Tutor notes

Session name & Code: Critical appraisal [PH ]

Session length: 120 minutes
Session format: LGW
Required resources:
• White board
• Internet connection and PC for search process on PubMed
Aims and objectives:
1. Critically Evaluate Research Methodologies: Students will critically assess the
methodology of different research studies.
2. Identify Bias and Confounding Factors: Students will identify potential biases
and understand their impact on research outcomes.
3. Apply Critical Appraisal Skills to Clinical Practice: Students will apply these
skills to enhance their decision-making in clinical settings.

Intended learning Outcomes: list of ILOS for the planned session e.g.
By the end of this session, students should be able to:
• assess the methodology of different research studies.
• identify potential biases and understand their impact on research outcomes
• apply these skills to enhance their decision-making in clinical settings.

Session layout and description:

• Introduction (10 minutes): Explanation of critical appraisal’s significance in
evidence-based practice, with a brief overview of CASP and its objectives.
• Section 1: Randomized Controlled Trials (25 minutes)
• Section 2: Cohort Studies (25 minutes)
• Section 3: Case-Control Studies (25 minutes)
• Section 4: Cross-Sectional Studies (25 minutes)
• Conclusion and Discussion (10 minutes): Review of key takeaways and
discussion on the application of CASP checklists in various research studies.

1
 Tutor notes

Section 1: Randomized Controlled Trials (RCTs) (25 minutes)

- Objective: Understand how to critically appraise an RCT using the CASP checklist.
- Activity: Review a sample RCT paper.
- Check for clarity in the research question, randomization, blinding, and outcome measurement.
- Discuss the application of the results to local settings.
- Analyze whether the study’s design addresses potential confounding factors.

CASP Checklist: For Randomised Controlled Trials (RCTs)

Manuscript title: [A brief diet intervention can reduce symptoms of depression in young adults –
A randomised controlled trial]

Section A Is the basic study design valid for a randomised controlled trial?

1. Did the study address a clearly formulated Yes No Can’t Tell

research question?

CONSIDER:
Was the study designed to assess the outcomes of an intervention?
Is the research question ‘formulated’ in terms of:
• Population studied yes
• Intervention given yes
• Comparator chosen yes
• Outcomes measured? yes

2. Was the assignment of participants to Yes No Can’t Tell

interventions randomised?
CONSIDER:
• How was randomisation carried out? Was the method appropriate? yes
• Was randomisation sufficient to eliminate systematic bias? yes
• Was the allocation sequence concealed from investigators and participants? Can’t tell

3. Were all participants who entered the study Yes No Can’t Tell
accounted for at its conclusion?
CONSIDER:
• Were losses to follow-up and exclusions after randomisation accounted for? yes
• Were participants analysed in the study groups to which they were randomised (intention-to-
treat analysis)? Yes (both methods)
• Was the study stopped early? If so, what was the reason? No

Section B Was the study methodologically sound?

4. (a) Were the participants ‘blind’ to Yes No Can’t Tell

intervention they were given?

2
 Tutor notes
(b) Were the investigators ‘blind’ to the Yes No Can’t Tell
intervention they were giving to
participants?
(c) Were the people assessing/analysing Yes No Can’t Tell
outcome/s ‘blinded’?
5. Were the study groups similar at the start of Yes No Can’t Tell
the randomised controlled trial?
CONSIDER:
• Were the baseline characteristics of each study group (e.g. age, sex, socio-economic group)
clearly set out? Yes
• Were there any differences between the study groups that could affect the outcome/s? No

6. Apart from the experimental intervention, Yes No Can’t Tell

did each study group receive the same level
of care (that is, were they treated equally)?
CONSIDER:
• Was there a clearly defined study protocol? Yes
• If any additional interventions were given (e.g. tests or treatments), were they similar between
the study groups? No additional interventions
• Were the follow-up intervals the same for each study group? Yes

Section C: What are the results?

7. Were the effects of intervention reported Yes No Can’t Tell

comprehensively?
CONSIDER:
• Was a power calculation undertaken? Yes
• What outcomes were measured, and were they clearly specified? Yes
• How were the results expressed? For binary outcomes, were relative and absolute effects
reported? Yes
• Were the results reported for each outcome in each study group at each follow-up interval?
Yes
• Was there any missing or incomplete data? Yes
• Was there differential drop-out between the study groups that could affect the results? No
• Were potential sources of bias identified? Yes
• Which statistical tests were used? Appropriate
• Were p values reported? Yes

8. Was the precision of the estimate of the Yes No Can’t Tell

intervention or treatment effect reported?

CONSIDER:
• Were confidence intervals (CIs) reported? No

9. Do the benefits of the experimental Yes No Can’t Tell

intervention outweigh the harms and costs?

CONSIDER:
• What was the size of the intervention or treatment effect? Cohen’s d = 0.65
• Were harms or unintended effects reported for each study group? No
• Was a cost-effectiveness analysis undertaken? (Cost-effectiveness analysis allows a
comparison to be made between different interventions used in the care of the same condition
or problem.) No

3
 Tutor notes

Section D: Will the results help locally?

10. Can the results be applied to your local Yes No Can’t Tell
population/in your context?

CONSIDER:
• Are the study participants similar to the people in your care? Yes
• Would any differences between your population and the study participants alter the outcomes
reported in the study? No
• Are the outcomes important to your population? Yes
• Are there any outcomes you would have wanted information on that have not been studied or
reported? No
• Are there any limitations of the study that would affect your decision? Small sample size

11. Would the experimental intervention Yes No Can’t Tell

provide greater value to the people in
your care than any of the existing
interventions?

CONSIDER:
• What resources are needed to introduce this intervention taking into account time, finances,
and skills development or training needs? Can’t tell
• Are you able to disinvest resources in one or more existing interventions in order to be able to
re-invest in the new intervention? Can’t tell

Section 2: Cohort Studies (25 minutes)

- Objective: Evaluate the validity of results from cohort studies.
- Activity: Engage with a cohort study example.
- Focus on recruitment methods, bias in exposure measurement, and outcome assessments.
- Consider the follow-up process and its adequacy in revealing effects.

4
 Tutor notes

CASP Checklist: For Cohort Studies

Manuscript title: Association between maternal multimorbidity and preterm birth, low birth
weight and small for gestational age: a prospective birth cohort study from the Japan
Environment and Children’s Study

Section A: Are the results valid?

1. Did the study address a clearly focused Yes No Can’t Tell

issue?
CONSIDER:
A question can be ‘focused’ in terms of
• the population studied Yes
• the risk factors studied Yes
• is it clear whether the study tried to detect a beneficial or harmful effect Yes
• the outcomes considered yes
2. Was the cohort recruited in an acceptable Yes No Can’t Tell
way?
CONSIDER:
• Look for selection bias which might compromise the generalisability of the findings:
• was the cohort representative of a defined population Yes
• was there something special about the cohort No
• was everybody included who should have been Yes
3. Was the exposure accurately measured to Yes No Can’t Tell
minimise bias?
CONSIDER:
Look for measurement or classification bias:
• did they use subjective or objective measurements both
• do the measurements truly reflect what you want them to (have they been validated) can’t tell
• were all the subjects classified into exposure groups using the same procedure yes
4. Was the outcome accurately measured to Yes No Can’t Tell
minimise bias?
CONSIDER:
Look for measurement or classification bias:

• did they use subjective or objective measurements objective

• do the measurements truly reflect what you want them to (have they been validated) Yes
• has a reliable system been established for detecting all the cases (for measuring disease
occurrence) Yes
• were the measurement methods similar in the different groups Yes
• were the subjects and/or the outcome assessor blinded to exposure (does this matter) No

5. (a) Have the authors identified all important Yes No Can’t Tell
confounding factors?
CONSIDER:
• list the ones you think might be important, and ones the author missed No

b) Have they taken account of the Yes No Can’t Tell

confounding factors in the design and/or
analysis?
CONSIDER:
• look for restriction in design, and techniques e.g. modelling, stratified-, regression-, or sensitivity
analysis to correct, control or adjust for confounding factors done

5
 Tutor notes

6. a) Was the follow up of subjects complete Yes No Can’t Tell

enough?
CONSIDER:
• the persons that are lost to follow-up may have different outcomes than those available for
assessment No loss of follow-up
• in an open or dynamic cohort, was there anything special about the outcome of the people leaving,
or the exposure of the people entering the cohort stable cohort
b) Was the follow up of subjects long Yes No Can’t Tell
enough?
CONSIDER:
• the good or bad effects should have had long enough to reveal themselves Yes

Section B: What are the results?

7. What are the results of this study? Yes No Can’t Tell

CONSIDER:
• what are the bottom line results
• have they reported the rate or the proportion between the exposed/unexposed, the ratio/rate
difference yes
• how strong is the association between exposure and outcome (RR) weak association
• what is the absolute risk reduction (ARR) not reported
8. How precise are the results? Yes No Can’t Tell

CONSIDER:
• look for the range of the confidence intervals, if given Narrow

9. Do you believe the results? Yes No Can’t Tell

CONSIDER:
• big effect is hard to ignore
• can it be due to bias, chance or confounding No
• are the design and methods of this study sufficiently flawed to make the results unreliable No
• Bradford Hills criteria (e.g. time sequence, dose-response gradient, biological plausibility,
consistency)

Section C: Will the results help locally?

10.Can the results be applied to the local Yes No Can’t Tell

population?
CONSIDER:
• Is a cohort study the appropriate method to answer this question Yes
• If the subjects covered in this study could be sufficiently different from your population to cause
concern No
• If your local setting is likely to differ much from that of the study No
• If you can quantify the local benefits and harms Yes

11.Do the results of this study fit with other Yes No Can’t Tell
available evidence?

12.What are the implications of this study for Yes No Can’t Tell
practice?

6
 Tutor notes

Section 3: Case-Control Studies (25 minutes)

- Objective: Assess the selection of cases and controls and measurement of exposure.
- Activity: Appraisal of a case-control study.
- Identify potential sources of bias and methods to minimize these issues.
- Evaluate the matching of cases and controls and the measures used to ensure accuracy.

CASP Checklist: For case control studies

Manuscript title: Aspirin Use and Risk of Pancreatic Ductal Adenocarcinoma: A Large Case
Control Study

Section A: Are the results of the study valid?

1. Did the study address a clearly focused issue? Yes No Can’t Tell
CONSIDER:
An issue can be ‘focused’ In terms of
• the population studied Yes
• whether the study tried to detect a beneficial or harmful effect Yes
• the risk factors studied Yes
2. Did the authors use an appropriate method Yes No Can’t Tell
to answer their question?
CONSIDER:
• is a case control study an appropriate way of answering the question under the circumstances Yes
• did it address the study question Yes
3. Were the cases recruited in an acceptable Yes No Can’t Tell
way?
CONSIDER:
We are looking for selection bias which might compromise validity of the findings
• are the cases defined precisely Yes
• were the cases representative of a defined population (geographically and/or temporally) Yes
• was there an established reliable system for selecting all the cases Yes
• are they incident or prevalent Incident
• is there something special about the cases No
• is the time frame of the study relevant to disease/exposure Yes
• was there a sufficient number of cases selected can’t tell
• was there a power calculation No
4. Were the controls selected in an acceptable Yes No Can’t Tell
way?
CONSIDER:
We are looking for selection bias which might compromise the generalisability of the findings

• were the controls representative of the defined population (geographically and/or temporally) Yes
• was there something special about the controls No
• was the non-response high, could non-respondents be different in any way Not reported
• are they matched, population based or randomly selected Matched hospital based
• was there a sufficient number of controls selected Can’t tell

7
 Tutor notes
5. Was the exposure accurately measured to Yes No Can’t Tell
minimise bias?
CONSIDER:
We are looking for measurement, recall or classification bias

• was the exposure clearly defined and accurately measured Yes

• did the authors use subjective or objective measurements Subjective
• do the measures truly reflect what they are supposed to measure (have they been validated) Yes
• were the measurement methods similar in the cases and controls Yes
• did the study incorporate blinding where feasible No
• is the temporal relation correct (does the exposure of interest precede the outcome) Yes

6. a) Aside from the exposure, did the Yes No Can’t Tell

groups have similar characteristics?
CONSIDER:
List the ones you think might be important, that the author may have missed NO missed factors
• genetic
• environmental
• socio-economic
6 b) Have the authors taken account of the Yes No Can’t Tell
potential confounding factors in the
design and/or in their analysis?
CONSIDER:
• restriction in design, and techniques e.g. modelling, stratified-, regression-, or sensitivity analysis
to correct, control or adjust for confounding factors Regression analysis and propensity score

Section B: What are the results?

7. Was the treatment effect large? Yes No Can’t Tell

CONSIDER:
• what are the bottom-line results No significant difference between two groups
• is the analysis appropriate to the design Yes
• how strong is the association between exposure and outcome (look at the odds ratio) weak
association
• are the results adjusted for confounding, and might confounding still explain the association No
association
• has adjustment made a big difference to the OR No
8. Was the estimate of the treatment effect Yes No Can’t Tell
precise?
CONSIDER:
• size of the p-value
• size of the confidence intervals Narrow
• have the authors considered all the important variables Yes
• how was the effect of subjects refusing to participate evaluated Not reported
9. Do you believe the results? Yes No Can’t Tell
CONSIDER:
• big effect is hard to ignore!
• can it be due to chance, bias, or confounding Yes
• are the design and methods of this study sufficiently flawed to make the results unreliable No
• consider Bradford Hills criteria (e.g. time sequence, does-response gradient, strength, biological
plausibility) No association

8
 Tutor notes

Section C: Will the results help locally?

10. Can the results be applied to your Yes No Can’t Tell

patients/the population of interest?
CONSIDER:
• the subjects covered in the study could be sufficiently different from your population to cause
concern No
• if your local setting is likely to differ much from that of the study No
• can you quantify the local benefits and harms Yes
11. Do the results of this study fit with other Yes No Can’t Tell
available evidence?
CONSIDER:
• all the available evidence from RCT’s Systematic Reviews, Cohort Studies, and Case Control Studies
as well, for consistency Yes

Section 4: Cross-Sectional Studies (25 minutes)

- Objective: Critique the methodology of cross-sectional studies.
- Activity: Critical analysis of a cross-sectional study.
- Examine the appropriateness of the sampling technique and the measurement’s accuracy.
- Discuss the implications of the study findings for practice.

CASP Checklist: For Descriptive/Cross-Sectional Studies

Manuscript title: A cross-sectional study on factors affecting the intention to quit smoking
among female call centre employees

Section A: Are the results valid?

1. Did the study address a clearly focused Yes No Can’t Tell

issue?
CONSIDER:
A question can be ‘focused’ in terms of
• the population studied yes
• the risk factors studied yes
• is it clear whether the study tried to detect a beneficial or harmful effect yes
• the outcomes considered yes
2. Did the authors use an appropriate method Yes No Can’t Tell
to answer their question?
CONSIDER:
• Is a descriptive/cross-sectional study an appropriate way of answering the question yes
• did it address the study question yes
3. Were the subjects recruited in an acceptable Yes No Can’t Tell
way?
CONSIDER:
We are looking for selection bias which might compromise the generalisability of the findings:

9
 Tutor notes
• Was the sample representative of a defined population No
• Was everybody included who should have been included Yes

4. Were the measures accurately measured to Yes No Can’t Tell

reduce bias?
CONSIDER:
Look for measurement or classification bias:

• did they use subjective or objective measurements objective

• do the measurements truly reflect what you want them to (have they been validated) yes

5. Were the data collected in a way that Yes No Can’t Tell

addressed the research issue?
CONSIDER:
• if the setting for data collection was justified yes
• if it is clear how data were collected (e.g., interview, questionnaire, chart review) yes
• if the researcher has justified the methods chosen yes
• if the researcher has made the methods explicit (e.g. for interview method, is there an indication of
how interviews were conducted?) no

6. Did the study have enough participants to Yes No Can’t Tell

minimise the play of chance?
CONSIDER:
• if the result is precise enough to make a decision No (wide CI)
• if there is a power calculation. This will estimate how many subjects are needed to produce a
reliable estimate of the measure(s) of interest. No
7. How are the results presented and what is Yes No Can’t Tell
the main result?
CONSIDER:
• if, for example, the results are presented as a proportion of people experiencing an outcome, such
as risks, or as a measurement, such as mean or median differences, or as survival curves and
hazards
• how large this size of result is and how meaningful it is
• how you would sum up the bottom-line result of the trial in one sentence
8. Was the data analysis sufficiently rigorous? Yes No Can’t Tell

CONSIDER:
• if there is an in-depth description of the analysis process Yes
• if sufficient data are presented to support the findings yes

9. Is there a clear statement of findings? Yes No Can’t Tell

CONSIDER:
• if the findings are explicit Yes
• if there is adequate discussion of the evidence both for and against the researchers’ arguments yes
• if the researchers have discussed the credibility of their findings yes
• if the findings are discussed in relation to the original research questions yes
10.Can the results be applied to the local Yes No Can’t Tell
population?

CONSIDER:
• the subjects covered in the study could be sufficiently different from your population to cause
concern. No
• your local setting is likely to differ much from that of the study No

10
 Tutor notes
11.How valuable is the research? Yes No Can’t Tell

CONSIDER:
• one descriptive/cross-sectional study rarely provides sufficiently robust evidence to recommend
changes to clinical practice or within health policy decision making
• if the researcher discusses the contribution the study makes to existing knowledge (e.g., do they
consider the findings in relation to current practice or policy, or relevant research-based
literature?) Yes
• if the researchers have discussed whether or how the findings can be transferred to other
populations Yes

Conclusion and Discussion (10 minutes)

- Review key takeaways from each study type.
- Open floor for questions and clarifications on applying CASP checklists.
- Discuss the importance of combining evidence from various types of studies to strengthen
recommendations.

11