Methodology Checklist 1: Systematic Reviews and Meta-analyses
SIGN Notes for completion of checklist
SIGN gratefully acknowledges the permission received from the authors of the AMSTAR tool to base this checklist on their work:
Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C,. et al. Development of AMSTAR: a measurement tool to
assess the methodological quality of systematic reviews. BMC Medical Research Methodology 2007, 7:10 doi:10.1186/1471-
2288-7-10. Available from http://www.biomedcentral.com/1471-2288/7/10 [cited 10 Sep 2012]
Must refers to a statement that has to be fulfilled for the question to receive a yes answer. Should
statements are a mark of quality but not a necessity for a yes answer. These should be used to
assess the overall quality of the paper.
Section 1: Internal validity
In a well conducted systematic Notes
review:
1.1 The research question The PICO must be clear in the paper even if not directly
is clearly defined and referred to. The research question and inclusion criteria
the should be established before the review is conducted.
inclusion/ exclusion
criteria must be listed in
the paper.
1.2 A comprehensive At least two relevant electronic sources must be searched. The
literature search is report must list the databases used (e.g., Central, EMBASE,
carried out. and MEDLINE). (Cochrane register/Central counts as two
sources; a grey literature search counts as supplementary).
(PubMed and MEDLINE count as one database.)
Key words and/or MESH terms must be stated and where
feasible the search strategy should be provided. Dates for the
search should be provided.
The paragraph above is the minimum requirement.
All searches should be supplemented by consulting current
contents, reviews, textbooks, specialized registers, or/and
experts in the particular field of study, and by reviewing the
references in the studies found.
The paragraph above is a quality criteria which affects the
overall rating of the review.
Notes
This criterion will not apply in the case of prospective meta-
analysis - this is where meta-analysis is based on pre-selected
studies identified for inclusion before the results of those
studies are known. Such reports must state that they are
prospective.
1.3 At least two people At least two people should select papers. There should be a
should have selected consensus process to resolve any differences
studies.
1.4 At least two people At least two people should extract data and should report that a
should have extracted consensus was agreed. One person checking the others data
data. extraction is accurate is acceptable.
1.5 The status of publication The authors should state that they searched for reports
was not used as an regardless of their publication status. The authors should state
inclusion criterion. whether or not they excluded any reports (from the systematic
review), based on their publication status.
If review indicates that there was a search for “grey literature”
or “unpublished literature,” indicate “yes.” SIGLE database,
dissertations, conference proceedings, and trial registries are
all considered grey for this purpose. If searching a source that
contains both grey and non-grey, must specify that they were
searching for grey/unpublished lit.
1.6 The excluded studies Limiting the excluded studies to references is acceptable.
are listed.
1.7 The relevant In an aggregated form such as a table, data from the original
characteristics of the studies should be provided on the participants, interventions
included studies are and outcomes. The ranges of characteristics in all the included
provided. studies e.g., age, race, sex, relevant socioeconomic data,
disease status, duration, severity, or other diseases should be
reported. (Note that a format other than a table is acceptable,
as long as the information noted here is provided).
Absence of this will make it impossible to form guideline
recommendations. Mark as (-) original papers would need to be
examined.
1.8 The scientific quality of It can include use of a quality scoring tool or checklist, e.g. risk
the included studies of bias assessment, or a description of quality items, with some
was assessed and kind of result for EACH study (“low” or “high” is fine, as long as
documented it is clear which studies scored “low” and which scored “high”; a
summary score/range for all studies is not acceptable.
Absence of this will make it impossible to form guideline
recommendations. Mark as (-)
1.9 Was the scientific Examples include sensitivity analysis based on study quality,
quality of the included exclusion of poor quality studies, and statements such as ‘the
studies used results should be interpreted with caution due to poor quality of
appropriately? included studies’
The results of the methodological rigor and scientific quality
should be considered in the analysis and the conclusions of
the review, and explicitly stated in formulating
recommendations.
Cannot score “yes” for this question if scored “no” for question
1.8.
1.10 Appropriate methods Studies that are very clinically heterogeneous should not be
are used to combine the combined in a meta-analysis.
individual study findings.
Look at the forest plot–do the results look similar across the
studies?
For the pooled result a test should be done to assess statistical
heterogeneity i.e. Chi-squared (2) test for homogeneity and/or
I2 test for inconsistency.
If significant heterogeneity is apparent the authors should have
explored possible explanations using methods such as
sensitivity analysis or meta-regression. A random effects
analysis may be used to take account of between-study
variation but is not a ‘fix’ for heterogeneity.
Planned subgroup analyses should be pre-specified and limited
in number because conducting many subgroup analyses
increases the probability of obtaining a statistically significant
result by chance. Conclusions based on post-hoc subgroup
analyses must be interpreted with caution.
Cannot score “yes” for this question if scored “no” for question
1.8.
1.11 The likelihood of The possibility of publication bias should be assessed where
publication bias was possible, commonly done by visual inspection of a funnel plot
assessed appropriately together with a statistical test for asymmetry (e.g., Egger
regression test) although other statistical and modelling
approaches may be reported.
Absence of a funnel plot doesn’t mean the likelihood of
publication bias was not assessed appropriately (there are
other methods); 10 studies is just a ball-park minimum number
for a funnel plot and a plot is of little use when there are few
studies.
1.12 Conflicts of interest are Potential sources of support should be clearly acknowledged in
declared. both the systematic review and the included studies.
SECTION 2: OVERALL ASSESSMENT OF THE STUDY
2.1 What is your overall Rate the overall methodological quality of the study, using the
assessment of the following as a guide:
methodological quality High quality (++): Majority of criteria met. Little or no risk of
of this review? bias..
Acceptable (+): Most criteria met. Some flaws in the study with
an associated risk of bias.
Low quality (-): Either most criteria not met, or significant flaws
relating to key aspects of study design.
Reject (0): Poor quality study with significant flaws. Wrong
study type. Not relevant to guideline.
