Focus on Statistics

Meta-analysis: a critical appraisal of

the methodology, benefits and drawbacks
■■ Bad studies may be included
ABSTRACT ■■ The data summarized may not be homogeneous
Meta-analysis has become an integral part of evidence-based decision-making ■■ Grouping different causal factors may lead to
processes and is being increasingly used in medical and non-medical disciplines. meaningless estimates of effects
Aggregate data or summary statistics continue to be the mainstay of meta- ■■ The theory-directed approach may obscure discrepancies
analysis and are used by many professional societies to support clinical practice (Eysenck, 1994).
guidelines. Meta-analyses synthesize the summary statistics from independent This article covers a range of issues relating to the meta-
trials by pooling them to estimate the underlying common effect size. The results analysis, and critically investigates its drawbacks and
represent the highest level of evidence but only if the chosen studies are of high benefits.
quality and the selection criteria are fully satisfied. It is important to address the
issues of defining an explicit and relevant question, exhaustively searching for
the totality of evidence, meticulous and unbiased data transfer or extraction,
Prelude to meta-analyses: systematic reviews
assessment of between study heterogeneity and the use of appropriate statistical A systematic review is the process of searching, gathering
methods for estimating summary effect measures. This article reviews the and investigating the relevant literature on a specific topic
methodology, benefits and drawbacks of performing a meta-analysis. of interest. Ideally the results of meta-analyses should
be rigorous, comprehensive, transparent, free from bias

I
and reproducible. Khan et al (2003) describe five steps in
n the 21st century, scientists and policy makers rely on an performing a systematic review to ensure its objectivity:
evidence-based decision-making process to guide them 1. Framing questions for a review
towards early interventions, better treatment methods 2. Identifying relevant work
and structured guidelines which work efficiently and 3. Assessing the quality of studies
reliably to produce the best possible outcomes. The 4. Summarizing the evidence
validity of evidence, and hence the decisions based on this, 5. Interpreting the findings.
relies on the quality of the data, the methodology used to To ensure that the evidence is of highest quality, various
extract them, the robustness and totality of the evidence researchers tried to devise processes, criteria and protocols
and use of relevant methods to analyse them. Hence, to prevent biases and design flaws to improve the quality of
decision makers need to be aware of factors that impact reviews and evidence. The Quality of Reporting of Meta-
the quality of evidence along with any shortcomings in analyses (QUOROM) addresses standards for improving
the process of gathering and processing such evidence. The the quality of meta-analysis of clinical randomized
use of meta-analysis as part of systematic review enables controlled trials (Moher et al, 1999). The Consolidated
inclusion of analysis of quantitative data from independent Standards of Reporting Trials (CONSORT) (Moher et
trials in decision-making processes (Khan et al, 2016). al, 2010) encompass various initiatives to deal with the
However, problems arise in meta-analysis. These include: problems arising from inadequate reporting of randomized
■■ Regressions are often non-linear controlled trials. The Preferred Reporting Items for
■■ Effects are often multivariate rather than univariate Systematic Reviews and Meta-Analyses (PRISMA) (Moher
■■ Coverage can be restricted et al, 2009) is an evidence-based minimum set of items
for reporting in systematic review and meta-analysis.
The authors of PRISMA later introduced the PRISMA
Professor Shahjahan Khan, Professor of Statistics, School of
Agricultural, Computing and Environmental Sciences, protocols (Moher et al, 2015; Shamseer et al, 2015). The
International Centre for Applied Climate Sciences and Meta-analysis Of Observational Studies in Epidemiology
Centre for Health Sciences Research, University of (MOOSE) was proposed by Stroup et al (2000) which
Southern Queensland, Toowoomba, Queensland, Australia contains a checklist of specifications for reporting meta-
Ms Breda Memon, Research and Clinical Nurse, South East analysis of observational studies.
Queensland Surgery and Sunnybank Obesity Centre,
McCullough Centre, Sunnybank, Queensland, Australia
Meta-analysis
© 2019 MA Healthcare Ltd

Professor Muhammed A Memon, Consultant Surgeon, Glass (1976) introduce meta-analysis as a statistical
Mayne Medical School, School of Medicine, University of
Queensland, Brisbane, Queensland, Australia procedure to re-analyse the published statistical results from
Correspondence to: Professor MA Memon
a large number of independent studies on a specific topic
(mmemon@yahoo.com) for the purpose of integrating the findings in the context
of educational research. He felt that when faced with an

636 British Journal of Hospital Medicine, November 2019, Vol 80, No 11

Downloaded from magonlinelibrary.com by 131.172.036.029 on November 16, 2019.
 Focus on Statistics

abundance of information, the goal should be to extract the

quantifiable information, group them in an orderly fashion S1 S2 S3 S4 S5 S6 S7
according to categories, pooling the data and summarizing Random sequence generation
these results. Meta-analysis is described as ‘a statistical analysis + + + + + + + (selection bias)
that combines or pools the results of several independent
Allocation concealment
clinical trials considered by the analyst to be “combinable” + + + + + + + (selection bias)
in which the primary aim is attaining an estimate of average
effect size attributable to a certain intervention presented in – + + + + + Blinding of participants and
the same metric’ (Egger and Smith, 1997; Smith and Egger,
? personnel (performance bias)
1998; Huedo-Medina et al, 2006). Blinding of outcome assessment
? + + + + + ? (detection bias)
Appraisal of meta-analysis
Incomplete outcome data
Quality of studies included + – – + + – + (attrition bias)
Quality assessment of a study is necessary in order to
prevent misleading results based on invalid or poor quality + + +
studies. Quality (or validity) involves some measure of
? ? ? ? Selective reporting (reporting bias)

the methodological strength of the relevant study, or how

able it is, through its design and its conduct, to prevent + + + ? ? + + Other bias
systematic errors, or bias. Pooling results from low levels
of evidence, e.g. retrospective trials, with those with a Key: + Low risk of bias ? Unclear risk of bias – High risk of bias
high level of evidence, e.g. randomized controlled trials,
reduces the quality of the synthesized results and may lead
to invalid conclusions. Figure 1. The Cochrane Risk of Bias Tool representing various domains. S1–S7
represent various randomized controlled trials.
Several methods have been described to evaluate trial
quality. The Jadad scale or Oxford Quality Scoring System
(Jadad et al, 1996), based on reporting of randomization, For the computation of any confidence interval for an
blinding and withdrawals, is the most widely used unknown population effect size, the point estimate of effect
assessment tool because of its simplicity. Another tool is the size for each of the individual studies, along with their
Cochrane Risk of Bias Tool, which is based on a number of standard deviation and sample size, is essential. Also, the
domains such as selection bias, performance bias, detection sampling distribution of the estimator of the population
bias, attrition bias, reporting bias and other bias (Figure 1) effect size must be identifiable in order to be able to
(Higgins et al, 2011). The Newcastle–Ottawa scale (Stang, determine the critical value of the underlying statistic at a
2010) (Figure 2) evaluates the quality of observational
studies based on three broad categories: the selection of Outcome
the study groups, the comparability of the groups, and the Author/ Selection Comparability assessment
ascertainment of either exposure or outcome of interest for Year 1 2 3 4 5 6 7 Total quality
case control or cohort studies respectively. A1/1990 * * * * * * * *******
Other tools for non-randomized study assessment
include the Risk Of Bias In Non-randomized Studies- B2/1995 * * * * * *****
of Interventions (ROBINS-I) assessment tool (Sterne et C3/2000 * * * * * * * *******
al, 2016). It outlines seven domains where biases might
occur: two in the ‘pre-intervention’ phase, one in the ‘at D4/2005 * * * * * * ******
intervention’ phase and four in the ‘post-intervention’ E5/2010 * * * * * * ******
phase. It is beyond the scope of this article to provide a
detailed account of these tools. F6/2015 * * * * * * ******
Selection: 1) Is the case definition described? 2) Was the sample truly representative of the
Reporting variability of summary statistics total population? 3) How was the ascertainment of exposure done? (Each affirmative answer
gets one star)
and how to combine the data Comparability: 4) Did the study have no difference between study 1 and study 2 groups?
The summary statistics or aggregate data of the outcome The main factors taken into account while calculating this were various outcomes, e.g. prior
variables are reported in the individual studies using treatment, pain score, complications, mentioned. 5) One more star was given if the following
factors were comparable in the two groups; age, gender of patient, operative time recorded. (If
different units of measurement. Depending on the type both were affirmative then two stars, even if one or more of the above mentioned criteria were
of outcome variable, the summary statistics could be absent. However, no stars were given if the groups differed entirely.)
© 2019 MA Healthcare Ltd

mean and standard deviation, or correlation coefficient Outcome assessment: 6) Clear assessment of outcomes via record linkage. 7) Adequacy of the
cohort follow ups – whether the follow ups were completed or less than 20% of patients were
(measuring the relationship of two quantitative outcome lost to follow up.
variables) for continuous outcome variables, and odds
ratio, risk ratio and risk difference for categorical outcome Figure 2. Newcastle–Ottawa scale for assessing the quality of non-randomized
variables, along with the sample size. studies. A1–F6 represent authors of these studies.

British Journal of Hospital Medicine, November 2019, Vol 80, No 11 637
Downloaded from magonlinelibrary.com by 131.172.036.029 on November 16, 2019.
Focus on Statistics

Meta-regression is an extension to subgroup plays a significant role in determining the estimate of the
effect size. It is therefore essential to check if heterogeneity
analyses that allows the effect of continuous is present in the data (Sauerland and Seiler, 2005; Ng et al,
as well as categorical characteristics 2006). If the between-studies variation is not significant,
to be investigated.  the meta-analysis becomes simple and straightforward.
However, in many cases the studies are heterogeneous and
predetermined confidence level. Such a critical value and therefore meta-analysis must address this fact in computing
the standard error of the estimator are used to calculate the the pooled effect size and the confidence interval. Some of
margin of error for the confidence interval. the commonly used methods to overcome the heterogeneity
If the raw (individual patient) data from all selected are discussed below. Unfortunately, not all of them are
studies are available, then one could analyse the data by equally effective or provide a real remedy for the problem.
using mega-analysis methods. Unfortunately, the sample
effect size (in individual studies) is a random variable as it Assessing heterogeneity
differs from study to study. The study-specific values of the The presence of heterogeneity among the effect size measure
estimated effect size are not only different but may have is assessed by performing the Cochrane’s Q-statistic or I2
opposing results, producing contradictory evidence. In the statistic (Huedo-Medina et al, 2006). The main problem
face of conflicting evidence from different primary studies, with the Q-statistic is that its value increases as the number
the challenge is to reconcile the results to come up with a of studies in a meta-analysis grows larger. While it is useful
valid estimated common effect size. to detect heterogeneity and inform on the degree of its
In many cases, different studies use different statistical significance, it is unable to describe the extent
measurements such as the median, the minimum and of the presence of true heterogeneity (Huedo-Medina et
maximum values and/or the interquartile range instead of al, 2006).
reporting mean and standard deviation of the quantitative An alternative method for assessing heterogeneity is
outcome variables. Therefore, the effect size of all selected the I2 statistic used in conjunction with its confidence
studies must be converted to the same unit of measurement intervals (Higgins and Thompson, 2002). The I2 statistic
before pooling them. Wan et al (2014), using Hozo’s presents as a percentage of the total variability that can
methodology and improving upon it, provided details on be attributed to true heterogeneity within a set of effect
estimating the sample mean and standard deviation from sizes. Therefore, an I2 statistic equalling 0% suggests that
the sample size, median, range and/or interquartile range. there is no between-study variability and that all variation
observed is a result of sampling error. Conversely, when I2
Inverse variance method and redistribution of weights approaches 100%, it suggests that the observed variation is
The main objective of any meta-analysis is to pool statistics the result of between-study variability rather than sampling
from independent studies with a view to synthesizing them error. A rough guide for the interpretation of I2 statistic is:
to calculate an estimate of the common effect size. There 0–40% might not be important; 30–60% may represent
are different ways of pooling statistics, depending on the moderate heterogeneity; 50–90% may represent substantial
type of weight used for individual studies in computing heterogeneity and 75–100% is considerable heterogeneity
the pooled estimate. (Huedo-Medina et al, 2006; Higgins and Green, 2011).
Conventionally, the inverse variance weights are
commonly used in meta-analysis (Borenstein et al, 2009). Subgroup analysis
This way the weights are redistributed among the studies One way to minimize the impact of heterogeneity in a
depending on the extent of spread of the individual studies. meta-analysis is to group the studies based on the value of
It is well known that the sample variance (a measure of the the individual study variance (Figures 3a and b) (Higgins
spread or the differences between the observed value of and Green, 2011). Studies with similar values of the
the relevant outcome variable and their arithmetic mean) variance are sub-grouped and separate meta-analyses are
is inversely related to the sample size (n), and hence larger conducted on each of the subgroups along with the overall
studies (with higher sample size) should receive higher meta-analysis. However, the problem remains with the
weights under the principle of inverse variance weight estimation of the common effect size of all studies based
than the smaller studies. But in many cases this principle on combining the results from all the subgroups because
may lead to allocating higher than appropriate weight of significant heterogeneity among the subgroups.
to the lower quality smaller studies and vice versa. The
redistribution of weights varies significantly under various Meta-regression
commonly used statistical models especially if there is Meta-regression is a technique for performing a regression
© 2019 MA Healthcare Ltd

significant heterogeneity among the studies. analysis to assess the relationship between the treatment
effects and the study characteristics of interest (e.g. suture vs
Heterogeneity prosthesis) or factors concerning the execution of the study
Since meta-analyses are based on the summary statistics (e.g. allocation sequence concealment) (Figures 3a and b)
of individual studies, the between-studies variation often (Thompson and Higgins, 2002; van Houwelingen et al,

638 British Journal of Hospital Medicine, November 2019, Vol 80, No 11

Downloaded from magonlinelibrary.com by 131.172.036.029 on November 16, 2019.
 Focus on Statistics

a Suture Mesh Total complications

Study Case No Case No OR L U Weight
2 34 0.49 0.04 5.61 0.06

2.01 1.507 1.005 0.502 0

Frantzides et al (2002) 1 35
0 50 1 0.02 51.39 0.02

Standard error
Granderath et al (2005) 0 50
Oelschlanger et al (2011) 10 47 12 39 0.69 0.27 1.77 0.38
Watson et al (2014) 8 35 8 75 2.14 0.74 6.18 0.3
Oor et al (2018) 6 30 7 29 0.83 0.25 2.76 0.23

Pooled OR 25 197 29 227 1 0.49 2.05 0.1 2.0 4.0 6.0 10.0 -6 -4 -2 0 2 4 6
Test for heterogeneity Q = 3.01 P = 0.56 I2 % = 0 Favours suture Favours mesh Log odds ratio
Test for overall effect Z = 0.028 P = 0.99

Suture Mesh Total complications

b
Study Case No Case No OR L U Weight

2.01 1.507 1.005 0.502 0

Frantzides et al (2002) 1 35 2 34 0.49 0.04 5.61 0.1

Standard error
Granderath et al (2005) 0 50 0 50 1 0.02 51.39 0.04
Watson et al (2014) 8 35 5 37 1.69 0.5 5.67 0.43
Oor et al (2018) 6 30 7 29 0.83 0.25 2.76 0.43

Pooled OR 15 150 14 150 1.07 0.49 2.35 -6 -4 -2 0 2 4 6

P = 0.77 I2 % = 0 0.1 2.0 4.0 6.0 10.0
Test for heterogeneity Q = 1.13 Log odds ratio
Test for overall effect Z = 0.28 P = 0.8 Favours suture Favours mesh

Figure 3. Forest and funnel plots. In these graphs which were created using random effects model, squares indicate point estimates of treatment
effect with the size of the squares representing the weight attribute to each study. The horizontal lines represent 95% confidence interval for odds
ration. The pooled estimate for complication rate (is the pooled odds ratio obtained by combining all odds ratios of the five studies using the inverse
variance weighted method) and is represented by the diamond and the size of the diamond depicts the 95% confidence interval. Funnel plot showing
no outliers (dots within the triangle) and therefore, no publication bias. a. Analysis based on a total sample of meshes (i.e. both absorbable and non-
absorbable meshes). b. Subgroup analysis based only on non-absorbable meshes. No. = total number of cases; Case = number of patients with
complication; OR = odd ratio; L = lower limit of confidence interval; U = upper limit of confidence interval.

2002; Greenland and O’Rourke, 2008). Meta-regression Fixed effects model

is an extension to subgroup analyses that allows the effect The fixed effects model assumes that there is a common
of continuous as well as categorical characteristics to be unknown true effect size for all the studies under
investigated, and in principle allows the effects of multiple investigation (Borenstein et al, 2009).
factors to be investigated simultaneously (although this is
rarely possible because there are inadequate numbers of Random effects model
studies) (Greenland and O’Rourke, 2008). Meta-regression This model is widely used to handle heterogeneity. It
should generally not be considered when there are fewer assumes that the true effect size is not identical for every
than ten studies in a meta-analysis. study. However, independent studies have enough in
common to justify the synthesis of results to produce an
Sensitivity analysis ‘average effect size’. The random effects model redistributes
Sensitivity analysis is often used to check the impact on the the weights to the individual studies in computing the
result of the estimate of the common effect size as a result of pooled estimate of the common effect size using two
the inclusion of one particular study or a group of similar sources of variation, i.e. within-study or random error
studies in a meta-analysis (Stroup et al, 2000; Sauerland and between-study or variation of the true effect size
and Seiler, 2005). Studies with significantly higher variance (Borenstein et al, 2009). Under the random effects model,
than others could be used in the sensitivity analysis to see larger studies normally receive smaller weights. Thus, the
how the results are impacted by these studies. This is not random effects model method of synthesizing the common
a solution to the heterogeneity problem, but it provides effect size takes away weights from the larger studies and
some useful insight into the problem and how individual redistributes them to smaller studies.
study effect size impacts on the meta-analysis. The DerSimonian and Laird approach is commonly used
for a random effects meta-analysis. However, it may lead to
Statistical models for meta-analyses too many statistically significant results when the number
Different statistical models are used for meta-analysis of studies is small and there is moderate or substantial
under different conditions. The main difference among the heterogeneity. An alternative method described by Hartung
© 2019 MA Healthcare Ltd

models is the way they allocate the inverse variance weights and Knapp and by Sidik and Jonkman (IntHout et al,
to the individual studies. The objective of redistribution 2014) is claimed to be simple and robust especially when
of weights among the studies, under different statistical there is heterogeneity and the number of studies in the
models, is to find a more precise estimator of the common meta-analysis is small. It is beyond the scope of this article
effect size to achieve a shorter confidence interval. to provide a detailed account of these methods.

British Journal of Hospital Medicine, November 2019, Vol 80, No 11 639
Downloaded from magonlinelibrary.com by 131.172.036.029 on November 16, 2019.
Focus on Statistics

Inverse variance heterogeneity model Benefits and drawbacks of meta-analyses

Doi et al (2015a) introduced the inverse variance Benefits
heterogeneity model. It emphasizes that the fixed effects Meta-analysis based on well-conducted randomized
model based estimator variance can be made closer to the controlled trials provides the highest level of objective
observed variance by modelling over-dispersion through evidence by controlling extraneous variation and biases. The
a quasi-likelihood approach. This implies that the meta- selection and implementation of the correct statistical model
analysis is performed under a fixed effects assumption produces highest quality of evidence. The results are accurate
and the variance of the estimator of the common effect if the underlying model assumptions are met and there is no
size is inflated to account for the heterogeneity. This has selection bias and no error in the extraction of data.
the advantage of being based purely on the variance- Meta-analysis can combine quantitative summary
to-mean relationship, rather than on distributional statistics of individual studies to estimate the common
assumptions, with variance appropriately inflated using effect size even if the results of the individual studies are
a scale parameter (Kulinskaya and Olkin, 2014; Doi et inconclusive, and conflicting. Meta-analyses provide more
al, 2015b). statistical power because of their increased sample size,
leading to precise and reliable results. Meta-analysis can be
Publication and reporting bias in meta-analysis performed to estimate the common effect size for a subset
Publication bias is a serious problem in meta-analysis. of the selected studies (subgroup analysis).
It arises because studies with negative or non-significant
effects are not normally submitted or accepted for Drawbacks
publication in professional journals (Duval and Tweedie, The validity of the results of meta-analysis depend on the
2000; Sauerland and Seiler, 2005; Higgins and Green, quality and the design of the trials included in the synthesis,
2011). This phenomenon impacts on the ultimate results of presence of publication bias and presence of heterogeneity.
the meta-analysis, sometimes without realizing the extent Meta-analysis cannot be used if the measurement of
of the exclusion and their potential impact. outcome variables is not similar for all studies.
The funnel plot (Figure 3) is used to assess the The validity of forest plots representing confidence
publication bias in a meta-analysis. It is used primarily as intervals depends on the correct identification of the
a visual aid for detecting bias or systematic heterogeneity. sampling distribution of the effect size estimator which
A funnel plot is a scatterplot of treatment effect against enables choosing the correct critical value in calculating
a measure of study size such as the estimated standard the margin of error.
error or sample size of each of the studies. Asymmetry in Selection of the wrong model for heterogeneity will
funnel plots indicate publication bias in meta-analysis, result in misleading results.
but the shape of the plot in the absence of bias depends In the presence of significant publication bias or
on the choice of axes (Sterne and Egger, 2001). Lastly, reporting anomalies, the results of meta-analysis will be
most articles are published in English language journals inaccurate, unreliable and invalid.
and therefore publications in other languages are excluded
from systematic review or meta-analysis. This not only Conclusions
introduces language bias but may also exclude some of As the practice of evidence-based decision-making
the evidence, leading to erroneous conclusions (Morrison continues to grow, it is important that everyone involved
et al, 2012). in conducting meta-analysis follows Khan et al’s (2003)
five steps to ensure its objectivity. Meta-analysis permits
Presentation and reporting of meta-analyses the detection of statistically significant differences among
The usual way to present the results of meta-analysis is study groups that may not have been possible in individual
to show the confidence intervals of individual studies reports because of their small sample size or underpowered
and the combined meta-analysis on the same graph in trials. However, the quality of the results produced by a
the form of a forest plot (Higgins and Thompson, 2002; meta-analysis will never be superior to the quality of the
Petrie and Sabin, 2009). The middle of the confidence statistics reported in the individual studies, which again
interval of the individual studies is marked by dark is directly dependent on the design of the study. Meta-
squares and the size or area of the associated squares analysis can provide much needed high-quality quantitative
represents the level of weight of the study. For the meta- evidence for making appropriate decisions if the underlying
analysis of the common effect size, the confidence interval processes, protocols and methods are properly and strictly
is represented by a diamond. The horizontal edges of the observed. Moreover, every step in any meta-analysis must
diamond represent the limits of the confidence interval. be scrutinized for potential bias, from the formulation of
© 2019 MA Healthcare Ltd

The relative location of the diamond with respect to the research question to the interpretation and discussion
the no-effect vertical line indicates which intervention is of the results, to ensure the quality and value of the final
favoured by the data. If appropriate, subgroup analyses product (Bernard, 2014).  BJHM
are also included in the forest plot along with the
combined meta-analysis. Conflict of interest: none.

640 British Journal of Hospital Medicine, November 2019, Vol 80, No 11

Downloaded from magonlinelibrary.com by 131.172.036.029 on November 16, 2019.
 Focus on Statistics

Bernard RM. Things I have learned about meta-analysis since 1990.

Reducing bias in search of ‘the big picture’. Can J Learn Technol. KEY POINTS
2014;40:1–17.
Borenstein M, Hedges LV, Higgins JPT, Rothstein RH. 2009. ■■ Meta-analysis is a systematic review that uses quantitative methods to
Introduction of Meta-analysis. Chichester: John Wiley & Sons summarize the results. It allows a re-examination of treatment effect of
Ltd:421. several studies and provides a single, overall measure of the treatment effect.
Doi SA, Barendregt JJ, Khan S, Thalib L, Williams GM. Advances
in the meta-analysis of heterogeneous clinical trials I: the inverse ■■ Meta-analysis is the aggregation of information (from several studies) leading
variance heterogeneity model. Contemp Clin Trials. 2015a to a higher statistical power and more robust point estimate when compared
Nov;45(Pt A):130–138. https://doi.org/10.1016/j.cct.2015.05.009 to any individual study.
Doi SAR, Barendregt JJ, Khan S, Thalib L, Williams GM. Simulation
comparison of the quality effects and random effects methods of ■■ Meta-analysis represents the highest level (level 1) of evidence among the
meta-analysis. Epidemiology. 2015b Jul;26(4):e42–e44. https://doi. hierarchy of evidence for evidence-based research.
org/10.1097/EDE.0000000000000289
■■ Meta-analysis results can be generalized to a large population (in the vast
Duval S, Tweedie R. Trim and fill: A simple funnel-plot-based method
of testing and adjusting for publication bias in meta-analysis. majority of cases).
Biometrics. 2000 Jun;56(2):455–463. https://doi.org/10.1111/ ■■ The validity of evidence produced by meta-analyses, and hence the decisions
j.0006-341X.2000.00455.x
based on them, are reliant on the quality of the data, the methodology used to
Egger M, Smith GD. Meta-analysis: potentials and promise. BMJ.
1997 Nov 22;315(7119):1371–1374. https://doi.org/10.1136/ extract them, the robustness and totality of the evidence and relevant methods
bmj.315.7119.1371 to analyse them.
Eysenck HJ. Systematic Reviews: meta-analysis and its problems.
BMJ. 1994 Sep 24;309(6957):789–792. https://doi.org/10.1136/
bmj.309.6957.789 Elaboration: updated guidelines for reporting parallel group
Glass G. Primary, secondary, and meta-analysis of randomised trials. J Clin Epidemiol. 2010 Aug;63(8):e1–e37.
research. Educ Res. 1976 Nov;5(10):3–8. https://doi. https://doi.org/10.1016/j.jclinepi.2010.03.004
org/10.3102/0013189X005010003 Moher D, Shamseer L, Clarke M et al; PRISMA-P Group. Preferred
Greenland S, O’Rourke K. 2008. Meta-analysis. In: Modern reporting items for systematic review and meta-analysis protocols
Epidemiology. Rothman KJ, Greenland S, Lash TL. 3rd edn. (PRISMA-P) 2015 statement. Syst Rev. 2015 Dec;4(1):1. https://
Wolters Kluwer/Lippincott Williams & Wilkins: Philadelphia: doi.org/10.1186/2046-4053-4-1
652–682. Morrison A, Polisena J, Husereau D et al. The effect of English-
Higgins JPT, Altman DG, Gøtzsche PC et al; Cochrane Bias Methods language restriction on systematic review-based meta-analyses:
Group; Cochrane Statistical Methods Group. The Cochrane a systematic review of empirical studies. Int J Technol Assess
Collaboration’s tool for assessing risk of bias in randomised trials. Health Care. 2012 Apr;28(2):138–144. https://doi.org/10.1017/
BMJ. 2011 Oct 18;343:d5928. https://doi.org/10.1136/bmj.d5928 S0266462312000086
Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta- Ng TT, McGory ML, Ko CY, Maggard MA. Meta-analysis in surgery.
analysis. Stat Med. 2002 Jun 15;21(11):1539–1558. https://doi. Arch Surg. 2006 Nov 01;141(11):1125–1130, discussion 1131.
org/10.1002/sim.1186 https://doi.org/10.1001/archsurg.141.11.1125
Higgins JP, Green S. 2011. Cochrane Handbook for Systematic Petrie A, Sabin C. 2009. Medical Statistics at a Glance. Oxford: Wiley-
Reviews of Interventions. Version 5.1.0. (accessed 6 September Blackwell: 180.
2019) https://handbook-5-1.cochrane.org/ Sauerland S, Seiler CM. Role of systematic reviews and meta-analysis
Huedo-Medina TB, Sánchez-Meca J, Marín-Martínez F, Botella in evidence-based medicine. World J Surg. 2005 May;29(5):582–
J. Assessing heterogeneity in meta-analysis: Q statistic or I² 587. https://doi.org/10.1007/s00268-005-7917-7
index? Psychol Methods. 2006;11(2):193–206. https://doi. Shamseer L, Moher D, Clarke M et al; PRISMA-P Group. Preferred
org/10.1037/1082-989X.11.2.193 reporting items for systematic review and meta-analysis protocols
IntHout J, Ioannidis JPA, Borm GF. The Hartung-Knapp- (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015 Jan
Sidik-Jonkman method for random effects meta-analysis is 02;349 jan02 1:g7647. https://doi.org/10.1136/bmj.g7647
straightforward and considerably outperforms the standard Smith GD, Egger M. Meta-analysis: unresolved issues and future
DerSimonian-Laird method. BMC Med Res Methodol. 2014 developments. BMJ. 1998 Jan 17;316(7126):221–225. https://doi.
Dec;14(1):25. https://doi.org/10.1186/1471-2288-14-25 org/10.1136/bmj.316.7126.221
Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Stang A. Critical evaluation of the Newcastle-Ottawa scale for the
Gavaghan DJ, McQuay HJ. Assessing the quality of reports of assessment of the quality of nonrandomized studies in meta-
randomized clinical trials: is blinding necessary? Control Clin analyses. Eur J Epidemiol. 2010 Sep;25(9):603–605. https://doi.
Trials. 1996 Feb;17(1):1–12. https://doi.org/10.1016/0197- org/10.1007/s10654-010-9491-z
2456(95)00134-4 Sterne JAC, Egger M. Funnel plots for detecting bias in meta-analysis.
Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a J Clin Epidemiol. 2001 Oct;54(10):1046–1055. https://doi.
systematic review. J R Soc Med. 2003 Mar;96(3):118–121. https:// org/10.1016/S0895-4356(01)00377-8
doi.org/10.1177/014107680309600304 Sterne JAC, Hernán MA, Reeves BC et al. ROBINS-I: a tool for
Khan S, Doi SAR, Memon MA. Evidence based decision and meta- assessing risk of bias in non-randomised studies of interventions.
analysis with applications in cancer research studies. Appl Math BMJ. 2016 Oct 12;355:i4919. https://doi.org/10.1136/bmj.i4919
Inf Sci. 2016 May 1;10(3):815–822. https://doi.org/10.18576/ Stroup DF, Berlin JA, Morton SC et al. Meta-analysis of observational
amis/100301 studies in epidemiology: a proposal for reporting. Meta-analysis
Kulinskaya E, Olkin I. An overdispersion model in meta- Of Observational Studies in Epidemiology (MOOSE) group.
analysis. Statistical Modelling: An International Journal. 2014 JAMA. 2000;283(15):2008–2012. https://doi.org/10.1001/
Feb;14(1):49–76. https://doi.org/10.1177/1471082X13494616 jama.283.15.2008
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Thompson SG, Higgins JPT. How should meta-regression analyses be
Improving the quality of reports of meta-analyses of randomised undertaken and interpreted? Stat Med. 2002 Jun 15;21(11):1559–
controlled trials: the QUOROM statement. Lancet. 1999 1573. https://doi.org/10.1002/sim.1187
Nov;354(9193):1896–1900. https://doi.org/10.1016/S0140- van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in
© 2019 MA Healthcare Ltd

6736(99)04149-5 meta-analysis: multivariate approach and meta-regression. Stat

Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Med. 2002 Feb 28;21(4):589–624. https://doi.org/10.1002/
Preferred reporting items for systematic reviews and meta-analyses: sim.1040
the PRISMA statement. PLoS Med. 2009 Jul 21;6(7):e1000097. Wan X, Wang W, Liu J, Tong T. Estimating the sample mean
https://doi.org/10.1371/journal.pmed.1000097 and standard deviation from the sample size, median, range
Moher D, Hopewell S, Schulz KF et al; Consolidated Standards and/or interquartile range. BMC Med Res Methodol. 2014
of Reporting Trials Group. CONSORT 2010 Explanation and Dec;14(1):135. https://doi.org/10.1186/1471-2288-14-135

British Journal of Hospital Medicine, November 2019, Vol 80, No 11 641
Downloaded from magonlinelibrary.com by 131.172.036.029 on November 16, 2019.