Psychotherapy Research

ISSN: 1050-3307 (Print) 1468-4381 (Online) Journal homepage: http://www.tandfonline.com/loi/tpsr20

How to conduct a qualitative meta-analysis:

Tailoring methods to enhance methodological
integrity

Heidi M. Levitt

To cite this article: Heidi M. Levitt (2018): How to conduct a qualitative meta-analysis:
Tailoring methods to enhance methodological integrity, Psychotherapy Research, DOI:
10.1080/10503307.2018.1447708

To link to this article: https://doi.org/10.1080/10503307.2018.1447708

Published online: 13 Mar 2018.

Submit your article to this journal

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=tpsr20
Psychotherapy Research, 2018
https://doi.org/10.1080/10503307.2018.1447708

CONSIDERATIONS OF HOW TO CONDUCT META-ANALYSES IN PSYCHOLOGICAL

INTERVENTIONS

How to conduct a qualitative meta-analysis: Tailoring methods to

enhance methodological integrity

HEIDI M. LEVITT

Department of Psychology, University of Massachusetts Boston, Boston, MA, USA

(Received 31 October 2017; revised 21 December 2017; accepted 22 December 2017)

Abstract
Although qualitative research has long been of interest in the field of psychology, meta-analyses of qualitative literatures
(sometimes called meta-syntheses) are still quite rare. Like quantitative meta-analyses, these methods function to
aggregate findings and identify patterns across primary studies, but their aims, procedures, and methodological
considerations may vary. Objective: This paper explains the function of qualitative meta-analyses and their
methodological development. Recommendations have broad relevance but are framed with an eye toward their use in
psychotherapy research. Rather than arguing for the adoption of any single meta-method, this paper advocates for
considering how procedures can best be selected and adapted to enhance a meta-study’s methodological integrity.
Method: Through the paper, recommendations are provided to help researchers identify procedures that can best serve
their studies’ specific goals. Meta-analysts are encouraged to consider the methodological integrity of their studies in
relation to central research processes, including identifying a set of primary research studies, transforming primary
findings into initial units of data for a meta-analysis, developing categories or themes, and communicating findings.
Conclusion: The paper provides guidance for researchers who desire to tailor meta-analytic methods to meet their
particular goals while enhancing the rigor of their research.

Keywords: qualitative meta-analysis; meta-synthesis; methodological integrity; psychotherapy; trustworthiness

Clinical or methodological significance of this article: This article describes the function of meta-analytic methods in
relation to various research goals, such as the development of theoretical models, comprehensive literature reviews, or
methodological reviews. It encourages researchers to design and carry out studies in a manner that strengthens the
methodological integrity of their work in relation to their specific goals.

Qualitative meta-analytic methods are tools that allow is psychotherapy effective?). This overlap is genera-
researchers to aggregate and synthesize findings from tive as methodologists with expertise in quantitat-
primary qualitative studies. These studies can assist ive, qualitative, and mixed methods have long
researchers in meeting a variety of goals, such as devel- argued the deliberate varying of methodological
oping broadly-based theoretical understandings, and epistemological approaches enhances research
cataloging sets of findings, developing measures, programs and deepens understanding (e.g., Levitt,
conducting comprehensive assessments of the state of Surace, et al., 2017; Shadish, 1986). Accordingly,
a body of literature, forming principles to guide researchers from any perspective can strengthen
in-session practice, or examining the methods and their framing of research questions by consulting
methodologies employed within a field. Many of these both qualitative and quantitative primary and
goals are similar to those of quantitative meta-analyses. meta-analytic research in their literature reviews.
Indeed, qualitative and quantitative researchers In this process, researchers will be aided by an
often examine the same exact questions (e.g., why understanding of the logic of design that is

Correspondence concerning this article should be addressed to Heidi M. Levitt, Department of Psychology, University of Massachusetts
Boston, Boston, MA, USA. Email heidi.levitt@umb.edu

© 2018 Society for Psychotherapy Research

2 H. M. LEVITT

particular to qualitative methods and in which rigor number of qualitative meta-analytic reviews (e.g.,
is associated especially with the strengthening of Timulak, 2007 on helpful therapy processes). Other
inductive processes. qualitative analyses have followed by multiple
The current paper provides an overview of the research teams, focusing on issues such as experi-
history of qualitative meta-analytic methods in psy- ences of master therapists (Jennings et al., 2016),
chology. Varied forms of qualitative meta-analyses the training of psychotherapists (McGillivray,
that have been put forward, each entailing prescribed Gurtman, Boganin, & Sheen, 2015) and computer-
sets of procedures (Barnett-Page & Thomas, 2009; ized psychotherapy (Knowles et al., 2014). Levitt,
Timulak, 2013). Instead of recommending the adop- Pomerville, and Surace’s (2016) omnibus meta-
tion of any one method whole cloth, however, the analysis reviewed psychotherapy studies with varied
current paper advocates for considering how pro- foci.
cedures can best be selected and tailored (potentially Although direction on how to conduct qualitative
from across forms of meta-analysis) to allow research- meta-analyses has come from outside psychology
ers to meet their specific study goals. To provide gui- (e.g., Sandelowski et al., 1997), Timulak (2009)
dance on navigating this process, the author reviews wrote a helpful article describing his meta-analytic
the steps of meta-analyses and considers critical method with a focus on psychotherapy, and in
researcher decisions using the framework of meth- 2013, he co-authored a paper on this topic with his
odological integrity, which has been developed to colleague, Mary Creaner. The current paper builds
guide the adaptation of procedures in relation to the on this work by discussing how to tailor meta-ana-
logic of qualitative research. lyses for varied purposes and how to strengthen
methodological integrity within the meta-analytic
process.
The Development of Qualitative
Meta-analysis
Methodological Integrity
Neither qualitative research, nor meta-analytic research
is new to either psychology or psychotherapy research- Methodological integrity is a concept put forward by
ers (Wertz, 2014). Qualitative research methods have a task force of the Society for Qualitative Inquiry in
become systematized only over the last half-century Psychology (SQIP; Levitt, Motulsky, Wertz,
as their procedures have come to be formally expli- Morrow, & Ponterotto, 2017) that generated rec-
cated, with phenomenological (Giorgi, 1970) and ommendations for both designing and reviewing
grounded theory researchers (Glaser & Strauss, 1967) qualitative research. The goal of this work was to
as early forerunners in this processes. These methods identify the methodological foundation for trust-
began to catch on gradually in psychological research worthiness in qualitative research to provide a
and leading psychotherapy researchers were at the van- clearer understanding of the underlying logic within
guard of this turn, such as Robert Elliott, Clara Hill, qualitative research design. In contrast to a cookbook
John McLeod, and David Rennie. They introduced approach to research evaluation that constrains
and popularized qualitative methods in psychology, research design by mandating adherence to estab-
and established early standards for their evaluation lished methods, this framework was developed to
(e.g., Elliott, Fischer, & Rennie, 1999). provide guidance to reviewers on how qualitative
Qualitative meta-analyses began to emerge a few research methods could best be adapted given a
decades later in nursing, sociology, and education study’s research goals, approach to inquiry or epis-
(e.g., Noblit & Hare, 1988; Paterson, Thorne, temology, and characteristics. Input was incorpor-
Canam, & Jillings, 2001; Sandelowski, Docherty, & ated from researchers from across a breadth of
Emden, 1997), although they remain relatively rare qualitative methods and perspectives to ensure the
in psychology. Researchers using multiple terms to framework’s capacity to be applied across
signify this form of review may have compromised approaches. Recently, APA Style, which determines
the visibility of this method (e.g., meta-ethnography, the research standards for both the Publication
meta-study, meta-synthesis). Although these Manual of the APA and APA Style Central, has used
approaches all aggregate qualitative findings, they the concept of methodological integrity to guide
each recommend differing sets of procedures. In authors, reviewers, and editors in reporting primary
this article, I retain the term qualitative meta-analysis qualitative research, mixed methods, and qualitative
as it is better recognized in both psychology and psy- meta-analyses (Levitt et al., in press). Given its
chotherapy research. broad applicability, the framework of methodological
Ladislav Timulak spearheaded this method in the integrity is applied in the current paper to the design-
field of psychotherapy research and contributed a ing of qualitative meta-analyses (Table I).
Table I. Conceptual issues and recommendations for designing a qualitative meta-analysis.

Issue Question Methodological recommendation

(1) Identifying and describing primary studies. How comprehensive does my search need to be? Consider whether your goal is to review the complete literature base or to collect enough
primary studies to reach a saturated set of findings or model that answers your question.
How should I consider the quality of primary articles? Consider restricting data collection to published studies, using methods that rate quality,
or coding methodological features.
How should I consider the fit of primary articles with Seek a diversity of studies within the scope of your research question or adjust your
the research question? question to fit the available studies. Articulate justifications for the fit between types of
diversity included or coded in the studies and your question. Be transparent in
describing your procedures for assessing fit (inter-rater coding, etc.).
(2) Transforming primary research findings How should I label my initial units reflecting the Using the primary research labels is helpful so long as the connection to your research
into initial units of data in the meta-analysis. primary findings to facilitate their use in the meta- question is clear, and the phrasing allows you to make connections across other studies.
analysis? If not, re-label findings to do so.
(3) Organizing initial units into categories or How many categories or themes should I create? Consider what would be useful in the context of your goal (an article, a training, a book,
themes. etc.).
How should I label categories? Generate category labels that are descriptive and specific enough to serve as meaningful
answers to your research question. Keep labels grounded in the distinctions within the
data.
Reconciling conflicting answers will increase the coherence of your findings.
(4) Increasing Methodological Integrity Should I use frequency counts of findings? The use of frequency counts can be used to indicate the salience of findings or focus of the
researchers, rather than their validity or the proportion of cases in which they are
experienced.
How can I review large numbers of studies? The use of hybrid qualitative methods has been found helpful, in which researchers
develop an attuned saturated model using one qualitative method (e.g., grounded
theory, discourse analysis, thematic analysis) and then use content analysis to sort the
rest of the database into that model (maintaining the ability to adjust the model as
needed).
Should I examine the methods used and their Examining methods can be an important function of meta-research but it may be
influence upon the findings? challenging to merge a meta-method study and a meta-analysis of findings within one
article. Researchers may need to write a separate article to examine methods and their
effect more closely.
Should I restrict my literature to certain features (e.g., Restricting your literature review to focus on certain features can allow for
therapy orientation or diagnoses) recommendations that are tailored to that context. It can provide insight into dynamics
that are localized and can challenge understandings that are based more broadly.
What procedures should I consider when using meta- Using methods that organize data in a hierarchical format can be helpful when developing

Psychotherapy Research 3
analyses to develop measures? measures because this form suggests both items for an initial scale as well as factors with
which they might be associated in exploratory factor analyses.
How should I manage my expectations, perspectives, Qualitative meta-analysts value being aware of their own expectations and being
and assumptions in the analysis and write up? transparent in the process of reporting findings. This transparency can strengthen
methodological integrity.
How should I examine the expectations, perspectives Researchers can examine the self-reflective statements of researchers, the perspectives
and assumptions of the primary researchers? revealed by their questions or findings, or search authors’ other publications for
identification of their perspectives.
How should findings be discussed to enhance Findings that are contextualized appropriately, that generate new insight into a
methodological integrity? phenomenon, that are coherent with one another and that lead to a meaningful and
useful contribution will be higher in methodological integrity.
4 H. M. LEVITT

There are two central components in the estab- General Process of Meta-analysis
lishment of methodological integrity; both have
Many forms of qualitative meta-analytic methods
been conceptualized as having four central features.
have been developed (e.g., meta-ethnography,
The first process, fidelity to the subject matter is the
Noblit & Hare, 1988; meta-study, Paterson et al.,
process by which researchers select procedures
2001; meta-summary, Sandelowski & Barroso,
that increase their findings’ allegiance to the
2003). These methods prescribe varied sets of pro-
phenomenon under study. It is strengthened when
cedures, with some emphasizing cataloging findings,
researchers collect data that includes variations in
developing new findings, conducting a critical analy-
the phenomenon that generate a more comprehen-
sis of findings, or assessing the quality of findings (see
sive understanding of their research question (data
Barnett-Page & Thomas, 2009; Timulak, 2013 for a
adequacy); when researchers recognize and are
detailed review). In addition, many qualitative
transparent about the influence of their own per-
methods can be adapted to generate a meta-analytic
spectives and appropriately limit that influence
approach by using primary findings in place of raw
within their data collection (perspective management
data (e.g., Kearney’s, 1998 grounded formal theory).
in data collection); when they consider how their
Before describing how to adapt qualitative meta-
own perspectives influenced or guided their analytic
analytic procedures to the goals of a given study,
process with the aim of strengthening their percep-
the general processes that form the foundation of
tiveness (perspective management in data analysis);
most forms of qualitative meta-analysis will be
and when findings are rooted in the data analyzed
reviewed. I will draw from some of my own research
(groundedness).
collaborations to provide examples of decision
The second process, utility in achieving research
points. This review will consider how to best adapt
goals is the process by which researchers select pro-
procedures using the framework of methodological
cedures that usefully answer their research questions
integrity.
and address their aims (e.g., developing theory, dee-
pening understanding, raising critical consciousness,
identifying social and discursive practices). Utility is Identifying and Describing Primary Studies
strengthened when findings are considered in
context—relaying, for instance, their location, time, The process of identifying studies for a qualitative
and cultural situation (contextualization of data); meta-analysis may be similar to the process of a quan-
when data collected are rich enough to support titative meta-analysis. Typically, researchers will
insightful analyses (catalyst for insight); when analyses need to use electronic databases (e.g., PsycINFO)
lead to insights that address the researchers’ ques- to locate the primary studies. Researchers will ident-
tions and goals (meaningful contributions); and when ify keywords and terms that will assist them in finding
conflicting findings are reconciled or explained studies that are relevant to their question. In their
(coherence among findings). reporting, they will identify the databases searched,
A central characteristic of the framework of meth- the time period searched, the keywords used, and
odological integrity is that fidelity and utility are the languages of studies reviewed. Researchers may
conceptualized in relation to the researchers’ goals, wish to use a flow diagram to demonstrate their
approaches to inquiry (e.g., philosophical assump- process of identifying studies (e.g., Moher, Liberati,
tions), and study characteristics (e.g., the particular Tetzlaff, & Altman, 2009; see Swift & Wampold,
subject matter, resources, participants, researchers). 2018 for a discussion of how this process in quantitat-
Many examples can be provided of times when these ive meta-analyses).
features could lead to changes in procedures. For An exception can occur when the researcher’s goal
instance, interviewing procedures might change is to develop a new understanding or theory of the lit-
when interviewing vulnerable participants to maxi- erature and plans to halt the collection of primary
mize the fidelity of the ensuing data. The types of studies once saturation is reached (e.g., Kearney,
diversity sought in participants might shift in 1998). In this case, the researchers might not need
relation to the forms of diversity that influence a to locate all the research in a literature base but
given topic to enhance the data adequacy. Or, a enough to satisfy that goal. (For a detailed description
critical qualitative approach would demonstrate of this process and its tie to utility, see the forthcom-
utility in relation to the goals of this approach ing section on Developing an understanding of find-
(e.g., social change, consciousness raising) in order ings versus a description of an entire literature.)
to lead to a meaningful contribution. In other
words, the fidelity and utility of procedures need Considering quality. As search terms often are
to be evaluated in relation to their function in the insufficient in assessing the eligibility of the primary
research design and aim. studies in light of a specific research question,
 Psychotherapy Research 5

researchers will need to describe how they narrowed question, there are several things to keep in mind.
down the studies reviewed. To do this, researchers In qualitative research, greater diversity of partici-
usually consider two issues—the fit of the primary pants within the scope of a question tends to be
study to the meta-analytic study question and the helpful as it allows for a more comprehensive under-
quality of the study. When considering the quality of standing of how a phenomenon is experienced across
a study, a key decision is whether unpublished a variety of people, improving the fidelity of a study
research will be retained. Including only published (Levitt, Surace, et al., 2017). Researchers will want
qualitative research can be a helpful form of quality to establish the scope of the question that they are
control (Timulak, 2009). asking and whether there is enough of a primary lit-
Criteria have been developed to evaluate the erature to support data adequacy. That is, is there a
quality of qualitative research for systematic reviews number of and diversity of primary studies to
(e.g., Critical Appraisal Skills Programme, 2017; provide an answer to the researchers’ question in its
Paterson et al., 2001), however, characteristics of current form? If not, they could describe this as a
these systems tend to be quite basic and are overwhel- limitation of the research and direct future research-
mingly present in published research (e.g., Was there ers to filling in that gap. For instance, if research on
a statement of research aims? Was a qualitative meth- insight in therapy had been conducted mostly from
odology appropriate?). A few meta-analytic the perspective of one type of client (e.g., depressed
approaches (e.g., framework synthesis, thematic nar- clients), this would be a strong limitation to making
rative analysis, and thematic synthesis) include within general claims.
them guidance for rating the research quality of the Because the meta-analytic researchers are not
primary studies, for instance by assessing their ration- recruiting clients themselves, they may have to
ale, methods, and findings. These evaluations can adapt their question to the characteristics of the
lead to the exclusion of studies if these ratings do data available in the primary studies for their
not meet an acceptable score. Because these answers to have methodological integrity. For
methods exclude certain studies, however, this instance, in the preceding example, it would be
process would not permit researchers to conduct an appropriate for the researchers to narrow the scope
evaluation of the methodological state of the litera- of their study question to insight within depressed
ture on a whole. Also, if researchers are interested clients. Although researchers will begin with a
in critiquing the use of methods, excising methodolo- general study question, as they code the primary
gically problematic articles would not permit a com- research studies, they may shape their study question
prehensive picture of the field and the analysis to be descriptive of the primary data. Researchers
would be limited. Also, a challenge in evaluating pub- may wish to consider the cultural backgrounds of par-
lished research is that the page limits for most jour- ticipants, their presenting issues, their diagnoses,
nals have been developed for quantitative research their professional training, and other features that
and may prohibit a full description of the methods are relevant to their research topic.
used, making it challenging to be certain of the meth- When designing inclusion and exclusion criteria to
odological rigor as procedures may go unreported. remove studies from the initial returns from a search
In my experience, qualitative meta-analyses are for primary research, providing a justification of these
robust to the occasional weaker study as poor quali- criteria in relation to the scope of the question will
tative research findings tend to be characterized by help reviewers and readers make sense of the
being less descriptive and vague. Because of this, decisions being made. For example, in our study on
their findings can be interpreted in light of other clients’ experiences of psychotherapy, we excluded
studies that display more insight and acuity within case studies that focused upon one client (Levitt
the meta-analytic process. Limiting the primary et al., 2016). We explained that case study research
research to studies that have passed the test of peer would be challenging to include both because our
review then may be sufficient quality control. question focused on typical experiences and case
Across forms of meta-analysis, researchers may wish studies often are used to represent more idiosyncratic
to document research quality by coding the specific experiences.
design features that enhance integrity in the literature When reviewing initial returns from a search, it is
or by rating of the eight components of methodologi- likely that most of these studies can be culled based
cal integrity (see Levitt, Surace, et al., 2017 for ques- upon the title and abstract of papers alone. Research-
tions and principles that can guide this evaluation). ers will want to record the numbers of articles that
were eliminated within the examination of the title
and abstract versus full text. If researchers use mul-
Considering fit. When evaluating the fit of tiple judges to make these determinations, a typical
primary studies to the meta-analytic research process in quantitative meta-analyses, they also will
6 H. M. LEVITT

report the numbers of judges, and how agreement extraneous units and categories into an analysis can
was indicated or disagreement was resolved. make the process of developing findings confusing
In generating a description of the primary studies and including irrelevant data can weaken the emer-
selected for analysis, researchers may or may not ging findings by compromising fidelity.
want to use multiple judges to check each other’s In the labeling of primary units, I often use the
coding depending on how challenging that infor- primary researchers’ category or thematic titles
mation is to identify within a study. Typically, directly or with small clarifying amendments. This
researchers will identify features of each article that is not always possible, however. Some qualitative
relate to the context of the paper (e.g., the research- methods do not present their results via category
ers’ location, the setting of the research, the psy- titles but use narrative descriptions instead. Also,
chotherapy orientations examined) and features that sometimes researchers form the category titles that
describe the methodological processes (e.g., the represent the research questions rather than the
numbers of participants, the central research ques- answers—a practice that I discourage. For instance,
tion, data collection methods, data analytic a study on clients who have interpersonal conflict
methods, epistemological perspectives, the checks might have categories called “interpersonal conflict
on methodological integrity used). For instance, in intimate relationships,” “interpersonal conflict in
Levitt, Pomerville, Surace, and Grabowski’s (2017) the workplace,” and “interpersonal conflict in
meta-method study on the literature on clients’ therapy.” These category titles reflect the question
experiences of psychotherapy described the use of of the researcher about the contexts of interpersonal
checks that included auditors, inter-rater reliability, conflict but they do not reveal anything substantive
consensus, saturation, reflexivity checks, participant about the character of the conflict in any of these set-
checks, and triangulation. The recommendation in tings. In both these times, I will form my own labels
identifying and describing the primary studies is to by summarizing the text to provide meaning unit
consider the function of the procedures and labels that distinguish the ways the phenomenon
whether they strengthen the transparency of reporting transpires in the contexts.
and the methodological integrity of the design with Labels can be written in such a way to permit both
regard to the goals of the meta-study. common and unique meanings to be identified across
studies. For instance, if one study describes tolerating
fear in a behavioral exposure intervention, a second
Conducting Meta-analyses portrays the emotion-focused process of resolving
sadness during a focusing exercise, and a third
After the primary literature is in hand, the central describes the experience of hypnosis to re-experience
processes in meta-analytic methods include generat- stressful early experiences, labeling these units “be-
ing units from the primary findings and labeling havioral exposure is effective,” “focusing resolves
them, creating categories based upon commonalities depression,” and “hypnosis reduces stress” will lose
and distinctions in the meanings within these units, the common processes of engaging with emotions
and then examining the relationship between these within these interventions. It would be preferable to
categories to develop a central finding or a set of avoid overreliance on the intervention labels and
main findings. instead articulate this common experience. For
instance, “behavioral exposure helps clients engage
Generating units and labels. A first step in with fear, contributing to their resolution,” “focusing
qualitative meta-analysis is to form initial units that exercises help clients maintain contact with sadness
each describe a finding from the primary research to alleviate it,” or “hypnosis structures engagement
and assign it a descriptive label. I find it helpful to with early stressors.” These labels that emphasize
follow Rennie’s (2000) approach in borrowing from process are more likely to be meaningfully grouped
phenomenology the process of generating meaning together. For this reason, it can be helpful to avoid
units (e.g., Giorgi, 2009) that each describe one jargon to increase the likelihood of a developing
central meaning. A finding in a primary study may insightful data that can lead to new understandings.
lead to more than one unit, if they include multiple
meanings that are relevant to the subject in the
meta-study. In this process, the groundedness of Creating categories and higher order
units can be increased by remaining close to the orig- categories. The labeled units become the foun-
inal researchers’ ideas as possible when labeling units, dation for categories (sometimes called themes or
but shaping them to answer the central question of simply findings) that describe patterns within the
the meta-analysis as directly as possible (and findings meta-analytic findings. The meaning units from one
that are not relevant are excluded). Introducing study, each summarizing a central finding, are
 Psychotherapy Research 7

compared to the units from the other studies in turn. I (6) Feeling and discussing sadness with the
conceptualize this process as akin to constant com- therapist led to feeling connection.
parison, in which each unit is compared with every
Grouping units 1, 3, and 5 into a category called,
other unit to identify points of similarity and differ-
“Sadness is scary because it is overwhelming and
ence, which then become the basis for categories
might lead to therapist judgment,” and grouping
that group those units together (Glaser & Strauss,
units 2, 4, and 6 together into a category called,
1967).
“Feeling sadness in session increases engagement, con-
Depending on the form of analysis in use, these
nection and acceptance” can be a crucial preliminary
categories may become the central findings in the
step in considering how to reconcile the meanings
meta-analysis, or the initial categories created may
within the units in an overarching category title that is
be subjected to a similar process of mutual compari-
more meaningful and grounded than simply saying
son in which they are grouped together to identify
that clients feel sadness in therapy. One such title in
broader, more encompassing themes. This process
our meta-analysis of clients’ therapy experiences
of seeking to integrate higher order categories may
(Levitt et al., 2016) was, “Fear of sadness and vulner-
continue until the set of findings appear to be convey-
ability prompts disengagement but experiencing and
ing discrete ideas that hold utility in light of the
exploring these emotions in therapy enhances engage-
research question at hand. Typically, it is challenging
ment and leads to acceptance” (p. 818). The reconcil-
to present more than eight main findings in a journal
ing of conflicting clients’ reactions guides therapists on
length article so when there are many initial cat-
how to intervene when they see clients avoiding their
egories, it can be especially helpful for researchers
emotions in session and demonstrates greater utility
to group them together so they become more man-
than a general vague category title. Being as specific
ageable to present. Alternatively, researchers may
as possible in terms of contexts and processes within
need to divide their analysis into separate articles
the findings is useful in this regard.
that focus on discrete sets of findings. For instance,
Third, when there are conflicts within the data, I
a meta-analysis might result in one paper that is
work to reconcile them in order to increase the coher-
focused upon the review of findings and a second
ence of my findings. For instance, if one finding
meta-method article focused on the use of qualitative
shows that clients do like when therapists have pro-
methods in a field of study.
fessional credentials but another states that they
Similar to the formation of initial unit labels, the
also are suspicious of credentials, therapists reading
categories are assigned labels to capture the pattern
the article will be uncertain how to draw guidance
observed in that set of data. There are a few tips
from that study. Conflicting findings allow research-
that can be helpful at this point. First, these labels
ers to identify patterns that may not have been articu-
are best when they make clear their connection to
lated before. To do this, researchers may ask when,
the central meta-analytic study question, for the
under what conditions, why, or contingent upon what dis-
reasons described previously. Second, when there
tinctive responses patterns hold. For instance, in this
are many units grouped together within a category
same meta-analysis, we developed the principle,
(>8), it can be useful to check if there are subcate-
“Professional structure creates credibility and clarity
gories that could be created to articulate the distinc-
but casts suspicion on care in the therapeutic
tive meanings among the units. The articulation of
relationship” (p. 820), to explain the varied functions
these common meanings in subcategory titles might
of credentials. By reconciling conflicting meanings,
inform the evolving broader category title. For
researchers enable more sophisticated theoretical,
instance, consider the following unit labels that
clinical, and research implications and increase the
might appear under a provision category title
utility of their research findings.
“Sadness in therapy”:

Identifying Goals and Tailoring

(1) Clients expressed worry about feeling over-
Meta-analysis
whelming sadness in therapy.
(2) Clients felt that exploring sadness in therapy While the previous section focused on processes that
helped them accept their emotions. are common across most forms of meta-analysis, this
(3) Clients felt vulnerable when sad topics arose present section is concerned with the tailoring of pro-
and tried to avoid the topic. cedures to specific research goals. There are a variety
(4) Clients felt more engaged in sessions when of reasons why one might wish to conduct a meta-
they got in touch with sadness. analysis, including the development of a new under-
(5) Clients worried that their therapist would be standing, a need to reconcile conflicts in the litera-
critical of them if they cried. ture, the identification of central findings in an
8 H. M. LEVITT

entire literature, the development of an outcome may extend deeper than those in the primary litera-
measure or a process measure, the desire to raise ture (e.g., Ma, Roberts, Winefield, & Furber,
critical consciousness about shortcomings or biases 2015). They do not need to analyze an entire litera-
in a literature, and an interest in reviewing the meth- ture, however, to develop and offer new understand-
odological soundness of a set of findings or the use of ings. Investigators being clear on their study aim will
a certain method. Research goals will be reviewed in be helpful as they set forth to design their meta-analy-
relation to adaptions that can be made to facilitate sis. This goal may be influenced, in part, by the
that goal and strengthen methodological integrity. number of primary studies available for analysis. A
distinction of qualitative meta-analysis is that the
numbers of primary studies can vary widely from 2
Cataloging Types of Findings versus to 100, with 12 being an average number (Timulak,
Re-analyzing Them 2009). Many psychotherapy qualitative meta-ana-
lyses seem to examine approximately 7–10 studies
When researchers describe meta-analyses, they typi-
(e.g., McGillivray et al., 2015; Timulak, 2007;
cally are referring to processes in which primary find-
Timulak & Creaner, 2010), which may well exhaust
ings are subjected to a synthesizing process of
the research that has been conducted in an area.
secondary analysis. In contrast, Sandelowski and
If the literature contains a large enough number of
Barroso (2003) have developed a method, qualitative
studies, however, it creates the possibility that
meta-summary, which collates findings into descriptive
researchers seek to develop a saturated understand-
groups but does not re-interpret them. Instead, it
ing of that literature in their meta-analysis. Saturation
guides researchers to consider the frequency of findings
is the point at which new data being analyzed ceases
across studies and to credit findings that appear more
to add new understandings to the analytic scheme
often with higher validity within an “effect size” that is
already developed, suggesting that the analysis is
“verifying the presence of a pattern or theme”
comprehensive (Glaser & Strauss, 1967). To
(p. 231). The use of frequency as a proxy for validity
provide an example, in our meta-analysis on clients’
in qualitative research can be seen as problematic,
experiences of psychotherapy (Levitt et al., 2016),
however (see Levitt, Butler, & Hill, 2006 for a descrip-
saturation was reached at 47 studies, as tested by
tion of this problem in primary qualitative research).
the addition of 20 more studies into the analysis
The concern with this calculation is that, even
that did not generate new categories. This could be
within the same area, researchers may not have the
a rationale to end the collection of primary research
same interests, ways of phrasing questions, capabili-
as the findings meet the research goal of developing
ties as researchers, or priorities in terms of generating
new understandings of the literature—even if all the
thematic foci. As a result, it can be that a finding
primary studies were not reviewed.
found only in one study, but by a highly attuned
When there is large number of studies and
research team, underlies and meaningfully explains
researchers wish to review them all, they might con-
superficial findings that are identified frequently
sider using a combination of methods. When we
often across other studies. Also, research can be con-
first conducted this analysis, we had planned to halt
ducted at times and regions when certain expla-
data collection at the point of saturation (reviewing
nations become more prevalent and accessible, even
67 studies) and submitted our paper to Psychological
though they might always be at play. For instance,
Bulletin. Feedback from reviewers, however, indi-
descriptions of fear of the government by immigrant
cated that they wanted our review to include all of
clients might emerge under one president and not
the research related to clients’ experience of psy-
under another, but its inconsistency in time does
chotherapy. A systematic search of the literature
not make the finding less valid. Or participants
revealed that there were 109 studies that met our
might agree with ideas that they fail to describe
inclusion criteria. Because we were using a grounded
because they think they are obvious, not relevant, or
theory meta-analytic approach, it would have been
not as pressing as other concerns. For these many
prohibitive to conduct constant comparison with all
reasons, the use of frequency to indicate the validity
the additional studies. As a result, we developed a
or representativeness of a finding within a population
hybrid method in which we used the categories that
(versus within a set of studies) is questionable.
we had developed in our grounded analysis that
were highly attuned to clients’ experiences and then
conducted a content analysis with the remaining lit-
Developing an Understanding of Findings
erature by sorting the unitized and labeled findings
versus a Description of an Entire Literature
into the already-established categories. Although we
Qualitative meta-analyses have been advocated for were open to adjusting the category labels to reflect
their ability to develop novel interpretations that incoming data, the hierarchy remained relatively
 Psychotherapy Research 9

stable. This additional analysis permitted us to been conducted, for instance, specifically examining
characterize a broader research literature and to research that is focused within specific orientations
speak more confidently about the prevalence of or diagnoses. It still can be challenging to identify
trends in our findings. The adoption of a similar enough research to conduct these meta-analyses.
hybrid approach to analysis might make feasible Within the research on clients in psychotherapy, for
other comprehensive reviews of large literature bases. instance, the vast majority of the primary qualitative
research studies appear to include clients from
across psychotherapy orientations (73.4%; Levitt
Reviewing Content versus Methods Being et al., 2016). This research can provide valuable
Used information, however, on processes and goals that
are associated within identified therapy approaches.
While some meta-analysts are interested primarily in
For instance, Timulak and Creaner (2010) examined
the content of the primary findings, others may be
8 qualitative studies on humanistic therapies and
more interested in examining how qualitative
identified 11 types of client-reported change. These
methods are used. A number of established methods
included a variety of outcomes, such as healthier
were developed to examine the use of both findings
emotional experiencing, experiences of appreciating
and methods within one qualitative approach, such
vulnerability, changed views of others, and empower-
as meta-ethnography to examine ethnographic
ment. This focused research usefully identified
studies (Noblit & Hare, 1988), and grounded formal
change processes that are not typically assessed
theory to review grounded theory studies (Kearney,
using standardized psychotherapy outcome measures
1998). While these methods can be adapted to
(Levitt, Stanley, Frankel, & Raina, 2005) and lead
review a wide range of methods, the parallel structure
toward a better understanding of how change
between the meta-analytic method and primary
unfolds within a treatment.
methods can facilitate a review.
Paterson et al. (2001) described a meta-study
process that incorporates a meta-analysis that
reviews a literature’s findings, a meta-theory that Process or Outcome Measure Development
reviews the theory and a meta-method that reviews
Within mixed methods programs of research, quali-
the methods used and considers how the methods
tative meta-syntheses can be used to develop
used to influence the findings reported. The meta-
measures that are grounded in a literature base.
method component can be used independently,
When developing a process measure, findings from
however, to examine how multiple methods are
a study can help researchers to identify the central
applied within an area of research. This approach to
change processes related to the resolution of types
research can lead to informative findings that can
of therapy events across primary studies. They also
advance qualitative methods in a given area. For
could lead to the development of typologies that
instance, a meta-method study indicated that it was
can help identify the internal processes that are
commonplace for qualitative psychotherapy research-
unfolding within a session. For instance, Li, Black,
ers to creatively integrate procedures from one
and Garland (2016) used a mixed method approach
method to enhance another approach, suggesting
including qualitative interviews, scale validation,
that reviewers’ evaluations should focus on methodo-
and a replication study to develop a process
logical integrity within a study rather than its adher-
measure of how mindfulness can be used to alleviate
ence to a given method to reflect this practice
psychological suffering. This process allowed them to
(Levitt, Pomerville, et al., 2017). Meta-method
begin their measure development with a set of
studies also can direct researchers toward methods
attuned items that conveyed the experiences they
that rarely are used but that might make a significant
were studying with fidelity.
contribution in a field. Because qualitative studies are
When developing an outcome measure, it can be
still relatively new to the field of psychology, examin-
useful to utilize a method of qualitative analysis that
ations of how they are being conducted can commu-
will result in at least two levels of categories, which
nicate and shape standard research practices.
can assist in identifying items that might be associ-
ated with one another in subscales. Grounded
theory meta-approaches can be particularly useful
Reviewing Research across or within a
in this respect as the hierarchical data structure
Certain Context
makes it possible to identify a large number of
Sometimes researchers’ goals focus on primary items that can then be reduced within a factor analy-
research conducted within a specific context or sis. For instance, Levitt identified a set of 50 items
setting. Some qualitative meta-analytic research has from a grounded-theory driven meta-analysis
10 H. M. LEVITT

(Levitt et al., 2016) and then, with colleagues Gra- transparent about this process so that their findings
bowski and Minami (2018), developed a 15-item can be read in light of their analytic approach.
measure to capture the change processes most criti- In addition to these forms of methodological integ-
cal to psychotherapy clients based upon an explora- rity, which also relate to primary data analysis, there is
tory factor analysis. The advantage of developing a form that relates specifically to meta-analyses. This
items from meta-analyses is that they can be both is the consideration of the perspectives of the primary
robust, as they are confirmed across a number of researchers. Because many primary qualitative
studies, and comprehensive, as they broadly reflect research studies include reflexive statements reveal-
the internal experiences of participants from ing the positions or attitudes of investigator with
across a range of primary study questions and inves- respect to the research question, meta-analytic
tigator lenses. Reflecting the qualitative research researchers can analyze this information. A problem
from which they derive, they also permit the though is that, although many primary studies
measurement of phenomena that are challenging (80.7% in one meta-analysis of psychotherapy
to evaluate because they may be subjective, covert, research; Levitt et al., 2016) contain some reflexive
and/or context-dependent. statement by the researchers, far fewer (27.5% of
that literature) described how they adjusted
methods in response to their personal hypotheses.
Researchers can examine other information as well
Additional Thoughts on Strengthening
such as the perspectives being revealed by research
Methodological Integrity
questions being asked (or not asked), or biases
Through this paper, guidance has been provided on present in research presentations. To provide an
how to use processes to improve methodological example of a study that examined the assumptions
integrity in meta-analyses and how to adapt pro- of primary researchers, in that same meta-analysis
cedures in relation to researchers’ goals. As on clients’ experiences, the researchers found that
described, to strengthen the fidelity of their findings the primary researchers tended to ask questions
with the primary literature, meta-analysts will wish about either professional power or cultural power
to consider the adequacy of their data in relation to but quite rarely about the intersection of the two—
the scope of their research question. They can revealing that researchers appeared to be motivated
increase the groundedness of their findings by generat- by one interest or the other, but tended to ignore
ing units and initial categories that are faithful to the the intersection of these types of power—generating
primary findings under analysis. There are two com- a limitation in the literature. Another way meta-ana-
ponents of fidelity that have not yet been discussed, lysts might examine researcher perspectives is to
however, which bear consideration. adopt the approach used in quantitative analyses of
The first is the management of the researchers’ per- investigator allegiance in which publications are
spectives during data collection. Throughout the searched for identification of authors’ expertise in
course of qualitative meta-analyses, researchers or bias toward a specific hypothesis (e.g., Munder,
being aware of their hopes and expectations can Gerger, Trelle, & Barth, 2011; Spielmans & Flücki-
allow them to consider how to collect data in a way ger, 2018).
that tests these expectations rather than build upon Through this paper, procedures that support the
their assumption. Researchers may use a variety of utility of meta-analytic findings have been indicated
methods to assist them in keeping this awareness as well. Contextualization can be useful when present-
forefront. These include using techniques such as ing quotes, when describing the location and time-
memo-ing, note-taking, and discussing expectations period of the data collection in the primary literature
and how to limit their effects within a research in the method section, and when describing limit-
team. In addition, meta-analysts strengthen fidelity ations of the meta-analysis in the discussion section.
by managing their perspectives during data analysis. By contextualizing their research, they assist readers
This might occur in the same manner as during to better apply findings in their own contexts.
data collection, but researchers also consider how Researchers are more likely to generate insight when
they employ viewpoints to increase the perspicacity they avoid jargon and code primary findings in such
of their analyses. For instance, researchers conduct- a way that they can be compared meaningfully
ing theory-driven analyses or using critical epistem- across studies. By reconciling conflicting findings,
ologies (e.g., feminist-multicultural approaches) meta-analysts can create coherent findings that can
might instead deliberately use a certain theoretical provide clarity in areas that had been complex and
perspective to structure their analysis. Regardless of confusing. Also, by generating meaningful contri-
whether researchers seek to limit or apply a perspec- butions, in relation to the research goals that are set
tive in their analysis, they increase fidelity by being forth, meta-analytic researchers can provide
 Psychotherapy Research 11

enlightening information to address a wide variety of methodological integrity. Qualitative Psychology, 4, 2–22.
aims. Rather than confining their research design to doi:10.1037/qup0000082
Levitt, H. M., Pomerville, A., Surace, F. I., & Grabowski, L. M.
predetermined sets of procedures, by considering (2017). Meta-method study of qualitative psychotherapy
these components of methodological integrity as research on clients’ experiences: Review and recommendations.
they design research, meta-analysts can adapt Journal of Counseling Psychology, 64, 626–644. doi:10.1037/
methods to support their goals, just as reviewers can cou0000222
use these ideas in the evaluation process. Levitt, H. M., Pomerville, A., & Surace, F. I. (2016). A qualitative
meta-analysis examining clients’ experiences of psychotherapy:
A new agenda. Psychological Bulletin, 142, 801–830. doi:10.
1037/bul0000057
Acknowledgment Levitt, H. M., Stanley, C. M., Frankel, Z., & Raina, K. (2005). An
evaluation of outcome measures used in humanistic psychother-
The author thanks Zenobia Morrill for her comments apy research: Using thermometers to weigh oranges. The
on a draft of this article. Humanistic Psychologist, 33, 113–130. doi:10.1207/
s15473333thp3302_3
Levitt, H. M., Surace, F. I., Wu, M. B., Chapin, B., Hargrove, J.
G., Herbitter, C., … Hochman, A. L. (2017). The Meaning of
Scientific Objectivity and Subjectivity: Methodologists’
References
Perspectives. Manuscript under review.
Barnett-Page, E., & Thomas, J. (2009). Methods for the synthesis Levitt, H. M., Grabowski, L. M., & Minami, T. (2018). The client
of qualitative research: A critical review. BMC Medical Research critical experiences in therapy scale (CETS). Unpublished
Methodology, 9, 1–11. Retrieved from https://www.ncbi.nlm.nih. manuscript, University of Massachusetts Boston.
gov/pmc/articles/PMC3224695/ Li, M. J., Black, D. S., & Garland, E. L. (2016). The Applied
Critical Appraisal Skills Programme. (2017). CASP Qualitative Mindfulness Process Scale (AMPS): A process measure for
Checklist. [online] Available at: http://www.casp-uk.net/ evaluating mindfulness-based interventions. Personality and
checklists. Individual Differences, 93, 6–15. doi:10.1016/j.paid.2015.10.027
Elliott, R., Fischer, C. T., & Rennie, D. L. (1999). Evolving guide- Ma, N., Roberts, R., Winefield, H., & Furber, G. (2015). Utility of
lines for publication of qualitative research studies in psychology qualitative metasynthesis: Advancing knowledge on the well-
and related fields. British Journal of Clinical Psychology, 38, 215– being and needs of siblings of children with mental health pro-
229. doi:10.1348/014466599162782 blems. Qualitative Psychology, 2, 3–28. doi:10.1037/
Giorgi, A. (1970). Psychology as a human science: A phenomenologi- qup0000018
cally based approach. New York, NY: Harper and Row. McGillivray, J., Gurtman, C., Boganin, C., & Sheen, J. (2015).
Giorgi, A. (2009). The descriptive phenomenological method in psy- Self-practice and self-reflection in training of psychological
chology: A modified Husserlian approach. Pittsburgh, PA: interventions and therapist skills development: A qualitative
Duquesne University Press. meta-synthesis review. Australian Psychologist, 50, 434–444.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded doi:10.1111/ap.12158
theory: Strategies for qualitative research. London: Wiedenfeld Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009).
and Nicholson. Preferred reporting items for systematic reviews and meta-ana-
Jennings, L., Sovereign, A., Renninger, S., Goh, M., Skovholt, T. lyses: The PRISMA statement. Journal of Clinical Epidemiology,
M., Lakhan, S., & Hessel, H. (2016). Bringing it all together: A 62, 1006–1012. doi:10.1016/j.jclinepi.2009.06.005
qualitative meta-analysis of seven master therapists studies from Munder, T., Gerger, H., Trelle, S., & Barth, J. (2011). Testing the
around the world. In L. Jennings, T. M. Skovholt, L. Jennings, allegiance bias hypothesis: A meta-analysis. Psychotherapy
& T. M. Skovholt (Eds.), Expertise in counseling and psychother- Research, 21, 670–684. doi:10.1080/10503307.2011.602752
apy: Master therapist studies from around the world (pp. 227– Noblit, G. W., & Hare, R. D. (1988). Meta-ethnography:
273). New York, NY: Oxford University Press. Synthesizing qualitative studies. Newbury Park, CA: Sage.
Kearney, M. H. (1998). Truthful self-nurturing: A grounded Paterson, B. L., Thorne, S. E., Canam, C., & Jillings, C. (2001).
formal theory of women’s addiction recovery. Qualitative Health Meta-study of qualitative health research: A practical guide to
Research, 8, 495–512. doi:10.1177/104973239800800405 meta-analysis and meta-synthesis. Thousand Oaks, CA: Sage.
Knowles, S. E., Toms, G., Sanders, C., Bee, P., Lovell, K., Rennie, D. L. (2000). Grounded theory methodology as methodi-
Rennick-Egglestone, S., … Bower, P. (2014). Qualitative cal hermeneutics: Reconciling realism and relativism. Theory &
meta-synthesis of user experience of computerised therapy for Psychology, 10, 481–502. doi:10.1177/0959354300104003
depression and anxiety. Plos ONE, 9. doi:10.1371/journal. Sandelowski, M., & Barroso, J. (2003). Creating metasummaries
pone.0084323 of qualitative findings. Nursing Research, 52, 226–233.
Levitt, H. M., Bamberg, M., Creswell, J. W., Frost, D., Josselson, Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/
R., & Suárez-Orozco, C. (in press). Journal article reporting 12867779
standards for qualitative research in psychology: The APA pub- Sandelowski, M., Docherty, S., & Emden, C. (1997). Focus on
lications and communications board task force report. American qualitative methods. Qualitative metasynthesis: Issues and tech-
Psychologist. doi:10.1037/amp0000151 niques. Research in Nursing & Health, 20, 365–371. Retrieved
Levitt, H., Butler, M., & Hill, T. (2006). What clients find helpful from https://www.ncbi.nlm.nih.gov/pubmed/9256882
in psychotherapy: Developing principles for facilitating Shadish, W. R. (1986). Planned critical multiplism: Some elabor-
moment-to-moment change. Journal of Counseling Psychology, ations. Behavioral Assessment, 8, 75–103. Retrieved from http://
53, 314–324. doi:10.1037/0022-0167.53.3.314 psycnet.apa.org/record/1987-00075-001
Levitt, H. M., Motulsky, S. L., Wertz, F. J., Morrow, S. L., & Spielmans, G. I., & Flückiger, C. (2018). Moderators in psy-
Ponterotto, J. G. (2017). Recommendations for designing and chotherapy meta-analysis. Psychotherapy Research. doi:10.1080/
reviewing qualitative research in psychology: Promoting 10503307.2017.1422214
12 H. M. LEVITT
Swift, J. K., & Wampold, B. E. (2018). Inclusion and exclusion Timulak, L. (2013). Qualitative meta-analysis. In U. Flick (Ed.),
strategies for conducting meta-analyses. Psychotherapy The Sage handbook of qualitative data analysis. (pp. 481–495).
Research. doi:10.1080/10503307.2017.1405169 Thousand Oaks, CA: Sage.
Timulak, L. (2007). Identifying core categories of client identified Timulak, L., & Creaner, M. (2010). Qualitative meta-analysis of
impact of helpful events in psychotherapy: A qualitative meta- outcomes of person-centred/experiential therapies. In M.
analysis. Psychotherapy Research, 17, 310–320. doi:10.1080/ Cooper, J. C. Watson, & D. Holledampf (Eds.), Person-centred
10503300600608116 and experiential psychotherapies work (pp. 65–90). Ross-on-
Timulak, L. (2009). Meta-analysis of qualitative studies: A tool for Wye: PCCS Books.
reviewing qualitative research findings in psychotherapy. Wertz, F. J. (2014). Qualitative inquiry in the history of
Psychotherapy Research, 19(4–5), 591–600. doi:10.1080/ psychology. Qualitative Psychology, 1, 4–16. doi:10.1037/
10503300802477989 qup0000007