The Kirkpatrick model for training

evaluation: bibliometric analysis after 60

years (1959–2020)
Aljawharah Alsalamah and Carol Callinan

Aljawharah Alsalamah and Abstract

Carol Callinan are both Purpose – A number of studies on Kirkpatrick’s four-level training evaluation model have been
based at the University of published, since its inception in 1959, either investigating it or applying it to evaluate the training process.
Lincoln, Lincoln, UK. The purpose of this bibliometric analysis is to reconsider the model, its utility and its effectiveness in
meeting the need to evaluate training activities and to explain why the model is still worth using even
though other later models are available.
Design/methodology/approach – This study adopts a ‘‘5Wsþ1H’’ model (why, when, who, where, what
and how); however, ‘‘when’’ and ‘‘how’’ are merged in the methodology. A total of 416 articles related to
Kirkpatrick’s model published between 1959 and July 2020 were retrieved using Scopus.
Findings – The Kirkpatrick model continues to be useful, appropriate and applicable in a variety of
contexts. It is adaptable to many training environments and achieves high performance in evaluating
training. The overview of publications on the Kirkpatrick model shows that research using the model is an
active and growing area. The model is used primarily in the evaluation of medical training, followed by
computer science, business and social sciences.
Originality/value – This paper presents a comprehensive bibliometric analysis to reconsider the model,
its utility, its effectiveness in meeting the need to evaluate training activities, its importance in the field
measured by the growth in studies on the model and its applications in various settings and contexts.
Keywords Kirkpatrick model, Bibliometric analysis, Training, Evaluation
Paper type literature review

Introduction
One of the best-known and most widely used models for evaluating training and
development programs is Kirkpatrick’s (1959) four-level evaluation model (the four levels
being a reaction, learning, behavioral and results). Between November 1959 and February
1960, Kirkpatrick published a series of four articles titled “Techniques for evaluation of
training programs” in the Journal of the American Society of Training Directors. Developed
for use within business organizations (Bates, 2004), the model has received much attention
from researchers and organizations (Lien et al., 2007), as it provides a sound foundation for
the examination of the effect of training on organizations (Watkins et al., 1998). Over the
years, it has evolved to become one of the most widely accepted and influential models for
the assessment of training and development programs in a range of settings (Phillips, 1991;
Bassi and Cheney, 1997; Tamkin et al., 2002). It has also formed the basis of other
approaches to the assessment of training and development; as Bates (2004, p. 342) states,
“[t]he Kirkpatrick model has been the seed from which a number of other evaluation models
Received 17 December 2020 have germinated.”
Revised 9 March 2021
22 June 2021
21 July 2021
As 60 years have now passed, as Kirkpatrick created his model, this bibliometric study
Accepted 28 July 2021 intends to reconsider the model, its utility, its effectiveness in meeting the need to evaluate

PAGE 36 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022, pp. 36-63, © Emerald Publishing Limited, ISSN 0019-7858 DOI 10.1108/ICT-12-2020-0115
training activities, its importance in the field measured by the growth in studies on the model
and its applications in various settings and contexts. Furthermore, a bibliometric analysis of
business and training journals will provide better insight for scholars in the field. The paper
also examines the criticisms leveled at the model.
Bibliometrics, which uses scientometric methods, is defined as “the application of
mathematical and statistical methods to books and other means of communication, which
are mainly in charge of the management of libraries and documentation centers”
(Repanovici, 2010, p. 2). Typically used within the field of library and information science
(Repanovici, 2010), a bibliometric review is a type of domain-based review and a type of
systematic literature review (Paul and Criado, 2020). It involves analysis of a large number
of different types of publications using statistical tools to explore specific research areas for
current trends and for citations and/or co-citations; the scope of the analysis can be
restricted by several criteria, such as spatial factors (e.g. country), temporal factors (e.g.
year), author, journal, method, theory and research problem (Paul and Criado, 2020).
The process of defining questions in mapping a study is ill-defined and guidance is needed
to define the research questions (Jia et al., 2016). To identify a starting point and the key
elements of the research to achieve the study’s aim, this study adopted a “5Ws þ1H” model
(why, when, who, where, what and how), commonly used in journalism to discover the most
important aspects of a story (Kipling, 1912). In this case, the model was used to explore
how the Kirkpatrick model has been applied. The 5Ws þ 1H model uncovers essential
information that needs to be considered for any story to be told, namely, motivation, time,
actor, location, content and causality (Jia et al., 2016). A helpful starting point for these
questions is shown below as follows (Tattersall, 2015a, 2015b; Jia et al., 2016; Calero et al.,
2020):
䊏 Why: why was this research conducted? Why was there a need for it?
䊏 When: when was this study conducted? When did the project start and when did it
finish?
䊏 Who: who conducted the research?
䊏 Where: where were the results published?
䊏 What: what were the results of this research?
䊏 How: how was this study conducted?
Various studies have performed bibliometric analysis using a 5Ws approach for information-
gathering or problem-solving. For example, Calero et al. (2020) conducted a bibliometric
analysis of green and sustainable software using a 5Ws or 5Ws þ 1H approach (Galindo
et al., 2019; Bibi et al., 2020) and Rahimi and Rosman (2020) used the 5Ws approach to
investigate enterprise content management.
For the purpose of this study, the when and how questions are related, showing the time of
the study and the mechanism adopted; therefore, when and how were merged. This is a
valid approach that has been adopted by other authors, such as Hart (1996), who reports
that the sixth item, how, can be rephrased or merged with any of the 5 W questions. The
approach allows researchers to explore key subjects while retaining control of what they say
and how they say it. The use of this approach may also help researchers to present their
findings to an audience without much effort (Tattersall, 2015a, 2015b).

Literature review
Related work
The goal of a systemic literature review (SLR) is to identify and evaluate relevant research to
answer a specific research question (Papamitsiou and Economides, 2014). Several SLRs

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 37

 have been conducted in the area of evaluation within different domains, one example being
the evaluation of health-care programs using Kirkpatrick’s model. Johnston et al. (2018)
conducted an SLR on studies using Kirkpatrick’s evaluation of simulation and debriefing in
health-care education while Campbell et al. (2019) conducted an SLR on studies
investigating the effectiveness of online cancer education for nurses and allied health
professionals, also using Kirkpatrick’s evaluation framework. However, to the best of our
knowledge, no SLR has been conducted using only the Kirkpatrick model within the field of
social sciences. The above-mentioned studies focused on health care and their findings
cannot be generalized to all fields. As Garousi and Mäntylä (2016, p. 403) note, “this kind of
study is needed regularly in the quest to keep up with the most recent research
developments.” Consequently, a bibliometric analysis of studies on the Kirkpatrick model
was performed to obtain the information needed to answer the 5Ws.

Kirkpatrick model
Between November 1959 and February 1960, Kirkpatrick published four articles titled
“Techniques for evaluation of training programs.” The purpose of Kirkpatrick’s model was to
provide an efficient means and systematic way for managers to evaluate training outcomes
among employees and in organizational systems (Cahapay, 2021). The Kirkpatrick model
comprises the following four levels: Level 1 – Reaction, which assesses participants’
satisfaction and interest in the training; Level 2 – Learning, which assesses the extent of
skills and knowledge gained; Level 3 – Behavioral, which measures trainees’ ability to apply
learned knowledge and skills in the workplace; and Level 4 – Results, which measures the
effect of training on the organization.
The Kirkpatrick model has inspired the development of several other assessment models
(Bates, 2004; Holton, 1996; Kaufman et al., 1995). Therefore, most of the evaluation models
found in the literature are based on the Kirkpatrick model (Holton, 1996; Nickols, 2005; Reio
et al., 2017).
However, the ascending order of the value of its levels and the assumptions surrounding
causality in the Kirkpatrick model have been criticized (Alliger and Janak, 1989; Bernthal,
1995; Clement, 1982). According to Giangreco et al. (2010), the implication of the first
criticism is that behavioral change has higher significance than positive reactions and a
positive outcome at Level 4 is the ultimate goal of every training program. Thus, the model is
hierarchical in nature, with its four levels arranged in ascending order (Reio et al., 2017).
However, in practice, an outcome is not necessary at each level in the training program; for
example, those designed to instill company pride in employees can only be expected to
have an output (that is, impact) at Level 1 (Reaction) while those involving learning about
organizational history and philosophy may only be evaluated at Level 2 (Learning) (Alliger
and Janak, 1989). Therefore, studies and empirical results do not furnish sufficient proof to
support the assumption that every subsequent level presents more valuable information
than the previous one (Bates, 2004).
The second assumption is the causal chain between levels, with a pragmatic response
required for learning to occur and learning being essential for the transfer (Holton, 1996).
However, Levels 1 and 2 can be evaluated simultaneously (Alliger and Janak, 1989),
suggesting that one of the levels does not necessarily have an impact on the other.
Furthermore, learning can often be difficult and methods of ensuring effective learning may
be uncomfortable for participants (Knowles, 1980; Rodin and Rodin, 1973). However,
research has found that an unpleasant experience in training can somewhat encourage
learning (Rodin and Rodin, 1973), which suggests the possibility of a negative correlation
between Levels 1 and 2. Moreover, a fun lecture does not necessarily cause more learning
(Kaplan and Pascoe, 1977), as good reactions do not imply good learning (Rowold, 2007).
Therefore, it could be argued that there is a relationship between the levels, but that this
relationship is very complex. The issue of causality, therefore, remains controversial within

PAGE 38 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

the research community. Moreover, some researchers argue that Kirkpatrick’s model
oversimplifies training effectiveness by concentrating only on the results while ignoring the
processes (Aldrich, 2002; Bates, 2004); therefore, its levels are incomplete (Kaufman et al.,
1995) and incomprehensive, as they do not account for the many factors and contexts that
influence training effectiveness in organizations (Bates, 2004). For example, the Kirkpatrick
model focuses only on the outcomes that occur after the training and ignores other aspects
of the learning process (Bushnell, 1990). To be effective, a training evaluation must consider
the organizational and work environment and the characteristics of individual trainees
(Cannon-Bowers and Salas, 1997). Giangreco et al. (2010) argue that this criticism is
unfounded, as training interventions are based on distinct objectives and designs, which
may place higher significance at distinct levels of Kirkpatrick’s model.
Despite the criticism of the Kirkpatrick model, it is still widely used. Many researchers and
practitioners have found it very useful (Phillips, 2003) and it is the most commonly used
model in organizations (Bernardino and Curado, 2020; Jain et al., 2021). Kirkpatrick’s four
levels of evaluation constitute one of several viable tools available to evaluate the
effectiveness of training. Although some have attempted to develop alternatives (Holton,
1996; Kaufman et al., 1995), the Kirkpatrick model remains the standard in the business
and education realms (Waddill, 2006).
However, the shortcomings identified have driven researchers to initiate creative studies to
enhance the existing understanding of the optimal evaluation design and its application
(Reio et al., 2017). Based on criticisms of the Kirkpatrick model, various other models have
been developed (Warr et al., 1970; Stufflebeam, 1983; Brinkerhoff, 1987; Bushnell, 1990;
Kraiger et al., 1993; Holton, 1996; Kaufman et al., 1995; Phillips, 1996). These extended
models on training effectiveness are still considerably influenced by Kirkpatrick’s four-level
training model (Jain et al., 2021). Moreover, according to Tamkin et al. (2002), some can be
considered direct descendants of Kirkpatrick’s model, as they adopt much from the original
model and extend it at the front, by including training design or needs analysis, at the back,
by adding evaluation of societal outcomes or, sometimes, at both ends. Some models
extend it with the addition of an evaluation of the return on investment (ROI), but this can be
included within the Results level (Level 4).
Therefore, this bibliometric study intends to reconsider the model, the extent of its use and
activity in the field and to investigate its usefulness and its features through the growth of its
applications in various settings and contexts.

Formulating 5Ws þ 1H
In this study, the researchers developed their own specific research questions, each one
corresponding to the appropriate “W” or “H.” Here, we present the research questions and
then answer the “why” question. The answers to the remaining questions on research into
the use of Kirkpatrick’s model are provided in the methodology and results in sections.

Research questions
Why
RQ1. Why is the field of research relevant?
When and how
RQ2. When was this study conducted? How?
Who
RQ3. Who are the main authors in this area?
RQ4. Which are the main institutions in this area?

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 39

 Where
RQ5. Which countries/regions produce the majority of publications on the Kirkpatrick
model?
RQ6. Which journals and conferences are the most effective (in terms of the number of
publications) for sharing information about the Kirkpatrick model?
What
RQ7. What are the most common keywords in publications?
RQ8. What are the most relevant domains in which the Kirkpatrick model is used?
RQ9. What types of studies use the Kirkpatrick model?
RQ10. What were the results of these studies on the model? Is the model applicable and
useful?

Answering the ‘‘why’’ question

RQ1. Why is the field of research relevant?
The purpose of this study is to conduct a bibliometric analysis of research using the
Kirkpatrick model to reconsider the model, its utility and its effectiveness in meeting the
need to evaluate training activities, as well as to determine how its use has evolved over
the past 60 years in all fields. The focus is on the model’s relevance to formal education in
schools in the social science domain. Therefore, the study will first present a data analysis
related to all fields of study and then focus on the field of social sciences.

Methodology (“when”)
RQ2. When was this study conducted? How?
There are many multidisciplinary bibliographic databases, including Scopus, Web of
Science (WoS), Google Scholar (Balstad and Berg, 2020; Wei, 2020) and Dimensions (van
den Besselaar and Sandström, 2020). In this study, the Scopus, Dimensions and WoS
databases were selected as sources of publications related to the study. Google Scholar
was omitted for several reasons. First, it does not always produce consistent search results,
as its indexing methods are not as strict as those of other databases, such as Scopus or
WoS (Shareefa and Moosa, 2020). Second, its search results cannot be downloaded, unlike
those from other databases, for example, WoS or Dimensions. Third, there is no application
programming interface (API) that can be used for search purposes (Moral-Muñoz et al.,
2020). Therefore, three databases were used to retrieve bibliometric data sets.
The research methodology is presented in Figure 1. The researchers set different
parameters for the search. Initially, it covered all languages to retrieve all publications
focusing on the search area. Similarly, in the first instance, the study focused on all subject
domains but later narrowed this to social sciences; however, a bibliometric analysis of both
areas (general and social sciences) was performed. The date was set to the default, which
showed no limits in terms of date. The focus in the search fields was set to title, abstract and
keywords for all the index databases.
The choice of keywords to be used in the search was based on different considerations to
obtain the highest degree of accuracy in search results. The study focuses on the
Kirkpatrick model and its use as a tool for assessment or evaluation. As the Kirkpatrick
model has been called “a model, system, framework, taxonomy, methodology, typography
and a vocabulary” (Holton, 1996, p. 50), all these terms were included within the search
string, as shown below.

PAGE 40 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

 Figure 1 Methodology of the study

An electronic search of the selected databases (Scopus, Dimensions and WoS) was
performed on July 14, 2020, using the keywords in Figure 2: String 1. The records from all
the databases can be seen in Figure 3, which shows that the highest number of records
was retrieved from the Scopus database. The chronological order of scientific publications
within these databases can be seen in Figure 4. Vı́lchez-Roma
n et al. (2020) give three
reasons for the superiority of Scopus over other databases as follows: Scopus holds twice
as many indexed journals as other databases (Shareefa and Moosa, 2020), as seen in
Figure 3; it includes non-English indexed sourcing; and previous bibliometric analysis
studies support the superiority of Scopus over other databases.
Several tools were used to analyze or visualize the data set retrieved from the Scopus
database. Two bibliometric programs were used as follows: Bibliometrix (Version 3.0) and
VOSviewer. Bibliometrix is an open-source program in the R language that was developed
to conduct a comprehensive quantitative science mapping analysis (Aria and Cuccurullo,
2017). It uses metadata from indexing databases, such as Scopus, WoS, Dimensions and
PubMed, to survey publications. Bibliometrix helps to calculate, evaluate and rank co-
citation, co-authorship, co-occurrence and other types of measures on metadata. Its results
show different types of outputs, such as co-wording of keywords, abstract, scientific
publications by country, publication sourcing and citations and network collaborations in
terms of countries and authors. It uses various types of algorithms to conduct these data-
mining techniques (Moral-Muñoz et al., 2020).

Figure 2 String 1 – keywords

(("Kirkpatrick model" OR "Kirkpatrick system" OR "Kirkpatrick framework" OR

"Kirkpatrick four level" OR "model of Kirkpatrick" OR "framework of Kirkpatrick" OR
"four level of Kirkpatrick" OR "system of Kirkpatrick " OR "Kirkpatrick's categories" OR
"Kirkpatrick taxonomy" OR " Kirkpatrick methodology" OR " Kirkpatrick typography" OR
"Kirkpatrick vocabulary" OR "typography of Kirkpatrick" OR "vocabulary of Kirkpatrick"
OR "methodology of Kirkpatrick" OR "taxonomy of Kirkpatrick" ) AND ( "train*" OR
"evaluat*" OR "assess*" ))

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 41

 Figure 3 Scientific publications related to Kirkpa`trick’s model in three databases (Scopus,
Dimensions and WoS)

Figure 4 Number of publications (1972–2020) for the nine most commonly used models

VOSviewer is a software tool that creates maps based on network data and visualizes and
explores these data (Eck and Waltman, 2020). It uses metadata from indexing databases,
such as Scopus, WoS and others, to show co-citation, bibliographic coupling, co-
authorship and co-occurrence using clustering methods; it also has the advantage over
other bibliometric tools of visualization (Moral-Muñoz et al., 2020). Other tools, such as MS
Excel and Google Sheets, were also used for data visualization.

Results (“how,” “who,” “where,” “what”)

The first step in this study was a comparison of the number of publications related to
Kirkpatrick’s model and the number of publications for other models referred to in the
literature review, that is, the commonly used models, as suggested by Kumpikaite_ (2007),
Chang (2010), Werner and De Simor (2012), Topno (2012) and Reio et al. (2017). Similarly,
other models were searched on the Scopus database using the same keywords [1] that we
used in the search for the Kirkpatrick model, but with changes to the model name and
number of levels. Figure 4 shows the records retrieved for each model.

PAGE 42 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

However, in the first search run for the Phillips model (1996), there were 38 retrieved
records. The first record was published in 1972 and it seemed unlikely that this would be
relevant. Therefore, these publications had to be examined manually to obtain detailed
data. After reviewing them, 29 studies were excluded, as they related to the
Bragg–Rice–Phillips model for the complex dielectric constant of oceans, the Phillips curve
model in economics, the Phillips and Van Vechten dielectric theory and the Phillips–Heffron
model for power electronics. Nine studies related to the Phillips model were retained as they
referred to evaluating training and the first study to discuss this model was published in
2005. Moreover, there were 425 retrieved records that related to Kirkpatrick’s model. These
were examined manually to ensure that they were within the study’s scope. After reviewing
them, nine studies related to the Sherrington–Kirkpatrick model of spin glasses were
excluded. The remaining 416 studies related to Kirkpatrick’s model were retained, as they
referred to evaluating training and the first study on this model was from 1984. A scan of
results for other models showed that they matched the keywords correctly.
Figure 4 shows two significant features. The first is that the results only go back to the
1970s, 1980s or 1990s; however, this does not indicate that the model was not used in this
period but may indicate that the main interest in publishing articles on the evaluation of
training started after 2000. The second is the considerable difference in the number of
publications on Kirkpatrick’s model and other evaluation models, which indicates the high
volume of research activity using the Kirkpatrick model; this suggests that it is effective for
evaluating training despite the existence of other evaluation models that can assess training
and its effects. This result may confirm features of the model that made it a widely accepted
and commonly used model. In contrast, most existing models for evaluating training are too
complicated, costly to implement and time-consuming (Jain et al., 2021). Warr et al.’s model
(1970) consisting of context, input, reaction and outcome (CIRO) focuses on conducting
measurement both before and after the training has been carried out (Topno, 2012);
however, it does not define how these measurements should be carried out (Tzeng et al.,
2007). While the reaction level is similar to that of Kirkpatrick’s model in that the views and
suggestions of participants concerning the training program are elicited, the CIRO model
focuses more on participant suggestions to alter the format of the training. The final level,
outcome, looks at the results of the training on an immediate, intermediate and final level;
this is similar to Kirkpatrick’s learning, behavioral and results in levels (Tamkin et al., 2002;
Phillips, 2003). Thus, the CIRO model borrows heavily from Kirkpatrick’s model (Jain et al.,
2021).
Stufflebeam’s (1983) model consists of context, input, process and product (CIPP) to carry
out a reliable evaluation. However, multiple data collection methods are usually required
and each data set must be analyzed using methods appropriate to the data and the
evaluation questions that are addressed (Frye and Hemmer, 2012). Furthermore, at the
planning stage, evaluators should consider the available time and resources. If this model
requires more time or resources than are available, another model may need to be
considered (Fitzpatrick et al., 2003).
The six stages of Brinkerhoff’s (1987) model are goal setting, program design, program
implementation, immediate outcomes, intermediate or usage outcomes and impacts and
worth. Brinkerhoff’s Stage 3 is similar to Kirkpatrick’s Level 1 and Brinkerhoff’s Stage 4 is
identical to Kirkpatrick’s Level 2 (Reio et al., 2017). Moreover, Brinkerhoff’s Stage 5 is the
same as Kirkpatrick’s Level 3 and Brinkerhoff’s Stage 6 is the same as Kirkpatrick’s Level 4
(Kumpikaite,_ 2007; Reio et al., 2017). Thus, Brinkerhoff’s model is a cycle of overlapping
steps (Werner and De Simor, 2012). The model is appropriate in situations where there is no
pressure to meet deadlines or reduce the budget (Holton and Naquin, 2005; Passmore and
Velez, 2012).
Bushnell (1990) developed the input, process, output (IPO) model, which emphasizes the
input to training (Tamkin et al., 2002). Bomberger (2003) argues that the IPO model goes

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 43

 beyond the Kirkpatrick model, as it provides both formative and summative information to
show the worth of training in financial terms (Reio et al., 2017). However, this model does
not provide sufficient information on program function and does not take into account the
factors that may influence the results (Passmore and Velez, 2012). Furthermore, its
application is largely theoretical rather than practical (Topon, 2012).
In Kraiger et al. (1993) model, they define learning as changes in cognitive outcomes, skill-
based outcomes or affective outcomes (Colquitt et al., 2000; Salas and Cannon-Bowers,
2001) and they propose that training evaluation should address these three outcomes. It
can be seen that the outcomes of this model are similar to Kirkpatrick’s learning, behavioral
and reactions criteria, respectively, (Tan et al., 2003). Kraiger et al. (1993) model provide a
solid theoretical basis on which to measure learning outcomes; however, it does not provide
a methodology for the measurement of mental models. Moreover, it does not provide
guidance for determining the financial value or cost-effectiveness of training and it focuses
on the effects of training on attendees while neglecting the effects on the organization.
Finally, it provides little opportunity to incorporate the subjective views of trainers or trainees
into the evaluation (Beech and Leather, 2006).
Kaufman and Keller (1994) proposed an extended version of Kirkpatrick’s model that
includes an evaluation of the impact of training beyond the organization. Their model also
considers societal impact, consumer satisfaction and the ways in which training benefits the
surrounding environment in the organization (Topno, 2012; Reio et al., 2017; Jain et al.,
2021). However, compared to other models, Kaufman and Keller’s is more theoretical in
focus and provides less in the area of practical application (Topno, 2012).
Phillips’ (1996) model expanded Kirkpatrick’s Level 4 (Results) and suggested the addition
of a fifth level called ROI. In this level, the data obtained in Kirkpatrick’s Level 4 are
converted into financial values and then compared with the cost of the training program
(Beech and Leather, 2006). However, Topno (2012) argues that Kirkpatrick and other
theorists have suggested that this fifth level (ROI) can easily be included in Kirkpatrick’s
original Level 4: Results. Therefore, the inclusion of the fifth level is only necessary if the
assessment of ROI might otherwise be ignored or forgotten when referring to the Results
level. Moreover, Russ-Eft and Preskill (2005) argue that defining ROI is a complex,
multifaceted task within a complex system and suggest that it has not produced accurate
measurements of investments for training. Therefore, ROI is difficult to measure (Watkins
et al., 1998).
Holton (1996), who proposed the human resources development (HRD) model, suggests
three outcome levels as follows: learning, individual performance and organizational
performance (Holton, 2005). These three outcomes are similar to Kirkpatrick’s Levels 2, 3
and 4. The missing outcome is the first level, Reaction (Holton, 1996, 2005).
While Holton’s model focuses on the expected outcomes of training and the influences that
promote or prevent their achievement, it has not been used as widely as the Kirkpatrick
model (Bomberger, 2003; Reio et al., 2017). This model cannot be used frequently, as it
may complicate the evaluation process to the point where practitioners have neither the
money nor the time to conduct extensive evaluations (Antos and Bruening, 2006).
Consequently, further refinement and testing are needed (Werner and De Simor, 2012).
Therefore, the results below present only the studies related to Kirkpatrick’s model.
The analysis of the extracted study data set shows that many types of publications,
including articles, conference papers and reviews, constitute the majority of the
publications on the Kirkpatrick model, both in all fields and in social sciences, as Table 1
shows. The majority of publications in all fields are articles (71.6%), followed by conference
papers (13.9%). In social sciences, the majority of publications are articles (72.8%),
followed by reviews (15.4%). There is no balance between the number of conference

PAGE 44 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

 Table 1 Main forms of publication relating to the Kirkpatrick model
All fields Social sciences
No. Form of publication Publication (%) Publication (%)

1 Article 298 71.6 123 72.8

2 Conference paper 58 13.9 16 9.5
3 Review 49 11.8 26 15.4
4 Book chapter 9 2.2 3 1.8
5 Short survey 1 0.2 0 0
6 Undefined 1 0.2 1 0.6

publications and journal publications. The majority of publications (more than 70%) are
journals articles, as Table 1 shows.
The growth in the number of publications in all fields, including social sciences, is shown in
Figure 5. This indicates the increased interest of researchers and publishers, in the
Kirkpatrick model for evaluating training programs. For social sciences, 1998 appears to be
the starting point for such research.
The annual growth rate (AGR) refers to the total number of publications that appeared
compared with the previous year. It is calculated using the number of publications in one
year and the number of publications from the previous year, as shown in equation (1):

!
N:publicationYear  N:publicationprevious Year
AGR ¼  100 (1)
N:publicationprevious Year

Table 2 shows the AGR of our data in all fields and in social sciences for a nine-year period
from 2011 to 2019 (we have removed 2020 since the year had not finished when the search
was conducted). As can be seen, for all fields, the positive values are maintained except in
2013 when there was a decrease (21.7%). Similarly, in the social sciences field, the positive
values are maintained except for two years in which they decreased, namely, 2013 (27.3%)
and 2016 (46.7%). This indicates that the research on Kirkpatrick’s model or its application
is an active area and is growing in all fields, including social sciences.
To estimate the trend in this field in the future and predict the number of publications in
coming years, the least squares method was used, as shown in equation (2):

Figure 5 Growth in publications relating to the Kirkpatrick model by year

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 45

 Table 2 AGR of publications relating to the Kirkpatrick model
General Social sciences
Year Publication AGR Publication AGR

2019 68 28.3 27 80.0

2018 53 55.9 15 0.0
2017 34 3.0 15 87.5
2016 33 10.0 8 46.7
2015 30 36.4 15 36.4
2014 22 22.2 11 37.5
2013 18 21.7 8 27.3
2012 23 27.8 11 22.2
2011 18 5.9 9 50.0

 P  P !
Y ðX  Y Þ
Y ¼ P þ P 2 X (2)
NYears X

A straight line was calculated using the data for the period 2008–2019, where Y is the
number of publications, N is the number of years and X is the base, which is assumed to be
zero. As per the analysis in Table 3, the estimate indicates that we would expect five
additional publications to be produced each year compared with the previous year in all
fields, including social sciences.

Who
This section presents statistics related to the authorship of papers on the Kirkpatrick model.
Therefore, it will answer RQ4 and RQ5.
RQ3. Who are the main authors in this area?
Table 4 shows the most prolific authors in general and in social sciences specifically, who
use or discuss the Kirkpatrick model. It shows that the most prolific authors in general are
Kumar, Liu, Reeves, Sotomayor, Steinert and Triviño, each with three papers; Reeves has a
further three papers in social sciences, making him the most prolific author in this field, as
Figure 6 confirms. Using VOSviewer to provide a graphical representation of the interaction

Table 3 Predicted trends in the publication using the least-squares method

All fields Social sciences
Publication Publication
Year Y X XY X2 trend Y X XY X2 trend

2025 21 83 21 29
2024 19 78 19 27
2023 17 73 17 26
2022 15 69 15 24
2021 13 64 13 23
2020 35 11 18 11
2019 68 9 612 81 27 9 243 81
2018 53 7 371 49 15 7 105 49
2017 34 5 170 25 15 5 75 25
2016 33 3 99 9 8 3 24 9
2015 30 1 30 1 15 1 15 1
2014 22 1 22 1 11 1 11 1
2013 18 3 54 9 8 3 24 9
2012 23 5 115 25 11 5 55 25
2011 18 7 126 49 9 7 63 49
2010 17 9 153 81 6 9 54 81

PAGE 46 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

 Table 4 Number of publications and number of authors
General Social sciences
Author No. of papers Author No. of papers

Kumar, A. 3 Reeves, S. 3
Liu, Z. 3 Abdulghani, H. M. 2
Reeves, S. 3 Ahten, S. M. 2
Sotomayor, T. M. 3 Baker, L. 2
Steinert, Y. 3 Barr, H. 2
Triviño, X. 3 Bezuidenhout, J. 2
Abdulghani, H. M. 2 Borduas, F. 2
Ahten, S. M. 2 Catalano, G. D. 2

Figure 6 Authors with the most co-authorship interaction on publications using the
Kirkpatrick model

of authors in social sciences, we can see that there are three clusters as follows: Cluster 1
comprises Birch, Boot, Davies, Fletcher, Kitto, McFadyen and Rivera; Cluster 2 comprises
Baker, Cameron, Egan-Lee, Esdaile, Friesen and Onyura; and Cluster 3 comprises Reeves
(with Barr, Freeth, Hammick and Koppel). We can conclude that Cluster 1 is the most
interrelated.
Figure 6 provides a graphical representation of co-authorship interaction on publications
using this model.
Table 5 shows the percentages of single-authored publications and multi-authored
publications by year over a 20-year period from 2000 to July 2020. The vast majority of
publications are multi-authored. Single-authored publications account for only 11.2% of the
total, the remaining 88.8% being multi-authored; this indicates a high degree of
collaboration among authors in this area.
RQ4. Which are the main institutions in this area?
An analysis of the most prolific institutions is shown in Table 6. Although papers on studies
using the Kirkpatrick model originate in several institutions in various countries, the three
most prolific institutions in publishing in all fields are also the most prolific institutions in
social sciences, which are the University of Toronto, the University of British Columbia and
the University of Alberta.

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 47

 Table 5 Single- and multi-authored publications by years
Year N. publication Single-authored publication (%) Multi-authored publication (%)

2020 35 0 0.0 35 100.0

2019 68 1 1.5 67 98.5
2018 53 1 1.9 52 98.1
2017 34 3 8.8 31 91.2
2016 33 2 6.1 31 93.9
2015 30 1 3.3 29 96.7
2014 22 1 4.5 21 95.5
2013 18 3 16.7 15 83.3
2012 23 3 13.0 20 87.0
2011 18 3 16.7 15 83.3
2010 17 6 35.3 11 64.7
2009 9 0 0.0 9 100.0
2008 14 3 21.4 11 78.6
2007 9 5 55.6 4 44.4
2006 4 1 25.0 3 75.0
2005 4 1 25.0 3 75.0
2004 3 2 66.7 1 33.3
2003 8 2 25.0 6 75.0
2002 4 1 25.0 3 75.0
2000 2 2 100.0 0 0.0

Table 6 Most prolific institutions with regard to publishing on the Kirkpatrick model
All fields Social sciences
No. of No. of
No. Institution publications (%) Institution publications (%)

1 University of Toronto 15 4.2 University of Toronto 8 3.7

2 University of British Columbia 7 1.9 University of British Columbia 6 2.8
3 University of Alberta 6 1.7 University of Alberta 6 2.8
4 Maastricht University 5 1.4 Stellenbosch University 3 1.4
5 Monash University 5 1.4 Dalhousie University 3 1.4
6 University of Sydney 5 1.4 Saint Michael’s Hospital, 3 1.4
University of Toronto
7 University of Oxford 5 1.4 Maastricht University 3 1.4
8 North China Electric Power 5 1.4 Monash University 3 1.4
University Baoding

Where
In this section, we will present the results of the analysis of publication types and countries.
RQ5. Which countries/regions produce the majority of publications on the Kirkpatrick
model?
Figures 7 and 8 show the distribution of scientific publications by country in all fields and in
social sciences, respectively.
As can be seen from both figures, the United States (USA) is the most prolific country and
the highest contributor to the body of knowledge in this area, followed by the United
Kingdom (UK), Canada and Australia. The Kirkpatrick model is well-established in the USA
and the abundance of publishing in the USA and the other countries listed above may be
attributed to the scientific progress in those countries, as well as to cooperation between
authors. Both figures reveal that the publication of scientific articles on the application of the
Kirkpatrick model in the evaluation of training is still in its initial stage of growth in the Middle

PAGE 48 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

 Figure 7 Scientific publications in all fields based on country

Figure 8 Scientific publications in social sciences based on country

East and Africa. It can be argued that the reasons some parts of the world still do not apply
this model may be cultural (e.g. research on the model was published in English and these
countries use their own language in research) or educational (e.g. there may be a
difference between developing and developed countries in terms of education and
experience). Furthermore, knowledge level and field maturity can make a significant
difference between region-specific contexts; for example, in developing countries, the
focus may be on fundamental issues to be solved, whereas, in developed countries, the
focus is usually on developing, promoting and enhancing systems in a specific context,
such as enhancing the evaluation of training in education, management, medicine or other
fields.
The analysis was taken a step further to determine the clustering and co-occurrences of
countries. This was done using VOSviewer, where parameters were set to “analysis type:
co-authorship” and “minimum number of country occurrences: five.” Regarding
collaboration among countries, results in both Figures 9 and 10 indicate that there are three
groups of countries. The USA, Australia and The Netherlands are in the first group; the UK,
Iran and India are in the second group; and Canada and South Africa are in the third group.

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 49

 Figure 9 Collaboration among countries in scientific publications

Figure 10 Collaboration among countries in scientific publications in social sciences

RQ6. Which journals and conferences are the most effective (in terms of the number of
publications) in sharing information about the Kirkpatrick model?
The analysis of the extracted data set shows that many types of publications, including
articles and conference papers, constitute the majority of the body of knowledge on the
Kirkpatrick model.
Table 7 shows the top 10 journals in all fields and in social sciences. According to Table 7,
the top journals, in general, were in the social science domain, with the evaluation of

Table 7 Most effective journals for sharing information about the Kirkpatrick model
All fields Social science
Forum N. publications Forum N. publications

Medical Teacher 14 Medical Teacher 14

BMJ Open, Medical Journal 9 BMC Medical Education 6
BMC Medical Education 6 Journal of Continuing Education in The Health Professions 6
Journal of Continuing 6 Journal of Dental Education 6
Education in The Health
Professions
Journal of Dental Education 6 Medical Education 4
Journal of Medical Internet 4 Academic Medicine 3
Research
Medical Education 4 Clinical Simulation in Nursing 3
Revista Medica De Chile 4 Journal of Education and Health Promotion 3
T and D 4 Journal of Surgical Education 3
Academic Medicine 3 Nurse Education in Practice 3

PAGE 50 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

medical programs the most common topic. This indicates that the most effective journals for
sharing information about the Kirkpatrick model are medical journals.
The results from the Scopus database highlighted a significant lack of scientific publishing
on the Kirkpatrick model in conference proceedings within all fields, which indicates that
there is no effective conference for publishing studies on the Kirkpatrick model in any field.
The most effective conference yielded only two papers, as shown in Table 8; the majority of
conference papers on the Kirkpatrick model tend to be within the field of information
technology.
Figure 11 compares the evolution of the number of papers published in general in journals
and in conference proceedings. The figure shows that the publication of conference papers
on the Kirkpatrick model started 16 years later than that of journal articles. There are still
very few conference papers on this topic and the number is not growing. On the other hand,
there is an upward trend in the publication of journal articles.
When we look specifically at publishing in social sciences in journals and conference
proceedings specifically, the data show that from around 2011, the publishing gap between
journals and conferences started to widen and journal papers took the lead over conference
papers by a significant margin, as shown in Figure 12.

What
In this section, we will present the results of the analysis of authors’ keywords and search
domains.
RQ7. What are the most common keywords in publications?

Table 8 Most effective conferences for sharing information about the Kirkpatrick model
All fields
Forum N. publications

Proceedings – 2008 2nd International Symposium on Intelligent Information 2

Technology Application, IITA 2008

Figure 11 Comparison of the number of papers published in journals and conference

proceedings in all fields by year

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 51

 Figure 12 Comparison of the number of papers published in journals and conference
proceedings in social science

The analysis was taken a step further to determine the clustering and co-occurrence of
authors’ keywords using VOSviewer, where parameters were set to “analysis type: co-
occurrence” and “minimum number of keyword occurrences: seven.” The retrieved
keywords in social sciences totaled 29, as seen in Figure 13 and were divided into seven
clusters. Table 9 shows four clusters. Figure 14 shows that the retrieved keywords in all
fields totaled 32, with the recent keywords being “teaching,” “students,” “quality
improvement,” “patient safety,” “continuing education,” “simulations,” “medical,” “dental
education” and “program evaluation.” This showed that the focus tends to be on higher
education, especially in medical schools; this is also shown in Figure 15, which presents a
timeline of trending topics in all fields.
As expected, Figures 13 and 14 indicate that research into the Kirkpatrick model focuses
on the field of social sciences, especially on assessments of curriculum, students and
education. According to the clustering table, SLR studies related to the Kirkpatrick model
occurred only in the field of medical education. Figure 15 affirms these findings by showing
that the most common topics in all fields in 2016 were evaluation, training, curriculum and

Figure 13 Co-occurrence map for author keywords in social sciences

PAGE 52 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

Table 9 Author keywords clustering based on their co-occurrence
Cluster 1 Cluster 2 Cluster 3 Cluster 4

Continuing education E-learning Assessment Curriculum

Dental education Effectiveness Education Simulation
Faculty development Incivility Higher education Students
Impact IL Learning Teaching
Medical education Teacher training Program evaluation
Systematic review

Figure 14 Co-occurrence map for author keywords in all fields

Figure 15 Trending topics in all fields based on author keywords 2010–2020

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 53

 medical education; in 2019, they were students and learning; and in 2020, the most
common topic was online learning.
RQ8. What are the most relevant domains in which the Kirkpatrick model is used?
In the first phase, we used the information provided by Bibliometrix to identify the most
relevant domains in which the Kirkpatrick model is used in all fields; in the second phase,
we focused on social sciences. Considering the extracted data set to show the most
relevant domains in which Kirkpatrick’s model is used, as seen in Figure 16, the most
common domain for research using the Kirkpatrick model was medical training evaluation,
both in all fields and in social sciences. Moreau (2017) and Maudsley and Taylor (2020)
confirm that the Kirkpatrick model has been popular for straightforward, practical-to-
evaluate medical education programs.
When we look in depth at the linking of school education (such as the evaluation of
students, teachers and head teachers) with the Kirkpatrick model, we find the education
field included in the arts and humanities domain. This domain includes only eight studies.
To obtain detailed data for these studies, they were examined manually. As Lewis et al.
(2013, p. 49) assert, “[a]s scholars increasingly take up such data sets, they may be better
served by interweaving computational and manual approaches throughout the overall
process to maximize the precision of algorithms and the context-sensitive evaluations of
human coders.” It was noted that the studies related to education were all conducted
during the past decade: Smidt et al. (2009), Chang and Chou (2015), Yahya et al. (2017),
Kazan et al. (2019), Liao (2019), Lin and Bazaluk (2020) and Suharto et al. (2020).
Moreover, Suharto et al. (2020) study linked the Kirkpatrick model with school education to
evaluate the effectiveness of a school-based disaster risk reduction program in a junior high
school in Indonesia.
This result confirms that the link between the Kirkpatrick model and school education (such
as the evaluation of students, teachers and head teachers) has been very weak; there is
clearly a lack of research in this area.
RQ9. What types of studies use the Kirkpatrick model?
The number of records retrieved in this study was 416. They were manually checked to
classify their types and the results were as follows: empirical studies – 79.3%, systematic
reviews – 9.6%, reviews – 9.4% and scoping reviews – 1.7%.

Figure 16 Most common domains in all fields and social sciences

PAGE 54 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

However, only one systematic review was about the levels of Kirkpatrick’s model, which in
this case were used in evaluating simulation and debriefing in health-care education
(Johnston et al., 2018). The other systematic reviews used Kirkpatrick’s model in the
following three ways: as an analytical framework for study (Brauckmann and Pashiardis,
2011; MacLure and Stewart, 2016; Ahmady et al., 2019); to synthesize the study data
(Steinert et al., 2006; Maiti and Bidinger, 2016); and to organize the study outcomes (Hill
et al., 2009; Lee et al., 2020).
RQ10. What were the results of these studies on the model? Is the model applicable and
useful?
All the studies (416) were sorted and analyzed manually to investigate their results on
Kirkpatrick’s model. The results of the studies can be divided into two types, which are
outlined below.
The first type consists of studies that criticize the model. There were 39 critical reviews for
this model, which, in general, discussed the three problematic assumptions of the model
that may be identified as follows: the levels are arranged in ascending order of information
provided; the levels are causally linked; and the levels are positively intercorrelated, as
discussed earlier in the literature review section (Alliger and Janak, 1989; Bates, 2004;
Moreau, 2017). Moreover, seven of the studies proposed an alternative approach to
Kirkpatrick’s model for several reasons: because of the criticisms of the model mentioned
above (MacKie, 2007); the Kirkpatrick model has rarely been implemented in its entirety
and evaluations of training are usually confined to Levels 1 and 2 (Cunningham, 2007;
Giangreco et al., 2010; Nickols, 2005); as new technology changes the how, when and
where of learning, Rossett (2007) proposed using new technologies to measure learning
instead of traditional methods of learning evaluation as Kirkpatrick’s model did; and the
model is based on outcomes, and therefore, Dubrowski and Morin (2011) proposed an
approach that integrates both outcomes- and process-based models with models of clinical
performance assessment to provide a deeper understanding of the function of education in
the health professions. Furthermore, Agarwal et al. (2019) proposed expanding the model
to include three gaps identified in it as follows: training motivation, organization citizenship
behavioral and the assessment of both the individual and the organization simultaneously.
The second type of study are those that mention the benefits and features of the model. In
total, 330 empirical studies used the Kirkpatrick model to evaluate training in various
contexts and settings, as Figures 7, 8 and 16 show and all these studies obtained results on
the effectiveness of the training being evaluated. Furthermore, several of these studies and
reviews indicated the benefits and advantages of the model. The studies were divided into
four groups.
The first group included studies providing general descriptions of using the model, such as
its usefulness (Frye and Hemmer, 2012; Ghorbandoost et al., 2018; Kang et al., 2015); its
effectiveness for organizational performance improvement (Pearlstein, 2010); its ability to
find the weaknesses and problems of training courses and attempt to fix them
(Ghorbandoost et al., 2018); its ease of implementation (Fadhilah et al., 2018); the fact that it
is a well-established assessment method for training models (Schuettfort et al., 2020); and
the fact that it has stood the test of time (Desilets, 2018).
The second group comprised studies emphasising the validity and reliability of the model
results, for example, stating that the results provided by this model are reliable (Liu et al.,
2008; Xiong et al., 2008; Yang and Zhu, 2008) and stable (Li et al., 2008).
The third group included studies emphasising the usefulness and strength of one of the
levels of the model. For example, researchers noted that the model provides detailed
insights into results (Steensma and Groeneveld, 2010), it gives a structure for the evaluation
of short- and long-term learning outcomes (Carlfjord et al., 2017) and the third level of the
model is robust (Raja Kasim and Ali, 2011).

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 55

 Studies in the fourth group emphasize the validity of the model for measuring the
effectiveness of training on a specific topic. The suitability and validity of the model for
measuring the effectiveness of 12 domains of training were mentioned in several studies.
For example, researchers state that it provides an excellent framework to determine the
strengths and weaknesses of health information management (Rouse, 2011); it is
suitable for organizers or institutions to evaluate the impact of continuing education
programs (Deodhar and Powdwal, 2017) or to evaluate e-training (Zheng et al., 2013);
it has come close to being a standard evaluation practice at studied hotels (Ho et al.,
2016); it is suitable for evaluating technology-enhanced learning (Ruiz and Snoeck,
2018); and it is effective for evaluating the training program for nurses in an
emergency surgery department (Li et al., 2020), for measuring the effectiveness of
information literacy (IL) courses (Chang and Chen, 2014) and for the evaluating a
health volunteer program (Vizeshfar et al., 2018).
In summary, despite the traditional use of the model in business and the criticism of it
in many studies, the classification and analysis of studies in this paper reveal the
continuing use of the Kirkpatrick model in research. The growth in its application
across industries and contexts indicates that the model is accepted by many as a
valid and credible model for evaluating training programs and verifies the model’s
robustness and its high performance in evaluating training. Moreover, the Kirkpatrick
model is useful, appropriate and applicable in a variety of contexts and adaptable to
many training environments.
This study has provided a theoretical contribution and highlighted implications related to the
knowledge gap in training evaluation research by demonstrating features of the Kirkpatrick
model – simplicity, flexibility, applicability and adaptability – that make it an attractive choice
for business leaders, trainers and evaluators.

Sensitivity analysis
A sensitivity analysis of the publications was conducted to ensure that the search string had
been built correctly and minimize the level of risk to the sample accuracy. To calculate the
required number of random papers to be manually examined, we used Cochran’s sample
size formula (Cochran, 1977), as shown in equation (3):

Z 2 pð1p Þ
 e  
2
(3)
1 þ Z 2  p 1p
e2 N

In the above formula, N is the total number of publications obtained from the Scopus
database using the search string (416). Z represents the parameter matching the
confidence interval. It is 95%, which implies that Z = 1.96. The population proportion or
population expected to have the characteristic of the study interest is 50%, which implies
that p = 0.5. Furthermore, e stands for the desired margin of error; it is set to 8%, which
implies that e = 0.08.
Of the 110 randomly selected papers, 107 (97%) were accurate in that they were about the
Kirkpatrick model; only 3% were inaccurate, namely, there was no relationship between the
papers and the search string. Therefore, around 6% out of 10% were estimated
(i.e. confidence level e showed an adequate level of quality for the obtained data to be
analyzed).

Limitations of the study

This study has two limitations. First, it was limited to papers that are accessible for reading
in sources (indexing databases). Second, it only incorporated studies in English, as this is

PAGE 56 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

the language used in research software (bibliometric analysis) and sources (including open
libraries).

Conclusion
This paper has discussed the emergence of the Kirkpatrick model for the evaluation of
training processes and the growth in its applications. It has presented the results of a
bibliometric analysis of publications on the Kirkpatrick model from 1959/1960 to July 2020
retrieved from Scopus. The search identified 416 papers in all fields, including 169
specifically in social sciences. The paper has answered the 5Ws þ 1H (why, when, who,
where, what and how) to explore key topics in this area. It concludes that the interest in
publishing on the evaluation of training started after 2000; however, Kirkpatrick’s model
remains the most commonly used model for evaluating training. The research using
Kirkpatrick’s model is an active and growing area. Moreover, our estimates indicate that five
additional publications are likely to be produced each year compared with the previous
year. The vast majority of publications in this area are multi-authored, with 88.8% of papers
demonstrating this trend.
The majority of the body of knowledge in this area comprises research papers and the most
effective journals for this type of paper are those in the medical field. Currently, there are no
effective conferences for publishing studies on the Kirkpatrick model.
Some parts of the world, such as the USA, the UK and Canada, are very active in research
in this area, whereas other parts of the world, including the Middle East, Africa and Russia,
are still not applying this model.
The most common subjects for the application of the Kirkpatrick model are social sciences,
medicine and computer science. This proves the popularity and applicability of using the
Kirkpatrick model in various fields as it was first developed for use within business
organizations.

The implications of the study findings

The classification and analysis of studies in this paper indicate growth in the Kirkpatrick
model’s application across industries and contexts, suggesting that the model is accepted
by many as a valid and credible model for evaluating training programs. It also highlights
the model’s robustness and its high performance in evaluating training. Moreover, the
Kirkpatrick model is useful, appropriate and applicable in a variety of contexts and is
adaptable to many training environments because of its features of simplicity, flexibility,
applicability and adaptability. The study results highlight some suggestions for future
research. Moreover, this study helps to understand the features of the Kirkpatrick model
that make it worth using even though other later models are available. Furthermore, the
findings are useful for human resource management, organizations and practitioners and
suggest using the Kirkpatrick model to evaluate training because of its robustness,
adaptability and suitability, thereby resulting in better evaluation of the effectiveness of
training.

Note
1. shorturl.at/hrxHQ

References
Agarwal, N., Pande, N. and Ahuja, V. (2019), “Expanding the Kirkpatrick evaluation model-towards more
efficient training in the IT sector”, Human Performance Technology, pp. 1092-1109.

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 57

 Ahmady, S., Khajeali, N., Sharifi, F. and Mirmoghtadaei, Z. (2019), “Educational intervention to improve
preclinical academic performance: a systematic review”, Journal of Education and Health Promotion,
Vol. 8 No. 38, doi: 10.4103/jehp.jehp_374_18.

Aldrich, C. (2002), “Measuring success: in a post-Maslow/Kirkpatrick world, which metrics matter”,

Online Learning, Vol. 6 No. 2, pp. 30-32.
Alliger, G. and Janak, E. (1989), “Kirkpatrick’s levels of training criteria: thirty years later”, Personnel
Psychology, Vol. 42 No. 2, pp. 331-342.

Antos, M. and Bruening, T. (2006), “A model hypothesizing the effect of leadership style on the transfer of
training”, Journal of leadership Education, Vol. 5 No. 3, pp. 31-52.
Aria, M. and Cuccurullo, C. (2017), “Bibliometrix: an R-tool for comprehensive science mapping
analysis”, Journal of Informetrics, Vol. 11 No. 4, pp. 959-975.

Balstad, M.T. and Berg, T. (2020), “A long-term bibliometric analysis of journals influencing management
accounting and control research”, Journal of Management Control, Springer Berlin Heidelberg, Vol. 30
No. 4, pp. 357-380.
Bassi, L.J. and Cheney, S. (1997), “Benchmarking the best’, training and development”, Training &
Development, Vol. 51 No. 11, p. 60.
Bates, R. (2004), “A critical analysis of evaluation practice: the Kirkpatrick model and the principle of
beneficence”, Evaluation and Program Planning, Vol. 27 No. 3, pp. 341-347.
Beech, B. and Leather, P. (2006), “Workplace violence in the health care sector: a review of staff training
and integration of training evaluation models”, Aggression and Violent Behavior, Vol. 11 No. 1, pp. 27-43.
Bernardino, G. and Curado, C. (2020), “Training evaluation: a configurational analysis of success and failure of
trainers and trainees”, European Journal of Training and Development, Vol. 44 Nos 4/5, pp. 531-546.
Bernthal, P.R. (1995), “Evaluation that goes the distance”, Training & Development, Vol. 49 No. 9,
pp. 41-46.
Bibi, S., Zozas, I., Ampatzoglou, A., Sarigiannidis, P.G., Kalampokis, G. and Stamelos, I. (2020),
“Crowdsourcing in software development: empirical support for configuring contests”, IEEE Access,
IEEE, Vol. 8, pp. 58094-58117.
Bomberger, D.W. (2003), Evaluation of Training in Human Service Organizations: A Qualitative Case
Study, The PA State University.
Brauckmann, S. and Pashiardis, P. (2011), “Contextual framing for school leadership training: empirical
findings from the commonwealth project on leadership assessment and development (Co-LEAD)”,
Journal of Management Development, Vol. 31 No. 1, pp. 18-33.
Brinkerhoff, R.O. (1987), “Achieving results from training”, San Francisco, CA: Jossey-Bass.
Bushnell, D.S. (1990), “Input, process, output: a model for evaluating training”, Training & Development
Journal, Vol. 44 No. 3, pp. 41-43.
Cahapay, M.B. (2021), “Kirkpatrick model: its limitations as used in higher education evaluation”,
International Journal of Assessment Tools in Education, Vol. 8 No. 1, pp. 135-144.
Calero, C., Mancebo, J., Garcia, F., Moraga, M.A., Berna, J.A.G., Fernandez-Aleman, J.L. and Toval, A. (2020),
“5Ws of green and sustainable software”, Tsinghua Science and Technology, Vol. 25 No. 3, pp. 401-414.
Campbell, K., Taylor, V. and Douglas, S. (2019), “Effectiveness of online cancer education for nurses and
allied health professionals; a systematic review using Kirkpatrick evaluation framework”, Journal of
Cancer Education, Vol. 34 No. 2, pp. 339-356.
Cannon-Bowers, J.A. and Salas, E. (1997), “A framework for developing team performance measures in
training”, in Brannick, M.T., Salas, E. and Prince, C. (Eds), Series in Applied Psychology. Team
Performance Assessment and Measurement: Theory, Methods, and Applications, Lawrence Erlbaum
Associates Publishers, NJ, pp. 45-62.
Carlfjord, S., Roback, K. and Nilsen, P. (2017), “Five years’ experience of an annual course on
implementation science: an evaluation among course participants”, Implementation Science, Vol. 12
No. 1, pp. 1-8.
Chang, Y.E. (2010), “An empirical study of Kirkpatrick’s evaluation model in the hospitality industry”,
Florida International University, Doctoral thesis, Florida International University, 12 November, doi:
10.25148/etd.FI10120807.

PAGE 58 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

Chang, N. and Chen, L. (2014), “Evaluating the learning effectiveness of an online information literacy
class based on the Kirkpatrick framework”, DE Gruyter, Vol. 64 No. 3, pp. 211-223.
Chang, N.C. and Chou, J.T. (2015), “Evaluating learning effectiveness of an information law course in a
blended learning environment based on the Kirkpatrick model”, Journal of Educational Media & Library
Sciences, Vol. 52 No. 4, pp. 417-450.
Clement, R.W. (1982), “Testing the hierarchy theory of training evaluation: an expanded role for trainee
reactions”, Public Personnel Management, Vol. 11 No. 2, pp. 176-184.
Cochran, W.G. (1977), Sampling Techniques, 3rd ed., John Wiley & Sons, New York, NY.
Colquitt, J.A., LePine, J.A. and Noe, R.A. (2000), “Toward an integrative theory of training motivation: a
meta-analytic path analysis of 20 years of research”, Journal of Applied Psychology, Vol. 85 No. 5,
pp. 678-707.
Cunningham, I. (2007), “Sorting out evaluation of learning and development: making it easier for
ourselves”, Development and Learning in Organizations: An International Journal, Vol. 21 No. 5,
pp. 4-6.
Deodhar, M. and Powdwal, S. (2017), “Impact of continuing education programs (CEPs) on LIS
professionals in academic libraries in Mumbai, India”, Library Management, Vol. 38 Nos 2/3, pp. 117-130.
Desilets, L.D. (2018), “An update on Kirkpatrick’s model of evaluation: part two”, The Journal of
Continuing Education in Nursing, Vol. 49 No. 7, pp. 292-293.
Dubrowski, A. and Morin, M.P. (2011), “Evaluating pain education programs: an integrated approach”,
Pain Research and Management, Vol. 16 No. 6, pp. 407-410.
Eck, N.J.V. and Waltman, L. (2020), VOSviewer Manual, Univeristeit Leiden, Leiden.
Fadhilah, M.K., Surantha, N. and Isa, S.M. (2018), “Web-based evaluation system using Kirkpatrick
model for high school education (a case study for vocational high school in Jakarta)”, 2018 International
Conference on Information Management and Technology (ICIMTech), IEEE, Jakarta, Indonesia,
pp. 166-171.

Fitzpatrick, J.L., Sanders, J.R. and Worthen, B.R. (2003), Program Evaluation: Alternative Approaches
and Practical Guidelines Hardcover, 3rd ed., Pearson, London.
Frye, A.W. and Hemmer, P.A. (2012), “Program evaluation models and related theories: AMEE guide no.
67”, Medical Teacher, Vol. 34 No. 5, pp. 288-299.

ndez, A.M. and Ruiz-Cortés, A. (2019),
Galindo, J.A., Benavides, D., Trinidad, P., Gutiérrez-Ferna
“Automated analysis of feature models: quo Vadis?”, Computing, Springer Vienna, Vol. 101 No. 5,
pp. 387-433.
Garousi, V. and Mäntylä, M.V. (2016), “Citations, research topics and active countries in software
engineering: a bibliometrics study”, Computer Science Review, Elsevier Inc, Vol. 19, pp. 56-77.
Ghorbandoost, R., Zeinabadi, H., Mohammadi, Z. and Shafiabadi, M.S. (2018), “Evaluating the
effectiveness of neonatal resuscitation training course on nurses of kowsar medical center in Qazvin
university of medical sciences based on Kirkpatrick model”, Annals of Tropical Medicine and Public
Health, Vol. 3.
Giangreco, A., Carugati, A. and Sebastiano, A. (2010), “Are we doing the right thing? Food for thought on
training evaluation and its context”, Personnel Review, Vol. 39 No. 2, pp. 162-177.
Hart, G. (1996), “The five W’s: an old tool for the new task of audience analysis”, Technical
Communication, Vol. 43 No. 2, pp. 139-145.
Hill, A.G., Yu, T.C., Barrow, M. and Hattie, J. (2009), “A systematic review of resident-as-teacher
programmes”, Medical Education, Vol. 43 No. 12, pp. 1129-1140.

Ho, A.D.D., Arendt, S.W., Zheng, T. and Hanisch, K.A. (2016), “Exploration of hotel managers’ training
evaluation practices and perceptions utilizing Kirkpatrick’s and Phillips’s models”, Journal of Human
Resources in Hospitality & Tourism, Taylor & Francis, Vol. 15 No. 2, pp. 184-208.

Holton, E.F. (1996), “The flawed four-level evaluation model”, Human Resource Development Quarterly,
Vol. 7 No. 1, pp. 5-21.
Holton, E.F. (2005), “Holton’s evaluation model: new evidence and construct elaborations”, Advances in
Developing Human Resources, Vol. 7 No. 1, pp. 37-54.

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 59

 Holton, E.F. and Naquin, S. (2005), “A critical analysis of HRD evaluation models from a decision-making
perspective”, Human Resource Development Quarterly, Vol. 16 No. 2, pp. 257-280.

Jain, G., Sharma, N. and Shrivastava, A. (2021), “Enhancing training effectiveness for organizations
through blockchain-enabled training effectiveness measurement (BETEM)”, Journal of Organizational
Change Management, Vol. 34 No. 2, doi: 10.1108/JOCM-10-2020-0303.

Jia, C., Cai, Y., Yu, Y.T. and Tse, T.H. (2016), “5Wþ1H pattern: a perspective of systematic mapping
studies and a case study on cloud software testing”, Journal of Systems and Software, Elsevier Ltd,
Vol. 116, pp. 206-219.

Johnston, S., Coyer, F.M. and Nash, R. (2018), “Kirkpatrick’s evaluation of simulation and debriefing in
health care education: a systematic review”, Journal of Nursing Education, Vol. 57 No. 7, pp. 393-398.
Kang, J., Yang, E.B., Chang, Y.J., Choi, J.Y., Jho, H.J., Koh, S.J., Kim, W.C., Choi, E., Kim, Y. and Park, S.
(2015), “Evaluation of the national train-the-trainer program for hospice and palliative care in Korea”,
Asian Pacific Journal of Cancer Prevention, Vol. 16 No. 2, pp. 501-506.
Kaplan, R.M. and Pascoe, G.C. (1977), “Humorous lectures and humorous examples: some effects upon
comprehension and retention”, Journal of Educational Psychology, Vol. 69 No. 1, pp. 61-65.
Kaufman, R. and Keller, J.M. (1994), “Levels of Evaluation: Beyond Kirkpatrick”, Human Resource
Development Querterly, Vol. 5, pp. 371-380.
Kaufman, R., Keller, J. and Watkins, R. (1995), “What works and what doesn’t: evaluation beyond
Kirkpatrick”, Performance þ Instruction, Vol. 35 No. 2, pp. 8-12.
Kazan, E., Usmen, M., Desruisseaux, B., Kaya, S. and Seyoum, M. (2019), “Training effectiveness
analysis of osha silica and excavation standards for construction”, in Passerini, G., Garzia, F. and
Lombardi, M. (Eds), 8th International Conference on Safety and Security Engineering, SAFE 2019, WIT
Press, Italy, pp. 33-42.
Kipling, R. (1912), Just so Stories, MacMillan, London.

Knowles, S.M. (1980), The Modern Practice of Adult Education: From Pedagogy to Andragogy,
Cambridge Book Company, Cambridge.
Kraiger, K., Ford, J.K. and Salas, E. (1993), “Application of cognitive, skill-based, and affective theories of
learning outcomes to new methods of training evaluation”, Journal of Applied Psychology, Vol. 78 No. 2,
pp. 311-328.
Kirkpatrick, D.L. (1959), “Techniques for evaluating training programs”, Journal of ASTD, pp. 1-13.
Kumpikaite,_ V. (2007), “Human resource training evaluation”, Engineering Economics, Vol. 55 No. 5,
pp. 29-36.
Lee, S.Y., Fisher, J., Wand, A.P.F., Milisen, K., Detroyer, E., Sockalingam, S., Agar, M., Hosie, E. and
Teodorczuk, A. (2020), “Developing delirium best practice: a systematic review of education
interventions for healthcare professionals working in inpatient settings”, European Geriatric Medicine,
Springer International Publishing, Vol. 11 No. 1, doi: 10.1007/s41999-019-00278-x.
Lewis, S.C., Zamith, R. and Hermida, A. (2013), “Content analysis in an era of big data: a hybrid approach
to computational and manual methods”, Journal of Broadcasting & Electronic Media, Vol. 57 No. 1,
pp. 34-52.
Li, T., Yang, Y. and Liu, Z. (2008), ““An improved neural network algorithm and its application on
enterprise strategic management performance measurement based on Kirkpatrick model”, Proceedings
– 2008 2nd International Symposium on Intelligent Information Technology Application, IITA 2008, Vol. 1,
IEEE, Washington, DC, pp. 861-865.
Li, Z., Cheng, J., Zhou, T., Wang, S., Huang, S. and Wang, H. (2020), “Evaluating a nurse training
program in the emergency surgery department based on the Kirkpatrick’s model and clinical demand
during the COVID-19 pandemic”, Telemedicine and e-Health, Vol. 26 No. 8, pp. 985-991.
Liao, P.W. (2019), “Experiential learning is an effective training model to improve self-esteem”,
Humanities & Social Sciences Reviews, Vol. 7 No. 5, pp. 165-173.
Lien, B.Y.H., Yu Yuan Hung, R. and McLean, G.N. (2007), “Training evaluation based on cases of
Taiwanese benchmarked high-tech companies”, International Journal of Training and Development,
Vol. 11 No. 1, pp. 35-48.

PAGE 60 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

Lin, Y.W. and Bazaluk, O. (2020), “Using brain science theory to analyze the Unity between language
input and output: methodology improvement substantiation”, Psiholingvistika, Vol. 27 No. 1, pp. 195-218.

Liu, Z., Zhang, S. and Xiong, F. (2008), ““A novel FNN algorithm and its application on M&A performance
evaluation based on Kirkpatrick model”, Proceedings – 2008 International Conference on MultiMedia and
Information Technology, MMIT 2008, 30 Decmber, IEEE, Gorges, China, pp. 66-69.

MacKie, D. (2007), “Evaluating the effectiveness of executive coaching: where are we now and where do
we need to be?”, Australian Psychologist, Vol. 42 No. 4, pp. 310-318.
MacLure, K. and Stewart, D. (2016), “Digital literacy knowledge and needs of pharmacy staff: a
systematic review”, Journal of Innovation in Health Informatics, Vol. 23 No. 3, pp. 560-571.
Maiti and Bidinger (2016), “A BEME systematic review of the effects of interprofessional education: BEME
guide no. 39”, Journal of Chemical Information and Modeling, Vol. 53 No. 9, pp. 1689-1699.
Maudsley, G. and Taylor, D. (2020), “Analysing synthesis of evidence in a systematic review in health
professions education: observations on struggling beyond Kirkpatrick”, Medical Education Online,
Vol. 25 No. 1, doi: 10.1080/10872981.2020.1731278.
Moral-Muñoz, J.A., Herrera-Viedma, E., Santisteban-Espejo, A. and Cobo, M.J. (2020), “Software tools


for conducting bibliometric analysis in science: an up-to-date review”, El Profesional de La Informacion,
Vol. 29 No. 1, pp. 1-20.
Moreau, K.A. (2017), “Has the new Kirkpatrick generation built a better hammer for our evaluation
toolbox?”, Medical Teacher, Vol. 39 No. 9, pp. 999-1001.
Nickols, F.W. (2005), “Why a stakeholder approach to evaluating training”, Advances in Developing
Human Resources, Vol. 7 No. 1, pp. 121-134.

Papamitsiou, Z. and Economides, A.A. (2014), “Learning analytics and educational data mining in
practice: a systemic literature review of empirical evidence”, Educational Technology and Society,
Vol. 17 No. 4, pp. 49-64.

Passmore, J. and Velez, M. (2012), “SOAP-M: a training evaluation model for HR”, Industrial and
Commercial Training, Vol. 44 No. 6, pp. 315-325.
Paul, J. and Criado, A.R. (2020), “The art of writing literature review: what do we know and what do we
need to know?”, International Business Review, Vol. 29 No. 4, pp. 1-7.
Pearlstein, R.B. (2010), “How to use Kirkpatrick’s taxonomy effectively in the workplace”, in Moseley, J.L.
and Dessinger, J.C. (Eds), Handbook of Improving Performance in the Workplace, International Society
for Performance Improvement, pp. 38-57.
Phillips, J.J. (1991), Handbook of training evaluation and measurement methods, Gulf Publishing
Company, Houston.
Phillips, J.J. (1996), “How much is the training worth?”, Journal of ASTDTraining and Development,
Vol. 50 No. 4, pp. 20-24.
Phillips, P. (2003), Training Evaluation in the Public Sector, The University of Southern MS, Hattiesburg.
Rahimi, M. and Rosman, M. (2020), “The 5Ws of enterprise content management (ECM) research: is it
worth?”, Open Journal of Science and Technology, Vol. 3 No. 1, pp. 46-70.
Raja Kasim, R.S. and Ali, S. (2011), “Measuring training transfer performance items among academic
staff of higher education institution in Malaysia using Rasch measurement”, 2011 IEEE Colloquium on
Humanities, Science and Engineering, CHUSER 2011, IEEE, No. Chuser 2011, pp. 756-760.
Reio, T.G., Rocco, T.S., Smith, D.H. and Chang, E. (2017), “A critique of Kirkpatrick’s evaluation model”,
New Horizons in Adult Education and Human Resource Development, Vol. 29 No. 2, pp. 35-53.

Repanovici, A. (2010), “Measuring the visibility of the university’s scientific production using
GoogleScholar, ‘publish or perish’ software and scientometrics”, World Library and Information
Congress: 76th IFLA General Conference and Assembly. Retrieved December, Science and Technology
Libraries, Gothenburg, Sweden, pp. 1-14.
Rodin, M. and Rodin, B. (1973), “Student evaluations of teachers”, The Journal of Economic Education,
Vol. 5 No. 1, pp. 5-9.

Rossett, A. (2007), “Leveling the levels”, T and D, Vol. 61 No. 2, pp. 49-53.

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 61

 Rouse, D. (2011), “Employing Kirkpatrick’s evaluation framework to determine the effectiveness of health
information management courses and programs”, Perspectives in Health Information Management,
Vol. 8.
Rowold, J. (2007), “Individual influences on knowledge acquisition in a call center training context in
Germany”, International Journal of Training and Development, Vol. 11 No. 1, pp. 21-34.
Ruiz, J. and Snoeck, M. (2018), “Adapting Kirkpatrick’s evaluation model to technology enhanced
learning”, Proceedings of the 21st ACM/IEEE International Conference on Model Driven Engineering
Languages and Systems: Companion Proceedings, ACM, Copenhagen, Denmark, pp. 135-142.

Russ-Eft, D. and Preskill, H. (2005), “In search of the holy grail: ROI evaluation in HRD”, Advances in
Developing Human Resources, pp. 71-85.

Salas, E. and Cannon-Bowers, J.A. (2001), “The science of training: a decade of progress”, Annual
Review of Psychology, Vol. 52 No. 1, pp. 471-685.
Schuettfort, V.M., Ludwig, T.A., Marks, P., Vetterlein, M.W., Maurer, V., Fuehner, C., Janisch, F., Soave,
A., Rink, M., Riechardt, S., Engel, O., Fisch, M., Dahlem, R. and Meyer, C.P. (2020), “Learning benefits of
live surgery and semi-live surgery in urology – informing the debate with results from the international
meeting of reconstructive urology (IMORU) VIII”, World Journal of Urology, Springer Berlin Heidelberg,
doi: 10.1007/s00345-020-03506-3.
Shareefa, M. and Moosa, V. (2020), “The most-cited educational research publications on differentiated
instruction: a bibliometric analysis”, European Journal of Educational Research, Vol. 9 No. 1, pp. 331-349.
Smidt, A., Balandin, S., Sigafoos, J. and Reed, V.A. (2009), “The Kirkpatrick model: a useful tool for
evaluating training outcomes”, Journal of Intellectual & Developmental Disability, Vol. 34 No. 3,
pp. 266-274.
Steensma, H. and Groeneveld, K. (2010), “Evaluating a training using the ‘four levels model’”, Journal of
Workplace Learning, Vol. 22 No. 5, pp. 319-331.
Steinert, Y., Mann, K., Centeno, A., Dolmans, D., Spencer, J., Gelula, M. and Prideaux, D. (2006), “A
systematic review of faculty development initiatives designed to improve teaching effectiveness in
medical education: BEME guide no. 8”, Medical Teacher, Vol. 28 No. 6, pp. 497-526.
Stufflebeam, D.L. (1983), “The CIPP model for program evaluation”, in Madaus, F.F., Scriven, M. and
Stufflebeam, D.L. (Eds.), Evaluation Models: Viewpoints on Educational and Human Services Evaluation,
Norwell, MA: Kluwer, pp. 117-141.
Suharto, N.T., Slamet, P.H., Jaedun, A. and Purwanta, H. (2020), “The effectiveness of a school-based
disaster risk reduction program in Indonesia: a case study in the klaten regency’S junior high schools”,
International Journal of Innovation, Creativity and Change, Vol. 12 No. 12, pp. 949-962.
Tamkin, P., Yarnall, J. and Kerrin, M. (2002), Kirkpatrick and beyond: A Review of Training Evaluation, The
Institute for Employment Studies, Brighton, The UK.
Tan, J.A., Hall, R.J. and Boyce, C. (2003), “The role of employee reactions in predicting training
effectiveness”, Human Resource Development Quarterly, Vol. 14 No. 4, pp. 397-411.
Tattersall, A. (2015a), “Who, what, where, when, why: using the 5 Ws to communicate your research j
impact of social sciences”, LSE Impact Blog, pp. 8-10.
Tattersall, A. (2015b), “Who, what, where, when, why: using the 5 Ws to communicate your research j
impact of social sciences”, LSE Impact Blog, pp. 8-10.
Topno, H. (2012), “Evaluation of training and development: an analysis of vmarious models”, IOSR
Journal of Business and Management, Vol. 5 No. 2, pp. 16-22.
Tzeng, G.H., Chiang, C.H. and Li, C.W. (2007), “Evaluating intertwined effects in e-learning programs: a
novel hybrid MCDM model based on factor analysis and DEMATEL”, Expert Systems with Applications,
Vol. 32 No. 4, pp. 1028-1044.
van den Besselaar, P. and Sandström, U. (2020), “Bibliometrically disciplined peer review: on
using indicators in research evaluation”, Scholarly Assessment Reports, Vol. 2 No. 1, doi:
10.29024/sar.16.
Vı́lchez-Roma
n, C., Sanguinetti, S. and Mauricio-Salas, M. (2020), “Applied bibliometrics and information
visualization for decision-making processes in higher education institutions”, Library Hi Tech, Vol. 39
No. 1, doi: 10.1108/LHT-10-2019-0209.

PAGE 62 j INDUSTRIAL AND COMMERCIAL TRAINING j VOL. 54 NO. 1 2022

Vizeshfar, F., Momennasab, M., Yektatalab, S. and Iman, M.T. (2018), “Evaluation of the effectiveness of
a first aid health volunteers’ training programme using Kirkpatrick’s model: a pilot study”, Health
Education Journal, Vol. 77 No. 2, pp. 190-197.
Waddill, D.D.W. (2006), “Action e-learning: an exploratory case study of action learning applied online”,
Human Resource Development International, Vol. 9 No. 2, pp. 157-171.

Warr, P., Bird, M. and Rackham, N. (1970), Evaluation of Management Training: A Practical Framework,
with Cases, for Evaluating Training Needs and Results, 2nd edition, Gower Press, CA: London, England.
Watkins, R., Leigh, D., Foshay, R. and Kaufman, R. (1998), “Kirkpatrick plus: evaluation and continuous
improvement with a community focus”, Educational Technology Research and Development, Vol. 46
No. 4, pp. 90-96.
Wei, M. (2020), “Research on impact evaluation of open access journals”, Scientometrics, Vol. 122 No. 2,
Springer International Publishing, pp. 1027-1049.
Werner, J.M. and De Simor, R.L. (2012), Human Resource Development, 6th ed., South-Western,
Cengage Learning, Mason.
Xiong, F., Zhang, Y., Li, T. and Liu, Z. (2008), “Integration of a novel neural network algorithm and
Kirkpatrick model and its application in R&D performance evaluation”, Proceedings - International
Conference on Computer Science and Software Engineering, CSSE. 12-14 December, Vol. 1, IEEE,
Wuhan, China, pp. 353-356.
Yahya, A.B., Noordin, M.K., Ali, D.F., Boon, Y., Hashim, S., Ahmad, J. and Ibrahim, M.A. (2017), “The
impact of ASNAF development training program on the quality of life of the poor and needy”, Man in India,
Vol. 97 No. 13, pp. 307-315.
Yang, S. and Zhu, Q. (2008), “Research on manager training effectiveness evaluation based on
Kirkpatrick model and fuzzy neural network algorithm”, 2008 4th International Conference on Wireless
Communications, Networking and Mobile Computing, IEEE, Dalian, China, pp. 1-4.
Zheng, L., Huang, R. and Yu, J. (2013), “Evaluation of the effectiveness of e-training: a case study on in-
service teachers’ training”, Proceedings – 2013 IEEE 13th International Conference on Advanced
Learning Technologies, ICALT 2013, 15-18 July, IEEE, Beijing, China, pp. 229-231.

Corresponding author
Aljawharah Alsalamah can be contacted at: 14587906@students.lincoln.ac.uk

For instructions on how to order reprints of this article, please visit our website:
www.emeraldgrouppublishing.com/licensing/reprints.htm
Or contact us for further details: permissions@emeraldinsight.com

VOL. 54 NO. 1 2022 j INDUSTRIAL AND COMMERCIAL TRAINING j PAGE 63