JMIR SERIOUS GAMES Graafland et al

Viewpoint

How to Systematically Assess Serious Games Applied to Health

Care

Maurits Graafland1, MD; Mary Dankbaar2, MSc; Agali Mert3, MD, PhD; Joep Lagro4, MD, PhD; Laura De
Wit-Zuurendonk5, MD; Stephanie Schuit6, MD, PhD; Alma Schaafstal7, PhD; Marlies Schijven1, MD, PhD, MHSc
1
Department of Surgery, Academic Medical Center, Amsterdam, Netherlands
2
Desiderius School, Department of Medical Education, Erasmus University Medical Centre, Rotterdam, Netherlands
3
National Military Rehabilitation Centre, Doorn, Netherlands
4
Department of Geriatrics, Haga Ziekenhuis, Den Haag, Netherlands
5
Department of Obstetrics and Gynaecology, Máxima Medical Centre, Veldhoven, Netherlands
6
Departments of Emergency Medicine and Internal Medicine, Erasmus University Medical Centre, Rotterdam, Netherlands
7
Department of ICT, Windesheim University of Applied Sciences, Zwolle, Netherlands

Corresponding Author:
Marlies Schijven, MD, PhD, MHSc
Department of Surgery
Academic Medical Center
PO Box 22660
Amsterdam, 1100DD
Netherlands
Phone: 31 20 566 4207
Fax: 31 20 691 4858
Email: m.p.schijven@amc.uva.nl

Abstract
The usefulness and effectiveness of specific serious games in the medical domain is often unclear. This is caused by a lack of
supporting evidence on validity of individual games, as well as a lack of publicly available information. Moreover, insufficient
understanding of design principles among the individuals and institutions that develop or apply a medical serious game compromises
their use. This article provides the first consensus-based framework for the assessment of specific medical serious games. The
framework provides 62 items in 5 main themes, aimed at assessing a serious game’s rationale, functionality, validity, and data
safety. This will allow caregivers and educators to make balanced choices when applying a serious game for healthcare purposes.
Furthermore, the framework provides game manufacturers with standards for the development of new, valid serious games.

(JMIR Serious Games 2014;2(2):e11) doi: 10.2196/games.3825

KEYWORDS
consensus; serious game; applied game; telehealth; mobile health; video game

rehabilitation in disabled patients, showing equal effectiveness

Background compared to conventional training programs [3]. Games have
Serious or applied games are digital games with the purpose to been applied to promote healthy behavior in children [4] and
improve individual’s knowledge, skills, or attitudes in the “real” educate patients [5,6]. Serious game-based patient education
world. Serious games applied to medical or health-related has also been shown to increase the treatment adherence among
purposes are growing rapidly in numbers and in types of adolescents with leukemia [7]. A third application for serious
applications. Serious games have been shown to be at least as games is training medical personnel [8]. Serious games have
effective as conventional tests in improving cognitive abilities been shown to add to Advanced Life Support training [9] and
in the elderly [1] and even more effective than conventional improve understanding of geriatrics principles among medical
neuropsychological interventions when it comes to improving students compared to conventional training methods [10].
neuropsychological abilities of alcoholic patients [2]. Serious Patients [11], students, and professionals [12] generally view
game-based interventions have been used to support game-based interventions as fun and challenging.

http://games.jmir.org/2014/2/e11/ JMIR Serious Games 2014 | vol. 2 | iss. 2 | e11 | p. 1

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR SERIOUS GAMES Graafland et al

Although results for serious games in terms of effectiveness for Healthcare (DSSH) [16] has developed a consensus-based
such purposes are promising, their implementation as “serious” framework, categorizing important items that assess a serious
modalities for prevention, treatment, or training in health care game’s safety and validity. Eight individuals (see
is hindered by lack of understanding of the underlying concepts Acknowledgements section for details) from six different
among health care professionals, or even distrust. Before doctors institutions experienced in designing, applying, or researching
and patients consider using serious games as a useful solution serious games for health care-related purposes participated. The
for a health care-related problem, it is important that they reporting standards for non-game mobile health apps for medical
understand what problem is being addressed by the game and purposes (mHealth), published by Lewis [17] and Albrecht [18],
that a proposed claim on effectiveness is indeed trustworthy. was used as a basis. This system is applied by the peer-reviewed
Many clinicians are currently undereducated in judging a serious mHealth app assessment initiative of the Journal of Medical
game’s safety or effectiveness. Information on individual games Internet Research [19]. Due to inherent differences in the
is often hard to find in disorganized app stores and websites functionality of games compared to purely informational
[13]. Studies on serious games’ validity and effectiveness remain mHealth applications, this framework required re-evaluation.
scarce [8,14]. The idea of applying a video game in health care
The panel reviewed the items from these reporting standards
may even be resentful to certain clinicians or patients. In
during two meetings. All items in the Albrecht framework [18]
addition, threats to data safety fuel distrust towards electronic
were systematically evaluated. For each of the 5 categories,
applications in health care altogether [15]. Such issues menace
items irrelevant to serious games were removed and if necessary,
the practical application of serious games throughout health
extra items were added. During the second panel meeting, the
care, subsequently limiting investments in smart solutions that
framework was re-evaluated and all members approved the final
may actually prove beneficial in the end.
version.
This article discusses the first tool for the systematic assessment
The framework described provides 62 items in 5 main themes
of serious games applied to medical use, for educators and
(Table 1), aimed at assessing a serious game’s rationale,
clinicians. The information collected and organized accordingly,
functionality, validity, and data safety. It specifically does not
will aid health care practitioners to understand and appraise the
aim to assess its effectiveness in terms of success or user
risks and benefits of specific serious games in health care in a
attractiveness. The panel defined serious games (other than a
uniform manner.
regular medical application) as digital applications instigating
a specific behavioral change to its user, in the form of skills,
Assessment Framework knowledge, or attitudes useful to reality [20]. The framework
To our knowledge, there is currently no systematic framework does therefore not apply to (mobile or Web-based) digital health
for the assessment of serious games in health care described in apps with a purely informational purpose, for which the mHealth
literature. Therefore, the Dutch Society for Simulation in app assessment framework is designed [18].

http://games.jmir.org/2014/2/e11/ JMIR Serious Games 2014 | vol. 2 | iss. 2 | e11 | p. 2

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR SERIOUS GAMES Graafland et al

Table 1. Items relevant for the assessment of a serious game used for health care-related purposes.
Category Item Question
Game description Meta-data Operating system Operating systems of the game
Version Version
Web-link Web-link
Project type Commercial, non-commercial, other
Access Public / restricted / other
Adjunct devices Is an adjunct device needed?
Development Funding How was development funded? Eg, funding agencies, investors
Sponsoring / Advertising Advertisement policy Is the game free of commercial pop-ups?
If not, what is advertised?
Sources of income Are there sources of income within the game?
Sources of income outside What are the sources of income of the owner/distributor?
game
Potential conflicts of inter- Affiliations What affiliations do the publishers have that could influence content
est or user group?
Conflicts of interest What interests do the publishers have that could influence the
game’s content or user group?
Disclosure Are conflicts of interest disclosed?
Rationale Purpose Goal or purpose What is (are) the purpose(s) of the game?
Disclosure Is (are) the purpose(s) disclosed to users?
Medical device Medical device Is the serious game a medical device, or not?
Class If yes, which class?
Approval by legal bodies If yes, does it comply to the necessary requirements (FDA-ap-
proval, CE-mark?).
User group Specific user groups For each user group: disease/condition, or health care profession.
Description Please specify gender, age (range), and other relevant descriptive
items.
Limits Are there age limits, or other limits?
Disclosure Is the intended user group disclosed?
Setting Patient care Is the game used in patient care?
Training courses Is the game used in training courses or -curricula?
SCORM compliancy If used in training courses or curricula, is the serious game
SCORM-compliant?

http://games.jmir.org/2014/2/e11/ JMIR Serious Games 2014 | vol. 2 | iss. 2 | e11 | p. 3

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR SERIOUS GAMES Graafland et al

Category Item Question

Functionality Purposes / didactic features For every purpose of the
game:
Learning or behavioral goals What content will the player learn?
Relation learning and game- How does the learning content relate to the gameplay?
play
Instruction What intervention leads to the learning transition (eg, tutorial, in-
structions (in-game))
Assessment (progress) in Through which parameters is progress in the game measured?
game
Assessment parameters Which parameters are to designers' opinion indicative for measuring
learning effects?
Content Management Content Management sys- Is the Content Management System restricted to specified persons
tem or institutions?
User uploaded content If no, are users allowed to upload their own content?
Content monitoring How is uploaded content checked?
Restrictions and limits of the Please describe restrictions and limits of the serious game. What
serious game content on the learning goals is not covered?
Potentially undesirable ef- Potentially undesirable ef- What potential undesirable effects could the game have?
fects fects
Disclosure Are such potential undesirable effects disclosed to the user?
Measures taken What measures are taken to prevent potential undesirable effects?
Validity Design process Medical expert complicity Were medical experts (content experts) involved in the design
process from the start?
User group complicity Were representatives from the user group involved in the design
process from the start?
Educationalist complicity Were educationalists involved in the design process from the start?
User testing User testing Did user testing take place? What were the results, and how were
these incorporated in the design?
Stability Platform stability Does the game produce the same results on different platforms?
Validity (effectiveness) Face validity Do educators and trainees view it as a valid way of instruction?
Content validity How is its content validated to be complete, correct, and nothing
but the intended medical construct?
Construct validity Is the game able to measure differences in skills it intends to
measure?
Concurrent validity How does learning outcome compare to other methods assessing
the same medical construct?
Predictive validity Does playing the game predict skills improvement in real life?

http://games.jmir.org/2014/2/e11/ JMIR Serious Games 2014 | vol. 2 | iss. 2 | e11 | p. 4

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR SERIOUS GAMES Graafland et al

Category Item Question

Data protection Data protection and priva- Data processing How is data collected in the serious game?
cy
Patient privacy Are patient-specific data stored in the game?
If yes, are patient informed consent criteria met according to rele-
vant national standards?
Data ownership Who owns and stores the data resulting from play?
Data storage period During what period are data stored?
Data removal Can the user delete data temporarily and/or permanently?
Data storage security Is the data storage secured in conformity with laws of the countries
stated above?
Data transmission security Is the data transmission secured in conformity with laws of the
countries stated above?
Disclosure Are all items on “data protection” disclosed to the user?

Functionality
Assessing Medical Serious Games
Functionality of a serious game clearly differs from that of an
Game Description mHealth app. These usually contain “dry” content (eg, medical
When evaluating a specific serious game, it should be information) or an obvious functionality (eg, communicating
thoroughly described and registered (including information or registering information), whereas a game requires the user
about the manufacturer or owner to whom the game should be to operate or interact with the content, with the ultimate goal to
attributed and the version). Equally to mobile applications, a change ones behavior in real life (ie, learning). To understanding
special interest is taken into the owner’s policy concerning this process, information is required on the game’s content, how
revenues from sponsoring and advertisements, both during the instruction is delivered, how performance is assessed and
development as well as its use. Sources of revenue and how these aspects are integrated in the gameplay [24,25].
affiliations (eg, pharmaceutical industry) may bias or threaten Consequently, it is important to register information on the
a serious game’s validity for obvious reasons. These should be game’s content management. For instance, users may be able
fully disclosed to the game’s users. Sources of income within to add content themselves, making content validation an
a game can be equally relevant to the costs required for the important issue. This directly influences the game’s content’s
initial purchase. validity.
Rationale Finally, undesired results or negative transfer of learning could
This clarifies the game’s purpose outside the game. This external occur in the interaction with a serious game, which is not the
purpose (eg, improving eye-hand coordination in laparoscopic same concept as “gaming the game” (ie, cheating), an effect
surgery) may differ from the actual goal in the game (eg, that may very well enhance learning [24]. If validation research
completing a quest in an underground world [21] or playing a is not present, at least a logical connection between gameplay
tennis game [22]). This clearly differs from the Albrecht and behavioral or learning goals should be present and disclosed
framework, because most mHealth apps have a single obvious by the developer.
purpose (internal goal = external goal). A game’s purpose relates Validity
to the intended user group and the setting in which it is used,
similar to mHealth apps. Validity determines whether an instructional instrument (such
as a serious game) adequately resembles the construct it aims
Additionally, serious games might fall within the scope of the to educate or measure. More formally, “the degree to which
medical devices, requiring specific guidelines to be evidence and theory supports the interpretations of [game] scores
implemented, set by the US Food & Drug Administration entailed by the proposed use of [the game]” [26]. The American
(FDA), European Committee (Conformité Européenne, CE), Psychological Association has set a series of standards to
or national equivalents. This specifically applies to games with measure validity [26]. Whereas many validity types have been
a distinct diagnostic or therapeutic purpose. Moreover, described, validity research in medical education usually
integration of serious games into electronic learning contains several consequential phases [27,28]. First, experts
environments may demand certain technical requirements. The should scrutinize the game’s content to determine its legitimacy
industry has set standards to improve the interoperability of (content validity). Second, experts and novices judge the
e-learning content (the Sharable Content Object Reference instrument’s apparent similarity to the construct it attempts to
Model; SCORM)[23]. Its implementation will improve the represent (face validity). Construct validity reflects the ability
integration of educative serious games in learning management of the instrument to actually measure what it intends to measure
software. (ie, the difference in performance between groups of users with
different levels of experience in reality). Concurrent validity

http://games.jmir.org/2014/2/e11/ JMIR Serious Games 2014 | vol. 2 | iss. 2 | e11 | p. 5

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR SERIOUS GAMES Graafland et al

reflects the correlation between performance on the serious The safe application of technology-enhanced solution remains
game and their performance on an instrument believed to the responsibility of the health care provider. Choosing if a
measure the same construct (eg, a simulator or course). The serious game answers to the user’s needs, can be based on
ultimate goal is to prove a game’s predictive validity: does information concerning 5 main areas described in this article.
performance in the game lead to better outcomes in reality? The majority of the items cannot be assessed using objective
Most validation research currently published in the medical parameters. For instance, claiming a specific serious game’s
domain uses these concepts [29]. For individual cases, relevance predictive validity should be supported by solid evidence. A
of specific validity types may differ. When considering mHealth comprehensive evaluation by a panel of experts in the form of
apps in general, content validity may be the sole source of a quality label could form a more practical solution.
validity.
Guidelines have been recently published reporting standards to
Validity research is frequently a long and costly enterprise. support clinicians and patients in distinguishing high quality
Many newly developed serious games have therefore not yet mhealth apps [17-19] and medical websites [30]. These
undergone validity research [8]. The framework therefore standards form the basis for the framework described in this
determines a number of steps to pre-assess a serious game’s article. These standards have two important shortcomings when
potential as a valid instrument, with regard to its design and it comes to games. First, explicit information on a serious
initial testing phases. This encompasses the involvement of user game’s content and didactic features is required, as the external
groups, content experts, or educationalists in the design (if purpose of a serious game is frequently less obvious to the user
relevant to the game’s purpose). Next, if a game has undergone than in the case of mHealth apps. Second, serious games require
user testing and stability testing, the game is more likely to have additional validation steps (eg, construct and predictive validity),
higher face- and content validity. compared to non-interactive information platforms. Gameplay
is dynamic and learning goals in gameplay are often not
Data Protection disclosed to the user. In fact, the user learns by playing the
Threats to user privacy are imminent in electronic and mobile game, whereas discovery in itself may be part of the gameplay.
health apps, especially when patient-specific data are measured Disclosing learning goals would thus be counterproductive.
or entered in the game [15]. This considers data “at rest” on
devices or servers, as well as data “in transit”. It must be clear There are several limitations to the framework described in this
whether data is collected by the game, who owns the data and study. It considers validity of the serious game’s content and
whether users can request to remove their data. Storage and its didactic functionality. Validity does not predict a game’s
analysis of personal data should be disclosed to users and must success nor its attractiveness to the user, which also depend on
be in conformity with the laws applied in countries the serious its entertainment capability and distribution method [31]. It does
game is distributed in. Special care must be taken if patient not wish to objectify which game is most fun, but merely which
information is collected. These items are in general conformity game is most valid. A second consideration is that in the
with the requirements for mHealth apps described in Albrecht’s scientific field of validity research in medicine, validity concepts
framework [18]. other than the one used in this framework have been proposed
[32]. The “classical” validity concepts (content-, face-,
Discussion construct-, concurrent-, and predictive validity) have been most
frequently used in validity research in medicine and therefore
When using serious games in health care, end users (clinicians, the most logical to encompass in the framework presented in
patients, or educators) must decide whether games are safe and this article [27,28].
effective enough to be used for their intended purposes. In order In summary, this consensus-based tool provides the end users
to do so, they need consistent, transparent, and reliable the support required when assessing the effectiveness and
assessments. Are applied games really stating their claim in this relevance of serious games in health care. An FDA-approval or
field? In the framework described in this article, both developers CE-mark is simply insufficient for this purpose. In order to
and end users are supported in assessing relevance, validity, prevent wrongful application and data theft of unsuspecting
and data safety of an applied game. In order to become a patients or medical students, this information on medical serious
“qualified game”, developers should disclose comprehensive games should become publically available to all end users. This
information on their products and claims. They must provide will aid the prescription of safe and effective games to patients
transparency to meet the standards. The Journal of Medical and the implementation of games into educational programs.
Internet Research and the Dutch Society for Simulation in
Healthcare [16] have launched an international peer-reviewing
initiative for serious games in health care.

Acknowledgments
The authors received funding from the Patient safety project (grant ref PID 101060), of the Pieken in de Delta-program by the
Dutch Ministry of Economic Affairs, Agriculture and Innovation, the city of Utrecht and the province of Utrecht, the Netherlands.
The funding agency had no role in the preparation of the consensus tool, nor the preparation of the manuscript.
The Committee for Serious Gaming committee of the Dutch Society for Simulation in Healthcare (DSSH) consist of the following
members: M Graafland (Department of Surgery, Academic Medical Centre, Amsterdam, The Netherlands); MEW Dankbaar

http://games.jmir.org/2014/2/e11/ JMIR Serious Games 2014 | vol. 2 | iss. 2 | e11 | p. 6

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR SERIOUS GAMES Graafland et al

(Department of Medical Education (Desiderius School), Erasmus University Medical Centre, Rotterdam, The Netherlands); A
Mert (National Military Rehabilitation Centre, Doorn, The Netherlands); J Lagro (Department of Geriatrics, Radboud University
Medical Centre, Nijmegen, The Netherlands); LD de Wit-Zuurendonk (Department of Obstetrics and Gynaecology, Máxima
Medical Centre, Veldhoven, The Netherlands); SCE Schuit (Departments of Emergency Medicine and Internal Medicine, Erasmus
University Medical Centre, Rotterdam, The Netherlands); A Schaafstal (Department of ICT, Windesheim University of Applied
Sciences, Zwolle, The Netherlands); and MP Schijven (Department of Surgery, Academic Medical Centre, Amsterdam, The
Netherlands).

Conflicts of Interest
None declared.

References
1. Kueider AM, Parisi JM, Gross AL, Rebok GW. Computerized cognitive training with older adults: a systematic review.
PLoS One 2012;7(7):e40588 [FREE Full text] [doi: 10.1371/journal.pone.0040588] [Medline: 22792378]
2. Gamito P, Oliveira J, Lopes P, Brito R, Morais D, Silva D, et al. Executive functioning in alcoholics following an mHealth
cognitive stimulation program: randomized controlled trial. J Med Internet Res 2014;16(4):e102 [FREE Full text] [doi:
10.2196/jmir.2923] [Medline: 24742381]
3. Prange GB, Kottink AI, Buurke JH, Eckhardt MM, van Keulen-Rouweler BJ, Ribbers GM, et al. The Effect of Arm Support
Combined With Rehabilitation Games on Upper-Extremity Function in Subacute Stroke: A Randomized Controlled Trial.
Neurorehabil Neural Repair 2014 May 29. [doi: 10.1177/1545968314535985] [Medline: 24878589]
4. Majumdar D, Koch PA, Lee H, Contento IR, Islas-Ramos AD, Fu D. "Creature-101": A Serious Game to Promote Energy
Balance-Related Behaviors Among Middle School Adolescents. Games Health J 2013 Oct;2(5):280-290 [FREE Full text]
[doi: 10.1089/g4h.2013.0045] [Medline: 24761326]
5. Cooper H, Cooper J, Milton B. Technology-based approaches to patient education for young people living with diabetes:
a systematic literature review. Pediatr Diabetes 2009 Nov;10(7):474-483. [doi: 10.1111/j.1399-5448.2009.00509.x] [Medline:
19490492]
6. Read JL, Shortell SM. Interactive games to promote behavior change in prevention and treatment. JAMA 2011 Apr
27;305(16):1704-1705. [doi: 10.1001/jama.2011.408] [Medline: 21447802]
7. Kato PM, Cole SW, Bradlyn AS, Pollock BH. A video game improves behavioral outcomes in adolescents and young
adults with cancer: a randomized trial. Pediatrics 2008 Aug;122(2):e305-e317 [FREE Full text] [doi: 10.1542/peds.2007-3134]
[Medline: 18676516]
8. Graafland M, Schraagen JM, Schijven MP. Systematic review of serious games for medical education and surgical skills
training. Br J Surg 2012 Oct;99(10):1322-1330. [doi: 10.1002/bjs.8819] [Medline: 22961509]
9. Buttussi F, Pellis T, Cabas Vidani A, Pausler D, Carchietti E, Chittaro L. Evaluation of a 3D serious game for advanced
life support retraining. Int J Med Inform 2013 Sep;82(9):798-809. [doi: 10.1016/j.ijmedinf.2013.05.007] [Medline: 23763908]
10. Lagro J, van de Pol MH, Laan A, Huijbregts-Verheyden FJ, Fluit LC, Olde Rikkert MG. A Randomized Controlled Trial
on Teaching Geriatric Medical Decision Making and Cost Consciousness With the Serious Game GeriatriX. J Am Med
Dir Assoc 2014 Jun 6. [doi: 10.1016/j.jamda.2014.04.011] [Medline: 24913210]
11. Forsberg A, Nilsagård Y, Boström K. Perceptions of using videogames in rehabilitation: a dual perspective of people with
multiple sclerosis and physiotherapists. Disabil Rehabil 2014 May 16:1-7. [doi: 10.3109/09638288.2014.918196] [Medline:
24833535]
12. Graafland M, Vollebergh MF, Lagarde SM, van Haperen M, Bemelman WA, Schijven MP. A Serious Game Can Be a
Valid Method to Train Clinical Decision-Making in Surgery. World J Surg 2014 Aug 27. [doi: 10.1007/s00268-014-2743-4]
[Medline: 25159118]
13. van Velsen L, Beaujean DJ, van Gemert-Pijnen JE. Why mobile health app overload drives us crazy, and how to restore
the sanity. BMC Med Inform Decis Mak 2013;13:23 [FREE Full text] [doi: 10.1186/1472-6947-13-23] [Medline: 23399513]
14. Connolly TM, Boyle EA, MacArthur E, Hainey T, Boyle JM. A systematic literature review of empirical evidence on
computer games and serious games. Computers & Education 2012 Sep;59(2):661-686. [doi: 10.1016/j.compedu.2012.03.004]
15. Kotz D. A threat taxonomy for mHealth privacy. In: Crowcroft J, Manjunath D, Misra A. editors Third International
Conference on Communication Systems and Networks (COMSNETS 2011).: IEEE; 2011 Presented at: Third International
Conference on Communication Systems and Networks; 2011; New York, NY p. 1. [doi: 10.1109/COMSNETS.2011.5716518]
16. Dutch Society for Simulation in Healthcare. Official Website Internet. DSSH official website 2014:2014.
17. Lewis TL. A systematic self-certification model for mobile medical apps. J Med Internet Res 2013;15(4):e89 [FREE Full
text] [doi: 10.2196/jmir.2446] [Medline: 23615332]
18. Albrecht UV. Transparency of health-apps for trust and decision making. J Med Internet Res 2013;15(12):e277 [FREE
Full text] [doi: 10.2196/jmir.2981] [Medline: 24449711]
19. Journal of Medical Internet Research (JMIR). 2014. Apps Peer-Review launched InternetURL: http://mhealth.jmir.org/
announcement/view/67 [accessed 2014-11-04] [WebCite Cache ID 6TpSDiKIJ]

http://games.jmir.org/2014/2/e11/ JMIR Serious Games 2014 | vol. 2 | iss. 2 | e11 | p. 7

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR SERIOUS GAMES Graafland et al

20. Michael DR, Chen S. Serious Games: Games That Educate, Train, and Inform. Boston, MA: Course Technology PTR;
2006.
21. Jalink MB, Goris J, Heineman E, Pierie JP, ten Cate Hoedemaker HO. Construct and concurrent validity of a Nintendo Wii
video game made for training basic laparoscopic skills. Surg Endosc 2014 Feb;28(2):537-542. [doi:
10.1007/s00464-013-3199-6] [Medline: 24061627]
22. Badurdeen S, Abdul-Samad O, Story G, Wilson C, Down S, Harris A. Nintendo Wii video-gaming ability predicts
laparoscopic skill. Surg Endosc 2010 Aug;24(8):1824-1828. [doi: 10.1007/s00464-009-0862-z] [Medline: 20108147]
23. SCORM. SCORM Explained InternetURL: http://scorm.com/scorm-explained/ [accessed 2014-11-04] [WebCite Cache
ID 6TpSOWvl4]
24. Anderson LW, Krathwohl DR, Bloom B. A taxonomy for learning, teaching, and assessing: a revision of Bloom's taxonomy
of educational objectives. New York, NY: Longman; 2001.
25. Van Staalduinen JP, de Freitas S. A Game-Based Learning Framework: Linking Game DesignLearning Outcomes. In:
Khine MS, editor. Learning to Play. New York, NY: Peter Lang Publishing, Inc; 2011:29.
26. American Educational Research Association, American Psychological Association, National Council on Measurement in
Education. Standards for educational and psychological testing. Washington, DC: American Educational Research
Association; 1999.
27. Schijven MP, Jakimowicz JJ. Validation of virtual reality simulators: Key to the successful integration of a novel teaching
technology into minimal access surgery. Minim Invasive Ther Allied Technol 2005;14(4):244-246. [doi:
10.1080/13645700500221881] [Medline: 16754170]
28. Gallagher AG, Ritter EM, Satava RM. Fundamental principles of validation, and reliability: rigorous science for the
assessment of surgical education and training. Surg Endosc 2003 Oct;17(10):1525-1529. [doi: 10.1007/s00464-003-0035-4]
[Medline: 14502403]
29. Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R. What counts as validity evidence? Examples and prevalence in
a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract 2014 May;19(2):233-250. [doi:
10.1007/s10459-013-9458-4] [Medline: 23636643]
30. Health on the Net Foundation Code of Conduct (HONcode). Operational definition of the HONcode principles InternetURL:
http://www.hon.ch/HONcode/Webmasters/Guidelines/guidelines.html [accessed 2014-11-04] [WebCite Cache ID 6TpScFyun]
31. Bellotti F, Kapralos B, Lee K, Moreno-Ger P, Berta R. Assessment in and of Serious Games: An Overview. Advances in
Human-Computer Interaction 2013;2013:1-11. [doi: 10.1155/2013/136864]
32. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application.
Am J Med 2006 Feb;119(2):166.e7-166.16. [doi: 10.1016/j.amjmed.2005.10.036] [Medline: 16443422]

Edited by G Eysenbach; submitted 31.08.14; peer-reviewed by I Paraskevopoulos; comments to author 10.09.14; revised version
received 15.10.14; accepted 17.10.14; published 11.11.14
Please cite as:
Graafland M, Dankbaar M, Mert A, Lagro J, De Wit-Zuurendonk L, Schuit S, Schaafstal A, Schijven M
How to Systematically Assess Serious Games Applied to Health Care
JMIR Serious Games 2014;2(2):e11
URL: http://games.jmir.org/2014/2/e11/
doi: 10.2196/games.3825
PMID: 25654163

©Maurits Graafland, Mary Dankbaar, Agali Mert, Joep Lagro, Laura De Wit-Zuurendonk, Stephanie Schuit, Alma Schaafstal,
Marlies Schijven. Originally published in JMIR Serious Games (http://games.jmir.org), 11.11.2014. This is an open-access article
distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which
permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Serious
Games, is properly cited. The complete bibliographic information, a link to the original publication on http://games.jmir.org, as
well as this copyright and license information must be included.

http://games.jmir.org/2014/2/e11/ JMIR Serious Games 2014 | vol. 2 | iss. 2 | e11 | p. 8

(page number not for citation purposes)
XSL• FO
RenderX