Computer Science Review 27 (2018) 33–44

Contents lists available at ScienceDirect

Computer Science Review

journal homepage: www.elsevier.com/locate/cosrev

Review article

Publication trends in gamification: A systematic mapping study

Jussi Kasurinen a , Antti Knutas b, *
a
South-Eastern Finland University of Applied Sciences, Kotka Campus, Pääskysentie 1, 48220 Kotka, Finland
b
Lero, The Irish Software Research Centre, Dublin 13, Glasnevin, Ireland

highlights

• This study presents four big trends and the core literature in gamification.
• This publication provides a meta-review of several other literature reviews.
• This publication identifies the most common publication venues on gamification.
• The study concludes that the most pressing research issue currently is to collect evidence on the practical applications of gamification.
• Most common theme in gamification studies currently is education.

article info a b s t r a c t
Article history: The term gamification and gamified systems are a trending area of research. However, gamification can
Received 22 October 2016 indicate several different things, such as applying the game-like elements into the design of the user
Received in revised form 29 October 2017 interface of a software, but not all gamification is necessarily associated with software products. Overall, it
Accepted 31 October 2017
is unclear what different aspects are studied under the umbrella of ‘gamification’, and what is the current
state of the art in the gamification research. In this paper, 1164 gamification studies are analyzed and
classified based on their focus areas and the research topics to establish what the research trends in
Keywords:
Gamification gamification are. Based on the results, e-learning and proof-of-concept studies in the ecological lifestyle
Systematic literature review and sustainability, assisting computer science studies and improving motivation are the trendiest areas
Proof-of-concept studies of gamification research. Currently, the most common types of research are the proof-of-concept studies,
Serious games and theoretical works on the different concepts and elements of gamification.
MOOCs © 2017 Elsevier Inc. All rights reserved.
Crowdsourcing
Games for health

Contents

1. Introduction......................................................................................................................................................................................................................... 34
2. Related research.................................................................................................................................................................................................................. 34
3. Research method................................................................................................................................................................................................................. 35
3.1. Data collection process .......................................................................................................................................................................................... 35
3.1.1. Inclusion and exclusion criteria ............................................................................................................................................................. 36
3.2. Review process ....................................................................................................................................................................................................... 36
4. Results.................................................................................................................................................................................................................................. 37
4.1. General classification ............................................................................................................................................................................................. 37
4.2. Proof-of-concept classifications ............................................................................................................................................................................ 39
4.3. Summary of the identified literature reviews ...................................................................................................................................................... 39
4.4. Identifying core literature related to the publications ........................................................................................................................................ 41
4.5. Implications of the analysis ................................................................................................................................................................................... 41
5. Discussion............................................................................................................................................................................................................................ 42
6. Conclusions.......................................................................................................................................................................................................................... 42
Acknowledgments .............................................................................................................................................................................................................. 43
References ........................................................................................................................................................................................................................... 43

* Corresponding author.
E-mail address: antti.knutas@dcu.ie (A. Knutas).

https://doi.org/10.1016/j.cosrev.2017.10.003
1574-0137/© 2017 Elsevier Inc. All rights reserved.
34 J. Kasurinen, A. Knutas / Computer Science Review 27 (2018) 33–44

Nomenclature

SMS Systematic Mapping Study

SLR Systematic Literature Review
MOOC Massive Online Open Course

1. Introduction

Gamification is a topic, which has been considered one of the

significant new trends in the development of services and ap-
plications in the software industry. Fundamentally, gamification Fig. 1. The axis of different gamification-related concepts as defined by Deterding
means that some system applies game-like elements to enhance et al. [2].
the user participation, the motivation to keep using the said system
or the retention rate to keep the existing customers. These benefits
are important, since for example in the mobile games industry, a 2. Related research
retention rate of 14.5 percent would be considered normal, with
as low as five percent of the customers paying money for the Gamification is a vague term, since there are several similar
service [1]. Simplified explanation also could be that gamification concepts such as playful design, serious gaming and even games for
is a design concept for user interface design, drawing its roots from health, which all have conceptually similar definitions and effects
the 80’s and the studies on ‘enjoyable user interfaces’ [2], or first to the objectives of gamified design. The most important aspect of
impressions [3]. However, the motivational context of using the
understanding gamification is to understand how the gamification
services is not a novel concept, and has been studied for example
differs from the concepts of playful design and serious games. For
in the context of computer science education [4].
this purpose, for example Deterding et al. [2] offer a classification
Within the context of gamification, there are several contexts,
of the different terms based on the extensity and method of their
which are considered sort of major application domains. These
use. Their definitions propose a following solution:
domains are computational fields or business domains, where the
motivational aspects, the increased retention rate and the en- • Gameful design (Gamification) implies that the software
hanced participation are very useful, such as in the serious games product is designed with the game-like components form-
including games for health [5], crowdsourcing [6], and in online ing a part of the system design, but the product itself has a
education [7]. Overall, it seems that the gamification domain is functional non-game purpose and elements, which are not
huge, and new areas of application are discovered continuously.
game-like.
In this paper, the objective is to assess the entire domain of gam-
• Serious Games (and other games) are products, which are
ification and its applications by the means of a systematic mapping
fully built from the game components and game-like ele-
study and literature review, conducted with two engineering-
ments. If the product has a ‘‘real’’ purpose, it can be classi-
oriented research search engines and with two general research
fied as a Serious Game, otherwise normal — entertainment-
search engines. The study systematically codifies 1164 items col-
game.
lected from ACM Digital library, IEEE Xplore, Web of Science and
Google Scholar, and overall identifies over 1100 different items
• Playful design implies a system, which has playful elements
from approximately 2800 authors. Out of the identified documents, in its design, but also has components which are not playful
over 900 papers were classified and categorized to understand and the system has a non-playful, real-life, purpose.
the focus and the current themes of the gamification research. • Toys are products which are fully designed for enabling
The research questions of this paper are ‘‘What are the current use in play, and do not have an intended non-play purpose
trends in the gamification research?’’ and ‘‘Where is the effort in the beyond entertainment.
gamification research focused?’’.
In essence, this model divides the field into the vectors of the
Based on the observations, the two most prominent types of
entertainment purpose of the product, and the extent of the effect
research domains in gamification are the applications of gamifi-
in the product design. It is worth observing, that the ‘‘traditional’’
cation in the online education, especially in the development of
software products are not visible in this model, since they are not
massive online open courses (MOOCs), and in the development
of prototype tools and systems, which apply some form of gam- designed for gaming or playing, and they only have the functional,
ification. Third large topic of published research is the studies of non-gamified purpose. In addition of Deterding’s model (2011),
applicability of the different gamification approaches, the general there is also a definition by Huotari and Hamari [8] from the
theory regarding the topic. Out of the prototyping disciplines, perspective of service marketing, where the gamification is defined
computer science education, ecological sustainability and general as ‘‘Gamification is a form of service packaging where a core service
motivation enhancement tools are the most common areas of is enhanced by a rules-based service system that provides feedback
applied gamification. Overall, the different research publications and interaction mechanisms to the user with an aim to facilitate
of gamification research form six major categories, from which the and support the users’ overall value creation’’ [9]. In the context
educational topics are significantly larger than the others, with of the ludification culture, there also exists an extended model by
the software development related solutions and crowdsourcing Deterding et al. [2], but for the purposes of this literature review
applications trending upwards. this simplified model can be applied; between the different areas
The rest of this paper is structured as follows: In Section 2, the of gamified design, the divisions are illustrated in Fig. 1.
terminology and research work related to this study is defined. Sec- Gamified design has also some areas of application with rel-
tion 3 defines the research approach and the classification method, atively common, recurring themes such as the crowdsourcing,
whereas Section 4 introduces the results. Section 5 discusses the Games for Health and MOOCs. For example, a definition for crowd-
limitations and implications of this work, and Section 6 closes the sourcing by Estelles-Arolas and Gonzalez-Ladron-de Guevara [6]
paper with the conclusions. dictates that the crowdsourcing is a type of participative online
 J. Kasurinen, A. Knutas / Computer Science Review 27 (2018) 33–44 35

Table 1
Data analysis steps.
Step Procedures Results
1. Determine search terms. Determine the search terms from accepted field keywords Boolean search terms that get the desired results from
that cover the desired topics. the databases in the next steps.
2. Determine databases. List the databases that cover most of the publications in the A list of databases for steps three and four.
chosen field of science.
3. Run a test search. Select one database and run a search to test the validity of the Verification that the search terms will return the
search terms. desired type of publications.
4. Run a full search and store the results. Search all the selected databases and store the results. A list of all publications that match the search terms.
5. Deduplicate and sort according to Remove duplicate results and then use the inclusion and The final list of articles that will be included in the
inclusion and exclusion criteria. exclusion criteria to select the articles for in-depth analysis. systematic map.
6. Analyze the query results. Review the articles and record the determined data and A systematic map of the chosen section of research
metadata. Analyze and compare the research articles and literature.
their research approaches.

activity, where a group of individuals undertake tasks of varying approaches can be used to identify research gaps in the current
difficulty for a mutual benefit. Basically, crowdsourcing is an act of state of research, but usually SMS is more applicable if the problem
using volunteering human resources to achieve ability to conduct or topic is more generic in the nature.
activities such as image labeling, which would be difficult for a A systematic mapping study classifies and structures a field
computer to handle, and laborious for a small group of people. In of interest in research by categorizing publications and analyzing
the gamified context, the system might for example keep a score their publication trends [11]. Additionally, SMS can analyze what
on the amount and accuracy of the labels each user has given, and kind of studies have been done in the field, and what are the
offer leaderboards or possibility to use some system feature after research methods and outcomes [14]. In Table 1 we present how
attaining a certain score or status. we have used the systematic mapping study method created by
On the other common domains of gamification, Games for Bailey et al. [14] for the field of software engineering, developed
Health are a subset of serious games, where the concept of the further by Petersen et al. [11].
game is to improve the health of the player, either via exercise-
inducing games (for example Göbel et al. [5]) or via promoting 3.1. Data collection process
the health-enhancing activities. The scale of the gamification may
change from offering differently themed backgrounds or trophies Analytical assessment of an entire research area or scientific do-
for achieving certain milestones as rewards, to a full-fledged game, main is a laborious task. To understand the current trends and the
which enables the player to physically train while playing the most relevant papers, we decided to include four search engines
game. into the collection of the studies: ACM Digital Library, IEEE Xplore
Finally, MOOCs (Massive Open Online Courses) (for example Digital Library, Google Scholar and Web of Science Core Collection.
Alario-Hoyos et al. [7]) are Internet courses, which are offered to Initially, the ACM and IEEE xplore databases found a total of 954
every interested party, regardless of their institutional association papers which had the term ‘‘gamification’’ in either in the body
or other background information. In the MOOC systems, the course text, title, abstract or as a keyword. These were all included to the
contents and participation is open to everyone, but for example analysis database, which was maintained with the Zotero reference
official endorsements or graduation diplomas require a separate tool2 and its reference database system.
payment and registration [7]. The MOOCs may also involve gam- As the ACM and IEEE databases are engineering- and computer
ified design in the courseware system to motivate the student, science-specific databases, Google Scholar and Web of Science
increase the retention rate or enhance the amount time spent with were included to the data sources to collect the relevant papers
the course assignments and other self-study material. For open, from the other disciplines. From the Web of Science, the Core
commercial examples, for example CodeCademy1 offers open, Collection was used to narrow the search scope to the scientific
gamified courses on learning introductory-level programming, and publications. In Google Scholar the additional conditions ‘‘papers
other computer science topics. in English only’’ and not from sites ‘‘dl.acm.org" or ‘‘ieeexplore.
ieee.org" were used to further limit the amount of unnecessary
3. Research method duplicate entries. Additionally, since Google Scholar and Web of
Science Core Collection are general-topic databases, only the re-
A systematic mapping study (SMS) is a secondary study that sults which had more than 2 reported citations were included into
aims at classification and thematic analysis of earlier research this analysis to maintain the relevance of the entries. This was
[10,11]. It is closely related to a wider secondary study, a system- especially important with the Google Scholar, since this filtering
atic literature review (SLR), which aims at gathering and evalu- reduced the amount of papers from 6810 to 214, which was in
ating all the research results on a selected research topic [12,13]. line with the other search engines for relevant hits, and enabled
Kitchenham and Charters [10] present the best practices of both for us to do classification to all of the collected papers. During the data
the field of software engineering and also compare the two. The collection some random inspections of the search engine accuracy
SMS is more general in its search terms, and aims at classifying was conducted by taking a list of references from one paper, and
and structuring the field of research, whereas the target of SLR seeing if snowballing [11] this reference list yielded objects which
is to conclusively summarize and evaluate the research results. were not on the dataset. However, due to the large amount of
Kitchenham and Charters [10] also discuss the applications and papers in the dataset, systematic approach on following reference
states that the SMS can be especially suitable if only a few literature trails (aka. snowballing) was not possible to conduct.
reviews have been done on the selected topic, and there is a need to The data was collected during the September of 2015, with the
get a general overview of the field. Regardless of the selection, both raw-data database with the duplicate entries removed is available

1 https://www.codecademy.com/. 2 https://www.zotero.org/.
36 J. Kasurinen, A. Knutas / Computer Science Review 27 (2018) 33–44

Table 2
Data analysis steps.
Round Step Action Amount of Documents after Action
1 Data Collection Gathering documents to the database. 1207
2 Automated inspection of collected data. Removal of the duplicates. 1164
3 Manual inspection of collected data. Classification based on the title and abstract, removal of 775
the items with missing info and meta papers.
4 Analysis of the identified proof-of-concept studies. Based on the classification, identification of POC themes. 310 (out of 775)
5 Analysis of the identified literature reviews. Based on classification. 18 (out of 775)

Table 3
Included data sources.
Source Hits Relevant Search and inclusion criteria
ACM Digital Library 707 707 ‘‘gamification’’ in document
IEE Explore Digital Library 247 247 ‘‘gamification’’ in document
Google Scholar 6810 214 ‘‘gamification’’ in document, English only,
-site:dl.acm.org
-site:ieeexplore.ieee.org.
more than 2 citations
Web of Science Core Collection 351 39 ‘‘gamification’’ in document, years 2002–2015,
more than 2 citations

online as a flat file database stored in CSV format.3 The data 3.2. Review process
collection took a total of two weeks, since some of the data sources
were strict against using any form of automation to collect the As a first step after collecting the data, an initial automated
document information and metadata into the database. This meant inspection and removal of duplicate entries was done based on
that in some occasions, the entries had to be imported manually, the suggestions by the Zotero tool. After this action, the database
one by one, to the Zotero database since there were no sufficiently was exported as a CSV file, and analyzed with Python scripts to
advanced automated tools available at the time. The analysis steps extract the titles, publication places, years, abstracts and authors to
are summarized in Table 2, and the data sources in Table 3. a separate Excel spreadsheet, which was used in the classification
and manual inspection of the collected data. In more technical
3.1.1. Inclusion and exclusion criteria detail, the CSV file was converted into a list-array with all the
A total of 1207 conference and journal articles were found in data simplified to lowercase letters without preceding or trailing
the database searches. These articles were first reviewed by the
whitespaces to minimize duplicate entries, and analyzed with the
title, keywords and abstract. In the first round of review, articles
list management tools of Python. The resulting calculations were
that did not in any way discuss gamification or were written in
then written to a new output-CSV-file, which was further analyzed
other languages than English were dropped from the study. After
in MS Excel tool.
the first round, 1164 articles were selected for an in-depth review
In addition of manual classification, the document abstracts
and comparison against the inclusion and exclusion criteria.
were also sorted into topics using the Latent Dirichlet Allocation
During the preliminary review that included data consolidation
(LDA) topic modeling algorithm [15] in order to triangulate the
and trimming process, the following inclusion and exclusion crite-
ria were applied to the remaining articles. The inclusion criteria in findings. LDA can be used as a statistical text mining method for
this study were discussion of the following topics: assigning documents into topics, which are detected using word
association and distributions [16]. It is a commonly used method
• gamification in any application domain (such as healthcare, for text analysis and equivalent methods have been used to statisti-
crowdsourcing, education etc.) cally analyze scientific texts in number of previous studies [17,18].
• relevance in general search engine terms; since Google A modified version of the nails script was used to perform the
Scholar and Web of Science combined gave several thousand topic modeling [19], which uses the R topicmodels library [20].
hits, only the papers with at least two recorded references Additionally, the LDAvis library was also used to calculate the
were included to the analysis to maintain relevance and distance between topics on a scatterplot, which approximates the
eliminate non-research and non-peer-reviewed papers dis- semantic relationships between the topics with multidimensional
cussing the topic, such as popularized articles, advertise- scaling [21]. It is a method similar to factor analysis and allows the
ments, thesis works, study reports etc. level of similarity between objects to be visualized.
During the manual inspection step, the documents were classi-
The excluded categories in the papers were:
fied based on their title and abstract. The classification was done
• literature surveys with no original research (study reports), following the basic principles of open coding (for example [22]):
• paper was not written in English, For each new type of paper a new class is created, or existing class
• papers not subject to peer review, or is extended to cover the topic presented in the paper. During the
• papers not considering the research topic from the perspec- codification, classes were also merged, for example ‘‘eLearning’’
tive of gamification. and ‘‘MOOCs’’, to form larger categories as the amount of codi-
fied entries increased. The only seed classes [23] were the ‘‘Not
If a paper discussed gamification and only marginally touched relevant/rejected’’ and ‘‘Meta/Discussion/Keynote’’ for identifying
the inclusion criteria, it was still included in this SMS study in order papers which were out of scope, or did not have peer-reviewed
to give as comprehensive a view of research as possible. After this content. After the classifying the identified papers, following gen-
final round of filtering, a total of 775 articles were selected to be eral classes were identified:
included in the systematic literature review. Theory/elements: Publication focuses on the theory of gam-
ification or on the elements, which are associated with the
3 http://www2.it.lut.fi/GRIP/free_pubs/gamification_sept2015_nodup_data.zip. gamification.
 J. Kasurinen, A. Knutas / Computer Science Review 27 (2018) 33–44 37

POC:what: Publication discusses a proof-of-concept-work

which goes beyond describing an idea for a system or a mechanism.
A short (1–3 words) description of the domain the POC is related
to was included for further classification.
Meta/Discussion/Keynote: The paper discusses a concept rele-
vant to the gamification, but is not a peer-reviewed scientific pub-
lication. This includes items such as editor’s notes, book reviews,
workshop summaries, track summaries and author interviews.
Serious: The publication self-identifies as a ‘‘serious game’’ ap-
plication rather than a gamified application or service.
G4H/healthcare: The publication self-identifies as a ‘‘Games for
Health’’-application or other health-related application.
Crowdsourcing: The publication self-identifies as a crowdsourc-
ing application.
Not-relevant/No-abstract/Other: The publication was not rele-
vant to the gamification domain, or it was mislabeled by the
search engine, or the data collection tool was unable to collect the
classification information (no abstract, or no publication venue).
Business: The publication focuses on the business management
aspects or the new business models related to the gamification.
Fig. 2. Topics discovered with the LDA analysis.
Technology: The publication focuses on the new technology,
hardware or technological solutions related to the gamification.
MOOC/eLearning: The publication focuses on the applications
of gamification in a learning and teaching context, including e- also identified. 122 papers were classified as meta topics, which
learning, MOOCs and gamified course structures. included discussion and position papers, book reviews and keynote
Literature review: The publication is another systematic map- summaries. On the divisions by the topics and domains, only 30 pa-
ping study or literature review on the gamification, or a study pers focused on the serious games-aspect, meaning that the paper
summarizing the existing studies on gamification. self-identified as a paper describing a serious game-related con-
In total, the data collection resulted in 1207 relevant items, out cept or theory. Similarly, 109 papers identified as games for health-
of which 43 were duplicates, leading to the analysis set of 1164 related paper, while crowdsourcing applications or enhancement
unique documents. From this dataset, 569 different publication of the motivation or accuracy of the crowdsourcing work was
venues were identified, with the publication types dividing to discussed on 104 papers. Out of the 1164, 50 papers were directly
813 conference papers, 249 journal articles, 78 book chapters, 20 related to the business aspects, usually management of team work,
complete books and 4 classified as ‘‘others’’. Out of the different and 57 papers discussed the new technology or technical aspects
venues, ten major publication venues with ten or more papers of implementing the gamified systems. Largest of the domains
were identified. was learning, with 206 papers discussing the e-learning, gamified
For the dataset, the next steps were to identify the domains, course structures or application of MOOC services. Also, out of
to which the different proof-of-concept-studies were related, and the 1164 papers, 18 literature reviews, mapping studies or other
collect more information on the literature reviews, which would papers discussing the current state of the gamification were iden-
be useful in establishing information on the current trends and tified.
research areas of gamification. On this manual inspection all the In total, 267 papers were rejected, mostly because the system
papers, which did not include abstract, or were behind paywall, or could not access an abstract, or did not include the publication
were considered not relevant or not peer reviewed, were rejected. venue. In some occasions, the paper did not discuss anything
Other papers were classified based on their abstracts, and besides gamification-related, even if the ‘‘gamification’’ was used as a key-
classification, also subjected to the statistical analysis with the LDA word. The 78 book chapters and 20 books which the data collection
approach. In addition of these actions, the identified systematic identified, were also classified as ‘‘meta/discussion’’, since the pub-
literature reviews were additionally manually inspected and dis- lication process for the book articles and books was not transparent
seminated for a short summary of contents. or clearly defined. Because of this, there was an ambiguity if these
papers had been rigorously peer-reviewed, so they were rejected.
4. Results All of the presented numbers are summarized in Table 4.
With the LDA topic modeling text mining method the abstracts
In this section, the results of the manual inspection and classifi- were divided into six topics. Fig. 2 shows a visualization of the
cation are introduced. In the first subsection, the general statistics topics and lists four most descriptive words for each topic. The
are discussed. In the second section, the division of the different relative sizes of the circles represent the prevalence of the topic in
proof-of-concept works are described, and in the third section, the the dataset and locations represent intertopic distances calculated
systematic mapping studies are listed. Finally, some implications with Jensen–Shannon divergence [21]. The six discovered topics
of the results are presented. can be summarized as education, industry, (children’s physical)
activity, crowdsourcing, mobile health and (software and research)
4.1. General classification development community.
The volume of publications in each LDA-detected topic, divided
Out of the 1164 papers identified via the data collection steps, and normalized by year, can be seen in Fig. 3. The most numerous
252 papers discussed the gamification theory in general, or were topic in each year is educational (T1), with (children’s physical)
focused on one aspect or component, which was studied further activity (T3) close and development community (T6) on the rise.
for in the use in the gamifying products and services. 308 dif- The volume for business-related publications (T2) remains low.
ferent proof-of-concept-papers, which described in detail some The author information was also assessed as a part of the
functional prototype or more sophisticated gamified service, were data analysis. The most published author in gamification research,
38 J. Kasurinen, A. Knutas / Computer Science Review 27 (2018) 33–44

Table 4
Identified paper classes from the 1164 papers.
Class Papers % of All Description
(out of 1164)
General Theory and Elements 252 21.6% This publication focuses on the general gamification
theory or gamification elements, providing an insight or a
new framework for studies.
Proof of Concepts 308 26.5% This publication focuses on the introduction of a new
gamified service or application, in some functional state of
usability.
POC — Computer Science teaching tool 29 9.4% The presented application focuses on teaching computer
(out of 308) (out of 308) science discipline; for example learning programming, or
learning test case design.
POC — Ecologically Friendly Lifestyle or 25 8.1% The presented application focuses on teaching ecologically
Sustainability-related tool (out of 308) (out of 308) friendlier lifestyle or the aspects of sustainability; for
example conserving electricity, recycling or otherwise
preserving the environment.
POC — Motivation Improvement tool 25 8.1% The presented application focuses on improving the
(out of 308) (out of 308) motivation for conducting some manual task; for example
classifying images or completing given task lists.
POC — Software Development tool 23 7.4% The presented application focuses on improving the
(out of 308) (out of 308) motivation to do software development; for example
identifying requirements or participating on the design
work.
POC — Social behavior 21 6.8% The presented application focuses on improving the social
(out of 308) (out of 308) interactions between different stakeholders.
POC — eGovernment 11 3.6% The presented application focuses on gamifying some
(out of 308) (out of 308) aspect of the interaction between the government and the
citizens.
POC — Non-CS STEM-topic Teaching tool 11 3.6% The presented application focuses on teaching the topics of
(out of 308) (out of 308) science, technology, engineering or medicine, but not
computer science or computer science-related field.
POC — Business Management tool 11 3.6% The presented application focuses on the business
(out of 308) (out of 308) management; for example participation to meetings, or
logistics management.
POC — Physiotherapy Self-Training tools 11 3.6% The presented application focuses on helping the patients
(out of 308) (out of 308) to conduct independent physiotherapy training sessions.
POC — Museums and history 10 3.2% The presented application focuses on enhancing museum
(out of 308) (out of 308) visiting experience, or in general teaching history.
Meta, Discussion or Keynote summary papers 122 10.5% This paper is a discussion or position paper, an
advertisement or a summary of a conference track or a
keynote speech, or some other meta-topic publication.
Serious Games 30 2.6% This paper self-identifies primarily as a study on serious
games-focused topic.
Games For Health 109 9.4% This paper self-identifies primarily as a study on Games for
Health-focused topic.
Crowdsourcing 104 8.9% This paper self-identifies primarily as a study on
crowdsourcing-focused topic.
Not Relevant, No Abstract, Other Reason to Reject 267 22.9% This was the catch-all class for papers, which were
rejected based on the topic and abstract-level
classification, or which did not have a peer-review process.
Business-Focused Study 50 4.3% This paper discusses the application of gamification in the
real-world business management, or as a commercial
activity.
Technology-Focused Study 57 4.9% This paper discusses the technology and the development
of new technical solutions to enable gamification.
MOOC or eLearning-Focused Study 206 17.7% This paper discusses the development of a MOOC, or
another application of gamification in enabling online
learning.
Literature Review 18 1.5% This paper is a literature review, mapping study or other
paper summarizing the research in some topic of
gamification.

according to the data, was Juho Hamari from University of Tam- publications). Overall, 17 authors were identified to have five or
pere (11 publications), followed by Lennart Nacke from University more publications, producing 9.1 percent of all identified publica-
of Waterloo (10 publications), Daniel Johnson from Queensland tions. The full list of these 17 authors is presented in Table 5. In
University of Technology, Sebastian Deterding from Northeast- general, the literature review identified 2860 different contribut-
ern University and Oliver Korn from Offenburg University (all 7 ing authors, with 400 authors having more than 1 published study.
 J. Kasurinen, A. Knutas / Computer Science Review 27 (2018) 33–44 39

Fig. 4. Number of publications by year.

4.2. Proof-of-concept classifications

Fig. 3. LDA analysis topics, volume per year.
On the proof-of-concept studies, 308 papers describing the
different types of proof-of-concept systems were identified. The
Table 5
Top 10 categories in the different types of proof-of-concepts were
List of authors with more than five publications in the dataset.
tools aimed to improve computer science – or closely related, such
Author Number of publications
as software engineering or information science – course curricula
Hamari, Juho 11 (29 studies), tools designed to steer lifestyle choices and behavior
Nacke, Lennart 10
Johnson, Daniel 7
towards more ecologically friendly or sustainable habits (25 stud-
Korn, Oliver 7 ies), tools aimed to enhance the participant motivation towards
Deterding, Sebastian 7 their given tasks (25 studies), gamification tools aimed to enhance
Wyeth, Peta 6 or support software development (23 studies), gamification to
Schmidt, Albrecht 6
improve learning in the STEM fields other than computer science
Fitz-Walter, Zachary 6
Coulton, Paul 6 (11 studies), tools related to offering electronic government ser-
Kranz, Matthias 5 vices or empowering people in the decision-making process (11
Koivisto, Jonna 5 studies), tools related to the management of business activities
Gutwin, Carl 5 such as meetings or design sessions (11 studies), physiotherapy
Hörz, Thomas 5
Costa, Carlos 5
tools (11 studies) and tools related to the history, for example
Barata, Gabriel 5 museum information systems (10 studies). The Top 10 categories
Gama, Sandra 5 were also all of the categories, which had ten or more published
Tjondronegoro, Dian 5 studies. In the position 11–20 there were additional papers in
healthcare, with seven papers discussing general healthcare topics
and six focusing on speech therapy. In Top 11–20, the domains
The publishing years were all relatively recent, similarly as and topics were of limited scope, such as learning how to drive car
observed by Hamari et al. [24]. Only one publication was identified safely, changing and managing eating habits, or managing online
to be published on the year 2010, and everything else was between payments and other bank transactions. In total, the Top 10 proof of
2011 and 2015. As identifiable from Fig. 4, the number of published concept-classifications covered 61 percent of all POC studies, and
papers has been on a near-linear rise, with the 2014 being the Top 20 covered 82.5 percent. The Top 1–21 topics are included in
busiest year so far with 432 publications on gamification. As for Fig. 5.
2015, since the entries for the database were collected during
the Q3/2015, it can be expected that the number of total pub-
4.3. Summary of the identified literature reviews
lications following the trend of +120 publications/year average
will be around 550 publications published in 2015. However, this
The data collection also included 18 papers, which were iden-
direct comparison against the calendar months may not give an
accurate view, since the relevant conferences and call for papers tified to be other systematic mapping studies, literature reviews,
for journals are not equally distributed throughout the year and or similar domain-spanning studies into the gamification. Reading
the proceedings and journal issues take time to be put available through the papers, five of these papers were rejected from being
online. On three papers the publication year was not captured by listed here as a summary, since they were either summarizing
the data collection system. certain subset such as conference events, did not contain an actual
The most common publication venues according to the analy- literature review, or were different versions of the same systematic
sis were the Computer Human Interaction Conference (CHI) pro- study. The following primary studies were identified:
ceedings with 46 entries from two years combined. Additionally,
the ACM conference on Human Factors in Computing was very • Hamari et al. [24] observed the empirical studies into gami-
active community with 25 entries. From journals, Personal and fication, identifying 24 peer-reviewed studies and reporting
Ubiquitous Computing by Springer, eLearn by ACM and Computers the outcomes and approaches applied. For example, the
in Human Behavior by Elsevier were the three most common different motivational affordances and motivators are iden-
journal publication venues. Overall, the data collection identified tified and classified from the different cases, psychological
570 different conferences and journal titles from the dataset, from and behavioral outcomes assessed, and the overall results
which the 20 most common venues are listed in Table 6. The classified. One of the major observations is that the applied
table separates the different types of proceedings according to gamification and its success is very context-dependent, be-
the publisher data, so some of the Top 20 publication venues are cause of the different intrinsic and extrinsic motivators as-
different types of contributions from the same main event. sociated to the different activities.
40 J. Kasurinen, A. Knutas / Computer Science Review 27 (2018) 33–44

Table 6
20 most common publication venues, journals and conferences.
Venue # of papers
CHI ’13 Extended Abstracts on Human Factors in Computing Systems 26
Proceedings of the First International Conference on Gameful Design, Research, and Applications 25
CHI ’14 Extended Abstracts on Human Factors in Computing Systems 20
eLearn 20
Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality 18
Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems 17
Proceedings of the First International Workshop on Gamification for Information Retrieval 15
Proceedings of the First ACM SIGCHI Annual Symposium on Computer–Human Interaction in Play 15
Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems 12
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 11
Personal Ubiquitous Computing 9
Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing 9
Proceedings of the 11th International Conference on Entertainment Computing 9
Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems 8
Proceedings of the 11th Conference on Advances in Computer Entertainment Technology 8
Proceedings of the 45th ACM Technical Symposium on Computer Science Education 8
Proceedings of the 24th International Conference on World Wide Web 8
Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing 7
2015 48th Hawaii International Conference on System Sciences (HICSS) 7
Computers in Human Behavior 7

Fig. 5. Proof-of-concept general topics with five or more publications (TOP 1–21).

• Khaleel et al. [25] identifies different gamification and reaches more population than any service or system ever
gameful design mechanics, based on the literature. The work before, allowing people to learn about different subjects,
itemizes several gamification and gameful design-related and having a new ‘‘digitally native’’ generation of software
user interface and content elements, explaining the differ- users. This study discusses the different motivational tactics
ences between the user satisfaction, useful interfaces and and applications with the means of gamification to enable
the aspect of fun from the viewpoint of UI design, and better results, and maintain the interest of the digitally
summarizes that the lack of gamification in any modern native generation, for which the traditional pedagogical
system might actually be more harmful, than the existence tools and support systems might be too restrictive and old-
of gamification that is not very functional. fashioned.
• Panchariya et al. [26] discusses the application of big data • de Sousa Borges et al. [28] systematically map studies, in
in different contexts, such as crowdsourcing and human– which the gamification is applied in the learning con-
computer interaction. Their work focuses on the data mining text, identifying 26 primary papers. Their mapping study
aspects of big data, enabling researchers to discover the indicates, that in the learning context, the most common
patterns and relationships between the data by applying types of papers either evaluate the existing systems, or
the citizen science and crowdsourcing approaches enriched offer solutions to the learning problems. Similarly, this study
with the gamification aspects. Their work also presents a identifies that there is a clear shortage of experience re-
geography-based example of the said systems called Geo- ports on the usage of gamified systems, and validation of
Tagger. the gamification-enabled learning. Additionally, most of the
• Sanmugam et al. [27] discusses the cognitive impact of dif- identified works study behavioral changes, improved learn-
ferent approaches to the gamification. The World Wide Web ing or engagement levels.
 J. Kasurinen, A. Knutas / Computer Science Review 27 (2018) 33–44 41

• Cheung et al. [3] analyzed over 200 game reviews and cus- • Morrison and DiSalvo [33] discuss the different motivational
tomer feedbacks to understand how the interface design aspects of teaching computer science with MOOC. Their
and first impressions affect the player retention rates. Their work focuses on the activities in Khan Academy, and discuss
observations on the first-hour analysis indicates, that the different objectives of goal setting, outcome expectations,
current game design does not help users who are seeking values and social comparisons. These topics are further dis-
information about the product or within the product, but is sected into number of processes, which all are aimed to-
designed to provide memorable activities and strong moti- wards increasing the motivational aspects of learning tech-
vation to keep using the product. This front-loaded design nically challenging topics such as software engineering.
does not take into account the user retention over a long • Pedreira et al. [34] performed a systematic mapping study
period of time, as for example there is little to no concern focusing on gamification in the software engineering, where
over the ‘‘last hour design’’, ensuring that the product offers they evaluate 29 primary studies published between Jan-
users satisfying end sequence. uary 2011 and June 2014. The authors conclude that many
of the studies focus on software development and to a lesser
• Knaving and Björk [29] derives the design principles for
extent on the requirements. The presented gamification
gamified systems from the successful gamification projects
mechanics are simple, such as points and badges and few
and applications. Their study implies, that several gamified
provide empirical evidence of the impact of gamification.
systems fail to capture their audience because the game-
They recommend future research focusing on the more ad-
like elements and reward systems are not as integrated to
vanced mechanics and software process areas that have not
the service as they could, but rather exist as an additional
been fully studied, such as maintenance or integration to
layer between the user and the content. The design and organizations.
especially the gamified elements should not take the focus
away from the content or working, and in many occasions
the gamification should be approached from the perspective 4.4. Identifying core literature related to the publications
of playfulness, not incorporating the game into the existing
system. A social network analysis was performed on the subset of gam-
• Normal et al. [9] discusses the impact and experiences of ification papers that were available from the Web of Science in
playing games, identifying 12 primary studies into the topic. order to find core literature that is cited from the gamification
Their classification defines different meanings of fun to en- papers. This kind of analysis can give a more accurate picture of
hance the different aspects, such as maintaining healthy diet influential papers than just counting the total number of citations.
or improving employee work satisfaction. Their study also In social network analysis, communication between individual
identifies that the adult learning is one of the areas of ‘‘fun’’, or social units are mapped into a communication matrix and
which is not very thoroughly examined in the academic then modeled as graphs. These graphs can be used to visualize
literature. communication patterns in social systems. First, the graph was
• Seaborn and Fels [30] conducted a survey of gamification presented visually by using Gephi’s ForceAtlas layout algorithm
publications, reporting the applied elements, trends and [35]. Then, the relative influence of the nodes was analyzed by
using the eigenvector centrality measure [36,37]. Compared to
theories used from 31 gamification studies. Their obser-
simpler geometrical measures like degree centrality, eigenvector
vations include that the gamification research is not very
centrality is more advanced in that it considers the influence of
strongly grounded in theory such as the existing frame-
the connected nodes, and takes the entire pattern of the graph
works, and that there is a lack of comparative and longi-
into account. Where degree centrality gives a simple count of the
tudinal case studies for gamification. Only one of the iden-
number of connections a node has, eigenvector centrality assigns
tified papers discussed crossover-designs with comparison
higher values to connections to higher-ranking nodes [38]. For
against the non-gamified system. Additionally, the defini- example, with this calculation method a node with few high-
tion of gamification seems to vary, but has two persistent ranking connections might outrank a node with a larger number
elements; the domain usage of non-entertainment, and the of low-ranking connections.
inspiration and design drawn from games, especially game Table 7 lists fifteen papers with the highest eigenvector cen-
elements. trality rank. In this case centrality is a normalized value from one
• Carter et al. [31] summarizes the research papers and game to zero, with the most central item being assigned the value of
research trends between the years 2003–2013 in the CHI one. Additionally, the full graph is available online for viewing.4
field. The paper conducts a grounded theory study and iden- The articles discovered with this method include two conference
tifies four paradigms of games and play—research in the articles, three books, and ten journal articles. Several papers by
human–computer interaction; operative research, in which authors who have written fundamental papers related to gamifi-
games are used to achieve desired results, epistemological cation are present: [2,39–43]. What is also notable is the presence
research, in which games are used to generate insight into of several books in the list. It appears that several papers have cited
other activities, ontological research, where games are stud- implementation or how-to type books on gamification.
ies as objects and practice studies, where the games are used
to practice other activities. The paper also provides different 4.5. Implications of the analysis
metrics and classifications of papers from the different HCI
venues. The original research questions for this study were ‘‘What are
• Tootell et al. [32] discuss the different implications of gam- the current trends in the gamification research?’’ and ‘‘Where is the
ification to the generation of people, who have been born effort in the gamification research focused?’’. Based on the obser-
and taught with the gamified systems. Their study dissects vations made with the data, it seems that the trends in gamifica-
different gamification mechanics and observes their effect tion research are in the proof-of-concept-systems and especially
on the motivational aspects and engagement. Their obser- in eLearning-applications. The research trends seem to gravitate
vations focus on the early education, and the effect of gam-
ification in it. 4 http://www.it.lut.fi/GRIP/datatools/gamification-gefx-js/.
42 J. Kasurinen, A. Knutas / Computer Science Review 27 (2018) 33–44

Table 7
Top 15 most central papers according to social network analysis.
Article Centrality
measure
‘‘From game design elements to gamefulness: defining gamification’’ [2] 1.0
‘‘Gamification by design: Implementing game mechanics in web and mobile apps’’ [44] 0.77
‘‘The gamification of learning and instruction: game-based methods and strategies for training and education’’ [45] 0.55
‘‘Reality is broken: Why games make us better and how they can change the world’’ [46] 0.47
‘‘Gamifying learning experiences: Practical implications and outcomes’’ [47] 0.41
‘‘A social gamification framework for a K-6 learning platform’’ [48] 0.36
‘‘Flow: The psychology of optimal performance’’ [49] 0.35
‘‘A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation’’ [39] 0.34
‘‘Defining gamification: a service marketing perspective’’ [43] 0.34
‘‘The theory of planned behavior’’ [50] 0.32
‘‘Transforming homo economicus into homo ludens: A field experiment on gamification in a utilitarian peer-to-peer trading service’’ [41] 0.31
‘‘Game design as marketing: How game mechanics create demand for virtual goods’’ [42] 0.31
‘‘How can exploratory learning with games and simulations within the curriculum be most effectively evaluated?’’ [51] 0.30
‘‘Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being’’ [40] 0.29
‘‘A systematic literature review of empirical evidence on computer games and serious games’’ [52] 0.29

towards the aforementioned eLearning, motivational aspects and of such data. The rejection rate of 22.9 percent included almost 270
supporting ecological, sustainable lifestyle, and conducting re- items from the search engine, meaning that every fifth object was
search which focuses on the certain solutions, or components of considered incomplete for classification. However, manual inspec-
gamified design. In addition, looking into the data as a whole, some tion of the no-abstract and other rejected items seems to indicate,
additional conclusions can be drawn on the different domains of that several of these items are separately indexed book chapters,
gamification research: workshop reports, clearly mislabeled research papers, book titles
and published opinion pieces from the non-peer-reviewed sources.
• In general, the gamification studies introduce new gamified Since the book chapters and book titles were discarded from the
services, or discuss the general theory, or apply the gamifi- analysis because of the unclear status of peer reviews, this alone
cation principles in the learning process. adds 98 items to the list of rejected items. This obviously does not
• The proof-of-concept studies are focused on few areas of mean that in the remaining 172 items there are no false negatives,
interest; the ten most common domains covered more than but this most likely does not majorly affect the codifications since
half of all identified POC studies. they also seem to cover different topics and based on random
• Business-focused studies and technology/hardware-focused sampling by authors, do not originate from one source such as one
studies were less frequent than anticipated by the authors search database or publishing venue.
prior to the analysis (both less than 5 percent of papers). Other concern over the accuracy is that the classification was
This conclusion is also supported by the LDA text mining of done primarily by one researcher, as pointed out by the Kitchen-
abstracts. ham et al. [13]. On this concern, the data algorithms for collecting
• Education is a dominant theme among the publications, the data and the classification scheme was discussed and reviewed
with development and crowdsourcing-themed publications
with other researchers, but the work was done by one author
on the rise.
classifying and the other author verifying. Since the applied clas-
• Most of the Serious Games topics were related to healthcare
sification scheme was rather straightforward and atomic, relying
or Games for Health domain.
heavily on the self-identification aspects of the authors of the
• Seventeen researchers, 0.6 percent of the identified re-
papers, and used of certain key terms such as crowdsourcing,
searchers, are involved with almost one tenth of the iden-
business model or MOOC, the classification should be at least
tified publications, with the overall author retention rate
acceptably accurate. This was also triangulated with the statistical
being 14 percent for the gamification research.
LDA analysis, which independently generated data and metrics
compatible with the observations made from the manual classi-
5. Discussion fication and inspection of the documents.

The objective of a systematic mapping study is to establish an 6. Conclusions

understanding of the field or topic, on which the data was collected
while avoiding bias. In this study, the validity of the dataset was In this study, the objective was to understand via a systematic
secured with several approaches: first of all, accessibility was used mapping study of 1164 research papers on gamification what the
as the first filter: if the paper is not available online, we could not current major trends in the research area are, and how the differ-
access or assess it and therefore rejected them from the dataset. ent gamification-related topics have been studied. Most common
Secondly, only peer-reviewed papers (journal articles, conference overall theme of the papers identified from text mining the ab-
papers) were accepted to the dataset and finally, the gamification stracts was education. Based on our expert classification the most
was used as the only search term since it semantically makes an common types of research in the gamification are development
ideal keyword: for the purposes of this study, the word does not of proof-of-concept prototypes, theoretical papers discussing the
have any secondary or different uses, which were out of scope components or applicability, and papers discussing the eLearning
for this study. This might also explain the age distribution of the concepts such as massive open online courses. On the trends of
papers: ‘gamification’ as a term was coined in 2008, but it became proof-of-concept prototypes, the most common domains are com-
the popular tech-word for game-like design around 2010 [2]. puter science education, ecological lifestyles and sustainability,
On the accuracy of the classification with the papers there are and motivational tools. The study also identified 400 authors, who
additional concerns. The method of manual classification can be have authored two or more studies in the gamification discipline,
considered problematic, since the classification was based on the and over 500 different publication titles and venues. The review
titles and abstracts, and the entire process relied on the availability also indicated that the gamification-related research is a new and
 J. Kasurinen, A. Knutas / Computer Science Review 27 (2018) 33–44 43

rising trend, with all identified publications being less than ten [17] B. Penzenstadler, A. Raturi, D. Richardson, C. Calero, H. Femmer, X. Franch,
years old. This study also identified thirteen similar systematic Systematic mapping study on software engineering for sustainability (SE4S),
in: Proceedings of the 18th International Conference on Evaluation and Assess-
reviews on the topics related to gamification, or some aspect of
ment in Software Engineering, ACM, 2014, p. 14.
gamification, such as empirical studies of gamification. These stud- [18] C. Wang, D.M. Blei, Collaborative Topic Modeling for Recommending Sci-
ies are discussed and listed in Section 4.4. entific Articles, ACM Press, 2011, p. 448. http://dx.doi.org/10.1145/2020408.
Based on the study it is plausible to argue that the most pressing 2020480.
[19] A. Knutas, A. Hajikhani, J. Salminen, J. Ikonen, J. Porras, Cloud-based bibliomet-
issue of the research work in gamification is to collect evidence on
ric analysis service for systematic mapping studies, in: Proceedings of the 16th
the practical applications and their impact. Applying gamification International Conference on Computer Systems and Technologies, ACM, 2015,
in the education is also an important part of the gamification pp. 184–191.
domain, games for health and serious games in general being [20] K. Hornik, B. Grün, topicmodels: An R package for fitting topic models, J. Statist.
acknowledged, but much more limited as topics of study. In the Softw. 40 (2011) 1–30.
future, the results of this study are useful in exploring new research [21] C. Sievert, K.E. Shirley, (2014) LDAvis: A method for visualizing and interpret-
ing topics, in: Proceedings of the Workshop on Interactive Language Learning,
gaps in the field of gamification, and gaining insight on what
Visualization, and Interfaces. pp. 63–70.
concepts and areas of application have already been studied. [22] A. Strauss, J. and Corbin, Basics of Qualitative Research: Grounded Theory
Procedures and Techniques, SAGE Publications, Newbury Park, CA, USA, 1990.
Acknowledgments [23] C.B. Seaman, Qualitative methods in empirical studies of software engineering,
IEEE Trans. Softw. Eng. 25 (1999) 557–572.
[24] J. Hamari, J. Koivisto, H. Sarsa, (2014) Does Gamification Work?—A Literature
This study was partially funded by the European Union Re- Review of Empirical Studies on Gamification, in: Proceedings of the 47th
gional Development Fund grant number A70554, ‘‘Kyberturval- Hawaii International Conference on System Sciences, HICSS.
lisuusosaamisen ja liiketoiminnan kehittäminen’’, administrated [25] F.L. Khaleel, N.S. Ashaari, T.S.M. Tengku Wook, A. Ismail, User-Enjoyable Learn-
by the council of Kymenlaakso. The work was also supported, in ing Environment Based on Gamification Elements, IEEE, 2015, pp. 221–226.
http://dx.doi.org/10.1109/I4CT.2015.7219570.
part, by Science Foundation Ireland grant 13/RC/2094. The second
[26] N.S. Panchariya, A.J. DeStefano, V. Nimbagal, R. Ragupathy, S. Yavuz, K.G.
author acknowledges the funding by Ulla Tuominen Foundation. Herbert, E. Hill, J.A. Fails, Current Developments in Big Data and Sustainability
Sciences in Mobile Citizen Science Applications, IEEE, 2015, pp. 202–212. http:
References //dx.doi.org/10.1109/BigDataService.2015.64.
[27] M. Sanmugam, Z. Abdullah, N.M. Zaid, Gamification: Cognitive Impact and
Creating a Meaningful Experience in Learning, IEEE, 2014, pp. 123–128. http:
[1] Swrve, The Swrve new players report 04/2014 [WWW Document], 2014, URL
//dx.doi.org/10.1109/ICEED.2014.7194700.
http://landingpage.swrve.com/rs/swrve/images/new-players-report-0414.pdf.
[2] S. Deterding, D. Dixon, R. Khaled, L. Nacke, From game design elements to [28] S. de Sousa Borges, V.H. Durelli, H.M. Reis, S. Isotani, A systematic mapping
gamefulness: Defining gamification, in: Proceedings of the 15th International on gamification applied to education, in: Proceedings of the 29th Annual ACM
Academic MindTrek Conference: Envisioning Future Media Environments, Symposium on Applied Computing, ACM, 2014, pp. 216–222.
ACM, 2011, pp. 9–15. http://dx.doi.org/10.1145/2181037.2181040. [29] K. Knaving, S. Björk, Designing for Fun and Play: Exploring Possibilities in
[3] G.K. Cheung, T. Zimmermann, N. Nagappan, The First Hour Experience: How Design for Gamification, ACM Press, 2013, pp. 131–134. http://dx.doi.org/10.
the Initial Play Can Engage (Or Lose) New Players, ACM Press, 2014, pp. 57–66. 1145/2583008.2583032.
http://dx.doi.org/10.1145/2658537.2658540. [30] K. Seaborn, D.I. Fels, Gamification in theory and action: A survey, Int. J. Human-
[4] M. Guzdial, E. Soloway, Teaching the nintendo generation to program, Comput. Stud. 74 (2015) 14–31. http://dx.doi.org/10.1016/j.ijhcs.2014.09.006.
Commun. ACM 45 (2002) 17. http://dx.doi.org/10.1145/505248.505261. [31] M. Carter, J. Downs, B. Nansen, M. Harrop, M. Gibbs, Paradigms of Games
[5] S. Göbel, S. Hardy, V. Wendel, F. Mehm, R. Steinmetz, Serious Games for Health: Research in HCI: A Review of 10 Years of Research At CHI, ACM Press, 2014,
Personalized Exergames, ACM Press, 2010, p. 1663 http://dx.doi.org/10.1145/ pp. 27–36. http://dx.doi.org/10.1145/2658537.2658708.
1873951.1874316.
[32] H. Tootell, M. Plumb, C. Hadfield, L. Dawson, (2013) Gestural Interface Technol-
[6] E. Estelles-Arolas, F. Gonzalez-Ladron-de Guevara, Towards an integrated
ogy in Early Childhood Education: A Framework for Fully Engaged Commu-
crowdsourcing definition, J. Inform. Sci. 38 (2012) 189–200. http://dx.doi.org/
nication, in: 2013 46th Hawaii International Conference on System Sciences
10.1177/0165551512437638.
[7] C. Alario-Hoyos, M. Pérez-Sanagustín, C.D. Kloos, P.J. Muñoz Merino, (2014) (HICSS). Presented at the 2013 46th Hawaii International Conference on Sys-
Recommendations for the design and deployment of MOOCs: Insights about tem Sciences, HICSS, pp. 13–20. http://dx.doi.org/10.1109/HICSS.2013.241.
the MOOC digital education of the future deployed in MiríadaX, in: Proceed- [33] B.B. Morrison, B. DiSalvo, Khan Academy Gamifies Computer Science, ACM
ings of the Second International Conference on Technological Ecosystems for Press, 2014, pp. 39–44. http://dx.doi.org/10.1145/2538862.2538946.
Enhancing Multiculturality. ACM, pp. 403–408. [34] O. Pedreira, F. García, N. Brisaboa, M. Piattini, Gamification in software en-
[8] K. Huotari, J. Hamari, (2011) Gamification from the perspective of service gineering –A systematic mapping, Inf. Softw. Technol. 57 (2015) 157–168.
marketing, in: Proc. CHI 2011 Workshop Gamification. http://dx.doi.org/10.1016/j.infsof.2014.08.007.
[9] M.J. Normal, K. MdNor, B.I. Ishak, Fun Beliefs in Digital Games from the [35] M. Bastian, S. Heymann, M. Jacomy, (2009) Gephi: An open source software
Perspective of Human Nature: A Systematic Review, IEEE, 2014, pp. 359–364. for exploring and manipulating networks, in: International AAAI Conference
http://dx.doi.org/10.1109/ISTMET.2014.6936534. on Weblogs and Social Media.
[10] B. Kitchenham, S. Charters, (2007) Guidelines for performing Systematic Liter- [36] P. Bonacich, Some unique properties of eigenvector centrality, Social Networks
ature Reviews in Software Engineering.
29 (2007) 555–564. http://dx.doi.org/10.1016/j.socnet.2007.04.002.
[11] K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson, (2008) Systematic mapping stud-
[37] P. Bonacich, Factoring and weighting approaches to status scores and clique
ies in software engineering, in: 12th International Conference on Evaluation
identification, J. Mater. Sci. 2 (1972) 113–120. http://dx.doi.org/10.1080/
and Assessment in Software Engineering, p. 1.
[12] J.C. de Almeida Biolchini, P.G. Mian, A.C.C. Natali, T.U. Conte, G.H. Travassos, 0022250X.1972.9989806.
Scientific research ontology to support systematic review in software engi- [38] M.E. Newman, The mathematics of networks, New Palgrave Encyclopedia
neering, Adv. Eng. Inform. 21 (2007) 133–151. http://dx.doi.org/10.1016/j.aei. Econom. 2 (2008) 1–12.
2006.11.006. [39] E.L. Deci, R. Koestner, R.M. Ryan, A meta-analytic review of experiments
[13] B. Kitchenham, O. Pearl Brereton, D. Budgen, M. Turner, J. Bailey, S. Linkman, examining the effects of extrinsic rewards on intrinsic motivation, Psychol.
Systematic literature reviews in software engineering –a systematic literature Bull. 125 (1999) 627.
review, Inf. Softw. Technol. 51 (2009) 7–15. http://dx.doi.org/10.1016/j.infsof. [40] R.M. Ryan, E.L. Deci, Self-determination theory and the facilitation of intrinsic
2008.09.009. motivation, social development, and well-being, Amer. Psychol. 55 (2000) 68.
[14] J. Bailey, D. Budgen, M. Turner, B. Kitchenham, P. Brereton, S.G. Linkman, [41] J. Hamari, Transforming homo economicus into homo ludens: A field exper-
Evidence relating to Object-Oriented software design: A survey, in: ESEM, iment on gamification in a utilitarian peer-to-peer trading service, Electron.
Citeseer, 2007, pp. 482–484. Commerce Res. Appl. 12 (2013) 236–245.
[15] D.M. Blei, A.Y. Ng, M.I. Jordan, Latent dirichlet allocation, J. Mach. Learning Res.
[42] J. Hamari, V. Lehdonvirta, Game design as marketing: how game mechanics
3 (2003) 993–1022.
create demand for virtual goods, Int. J. Bus. Sci. Appl. Manage. 5 (2010) 14–29.
[16] D.M. Blei, Probabilistic topic models, Commun. ACM 55 (2012) 77. http://dx.
doi.org/10.1145/2133806.2133826.
44 J. Kasurinen, A. Knutas / Computer Science Review 27 (2018) 33–44

[43] K. Huotari, J. Hamari, Defining gamification: a service marketing perspective, [48] J. Simões, R.D. Redondo, A.F. Vilas, A social gamification framework for a K-6
in: Proceeding of the 16th International Academic MindTrek Conference, ACM, learning platform, Comput. Hum. Behav. (2012).
2012, pp. 17–22. [49] M. Csikszentmihalyi, Flow: The Psychology of Optimal Performance, Cambridge
[44] G. Zichermann, C. Cunningham, Gamification By Design: Implementing Game University Press, NY, 1990.
Mechanics in Web and Mobile Apps, O’Reilly Media, Inc., 2011. [50] I. Ajzen, The theory of planned behavior, Organiz. Behav. Human Decision
[45] K.M. Kapp, The Gamification of Learning and Instruction: Game-Based Meth- Process. 50 (1991) 179–211.
ods and Strategies for Training and Education, John Wiley & Sons, 2012. [51] S.De. Freitas, M. Oliver, How can exploratory learning with games and simula-
[46] J. McGonigal, Reality Is Broken: Why Games Make Us Better and how They Can tions within the curriculum be most effectively evaluated?, Comput. Educ. 46
Change the World, Penguin, 2011. (2006) 249–264.
[47] A. Domínguez, J. Saenz-de Navarrete, L. de Marcos, L. Fernández-Sanz, C. Pagés, [52] T.M. Connolly, E.A. Boyle, E. MacArthur, T. Hainey, J.M. Boyle, A systematic
J.-J. Martínez-Herráiz, Gamifying learning experiences: practical implications literature review of empirical evidence on computer games and serious games,
and outcomes, Comput. Educ. 63 (2013) 380–392. http://dx.doi.org/10.1016/j. Comput. Educ. 59 (2012) 661–686.
compedu.2012.12.020.