From Chess and Atari to StarCraft and Beyond:

How Game AI is Driving the World of AI

Sebastian Risi and Mike Preuss
arXiv:2002.10433v1 [cs.AI] 24 Feb 2020

Abstract This paper reviews the field of Game AI, strengthening it (e.g. [45]). The main arguments have
which not only deals with creating agents that can play been these:
a certain game, but also with areas as diverse as cre-
ating game content automatically, game analytics, or – By tackling game problems as comparably cheap,
player modelling. While Game AI was for a long time simplified representatives of real world tasks, we can
not very well recognized by the larger scientific com- improve AI algorithms much easier than by model-
munity, it has established itself as a research area for ing reality ourselves.
developing and testing the most advanced forms of AI – Games resemble formalized (hugely simplified) mod-
algorithms and articles covering advances in mastering els of reality and by solving problems on these we
video games such as StarCraft 2 and Quake III appear learn how to solve problems in reality.
in the most prestigious journals. Because of the growth
of the field, a single review cannot cover it completely.
Therefore, we put a focus on important recent develop- Both arguments have at first nothing to do with
ments, including that advances in Game AI are start- games themselves but see them as a modeling / bench-
ing to be extended to areas outside of games, such as marking tools. In our view, they are more valid than
robotics or the synthesis of chemicals. In this article, ever. However, as in many other digital systems, there
we review the algorithms and methods that have paved has also been and still is a strong intrinsic need for im-
the way for these breakthroughs, report on the other provement because the performance of Game AI meth-
important areas of Game AI research, and also point ods was in many cases too weak to be of practical use.
out exciting directions for the future of Game AI. This could be both in terms of playing strength, or sim-
ply because they failed to produce believable behav-
ior [44]. The latter would be necessary to hold up the
suspension of disbelief, or, in other words, the illusion
1 Introduction
to willingly be immersed in a game world.
For a long time, games research and especially research But what exactly is Game AI? Opinions on that
on Game AI was in a niche, largely unrecognized by have certainly changed in the last 10 to 15 years. For a
the scientific community and the general public. Propo- long time, academic research and game industry were
nents of Game AI research wrote advertisement articles largely unconnected, such that neither researchers tack-
to justify the research field and substantiate the call for led AI-related problems game makers had nor the game
makers discussed with researchers what these problems
S. Risi actually were. Then, in research some voices emerged,
IT University of Copenhagen and modl.ai calling for more attention for computer Game AI (partly
Copenhagen, Denmark
E-mail: sebr@itu.dk
as opposed to board game AI), including Nareyek [52,
53], Mateas [48], Buro [11], and also Yannakakis [88].
M. Preuss
LIACS, Universiteit Leiden Proponents of a change included Alex Champan-
Leiden, Netherlands dard in his computational intelligence and games con-
E-mail: m.preuss@liacs.leidenuniv.nl ference (CIG) 2010 tutorial [94] and Youichiro Miyake
2 Sebastian Risi and Mike Preuss

in his GameOn Asia 2012 keynote1 . At that time, a successfully tackling difficult problems of human deci-
large part of Game AI research was devoted to board sion making, such as Go, Dota2, and StarCraft.
games as Chess and Go, with the aim to create the best It is, however, a fairly open question how we can uti-
possible AI players, or to game theoretic systems with lize these successes for solving other problems in Game
the aim to better understand these. AI and beyond. As it appears to be possible but utterly
Champandard and Miyake both argued that research difficult to transfer whole algorithmic solutions, e.g., for
shall try to tackle problems that are actually relevant a complex game as StarCraft, to a completely different
also for the games industry. This led to a shift in the fo- domain, we may rather see innovative recombinations of
cus of Game AI research that was further intensified by algorithms from the recently enriched portfolio in order
a series of Dagstuhl meetings on Game AI that started to craft solutions for new problems.
in 20122 . The panoramic view [91] explicitly lists 10 In the next sections, we start with enlisting some
subfields and relates them to each other, most of which important terms that will be repeatedly used (Sect.2)
were not widely known as Game AI at that time, and before tackling state / action based learning in Sect. 3.
even less so in the game industry. Most prominently, We then report on pixel-based learning in Sect. 4. At
areas with a focus on using AI for design and pro- this point, PCG comes in as a flexible testbed generator
duction of games emerged, such as procedural content (Sect. 5). However, it is also a viable aim on its own
generation (PCG), computational narrative (nowadays to be able to generate content. Very recently, different
also known as interactive storytelling), and AI-assisted sources of game information, such as pixel and state
game design. Next to these, we find search and plan- information, are given as input to these game-playing
ning, non-player character (NPC) behavior learning, agents, providing better methods for rather complex
AI in commercial games, general Game AI, believable games (Sect. 6). While many approaches are tuned to
agents, and games as AI benchmarks. A third impor- one game, others explicitly strive for more generality
tant branch that came up at that time (and resembles (Sect. 7). Next to game playing and generating content,
the 10th subfield) considers modeling players and un- we also shortly discuss AI in other roles (Sect. 8). We
derstanding what happens in a running game (game conclude the article with a short overview of the most
analysis). important publication venues and test environments in
The 2018 book on AI and Games [92] shows the pre- Sect. 9 and some reasoning about the expected future
game (design / production) during game (game playing) developments in Game AI in Sect. 10.
and after-game (player modeling / game analysis)3 uses
of AI together with the most important algorithms be-
hind it and gives a good overview of the whole field.
2 Algorithmic approaches and game genres
Due to space restrictions, we cannot go into details on
developments in each sub-area of Game AI in this work We provide an overview of the predominant paradigms-
but rather provide an overview over the ones considered / algorithm types and game genres, focusing mostly on
most important, including highlighting some amazing game playing and more recent literature. These algo-
recent achievements that for a long time have not been rithms are used in many other contexts of AI and ap-
deemed possible. These are mainly in the game playing plication areas of course, but some of their most popular
field but also draw from generative approaches such as successes have been achieved in the Game AI field.
PCG in order to make them more robust.
Most of the popular known big recent successes are
Reinforcement Learning (RL). In reinforcement
connected to big AI-heavy IT companies entering the
learning an agent learns to perform a task through
field such as DeepMind (Google), Facebook AI and Ope-
interactions with its environment and through re-
nAI. Equipped with rich computational and human re-
wards. This is in contrast to supervised learning, in
sources, these new players have especially profited from
which the agent is directly told the correct action in
Deep (Reinforcement) Learning to tackle problems that
different states. One of the main challenges in RL
were previously seen as important milestones for AI,
is to find a balance between exploitation (i.e. seek-
1
http://igda.sakura.ne.jp/sblo_files/ai-igdajp/ ing out states that are known to give a high reward)
academic/YMiyake_GameOnAsia_2012_2_25.pdf vs. exploration (i.e. trying out something new that
2
see http://www.dagstuhl.de/12191, http://www. might lead to higher rewards in the long run).
dagstuhl.de/15051, http://www.dagstuhl.de/17471,
http://www.dagstuhl.de/19511
3
We are aware that this division is a bit simplistic, of
course players can be also modeled online or for supporting
the design phase. Please consider this a rough guideline only.
From Chess and Atari to StarCraft and Beyond: How Game AI is Driving the World of AI 3

Deep Learning (DL). Deep learning is a broad Evolutionary Algorithms (EA). Also known as
term and comes in a variety of different shapes and bio-inspired optimization algorithms, Evolutionary
sizes. The main distinguishing feature of deep learn- Algorithms take inspiration from natural evolution
ing is the idea to learn progressively higher-level for solving black-box optimization problems. They
features through multiple layers of non-linear pro- are thus applied when classical optimization meth-
cessing. The most prevalent deep learning methods ods fail or cannot be employed because no gradient
are based on deep neural networks, which are artifi- or not even numeric objective value information (but
cial neural networks with multiple different layers (in ranking of solutions) is available. A key idea of EAs
new neural network models these can be more than is parallel search by means of populations of can-
100 layers). Recent advances in computing power, didate solutions, which are concurrently improved,
such as more and more efficient GPUs (which were making it a global optimization method. EAs are es-
first developed for fast rendering of 3D games), more pecially well suited for multi-objective optimization,
data, and various training improvements have al- and the well-known GA, NSGA-II, CMA-ES algo-
lowed deep learning methods to surpass the previ- rithms are all EAs, see also the introduction/survey
ous state-of-the-art in many domains such as im- book [18].
age recognition, speech recognition or drug discov-
ery. LeCun et al. [38] provide a good review paper Which are the most important games to serve as
on this fast-growing research area. testbeds in Game AI? The research-oriented frameworks
general game playing (GGP), general video Game AI
(GVGAI) and the Atari learning environment (ALE)
Deep Reinforcement Learning. Deep Reinforce-
play an important role but are somewhat far from mod-
ment Learning combines reinforcement learning with
ern video games. This also holds true for the traditional
deep neural networks to create efficient algorithms
AI challenge board games Chess and Go and card games
that can learn directly from high-dimensional sen-
as Poker or Hanabi. In video games, the predominant
sory streams. Deep RL has been the workhorse be-
genres are real-time strategy (RTS) games such as Star-
hind many of the recent advances in Game AI, such
Craft, Multiplayer online battle arena (MOBA) games
as beating professional players in StarCraft and Dota2.
such as Dota2, and first person shooter (FPS) games
[3] provides a good overview over deep RL.
such as Doom. Sports games currently get more im-
portant [43] as they often represent a competitive team
Monte Carlo Tree Search (MCTS). Monte Carlo situation that is seen as similar to many real-world hu-
Tree Search is a fairly recent [14] randomized tree man/AI collaborative problems. In a similar way, co-
search algorithm. States of the mapped system are operative (capture-the-flag) variants of FPS games [31]
nodes in the tree, and possible actions are edges that are used. Figures 1 and 2 provide an overview of the
lead to new states. In contrast to older methods such different properties of the games used as AI testbeds.
as alpha-beta pruning, it does not attempt to look
at the full tree but uses controlled exploration and
3 Learning to play from states and actions
exploitation of already obtained knowledge (success-
ful branches are preferred) and often fully random-
Games have for a long time served as invaluable testbeds
ized playouts, meaning that a game is played until it
for research in artificial intelligence (AI). In the past,
ends by applying randomized actions. If that takes
particularly board games such as Checkers and Chess
too long, state value heuristics can be used alterna-
have been tackled, later on turning to Go when Check-
tively. Loss/win information is propagated upwards
ers had been solved [70] and with DeepBlue [12] an
up to the tree root such that estimations of the win
artificial intelligence had defeated the world champion
ratio at every node get available for directing the
in Chess consistently. All these games and many more,
search. MCTS can thus be applied to much larger
up to Go, have one thing in common: they can be ex-
trees, but provides no guarantees concerning obtain-
pressed well by states and actions, where the number
ing optimal solutions. [10] is a popular introductory
of actions is usually a not-too-large number of often
survey.
around 100 or less reasonable moves from any possible
position. For quite some time, board games have been
tackled with alpha-beta pruning (Turing Award Win-
ners Newell and Simon explain in [54] how this idea
came up several times almost at once) and very so-
4 Sebastian Risi and Mike Preuss

perfect information partial information 1 player 2 player multiplayer

deterministic

cooperative
Chess, Checkers
Go, Othello Battleship Hanabi
Atari, GGP

non-deterministic

teams
StarCraft, Poker Bridge, MOBA
Backgammon FPS, sports games
Hanabi, Bridge,
GVGAI, sports games
MOBA, RTS, FPS

Backgammon

competitive
sports games
Fig. 1 Available information and determinism as separating GGP, GVGAI, Poker
StarCraft, Battleship
Atari, FPS RTS, FPS
properties for different games treated in Game AI. Chess, Checkers
Go, Othello, RTS

Fig. 2 Player numbers and style from cooperative to com-

phisticated and extremely specialized heuristics before petitive for different games or groups of games treated in
Coulom invented Monte Carlo Tree Search (MCTS) [14] Game AI. Note that for several games, multiple variants are
in 2006. MCTS gives up optimality (full exploration) possible, but we use only the most predominant ones.
in exchange for speed and is therefore now dominating
AI solutions for larger board games such as Go with Seemingly not, as [85] suggests (see Sect. 6). However,
about 10170 possible states (board positions). MCTS- this may be a question of the number of actions and
based Go algorithms had greatly improved the state-of- states in a game and remains to be seen. Nevertheless,
the-art up to the level of professional players by incor- board games and card games are obviously good can-
porating sophisticated heuristics as Rapid Action Value didates for such AI approaches.
Estimation (RAVE) [21]. In the following, MCTS based
approaches were shown to cope well also with real-time
conditions as in the PacMan game [59] and also hidden 4 Learning to play from pixels
information games [62].
However, only the combination of MCTS with DL For a long time, learning directly from high-dimensional
led to a world-class professional human-level Go AI input data such as the pixels of a video game was an
player named AlphaGo [76]. At this stage, human ex- unsolved challenge. Earlier neural network-based ap-
perience (recorded grandmaster games) had been used proaches for playing games such as Pac-Man relied on
for ”seeding” the learning process that was then accel- careful engineered features such as the distance to the
erated by self-play. By playing against itself, the Al- nearest ghost or pill, which are given as input to the
phaGo algorithm was able to steadily improve its value neural network [67].
(how good is the current state?) and policy (what is While some earlier game-playing approaches, espe-
the best action to play?) artificial neural networks. The cially from the evolutionary computation community,
next step, AlphaGo Zero [77] removed all human data, showed initial success in learning directly from pixels
relying on self-play alone, and learned to play Go better [20, 29, 57, 82], it was not until DeepMind’s seminal
than the original AlphaGo approach but from scratch. paper on learning to play Atari video games from pix-
This approach has been further developed to AlphaZero els [50, 51] that these approaches started to compete
[75] and shown to be able to learn to play different and at times outperform human players. Serving as a
games, next to Go also Chess and Shogi (Japanese Chess). common benchmark, many novel AI algorithms have
In-depth coverage of most of these developments is also been developed and compared on Atari video games
provided in [61]4 . first [33] before being applied to other domains such
From the last paragraphs, it may appear as if learn- as robotics [1]. A computationally cheap and thus in-
ing via self-play is limited to two-player perfect infor- teresting end-to-end pixel-based learning environment
mation games only. However, also multi-player partial is VizDoom [36], a competition setting that relies on a
information games such as Poker [9] and even cooper- rather old game that is run in very small screen resolu-
ative multi-player games such as Hanabi [39] have re- tions. Low resolution pixel inputs are also employed in
cently been tackled and AI players now exist that can the obstacle tower challenge (OTC) [32].
play these games at the level of the best human players. DeepMind’s paper ushered in the area of Deep Rein-
Thus, is self-play the ultimate AI solution for all games? forcement Learning, combining reinforcement learning
with a rich neural network-based representation (see in-
4
https://learningtoplay.net/ fobox for more details). Deep RL has since established
From Chess and Atari to StarCraft and Beyond: How Game AI is Driving the World of AI 5

Fig. 3 A visualisation of the AlphaStar agent playing against the human player MaNa, from [84]. Shown is the raw observation
that the neural network gets as input (bottom left), together with the internal neural network activations. On the lower right
side are shown actions considered by the agent together with a prediction of the outcome of the game.

itself as the prevailing paradigm is to learn directly Instead of first training a policy on random roll-
from high-dimensional input such as images, videos, or outs, follow-up work showed that end-to-end learning
sounds without the need for human-design features or through reinforcement learning [28] and evolution [65,
preprocessing. More recently, approaches based on evo- 66] is also possible. We will discuss MuZero as another
lutionary algorithms have shown to also be competitive example of planning in latent space in Section 6.
with approaches based on gradient descent-based meth-
ods [80].
5 Procedural content generation
However, some of the Atari games, namely Mon-
tezuma’s Revenge, Pitfall, and others proved to be too In addition to playing games, another active area of AI
difficult to solve with standard deep RL approaches research is procedural content generation (PCG) [68,
[50] because of sparse and/or late rewards. These hard- 74]. PCG refers to the algorithmic creation of game
exploration games can be handled successfully by evo- content such as levels, textures, quests, characters, or
lutionary algorithms that explicitly favor exploration even the rules of the game itself.
such as Go-Explore [17]. One of the appeals of employing PCG in games is
A recent trend in deep RL is to allow agents to learn that it can increase their replayability by offering the
a general model of how their environment behaves and player a new experience every time they play. For ex-
use that model to explicitly plan ahead. For games, one ample, games such as No Man’s Sky (Hello Games,
of the first approaches was the World Model introduced 2016) or Spelunky (Mossmouth, LLC, 2013) famously
by [26], in which an agent learns to solve a challenging featured PCG as part of their core gameplay, allowing
2D car racing game and a 3D VizDoom environment players to explore an almost unlimited variety of plan-
from pixels alone. In this approach, the agent first learns ets or caves. One of the most important early benefits of
by collecting observations from the environment, and PCG methods was that it allowed the creation of larger
then training a forward model that takes the current game worlds than what would normally fit on a com-
state of the environment and action and tries to predict puter’s hard disk at the time. One of the first games us-
the next state. Interestingly, this approach also allowed ing PCG-based methods was Elite (Brabensoft, 1984),
an agent to get better by training inside a hallucinated a space trading video game featuring thousands of plan-
environment created through a trained world model. ets. The whole starsystem with each visited planet and
6 Sebastian Risi and Mike Preuss

space stations could be recreated from a given random conceptual simplicity, but on the other hand, it is intu-
seed. itively clear that adding more information – if available
While the origin of PCG is rooted in creating a – may be of advantage. More recently, these two ways
more engaging experience for players [93], more recently of obtaining game information were joined in different
PCG-based approaches have also found important other ways.
use cases. With the realisation that methods such as The hide-and-seek approach [4] depends on visual
deep reinforcement learning can surpass humans in many and state information of the agents but also heavily
games, also came the realisation that these methods on the use of co-evolutionary effects in a multi-agent
overfit to the exact environment they are trained on environment that very much reminds of EA techniques.
[35, 96]. For example, an agent trained to reach the In AlphaStar (Fig. 3) that was designed to play
level of a human expert in a game such as Breakout, StarCraft at human professional level, both state in-
will fail completely when tested on a Breakout version formation (location and status of units and buildings)
where the game pedal has a slightly different size or is as well as pixel information (minimap) is fed into the
at a slightly different position. Recent research showed algorithm. Interestingly, self-play is used heavily, but is
that by training agents on many procedurally generated not sufficient to generate human professional competi-
levels allows them to become significantly more general tive players because the strategy space is huge and hu-
[35]. In an impressive extension of this idea, DeepMind man opponents may come up with very different ways
trained agents on a large number of randomly created to play the game that must all be handled. Therefore,
levels to reach human-level performance in the Quake as in AlphaGo, human game data is used to seed the
III Capture the Flag game [31]. This trend to make AI algorithm. Furthermore, also co-evolutionary effects in
approaches more general by training them on endless a 3 tier league of different types of agents are driving
variations of environments was continued in the hide- the learning process. It shall be noted that the success
and-seek work by OpenAI [4] and also in the obstacle of AlphaStar was hard to imagine only some years ago
tower challenge (OTC) [32] and will certainly also be because RTS games were considered the hardest possi-
employed in many future approaches. ble testbeds for AI algorithms in games [55]. These suc-
Meanwhile, PCG has been applied to many differ- cesses are, however, not without controversy and people
ent types of game components or facets (e.g. visuals, argue if the comparisons of AIs playing against humans
sound), but most often to only one of these at once. are fair [13, 34].
One of the open research questions in this context is MuZero [71] is able to learn playing Atari games
how generators for different facets can be combined [41]. (pixel input) as well as Chess and Go (state input) by
Similar to some of the other techniques described generating virtual states according to reward/position
in this article, PCG has also more recently found to be value similarity. These are managed in a tree-like fash-
applicable to areas outside of games [68]. For example, ion as in MCTS but costly rollouts are avoided. The
training a humanoid robot hand to manipulate a Ru- elegance of this approach lies in the ability to use dif-
bik’s cube in a simulator on many variants of the same ferent types of input and the construction of an internal
problem (e.g. varying parameters such as the size, mass, representation that is oriented only at values and not
and texture of the cube) has allowed a policy trained at exact game states.
in a simulator to sometimes work on a physical robot
hand in the real world. For a review of how PCG has
increased generality in machine learning we refer the
7 Towards more general AI
interested reader to this survery [68] and for a more in-
depth review of PCG in general to the book by Shaker
While AI algorithms have become exceedingly good at
et al. [74].
playing specific games [33], it is still an unsolved chal-
lenge how to make an AI algorithm that can learn to
6 Merging state and pixel information quickly play any game it is given, or how to trans-
fer skills learned in one game to another. This chal-
Whereas the AI in AlphaGo and its predecessors for lenge, also known as General Video Game Playing [22],
playing board games dealt with board positions and has resulted in the development of the General Video
possible moves, deep RL and recent evolutionary ap- Game AI framework (GVGAI), a flexible framework
proaches for optimising deep neural networks (a re- designed to facilitate the development of general AI
search field now referred to as deep neuroevolution [79]), through video game playing [60].
learn to play Atari games directly from pixel informa- With increasingly complicated worlds and graphics,
tion. On the one hand, these approaches have some video games might be the ideal environment to learn
From Chess and Atari to StarCraft and Beyond: How Game AI is Driving the World of AI 7

more general intelligence. Another benefit of games is called Experience-Driven Procedural Content Genera-
that they often share similar controllers and goals. To tion [93], allows these algorithms to automatically gen-
spur developments in this area, the GVGAI framework erate unique content that induces a desired experience
now also includes a Learning Track, in which the goal of for a player. For example, [58] trained a model on play-
the agent is to learn a new game quickly without being ers of Super Mario, which could then be used to auto-
trained on it beforehand. The hope is that methods that matically generate new Mario levels that maximise the
can quickly learn any game they are given, will also modelled fun value for a particular player. Exciting re-
ultimately be able to quickly learn other tasks such a cent work can even predict a player’s affect in certain
robot manipulation in the real world. situation from pixels alone [47].
Whereas most successful approaches for GVGAI games There is also a large body of research on human-like
employ MCTS, it shall be noted that there are also non-player characters (NPC) [30], and some years ago,
other competitive approaches as the rolling horizon evo- this research area was at the core of the field, but with
lutionary algorithm (RHEA) [42] that evolve partial ac- the upcoming interest in human/AI collaboration it is
tion sequences as a whole through an evolutionary op- likely to thrive again in the next years.
timization process. Furthermore, DL variants start to Other roles for Game AI include playtesting and
get used here as well [83]. balancing which both belong to game production and
mostly happen before games are published. Testing for
bugs or exploits in a game is an interesting application
8 AI for player modelling and other roles
area of huge economic potential and some encourag-
ing results exist [15]. With the rise of machine learn-
In this section, we briefly mention a few other use cases
ing methods that can play games at a human or be-
for current AI methods. In addition to learning to play
yond human level and methods that can solve hard-
or generating games and game content, another impor-
exploration games such as Montezuma’s Revenge [17],
tant aspect of Game AI – and potentially currently the
this area should see a large increase of interest from
main use case in the game industry – is game analytics.
the game industry in the coming years. Mixed-initiative
Game analytics has changed the game landscape dra-
tools that allow humans to create game content to-
matically over the last ten years. The main idea in game
gether with a computational creator often include an
analytics is to collect data about the players while they
element of automated balancing, such as balancing the
play the game and then update the game on the fly. For
resources on a map in a strategy game [40]. Game bal-
example, the difficulty of levels can be adjusted or the
ancing is a wide and currently under-researched area
user interface can be streamlined. At what point players
that may be understood as a multi-instance parame-
stopped playing the game can be an important indica-
ter tuning problem. One of the difficulties here is that
tion of what to change to reduce the game’s churn5 rate
many computer games do not allow headless acceler-
[27, 37, 69]. We refer the interested reader to the book
ated games and APIs for controling these. Some auto-
on game analytics by El-Nasr et al. [19].
mated approaches exist for single games [63] but they
Another important application area of Game AI is
usually cannot cope with the full game and approaches
player modelling. As the name suggests, player mod-
for more generally solving this problem are not well es-
elling aims to model the experience or behavior of the
tablished yet [86]. Dynamic re-balancing during game
player [5, 95]. One of the main motivations for learn-
runtime is usually called dynamic difficulty adaptation
ing to model players is that a good player model can
(DDA) [78].
allow the game to be tailored even more to the individ-
ual player. A variety of different approaches to model
players exist, such as supervised learning (e.g. train-
ing a neural network in a supervised way on recorded 9 Journals, conferences, and competitions
plays of human players to behave the same way), to
unsupervised approaches such as clustering that aim The research area of Game AI is centered in computer
to group similar players together [16]. Based on which science, but influenced by other disciplines as i.e. psy-
cluster a new player belongs to, different content or chology, especially when it comes to handling humans
other game adaptations can be performed. Combin- and their emotions [89, 90]. Furthermore, (computa-
ing PCG (Sect. 5) with player modelling, an approach tional) art and creativity (for PCG), game studies (for-
mal models of play) and game design are important
5
In the game context, churn means that a player who has
neighboring disciplines.
played a game for some time completely stops playing it. This
is usually very hard to predict but essential to know especially In computer science, Game AI is not only limited
for online game companies. to machine learning and traditional branches of AI but
8 Sebastian Risi and Mike Preuss

Fig. 4 Chemical retrosynthesis on basis of the AlphaGo approach; figure from [73]. The upper subfigure shows the usual
MCTS steps, and the lower subfigure links these steps to the chemical problem. Actions are now chemical reactions, states
are the derived chemical compounds. Instead of preferred moves in a game, the employed neural networks learn reaction
preferences. In contrast to AlphaGo, possible moves are not simply provided but have to be learned from data, an approach
termed ”world program”[72].

also has links to information systems, optimization, com- 10 The future of Game AI
puter vision, robotics, simulation, etc. Some of the core
conferences for Game AI are: More advanced AI techniques are slowly finding their
– Foundations of Digital Games (FDG) way into the game industry and this will likely increase
– IEEE Conference on Games (CoG), until 2018 the significantly over the coming years. Additionally, com-
Conference on Computational Intelligence and Games panies are more and more collaborating with research
(CIG) institutions, to bring the latest innovations out to the
– Artificial Intelligence for Interactive Digital Enter- industry. For example, Massive Entertainment and the
tainment (AIIDE) University of Malta collaborated to predict the moti-
vations of players in the popular game Tom Clancys
Also, many computer science conferences have tracks The Division [49]. Other companies, such as King, are
or co-located smaller conferences on Game AI, as e.g. investing heavily in deep learning methods to automat-
GECCO and IJCAI. The more important journals in ically learn models of players that can then be used for
the field are the IEEE Transactions on Games ToG (for- playtesting new levels quickly [25].
merly TCIAIG) and the IEEE Transactions on Affec- Procedural content generation is already employed
tive Computing. The most active institutes in the area for many mainstream games such as Spelunky (Moss-
can be taken from a list (incomplete, focused only on mouth, LLC, 2013) and No Man’s Sky (Hello Games,
the most relevant venues) compiled by Mark Nelson.6 2016) and we will likely see completely new types of
A large part of the progress of the last years is games in the future that would be impossible to re-
due to the free availability of competition environments alise without sophisticated AI techniques. The recent
as: StarCraft, GVGAI, Angry Birds, Hearthstone, Han- AI Dungeon 2 game (www.aidungeon.io) points to what
abi, MicroRTS, Fighting Game, Geometry Friends and type of direction these games might take. In this text
more, and also the more general frameworks as: ALE, adventure game players can interact with Open AI’s
GGP, OpenSpiel, OpenAIGym, SC2LE, MuJoCo, Deep- GPT-2 language model, which was trained on 40 giga-
RTS. bytes from text scraped from the internet. The game
6
http://www.kmjn.org/game-rankings responds to almost anything the player types in a sen-
REFERENCES 9

sible way, although the generated stories also often lose crisis7 . Some of that can be cured by better experimen-
coherence after a while. This observation points to an tal methodology and statistics as also worked well in
important challenge: For more advanced AI techniques Evolutioanry Computation some time ago [7]. First at-
to be more broadly employable in the game industry, tempts in Game AI also try to approach this problem
approaches are needed that are more controllable and by defining guidelines for experimentation, e.g. for the
potentially interpretable by designers [97]. ALE [46], but replicating experiments that take weeks
We predict that in the near future, generative mod- is an issue that will probably not easily be solved.
elling techniques from machine learning, such as Gen- It is definitively desired to apply the algorithms that
erative and Adversarial Networks (GANs) [24], will al- successfully deal with complex games also to other ap-
low users to personalise their avatars to an unprece- plication areas. Unfortunately, this is usually not triv-
dented level or allow the creation of an unlimited vari- ial, but some promising examples already exist. The Al-
ety of realistic textures and assets in games. This idea phaGo approach that is based on searching by means of
of Procedural Content Generation via Machine Learn- MCTS in a neural network representation of the treated
ing (PCGML) [81], is a new emerging research area that problem has been transfered to the chemical retrosyn-
has already led to promising results in generating levels thesis problem [73] that consists of finding a synthesis
for games such as Doom [23] or Super Mario [87]. path for a specific chemical component as depicted in
From the current perspective, we would expect that Fig. 4. As for the synthesis problem, in contrast to play-
future research (next to playing better on more games) ing Go, the set of feasible moves (possible reactions) is
in Game AI will focus on these areas: not given but has to be learned from data, the approach
bears some similarity to MuZero [71]. The idea to learn
– AI/human collaboration and AI/AI agent collabo- a forward model from data has been termed world pro-
ration is getting more important, this may be sub- gram [72].
sumed under the term team AI. Recent attempts in Similarly, the same distributed RL system that Ope-
this direction include e.g.: Open AI five [64], Han- nAI used to train a team of five agents for Dota 2 [8],
abi [6], capture the flag [31] was used to train a robot hand to perform dexterous
– More natural language processing enables better in- in-hand manipulation [2].
terfaces and at some point free-form direct com- We believe Game AI research will continue to drive
munication with game characters. Already existing innovations in the world of AI and hope this review ar-
commercial voice-driven assistance systems as the ticle will serve as a useful guide for researchers entering
Google Assistant or Alexa show that this is possi- this exciting research field.
ble.
– The previous points and the progress in player mod-
eling and game analysis will lead to more human-like Acknowledgements
behaving AI, this will in turn enable better playtest-
ing that can be partly automated. We would like to thank Mads Lassen, Rasmus Berg
– PCG will be applied more in the game industry and Palm, Niels Justesen, Georgios Yannakakis, Marwin Segler,
other applications. For example, it is used heavily in and Christian Igel for comments on earlier drafts of this
Microsoft’s new flight simulator version that is now manuscript.
(January 2020) in alpha test mode. This will also
trigger more research in this area.
References
Nevertheless, as in other areas of artificial intelli-
gence, Game AI will have to cope with some issues that [1] I. Akkaya et al. “Solving Rubik’s Cube with a Robot
Hand”. In: arXiv:1910.07113 (2019).
mostly stem from two newer developments: theory-light [2] O. M. Andrychowicz et al. “Learning dexterous in-
but very successful deep learning methods, and highly hand manipulation”. In: The International Journal of
parallel computation. The first entails that we have Robotics Research 39.1 (2020), pp. 3–20.
very little control over the performance of deep learn- [3] K. Arulkumaran et al. “A brief survey of deep rein-
forcement learning”. In: arXiv:1708.05866 (2017).
ing methods, it is hard to predict what works well with [4] B. Baker et al. Emergent Tool Use From Multi-Agent
which parameters, and the second one means that many Autocurricula. 2019. arXiv: 1909.07528.
experiments can hardly ever be replicated due to hard- [5] S. C. Bakkes, P. H. Spronck, and G. van Lankveld.
ware limitations. E.g., Open AI Five has been trained “Player behavioural modelling for video games”. In:
Entertainment Computing 3.3 (2012), pp. 71–79.
on 256 GPUs and 128,000 CPUs [56] for a long time.
More generally, large parts of the deep learning driven 7
https://www.wired.com/story/
AI are currently presumed to run into a reproducibility artificial-intelligence-confronts-reproducibility-crisis/
10 REFERENCES

[6] N. Bard et al. “The Hanabi Challenge: A New Frontier tational Intelligence and Games (CIG). IEEE. 2018,
for AI Research”. In: CoRR abs/1902.00506 (2019). pp. 1–8.
arXiv: 1902.00506. [26] D. Ha and J. Schmidhuber. “World models”. In: arXiv:
[7] T. Bartz-Beielstein et al. Experimental methods for the 1803.10122 (2018).
analysis of optimization algorithms. Springer, 2010. [27] F. Hadiji et al. “Predicting player churn in the wild”.
[8] C. Berner et al. “Dota 2 with Large Scale Deep Rein- In: 2014 IEEE Conference on Computational Intelli-
forcement Learning”. In: arXiv:1912.06680 (2019). gence and Games. IEEE. 2014, pp. 1–8.
[9] N. Brown and T. Sandholm. “Superhuman AI for mul- [28] D. Hafner et al. “Learning latent dynamics for planning
tiplayer poker”. In: Science 365.6456 (2019), pp. 885– from pixels”. In: arXiv:1811.04551 (2018).
890. [29] M. Hausknecht et al. “A neuroevolution approach to
[10] C. B. Browne et al. “A Survey of Monte Carlo Tree general atari game playing”. In: IEEE Transactions
Search Methods”. In: IEEE Transactions on Computa- on Computational Intelligence and AI in Games 6.4
tional Intelligence and AI in Games 4.1 (2012), pp. 1– (2014), pp. 355–366.
43. [30] P. Hingston. Believable Bots: Can Computers Play Like
[11] M. Buro. “Real-Time Strategy Games: A New AI Re- People? Springer, 2012.
search Challenge”. In: IJCAI-03, Proceedings of the [31] M. Jaderberg et al. “Human-level performance in 3D
Eighteenth International Joint Conference on Artifi- multiplayer games with population-based reinforcement
cial Intelligence, Acapulco, Mexico, August 9-15, 2003. learning”. In: Science 364.6443 (2019), pp. 859–865.
Ed. by G. Gottlob and T. Walsh. Morgan Kaufmann, [32] A. Juliani et al. “Obstacle Tower: A Generalization
2003, pp. 1534–1535. Challenge in Vision, Control, and Planning”. In: Pro-
[12] M. Campbell, A. J. Hoane Jr, and F.-h. Hsu. “Deep ceedings of the Twenty-Eighth International Joint Con-
blue”. In: Artificial intelligence 134.1-2 (2002), pp. 57– ference on Artificial Intelligence, IJCAI 2019, Macao,
83. China, August 10-16, 2019. Ed. by S. Kraus. ijcai.org,
[13] R. Canaan et al. “Leveling the Playing Field-Fairness 2019, pp. 2684–2691.
in AI Versus Human Game Benchmarks”. In: arXiv:1903.- [33] N. Justesen et al. “Deep Learning for Video Game
07008 (2019). Playing”. In: IEEE Transactions on Games (2019),
[14] R. Coulom. “Efficient Selectivity and Backup Opera- pp. 1–1.
tors in Monte-Carlo Tree Search”. In: Computers and [34] N. Justesen, M. S. Debus, and S. Risi. “When Are
Games, 5th International Conference, CG 2006, Turin, We Done with Games?” In: 2019 IEEE Conference on
Italy, May 29-31, 2006. Revised Papers. Ed. by H. J. Games (CoG). IEEE. 2019, pp. 1–8.
van den Herik, P. Ciancarini, and H. H.L. M. Donkers. [35] N. Justesen et al. “Illuminating generalization in deep
Vol. 4630. Lecture Notes in Computer Science. Springer, reinforcement learning through procedural level gener-
2006, pp. 72–83. ation”. In: arXiv:1806.10729 (2018).
[15] J. Denzinger et al. “Dealing with Parameterized Ac- [36] M. Kempka et al. “Vizdoom: A doom-based ai research
tions in Behavior Testing of Commercial Computer platform for visual reinforcement learning”. In: 2016
Games.” In: CIG. Citeseer. 2005. IEEE Conference on Computational Intelligence and
[16] A. Drachen, A. Canossa, and G. N. Yannakakis. “Player Games (CIG). IEEE. 2016, pp. 1–8.
modeling using self-organization in Tomb Raider: Un- [37] L. B. M. Kummer, J. C. Nievola, and E. C. Paraiso.
derworld”. In: 2009 IEEE symposium on computational “Applying Commitment to Churn and Remaining Play-
intelligence and games. IEEE. 2009, pp. 1–8. ers Lifetime Prediction”. In: 2018 IEEE Conference
[17] A. Ecoffet et al. “Go-Explore: a New Approach for on Computational Intelligence and Games, CIG 2018,
Hard-Exploration Problems”. In: (2019). arXiv: 1901. Maastricht, The Netherlands, August 14-17, 2018. IEEE,
10995. 2018, pp. 1–8.
[18] A. E. Eiben and J. E. Smith. Introduction to Evolu- [38] Y. LeCun, Y. Bengio, and G. Hinton. “Deep learning”.
tionary Computing. 2nd. Springer, 2015. In: nature 521.7553 (2015), p. 436.
[19] M. S. El-Nasr, A. Drachen, and A. Canossa. Game [39] A. Lerer et al. Improving Policies via Search in Coop-
analytics. Springer, 2016. erative Partially Observable Games. 2019. arXiv: 1912.
[20] M. Gallagher and M. Ledwich. “Evolving pac-man play- 02318 [cs.AI].
ers: Can we learn from raw input?” In: 2007 IEEE [40] A. Liapis, G. N. Yannakakis, and J. Togelius. “Sen-
Symposium on Computational Intelligence and Games. tient Sketchbook: Computer-aided game level author-
IEEE. 2007, pp. 282–287. ing.” In: FDG. 2013, pp. 213–220.
[21] S. Gelly and D. Silver. “Monte-Carlo tree search and [41] A. Liapis et al. “Orchestrating Game Generation”. In:
rapid action value estimation in computer Go”. In: Ar- IEEE Trans. Games 11.1 (2019), pp. 48–68.
tificial Intelligence 175.11 (2011), pp. 1856 –1875. [42] D. P. Liebana et al. “Rolling horizon evolution ver-
[22] M. Genesereth, N. Love, and B. Pell. “General game sus tree search for navigation in single-player real-time
playing: Overview of the AAAI competition”. In: AI games”. In: Genetic and Evolutionary Computation
magazine 26.2 (2005), pp. 62–62. Conference, GECCO ’13, Amsterdam, The Nether-
[23] E. Giacomello, P. L. Lanzi, and D. Loiacono. “DOOM lands, July 6-10, 2013. Ed. by C. Blum and E. Alba.
level generation using generative adversarial networks”. ACM, 2013, pp. 351–358.
In: 2018 IEEE Games, Entertainment, Media Confer- [43] S. Liu et al. “Emergent Coordination Through Compe-
ence (GEM). IEEE. 2018, pp. 316–323. tition”. In: International Conference on Learning Rep-
[24] I. Goodfellow et al. “Generative adversarial nets”. In: resentations. 2019.
Advances in neural information processing systems. [44] D. Livingstone. “Turing’s Test and Believable AI in
2014, pp. 2672–2680. Games”. In: Comput. Entertain. 4.1 (2006).
[25] S. F. Gudmundsson et al. “Human-like playtesting with
deep learning”. In: 2018 IEEE Conference on Compu-
REFERENCES 11

[45] S. M. Lucas and G. Kendall. “Evolutionary computa- [64] J. Raiman, S. Zhang, and F. Wolski. “Long-Term Plan-
tion and games”. In: IEEE Computational Intelligence ning and Situational Awareness in OpenAI Five”. In:
Magazine 1.1 (2006), pp. 10–18. arXiv preprint arXiv:1912.06721 (2019).
[46] M. C. Machado et al. “Revisiting the Arcade Learning [65] S. Risi and K. O. Stanley. “Deep Neuroevolution of
Environment: Evaluation Protocols and Open Prob- Recurrent and Discrete World Models”. In: Proceedings
lems for General Agents”. In: CoRR abs/1709.06009 of the Genetic and Evolutionary Computation Confer-
(2017). arXiv: 1709.06009. ence. GECCO 19. Prague, Czech Republic: Association
[47] K. Makantasis, A. Liapis, and G. N. Yannakakis. “From for Computing Machinery, 2019, 456462.
Pixels to Affect: A Study on Games and Player Expe- [66] S. Risi and K. O. Stanley. “Improving Deep Neuroevo-
rience”. In: 2019 8th International Conference on Af- lution via Deep Innovation Protection”. In: arXiv: 2001.
fective Computing and Intelligent Interaction (ACII). 01683 (2019).
IEEE. 2019, pp. 1–7. [67] S. Risi and J. Togelius. “Neuroevolution in games: State
[48] M. Mateas. “Expressive AI: Games and Artificial Intel- of the art and open challenges”. In: IEEE Transactions
ligence”. In: DiGRA &#3903 - Proceedings of the 2003 on Computational Intelligence and AI in Games 9.1
DiGRA International Conference: Level Up. 2003. (2015), pp. 25–41.
[49] D. Melhart et al. “Your Gameplay Says it All: Mod- [68] S. Risi and J. Togelius. Procedural Content Gener-
elling Motivation in Tom Clancy’s The Division”. In: ation: From Automatically Generating Game Levels
arXiv:1902.00040 (2019). to Increasing Generality in Machine Learning. 2019.
[50] V. Mnih et al. “Human-level control through deep rein- arXiv: 1911.13071 [cs.AI].
forcement learning”. In: Nature 518.7540 (2015), p. 529. [69] J. Runge et al. “Churn prediction for high-value players
[51] V. Mnih et al. “Playing Atari with Deep Reinforcement in casual social games”. In: 2014 IEEE conference on
Learning”. In: CoRR abs/1312.5602 (2013). Computational Intelligence and Games. IEEE. 2014,
[52] A. Nareyek. “Game AI is Dead. Long Live Game AI!” pp. 1–8.
In: IEEE Intelligent Systems 22.1 (2007), pp. 9–11. [70] J. Schaeffer et al. “Checkers Is Solved”. In: Science
[53] A. Nareyek. “Review: Intelligent Agents for Computer 317.5844 (2007), pp. 1518–1522. eprint: https://science.
Games”. In: Computers and Games. Ed. by T. Mars- sciencemag.org/content/317/5844/1518.full.pdf.
land and I. Frank. Berlin, Heidelberg: Springer Berlin [71] J. Schrittwieser et al. Mastering Atari, Go, Chess and
Heidelberg, 2001, pp. 414–422. Shogi by Planning with a Learned Model. 2019. arXiv:
[54] A. Newell and H. A. Simon. “Computer science as em- 1911.08265 [cs.LG].
pirical inquiry: symbols and search”. In: Communica- [72] M. H. S. Segler. World Programs for Model-Based Learn-
tions of the ACM 19.3 (1976), pp. 113–126. ing and Planning in Compositional State and Action
[55] S. Ontañón et al. “A Survey of Real-Time Strategy Spaces. 2019. arXiv: 1912.13007 [cs.LG].
Game AI Research and Competition in StarCraft”. In: [73] M. H. Segler, M. Preuss, and M. P. Waller. “Planning
IEEE Trans. Comput. Intellig. and AI in Games 5.4 chemical syntheses with deep neural networks and sym-
(2013), pp. 293–311. bolic AI”. In: Nature 555.7698 (2018), p. 604.
[56] OpenAI. OpenAI Five. https : / / blog . openai . com / [74] N. Shaker, J. Togelius, and M. J. Nelson. Procedural
openai-five/. 2018. Content Generation in Games. Computational Syn-
[57] M. Parker and B. D. Bryant. “Neurovisual control in thesis and Creative Systems. Springer, 2016.
the Quake II environment”. In: IEEE Transactions [75] D. Silver et al. “A general reinforcement learning algo-
on Computational Intelligence and AI in Games 4.1 rithm that masters chess, shogi, and Go through self-
(2012), pp. 44–54. play”. In: Science 362.6419 (2018), pp. 1140–1144.
[58] C. Pedersen, J. Togelius, and G. N. Yannakakis. “Mod- [76] D. Silver et al. “Mastering the game of Go with deep
eling player experience for content creation”. In: IEEE neural networks and tree search”. In: Nature 529.7587
Transactions on Computational Intelligence and AI in (2016), pp. 484–489.
Games 2.1 (2010), pp. 54–67. [77] D. Silver et al. “Mastering the game of Go without hu-
[59] T. Pepels, M. H. M. Winands, and M. Lanctot. “Real- man knowledge”. In: Nature 550 (Oct. 2017), pp. 354–
Time Monte Carlo Tree Search in Ms Pac-Man”. In: 359.
IEEE Trans. Comput. Intellig. and AI in Games 6.3 [78] P. Spronck et al. “Adaptive game AI with dynamic
(2014), pp. 245–257. scripting”. In: Machine Learning 63.3 (2006), pp. 217–
[60] D. Perez-Liebana et al. “General video game ai: Com- 248.
petition, challenges and opportunities”. In: Thirtieth [79] K. O. Stanley et al. “Designing neural networks through
AAAI Conference on Artificial Intelligence. 2016. neuroevolution”. In: Nature Machine Intelligence 1.1
[61] A. Plaat. Learning to Play — Reinforcement Learning (2019), pp. 24–35.
and Games. https://learningtoplay.net/. 2020. [80] F. P. Such et al. “Deep neuroevolution: Genetic algo-
[62] E. J. Powley, P. I. Cowling, and D. Whitehouse. “In- rithms are a competitive alternative for training deep
formation capture and reuse strategies in Monte Carlo neural networks for reinforcement learning”. In: arXiv:
Tree Search, with applications to games of hidden in- 1712.06567 (2017).
formation”. In: Artificial Intelligence 217 (2014), pp. 92 [81] A. Summerville et al. “Procedural content generation
–116. via machine learning (PCGML)”. In: IEEE Transac-
[63] M. Preuss et al. “Integrated Balancing of an RTS Game: tions on Games 10.3 (2018), pp. 257–270.
Case Study and Toolbox Refinement”. In: 2018 IEEE [82] J. Togelius et al. “Super mario evolution”. In: 2009 ieee
Conference on Computational Intelligence and Games, symposium on computational intelligence and games.
CIG 2018, Maastricht, The Netherlands, August 14- IEEE. 2009, pp. 156–161.
17, 2018. IEEE, 2018, pp. 1–8. [83] R. Torrado et al. “Deep Reinforcement Learning for
General Video Game AI”. In: Proceedings of the 2018
12 REFERENCES

IEEE Conference on Computational Intelligence and

Games, CIG 2018. IEEE, Oct. 2018.
[84] O. Vinyals et al. AlphaStar: Mastering the Real-Time
Strategy Game StarCraft II. https://deepmind.com/
blog / alphastar - mastering - real - time - strategy -
game-starcraft-ii/. 2019.
[85] O. Vinyals et al. “Grandmaster level in StarCraft II
using multi-agent reinforcement learning”. In: Nature
575 (Nov. 2019).
[86] V. Volz. “Uncertainty Handling in Surrogate Assisted
Optimisation of Games”. In: KI - Künstliche Intelli-
genz (2019).
[87] V. Volz et al. “Evolving mario levels in the latent space
of a deep convolutional generative adversarial network”.
In: Proceedings of the Genetic and Evolutionary Com-
putation Conference. ACM. 2018, pp. 221–228.
[88] G. N. Yannakakis. “Game AI Revisited”. In: Proceed-
ings of the 9th Conference on Computing Frontiers.
CF 12. Cagliari, Italy: Association for Computing Ma-
chinery, 2012, 285292.
[89] G. N. Yannakakis, R. Cowie, and C. Busso. “The or-
dinal nature of emotions: An emerging approach”. In:
IEEE Transactions on Affective Computing (2018).
[90] G. N. Yannakakis and A. Paiva. “Emotion in games”.
In: Handbook on affective computing (2014), pp. 459–
471.
[91] G. N. Yannakakis and J. Togelius. “A Panorama of
Artificial and Computational Intelligence in Games”.
In: IEEE Trans. Comput. Intellig. and AI in Games
7.4 (2015), pp. 317–335.
[92] G. N. Yannakakis and J. Togelius. Artificial Intelli-
gence and Games. Springer, 2018.
[93] G. N. Yannakakis and J. Togelius. “Experience-driven
procedural content generation”. In: IEEE Transactions
on Affective Computing 2.3 (2011), pp. 147–161.
[94] G. N. Yannakakis and J. Togelius. “The 2010 IEEE
Conference on Computational Intelligence and Games
Report”. In: IEEE Comp. Int. Mag. 6.2 (2011), pp. 10–
14.
[95] G. N. Yannakakis et al. “Player modeling”. In: Schloss
Dagstuhl-Leibniz-Zentrum fuer Informatik. 2013.
[96] C. Zhang et al. “A study on overfitting in deep rein-
forcement learning”. In: arXiv:1804.06893 (2018).
[97] J. Zhu et al. “Explainable AI for designers: A human-
centered perspective on mixed-initiative co-creation”.
In: 2018 IEEE Conference on Computational Intelli-
gence and Games (CIG). IEEE. 2018, pp. 1–8.