Game Servers Deployment Automation Case Study

Zane Ouimet, Heath Caswell* and Youry Khmelevsky∗ Rob Bartlett and Alex Needham
Computer Science, Okanagan College, Kelowna, BC Canada WTFast, Kelowna, BC Canada
Emails: zane.ouimet@outlook.com, heath.caswell@gmail.com and Emails: {rob, alex}@wtfast.com
ykhmelevsky@okanagan.bc.ca

∗ Also Affiliated with Mathematics, Statistics, Physics, and Computer Science Department
Irving K. Barber School of Arts and Sciences, UBC, BC Canada

Abstract—This paper describes a software system prototype servers provide new flexibility in hardware reservation and al-
for automated game servers deployment and configuring cus- location but their use can make resource optimization difficult
tomized game servers on demand. The described system has a web by making it context sensitive i.e. dependent on the allocation
interface which allows customers to create accounts, purchase, of virtual machines (VMs) to hardware.
services, and gain access to and configure their purchased web
servers. This service consists of a website which acts as an The main project’s objective was to study predictive mon-
interface for customers to purchase subscriptions, and gain access itoring and optimization for game server clusters. The first
to and configure their purchased servers. Behind the website is phase of the project was to gather performance data about
a system which dynamically deploys virtual machines with the game servers, then analyze its time behaviour to allow the
requested configurations, handles all of the networking details, creation of a performance-prediction software module [5].
and provides information back to the customer on how to connect
to their server.
The initial module version applied virtualized game servers
in various configurations, and later versions were tested with
The proposed prototype was tested by deploying popular physical servers as well as parallel (cluster) game servers.
Minecraft game server. This facilitated network research by Later, the project investigated performance optimization based
allowing users to have a more scalable testing environment and on short-term predictions.
thus enable controlled laboratory experiments. This paper goes
through the entire life-cycle of the project, starting with some Our main contributions are: (1) a unique network and
information on existing research about the subject, and how it gaming servers infrastructure created for the emulation ex-
relates to ours. Following that we describe our project require- periments, which is also being used to perform stress testing
ments, the solution we ended up using and how it was modified to and data analysis of network game applications, as well as to
fit our requirements. We then have a section showing performance monitor the performance of game servers within a proprietary
experiments we ran. The final section is the conclusion which talks Gamers Private Network (GPN) [13]; (2) an automated soft-
about the outcome of our project in relation to our original goals,
ware system prototype for creating and configuring customized
and how it will impact future research in this area.
game servers on demand, and (3) using this automated software
system prototype we are able to improve the game servers’
I. I NTRODUCTION utilization. Two networking and servers optimization research
projects GPN-Perf1: ”Investigating performance of gamers
Minecraft [28], [1] is a popular video game played world- private networks” (2014) and GPN-Perf2 research project
wide, and is built simply enough to be used for network application (2015-2018) were funded by Natural Sciences
analysis and research. This paper describes a software system and Engineering Research Council of Canada (NSERC) [21].
prototype for automated game servers deployment and con- This research potentially has far reaching impacts not only in
figuring customized game servers on demand. The described reducing game latency but also in optimizing other types of
system has a web interface which allows customers to create network traffic via the prioritizing of the most important data
accounts, purchase, services, and gain access to and configure packets sent over the network.
their purchased web servers. This service consists of a website
which acts as an interface for customers to purchase subscrip- II. BACKGROUND AND T HEORY
tions, and gain access to and configure their purchased servers.
Behind the website is a system which dynamically deploys The ”large-scale network simulation is an important tech-
virtual machines with the requested configurations, handles all nique for studying the dynamic behavior of networks, network
of the networking details, and provides information back to protocols, and emerging class of distributed applications, in-
the customer on how to connect to their server. cluding Peer-to-Peer and Grid applications” [18], especially
for the game servers behaviour on the internet. We already
The project prototype development was the next step in discussed a Gamers Private Network (GPN) infrastructure
the design, construction and test of a new layer of game prototype in [5], which was developed to investigate traffic
server software that can optimize and monitor in real-time and game servers behaviour in an artificially created testbed
game services [3], [4]. It stems from the observation that game by using a popular game called Minecraft by using a simple
servers place demands on computing resources - hardware bot that was found online. GPN is a WTFast’s client/server
and network - that can vary with user behaviour and whose solution to make online games faster by increasing game speed,
optimization is the key to customer satisfaction. Virtualized reducing games disconnections, deviations and lag caused by

978-1-4673-9519-9/16/$31.00 ©2016 IEEE

spikes in packet traffic [13]. Our goal was to generate realistic product worked like so; a user can sign up, login in and order
network traffic related to video game environment on the a game server subscription of their choice. The subscription
Internet by ready made game emulation applications, available along with the user are stored in the database. Puppet [25] -
online. a configuration management tool- referred to the database, to
configure and deployed a custom game server subscription to
”A video game network is a distributed set of apparatus the user as well as handled configuration updates. The Domain
which are capable of exhibiting an interactive single identity Model of the developed system is shown at Fig. 1.
game” [26]. Response time and network latencies are very
important video game parameters, which can be a reason The system architecture of the developed prototype is
for the gamers’ frustrations and dissatisfaction, especially in shown on Fig. 2 and the Web subsystem architecture is shown
the multi-user environment. On the other hand, ”the online on Fig. 3.
service’s computers themselves introduce latencies, typically
increasing as the number of active users increases” [24]. Originally we used Foreman [27] which is an open source
tool that allows us to configure and automate repetitive de-
In [12] and in [14] multiuser online video game architec- ployment of modules on our servers in orchestration with
tures been discussed in order to reduce the bandwidth and the puppet. However Foreman proves to be difficult to use in an
servers processing time. Their approach may improve scaling, environment that requires redundancy as it creates a single
but it ”opens the game to additional cheating, since players failpoint. The end user software (game server) was run on a
are responsible for distributing events and storing state”. A linux based kernel and a WINE module [29] was created in
”hybrid P2P (Peer to Peer) architecture” with a central arbiter order to run Windows applications and is manifested when
was proposed in [23]. needed.
Authors in [31] have made an observation that ”Internet
delay is important for First Person Shooter (FPS) games A. System Redesign
because it can determine who wins or loses a game”. ”The time
the information reaches the server that matters, not the time Later, during the Summer 2015 the whole system was
the player actually pushes the button” [5]. Related research redesigned. The WTFast Game servers application is divided
has discovered that the latency in Internet is stronger for the into several parts on. For the purposes of this paper only the
”first-person perspective and a changing model” online games things required for the user interface operation namely the
[6]. In our case, Minecraft is a ”first-person perspective and Representational State Transfer (REST) Application Program
a changing model” online game. A study of a different first- Interface (API) and User Interface (UI). ”A service based on
person games show that ”client traffic is characterized by an REST is called a RESTful service”. The Web UI (Fig. 4) is the
almost constant packet and data rate” [9]. front end of the application. It is built using AngularJS [10]
and utilizes Satellizer [2] as a starting point. Additional angular
Many authors tried to emulate and investigate game traffic components are also required since they split from the main
for online games over the Internet. In [17] the author inves- Angular branch. The charts are generated by using an Angular
tigate problems related to the StarCraft II game, which were library called Angular ChartJS [20] which is essentially a
found ”due to various network conditions such as delay, jitter wrapper for the JQuery library ChartJS.
and loss”. The traffic characteristics and traffic models are
discussed, along with the suggested simulation and emulation The API (Fig. 4) uses a number of built in Laravel
tools. In [7] authors investigate traffic, generated by a thin functionality including Queues, Jobs, Commands. See the
client of OnLive games. They investigated traffic characteris- Laravel documentation [15] for more information on those.
tics such as the bitrates, inter-packet times as well as packet Cloudstack/Openstack keys and credentials are stored in
sizes. In [19] the authors discussed Massively Multiplayer app/services.conf.
Online Games problems, especially related to data centre
computing resources, and the related workload models. In the This implementation utilizes Cloudstack for a backend
paper they are trying to predict the resource demand by using management of Virtual Machines. Cloudstack’s ReST API
trace-based simulation as well. It was already discovered in has two types of requests, async (Fig. 7) and sync (Fig. 6).
[30], that ”players with higher delays (whether due to slower The latter can be processed normally with only the overhead
connections, congestion or a larger distance to the server) are involved in making an HTTP request. These HTTP requests are
at a clear disadvantage relative to players with low delay”. pushed onto the Beanstalk job queue to be processed without
The authors used bots instead of gamers as well to evaluate slowing down the API. The first type however will respond
eliminating delay differences between players [30]. A traffic with a job ID which is meant to be used to poll the API for
generation tool OpenAirInterface Traffic Generator (OTG), updates to that job, when that job completes it returns the
which can be used for online gaming testing and evaluation, requested response. As a result, we cannot couple the requests
as well as for modelling and simulation was discussed in [11]. to the HTTP request to Cloudstack. Instead the job proceeds
as in a synchronous request but instead of updating the server
III. S YSTEM D ESIGN immediately with the new information, instead a Laravel job
[16] is created and stored in the database first. This job is
We began our project with the search for an existing then processed by an Artisan command run on a schedule
program that would meet our criteria. During the ongoing (every minute currently) this command will retrieve all pending
research project we have designed a system made up of four jobs and query Cloudstack for updates on them. The updated
parts; database, server, server deployment and web which are information is then saved in the database for retrieval by the UI.
integrated together creating a functional system. In the end, the This makes necessary the UI polling to get updated information
Fig. 1. Domain Model of the Developed Prototype

Fig. 2. System Architecture

Fig. 4. WebUI

Fig. 3. Web Subsystem Architecture

Fig. 5. WebAPI

as well as the API polling the Cloudstack API to ascertain the

status of asynchronous tasks.
systems and may require additional effort to either maintain
or replace parts with a singular language or technology such
IV. D EVELOPMENT
as replacing PHP with NodeJS. At this time the PHP API is
The entirety of the project consisted of two main areas the interacting directly with Cloudstack to create and control VMs
”web” and ”infrastructure” parts. The prototype development while utilizing Puppet to install and configure software. The
part of the project was broken into two parts: a RESTful prototype makes use of PHP’s ability to access and write to
API (written in PHP) and a JavaScript UI. The diverse range the file structure allowing for the automated creation of the
of technologies being used can create issue while integrating Puppet Node files, or more simply put, machine configuration
Fig. 6. Synchronous Request.

Fig. 8. Premodification issues with Foreman network provisioning

While steps have been taken to manage problems com-

municating with Cloudstack the system still requires it to
acknowledge the commands to report the outcome of a given
command. A response of nothing will simply go unnoticed un-
til the systems flood protection kick in. If Cloudstack recovers
or finally reports a failed command it will be inundated and
flooded by the requests being sent from the API. At the time of
this writing Cloudstack will simply lock up and not process or
acknowledge commands causing both the API and Cloudstack
to back up. The API will log an error message when a com-
mand to Cloudstack takes longer than 5 minutes an example
of Cloudstack lockup can be seen in Fig. 9 (screenshot) where
the error log graph shows a large spike in problems. One of
the major problems was anytime our hypervisor would raise an
Fig. 7. Asynchronous Request. alarm, even if it was only a warning, Openstack would treat
it as a system error. When the configured 25% disk space
warning sounded, we lost the ability to generate VM’s for a
files. Both the UI and API are incomplete; however given a week, without any useful log files to help pin point them. The
functioning Cloudstack [8] install the system is able to: create, issues may be traceable to problems with the virtual network
destroy, start, stop, reboot, and configure a game server on the or physical storage problems.
VM through puppet files. The current issue that is blocking
further progress and development is the lack of reliability from The problems with Cloudstack led to the beginning of
Cloudstack when an error or exception occurs processing a integration with Openstack [22] as an alternative at this time
request. however the system uses only Cloudstack and the hope is to
resolve the issue and resume working with Cloudstack. API
Before changing to Cloudstack, various experiments were ”connectors” have been written for both Cloudstack and Open-
put into attempting to alter Foreman to allow scalability, stack. These classes can be easily swapped to perform similar
starting on API calls per Foreman server; It would only process functions on the respective hypervisors with the Cloudstack
5 API calls at a time, often timing out when communicating version containing far more functionality and the Openstack
between it’s own components. We were able to make each API system deprecated. System stability is the one of the most
request more reliable by adding retries, however the reasoning important concerns and single fail points should be avoided
behind 5 connections could not be found, and on the contrary, at all cost with ready redundancy and high quality exception
evidence was found suggesting that each instance should logging being necessary features to avoid users losing access
accept up to 30 connections. to their servers.
With the modifications, we where able to generate around A. User’s Interface
30 network provisioned machines at once, with only rare minor
errors; twice as many as before modifications. Likewise, with From a user’s perspective the application is capable:
image based provisioning, we where able to increase from 85% • Registering users
success, to 96% success, but this never solved the core issue,
which was the max number of communications to the Foreman • Authenticating users
server; If a user couldn’t change their machine configuration,
• Creating and controlling servers
it was a failure. This lead to changing the infrastructure to
Cloudstack; a scalable enterprise machine deployment service, • Creating new software (puppet module generation)
and relying on our web server writing the Puppet Node files. [admin]
Fig. 9. Admin Dashboard Showing Error Graph.

• Viewing error logs [admin] • Advanced resource allocation

• Changing user role [admin] • Failsafes and issue handling
• Updating profile • Payment and store
• Deleting server [admin] • Improved UI design

Our initial attempt to deploy a gaming server was with • Better user profiles
Minecraft. Using foreman and puppet we have automated
By using predictive models, new servers may be auto-
deployment and configuration conceding a purchase from a
matically added when traffic levels require more resources to
user via an external purchasing web system. The server page
maintain optimal performance. These models will likely be
is shown at Fig. 10 and process of virtual machines deployment
altered on a case by case basis and require operator interaction
is shown at Fig. 11.
to deal with high variance events such as a new game launch
The interface was built on a two tier system that separated or an unpredictable flood to a particular game.
administrators and users. It was meant to be extensible in the
Testing of the network infrastructure and the network
event that further user roles where required. A user was granted
software will be done using a program that emulates many
control over their servers and virtual machines but no other
players connecting to multiple game servers on several virtual
whereas an admin was capable of modifying and accessing
machines in the network. The goal of these network tests is
any users accounts or their servers. Additional account levels
to identify the point at which the network software can no
will be added as needed to accommodate new staff or users.
longer keep up to the flow of traffic, i.e. the point where the
network software becomes a cause of latency. The tests will
V. F UTURE W ORK also serve to identify the capacity of the game servers and
their host virtual machines. Initially the tests will be performed
Our future research will be related to utilizing the in- under ideal conditions, that is, on a local network with almost
frastructure created in the first phase of the project as the no wire latency. Later, network factors such as latency and
framework on which to test and build game server software that jitter can be artificially added to the network to emulate real
can optimize and monitor WTFast game services in real time. conditions of the game being played over the Internet on a
An effort will also be made to create and utilize predictive geographically remote server in order to confirm our tests in
models for handling anticipated demand for a particular server. a more realistic environment.
This infrastructure will allow new servers to be automatically
deployed and configured for use as private game servers, In this paper we discussed latency for gaming, which is
while also monitoring their performance and usage statistics, vital and it is discussed qualitatively both singularly and via
including the following additional features: networks. In our future research we will try to capture the
latency issues quantitatively. Then we will try to do relevant
• Authentication with WTFast comparisons under ideal versus the point where it ’gracefully
• Integration with the GPN degrades’ or the point where we can provide a consistent
performance across the spectrum of users. On the other hand,
• Load balancing for both the API and UI will try to go more in depth qualitatively as to those measures
Fig. 10. The Prototype’s Server Page.

Fig. 11. Server List With Deploy Machine.

that make it good or bad so we can perform some sort of offline, controlled environment. Such studies will then allow
evaluation to provide better conclusions and recommendations. intelligent, dynamic and online methods for game network
optimization which could provide benefits to a number of
Future research will investigate the performance of very network applications.
large clusters of the game servers, for example with par-
allel hardware, and time-series models of such populations VI. C ONCLUSION
for ultra-large-scale emulations of game environments. The
resulting methods will constitute a full laboratory toolbox to Network analysis is a broad spectrum of research, that can
explore latency and general online-gaming performance in an be studied in many different ways. Our focus on latency within
video games led us to experimentation with Minecraft clients [8] Apache CloudStackTM . Open source cloud computingTM :
and servers due to its simplicity and the nature of Java. The https://cloudstack.apache.org.
need for a controlled environment to ensure consistent and [9] Johannes Färber. Traffic modelling for fast action network games.
accurate results led to the requirement of multiple automated Multimedia Tools and Applications, 23(1):31–46, 2004.
bots [3], [4], that could interact with the servers in a predictable [10] Google. Angularjs - html enhanced for web apps: https://angularjs.org.
way and for the automated game servers deployment system [11] Aymen Hafsaoui, Navid Nikaein, and Christian Bonnet. Analysis and
experimentation with a realistic traffic generation tool for emerging
prototype and configuring customized game servers on de- application scenarios. In Proceedings of the 6th International ICST
mand. This project was successful in creating just that, meeting Conference on Simulation Tools and Techniques, SimuTools ’13, pages
all originally set criteria. We anticipate that the automated 268–273, ICST, Brussels, Belgium, Belgium, 2013. ICST (Institute
deployment system prototype will greatly help out future work for Computer Sciences, Social-Informatics and Telecommunications
in this field. Based on our experiments, our infrastructure gives Engineering).
to us a suitable tool for the network performance investigation [12] Takuji Iimura, Hiroaki Hazeyama, and Youki Kadobayashi. Zoned
federation of game servers: A peer-to-peer approach to scalable multi-
and to generate a stress test for the game servers. player online games. In Proceedings of 3rd ACM SIGCOMM Workshop
on Network and System Support for Games, NetGames ’04, pages 116–
120, New York, NY, USA, 2004. ACM.
ACKNOWLEDGMENTS
[13] AAA Internet Publishing Inc. Wtfast: https://www.wtfast.com.
The described WTFast Gateway was developed by Com- [14] Jared Jardine and Daniel Zappala. A hybrid architecture for massively
puter Science students at Okanagan College within COSC 470 multiplayer online games. In Proceedings of the 7th ACM SIGCOMM
Software Engineering and COSC 471 Software Engineering Workshop on Network and System Support for Games, NetGames ’08,
pages 60–65, New York, NY, USA, 2008. ACM.
Project courses in the Fall 2014 and in the Winter 2015
[15] laravel.com. Documentation - laravel: http://laravel.com/docs.
(Bachelor of Information Systems program, Computer Science
[16] laravel.com. Queues - laravel: http://laravel.com/docs/5.1/queues.
department). Our thanks to the students Marc Schroth, Mike
[17] Choong-Soo Lee. The revolution of starcraft network traffic. In
Adkins, J. Riley Dunkin, Simon Detlor and Brad French for Proceedings of the 11th Annual Workshop on Network and Systems
their work on the WTFast Gateway. Support for Games, NetGames ’12, pages 18:1–18:2, Piscataway, NJ,
USA, 2012. IEEE Press.
We would like to thank NSERC’s CCI Engage College
[18] Xin Liu and Andrew A. Chien. Traffic-based load balance for scalable
program for supporting our ”GPN-Perf1: Investigating perfor- network emulation. In Proceedings of the 2003 ACM/IEEE Conference
mance of gamers private networks” in 2014 and GPN-Perf2 on Supercomputing, SC ’03, pages 40–, New York, NY, USA, 2003.
research project application in 2015. Our thank to Amazon ACM.
Web Services, Inc. for supporting our capstone student soft- [19] Vlad Nae, Alexandru Iosup, Stefan Podlipnig, Radu Prodan, Dick
ware engineering and our student research projects by AWS Epema, and Thomas Fahringer. Efficient management of data center
(Amazon Web Services) Grants for Research and Education. resources for massively multiplayer online games. In Proceedings of
the 2008 ACM/IEEE Conference on Supercomputing, SC ’08, pages
We would like to thank both reviewers for their insightful 10:1–10:12, Piscataway, NJ, USA, 2008. IEEE Press.
comments on the paper, as these comments led us to an [20] Ngmodules.org. angular-chartjs - a fully functional directive set for the
chartjs library: http://ngmodules.org/modules/angular-chartjs.
improvement of the work. We thank the reviewers because
[21] NSERC. Natural sciences and engineering research council of canada
these points were useful to clarify the main focus of the paper. (nserc): http://www.nserc-crsng.gc.ca.
[22] Openstack. Open source software for creating private and public clouds:
R EFERENCES https://www.openstack.org.
[23] Joseph D. Pellegrino and Constantinos Dovrolis. Bandwidth require-
[1] Mojang Synergies AB./Microsoft. Minecraft home page: ment and state consistency in three multiplayer game architectures. In
https://minecraft.net/., June 2014. Proceedings of the 2Nd Workshop on Network and System Support for
[2] Simonas Gildutis Dimitris Bozelos. Alice Chen, Salim Alami. Satellizer Games, NetGames ’03, pages 52–59, New York, NY, USA, 2003. ACM.
- token-based angularjs authentication: https://satellizer.herokuapp.com. [24] S. G. Perlman. Network architecture to support multiple site real-time
[3] T. Alstad, J.R. Duncan, S. Detlor, B. French, H. Caswell, Z. Ouimet, video games. United States Patent number 5,586,257, Dec. 17, 1996.
Y. Khmelevsky, G. Hains, R. Bartlett, and A. Needham. Minecraft [25] Puppetlabs. Puppet labs: https://puppetlabs.com.
computer game performance analysis and network traffic emulation by
a custom bot. In Science and Information Conference (SAI), 2015, pages [26] D. H. Sitrick. Video game network. United States Patent number
227–236, July 2015. 4,572,509, Feb. 25, 1986.
[4] T. Alstad, J. Riley Dunkin, S. Detlor, B. French, H. Caswell, Z. Ouimet, [27] Theforeman.org. Foreman - a complete lifecycle management tool for
Y. Khmelevsky, and G. Hains. Game network traffic simulation by a physical and virtual servers: http://theforeman.org.
custom bot. In Systems Conference (SysCon), 2015 9th Annual IEEE [28] Wikipedia. Minecraft: http://en.wikipedia.org/wiki/minecraft., May
International, pages 675–680, April 2015. 2014.
[5] Trevor Alstad, J. Riley Dunkin, Rob Bartlett, Alex Needham, Gaétan [29] WineHQ. Run windows applications on linux, bsd, solaris and mac os
Hains, and Youry Khmelevsky. Minecraft computer game simulation x: https://www.winehq.org.
and network performance analysis. In Second International Conferences [30] Sebastian Zander, Ian Leeder, and Grenville Armitage. Achieving
on Computer Graphics, Visualization, Computer Vision, and Game fairness in multiplayer network games through automated latency
Technology (VisioGame 2014), Bandung, Indonesia, November 2014. balancing. In Proceedings of the 2005 ACM SIGCHI International
Accepted for publication. Conference on Advances in Computer Entertainment Technology, ACE
[6] Mark Claypool and Kajal Claypool. Latency and player actions in ’05, pages 117–124, New York, NY, USA, 2005. ACM.
online games. Commun. ACM, 49(11):40–45, November 2006. [31] Qili Zhou, C.J. Miller, and Victor Bassilious. First person shooter
[7] Mark Claypool, David Finkel, Alexander Grant, and Michael Solano. multiplayer game traffic analysis. In Object Oriented Real-Time Dis-
Thin to win?: Network performance analysis of the onlive thin client tributed Computing (ISORC), 2008 11th IEEE International Symposium
game system. In Proceedings of the 11th Annual Workshop on on, pages 195–200, May 2008.
Network and Systems Support for Games, NetGames ’12, pages 1:1–1:6,
Piscataway, NJ, USA, 2012. IEEE Press.