Testing Strategies for Smart Cities applications: A Systematic

Mapping Study
Alex Costa Leopoldo Teixeira
Federal University of Pernambuco Federal University of Pernambuco
Recife, Pernambuco Recife, Pernambuco
ajc2@cin.ufpe.br lmt@cin.ufpe.br

ABSTRACT 1 INTRODUCTION
Context: Smart Cities are urban areas that enable the development Smart Cities are urban areas that enable the development of appli-
of applications to improve city resources management, through the cations to achieve improvements in resource management of the
use of information technology such as Internet of Things and cloud city, through the use of information technology such as Internet of
computing, as well as government data availability and citizen Things (IoT), cloud computing, government data availability and
participation. Some challenges identified in the development of citizen participation. Smart Cities are an approach that has already
solutions for this context are: scalability, modularity and security. been discussed for some time [12], which are imagined as capable
Goal: Testing activities are critical to the verification and validation of managing the use of their resources, providing improvements in
of Smart City solutions, so our goal is to develop a map of test citizens’ quality of life, in different dimensions, such as mobility,
strategies for applications developed in the context of Smart Cities. governance, environment, economy, people and life [5]. Through
Method: For this study we defined a systematic literature review recent technological advances and increasing access to such tech-
protocol to identify, select, analyze and synthesize the results of nology, the number of IoT devices already exceeds the amount of
previously published empirical studies in the software engineering mobile devices that are used by people. IoT is then expected to
literature, related to testing Smart Cities applications. Results: consolidate as an essential part of Smart Cities [12].
In this systematic mapping, 13 articles were selected, that have Testing these large scale applications is usually hard, since they
identified test strategies used by developers for the application involve a large number of devices, sensors, and vast amounts of
testing process. We have also identified some difficulties faced in generated data. Moreover, what works well in one application may
the process of testing these applications through reports present not work in another. As an example, in the context of solutions
in the works selected in this mapping. Conclusion: Our research proposed for urban traffic management, some test beds already
synthesized evidence that we hope might serve as a data source for exist [11, 13, 16]. These may work fine, however we can not say
academic research and industrial practice. As future work we plan that they will work well in all cities in which such solution is imple-
to apply our results highlighting a case study in real applications mented. Currently, no standard has been found in the literature to
to validate the collected evidence. be adopted or even a catalog of best practices in which the commu-
nity can rely to define which test strategies fits best for a particular
CCS CONCEPTS solution to be implemented.
Therefore, based on this lack of evidence, the motivation to carry
• Software and its engineering → Software testing and de-
out a study to propose a solution to this identified gap arises. We
bugging; Empirical software validation; • General and reference
are interested in trying to identify the strategies used to test appli-
→ Empirical studies;
cations developed for the context of Smart Cities. To achieve this
goal, we systematically examined the literature with the objective
KEYWORDS of identifying, analyzing and mapping the evidence about this issue,
Smart City, Tests, Mapping Study seeking to answer some questions directed to the strategies that are
used by the developers to test. Moreover, we are also interested on
ACM Reference Format: trying to identify possible patterns of existing test strategies and
Alex Costa and Leopoldo Teixeira. 2018. Testing Strategies for Smart Cities potential difficulties to test Smart Cities applications. In the search
applications: A Systematic Mapping Study. In III Brazilian Symposium on process in the literature that was carried out, we found some studies
Systematic and Automated Software Testing (SAST ’18), September 17–21, that involve Smart Cities [4, 9, 14, 24]. However, these studies do
2018, SAO CARLOS, Brazil. ACM, New York, NY, USA, 9 pages. https://doi.
not deal with testing strategies, which are the focus of our research.
org/10.1145/3266003.3266005
By answering these questions based on the existing literature,
we identified three main strategies that are commonly used to help
developers in the application testing process for the context of Smart
Publication rights licensed to ACM. ACM acknowledges that this contribution was Cities. The most commonly reported strategy among those three is
authored or co-authored by an employee, contractor or affiliate of a national govern-
ment. As such, the Government retains a nonexclusive, royalty-free right to publish or that of test-beds, which are platforms that attempt to be as close
reproduce this article, or to allow others to do so, for Government purposes only. to reality as possible, to enable verifying and experimenting with
SAST ’18, September 17–21, 2018, SAO CARLOS, Brazil applications [6]. We have also identified a number of challenges
© 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-6555-0/18/09. . . $15.00
faced by developers that were reported in the selected studies. A
https://doi.org/10.1145/3266003.3266005 recurring challenge is that proposed solutions are usually very
SAST ’18, September 17–21, 2018, SAO CARLOS, Brazil Alex Costa and Leopoldo Teixeira

narrow and limited to a specific context. Based on the results we

These are those that combine the facilities of information
obtained, we hope to provide relevant data that might be useful for
and communication technologies and Web 2.0 with the or-
the testing processes of Smart Cities applications. Moreover, we
ganizational, design and planning efforts to dematerialize
also pose some challenges and implications for future research and
and accelerate bureaucratic processes, helping to identify
industrial practice.
and implement innovative solutions for city complexity
The rest of this article is organized as follows. In Section 2, we
management. Toppeta [23]
present the basic concepts and related works. In Section 3, we
present the systematic review method followed in this study. In
Section 4, we present the results of our review and the answers From the examined literature, we found the definitions that were
to the research questions. In Section 5, we discuss the findings of presented above. It is possible to observe that some common points
our review. Finally, in Section 6, we present our conclusions and are reported by the authors. For instance, we notice that in the defi-
proposals for future work. nitions, the use of (1) information and communication technologies
is very common, as well as that the authors report that the solutions
2 BACKGROUND AND RELATED WORK proposed for a Smart City should consider (2) improving the quality
of life for their citizens. It is also possible to identify the reports on
This section presents existing definitions of Smart Cities. We then the (3) governmental aspects that should enable improvements in
show some related work, not specifically in the area of mapping city management, facilitating and accelerating existing processes,
studies, but on studies related to developing and testing Smart City which might be too bureaucratic. We are aware that there are other
applications. definitions for Smart Cities, but in this work we use these three
points as the baseline for our study.
2.1 Definitions
After conducting an exploratory search in the literature, we iden- 2.2 Related Work
tified several definitions for the concept of Smart Cities, where A well-known Smart City project is the SmartSantander [2], which
several characteristic aspects are informed, such as mobility, gover- was developed in the city of Santander, in Spain. In this project,
nance, participation of society, among others. However, the use of several IoT sensors were deployed providing a large infrastructure
information technology is always included in author definitions, for collecting relevant information through a test scenario, also
so in the following we present some of the definitions we found in known as a test-bed. Test scenario means a standardized laboratory
the literature: environment that is used to test new applications [5].
The Santander test-bed consists of 10 case study implementations
They are those that make systematic use of information that have an environmental monitoring consisting of approximately
and communication technologies to promote efficiency in 2000 IoT devices on public lighting poles that provide different data
the planning, execution and maintenance of urban services such as noise, temperature, lighting and presence sensors. Moreover,
and infrastructures, in the best interests of the actors who outdoor parking management using 400 parking sensors buried un-
work in these cities. Harrison and Donnelly [15] der the asphalt in the main parking lots of the city, the management
has the objective of detecting vacant parking spots in the parking
lots. An extension named environmental monitoring mobile, used
sensors installed in 150 public vehicles with the objective of col-
lecting data in several points of the city. The intensity of traffic
An Smart City has a defined geographical demarcation, that was also monitored with 60 devices located at strategic points in
information and communication technologies, logistics, en- the city entrances to measure the main aspects related to traffic.
ergy production act to create benefits for citizens as quality At some crossroads of the streets there are panels which inform
of life, inclusion and participation and improvement in en- the available parking spaces. Parks and garden irrigation was also
vironmental quality, it is governed by a set of well-defined monitored with 50 devices to collect data. Augmented reality was
subjects, capable of declaring the rules and policies for used in points of interest in the city, with RFID and QR code tags.
municipal government and development. Dameri [10] Participative sensing was also considered, with users using mobile
phones to send relevant data to the SmartSantander platform. They
also experimented with transceivers, where researchers can con-
duct tests in their experiments, such as data mining techniques and
They are those capable of efficiently and effectively con- routing protocols. Finally, there was also experimentation in the
necting the physical infrastructures and information and service level, which differs from the previous one, in the sense that
communication technologies, converging organizational, data that are not available by the native experimentation can be
normative, social and technological aspects in order to im- offered to the researchers for the development of new services. The
prove the conditions of sustainability and quality of life of SmartSantander test scenario enables service delivery and experi-
the population. Moss Kanter and Litow [19] mentation simultaneously, however the project does not process
or store the generated data, which poses a challenge for its use in
other Smart City applications.
Testing Strategies for Smart Cities applications: A Systematic Mapping Study SAST ’18, September 17–21, 2018, SAO CARLOS, Brazil

The work of Dameri Dameri [10] investigated IoT applications. question that guides the conduction of the study is presented in the
One point that was strongly emphasized in the study was the dif- following:
ficulty of maintaining scalability during the development process. RQ1 - What are the strategies used to test applications developed
They report that testing such applications in a controlled environ- for the context of Smart Cities?
ment may not be enough to understand and evaluate applications. In order to help on answering this research question, we have
As a proposed solution, they propose a simulation platform to test devised the following specific questions presented as follows:
and debug applications before the solution is deployed. The pro- RQ1.1 Is there any testing pattern adopted by developers to test
posal focuses on testing for IoT applications with a large number applications developed for the context of Smart Cities?
of devices developed for urban environments. For the platform, the RQ1.2 What are the difficulties identified by developers to test
authors define three main advantages: the ability to simulate large- applications developed for the context of Smart Cities?
scale systems, maximize code reuse and finally the high generality
of simulated nodes. 3.2 Data Sources and Search Strategy
In the United Arab Emirates, there is a project called Masdar We followed the Budgen and Brereton [8] approach for using Boolean
City [1], that is being built to be a reference in sustainability and operators AND and OR to help construct search strings. Table 1
technology. One of the objectives of this project is to test initiatives shows the engines used in the automatic search of the research. It
developed for the context of Smart Cities with particular emphasis is important to point out that the engines were selected based on
on the use of renewable power sources. During the year of 2018, in their relevance to the research community and the availability to
partnership with a electric vehicles company, the city is being used perform the downloads of the selected papers.
as a living labs for autonomous driving initiative, further enhancing
the portfolio of initiatives in the context of Smart Cities. The city Table 1: Search Engines
is still in its development process, with a number of other projects
likely to emerge in the context of Smart Cities, and the prediction Search Engine Link
for the overall conclusion of this project is estimated to be 2025. ACM Digital Library https://dl.acm.org
The municipality of West Hollywood is a small city that is lo- IEEE Digital Library https://ieeexplore.ieee.org
cated in Southern California with about 35,000 inhabitants. The ScienceDirect https://www.sciencedirect.com/
city operates a test-bed initiative, which is used to test applications Scopus https://www.scopus.com/
developed in the context of Smart Cities, relying on five key areas: EI Compendex https://www.engineeringvillage.com/
sustainability, mobility, access to government data, resilience, and Springer Link http://www.springer.com.br/
transparency. The city also developed the "WeHo Smart City" strate-
gic plan to help cities better integrate applications developed for
Smart Cities and make cities smarter [3]. After several tests, we have defined a search string that is pre-
sented in the following. Some adjustments were necessary to get
as many works as possible relevant to this scientific study. It is
3 METHOD important to note that some syntax formatting adjustments have
The research developed in this work can be characterized as ex- been applied to the string according to the search engine rules.
ploratory, considering that the exploratory research establishes
methods, techniques and criteria and aims to provide relevant in- application AND ("Smart City" OR "Smart Cities") AND
formation about the research topic [7]. For the development of this test
research the adopted methodology was that of a systematic mapping
of the literature. Petersen et al. [20] defines systematic mapping as
a search method to construct a classification of a particular field 3.3 Inclusion and Exclusion Criteria
of the area of interest searched. For the correct implementation of From the initial set of 2,414 articles, we selected papers that present
this methodology, the definition of objectives, the development of strategies, concepts, guidelines, discussions and lessons learned
a protocol to be followed by the participants of the research and used to test applications developed for the context of Smart Cities.
the selection, extraction and analysis of data relevant to research We excluded studies that failed to satisfy any of the seven Exclusion
of primary studies that are of research related to the theme of the Criteria (CE) which are shown following:
mapping. EC01 Written in any language but English.
In the methodology of systematic mapping studies, the research EC02 Not accessible on the Web.
questions are broader and more general when compared to those EC03 Invited papers, keynote speeches, workshop reports, books,
found in Systematic Literature Reviews (SLRs) [17] [8]. theses, and dissertation.
EC04 Incomplete documents, drafts, slide of presentations, and
extended abstracts.
3.1 Research Questions
EC05 Addressing other areas besides computer science.
According to the presented context of this research, the research EC06 Papers that do not present any type of findings or discussions
problem identified and addressed in this work is related to the about strategies used to test applications developed for the context of
absence of information about which test strategies are used in Smart Cities.
applications developed in the context of Smart Cities. The research EC07 Non-accessible studies.
SAST ’18, September 17–21, 2018, SAO CARLOS, Brazil Alex Costa and Leopoldo Teixeira

3.4 Study Selection Table 3: Data Extraction Form

The selection stage of the primary studies was carried out by two
researchers, working independently. In this stage, we selected the Data Description
studies by reading the title, abstract and keywords. Table 2 presents Title Title of the paper.
the results of the automatic search performed in this mapping. Year Year of publication of the paper.
Study Goal Main objective of the research analysed.
Study Motivation Motivations for test applications.
Table 2: Search Engines and their corresponding results in Strategies, concepts, guidelines, discussions,
number of papers Test
lessons learned used to test applications.
Evidence Evidence of use of test.
Engine Papers
ACM Digital Library 159
IEEE Digital Library 330 3.6 Data Synthesis
ScienceDirect 883 To synthesize the obtained data, we performed the following steps:
Scopus 121 Identify test strategies: For each article that was selected in
EI Compendex 276 the mapping, we aimed to identify reports of use of testing strategies
Springer Link 645 that were described by the author.
2414 Identify Evidence of Test Usage: Based on the reports of test
usage in selected papers, we tried to identify evidence of implemen-
tation and testing.
In this stage, 166 articles were selected to continue in the map- Grouping the test strategies: After obtaining the information
ping selection stage. After we carried out the selection phase of described above, we grouped the test strategies used by the works
the studies, each researcher applied the exclusion criterion (EC06) in applications developed for the context of Smart Cities.
to keep only those articles that contain relevant information to
answer at least one of the research questions. At this stage, articles 4 RESULTS
dealing with different themes from the research carried out in this
study were excluded. Duplicated results were also deleted. When it
4.1 Mapping Study Overview
was identified that a study was published in more than one event In this systematic mapping study, we analyzed 13 primary articles,
(conference or journal), a review was undertaken in the publica- with no publication period limit. The selection process occurred
tions for data extraction purposes. We conclude the selection stage between September of 2017 and March of 2018. Table 4 summarizes
with 13 articles. To resolve the disagreements and conflicts between the analyzed papers in chronological publication order.
the two researchers during the selection stage, consensus meetings A temporal distribution of the selected papers for this review is
were held. Figure 1 illustrates the selection stage of the articles. presented in Figure 2, where we can observe a slight increase in
the number of studies during the temporal evolution. We can also
observe that 41.67% of the selected studies (TS9, TS2, TS7, TS2 and
TS1 ) are recent, published in 2017.
Figure 1: Search Strategy - Study Selection

Figure 2: Temporal distribution of primary studies

3.5 Data Extraction

After the final selection, the researchers, working independently,
began the process of extracting relevant data from each selected
article. They analyzed each article to fill out the form information,
as described following in Table 3.
During the data extraction process, meetings were held to clarify
conflicts and disagreements between researchers, aiming to improve 4.2 Test Strategies
the accuracy of the process, and therefore the reliability of extracted In this section we present the collected evidence about the use of
data. test strategies in applications developed in the context of smart
Testing Strategies for Smart Cities applications: A Systematic Mapping Study SAST ’18, September 17–21, 2018, SAO CARLOS, Brazil

Table 4: Summary of Selected Papers Table 5: Test Strategies and their corresponding evidence
sources
Title Year Ref. Source
Streaming the Sound of Test Strategies Evidence
Smart Cities: TS10 - TS11 - TS9 - TS6
2013 TS10 Conference
Experimentations on the Test-bed TS4 - TS3 - TS12 - TS8
SmartSantander test-bed TS1 - TS13
SmarSantander: IOT Living Labs TS2 - TS5 - TS8
experimentation over a 2014 TS11 Conference Simulator TS7 - TS12
smart city testbed
TRESCIMO: European Union
and South African Smart City 2015 TS4 Conference After performing the analysis in the selected works, we managed
Contextual Dimensions to identify 3 common test strategies which are reported by the
A Testbed for SCADA Cyber selected papers. These are strategies used to verify and validate
Security and Intrusion 2015 TS12 Conference solutions developed for the particular context of Smart Cities. It
Detection is likely that other testing strategies used in traditional software
Test of New Control Strategies development projects, such as unit tests, may also be used on such
for Room Temperature Control 2015 TS13 Conference projects.
Systems Detection We noticed that most articles report on the development and
Designing a smart city use of experimental testing infrastructure for conducting research
playground: real-time air quality and experiments of architectures, services and IoT applications in
2016 TS3 Conference the context of some city. These are commonly known as Test-beds,
measurements and visualization
in the City of Things testbed which are platforms that make possible verifying and experimenting
A Framework for IoT Service with projects using a near-to-reality environment [6].
Experiment Plataforms in 2016 TS5 Conference A well-known test-bed is the one from the city of Santander,
Smart-City Environments reported in the TS10 and TS11 articles. The authors of the TS13
Living Labs for Smart Cities 2016 TS8 Conference study use the same strategy, but adopting a different terminology,
Lessons from SmartCampus: namely that of Test-bench. They define a bench of tests to develop
External Experimenting and carry out tests for controlling room temperature. From the
2017 TS9 Conference test-bench it is possible to test the heating system of an apartment
with User-Centric
Internet-of-Things Testbed in a controllable environment under dynamic contour conditions.
Living Lab Bamberg: an From the reports on such articles, we have not identified a sig-
infrastructure to explore nificant difference between test-bed and test-bench strategies. We
2017 TS2 Conference believe that it is only a matter of terminology between the two
smart city research
challenges in the wild terms and that the two strategies are equivalent.
Design of large scale network According to Schumacher et al. Schumacher et al. [21], since 2003
simulator using device 2017 TS7 Conference the Living Labs concept has been used as a strategy for combining
emulator for Internet of Things the government, academia and industry to develop solutions in
certain areas of a city. In TS8, the main focus of the Living Lab
IoTbed: A Generic Architecture
for Smart Cities at the University of Guadalajara was to generate
for testbed as a Service
2017 TS6 Conference metrics that can be used to evaluate the performance of a city based
for Internet of
on the current infrastructure and propose innovative solutions
Things-Based Systems
using Information Technologies.
Poster Abstract: An Open
2017 TS1 Conference The main objective of the Living Lab presented by TS2 is to
Smart City IoT Test Bed
provide an open infrastructure for research in applications devel-
oped based on sensors, providing researchers with the possibility
of developing, testing and evaluating the developed applications.
The main focus in the development of test-beds is the creation
of verification and experimentation platforms with maximum ap-
proximation to reality. The strategy of Living Labs comes with the
cities following the research questions that were decided for this purpose of trying to connect the governmental aspects, the acad-
research. emy, and also the industry. It also considers the development of
RQ1 - What are the strategies used to test applications developed platforms to test the proposed solutions to the context of Smart
for the context of Smart Cities? Cities.
To answer this research question, we have analyzed all of the The TS7 article proposes using the strategy of a simulator archi-
selected studies looking for evidence of the use of test strategies tecture to test and debug IoT application algorithms before they
carried out by the authors. The results are summarized in Table 5. are deployed in the real environment. The focus of the simulator
SAST ’18, September 17–21, 2018, SAO CARLOS, Brazil Alex Costa and Leopoldo Teixeira

is on emulating a large scale system and proposed networking en- to the creation of test plans for applications developed for the Smart
vironments to perform interaction tests that occur within the IoT Cities context. Some of these reports are presented in the following:
application.
The TS12 work reports on the catastrophic impacts that can "...new technologies and solution-optimisations are con-
occur in the case of attacks on intelligent traffic infrastructures. It strained in terms of applicability to the context under
also states how it is often unfeasible to enhance attack tests and which they have been tested..." - TS11
mitigation strategies in real networks. The proposed solution is
a simulated test-bed that simulates a traffic generator to perform
network attack testing by providing system intrusion analysis. The "...there is a need for a test bed, which can enable explo-
authors report that this solution may help researchers to study ration and experimentation..." - TS1
the impact of different attack scenarios and to test new security
solutions.
The strategy of using simulation to perform validation and exper- "...Creating Internet-of-Things (IoT) solutions that can be
imentation in the proposed solutions is similar to the approaches deployed at scale requires adequate experimentation envi-
of test-beds and living labs. Nonetheless, we understand that the ronments..." - TS9
main difference is related to the issue of creating environments
close to reality. For instance, in the TS07 study, the authors report
that the simulator is capable of providing infrastructure to easily "...Experimenting on live systems is generally not advis-
test and debug application algorithms before they are deployed in able and impractical as this may deem the system unstable.
a real environment. Therefore, such strategy might not be limited Such situation calls for the need of an experimental setup
to a specific solution as we see in most of the test-beds seen in this equivalent or quite close to the real scenario for develop-
mapping study. Another difference is related to the issue of scala- ment and testing of security solutions...." - TS12
bility. Still on the TS07 study, authors claim that the architecture
developed for the simulator focuses on large-scale systems, provid-
ing simulated environments for conducting interaction tests within "...The challenge for many smart-city test and experimen-
the application. Therefore, it is important to take into account such tation platforms (TEPs), like living labs, has been the lack
differences when deciding which strategy to choose, based on the of sustainable value creation model..." - TS5
specific needs of the project. Through the needs and requirements
of each solution, the developers can verify which strategy fits best
avoiding possible problems in the verification and experimentation "...Developing Smart Cities solutions faces the challenge of
processes. validating prototypes with respect to the following crite-
From what we identified in the articles selected in this systematic ria: scalability, interoperability, modularity, resiliency, and
review, the identified strategies tend to prepare an environment security...." - TS8
of verification and experimentation that is as close to the reality
as necessary for the solutions that are developed. However, we
Based on such reports, we can observe that testing such appli-
observed that it was not possible to identify a standard to be adopted
cations is still challenging. There is a recurrent need for testing
for the development of test strategies for Smart City solutions.
and experimentation environments, and the inherent difficulty of
RQ1.1 Is there any testing pattern adopted by developers to test
testing such large-scale systems in terms of scalability, security,
applications developed for the context of Smart Cities?
and other non-functional aspects also poses a challenge, since it
After collecting evidence of testing for applications developed in
is usually ill-advised to conduct tests on the actual systems while
the context of Smart Cities, we were able to identify the previously
they are in production. In the following section we discuss poten-
listed strategies. However, even though there is a commonality in
tial implications for research and industrial practice related to our
terms of the used strategies, we could not detect or identify some
findings.
kind of test pattern to be followed by the researchers or developers
of such applications. This is mainly due to the fact that most of the
5 DISCUSSION
test strategies are developed for some specific context which is not
guaranteed to be reused in another application. Another relevant During the process of analyzing the articles we noticed that most
observation is that the financial investment in the majority of cases of the works report on the need for devising test strategies and
is confined to the project duration and does not aim at continuity environments that support different applications developed in the
after the project completion, causing a halt in the development context of Smart Cities. However, we also realize that there is no
of the test strategy, and potentially reducing the incentives for standard to be followed by developers because usually the test
developing strategies that can later be reused in other contexts. strategies which are used have been developed for a specific case
RQ1.2 What are the difficulties identified by developers to test and do not continue after the project completion.
applications developed for the context of Smart Cities?
During the process of analyzing the studies, we were able to 5.1 Implications for Research
identify some of the challenges and difficulties related to testing In this section, we discuss the implications of this mapping study
reported by authors in their research. Some of these are also related for future software engineering research. From the results of this
Testing Strategies for Smart Cities applications: A Systematic Mapping Study SAST ’18, September 17–21, 2018, SAO CARLOS, Brazil

study we were able to identify some challenges faced by developers particular test strategy to be adopted for the applications that have
to test applications for the context of Smart Cities. been developed in the Smart Cities context.
Based on the answers from RQ1.2, we observed that one of the In the results section of this study we present the strategies
main challenges for the developers is that the applications are reported by the works that were selected in the mapping. Based
developed for specific and narrow domains. Moreover, they are on these results, developers can save the time of searching the
limited to the context where they have been tested, as TS2 reports. literature to try to find data that helps on choosing and establishing
A considerable research challenge would be to attempt to develop the testing process. This might also depend directly on the adopted
testing patterns and strategies that are not limited to a specific culture for each company. For instance, if the company has the
solution but can be reusable in other solutions that are developed culture to seek information in the literature, our work might serve
in the context of Smart Cities. as a good data source for directing this search, and potentially
It is the case that, even though each application has its own helping to define the testing process that might be adopted in the
set of specific features, in many cases there are recurring patterns company.
related to these features, such as the use of localization technologies. Second, our study might also be able to collaborate in the de-
Therefore, a direction towards this challenge would be to have at velopment of test strategies. One of our research questions is to
least a test-bed framework that could abstract away these common try to identify the difficulties faced by the developers to test the
features, that could be then instantiated for each specific Smart City applications developed in the context of Smart Cities. From our
application. results related to this research question, when a certain company
Besides considering functional properties, we also have the chal- develops a test strategy, it can take into account the difficulties
lenge of limited context related to non-functional aspects of such already reported by the developers that are listed in our results.
applications, as well. For instance, the TS8 work reports on the This way, it can save time and reduce financial costs in the develop-
difficulty of validating prototypes according to some criteria, such ment process and also prevent itself from making possible failures
as scalability, interoperability, modularity, resilience, and security. that could happen if it did not have the previous knowledge of the
In order to try to cover these criteria, it is necessary to have or difficulties already existent for these testing processes.
develop strata of tests that meet these needs. In the TS9 study, it is
reported that in order to create IoT solutions at scale, it is necessary
5.3 Validity and Limitations
to have environments of adequate experimentation for the devel-
oped solutions. From the knowledge that the five criteria mentioned For the accomplishment of this systematic review of the literature,
above are difficulties faced to carry out the validation of prototypes we are based on the guidelines presented by Budgen and Brereton
of solutions, the need arises to develop scientific studies that can [8], in the study of threats to the validity of Teixeira et al. [22] and
direct the developers to develop tests strategies that can cover such also the four main threats to validity presented by Wohlin et al.
non-functional criteria. [25] which are: internal, external, construct and conclusion.
The TS6 study raises a rather interesting question for future Internal validity: Skill levels in the development of systematic
research: How can a resource-constrained IoT device researcher mapping of the literature can be considered a threat to the inter-
develop and test their IoT-based applications without making a nal validity of the research. In trying to avoid this threat, several
significant investment in buying and maintaining a large number research planning meetings were held before starting the research.
of IoT devices? We saw that another difficulty is the lack or conti- Another threat to the internal validity to be considered is the com-
nuity of resources for the developers, causing many experimental munication between the researchers. We tried to avoid that, by
platforms to perish external funding ends, as the TS5 study reports. having researchers working independently in the selection, extrac-
In the research presented by TS9, some desirable properties for tion and analysis stages, so that results were later merged, with
the test strategies are presented and can serve as motivation for possible divergences discussed. The number of researchers partic-
new scientific researches. In the following, we present the needs: ipating in this study is also a threat to be considered, since the
participation of more researchers could offer a broader variety of
• Experimental environment realism. opinions in consensus meetings in the selection, extraction and
• Heterogeneity of IoT devices. analysis stages of the study.
• End User Involvement. External validity: In order to mitigate the generalization of the
results obtained in this study, we tried to avoid some threats. First,
Therefore, through the results presented in this study we hope
we considered the use of six search engines and also did not restrict
to motivate researchers by means of these reports to carry out
the publication time of the studies to try to collect the maximum
scientific research in this context providing data for the developers
number of papers relevant to the search. Another threat is related
to use in their applications.
to the search string used in the search engines. In trying to reduce
this risk, we performed some refinements in the string. For instance,
5.2 Implications for Industrial Practice we performed some tests with the inclusion of more terms in the
The implications for industrial practice that this study can offer search string, such as the term “intelligent cities”. However, the
are presented in two forms. First, the choice of a suitable testing number of retrieved articles was not relevant in relation to the
strategy for the solution that was developed. Based on the common search string used in this study. Nonetheless, we believe that there
strategies reported on our results, developers already have an initial may be a need for more refinements in the string to try to improve
direction on decision making, in terms of how they can choose a the obtained results.
SAST ’18, September 17–21, 2018, SAO CARLOS, Brazil Alex Costa and Leopoldo Teixeira

Construct validity: We can consider as a threat to construct va- by INES 2.0,1 FACEPE grants PRONEX APQ 0388-1.03/14 and APQ-
lidity the definition of our search strategy. In order to reduce the risk 0399-1.03/17, and CNPq grant 465614/2014-0. We also acknowl-
of this threat, we followed the steps reported by Kitchenham [18], edge additional support from FACEPE grant APQ-0570-1.03/14, and
searching for studies that were derived from our research question CNPq grant 409335/2016-9.
and also studying other systematic reviews already published in
the literature. REFERENCES
Conclusion validity: For the results obtained in this study, a [1] [n. d.]. Masdar City project. http://www.masdar.ae/. accessed 05/18/2018.
threat to conclusion validity is related to the fact that our results [2] [n. d.]. SmartSantander project. http://http://www.smartsantander.eu//. accessed
have not yet been applied in some possible scenario or Smart City 05/20/2018.
[3] [n. d.]. West Hollywood the creative city. http://www.weho.org//. accessed
application. Another issue is since there are no other mapping 06/05/2018.
studies in this context that we are aware of, our results have not [4] Leonidas G Anthopoulos. 2015. Understanding the smart city domain: A literature
review. In Transforming city governments for successful smart cities. Springer, 9–21.
been confronted with other results from other studies. [5] Pieter Ballon, Jo Pierson, and Simon Delaere. 2005. Test and experimentation
platforms for broadband innovation: Examining European practice. (2005).
[6] Emanoel Barreiros, Adauto Almeida, Juliana Saraiva, and Sergio Soares. 2011. A
systematic mapping study on software engineering testbeds. In Empirical Software
Engineering and Measurement (ESEM), 2011 International Symposium on. IEEE,
6 CONCLUSIONS 107–116.
Application testing activities are a very important step in the soft- [7] Pedro Alcino Bervian, Amado Luiz Cervo, and Roberto da Silva. 2002. Metodologia
científica. São Paulo: Pretence Hall (2002), 482–493.
ware development process. When ignored, the consequences might [8] David Budgen and Pearl Brereton. 2006. Performing Systematic Literature Reviews
be costly and often lead to considerable problems such as rework in Software Engineering. In Proceedings of the 28th International Conference on
Software Engineering (ICSE ’06). ACM, New York, NY, USA, 1051–1052. https:
in the application development process. Therefore, considering the //doi.org/10.1145/1134285.1134500
fact that developing applications for the context of Smart Cities is [9] Annalisa Cocchia. 2014. Smart and digital city: A systematic literature review. In
still a recent development in industrial and research practice, in Smart city. Springer, 13–43.
[10] Renata Paola Dameri. 2013. Searching for smart city definition: a comprehensive
this study we performed a systematic review of the literature on proposal. International Journal of Computers & Technology 11, 5 (2013), 2544–2551.
the test strategies used in applications developed for the context [11] Adnan A Farooqui, Syed Sajjad Haider Zaidi, Attaullah Y Memon, and Sameer
of Smart Cities. We obtained in the automatic searches more than Qazi. 2014. Cyber security backdrop: A scada testbed. In Computing, Communica-
tions and IT Applications Conference (ComComAp), 2014 IEEE. IEEE, 98–103.
2,000 primary studies and from these we selected 13 relevant studies [12] Mark S Foster. 1990. Cities of Tomorrow: An Intellectual History of Urban
to carry out our research. Planning and Design in the Twentieth Century.
[13] Haihui Gao, Yong Peng, Zhonghua Dai, Ting Wang, and Kebin Jia. 2013. The
From the results of our study, we present evidences of test strate- design of ics testbed based on emulation, physical, and simulation (eps-ics testbed).
gies used in the applications, mainly test-beds, living labs, and In Intelligent Information Hiding and Multimedia Signal Processing, 2013 Ninth
simulators. We also identified through the reports of the selected International Conference on. IEEE, 420–423.
[14] Francesco Gianni and Monica Divitini. 2015. Technology-enhanced Smart City
works that there is no defined pattern of test strategies to be fol- Learning: a Systematic Mapping of the Literature. IxD&A 27 (2015), 28–43.
lowed by the developers for the development of tests. Finally, we [15] Colin Harrison and Ian Abbott Donnelly. 2011. A theory of smart cities. In
also presented the difficulties reported by the authors related to Proceedings of the 55th Annual Meeting of the ISSS-2011, Hull, UK, Vol. 55.
[16] Junho Hong, Shinn-Shyan Wu, Alexandru Stefanov, Ahmed Fshosha, Chen-Ching
testing applications developed for the context of Smart Cities. Liu, Pavel Gladyshev, and Manimaran Govindarasu. 2011. An intrusion and defense
We discussed the possible implications of this research for fu- testbed in a cyber-power system environment. In Power and Energy Society General
Meeting, 2011 IEEE. IEEE, 1–5.
ture scientific research, proposing topics for researchers to conduct [17] Barbara Kitchenham. 2004. Procedures for performing systematic reviews. Keele,
studies to deepen knowledge about testing strategies for Smart UK, Keele University 33, 2004 (2004), 1–26.
Cities applications. We also discussed the implications for the in- [18] Barbara A Kitchenham and S Charters. 2007. Guidelines for performing System-
atic Literature Reviews in Software Engineering. (2007).
dustrial practice reporting on the benefits our study could offer [19] Rosabeth Moss Kanter and Stanley S Litow. 2009. Informed and interconnected:
developers in the testing process for these applications. We hope A manifesto for smarter cities. (2009).
that the results of this study will contribute to future research on [20] Kai Petersen, Robert Feldt, Shahid Mujtaba, and Michael Mattsson. 2008. System-
atic Mapping Studies in Software Engineering.. In EASE, Vol. 8. 68–77.
this topic and that it may also benefit industry professionals in the [21] Jens Schumacher, Karin Feurstein, and Manfred Gschweidl. 2009. Technology
application testing process. Roadmap for Living Labs. In Handbook of Research on Social Dimensions of Semantic
Technologies and Web Services. IGI Global, 838–864.
From the results obtained in this study, some future work oppor- [22] Eudis Teixeira, Liliane Fonseca, and Sergio Soares. 2018. Threats to Validity in
tunities may be cited as a means of contributing to the improvement Controlled Experiments in Software Engineering - What the Experts Say and Why
of research related to this context, such as: (1) conducting a case This Is Relevant. Brazilian Symposium On Software Engineering (2018).
[23] Donato Toppeta. 2010. The smart city vision: how innovation and ICT can build
study from our results in real applications in the context of Smart smart,“livable“, sustainable cities. The Innovation Knowledge Foundation 5 (2010),
Cities; (2) extending this systematic mapping of literature consid- 1–9.
ering a broader search string and other search engines; and (3) [24] Evelin Priscila Trindade, Marcus Phoebe Farias Hinnig, Eduardo Moreira da
Costa, Jamile Sabatini Marques, Rogério Cid Bastos, and Tan Yigitcanlar. 2017.
evaluating our results through a replication of this research. Sustainable development of smart cities: A systematic review of the literature.
Journal of Open Innovation: Technology, Market, and Complexity 3, 1 (2017), 11.
[25] Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and
Anders Wesslén. 2012. Experimentation in software engineering. Springer Science
& Business Media.
ACKNOWLEDGMENTS
We thank Liliane Fonseca and Eudis Teixeira for the feedback on
the threats to validity section. This research was partially funded
1 http://www.ines.org.br
Testing Strategies for Smart Cities applications: A Systematic Mapping Study SAST ’18, September 17–21, 2018, SAO CARLOS, Brazil

SELECTED PAPERS
[TS1] Bharadwaj Amrutur, Vasanth Rajaraman, Srikrishna Acharya, Rakshit Ramesh,
Ashish Joglekar, Abhay Sharma, Yogesh Simmhan, Abhijit Lele, Ashwin Ma-
hesh, and Sathya Sankaran. An open smart city iot test bed: street light poles
as smart city spines. In Proceedings of the Second International Conference on
Internet-of-Things Design and Implementation, pages 323–324. ACM, 2017.
[TS2] Aboubakr Benabbas, Golnaz Elmamooz, Brent Lagesse, Daniela Nicklas, and
Ute Schmid. Living lab bamberg: an infrastructure to explore smart city
research challenges in the wild. KI-Künstliche Intelligenz, 31(3):265–271, 2017.
[TS3] Bart Braem, Steven Latré, Philip Leroux, Piet Demeester, Tanguy Coenen,
and Pieter Ballon. Designing a smart city playground: Real-time air quality
measurements and visualization in the city of things testbed. In Smart Cities
Conference (ISC2), 2016 IEEE International, pages 1–2. IEEE, 2016.
[TS4] Louis Coetzee, Andrew Smith, Alejandra Escobar Rubalcava, Andreea Ancuta
Corici, Thomas Magedanz, Ronald Steinke, Marisa Catalan, Josep Paradells,
Hinesh Madhoo, Tiaan Willemse, et al. Trescimo: European union and south
african smart city contextual dimensions. In Internet of Things (WF-IoT), 2015
IEEE 2nd World Forum on, pages 770–776. IEEE, 2015.
[TS5] Mervi Hamalainen and Pasi Tyrvainen. A framework for iot service experiment
platforms in smart-city environments. In Smart Cities Conference (ISC2), 2016
IEEE International, pages 1–8. IEEE, 2016.
[TS6] Mahmud Hossain, Shahid Noor, Yasser Karim, and Ragib Hasan. Iotbed:
A generic architecture for testbed as a service for internet of things-based
systems. In Internet of Things (ICIOT), 2017 IEEE International Congress on,
pages 42–49. IEEE, 2017.
[TS7] Bumho Kim, Haeyong Kim, and Seon-Tae Kim. Design of large scale network
simulator using device emulator for internet of things. In Advanced Com-
munication Technology (ICACT), 2017 19th International Conference on, pages
855–857. IEEE, 2017.
[TS8] Victor M Larios, Leopoldo Gomez, Olga B Mora, Rocio Maciel, and Natalia
Villanueva-Rosales. Living labs for smart cities: A use case in guadalajara city
to foster innovation and develop citizen-centered solutions. In Smart Cities
Conference (ISC2), 2016 IEEE International, pages 1–6. IEEE, 2016.
[TS9] Michele Nati, Alexander Gluhak, Jaroslaw Domaszewicz, Spyros Lalis, and
Klaus Moessner. Lessons from smartcampus: external experimenting with
user-centric internet-of-things testbed. Wireless Personal Communications,
93(3):709–723, 2017.
[TS10] Congduc Pham and Philippe Cousin. Streaming the sound of smart cities:
Experimentations on the smartsantander test-bed. In Green Computing and
Communications (GreenCom), 2013 IEEE and Internet of Things (iThings/CP-
SCom), IEEE International Conference on and IEEE Cyber, Physical and Social
Computing, pages 611–618. IEEE, 2013.
[TS11] Luis Sanchez, Luis Muñoz, Jose Antonio Galache, Pablo Sotres, Juan R Santana,
Veronica Gutierrez, Rajiv Ramdhany, Alex Gluhak, Srdjan Krco, Evangelos
Theodoridis, et al. Smartsantander: Iot experimentation over a smart city
testbed. Computer Networks, 61:217–238, 2014.
[TS12] Prateek Singh, Saurabh Garg, Vinod Kumar, and Zia Saquib. A testbed for
scada cyber security and intrusion detection. In Cyber Security of Smart
Cities, Industrial Control System and Communications (SSIC), 2015 International
Conference on, pages 1–6. IEEE, 2015.
[TS13] Nina Kopmann, Rita Streblow, and Dirk Müller. Test of new control strategies
for room temperature control systems. In Proceedings of the 4th International
Conference on Smart Cities and Green ICT Systems, pages 277–282. SCITEPRESS-
Science and Technology Publications, Lda, 2015.