2020 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS)

Behavior-driven development (BDD) Cucumber

Katalon for Automation GUI testing case CURA
and Swag Labs
2020 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) | 978-1-7281-9167-6/20/$31.00 ©2020 IEEE | DOI: 10.1109/ICIMCIS51567.2020.9354325

1st Rizal Broer Bahaweres 2nd Elda Oktaviani 3rd Luh Kesuma Wardhani
Department of Informatics Department of Informatics Department of Informatics
Syarif Hidayatullah State Islamic Syarif Hidayatullah State Islamic Syarif Hidayatullah State Islamic
University University University
Jakarta, Indonesia Jakarta, Indonesia Jakarta, Indonesia
Computer Science Dept. elda.oktaviani15@mhs.uinjkt.ac.id Computer Science Dept.
IPB University IPB University
Bogor, Indonesia 5th Arif Imam Suroso Bogor, Indonesia
rizalbroer@computer.org Computer Science Dept luhkesuma@uinjkt.ac.id
School of Bussiness
4th Irman Hermadi IPB University 6th Indra Permana Solihin
Computer Science Dept. Bogor, Indonesia Computer Science Dept.
IPB University arifimamsuroso@apps.ipb.ac.id IPB University
Bogor, Indonesia Bogor, Indonesia
irmanhermadi@apps.ipb.ac.id 7th Yandra Arkeman Informatics
Computer Science Dept Faculty of Computer Science.
Agro-Industrial Tech. Dept Universitas Pembangunan Nasional
IPB University Veteran Jakarta
Bogor, Indonesia Jakarta, Indonesia
yandra.arkeman@gmail.com indrapermana@upnvj.ac.id

Abstract— Graphical user interface (GUI) is widely used in of the objective GUI is makes it easier to interact with a system
software applications. About 50% of the application code is for [2]. Because labor costs are high and demands for high-quality
GUI implementation. Because labor costs are high and demands software are increasing. Therefore automation GUI testing
for high-quality software are increasing, automation GUI will be a good investment for any software company [1].
testing will be a good investment for any software company. We
will apply the Behavior-driven development (BDD) method for Software testing is one of the most important and crucial
automation GUI testing with the Cucumber framework on the phases in the software development life cycle process,
CURA and Swag Labs web applications. The BDD methodology spending an average of 40% to 70% of the software
used is based on the NLP (natural language program) concept. development process [3].
Where the information of semi-automatically produce a step
The Agile software development methodology is
definition and code framework. The scenarios are given on the
natural language and then applied to Software Testing. Class increasingly popular in the software industry because of their
diagrams and sequence diagrams are also applied to each higher chances of success. TDD, BDD, Acceptance Test
represent a code framework automatically. The author uses Driver Development (ATDD) are examples of the widely used
Katalon Studio software. It was found that the results of the test Agile development process where acceptance testing plays a
case automation GUI testing on the CURA website with 210 large role in their success [4]. Based on the implementation of
steps passed, 3 step warnings with average responses times in testing, testing can be classified into two categories. Manual
the test case were 1.92 minutes. The results of the test case testing and automation testing [5]. TDD and ATDD focus on
automation GUI testing on the Swag Labs web with 87 where 36 verifying the state of the system rather than the desired
steps failed, 4 step warning and 44 step passed with an average behavior of the system. Problems with manual testing are time
responses times in the test case are 2,122 minutes. consuming and tedious.
Keywords— Empirical Software Engineering, Software Behavior-driven development (BDD) is an Agile approach
Testing Automation, GUI Testing, Behavior-driven to software development that encourages teams to deliver
Development, Cucumber. software by emphasizing interaction among stakeholders. It
considers business and technical perspectives - and one of the
I. INTRODUCTION main ways to achieve this is by writing test cases in a language
Graphical user interface (GUI) is widely used in software that can be understood by individuals and non-technical
applications. The GUI makes the application easier to use and programmers [6]. Losses and potential challenges of BDD.
user friendly [1]. In modern software applications, around BDD requires high business involvement and collaboration.
50% of the application code is for GUI implementation. One

978-1-7281-9167-6/20/$31.00 ©2020 IEEE

87 on May 28,2021 at 17:28:21 UTC from IEEE Xplore. Restrictions apply.

Authorized licensed use limited to: University of Prince Edward Island. Downloaded
 2020 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS)

BDD works best in an Agile or iterative context, poorly

written tests can lead to higher testing maintenance costs [7].
BDD is a behavior test, a test that focuses on user Fig. 1. BDD Flow Process
acceptance tests, where the User Acceptance Criteria (UAC)
illustrates the treatment of users who will use the application. BDD methodology which exploits such information in
BDD supports script base testing. the tester uses the script base order to semi-automatically generate step definitions and code
for retesting. Cucumber is the most popular BDD framework skeletons from scenarios given in natural language. For this
used in software development [8]. The BDD concept has long purpose, we are making use of UML class diagrams and UML
been known to be built on the concept of natural language sequence diagrams that are proper abstractions of code
processing (NLP). Where automatically extracts information skeletons and step definitions, respectively, from which the
contained in scenarios into step definitions and code skeletons required pieces for the BDD flow can easily be generated. In
using the help of UML class diagrams and sequence diagrams the following, the general idea is briefly illustrated in Figs. 2
[9]. and 3 by means of the telephone scenario[9]

There are many web applications that we can use for

automation GUI testing. The authors choose a web application
that has not been tested. Katalon Studio and the CURA web
are good candidate for this purpose. The Katalon was chosen
because it has complex features and is easy to use for
beginners [10]. Use Case to be tested are logging to Swag
Labs. GUI testing [11] is to analyze the response time of
application web pages.
Fig. 2. Extract class diagrams from scenarios for creating a code sceleton
The results of testing based on BDD test cases in the form
of passed, warning, failed, and the results of responses times
can be analyzed whether it affects the complexity of the web.
The formulation of problem can be determined in this
study as follows:
1. What are the results of the GUI testing automation with
the implementation of Cucumber BDD using Katalon
Studio?
2. How many responses times are spent in BDD
automation GUI testing using Catalone Studio? Fig. 3. Extract sequence diagrams from scenarios for creating step definitions

II. LITERATURE REVIEW D. Semi Automation Information Extraction

A. Related Research BDD contains the concept of NLP that has been explained
previously that by determining class diagrams and UML
TABLE I. RELATED WORK sequence diagrams. From UML it will semi-automatically
extract information from BDD scenarios using natural
Ref Automation GUI Cucumber BDD Response language [9].
Testing Testing Times
[11]   - - -
[12]   - - -
[13]  -   -
[14]   -  -
[15]   -  
[16]  - - - -
Author     

Table I illustrates some of the related research in this

study. [11] [12] [13] [14] [15] [16]. Based on previous
research, the author will conduct research on automation
testing using the BDD method with Cucumber and BDD tools Fig. 4. Noun extraction using PST and WordNet dictionary
namely Katalon Studio based on the NLP concept in its
application. E. Cucumber
B. Automation Testing Cucumber facilitates the BDD process into automation
testing through the Gherkin language and can, therefore, be
Automation testing is the process of using software that is called more as a BDD framework [18].
separate from the software being tested to control the
implementation of the test and the comparison of actual results III. RESEARCH METHODOLOGY
with expected results [17].
Figure 5. shows the steps of this research as follows: (1)
C. Process Flow BDD study of literature, (2) create user stories, (3) create steps on
The typical BDD design process, as currently applied, cucumber, (4) create test suite, (5) analysis test results and (6)
involves the steps illustrated through Figure 1 [10]. analysis of responses time.

88 on May 28,2021 at 17:28:21 UTC from IEEE Xplore. Restrictions apply.

Authorized licensed use limited to: University of Prince Edward Island. Downloaded
 2020 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS)

usecase diagram of the Swag Labs web application. On the

Swag Labs e-commerce website, the users are the buyers who
want to buy some goods
C. Cucumber Deployment
There are a few steps to be done. First step is creating file
feature and second step is creating step definitions.
D. Create Test Suite
Test Suites is a runner which are the collection of test
cases. Every test case will be test and evaluate.
E. Analysis Test Result
At this step, the tester identifies whether the features that
are run are passed (green), failed (red), or warning (yellow).
F. Analysis Response Times
According to [11] a GUI with high complexity, will have
a high response time so that access time is longer. On the other
hand, a simpler GUI will be more quickly accessed. It is
Fig. 5. Research Methodology
possible that a more complex GUI responds faster than a lower
complicated GUI, if using advanced algorithms.
A. Create User Stories
IV. RESULT AND DISCUSSION
User stories are descriptions of system requirements in the
form of natural language that can be easily understood by A. Create User Stories
end-users. User stories describe the types of users, what they Table II illustrates user stories based on features and GUIs
want, and why. Below is a table of user stories that will be on the CURA and Swag Labs web applications.
tested based on stories and User Acceptance Criteria (UAC)
written in English. TABLE II. USER STORIES WEB CURA

B. Usecase diagram No Story Descriptions UAC

1 A001 Login As a user, GIVEN I in Login Page
I want to login to WHEN I input Valid
Cura System, so Username & Password
that I can make an account CURA system
appointment. THEN
1. I Click button Login
& Login is success
2. I Back to screen
Login & show message
"Login failed! Please
ensure the username
and password are
valid.”
2 A002 Make As a user, GIVEN I Make
Appoinment I want to make an appointment Page
appointment WHEN I fill in make
Fig. 6. Usecase Web CURA So I can book appointment data
appointment THEN I click "Book
Appointment"
3 A003 As a user, GIVEN I in
Appointment I want to see Appointment
Confirmation Appointment Confirmation page
Confirmation WHEN I see the detail
So I can see the of my appointment
detail of my information
appointment THEN I can click “Go
to Homepage” button
4 A004 History As a user GIVEN I in History
I want to see page
history menu WHEN I see the history
So I can see of my appointment
history of my THEN I can click “Go
appointment to Homepage” button
5 A005 Logout As a user GIVEN I in CURA
Fig. 7. Usecase Web Swag Labs I want to logout system
So I can leave WHEN I click “Logout”
Figure 6. show usecase diagrams of the CURA web from the website button
application. Users are buyers that want to book medical THEN I go to Home
treatment schedules at CURA Hospital. Figure 7. Show page

89 on May 28,2021 at 17:28:21 UTC from IEEE Xplore. Restrictions apply.

Authorized licensed use limited to: University of Prince Edward Island. Downloaded
 2020 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS)

TABLE III. USER STORIES WEB SWAG LABS

No Story Descriptions UAC
1 B001 Login As a user, I want GIVEN I in Login Page
to login to Web WHEN I input Valid
Swag Labs Username & Password
So that I can buy account Swag Labs
something from THEN
the web. 1. I Click button Login
& Login is success
2. I Back to screen
Login & show message "
Epic sadface: Username
and password do not
match any user in this Fig. 8. Login.feature
service”
2 B002 Select As a user, I want GIVEN I Products Page We use step definitions by the groovy language that
Products to select product WHEN I select the title available at Katalon Studio.
from Web Swag of one product
Labs THEN I click "ADD TO
So that I can add CART"
product to chart.
3 B003 Remove As a user, I want GIVEN I Products Page
Products to remove the WHEN I select the title
product that I of one product
chose AND I click "ADD TO
So that I can CART"
choose another THEN I click
product “REMOVE”
4 B004 As a user, I want GIVEN I Products Page
Checkout to checkout WHEN I select the title
Products product purchased of one product
So that I can THEN
check the proct I click "ADD TO CART"
that i want to buy I go to cart page Fig. 9. Step Definitions.groovy
I click “CHECKOUT”
button
Next step is creating runners. Runners are created in
I fill checkout
information package step definitions but are separated from the
I verify checkout stepdefiniton.groovy file. The file runner uses the groovy
information language.
5 B005 Logout as a user i want to GIVEN I in Swag Labs
logout system
So i can leave WHEN I click “Logout”
from the website button
THEN I go to Login
page

Table III. user stories web swag labs are showing use case
test on login, select products, remove products, checkout, and
logout. The last column of table II and table III are statement
case that follow the patern of BDD

B. Create Feature File, Step Definitions and Runner

User Story A001 is made in the feature login file which
consists of 2 scenarios, the first scenario is login with a valid Fig. 10. Runner File
username and password, the second scenario is logging in with
an invalid username and password. C. Create Test Case and Test Suites
Katalon Studio supports Keywords-Driven testing where
the test case consists of keywords representing user actions on
AUT (Applications Under Test). This allows users with less
experience in programming to easily generate automated tests.
At this time, the writer made a test case to run the previously
made runner.

90 on May 28,2021 at 17:28:21 UTC from IEEE Xplore. Restrictions apply.

Authorized licensed use limited to: University of Prince Edward Island. Downloaded
 2020 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS)

warning and 36 step is red, this means that 41.37% of the

results of the automation testing failed. A total of 6 feature
files created overall were successfully tested with the results
passed, warning and failed.
E. Results Responses Times

Fig. 11. Test Case Login in keywords

Figure 15. CURA Web Response Times Graph

Figure 15. means LC: Login CURA; MA: Make

Appointment; AC: Appointment Confirmation; H: History;
Fig. 12. Test Suite web CURA application LgC: Logout CURA, figure 14 is a graphic display of data
from the automation testing response time following the paper
D. Test Case Result [15]. The results of the Swag Labs web automation test are
GUI test case test results on the CURA web application. based on features described in the previous user stories.
Response time results in minutes (m). The graph shows that
the highest response time on the make appointment function
test or feature is 4.57 minutes and the test result with the
lowest response time on the logout feature is 0.35 minutes.
Below this is the calculation of the average test case response
time results on the CURA web application.
ℎ
ℎ

1.0 4.57 1.52 2.16 0.35

5
Fig. 13. CURA Web Test Case Results 9.6
From the 213 steps made there are 210 green testing steps 5
(figure 13). This means that 98.59% of the results of the 1.92 % &
automation testing are passed. Three step automation testing
is orange, this means that 1.40% of the results of the Based on the average calculation results, the response time
automation testing are warning. A total of 6 feature files results in the CURA web application test case, obtained an
created overall were successfully tested with passed and average of 1.92 minutes. In testing by running an existing test
warning results. case, the response time in one test case varies if it is run several
times [11]. Based on these data the authors know the response
time on the CURA web application fluctuates. Each process
of testing features per test case is based on the number of steps
in the gherkin in the feature file. The more steps the gherkin
makes, the more response time are generated.

Fig. 14. Swag Labs web Test Case Results

Out of the 87 steps made there are 44 green test steps

(Figure 14). This means that 50, 57% of the automation testing
results are passed, the 4 step automation testing is orange, this
means that 4.59% of the automation testing results are

91 on May 28,2021 at 17:28:21 UTC from IEEE Xplore. Restrictions apply.

Authorized licensed use limited to: University of Prince Edward Island. Downloaded
 2020 International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS)

2. The complexity of a page partially affects the response

times. May be other factors beside complexity of the
appearance that effect respons times

Further work on automation functional testing and web

APIs testing are the next interisting research agenda.
REFERENCES
[1] L. Wang, “GUI test automation for Qt application,” 2015.
[2] T. D. Hellmann, “Automated GUI Testing for Agile Development
Environments,” 2015, doi: 10.11575/PRISM/25075.
Fig. 16. Swag Lab Web Response Times Graph [3] D. Kumar and K. K. Mishra, “The Impacts of Test Automation on
Software’s Cost, Quality and Time to Market,” Procedia Comput. Sci.,
On Figure 16, LSL: Login Swag Labs; SP: Select Product; vol. 79, pp. 8–15, 2016, doi: 10.1016/j.procs.2016.03.003.
RP: Remove Product; C: Checkout; LgSL: Logout Swag [4] M. Rahman and J. Gao, “A reusable automated acceptance testing
architecture for microservices in behavior-driven development,” Proc.
Labs. Figure 16 is a graph of data from the results of - 9th IEEE Int. Symp. Serv. Syst. Eng. IEEE SOSE 2015, vol. 30, pp.
automation testing response time. The results of the 321–325, 2015, doi: 10.1109/SOSE.2015.55.
automation testing are based on features described in the [5] R. M. Sharma, “Quantitative Analysis of Automation and Manual
previous user stories. Response time results in minutes (m). Testing,” vol. 4, no. 1, pp. 252–257, 2014.
The graph shows that the highest response time results on [6] L. Raed and M. Besse, “Use BDD To Make Your Software
testing the checkout function or feature is 4.15 minutes and Development Project More Successful,” vol. 1, no. 617, pp. 1–9, 2015.
the test results with the lowest response time on the logout [7] J. Ferguson and D. North, “Behavior Driven Development for the
feature are 1.3 minutes. Below this is the calculation of the whole software lifecycle,” 2014.
average test case response time results on the Swag Labs web [8] Adoriasoft, “TDD vs BDD or Why BDD is the New Black,” 2016.
application. [9] M. Soeken, R. Wille, and R. Drechsler, “Assisted behavior driven
development using natural language processing,” Lect. Notes Comput.
'( ) Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes
ℎ Bioinformatics), vol. 7304 LNCS, pp. 269–287, 2012, doi:
ℎ 10.1007/978-3-642-30561-0_19.
[10] S. T. Help, “Selenium Vs Katalon Studio: How To Simplify Selenium
1.34 1.49 2.33 4.15 1.3 Tests In Katalon Studio,” 2020. [Online]. Available:
'( )
5 https://www.softwaretestinghelp.com/selenium-vs-katalon-studio/.
[Accessed: 16-Apr-2020].
10.61 [11] M. M. Muhtadi, M. D. Friyadi, and A. Rahmani, “Analisis GUI Testing
'( ) pada Aplikasi E-Commerce menggunakan Katalon,” Pros. Ind. Res.
5
Work. Natl. Semin., vol. 10, no. 1, pp. 1387–1393, 2019.
'( ) 2.122 % & [12] H. Herlinda, D. Katarina, and E. W. Ambarsari, M.Kom., “Automation
Testing Tool dalam Pengujian Aplikasi Belajar Tajwid pada Platform
Android,” STRING (Satuan Tulisan Ris. dan Inov. Teknol., vol. 4, no.
Based on the average calculation results, the response time 2, p. 205, 2019, doi: 10.30998/string.v4i2.5285.
results on the Swag Labs web application, obtained an average [13] H. Naik, “Behavior Driven Development: An Effective Technical
of 2.122 minutes. In testing by running an existing test case, Practice to Develop Good Software,” Int. J. Comput. Appl., vol. 149,
the response time in one test case varies if it is run several no. 5, pp. 23–27, 2016, doi: 10.5120/ijca2016911400.
times [11], but the results are not much different from those [14] V. Garousi and E. Yildirim, “Introducing automated GUI testing and
listed in graph (figure 15,16). Based on these data, we find observing its benefits: An industrial case study in the context of law-
response time in the Swag Labs web application are increases. practice management software,” Proc. - 2018 IEEE 11th Int. Conf.
Softw. Testing, Verif. Valid. Work. ICSTW 2018, no. July, pp. 138–145,
V. CONCLUSION AND FUTURE WORK 2018, doi: 10.1109/ICSTW.2018.00042.
[15] N. Cruz Quental, C. de Albuquerque Siebra, J. Peixoto Quintino, F.
The conclusions of this study are as follows : Florentin, F. Queda Bueno da Silva, and A. L. de Medeiros Santos,
“Automating GUI Response Time Measurements in Mobile and Web
1. Automation GUI testing is needed to determine the Applications,” pp. 35–41, 2019, doi: 10.1109/ast.2019.00011.
quality of the appearance of an application. One thing [16] R. B. Bahaweres, K. Zawawi, D. Khairani, and N. Hakiem, “Software
that can be measured is the response time of the GUI. Testing With Genetic Algorithm,” no. September, pp. 19–21, 2017.
The results of the test cases on the CURA web [17] M. A. Umar and Z. Chen, “A Study of Automated Software Testing :
application with the Katalon are passed and warnings. Automation Tools and Frameworks,” no. December, 2019.
The test case results on the Swag Labs web application [18] S. Rose, M. Wynne, and A. Hellesoy, The Cucumber For Java Book.
2015.
with a Catalyst are failed, warning and passed. With
an average response time on the CURA web test case
1.92 minutes and an average response time on the
Swag Labs web test case 2.122 minutes.

92 on May 28,2021 at 17:28:21 UTC from IEEE Xplore. Restrictions apply.

Authorized licensed use limited to: University of Prince Edward Island. Downloaded