Architecture  Development  DevOps  Test Automation  Downloads

About Me

Topics


search..

Recen
t Posts
gRPC –

JMeter – Continuous Performance Reactor

Testing – JMeter + ANT + Jenkins Integratio

n
Integration – Part 2 gRPC –
28 Comments / Articles, Best Practices, CI / CD / DevOps, Framework,
Performa
Jenkins, JMeter / By vIns / February 14, 2016
nce
Goal: Comparis

on With
To create a Continuous Delivery Pipeline which includes Performance
Testing process to detect any performance related issues as early as REST

possible. NATS –
Server 
Usually the full scale Performance Test will be done in the Staging/Pre- Clustering
Production environment which could be identical to your Production RSocket –
environment. Code push to Staging happens after thorough QA
Real Time
functional/regression verification is done. So even if QA certifies the build,
Applicatio
there is a chance that code might not work as expected at certain load which
n
could be found only in Staging environment. This performance issue requires
code change -> build -> QA verification once again which could postpone Developm

your Production push! ent With

RSocket +
So, Can we fit the performance testing into the existing continuous
development & testing process to detect the performance issues as early as Spring
possible? Boot +

Yes! We absolutely can! The idea here is to run the performance test in the WebSocke

lower environments [Dev/QA] (we might have to adjust the load t

accordingly) & to compare the results against some baseline metrics. Note NATS –
that this is NOT going to replace the full scale performance test which we Cloud
will do in the Staging environment.
Native
Note: If you have not read this post – I would advise you to read the Part 1 of Messaging
this post which talks about JMeter + Ant integration. Server
 

Jenkins Install:
Please check this link for the detailed steps on installing Jenkins on various Seleniu

OS. m

WebDri
 
ver -
Create Jenkins Job: How To

Create a simple free style project. Test

REST

API

Introdu

cing
 Build step should be ‘Invoke Ant‘ PDFUtil
-
[We will use Default Ant assuming the slave machine has Ant installed
Compa
already. if not, Let jenkins install ANT automatically.]
re two
PDF

files

textuall
y or

Visually

JMeter

- How

We will invoke the target ‘all‘ as the target ‘all’ executes below tasks To Run
we wanted to execute. Multipl
clean e

show-test-properties Thread

run Groups

in
generate-report
Multipl
generate-charts
e Test

our test uses below properties. We will be passing the values for Environ

those properties from Jenkins to JMeter through Ant as given above. ments
threadgroup.count
Seleniu
threadgroup.rampup
m
threadgroup.duration WebDri

ver -
Our project folder is present under C:/workspace/CPT. So set the
Design
custom workspace accordingly.
Pattern

s in

Test
 Autom
ation -

Factory
Pattern

Kafka
Stream
s-

Real-
Time
Our tests creates the results under result folder. We need to archive
Data
the files we need as given below. To archive all files under result, use
Proces
result\*.*
sing

Note: Jenkins will look for the files under the workspace. So set the Using
path relative to the workspace. Spring
Boot

JMeter

  - Real
Time
Invoke JMeter Test from Jenkins: Results

If all the above steps have been done correctly, Clicking on ‘Build’ will -
do to run the jmeter test. We can see a nice output as shown below. It InfluxD
runs the test for 3 users with 3 seconds as rampup period for 100 B&
seconds.
Grafan
a - Part
1-

Basic
Setup

JMeter 
-
 Distribu
ted

Load
Testing
using

Docker

JMeter
- How
Once the test run is complete, Jenkins archives all the files under
To Test
result folder. It includes the HTML file + PNG charts we have created.
It archive the results for each and every run & they get stored in the REST

server where jenkins is running. API /

MicroS
ervices

JMeter
-
Propert

y File
Reader
Instead of using hardcoded test properties, we can let our team -A
member enter these values in Jenkins UI. To make it work, we need to custom
make this job parameterized & Let’s create 3 parameters as our test
config
expects 3 parameters to run. We can also have default values for
elemen
those parameters.
t

Seleniu
m

WebDri
ver -
How To

Run
 Autom
ated
Tests

Inside
A
Docker

Contain
er -
Part 1

Categ
ories
Architectu
re (43)
Arquillian
(9)
Modify the Build -> Properties section to use the Jenkins parameters
Articles
we have created.  This step is required because Ant expects values for
(174)
these variables ‘threadgroup.count’..etc.  [You can also avoid this step
AWS /
by creating the jenkins parameters like this : Instead of
USER_COUNT,  We can have create threadgroup.count as Jenkins Cloud (17)

Parameter name. ]. Since Jenkins parameters and Ant parameters are AWS (4)
different , we should map as shown below. Best

Practices
(75)

 CI / CD /
DevOps
(51)
Data
Stream /

Event
Stream
(20)
Database
(2)

That’s it – Now, We should be able to pass the parameter’s values Design

required for the JMeter test directly from Jenkins UI. Enter some Pattern
values and click on Build. Console will show the new values passed to (37)
the test and JMeter will run the test accordingly. Below output shows
Architectu
the JMeter invokes 5 threads.
ral Design
Pattern
(22)

Factory
Pattern (1)
Kubernete
s Design
Pattern

  (17)

Strategy
Pattern (1)
Distribute
d Load
Test (9)
Docker

(23)
 ElasticSea
rch (2)
EMail

Validation
(1)
Framewor
k (100)
Functional
Test

Automatio
n (83)
Puppeteer
  (1)

Jenkins-Performance Plugin: QTP (10)

Selenium
Jenkins has a plugin for JMeter to parse the result files, create an aggregate (76)
report, create charts and to compare the current result with previous results
Extend
etc.
WebDriver
You can find more details on the plugin here. 
(11)
[Note that: This plugin parses the result file only if the results are stored in Ocular (2)
the XML format. If you prefer to use CSV format, It will not work]
Page
Once the plugin is installed, for the Jenkins Job, we can find ‘Publish
Object
Performance test result report‘ under Post build actions.  In the Report files
section, give the relative path (to the workspace) of the output jtl file.  Design (17)
Report (8)
Selenium
Grid (10)
TestNG (7)

gRPC (9)
Java (50)

Guice (2)
When we run our tests couple of times, we can see Jenkins starts Reactor
comparing the result of the performance test with previous test and (25)
shows the nice summary output. [You see a lot of 0 as response
Jenkins
times in the below summary because i use a debug sampler for this
(17)
demo which does not take time to execute. Response time chart
Kafka (9)
shows a starighline because they are always 0 in my tests.]
Kubernete

s (6)
Maven (7)
messagin
g (3)
MicroServi
To get more details on each sampler, click on the sampler. Jenkins will
ce (56)
provide the response times charts for each and every sampler as
shown below. It also shows the data in the tabular format along with Monitorin

HTTP response codes which is nice!!. g (13)

FileBeat
(1)
Grafana
(5)

InfluxDB
(7)
Kibana (2)
Multi
Factor
Authentic

ation (2)
nats (2)
Performa
nce
Testing

(44)
EMailing the Results: Extend
JMeter (5)
Jenkins has a nice plugin for Emailing the results. Please check this link for
JMeter
more details.
(43)
Once the plugin is installed, You can find a post build action ‘Editable Email
Workload
Notification‘
Model (2)
Under triggers of the Editable Email Notification, We will be setting Little's
the trigger as ‘Always’ [Always -> whether test fails or not. You can
Law (1)
also change the trigger to send the mail only if the test passes etc]
Web
Once JMeter runs the test, the post build actions are getting executed Scraping
consecutively in the order they are present in Jenkins Job/Project
(1)
config.  So, When this ‘post build action’ gets executed, the result files
Protocol
would have been already created under the result folder.  We might
want to include the HTML file in the Email body and attach the Buffers

images. [If you do not want HTML in the Email body or you use TXT (10)
format, attach the HTML file as well as attachments along with Reactive
images] Program

Below screenshot will give an idea of the basic config to send the ming (28)
mail. [I assume you have already setup email server configuration or Redis (6)
your Jenkins administrator will do that] rsocket (4)
Slack (2)
SMS (1)
Spring

(48)
Spring
Boot (41)
Lets run our test one more time to see if Jenkins can send out the
Spring
results for us after running the test.
WebFlux
And..Yes….I got the email as shown here!!! (41)

Udemy

Courses
 (4)
Utility (20)

WebSocke
t (1)

All the charts files are also attached in the email received.

Summary:
We did a nice job by integrating JMeter with Ant and Jenkins. Thus
Continuous Performance Testing process setup is implemented.   We are
also able to run the test as and when we want by a simple click in Jenkins.
While you are concentrating on other tasks, Jenkins takes care of running the
test, creating the results and sending out the results for you!! It also reduces
the dependency on Performance Testers. YES..!! Anyone can run the test
now. You have to just the share the link for the Jenkins job you have created.

We can also make this job integrated with ‘development deployments’ jobs
of Jenkins – That is…whenever a code is pushed to the given test
environment,  this job gets executed automatically without any manual
intervention. Any functional test and performance test can be done as early

as possible to detect any issues upfront!!
  

Grafana Implementation for Real Time

results:
The JMeter-Jenkins integration, emailing the results are great!! But If you had
noticed, to get the results, we need to wait for the test to finish. Would it
not be awesome to see the results while Jenkins is running the test!!??

If you have a long running test like mine and you are curious of seeing the
current results while Jenkins is running the test – please see one of my
favorite posts on getting real time results.

Happy Testing & Subscribe

Share This:

 JMeter – Continuous Performance QTP/UFT – How to Pass Test 

Testing – JMeter + ANT + Jenkins Parameters from Jenkins
Integration – Part 1

28 thoughts on “JMeter – Continuous Performance

Testing – JMeter + ANT + Jenkins Integration –
Part 2”

vvs
February 29, 2016 at 11:26 AM
Great Post !!
May I know if we can integrate this with the JMeter plugins thread group. So that I can pass the 
parameters like RampUp, Ramp Down, Hold For ..etc
A d l ti d th t t d t il lik th h t 90th %il i th HTML t
And also you mentioned that we can get details like throughput 90th %ile in the HTML report
generated by ANT task I tried but failed, it will be great if you can eloborate the above two points
Thanks in advance.
Reply

vIns
February 29, 2016 at 3:24 PM
Thanks.
Yes, You can use the jmeter-plugins thread groups as well. Ensure that you parameterize
correctly and place the jars in the slave machine as well.
Regarding the HTML report, you can update the XSLT file to add any additional information you
want. But when i mean by ‘update’, you need to include the logic to calculate the information
yourself. So, it might not be very easy in some cases.
If I find time, i will try to do that as a separate post.
Reply

vvs
March 2, 2016 at 12:51 PM
Thanks, that would be useful
Reply

Dean
June 13, 2016 at 2:43 PM
Do you have issues with Ant failing the Jmeter build, with no report?
I have, and expect it’s a java configuration, but would like to know more about how you configured your
java/ant/maven heap settings.
Reply


vIns
 June 25, 2016 at 3:20 PM
Do you mean Ant does not execute JMeter task? I saw in StackOverFlow that there is some
compatibility issue with Ant & Java versions. Please try to use latest versions and see If it fixes
the issue you are facing.
Reply

Prem Prakash
July 30, 2016 at 3:01 PM
Excellent and good one .
I am wondering what extra i would need to configure if i want to run more than one script .
Do i need to add all the jmeter script under the test ? IF yes how does ant understand which jmeter
would run if i want to trigger some specifci script only .
Reply

vIns
August 1, 2016 at 6:27 PM
Use testplans as shown here instead of test plan.
http://www.ibm.com/developerworks/library/j-ap04088/index.htm
Reply

Kevin
September 21, 2016 at 6:20 PM
Hello, Thanks for the very informative articles.
I was wonder how much value has performance testing CI provided for you in practice? Did you catch
many significant performance issues early on?
Reply

vIns
vIns
September 21, 2016 at 7:45 PM
Glad that you found this informative.
Yes, This process helped us from injecting 28 performance issues in PROD this year.
Reply

Kevin
September 21, 2016 at 8:24 PM
That sounds good!
Did you do full scale performance testing in CI or slimmed down test with limited user
and limited scripts?
How much maintenance work on updating JMeter scripts what required as changes were
made to the system?
Reply

vIns
September 22, 2016 at 6:50 PM
We get frequent builds. Even before doing the full scale perf test we do an initial
test with limited users & compare with the baseline. Mostly these are all APIs –
maintenance is very easy. Even for functional testing (not for all functionalities of
the application – focus is to test the server side logic), we use JMeter. These perf
test scripts are sub set of those regression scripts.
Reply

Kevin
September 26, 2016 at 1:47 PM
Thank you!
I see a lot of article online regarding the technicalities of how to
implement continuous performance test but not much on how successful
its been on projects in various case studies.

Vvxxsz
December 14, 2016 at 7:28 AM
Found the issue – I was setting same variables both from Jenkins and build.properties.
After I’ve removed build.properties and set up all variables in Jenkins only it works fine.
Thanks once again for your contribution – you are giving amazing tutorials and out-of-box solutions.
Reply

vIns
December 14, 2016 at 2:43 PM
Glad that it worked.
Reply

TK46
January 17, 2017 at 6:22 PM
Hi,
We have designed UI tests using Selenium webdriver in JMeter. We wanted to run these UI tests in
Jenkins without using headless approach. Do we have any references or steps to run JMeter UI tests in
Jenkins?
Reply

vIns
January 17, 2017 at 6:49 PM
Is your aim to do functional testing using JMeter + WebDriver? If the test is working fine in your

local and all the dependencies are present in remote machine(like java, webdriver, jmeter ect),
there should not be any issues in running using Jenkins. you do not need Ant/Maven. Just this
 y g g y
command (windows batch command/shell script in jenkins) – “jmeter -n -t test.jmx” will do. Do
you face any specific issue in running the test using Jenkins?
Reply

yan
April 10, 2017 at 2:19 PM
Hi,vlus
while executing generate-chart target on jenkins,there is exception throwing,It would be great if you
have time to help me figure out why
] Exception in thread “main” java.lang.UnsupportedOperationException: Seems you’ve requested
aggregate mode for graph that don’t support it. We apologize…
[java] at
kg.apc.jmeter.graphs.AbstractGraphPanelVisualizer.switchModel(AbstractGraphPanelVisualizer.java:363)
Reply

vIns
April 10, 2017 at 6:27 PM
Yan, All the reports can not be generated in the aggregate mode. So, The exception is normal
for certain reports. If you want to disable it, you can remove the “–aggregate-rows” and the next
“yes” for the “generate-graph” target in the build.xml file.
Use this latest sample project – https://github.com/vinsguru/jmeter-ant
Reply

yan
April 11, 2017 at 7:10 AM
vlns,Thank you very much for your quick reply,issue has been fixed after remove the “–aggregate-rows”
and the next “yes” 
the latest project looks more excellent ,thanks again for your contribution
Reply

yan
April 17, 2017 at 2:42 PM
Hi,vlns
I use testplans to run several scripts once time in build.xml,but there is a problem for me,is it possible to
generate jmeter results for all scripts into one test.jtl
Looking forward to your reply
Thanks
Reply

vIns
April 17, 2017 at 5:07 PM
Check this JMeter plugin – https://jmeter-plugins.org/wiki/MergeResults/. It merges the results.
If you want 1 combined result file, do not added any prefix label. If you are merging the results to
compare the results, then you would need the prefix label.
Reply

Aaron
April 25, 2017 at 1:56 PM
This is an excellent post! I have question though, how is it different from running manual Jmeter GUI
mode? Does it run on Jenkins server or the actual application servers?
Thanks!
Reply

vIns
April 25, 2017 at 2:37 PM
By using Jenkins, you can configure the performance testing job as part of your CD pipeline.
Jenkins will run the JMeter performance test using one of the slaves machines it has been
 configured to run. Not on the actual appn server.
Reply

nisar
March 9, 2018 at 10:48 AM
I Have Already set Environment Variables But Geetin Errors
Started by user nisar
Building in workspace C:\Users\nisarahs\Documents\jmeter-ant-master
[jmeter-ant-master] $ cmd.exe /C “ant.bat -Dthreadgroup.duration=180 -Dthreadgroup.rampup=3 -
Dthreadgroup.count=3 all && exit %%ERRORLEVEL%%”
Buildfile: C:\Users\nisarahs\Documents\jmeter-ant-master\build.xml
clean:
[delete] Deleting directory C:\Users\nisarahs\Documents\jmeter-ant-master\result
[delete] Deleting directory C:\Users\nisarahs\Documents\jmeter-ant-master\log
[delete] Deleting directory C:\Users\nisarahs\Documents\jmeter-ant-master\output
show-test-properties:
[echo] JMETER_HOME : C:\Users\nisarahs\Downloads\apache-jmeter-4.0
[echo] TEST_PATH : C:\Users\nisarahs\Documents\jmeter-ant-master/test/Apsrtc.jmx
[echo] THREADGROUP_COUNT : 3
[echo] THREADGROUP_RAMPUP : 3
[echo] THREADGROUP_DURATION : 180
run:
[mkdir] Created dir: C:\Users\nisarahs\Documents\jmeter-ant-master\result
[mkdir] Created dir: C:\Users\nisarahs\Documents\jmeter-ant-master\log
[jmeter] Executing test plan: C:\Users\nisarahs\Documents\jmeter-ant-master\test\Apsrtc.jmx ==>
C:\Users\nisarahs\Documents\jmeter-ant-master\result\result.jtl
generate-report:
[mkdir] Created dir: C:\Users\nisarahs\Documents\jmeter-ant-master\output
[java] An error occurred: Cannot read test results file : C:\Users\nisarahs\Documents\jmeter-ant-
master\result\result.jtl
[java] Java Result: 1 
generate-chart:
[ ] ‘C \U \ i h \D l d \ h j /bi /JM Pl i CMD’ i i d
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as
an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as
an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as
an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as
an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as
an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as
an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
[exec] ‘C:\Users\nisarahs\Downloads\apache-jmeter-4.0/bin/JMeterPluginsCMD’ is not recognized as
an internal or external command,
[exec] operable program or batch file.
[exec] Result: 1
all:
BUILD SUCCESSFUL
Total time: 1 second
Archiving artifacts
ERROR: No artifacts found that match the file pattern “log*.log,result*.“. Configuration error?
ERROR: ‘log*.log’ doesn’t match anything, but ‘.log’ does. Perhaps that’s what you mean? 
Build step ‘Archive the artifacts’ changed build result to FAILURE
Fi i h d FAILURE
Finished: FAILURE
Reply

nisar
March 9, 2018 at 12:11 PM
did we need to keep ant and ant master inside jenkins Folder ?
Reply

vIns
March 9, 2018 at 5:39 PM
Hi Nisar,
You seem to be using the latest version JMeter 4.0. The articles was created for JMeter version
3.0. Soemthing might have been changed. I need to take a look.
Reply

sowjanya
January 15, 2019 at 7:13 PM
Hi,
It was a very nice article Im trying to implement this one challenge im facing Im using load genearators
(Master slave concept) how can I leverage this issue with using jenkins ?
Reply

vIns
February 11, 2019 at 5:43 PM
The concept is still same. You always connect the master to jenkins. Not slaves. Master is
responsible for connecting to slaves and running the test.
Reply

Leave a Reply
Your email address will not be published. Required fields are marked *
Comment

Name *

Email *

Website

Save my name, email, and website in this browser for the next
time I comment.
Notify me of follow-up comments by email.

Notify me of new posts by email.

Post Comment

This site uses Akismet to reduce spam. Learn how your comment data is
processed.