White Paper
On
Performance Testing Process
By
Applabs Technologies
Applabs House,
Plot No. 564-A-26-III, Road No. 92,
Jubilee Hills, Hyderabad – 500 033
Load, Stress, Capacity Testing Guidelines
TABLE OF CONTENTS
Introduction…………………………………………………………………….…. 2
Tools Used………………………………………………………………………... 2
General Summary………………………………………………………………… 2
Definitions
Keys to Accuracy
Scalability Issues
Process Steps………………………………………………………………………. 4
Understanding the Nature of the Load
Determine User Session Variables
Determine Range & Distribution of Values
Estimating Target Load Levels
Load Test Design
Execution……………………………………………………………………………6
Reporting……………………………………………………………………………7
Load, Stress &
Capacity Testing Page 2 of 9
Introduction
Performance testing is conducted to assess how the site/product holds up under different
loads and provide the customer with extensive data to help scale the site/product up to
meet higher loads.
Market-leading tools such as Rational's Test RealTime, Compuware's QA Load, Mercury
Interactive's LoadRunner, and Radview's Webload are used to assess how our clients'
applications perform under varying loads.
Tools Used
(1) Mercury Interactive’s LoadRunner®
(2) Radview’s Webload®
General Summary
(1) Definitions
a. Testing Types
i. Load: Testing an application against a requested number of
users. The objective is to determine whether the site can
sustain this requested number of users with acceptable response
times.
ii. Stress: Load testing over an extended period of time. The
objective is to validate an application’s stability and reliability.
iii. Capacity: Testing to determine the maximum number of
concurrent users an application can manage. The objective is
to benchmark the maximum loads of concurrent users a site can
sustain before experiencing system failure.
b. Reporting Terms:
i. Load size: The number of concurrent Virtual Clients trying to
access the site.
ii. Throughput: The average number of bytes per second
transmitted from the ABT (Application being tested) to the
Virtual Clients running this Agenda during the last reporting
interval
iii. Round Time: It is the average time it took the virtual clients to
finish one complete iteration of the agenda during the last
reporting interval.
iv. Transaction Time: The time it takes to complete a successful
HTTP request, in seconds. (Each request for each gif, jpeg,
html file, etc. is a single transaction.) The time of a transaction
Load, Stress &
Capacity Testing Page 3 of 9
is the sum of the Connect Time, Send Time, Response Time,
and Process Time
v. Connect Time: The Time it takes for a Virtual client to
connect to the Application Being Tested.
vi. Send Time: The time it takes the Virtual Clients to write an
HTTP request to the ABT (Application being tested), in
seconds.
vii. Response Time: The time it takes the ABT (Application being
tested) to send the object of an HTTP request back to a Virtual
Client, in seconds. In other words, the time from the end of the
HTTP request until the Virtual Client has received the
complete item it requested.
viii. Process Time: The time it takes WebLoad to parse an HTTP
response from the ABT (Application being tested) and then
populate the document-object model (the DOM), in seconds.
ix. Wait Time (Average Latency): The time it takes from when a
request is sent until the first byte is received.
x. Receive Time: The elapsed time between receiving the first
byte and the last byte.
(2) Keys to Accuracy
a. Recording ability against a real client application
b. Capturing protocol-level communication between client application
and rest of system
c. Providing flexibility and ability to define user behavior
d. Verifying that all requested content returns to the browser to ensure a
successful transaction
e. Showing detailed performance results that can easily be understood
and analyzed to quickly pinpoint the root cause of problems
f. Measuring end-to-end response time
g. Using real-life data
h. Synchronizing virtual user to generate peak loads
i. Monitoring different tiers of the system with minimal intrusion
(3) Scalability
a. Generating the maximum number of virtual users that can be run on a
single machine before exceeding the machine’s capacity
b. Generating the maximum number of hits per second against a Web
server
c. Managing thousands of virtual users
d. Increasing the number of virtual users in a controlled fashion.
Load, Stress &
Capacity Testing Page 4 of 9
Process Steps
Diagram 1: Explained in Detail Below
Determine User
Understand the Nature Determine Range &
Session
of the Load Distribution of values
Variables
Estimate Target Develop Base Create Load-testing
Load Levels Test Scripts Scripts combining scripts
in a logical
Determine frequency Execute Test
in which scripts should Scripts
run
Monitor Collect & Provide Reports and
Execution Examine Reports Suggestions to Client
(1) Understanding the Nature of the Load via contents of Log Files:
a. Visitors’ IP addresses
b. Date & time each page visited
c. Size of the file or page accessed
d. Code indicating access successful or failed
e. Visitor's browser & OS
(2) Determine User Session Variables / Page Request Distribution
a. Length of the session (measured in pages)
b. Duration of the session (measured in minutes and seconds)
c. Type of pages that were visited during the session
i. Home page
ii. Product info page
iii. Credit Card Info page
d. Varying User Activities from page ‘types’.
i. Uploading information
ii. Downloading information
iii. Purchasing
Load, Stress &
Capacity Testing Page 5 of 9
e. Page Rendering Issues
i. Dynamic pages (including time sensitive Auction/trade pages)
ii. Static pages
f. New users vs. Existing users
(3) Determine Range & Distribution of values for these variables
a. Average (i.e. 4pg views per session)
b. Standard deviation
c. Discrete distribution
(4) Estimating Target Load Levels
a. Overall traffic growth (historic data vs. sales/marketing expectations)
b. Peak Load Level (day of week, time of day, or after specific event)
c. How long the peak load level is expected to last
d. Options on getting these numbers
i. Concurrent users
ii. Page views
iii. User session per unit of time
(5) Load Test Design
a. Test Objective
SAMPLE: The objective of this load test is to determine if the Web
site, as currently configured, will be able to handle the 12,000-
sessions/hr-peak load level anticipated for the coming holiday season.
If the system fails to scale as anticipated, the results will be analyzed
to identify the bottlenecks, and the test will be run again after the
suspected bottlenecks have been addressed.
b. Pass / Fail Criteria
SAMPLE: The load test will be considered a success if the Web site
will handle the target load of 12,000 sessions/hr while maintaining the
average page response times defined below. The page response time
will be measured over T1 lines and will represent the elapsed time
between a page request and the time the last byte is received:
c. Script Labels and Distribution
i. Type of scripts – (keeping Page Request Distribution as the
main concept here)
ii. Number of Scripts (usually 10 -15)
iii. Naming of Scripts
SAMPLE:
Browse NoBuy: Home >> Product Information(1) >> Product
Information(2) >> Exit
View Calendar: Home >>Event Calendars Page >> Next 30
days >> back >>Exit
iv. Calculate the target page distribution based on the types of
pages hit by each script, and the relative frequency with which
each script will be executed.
Load, Stress &
Capacity Testing Page 6 of 9
Execution
(1) Combine Scripts to create load-testing Scenario
a. Scripts Executed
b. Percentages in which those scripts are run
c. Description of how load will be ramped up
Sample Table 1: Script Table for static eCommerce site
Script Name Basic Description
Home Only Home Æ Exit
BrowseNoBuy Home Æ Product Information(1) Æ Product
Information(2) Æ Exit
BrowseAndBuy Home Æ Product Information(1) Æ Product
Information(2) Æ Order Entry Æ Credit Card
Validation Æ Order Confirmation Æ Exit
Sample Table 2: Spreadsheet (script table vs. user frequency)
Script Name Relative Home Product Order Credit Order
Frequency* Info Entry Card Confirmation
Home Only 55% 1
BrowseNoBuy 34% 1 2
BrowseAndBuy 11% 1 2 1 1 1
Total
Relative pg 1 .9 .11 .11 .11 2.23
Views
Resulting pg 45% 40% 5% 5% 5% 100%
view
distribution
Target pg view 48% 37% 6% 5% 4% 100%
distribution
Difference** 3% -3% 1% 0% 1%
* Obtained from historic log files or created from projected
user patterns
** Target Difference should be < 5% for non-transaction pages
and <2% for transaction pages.
Load, Stress &
Capacity Testing Page 7 of 9
Reporting
(1) Graphs: Utilize a suite of graphs from either LoadRunner or WebLoad to
provide the client with easy to use, relevant information.
Sample Graph1: Transaction Summary
This performance graph displays the number of transactions that passed, failed, aborted or ended with errors.
For example, these results show the Submit_Search business process to passed all its transactions at a rate of
approximately 96%.
Sample Graph 2: Throughput
This web graph displays the amount of throughput (in bytes) on the Web server during
load testing. This graph helps testers evaluate the amount of load X users generate in
terms of server throughput. For example, this graph reveals a total throughput of over 7
million bytes per second.
(2) Additional Report Info Available during & after Scenario Execution
The below are additional reporting metrics available to those item
mentioned in ‘definitions’ section.
a. Rendezvous – Indicates when and how virtual users are released at
each point.
Load, Stress &
Capacity Testing Page 8 of 9
b. Transactions/sec (passed) – The number of completed, successful
transactions performed per second. (see sample table 2)
c. Transactions/sec (failed) – The number of incomplete, failed
transactions per second. (see sample table 2)
d. Percentile – Analyzes percentage of transactions that were performed
within a given time range. (see sample table 2)
e. Performance Under Load – Transaction times relative to the number
of virtual users running at any given point during the scenario.
f. Transaction Performance – Average time taken to perform
transactions during each second of scenario.
g. Transaction Performance Summary – Minimum, maximum, and
average performance times for all transactions in scenario.
h. Transaction Performance by Virtual User – Time taken by an
individual virtual user to perform transactions during the scenario.
i. Transaction Distribution – The distribution of the time taken to
perform a transaction.
j. Connections per Second – Shows the number of connections made to
the Web server by virtual users during each second of the scenario run.
Load, Stress &
Capacity Testing Page 9 of 9
