Software Testing

Rajib Mall
CSE Department
IIT KHARAGPUR

1
 Design of Test Cases
• Systematic approaches are required to design an
effective test suite:
–Each test case in the suite should target different
faults.

2
 Testing Strategy
• Test Strategy primarily addresses:
– Which types of tests to deploy?

– How much effort to devote to which type of testing?

• Black-box: Usage–based testing (based on customers’ actual
usage pattern)

• White-box testing can be guided by black box testing results

3
 Quiz: How would you use this information
40% for planning test effort?

# of Bugs
Considering
Detected 25% Past Bug
Detection
Data…
15%
10%
10%

Reviews Unit Integration System customer

test test test reported

4
 Quiz: How would you use
this for planning unit test
effort?
Problems 50% Consider Past
Detected
Bug Detection
30%
Data…
10% 10%

test test test customer

Technique 1 Technique 2 Technique 3 reported

5
 Quiz: How would you use
Distribution of
this for planning Release 2 testing?
Error Prone
Number of Modules
bugs customer
Detected
reported bugs
for Release 1

M1 M2 M3 M4 M5 M6

Defect clustering: A few modules usually contain most defects… 6

Unit Testing

7
 When and Why of Unit Testing?
• Unit testing carried out:
• After coding of a unit is complete and it compiles
successfully.

• Unit testing reduces debugging effort

substantially.
8
 Why unit test?
• Without unit test:
– Errors become
difficult to track
down.
– Debugging cost
increases
substantially… Failure
• Testing of individual methods, modules, classes, or
components in isolation: Unit Testing
– Carried out before integrating with other parts of the
software being developed. Driver
• Following support required for Unit testing: Unit
– Driver Stub

• Simulates the behavior of a function that calls and supplies necessary data
to the function being tested.
– Stub
• Simulates the behavior of a function that has not yet been written.

10
 Unit Testing

PROCEDURE
STUB UNDER TEST DRIVER
CALL

Access To Nonlocal Variables

11
 Quiz
• Unit testing can be considered as which one of the
following types of activities?

– Verification

– Validation

12
 Design of Unit Test Cases
• There are essentially three main approaches to
design test cases:

–Black-box approach

–White-box (or glass-box) approach

–Grey-box approach
13
 Black-Box Testing
• Test cases are designed using only functional
specification of the software:
Input Software Output
–Without any knowledge of the
internal structure of the software.

• Black-box testing is also known as functional testing.

14
 What is Hard about BB Testing
• Data domain is large

• A function may take multiple parameters:

– We need to consider the combinations of the values of

the different parameters.

15
 What’s So Hard About Testing?
• Consider int check-equal(int x, int y)
• Assuming a 64 bit computer
–Input space = 2128

• Assuming it takes 10secs to key-in an integer pair:

-It would take about a billion years to enter all possible values!
-Automatic testing has its own problems!

16
 Solution
• Construct model of the data domain:

– Called Domain based testing

– Select data based on the domain model

17
 White-box Testing
• To design test cases:
–Knowledge of internal structure of software
necessary.

–White-box testing is also called structural testing.

18
• Software considered as a black box: Black Box
–Test data derived from the specification Testing
• No knowledge of code necessary

• Also known as:

– Data-driven or
– Input/output driven testing
Input System Output

• The goal is to achieve the thoroughness of exhaustive

input testing:
–With much less effort!!!!

19
• Scenario coverage Black-Box Testing
• Equivalence class partitioning
• Boundary value testing
• Cause-effect (Decision Table) testing
• Combinatorial testing
• Orthogonal array testing

20
 Start use case Use Case
Scenarios
Alternative flow 3 Alternative flow 1

Alternative flow 2

Alternative flow 4
end use case
end use case
end use case
 Deriving test cases from use cases
1. Identify the use case scenarios
2. For each scenario, identify one or more test cases
3. For each test case, identify the conditions that will
cause it to execute.
4. Complete the test case by adding data values

22
Scenario Originating Alternative Next Next
number flow flow alternative alternative Identify
1 Basic flow
use case
scenarios:
2 Basic flow Alt. flow 1
Example
3 Basic flow Alt. flow 1 Alt. flow 2

4 Basic flow Alt. flow 3

5 Basic flow Alt. flow 3 Alt. flow 1

6 Basic flow Alt. flow 3 Alt. flow 1 Alt. flow 2

7 Basic flow Alt. flow 4

8 Basic flow Alt. flow 3 Alt. flow 4

23
• Parameters of any test case:
Identify the
– Conditions
test cases
– Input (data values)
– Expected result
– Actual result
Test Scenario/ Data Data Data Exp. Actual
case ID conditon value 1 value 2 value N results results
1 Scenario 1

2 Scenario 2

3 Scenario 3

24
 Equivalence Class
Testing

03/08/10 25
 Equivalence Class Partitioning Start use case

• The input values to a program:

– Partitioned into equivalence classes.
• Partitioning is done such that: end use case

–Program behaves in similar ways to every input value

belonging to an equivalence class.
–At the least, there should be as many equivalence classes
as scenarios.
26
 Why Define Equivalence Classes?
E1 E2 E3
• Premise:
–Testing code with any one representative value from
a equivalence class:

–As good as testing using any other values from the

equivalence class.

27
 Equivalence Class Partitioning
• How do you identify equivalence classes?
–Identify scenarios
–Examine the input data.
–Examine output
• Few guidelines for determining the equivalence classes
can be given…

28
•If an input is a range, one valid and two invalid equivalence classes are defined.
Example: 1 to 100 1 100
•If an input is a set, one valid and one invalid equivalence classes are defined. Example:
{a,b,c}

•If an input is a Boolean value, one valid and one invalid class are defined.

Example: Guidelines
to Identify
•Area code: input value defined between 10000 and 90000--- range
Equivalence
•Password: string of six characters --- set Classes

29
 Equivalent class partition: Example
• Given three sides, determine the type of the triangle:
– Isosceles

– Scalene

– Equilateral, etc.

• Hint: scenarios correspond to output in this case.

30
 Equivalence Partitioning
• First-level partitioning:
– Valid vs. Invalid test cases
Valid Invalid

31
 Equivalence Partitioning

• Further partition valid and invalid test

cases into equivalence classes

Invalid
Valid

32
 Equivalence Partitioning
• Create a test case using at least
one value from each
equivalence class

Invalid
Valid

33
 Equivalence Class Partitioning
• If the input data to the program is specified by a range
of values:
–e.g. numbers between 1 to 5000.

–One valid and two invalid equivalence classes are defined.

1 5000

34
 Equivalence Class Partitioning
• If input is an enumerated set of values, e.g. :
–{a,b,c}
• Define:
–One equivalence class for valid input values.
–Another equivalence class for invalid input values..

35
 Example

• A program reads an input value in the range of 1

and 5000:
–Computes the square root of the input number

SQRT

36
 Example (cont.)
• Three equivalence classes:
–The set of negative integers,
–Set of integers in the range of 1 and 5000,
–Integers larger than 5000.
1 5000

37
 Example (cont.)
• The test suite must include:
–Representatives from each of the three equivalence
classes:

–A possible test suite can be:

1 5000
{-5,500,6000}.

38
 Equivalence Partitioning
• A set of input values constitute an equivalence class if the tester
believes that these are processed identically:
– Example : issue book(book id);

– Different set or sequence of instructions Book

may be executed based on book type.
Issue book Reference book

Single volume Multiple volume

39
 Equivalence Partitioning: Example 1 File Type
URL

• Example: Image Fetch-image(URL)

– Equivalence Definition 1: Partition based on URL
protocol (“http”, “https”, “ftp”, “file”, etc.)

– Equivalence Definition 2: Partition based on type of

file being retrieved (HTML, GIF, JPEG, Plain Text, etc.)

40
 Equivalence Partitioning: Single Parameter Function
• issue-book(int book-id)

• Create a test case for at least

one value from each equivalence
Book
class Invalid
Issue book Reference book
Valid
Single volume Multiple volume

41
 Multiparameter Functions
• postGrade(Roll,CourseNo, Grade)

Invalid Invalid
Valid Invalid
Valid Valid

42
 Multiparameter
int Normalization Factor; Function Accessing
postGrade(Roll,CourseNo, Grade) Global Variables
{ Grade=Grade*NormalizationFactor
-------}

Invalid Invalid Invalid Invalid

Valid Valid Valid Valid

43
 Multi Parameter Equivalence Partitioning: Example
Input Parameter Valid Equivalence Classes Invalid Equivalence
Classes

? ?
An integer N such that:
-99 <= N <= 99

? ?
Phone Number
Area code: [11,…, 999]
Suffix: Any 6 digits

44
 Equivalence Partitioning: Example
Input Valid Equivalence Classes Invalid Equivalence Classes

A integer N such that: [-99, 99] < -99

> 99
-99 <= N <= 99
Malformed numbers
{12-, 1-2-3, …}
Non-numeric strings
{junk, 1E2, $13}
Empty value
Phone Number [11,999][000000, Invalid format 5555555,
Area code < 11 or > 999
Prefix: [11, 999] 999999]
Area code with non-numeric
Suffix: Any 6 digits characters

45
 age predict-employability(age, education)

>30
Weak
Equivalence
Class Testing
20-30

School UG PG yrs of education

46
 predict-employability(age, education)
age

>30 Strong
Equivalence
Class Testing
20-30

School UG PG Yrs of ed.

47
 age predict-employability(age, education)
Strong
Robust
Equivalence
>30
Class Testing

20-30

School UG PG
yrs of education
48
• Design Equivalence class test cases: compute-interest(days)
• A bank pays different rates of interest on a deposit depending on
the deposit period.
Quiz 1
– 3% for deposit up to 15 days
– 4% for deposit over 15days and up to 180 days
– 6% for deposit over 180 days upto 1 year
– 7% for deposit over 1 year but less than 3 years
– 8% for deposit 3 years and above

49
• Design Equivalence class test cases: compute-interest(principal, days)
• For deposits of less than or equal to Rs. 1 Lakh, rate of interest:
– 6% for deposit upto 1 year
– 7% for deposit over 1 year but less than 3 years
Quiz 2
– 8% for deposit 3 years and above
• For deposits of more than Rs. 1 Lakh, rate of interest:
– 7% for deposit upto 1 year
– 8% for deposit over 1 year but less than 3 years
– 9% for deposit 3 years and above

50
• Design equivalence class test cases. Quiz 3

– Consider a program that takes 2 strings of maximum

length 20 and 5 characters respectively

– Checks if the second is a substring of the first

– substr(s1,s2);

51
 Special Value
Testing

03/08/10 52
52
 Special Value Testing
• What are special values?
– The tester has reasons to believe that execution with certain
values may expose bugs:

–General risk: Example-- Boundary value testing

–Special risk: Example-- Leap year not considered

53
• Some typical programming errors occur:
Boundary
–At boundaries of equivalence classes
Value Analysis
–Might be purely due to psychological factors.
1 100
• Programmers often commit mistakes in the:
–Special processing at the boundaries of equivalence
classes.

54
 Boundary Value Analysis
• Programmers may improperly use < instead of
<=
• Boundary value analysis: 1 100

–Select test cases at the boundaries of different

equivalence classes.
55
 Boundary Value Analysis: Guidelines
• If an input is a range, bounded by values a and b:
– Test cases should be designed with value a and b, just above
and below a and b.
• Example 1: Integer D with input range [-3, 10],
– test values: -3, 10, 11, -2, 0
• Example 2: Input in the range: [3,102]
– test values: 3, 102, -1, 200, 5

56
 Boundary Value Testing Example
• Process employment applications based on a person's age.
0-16 Do not hire
16-18 May hire on part time basis
18-55 May hire full time
55-99 Do not hire

• Notice the problem at the boundaries.

– Age "16" is included in two different equivalence classes (as are

18 and 55).
57
 Boundary Value Testing: Code Example

• If (applicantAge >= 0 && applicantAge <=16) hireStatus="NO";

• If (applicantAge >= 16 && applicantAge <=18) hireStatus="PART";

• If (applicantAge >= 18 && applicantAge <=55) hireStatus="FULL";

• If (applicantAge >= 55 && applicantAge <=99) hireStatus="NO";

58
• Corrected boundaries: Boundary Value Testing Example (cont)
0–15 Don't hire
16–17 Can hire on a part-time basis only
18–54 Can hire as full-time employees
55–99 Don't hire

• What about ages -3 and 101?

• The requirements do not specify how these values should be

treated.

59
 Boundary Value Testing Example (cont)
• The code to implement the corrected rules is:
If (applicantAge >= 0 && applicantAge <=15) hireStatus="NO";
If (applicantAge >= 16 && applicantAge <=17) hireStatus="PART";
If (applicantAge >= 18 && applicantAge <=54) hireStatus="FULL";
If (applicantAge >= 55 && applicantAge <=99) hireStatus="NO";
• Special values on or near the boundaries in this example are {-1, 0, 1}, {14, 15,
16}, {17, 18, 19}, {54, 55, 56}, and {98, 99, 100}.

60
 Boundary Value Analysis
• Create test cases to test boundaries of equivalence classes

61
 Example 1
• For a function that computes the square root of an
integer in the range of 1 and 5000:
–Test cases must include the values:
{0,1,2,4999,5000,5001}.
1 5000

62
• Consider a program that reads the “age” of employees and
computes the average age.
Example 2
input (ages) → Program → output: average age
Assume valid age is 1 to 150
1 150
• How would you test this?
– How many test cases would you generate?
– What type of test data would you input to test this program?

63
 Boundaries of the inputs
The “basic” boundary value testing
would include 5 test cases:
predict-longevity(age)
1. - at minimum boundary
2. - immediately above minimum
1 <= age <= 150
3. - between minimum and
1 age 150
maximum (nominal)
4. - immediately below maximum
5. - at maximum boundary

64
• How many test cases for the example ? Test Cases for the
Example
– answer : 5

• Test input values? :

1 at the minimum predict-longevity(age)
2 at one above minimum
45 at middle
149 at one below maximum
150 at maximum

65
 Multiple Parameters: Independent distinct Data
• Suppose there are 2 “distinct” inputs that are assumed to be independent
of each other.
– Input field 1: years of education ( say 1 to 23 )
– Input field 2: age (1 to 150)

• If they are independent of each other, then we can start with 5 + 5 = 10

sets. coverage of input data: yrs of ed coverage of input data: age

1. n= 1 ; age = whatever(37) 1. n= 12; age = 1

2. n =2; age = whatever 2. n =12; age = 2
3. n = 12; age = whatever 3. n = 12; age = 37
4. n = 22; age = whatever 4. n = 12; age = 149
5. n = 23; age = whatever 5. n = 12; age = 150

66
age
2–
Independent
inputs

yrs of ed.
- Note that there needs to be only 9 test cases for 2 independent inputs.
- In general, need (4z + 1) test cases for z independent inputs.

67
 Boundary Value Test
Given f(x,y) with constraints a ≤ x ≤ b
Boundary Value analysis focuses on the c ≤ y ≤ d
boundary of the input space to identify
test cases. y

Defined as input variable value at min, just d

above min, a nominal value, just above c

max, and at max.
x
a b

68
 Weak Testing: Single Fault Assumption
• Premise: “Failures rarely occur as the result of the
simultaneous occurrence of two (or more) faults”
• Under this:
– Hold the values of all but one variable at their nominal
values, and let that one variable assume its extreme
values.
69
 Boundary Value Analysis: Robustness
• Numeric values are often entered as strings :

– Internally converted to numbers [int x = atoi(str); val=x-48;]

• This conversion requires the program to distinguish between digits

and non-digits

• A boundary case to consider: Will the program accept / and : as

digits? Char / 0 1 2 3 4 5 6 7 8 9 :
ASCII 47 48 49 50 51 52 53 54 55 56 57 58

70
 Robustness testing
• This is just an extension of the Boundary Values to include invalid
values:
X

– Less than minimum

– Greater than maximum

• There are 7 test cases for each input

Y
• The testing of robustness is really a test of “error” handling.
1. Did we anticipate the error situations?
2. Do we issue informative error messages?
3. Do we support some kind of recovery from the error?

71
 2 – independent inputs for robustness test
X

•Note that there needs to be

only 13 test cases for 2
independent variables or
inputs.

•In general, there will be (6n+

1) test cases for n independent
inputs.

72
 Some Limitations of Boundary Value Testing
• How to handle a set of values?
• How about set of Boolean variables?
– True
– False
• May be radio buttons
• What about a non-numerical variable whose values are
text?

73
 Quiz: BB Test Design
• Design black box test suite for a function that
solves a quadratic equation of the form
ax2+bx+c=0.
Complex
• Equivalence classes Coincident
Real
– Invalid Equation Unique

– Valid Equation: Roots?

74