Testing Techniques

Once the software is developed it should be tested in a proper manner before the system is
delivered to the user. For this, two techniques that provide systematic guidance for designing
tests are used. These techniques are discussed here:

1. Once the internal working of software is known, tests are performed to ensure that all
internal operations of the software are performed according to specifications. This is
referred to as white box testing.
2. Once the specified function for which the software has been designed is known, tests are
performed to ensure that each function is working properly. This is referred to as black
box testing.

White Box Testing

White box testing (also called structural testing or glass box testing) is performed to test the
program internal structure. To perform white box testing, the tester should have a thorough
knowledge of the program internals along with the purpose of developing the software. During
this testing, the entire software implementation is also included with the specification. This helps
in detecting errors even with unclear or incomplete software specification.
The goal of white box testing is to ensure that the test cases (developed by software testers by
using white box testing) exercise each path through a program. That is, test cases ensure that all
internal structures in the program are developed according to design specifications. The test
cases also ensure the following.

1. All independent paths within the program have been exercised at least once.
2. All internal data structures have been exercised.
3. All loops (simple loops, concatenated loops, and nested loops) have been executed at and
within their specified boundaries.
4. All segments present between the control structures (like ‘switch’ statement) have been
executed at least once.
5. Each branch (like ‘case’ statement) has been exercised at least once.
6. All the logical conditions as well as their combinations have been executed at least once
for both true and false paths.

Various advantages and disadvantages of white box testing are listed in Table.

Table Advantages and Disadvantages of White Box Testing

Advantages Disadvantages
1. Covers the larger part of the 1. Tests that cover most of the
program code while testing. program code may not be good for
2. Uncovers typographical errors. assessing the functionality of
3. Detects design errors that occur surprise (unexpected) behaviors and
when incorrect assumptions are other testing goals.
made about execution paths. 2. Tests based on design may miss
other system problems.
3. Tests cases need to be changed if
implementation changes.

The effectiveness of white box testing is usually measured in terms of test or code coverage
metrics, that is, the fraction of code exercised by test cases. Various types of testing, which occur
as part of white box testing are basis path testing, control structure testing, and mutation testing.
Basis path testing enables to generate test cases such that every path of the program has been
exercised at least once. This technique is used to specify the basic set of execution paths that are
required to execute all the statements present in the program. Note that with the increase in the
size of the software the number of execution paths also increase, thereby degrading the
effectiveness of basis path testing.

Creating Flow Graph

A flow graph represents the logical control flow within a program. For this, it makes use of a
notation.

A flow graph uses different symbols, namely, circles and arrows to represent various statements
and flow of control within the program. Circles represent nodes, which are used to depict the
procedural statements present in the program. A sequence of process boxes and a decision box
used in a flowchart can be easily mapped into a single node. Arrows represent edges or links,
which are used to depict the flow of control within the program. It is necessary far every edge to
end in a node irrespective of whether it represents a procedural statement. In a flaw graph, the
area bounded by edges and nodes is known as a region. In addition, the area outside the graph is
also counted as a region while counting regions. A flow graph can be easily understood with the
help of a diagram.
Note that a node that contains a condition is known as predicated node, which contains one or
more edges emerging out of it.

Finding Independent Paths

A path through the program, which specifies a new condition or a minimum of one new set of
processing statements, is known as an independent path. For example, in nested ‘if’ statements
there are several conditions that represent independent paths. Note that a set of all independent
paths within a program is known as its basis set.

A test case is developed to ensure that while testing all statements of the program get exercised
at least once.

P1: 1-9

P2: 1-2-7-8-1-9

P3: 1-2-3-4-6-8-1-9

P4: 1-2-3-5-6-8-1-9

Where P1, P2, P3, and P4 represent different independent paths present in the program.

To determine the number of independent paths through a program, the cyclomatic complexity
metric is used that provides a quantitative measure of the logical complexity of a program. The
value of this metric defines the number of test cases that should be developed to’ ensure that all
statements in the program get exercised at least once during testing.

Cyclomatic complexity of a program can be computed using any of the following three methods.

 By counting the total number of regions in the flow graph of a program. For example,
there are four regions represented by R1, R2, R3, and R4; hence, the cyclomatic
complexity is four.
 By using the following formula.

CC = E – N + 2

Where

CC = the cyclomatic complexity of the program

E = the number of edges in the flaw graph

N = the number of nodes in the flaw graph.

For example, E = 11, N = 9. Therefore, CC = 11 – 9 + 2 = 4.

  By using the following formula.

CC= P + 1

Where

P= the number of predicate nodes in the flow graph.

For example, P = 3. Therefore, CC = 3 + 1 = 4.

Note: Cyclomatic complexity can be calculated either manually (generally for small
program suites) or using automated tools. However, for most operational environments,
automated tools are preferred.

Deriving Test Cases

In this, basis path testing is presented as a series of steps and test cases are developed to ensure
that all statements within the program get exercised at least once while performing testing. While
performing basis path testing, initially the basis set (independent paths in the program) is
derived. The basis set can be derived using the steps listed below.

1. Draw the flow graph of the program: A flow graph is constructed using symbols
previously discussed. For example, a program to find the greater of two numbers is given
below.

Procedure greater;

integer: x, y, z = 0;

1. enter the value of x;

2. enter the value of y;
3. if x > y then
4. z = x;

else

5. z = y;
6. end greater
 2. Compute the cyclomatic complexity: Thecyc10matic complexity of the program can be
computed using the flow graph depicted.

CC = 2 as there two regions R1 and R2

or

CC 6 edges – 6 nodes + 2 =2

or

CC 1 predicate node + 1 = 2.

3. Determine all independent paths through the program: For the flow graph depicted
the independent paths are listed below.

P1: 1-2-3-4-6

P2: 1-2-3-5-6

4. Prepare test cases: Test cases are prepared to implement the execution of all
independent paths in the basis set. The program is then tested for each test case and the
produced output is compared with the desired output.

Graph matrix is used to develop software tool that in turn helps in carrying ‘out basis path
testing. It is defined as a data structure used to represent the flow graph of a program in a tabular
form. This matrix is also used to evaluate the control structures present in the program during
testing.
Graph matrix is a square matrix of the size NxN, where Nis the number of nodes in the flow
graph. An entry is made in the matrix at the intersection of ith row and jth column if there exists
an edge between ith and jth node in the flow graph. Every entry in the graph matrix is assigned
some value known as link weight. Adding link weights to each entry makes the graph matrix a
useful tool for evaluating the control structure of the program during testing.

In the flow graph, numbers and letters are used to identify each node and edge respectively. A
letter entry is made if there is an edge between two nodes of the flow graph. For example, node 3
is connected to the node 6 by edge d and node 4 is connected to node 2 by edge c, and so on.

Control Structure Testing

Control structure testing is used to enhance the coverage area by testing various control
structures (which include logical structures and loops) present in the program. Note that basis
path testing is used as one of the techniques for control structure testing. Various types of testing
performed under control structure testing are condition testing, data-flow testing, and loop
testing.

Condition Testing

In condition testing, the test cases are derived to determine whether the logical conditions and
decision statements are free from errors. The errors presenting logical conditions can be incorrect
Boolean operators, missing parenthesis in a Boolean expression, error in relational operators,
arithmetic expressions, and so on.

The common types of logical conditions that are tested using condition testing are listed below.

1. A relational expression such as E1 op E2, where E1 and E2 are arithmetic expressions

and op is an operator.
2. A simple condition such as any relational expression preceded by a NOT (~) operator for
example, (~E1), where E1 is an arithmetic expression.
 3. A compound condition, which is formed by combining two or more simple conditions
using Boolean operators. For example, (E1 & E2) | (E2 & E3), where El, E2, and E3 are
arithmetic expressions and & and | represent AND and OR operators respectively.
4. A Boolean expression consisting of operands and a Boolean operator such as AND, OR,
or NOT. For example, A|B is a Boolean expression, where A and B are operands and |
represents OR operator.

Condition testing is performed using different strategies, namely, branch testing, domain testing,
and branch and relational operator testing. Branch testing executes each branch (like ‘if
statement) present in the module of a program at least once to detect all the errors present in the
branch. Domain testing tests relational expressions present in a program. For this, domain testing
executes all statements of the program that contain relational expressions. Branch and relational
operator testing tests the branches present in the module of a program using condition
constraints. For example,

if a > 10

then

print big

In this case, branch and relational operator testing verifies when the above code is executed, it
produces the output ‘big’ only if the value of variable a is greater than 10.

In data flow, testing, test cases are derived to determine the validity of variables definitions and
their uses in the program. This testing ensures that all variables are used properly in a program.
To specify test cases, data-flow-based testing uses information such as location at which the
variables are defined and used in the program.

For performing data-flow testing, a definition-use graph is built by associating the program
variables with nodes and edges of the control flow graph. Once these variables are attached, test
cases can easily determine which variable is used in which part of a program and how data is
flowing in the program. Thus, data-flow of a program can be tested easily using specified test
cases.

Loop Testing

Loop testing is used to check the validity of loops present in the program modules. Generally,
there exist four types of loops, namely, simple loop, nested loops, concatenated loops, and
unstructured loops.
 Simple loop: Refers to a loop that has no other loops in it. Consider a simple loop of size
n. Size n of the loop indicates that the loop can be traversed n times, that is, n number of
passes are made through the loop. The steps followed for testing simple loops are listed
below.

1. Skip the entire loop.

2. Traverse the loop only once.
3. Traverse the loop two times.
4. Make ill number of passes through the loop, where ill < n.
5. Traverse the loop n – I, n, n + 1 times.
  Nested loops: Loops within loops are known as nested loops. The numbers of tests
required for testing nesting loops depends on the level of nesting. More is the level of
nesting, more will be the number of tests required. The steps followed for testing nested
loops are listed below.

1. Start with the inner loop and set values of all the outer to minimum.
2. Test the inner loop using the steps followed for testing simple loops while keeping the
iteration parameters of the outer loops at their minimum values. Add other tests for
values that are either out-of-range or are eliminated.
3. Move outwards and perform tests for the next loop while holding other nested loops to
‘typical’ values and the iteration parameters of the outer loops at their minimum values.
4. Continue performing tests until all the loops are tested.

1. Concatenated loops: The loops containing several loops that may be dependent or
independent In case the loops are dependent on each other, the steps in nested loops are
followed. On the other hand, if the loops are independent of each other, the steps in
simple loops are followed.
2. Unstructured loops: Such loops of difficult to test; therefore, they should be redesigned
so that the use of structured programming constructs can be reflected.

Mutation testing is a white box method where errors are ‘purposely’ inserted into a program
(under test) to verify whether the existing test case is able to detect the error. In this testing,
mutants of the program are created by making some changes in the original program. The
objective is to check whether each mutant produces an output that is different from the output
produced by the original program.

In mutation testing, test cases that are able to ‘kill’ all the mutants should be developed. This is
accomplished by testing mutants with the developed set of test cases. There can be two possible
outcomes when the test cases test the program either the test case detects the faults or fails to
detect faults. If faults are detected, then necessary measures are taken to correct them.

When no faults are detected, it implies that either the program is absolutely correct or the test
case is inefficient to detect the faults. Therefore, it can be concluded that mutation testing is
conducted to determine the effectiveness of a test case. That is, if a test case is able to detect
these ‘small’ faults (minor changes) in a program, then it is likely that the same test case will be
equally effective in finding real faults.

To perform mutation testing, a number of steps are followed, which are listed below.

1. Create mutants of a program.

2. Check both program and its mutants using test cases.
3. Find the mutants that are different from the main program. A mutant is said to be
different from the main program if it produces an output, which is different from the
output produced by the main program.
4. Find mutants that are equivalent to the main program. A mutant is said to be equivalent to
the main program if it produces the same output as that of, the main program.
5. Compute the mutation score using the formula given below.

M = D/ (N – E)

Where

M = Mutation score

N = Total number of mutants of the program.

D= Number of mutants different from the main program.

E = Total number of mutants that are equivalent to the main program.

6. Repeat steps 1 to 5 till the mutation score is ‘1’.

However, mutation testing is very expensive to run on large programs. Thus, certain tools are
used to run mutation tests on large programs. For example, ‘Jester’ is used to run mutation tests
on java code. This tool targets the specific areas of the program code such as changing constants
and Boolean values.

Black Box Testing

Black box (or functional) testing checks the functional requirements and examines the input and
output data of these requirements. When black box testing is performed, only the sets of ‘legal’
input and corresponding outputs should be known to the tester and not the internal logic of the
program to produce that output. Hence to determine the functionality, the outputs produced for
the given sets of input are observed.
The black box testing is used to find the errors listed below.

1. Interface errors such as functions, which are unable to send or receive data to/from other
software.
2. Incorrect functions that lead to undesired output when executed.
3. Missing functions and erroneous data structures.
4. Erroneous databases, which lead to incorrect outputs when the software uses the data
present in these databases for processing.
5. Incorrect conditions due to which the functions produce incorrect outputs when they are
executed.
6. Termination errors such as certain conditions due to which a function enters a loop that
forces it to execute indefinitely.
In this testing, tester derives various test cases to exercise the functional requirements of the
software without considering implementation details of the code. Then, the software is run for
the specified sets of input and the outputs produced for each input set is compared against the
specifications to conform the correctness. If they are a23pecified by the user, then the software is
considered to be correct else the software is tested for the presence of errors in it. The advantages
and disadvantages associated with black box testing are listed in Table.

Table Advantages and Disadvantages of Black Box Testing

Advantages Disadvantages
1. Tester does not require knowledge 1. Exercising software for every
of internal logic of the program and possible test case requires a lot of
the programming language used. time, thus, only a small number of
2. Reveals any ambiguities and test cases are used to test the
inconsistencies in the functional functional requirements.
specifications. 2. As test cases are developed without
3. Efficient when used on larger looking at the internal logic of
systems. program, testing may leave many
4. A non-technical person can also paths in the program unexercised.
perform black box testing. 3. There may be duplication of test
cases if tester is unaware of the test
cases that have already been tried
by the software developer.
4. Cannot be targeted towards
particular code segments, hence is
more error prone.

Various methods used in black box testing are equivalence class partitioning, boundary value
analysis, and cause-effect graphing. In equivalence class partitioning, the test inputs are
classified into equivalence classes such that one input checks (validates) all the input values in
that class. In boundary value analysis, the boundary values of the equivalence classes are
considered and tested. In cause-effect graphing, cause-effect graphs are used to design test cases,
which provides all the possible combinations of inputs to the program.
This method tests the validity of outputs by dividing the input domain into different classes of
data (known as equivalence classes) using which test cases can be easily generated. Test cases
are designed with the purpose of covering each partition at least once. A test case that is able to
detect every error in a specified partition is said to be an ideal test case.

An equivalence class depicts valid or invalid states for the input condition. An input condition
can be either a specific numeric value, a range of values, a Boolean condition, or a set of values.
The general guidelines that are followed for generating the equivalence classes are listed in
Table.

Table Guidelines for Generating Equivalence Classes

Input Condition Number of equivalence Description

classes
Boolean Two One valid and one invalid
Specific numeric value Three One valid and two invalid
Range Three One valid and two invalid
Member of a set Two One valid and one invalid
To understand equivalence class partitioning properly, let us consider an example. This example
is explained in the series of steps listed below.

1. Suppose that a program P takes an integer X as input.

2. Now, either X < 0 or X >0.
3. In case X < 0, the program is required to perform task T1; otherwise, the task T2 is
performed.
4. The input domain is as large as X and it can assume a large number of values. Therefore
the input domain (p) is partitioned into two equivalence classes and all test inputs in the
X < 0 and X > 0 equivalence classes are considered to be equivalent.
5. Now, as shown in Figure independent test cases are developed for X < 0 and X > 0.

In boundary value analysis (BVA), test cases are derived on the basis of values that lie on an
edge of the equivalence partitions. These values can be input or output values either at the edge
or within the permissible range from the edge of an equivalence partition.

BVA is used since it has been observed that most errors occur at the boundary of input domain
rather than at the middle of the input domain. Note that boundary value analysis complements
the equivalence partitioning method. The only difference is that in BVA, test cases are derived
for both input domain and output domain while in equivalence partitioning test cases are derived
only for input domain.

Generally, the test cases are developed in boundary value analysis using certain guidelines,
which are listed below.

1. If an input condition specifies a range of values, test cases should be developed on the
basis of both the values at the boundaries and the values that are just above and below the
boundary values. For example, for the range -0.5≤X≤0.5, the input values for a test case
can be ‘-0.4’, -‘0.5’, ‘0.5’, ‘0.6’.
 2. If an input condition specifies a number of values, test cases should be designed to
exercise the minimum and maximum numbers as well as values just above and below
these numbers.
3. If an input consists of certain data structures (like arrays), then the test case should be
able to execute all the values present at the boundaries of the data structures such as the
maximum and minimum value of an array.

Equivalence partitioning and boundary value analysis tests each input given to a program
independently. It means none .of these consider the case of combinations of inputs, which may
produce situations that need to be tested. This drawback is avoided in cause-effect graphing
where combinations of inputs are used instead of individual inputs. In this technique, the causes
(input conditions) and effects (output conditions) of the system are identified and a graph is
created with each condition as the node of the graph. This graph is called cause-effect graph.
This graph is then used to derive test cases. To use the cause-effect graphing method, a number
of steps are followed, which are listed below.


1. List the cause (input conditions) and effects (outputs) of the program.
2. Create a cause-effect graph.
3. Convert the graph into a decision table.
4. Generate test cases from the decision table rules.

In order to generate test cases, all causes and effects are allocated unique numbers, which are
used to identify them. After allocating numbers, the cause due to which a particular effect
occurred is determined. Next, the combinations of various conditions that make the effect ‘true’
are recognized. A condition has two states, ‘true’ and ‘false’. A condition is ‘true’ if it causes the
effect to occur; otherwise, it is ‘false’. The conditions are combined using Boolean operators
such as ‘AND’ (&), ‘OR’ (I), and ‘NOT’ (-). Finally, a test case is generated for all possible
combinations of conditions.

Various symbols are used in the cause-effect graph. The figure depicts various logical
associations among causes ci and effects ei. The dashed notation on the right side in the figure
indicates various constraint associations that can be applied to either causes or effects.
To understand cause-effect graphing properly, let us consider an example. Suppose a triangle is
drawn with inputs x, y, and z. The values of these inputs are given between ‘0’ and ‘100’. Using
these inputs, three outputs are produced, namely, isosceles triangle, equilateral triangle or no
triangle is made (if values of x, y, z are less than. 60°).

1. Using the steps of cause-effect graphing, initially the causes and effects of the problem
are recognized, which are listed in Table.

Table Causes and Effects

Causes Effects
C1: side x is less than the sum of sides y El: no triangle is formed.
and z.
E2: equilateral triangle is formed.
C2: sides x, y, z are equal.
E3: isosceles triangle is formed.
C3: side x is equal to side y.

C4: side y is equal to side z.

C5: side x is equal to side z.

 2. A decision table (a table that shows a set of conditions and the actions resulting from
them) is drawn as shown in Table.

Table: Decision Table

Conditions
C1: x < y + z 0 X X X X
C2: x=y=z X 1 X X X
C3: x=y X X 1 X X
C4: y=z X X X 1 X
C5: x=z X X X X 1
El: not a triangle 1
E2: equilateral triangle 1
E3: isosceles triangle 1 1 1

3. Each combination of conditions for an effect in Table is a test case.

Differences between White Box and Black Box Testing

Although both the testing techniques are used together to test many programs, there are several
considerations that make them different from each other. Black box testing detects errors of
omission, which are errors occurring due to non-accomplishment of user requirements. On the
other hand, white box testing detects errors of commission which are errors occurring due to
non-implementation of some part of software code. Other differences between these two
techniques are listed in Table.
 Table Differences between White Box and Black Box Testing

Basis White Box Testing Black Box Testing

Purpose  To test the internal  To test the
structure of functionality of
software. software.
 Test the software  Concerned with
but does not ensure testing the
the complete specifications and
implementation of does not ensure that
all the specifications all the components
mentioned in user of software that are
requirements. implemented are
 Addresses flow and tested.
control structure of a  Addresses validity,
program. behavior and
performance of
software

Stage  Performed in the  Performed in the

early stages of later stages of
testing. testing.

Requirement  Knowledge of the  No knowledge of

internal structure of the internal structure
a program is of a program is
required for required to generate
generating test case. test case.

Test Cases  Test cases are  Internal structure of

generated on the modules or
basis of the internal programs is not
structure or code of considered for
the module to be selecting test cases.
tested.

Example  The inner software

present inside the
calculator (which is
known by the
developer only) is
checked by giving
inputs to the code.
In this testing, it is checked whether the calculator is working properly by giving inputs by
pressing the buttons in the calculator.

Gray Box Testing

Gray box testing technique is often defined as a mixture of black box testing and white box
testing techniques. This technique is especially used in web applications. In this technique, the
complete knowledge of the internal structure of the program is not required; rather partial
knowledge that is, how the components of the program operate and interact is sufficient. This
knowledge is used in deriving test cases. Some points noted in gray-box testing are listed below.

1. Gray box testing is platform and language independent.

2. The current implementation of gray box testing is heavily dependent on the use of a host
platform debugger(s) to execute and validate the software under test.
3. It can be applied in real-time systems.
4. It utilizes automated software testing tools to generate test cases
5. Module drivers and stubs are created by automation means, thus saving time of testers.