TESTING
CS 404 – SOFTWARE ENGINEERING 2 - FINALS
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 1
Some Observations
It is impossible to completely test any nontrivial module or any
system
Theoretical limitations: Halting problem ??
Practial limitations: Prohibitive in time and cost
Testing can only show the presence of bugs, not their absence
(Dijkstra)
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 2
Testing Activities
Subsystem Requirements
Unit System
Code Test Analysis
Design Document
Tested Document User
Subsystem
Subsystem Manual
Unit
Code Test
Tested Integration Functional
Subsystem
Test Test
Integrated Functioning
Subsystems System
Tested Subsystem
Subsystem Unit
Code Test
All tests by developer
Cf. levels of testing
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 3
Testing Activities continued
Client’s
Global Understanding User
Requirements of Requirements Environment
Validated Accepted
Functioning
System
System PerformanceSystem Acceptance Installation
Test Test Test
Usable
Tests by client System
Tests by developer
User’s understanding
System in
Use
Tests (?) by user
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 4
Level of abstraction
Levels of Testing in V Model
system system
requirements integration
software acceptance
requirements test
preliminary software
design integration
detailed component
design test
code & unit
debug test
Time
N.B.: component test vs. unit
Bernd Bruegge & Allen H. Dutoit
test; acceptance test vs. system integration
Object-Oriented Software Engineering: Using UML, Patterns, and Java 5
Test Planning [Pressman]
A Test Plan: A test plan includes:
covers all types and phases of test objectives
testing schedule and logistics
guides the entire testing process test strategies
who, why, when, what test cases
developed as requirements, ⧫ procedure
functional specification, and high- ⧫ data
level design are developed
⧫ expected result
should be done before
implementation starts procedures for handling
problems
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 6
Fault Handling Techniques
Fault Handling
Fault Tolerance
Fault Avoidance Fault Detection
Design Atomic Modular
Reviews
Methodology Transactions Redundancy
Configuration
Verification
Management
Testing Debugging
Unit Integration System Correctness Performance
Testing Testing Testing Debugging Debugging
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 7
Quality Assurance encompasses Testing
Quality Assurance
Usability Testing
Scenario Prototype Product
Testing Testing Testing
Fault Avoidance Fault Tolerance
Atomic Modular
Configuration Transactions Redundancy
Verification
Management
Fault Detection
Reviews
Debugging
Walkthrough Inspection
Testing
Correctness Performance
Unit Integration System Debugging Debugging
Testing Testing Testing
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 8
Types of Testing
Unit Testing:
Individual subsystem
Carried out by developers
Goal: Confirm that subsystems is correctly coded and carries out
the intended functionality
Integration Testing:
Groups of subsystems (collection of classes) and eventually the
entire system
Carried out by developers
Goal: Test the interface among the subsystem
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 9
System Testing
Terminology:
System Testing: system testing here = validation testing
The entire system
Carried out by developers
Goal: Determine if the system meets the requirements (functional
and global)
Acceptance Testing: 2 kinds of Acceptance testing
Evaluates the system delivered by developers
Carried out by the client. May involve executing typical
transactions on site on a trial basis
Goal: Demonstrate that the system meets customer requirements
and is ready to use
Implementation (Coding) and testing go hand in hand
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 10
Unit Testing
Informal:
Incremental coding Write a little, test a little
Static Analysis:
Hand execution: Reading the source code
Walk-Through (informal presentation to others)
Code Inspection (formal presentation to others)
Automated Tools checking for
⧫ syntactic and semantic errors
⧫ departure from coding standards
Dynamic Analysis:
Black-box testing (Test the input/output behavior)
White-box testing (Test the internal logic of the subsystem or object)
Data-structure based testing (Data types determine test cases)
Which is more effective, static or dynamic analysis?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 11
Black-box Testing
Focus: I/O behavior. If for any given input, we can predict the
output, then the module passes the test.
Almost always impossible to generate all possible inputs ("test
cases") why?
Goal: Reduce number of test cases by equivalence partitioning:
Divide input conditions into equivalence classes
Choose test cases for each equivalence class. (Example: If an object
is supposed to accept a negative number, testing one negative
number is enough)
❑ If x = 3 then …
❑ If x > -5 and x < 5 then …
What would be the equivalence classes?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 12
Black-box Testing (Continued)
Selection of equivalence classes (No rules, only guidelines):
Input is valid across range of values. Select test cases from 3
equivalence classes:
⧫ Below the range
Are these complete?
⧫ Within the range
⧫ Above the range
Input is valid if it is from a discrete set. Select test cases from 2
equivalence classes:
⧫ Valid discrete value
⧫ Invalid discrete value
Another solution to select only a limited amount of test cases:
Get knowledge about the inner workings of the unit being tested =>
white-box testing
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 13
White-box Testing
Focus: Thoroughness (Coverage). Every statement in the component is
executed at least once.
Four types of white-box testing
Statement Testing
Loop Testing
Path Testing
Branch Testing
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 14
White-box Testing (Continued)
Statement Testing (Algebraic Testing): Test single statements
Loop Testing:
Cause execution of the loop to be skipped completely. (Exception:
Repeat loops)
Loop to be executed exactly once
Loop to be executed more than once
Path testing:
Make sure all paths in the program are executed
Branch Testing (Conditional Testing): Make sure that each
possible outcome from a condition is tested at least once
if ( i = TRUE) printf("YES\n");else printf("NO\n");
Test cases: 1) i = TRUE; 2) i = FALSE
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 15
White-Box Testing
Loop Testing
[Pressman]
Simple
loop
Nested
Loops
Concatenated
Loops Unstructured
Loops
Why is loop testing important?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 16
Who Tests the Software?
[Pressman]
developer independent tester
Understands the system Must learn about the system,
but, will test "gently" but, will attempt to break it
and, is driven by "delivery" and, is driven by quality
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 17
Examples of Faults and Errors
Faults in the Interface Mechanical Faults (very
specification hard to find)
Mismatch between what the Documentation does not
client needs and what the match actual conditions or
server offers operating procedures
Mismatch between Errors
requirements and
implementation Stress or overload errors
Capacity or boundary errors
Algorithmic Faults
Timing errors
Missing initialization
Throughput or performance
Branching errors (too soon, errors
too late)
Missing test for nil
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 18
Dealing with Errors
Verification:
Assumes hypothetical environment that does not match real
environment
Proof might be buggy (omits important constraints; simply wrong)
Modular redundancy:
Expensive
Declaring a bug to be a “feature”
Bad practice
Patching
Slows down performance
Testing (this lecture)
Testing is never good enough
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 19
Another View on How to Deal with Errors
Error prevention (before the system is released):
Use good programming methodology to reduce complexity
Use version control to prevent inconsistent system
Apply verification to prevent algorithmic bugs
Error detection (while system is running):
Testing: Create failures in a planned way
Debugging: Start with an unplanned failures
Monitoring: Deliver information about state. Find performance bugs
Error recovery (recover from failure once the system is released):
Data base systems (atomic transactions)
Modular redundancy
Recovery blocks
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 20
What is this?
A failure?
An error?
A fault?
Need to specify
the desired behavior first!
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 21
Erroneous State (“Error”)
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 22
Algorithmic Fault
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 23
Mechanical Fault
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 24
How do we deal with Errors and Faults?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 25
Verification?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 26
Modular Redundancy?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 27
Declaring the Bug
as a Feature?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 28
Patching?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 29
Testing?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 30
Testing takes creativity
Testing often viewed as dirty work.
To develop an effective test, one must have:
⧫ Detailed understanding of the system
⧫ Knowledge of the testing techniques
⧫ Skill to apply these techniques in an effective and efficient manner
Testing is done best by independent testers
We often develop a certain mental attitude that the program should
in a certain way when in fact it does not.
Programmer often stick to the data set that makes the program
work
"Don’t mess up my code!"
A program often does not work when tried by somebody else.
Don't let this be the end-user.
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 31
Test Cases
Test case 1 : ? (To execute loop exactly once)
Test case 2 : ? (To skip loop body)
Test case 3: ?,? (to execute loop more than once)
These 3 test cases cover all control flow paths
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 32
