Software Testing and Quality Assurance
SEng5441
Chapter 1: Software Quality Assurance
(Introduction)
April 22, 2024
KIoT
Department of software engineering
Quality?
• Quality is an intangible concept.
• Most people can recognize it easily but they find it difficult to give a clear
description of the term.
• The terms good quality, poor quality are used in our everyday life to tell
how good or bad a product functions.
• A number of factors influence the making and buying of software
products. These factors include:
• User’s needs and expectations,
• The manufacturer’s considerations,
• The inherent characteristics of a product,
• The perceived value of a product.
• To be able to capture the quality concept, it is important to study quality
from a broader perspective. This is because the concept of quality
predates software development.
4/22/2024 2
Views of Quality
Transcendental View
• In the transcendental view, quality is something that can be
recognized through experience but is not defined in some tractable
form.
• Quality is viewed to be something ideal, which is too complex to
lend itself to be precisely defined. However, a good-quality object
stands out, and it is easily recognized.
User’s View
• It perceives quality as fitness for purpose. According to this view,
while evaluating the quality of a product, one must ask the key
question: “Does the product satisfy user needs and expectations?”
4/22/2024 3
Views of Quality
Manufacturing View
• Here quality is understood as conformance to the specification. The
quality level of a product is determined by the extent to which the product
meets standards and specifications set by the organization.
Product View
• In this view, quality is viewed as quantifiable and measurable internal
characteristics of a product. That is, internal qualities, determine its
external qualities. Example: reliability of a product can be measured by
mean time between failure
Value-Based View
• Quality, in this perspective, depends on the amount a customer is willing
to pay for it (the cost and price associated with the product).
4/22/2024 4
Software ?
Based on IEEE definition,
• Software is: computer programs, procedures, and possibly
associated documentation and data pertaining to the
operation of a computer system
• ISO definition lists out the following components of software:
• Computer programs (the “code”)
• Procedures
• Documentation
• Data necessary for operating the software system
4/22/2024 5
Software Quality
Pressman’s definition:
• Conformance to explicitly stated functional and performance
requirements, explicitly documented development standards,
and implicit characteristics that are expected of all
professionally developed software.
IEEE definitions:
1. The degree to which a system, component, or process meets
specified requirements.
2. The degree to which a system, component, or process meets
customer or user needs or expectations.
4/22/2024 6
Software Quality…
• Software development process is unique:
• High complexity, as compared to other industrial products
• Invisibility of the product
• Opportunities to detect defects (“bugs”) are limited to the
product development phases.
• The uniqueness of software products creates the need to
have a special method and tools for software quality
assurance
The big question?
How do we assure quality?
4/22/2024 7
Software Quality Assurance
IEEE definitions:
1. A planned and systematic pattern of all actions necessary
to provide adequate confidence that an item or product
conforms to established technical requirements.
2. A set of activities designed to evaluate the process by
which the products are developed or manufactured.
4/22/2024 8
4/22/2024 9
Verification and Validation
• Verification
• Evaluation of software system that help in determining whether the product of a given
development phase satisfy the requirements established before the start of that phase.
• The process of determining if the software is designed and developed as per the
specified requirements.
• Building the product correctly
• Validation
• Evaluation of software system that help in determining whether the product meets its
intended use
• The process of checking if the software (end product) has met the client’s true needs
and expectations.
• Building the correct product
10
Role of Testing
• Software quality assessment divide into two categories:
• Static analysis
• It examines the code and reasons over all behaviors that might
arise during run time.
• Examples: Code review, inspection, and algorithm analysis.
• Dynamic analysis
• Actual program execution to expose possible program failure
• One observe some representative program behavior, and reach
conclusion about the quality of the system.
• Focus is to combines the strengths of both approaches.
• Static and Dynamic Analysis are complementary in nature.
11
Software Errors, Faults and Failures
• Software Errors: are sections of the code that are partially or totally
incorrect as a result of a grammatical, logical or other mistake made by
a systems analyst, a programmer, or another member of the software
development team.
• Software Fault: are software errors that cause the incorrect functioning
of the software during a specific application.
• The origin of software failure is from software errors made by
programmers.
• However, not all errors can become software faults.
• All software faults do not necessarily end with software failure
• Software faults cause software failure only when activated
• i.e when the specific faulty application is used
4/22/2024 12
Software Errors, Faults and Failures
Types of Errors:
1. code error
2. procedure error
3. documentation error
4. software data error
4/22/2024 13
Software Errors, Faults and Failures
Example1:
• The “Meteoro-X” meteorological equipment firmware
(software embedded in the product) were meant to block the
equipment’s operation when its internal temperature rose
above 60°C.
• A programmer error resulted in a software fault when the
temperature limit was coded as 160°.
• This fault could cause damage when the equipment was
subjected to temperatures higher than 60°.
• Since the equipment was used only in those coastal areas
where temperatures never exceeded 60 ,the software fault
never turned into a software failure.
4/22/2024 14
Software Errors, Faults and Failures
• Some Famous Software Errors
• Airane 5
• Therac-25
• Patriot Missile System
• NASA's Mars Polar Lander
• ESA's Ariane 5 Launch System
• 2003 Blackout
• Y2K
4/22/2024 15
Famous Software Failures (Examples)
Airane 5
• Cost: $500 million
• Disaster:
• Ariane 5, Europe’s newest unmanned rocket, was intentionally
destroyed seconds after launch on its maiden flight. Also destroyed was
its cargo of four scientific satellites that were to study how the Earth’s
magnetic field interacts with solar winds.
• Cause:
• Shutdown occurred when the guidance computer tried to convert the
sideways rocket velocity from 64-bits to a 16-bit format. The number
was too big, and an overflow error resulted, as the floating point
number had a value too large to be represented by a 16-bit signed
integer . Efficiency considerations had led to the disabling of the
exception handler for this error. This led to a cascade of problems,
culminating in destruction of the entire flight
4/22/2024 16
Famous Software Failures (Examples)
British Passports to Nowhere (1999)
• Cost: £12.6 million, mass inconvenience
• Disaster:
• The U.K. Passport Agency implemented a new Siemens computer
system, which failed to issue passports on time for a half million British
citizens.
• The Agency had to pay millions in compensation, staff overtime and
umbrellas for people queuing in the rain for passports.
• Cause:
• The Passport Agency rolled out its new computer system without
adequately testing it or training its staff. At the same time, a law
change required all children under 16 traveling abroad to obtain a
passport, resulting in a huge spike in passport demand that
overwhelmed the buggy new computer system
4/22/2024 17
Common Causes of Software Errors
• Faulty requirements definition
• Client-developer communication failures
• Deliberate deviations from software requirements
• Logical design errors
• Coding errors
• Non-compliance with documentation and coding
instructions
• Shortcomings of the testing process
• Procedure errors
• Documentation errors
4/22/2024 18
Importance of Software Quality
• Software is a major component of computer systems (about
80% of the cost)
• It is used for:
• Communication (e.g. phone system, email system)
• Health monitoring
• Transportation (e.g. automobile, aeronautics)
• Economic exchanges (e.g. e-commerce).
• Entertainment.
• etc.
• Software defects are extremely costly in terms of
• money
• reputation
• loss of life
4/22/2024 19
Cost of Software Errors
• "Software bugs, or errors, are so prevalent and so detrimental that
they cost the U.S. economy an estimated $59.5 billion annually, or
about 0.6 percent of the gross domestic product. …
• Although all errors cannot be removed, more than a third of these
costs, or an estimated $22.2 billion, could be eliminated by an
improved testing infrastructure that enables earlier and more effective
identification and removal of software defects.
• These are the savings associated with finding an increased
percentage (but not 100 percent) of errors closer to the
development stages in which they are introduced.
• Currently, over half of all errors are not found until "downstream" in
the development process or during post-sale software use."
US Dept of Commerce
June 2002
4/22/2024 20
Measuring Software Quality
• The five viewpoints help us in understanding different aspects of
the quality concept. On the other hand, measurement allows us to
have a quantitative view of the quality concept.
• Developers must know the minimum level of quality they must
deliver for a product to be acceptable.
• Organizations make continuous improvements in their process
models and an improvement has a cost associated with it.
• Organizations need to know how much improvement in quality
is achieved at a certain cost incurred due to process
improvement. This causal relationship is useful in making
management decisions concerning process improvement.
• The present level of quality of a product needs to be evaluated so
the need for improvements can be investigated.
4/22/2024 21
Software Quality Standards
• Various software quality models have been proposed to
define quality and its related attributes. The most influential
ones are:
• McCall's Software Quality Model
• ISO 9126/1-2
• Boehm’s model
• FURPS/FURPS + model
• Dromey’s model
• CMM (Capability Maturity Model)
4/22/2024 22
McCall's Quality Factors
• McCall's Quality Factors have been grouped into three broad
categories as follows:
• Product operation
which are related to the requirements that directly
affect the operation of the software
• Product revision
which are required for testing and maintenance
of the software
• Product transition
that allow the software to adapt to the change of
environments in the new platform or technology
from the previous.
4/22/2024 23
McCall's Quality Factors
4/22/2024 24
McCall's Quality Factors
4/22/2024 25
ISO 9126 Software Quality Factors
• The ISO 9126–1 software quality model identifies 6
main quality characteristics, namely:
1. Functionality
2. Reliability
3. Usability
4. Efficiency
5. Maintainability
6. Portability
4/22/2024 26
FURPS/FURPS + model
• FURPS/FURPS + model stand for:
4/22/2024 27
Capability Maturity Model (CMM)
• The Capability Maturity Model for Software developed by the
SEI is a framework that describes the key elements of an
effective software process.
• The CMM describes an evolutionary improvement path for
software organizations from an ad hoc, immature process to a
mature, disciplined one.
• The CMM covers practices for planning, engineering, and
managing software development and maintenance.
• When followed, these practices improve the ability of
organizations to meet goals for cost, schedule, functionality,
and product quality.
4/22/2024 28
Levels of Capability Maturity Model
1. Initial
2. Repeatable
3. Defined
4. Managed
5. Optimizing
4/22/2024 29
The Objectives of Testing
• It does work
• It does not work
• Reduce the risk of failures
• Reduce the cost of testing
30
What is a Test Case?
• Test Case is a simple pair of <input, expected outcome>
State-less systems: A compiler is a stateless system. to compile a program it
does not need to know about the programs it compiled previously.
• Test cases are very simple
• Outcome depends solely on the current input.
State-oriented: where the program outcome depends both on the
current state of the system and the current input.
ATM is a state oriented system
• Test cases are not that simple. A test case may consist of a sequences of
<input, expected outcome>
• The outcome depends both on the current state of the system and the current
input
ATM example:
• < check balance, $500.00 >,
• < withdraw, “amount?” >,
31
• < $200.00, “$200.00” >,
• < check balance, $300.00 >
Expected Outcome
• An outcome of program execution may include
• Value produced by the program
• State Change
• A sequence of values which must be interpreted together for the outcome
to be valid
• A test oracle is a mechanism that verifies the correctness of program
outputs.
• Generate expected results for the test inputs
• Compare the expected results with the actual results of execution of the
implementation under test (IUT)
32
The Concept of Complete Testing
• Complete or exhaustive testing means
“There are no undisclosed faults at the end of test phase”
• Complete testing is near impossible for most of the system
• The domain of possible inputs of a program is too large
• Valid inputs
• Invalid inputs
• The design issues may be too complex to completely test
• It may not be possible to create all possible execution environments of
the system
33
The Central Issue in Testing
Figure 1: A subset of the input domain exercising a subset of the program behavior
• Divide the input domain D into D1 and D2
• Select a subset D1 of D to test program P
• It is possible that D1 exercise only a part P1 of P
34
Testing Activities
Figure 2.Different activities in process testing
• Identify the objective to be tested
• Select inputs
• Compute the expected outcome
• Set up the execution environment of the program
• Execute the program
• Analyze the test results
35
Testing Level
• Unit testing
• Individual program units, such as
procedure, methods in isolation
• Integration testing
• Modules are assembled to construct
larger subsystem and tested
• System testing
• Includes wide spectrum of testing such
as functionality, and load
Figure 3: Development and testing phases in
• Acceptance testing the V model
• Customer’s expectations from the
system
36
Source of Information for Test
Selection
• Requirement and Functional Specifications
• Source Code
• Input and output Domain
• Operational Profile
• Fault Model
• Error Guessing
• Fault Seeding
• Mutation Analysis
37
White-box and Black-box Testing
• White-box testing a.k.a. structural
• Black-box testing a.k.a. functional
testing testing
• Examines source code with focus on: • Examines the program that is
• Control flow accessible from outside
• Data flow • Applies the input to a program and
• Control flow refers to flow of control observe the externally visible
from one instruction to another outcome
• Data flow refers to propagation of • It is applied to both an entire
values from one variable or constant to program as well as to individual
another variable program units
• It is applied to individual units of a • It is performed at the external
program
interface level of a system
• Software developers perform structural
testing on the individual program units • It is conducted by a separate
they write software quality assurance group
38
Thank you
???
39
