Software Quality
Assurance
(Lecture 14)
Dr. R. Mall
1
Organization of this
Lecture:
Introduction Quality Engineering.
Quality control and Quality
Assurance
ISO 9000
SEI CMM
Summary
2
Introduction
Traditional definition of
quality:
fitness of purpose,
a quality product does exactly
what the users want it to do.
3
Fitness of purpose
For software products,
fitness of purpose:
satisfaction of the
requirements specified in
SRS document.
4
Fitness of purpose
A satisfactory definition of
quality for many products:
a car, a table fan, a food mixer,
microwave oven, etc.
But, not satisfactory for software
products.
5
Introduction
Consider a software product:
functionally correct,
i.e. performs all functions as
specified in the SRS document,
but has an almost unusable
user interface.
cannot be considered as a
quality product. 6
Introduction
Another example:
a product which does
everything that users want.
but has an almost
incomprehensible and
unmaintainable code.
7
Modern view of quality
Associates several quality factors with a
software product :
Correctness
Reliability
Efficiency (includes efficiency of resource utilization)
Portability
Usability
Reusability
Maintainability
8
Correctness
A software product is correct,
if different requirements as
specified in the SRS document
have been correctly
implemented.
Accuracy of results.
9
Portability
A software product is said to be
portable,
if it can be easily made to work in
different operating systems,
in different machines,
with other software products, etc.
10
Reusability
A software product has good
reusability,
if different modules of the
product can easily be reused
to develop new products.
11
Usability
A software product has good
usability,
if different categories of users
(i.e. both expert and novice users)
can easily invoke the functions of
the product.
12
Maintainability
A software product is maintainable,
if errors can be easily corrected as and
when they show up,
new functions can be easily added to
the product,
functionalities of the product can be
easily modified, etc.
13
Software Quality
Management System
Quality management system
(or quality system):
principal methodology used by
organizations to ensure that
the products have desired
quality.
14
Quality system
A quality system consists of the
following:
Managerial Structure
Individual Responsibilities.
Responsibility of the
organization as a whole.
15
Quality system
Every quality conscious organization has
an independent quality department:
performs several quality system activities.
needs support of top management.
Without support at a high level in a
company,
many employees may not take the quality
system seriously.
16
Quality System Activities:
Auditing of projects
Development of:
standards, procedures, and guidelines,
etc.
Production of reports for the top
management
summarizing the effectiveness of the
quality system in the organization.
Review of the quality system itself.
17
Quality system
A good quality system must be well
documented.
Without a properly documented
quality system,
application of quality procedures become
ad hoc,
results in large variations in the quality of
the products delivered.
18
Quality system
An undocumented quality system:
sends clear messages to the staff about the
attitude of the organization towards quality
assurance.
International standards such as ISO
9000 provide:
guidance on how to organize a quality
system.
19
Evolution of Quality
Systems
Quality systems have evolved:
over the last five decades.
Prior to World War II,
way to produce quality products:
inspect the finished products
eliminate defective products.
20
Evolution of Quality
Systems
Since that time,
quality systems of
organizations have undergone
four stages of evolution.
21
Evolution of Quality
Systems
22
Evolution of Quality
Systems
Initial product inspection method :
gave way to quality control (QC).
Quality control:
not only detect the defective products
and eliminate them
but also determine the causes behind
the defects.
23
Quality control (QC)
Quality control aims at correcting
the causes of errors:
not just rejecting defective products.
Statistical quality control
quality of the output of the process is
inferred using statistical methods
in stead of inspection or testing of all
products
24
Quality control (QC)
The next breakthrough,
development of quality assurance
principles
25
Quality assurance
Basic premise of modern quality
assurance:
if an organization's processes are
good and are followed rigorously,
the products are bound to be of
good quality.
26
Quality assurance
All modern quality paradigms
include:
guidance for recognizing, defining,
analyzing, and improving the
production process.
27
Total quality management
(TQM)
Advocates:
continuous process
improvements through process
measurements.
28
Business Process
reengineering
A term related to TQM.
Process reengineering goes a
step further than quality
assurance:
aims at continuous process
improvement.
29
Business Process
reengineering
Our focus is reengineering of
the software process.
Whereas BPR aims at
reengineering the way business is
carried out in any organization
not just software development
organizations.
30
Total quality management
(TQM)
TQM goes beyond documenting
processes
optimizes them through redesign.
Over the years the quality paradigm
has shifted:
from product assurance to process
assurance.
31
ISO 9000
ISO (international Standards
Organization):
a consortium of 63 countries
established to formulate and foster
standardization.
ISO published its 9000 series of
standards in 1987.
32
What is ISO 9000
Certification?
ISO 9000 certification:
serves as a reference for contract
between independent parties.
The ISO 9000 standard:
specifies guidelines for
maintaining a quality system.
33
What is ISO 9000
Certification?
ISO 9000 specifies:
guidelines for repeatable and high
quality product development.
Also addresses organizational aspects
responsibilities, reporting, procedures,
processes, and resources for
implementing quality management.
34
ISO 9000
A set of guidelines for the
production process.
not directly concerned about the
product it self.
a series of three standards:
ISO 9001, ISO 9002, and ISO 9003.
35
ISO 9000
Based on the premise:
if a proper process is followed
for production:
good quality products are bound
to follow.
36
ISO 9001:
Applies to:
organizations engaged in design,
development, production, and
servicing of goods.
applicable to most software
development organizations.
37
ISO 9002:
ISO 9002 applies to:
organizations who do not design products:
but are only involved in production.
Examples of this category of industries:
steel or car manufacturing industries
buy the product and plant designs from external
sources:
only manufacture products.
not applicable to software development
organizations.
38
ISO 9003
ISO 9003 applies to:
organizations involved only in
installation and testing of the
products.
39
ISO 9000 for Software
Industry
ISO 9000 is a generic standard:
applicable to many industries,
starting from a steel manufacturing industry to a
service rendering company.
Many clauses of ISO 9000 documents:
use generic terminologies
very difficult to interpret them in the
context of software organizations.
40
Software vs. other
industries
Very difficult to interpret many
clauses for software industry:
software development is radically
different from development of
other products.
41
Software vs. other
industries
Software is intangible
therefore difficult to control.
It is difficult to control anything that we cannot see and
feel.
In contrast, in a car manufacturing unit:
we can see a product being developed through stages
such as fitting engine, fitting doors, etc.
one can accurately tell about the status of the product
at any time.
Software project management is an altogether
different ball game.
42
Software vs. other
industries
During software development:
the only raw material consumed is data.
For any other product development:
Lot of raw materials consumed
e.g. Steel industry consumes large volumes of iron
ore, coal, limestone, etc.
ISO 9000 standards have many clauses
corresponding to raw material control .
not relevant to software organizations.
43
Software vs. other
industries
Radical differences exist between
software and other product
development,
difficult to interpret various clauses
of the original ISO standard in the
context of software industry.
44
ISO 9000 Part-3
ISO released a separate document
called ISO 9000 part-3 in 1991
to help interpret the ISO standard for
software industry.
At present,
official guidance is inadequate
45
Why Get ISO 9000
Certification?
Several benefits:
Confidence of customers in an
organization increases
if organization qualified for ISO
9001 certification.
This is especially true in the
international market.
46
Why Get ISO 9000
Certification?
Many international software
development contracts insist:
development organization to have
ISO 9000 certification.
47
Why Get ISO 9000
Certification?
Requires:
a well-documented software
production process to be in place.
contributes to repeatable and higher
quality software.
Makes development process:
focussed, efficient, and cost-effective
48
Why Get ISO 9000
Certification?
Points out the weakness of an
organizations:
recommends remedial action.
Sets the basic framework:
for development of an optimal
process and TQM.
49
How to Get ISO 9000
Certification?
An organization intending to obtain
ISO 9000 certification:
applies to a ISO 9000 registrar for
registration.
ISO 9000 registration process
consists of several stages.
50
How to Get ISO 9000
Certification?
Application stage:
Applies to a registrar for
registration.
Pre-assessment:
the registrar makes a rough
assessment of the organization.
51
How to Get ISO 9000
Certification?
Document review and adequacy
audit:
process and quality-related
documents.
the registrar reviews the
documents
makes suggestions for
improvements.
52
How to Get ISO 9000
Certification?
Compliance audit: the
registrar checks
whether the suggestions made by it
during review have been complied.
53
How to Get ISO 9000
Certification?
Registration:
The registrar awards ISO 9000
certificate after successful
completions of all previous phases.
Continued surveillance:
The registrar continues monitoring
the organization periodically.
54
ISO 9000 Certification
An ISO certified organization
can use the certificate for corporate
advertizements
cannot use the certificate to advertize
products.
ISO 9000 certifies organization's process
not any product of the organization.
An organization using ISO certificate
for product advertizements:
risks withdrawal of the certificate.
55
Summary of ISO 9001
Requirements
Management responsibility(4.1):
Management must have an
effective quality policy.
The responsibility and authority of
all those whose work affects
quality:
must be defined and documented.
56
Management
responsibility(4.1)
Responsibility of the quality system.
independent of the development
process,
can work in an unbiased manner.
The effectiveness of the quality
system:
must be periodically by audited.
57
Quality system (4.2) and contract
reviews (4.3):
A quality system must be
maintained and documented.
Contract reviews (4.3):
Before entering into a contract, an
organization must review the
contract
ensure that it is understood,
organization has the capability for
carrying out its obligations. 58
Design control (4.4):
The design process must be
properly controlled,
this includes controlling coding
also.
A good configuration control
system must be in place.
59
Design control (4.4):
Design inputs must be verified as
adequate.
Design must be verified.
Design output must be of required
quality.
Design changes must be controlled.
60
Document control (4.5):
Proper procedures for
document approval, issue and
removal.
Document changes must be
controlled.
use of some configuration
management tools is necessary.
61
Purchasing (4.6):
Purchased material, including
bought-in software:
must be checked for conforming to
requirements.
62
Purchaser Supplied Products (4.7):
Material supplied by a
purchaser,
for example,
client-provided software must
be properly managed and
checked.
63
Product Identification (4.8):
The product must be identifiable
at all stages of the process.
In software development context
this means configuration
management.
64
Process Control (4.9) :
The development must be properly
managed.
Quality requirements must be
identified in a quality plan.
65
Inspection and Testing
(4.10) :
In software terms this requires
effective testing i.e.,
unit testing, integration testing
and system testing.
Test records must be
maintained.
66
Inspection, measuring and test
equipment(4.11):
If integration, measuring, and test
equipments are used,
must be properly maintained and
calibrated.
67
Control of nonconforming product
(4.13) :
In software terms,
keeping untested or faulty
software out of released
product,
or other places whether it might
cause damage.
68
Corrective Action (4.14) :
This is both about correcting errors
when found,
investigating why they occurred
improving the process to prevent further
occurrences.
If an error reoccurs despite the quality
system,
the system needs improvement.
69
Handling (4.15) and Quality audits
(4.17):
Handling (4.15) Deals with:
storage, packing, and delivery
of the software product.
Quality Audits (4.17) :
quality system audit must be
carried out to ensure its
effectiveness.
70
Training (4.18) :
Training needs must be
identified and met.
Most items of ISO standard
are largely common sense.
71
Salient features of ISO 9001
requirements:
All documents concerned with the
development of a software product
should be properly managed,
authorized, and controlled.
Proper plans should be prepared
progress against these plans should
be monitored.
72
Salient features of ISO 9001
requirements:
Important documents independently
checked and reviewed:
for effectiveness and correctness.
The product should be tested :
against specification.
Several organizational aspects:
e.g., management reporting of the
quality team.
73
Shortcomings of ISO 9001
Certification (1)
ISO 9000 requires a production
process to be adhered to:
but does not guarantee the
process to be of high quality.
Does not give any guideline for
defining an appropriate process.
74
Shortcomings of ISO 9001
Certification (2)
ISO 9000 certification process
not fool-proof
no international accredition agency
exists.
likely variations in the norms of
awarding certificates:
among different accredition agencies
and among the registrars.
75
Shortcomings of ISO 9001
Certification (3)
Organizations qualifying for ISO
9001 certification:
tend to downplay domain expertise.
tend to believe that since a good
process is in place,
any engineer is as effective as any other
engineer in doing any particular activity
relating to software development.
76
Shortcomings of ISO 9001
Certification (4)
In manufacturing industry
clear link between process quality and
product quality
once a process is calibrated:
can be run again and again producing
quality goods
Software development is a creative
process:
individual skills and experience is
significant
77
Shortcomings of ISO 9001
Certification (5)
Many areas of software development
are very specialized:
special expertize and experience
(domain expertize) required.
ISO 9001
does not automatically lead to
continuous process improvement,
does not automatically lead to TQM.
78
Shortcomings of ISO 9001
Certification (6)
ISO 9001 addresses mostly management
aspects.
Techniques specific to software development
have been ignored
Configuration management
Reviews
Release builds
Problem Notification system
Intranets
79
SEI Capability Maturity
Model
Developed by Software Engineering
Institute (SEI) of the Carnegie
Mellon University, USA:
to assist the U.S. Department of
Defense (DoD) in software acquisition.
The rationale was to include:
likely contractor performance as a factor
in contract awards.
80
SEI Capability Maturity
Model
Major DoD contractors began CMM-
based process improvement initiatives:
as they vied for DoD contracts.
SEI CMM helped organizations:
Improve quality of software they developed
Realize adoption of SEI CMM model had
significant business benefits.
Other organizations adopted CMM.
81
SEI Capability Maturity
Model
In simple words,
CMM is a model for apprising the
software process maturity of a
contractor into different levels.
Can be used to predict the most likely
outcome to be expected from the next
project that the organization
undertakes.
82
SEI Capability Maturity
Model
Can be used in two ways:
Capability evaluation
Software process assessment.
83
Capability Evaluation
Provides a way to assess the
software process capability of
an organization
Helps in selecting a contractor
Indicates the likely contractor
performance
84
Software Process
Assessment
Used by an organization to assess
its current process:
Suggests ways to improve the
process capability.
This type of assessment is for
purely internal use.
85
SEI Capability Maturity
Model
The SEI CMM classifies software
development industries into:
Five maturity levels.
Stages are ordered so that improvements
at one stage provide foundations for the
next
Based on the pioneering work of Philip
Crosby
86
SEI Capability Maturity
Model
Optimizing (5)
Managed (4)
Defined (3)
Repeatable (2)
Initial (1)
87
Level 1: (Initial)
Organization operates
without any formalized process or
project plans
An organization at this level is
characterized by
ad hoc and often chaotic activities.
88
Level 1: (Initial)
Software production processes are
not defined,
different engineers follow their own
process
development efforts become chaotic.
The success of projects depend on
individual efforts and heroics.
89
Level 2: (Repeatable)
Basic project management practices
tracking cost, schedule, and
functionality are followed.
Size and cost estimation techniques
function point analysis, COCOMO, etc.
used.
Production process is ad hoc
not formally defined
also not documented.
90
Level 2: (Repeatable)
Process used for different projects
might vary between projects:
earlier success on projects with similar
applications can be repeated.
Opportunity to repeat process exist
when a company produces a family of
products.
91
Level 3: (Defined)
Management and
development activities:
defined and documented.
Common organization-wide
understanding of activities,
roles, and responsibilities.
92
Level 3: (Defined)
The process though defined,
process and product qualities
are not measured.
ISO 9001 aims at achieving
this level.
93
Level 4: (Managed)
Quantitative quality goals for products
are set.
Software process and product quality are
measured:
The measured values are used to control the
product quality.
Results of measurement used to evaluate
project performance
rather than improve process.
94
Level 4: (Managed)
Organization sets quantitative
quality goals
World-wide about 100
organizations assessed at this
level.
95
Level 5: (Optimizing)
Statistics collected from process and
product measurements are
analyzed:
continuous process improvement
based on the measurements.
Known types of defects are prevented
from recurring by tuning the process
lessons learned from specific projects
incorporated into the process
96
Level 5: (Optimizing)
Identify best software engineering
practices and innovations:
tools, methods, or process are
identified
transferred throughout the organization
World-wide about 50 organizations have
been assessed at this level.
97
Key Process Areas
Each level is associated with a
key process area (KPA) identifies
where an organization at the
previous level must focus to reach
this level
98
Level 2 KPAs
Software project planning
Size, cost, schedule.
project monitoring
Configuration management
Subcontract management
99
Level 3 KPAs
Process definition and
documentation
Reviews
Training program
100
Level 4 KPAs
Quantitative measurements
Process management
101
Level 5 KPAs
Defect prevention
Technology change management
Process change management
102
Comparison between ISO 9001
and SEI CMM
ISO 9001 awarded by an international
standards body
can be quoted in official documents and
communications
SEI CMM assessment is purely for
internal use.
103
Comparison between ISO 9001
and SEI CMM
SEI CMM was developed specifically
for software industry:
addresses many issues specific to
software industry.
SEI goes beyond quality assurance
aims for TQM
ISO 9001 correspond to SEI level 3.
104
Comparison between ISO 9001
and SEI CMM
SEI CMM provides a list of key areas
on which to focus to take an organization
from one level to the other
Provides a way for gradual quality
improvements over several stages.
e.g trying to implement a defined process
before a repeatable process:
counterproductive as managers are
overwhelmed by schedule and budget pressure.
105
Remarks on Quality Model
Usage
Highly systematic and measured approach to
software development process suits certain
circumstances
negotiated software, safety-critical software, etc
What about small organizations?
Typically handle applications such as internet, e-comm.
without an established product range,
without revenue base, experience on past projects, etc.
CMM may be incompatible
106
Small Organizations
Small organizations tend to believe:
We are all competent people hired to do a job,
we can’t afford training
We all communicate with one another
Osmosis works because we are so close
We are all heroes
We do what needs to be done
Therefore rules do not apply to us
107
Small Organizations
Often have problems:
Undocumented requirements
Inexperienced managers
Documenting the product
Resource allocation
Training
Peer reviews
108
Small Organizations
A two week CMM-based appraisal is
probably excessive:
Small organizations need to operate more
efficiently at lower levels of maturity
Must first fluorish if eventually they are to
mature
109
Personal Software Process
(PSP)
Based on the work of Humphrey
PSP is a scaled down version of industrial
software process
suitable for individual use
Even CMM assumes that engineers use
effective personal practices
110
Personal Software Process
(PSP)
A process is the set of steps for doing a
job
The quality and productivity of an
engineer
largely determined by his process
PSP is framework that
helps software engineers to measure and
improve the way they work.
111
Personal Software Process
(PSP)
Helps developing personal skills and
methods
Estimating and planning method
Shows how to track performance against
plans
Provides a defined process
can be fine tuned by individuals
Recognizes that a process for individual use is
different from that necessary for a team project.
112
Time Management
Track the way you spend time
Boring activities seem longer then actual
Interesting activities seem short
Record time for
Designing
Writing code
Compiling
Testing
113
Personal Software Process
(PSP)
Planning
Design
Code Logs
Compile
Test Project plan
Postmortem
summary
114
PSP-Planning
Problem definition
Estimate max, min, and total LOC
Determine minutes/LOC
Calculate max,min, and total development
times
Enter the plan data in project plan
summary form
record the planned time in Log
115
PSP-Design
Design the program
Record the design in specified format
Record the Design time in time recording
log
116
PSP-Code
Implement the design
Use a standard format for code text
Record the coding time in time recording
log
117
PSP-Compile
Compile the program
Fix all the defects
Record compile time in time recording log
118
PSP-Test/Postmortem
Test
Test the program
Fix all the defects found
Record testing time in time recording log
Postmortem
Complete project plan summary form with
actual time and size data
Record postmortem time in time record
119
Personal Software Process
(PSP)
PSP 3 Personal process
evolution
PSP 2 Personal quality management
Design and code reviews
PSP 1 Personal planning
Time and schedule
PSP 0 Personal measurement
Basic size measures
120
Six Sigma
Six sigma is a quantitative approach to
eliminate defects
Applicable to all types of industry - from
manufacturing, product development, to
service
The statistical representation of Six Sigma
quantitatively describes
how a process is performing
121
Six Sigma
To achieve six sigma
a process must not produce more than 3.4
defects per million opportunities.
5 Sigma -> 230 defects per million
4 Sigma -> 6210 defects per million
Six sigma methodologies
DMAIC (Define, Measure, Analyze, Improve,
Control)
DMADV: (Define, Measure, Analyze, Design,
Verify) 122
Six Sigma Methodologies
The methodologies are implemented by
Green belt and Black belt workers
Supervised by Master black belt worker
Pareto Chart:
Simple bar chart to represent defect data
Identify the problems that occurs with greatest
frequency
or incur the highest cost
123
Summary
Evolution of quality system:
product inspection
quality control
quality assurance
total quality management (TQM)
Quality paradigm change:
from product to process
124
Summary
ISO 9000:
basic premise:
if a good process is followed
good products are bound to follow
provides guidelines for
establishing a quality system.
125
Summary
ISO 9000
series of three standards
9001, 9002, and 9003
9001 is applicable to software
industry
126
Summary
SEI CMM
developed specially for software
industry
classifies software organizations
into five categories.
According to the maturity of their
development process.
127
Current Trends
Many organizations have already tuned
their process for
Budget,
Schedule, and
Quality product.
Competition is challenging them to:
Reduce time for delivery
Adopt Six-Sigma methodology
128
