77R-

15

E
QUALI
TYASSURANCE/
QUALI
TY
PL
CONTROLFORRI
SKMANAGEMENT
M
SA
 AACE® International Recommended Practice No. 77R-15

QUALITY ASSURANCE/QUALITY CONTROL FOR RISK

E
MANAGEMENT
TCM Framework: 7.6 – Risk Management
PL Rev. June 3, 2016
Note: As AACE International Recommended Practices evolve over time, please refer to www.aacei.org for the latest
revisions.
M
SA

Contributors:
Disclaimer: The opinions expressed by the authors and contributors to this recommended practice
are their own and do not necessarily reflect those of their employers, unless otherwise stated.

David C. Brady, P.Eng. DRMP (Author) Dr. Alexia A. Nalewaik (Author)

John K. Hollmann, PE CCP CEP DRMP (Author) James E. Arrow, DRMP

Copyright © AACE® International AACE® International Recommended Practices

 ®
AACE International Recommended Practice No. 77R-15
QUALITY ASSURANCE/QUALITY CONTROL FOR RISK
MANAGEMENT
TCM Framework: 7.6 – Risk Management

June 3, 2016

INTRODUCTION

SCOPE

This recommended practice (RP) of AACE International shall define the expectations, requirements, and practices
for developing a risk management (RM) quality program. This RP will identify the quality assurance (QA) process,
quality control (QC) process, and related risk management auditing methods for a capital asset portfolio, program
[2]
or project management organization. The RP expands on TCM sections 11.4, Quality and Quality Management
and 11.5, Value Management and Value Improving Practices (VIPs) as applicable to the Risk Management process
as described in TCM section 7.6. It includes practices for planning how to develop and implement a quality
assurance and quality control platform and a proper auditing program for risk management within a project, a
program, or a portfolio. Emphasis will be on a continuous improvement suite of tools aligned with quality
management principles applied to the risk management process and considers the cost of quality. This RP is not

E
about governance, risk management and/or compliance for an overall TCM process or management of an
enterprise as a whole. It only applies to the risk management process or program within that framework.
PL
The process of risk management interfaces with investment decision making (TCM section 3.3) in consideration of
risk prior to initiation of projects, and change management (TCM section 10.3) in which risk cost and time
allowances (e.g. contingency and reserves) are assessed and managed during program or project execution. While
this RP does not cover those processes per se, the user should ensure that the quality program interfaces
appropriately with them.
M
PURPOSE

This RP is intended to provide guidelines (i.e. not a standard) for developing and using a quality assurance and
SA

quality control program applicable to risk management.

This recommend practice is intended to be a model that can be used as a basis for planning quality improvement
programs for risk management; to help risk, project, and asset program managers discern whether their risk
management program is working; and identify where performance improvements are required. It will provide a
foundation for not just addressing the overall risk management QA/QC program but specifically developing QA/QC
[4]
ideas for risk treatment plans as described in RP 63R-11, Risk Treatment . Ideally, the risk management process
provides an opportunity for all stakeholders and contracting parties to work together and manage risk (i.e. threats
or opportunities) in a way that increases the probability of success of the portfolio, program or project. The
implementation of all or part of this RP will depend on the size and complexity of the program or project but the
basic processes described should be used in all cases.

BACKGROUND

The Sarbanes-Oxley (SOX) Act of 2002 in the United States, and similar laws in other countries, resulted in a
significant increase in financial governance required of public companies. These regulations focus on the accuracy
and completeness of financial reporting. This directly affects the TCM process which is largely focused on
improving the financial success of capital investment portfolios, programs and projects (among other measures of
success). To the extent that uncertainty (i.e. risk) in capital asset and project management affects the accuracy of
company financial reporting, risk management’s performance and hence governance is crucial. Return-on-capital
Copyright © AACE® International AACE® International Recommended Practices
77R-15: Quality Control/Quality Assurance for Risk Management 2 of 24

June 3, 2016

and similar metrics are key performance metrics for most public companies and governments have similar goals to
optimize the value obtained from investment of company or government revenues.

The starting point for a risk management quality process is to understand and implement a TCM process upon
which it will be applied. As stated in TCM 11.4.1:
“There are many definitions and perceptions of what quality and quality management are. In simple terms,
quality in TCM is conformance of an asset (product, service, process, etc.) with requirements and expectations.
Quality management is what an enterprise does to ensure that its assets meet these requirements and
expectations. In TCM, quality management is not a separate process; TCM, including strategic asset
management and project control, are quality processes. The TCM processes, as discussed in Section 2.1.2, are
based on the plan, do, check, act (PDCA) model; this model is a time honored quality management approach
sometimes called the Deming or Shewhart cycle.”

TCM is aligned with the International Organization of Standardization’s (ISO) eight principles that guide quality
[10]
management practice . These principles and how TCM (11.4.1) and this RP address the principles are shown in
Table 1:

E
ISO Principles TCM and RP Approach to the Principles

Customer focus. TCM and risk management elicit and identify stakeholder risk expectations,
requirements, and how they define success in order to determine how the
PL
QA/QC program will address them.

Leadership. TCM and risk management assure that the risk program addresses corporate
strategic objectives, the risk management’s objectives are communicated,
and buy-in and support are obtained.

Involvement of people. TCM and risk management address stakeholder management, resource
M
management and team development through the use of risk management
organization charts, QA/QC RASCI tables, communication matrices and
similar tools.

Process approach. TCM, including its risk management process (7.6), provide a framework for
SA

governance that can be used to assure alignment with the company’s

current quality control/assurance program.

System approach to management. The TCM and risk management process maps address management as an
integrated, quality management system (e.g. integration of risk management
and project control, etc.).

Continuous improvement. TCM and risk management processes include measurement and assessment
of performance, benchmarking of processes and practices, corrective
actions, and feedback loop to improve future practice and outcomes, i.e. a
quality management program.

Factual approach to decision making. TCM and risk management are predicated on decision analysis that is
supported by objective data obtained through performance measurement
and empirical data management as appropriate. In consideration of
uncertainty where subjectiveness is applied or evident in risk management,
it is noted and treated as such (e.g. recognizing bias, uncertainty, etc.).

Mutually beneficial supplier relationships. TCM and risk management are integrated processes that include suppliers
among the stakeholders whose expectations and requirements are
addressed.
Table 1 – ISO Quality Management Principles and their Alignment with the TCM Framework

Copyright © AACE® International AACE® International Recommended Practices

77R-15: Quality Control/Quality Assurance for Risk Management 3 of 24

June 3, 2016

RECOMMENDED PRACTICE

As a guideline, each organization should build upon this RP to develop and manage its own risk management
quality program.

Some examples of enterprise quality management initiatives that align with a risk management quality program
include:
• Total quality management (TQM)
• Six Sigma
• Stage or phase-gate project systems

In all cases, these are built on a managed integrated processes and systems. The intent of this RP is to focus on the
risk management process as defined in section 7.6 of the TCM Framework and assuring, auditing and controlling
the quality of its four main steps: plan, assess, treat and control. To do this, the RP provides some guideline
matrices or checklists that can be used to evaluate the content of a risk program’s quality management. In
essence, quality is about meeting requirements. Scorecards or similar measurements of the how well those
requirements have been met are a necessary element of assuring, auditing and controlling quality.

E
Per TCM Framework section 7.6, the goal of risk management is to “increase the probability that a planned asset,
project or portfolio achieves its objectives.” The quality of a risk management program ultimately is defined by
PL
how well it meets this goal. Demonstrating that this goal has been achieved is difficult, since the success of an
outcome without risk management is unknown. However, one can readily monitor success over time for a
portfolio by measuring improvement trends in such things as accuracy in a cost and schedule goal. This is not to
confuse accuracy with quality (e.g. the accuracy of an estimate for a risky project will always be less than that for a
non-risky project regardless of the quality of the estimating process) but the takeaway is that success of risk
management is measurable.
M
GENERAL REQUIREMENTS OF A RISK MANAGEMENT QUALITY PROGRAM
SA

Key Quality Program Concepts: Assurance, Control and Governance

The following are key concepts and terminology relative to a quality management program for risk management:
• Governance – In the TCM Framework, assuring the alignment of the portfolio, program or project risk
management process objectives with the strategy of the overall enterprise. To paraphrase one author, it
is the rules, laws and processes to guide the successful management of a portfolio, program or project.
[1]
• Quality – Conformance to established requirements (not a degree of goodness). (10S-90 )
• Quality is the characteristics of a product that allow it to meet the expectations of the project.
• Quality is all about fulfilling requirements.

• Quality control – Inspection, test, evaluation or other necessary action to verify that a product, process, or
service conforms to established requirements and specifications. (10S-90)

• Quality assurance – All those planned or systematic actions necessary to provide adequate confidence
that a product, process, or service will conform to established requirements. (10S-90)

• Quality audit – A formal, independent examination with intent to verify conformance with the acceptance
criteria. An audit does not include surveillance or inspection for the purpose of process control or product
acceptance. (10S-90)

Copyright © AACE® International AACE® International Recommended Practices

77R-15: Quality Control/Quality Assurance for Risk Management 4 of 24

June 3, 2016

Elements of Quality Measurement and Audit

Given that the objective or requirement of risk management in TCM is to increase the probability of achieving
objectives, and quality is conformance to requirements, both precision and accuracy are important to quality
measures. Audits are generally more qualitative in nature. The following are key concepts related to
measurements and audits in a quality program for risk management:
• Measures  Quantification of any attribute of a process or deliverable.

• Metrics  Qualitative (e.g. more or less) or quantitative (e.g. numerical value) measures indicative of
quality; i.e. a measure of conformance to a requirement, objective or baseline.

• Key performance indicators  Selected (i.e. key) metrics considered to be reliably indicative of
performance relative to strategic objectives.

• Precision  Consistency of repeated measured values regardless of the values nearness to the true value.

E
• Accuracy  Nearness of measured values to the true values.

• Variances:
PL
• Random variations might be normal (i.e. noise), depending on the processes used but the variance
has no discernible trend and when significant are generally unexpected; in the worst case, this can
indicate a process that is out of control.
• Known or predictable variances are those known to exist in the process because of particular
characteristics of the process or its outputs. These are generally unique to a particular application.
They may display a “trend” (e.g. increase or decrease over time) that indicates improving or
M
deteriorating performance.
• Variances that are always present in the process across all applications. The process itself will have
inherent variability that is perhaps caused by human mistakes, machine variations or malfunctions,
the environment, and so on. Variances that do not fall within the acceptable range usually require
SA

process improvement. Decisions to change the process always require management approval as part
of governance.

• Quality audit: a structured, independent review to determine whether process activities and deliverables
comply with enterprise, program and project requirements, policies, standards, processes, and
procedures as applicable. The objectives of a quality audit are:
• Identify the enterprise, program and project requirements, policies, standards, processes, and
procedures against which the activities and deliverables are being measured.
• Identify all the gaps/shortcomings.
• Identify any overlaps/duplication of effort.
• Identify all the good/best practices being implemented.
• Share the good practices introduced or implemented in similar programs or projects in the
organization and/ or industry.
• Proactively offer assistance in a positive manner to improve implementation of processes to help the
team meet its goals.
• Highlight contributions of each audit in the lessons learned repository of the organization.

The subsequent effort to correct any deficiencies should result in a reduced cost of quality and an increase in
sponsor or customer acceptance of the products of the process. Quality audits may be scheduled or random and
may be conducted by internal or external auditors. Quality audits can confirm the implementation of approved

Copyright © AACE® International AACE® International Recommended Practices

77R-15: Quality Control/Quality Assurance for Risk Management 5 of 24

June 3, 2016

change requests including corrective actions, defect repairs, and preventive actions. Experienced specialists
generally perform quality audits; the specialist’s job is to produce an independent evaluation of the quality
process. Some organizations are large enough to have their own quality assurance departments or quality
assurance teams; others might have to hire contract personnel to perform this function. Internal quality assurance
teams report results to the program, project team, and management team of the organization as appropriate.
External quality assurance teams report results to the customer, i.e. the entity that hired them.

Quality assurance and control focus on consistent (predictable variance) performance of a process or practice, i.e.
improving precision in measures. However, over time, continuous improvement of the process also seeks to
improve the accuracy of its outcomes given the inherent risk situation. Keep in mind that accuracy itself is not a
measure of quality; it is largely an artifact of risk (i.e. the fact that one project has more risk and a wider accuracy
range than another project does not mean that the riskier project’s management or deliverables are of lesser
quality).

Risks in Risk Management Quality Programs

Governance of risk management must consider the enterprise’s appetite for risks and expectations for innovation

E
and dynamic capturing of opportunities. A potential risk of excessive governance and QA/QC is that these sub-
processes can contribute to bureaucracy (e.g. measurement for measurement’s sake) and paralysis (e.g. failure to
act for fear of deviation in metrics). In establishing a quality program, its flexibility to deal with evolving
PL
organizational and process maturity, changing environments, events and so on should be considered. Similarly, the
cost of quality must be considered, i.e. at some point the cost of quality management may exceed its benefit.

ELEMENTS OF A QUALITY PROGRAM

M
In TCM, quality is the conformance to and improvement of internal processes and procedures in order to meet
stakeholder requirements and thus the focus is internal to the enterprise and its portfolio, program and project
management.
SA

The plan, do, check, act (PDCA) cycle as shown in Figure 1 is the framework for TCM because it is:
1. Time-proven and widely accepted as a valid management model
2. Quality driven

PLAN
(plan activities)

ACT
(evaluate PDCA DO
measures, act Cycle (perform activities)
upon variances)

CHECK
(measure
performance
of activities)
Figure 1 – The Plan, Do, Check, Act Cycle for Improvement

For the risk management process as described in TCM Section 7.6, and summarized in Figure 2, the PDCA process is
reflected in four steps - plan, assess, treat and control. While each enterprise, program and project will develop its

Copyright © AACE® International AACE® International Recommended Practices