Workshop

Managing Analytical
Quality
Handout Materials
Problem Sets and Tools
Method Decision Chart
Sigma-metric QC Selection Tool
Charts of Operating Specifications
Normalized OPSpecs Charts
Total QC Strategies

James O. Westgard, PhD

Westgard QC, Inc.
Madison, WI
www.westgard.com

Copyright Westgard QC, Inc. 2009 http://www.westgard.com 1

 Plan for Managing Analytical Quality

(1a) Intended (1) Define Quality (1b) CLIA PT

Clinical Use, Dint Goals, Dint, TEa Criteria, TEa

(2a) Evaluate Lab (3a) FDA 510k

(2) Select Method
Needs/Usage clearance

(3) Validate Method (3b) Manufacturer’s

Performance (CV,bias) Claims

(4) Validate QC Design (5a) Manufacturer’s

(rules, N, F) Risk Analysis
(8) Improve TQC
Effectiveness
(5) Formulate (5b) Lab Evaluation
Total QC Strategy Of Residual Risk
(7) Monitor TQC
Effectiveness (f)
(6) Implement Right
TQC Right

(9) Validate
Test Results

Reference: Westgard JO. Assuring the Right Quality Right: Good Laboratory
Practices for Verifying the Attainment of the Intended Quality of Test Results.
Madison WI:Westgard QC, Inc., 2007.

Copyright Westgard QC, Inc. 2009 http://www.westgard.com 2

 Problem Set #1. Construction of a Method Decision
Chart for TEa=10.0%

10.0

9.0
Observed Inaccuracy, %Bias

8.0

7.0
6.0

5.0
4.0

3.0

2.0

1.0

0.0
0.0 1.0 2.0 3.0 4.0 5.0

Criterion Y-int
Observed X-int
Imprecision, Sigma
%CV
Bias + 2s TEa TEa/2 2
Bias + 3s TEa TEa/3 3
Bias + 4s TEa TEa/4 4
Bias + 5s TEa TEa/5 5
Bias + 6s TEa TEa/6 6

Copyright Westgard QC, Inc. 2009 http://www.westgard.com 3

 Problem Set #2. Judging performance using a
Method Decision Chart

Method Decision Chart TEa=10%

10.0

9.0
Observed Inaccuracy, %Bias

8.0

7.0

6.0

5.0

4.0

3.0

2.0 World
1.0 Class

0.0
0.0 1.0 2.0 3.0 4.0 5.0
Observed Imprecision, %CV

Method Bias CV Sigma

A 0.0% 2.0%
B 2.0% 2.0%
C 0.0% 3.0%
D 3.0% 3.0%

Copyright Westgard QC, Inc. 2009 http://www.westgard.com 4

 Problem Set #3. Selection of QC Procedures using
Sigma-metrics QC Selection Tool

Sigma Scale
1.65
1.0
2.65 3σ 3.65 4σ 4.65 5σ 5.65 Pfr Ped N R

13s /2of32s /R4s /31s /6x

0.9
0.07 ----- 6 1
Probability for Rejection (P)

13s /22s /R4s /41s /8x

0.8
0.03 ----- 4 2
0.7 1 /2 /R /4
3s 2s 4s 1s
0.03 ----- 4 1
0.6 1
2.5s
0.5 0.04 ----- 4 1
12.5s
0.4 0.03 ----- 2 1
13s /22s /R4s
0.3
0.01 ----- 2 1
13s
0.2
0.00 ----- 2 1
0.1 13.5s
0.00 ----- 2 1
0.0
0.0 1.0 2.0 3.0 4.0
Systematic Error (SE, multiples of s)

Method Bias CV Sigma QC

A 0.0% 2.0%
B 2.0% 2.0%
C 0.0% 3.0%
D 3.0% 3.0%

Copyright Westgard QC, Inc. 2009 http://www.westgard.com 5

 Directions for Using the Sigma-Metrics QC Selection Tool

1. Identify the test and method for the intended application.

2. Define a quality requirement in form of allowable Total Error (TEa).

3. Identify a clinical decision concentration (Xc) that is of interest for the

test and its application with your patient population.

4. Decide whether to use 2 or 3 levels of controls.

5. Determine the imprecision of your method (SD, CV%) at or near Xc.

6. Determine the inaccuracy of your method (Bias, Bias%) at or near Xc.

7. Calculate the sigma-metric from the defined TEa and the observed
imprecision and the observed inaccuracy.
a. Sigma = (TEa% - Bias%)/CV% where all terms are in %, or
b. Sigma = (TEa – Bias)/SD where all terms are in concentrations
units.

8. Calculate the critical Systematic Error (∆SE) that needs to be detected,

which is simply Sigma – 1.65.

9. Utilize the Sigma-metrics tool for either 2 or 3 levels of control,

whichever is chosen for your application. Locate either the Sigma value
on the x-scale at the top of the graph, or ∆SE value on the x-scale at the
bottom of the graph. Draw a vertical line that intersects the power
curves.
a. Try to achieve a Ped of 0.90 or 90% chance of detection, with the
lowest N and the simplest control rules.
b. If a Ped of 0.90 cannot be achieved, then select the control
procedure that gives you maximum error detection.

10. Recommend a QC procedure for use with your method.

11. Document your recommendation.

Copyright Westgard QC, Inc. 2009 http://www.westgard.com 6

 Sigma-Metrics QC Selection Tool for 2 Levels Control
Sigma Scale
1.65
1.0
2.65 3σ 3.65 4σ 4.65 5σ 5.65 Pfr Ped N R

13s /2of32s /R4s /31s /6x

0.9
0.07 ----- 6 1
Probability for Rejection (P)

13s /22s /R4s /41s /8x

0.8
0.03 ----- 4 2
0.7 13s /22s /R4s /41s
0.03 ----- 4 1
0.6 12.5s

0.5 0.04 ----- 4 1

1
2.5s
0.4 0.03 ----- 2 1
13s /22s /R4s
0.3
0.01 ----- 2 1
13s
0.2
0.00 ----- 2 1
0.1 13.5s
0.00 ----- 2 1
0.0
0.0 1.0 2.0 3.0 4.0
Systematic Error (SE, multiples of s)

Test _____________________
Allowable Total Error (TEa%)
or (TEa units) _____________________

Clinical Decision Concentration _____________________

Observed Imprecision (CV%)
or (SD units) _____________________
Observed Inaccuracy (Bias %)
or (Bias units) _____________________
Sigma [(TEa%-Bias%)/CV%)]
or [(TEaUnits-BiasUnits)/SDUnits] _____________________
Critical Systematic Error
[Sigma – 1.65] _____________________

Selected QC Procedure _____________________

Analyst _____________________
Copyright Westgard QC, Inc. 2009 http://www.westgard.com Date _____________________ 7
 Sigma-Metrics QC Selection Tool for 3 Levels Control
Sigma Scale
1.65 2.65 3σ 3.65 4σ 4.65 5σ 5.65 P
fr
P
ed
N R
1.0
1 /2of3 /R /3 /6
3s 2s 4s 1s x
0.9
0.07 ----- 6 1
Probability for Rejection (P)

1 /2of3 /R /3 /6
0.8 3s 2s 4s 1s x
0.03 ----- 3 2
0.7 1 /2of3 /R /3
3s 2s 4s 1s
0.05 ----- 6 1
0.6 1 /2of3 /R /3
3s 2s 4s 1s
0.5 0.02 ----- 3 1
1
3s
0.4 0.01 ----- 6 1
12.5s
0.3
0.03 ----- 3 1
13s
0.2
0.01 ----- 3 1
0.1 13s
0.00 ----- 1 1
0.0
0.0 1.0 2.0 3.0 4.0
Systematic Error (SE, multiples of s)

Test _____________________
Allowable Total Error (TEa%)
or (TEa units) _____________________

Clinical Decision Concentration _____________________

Observed Imprecision (CV%)
or (SD units) _____________________
Observed Inaccuracy (Bias %)
or (Bias units) _____________________
Sigma [(TEa%-Bias%)/CV%)]
or [(TEaUnits-BiasUnits)/SDUnits] _____________________
Critical Systematic Error
[Sigma – 1.65] _____________________

Selected QC Procedure _____________________

Analyst _____________________
Copyright Westgard QC, Inc. 2009 http://www.westgard.com Date _____________________ 8
 Problem Set #4. Selection of QC Procedure using
OPSpecs Chart

OPSpecs Chart TEa=10%

Pfr N R
10.0
13s /2of32s /R4s /31s /6x
9.0
Observed Inaccuracy, %Bias

0.07 6 1
13s /22s /R4s /41s /8x
8.0
0.03 4 2
7.0 1 /2 /R /4
3s 2s 4s 1s
0.03 4 1
6.0 1
3.

2.5s
00

5.0 0.04 4 1
sig

12.5s
m
a
pe

4.0 0.03 2 1
rf o

13s /22s /R4s

rm

3.0
an

0.01 2 1
ce

13s
2.0
0.00 2 1
1.0 13.5s
0.00 2 1
0.0
0.0 1.0 2.0 3.0 4.0 5.0

Observed Imprecision, %CV

Method Bias CV QC
A 0.0% 2.0%
B 2.0% 2.0%
C 0.0% 3.0%
D 3.0% 3.0%

Copyright Westgard QC, Inc. 2009 http://www.westgard.com 9

 Problem Set #5. Selection of QC Procedures using
Normalized OPSpecs Charts
Cholesterol Application where TEa=10%
OPSpecs Chart TEa=100% with 90% AQA(SE)
Allowable inaccuracy (bias, %)

OPSpecs Chart TEa=100% with 90% AQA(SE)

100.0 100.0

Allowable inaccuracy (bias, %)

90.0

1 2
90.0

80.0 80.0

70.0 70.0

3.0
3.0

0s
0s

60.0 60.0

igm
igm

Pfr N R Pfr N R

ap
ap

50.0 50.0

erf
erf

12.5s 13s /22s /R4s /41s /8x

orm
orm

40.0

an
40.0 0.03 4 2
an

0.03 2 1

ce
ce

13s /22s /R4s 13s /22s /R4s /41s

30.0 30.0
0.01 2 1 0.03 4 1
13s 12.5s
20.0 20.0
0.00 2 1 0.04 4 1
10.0 13.5s 10.0 13s
0.00 2 1 0.01 4 1
0.0 0.0
0.0 10.0 20.0 30.0 40.0 50.0 0.0 10.0 20.0 30.0 40.0 50.0
Allowable imprecision (s,%) Allowable imprecision (s,%)
Probability for Rejection (P)

Sigma-Scale
1.65
1.0
2.65 3.65 4.65 5.65 P
fr Ped N R OPSpecs Chart TEa=100% with 50% AQA(SE)
Allowable inaccuracy (bias, %)
100.0
13s /22s /R4s /41s /8x
0.9

3
0.03 0.91 4 2 90.0
12.5s
0.8 80.0
0.04 0.70 4 1
0.7 13s /22s /R4s /41s 70.0
0.03 0.73 4 1 2.
0.6 12.5s 60.0 00
sig
m Pfr N R
0.5 0.03 0.48 2 1 a
50.0 pe
13s rf o 13s /22s /R4s /41s /8x
rm
0.4 0.01 0.41 4 1 40.0 an 0.03 4 2
ce
13s /22s /R4s 12.5s
0.3 30.0
0.01 0.35 2 1 0.04 4 1
13s 13s /22s /R4s /41s
0.2 20.0
0.00 0.25 2 1 0.03 4 1
0.1 13.5s 10.0 13s
0.00 0.09 2 1 0.01 4 1
0.0 0.0
0.0 1.0 2.0 3.0 4.0 0.0 10.0 20.0 30.0 40.0 50.0

Systematic Error (SE, multiples of s) Allowable imprecision (s,%)

Meth Bias CV Norm Norm QC

Y-coord X-coord
A 0.0% 2.0%
B 2.0% 2.0%
C 0.0% 3.0%
D 3.0% 3.0%

Copyright Westgard QC, Inc. 2009 http://www.westgard.com 10

 Directions for Using Normalized OPSpecs Charts

1. Define the analytical quality requirement for the test in the form of an
allowable Total Error (TEa), such as given by the CLIA PT criteria for
acceptable performance in a proficiency testing event.

2. Evaluate the imprecision and inaccuracy of your method. Express in

percent by reference to a clinical decision level (Xc) of interest.
a. (SD/Xc)*100 = CV%
b. (Bias/Xc)*100 = Bias%

3. Calculate a “normalized operating point” by expressing the observed

imprecision and inaccuracy as a percent of the allowable Total Error.
a. X-coordinate = (CV%/TEa%)*100
b. Y-coordinate = (Bias%/TEa%)*100

4. Plot the normalized operating point on the Normalized OPSpecs Charts

using the set of charts for either 2 control levels or 3 control levels.
a. Plot 1st on the chart having 90% AQA for low N (i.e., 2 or 3)
b. Plot 2nd on chart having 90% AQA for high N (i.e., 4 or 6)
c. Plot 3rd on chart having 50% AQA for high N (i.e., 4 or 6)

5. Inspect the normalized OPSpecs charts and select control rule(s) whose
allowable limits for imprecision and inaccuracy are above the operating
point. Identify the control rule(s) from the key at the right.
a. If no QC procedure can be selected on the 90%AQA low N chart,
then try the 90%AQA high N chart, and finally try the 50%AQA
high N chart.
b. If no QC procedure can be selected following these steps, then it
will be necessary to implement the “maximum QC” possible and
affordable, such as multirule procedures with N’s of 4 to 6.

6. For a more exact assessment of the probability for error detection (Ped),
you can calculate the Sigma-metric (or critical Systematic Error), then
use the power curves shown in the “Sigma-Metrics Graph” at the bottom
of the low N page.

7. Select the right control rules and right number of control measurements.
Aim for 90% error detection and less than 5% false rejection (the lower
the better).

Copyright Westgard QC, Inc. 2009 http://www.westgard.com 11

 Normalized OPSpecs Charts for 2 Levels Control
OPSpecs Chart TEa=100% with 90% AQA(SE)
100.0
Allowable inaccuracy (bias, %)

90.0 1
80.0

70.0
3.
00

60.0
sig
m

Pfr N R
a
pe

50.0
rf o

1
2.5s
rm

40.0
an

0.03 2 1
ce

1 /2 /R
3s 2s 4s
30.0
0.01 2 1
13s
20.0
0.00 2 1
10.0 13.5s
0.00 2 1
0.0
0.0 10.0 20.0 30.0 40.0 50.0
Allowable imprecision (s,%)

Sigma-Scale
1.65
1.0
2σ 2.65 3σ 3.65 4σ 4.65 5σ 5.65 Pfr Ped N R

13s /22s /R4s /41s /8x

Probability for Rejection (P)

0.9
0.03 0.91 4 2
12.5s
0.8
0.04 0.70 4 1
0.7 13s /22s /R4s /41s
0.03 0.73 4 1
0.6 12.5s

0.5 0.03 0.48 2 1

13s
0.4 0.01 0.41 4 1
13s /22s /R4s
0.3
0.01 0.35 2 1
13s
0.2
0.00 0.25 2 1
0.1 13.5s
0.00 0.09 2 1
0.0
0.0 1.0 2.0 3.0 4.0
Systematic Error (SE, multiples of s)
Copyright Westgard QC, Inc. 2009 http://www.westgard.com 12
 Normalized OPSpecs Charts for 2 Levels Control
OPSpecs Chart TEa=100% with 90% AQA(SE)
100.0
Allowable inaccuracy (bias, %)

90.0 2
80.0

70.0
3.
00
60.0 sig
m
a
pe Pfr N R
50.0 rf o
1 /2 /R /4 /8
3s 2s 4s 1s x
rm

40.0
an
0.03 4 2
ce

1 /2 /R /4
3s 2s 4s 1s
30.0
0.03 4 1
12.5s
20.0
0.04 4 1
10.0 13s
0.01 4 1
0.0
0.0 10.0 20.0 30.0 40.0 50.0
Allowable imprecision (s,%)

OPSpecs Chart TEa=100% with 50% AQA(SE)

100.0
Allowable inaccuracy (bias, %)

90.0 3
80.0

70.0
2.
60.0 00
sig
m Pfr N R
50.0 a
pe
rf o 13s /22s /R4s /41s /8x
rm
40.0 an 0.03 4 2
ce
1
2.5s
30.0
0.04 4 1
13s /22s /R4s /41s
20.0
0.03 4 1
10.0 13s
0.01 4 1
0.0
0.0 10.0 20.0 30.0 40.0 50.0
Allowable imprecision (s,%)
Copyright Westgard QC, Inc. 2009 http://www.westgard.com 13
 Normalized OPSpecs Charts for 3 Levels Control
OPSpecs Chart TEa=100% with 90% AQA(SE)
100.0
Allowable inaccuracy (bias, %)

90.0 1
80.0

70.0
3.
00

60.0
sig
m

Pfr N R
a
pe

50.0
rf o

1 /2of3 /R /3
3s 2s 4s 1s
rm

40.0
an

0.02 3 1
ce

1
2.5s
30.0
0.03 3 1
13s /2of32s /R4s
20.0
0.01 3 1
10.0 13s
0.01 3 1
0.0
0.0 10.0 20.0 30.0 40.0 50.0
Allowable imprecision (s,%)

Sigma-Scale
1.65
1.0
2σ 2.65 3σ 3.65 4σ 4.65 5σ 5.65 Pfr Ped N R

13s /2of32s /R4s /31s /6x

Probability for Rejection (P)

0.9
0.07 0.95 6 1
12.5s
0.8
0.06 0.84 6 1
0.7 13s /2of32s /R4s
0.02 0.74 6 1
0.6 13s /2of32s /R4s /31s

0.5 0.02 0.64 3 1

13s
0.4 0.01 0.55 6 1
12.5s
0.3
0.03 0.55 3 1
13s /2of32s /R4s
0.2
0.01 0.47 3 1
0.1 13s
0.01 0.30 3 1
0.0
0.0 1.0 2.0 3.0 4.0
Systematic Error (SE, multiples of s)
Copyright Westgard QC, Inc. 2009 http://www.westgard.com 14
 Normalized OPSpecs Charts for 3 Levels Control
OPSpecs Chart TEa=100% with 90% AQA(SE)
100.0
Allowable inaccuracy (bias, %)

90.0 2
80.0

70.0
3.
00
60.0 sig
m
a
pe Pfr N R
50.0 rf o
1 /2of3 /R /3 /6
3s 2s 4s 1s x
rm

40.0
an
0.07 6 1
ce

1
2.5s
30.0
0.06 6 1
13s /2of32s /R4s
20.0
0.02 6 1
10.0 13s
0.01 6 1
0.0
0.0 10.0 20.0 30.0 40.0 50.0
Allowable imprecision (s,%)

OPSpecs Chart TEa=100% with 50% AQA(SE)

100.0
Allowable inaccuracy (bias, %)

90.0 3
80.0

70.0
2.
60.0 00
sig
m Pfr N R
50.0 a
pe
rf o 13s /2of32s /R4s /31s /6x
rm
40.0 an 0.07 6 1
ce
1
2.5s
30.0
0.06 6 1
13s /2of32s /R4s
20.0
0.02 6 1
10.0 13s
0.01 6 1
0.0
0.0 10.0 20.0 30.0 40.0 50.0
Allowable imprecision (s,%)
Copyright Westgard QC, Inc. 2009 http://www.westgard.com 15
 Directions for Using a Sigma-Metrics Graph along with
Normalized OPSpecs Charts

1. When a QC procedure is selected from a 50% AQA chart, or if no

selection is possible from the 50% AQA chart, a more exact estimate of
the error detection capability of candidate QC procedures can be obtained
from the Sigma-Metrics Graph.

2. Calculate the sigma-metric from the defined TEa and the observed
imprecision and the observed inaccuracy.
a. Sigma = (TEa% - Bias%)/CV% where all terms are in percentages,
or
b. Sigma = (TEa – Bias)/SD where all terms are in concentrations
units.
c. Sigma = (100 – Normalized Bias)/(Normalized CV), where the
Normalized Bias is the Y-coordinate and the Normalized CV is the
X-coordinate in your earlier calculations of a normalized operating
point.

3. Calculate the critical Systematic Error (∆SE) that needs to be detected,

which is simply Sigma – 1.65.

4. Locate either the Sigma value or the critical SE value on the x-axis
(sigma-scale at top of graph, SE scale at bottom), then draw a vertical
line that intersects the power curves.

5. Assess the probability for error detection from the point of intersection
with a power curve and the Probability for false rejection (Pfr) from the
y-intercept of that power curves.

Copyright Westgard QC, Inc. 2009 http://www.westgard.com 16

 Total QC Strategy

Validate QC
Design with
QC planning
OPSpecs charts

HI-AQA (90%) LO-AQA (<50%)

or HI-Sigma AQA or or LO-Sigma
Sigma

Minimize cost of MOD-AQA Maximize

statistical QC (50%) or error detection
Sigma Maximize
non-statistical QC
Minimize cost of Maximize Improve method
non-statistical QC error detection performance

Maximize Deploy skilled

non-statistical QC analysts

Improve method Add patient

performance Data

Implement Right Optimize for

TQC Right Error Frequency

Copyright Westgard QC, Inc. 2009 http://www.westgard.com 17

 Problem Set #6. Identification of Total QC Strategy
Using Normalized OPSpecs Charts
Cholesterol Application where TEa=10%
OPSpecs Chart TEa=100% with 90% AQA(SE)
Allowable inaccuracy (bias, %)

OPSpecs Chart TEa=100% with 90% AQA(SE)

100.0 100.0

Allowable inaccuracy (bias, %)

90.0

1 2
90.0

80.0 80.0

70.0 70.0

3.0
3.0

0s
0s

60.0 60.0

igm
igm

Pfr N R Pfr N R

ap
ap

50.0 50.0

erf
erf

12.5s 13s /22s /R4s /41s /8x

orm
orm

40.0

an
40.0 0.03 4 2
an

0.03 2 1

ce
ce

13s /22s /R4s 13s /22s /R4s /41s

30.0 30.0
0.01 2 1 0.03 4 1
13s 12.5s
20.0 20.0
0.00 2 1 0.04 4 1
10.0 13.5s 10.0 13s
0.00 2 1 0.01 4 1
0.0 0.0
0.0 10.0 20.0 30.0 40.0 50.0 0.0 10.0 20.0 30.0 40.0 50.0
Allowable imprecision (s,%) Allowable imprecision (s,%)
Probability for Rejection (P)

Sigma-Scale
1.65
1.0
2.65 3.65 4.65 5.65 P
fr Ped N R OPSpecs Chart TEa=100% with 50% AQA(SE)
Allowable inaccuracy (bias, %)
100.0
13s /22s /R4s /41s /8x
0.9

3
0.03 0.91 4 2 90.0
12.5s
0.8 80.0
0.04 0.70 4 1
0.7 13s /22s /R4s /41s 70.0
0.03 0.73 4 1 2.
0.6 12.5s 60.0 00
sig
m Pfr N R
0.5 0.03 0.48 2 1 a
50.0 pe
13s rf o 13s /22s /R4s /41s /8x
rm
0.4 0.01 0.41 4 1 40.0 an 0.03 4 2
ce
13s /22s /R4s 12.5s
0.3 30.0
0.01 0.35 2 1 0.04 4 1
13s 13s /22s /R4s /41s
0.2 20.0
0.00 0.25 2 1 0.03 4 1
0.1 13.5s 10.0 13s
0.00 0.09 2 1 0.01 4 1
0.0 0.0
0.0 1.0 2.0 3.0 4.0 0.0 10.0 20.0 30.0 40.0 50.0

Systematic Error (SE, multiples of s) Allowable imprecision (s,%)

Meth Bias CV Chart Ped TQC Strategy

A 0.0% 2.0%
B 2.0% 2.0%
C 0.0% 3.0%
D 3.0% 3.0%

Copyright Westgard QC, Inc. 2009 http://www.westgard.com 18