R E S E A R C H AN D R E P O R T S

Performance Standards for Quality Control Systems

SELOM A GBOYI , SHASHI MEHTA

LEARNING OBJECTIVES INTRODUCTION

1. Identify the steps to design an evidence-based quality Laboratory testing plays a tremendous role in the diagno-
control (QC) system. sis, treatment, monitoring, and management of patients’
2. Set QC limits and frequency based on the sigma value conditions. The accuracy of laboratory results is critical
of each assay. for the delivery of quality healthcare.1 Laboratories ensure
3. Use the concept of total error to evaluate method the accuracy of their test results through the implementa-
performance. tion of an internal quality control (IQC) system and through
the participation of external quality assurance (EQA) pro-
grams. A properly designed quality control (QC) system
ABSTRACT
should effectively detect errors in the analytical system

Downloaded from http://hwmaint.clsjournal.ascls.org/ on August 27 2024

Standard statistical quality control (QC) has been around while minimizing false rejection.2,3
since its introduction in 1950 by Levey and Jennings. The standard QC practice in most laboratories is a stat-
Today, while most laboratories use standard statistical istical QC system in which the control limits are set at
QC processes, with or without Westgard multi-rules to ±2 standard deviations (SDs) from the mean.3 With the
evaluate quality, few incorporate performance standards improvement of technology and methodology over the
in their QC systems. With performance standards, the years, analytical systems have shown better accuracy
quality level of each test is determined on a sigma scale. and precision, and small changes in analyzers’ perfor-
This guides the selection of optimal QC frequency and mance may not be easily identifiable by standard QC
rules to monitor the analytical system. QC rules are processes.2,3 In order to optimize QC processes, clinical
selected based on the sigma score and are specific to each laboratories need to implement evidence-based QC.
test. While a single rule (13s) is sufficient to efficiently mon- Evidence-based QC requires quality goals against which
itor tests with sigma quality at or above 6, a more stringent the analytical system performance will be compared.3,4
rule (13s/22s/R4s/41s) is required for those performing at Quality goals, also known as performance standards, are
four sigma. The comparison of analytical total error (TE) the acceptable limits for both random and systematic
to the total error allowable (TEa) helps ensure that the sys- analytical errors and are used to determine if analytical
tem is operating within the defined quality specifications; methods are producing clinically acceptable results.4,5
therefore, accurate patient results are produced. Performance standards are expressed as total error allow-
able (TEa). With the incorporation of quality goals, not only
ABBREVIATIONS: ΔSEc - critical systematic error, CAP - can standard QC systems detect an error in the operating
College of American Pathologists, CLIA - Clinical system but they can also safely determine if the error will
Laboratory Improvement Amendments, DPM - defects per lead to unacceptable patient results.3
million, EQA - external quality assurance, IQC - internal
quality control, PT - proficiency testing, QC - quality con- ANALYTICAL SYSTEMS
trol, RCPA - Royal College of Pathologists of Australasia,
SD - standard deviation, TE - total error, TEa - total error The laboratory testing process can be divided into three
allowable. phases (pre-analytical, analytical, and post-analytical)
and errors can occur at any stage.1 Errors associated with
INDEX TERMS: performance standards, quality goals, analytical systems are monitored through the implemen-
evidence-based quality control, critical systematic error. tation of an IQC system and are the main focus of this
paper. Laboratories also participate in EQA or proficiency
Clin Lab Sci 2018;31(1):32–36 testing (PT) programs to ensure that they are meeting the
quality requirements set by regulatory bodies.
Analytical systems are subject to systematic and ran-
dom errors. Bias is an estimate of systematic error and SD is
Selom Agboyi, SHP-Rutgers University the measure of a random error. When a stable QC material
is measured for a period of time, it creates a population
Shashi Mehta, SHP-Rutgers University
of data points that, when plotted, will display a normal
Address for Correspondence: Shashi Mehta, SHP-Rutgers or Gaussian distribution. With a single data population,
University, mehtas1@shp.rutgers.edu it can be predicted that approximately 68% of the data,

32 | VOL 31, NO 1, WINTER 2018, CLINICAL LABORATORY SCIENCE

 RESEARCH AND REPORTS

or area under the curve, will fall within ±1 SD of the mean, Union of Pure and Applied Chemistry (IUPAC), the Inter-
95% between ±2 SDs, and 99.7% within ±3 SDs. Gaussian national Federation of Clinical Chemistry and Laboratory
statistics and predictions are the key to understanding Medicine (IFCC), and the World Health Organization
statistical QC in clinical laboratory settings.3-5 (WHO), included participants from 27 countries.7 The con-
sensus statement laid out the hierarchy of models from the
highest to the lowest. Five models were described and
HISTORY OF QC based on the following:
QC has been utilized in medical laboratories for several
decades and has evolved over the years with advances 1. Clinical requirements or clinical outcomes
in technology. Before the introduction of statistical QC 2. Biological variation within and between individuals
by Levey and Jennings, laboratory tests were performed 3. Professional recommendations such as those seen in
manually in batches where QC samples were run along professional expert publications
with patients’ samples. Statistical QC arose with the intro- 4. Quality specifications set by regulatory bodies (eg,
duction of the Levey-Jennings chart in 1950, and QC sam- the Clinical Laboratory Improvement Amendments
ples were run in duplicate with the range of acceptability [CLIA]) and the PT organizer (eg, College of American
set at ±3 SDs from the statistical mean.4,5 Pathologists [CAP])
With the introduction of automation in 1960, QC sam- 5. State of the art performance

Downloaded from http://hwmaint.clsjournal.ascls.org/ on August 27 2024

ples were reduced to a single measurement and the QC
limits were set at ±2 SDs.4,6 Statistical QC became the basis These five models were later reduced to three
of QC for quantitative tests in clinical laboratories.6 This after the Stockholm consensus was revised in 2014 in
QC acceptability criteria worked well with the single test Milan. The first and second models were kept intact,
auto-analyzer until multichannel analyzers made their whereas the remaining three were combined under the
way into clinical laboratories. There is a 5% false rejection umbrella of state-of-the-art performance.8
rate when the ±2 SDs rule is used for one analyte. For two
levels of control, the probability of false rejection increases
to 9.5%.6 EVIDENCE-BASED APPROACH TO QC
With multitest systems, several different tests can Unlike traditional QC design, evidence-based QC requires
be performed simultaneously on one analyzer. With the laboratory to set quality goals for each analyte tested.
simultaneous measurements, the rate of false rejection Expressed as TEa, quality goals or performance standards
increases because of a multiplier effect.6 In order to maxi-
are the acceptable limits for both random and systematic
mize error detection and minimize the false rejection rate,
errors and are used to determine if analytical methods are
Dr. Westgard developed a set of rules, known as Westgard
producing clinically acceptable results.3
rules,7 in which QC limits were set at ±3 SDs and ±2 SDs
Method performance is assessed by evaluating the
violation was used as a warning.4,6
analytical total error (TE), which is the combined effect
Today, most laboratories still use the standard statis-
of bias and imprecision.3,9 By comparing the TE to TEa,
tical QC with or without Westgard multi-rules. Statistical
the laboratory can determine if their analytical system is
QC design is based on the mean and SD mostly calculated
from 20 QC data points when a new QC material is started.3 operating within their defined quality specifications. In this
This standard QC system sets limits at ±2 SDs from the model of design, QC limits are determined based on the
mean to assess quality.2,3 The question arises: to what level analytical system capability in terms of the sigma metric
of quality are we controlling? (σ). Another useful quality indicator of method perfor-
mance is critical systematic error (ΔSEc), which is the num-
ber of SDs that the mean can shift before 5% of results fall
PERFORMANCE STANDARDS outside a defined quality specification (TEa).3,4,10
Expressed as TEa, quality goals, also known as perfor-
mance standards, are the acceptable limits for both ran-
TRUE/TARGET VALUE
dom and systematic errors and are used to determine if
analytical methods are producing clinically acceptable The analyte true value, used to monitor a method’s accu-
results.3-5 While selecting quality goals, laboratories should racy, is the best available estimate of the analyte value and
ensure that they are based on scientific evidence and are is preferably based on a peer group method mean.11 True
attainable by the analytical system. value should not be confused with the laboratory ob-
Various quality requirements have been identified served mean. If a laboratory does not participate in an
over the years, leading to several sources of performance interlaboratory comparison program, the manufacturer’s
standards. In 1999, a conference was held in Stockholm to target value may be used. When those two options are
reach a consensus on how to set quality goals in laboratory not available, the laboratory’s own historical data may
medicine. The conference, sponsored by the International be used.3,4

VOL 31, NO 1, WINTER 2018, CLINICAL LABORATORY SCIENCE | 33

 RESEARCH AND REPORTS

TOTAL ERROR SIGMA METRIC

Once the true value is determined, the analytical mean By definition, the sigma metric is the number of SDs that
bias can be calculated. The bias represents the systematic the closest tolerance limit is from the mean of the assay.3,4
error and is the measure of the difference between the lab- The concept of six sigma is widespread in industry quality
oratory observed mean and the true value. Random error, management and requires defined tolerance limits
also known as imprecision, is expressed as SD and is the in order to evaluate quality in terms of a sigma met-
measure of dispersion around the mean. The total varia- ric.3,4,15-17 In the industry, a sigma value ranges from
tion of a test result from the true value combines the one to six and relates to the number of defects per million
effects of bias and random error and is defined as TE.3,4 (DPM). A system with one sigma produces 697,700 DPM,
Figure 1 shows the visual representation of TE.9,18 whereas a six-sigma process generates 3.4 DPM. Six
The following formula is used to calculate the sigma is regarded as world-class quality, and three sigma
analytical TE: (66,807 DPM) is the minimum recommended for routine
production.4,11,12 In the laboratory, total allowable error
TE = Bias þ 2SD: (TEa) represents the tolerance limit and is used to calcu-
Method performance is evaluated by comparing late an assay performance on a sigma scale (Figure 2).18
TE to the TEa. The sigma metric is calculated as follows: Sigma metric
(σ) = (TEa − Bias)/SD

Downloaded from http://hwmaint.clsjournal.ascls.org/ on August 27 2024

Assuming a laboratory test has a bias equal to 3, an SD
TOTAL ERROR ALLOWABLE of 2, and the TEa set at 13, the sigma metric for the assay is
(13 − 3)/2 = 5. If the same test has a larger bias equal to
TEa is the tolerance limit, the maximum acceptable varia-
4 and an SD of 3, the sigma value will be (13 − 4)/3 = 3.
tion from the true value of an assay.3,4 As pointed out
There is a positive correlation between the sigma score
earlier, there are several sources of TEa limits. In the
and a test performance. The higher the sigma metric,
United States, CLIA sets limits for acceptable method per-
the better the test performance.
formance. For example, the CLIA limit for chloride is the
target value ±5%. The TEa in this case is ±5% of the true
value of the analytes.3,4 This criterion is also used by CAP. SETTING RULES BASED ON SIGMA VALUE
Regulatory agencies from different countries may have dif-
Standard QC systems use the same QC rules (mostly ±2 SD)
ferent limits. In Australia, the Royal College of Pathologists
for all analytes regardless of the level of performance,
of Australasia (RCPA) sets the limit for acceptable analytical
whereas the rules in evidence-based QC systems are based
performance for chloride at 3 mmol/L when levels are
on each analyte’s performance expressed in the sigma
below or equal to 100 mmol/L and at ±3% if levels are
metric.10 Westgard Sigma rules allow laboratories to
above 100 mmol/L.13 Chloride’s quality specification is
select QC limits and the number of control measurements
more stringent when based on biological variation and
is set at ±1.5% for desirable performance.14

Figure 2. Sigma metric. The graph shows how six-sigma toler-

Figure 1. Illustration of the concept of total error, which is the ance limits apply to QC in a clinical laboratory.
combination of systematic error (bias) and random The sigma value of this essay is four, as the mean is at
error (2 SD). 4 SD from the upper tolerance limit.

34 | VOL 31, NO 1, WINTER 2018, CLINICAL LABORATORY SCIENCE

 RESEARCH AND REPORTS

Table 1. QC rules based on sigma metric for two control level. Quality varies between tests and from one analytical
measurements per run system to another; therefore, it may not be appropriate to
Sigma QC Rules Number of Runs set the same QC rules for different analytes.
Setting quality goals allows laboratories to design a
≥6 13s 1 more efficient QC system and select error detection limits
5 13s/22s/R4s 1 specific to the quality level (sigma score) of each analyte.
4 13s/22s/R4s/41s 2 Such systems can be objectively monitored for improve-
<4 13s/22s/R4s/41s/8x 4 ment. The calculated monthly ΔSEc tracks the perfor-
mance of the analytical system relative to the quality
goals (TEa) set. The ΔSEc decreases as the TE produced
needed to achieve optimal error detection (≥90%), and moves closer to the error limits set (TEa). A ΔSEc of 0 indi-
keep false rejection below 5%.6,10,19,20 cates that the probability of producing clinically mislead-
Table 1 shows QC limits and frequency based on the ing patient results or unacceptable proficiency tests is
assay’s capability. Highly capable tests with sigma scores greater than 5%.7
at or above six do not require stringent QC rules, as they
can be controlled easily and efficiently.4,9 Two levels of REFERENCES
control run once every 24 hours with limits set at 13s are

Downloaded from http://hwmaint.clsjournal.ascls.org/ on August 27 2024

enough to control the quality of the results. The probability 1. Hammerling JA. A review of medical errors in laboratory diag-
nostics and where we are today. Lab Med. 2012;42(2):41–44.
of false rejection is 0%, meaning any error detected is doi:10.1309/LM6ER9WJR1IHQAUY
significant and should be addressed. At five sigma, the 2. Stackhouse JW. Into Quality Control. Cola’s Insights. 2013.
13s/22s/R4s QC rule is needed to bring about error detection Available at: http://www.cola.org/insights-newsletters/2013/
above 90% and false rejection below 5%.5,20 April/insights.pdf.
Tests performing at four sigma require additional rules 3. Brooks ZC. Performance-driven quality control. Washington,
DC: AACC Press; 2001.
(13s/22s/R4s/41s) and increased QC frequency (two levels of 4. Westgard JO. Six sigma quality design & control: desirable
control run twice a day or four levels run once a day). precision and quantitative measurement processes. 2nd ed.
The maximum QC rule (13s/22s/R4s/41s/8x) and higher Madison, WI: Westgard QC Inc; 2006.
QC frequency are recommended for performance below 5. Westgard JO. Basic QC practice: training in statistical quality
four sigma. The appropriate QC frequency is four times control for medical laboratories. 3rd ed. Madison, WI:
Westgard QC, Inc; 2010.
in 24 hours when two levels of control are used, and twice 6. Westgard JO. Basic QC Practices: Historical Perspective on
a day when four levels of control are measured per run.5,20 Laboratory QC: Where We’ve Been and Where We’re Going!
2011. Available at: https://www.westgard.com/history-and-
future-of-qc.htm.
CRITICAL SYSTEMATIC ERROR 7. Petersen PH, Fraser CG, Kallner A, Kenny D. Strategies to set
ΔSEc is used to monitor method performance over time. global analytical quality specifications in laboratory medicine.
As defined previously, ΔSEc measures how far the mean Scand J Clin Lab Invest. 1999;59:473–585.
8. Special issue: 1st EFLM strategic conference defining analyti-
can shift before 5% of the results exceed a TEa limit.3,4 cal performance goals 15 years after Stockholm conference.
ΔSEc is calculated as follows: Clin Chem Lab Med. 2015;53:829–958.
9. Westgard JO, Westgard SA. Total Analytical Error From Concept
ΔSEc = ðTEa − BiasÞ=SD − 1:65 or to Application. Clinical Laboratory News. Available at: https://
ΔSEc = sigma metric − 1:65: www.aacc.org/publications/cln/articles/2013/september/total-
analytic-error.
The figure 1.65 represents the z-score associated with 10. Westgard JO, Westgard SA. Quality control review: imple-
5% risk of producing erroneous results. It corresponds to menting a scientifically based quality control system. Ann
Clin Biochem. 2016;53(Pt 1):32–50. doi:10.1177/000456321
95% confidence limits.3,4 For example, a test method with 5597248
a quality level at six sigma is associated with a ΔSEc of 11. Schoenmakers CH, Naus AJM, Vermeer HJ, van Loon D,
4.35 (6 − 1.65 = 4.35); therefore, the mean can shift Steen G. Practical application of Sigma Metrics QC procedures
4.35 SD before 5% of results exceed TEa. On the other in clinical chemistry. Clin Chem Lab Med. 2011;49(11):1837–
hand, a test method with 1.65 sigma value is already pro- 1843. doi:10.1515/cclm.2011.249
12. Jassam N, Bosomworth M, Thompson D, Lindsay C, Barth JH.
ducing 5% erroneous results as ΔSEc is 0 (1.65 − 1.65 = 0). Managing quality in networked laboratories: a system for
The probability of producing clinically misleading patient managing quality in networked laboratories – statistical con-
results or unacceptable proficiency tests will exceed 5% trol rules design. Available at: https://www.westgard.com/
with any increase in bias or imprecision. networked-lab-quality.htm.
13. RCPA QAP Allowable limits of performance (ALP). Available at:
http://www.rcpaqap.com.au/docs/2014/chempath/ALP.pdf.
CONCLUSION 14. Westgard JO. Desirable Specifications for Total Error, Im-
precision, and Bias, Derived from Intra- and Inter-Individual
Even though multiple analytes are run on the same ana- Biologic Variation. Available at: https://www.westgard.com/
lyzer platform, they do not always perform at the same biodatabase1.htm.

VOL 31, NO 1, WINTER 2018, CLINICAL LABORATORY SCIENCE | 35

 RESEARCH AND REPORTS

15. Bais R. Use of Capability Index to improve laboratory analyti- 18. Westgard JO, Westgard SA. Assessing quality on the Sigma
cal performance. Clin Biochem Rev. 2008;29(suppl 1): scale from proficiency testing and external quality assess-
S27–S31. ment surveys. Clin Chem Lab Med. 2015;53(10):1531–1535.
16. Singh B, Goswami B, Gupta VK, Chawla R, Mallika V. doi:10.1515/cclm-2014-1241
Application of sigma metrics for the assessment of quality 19. Jones G, Calleja J, Chesher D, et al. Collective opinion paper on
assurance in clinical biochemistry laboratory in India: a pilot a 2013 AACB workshop of experts seeking harmonization of
study. Indian J Clin Biochem. 2011;26(2):131–135. doi:10. approaches to setting a laboratory quality control policy. Clin
1007/s12291-010-0083-1 Biochem Rev. 2015;36(3):87–95.
17. Gami B, Patel D, Chauhan K, Shah H, Haridas N. Sigma metrics 20. Westgard JO, Westgard SA. Westgard Sigma Rules. 2014.
as quality marker for analyzing electrolytes in laboratory. Int J Available at: https://www.westgard.com/westgard-sigma-
Adv Res (Indore). 2013;7(1):197–201. rules.htm.

Downloaded from http://hwmaint.clsjournal.ascls.org/ on August 27 2024

36 | VOL 31, NO 1, WINTER 2018, CLINICAL LABORATORY SCIENCE