Measurement System Analysis

Measurement Purpose

• In order to be worth collecting, measurements must provide value -

that is, they must provide us with information and ultimately,
knowledge

• The question… What do I need to know? … must be answered

before we begin to consider issues of measurements, metrics,
statistics, or data collection systems.

• Too often, organizations build complex data collection and

information management systems without truly understanding how
the data collected and metrics calculated actually benefit the
organization.
 Examples of What and When to Measure

Examples of what and when to measure:

• Product characteristics
• “X’s” in the process
• Assessing process capability
• Monitoring a process with control charts
• Prior to improvement projects
• After improvement projects
• As part of designed experiments
• To qualify operators

Low Quality Data is a Showstopper!!!

 Measurement Process

• Industry has traditionally viewed the measurement and analysis activity as

a “black box”. Measurement equipment was the major focus.
• Howevern equipment is only one part of the measurement process. The
owner of the process must know how to correctly use this equipment and
how to analyze and interpret the results.
 Introduction to MSA

• Analytic studies (Design of Experiments, Regression Analysis) and Statistical

Process Control are among the most important uses of measurement data. If
the data quality is low, the benefit of these procedures is likely to be low.

• How do you know that the data you have used is accurate and precise?
• How do you know if a measurement is repeatable and reproducible?

How good are these?

Measurement System Analysis

 Measurement System Analysis

MSA is a mathematical procedure to quantify variation introduced to a

process or product by the act of measuring.

S.W.I.P.E. Workpiece
Item to be (Part)
Measured Measurement
Standard Measurement Instrument
Process
Person /
Environment
Procedure

The item to be measured can be a physical part, document or a scenario for customer service.
Standard is a reference that is used to calibrate the equipment.
Workpiece is the part being measured
Instrument is the device used to measure the product.
Person can refer to an operator measuring the product.
Procedure is the method used to perform the measurement.
Environment is the surroundings where the measures are performed.
 Measurement System Analysis

Measurement System Analysis is important to:

• Study the % of variation in our process that is caused by our
measurement system.
• Compare measurements between operators.
• Compare measurements between two (or more) measurement
devices.
• Provide criteria to accept new measurement systems (consider new
equipment).
• Evaluate a suspect gage.
• Evaluate a gage before and after repair.
• Determine true process variation.
• Evaluate effectiveness of training program.
 Measurement System Analysis

MSA can be used to:

• Validate measurement systems and processes
• Provide a great way to qualify new inspection equipment
• Compare internal measurement standards with the standards of your
customer.
• Provide a method to evaluate inspector training effectiveness
 Types of MSA’s

• MSA’s fall into two categories:

o Variable
o Attribute

Variable Attribute
• Continuous scale • Pass/Fail, Good/Bad
• Discrete scale • Visual inspections
• Critical dimensions • Customer service quality

• Transactional projects often have Attribute based measurement

systems.
• Manufacturing projects generally use Variable studies more often,
but do use Attribute studies to a lesser degree.
 Poor Measures

Poor Measures can result from:

• Poor or non-existent operational definitions
• Difficult measures
• Lack of understanding of the definitions
• Inadequate training
• Inaccurate, insufficient or non-calibrated measurement devices
• Environmental factors

Measurement Error compromises decisions that affect:

• Customers
• Producers
• Suppliers
Measurement System Variability
Cause and Effect Diagram
 IATF 16949:2016 Requirements

7.1.5.1.1 Measurement System Analysis

Statistical studies shall be conducted to analyze the variation present in
the results of each type of inspection, measurement, and test equipment
system identified in the control plan. The analytical methods and
acceptance criteria used shall conform to those in reference manuals on
measurement systems analysis. Other analytical methods and acceptance
criteria may be used if approved by the customer.

Records of customer acceptance of alternative methods shall be retained

along with results from alternative measurement systems analysis

NOTE: Prioritization of MSA studies should focus on critical or special

product or process characteristics.
IATF 16949:2016
 AIAG MSA Manual 4th Edition

The Automotive Industry Action Group (AIAG) is

a not-for profit association that was originally
created to develop recommendations and a
framework for the improvement of quality in the
North American automotive industry.
The organization was founded by representatives
of the three largest North American automotive
manufacturers: Ford, General
Motors and Chrysler. Membership has grown to
include Japanese companies such as Toyota
and Honda.
The AIAG publishes automotive industry
standards and guidelines, including the MSA
manual 4th Edition (published in June 2010).
AIAG Publications - The AIAG Core Tools
VDA 5 - Capability of Measurement Processes

The German Association of the Automotive

Industry or VDA (German: Verband der
Automobilindustrie) is a German interest
group of the German automobile industry.

The VDA represents both automobile car

manufacturers including BMW, Volkswagen
and Daimler and automobile component
suppliers.

The VDA publishes a series of standards and

recommendations. The VDA Vol. 5 (Capability of
Measurement Processes) 2nd Edition is available
since November 2010.
 AIAG MSA Manual vs. VDA Vol. 5

• The procedure used in order to evaluate the applied measuring system is

basically very similar in the MSA manual and VDA Volume 5.

• The MSA and VDA Volume 5 mainly differ in the calculation of statistical
values.

• In the past, the german automotive groups demanded a certification according

to VDA 6.x from their suppliers. This standard required that suitable
procedures be used in order to evaluate the applied measurement system.
Since the VDA Volume 5 was not published until 2003, suppliers based their
inspection procedures mainly on the MSA manual.

• As of 2003, many suppliers had to evaluate their management systems

according to the MSA manual and the VDA Volume 5 because the German
automotive groups included VDA Volume 5 in their customer-supplier rating as
another applicable document.
 Components of Variation

Whenever you measure anything, the variation that you observe can be
segmented into the following components…

Observed Variation

Unit-to-unit (true) Variation Measurement System Error

Precision Accuracy

Repeatability Reproducibility Stability Bias Linearity

All measurement systems have error. If you don’t know how much of the
variation you observe is contributed by your measurement system, you cannot
make confident decisions.
 Measurement System Error

• An ideal measurement system would produce only “correct”

measurements each time it is used.

• A measurement system that could produce measurements like that

would be said to have the statistical properties of zero variance, zero
bias, and zero probability of misclassifying any product it measured.

• Unfortunately, measurement systems with such desirable statistical

properties seldom exist, and so process managers are typically forced to
use measurement systems that have less desirable statistical properties.

Ideal Measurement System ?

 Purpose of MSA

The purpose of MSA is to assess the error due to measurement

systems.
The error can be partitioned into specific categories:

• Precision errors
o Repeatability - within an operator or piece of equipment
o Reproducibility - operator to operator or attribute gage to attribute gage

• Accuracy errors
o Stability - accuracy over time
o Linearity- accuracy throughout the measurement range
o Resolution
o Bias – Off-set from true value (Constant or variable Bias)
 Accuracy and Precision

Accurate but not precise - On Precise but not accurate - The

average, the shots are in the average is not on the center, but
center of the target but there is a the variability is small
lot of variability
 Accuracy vs. Precision

ACCURATE PRECISE BOTH

+ =

Accuracy relates to how close the

average of the shots are to the Master
or bull's-eye.

Precision relates to the spread of the

shots or Variance.
NEITHER
 Precision

• A precise measure is one that returns the same value of a given

attribute every time an estimate is made.

• Precise data are independent of who estimates them or when the

estimate is made.

• Precision errors can be partitioned into two components:

o Repeatability errors
o Reproducibility errors

Repeatability and Reproducibility = Gage R+R

 Precision: Repeatability

Repeatability is the variation in measurements obtained with one

measurement instrument used several times by one appraiser while
measuring the identical characteristic on the same part.

Repeatability

For example:
• Manufacturing: One person measures the purity of multiple samples of the
same vial and gets different purity measures.
• Transactional: One person evaluates a contract multiple times (over a period of
time) and makes different determinations of errors.
 Precision: Reproducibility

Reproducibility is the variation in the average of the measurements

made by different appraisers using the same measuring instrument
when measuring the identical characteristic on the same part.

Reproducibility

Y Operator A
Operator B

For example:
• Manufacturing: Different people perform purity test on samples from the
same vial and get different results.
• Transactional: Different people evaluate the same contract and make
different determinations.
 Repeatability and Reproducibility Problems

Repeatability Problems:
• Calibrate or replace gage.
• If only occurring with one operator, re-train.

Reproducibility Problems:
• Measurement machines:
o Similar machines: Ensure all have been calibrated and that the standard
measurement method is being utilized.
o Dissimilar machines: One machine is superior.
• Operators:
o Training and skill level of the operators must be assessed.
o Operators should be observed to ensure that standard procedures are
followed.
• Operator by part interactions:
o Understand why the operator had problems measuring some parts and not
others.
• Re-measure the problem parts
• Problem could be a result of gage linearity
• Problem could be poor gage design
 Accuracy

An accurate measurement is the difference between the observed average

of the measurement and a reference value.
– When a measurement system consistently over or under estimates the value
of an attribute, it is said to be “inaccurate”
Accuracy can be assessed in several ways:
– Measurement of a known standard
– Comparison with another known measurement method
– Prediction of a theoretical value
What happens if we don’t have standards, comparisons or theories?
True
Average

Accuracy
Warning, do not assume your
metrology reference is gospel.

Measurement
 Accuracy: Bias

Bias is defined as the deviation of the measured value from the actual value.

Calibration procedures can minimize and control bias within acceptable

limits. Ideally, Bias can never be eliminated due to material wear and tear!

Bias Bias
 Accuracy: Linearity

Linearity is defined as the difference in Bias values throughout the

measurement range in which the gauge is intended to be used. This tells
you how accurate your measurements are through the expected range of
the measurements. It answers the question, "Does my gage have the
same accuracy for all sizes of objects being measured?"

Low Nominal High

+e

B i a s (y)
0.00
*
-e
*
*
Reference Value (x)
y = a + b.x
y: Bias, x: Ref. Value
a: Slope, b: Intercept
 Accuracy: Stability

Stability of a gauge is defined as error (measured in terms of Standard

Deviation) as a function of time. Environmental conditions such as
cleanliness, noise, vibration, lighting, chemical, wear and tear or other
factors usually influence gauge stability. Ideally, gauges can be maintained
to give a high degree of Stability but can never be eliminated unlike
Reproducibility. Gage Stability studies would be the first exercise past
calibration procedures.
Control Charts are commonly used to track the Stability of a measurement
system over time.

Drift

Stability is Bias characterized

as a function of time!
 Guidelines for Determining Stability

1. Select a production part that falls in the mid-range of the production

measurements and designate it as the master sample for stability
analysis. The known reference value is not required for tracking
measurement system stability.

2. On a periodic basis (daily, weekly), measure the master sample three

to five times. The readings need to be taken at differing times to
represent when the measurement system is actually being used. This
will account for warm-up, ambient or other factors that may change
during the day.

3. Plot the data on an X-bar & R or X-bar & S control chart in time order.
Establish control limits and evaluate for out-of-control or unstable
conditions using standard control chart analysis.
4. If the measurement process is stable, the data can be used to
determine the bias of the measurement system.
 Guidelines for Determining Stability

• It may be desirable to have master samples for the low end, the high
end, and the mid-range of the expected measurements. Separate
measurements and control charts are recommended for each.