Dr.

Dennis Jackson
909-273-4492
DSN 933-4492
jacksondh@corona.navy.mil

TBBY0711 1
 MEASUREMENT
UNCERTAINTY AND RISK
FOR NAVY CALIBRATIONS

Dr. Dennis Jackson

NSWC Corona Division (MS-00A4)
7 August 2002

MSDJ 2
 Overview

 Calibration Intervals and Tolerance

Testing

 Tolerance Testing and Measurement

Uncertainty

 Calibration Risk

 Tolerance Testing and System

Reliability Risk
MSDJ 3
 What Is a Calibration?

 Compare the measurement values from a UUT with

the measurement values from a calibrator.
– Deviation = UUT Measurement – Calibrator Measurement

 A UUT is considered in tolerance if:

– Lower Tolerance < Deviation < Upper Tolerance

 Measurement Reliability is the probability of being in

tolerance.

 A Calibration Interval is the amount of time between

calibrations that will meet a measurement reliability
target (keeps the UUT in tolerance).
MSDJ 4
 Calibration Interval
Determination
100

90
Test Equipment Reliability
Measurement Reliability (%) vs. Calibration Interval
80

70

60

50

40

30

20

10

0 6 12 18 24 30 36 42 48
Calibration Interval (Months)

72% EOP Reliability for GPTE

85% EOP Reliability for Safety-of-Flight and Mission Critical
MSDJ 5
 Measurement Model

Measurement
Test
Measurand
Instrument

easurement = True Value + Error

MSDJ 6
 Basic Measurement
Uncertainty Concept

Distribution of Measurements

Measured True
Value Value

True Value = Measured Value ± Uncertainty

MSDJ 7
 Expression of Uncertainty

1. Expanded Uncertainty Value (  Uncertainty )

2. Multiplier
• Standard Uncertainty = u(Error) = 
• Expanded Uncertainty = K  (e.g., 2 )

3. Confidence (e.g., 95%)

4. Time Limit (e.g., 12 months)

• Uncertainty should be projected for end of time limit
• Uncertainties without time limits are immediately invalid

MSDJ 8
 Calibration Tolerance
Testing Model

Measures Measures
Calibrator Artifact UUT

rement = True Value + Calibrator Error

nt = True Value + UUT Error

n = UUT Measurement – Calibrator Measurement

= UUT Error – Calibrator Error

= u2( UUT Error) + u2(Calibrator Error)

MSDJ 9
 Measurement Uncertainty and
Calibration Tolerance Limits

 Tolerance Testing does not test the UUT error – it

tests the deviation between the UUT and the
Calibrator

 Test Tolerance Limit  Measurement Uncertainty

(though it is related)

 Measurement Reliability  Uncertainty Confidence

Level (though it is related)

MSDJ 10
 The Relationship: The GUM vs
Calibration Tolerance Limits

 Measurement uncertainty and engineering

tolerance limits are related conceptually and
mathematically.

 Given engineering tolerance limits and a

measurement reliability, you can determine 95%
uncertainty limits.

 For comparability and consistency, 95%

uncertainty limits should be provided in addition
to tolerance limits.
MSDJ 11
 Traceability Chain
Uncertainty Results

Unc
Lab Std Unc Tolerance TAR Obs Rel
(95%)
NIST 2.70 1.350
Primary 11.25 5.626 6.250 4.17 0.7200
Cal Lab 45.53 22.767 25.000 4.00 0.7200
Test Eq 182.01 91.004 100.000 4.00 0.7200

The complete uncertainty equation uses

uncertainty information from the traceability chain
since each level is tested and adjusted by the
previous level.

MSDJ 12
 Approximate 95% GUM Uncertainty
Based on Tolerance Testing
If there is a consistent TAR, and you obtain the
uncertainty from the tolerance limit by:
2
  
2

   
  
1 1 1
u(UUT)  (UUT Tolerance Limit )    
  TAR  1 
2
 Z  1 Ro (UUT )    Z  1 Ro ( Calibrator ) 
  2     

2 
Uncertainty Multiplier = K = 2
Expanded 95% Uncertainty Confidence Level  2 u(UUT)

where:
UUT Tolera nce Limit
Ro = Reliability Target TAR =
Calibrator Tolerance Limit

( Zp = The pth percentile of the Standard Normal Distribution

F(Zp) = The cumulative Standard Normal Distribution Function = p
)
MSDJ 13
 Example
Expressions of Uncertainty

Based on Navy Interval Data (72% Reliability Target)

1.  100 ppm (Engineering Tolerance)

2. 72% Measurement Reliability Target
3. Test Accuracy Ratio (TAR) – 4:1
4. Calibration Interval: 1 year

Industry (GUM) Translation (95% Confidence)

1. ± 182.006 ppm (Expanded measurement uncertainty)

2. The expanded uncertainty is 2 times the standard uncertainty
(91.003 * 2)
3. 95% Confidence (Actually 95.45%, since 2 is an approximation)
4. Valid for 1 year

MSDJ 14
 Example
Expressions of Uncertainty

Based on Navy Interval Data (85% Reliability Target)

1.  100 ppm (Engineering Tolerance)

2. 85% Measurement Reliability Target
3. Test Accuracy Ratio (TAR) – 4:1
4. Calibration Interval: 1 year

Industry (GUM) Translation (95% Confidence)

1. ± 136.162 ppm (Expanded measurement uncertainty)

2. The expanded uncertainty is 2 times the standard uncertainty
(68.081 * 2)
3. 95% Confidence (Actually 95.45%, since 2 is an approximation)
4. Valid for 1 year

MSDJ 15
 Calibration Risk

 UUT’s observed to be in tolerance can

actually be out of tolerance (False Accept)

 UUT’s observed to be out of tolerance can

actually be in tolerance (False Reject)

 Risk is driven by the test accuracy ratio

(TAR)

MSDJ 16
 False Accept Due to
Measurement Error
Calibrator
Error

UUT
Error

Lower Limit True Value Upper Limit

PFA = Pr(UUT Out of Tolerance given Comparison In Tolerance)

(Proportion of the In Tolerance results that are in error)

MSDJ 17
 False Reject Due to
Measurement Error
Calibrator
Error

UUT
Error

Lower Limit True Value Upper Limit

PFR = Pr(UUT In Tolerance given Comparison Out of Tolerance)

(Proportion of the OOT results that are in error)

MSDJ 18
 Consumer Risk

BOPN = Proportion of tested instruments that are

still out of tolerance when tested using
Navy policy.

Navy Policy: Adjust only when OOT

MSDJ 19
 Calibration Risk Example

Unc BOPN
Cal AOP EOP Engineering
(95% TAR Obs Rel PFA PFR (Adj
Level Std Unc Std Unc Tolerance
EOP) OOT)
Primary 2.806 1.403 1.403
STD 11.225 4.125 5.613 6.250 0.7200 0.050 0.179 0.036
STD 45.541 16.934 22.771 25.000 4.000 0.7200 0.039 0.127 0.028
TE 182.006 91.003 100.000 4.000 0.7200 0.039 0.131 0.028

MSDJ 20
 Unnecessary Calibration Risk
(PFR)

80%
Unnecessary Calibrations (%)

70%
60%
50%
40%
30%
20%
10%
0%
0 2 4 6 8 10 12 14 16
Test Accuracy Ratio (K:1)

MSDJ 21
 Initial OOT Risk (BOPN)

16%
14%
12%
Initial OOT Rate (%)

10%
8%
6%
4%
2%
0%
0 2 4 6 8 10 12 14 16
Test Accuracy Ratio (K:1)

MSDJ 22
 Tested Equipment Risk:
Bad Test Decisions

MSDJ 23
 Effects of Bad Test Decisions

 Maintenance Costs  Fleet Costs

– Unnecessary Repair – Lower Reliability
– Lower Loadouts – Shorter Certification

MSDJ 24
 Effects of Bad Test Decisions on
Fleet Reliability
Bad decisions lower initial reliability
1
Perfect Test
0.9
Combat Sys Reliability

Partial Test
0.8
Partial Test w/
0.7 Uncertainty
Requirement
0.6
Stockpile Reliability
0.5
0 2 4 6 8 10 Certification Interval

Time Since Test (Years)

Lower initial reliability means:

 Lower stockpile reliability (* .88 vs .90 vs .92)
 Shorter maintenance cycles (+ 3 vs 5.5 vs 8 yrs)

MSDJ 25
 Tested Equipment Risk
Unc
AOP EOP Engineering
Level (95% TAR Obs Rel PFA PFR BOPR
Std Unc Std Unc Tolerance
EOP)
Std 0.02810 0.01405 0.01405
Cal 0.11224 0.04125 0.05612 0.06250 0.7200 0.050 0.180
TE 0.33740 0.13575 0.16870 0.25000 4.000 0.8500 0.024 0.216
Sys 1.53681 0.76841 1.00000 4.000 0.8000 0.026 0.139 0.026

System Risk Definitions:

PFA = Proportion of the in tolerance systems that aren’t
PFR = Proportion of unnecessary rework on rejected systems
BOPR = Initial out of tolerance rate after test for systems
tested and removed for repair if OOT.

MSDJ 26
 Unnecessary System Rework
(PFR)

80%
70%
Unnecessary Rework (%)

60%
50%
40%
30%
20%
10%
0%
0 2 4 6 8 10 12 14 16
Test Accuracy Ratio (K:1)

MSDJ 27
 Initial System OOT Rate After
Test (BOPR)

5%

4%
Initial OOT Rate (%)

3%

2%

1%

0%
0 2 4 6 8 10 12 14 16
Test Accuracy Ratio (K:1)

MSDJ 28
 Conclusions

 GUM measurement uncertainty and ICP engineering

tolerances are mathematically related.
 Expressions of measurement uncertainty should
contain at least the following:
– Expanded Uncertainty Value (±Uncertainty)
– Multiplier Used (e.g., Uncertainty = 2*standard uncertainty)
– Confidence Level (e.g., 95%)
– Time Limit (e.g., 12 months)
 Recommend publishing 95% measurement
uncertainties in addition to engineering tolerances.
 Calibration and system risks are the bottom line
metrics for a calibration system.
MSDJ 29