SMILE

Johns Hopkins University

Baltimore, MD USA

Author: Document Number: Equ35-A-04

Validation Committee Effective (or Post) Date: 19 Dec 2008

Review History Date of last review: 11 Aug 2010

Reviewed by: Heidi Hanes

SMILE Comments: This document is provided as an example only. It must be revised to accurately reflect your lab’s
specific processes and/or specific protocol requirements. Users are directed to countercheck facts when considering
their use in other applications. If you have any questions contact your SMILE representative.

SMILE Accuracy Guidelines-Chemistry

SMILE Accuracy Guidelines-Chemistry Document Number 210

Effective Date 19 Dec 2008

Subject Page 1 of 1

Guidelines for performing accuracy testing on a Supercedes New

chemistry analyzer

Review History Date of last review: 19 Dec 2008

Reviewed by: Erin Gover

Version # [0.0] Revision Date Description (notes)

[dd/mm/yy]

Revision
History

476105914.doc 1 of 9 30-Apr-09
 SMILE Chemistry Accuracy Flowchart
Determine your comparison or
reference method

Test 20 samples in duplicate

and average the results

Evaluate data
following step III.
A in attached
guideline

No
Collect additional data
R >0.975?
over larger range
Yes

Evaluate Error indices

following step III. B in
attached guideline
(Two Instrument Comparison)

95% of error No Plot the data following step III. D

indices between - in attached guideline
1.0 and 1.0? (Alternate Method Comparison)

Yes

Yes 95% of error No

indices between -
Accuracy experiment
1.0 and 1.0?
PASSES

Accuracy experiment
Evaluate data following step
FAILS. Troubleshoot
III. D. 6 in attached guideline
before repeating accuracy
(Six Sigma)
studies

Yes Sigma Metric No

>2.0?

Accuracy experiment
Accuracy experiment FAILS. Troubleshoot
PASSES before repeating accuracy
studies

476105914.doc 2 of 9 30-Apr-09
 SMILE Chemistry Accuracy Guidelines

ACCURACY is the true value of a substance being measured. Verification of accuracy is the
process of determining that the test system is producing true, valid results.

I. Determine your comparison or reference method

A. The comparison method must be previously validated.
B. The comparison method must be currently performing successfully on EQA
C. The ideal comparison method is a similar instrument/method
D. Comparison to an in-house method is preferred if the in-house instrument meets the
above criteria.
E. Samples with known values, such as proficiency testing samples or commercial
standards, may be used as the reference method.
II. Sample Criteria
A. A minimum of 20 samples that cover the reportable range of the method and include
points near the Medical Decision Points.
B. Patient, quality control, and proficiency testing materials may be used.
C. Testing: Run each sample in duplicate on each instrument
1. Ideally samples should be run within 2 hours of each other.
2. Duplicates should be averaged.
3. Data should be plotted immediately to identify and correct any outliers.
III. Acceptability criteria—evaluate the data using one of the options below:

476105914.doc 3 of 9 30-Apr-09
A. Plot the data in a regression analysis
program or spreadsheet. The SMILE
Accuracy Pack is included for this use
(Appendix 1).
1. Plot the reference method on the X axis
and the method being validated on the Y
axis.
2. Enter the Allowable Total Error (TEa)
concentration and percent. Refer to
SMILE Chemistry TE Limits table.
(Appendix 2)
3. Enter the results for the reference (X)
and the new (Y) methods.
4. Evaluate the statistics.
5. The correlation coefficient (R) must be
>0.975
A. Plot the data in EP Evaluator Two
Instrument Comparison Module or a similar
regression analysis program.
1. Plot the reference method on the X axis
and the method being validated on the Y
axis.
2. Enter the Allowable Total Error (TEa)
concentration and percent. Refer to
SMILE Chemistry TE Limits table.
(Appendix 2)
3. Enter the results for the reference (X)
and the new (Y) methods.
4. Evaluate the statistics by clicking on Print
Preview or on the statistics tab.
5. The correlation coefficient (R) must be
>0.975

If Then
R <0.975  Data does not extend over acceptable range.
 More data must be evaluated over larger range.
R >0.975  Proceed with step B below to evaluate acceptability.

B. Using this method, 95% of the data points from the comparative method must be
within Total Allowable Error limit of the reference method (refer to SMILE
Chemistry TE Limits table, Appendix 2). Use one of the methods below to
evaluate.

476105914.doc 4 of 9 30-Apr-09
1. Using a regression analysis program or
spreadsheet (such as the SMILE Accuracy
Pack, Appendix 1) calculate the “Error
Index” to measure the difference between
the two methods as a ratio of the Total
Allowable Error.
a. The Error Index can be calculated by
subtracting the reference method data
point (X) from the method being validated
data point (Y) and dividing by the Total
Allowable Error (TEa). The equation is:
(Y-X)/TEa.
b. Use the percent TEa or the minimum
detectable difference (absolute TEa),
whichever is greater.
c. The absolute TEa value is calculated by
converting the TEa percentage to a
decimal and multiplying by the reference
method data point.
d. The Error Index is measured for each X-
Y pair, and must fall within -1 and 1. If
more than 5% of the specimens have an
Error Index of less than -1 or greater than
1, the accuracy experiment fails.
1. The EP Evaluator program uses a statistic
called the “Error Index” to measure the
difference between the two methods as a
ratio of the Total Allowable Error.
a. The Error Index can be calculated by
subtracting the reference method data
point (X) from the method being validated
data point (Y) and dividing by the Total
Allowable Error (TEa). The equation is:
(Y-X)/TEa.
b. EP Evaluator uses the percent TEa or
the minimum detectable difference
(absolute TEa), whichever is greater.
c. The absolute TEa value is calculated by
converting the TEa percentage to a
decimal and multiplying by the reference
method data point.
d. The Error Index is measured for each X-
Y pair, and must fall within -1 and 1. If
more than 5% of the specimens have an
Error Index of less than -1 or greater than
1, the accuracy experiment fails. (See
Appendix 3 for an example EP report).

C. Evaluate results using the table below:

If Then
95% of Error Indices  Accuracy is acceptable
are acceptable  Proceed with Linearity experiments

<95% of Error Indices  Proceed with step D below.

are acceptable

D. Plot the data in a regression analysis program such as the SMILE Accuracy
Pack (Appendix 1) or EP Evaluator Alternate (Quantitative) Method
Comparison module. Deming Regression is preferred, however if not
available regular regression is acceptable provided that the Correlation
Coefficient (R) is >0.975.
1. Plot the reference method on the X axis and the method being validated
on the Y axis.
2. Enter the Allowable Total Error (TEa) concentration and percent. Refer to
SMILE Chemistry TE Limits table (Appendix 2).
3. Enter Medical Decision Points
i. A Medical Decision Point (MDP) is the concentration of the analyte at
which a medical decision is triggered. You may enter up to five values.

476105914.doc 5 of 9 30-Apr-09
 These values are plotted on the graph, and the program computes a
predicted Y value and confidence interval at each X decision point.
ii. For many analytes, the medical decision points correspond to the
lower and upper limits of the normal range. It is advisable to also
include medical decision points at the DAIDS Toxicity limits and/or
laboratory established critical values.
4. Evaluate using one of the methods described below:

b. If using a regression analysis program or a. If using EP Evaluator program create the

spreadsheet calculate and evaluate the
95% confidence limits and error indices.
The SMILE Accuracy Pack, (Appendix 1)
performs these calculations automatically.
i. Subtract the X Method MDP from the
low and the high 95% confidence limits
for each Medical Decision Point.
ii. Calculate Error Indices for each end of
the 95% confidence limits for each
Medical Decision Point as follows:
Divide the difference (from step i. above)
by the SMILE TE Limit absolute value.
1. Use the percent TEa or the minimum
detectable difference, whichever is
greater.
2. To calculate the absolute TEa value,
convert the TEa percentage to a
decimal and multiply by the reference
method data point.
i. All error indices must fall within -1
and 1. An Excel spreadsheet is
provided to calculate Error Indices
(See Appendix 5 SMILE Error Index
Calculator).
report by selecting the Print Preview
button.
b. Review the Medical Decision Point
Analysis data on page 1 of the report. (See
Appendix 4 for an example EP report)
c. Subtract the X Method MDP from the low
and the high 95% confidence limits for
each Medical Decision Point.
d. Calculate Error Indices for each end of the
95% confidence limits for each Medical
Decision Point as follows: Divide the
difference (from step c above) by the
SMILE TE Limit absolute value.
i. Use the percent TEa or the minimum
detectable difference, whichever is
greater.
ii. To calculate the absolute TEa value,
convert the TEa percentage to a
decimal and multiply by the reference
method data point.
iii. All error indices must fall within -1 and
1. An Excel spreadsheet is provided to
calculate Error Indices (See Appendix
5 SMILE Error Index Calculator).

5. Evaluate acceptability using the table below:

If Then
All error indices fall  Proceed with step 6 below.
between -1 and 1
Any error indices fall  Consider possible causes of inaccuracy.
outside -1 and 1  Troubleshoot before repeating accuracy studies.
 Consult SMILE for assistance.

6. Six Sigma method acceptability evaluation

a. Plot the data in a Six Sigma analysis program such as the SMILE
Accuracy Pack (Appendix 1) or EP Evaluator Six Sigma Metrics module.
b. Enter the Allowable Total Error (TEa) concentration and percent. Refer to
SMILE Chemistry TE Limits table.
c. Enter Medical Decision Points.
d. Enter the slope and intercept calculated in step D above.

476105914.doc 6 of 9 30-Apr-09
 i. If using EP Evaluator, use the Deming Regression slope and
intercept calculated by the program.
ii. If EP Evaluator is not used, it is acceptable to use slope and intercept
calculated using regular regression.
e. Enter the Means and %CVs from your Between Day Precision
experiments.
i. If using EP Evaluator create the report by selecting the Print Preview
button. (See Appendix 6 for an example EP report).

476105914.doc 7 of 9 30-Apr-09
 ii. The SMILE Accuracy Pack and EP Evaluator calculate a Sigma
metric for each Medical Decision point.
iii. Sigma metric=(TEa-Systematic Error)/Random Error
f. Evaluate acceptability at each MDP using the table below:

If Then
The Sigma  The method has unacceptable performance and does not meet
metric less your requirement for quality, even when the method is working
than 2.0 properly.
 It is not acceptable for routine operation.
The Sigma  The method has marginal performance and provides the
metric is necessary quality when everything is working correctly.
between 2.0-  This method will require:
3.0 o 4-8 controls per run
o well-trained operators
o reduced rotation of personnel
o more aggressive preventive maintenance
o careful monitoring of patient test results
o continual efforts to improve method performance
If the Sigma  The method has fair performance and meets your requirement for
metric is quality and can be well-managed in routine operation.
between 3.0-  This method will require 2 control measurements per run using
4.0 standard Westgard QC rules.
If the Sigma  The method has good performance and is clearly acceptable and
metric is can be well-managed in routine operation with only 2 control
between 4.0- measurements per run using standard Westgard QC rules.
6.0
If the Sigma  The method has Six Sigma performance and is even easier to
metric is >6.0 manage and control.

IV. References
A. GCLP Workshop and Workbook18-20 May 2008, Verification of Performance
Specifications, pages 1-33.
B. Clinical and Laboratory Standards Institute (CLSI). User Verification of
Performance for Precision and Trueness: Approved Guideline-Second Edition.
CLSI document EP15-A2 (ISBN 1-56238-574-7). Clinical and Laboratory
Standards Institutes, 940 West Valley Road, Suite 100, Wayne, Pennsylvania
19098-1898 USA, 2005.
C. NCCLS. (Currently CLSI) Evaluation of Precision Performance of Quantitative
Measurement Methods; Approved Guideline—Second Edition. NCCLS document
EP5-A2 (ISBN 1-56238-542-9). NCCLS, 940 West Valley Road, Suite 1400,
Wayne, Pennsylvania 19087-1898 USA, 2004.
D. Clinical and Laboratory Standards Institute (CLSI). Method Comparison and Bias
Estimation Using Patient Samples: approved Guidelines- Second Edition. CLSI
document EP9-A2 (ISBN 1-56238-472-4). Clinical and Laboratory Standards
Institutes, 940 West Valley Road, Suite 100, Wayne, Pennsylvania 19098-1898
USA, 2005.

476105914.doc 8 of 9 30-Apr-09
 E. Clinical and Laboratory Standards Institute (CLSI).Preliminary Evaluation of
Quantitative Clinical Laboratory Measurement Procedure: Approved Guidline –
Third Edition. CLSI document EP10-A3 (ISBN 1-56238-622-0). ). Clinical and
Laboratory Standards Institutes, 940 West Valley Road, Suite 100, Wayne,
Pennsylvania 19098-1898 USA, 2005.
F. Clinical and Laboratory Standards Institute (CLSI).Estimation of Total analytical
Error for Clinical Laboratory Methods. Approved 2003. CLSI document EP21-A
(ISBN 1-56238-502-xX. Clinical and Laboratory Standards Institutes, 940 West
Valley Road, Suite 100, Wayne, Pennsylvania 19098-1898 USA, 2005.
G. EP Evaluator Release 8, David G. Rhoads Associates Inc., www.dgrhoads.com.
H. James O. Westgard, Online Validation Training, Westgard QC, Inc.
www.westgard.com, Sections 11-Determining Bias,12- Estimating Trueness, and
13- Judging Method Acceptability.

476105914.doc 9 of 9 30-Apr-09