William Horwitz
Bureau of Foods HFF-7
Food and Drug Administration
Washington, D.C. 20204
Evaluation of
Analytical Methods Used for
Regulation of Foods and Drugs
Although the Association of Official
Analytical Chemists (AOAC) has been
evaluating and approving methods of
analysis for almost 100 years, there is
practically no discussion in the Jour-
nal of the Association of Official Ana-
lytical Chemists of the criteria for de-
termining which methods should be
approved for regulatory use. These de-
cisions are usually made on the basis
of a method's performance in interlab-
oratory collaborative studies.
John Mandel of the National Bu-
reau of Standards, in his 1981 Shew-
hart medal address (1), pointed out
that the basic objective of conducting
interlaboratory tests is not to detect
the known statistically significant dif-
ferences among laboratories: "The real
FDA
aim is to achieve the practical inter-
changeability of test results." Inter-
laboratory tests are conducted to de-
termine how much allowance must be
made for variability among laborato-
ries in order to make the values inter-
changeable.
An irreducible difference exists be-
tween supposedly identical measure-
ments made in different laboratories.
This point was recently demonstrated
by a group of New Zealand govern-
ment laboratories in attempting to
minimize the discrepancies in values
for blood alcohol between laboratories.
The laboratories went to great pains
to discover every source of error, even
to the extreme of moving analysts
from one laboratory to another. They
found that an analyst increases his or
her intra-analyst variability when
moved to a different laboratory envi-
ronment. They concluded that the
only way to eliminate interlaboratory
variability was to conduct all analyses
Presented at the 95th Annual Meeting of the As-
sociation of Official Analytical Chemists, Wash-
ington, D.C., Oct. 19,1981.
This article not subject to U.S. Copyright
Published 1981 American Chemical Society
Regulations
in a single laboratory and presumably
by the same analyst. Under our legal
system this solution is impossible,
since defendants accused by laborato-
ry evidence have a constitutional right
to produce rebuttal evidence from any
laboratory of their choice. Therefore,
the important question to be answered
in the evaluation of methods of analy-
sis is how much allowance must be
made for between-laboratory variabil-
ity in interpreting the values produced
by different laboratories. If the vari-
ability or error produced by the meth-
od is excessive—that is, it does not
permit effective regulation as required
by the statute—the method must be
judged unacceptable for the intended
purpose.
The purpose of this paper is to
suggest some practical limits of ac-
ceptable variability in methods of
analysis required by AOAC's custom-
ers—the regulatory agencies and the
regulated industries. The collabora-
tive study procedure has provided the
essential data for developing this in-
formation.
Method Characteristics
Methods are usually evaluated on
the basis of three characteristics: reli-
ability, applicability, and practicabili-
ty. For our present purpose, reliability
is the overriding consideration. In gen-
eral, when a need exists for a method,
we have to accept any reasonable de-
gree of reliability. Applicability to a
wide range of sample types and practi-
cability with respect to cost, time, and
training constraints both assume
greater importance when there are
several competing methods.
The important aspects of reliability,
listed in their approximate order of
importance for most purposes, are:
• Reproducibility, or total be-
tween-laboratory precision. This is the
measure of the ability of different lab-
ANALYTICAL CHEMISTRY, VOL. 54, NO. 1, JANUARY 1982 · 67 A

I Drugs
in Feeds
Pharma­
ceuticals Pesticide
Residues
•ε
Major Minor
Nutrients Nutrients
! between-laboratory" variability,
Concentration
Figure 1. The general curve relating interlaboratory coefficients of variation (ex­
pressed as powers of two on the right) with concentration (expressed as powers of
10) along the horizontal center axis
oratories to check each other. It is the
overall measure of variability, includ­
ing the within-laboratory component.
• Repeatability, or within-labora­
tory precision. This is the measure of
the ability of a laboratory (or analyst)
to check itself.
• Systematic error or bias (some­
times also called "accuracy or inaccu­
racy"). This is the difference of the
value(s) obtained from the true, as­
signed, or consensus value(s).
• Specificity (when required). This
is the ability of the method to measure
what it is intended to measure.
• Limit of reliable measurement
(when required). This is the smallest
amount (or concentration) of a mate­
rial that can be measured with a stat­
ed degree of confidence.
Which of these factors is most im­
portant depends upon the purpose for
which the data will be used. In regula­
tory analysis, analytical values are
used for three major purposes: to sur­
vey a field to determine the extent of
a problem; to monitor trends to deter­
mine if any corrective action has to be
taken; and to determine compliance
with an economic or legal specifica­
tion. A different emphasis on the vari­
ous method characteristics is required
to accomplish each purpose. In
surveying a field, the normal variabili­
ty of the measurement of a commodity
or the environment is usually so large
68 A · ANALYTICAL CHEMISTRY, VOL. 54, NO. 1, JANUARY 1982
Aflatoxlns
Trace
Elements
that a high degree of accuracy and
precision is not an important require­
ment. The averaging of numerous im­
precise determinations often provides
a surprisingly good mean. Sometimes
all that is needed is to differentiate
samples with "none" of the analyte
from those that contain "significant"
amounts. In monitoring trends, the
systematic error, as long as it is con­
stant, is not important. The precision
must be good enough to detect when a
"significant" difference occurs. In
compliance activities a high degree of
accuracy (as lack of bias) and preci­
sion are required at the specification
level, unless the specification is based
upon the method itself, in which case
only precision is pertinent. The preci­
sion requirement may decrease as the
distance from the specification value
increases. When the "no residue" re­
quirements of the Federal Food, Drug,
and Cosmetic Act are involved, speci­
ficity and limit of reliable measure­
ment are the most important consid­
erations. In practical work, other re­
quirements come into play. In surveys,
the need to analyze many samples
makes a rapid method a necessity; in
monitoring, repeated sampling of the
same population is important; in com­
pliance, practicality, although impor­
tant, is secondary to reliability.
Therefore, it is apparent that there
is no such thing as a "best" method
since the definition of "best" will vary
with the purpose for which the meth­
od will be used. Since this is not usual­
ly known beforehand, we must usually
assume that our primary interest will
be in the achievement of a suitable de­
gree of precision and bias; require­
ments for specificity and limit of reli­
able measurement will usually be self-
evident.
In regulatory work, or even in ana­
lyzing for adherence to commercial
specifications, between-laboratory
variability is the most important fac­
tor. Bias can be tolerated. If it is con­
stant, a correction factor can be used.
If it is variable, it becomes a compo­
nent of reproducibility (between-labo­
ratory variability). In fact, Youden (2)
equates systematic error to the "true
which in our terminology is the repro­
ducibility adjusted for the within-lab­
oratory variability (repeatability).
Bias, as a recovery factor, particularly
in modern trace analysis, is generally
permitted to seek its own level, pro­
vided it is above 60-80% (3).
Interlaboratory Precision
It would appear that any systematic
approach to estimating what consti­
tutes a reasonable precision would be
an almost impossible task. Methods
are composed of almost infinite com­
binations of dissolution, cleanup, and
measurement procedures. These innu­
merable combinations are applied to
pure substances and complex mixtures
as solids, liquids, and gases by ana­
lysts with various degrees of compe­
tency. Yet, despite this complexity, we
have found that analytical variability
can be summarized (in an oversimpli­
fied fashion to be sure) by plotting the
determined mean coefficient of varia­
tion (CV), expressed as powers of two,
against the analyte level measured, ex­
pressed as powers of 10, as shown in
Figure 1, taken from our recent paper
on quality control (4).
The sources of these data are an ex­
amination of over 150 independent
AOAC interlaboratory collaborative
studies covering numerous AOAC top­
ics, from drug preparations and pesti­
cide formulations on the high end of
the concentration scale to aflatoxin
contaminants at the low end, with im­
portant stops in between at pesticide
residue and trace element concentra­
tions. At least five analytical meth­
ods—chromatography, atomic absorp­
tion spectrometry, spectrophotometry,
polarography, and bioassay—are in­
volved. A convenient, easily remem­
bered reference point is that at 1 ppm
(HT 6 ), the CV is 24 = 16%. Other
points are given in Figure 1.
The most important and startling
point is that this idealized smoothed
curve is independent of the nature of
 "S
c
«
ô
3"

"S ο
+i
c
_o
s^t°r::-
ϋ a
1—

>π

ι
σ>
Ο

I ?
0)
'δ
C ^ W c i e n t of Variation

ε<ο
ο
υ
Year Sample Fat Content (%)

Figure 2. The performance of laboratories analyzing EPA's Figure 3. The interlaboratory coefficient of variation and
quality-control samples for pesticide residues in fat and blood standard deviation (absolute) of the gravimetric ether extrac­
over a 13-year period (4). Fat · , blood Ο tion method for the determination of fat in meat as a function
of the concentration (%) of fat (6)

metric) (6) and methyl esters of fatty

50
«-· 40
c tion) (10). These examples are given
"S
U 30
> for reference in Figures 5 and 6. When
ο the specific method curve deviates
c 20
1 10
Ο that method.
ο There is one other useful piece of
100 32 10 3.2 1 0.3 0.1
Concentration (%)
Figure 4. The interlaboratory coefficient of variation (center
curve) and the 9 5 % confidence limits (two outer curves) for
the gas chromatographic determination of methyl esters of
fatty aeids ( 7)
the analyte or of the analytical tech­ (5), show that the between-laboratory
nique that was used to make the mea­ CVs improved with analytical experi­
surement. This curve is merely a sum­ ence, but only to a minimum value ap­
mary of available interlaboratory data, proximating the 16% found in the col­
independent of such external influ­ laborative studies for pesticide resi­
ences as sampling and contamination. dues in food. Similarly, the quality
The significant data points are aver­ control monitoring of laboratories de­
ages of a number of studies of similar termining aflatoxin in peanuts for cer­
analytes whose CVs may cover a factor tification purposes (4) gives a value
of two in either direction; similarly the that corresponds to the 32% CV on the
concentration range may also extend general curve for aflatoxin at the 10
in both directions by an order of mag­ ppb concentration level.
nitude or so. But in general, the values
Other evidence suggests that this
taken from this curve are indicative of curve may represent a floor for the
achievable and acceptable perfor­
precision of analytical methods in the
mance of an analytical method by dif­ interlaboratory environment. In fact,
ferent laboratories. it appears that all methods more or
Independent evidence also supports less follow such a curve up to a point
this general precision curve. Quality- where the precision begins to deterio­
control studies of pesticide residue de­ rate at an even faster rate. This phe­
terminations in fat and blood by EPA nomenon is shown by methods in the
contractors, summarized in Figure 2 macro scale such as fat in meat (gravi­
acids (gas chromatographic) (7), as
well as methods in the micro scale
such as pesticide residues (gas chro­
matographic) (8, 9), and trace ele­
ments (predominantly atomic absorp­
in Figures 3-6, with the general preci­
sion curve labeled "AOAC" drawn in
markedly from the general precision
curve, we have obviously gone beyond
the limit of reliable measurement for
information that has been extracted
from the data used to construct the
general precision curve. The precision
component due essentially to analysts
(within-laboratory error or repeata-
bilty) in AOAC studies is usually one-
half to two-thirds of the total variabil­
ity (the combined effect of the within-
and between-laboratory variability),
as given in Figure 1. Ratios of repeat­
ability to reproducibility considerably
less than 0.5 indicate a very personal
method: Analysts can check them­
selves very well but they cannot check
other analysts in other laboratories.
This situation suggests that the direc­
tions require reworking or that the
reference standards may differ from
laboratory to laboratory. However,
this low ratio is also typical of meth­
ods requiring considerable personal
skill, such as counting filth elements
or mold. A very high ratio can indicate
that individual analyst replications
are so poor that they swamp out the
between-laboratory component.
Outliers
Another index that may prove use­
ful for evaluating methods is the per­
centage of outliers reported in a col­
laborative study. Every analyst has

70 A · ANALYTICAL CHEMISTRY, VOL. 54, NO. 1, JANUARY 1982

 100
80
I-
Ô
40
ο c
20
100 10 1.0
Concentration (ppm)
Figure 5. The interlaboratory coefficients of variation for the
determination of pesticide residues in butterfat (8) and in
wildlife (9) by gas chromatographic methods. Wildlife O, but-
terfat ·
implicit faith that if a method is fol-
lowed exactly, the correct result will
automatically be produced. However,
our review of several hundred collabo-
rative studies in which the samples
were examined as true unknowns re-
veals that often 5 to 15% of the re-
ported values are statistical outliers—
values that are far outside the region
where most of the other values reside.
Outliers are produced by experienced
chemists as well as by novices, at
macro as well as at trace levels. Schul-
ler et al. (11), in their review of afla-
toxin methods, noted that they had to
tolerate a 10% outlier rate in recom-
mending methods for international
referee status. In the analysis of moon
rocks from the Lunar Analysis Pro-
gram of the U.S. National Aeronautics
and Space Administration, Morrison
reported (12) that almost 7% of the
values had to be discarded as outliers.
Outliers produced by a single ana-
lyst or within a laboratory are usually
inconspicuous, since they are either
unrecognized from analysis of single
samples where there is nothing to
72 A · ANALYTICAL CHEMISTRY, VOL. 54, NO. 1, JANUARY 1982
120
£
c
ο
.S
« 80
«*-ο
φ
δ
«
ο
υ
40
01 0.01
Figure 6. The interlaboratory coefficients of variation of trace
elements in blood by various methods as calculated from the
data reviewed by Versieck and Cornells ( 10)
compare the result with, or they are
eliminated by repetition. In an inter-
laboratory situation with blind sam-
ples, where there is no opportunity to
censor the data, outliers are more ob-
vious. In current AOAC collaborative
studies, outliers are usually eliminated
by the techniques suggested by You-
den (2): a ranking test to remove con-
sistently high or low laboratories, fol-
lowed by the elimination of outlying
individual values by a Dixon test in-
volving the deviations of extreme
values.
We have only recently realized the
importance of outliers in the evalua-
tion of methods of analysis for approv-
al by the AOAC. Outliers are a fact of
laboratory life and allowance must be
made for them. By definition, they lie
at the extreme points of the statistical
frequency distributions of a series of
analytical values. Therefore, they have
a large influence on the magnitude of
the indices used to measure the per-
formance of methods.
Figures 7 and 8, using the data from
the collaborative study of aflatoxin in
1000 100 10.0 1.0
Concentration (ppb)
green cacao beans (13), illustrate this
effect. The repeatability (within-labo-
ratory variability) changes from an
unacceptable CV of 50% for the 20 val-
ues from 10 laboratories to a marginal-
ly acceptable 36% by the omission of
one value classified as an outlier by
the Dixon test. In this case, as in many
others, there is no question about the
classification as an outlier since 18
ppb of aflatoxin had been added to
each sample. In this study, the mean
changed little by the elimination of
the outlier: from 16.0 to 14.6 ppb, or
expressed in terms of recovery, from
89 to 81%.
This particular example shows only
a 5% outlier rate. We have seen meth-
ods approved by the AOAC with an
outlier rate as large as 50%. There is
only one legitimate excuse for elimina-
tion of laboratories without a statisti-
cal test: intentional or unintentional
failure to follow the method. An inten-
tional failure occurs when the speci-
fied equipment or reagent is unavail-
able and failure to substitute will
mean dropping out of the study. But
 Added

11
0.08
10
0.07 CV = 36%
9
CV = 50%
„ 0.06
8 a
Ο
Ζ 7 % 0.05
-Q to
9
_i 5 SE 0.04
ο
4
3 Ι °·03
ο
2
ι? 0.02
1 0.01
0 10 20 30 40 0 10 20 30 40
Total Aflatoxin (ppb) Concentration (ppb)

Figure 7. Original data from the interlaboratory study of the Figure 8. The data from Figure 7 plotted as a normal frequen­
determination of aflatoxin in cacao beans ( 13). The two val­ cy distribution with the outlier included (20 points, broken
ues from each laboratory are plotted horizontally. The circled line) and the outlier excluded (18 points, solid line)
value is an outlier by the Dixon test

substitution on the basis of "it cannot
possibly affect the results" is inexcus­
able. Collaborative studies are very
expensive in terms of time and man­
power. Jeopardizing their success with
untested changes undermines the en­
tire collaborative study.
Although many AOAC studies show
no outliers, a 5 to 15% outlier rate,
particularly at the ppm and ppb lev­
els, is not at all unusual. If but one of
five or six laboratories required in a
minimum statistical pattern turns out
to be an outlier by the Youden rank­
ing test, we have reached a 20% rejec­
tion point. It appears that we may
have to tolerate a 20% outlier rate, be­
cause that is the penalty to be paid if
we use a minimum number of partici­
pating laboratories. This is one of the
reasons why having at least 10 labora­
tories in a study will improve the
chances of acquiring adequate data for
statistical evaluation.
There is one important statistical
problem with outliers: What outlier
test should be used? This is a complex
statistical problem whose solution de­
pends on the true distribution of val­
ues. Chemists seem to have little diffi­
culty in applying intuition and experi­
ence to this problem, but many statis­
ticians are appalled at this approach.
We hope to apply a number of outlier
tests to a number of AOAC collabora­
tive studies to determine if any of the
several dozen procedures described in
the statistical literature is best suited
for application to interlaboratory
work.
False Positives and False
Negatives
Another potentially useful suitabili­
ty index for evaluation of methods
may be the percentage of false posi­
tives and false negatives. False posi­
tives (excessively high blanks) may
appear when working at any concen­
tration level, but the appearance of
both false positives and false negatives
is characteristic of trace analysis.
These values are not necessarily out­
liers, but they can be. We have noted
in our examination of the available
aflatoxin studies that the percentage
of false negatives increases much like
our CV/concentration curve as the
concentration approaches zero. This
phenomenon may be more useful for
delineating a limit of reliable measure­
ment, i.e., the concentration at which
the proportion of false negatives is
more than 20% (or some other num­
ber), than for evaluating the perfor­
mance of methods in general.
Bias or Systematic Error
Up to now I have paid little atten­
tion to the matter of bias or systemat­
ic error because in most cases it takes
care of itself. Very few methods are re­
jected because of low or high recov­
eries. In the case of macro methods,
recovery is frequently very close to
theoretical, because these methods are
usually based upon stoichiometry or
basic physical principles of extraction
or separations, and the amount of ana-
lyte available for measurement is not
limited. Furthermore, many methods
in food chemistry are empirical—they
are based upon the faith that other
participants will adhere to the speci­
fied directions to produce equivalent
results. Empirical methods by defini­
tion have no bias. In trace analysis,
the precision characteristic usually
takes care of recovery, because the
random error is often as large or larger
than the systematic error. If the recov­
ery is low but repeatable, as in isotope
dilution methods, any recovery is ac­
ceptable since correction can be made
back to the 100% level. If it is variable,
even though within acceptable limits,
the correction factor procedure will
not do. The method must then be ac­
companied by sufficient recovery data
to indicate the boundaries of perfor­
mance. In the proposed "SOM" docu­
ment of the FDA, recovery limits were
given as more than 80% for concentra­
tions of 0.1 ppm and above, and more
than 60% for lower concentrations (3).
Naturally, we would prefer higher re­
coveries. But these figures do appear
to be reasonable in light of actual re­
coveries under ordinary, and not col­
laborative, conditions.
Summary
The primary objective of interlabo­
ratory studies is to determine if we
have achieved interchangeability of
test results among laboratories. But
interchangeability is a function of the
purpose for which the results will be
used: to survey a field; to monitor
trends; or to determine compliance

74 A · ANALYTICAL CHEMISTRY, VOL. 54, NO. 1, JANUARY 1982

 with a specification. E a c h of t h e s e

How to choose p u r p o s e s places different e m p h a s i s

u p o n t h e characteristics or a t t r i b u t e s

an ICP Spectrometer of m e t h o d s : their reliability, applica­

bility, a n d practicability. For regulato­
ry p u r p o s e s reliability is p a r a m o u n t
a n d t h e between-laboratory precision
is t h e critical c o m p o n e n t . I n general,
t h i s precision can be r e p r e s e n t e d by
t h e following equation:
CV (%) = 2 ( 1 -°- 6 1 o « c )
where C is t h e concentration ex­
pressed as powers of 10 (e.g., 1 p p m =
1 0 - 6 ) . T h e coefficient of variation
doubles for each decrease of concen­
t r a t i o n of two orders of m a g n i t u d e .
T h e between-laboratory coefficient of
variation a t 1 p p m is 16% (2 4 ). T h e
w i t h i n - l a b o r a t o r y CV should ordinari­
ly b e one-half t o t w o - t h i r d s t h e b e ­
t w e e n - l a b o r a t o r y CV. O t h e r potential
evaluation criteria include an outlier
r a t e of 20% or less a n d an acceptable
level of false positive values a n d false
negative values. Acceptable specificity
a n d limit of reliable m e a s u r e m e n t m a y
and an ICP company. also have t o be based on t h e level of
false positives a n d false negatives. R e ­
#2 Look for superior performance in covery values ordinarily t a k e care of
any matrix. t h e m s e l v e s a t macro levels, b u t a t
Yes No t r a c e levels, 60% at t h e p p b level a n d
1. Are low detection limits (ppb) 80% a t t h e 0.1 p p m level m a y be t h e
lowest acceptable recoveries.
important?
2. Has the instrument been References
designed specifically for ICP? (1) Mandel, J. Quality Progress August
3. Can alignment be performed and 1981, 34-36.
verified easily- graphically? (2) Youden, W. J.; Steiner, Ε. Η. Statisti­
cal Manual of the AOAC (1975). Associa­
4. Does the instrument have the tion of Official Analytical Chemists:
stability and accuracy you need? Washington, D.C.
(3) Fed. Regist. March 20,1979, 44 (55),
17 070-114.
The more "yeses" you checked, the more reasons (4) Horwitz, W.; Kamps, L. R.; Boyer,
you have to learn more about the Baird Plasma K. W. J. Assoc. Off. Anal. Chem. 1980,
63,1344-54.
Spectromet, the most advanced system in the ICP (5) Watts, Randall R. "Proficiency Testing
field today. Designed from the ground up for ICP, the and Other Aspects of a Comprehensive
Plasma Spectromet combines Baird quality optics Quality Assurance Program." In "Opti­
mizing Chemical Laboratory Perfor­
with the most advanced innovations in plasma mance through the Application of Quali­
spectroscopy and high speed graphic computer ty Assurance Principles"; Garfield, Fred­
erick M. et al., Eds.; Association of Offi­
technology. With the Baird Plasma Spectromet, you cial Analytical Chemists: Arlington, Va.,
can have simultaneous multielement analysis in a 1980, pp 87-115.
matter of seconds. (6) Pettinati, Julio D.; Swift, Clifton E.
J. Assoc. Off. Anal. Chem. 1977, 60,
As the world leader in optical emission spectrom­ 600-608.
eters since 1936, Baird can be depended upon for (7) Firestone, D.; Horwitz, W. J. Assoc.
superior performance, quality workmanship, and a Off. Anal. Chem. 1979, 62, 709-21.
(8) Snelson, J. T. "Analysis of Organochlo-
total commitment to our customers. rine Residues in Butter Fat." Document
For more information about the Baird Plasma CX/PR 77/9 (1976); Food and Agricul­
ture Organization: Rome, Italy.
Spectromet and a free copy of the entire "How to (9) Holden, A. V. "The OECD Interna­
Choose" series, call or write us at Baird Corporation, tional Co-operative Studies of Organo-
125 Middlesex Turnpike, Bedford, MA 01730. chlorine Residues in Wildlife." In "Envi­
ronmental Quality and Safety"; Coul-
Tel: (617) 276-6094. Telex: 923491. ston, F.; Korte, F., Eds.; George Thieme
Publishers: Stuttgart; Supplement Vol.
Ill, pp 40-46.
(10) Versieck, Jacques; Cornells, Rita,
Anal. Chim. Acta 1980,118, 217-54.
(11) Schuller, P. L.; Horwitz, W.; Stoloff,
L. J. Assoc. Off. Anal. Chem. 1980, 59,

BAIRD
The Spectroscopy People
1315-43.
(12) Morrison, G. H. Anal. Chem. 1971, 43
(7), 22-31A.
(13) Scott, P. M.; Pryzybylski, W. J. Assoc.
Off. Anal. Chem. 1971,54, 540-44.
CIRCLE 26 ON READER SERVICE CARD
76 A · ANALYTICAL CHEMISTRY, VOL. 54, NO. 1, JANUARY 1982