This article was downloaded by: [Moskow State Univ Bibliote]

On: 17 October 2013, At: 04:25

Publisher: Taylor & Francis
Informa Ltd Registered in England and Wales Registered Number:
1072954 Registered office: Mortimer House, 37-41 Mortimer Street,
London W1T 3JH, UK

Analytical Letters
Publication details, including instructions for
authors and subscription information:
http://www.tandfonline.com/loi/lanl20

Statistical Estimation of
Linear Calibration Range
a a
L. Cuadros Rodríguez , A. M. García Campaña
a
& J.M. Bosque Sendra
a
Department of Analytical Chemistry , Faculty
of Sciences University of Granada , Fuentenueva
s.n. E-18071, Granada, Spain
Published online: 22 Aug 2006.

To cite this article: L. Cuadros Rodríguez , A. M. García Campaña & J.M. Bosque
Sendra (1996) Statistical Estimation of Linear Calibration Range, Analytical
Letters, 29:7, 1231-1239, DOI: 10.1080/00032719608001471

To link to this article: http://dx.doi.org/10.1080/00032719608001471

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all
the information (the “Content”) contained in the publications on our
platform. However, Taylor & Francis, our agents, and our licensors
make no representations or warranties whatsoever as to the accuracy,
completeness, or suitability for any purpose of the Content. Any opinions
and views expressed in this publication are the opinions and views of
the authors, and are not the views of or endorsed by Taylor & Francis.
The accuracy of the Content should not be relied upon and should be
independently verified with primary sources of information. Taylor and
Francis shall not be liable for any losses, actions, claims, proceedings,
demands, costs, expenses, damages, and other liabilities whatsoever
or howsoever caused arising directly or indirectly in connection with, in
relation to or arising out of the use of the Content.

This article may be used for research, teaching, and private study
purposes. Any substantial or systematic reproduction, redistribution,
 reselling, loan, sub-licensing, systematic supply, or distribution in any
form to anyone is expressly forbidden. Terms & Conditions of access
and use can be found at http://www.tandfonline.com/page/terms-and-
conditions
Downloaded by [Moskow State Univ Bibliote] at 04:25 17 October 2013
 ANALYTICAL LETTERS, 29(7), 1231-1239 (19%)

STATISTICAL ESTIMATION OF LINEAR CALIBRATION RANGE

Downloaded by [Moskow State Univ Bibliote] at 04:25 17 October 2013

Key words. Linearity. Goodness-of-fit, Lack-of-fit, Precision, Linear range

L. Cuadros Rodriguez(*),A.M. Garcia Campaiia and J.M. Bosque Sendra

Department ofAnalytica1 Chemistry. Faculty of Sciences. University of Granah.

Fuentenueva s.n. E-18071 G r a n d , Spain

Abstract.

The "in-line'' and "on-line" linearity terms of a linear calibration are defined. "On-line"
linearity, which is calculated from the standard deviation of the slope, is proposed as a
performance characteristic for the analytical method. The relation between the precision of an
analytical result, and the analytical sensitivity and "on-lime'' linearity of the corresponding chemical
calibration is established. The simplified equation that is obtained makes it possible to rapidly
estimate the standard deviation with a sufficient degree of freedom, even when few analysis
replicates are available. An indication is also given of the suitable equation for a more accurate
calculation. The use of the two linearities, both "in-line'' and "on-line", is discussed, concerning
the determination of the linear calibration range, for those techniques with which curvatures are
obtained when the concentration of the standard analyte exceeds a particular value.

*To whom correspondence should be addressed.

1231

Copyright 0 1996 by Marcel Dekker, Inc.

 1232 CUADROS RODRiGUEZ, GARCiA CAMPARA, AND BOSQUE SENDRA

Introduction
Instrumental analytical methods require the establishing of a chemical calibration, that is,
the estimation of the mathematical function that will relate the analytical signal (instrumental
response) to the analytical property to be determined, usually the concentration.
When there exists a first order calibration hnction (Equation 1. R, instrumental response,
c, concentration, a, intercept, and b, slope), the estimation of the corresponding equation is
normally performed by applying linear least squares regression
Downloaded by [Moskow State Univ Bibliote] at 04:25 17 October 2013

R = a +b.c (1)
Using this model, it is possible t o perform inferential statistics both on predicted signal
values for a standard analyte concentration and on concentration values that correspond to a
particular signal value In both cases, uncertainty can be quantified by means of signal or
concentration standard deviations
This is why the analytical chemist must make sure that the linear model is correct.
Linearity is usually characterised by Pearsons linear correlation coefficient' or by determination
coefficient*, although in a recent paper the Analytical Methods Committee of the Royal Society
ofchemistry recommended the use of the "lack-of-fit" F-test to detect non linearit? This test
is based on the comparison of the variability o f a set of measurements with respect to the mean
(due to pure error) in relation to the variability of these means with respect to the line calibration
(due to lack-of-fit), if the latter is significantly greater than the former the fit is not good, and thus
the linearity is bad
Clearly the acceptance or otherwise of the linear model depends on the magnitude of the
errors of repeatability ofthe corresponding signals, since, if these are large then small curvatures
may pass unobserved

"In-line" and "on-line" linearity

It is necessary to distinguish two characteristics that are both described by the term
"linearity",but which refer to different properties. "In-line" linearity refers to the straight line fit
(goodness-of-fit) of the analytical signal values obtained experimentally in relation to the analyte
concentration, that is it describes the absence of curvature on the calibration line. "On-
line" linearity indicates the greater OJ lesser dispersion of the data around the calibration line
"In-line'' linearity (also termed rectilinearity) is that which is verified by the "lack-of-
fit" test. I n addition, some methods to detect curvature have been used, such as the study of
residual plots', log-log plots, response indexes from log-log plots and linearity plots (plots of
 LINEAR CALIBRATION RANGE 1233

sensitivity vs. c~ncentrationj~,the slope ranking m e t h ~ d ~ or

. ~ ,the study of the level of
significance of the coefficient of the quadratic term of the calibration equation'. "On-line"
linearity is not usually quantified with a specific parameter and the residual standard deviation is
used for its evaluation, although proposals have been made for the use of the relative standard
deviation of the slope', RSD(bj = sbh.
"In-line'' linearity can not be evaluated separately from "on-line" linearity, since one
depends on the other. A low degree of "on-line" linearity means that it is not possible to
Downloaded by [Moskow State Univ Bibliote] at 04:25 17 October 2013

accurately distinguish the curvature by means of the "lack-of-fit" test.

Therefore, we think it of interest to use the relative standard deviation of the slope as a
measure of the precision of the chemical calibration, once a non significant "lack-of-fit'' has been
obtained. Thus "on-line" linearity can be quantified by a parameter defined as:
L I N o , ( % ) = 1 0 0 [l -RSD ( b )I (2)
The above defined linearity may be considered another performance characteristic of the
analytical method, in addition to the classical ones of sensitivity, precision or lower limits'.

Precision
Precision measures the degree of uncertainty of the analytical result. This uncertainty is
due to the dispersion of the instrumentalresponses and to the use of estimated slope and intercept
values in the calibration equation when transforming the signals into concentrations. Both these
factors are present in the residual standard deviation, s ~ ,on~ the
, regression line. The precision,
which is expressed by the standard deviation of the estimated concentration values, sc, therefore
depends on the dispersion of the residuals. However, as the magnitude of sR,cin absolute terms
depends on the magnitude of the instrumental responses, it is more representative to use
normalized values with the slope of the calibration line. For this purpose, the precision really
depends on parameters such as the analytical sensitivity ofthe calibration', S = s&b, and on "on-
line" linearity.
Certainly, the variance on a estimated concentration value, szc0, can be calculated, using
linear regression analysis, from Equation 3:

(the significance of different terms is shown in the glossary). This is an approximate equation of
the general equation that is provided in the Appendix; a detailed discussion of its validity may be
found in Ref. 8.
 1234 CUADROS RODRfGUEZ, GARCfA C A M P ~ AAND
, BoSQUE SENDRA

As the variance of the slope is expressed as:

and by u s i g the term 4 for the difference (c,, - y),calculated from (R,- R)h, Equation 3 may
be expressed as.
1 1
57 = s (-+-) +RSD ( b ).d,' (5)
n m
Downloaded by [Moskow State Univ Bibliote] at 04:25 17 October 2013

and thus it is shown that the precision depends on the analytical sensitivity of the calibration, S,
on the "on-line" linearity, RSD(b), and on the distance from the analytical result to the central
concentration of the calibration line, do. The precision will be maximized, and independent of
linearity, when the concentration found coincides with the centre of the interval used in the
calibration, do=O
Equation 5 provides a fast and simple estimation of the standard deviation of an analytical
result, even when few replicates ofthe analysis are performed, since the values thus calculated use
the information from all the measures taken to establish the calibration, and not only those
corresponding to the sample to be analysed This value better represents the precision of the
result than that obtained only from the repetition of the measures from the sample (the standard
deviation thus obtained has n-2 degrees of freedom (DOF), compared to that derived from
repeatability, which only has m-1)

Linear calibration range

The conclusions of the "lack-of-fit" test may be used to determine the amplitude of the
hear dynamic range of a calibration, above all in those techniques for which the response is linear
at a low analyte concentration, but which exhibits curvature as the concentration increases' In
cases where replicates are not available, the use of robust regression (of least median squares) has
also been proposed, with the same intention"
As far as we know, no generally accepted definition of the upper boundary and thus of the
linear range for characterizing a procedure has been proposed, and the concentration where the
response differs by a certain percentage (i e 3 or 5%) was usually used''
The "lack-of-fit'' test leads to a P, value that indicatesthe probability that the bias found
between the experimental values and those predicted by the calibration equation is due only to
random experimental errors, and thus a high value for P-value corroborates the "in-line'' linearity
of the calibration, while values of less than 0.05 might indicate the existence of curvature
 LINEAR CALIBRATION RANGE 1235

Nevertheless, the magnitude of the P, value is not proportional to the degree of "in-line"
linearity, that is, a P, of 80% does not indicate an "in-line'' linearity twice that of a value of
40%, but rather that the conclusions only have a probabilistic value
The P-value is joined to experimental statistic Snedecor-F of the lack-of-fit test The
calculation implies the use of statistic-mathematic equations and its description overcomes the
object of this work A detailed and descriptive example is shown in the text of Massart ei all2
However, the P-value can be easily calculated by using readily available statistics packages In
Downloaded by [Moskow State Univ Bibliote] at 04:25 17 October 2013

this paper, all the calculations were carried out using the STATGRAPHICS data analysis
package13
In those cases in which this test is used to determinethe linear interval of a calibration with
an "on-line" linearity that is not very high (LINOL<99%), it is advisable, even when a non
significant "lack-of-fit'' has been obtained, to remove the pairs that correspond to the greatest
concentration and to repeat the linearity study, noting whether an appreciable increase in the PLoF
value is observed If this occurs, the analyst should consider the possibility of reducing the
calibration interval, in order to improve the precision of the results

Example 1
The data given in Table 1 represent the simulated fluorescence signals for two calibrations
with equal analyte concentrations. Three replicates were performed for each concentration, such
that the mean values of each calibration were equal, but the means of the second one featured a
greater dispersion. The final point was chosen such that the values were inferior to those
predicted for a linear behaviour over the whole interval (see Fig. 1).
Treatment of the data produces PLoF=2.3% and LINOL=99.1%for calibration 1, and
PL0r26.8% and LINOL=98.8%for calibration II. It may be seen that while curvature is detected
in the first calibration, the greater dispersion of means in the second conceals this fact and leads
to an incorrect conclusion.
However, when we remove the three signal values corresponding to the concentration
value of 250, we obtain PLoF'71.2% and LINOL=99.2%for calibration I, and PLoF=90.9% and
LINoL=98.8% for calibration 11. There has been a large increase in the&, values of both
calibrations, which in the case of the second indicates that curvature may exist in the initial
calibration. In such a situation, the analyst should choose the interval lying between 0 and 200
concentration units.
If the procedure is again performed, but additionally the 200 value is removed, this
produces PLo~88.2%and LINoL=99.00/0for calibration I, and PLoF=95.6% and LINoL=98.3%
 1236 CUADROS RODdGUEZ, GARCfA C A M P ~ AAND
, BOSQUE SENDRA

TABLE 1
Simulated data sets for calibrations I and I1
-___
Concentrations Responses
(Arbitrary units) I 11 R,
0 15 20 16 14 22 15 17
50 79 85 79 78 87 78 81
100 119 148 144 151 149 141 147
150 309 213 214 207 214 215 212
200 278 271 273 280 270 272 274
Downloaded by [Moskow State Univ Bibliote] at 04:25 17 October 2013

250 333 329 327 336 326 328 330

K, mdicstes the means of three repllcstes ofeach smndard

0-' ' ' '

0 60 100 160 200 160 3W 0 w loo 160 200 260 800
Concentratbn Concentration

FlGURE 1
Calibration curves for data sets I and I1 of Table 1 (.), response replicated for
each concentration, (n), mean of three replicates

for calibration I1 These results are slightly better than the previous ones, which indicates that the
maximum concentration for the calibration should be the value of 200.

Example 2
The data in Table 2 refer to the signal obtained for five replicates of analyses made from
the same sample Data treatment was performed using calibration I1 (see Table 1) in the interval
0-250 (A) and 0-200 (B)
 LINEAR CALIBRATION RANGE 1237

TABLE 2
Results for five replicates of a simulated analysis

Analysis No Response

1 225
2 218
3 219
4 224
5 22 1
Downloaded by [Moskow State Univ Bibliote] at 04:25 17 October 2013

%=221 4
SRo'3 0s

Q mhcates the means of five replicates. and sRaIS the standard

deswation with 4 degrees of freedom

The mean analyte concentration calculated from (A) was co(A)=160 3 with a precision
(Equation 5) of s1603(A)=2 16 (16 DOF), while the (B) data set producedoc (B)=158 3 and
~ 1 5 3(8)=1
8 90 (13 DOF)
It is apparent that the result obtained with (B) produces greater precision by using a higher
quality calibration Moreover, the result of (A) is biased with respect to (B), since the curvature
produces biased estimates of the slope and the intercept of the calibration equation

Appendix
The complete equation that permits an exact calculation of the concentration variance,
derived from the inverse regression, i d 4

where g is a fimction of "on-line" linearity, given by:

and using a Student-t with a 95% confidence level and n-2 degrees of freedom
Combining Equations 6 and 7, the following precision is obtained:

RSD- (bj
 1238 CUADROS RODRIGUEZ, GARCf.4 CAMPMA, AND BOSQUE SENDRA

Conclusions
'The application of the "lack-of-fit'' test to the calibration data produces an estimation of
the "in-line" linearity that may be incorrect, depending on the magnitude of the dispersion of the
experimental responses Moreover, if the test is used to determine the linear interval, the "in-line''
and "on-line" linearities must be evaluated in conjunction before any decisions are taken, as the
evidence of curvature may be hidden The incorrect choice of this linear calibration range will
produce biased results that are less accurate than those obtained when the correct linear interval
Downloaded by [Moskow State Univ Bibliote] at 04:25 17 October 2013

is employed
The proposed model could be helpfkl to determine the linear calibration range of analytical
tectuuques, such as spectrofluorimetry,which show significant curvature when the concentration
of standard increases

Glossary
a Intercept of calibration line
b Slope of calibration line
c Concentration
c,
- Each analyte standard concentration used in calibration
c Arithmetic mean of the ci concentration values
c,) Estimated concentration of analyte of an analyzed sample
do Difference between any "co" concentration value and c
LINIL "ln line" linearity
LIN,, "On line" linearity
m Repetitions of the analysis
n Total number of pair of values used in calibration
PLOF P-value of lack-of-fit F-test (indicating the probability that the bias found between
the experimental values and those predicted by the Calibration equation is due to
random experimental error)
R
- Instrumental response
R Arithmetic mean of the n instrumental response values for calibration
&, Anthmetic mean of the signal values for m analyses camed out on sample solution
of analyte concentration cg
RSD(b) Relative standard deviation of slope
S Analytical sensitivitv
sr," Standard deviation on residuals of regression
s R,c Variance on residuals of regression
:> Standard deviation on slope
Variance on slope
Standard deviation on predicted concentration
sc
sLc0 Variance on a estimated concentration value, co- in a determination

References
1. Analytical Methods Committee. Analyst, 113, 1469-71 (1988)
 LINEAR CALIBRATION RANGE 1239

2. M.J. Adams, "Chemometricsin Analytical Spectroscopy", RSC Analytical Spectroscopy

Monographs. N. Bamett (Ed.), Royal Society of Chemistry (1999, p. 160.

3. Analytical Methods Committee. Analyst, 119,2363-6 (1994)

4. R. Cassidy and M. Janoski. LC-GC, 10,692-4 (1992)

5. P. Vankeerberghen, J. Smeyers-Verbeke, A. Thielemans and D.L. Massart, Analusis, 20,

103-9 (1992)

6. A.H. Kalantar. Analusis, 21, 119-22 (1993)

Downloaded by [Moskow State Univ Bibliote] at 04:25 17 October 2013

7. D.G. Mitchell and J.S. Garden. Talanta, 29, 921-9 (1982)

8. L. Cuadros Rodrigueq AM. Garcia Campaiia, C. Jimenez Linares and M. Roman Ceba.
A d . Lett., 26, 1243-58 (1993)

9. J.N. Miller. Spectrosc. Znt.,3, 42-3 (1991)

10 M.C. Ortiz, J. Arcos, J.V. Juarros, J. Lopez-Palacios and L.A. Sarabia. Anal. Chem., 65,
678-82 (1993)

11 D.G. Mitchell and J.S. Garden. Talantu, 29, 921-9 (1982)

12 D.L. Massart, B.G.M. Vandeginste, S.N. Deming, Y. Michottem and L. Kaufinan,

"Chemometrics:a textbook". Elsevier (1988), p. 80-4

13 STATGRAPHICS 7.0, Statistical Graphics Corporation, 1994

14 M.A. Sharaf, D.L. Illman and B.R. Kowalski, "Chemometrzcs".Wiley (1986), p. 127

Received: Decanber 14, 1995

Accepted: January 31, 1996