NIR - Multivariate Calibration - 3rd Edition 2014 PDF

Jörg-Peter Conzen
Multivariate Calibration
A practical guide
for developing methods in the
quantitave analytical chemistry
Order No.: I 26009

ISBN 3-929431-13-3
Multivariate
Calibration
A Practical Guide for the

Method Development in the
Analytical Chemistry
3rd English edition.
© 2014 Bruker Optik GmbH

7KHWH[W¿JXUHVDQGSURJUDPVKDYHEHHQZRUNHGRXWZLWKWKHXWPRVW
FDUH+RZHYHUZHFDQQRWDFFHSWQHLWKHUOHJDOUHVSRQVLELOLW\QRUDQ\OLD-
ELOLW\ IRU DQ\ LQFRUUHFW VWDWHPHQWV ZKLFK PD\ UHPDLQ QRU WKHLU FRQ
sequences.
7KHIROORZLQJSXEOLFDWLRQLVSURWHFWHGE\FRS\ULJKW$OOULJKWVUHVHUYHG
1RSDUWRIWKLVSXEOLFDWLRQPD\EHUHSURGXFHGLQDQ\IRUPE\SKRWRFR-
S\PLFUR¿OPRURWKHUSURFHGXUHVRUWUDQVPLWWHGLQDXVDEOHODQJXDJHIRU
PDFKLQHVLQSDUWLFXODUGDWDSURFHVVLQJV\VWHPVZLWKRXWRXUZULWWHQDX-
WKRUL]DWLRQ7KHULJKWVWRGLVWULEXWLRQE\PHDQVRIOHFWXUHVUDGLRDQG
WHOHYLVLRQDUHDOVRUHVHUYHG
7KHVRIWZDUHDQGKDUGZDUHGHVFULSWLRQVUHIHUUHGWRLQWKLVPDQXDODUHLQ
PDQ\FDVHVUHJLVWHUHGWUDGHPDUNVDQGDVVXFKDUHVXEMHFWWROHJDOUHTXL-
rements.
TABLE OF CONTENTS
CHAPTER1
Table of Symbols ..........................................................................................................v

Introduction ..................................................................................................................1
Multivariate Calibration Methods in Analytical Chemistry..........................................3
Theoretical Considerations .....................................................................................3
PLS Regression .......................................................................................................7
Validation of Chemometric Models and Analysis of Unknown Samples ...................13
Practical Procedure for Setting Up a Model ...............................................................19
Selection Criteria for a PLS Calibration; Method Optimization ................................23
Selecting the Concentration Range and Component Values..................................23
Creating representative Calibration Samples ........................................................25
$PELHQW,QÀXHQFHVRQWKH,QVWUXPHQW ..................................................................28
The Problem of “Collinearity” ..............................................................................29
Performing a Background Measurement ..............................................................32
The Importance of Measurement Parameters and Reference Values .....................35
Selecting the Spectral Data Block .........................................................................36
Selecting Internal or External Validation ..............................................................36
Selecting Spectral Ranges .....................................................................................39
Selecting the Data Preprocessing Method ............................................................42
Selecting the Appropriate Number of Factors .......................................................44
Selecting Suitable Calibration Samples; Recognizing and Eliminating Outliers ..46
Evaluating the Validation Results .........................................................................47
Implementation and Maintenance of Methods ......................................................49
A Practical Example ...................................................................................................53
Method Development and Validation ....................................................................53
Analysis and Determination of Outliers ................................................................62
3/65HJUHVVLRQ$0HWKRG3URYLGLQJ,Q¿QLWH$FFXUDF\" ....................................64
'H¿QLWLRQRI,PSRUWDQW$EEUHYDWLRQV ........................................................................71
Summary and Outlook ...............................................................................................83
Frequently Asked Questions.......................................................................................87
NIR Tables of Functional Groups .............................................................................109
References ...............................................................................................................113
Multivariate_Calibration iii
iv Multivariate_Calibration
TABLE OF SYMBOLS
CHAPTER1
The following symbols are used in this tutorial:
Ai absorbance value of sample i

Am mean absorbance value
b FDOLEUDWLRQIXQFWLRQUHJUHVVLRQFRHI¿FLHQWRUEFRHI¿FLHQW
M
b UHJUHVVLRQFRHI¿FLHQWUHVXOWLQJIURPWKHFRPELQDWLRQRIVSHFWUDOGDWDDQG
concentration data of the master measurement
Ci concentration of sample i
Differi difference between reference and analysis value of sample i

F residual matrix of spectral data
G residual matrix of concentration data
hi Mahalanobis distance of sample i
i sample index
Leveri leverage of sample i
M number of samples
pR loadings vector for spectral data at rank R
qR loadings vector for concentration data at rank R
R rank
R2 FRHI¿FLHQWRIGHWHUPLQDWLRQH[SODLQHGYDULDQFHRIVSHFWUDOGDWD
Resi residual value of sample i (residual of concentration data)
RMSECV root mean square error of cross validation
RMSEP root mean square error of prediction (test set validation)
SpecResi spectral residuum of sample i
tR scores vector of the spectral or concentration data for the rank R
T transponent
W wavelength
X spectral data
XAnalysis spectral data of the analysed sample
Xi spectral data of sample i
M
X matrix of spectral data resulting from sample measurements with the master
S
X matrix of spectral data resulting from sample measurements with the slave
Y concentration value
Multivariate Calibration v
YAnalysis concentration values of analysed samples
Yi concentration value of sample i
M
Y Concentration data matrix resulting from the regression of the extinction and
concentration data measured with the master instrument
S
Y Concentration data matrix resulting from the nexus of the spectral data
PHDVXUHGZLWKWKHVODYHDQGWKHUHJUHVVLRQFRHI¿FLHQWMb
Ym
mean concentration value
meas
Yi measured concentration value of sample i
Yipred predicted concentration value of sample i
vi Multivariate Calibration
1
C H A P T E R
CHAPTER1
INTRODUCTION
CHAPTER2
The appearance of modern analytical chemistry has changed

fundamentally over the last few years, due to the introduction
of so-called chemometric evaluation techniques. The term
“Chemometrics” nowadays represents all multivariate
calibration methods in analytical chemistry. Compared to the
classical univariate calibration, this technique uses not only
one spectral data point for the calibration, but the whole
spectral structure. The advantage of this type of calibration is
the amount of spectral information used, so that even minor
GLIIHUHQFHVLQWKHVDPSOHVSHFWUDFDQEHLGHQWL¿HG
Although comprehensive literature on this subject already
exists, the largely mathematical language used in these pub-
lications is often hard to understand for the analytical chem-
ist. Nevertheless, these new statistical evaluation techniques
are an essential tool in modern instrumental analysis. They
permit an investigation of complex systems which would
have been unthinkable a few years ago.
One important intention of this book is to describe the func-
tionality of multivariate calibration in easy terms. This was
done intentionally without a strongly mathematical descrip-
tion.
First, the theoretical basics of chemometric evaluation tech-
niques are described. Chapter 2 explains their functionality
using the PLS algorithm as an example, and emphasizes the
advantages of multivariate calibration compared to the clas-
Multivariate Calibration 1
sical univariate calibration. Chapter 3 shows how unknown
samples can be analyzed using a chemometric model, and
how the quality of the analysis can be evaluated using simple
URXWLQHV7KHVH¿UVWFKDSWHUVRQO\GHDOZLWKWKHWKHRUHWLFDO
aspects of multivariate calibration. They do not give practical
advice on how to optimize a PLS model yet.
The practical procedure of setting up chemometric methods
is described in the chapters 4, 5 and 6. They are very impor-
tant for the analyst as they give hints on how to optimize all
relevant parameters. The study of these chapters should
enable even the amateur to set up a chemometric model
quickly that provides optimum results at a given measure-
ment situation. Here, mainly the practical analyst is
addressed.
These three chapters build the core of this tutorial. The most
important task is to enable the user to achieve an easy and
successful application of multivariate calibration techniques.
With the help of these chapters, an effective and systematic
method is demonstrated. A detailed knowledge of the
theoretical background of the statistical methods (i.e. an in-
depth knowledge of chapters 2 and 3) is not neccessary.
Chapter 7 is a glossary of all relevant statistical terms that are
used in the specialized literature. It only serves as a quick
reference to help you to understand new terms. Once again,
learning the mathematical terms is not necessary to develop
chemometric methods successfully. The last chapter is a
summary and an outlook, where importance of the
multivariate calibration is discussed compared to the classi-
cal univariate calibration.
This manual is by no means a comprehensive summary on
chemometric methods in analytical chemistry. It is designed
as a tutorial which aims at giving practical advice for the
daily laboratory routine. It should enable the user to set up
optimum chemometric models systematically, without
deeper knowledge of the theory of multivariate calibrations.
Nevertheless, the impression should not be given that a quick
and carefree usage of chemometric software is advisable. It
LVDFDUHIXODQGDWWHQWLYHDSSURDFKWKDWLQÀXHQFHVWKHTXDOLW\
of an analysis considerably.
2 Multivariate Calibration
2
C H A P T E R
CHAPTER1
MULTIVARIATECHAPTER2
CALIBRATION METHODS IN
ANALYTICAL CHEMISTRY
This and the following chapter will explain the theoretical

background of multivariate calibration techniques. It was
tried to avoid a strictly mathematical description of the
algorithms. Still, the procedure and the representation will be
unusual for some analyst and requires a certain change in the
way of thinking. The main task of this tutorial is to give
practical hints in method development, but a coarse under-
standing of the theoretical background does help to get
acquainted with the multivariate calibration techniques.
A. Theoretical Considerations
To describe multivariate calibration techniques, the practical

procedure of developing a method will be explained for-
PDOO\¿UVW*HQHUDOO\VSHDNLQJHYHU\TXDQWLWDWLYHDQDO\WLFDO
method aims to determine a system property Y quantitatively
from a measured system parameter X. This determination
normally requires two steps: the calibration and the analysis
(prediction). During calibration, a correlation of the
measured quantity X and the system property Y is searched
for.
This correlation is described in the calibration model:1
< ;ÂE
with the calibration function b1, which is often called
³UHJUHVVLRQFRHI¿FLHQW´RU³EFRHI¿FLHQW´
b = (XTÂ;Â;TÂ<
In this representation, the individual parameters X and Y are
written in matrix form. If they were to represent a spectro-
scopic measurement, for example, the spectral intensities
would be written into the X-matrix in rows, point by point.
Each additional sample would thus correspond to an addi-
tional row in the matrix. The corresponding component val-
ues would then be written into the rows of the Y-matrix. T
denotes the transpose of the associated matrices.
After the calibration, the analysis is performed. By connect-
ing the calibration model to the measured parameter X, the
system property Y of an unknown sample is determined. This
LVGHSLFWHGVFKHPDWLFDOO\LQ)LJXUH
Step 1: Calibration
X-Data + Y-Data Calibration Function b
Step 2: Analysis
X-Data + Calibration Function b Y-Data
Figure 2.1 Schematic procedure of the quantitative determination of

substances
In case of a quantitative evaluation of (near-)infrared or

Raman spectra, the measured value is typically an absorption
or emission spectrum, and the system value to determine is
the concentration of the analyte. There are two methods of
VHWWLQJ XS D FDOLEUDWLRQ PRGHO IRU VXFK D V\VWHP ¿UVWO\
univariate (single variable) calibration, and secondly, the
increasingly popular method of multivariate calibration:
Figure 2.2 Calibration of absorbance spectra
)LJXUH DQG )LJXUH VKRZ WKH SURFHGXUH GHVFULEHG

above with a univariate evaluation of an absorption band.
Five samples with the concentrations C to C5 were mea-
VXUHG7KHVHYDOXHVFRUUHVSRQGWRWKH³<YDOXHV´7KHPHD
VXUHPHQWV UHVXOW LQ ¿YH DEVRUSWLRQ YDOXHV $ to A5; the
³;YDOXHV´VHHOHIWSLFWXUHLQ)LJ
In univariate calibra- In case of a univariate calibration of the system, the absor-
tion, only one spectral bance values of the peak maximum are plotted versus the
information, e.g. the
peak height or peak concentration of the analyte (see picture on the right in Fig-
area, is correlated to XUH 7KH ¿W IXQFWLRQ FDOFXODWHG IURP WKH DEVRUEDQFH
the reference value. concentration data then allows to calculate the concentration
from the measured absorbance values and vice versa.
The analysis of a new, unknown sample is carried out by
measuring it spectroscopically and determining the absor-
bance value Ap at the peak maximum. This value is then
correlated with the calibration function b, which was calcu-
ODWHGHDUOLHUDQGUHVXOWVLQWKHDQDO\WHYDOXH)LJXUH,Q
FDVHRIDVLQJOHFRPSRQHQWV\VWHPLWLVVXI¿FLHQWWRHYDOXDWH
the spectra at only one wavelength. The determination of a
two-component system is accomplished by measuring a
second wavelength, which is characteristic for the substance
to be analyzed2. In this way, multi-component systems can be
analyzed such that for each additional component the
absorbance value at one additional, suitable wavelength is
added to the calibration set.
Figure 2.3 Analysis of absorbance spectra
,QPDQ\FDVHVDXQLYDULDWHFDOLEUDWLRQOHDGVWRDQLQVXI¿
cient prediction capability, as this method has several short-
comings:
The concentration of the analyte is only correlated to a single
point in the spectrum, and therefore, when evaluating new,
uncalibrated samples, neither outliers nor the presence of
unknown interfering components can be recognized. In
other words: from the peak height or peak area, one cannot
DVVXPHWKDWWKHVWUXFWXUHRIWKHPHDVXUHGVSHFWUD¿WVWKH
calibration data.
Statistical variation of the signal, such as detector noise, is
directly incorporated into the concentration data. The
resulting uncertainty usually has to be minimized by
multiple sample measurements and subsequent averaging of
the results.
A satisfactory calibration of multi-component systems requires
VXI¿FLHQWVHSDUDWLRQRIWKHSHDNPD[LPD+RZHYHULQPDQ\
cases this is simply not possible, especially in near infrared
spectroscopy.
In the analysis of multi-component systems, a linear additivity
of the absorbance values of all analytes at the measured
wavelength is assumed (Beer-Lambert Law). Plotting the
absorbance values against the concentration leads to a linear
FDOLEUDWLRQIXQFWLRQEVHH)LJXUH,QPDQ\FDVHVWKLVLV
not valid for real systems. Intermolecular forces or
temperature effects can lead to distortions of the respective
analyte bands. Furthermore, there are several techniques for
which the Beer-Lambert law is not valid, such as diffuse
UHÀHFWDQFHPHDVXUHPHQWVDWHFKQLTXHRIWHQXVHGLQ,5
spectroscopy.
Thus an analysis using classical univariate calibration meth-
ods may often lead to useless results for multi-component
systems. One method for solving such problems is the use of
multivariate calibration methods, such as Multiple Linear
Regression (MLR), Principal Component Regression
(PCR), or Partial Least Squares (PLS)-Regression.
B. PLS-Regression
A comparative review of these chemometric methods used in

analytical chemistry can be found in [1, 3, 4, 5, 6, 7]. As the PLS
algorithm is established as the one most commonly used,
only this method will be described here. Due to its extent and
mathematical complexity, a thorough explanation will not be
presented. For further reading refer to [1, 5, 6, 11].
In order to carry out a PLS regression for a given system, the
information of the substance spectra must be compared to the
corresponding concentration values. Changes that occur in
both data point structures must be recognized and correlated
with each other.
Multivariate For this purpose, a large number of samples must be mea-
calibration generally sured. For a mathematical display of the changes in both data
combines a large
amount of spectral sets, they are written down in a data point matrix (see Figure
information with the DQGWKHLUHLJHQYHFWRUVDUHJHQHUDWHG7KHVHHLJHQYHFWRUV
corresponding are called factors or primary components. They can be used
reference values of the
sample. This leads to for the prediction of concentrations instead of the original
higher precision and spectra, as they contain all relevant information of the
error stability. investigated system. The advantage of this decomposition is
obvious: the analytically relevant information from large
data sets is compressed into factors and can then be used for
the calibration.
In case of a PLS calibration, the eigenvectors are sorted in
GHVFHQGLQJ RUGHU 7KH ¿UVW IDFWRU FKDUDFWHUL]HV WKH PDLQ
changes of the observed spectrum. It has the largest impor-
tance for the calibration model. With an increasing number
of factors, ever smaller changes in the data structure are
characterized. This has an important consequence for the
evaluation of the substance spectra: The lower factors mostly
characterize the important changes in spectral structures,
whereas the higher factors mainly represent the disturbing
part of the spectral noise.
Figure 2.4 Encoding the spectral and concentration data in matrix form. In this example M
calibration samples were measured and - in a second step - all N wavelengths of
the resulting spectra are written in rows into a (M, N)-matrix. This matrix is
equivalent to the spectral data matrix X. In the same way, all L component values
are written into a (M, L)-concentration data matrix.
The selection of the optimum number of factors is of central

importance for the quality of the PLS model. With too few
IDFWRUVWKHVSHFWUDOVWUXFWXUHVDUHQRWUHFRJQL]HGVXI¿FLHQWO\
The corresponding regression can therefore never lead to
satisfactory analysis results. This is described as
³XQGHU¿WWLQJ´,IWKHQXPEHURIIDFWRUVLVWRRODUJHWKLVOHDGV
to a deterioration of the analysis, as too many parts of the
GLVWXUELQJVSHFWUDOQRLVHDUHLQFRUSRUDWHG³RYHU¿WWLQJ´
In a PLS regression, the spectral data matrix X and the con-
centration data matrix Y are reduced to only a few factors.
The original matrices are then represented as the sum of A
products of a so-called scores vector ti with the loadings
vector pi, or qi respectively3:
spectral data:
X = tpT + tpT + t3p3T + ...tRpRT)
concentration data:
Y = tqT + tqT + t3q3T + ...tRqRT*
In all cases, the scores and loading values are displayed as
vectors. This becomes more apparent in the schematic rep-
UHVHQWDWLRQRIHTXDWLRQ3:
Figure 2.5 Schematic diagram for the factorization of the spectral data matrix X
The rank R represents the number of factors and T denotes

the transpose of the respective loading vectors. F and G are
the so-called residual matrices of the spectral data and con-
centration data respectively. These correspond to changes in
the data structure which were not accounted for in the previ-
ous factorization.
*HQHUDOO\ WKH QXPEHU RI PHDVXUHG DEVRUEDQFH YDOXHV IDU
exceeds the number of components present. Thus the system
LV ³RYHUGHWHUPLQHG´ DQG FDQ WKHUHIRUH EH FRUUHODWHG QRW
only with a single spectral data point (like a peak maximum
in univariate calibration), but with the whole spectral data
structure. The information content of a data set calibrated in
this way is much larger than one obtained by univariate
calibration. Furthermore, it is possible to determine outliers
during the analysis, and one can decide whether unknown
disturbance components which do not correlate with the data
set have lead to spectral changes. As it is possible - in contrast
to univariate calibration - to take the spectroscopic
LQIRUPDWLRQRIWKHSHDNÀDQNVLQWRDFFRXQWVSHFWUDFDQDOVR
be evaluated by means of their structure. For this reason,
strongly overlapping bands can be distinguished in the
spectra, as long as there is a small variance in their shape8. In
a similar way, spectral structures can be recognized in very
noisy regions, which leads to a corresponding improvement
in predictive accuracy of the concentration data.
In a PLS regression, The special importance of PLS regression in analytical
WKHGDWDVHWVDUH¿UVW chemistry arises from its simultaneous and mutually depen-
decomposed into their
principal components. dent factorization of the X- and Y-data. When evaluating
7KHQD¿WWLQJRIWKH absorption spectra, one can assume that changes of the spec-
scores vectors from tral data have their origin in variations of the corresponding
spectral data and
concentration data is analyte concentration. This means that a variation in the
carried out. spectral data should lead to a corresponding change in the
Consequently, the spectrum. Therefore, the scores vectors of concentration data
method is more robust
against inaccuracies in
DQGVSHFWUDOGDWDPDWUL[VKRXOGEHLGHQWLFDO+RZHYHULQWKH
the reference and case of real samples, errors in sample preparation and in the
sample measurements. reference methods used to determine the concentration
values as well as instrument drifts and spectral noise will
lead to different scores vectors, if the matrices were reduced
purely by mathematical methods (i.e. independently).
Therefore, in the PLS method, identical scores vectors for
both data sets at the given factor numbers are assumed. They
are chosen so that they have the smallest possible deviation
from the original values. This is a compromise between the
factors’ suitability for describing the samples and the
increase in correlation between the data sets.
7KHVRFDOOHG3/6DOJRULWKPRQO\WDNHVWKHFRQFHQWUDWLRQ
values of one analyte into account. All other data are inter-
preted as disturbances, i.e. the Y-matrix of the concentration
GDWDLVDYHFWRU,QWKH3/6DOJRULWKPWKHFRQFHQWUDWLRQVRI
all components in the system are taken into account for the
calibration. For the prediction of new samples, the model
delivers a simultaneous analysis of all calibrated substances.
$V LQ FRQWUDVW WR WKH 3/6 FDOLEUDWLRQ DOO GDWD RI WKH
concentration matrix must be correlated with the data of the
VSHFWUDOPDWUL[WKH3/6SUHGLFWLRQUHWXUQVW\SLFDOO\SRRUHU
UHVXOWV WKDQ WKH 3/6 SUHGLFWLRQ6. For this reason, it is
JHQHUDOO\DGYLVDEOHWRSHUIRUPDFDOLEUDWLRQZLWKWKH3/6
algorithm. To carry out an analysis of a multi-component
system, this algorithm is applied successively to all cal-
ibrated components, so that a model for all desired
FRPSRQHQWVLVHVWDEOLVKHGDVZLWKWKH3/6DOJRULWKP
3
C H A P T E R
CHAPTER1
VALIDATION OF
CHAPTER2
CHEMOMETRIC MODELS
AND ANALYSIS OF
UNKNOWN SAMPLES
The previous chapter described how to set up a calibration

model using a PLS regression of spectroscopic data and the
corresponding concentration values. This model supplies a
limited number of scores and loadings vectors, by means of
which the spectral and concentration data respectively are
encoded. These are used to calculate the calibration function
b which allows the analysis of unknown samples.
In this context, the spectra of the samples to be evaluated are
measured. By combination of spectral data with the cali-
bration function b, the concentration values of the analytes
can be derived from equation (2-1):
YAnalysis = XAnalysisÂE
with XAnalysis: spectral data of analyzed sample.

YAnalysis: concentration values of analyzed sample.
Calculation of the calibration function b directly yields the

analyte concentrations from the respective spectra. The
degree of correlation gained by the combination of spectral
and concentration data is of major importance for the quality
of an analysis. With a good correlation, the analysis is
relatively exact. In contrast, a bad correlation can never lead
WRJRRGDQDO\VLVUHVXOWV7KHUHIRUHLWLVQHFHVVDU\WR¿QGWKH
function b which yields the best correlation (and hence the
best possible analysis results).
Generally, a validation For this purpose, the acquired model must be validated, i.e.
is possible only with evaluated. Such an evaluation is performed by predicting a
“independent”
samples, i.e. the certain number of samples with known analyte concentration
spectra of these with the chemometric model. A comparison of the predicted
samples must not be values with the actual values shows the precision of the
included in the
calibration data set. model. This is carried out with a large number of different
model parameters. That parameter set which leads to the
smallest error in prediction characterizes the best method.
Validation of the different chemometric methods permits the
recognition of outliers and the most suitable frequency
ranges, and particularly allows to determine the optimum
number of factors. Two types of validation are possible:
internal validation (cross validation) and external validation
(test set validation).
In the case of an internal validation, individual samples
GH¿QHGE\WKHXVHUDUHWDNHQIURPWKHFDOLEUDWLRQVHW8VLQJ
the remaining samples, a chemometric model is established
and used to analyze the previously extracted samples. A
comparison of the results with the actual concentration
values shows how precisely the model predicts the samples.
By extracting the samples beforehand, it is guaranteed that
they are not known to the calibration model and are thus
independent. An independent data set is very important.
Only in this way, the actual preciseness in prediction can be
assessed realistically.
To assess the complete data set, the samples analyzed previ-
ously are returned to the data set, and a second set of test
spectra is removed for analysis. This procedure of removing
samples, analysing them, and returning them to the calibra-
tion data set is continued successively until all samples have
been analyzed once. A comparison of the resulting analysis
values with the original raw data allows the calculation of the
predictive error of the complete data system, the RMSECV
(„Root Mean Square Error of Cross Validation“). This is a
quantitative measure for the mean accuracy of the predictive
capability of the chemometric model. The smaller this error,
the better the quality of the model.
For a cross validation it is important to remove only few
samples from tha data set, as the model built from the
remaining data set must be very similar to the model created
from the original data. For data sets with less than 50 samples
it is strongly recommended to remove not more than one
sample for the cross validation.
The whole process is schematically illustrated in the follow-
LQJ¿JXUH
Figure 3.1 Schematic of a cross validation
The steps of a cross validation (internal validation):

1 Remove a sample from the calibration data set.
2 Set up a model with the remaining samples.
3 Analyze the removed test sample; calculate the error of

analysis for this sample: Y1meas – Y1pred.
4 Return the removed sample to the data set and remove a new
sample. Calculate new model and predict new sample: Y2meas
– Y2pred.
5 Repeat step 4 until all M samples of the calibration data set

have been analyzed once; calculate the mean error of predic-
tion RMSECV.
(3-2)
A second method for assessing the quality of a method is the

external validation. In contrast to internal validation, all
samples of the calibration data set are used to generate the
model. This model remains constant for further validation,
i.e. the analysis spectra are no longer removed from the cal-
ibration data set. In order to estimate the error of prediction
for the model, further samples are measured and added to a
so-called “test set”. For validation, only the samples of the
test set are analyzed, i.e. the data set for external validation is
divided into calibration samples and analysis samples. In
contrast to cross validation, there is no exchange between the
two sample sets. A comparison of the analysis results with
the original concentration data of the test set is used to
calculate the RMSEP („Root Mean Square Error of Predic-
tion“). This value also represents a quantitative measure for
the predictive accuracy of the model. Good models are char-
acterized by low RMSEP values. Internal and external vali-
dation should lead to comparable results. If this is not the
case, too few samples were analyzed to establish a reliable
method. Figure 3.2 shows a schematic diagram of the test set
validation:
Figure 3.2 Schematic diagram of a test set validation
The steps of a test set validation (external validation):
1 Set up the model, using all calibration spectra.
2 Evaluate the model, using a separate data set of test samples

with known concentrations; calculate the mean error of
prediction RMSEP from the respective analysis errors:
(3-3)
To summarize: External validation can be distinguished from

internal validation by the fact that, in external validation, no
samples are excluded from the calibration data set. The
established model is constant for all analyses. In contrast,
cross validation allows the evaluation of the model from the
calibration data alone. Here, even with a limited number of
VDPSOHVLWLVSRVVLEOHWRJHQHUDWHVXI¿FLHQWO\ODUJHGDWDVHWV
for setting up and validating the method.
C H A P T E R
PRACTICAL PROCEDURE FOR

4
SETTING UP A MODEL
The theoretical procedure of setting up a PLS method was

discussed in the last chapters. This chapter describes the
practical considerations from collecting the spectra to ana-
lyzing the unknown samples. Usually, six steps are neces-
sary:
1. Step: Entering Spectral Data and

Concentration Data
Reference values are Before calculating the model, the spectra must be loaded and
generated by mixing the corresponding concentration values for the individual
the samples in the
laboratory, or by components must be entered. Furthermore, it is necessary to
TXDQWLWDWLYHGHWHU GH¿QH WKH FDOLEUDWLRQ DQG YDOLGDWLRQ VSHFWUD VHWV )RU
mination using a VXI¿FLHQWO\ODUJHGDWDVHWVLWLVDGYLVDEOHWRDVVLJQDQHTXDO
different analytical
WHFKQLTXH QXPEHURIVSHFWUDWRERWKVHWV,IRQO\DQLQVXI¿FLHQWDPRXQW
of data is available, it is important to create a comprehensive
calibration data set. In this case, no test spectra should be
GH¿QHG 7KH VXEVHTXHQW HYDOXDWLRQ RI WKH DQDO\VLV PRGHO
will then exclusively be carried out by cross validation.
2. Step: Data Preprocessing

In this step, a method for the preprocessing of the spectral
data is selected. This is often necessary to eliminate e.g. dis-
turbing baseline drifts. In practice, subtraction of a straight
OLQHYHFWRUQRUPDOL]DWLRQRUWDNLQJWKH¿UVWGHULYDWLYHRID
spectrum often leads to optimized PLS models.
6WHS'H¿QLWLRQRIDQ$SSURSULDWH)UHTXHQF\
Range
7KHVHOHFWLRQRIDQDSSURSULDWHIUHTXHQF\UDQJHLVRIFUXFLDO
LPSRUWDQFHIRUWKHTXDOLW\RID3/6PRGHO:KHQVHWWLQJXSD
PRGHORQHVKRXOGXVHWKHIUHTXHQF\UDQJHRIWKHVSHFWUXP
where a good correlation between the changes in the spectral
and the concentration data can be found. The extent of
FRUUHODWLRQ FDQ EH MXGJHG HDVLO\ E\ WKH FRHI¿FLHQW RI
determination R2 (see 4. Step).
6WHS9DOLGDWLRQDQG2SWLPL]DWLRQRIWKH0HWKRG
The suitability of the chosen data preprocessing methods and
RI WKH IUHTXHQF\ UDQJH IRU WKH JLYHQ PHDVXUHPHQW WDVN LV
evaluated during the validation. In this step, important
SDUDPHWHUVOLNHWKHFRHI¿FLHQWRIGHWHUPLQDWLRQR2 and the
mean errors of prediction RMSECV or RMSEP are calcu-
lated. In addition, an automatic outlier recognition is carried
out (see Chapter 5). The results are summarized in a report.
It is very important to 7RDFKLHYHDQRSWLPXPPHWKRGWKHFRHI¿FLHQWRIGHWHUPL
WDNHWKHFRHI¿FLHQWRI nation R2 and the respective mean error of prediction are
determination into
account, as a high summed up in a table for all “sensible” combinations of data
value for R2 indicates a SUHSURFHVVLQJDQGIUHTXHQF\ZLQGRZVLWLVWKHWDVNRIWKH
good correlation DSSOLFDWLRQ VSHFLDOLVW WR ¿QG PHDQLQJIXO FRPELQDWLRQV
between the spectral
data and the a priori no general recommendation can be given). Table 4.1
concentration data shows a way of summarizing the validation results.
7DEOH)RUPIRU&RPSDULQJWKH4XDOLW\RI9DOLGDWLRQ5HVXOWV
No. Data )UHTXHQF\ 2SWLPXPUDQN &RHIIRIGHWHU 0HDQHUURURI 5HPDUNV
preprocessing ranges [cm-1] mination R2 [%] prediction
1 1st Derivative 7,835-8,905 9 99,78 0.035 high rank
2 MSC 4,755-5,235 5 99,80 0.031 optimum
3 1st Der. + VN 4,755-5,825 7 99,23 0.041 outliers
4 Vector Norm. 5,745-6,105 6 99,05 0.052
The settings which lead to a high R2 value and a corre-
sponding low mean error of prediction should be used for the
calibration. Moreover, in many cases, it is sensible to choose
a setting which delivers a smaller number of factors with
HTXDOO\JRRGYDOLGDWLRQUHVXOWV
During the validation, potential outliers can be detected eas-
ily. They are distinguished, for example, by exceptionally
high F-values or FProb-values. If an independent check of
WKHVH VDPSOHV FRQ¿UPV WKDW WKH YDOXHV ZHUH FDXVHG E\ DQ
erroneous measurement, they should be removed from the
data set.
6WHS7KH&DOLEUDWLRQ
Only after all outliers have been removed from the calibra-
tion data set, and after the optimum system parameters have
EHHQ IRXQG WKH ¿QDO YHUVLRQ RI WKH PRGHO LV FRQVWUXFWHG
During the calibration, the scores and loading vectors are
calculated and thus the calibration function b is determined
(see Chapter 2). These values are stored internally, and they
are now available for the analysis of new samples.
6WHS7KH$QDO\VLV
For the analyte In this last step, the optimized chemometric model is used to
samples, the true analyze new samples. Simultaneously, the credibility of the
concentration values
are not known. analysis is checked by using characteristic parameters. One
Therefore, it is option is the calculation of the so-called “Mahalanobis” dis-
neces-sary to check by tance. Here the spectral structures of the complete calibra-
using statistical
parameters whether the tion data set are compared to the structure of the analyte
VDPSOHV³¿W´WKH spectrum. If the spectrum contains structures which do not
calibration spectra. ³¿W´RULIWKHFRPSRQHQWYDOXHVRIWKHDQDO\WHLVRXWVLGHWKH
calibration range, an increase of the Mahalanobis distance
can be observed (see Chapter 7).
$QDGGLWLRQDOPHWKRGZKLFKLVIUHTXHQWO\XVHGWRGHWHUPLQH
outliers, is the calculation of the spectral residuae. In this
method, the difference is calculated between the measured
spectrum and the spectrum which is theoretically expected
from the factor analysis of the calibration spectra. The
smaller this difference (i.e. the smaller the residuum), the
more credible the analysis result (see Chapter 7). The value
of the spectral residuum and the Mahalanobis distance are a
TXDQWLWDWLYHPHDVXUHIRUWKHTXDOLW\RIWKHDQDO\VLVUHVXOW,Q
addition, there is a number of further statistical parameters
for determining outliers, which shall not be discussed at this
point.
Hence, the analysis delivers two relevant pieces of informa-
tion: It indicates the analysis value of the sample, and it pro-
vides an outlier determination. This ensures that the user is
alerted if, by mistake, an erroneous measurement causes in-
correct analysis results.
5
C H A P T E R
CHAPTER1
SELECTION CRITERIA
CHAPTER2
FOR A PLS CALIBRATION;

METHOD OPTIMIZATION
In principle, there is an unlimited number of possibilities for

setting up a chemometric model. However, in general it is
possible to get a satisfactory analysis result by optimizing
only a few system parameters.
In this context, the selection of the calibrated spectral range,
the choice of data preprocessing methods, and the selection
and quality of the calibration standards are of central impor-
tance. This chapter explains the most important selection
criteria which allow the analyst to optimize the measurement
task.
A. Selecting the Concentration Range

and Component Values
The calibrated concentration range should be slightly larger

than the range of concentrations spanned by the samples. In
this way, the model will be more „robust“ with regard to
component values lying at the limits of the calibration range.
This is important, for example, if a reliable analysis of „bad
batches“ from a production process is required. These often
KDYHVLJQL¿FDQWGHYLDWLRQVLQWKHFRQFHQWUDWLRQYDOXHVDQG
WKHFRQFHQWUDWLRQUDQJHXVHGWRFDOLEUDWHPXVWEHVXI¿FLHQWO\
large in order to permit reliable analysis of these samples.
Figure 5.1 shows a comparison of reference values and
analysis values.
Figure 5.1 Selection of the concentration range: The samples of a

“typical” concentration range normally yield precise
results. This is not true for a sample with an analyte
concentration well outside the concentration range.
The individual component values should span the entire

concentration range evenly. It is not advisable to incorporate
individual samples into the calibration data set if the
concentration values of these samples lie well outside the
normal range (see Figure 5.1). If the concentration range of
an existing model needs to be expanded, it is necessary to
generate a more extensive data set.
B. Creating Representative
Calibration Samples
Another important point is the consideration of external dis-
turbances, which can occur during the sample measurement
and which cannot be avoided by a suitable sample prepara-
tion. An example is the spectroscopic in-line measurement of
chemical processes. Frequently, preparation or thermostatting
of the sample is not possible. In the same way, modern quality
control often does not allow extensive sample preparation,
which would be technically possible, but is too expensive.
Here, the methods of chemometrics are an important tool to
solve these tasks. An overlapping or deformation of the ana-
lyte signals - caused by contamination or by temperature
ÀXFWXDWLRQVRIWKHVDPSOHFDQEHDFFRXQWHGIRULQWKHIDF
torization of the spectra. Since all relevant system parameters
are stored as independent information blocks (i.e. as factors),
disturbances that overlap the spectrum can easily be elimi-
nated. For this purpose, all potential disturbances are simu-
lated during the calibration and stored as independent factors.
When comparing these values with the reference values of the
analyte, the algorithm will “detect” that this information is not
from the analyte itself. The corresponding structures will thus
not be used for the prediction of new, unknown samples. In
other words: The PLS algorithm can distinguish between
analytically relevant and analytically useless structures in the
spectrum. Disturbances are detected during calibration and
eliminated in further analysis.
The task of the method developer is to measure the samples in
realistic conditions. Consequently, all potential disturbances
that interfere with the system should be taken into account in
order to make the algorithm “learn” to recognize and eliminate
WKHP $OO ÀXFWXDWLRQV WKDW FDQ RFFXU LQ UHDOLW\ VKRXOG EH
accounted for in the calibration sample set. Only this will
ensure that the sample set is representative and suitable for
developing a method. Therefore, it is not advisable to carry
out a calibration under the most ideal conditions. Measuring
purest, well thermostatted samples leads to a very small error
in analysis, but the model is not robust enough to work reliably
in practice.
7KHLQÀXHQFHRIWKHVDPSOHWHPSHUDWXUHLVGHPRQVWUDWHGE\
the following example:
Due to the ability to build H-bridges, the spectrum of water is
very sensitive to temperature changes. With increasing
temperature, the H2O molecules move faster, which leads to
a cracking of the hydrogen-bridges. The existing electron
density moves back to the chemical OH-bonding, which
leads to an increase in the force constant. The molecule
oscillates faster, and the respective absorption band in the
NIR spectrum is moved to higher frequencies (wavenum-
bers). The more the water is heated, the further the band
moves to higher wavenumbers. Moreover, the water expands
with increasing temperature. The resulting reduction of
density leads to a decrease in signal height. Thus, with
increasing temperature, the water spectrum shows a decrease
in signal intensity and a simultaneous shift of the peak
maxima to higher wavenumbers (see Figure 5.2).
)LJXUH 7KHLQÀXHQFHRIWKHWHPSHUDWXUHRQWKHVWRYHUWRQHRIZDWHU
(optical pathlength 1 mm, reference: air)
For even larger temperature differences, a broadening and
distortion of the band can be observed, resulting from the
different population density of the single rotation-vibration
levels. For the sake of simplicity, a graphical description of
these deformations is left out here.
It is obvious that a calibration with thermostated aqueous
solutions can never supply reliable results if the actual anal-
ysis is done at different temperatures later.
,IWKHFDOLEUDWLRQLVFDUULHGRXWXQGHUVWULFWO\GH¿QHGFRQGL
tions, it is necessary to do the same during analysis. Addi-
tionally, it has to be taken into account that chemometric
methods must be adapted to changes in product composition
or product quality. This is also true if the measurement
parameters change. A method generally only works under
those conditions under which it was set up, and only those
disturbances are recognized which were accounted for in the
calibration. For this reason, a maintenance of the method is
often necessary at a later point.
Setting up and maintaining a method is quite easy, as it is not
necessary to correlate the reference values to the respective
disturbances or temperature values. This is also a result from
the factorization. To compensate for unwanted disturbances
in a multivariate calibration, the calculated analysis values
are not - as one could assume - corrected by an adjustment
function set up internally. Rather, the PLS1 algorithm tries to
¿QG D UHODWLRQ EHWZHHQ WKH FRPSRQHQW DQG WKH UHVSHFWLYH
spectral structure. Any other information is not accounted
for. There is no difference whether it is caused by an impurity,
a further analyte, or spectral noise (see Chapter 2).
7KHUHIRUHLWLVVXI¿FLHQWIRUDFDOLEUDWLRQWRKDYHDGDWDVHW
which was measured under authentic conditions. The refer-
ence values of the disturbing components are of no impor-
tance. Referring to the above example, for a calibration of an
DTXHRXV VROXWLRQ LW LV WRWDOO\ VXI¿FLHQW WR PHDVXUH WKH
samples at a representative range of temperatures. The
recording of the temperature itself is not necessary. Also, if a
change in product quality should ask for maintenance, a
further representative set of samples is added to the existing
model. Afterwards, the model should work reliably again.
C. $PELHQW,QÀXHQFHVRQWKH,QVWUXPHQW
Representative calibration samples are the perquisite of sta-

ble chemometric models. It is obvious that disturbing envi-
URQPHQWDOLQÀXHQFHVDIIHFWQRWRQO\WKHFKDUDFWHULVWLFVRIWKH
samples, but also the characteristics of the measuring
instrument. These are in case of the optical spectroscopy
LQÀXHQFHVRIWKHDPELHQWWHPSHUDWXUHDVZHOODVDWPRVSKHULF
LQÀXHQFHVOLNHWKHGLIIXVLRQRI&22 or gaseous water in and/
or from the spectrometer. This is described in the following
example.
While the diffusion of CO2 and H2O can lead to a substantial
disturbance of MIR-spectra, in NIR-spectroscopy, „only“
the water dissolved in the ambient air, which penetrates into
the equipment, affects the measurement noticeably.
)LJXUHVKRZVWKHLQÀXHQFHRIWKHPRLVWXUHGLIIXVLRQLQWR
a NIR spectrometer. The upper spectrum was measured
immediately after the reference measurement. No distur-
bances can be seen by additional water vapor absorptions.
After the measurement the equipment was placed into an
area with an unusually high humidity. The lower spectrum
was measured approximately one week later, without taking
a new reference measurement. One recognizes very sharp
structures in a spectral region of approximately 7,500 - 6,700
cm-1 and of approx. 5,800 - 5,000 cm-1 which overlay the
original spectrum. The respective disturbances are so strong
that the original structures can no longer be recognized.
Spectral regions which show such a disturbance must not be
included in the evaluation. Methods have to be set up which
work without these spectral ranges. If this is not possible, a
new reference spectrum must be measured in short periods,
or the instrument must be purged with dry air.
Figure 5.3 Absorbance spectra before (above) and after (below) the diffusion of humid
air into a NIR-spectrometer (MATRIX-F, Bruker Optik GmbH). The
spectra are shifted in y-direction for a better overview.
D. The Problem of “Collinearity”
In principle, there are two methods for setting up calibration

sample sets. On one hand, material samples can be measured,
and the reference values can be determined with an
independent analytical method. On the other hand, the sam-
ples can be manufactured synthetically in the laboratory.
Apart from an even distribution of the concentration values
over the entire concentration range (see above), in the case of
multi-component mixtures it is important that the individual
concentration values do not change collinearly. This means
that the concentrations of the respective components must
not increase or decrease equally in the different samples (see
Figure 5.4).
In the case of a collinear data set, the PLS algorithm cannot
assign the individual spectral bands clearly to the respective
component values. A method set up in this way is useless for
the analysis of non-collinear data sets.
Figure 5.4 Top row:&ROOLQHDUGDWDVHWVH[HPSOL¿HGZLWKWKUHH

samples containing components “A” and ”B”. The single
component values change for all three samples in the same
manner, i.e .the data set is collinear
Bottom Row: The data sets are not collinear, i.e .the
composition does not change systematically.
Collinear calibration data sets are only permitted if it is

ensured that the component values of the samples to be ana-
lyzed are always collinear. For example, this true for any
two-component system. Here, the increase of the concentra-
tion of one component leads to a corresponding (i.e. col-
linear) decrease of the other component, since both values
always add up to 100%. But a multiplicity of complex multi-
component systems also shows a natural collinearity. For
example, in the production of plastics, the polymerization
degree of the product often rises equally to the consumption
of the educts. In this way, a very accurate determination of
the polymerization degree succeeds with the help of a
spectroscopic analysis. The concentration of the basic
materials (not the polymerization degree itself) is determined,
but due to the direct (collinear) connection of both
parameters, the analysis succeeds easily. This applies
generally. Today, many physical parameters, IR-inactive
substances, or analyte concentrations below the detection
limit can be determined easily with the help of an (N)IR-
spectroscopic measurement, if these values are connected
collinearly with a further detectable component.
This is obviously not possible if the component values can
change independently of each other. In this case, it should be
observed that the values have a statistic distribution across
all samples. It is therefore not recommended to produce the
standards for the calibration from a dilution series, as not
only the analyte but all other components are diluted in the
same way. Again, the algorithm cannot differentiate between
the individual values, and a validation will lead to completely
useless results.
In this context, it is recommended not to measure a set of
calibration standards in ascending or descending order of the
concentration values. Systematic changes in the samples, for
example a rise in temperature during a series of
measurements, can simulate a correlation with a system
property. Measuring the samples again at a later stage and
comparing them with the spectra measured previously guar-
antees that no systematic changes are included in the cali-
bration.
It is useful to keep the problem of collinearity in mind right
from the beginning when setting up calibration data sets.
Afterwards, it may not be easy to recognize whether single
component values change in a collinear way. This is illus-
trated by the following example:
Table 5.1 Examples for Collinear Data Sets

No. X X . 3,9 X . 0,4 X / 120 (X . 0,3) + 15 100 – 2X 50 - (X . 1,7)
1 1 3,9 0,4 0,00833333 15,3 98 48,3
2 2 7,8 0,8 0,01666667 15,6 96 46,6
3 5 19,5 2 0,04166667 16,5 90 41,5
4 7 27,3 2,8 0,05833333 17,1 86 38,1
5 8 31,2 3,2 0,06666667 17,4 84 36,4
6 9 35,1 3,6 0,075 17,7 82 34,7
... ... ... ... ... ... ... ...
The numbers in each column can be generated by a simple
conversion from the numbers in the column „X“. They are
WKXVFROOLQHDU7KLVDSSOLHVQRWRQO\ZLWKUHJDUGVWRWKH¿UVW
column. The numbers of each column behave collinearly to
the numbers of every other column. The relative changes of
individual component values are the same for each sample.
Nevertheless, the collinearity may not be recognizable at
¿UVWVLJKW7KHUHIRUHLWLVXVHIXOWRSD\DWWHQWLRQWRDQLQGH
pendent distribution of the individual component values
right from the beginning.
E. Performing a Background Measurement

For most spectroscopic measurements, it is advisable to
carry out a new background measurement from time to time.
,Q WKLV ZD\ DPELHQW LQÀXHQFHV RQ WKH PHDVXUHPHQW VHWXS
HJWKHVSHFWURPHWHUWKHOLJKW¿EHUVWKHFXYHWWHVHWFDUH
compensated for. So, highly precise analyses are possible.
When selecting the reference substances, care should be

taken that these are as inert as possible and that they show no
absorption bands in the spectral range which is relevant for
the evaluation. The following background measurements
have proved useful in (N)IR-spectroscopy:
Gaseous measurements:
MIR: vacuum or dry nitrogen.
NIR: ambient air, rarely vacuum or dry nitrogen
Liquid measurements:
MIR: ambient air, in some cases pure solvent
(only at constant temperatures; see below).
NIR: ambient air.
Solid measurements:
MIR: dry potassium bromide or caesium jodide.
1,5 7HÀRQ RU URXJK PHWDO VXUIDFHV ZKLFK VFDWWHU
light (mainly gold).
In Raman spectroscopy, normally no background measure-
ment is carried out.
7KHLQÀXHQFHRIDWPRVSKHULFJDVHVHVS&22 and H2O) on

measurements in the mid-infrared region are usually so
strong that the spectrometer needs to be purged with nitro-
gen.
In contrast, measurements in NIR are less sensitive and are

affected noticeably only by atmospherically dissolved water.
Its disturbance is generally compensated by a regular
reference measurement (e.g. once every hour). Thus, typical
laboratory applications can be performed in the NIR without
further precautionary action. However, this does not apply
to most process applications. Here, the equipment usually
works for several months with the same background
measurement. Changes in humidity content of the ambient
air is known to affect the analysis in a spectral region of
approx. 7,500 - 6,700 cm-1 and of approx. 5,800 - 5,000 cm-1
(see above). If this range is important for the evaluation,
these devices are purged with nitrogen or oil-free air.
The use of inert, low-band or band-free reference materials is

an important requirement for the standardization of spectra
and methods. As already mentioned, in (N)IR-spectroscopy,
the height, width, shape, and position of the absorption bands
depend on the ambient temperature. If reference and sample
measurements are not recorded at exactly the same
temperature, artefacts of the reference material can be
observed, if this material shows absorbance bands in the
observed spectral region. They appear exactly at the same
frequencies where the absorbance bands of the material
occur. This is shown in Figure 5.5 by the example of the NIR-
spectroscopic measurement of water.
Figure 5.5 Spectra structure which emerges from the shift of the 1. overtone of water with
rising temperature. For the background and sample measurement pure water was
used. The background was taken at 30°C. The samples were measured with rising
temperature up to 60°C.
The same samples as in Figure 5.2 were measured, where the

reference was measured against air and no disturbing
artefacts could be observed. Now, with water as reference
material, structures can be observed which overlay the orig-
LQDOVSHFWUXP7KLVKDVDGLVWXUELQJLQÀXHQFHRQWKHFKHPR
metric evaluation. The measurements against air only show a
VOLJKWRIIVHWVHH)LJXUHZKLFKKDVDIDUVPDOOHULQÀXHQFH
on the analysis. It is therefore clear that, in vibrational
spectroscopy, it is generally not useful to use the sample
matrix for a background measurement. The equation mostly
valid in chemical analysis
[Substance Spectrum] - [Spectrum of Matrix] =

[Pure Analyte Spectrum]
or, in simple words:
[Matrix+Analyte] - [Matrix] = [Analyte]
does not apply in (N)IR and Raman-spectroscopy (or only

with exactly identical temperatures for background and
sample measurement). Preferably, the background measure-
ment should be carried out against air (or nitrogen), or against
a reference material which does not show absorption bands
in the relevant spectral region.
F. The Importance of Measurement

Parameters and Reference Values
In the analysis of FT-(N)IR and Raman spectra, the number

of spectral data points is usually substantially larger than the
number of individual components. Every component value is
thus associated not with an individual spectral point (as is the
case in a univariate calibration), but rather with an entire
spectral feature. The system retains a substantially higher
information content. Therefore, it is possible for the PLS
algorithm to compensate (to some extent) for statistical
variations in the calibration data. Nevertheless, it is also
obvious that an exact model can never be derived from an
LQVXI¿FLHQWGDWDVHW+HQFHWKHTXDOLW\RIWKHVSHFWURPHWHU
(such as its resolution, stability, signal-to-noise ratio, and
especially its precision of measurement) affects the quality
of the model constructed.
Of central importance in this context is the quality of the
reference data and the care with which the samples were
prepared. Experience shows that, in most cases, poor
FKHPRPHWULFPRGHOVUHVXOWHLWKHUIURPDQLQVXI¿FLHQWDFFX
racy of the reference technique or from bad sample prepara-
WLRQ ,W LV FOHDU WKDW IRU H[DPSOH ÀXFWXDWLRQV LQ VDPSOH
temperature during measurement or investigation of inho-
mogeneous or impure sample material can also lead to seri-
ous errors in the analysis, if they were not taken into account
in the method. The measurement error of the spectrometer or
the inaccuracy associated with the PLS model itself are
usually of negligible importance. Hence, it is primarily a
careful analytical technique as well as a reliable reference
method that determine the quality of the analysis.
,IWKHTXDOLW\RIWKHUHIHUHQFHGDWDLVGLI¿FXOWRULPSRVVLEOHWR
improve, it is advisable to make several repeat measurements
of the same sample and to construct an average. Statistical
variations are then averaged out, and possible outliers have a
much smaller effect on the quality of the analysis. This is also
true for spectrometric analysis, where, in critical cases, errors
can be minimized by repeat sample measurements and
averaging.
G. Selecting the Spectral Data Block
In principle, there is no restriction in selecting spectral data

blocks for a PLS calibration. The selection can be carried out
using interferograms, single channel spectra, transmission
spectra, absobance spectra, etc. In most cases, it is advisable
to use absobance spectra. As long as the Beer-Lambert law is
followed, absobance spectra have the advantage of presenting
a linear relationship between absobance values and
concentration data. Such a linear relationship is generally
advantageous in a PLS-algorithm factorization4.
H. Selecting Internal or External Validation
The chemometric model is set up from the calibration data

and is then tested for quality with the aid of test spectra. There
are two methods to perform such a validation: Internal
validation (Cross Validation) or external validation (Test Set
Validation):
Test Set Valiation: For a test set validation, two independent
sets of samples are used, one for system calibration, and the
other to validate the corresponding model. Both sets should
consist of about the same number of samples, and each set
should cover the whole concentration range of the system
evenly.
Cross Validation: If the number of available samples is lim-
ited (50 samples or less), one should do without a separate
test set. Here, only a cross validation can make reliable pre-
dictions about the quality of a model.
,QSUDFWLFHDFURVVYDOLGDWLRQLVSHUIRUPHG¿UVWWRDVVHVVWKH
method. Only if it is guaranteed that a representative selec-
WLRQRIVSHFWUDKDYHEHHQPHDVXUHGWKHGH¿QLWLRQRIDVHSD
rate test set is sensible. In contrast, the test set method has
advantages where large amounts of data are available, as the
calculation time is considerably shorter than with cross vali-
dation.
Most software packages offer both possibilities: internal and
external validation. This enables the analyst to estimate the
robustness of the method. This is carried out as follows:
,QWKH¿UVWVWHSDVGHVFULEHGDERYHWKHEHVWPRGHOSDUDP
eters are found by performing a cross validation. The result-
ing value of R2 and RMSECV are written down. In a second
VWHSWKHGDWDVHWLVVSOLWLQWRWZRHTXDOO\VL]HGSDUWV7KH¿UVW
³SDFNHW´LVGH¿QHGDVWKHFDOLEUDWLRQGDWDVHWDQGWKHVHFRQG
“packet” is the test set. Again, a method is set up with the
SDUDPHWHUV GH¿QHG SUHYLRXVO\ YDOLGDWDWHG E\ D WHVW VHW
validation and documented by writing down the R2 and
RMSEP value. In a third step, the test set and the calibration
data set are exchanged and validated with the same model
parameters (2nd test set validation). The resulting values of R2
and RMSEP are compared to the previously calculated values
of the test set validation as well as for the cross validation.
Table 5.2 shows this for the error of analysis.
Table 5.2 Checking the Stability of a Calibration
Analysis Error of Analysis Error of
Analysis Error of 1. Test Set 2. Test Set
Cross Validation Validation Validation
Stable 0,10 0,11 0,10
Calibration
Unstable 0,10 0,25 0,33
Calibration
In the case of a representative and suitably sized data set (this

is the condition for a stable method) all values should lie
within small error tolerances. If, however, the error of
prediction for the cross validation is far better than for the test
set validations, this may indicate that not enough samples
were measured. This results from the fact that for a cross
validation, compared with a test set validation, approximately
twice as many samples are included in the calibration data
VHW,IDQLQVXI¿FLHQWQXPEHURIVDPSOHVZHUHPHDVXUHGWKLV
has of course a stronger effect on a test set validation, due to
the reduced size of the calibration data set.
,Q FRQWUDVW WR WKLV LI WKHUH DUH VXI¿FLHQWO\ ODUJH GDWD VHWV
halving the number of calibration spectra should lead to
similar errors of analysis. If this is the case, one can assume
that the calibration will work robustly in the future. More-
over, it is generally recommended to carry out a method
optimization also with the 1st test set validation and to check
the robustness with the cross validation and the 2nd test set
validation. The more different combinations are tested, the
HDVLHULWLVWRVHHZKHWKHUKDOYLQJWKHGDWDVHWOHGWRVLJQL¿
cant loss in analysis precision, and whether more samples
need to be measured.
Irrespective of the type of validation, care needs to be taken
not to include identical samples in the calibration and in the
test data set. Similarly, if multiple measurements of the same
sample were carried out, it is not permissible to divide the
individual spectra between the two data sets (see Chapter
³3/6UHJUHVVLRQ D 0HWKRG 3URYLGLQJ ,Q¿QLWH $FFX
racy?”).
I. Selecting Spectral Ranges
PLS regression is a so-called „full spectrum method“, i.e. the

more spectral data points are present, the more spectral
information is available per component, and the better the
respective model will be. However, it has to be taken into
account that contributions from spectral noise or absorption
bands of interfering components can deteriorate the quality
of the model. In these cases, the PLS algorithm tries to
account for spectral structures which do not originate from
the component value, and the analysis results will deterio-
rate.
This is shown in Figure 5.6 in the example of an NIR-spec-
trum of water (transmission measurement, optical path
length: 2 mm). The spectrum shows a total absorption
between 5,200 cm-1 and 4,000 cm-1 and below 4,000 cm-1 a
strong contribution of spectral noise. Both regions must not
be used for setting up a calibration.
Figure 5.6 Cuvette spectrum of water (optical path length: 2 mm)
Therefore, it is often useful to measure all samples over the
whole spectral range and then look for spectral features that
enhance the model. In this context, absorbance bands with
absorbance values between 0.7 and 1.0 generally lead to the
best results. When using modern FT-spectrometers, typically
absorbance values of up to 2.5 can be used for the calibration.
However, it is necessary that an instrument is set up with
detectors which work linearly in the whole range (e.g. Peltier-
cooled InAs- or InGa-As detectors). Larger signals should
not be taken into account, as they will lead to a higher
uncertainty of the measurement.
Anyway, it is advisable to remove whole signal groups suc-
cessively. As, in IR-, NIR and Raman-spectroscopy, most
substances (with very few exceptions) have signals in a very
large spectral range, the search for a dedicated structure is
often not necessary. Table 5.3 und Figure 5.7 show a brief
overview of the frequency ranges which are normally
observed in near-infrared spectroscopy (more detailed infor-
mation are given in the appendix):
Table 5.3 Absorbance Signals in the Near Infrared
Group Frequency Range [cm-1] Name
Aliphatic 6.300 - 5.500 1. Overtone CH-stretching
hydrocarbons 9.100 - 7.800 2. Overtone CH-stretching
5.000 - 4.100 combination
Aromatic ca. 6.000 1. Overtone CH-stretching
hydrocarbons ca. 9.000 2. Overtone CH-stretching
Carboxylic acid ca. 6.900 1. Overtone CH-stretching
ca. 5.250 2. Overtone CO-stretching
Amines 7.000 - 6.500 1. Overtone NH-stretching
Water (very strong 7.500 - 6.400 1. Overtone OH-stretching
absorptions) 5.400 - 4.900 combination
Figure 5.7 Absorbance bands of various functional groups in the NIR region.
J. Selecting the Data Processing Method
Apart from choosing the right frequency window, the data

preprocessing method is the second important model
parameter. Here, the purpose is to model the spectra in such a
way that the PLS algorithm can establish a good correlation
between the spectral and the concentration data.
No Data Preprocessing: no data preprocessing.
Subtraction of a Constant Offset: The spectra are shifted
linearly so that the minimum y-value is equal to zero.
Application: Linear baseline shifts are eliminated. These
FDQUHVXOWHJIURPGLIIHUHQWYDOXHVRIWKHGHWHFWRUDPSOL¿
cation.
Subtraction of a Straight Line: In each selected frequency
UDQJHDVWUDLJKWOLQHLV¿WWHGWRWKHVSHFWUXPXVLQJWKH
partial least squares method. This straight line is then
subtracted from the respective spectrum.
Application: A linear tilt of the baseline shift is eliminated.
(see Figure 5.8).
Vector Normalization: First, the spectra are centered. Then,
the sum of all squares of all Y-values is calculated, and the
respective spectrum is divided by the square root of this
sum. The so-called vector norm of the resulting spectrum is
always 1.
Application: In principle, a spectrum contains two parts of
information: the height of the band as well as the structure.
After normalization, the height information is lost; only the
structural information remains. Normalization is used e.g. to
HOLPLQDWHWKHLQÀXHQFHRIGLIIHUHQWRSWLFDOSDWKOHQJWKVLQ
case of transmission measurements. The path length
changes the height of the signal, but not its structure. In a
VLPLODUZD\LQPHDVXUHPHQWVLQGLIIXVHUHÀHFWLRQWKH
LQWHUIHULQJLQÀXHQFHVRIGLIIHUHQWPDWHULDOGHQVLWLHVRU
particle sizes can often be minimized.
Min-Max-Normalization (for absorbance spectra): The
spectra are shifted linearly, so that the minimum Y-value
equals zero. Then the spectra are expanded, so that the
maximum Y-value equals two absorbance units (Figure 5.8).
Application: comparable to vector normalization.
Multiplicative Scatter Correction: First, a mean spectrum is
calculated from all spectra of the calibration data set. Then,
each spectrum X(i) is transformed according to
;L‫ މ‬XYÂ;L
7KHFRHI¿FLHQWVu and v are chosen such that the difference
between the transformed spectrum X(i)‘ and the mean
spectrum has a minimum.
Application: This method is often used for measurements in
GLIIXVHUHÀHFWLRQ
First Derivative:FDOFXODWHVWKH¿UVWGHULYDWLYHRIWKHVSHFWUXP
(Figure 5.8).
Application:%\FDOFXODWLQJWKH¿UVWGHULYDWLYHWKHVLJQDOV
ZLWKVWHHSHGJHVJHWPRUHHPSKDVLVWKDQUHODWLYHO\ÀDW
bands. This method is used to emphasize pronounced, but
small features compared to huge broad-banded structures.
Another important application is the evaluation of broad
bands. This is often done in NIR-technology. By calculating
the derivative, these structures get a steeper shape which can
be evaluated more easily.
When using the derivative as a data preprocessing method,
it has to be taken into account that the spectral noise is
enhanced as well. This superimposes the spectrum as an
additional disturbance and can deteriorate the analyte signal.
Second Derivative: calculates the second derivative of the
respective spectrum.
Application:&RPSDUHGWRWKH¿UVWGHULYDWLYHHYHQ
H[WUHPHO\ÀDWVWUXFWXUHVFDQEHHYDOXDWHG7KHGLVWXUELQJ
LQÀXHQFHRIWKHVSHFWUDOQRLVHLVJHQHUDOO\VRVWURQJWKDW
spectra can only be evaluated in a very restricted spectral
range.
)LJXUHVKRZVWKHLQÀXHQFHRIYDULRXVGDWDSUHSURFHVVLQJ
methods on the appearance of an NIR-spectrum (mea-
VXUHPHQW RI D KXPDQ KDQG ZLWK D ¿EHURSWLF SUREH 7KH
original spectrum shows a slight offset in the baseline as well
as a slight drift. This drift can be eliminated by the sub-
traction of a straight line (broken line) and the offset by a
0LQ0D[1RUPDOL]DWLRQGRWWHGOLQH7KH¿UVWGHULYDWLYHRI
the original spectrum (line-dotted line) has been expanded
for reasons of display and shifted to higher absorbance
values. A relative enhancement of the sharp structures
compared to the original spectrum can be observed.
)LJXUH 1,5VSHFWUXPRIDKXPDQKDQGPHDVXUHGLQGLIIXVHUHÀHFWLRQ
The optimum method depends strongly on the system to be

analyzed. Experience shows that in a great number of cases
the subtraction of a straight line, a normalization of the
VSHFWUD RU WKH ¿UVW GHULYDWLYH OHDGV WR WKH EHVW FDOLEUDWLRQ
results. In few cases, the combination of two data prepro-
cessing methods can lead to the best result. Often a number
of settings show equally good results (see Chapter 6), so that
normally all possible data preprocessing variations should be
tried.
K. Selecting the
Appropriate Number of Factors
In PLS regression, the spectral data and the concentration
GDWDDUH¿UVWHQFRGHGLQDPDWUL[IRUPDQGWKHQUHGXFHGWR
only a few factors. The number of factors of a chemometric
model is called “rank”. The section of this rank is of major
importance for the quality of the analysis. Choosing too few
IDFWRUVZLOOOHDGWRDQLQVXI¿FLHQWH[SODQDWLRQRIWKHFKDQJHV
LQWKHVSHFWUDODQGFRQFHQWUDWLRQGDWD³XQGHU¿WWLQJ´7KHUH
is only little correlation between the two data sets, and the
DQDO\VLVUHVXOWVIURPWKLVPRGHODUHLQVXI¿FLHQW,IWRRPDQ\
factors are chosen, the model tries to account for even the
smallest changes in the data set, such as spectral noise
³RYHU¿WWLQJ´7KLVZD\VSHFWUDOLQIRUPDWLRQunVSHFL¿FIRU
the sample is included in the model. A deterioration of the
analysis results is also to be expected from these models.
Thus, every PLS model has the optimum number of factors
which guarantees the smallest possible error of analysis.
There are numerous hints that lead to the optimum number of
factors for a certain model: The values of the mean error of
prediction (RMSECV for cross validation, and/or RMSEP
for test set validation) go through a minimum for the optimal
UDQN,QFRQWUDVWWKHYDOXHRIWKHFRHI¿FLHQWRIGHWHUPLQDWLRQ
R2 possesses a maximum. Thus, the optimum rank for a
certain calibration model can be found easily: First, the R2-
values and the mean error of prediction are computed. Then,
these values are plotted as function of the rank. The rank is to
be regarded as optimal, when the characteristics mentioned
go through an optimum value, and/or do not change
VLJQL¿FDQWO\IRUKLJKHUIDFWRUQXPEHUV,IVHYHUDOUDQNVOHDG
to comparably good results, it is recommended to select the
model with the smallest number of factors (see Chapter 6).
Caution: The validation of a method is only possible using
independent test spectra, i.e. the spectra must by no means
be part of the calibration data set. This applies if - in a cross
validation - all measurements of one sample are declared as
“leave-out spectra”. In case of a test set validation, new
samples should be measured for the test set.
L. Selecting Suitable Calibration Samples;
Recognizing and Eliminating Outliers
If the calibration data set contains outliers, they can be iden-

WL¿HGHDVLO\DIWHUWKHYDOLGDWLRQ2XWOLHUVDUHHJUHSUHVHQWHG
by
a large value for the spectral residuum;
a large “F-value”;
an “FProb” value of 0.99 or more;
a relatively large analysis error, which can be seen in high
“Differ” values (this value shows the difference between the
reference value and the analysis value).
In most cases, it is helpful to compare the reference and
analysis values graphically. The outliers show a large dis-
tance from the respective reference values. After removing
the outliers from the calibration data set, a new model can be
set up. Its validation should display no more outliers. With
FULWLFDOGDWDVHWVLWLVSRVVLEOHWKDWDIWHUD¿UVWUHPRYDORI
outliers the model is still not optimal. This is the case if the
number of outliers is so high that they affect the model
noticeably. Here, the model must be optimized step by step
by a successive removal of outliers.
Caution: At any rate it is necessary to examine critically
whether the outliers result from a faulty measurement. The
elimination of typical but “unpleasant” samples is not
SHUPLWWHG7KHFDOLEUDWLRQPRGHOZRXOGWKHQODFNVXI¿FLHQW
stability to handle the natural variety of different samples.
Frequently, samples are marked as outliers which lie far
outside the calibration range. In this case, however, the cause
IRUWKHRXWOLHULGHQWL¿FDWLRQLVQRIDXOW\PHDVXUHPHQWEXWD
lacking robustness of the method. Here it would be the task
of the analyst to generate a homogeneous distribution of the
component values over the entire range by further sample
measurements instead of removing the few existing values.
Also one should not trust the outlier marking in the valida-
tion report blindly and simply remove marked samples.
Generally, a chemometric software can measure the reliabil-
ity of a value only on the basis of statistic parameters. Obvi-
ously, only by considering these values without any
knowledge of the corresponding application it is not possible
to judge if a particular spectrum really is an outlier. Therefore,
only the analyst himself can decide, on the basis of an
independent examination of the spectrum, whether this value
must actually be rejected for the particular application.
M. Evaluating the Validation Results
One of the most important tasks of the method developer is

to examine the usefulness of a data set. The easiest way to
prove its reliability are multiple measurements of the sam-
ples and a subsequent plotting of the true values against the
predicted values. In the ideal case, reference and analysis
values for all measurements are identical within the stipu-
lated error tolerance.
However, this will not apply to every calibration model. In
some cases, the mean error of analysis (RMSECV or
RMSEP) will not have the desired quality. Now it is the task
RIWKHDQDO\VWWR¿QGDQH[SODQDWLRQIRUWKLVUHVXOWDQGWRWDNH
suitable measures, in order to improve the result. If this is not
possible, it is important to realize quickly that the selected
analytical method is generally not suitable for analyzing the
VDPSOHVZLWKVXI¿FLHQWSUHFLVLRQDQGWKDWWKHUHIRUHIXUWKHU
activities are not reasonable. This is described by the
following two examples.
)LJXUH 3UHVHQWDWLRQRIWZRFDOLEUDWLRQVZLWKLQVXI¿FLHQWSUHFLVLRQ
Figure 5.9 shows a comparison of the true and predicted

YDOXHVRIWZRFDOLEUDWLRQVZLWKRQO\LQVXI¿FLHQWSUHFLVLRQ
Each sample was measured three times. One generally rec-
ognizes a large distance of the individual points from the
bisector. However, the illustration on the left shows that the
individual measurements are plotted with high accuracy. The
individual points show approximately the same distance
from the straight line. In other words: The measurement is
wrong, but precise. This leads to the conclusion that the
analytical procedure should be suitable to solve the mea-
suring task. The cause for the lack of accuracy probably lies
in other factors.
For example, statistic dispersions of the analysis values
around the true value are caused by inaccurate reference
values or by an impure sample preparation. In contrast to
this, measurements at a different temperature or the use of a
different measurement setup cause a systematic error. Here,
the individual point collectives would all be above or below
WKHVWUDLJKWOLQHVLQ)LJXUH)XUWKHUPRUHLQVXI¿FLHQFLHV
of the chemometric method can lead to such a result. A
complex multi-component mixture, for example, requires a
relatively large number of representative calibration stan-
GDUGV ,I WKHVH DUH QRW PHDVXUHG LQ VXI¿FLHQW QXPEHU WKH
DOJRULWKPLVQRWDEOHWRDVVLJQWKHDQDO\WHVSHFL¿FVWUXFWXUHV
in the spectrum properly to the respective components. Here,
an analysis of new unknown samples would lead to a precise,
but wrong analysis. The model must therefore be extended
by further calibration spectra.
In many cases, the causes for a bad result of an analysis can
be found easily, if the individual measured values are pre-
cisely illustrated with the analysis. It is far more critical, if
the individual points show a large statistic dispersion (right
picture of Figure 5.9). Here, the calculated values are not
only wrong, but also not reproducible. Usually, an unsuitable
sample preparation or measurement setup is responsible. An
example is the investigation of heterogeneous samples. If the
material is not homogenized, or if the measurement spot used
is too small, no accurate values can be found despite a precise
local measurement. This must be considered during the
evaluation of the results. Even if the measurement values
cannot be improved in suitable measures, it is frequently
recommendable to add a further number of samples to the
calibration model. If even this does not lead to a considerable
improvement of the predictive accuracy, it is to be feared that
the selected analytical method is not suitable.
N. Implementation and
Maintenance of Methods
Certainly, the most important tasks of the method developer

are the compilation of representative calibration data sets and
the consideration of possible disturbances on the mea-
surement or the equipment. This usually requires analytical
expert knowledge, since not only the samples which can be
examined (and/or the process which can be examined) must
be understood, but also the measuring and evaluation tech-
nique.
Thus, every company usually has an expert who compiles
and maintains the calibrations. In many industries, this is
RXWVRXUFHG WR FRPSDQLHV RIIHULQJ D TXDOL¿HG FDOLEUDWLRQ
service. Apart from the actual calibration, they should deliver
an estimation of the expected accuracy of the method in the
context of a feasibility study. Also, consecutive maintenance
of the method should be offered, e.g. within a service contract.
It is generally recommended to examine the reliability of a
chemometric method from time to time. This can simply be
done by measuring the validity of the method with an inde-
pendent test set at regular intervals (e.g. every „n“ months,
every „n“ batches, after „n“ measurements, etc.). In some
industries, for example in the pharmaceutical industry,
appropriate procedure guidelines force the user to carry out
VXFKH[DPLQDWLRQVUHJXODUO\DPHGLFLQHUHFHLYHVWKHRI¿FLDO
release for sale only if the unrestricted usefulness of the
release analytics is documented in written form).
But even without appropriate regulations, “observation” and
maintenance of the method is advisable. It can often be
REVHUYHGWKDWDPHWKRGDOWKRXJKLWZDVXVHIXODW¿UVWJUDG
ually yields worse results. This can have a number of rea-
sons: First, a creeping change in the instrument (and changes
LQ FXYHWWHV SUREHV DQG ¿EHU RSWLFV FDQ EH WKH FDXVH
Furthermore, changing site conditions or even different raw
material qualities affect the quality of the analysis.
Particularly the calibration of natural materials or of petro-
chemical raw materials can make method maintenance nec-
essary over a long period.
For many applications, it is important that slow degradation
in the quality of analysis is recognized early. Modern prob-
lems of release analytics or process technology do not permit
improvement of the method, once it has lost its usefulness for
the given task. The maintenance of a method should be a
continuous process in the background of daily routine
analytics.
If the method developer realizes that a number of test sam-
SOHV ZHUH QRW DQDO\]HG FRUUHFWO\ DW ¿UVW LW LV LPSRUWDQW WR
review it and to compare the spectra with the earlier calibra-
tion spectra. In most cases, the reason for outliers can be
found relatively quickly. Once the reason is understood,
suitable strategies can be developed to extend the sample set
with suitable samples. It should be mentioned here that it
does not make sense to add new spectra to the data set con-
tinuously without prior examination. There is the danger that
the information necessary for a stable method is not included
in these samples. In this way, very large data sets are produced
which are by no means representative.
A further problem which we often encounter in process ana-
lytics is the impossibility to imitate process conditions in the
laboratory. Also, taking a sample and analyzing it with a
reference method is often not possible. In other words: it is
not possible to collect representative calibration samples.
+HUHLWLVRIWHQXVHIXOWRFROOHFWWKHVDPSOHV¿UVWDQGWKHQWR
draw conclusions for the respective values of the process
from the results of the established routine analytics. If this is
not possible, one has to be content with a strongly limited
method that can only work as a process monitor stating
“Process OK” or “Process not OK”. Sometimes a provision-
ary method set up in the laboratory must be used which
delivers imprecise results but mirrors the relative progres-
sion of the reaction correctly.
$JDLQWKHMXGJPHQWRIDTXDOL¿HGDQDO\VWLVQHFHVVDU\WRGH-
WHUPLQHLIWKHVSHFL¿HGPHWKRGVDWLV¿HVWKHUHTXLUHPHQWVRI
the process.
6
C H A P T E R
CHAPTER1
A PRACTICAL EXAMPLE
CHAPTER2
In the previous chapter, all relevant system parameters were

described and catalogued, describing how to carry out an
optimal PLS regression. These will now be demonstrated by
a practical example, whereby the most effective means of
developing a calibration method is shown. It is also illus-
trated to the analyst, how the use of an inappropriate valida-
tion can result in a completely meaningless calibration
model, and what criteria can be used to recognize such mod-
els.
A. Method Development and Validation
Every PLS calibration requires the choice of suitable fre-

quency ranges, optimum data preprocessing, and an opti-
mum number of factors. These factors depend on a multitude
of system parameters and it is not possible to calculate them
by theoretical considerations. Therefore they must be
evaluated empirically. This can be done as follows:
Even for a very simple A large number of test samples should be measured.
analysis task, one Depending on the task, a calibration data set of approxi-
should measure at least
20 samples to create mately 20 to 200 samples is necessary. The more complex
statistically relevant the composition of the sample, the more spectra are neces-
data sets. sary for the calibration. For example, two-component mix-
tures can be calibrated using only a few spectra. Complex
systems like natural compounds or the determination of
physical parameters in petrochemical products demand a far
higher effort.
To set up a method, the calibration spectra and their respec-
tive reference values are read into the PLS software. Appro-
priate frequency windows as well as data processing methods
DUHGH¿QHGDQGDFDOLEUDWLRQLVFDUULHGRXW7KHTXDOLW\RIWKH
calibration is evaluated by means of a validation (see
Chapter 4). It is left to the user to decide whether he wants to
carry out the evaluation using an external (test set) validation
or an internal (cross) validation. Very often, cross validation
offers advantages: All spectra are used for the calibration and
the consecutive validation. No part of the measurements is
ORVWE\GH¿QLQJDQH[WHUQDOWHVWGDWDVHWVHH&KDSWHU
The quality of a calibration can easily be assessed by the
YDOXHRIWKHFRHI¿FLHQWRIGHWHUPLQDWLRQR2 and the error of
analysis (RMSECV or RMSEP). This is illustrated in the
following example with an NIR-spectroscopic analysis of a
mixture of Methanol CH3OH, Ethanol C2H5OH and Pro-
panol C3H7OH. The measurement was carried out using the
Bruker near-infrared spectrometer MATRIX-F with a spec-
tral range from 10,000 cm-1 to 4,000 cm-1 (optical path length
of the cuvettes: 2 mm; spectral resolution: 8 cm-1, connected
YLDPOLJKW¿EHUV$WRWDORIPL[WXUHVZHUHPHDVXUHG
with a concentration range of 0 – 100%. Figure 6.1 shows a
selection of the corresponding spectra.
NIR spectra in A strong overlapping of the analyte signals can be observed
particular show even with this three-component mixture. There are mainly
strongly overlapping
bands. Multivariate four major signal groups: The COH-combination vibrations
calibration has distinct (at 4,800 cm-1 WKH ¿UVW RYHUWRQHV RI WKH &+2- and CH3-
advantages over the groups (6,000 cm-1- 5,500 cm-1) and the COH-group
univariate calibration
methods. (7,300 cm-1- 6,000 cm-1), as well as the second overtones of
the CH2- and CH3-groups (8,800 cm-1- 7,800 cm-1). The
spectrum shows no relevant signals above approx. 9,000 cm-
1
. Below 4,400 cm-1 large parts of spectral noise can be found,
which can be explained by strong light loss in the used light
¿EHU
Figure 6.1 Near-Infrared Spectra of mixtures from methanol, ethanol and propanol (cuvette
FRQQHFWHGYLDPOLJKW¿EHUVSDWKOHQJWKPP
This system is now subjected to a PLS calibration. It is gen-

HUDOO\UHFRPPHQGHGWRSHUIRUPWKH¿UVWFDOLEUDWLRQRYHUWKH
entire frequency range. However, one should take care to
select low-noise spectral data. Ranges at the borders of the
maximum possible requency window, or absorption bands
with more than 2.5 absorption units, are generally character-
ized by very small light intensities. Consequently, the
resulting signals contain more noise and should not be used.
Therefore, signals below 4,400 cm-1 and the absorption band
at approx. 4,800 cm-1 should not be considered. Taking
further into account that the spectra do not show considerable
absorptions above 9,000 cm-1 WKH ¿UVW VHQVLEOH IUHTXHQF\
window for the calibration ranges from approx. 9,000 -
5,200 cm-1.
If one performs a validation for an increasing number of
factors, one will usually see an improvement of the results of
DQDO\VLV DW ¿UVW 7KH KLJKHU WKH VHOHFWHG UDQN WKH PRUH
spectral information is processed, and the better the results.
However, this cannot be continued arbitrarily. From a certain
critical number of factors, ever more portions of spectral
noise are added to the analysis model, and the quality of the
UHVXOWVGHWHULRUDWHVRYHU¿WWLQJ
This is shown in Figure 6.2. Here, the mean errors of predic-
tion for the PLS regression of the methanol concentration of
the 30 mixtures from CH3OH, C2H5OH and C3H7OH are
represented exemplarily for a rising number of factors. The
method evaluation was carried out using a cross validation.
First, one recognizes an improvement of the results of anal-
ysis for a rising number of factors. For seven or more factors
the results deteriorate again, since the model becomes
RYHU¿WWHG7KHUHIRUHVL[IDFWRUVVKRXOGEHEHVWVXLWDEOHLQ
order to obtain optimum analysis results for the example
shown. A mean error of prediction of 0,07% is found. This is
a realistic value for the NIR spectroscopic investigation of
pure, liquid multi-component mixtures.
)LJUXH 0HDQ(UURURI3UHGLFWLRQIRUPHWKDQROSORWWHGDJDLQVWWKH
UDQNRID3/6UHJUHVVLRQIRUPHWKDQROHWKDQROSURSDQRO
A comparison of the analysis values and the corresponding
reference data is shown for the 6-factor-model in Figure 6.3.
One recognizes that a good match between both values is
found generally. The analysis was always performed using
independent test spectra, i.e. the query spectrum was not
contained in the calibration data set. Hence, one can assume
similarly good results to be obtained with the future analysis
of new alcohol mixtures.
)LJXUH &RPSDULVRQRIWKHUHIHUHQFHDQGDQDO\VLVYDOXHVIRUD3/6
regression determining the methanol concentration from a
PL[WXUHRIPHWKDQROHWKDQROSURSDQRO506(&9min =
IUHTXHQF\UDQJHFP-1FP-1UDQN
GDWDSUHSURFHVVLQJÄ6XEWUDFWLRQRIDVWUDLJKWOLQH³
6RWKHRSWLPXPUDQNIRUDFHUWDLQGH¿QHGFDOLEUDWLRQPRGHO
can be found easily. Therefore, the only question to be
FODUL¿HGLVZKLFKLVWKHPRVWVXLWDEOHPHWKRGIRUDJLYHQ
task, i.e. which combination of data preprocessing and fre-
quency ranges. Since this question cannot be answered gen-
erally, the optimum frequency ranges and data preprocessing
methods must be determined empirically by „trial and error“.
For this purpose these values are changed systematically and
calculated in each case for a rising number of factors. That
VHWWLQJZKLFKVKRZVWKHODUJHVWYDOXHVRIWKHFRHI¿FLHQWRI
determination R2, and/or a minimum error of prediction,
characterizes the best analysis model. Thus all meaningful
variants at frequency ranges and data preprocessing methods
are tested successively until the optimum model is found.
For the selection of suitable frequency ranges, in most cases
LW LV VXI¿FLHQW WR JURXS WRJHWKHU DSSURSULDWHO\ ODUJH
frequencies (see “Selecting the Spectral Regions” in Chapter
5). Finding of individual spectral data points is typically not
necessary.
A deeper understanding of the basic mathematics is not nec-
essary for the selection of the different data preprocessing
methods and frequency ranges. Good values for R2 are larger
than 90% for solids and larger than 99% for liquid
measurements. Noticably worse values refer to a calibration
PRGHORILQVXI¿FLHQWTXDOLW\DQGVKRXOGXVXDOO\QRWEHWDNHQ
into account.
In order to ensure an uncomplicated comparison of the indi-
vidual models, it is advisable to sum up the most important
parameters in a table. This is shown below for the NIR-
spectroscopic analysis of the Methanol/Ethanol/ Propanol-
mixture discussed previously. (For reasons of a better over-
YLHZWKHUHVXOWVIRURQO\¿YHYDOLGDWLRQVDUHVKRZQKHUHLQ
the case of a real method optimization, all conceivable
meaningful combinations of data preprocessing and fre-
quency ranges would have to be tried one after the other. This
often results in tables with 30 and more documented
methods.)
7DEOH0HWKRGRSWLPL]DWLRQIRUWKH1,5VSHFWURVFRSLFDQDO\VLVRIWKHPHWKDQRO
FRQFHQWUDWLRQLQPHWKDQROHWKDQROSURSDQROPL[WXUHV
No. Data Frequency Optimum Coeff. of Mean error of 5HPDUNV
preprocessing ranges [cm-1] rank determination prediction
5 [%]
1 none 9,000-5,200 9 99.8 0.16% total. Spec.
2 SSL 9,000-5,200 6 >99.9 0.07% total. Spec.
3 Vec. Norm 9,000-5,200 8 99.6 0.42% total. Spec.
4 SSL 7,000-5,200 8 >99.9 0.07% 1.overtones
5 SSL 6,000-5,200 7 >99.9 0.07% no OH
... ... ... ... ... ... ...
,QWKH¿UVWWKUHHOLQHVFDOLEUDWLRQVRYHUWKHHQWLUHVSHFWUDO
region of 9,000 cm-1 - 5,200 cm-1 for different data prepro-
cessing methods are shown. One recognizes that the sub-
traction of a straight line (SSL) is the best suitable data
preprocessing option (Method No.2 in Table 6.1) in order to
REWDLQDJRRGUHVXOWRIDQDO\VLV7KHFRHI¿FLHQWRIGHWHUPL
nation R2 is larger, and/or the mean error of analysis is
smaller than in the two other models (method No.1 and 3).
Looking at further frequency ranges (method No. 4 and 5),
the model cannot be improved any further.
Similarly, neglecting the second overtone of the CH2- and
CH3-vibrations between 8,800 cm-1 and 7,800 cm-1 does not
lead to any measurable losses in the analysis quality, neither
does neglecting the strong OH absorption at approx. 6,900
cm-1. Several models result in a mean error of analysis of
7KHUHIRUH RQ WKH ¿UVW YLHZ DOO WKUHH PRGHOV DUH
equally well suitable to determine the methanol concen-
tration. Nevertheless, it is recommendable to use the model
with the lower number of factors. Methods which work with
fewer ranks usually have a higher stability. Therefore, in the
example shown here, it would be useful to perform a cali-
bration in the spectral region between 9,000 cm-1 - 5,200 cm-1
with the use of a „SSL“ as data preprocessing option (method
No.2).
With these settings, a method can be compiled which is suit-
able for the analysis of new unknown samples. The most
important results should always be written down. Fig. 6.4
shows an example for a validation report for the application
shown here.
Figure 6.4 Validation report
It may surprise some analysts that, in the example shown
here, several very different analytical models lead to almost
LGHQWLFDOUHVXOWV7KHUHIRUHWKLVSRLQWZLOOEHFODUL¿HGIXU
ther: The equivalence of several chemometric models can be
explained on the basis of the factorization of the spectra. To a
certain extent, the individual factors represent “information
units” which represent certain properties (and/or the
combination of certain properties) of the samples. The con-
centration of the analyte, for example, is such a system
property. In the case of a successful factorization, the PLS
algorithm recognizes the factors that are relevant for analysis
and correlates these with the appropriate system properties
(e.g. the analyte concentration). Generally, this succeeds for
a larger number of spectral regions, since most substances
possess analytically evaluable signals in more than one
frequency window of the spectrum. Since each of these
ranges consists of a multitude of data points (i.e. has an
accordingly high analytic information content) often the
system is determined statistically safely for all of these
ranges. Thus, in most cases there is a selection of calibration
models of comparable quality, which lead to similarly good
results of analysi.s
A further important point results from the factorizing of the
spectra. In the case of a univariate calibration, the analysis of
PXOWLFRPSRQHQWPL[WXUHVUHTXLUHVDVXI¿FLHQWVHSDUDWLRQRI
the individual analyte signals. Each component is assigned
to a certain wavelength or a certain area. This is not
necessary in a multivariate calibration. Here, an evaluation
for several components can often be accomplished from
identical spectral structures and data preprocessing methods.
Since with factorization the spectra are separated into
independent information units, it is not necessary to separate
spectral structures manually. Particularly if the single signals
overlap strongly, this is an advantage over the classical
univariate evaluation.
B. Analysis and Determination of Outliers
For the analysis of new unknown samples, they must be

measured spectroscopically and analyzed using the method
which was set up and optimized before. Furthermore, the
Mahalanobis distance or the spectral residuae are calculated.
These values can be used directly to determine outliers.
The recognition of outliers is of particular importance for the
evaluation of the results. It is often quite possible that a given
substance to be analyzed was contaminated or not correctly
measured. This is easily recognizable by a growth in the
Mahalanobis distance or in the spectral residuum. The worse
the correlation between the test spectrum and the calibration
data set, the higher the corresponding values. In case of the
analysis of an outlier, the Mahalanobis distance is greater
than its corresponding threshold, and the sample is labeled as
an outlier.
The Mahalanobis distance and the spectral residuum are a
quantitative measure for the quality of an analysis. If the
values lie below the threshold, it is guaranteed that the anal-
ysis result is reliable. Thus, it is impossible that the analyst
measures an unsuitable sample without being warned by the
software.
The results of the analysis are documented in a report. Figure
6.5 shows an example of such an analysis report, which
contains all important information.
Figure 6.5 Analysis report
C. 3/6UHJUHVVLRQD0HWKRG3URYLGLQJ
,Q¿QLWH$FFXUDF\"
The optimization of chemometric models and the analysis of

QHZ XQNQRZQ VDPSOHV ZHUH GHVFULEHG LQ WKH ¿UVW WZR
sections of this chapter. Here, the focus is set on the possible
sources of error that can occur during the optimization. The
choice of suitable test spectra for the evaluation of the model
DQGWKHYHUL¿FDWLRQRIWKHOHJLWLPDWLRQRIDYDOLGDWLRQLVRI
special importance.
To verify a PLS calibration model, a set of representative test
samples is absolutely necessary. They should cover the entire
range of concentrations to be calibrated and should represent
all possible natural variations of potential samples.
Furthermore, variations in the ambient conditions of the
LQVWUXPHQWVXFKDVÀXFWXDWLRQVLQWHPSHUDWXUHRUSRVVLEOH
diffusion of humidity into the optics, should be taken into
account. Only in this way, the validation of the method can
lead to a reliable statement concerning the expected error of
analysis.
In this context, it is not permitted to choose test samples
which are already part of the calibration data set10. For
example, if a data set is generated by measuring each sample
several times, then all measurements of this sample should
either be included in the test set or in the calibration set. In
case of cross validation, all spectra of a sample must belong
to “spectra to be tested” (leave-out spectra).
The choice of identical data for the test set and the calibration
set is particularly critical. This can bee easily understood
when looking at the equations (2-1) and (2-3). When
FDOFXODWLQJ WKH UHJUHVVLRQ FRHI¿FLHQW b, spectral and con-
centration data are inserted directly into the equation. If,
during validation, the function b is directly correlated to the
spectral data it was calculated from (i.e. test set and calibra-
tion set identical), this results in the concentration data
entered earlier (see equation (2-3)). This “reconstruction” of
the reference values entered originally is the more precise the
more factors were chosen for the calibration.
So, using a number of factors which is large enough would
EHVXI¿FLHQWWRREWDLQD³SHUIHFW´PDWFKEHWZHHQDWHVWVDP
ple and the corresponding spectrum in the calibration set, i.e.
the very analysis value is returned which was initially fed to
the model during calibration. In this case, spectral noise
cannot reduce the quality of the results because the noise
amplitudes of the test and calibration spectra are identical.
This is referred to as a validation using “dependent samples”,
because the samples are already known to the model.
Obviously, the analysis of dependent samples to validate the
method yields results which are completely useless10. This
will be shown in the following example using the 30 spectra
obtained from the mixtures described earlier, containing
methanol, ethanol and propanol. This time, instead of using
the accurate concentrations of the mixtures, random num-
bers between 0 and 100% are chosen, numbers which are
completely unrelated to the actual component values.
)LJXUH 9DOLGDWLRQRID3/6UHJUHVVLRQIRUWKHGHWHUPLQDWLRQRI
0HWKDQROIURPDPL[WXUHRI0HWKDQRO(WKDQRO3URSDQRO
with independent samples for a 13 factor model. As
UHIHUHQFHGDWDDUELWUDU\QRQVHQVHFRQFHQWUDWLRQYDOXHV
EHWZHHQDQGZHUHFKRVHQ7KHYDOLGDWLRQVKRZV
WKDWDQDQDO\VLVLVQRWSRVVLEOHLQWKLVFDVH
An analytically correct validation of this model shows that it
is obviously not suitable for predicting the stated nonsense
concentration values. This is shown in Figure 6.6. Reference
value and analysis value show no observable connection.
E.g., a PLS analysis with a methanol content of 5% results in
a analysis value of 102%. Another sample with a content of
96% is predicted with 29%. As expected, in this example, an
analysis (or in better words, a reconstruction of the
meaningless input values) is not possible.
)LJXUH 9DOLGDWLRQRID3/6UHJUHVVLRQIRUWKHGHWHUPLQDWLRQ
RI0HWKDQROIURPDPL[WXUHRI0HWKDQRO(WKDQRO3URSDQRO
with dependent samples for a 7 factor model. As reference
GDWDDUELWUDU\QRQVHQVHFRQFHQWUDWLRQYDOXHVEHWZHHQ
DQGZHUHFKRVHQ506((
Figure 6.8 The same validation as Fig. 6.7 for a 13 factor model
506((
)LJXUH 7KHVDPHYDOLGDWLRQDV)LJIRUDIDFWRUPRGHO
506(E
The situation changes drastically if the samples used for the

development of the method are also used in the validation of
that method, i.e. if the validation data set is made from
dependent samples. Even for a 7-factor model, a rough cor-
relation between the “real” (nonsense) values and the pre-
dicted “analysis” values can be found (see Figure 6.7).
The accuracy can even be „improved“ by using 13 or 16 fac-
tors (Figure 6.8 and Figure 6.9). The corresponding mean
errors of prediction are obtained as 17,5% for the 7-factor
model, 0,42% for the 13-factor model and 0.04% for the
16-factor model. Therefore, a calibration using 16 factors
seems to yield even better results than the model which is
analytically correct (see Figure 6.3).
This shows impressively that even with a relative small
number of factors the (completely nonsense) values can be
well reproduced. Thus, it is possible to obtain arbitrarily
„good“ analysis results by selecting inadmissible test spec-
tra. However, the method cannot withstand a validation with
real samples.
The credibility of a validation can be assessed easily in
practice. On the one hand, this is possible by checking
characteristic parameters, such as the mean error of analysis.
As already mentioned, the error of prediction must go
through an optimum value for an increasing number of fac-
tors. If, however, a steady improvement is achieved for a
growing number of factors, dependent test spectra have been
used for the validation. This is shown in Figure 6.10 for the
example described above. With a growing number of factors,
the error of analysis decreases to drop down at 16 factors
close to 0%. In contrast, the correctly validated model shows
its minimum at 6 factors with a mean error of analysis of
0.07% which cannot be decreased any more (Figure 6.2).
On the other hand the legitimation of a model can be checked
by a simple sample measurement. For this purpose, a small
number of samples is measured and analyzed. The resulting
error of analysis must be in the same range as the error of
prediction (RMSECV or RMSEP), which was determined
beforehand. If the mean analysis error of the validation data
set is well below the error of the measured samples, the
model might have been validated using inadequate samples.
Figure 6.10 Mean error of estimation plotted against the rank for a
PL[WXUHRI0HWKDQRO(WKDQRO3URSDQRO$VUHIHUHQFH
YDOXHVDUELWUDU\QRQVHQVHFRQFHQWUDWLRQYDOXHVEHWZHHQ
0 and 100% were chosen and validated using dependent
WHVWVSHFWUD7KLVLQDGPLVVLEOHYDOLGDWLRQFDQEHUHFRJQL]HG
E\DQHUURURIDQDO\VLVZKLFKGHFUHDVHVVWHDGLO\ZLWKKLJKHU
ranks.
7
C H A P T E R
CHAPTER1
DEFINITION OF
CHAPTER2
IMPORTANT
ABBREVIATIONS
In chemometric literature, a great number of statistical para-

meters is used to characterize the various methods. A detai-
led knowledge of the individual formulae is generally not
necessary for an execution and optimization of a quantitative
DQDO\VLV+RZHYHUWKLVFKDSWHUJLYHVDPDWKHPDWLFDOGH¿QL-
tion of commonly used abbreviations to help the reader in
studies of specialized literature.
E&RHI¿FLHQW see Calibration Function.
%LDV Generally, the bias is the systematic averaged
deviation between the data set of the true and the
predicted values. It is calculated by averaging all
particular deviations of the samples inside the data sets:
(7-1)
%LDV&RUUHFWLRQ To use calibrations on different

instruments or at different sites sometimes an adaptation
is required, before the calibration can be fully utilized. If
identical samples are measured in different laboratories
but on the same spectrometer type, often systematic
deviations can be found. This bias is case of site change
due to the systematic deviations in the reference values of
the different laboratories. When changing from one
instrument to another, slightest constructional changes in
the spectrometers can lead to different spectra. To adapt
the original analysis values to the expected new values,
the difference (i.e. the bias) is subtracted from the
predicted value. This way, existing calibrations can be
used at different sites or on different instruments.
The bias correction should not be confused with the
Offset and Slope correction. Here the predicted analysis
values Yipred are plotted against the true values Yimeas and
the regression line is calculated. The correction now
adjusts this regression line towards the bisector, so that on
an average the predicted, new values and the original
values match (for more details see Offset and Slope
correction).
Typically the slope of the regression line is “1” or close to
“1”. Thus the ordinate (i.e. the offset) is equal to the mean
systematic deviation of the predicted from the original
values (i.e. the bias). Therefore, offset and bias are
normally almost identical. In the literature, it is often
referred to as “Bias and Slope correction”, although
actually the offset of the regression line is taken for the
correction. If there is however no linear correlation
between the original and the predicted value, the slope of
the corresponding regression line will be different from
“1”. Bias and offset will be different and a bias correction
will lead to different values compared to an offset
correction.
&DOLEUDWLRQ)XQFWLRQ During the calibration, a number
of samples of known concentration is measured. The
calibration function b correlates a property Y (e.g. the
concentration of the analyte) of a system with an
experimentally observable X (e.g. the spectrum):
b = (XTÂ;-1Â;TÂ<
X and Y are written in matrix form. In expert literature,
the function b is often called bFRHI¿FLHQWRUUHJUHVVLRQ
FRHI¿FLHQW:LWKWKLVIXQFWLRQWKHDQDO\WHFRQFHQWUDWLRQ
can be calculated directly from the spectrum (the
procedure is described in Chapters 2 and 3):
< ;ÂE
Experienced method developers often observe the
bFRHI¿FLHQWRIDFDOLEUDWLRQWR¿QGVXLWDEOHVSHFWUDO
regions for method development. These ranges of a
spectrum which contain important information of the
analyte often contribute the largest portions of the
UHJUHVVLRQFRHI¿FLHQW
&RHI¿FLHQWRI'HWHUPLQDWLRQ see R2.
&RUUHORJUDP The correlogram shows the amount of
consistence between spectral and concentration data for a
given rank. Values close to +1 or -1 mark those spectral
regions where a good correlation occurs. Values close to
]HURVKRZDQLQVXI¿FLHQWFRUUHODWLRQ7KHVHUDQJHV
should not be used for the calibration:
(7-4)
'LIIHU The “Differ” value is the difference between the

true concentration of a sample i and the predicted
concentration.
meas pred
Differi = Yi - Yi (7-5)
(UURURI$QDO\VLVsee Error of Prediction.
(UURURI3UHGLFWLRQ The quality of a calibration is
normally determined by its precision in the analysis of
new unknown samples. This is checked during method
development. Here, the overall averaged analysis error is
calculated for all samples. This value is called RMSECV
(see below) for the internal validation, and RMSEP (see
below) for the external validation.
([SODLQHG9DULDQFH see R2.
)DFWRU The concentration data matrix and the spectral
data matrix are broken down into pairs of scores and
loadings vectors by the PLS algorithm (see equation 2-3
and 2-4). Each of these pairs of scores and loadings
vectors are called a factor.
)9DOXH and )3URE: F-values are used for recognizing
outliers in the calibration data set. They can generally be
derived from the spectral- and concentration values of the
measured sample. There are two kinds of F-values: those
which are calculated directly from the spectral residuae,
and those which result from the difference of the true and
the predicted values (predicted by the chemometrical
model). The larger the F-value of the analyzed sample,
the more likely it is an outlier.
F-value calculation for the determination of spectral
outliers:
(7-6)
F-value calculation for the determination of concentration

outliers:
(7-7)
The F-value of a concentration outlier can be calculated

from the difference between the measured values and the
predicted values of the sample i. M is the number of
samples in the calibration data set.
The value and distribution of the F-values of the single
components depends on the application. In order to judge
whether an F-value might indicate an outlier, it must be
compared with the F-values of the other samples of the
data set. Therefore, the so called FProb value is
calculated. This value indicates the probability of the
existence of an outlier in the distribution of all F-values:
(7-8)
If the F-value is “zero”, the resulting FProb is “zero” (i.e.

the probability that the sample is an outlier is 0%). An
LQ¿QLWHVLPDOO\ODUJH)YDOXHOHDGVWRDQ)3URERILH
100% probability).
/HYHUDJH7KHOHYHUDJHLVDPHDVXUHIRUWKHLQÀXHQFHRID
sample on the PLS model. Mathematically, it is the
Mahalanobis distance of the single calibration samples
(see Mahalanobis distance). The leverage value of
outliers is unnaturally high, compared to other samples.
0DKDODQRELV'LVWDQFHDuring factorization, the
measured spectra are decomposed into different factors
and a spectral residuum (see equation 2-3).This applies to
the calibration samples as well as to the analysis samples.
If an analysis of unknown samples is to be performed
with the PLS algorithm, it should be checked whether the
spectra can be analyzed reliably with this method. This is
possible with the calculation of the Mahalanobis distance.
Here, it is checked how well the spectrum of the analyte
³¿WV´WKHVSHFWUDRIWKHFDOLEUDWLRQGDWDVHW
7KH0DKDODQRELVGLVWDQFHLVGH¿QHGDVWKHGLIIHUHQFHRI
the measured spectrum of the analyte to the mean value of
all spectra in the calibration data set, which was used
when reconstructing the spectral data matrix for the given
number of samples. The larger this difference, the larger
the value of the Mahalanobis distance. There are various
possible reasons. External disturbances, such as the
contamination of the samples or disturbing temperature
drifts, lead to a distortion of the peak symmetry, which
results in a growing of the Mahalanobis distance. In the
same way, this value grows if samples are analyzed which
lie outside the concentration range. The Mahalanobis
distance is a quantitative measure for the reliability of an
analysis, because outliers as well as samples with
reference values outside the valid concentration range are
detected.
During the PLS regression, the Mahalanobis distances of
all calibration spectra are calculated. From these results,
the maximum tolerable value of this distance is
determined, for which the spectrum can be analyzed
safely with the given calibration function. Samples which
lie above this threshold cannot be determined reliably.
These samples may be potential outliers.
If the spectral data are factorized according to equation
(2-3), the Mahalanobis distance hi is determined as:
(7-9)
where the calculation is performed for R factors. If the
single scores vectors ti were not calculated from an
unknown sample but from a calibration spectrum, it is
also called “Leverage”. The leverage values of the
FDOLEUDWLRQVDPSOHVDUHDPHDVXUHIRUWKHLULQÀXHQFHRQ
the PLS model.
0'/(Minimum Description Length): An empirical
number to determine the optimum number of factors:
(7-10)
2IIVHWDQG6ORSH&RUUHFWLRQ The offset correction sets

the values of a regression line to the values of the origin
by subtracting the offset (i.e. the ordinate) from the
respective points.
Offset- and slope correction are important in
spectroscopy for the calibration transfer from one
instrument to another. The method is set up on a central
instrument, the “master”. This method is then
implemented on a number of further instruments, the
³VODYHV´,QVWUXPHQWVSHFL¿FV\VWHPDWLFYDULDWLRQVRIWKH
slaves are calculated back to the expected values for the
master system with an appropriate correction. This
LQVWUXPHQWWKHQGH¿QHVDVWDQGDUGWRZKLFKDOORWKHU
measurements are related. Evaluations on a slave
spectrometer can be performed that were set up on the
FHQWUDOPDVWHUVSHFWURPHWHU7KLVLVEULHÀ\GHVFULEHGIRU
a multivariate calibration:11
The spectra which were generated by master and slave
during measurement of the single samples were read into
a spectral data matrix. MX is the spectral data matrix of the
master, and SX is the spectral data matrix of the slave.
From the known concentration values of the calibration
VDPSOHVWKHUHJUHVVLRQFRHI¿FLHQWMb is calculated which
connects the spectral and the concentration data for the
measurement on the master, compare to equation (2-1):
M
Y = M;ÂMb (7-11)
For the measurements with the slave, no extra calibration
is performed. To calculate the concentration data matrix
for the slave measurements, they are linked to the
FRHI¿FLHQWMb, which was generated from the master
measurement:
S
Y = S;ÂMb (7-12)
If master and slave produce identical spectra (MX = SX),

the analysis results are equal for all cases (MY =SY), i.e.
the sample measurements can be performed at any
instrument. This is normally the case with high precision
FT-spectrometers.9
If master and slave generate different spectra from
identical samples, one tries to adjust the differences with
an offset/slope correction. For the predicted concentration
values of the slaves, the corresponding true values of the
slave are plotted which have been determined with an
additional sample measurement. In case of identical
results of both measurements, a regression from the
single Y-values would result in a bisector. As master and
slave generate different spectra, in reality there is a
deviation from the bisector. Therefore, a corrective term
is calculated which converts the determined regression
line into the bisector:
S
Ycorrected = (S;ÂMb) – offset (7-13)
If, however, offset and slope of the regression line need to

be corrected, it is called an offset/slope correction:
S
Ycorrected = ((S;ÂME±RIIVHWÂVORSH-1 (7-14)
Slope and Offset are the respective correction terms with

which a direct calculation of the analyte concentration is
possible. Their calculation is described in 11.
35(66 (Predictive Residual Error Sum of Squares): The
sum of all squared errors of prediction. This value helps
to optimize the number of PLS vectors:
(7-15)
3:6 (Property Weighting Spectrum): The PWS is the

counter of the correlogram equation. High PWS values
mark those spectral ranges, where not only a good
correlation of the data sets can be found (like the
FRUUHORJUDPEXWDOVRDVXI¿FLHQWEDQGLQWHQVLW\FDQEH
observed. To set up a calibration method, spectral ranges
with high correlogram and high PWS values should be
selected:
(7-16)
527KHFRHI¿FLHQWRIGHWHUPLQDWLRQR²) gives the

percentage of variance present in the component values,
which is reproduced in the prediction. This is often called
explained variance7KHKLJKHUWKHFRHI¿FLHQWWKHEHWWHU
the correlation between the concentration data and
spectral data. Low R2 values generally lead to bad
analysis results. There are various possible reasons for
these bad correlations: Selecting inappropriate model
SDUDPHWHUVOHDGVWRDORZFRHI¿FLHQWRIGHWHUPLQDWLRQR2
DVGRHVIRUH[DPSOHDQLQVXI¿FLHQWSUHFLVLRQRI
the reference data, or the existence of outliers in the
calibration data set.
(7-17)
5HJUHVVLRQ&HI¿FLHQW see Calibration function.

5DQNNumber of PLS factors.
5HVLGXXP A factorization can never explain the total
variance of the concentration data or spectral data matrix.
The rest that is not explained by the factorization is called
residuum; the respective values are the residual values
(see equation 2-3 and 2-4). The residuum is the difference
between the real data and the data reconstructed by
factorization. This applies to spectral data as well as to
concentration data.
There is a number of residuae which are of interest in
chemometrics, e.g. the residual value of the spectral
residual matrix F (see equation (2-4)):
(7-18)
The larger the value, the smaller the part of the spectral
structures which can be explained by the factorization.
In the same way, the residual matrix of the concentration
data G describes that part of the components of the
calibration data set which cannot be explained by
factorization:
(7-19)
The calculation of the residual values is useful not only

for the evaluation of calibration models. The residuum of
an analyte spectrum is an important clue for the detection
of outliers. Here, the difference is calculated between the
measured spectrum xi and the spectrum si which was
“expected” from the factorization:
(7-20)
The sum is calculated over all frequency values (index

“j”). Here xi are the measured and si the combined spectra:
(7-21)
The larger the residuum (i.e. the larger the difference

between the real and the spectrum which is expected
according to the factorization) the more likely the sample
is an outlier. In this way, contaminations in the spectrum
can be detected, which overlay the peaks of the analyte
spectrum and lead to an increase in the residual value.
Besides the calculation of the Mahalanobis distance, this
value is most often used for outlier determination.
506(&9(Root Mean Square Error of Cross Validation):
This value is a quantitative measure for the preciseness
with which the samples are predicted during a validation.
The RMSECV is comparable to the RMSEP for the
external validation.
(7-22)
506((: (Root Mean Square Error of Estimation). This

value calculates the analysis error of the calibration
samples. Caution: The RMSEE is not suitable for a
method validation, as no independent test spectra are
analyzed, see Chapter 6-C:
(7-23)
506(/&(Root Mean Square Error of Leverage

Correction): This value is calculated during the
calibration. It estimates the size of an expected error of
analysis for new unknown samples.
(7-24)
506(3 (Root Mean Square Error of Prediction): This

value is a quantitative measure for the preciseness of the
analysis of test set samples. It is comparable to the
RMSECV for the internal validation.
(7-25)
53' (Residual Prediction Deviation): The RPD value is

the quotient of the standard deviation of the reference
values (SD) and the bias-corrected mean error of
prediction of the validation (SEPbias).
(7-26)
with
(7-27)
and
(7-28)
It is a qualitative measure for the assessment of the

validation results. The smaller the SEPbias is, compared to
the variance of the reference values, the better is the
model, i.e. the larger the RPD, the better is the calibra-
tion.
To evaluate the quality of a validation, calculating the
RPD is more meaningful than only looking at the error of
prediction. For example, validating a calibration with a
relatively small range will most likely lead to a small
error of prediction. The model “knows” only little
variation in the reference data set and will therefore
predict the analysis values with a small variance (i.e. an
apparently small error), even if the model itself is not
good. To avoid an over-optimistic interpretation of the
result, the RPD value is calculated: This way it can be
seen that the standard deviation of the reference values is
relatively small (low SD values) and that even a minor
deterioration of the predictive error (growing values for
SEPbias) will lead to a clearly worse RPD value. So even
small prediction errors can belong to a bad calibration
model - clearly indicated by the RPD value.
Looking at the RPD has another advantage: It was
already said that calibration samples should span the
whole range of the calibration range homogeneously.
Methods, which have a large amount of samples in the
FHQWHUEXWRQO\DIHZDWWKHHQGRIWKHUDQJHOLNHLQ¿JXUH
5.1) are less robust. This is being accoutned for in the
RPD value. A more homogeneous distribution of the
reference values leads to a higher standard deviation SD
than clustering at one position. The resulting RPD is
accordingly larger and the calibration model is rated
better.
For historical reasons, the RPD value is often used in the
agro market, for example in the NIR spectroscopic
analysis of wheat. The following rule of thumb is valid
for assesing the quality of a calibration:
RPD between 2.5 -3: method OK for rough screening
RPD > 3: method OK for screening
RPD > 5: method OK for quality control
RPD > 8: method excellent for all analytical tasks.
These values are very dependent on the kind of sample
and the calibration range. The rule of thumb above should
be valid for most applications in the food industry (for
natural, heterogeneous and solid samples). For chemical
analysis (synthetic, homogeneous liquid samples), the
calibrations should show higher RPD values.
The RPDYDOXHLVE\WKHZD\RIWKHVDPHVLJQL¿FDQFHDV
the so called “explained variance” R2. The R2also allows
a qualitative evaluation of the error of prediction during a
validation. For a non-biascorrected calibration (bias = 0)
the following correlation applies:
(7-29)
For a relatively small bias correction, the equation is

approximately valid.
6(&9 (Standard Error of Cross Validation): see
RMSECV.
6(( (Standard Error of Estimation): see RMSEE.
6(3 (Standard Error of Prediction): see RMSEP.
6SHFWUDO5HVLGXXP: see Residuum.
6ORSH&RUUHFWLRQ see Bias Correction.
66((Sum of Square Errors): This value is the sum of the
square residuae. The larger the resulting SEE, the worse
the model explains the variance of the data sets:
(7-30)
8
C H A P T E R
CHAPTER1
SUMMARY AND OUTLOOK

CHAPTER2
The advantages of the multivariate calibration technique are

undisputed today.12,13,14 By taking all spectral information
into account when setting up a method, analyses can be car-
ried out with high precision. Furthermore, it is possible to
identify outliers reliably, and even to evaluate bands which
are massively overlapped with signals of other components.
Yet, at the beginning, multivariate calibration made its way
to the analytical laboratory rather hesitatingly. A reason for
this might have been a certain reluctance on the part of the
method developers to use statistical algorithms. Near-infra-
UHG VSHFWURVFRSLVWV ZHUH WKH ¿UVW ZKR PDGH XVH RI WKLV
method. However, even this was due to a need rather than an
“inexplicable” passion for statistical mathematics: Near-
infrared spectra are often characterized by strong overlap-
ping and broad bands, a task for which the classical univari-
ate method fails.
In the meantime, many analysts make use of the advantage of
chemometrics. Particularly, original doubts could be
UHVROYHG UHJDUGLQJ WKH DSSOLFDWLRQ RI GLI¿FXOW VWDWLVWLFDO
algorithms in routine laboratories. Exact knowledge of these
algorithms is not necessary for practical work. It is possible
WR¿QGWKHRSWLPXPSDUDPHWHUVIRUDFDOLEUDWLRQPRGHOUHOL
ably, simply by a thorough and systematic approach. There is
no advantage of experience or knowledge, which would
work in favor of analysts with an education in mathematics.
Another advantage of multivariate calibration results from
the use of large spectral ranges when setting up a calibration
model: In the region of (near-) infrared spectroscopy or
Raman technology, there are only few substances that have a
spectrum which can be observed in a small frequency win-
dow. Normally, the analyte bands are found in a wide fre-
quency range, which can be accounted for in the factorization.
Therefore, it is highly unlikely that the spectrum contains a
hidden signal which alone leads to a successful analysis, and
ZKLFKWKHDQDO\VWQHHGVWR¿QG7KHRSSRVLWHLVPRUHOLNHO\
,WLVRIWHQHQWLUHO\VXI¿FLHQWWRWDNHODUJHUVLJQDOJURXSVLQWR
account when developing a method. Furthermore, the result
of a PLS-regression usually improves if a higher number of
analytically relevant data points is used.
Therefore, the approach of method development used in
multivariate calibration is entirely different from the
approach used in univariate calibration. In the latter method,
RQH QRUPDOO\ WULHV WR ¿QG D VLJQDO WKDW FDQ EH HYDOXDWHG
quantitatively without any overlapping from other spectral
structures. In multivariate calibration, larger spectral areas
are combined intentionally. It does not matter whether the
signals of various other system properties overlap in these
areas. The combination which is best suited for the analytical
method can be found reliably by testing all sensible
combinations and performing a consecutive validation.
Usually, no expert knowledge is required regarding the
spectra of the pure substances which are to be analyzed.
The obvious conclusion would be to automate the entire
process of method development and optimization. As the
position of most functional groups in (N)IR and Raman
technology are well known, it should be possible to automate
the “testing” of all possible parameters with an appropriate
software routine.
Figure 8.1 shows the result of such an automated optimiza-
tion for the application sample in Chapter 6.
Figure 8.1 The result of an automatic method optimization with the OPUS/QUANT software
%UXNHU2SWLN*PE+(WWOLQJHQ9HUV7KH¿JXUHVKRZVDOLVWRIWKH
calibration models with the smallest analysis error RMSECV.
The result is the same as that from a manual optimization of

the method: There is a large number of different models that
lead to comparably good analysis results. Experience shows
that this is generally true. Method developers do not need to
be burdened with manual optimization of the calibration
model. This task is now carried out by suitable software
routines.
Today, analytical skills are not required primarily for opti-
mizing a method, but for selecting representative calibration
samples, for considering the possible environmental effects
on sample and instrument, for evaluating the achieved results
and for choosing appropriate methods for sample preparation
DQGVXLWDEOHPHDVXUHPHQWFRQ¿JXUDWLRQ7KHVHWDVNVFDQQRW
be performed by a computer.
APPENDIX
CHAPTER1
CHAPTER2
FREQUENTLY ASKED QUESTIONS

CHAPTER3
Experience shows that even after studying chemometrical

literature, problems can arise in the practical day to day
work, which the user has to solve with expertise and skill.
Below, several frequently asked questions are discussed,
which will help the user to integrate NIR spectroscopy into
his daily laboratory work:
1 How many samples do I need for a PLS calibra-

tion?
It depends very much on the application. A calibration should
UHÀHFW DOO QDWXUDO YDULDQFH RI WKH JLYHQ V\VWHP 7KH PRUH
complex an application is, the more different samples are
needed in order to take all variation into account. Furthermore
WKHUHDUHLQÀXHQFHVUHJDUGLQJGLIIHUHQWWHPSHUDWXUHVSHOOHW
size of solids, moisture contents etc. Easy applications like
mixtures of clear, organic liquids require approximately
from 20 samples onwards, whereas petrochemical
applications or natural products like wheat may require data
sets of several hundred up to thousands of samples.
In this context it is important to cover the whole concen-
tration range homogeneously. Experience has shown that a
lot of users start to collect as many data as possible, before
they start building the model. A lot of work is invested and at
a later stage, they realize that a lot of spectra contain
redundant information and do not contribute additional value
WRWKHPRGHO7KHUHIRUHLWLVKLJKO\UHFRPPHQGHGWREXLOGXS
preliminary methods from the beginning, so that further
samples can be measured more selectively.
2 Why can’t I calibrate with only a few samples?
PLS is a statistical procedure. With only a few samples, no
reliable statistics can be achieved. A simple thought
experiment can illustrate this: If one assumes, that an NIR
spectrum consists of about 1,000 data points, there will be
VHYHUDOVSHFWUDOVWUXFWXUHVZKLFK¿WWRWKHJLYHQFRQFHQWUDWLRQ
values accidentally, if only few measurements are available.
Not until a substantial number of samples are collected, real
LHSK\VLFDOO\MXVWL¿HGFRUUHODWLRQVFDQEHIRXQGEHWZHHQ
the spectral data and the concentration values. With sample
sets larger than 20 spectra, the statistical probability of
DFFLGHQWDOFRUUHODWLRQVWHQGVWR]HUR7KHUHIRUHFDOLEUDWLRQV
or feasibility studies with only 5 or 6 samples are in my
opinion not legitimate.
3 Is it possible to generate calibration spectra syn-

thetically, e.g. by mathematical combinations of the
pure component spectra using the spectra
calculator?
No. Real spectra of liquids or solids cannot be generated by
simple adding or subtracting of raw spectra (e.g. the pure
FRPSRQHQWVSHFWUD0RUHRYHULWLVQRWSRVVLEOHWRVLPXODWH
the natural variances of the real system mathematically.
4 Is it at least possible to mix samples in the labora-

tory to comfortably generate a larger number of
samples?
Yes, if the laboratory samples represent the natural variance
RIWKHUHDOV\VWHPDQGWKLVLVWKHGLI¿FXOW\2QO\IHZUHDO
V\VWHPVFDQEHVLPXODWHGXQGHUODERUDWRU\FRQGLWLRQV7KH
easiest was to achieve a robust calibration is collecting
representative calibration samples and their analysis with the
respective reference method.
5 Can I work with stock standards or dilution series?
Unfortunately not. Stock standards only vary in the concen-
tration of the analyte and do not take the variance of the other
components or the matrix into account. Stock standards can
only be used if there are enough samples which represent all
V\VWHP YDULDQFHV 7KH\ DUH WKHQ RQO\ XVHG WR ¿OO WKH
calibration range homogeneously with measurement values.
Working solely with stock standards is only possible if the
samples contain all possible variation of all components,
which is hardly ever the case.
Dilution series are completely unsuitable. By diluting, all
concentration values change collinearly (i.e. in the same
ZD\ 7KH 3/6 DOJRULWKP FDQ WKHUHIRUH QRW GLVWLQJXLVK
between changes of the component and changes of the
PDWUL[7KHUHVXOWLQJFDOLEUDWLRQVDUHFRPSOHWHO\XVHOHVV
6 For my application it is not possible to cover the

whole application range with given samples, i.e.
certain concentration or parameter values do not
occur in reality. What should I do?
7KHUHDUHVHYHUDOSRVVLELOLWLHV
7KH FDOLEUDWLRQV DUH VROHO\ PDGH ZLWK WKH VDPSOHV
available. However, it is not very reliable for the
¶HPSW\¶UHJLRQVRIWKHFDOLEUDWLRQ7KLVLVDFFHSWDEOHLI
in real life there will be no samples at all in this region
and/or if analyses in this region are not important.
Besides, a calibration works quite well for interpola-
tion opposed to extrapolation.
7KHPLVVLQJVDPSOHVDUHJHQHUDWHGLQWKHODERUDWRU\,W
is recommended to mix real samples from different
H[WUHPHUHJLRQVRIWKHFDOLEUDWLRQFXUYHDQGPHDVXUH
WKHP7KH UHVXOWLQJ VDPSOHV DUH DUWL¿FLDOO\ EXLOW EXW
contain the given original matrix.
Several small calibrations are set up in those regions
where samples are available.
7 Are there frequency ranges or data preprocessing
methods which are generally favorable?
'LIIHUHQWXVHUVZLOOVXUHO\JLYHGLIIHUHQWDQVZHUV7KHUHDUH
even well-known manufacturers of NIR spectrometers, who
XVHWKHIXOOVSHFWUXPUDQJHDQGWKH¿UVWGHULYDWLYHDVDVWDQ
GDUG SURFHGXUH7KLV ZD\ DOO VSHFWUDO LQIRUPDWLRQ LV XVHG
DQG ZLWK WKH ¿UVW GHULYDWLYH GLVWXUELQJ EDVHOLQH VKLIWV DUH
reduced. However, this way, disturbing water bands cannot
be eliminated and a derivative often enhances spectral noise
or instrument drifts along the wavelength scale. I personally
tend to think of a PLS calibration as a ’black box’, i.e. I
FKRRVHDVHWRIPRGHOVZLWKSUHRSWLPL]HGSDUDPHWHUV2YHU
the following weeks, the models are tested with real samples.
7KH¿QDOGHFLVLRQIRUDPRGHOLVPDGHDIWHU,KDYHGHYHORSHG
DIHHOLQJIRUWKHLQÀXHQFHRIWKHQDWXUDOGLVWXUEDQFHVRQWRWKH
precision of the calibration and I can choose on the most
robust model. I have stopped relying on my analytical
expertise when it comes to estimating the robustness of PLS
PRGHOVIURPWKHRXWVHWVHHDOVRTXHVWLRQ
8 But how do I treat areas, which hardly show any

DEVRUSWLRQEDVHOLQHRUEDQGVVHQVLWLYHWRGLVWXU
EDQFHVOLNHWKH2+YLEUDWLRQVDWFP-1 and
6,900 cm-1, which are temperature dependent?
Areas which show total absorption or severe spectral noise
VKRXOGREYLRXVO\QRWEHLQFOXGHGLQWRWKHFDOLEUDWLRQ7KH\
ZRXOG JHQHUDOO\ ZRUVHQ WKH UHVXOWV 2WKHU JHQHUDO DGYLFH
FDQQRW EH JLYHQ (YHQ LI DW ¿UVW VLJKW LW ORRNV DQDO\WLFDOO\
SODXVLEOHQRWWRXVHVWUXFWXUDOO\ZHDNDUHDVHJEDVHOLQHVRU
’interference-prone’ bands, it is not always wise to cut those
UHJLRQV RXW 7KHUH DUH PDQ\ FDOLEUDWLRQV ZKLFK QHHG D
baseline to identify and compensate e.g. baseline drifts. And
sometimes also the ’interference-prone’ bands contain
information which are highly relevant for the given applica-
tion, even if it does not happen so often.
In this context the diffusion of ambient water vapor into the
spectrometer is more relevant in terms of the log-term sta-
bility of calibrations. In Figure 5.3 NIR spectra are compared
which are measured with an instrument which is totally dry
inside and another where ambient humidity can diffuse
inside. It can be seen that the second instrument shows strong
absorbance bands of water vapor (7,500-6,700 cm-1 and
5,800 - 5,000 cm-1DFRXSOHRIGD\VDIWHUWDNLQJUHIHUHQFH
7KDWLVDWOHDVWQRSUREOHP1HDUO\HYHU\1,5LQVWUXPHQWLQ
the world has got humidity inside and there is no need to
work with sealed and dry spectrometers.
But due to the exchange of gaseous water vapor with the
surrounding atmosphere, small sharp banded peaks are
DGGHGWRWKHVSHFWUD2EYLRXVO\WKHHIIHFWLVYHU\ORZDWWKH
beginning, if samples are measured immediately after taking
reference. But even though the effect seems to be minor (and
cannot be clearly seen in the spectra shortly after taking
UHIHUHQFHWKHLQÀXHQFHRQKHFDOLEUDWLRQLVUHODWLYHO\KLJK
7KH3/6DOJRULWKPLVPDLQO\ORRNLQJIRUFKDQJHVLQVSHFWUD
LHVKDUSEDQGHGVWUXFWXUHVOLNHYDSRUVSHFWUDLQÀXHQFHWKH
analysis relatively strong. And the better the spectral
resolution of the collected spectra, the more badly this effect
will be.
7KHUHIRUHLWLVKLJKO\UHFRPPHQGHGHUDVLQJDOOIUHTXHQFLHV
from calibration equations which incorporate the regions
ZKHUHZDWHUYDSRUDEVRUEV7KLVLVWUXHHYHQLIWKHDQDO\WHV
themselves show nice bands in the corresponding regions.
7KLV PDLQO\ RFFXUV HJ LQ DJULFXOWXUH DSSOLFDWLRQV ZKHUH
water or protein need to be determined. Especially for the
calibration of the humidity itself (i.e. the content of liquid
ZDWHULQVLGHRIWKHVDPSOHVLWVRXQGVa priori not logical to
neglect the water vapor bands.
Nevertheless, for protein there are enough frequencies left to
guarantee that all relevant spectral information is inside of
the calibration. Protein frequencies in the water vapor region
RIIHURQO\UHGXQGDQWLQIRUPDWLRQ7KHVDPHLVWUXHIRUWKH
calibration of water inside of the samples. Here also the
edges of the big water bands contain enough information to
guarantee a disturbance free analysis. It has been proven in
numerous studies that mostly calibrations which avoid the
incorporation of water vapor bands outperformed the other
ones in terms of stability. Even if smaller frequency ranges
has been chosen. So, the frequency range of the water vapor
bands seems to be a „sleeping risk“ for any calibration and it
is pretty useful to neglect the corresponding areas inside the
calibration curves.
9 During the optimization of the model parameters

(optimum frequency ranges, data preprocessing
DQGUDQN,IRXQGDODUJHQXPEHURIPRGHOVZKLFK
lead to similar validation results. Are all these
models equally good in practice in determining
unknown samples?
1RLQPRVWFDVHVQRW7KLVLVRQO\WKHFDVHLIDODUJHQXPEHU
of samples have been collected and measured over a long
SHULRG2QO\WKLVZD\LWLVDVVXUHGWKDWDOOQDWXUDOYDULDQFHV
are represented in the sample set. In reality, only a limited
number of samples is accessible and the time frame for the
method development is narrow. Here the reliability of a
method should be tested occasionally by measuring new
validation samples and be exchanged if necessary by a better
PHWKRGDQGRUDQLPSURYHGSDUDPHWHUVHW7KHH[SHULHQFH
shows that methods, which showed comparable validation
results during the method evaluation (i.e. comparable
RMSEP/RMSECV and R2YDOXHV FDQ KDYH VWURQJO\
different precision in reality.
7KLVZLOOEHVKRZQE\DUHDOZRUOGH[DPSOH)RUWKHH[DP
ination of the „content uniformity“, a pharmaceutical com-
pany wants to set-up an NIR spectrometer which measures
WKHDFWLYHLQJUHGLHQWLQWKHLUWDEOHWV7RPDNHVXUHJHQHUDWLQJ
a feasible data set, samples has been collected from different
batches by using reference values which cover the whole
concentration range homogeneously. So, more than 40
samples has been measured and the method developer was
VXUH WKDW PRUH RU OHVV DOO UHOHYDQW YDULDWLRQV IURP KLV
production process has been incorporated in his chemomet-
ric method. Figure A.1 shows the results of the validation:
Again a higher number of different models has lead to com-
SDUDEOHYDOLGDWLRQUHVXOWV506(3DURXQGÄ³$SULRULDOO
of these models seem to be equally „good“. But to check,
how robust the different models are, a separate test set from
further batches was generated and analysed.
Figure A.1 Example of the prediction error of several chemometric models during method
development and by using a separate sample set
7KH UHVXOWV DUH DOVR SORWWHG LQ )LJXUH $ 1RZ GLIIHUHQW
models show enormous differences in their performance.
Model No.1 gives comparable RMSEP-values related to the
validation results. Model No. 10 is approximately one order
RIPDJQLWXGHZRUVH(YHQWKDWDUHSUHVHQWDWLYHQXPEHURI
samples and batches has been used to set-up the calibration
this models is so sensitive to minor differences which
occurred in separate test set that it gives totally awful results.
In contrast model No. 1 is robust - by using the same
calibration spectra and reference values from model No. 10 -
just due to a better setting.
7KLVH[DPSOHVKRZVKRZLPSRUWDQWLWLVWRFKHFNWKHPRGHO
stability continuously. Just by looking at the R2 and RMSEP
values during the method development is not enough. Espe-
cially if this model is chosen blindly, which gives the lowest
prediction errors during the validation routine. It can be seen
in Figure A.1 that model No. 9 gives the lowest RMSEP for
the validation. But this model is enormously unstable in
gives unacceptably bad results of the RMSEP for the separate
sample set. Model No. 9 should be chosen in no case.
2IWHQLWVKRZVXSWKDWWKHRSWLPDOSDUDPHWHUVIRUDUREXVW
model cannot be recognized a priori, since during the method
RSWLPL]DWLRQWKHLQÀXHQFHVRIDOOIXWXUHGLVWXUEDQFHVDUHQRW
known. Also analytically plausible measures (like calculating
WKHGHULYDWLYHIRUWKHFRPSHQVDWLRQRIEDVHOLQHGULIWVGRQRW
always lead to the development of the most stable model.
7KLVFDQLQP\RSLQLRQRQO\EHDFKLHYHGE\WKHDGGLWLRQDO
validation of the calibration from time to time. However only
VXI¿FLHQWO\ODUJHVDPSOHVHWVZKLFKUHÀHFWall characteristics
of the system representatively, guarantees an exact estimation
of analysis errors during the optimization of the method.
10 Can I be really sure that a good result for the vali-

dation leads also to an accordingly good results
with the analysis of unknown samples?
7KLVLVRQO\WKHFDVHLIWKHQXPEHURIVDPSOHVLQWKHFDOLEUD
WLRQVHWLVODUJHHQRXJKVHHDERYH
11 Is it generally better to choose models with a low

rank?
1RUPDOO\PRGHOVZLWKORZUDQNDUHSUHIHUDEOH7KHUXOHRI
thumb is: the less ranks are chosen, the more stable is the
calibration. However there are examples which show that
those models can be inferior to those with higher rank. As
mentioned before: For a limited number of samples, not all
SRVVLEOHGLVWXUEDQFHVDUHNQRZQ\HWDQGQRGH¿QLWHFRQ
clusion can be made regarding the best possible method. It
cannot be ruled out that when re-checking the method with
new samples at a later stage, a different method with a dif-
ferent parameter set - and probably with higher rank - might
be better.
12 Is it helpful to enter the values of the disturbing
parameter, or the sample temperature into the
software when calibrating the system, to make sure
that they are taken into account?
Most of the chemometric software packages us the PLS1
algorithm, as it generally leads to good results. With this
algorithm, only the component values of the analyte are
taken into account and are correlated to the corresponding
spectral features. Anything else will be detected as distur-
bances, i.e. the concentration data are treated as a vector and
the other component values or temperature values do not
OHDGWRDFKDQJHRIWKHUHVXOWV7KHDGGLWLRQDOYDOXHVDUHQRW
taken into account during the analysis. If you want to analyze
a multiple-component system with PLS1, it is applied
successively to all calibrated components. Specifying the
sample temperature only makes sense if you want to calcu-
late the temperature of the unknown samples, i.e. use the
spectrometer as a thermometer.
13 What exactly are disturbances? How can I avoid

them or consider them in the model?
7KH 3/6 FDOLEUDWLRQ WULHV DVVLJQ VSHFWUDO VWUXFWXUHV RI WKH
DQDO\WHWRLWVFRPSRQHQWYDOXHVHJFRQFHQWUDWLRQ&KDQJHV
in the spectrum are correlated to changes in the component
YDOXH 7KLV FRUUHODWLRQ JHWV ZRUVH LI WKH DQDO\WHVSHFL¿F
FKDQJHV LQ WKH VSHFWUXP RYHUODS ZLWK QRQVSHFL¿F 7KHVH
structures can have multiple reasons: absorptions by the
other components of the sample, straylight effects of different
size particles, changes in the sample temperature etc. It is
obvious that all structures that over lap the analyte spectrum
PDNH WKH FDOLEUDWLRQ PRUH GLI¿FXOW 7KH VPDOOHU WKH
concentration of the analyte is, the more structures will
RYHUODSLWVVLJQDO7KHUHIRUHWKHGHWHFWLRQOLPLWRIDVXEVWDQFH
LQ D VSHFL¿F DSSOLFDWLRQ LV PXFK PRUH GHSHQGHQW RQ WKH
natural variance of the matrix than on the spectral noise of
the instrument. Spectroscopic methods like e.g. the NIR
spectroscopy are normally not suitable for trace analysis
DSSOLFDWLRQV 7KH UHDVRQ LV WKH JHQHUDO SUREOHP WR
GLIIHUHQWLDWH WKH DQDO\WHVSHFL¿F VLJQDOV PDWKHPDWLFDOO\
from the other signals. Here, chromatographic methods (like
+/3& DUH LQ DGYDQWDJH DV WKH\ SK\VLFDOO\ VHSDUDWH WKH
different components and thus making them more clearly
visible.
7KHVHGLVWXUEDQFHVFDQQRUPDOO\QRWEHDYRLGHG+RZHYHU
by choosing a good sample preparation and by intelligent
PHDVXUHPHQW FRQ¿JXUDWLRQV WKHVH LQÀXHQFHV FDQ EH PLQL
mized. Some examples:
Heterogeneous samples: Milling or using an integrat-
ing sphere with sample rotator.
7HPSHUDWXUHGULIWV0HDVXUHPHQWLQWKHUPRVWDWWHG
cuvettes or vials.
Incidence of extraneous light: Shading of the mea-
VXUHPHQWZLQGRZRUXVLQJDQG)7LQVWUXPHQW
Filming or adhesion of the sample (e.g. with food
VDPSOHV0HDVXUHPHQWLQWUDQVPLVVLRQ
6WURQJVFDWWHULQJHJRIDWDEOHW0HDVXUHPHQWLQ
transmission with focussed beam.
Coated tablets: Measurement in transmission with
focussed beam.
Measurement through packaging material with probes.
But also the mathematical treatment of the raw spectra can
minimize perturbations. An MSC (Multiple Scattering Cor-
UHFWLRQLVIRUH[DPSOHXVHIXOIRUVWURQJO\VFDWWHULQJVDPSOHV
and a derivative helps against baseline drifts.
14 Aqueous solutions react quite sensitively on tem-

SHUDWXUHÀXFWXDWLRQV,VLWEHWWHUWRFROOHFWDODUJH
number of samples in a wide temperature range (to
DFKLHYHDUREXVWPRGHORUVKRXOG,JHQHUDOO\
thermostat the samples during the analysis (to
DYRLGWHPSHUDWXUHÀXFWXDWLRQV"
Both is possible, but normally the samples are thermostatted.
7KLVZD\WKHEHVWUHVXOWVDUHDFKLHYHGZLWKDFFHSWDEOHHIIRUW
15 :KHQGRHQYLURQPHQWDOLQÀXHQFHVRQWKHLQVWUX
ment itself need to be compensated?
In general, the instruments are stable, at least for a few hours,
VR WKDW HQYLURQPHQWDO LQÀXHQFHV VKRXOG QRW PDWWHU LI D
background is taken from time to time. However, this does
not apply for many in-line applications. For these kinds of
measurements, it is often not possible to perform easily a
background measurement as the measurement probe is inte-
grated directly into the process. Long-term changes in tem-
perature and humidity can impair the measurement results
FRQVLGHUDEO\6HH)LJXUH,QWKLVFDVHWKHVSHFWURPHWHU
VKRXOGEHWKHUPRVWDWWHGLQDQDLUFRQGLWLRQHGFDELQHW7KH
LQÀXHQFHRIKXPLGLW\FDQEHHOLPLQDWHGE\XVLQJHQFDSVX
lated instruments, purging the spectrometer with dry air or
excluding water bands within the calibration.
16 How do I recognize collinear data sets?

In mathematics, two vectors are collinear if one of them is a
multiple of the other. In terms of analytical chemistry, two
data sets are collinear if two or more component values of the
UHIHUHQFHGDWDFKDQJHÃLQWKHVDPHZD\µ6HH7DEOH,Q
this context, it does not matter whether the different com-
ponent values change to the same extend or whether they
increase or decrease. If the reference data have two or more
components which show collinearity they should not be used
for the calibration as in this case the PLS algorithm is not
able to assign the spectral structures to the correct corre-
sponding component (Exception: systems where the com-
ponent values are always collinear, e.g. two component
V\VWHPV&ROOLQHDUGDWDVHWVFDQEHUHFRJQL]HGYHU\HDVLO\
by plotting the component values in an x/y diagram (see
)LJXUH$
In the vector algebra, there is no graduation for collinearity.
Nevertheless, in the context of PLS calibration one is often
talking about a ‚strong‘ or a ‚weak‘ collinearity depending
on whether there is a close connection between the reference
data or not.
D
Figure A.2 Comparison of different component values a) collinear,

b) slightly collinear, c) non-collinear
17 Are collinear data sets completely of no use for

calibration purposes?
Yes, if the collinearity has been caused by a careless selec-
tion of the reference values (e.g. if they are the result of a
GLOXWLRQVHULHVDQGLILWKDVQRWQDWXUDOUHDVRQV0DQ\V\VWHP
properties, however, have such a ‚natural‘ relation and
consequently the reference values are generally collinear. As
in this case, the component values are always related with
each other, there is no need to distinguish between them.
From one component value you can deduce all other
component values (which are in a collinear relation to the
FRPSRQHQWYDOXHLQTXHVWLRQ,QWKLVFDVHFROOLQHDUGDWDVHWV
can be used for the calibration without any problems.
An example for it is the spectroscopic determination of the
RFWDQHQXPEHU$W¿UVWVLJKWLWVHHPVWREHP\VWHULRXVWKDW
the knock behavior of SI engines for different fuel mixtures
can be predicted by a NIR measurement. Nevertheless, there
LV QR GRXEW DERXW WKH VHULRXVQHVV RI WKH FDOLEUDWLRQ 7KH
NQRFN EHKDYLRU LV GLUHFWO\ FROOLQHDUO\ UHODWHG WR WKH
individual fuel components. As these components can be
measured spectroscopically, from their values the octane
number can be deduced directly.
18 I am looking for a low-concentration component in

a complex matrix. Is it appropriate to use this
matrix as background measurement in order to
‚extract mathematically‘ the component which I‘m
looking for, and in doing so, to make it more
visible?
7KLVSURFHGXUHLVLQQRFDVHUHFRPPHQGDEOH$VVRRQDVWKH
matrix shows the slightest changes during future analyses
ZKDWLVQHDUO\LQHYLWDEOHDGGLWLRQDOLQWHUIHULQJVWUXFWXUHV
RFFXU LQ WKH VSHFWUXP VHH )LJXUH 7KHVH DGGLWLRQDO
LQWHUIHUHQFHV ZLOO OHDG ZLWK D JUHDW SUREDELOLW\ WR VLJQL¿
cantly poorer analysis results as if air had been used as
background. Besides the PLS algorithm is able to compen-
VDWHLQWHUIHULQJPDWUL[LQÀXHQFHVRQWKHPRGHOSURYLGHGWKDW
these interference have already occurred during the
DFTXLVLWLRQRIWKHFDOLEUDWLRQVSHFWUD7KHUHIRUHWKH\VKRXOG
not be ‚blended out‘.
19 Is it allowed to improve the analysis results by
performing multiple measurements and averaging
the spectra afterwards?
<HV7KLVSURFHGXUHLVHVSHFLDOO\UHFRPPHQGDEOHLQFDVHRI
heterogeneous samples.
20 Can I apply this procedure also to the calibration

spectra, i.e. after a multiple measurement only the
average spectra are used for the calibration?
Yes, but the resulting for the validation (RMSEP or
506(&9 DOORZV RQO\ D URXJK HVWLPDWLRQ IRU D FRUUH
sponding future multiple PHDVXUHPHQWV 7KH LQGLYLGXDO
measurements will show a greater scattering of the values. In
addition, take into consideration that this procedure will
probably not improve the PLS regression (i.e. the resulting b
FRHI¿FLHQW )RU WKLV FDVH JHQHUDO LQIRUPDWLRQ FDQQRW EH
JLYHQ EHFDXVH WKH FKDQJHG DYHUDJHG FDOLEUDWLRQ GDWD VHW
will also lead to different model parameters (different fre-
quency ranges, different data preprocessing methods and
GLIIHUHQW UDQNV &RQVHTXHQWO\ WKH UHVXOWLQJ YDOLGDWLRQ
results can not be compared with each other.
Nonetheless, an analysis of strongly heterogeneous samples
is often only possible if a multiple measurement is performed
and the results are averaged afterwards. In this case, the
calibration spectra should have been acquired under similar
conditions, i.e. the calibration should be performed on the
basis of the average spectra.
21 Can I correct outliers in the calibration data set on

the basis of the existing calibration curve, i.e. for
the calibration I substitute the current reference
value for the value I would expect on the basis of
the already existing calibration curve?
I have already discussed this question several times in a
FRQWURYHUVLDOPDQQHU7KHVXSSRUWHUVSUHVHQWWKHIROORZLQJ
arguments: In general, the quality of a calibration is mainly
GHWHUPLQHGE\WKHTXDOLW\RIWKHUHIHUHQFHYDOXHV7KLVPHDQV
that the spectroscopic measurement is generally much more

reliable than the value provided by the reference lab.
7KHUHIRUHLWVHHPVWRVXJJHVWLWVHOIWKDW\RXFDQKDYHSHUVH
PRUH FRQ¿GHQFH LQ VSHFWURVFRSLF PHDVXUHPHQWV DQG WKDW
you can substitute the reference values for the values
calculated by the PLS regression. If you follow this idea
consequently till end, it would mean that you can improve
your calibration gradually by correcting each time the worst
values as described above and re-performing a calibration
with the improved parameter set in order to obtain step by
step a constantly improving calibration function.
,P\VHOIFRQVLGHUWKLVSURFHGXUHDVQRWWUXVWZRUWK\7KHQHZ
reference value does not include any additional information
that is relevant to the calibration because the corrected value
has been taken from an already existing calibration curve. A
calibration model can not be improved on the basis of
reference values which originate from the calibration model
LWVHOI 7KLV SURFHGXUH RQO\ JHQHUDWHV DGGLWLRQDO GDWD SRLQW
that lie ideally on the calibration curve, giving the impression
RI D SHUIHFW ¿W WR WKH H[LVWLQJ GDWD %XW WKHVH ÃLGHDOµ GDWD
points are deceptive and lead to the misapprehension of
having produced a good calibration model (with a seemingly
ORZQXPEHURIDQDO\VLVHUURUV%XWWKHWUXHTXDOLW\RIWKH
calibration remains hidden.
22 Are there any disadvantages if I exclude outliers

unhesitatingly from the calibration data set?
Yes! It is not obvious a priori whether the outlier results from
a faulty measurement or whether the PLS model is not good
HQRXJK WR REWDLQ UHOLDEOH DQDO\VLV UHVXOWV Ã2XWOLHUVµ DUH D
great thing. Especially such sample that show poor mea-
surement errors are well suited for testing the robustness of
FKHPRPHWULFPRGHOV<RXZLOO¿QGRXWWKDWGLIIHUHQWPRGHOV
differ in their sensitivity with respect to these spectra that do
not run true to type. For future routine analyses, it is
recommendable to prefer those models which also analyze
those ‚troublesome‘ spectra reliably.
2QO\ WKRVH VSHFWUD ZKLFK KDYH EHHQ UHYHDOHG E\ DQ LQGH
pendent reference method being the result of a faulty mea-
surement should be exclude from the data set. If, however,

the reference method reveals that the so called ‚outlier‘ is the
result of a correct measurement, this sample is important for
setting up the model. Due to the fact that in practice all
occurring sample are to be analyzed, these samples should be
included into the model and they should be analyze reliably
in the course of the validation. In the end, especially those
VDPSOHV DUH YHU\ LPSRUWDQW WR ¿QG D VXLWDEOH FDOLEUDWLRQ
function.
23 After having excluded the outliers, do I have to

UHSHDWWKHYDOLGDWLRQLQRUGHUWR¿QGWKHRSWLPDO
model parameters?
<HVEHFDXVHWKHRXWOLHUVKDGDQXQIDYRUDEOHLQÀXHQFHRQ
the calibration function and they have lead to an unnatural
LQFUHDVHRIWKHDQDO\VLVHUURU506(3DQG506(&9
24 In many studies and publications, the analysis error

is indicated on the basis of a validation of the
spectra which have already been used for the
calibration, i.e. the spectra in the calibration set and
in the validation set are identical. In this case, the
resulting reports indicate RMSEE or SEE (also
506(&RU6(&LQVWHDGRI506(&9RU6(&9
RU506(36(3,VWKLVSURFHGXUHWUXVWZRUWK\"
If the calibration set and the validation set include identical
spectra it is not possible to make a reliable statement about
the actual error that can be expected for an application in
UHDOLW\ 7KHVH PRGHOV WHQG WR ÃRYHU¿WWLQJµ ZLWK LQFUHDVLQJ
UDQNQXPEHUV7KHUHDVRQIRULWLVWKDWWKHYDOLGDWLRQVSHFWUD
DUHDOUHDG\LQFOXGHGLQWKHPRGHO7KHKLJKHUWKHVHOHFWHG
rank is the more it is simply reconstructed and the lower the
VHHPLQJDQDO\VLVHUURULV:LWKDQDSSURSULDWHRYHU¿WWLQJ
you can reconstruct nearly all values you like (see Figure
7KH 506(( YDOXH RU 506(& YDOXH PDNHV RQO\
sense in comparison with the RMSECV or RMSEP value up
to the optimum rank in order to make the difference between
WKH LGHDOL]HG DQG WKH UHDO PRGHO YLVLEOH 2SWLPL]LQJ WKH
model only on the basis of the RMSEE value (or RMSEC

YDOXHRUDUHDOLVWLFHVWLPDWLRQRIWKHDQDO\VLVHUURUWKDWLVWR
be expected is not trustworthy at all.
25 Do I have to use only absorbance spectra for the

calibration or can I use also single channel spectra,
interferograms etc.?
In general, all structures can be calibrated. But absorbance
spectra have the advantage to show a linear relation between
absorbance and concentration due to the Lambert-Beer law.
Such a linear relation is quite useful to calculate precise
PRGHOVZLWKDKLJKVWDELOLW\ORZUDQNV
26 Does it make sense to acquire the calibration

spectra always with the highest possible resolution
of my spectrometer (in order to measure most of the
SK\VLFDOLQIRUPDWLRQ"
No. Most systems are ‚physically overdetermined‘, i.e. the
spectrum includes more data points than it is necessary for
the evaluation. Fort most applications, a high resolution does
QRW PHDQ DQ DGGLWLRQDO DGYDQWDJH 2Q WKH FRQWUDU\ WKH
measurement will take unnecessarily longer or the spectral
noise increases. For most applications, the manufacturers of
)7 VSHFWURPHWHUV UHFRPPHQG D UHVROXWLRQ RI FP-1 to
32 cm-17KLVLVDJRRGFRPSURPLVHEHWZHHQPHDVXUHPHQW
WLPHDURXQGVHFDQGOLJKWJDLQ'LVSHUVLYHLQVWUXPHQWV
work even with a much more lower resolution (ca. 25 -
100 cm-1GHSHQGLQJRQWKHREVHUYHGSRVLWLRQLQWKHVSHF
WUXP%XWIRUPRVWDSSOLFDWLRQVLWLVXQFULWLFDO2QO\DIHZ
applications require a higher resolution. An example is the
determination of the concentration of the active ingredients
in pharmaceutical tablets. In most pharmaceuticals, the con-
centration of the active ingredient is so low that indeed a high
resolution is required to obtain analytically useful results.

27 If a high resolution is not required for most applica-
tions so I do not need a spectrometer with a high
resolution capability. Is this true?
No. You need a high resolution spectrometer in order to
generate stable analysis results or if you intend to transfer the
calibrations to other instruments. But as already said: for
most applications a high resolution is not necessary. In most
cases, the analysis error depends on the accuracy of the ref-
erence method and on an appropriate sample selection. Nev-
ertheless, the resolution capability of a spectrometer is of
crucial importance, however, at a point the user would not
H[SHFWLW7KHZDYHQXPEHUVFDOH[D[LVLQWKHVSHFWUXPRI
the spectrometer is factory-calibrated. For this purpose,
typically a spectrum of the ambient air is acquired and the
peak positions of the water vapor is determined. As the water
vapor peaks remain more or less unchanged they are well-
VXLWHGIRUFDOLEUDWLQJWKHVSHFWURPHWHU7KHKLJKHUWKHZDWHU
vapor peaks are resolved the better the x-axis calibration will
EH 1RZDGD\V )7VSHFWURPHWHUV RIIHU D PHDVXUHPHQW
accuracy of 0.1 cm-1 (i.e. all spectrometers leaving the factory
KDYHORZHUWROHUDQFHVUHJDUGLQJWKHZDYHQXPEHUVFDOH7R
ensure this measurement accuracy, the spectrometer has to
be capable of detecting the water vapor peaks with a
resolution of at least 2 cm-1.
A lower resolution capability of the spectrometer will lead to
a loss of measurement accuracy at the factory calibration.
7RGD\GLVSHUVLYHLQVWUXPHQWVSURYLGHDPHDVXUHPHQWDFFX
racy of about 2 cm-1 on the wave number scale (spectral res-
olution capability of about 25 - 100 cm-1GHSHQGLQJRQWKH
REVHUYHG IUHTXHQF\ LQ WKH VSHFWUD VHH )LJ $ 7KHVH
inaccuracies make it impossible to transfer the calibrations
GLUHFWO\WRDQRWKHULQVWUXPHQW7KLVHIIHFWLVZHOONQRZQWR
WKHXVHUV7KHQHFHVVDU\DGDSWDWLRQRIWKHZDYHQXPEHUVFDOH
is called ‚standardization‘. In the course of the
standardization, the x-axis of a slave instruments is adapted
to a so called ‚master‘ of which the measurements are
SRVWXODWHGWREHÃWUXHµ6HHDOVRELDVDQGVORSHFRUUHFWLRQ
Such an adaptation is not only necessary if you want to
transfer calibrations to other instruments but also after the
H[FKDQJHRIDQRSWLFDOFRPSRQHQWHJOLJKWVRXUFH

Figure A.3 Single channel spectrum of water vapor with a resolution
of 2 cm-1 and 25 cm-1
7KLVDGDSWDWLRQFDQEHDYRLGHGLI\RXXVHDQ)7VSHFWURPH
ter with a measurement accuracy about 20 times better
(0.1 cm-16RDKLJKUHVROXWLRQFDSDELOLW\RIDVSHFWURPHWHU
is indirectly decisive for the usefulness of the spectrometer
during its routine operation, even if most applications do not
require a high resolution.
28 I only want to determine the moisture content in my

WHVWPDWHULDOXVLQJ1,57KLVLVDYHU\VLPSOH
application. Besides, the water spectrum shows
such broad bands that at least for this application a
high-resolution spectrometer is not necessary.
It is a very popular mistake that applications with ‚broad-
band‘ analyte signal (e.g. determination of the moisture con-
WHQWUHTXLUHRQO\DQLQVWUXPHQWZLWKDORZUHVROXWLRQFDSD
bility. As already explained in question 26 and 27, the
resolution capability determines the required measurement
accuracy but not if a sharp-banded or broad-banded compo-
nent needs to be analyzed. In other words: even applications
with sharp analyte structures or more complex analytical
problems can be solved with a lower resolution, provided

you are willing to accept cuts in the long-term stability of the
PHWKRGVRULWVWUDQVIHUDELOLW\7KLVLVWKHGHFLVLYHSRLQW
29 Should I prefer in general a cross validation to a test

set validation as a cross validation requires less
spectra?
,IRQO\DOLPLWHGVDPSOHVHWZLWKWRVDPSOHVLVDYDLO
able you should prefer a cross validation. But nevertheless,
you should try to get more samples in order to check the sta-
bility of the different models with a corresponding test set.
6HHFKDSWHU+
30 How do I optimize the optical path length for NIR

measurements?
,Q1,5WKHH[WLQFWLRQFRHI¿FLHQWVDUHFRPSDUDEOHIRUPRVW
substances. And in most cases, a path length of 1mm (some-
WLPHVPPLVRSWLPDOIRUWUDQVPLVVLRQPHDVXUHPHQWRIOLT
uids. A path length can be considered as ideal if the spectrum
shows absorbance values of 0.7 to 1.0 in the calibration
range. And it does not matter whether there other peaks
outside the calibration range showing total absorbance.
7KHVHSHDNVGRQRWKDYHDQLQÀXHQFHRQWKHFDOLEUDWLRQ,Q
this context absorbance values above 2.5 should not be used
(because generally they include a high portion of spectral
QRLVH
Sometimes, an application requires a larger path length, for
example for the analysis of heterogeneous samples (to cover
DPRUHUHSUHVHQWDWLYHVDPSOHFURVVVHFWLRQ,QWKLVFDVH\RX
can use alternatively higher overtones for the evaluation
XVLQJ SDWK OHQJWK RI PP RU PRUH 7KLV DSSOLHV IRU
example, to food samples. In general, milk and meat prod-
ucts are measured in NIR using these larger path lengths.
Even if you analyze very low analyte concentrations it is
recommended to use larger path length. In doing so, the
DQDO\WHEDQGVDUHLQWHQVL¿HGDQGWKHVLJQDOVRIWKHXQLQWHU
HVWLQJPDLQFRPSRQHQWVGLVDSSHDUGXHWRWRWDODEVRUEDQFH
If, in addition, the analysis of the main components is
UHTXHVWHGLWLVRIWHQSRVVLEOHWR¿QGVXLWDEOHVSHFWUDOUDQJHV

outside the total absorption regions allowing also their
GHWHUPLQDWLRQ 7KLV FDQ EH IRU H[DPSOH SHDNV RI ORZHU
intensity or band shoulders of higher peaks with total absor-
bance on top. So main components and minor components
can be analyzed together using larger path length (i.e. you
need not repeat the measurement for the main component
XVLQJDVPDOOHUSDWKOHQJWK
31 +RZFDQ,¿QGRXWWKHRSWLPDOPHDVXUHPHQWVSRW
VL]HHJIRU'5,)7PHDVXUHPHQWV"
7KHQHFHVVDU\PHDVXUHPHQWVSRWVL]HGHSHQGVPDLQO\RQWKH
particle size and the heterogeneity of the sample. Finally, the
measurement spot must cover a representative sample cross-
VHFWLRQ 2IWHQ D PHDVXUHPHQW VSRW GLDPHWHU RI VHYHUDO
centimeters is required (e.g. in the food and animal food
LQGXVWU\ SRO\PHU LQGXVWU\ HWF $ VDPSOH URWDWRU ZLOO
improve the result once more as it rotates the sample
aFHQWULFDOO\RYHUWKHPHDVXUHPHQWVSRW
32 ,VLWUHFRPPHQGDEOHWR¿QGWKHRSWLPDOVSHFWUDO
ranges for the calibration on the basis of spectra
WDEOHVVHH7DEOHRU)LJXUHRUZLWKWKHDLG
of the spectra of the corresponding pure substances
(by measuring the exact positions of the main
absorption bands and entering these values into the
FKHPRPHWULFVRIWZDUH"
7KLV SURFHGXUH LV QRW UHFRPPHQGDEOH EHFDXVH WKH VSHFWUD
tables do not reveal for a special case which bands are over-
ODSSHGE\LQWHUIHUHQFHVWRZKDWGHJUHH7KHSHUVRQLQFKDUJH
with developing chemometric method should invest time in
providing representative calibration data sets and checking
critically the robustness of the method from time to time.

33 Is it recommendable for the analysis of multi
component mixtures to add the spectra of the
corresponding pure substances to the calibration set
because in these spectra the peaks are not over-
lapped by anything?
By no means! Because there is a risk of adding a sample to
the data set which is far beyond the relevant calibration range
but which is now to be analyzed as well. It is obvious that this
SURFHGXUHFDQPDNHWKHFDOLEUDWLRQPRUHOLNHO\ZRUVH7KLV
is a general fact. It also does not make sense if you add the
spectra of the corresponding binary mixture to a ternary
mixture. Also in this case, there is the risk that these binary
mixtures do not exist in reality and adding them to the model
ZLOOLPSDLULWXQQHFHVVDULO\7KHUXOHRIWKXPEIRUVHWWLQJXS
DXVHIXOFDOLEUDWLRQLV7KHFDOLEUDWLRQVDPSOHVVKRXOGFRYHU
homogeneously the complete natural variance of the
component values. Not more, but also not less.
34 7KHPDLQWHQDQFHRIFKHPRPHWULFPHWKRGLVrec-
ommended, i.e. the quality should be checked from
time to time. But how am I to proceed if I detect a
JUDGXDOZRUVHQLQJRIWKHPHWKRGRYHUDORQJHU
time period?
Add further samples with new properties to an existing
PHWKRGDQGUHYDOLGDWHWKHPHWKRG7KHUHYDOLGDWLRQPD\
lead to a new parameter set (spectral range, data preprocess-
LQJPHWKRGVDQGUDQNZKLFKWDNHVWKHQHZV\VWHPSURSHU
ties better into consideration.

NIR TABLES OF FUNCTIONAL GROUPS
CHAPTER4
14
Table A.1 Groups containing only C- or H- Atoms

Overtone, Combination Frequency Range Remarks
[cm-1]
1) -CH3 methyl
combination 4,400 4,380 CH-stretching + CH-bending
7,380 7,330 2x CH-stretching + CH-bending
9,900 9,800 2x CH-stretching + 3x CH-bending
1st overtone 5,850 5,780 1st overtone of anti-sym. stretching
5,650 5,600 1st overtone of sym. stretching
2nd overtone 8,700 8,580 2nd overtone of anti-sym. stretching
8,400 8,330 2nd overtone of sym. stretching
3rd overtone 11,490 11,300 3rd overtone of anti-sym. stretching
11,110 10,990 3rd overtone of sym. stretching
2) -CH2 methylene
combination 4,310 4,290 CH-stretching + CH-bending
4,340 4,320
7,190 7,140
9,520 9,430
1st overtone 5,760 5,710 1st overtone of anti-sym. stretching
5,620 5,570 1st overtone of sym. stretching
2nd overtone 8,550 8,470 2nd overtone of anti-sym. stretching
8,330 8,260 2nd overtone of sym. stretching
3rd overtone 11,300 11,170 3rd overtone of anti-sym. stretching
10,990 10,870 3rd overtone of sym. stretching
3) -CH aliphatic
1st overtone 5,700 5,630
2nd overtone 8,440 8,370
3rd overtone 11,110 10,990
4) -C=C- alkenes
combination 4,270 4,260 CH2-stretching + =CH2-bending
4,580 4,560 CH2-stretching + C=C-stretching
4,680 4,660 =CH-stretching + C=C-stretching
6,080 6,020 vinyl group
rd
3 overtone 11,630 11,490

Table A.1 Groups containing only C- or H- Atoms
[cm-1]
5) -C C- alkynes
3rd overtone 12,820 12,660
6) -CH aromatic
combination 6,940 6,900 2x CH-stretching + CH-bending
9,350 9,220 2x CH-stretching + 2x C-C-stretching
3rd overtone 11,760 11,630
Table A.2 Groups containing O-Atoms

[cm-1]
7) H2O
combination 5,180 5,150 2+VWUHWFKLQJ2+EHQGLQJ
7,270 7,220 2+DQWLV\PVWUHWFK2+V\PVWUHWFK
2nd overtone 12,260 10,150
3rd overtone 13,510 13,330
8) free -OH alcohol
combination 4,850 4,780 2+VWUHWFKLQJ2+EHQGLQJ
2nd overtone 4,220 4,180 2ndRYHUWRQHRI2+EHQGLQJ
10,640 10,470 2ndRYHUWRQHRI2+VWUHWFKLQJ
3rd overtone 13,700 13,420
9) bound -OH alcohol
1st overtone 6,970 6,760 intermolecular hydrogen bond
6,670 6,270 intermolecular hydrogen bond
2nd overtone 10,200 10,100 intermolecular hydrogen bond
10) -COOH, -COOR
2nd overtone 5,290 5,210 [& 2VWUHWFKLQJFDUER[DFLGV
5,180 5,130 [& 2VWUHWFKLQJHVWHUV
11) -C=O ketones
12) -CHO aldehydes
combination 4,570 4,520 &+VWUHWFKLQJ& 2VWUHWFKLQJ
13) -COC- epoxides

1st overtone 6,100 6,060 1st overtone of CH-stretching

Table A.3 Groups containing N-Atoms
[cm-1]
14) -NH2 primary amines
combination 5,080 4,980 NH-stretching + NH-bending
1st overtone 6,580 6,490 1st overtone of NH2 sym. stretching
6,670 6,580 1st overtone of NH2 anti-sym. stretching
6,900 6,780 ArNH2
2nd overtone 9,800 9,620 2nd overtone of NH2 sym. stretching
10,000 9,800 2nd overtone of NH2 anti-sym. stretching
10,200 9,800 ArNH2
3rd overtone 12,500 12,200 3rd overtone of NH2 anti-sym. stretching
12,820 12,500 ArNH2
12,990 12,660 3rd overtone of NH2 sym. stretching
15) >NH secondary amine
16) -CONH2 prim. amides
combination 4,670 4,610 2x amide + amide
4,760 4,690 NH-stretching + amide
6,710 6,620 1st overtone of NH2 sym. stretching
2nd overtone 6,940 6,850 1st overtone of NH2 anti-sym. stretching
4,950 4,900 2nd overtone of amide
10,000 9,800 2nd overtone of NH2 sym. stretching
10,310 10,100 2nd overtone of NH2 anti-sym. stretching
17) -CONH- second. amides
combination 4,650 4,610 2x amide + amide
2nd overtone 5,240 5,180 2nd overtone of amide
Table A.4 Groups containing S- or P-Atoms

[cm-1]
18) -SH thiols
19) P-OH
1st overtone 5,260 5,240 1stRYHUWRQHRI2+VWUHWFKLQJ
20) -PH

CHAPTER5

REFERENCES
CHAPTER6
1 '0+DDODQG(97KRPDV$QDO&KHP

2 G. Schwedt, Taschenatlas der Analytik 7KLHPH 9HUODJ
6WXWWJDUW(GLWLRQ6I
3 3*HODGL%5.RZDOVNL$QDO&KLP$FWD
4 .5%HHEH%.RZDOVNL$QDO&KHP$
5 +0DUWHQV71DHVMultivariate Calibration, J. Wiley &
6RQV1HZ<RUN&KDSWHU
6 6'%URZQ$SSO6SHFWURVF1R$
7 3*HODGL-&KHPRPHWULFV
8 J.P. Conzen, J. Bürck, H.J. Ache, Fresenius J. Anal. Chem.

9 -3 &RQ]HQ 7 6WDGHOPDQQ + :HLOHU 7KLHUU\ 'UR]
*HRUJHW/DERU3UD[LV
10 -3&RQ]HQ*,7/DERU)DFK]HLWVFKULIW
11 E. Bouveresse, C. Hartmann, D.L. Massard, I.R. Last, K.A.
3UHEEOH$QDO&KHP1R
12 02WWR&KHPRPHWULHStatistik und Computereinsatz in
der Analytik9&+9HUODJVJHVHOOVFKDIW:HLQKHLP
13 H. Martens, M. Martens, Multivariate Analysis of Quality,
-:LOH\ 6RQV1HZ<RUN
14 +: 6LHVOHU < 2]DNL 6 .DZDWD +0 +HLVH 1HDU
,QIUDUHG 6SHFWURVFRS\ :LOH\9&+ 9HUODJV *PE+
:HLQKHLP

CHAPTER7

NOTE OF THANKS
CHAPTER8
,ZLVKWRWKDQNP\FROOHDJXHVIURP%UXNHU2SWLN*PE+0U
7LP6WDGHOPDQQIRUDQXPEHURISUDFWLFDOKLQWV'U0DUNXV
Arnold for the expert review of the formulas in Chapter 7,
and Ms. Dagmar Behmer for proof-reading and layouting the
manuscript.

INDEX
Numerics
2-component system 30
A
analysis 21, 62
average 36
B
EFRHI¿FLHQW
ELDV
C
calibration 21
FDOLEUDWLRQIXQFWLRQ
FDOLEUDWLRQVHUYLFH
FDOLEUDWLRQWUDQVIHU
CO2 28
FRHI¿FLHQWRIGHWHUPLQDWLRQ
FROOLQHDUGDWDVHWV
FRPELQHGVSHFWUD
FRPSRQHQWYDOXHV
FRQFHQWUDWLRQGDWD
FRUHOORJUDP
FURVVYDOLGDWLRQ
D
GDWDSUHSURFHVVLQJ
GH¿QHGYDULDQFH
GHIRUPDWLRQ
GHSHQGHQWVDPSOHV
GLIIHU
GLIIXVHUHÀHFWLRQ
GLVWXUEDQFHV
E
HLJHQYHFWRUV
HQYLURPHQWDOLQÀXHQFHV
HUURURIDQDO\VLV
HUURURISUHGLFWLRQ
H[WHUQDOYDOLGDWLRQ
F
factorization 61
IDFWRUV
IHDVLELOLW\VWXG\
force constant 26
)3URE
)7VSHFWURPHWHUV
IXOOVSHFWUXPPHWKRG
)9DOXH
H
H2O 28
K\GURJHQEULGJHV
I
LGHDOFRQGLWLRQV
LQWHUQDOYDOLGDWLRQ
L
OHDYHRXWVSHFWUD
OHYHUDJH
OLQHDUEDVHOLQHVKLIW
ORDGLQJVYHFWRU
M
0DKDODQRELVGLVWDQFH
PDLQWHQDQFHRIWKHPHWKRG
PDVWHU
PDWHULDOGHQVLW\
0'/
PHDVXUHPHQWSDUDPHWHUV
PLQPD[QRUPDOL]DWLRQ
multi-component systems 30
PXOWLSOLFDWLYHVFDWWHUFRUUHFWLRQ
N
QDWXUDOFRPSRXQGV
QHDULQIUDUHGVSHFWURVFRS\
QXPEHURIIDFWRUV
O
RIIVHWFRUUHFWLRQ
optimization 20
RXWOLHUGHWHUPLQDWLRQ
RXWOLHUV
RYHU¿WWLQJ
P
SDUWLFOHVL]H
SK\VLFDOSDUDPHWHUV
3/65HJUHVVLRQ
35(66
SULPDU\FRPSRQHQWV
3:6
R
R2
UDQN
UHIHUHQFHYDOXHV
UHJUHVVLRQFRHI¿FLHQW
UHSHDWHGPHDVXUHPHQWV
UHVLGXDOPDWUL[
UHVLGXXP
506(&9
RMSEE 80
RMSELC 80
RMSEP 16, 20, 80
RPD 80
UREXVWQHVV
URRWPHDQVTXDUHHUURURIFURVVYDOLGDWLRQ
URRWPHDQVTXDUHHUURURISUHGLFWLRQ
S
VDPSOHVHW
VFRUHVYHFWRU
VHFRQGGHULYDWLYH
6(&9
SEE 81
SEP 81
VHUYLFHFRQWUDFW
VHWWLQJXSDPRGHO
VORSH
slope correction 80
VSHFWUDOGDWD
VSHFWUDOGDWDEORFN
VSHFWUDOQRLVH
spectral ranges 38
VSHFWUDOUHVLGXXP
SSE 82
VXEWUDFWLRQRIDFRQVWDQWRIIVHW
T
temperature 26
WHPSHUDWXUHÀXFWXDWLRQV
WHVWVHWYDOLGDWLRQ
U
XQGHU¿WWLQJ
V
YDOLGDWLRQ
YDOLGDWLRQUHSRUW
YDOLGLW\
YHFWRUQRUPDOL]DWLRQ

NIR - Multivariate Calibration - 3rd Edition 2014 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

NIR - Multivariate Calibration - 3rd Edition 2014 PDF

Uploaded by

Copyright:

Available Formats

Jörg-Peter Conzen

Order No.: I 26009

A Practical Guide for the

© 2014 Bruker Optik GmbH

Table of Symbols ..........................................................................................................v

The following symbols are used in this tutorial:

Ai absorbance value of sample i

Differi difference between reference and analysis value of sample i

The appearance of modern analytical chemistry has changed

This and the following chapter will explain the theoretical

To describe multivariate calibration techniques, the practical

X-Data + Y-Data Calibration Function b

X-Data + Calibration Function b Y-Data

Figure 2.1 Schematic procedure of the quantitative determination of

In case of a quantitative evaluation of (near-)infrared or

)LJXUH  DQG )LJXUH  VKRZ WKH SURFHGXUH GHVFULEHG

Figure 2.3 Analysis of absorbance spectra

A comparative review of these chemometric methods used in

The selection of the optimum number of factors is of central

The rank R represents the number of factors and T denotes

The previous chapter described how to set up a calibration

YAnalysis = XAnalysisÂE 

with XAnalysis: spectral data of analyzed sample.

Calculation of the calibration function b directly yields the

Figure 3.1 Schematic of a cross validation

The steps of a cross validation (internal validation):

2 Set up a model with the remaining samples.

3 Analyze the removed test sample; calculate the error of

5 Repeat step 4 until all M samples of the calibration data set

A second method for assessing the quality of a method is the

Figure 3.2 Schematic diagram of a test set validation

2 Evaluate the model, using a separate data set of test samples

To summarize: External validation can be distinguished from

PRACTICAL PROCEDURE FOR

The theoretical procedure of setting up a PLS method was

1. Step: Entering Spectral Data and

2. Step: Data Preprocessing

FOR A PLS CALIBRATION;

In principle, there is an unlimited number of possibilities for

A. Selecting the Concentration Range

The calibrated concentration range should be slightly larger

Figure 5.1 Selection of the concentration range: The samples of a

The individual component values should span the entire

Representative calibration samples are the perquisite of sta-

D. The Problem of “Collinearity”

In principle, there are two methods for setting up calibration

Figure 5.4 Top row:&ROOLQHDUGDWDVHWVH[HPSOL¿HGZLWKWKUHH

Collinear calibration data sets are only permitted if it is

Table 5.1 Examples for Collinear Data Sets

2 2 7,8 0,8 0,01666667 15,6 96 46,6

3 5 19,5 2 0,04166667 16,5 90 41,5

4 7 27,3 2,8 0,05833333 17,1 86 38,1

5 8 31,2 3,2 0,06666667 17,4 84 36,4

6 9 35,1 3,6 0,075 17,7 82 34,7

... ... ... ... ... ... ... ...

E. Performing a Background Measurement

When selecting the reference substances, care should be

7KHLQÀXHQFHRIDWPRVSKHULFJDVHV HVS&22 and H2O) on

In contrast, measurements in NIR are less sensitive and are

The use of inert, low-band or band-free reference materials is

The same samples as in Figure 5.2 were measured, where the

[Substance Spectrum] - [Spectrum of Matrix] =

or, in simple words:

)LJXUH DQG )LJXUH VKRZ WKH SURFHGXUH GHVFULEHG

YAnalysis = XAnalysisÂE

7KHLQÀXHQFHRIDWPRVSKHULFJDVHVHVS&22 and H2O) on

%LDV&RUUHFWLRQ To use calibrations on different

'LIIHU The “Differ” value is the difference between the

2IIVHWDQG6ORSH&RUUHFWLRQ The offset correction sets

527KHFRHI¿FLHQWRIGHWHUPLQDWLRQR²) gives the

5HJUHVVLRQ&HI¿FLHQW see Calibration function.