PROCESS FAULT DIAGNOSIS USING ONE-CLASS SUPPORT VECTOR MACHINES

G.T. Jemwa and C. Aldrich*

Department of Process Engineering,

University of Stellenbosch, Stellenbosch, South Africa
Fax +27(21)808 2059, *E-mail: ca1@sun.ac.za

Abstract: The process fault diagnosis problem is usually considered in classification

framework. Although used widely in diagnostic applications, artificial neural networks
and other nonlinear classifiers perform poorly under nonideal conditions encountered in
practice, owing to the arbitrary placement of decision boundaries in empty regions of the
input space and unbounded normal class region. This is particularly problematic where
few and noisy data are available. In this paper, the use of one-class support vector
machines for the diagnosis of process operations is proposed and their performance under
practical conditions assessed. One-class classifiers are shown to be superior to and more
robust than competing approaches previously proposed for diagnostic applications.
Copyright © 2007 IFAC

Keywords: Fault diagnosis; Process Control, Neural Networks; Pattern Recognition;

Machine Learning

1. INTRODUCTION commonly referred to as kernel methods or

Support vector machines (SVMs) are state-of-the-art
learning algorithms inspired by insights from
statistical learning theory. Instead of optimizing a
model by only minimizing the training error, SVMs
incorporate the idea of large margin maximization
that gives probability guarantees of how well the
trained classifier will perform when presented with
data not used in training. SVMs also provide for a
method for controlling capacity or flexibility of the
resulting model by using kernel functions. Although a
recent innovation, they have already been applied
with success in many areas, such as image analysis,
signal processing, text mining, and bioinformatics,
among other (Shawe-Taylor and Cristianini, 2004).
Originally formulated in a pattern recognition
context, the ideas underlying SVMs have since been
extended to other learning problems such as
regression, density estimations and transductive
learning. The resulting group of learning tools is
algorithms (Schölkopf and Smola, 2002).
In this paper, the problem of fault diagnosis of
process operations is investigated. Compared to fault
detection and identification, diagnosis is a more
important yet difficult problem since knowledge of all
faults that can possibly occur on a plant is impossible
to obtain. However, given sufficient historical data of
a certain class of faults, e.g. sensor faults, as well as
fault-free data, it is possible, at least in principle, to
use pattern recognition methods such as multilayer
perceptrons to learn to predict what type of fault may
have occurred when a fault is detected.
Unfortunately, it has been observed that the
performance of multilayer perceptrons degrades when
deployed in nonideal conditions as often encountered
on industrial plants (Kramer and Leonard, 1990). This
paper proposes the use of one-class SVM classifiers
as an alternative. Because of the large margin bias,
SVMs can be expected to yield improved
performance rates compared to other methods.
The paper is structured as follows. The next section
discusses the problem of process fault diagnosis from
a classification perspective. A brief review of the use
artificial neural networks in fault diagnosis is
presented. The one-class classification framework is
then considered, including a basic formulation of the
relevant algorithm. An analysis of the performance of
one-class SVM in diagnostic situations is then
presented using a simulated example representative of
many chemical reactor models.
2. THE FAULT DIAGNOSIS PROBLEM
To identify the root cause(s) of a detected fault
condition in a process, knowledge of all possible
faults associated with the process must be available
(Stephanopoulos and Han, 1996). Clearly, this is an
ill-posed decision logic problem, since it is
impossible to have a priori information of everything
that can go wrong in a process. Although in some
cases certain fault conditions, particularly those
associated with sensors, can be simulated by use of a
process model, the design of diagnostic system must
ideally include the possibility of novel conditions that
have not yet been realized.
Generally, given known fault conditions associated
with a process, including fault-free or normal
operating conditions, the task of fault diagnosis
involves fitting a model relating inputs and outputs in
a training data set. The outputs typically take the form
of an indicator matrix, such that for a process with N
known faults, the output vectors lie in an N+1-
dimensional space, with each column representing a
specific fault. The extra column in the output vectors
is used to accommodate the normal operating
condition, say f0. Each column is assigned a unit entry
if the input vector corresponds to that fault; otherwise
it is assigned a zero entry. For a problem with N
classes, the decision function generator builds a
model for each j; for j = 0,1,2…,N. When presented
with a new instance x (typically, a symptom or
residual vector from a preceding fault detection and
identification phase), the vector is fed to each model,
which then computes an output fj(x). These outputs
are evaluated in the fault decision logic, that is
f * ( x) = G ( f 0 ( x ), f1 ( x ), , f N (x) )
where G is the decision logic evaluation function and
f* the most probable fault condition, as indicated in
Fig. 1.
Since fault diagnosis is essentially a pattern
recognition problem, pattern recognition algorithms
are typically used to solve the problem. These include
distance-based classifiers, multilayer perceptrons
(MLP), radial basis functions, support vector
machines, and so on. Of these, multilayer perceptron
(MLP) networks are among the most widely used
models in diagnostic applications (Sorsa and Koivo,
1991; Venkatasubramanian and Chan, 1989).
REFERENCE INPUT x(t)
NOMINAL 1
ST
FAULT 2
ND
FAULT N TH FAULT
MODEL MODEL MODEL MODEL
DECISION FUNCTION
GENERATOR
FAULT DECISION LOGIC:
f*=G(f0(x),f1(x),…,fN(x))
f*(x)
FAULT ANALYSIS AND
INTERPRETATION
Fig. 1. A schematic representation of fault diagnosis
as multiclass pattern recognition problem
3. LIMITATIONS OF DIAGNOSIS WITH
NEURAL NETWORKS
In a critical analysis of fault diagnosis using MLPs,
Kramer and Leonard (1990) investigated performance
of neural classifiers under non-ideal conditions often
encountered in practice. The objective of their study
was to identify process fault conditions not easily
detected by MLPs. It was observed that the tendency
of MLPs to extrapolate when a new sample falls
outside the training data range led to some serious
problems. Five such situations were identified in
which extrapolation from the training data was
required viz.:
(i) small training sets,
(ii) changes in the parent distributions of the
classes that occur after training,
(iii) corrupted data,
(iv) appearance of novel fault classes, and
(v) training of the network with synthetic
data.
Through a series of investigations, it was concluded
that distance-based classifiers should be instead of
MLPs in fault diagnosis problems, because of their
(1) greater reliability when dealing with non-
representative training data.
4. ONE-CLASS CLASSIFICATION
The goal of one-class classification is to estimate the
regions of data space in which the mass of the data is
concentrated, sometimes referred to as the support of
the distribution. Estimation of the support of the
distribution is a considerably more tractable problem
than density estimation, particularly for small-sized
samples. Mathematically, the problem can be
formulated as finding a function f that is positive in a
“small” region capturing most of the data points and
negative elsewhere (Schölkopf et al., 2001; Rätsch, et
al., 2002). In this sense, the function estimation
problem can be considered as a quantile (or minimum
volume) estimation problem (Polonik, 1997).
4.1 One-class support vector machines
Inspired by support vector learning theory, a few
algorithms have been proposed for solving one-class
classification problems. These include the one-class
SVM which seeks a hyperplane that separates the
unlabelled data set X from the origin with maximum
margin (Schölkopf et al., 2001), and the SV data
descriptor that finds an optimal enclosing hypersphere
with minimal volume (Tax and Duin, 1999).
The above algorithms are restricted as far as the kind
of ‘abnormality’ that can be detected is concerned.
Prior information of what the abnormal class looks
like can be encoded in the one-class SVM. Instead of
finding an f that is maximally separate from the
origin, one considers a large margin hyperplane w
that maximizes the separation of the set X =
{x1,x2,…,xm} from the centroid of another set Z =
{z1,z2,…,zt} (Schölkopf et al., 2000; Hayton et al.,
2000). More formally, the decision function is
obtained by minimizing a weighted sum of a
regularizer and a training error term that depends on
an overall margin ρ and training errors ξi
1 2 1 m
min w +
w, ,ρ 2 νm
1 t
w⋅ x−
subject to t j =1
ξi ≥ 0
with a trade-off parameter. The corresponding
decision function is given by
f ( x ) = sgn w⋅ x−
which takes positive values for most patterns in X,
while the regularizer ||w||2 is still smaller. This is
equivalent to a large margin separation from the
centroid of set Z. This formulation will be refereed to
as the generalized one-class SVM in the subsequent
sections.
The solution to Equation 2 is typically obtained by
first transforming it to an equivalent dual formulation
in terms of Lagrange multipliers αi , that is
1 m
min α iα j ( x ⋅ x) + Q − Q j − Qi
w, ,ρ 2 i, j
subject to {0 ≤ α i ≤
where
Q≡ 1
t2 np
(z n
⋅ z p ) , and
Qi ≡ 1
t n
(x i
⋅ zn ) .
Finally the function value of test point x is evaluated
according to
f ( x ) = sgn { i
αi ( xi ⋅ x ) −
1
t n
(z n }
⋅ x) . (5)
The threshold parameter ρ is computed by use of the
Karush-Kuhn-Tucker (KKT) conditions from
optimization theory. Thus, the algorithm outputs large
values for points similar to the image of X and small
values for generic points from Z.
It turns out that only relatively few of the training
patterns with non-zero αi’s define the decision
function, f. These are termed support vectors. Also,
the trade-off parameter has a special meaning; it is a
lower bound on the number of support vectors, an
upper bound on the number of outliers, and
asymptotically equals both the fraction of support
vectors and fraction of outliers (Schölkopf et al.,
2000).
The algorithm as formulated is restricted to linear
decision functions. Noting the data appear as dot
products in both the objective and evaluation
functions (Equations 4 and 5), the “kernel trick” can
be used to improve the flexibility or capacity of the
model (Schölkopf and Smola, 2002). A kernel k is a
ξi − ρ
i =1
mathematical function that implicitly computes the
dot product of images of input space data induced by
(2) some transformation ϕ, that is
zj ≥ ρ − ξi ,
k ( x, z ) = ϕ ( x), ϕ ( z ) (6)
where the angle brackets denote dot product in the
transformed or feature space. An example of
commonly used kernel function is the Gaussian radial
basis function:
1 t
zj −ρ (3)
t 2
j =1
x−z
k ( x, z ) = exp
2σ
2
(7)
≡ ϕ ( x), ϕ ( z )
with width σ a kernel hyperparameter that must be
specified by the user. Cross-validation is typically
used to determine the optimal kernel width.
The advantage of using kernels is that it is not
necessary to know the exact form of the mapping ϕ as
long as it satisfies the so-called Mercer conditions. By
using kernels one avoids computational difficulties
associated with working in a possibly high
dimensional feature space while retaining flexibility.
Hence, by replacing all occurrences of dot products in
(4)
the preceding formulations with kernels, statistically
νm
1
, i
αi = 1 simple yet rich class of models are possible.
4.2 Diagnosis Using One-class SVMs: An Empirical
Analysis
To investigate the use of one-class SVMs for
diagnosis, we consider the following generalized fault
diagnosis problem previously considered in Kramer
and Leonard (1990):
 x = x0 + α p + v (8)
where x is a two-dimensional vector of observed
variables sampled from a static process with a
nominal steady state x0 = (0,0), p is the fault
parameter vector of size np, α = [1 1; 1 -1] is a linear
operator incorporating the directional effect of one of
the fault parameters on the measurements, and
v∼N(0,0.015) is a vector of Gaussian distributed
measurement disturbances. Fault conditions are
classified into Ck by defining inequalities on the
failure parameters according to
unavoidable. Very few (in some case none)
extrapolation errors were observed. This is a
significant advantage over MLPs that, because of the
arbitrary placement of the decision boundary in
empty regions as well as the influence of extreme
points, give models with extrapolation errors.
Hence, it can be concluded that the generalized one-
class SVM is a better alternative to multilayer
perceptrons, particularly for fault conditions
characterized by small sets of noisy samples
commonly found in practice.

Norma l (C0 ) :| p1 | < 0.05, | p2 |< 0.05

(a)
Fault 1 (C1 ) :| p1 | > 0.05 1
R2 R1
Fault 2 (C2 ) :| p2 | > 0.05
R0
x2 0
Fault 1 causes both process variables x1 and x2 to
deviate in the same direction, while Fault 2 results in R1 R2
deviations in opposite directions. The normal class
(C0) occupies the intersection region, as indicated in -1
-1 0 1
Fig. 2. x1
(b)
1
1
R1

R0

Normal
x2 0
x 0 Fault 1
2
Fault 2 R2

-1
-1 0 1
x1
-1 (c)
-1 0 1
x 1
1 R1
R2
Fig. 2. Scatter plot of the two-dimensional fault
diagnosis problem 0
x2

A generalized one-class SVM model was constructed,
using Gaussian kernels with the following widths: σ =
[0.25 0.5 1 2 4 8]. For comparative purposes,
corresponding models for the standard one-class
SVM (no abnormal class information) and a standard
binary SVM classifier were also fit using the same
kernel and kernel parameters. Fig. 3 shows the
resulting decision functions for the three classifiers
for a kernel width = 4. The generalized one-class
SVM identified the different process operating
regions satisfactorily, as shown in Fig. 3(a). In
contrast, the other two classifiers could not separate
the different regions very well.
Fig. 4(a) summarizes the distribution of errors for 100
independent test sets, each with a sample size of
1000. The scaling of the figure corresponds with that
of Fig. 2. As indicated, the classification error did not
vary too widely over the range of kernel widths
considered (0.25 to 8). Fig. 4(b) shows a scatter plot
of a sample of misclassified data points from an
independent test set observed in the simulations. The
misclassified data are distributed in the overlapping
regions between the three classes and are generally
R0
-1
-1 0 1
x1
Fig. 3 Decision regions generated using (a) the
generalized one-class SVM (b) the standard one-
class SVM and (c) a binary SVM classifier. The
fault model representing each region is indicated
by R0, R1, and R2 for the Normal, Fault 1, and
Fault 2 conditions respectively.
4.3 Robustness Analysis
To investigate the effect of small changes in class
distributions after the classifier has been trained,
small and large shifts for different types of possible
fault modes were simulated as summarized in Table
1. These changes are similar to those investigated by
Kramer and Leonard (1990) and were chosen to allow
for a comparison with the 1-nearest neighbour
classifier that had the best performance in their study.
(a)
0.06
classification error
0.05
x2 0
0.04
0.03
0.25 0.5 1 2 4 8 -1
kernel width σ
Fig. 4. (a) Variation of misclassification error with
kernel width (b) Typical distribution of
misclassified patterns. The errors are associated
with class overlap and generally are unavoidable.
Table 1. Simulated changes for Robustness Analysis
of Diagnosis with one-class SVMs
Error
Small Shift
Type
Sensor
±0.025
Bias
Sensor
2×N(0,0.015)
Noise
Fault
±7.5°
Direction
Process
x0=(±0.025,±0.025)
Drift
5. DISCUSSION
An additive change in the sensor bias resulted in
moderate increases in the classification error, with
small and large shifts increasing the error from a
nominal 4% to 6% and 4% to 8% respectively.
Doubling and tripling the sensor noise uncertainty
increased the error rate to 7% and 12% respectively.
Small quantitative shifts in the fault directions can
arise from process changes. To simulate these
changes, small rotational changes in the fault
directions were induced to the base case scenario
(angle = ±45°). A small rotation increased the
nominal error rate by 1%, while a large rotation
resulted in a 5% change. Finally, changes to the
nominal operating point incurred errors of 7% and 9%
respectively for the different shifts.
With the exception of the rotational changes, the one-
class SVM gave marginal to significant improvement
in performance compared to the one-nearest
neighbour distance-based classifier reported in
Kramer and Leonard (1990). Fig 5 summarizes the
performance of the one-class SVM against the
competitive distance-based one-nearest neighbour (1-
NN) classifier. The one-class SVM results are for a
kernel of width 4. Similar results were obtained for
kernels with other widths for the same system.
0.25
Nominal
Small Shift
(b) 0.2 Large Shift
1
One-class SVM 1-NN Classifier
0.5 0.15
Mean Error
0.1
-0.5
-1 0.05
-0.5 0
x1
DRIFT
DRIFT
NOISE
NOISE
DIREC
DIREC
BIAS
BIAS
Fig. 5. Results from robustness analysis of the
performance of one-class SVM under changes in
the underlying fault class distribution. Also shown
for comparison are results for a distance based
classifier (1-NN) as investigated in Kramer and
Leonard (1990).
Large Shift 6. APPLICATION EXAMPLE:
PGM FLOTATION PLANT
±0.05
As an illustration, the proposed methodology is
applied textural information of froth structures from
3×N(0,0.015)
image data obtained from a platinum group metals
(PGM) flotation plant in South Africa (Aldrich et al.
±15°
2004). The original data are composed of the
following features characterizing the froth: (a) small
x0=(±0.05,±0.05)
number emphasis (SNE), (b) ENTROPY (c) image
local variations metric or ENTROPY (d) local
homogeneity metric (LOCHOM) and (e) gray level
dependencies (GLD). For visualization purposes two-
dimensional projections of the data that explain 94%
of the total variance (or principal components) are
considered.
The data were grouped into three classes (‘ ’,‘+’,‘ ’)
related to different operating conditions, with the ‘ ’
group representing the desirable conditions (see
Figures 6 and 7). The ‘ ’ class is associated with
ellipsoidal froth structures containing relatively
higher gangue minerals, whereas the other classes
with less gangue had spherical bubbles as well as a
darker colour. However, the stability of the ‘+’ class
was poor compared to the normal or ‘ ’ class. The
fault diagnosis objective in this case would be to
identify the specific abnormal condition when a fault
is detected by, for example, statistical process
monitoring techniques.
The decision functions separating the normal class
from abnormal conditions obtained using multilayer
perceptron networks and generalized one-class SVMs
are shown in Figures 6 and 7 below respectively.
(Although not shown, a 1-NN classifier gave a more
or less similar decision boundary as MLP networks.)
It can be seen that the presence of empty regions and
uneven distribution of the data in the measurement
space leads to spurious placement of the decision
function in the case of an MLP-based approach. Also,
the absence of data of the negative classes results in a
decision boundary extending outside the training data
regions, as indicated in the top left of Figure 6.
 insensitive to the influence of data lying in extreme
5
regions, and did not exhibit serious extrapolation
C
1,2
C
1,2 errors. Moreover, the one-class approach gave better
C performance than a one-nearest neighbour classifier,
0
previously proposed as a preferable alternative to
artificial neural networks. Not only does the use of
X2
0 C
1,2
kernels allow for a larger class of functions (including
distance-based classifiers) but also induces
C
0 C
1,2 appropriate regularization for the learning algorithm.
C
0
The study will be extended to larger systems in a
C
1,2
C
separate paper.
0
-5
-5 0 5 REFERENCES
X1
Aldrich, C., N.J. Le Roux and S. Gardner (2004).
Fig. 6. Fault diagnosis decision boundaries obtained Monitoring of metallurgical process plants by use
by using a single hidden layer MLP with four of biplots. AIChE Journal, 50, 2167-2186.
sigmoidal transfer functions. Regions associated Hayton, P., B. Schölkopf, L. Tarassenko and P.
with normal and abnormal operating conditions Anuzis (2000). Support vector novelty detection
are labelled C0 and C1,2 respectively. applied to jet engine vibration spectra. In:
Advances in Neural Information Processing
5
Systems, 13, 946-952.
Kramer, M.A. and J.A. Leonard (1990). Diagnosis
C
1,2 C
1,2 using backpropagation neural networks – Analyis
and critisicm, Computers and Chem Engng, 14,
1323-1338.
C
0 Polonik, W. (1997). Minimum volume sets and
X2
0 generalized quantile processes. Stochastic
C
1,2
Processes and their Applications, 69, 1-24.
C
0 Rätsch, G., S. Mika, B. Schölkopf, and K.-R. Müller
C
1,2 (2002). Constructing boosting algorithms from
SVMs: An application to one-class classification.
C
1,2 IEEE Transactions on Pattern Analysis and
-5
-5 0 5 Machine Intelligence, 24, 1184-1199.
X1 Schölkopf, B, J.C. Platt, and A.J. Smola (2000).
Kernel method for percentile feature extraction.
Fig. 7. Fault diagnosis decision boundaries obtained Technical Report, Microsoft Research 2000-22.
by using a generalized one-class SVM with a Schölkopf, B., J.C. Platt, J. Shawe-Taylor, A.J.
Gaussian kernel of unit width and nu-parameter Smola, and R.C. Williamson (2001). Estimating
value of 0.2. C0 and C1,2 denote normal and the support of a high-dimensional distribution.
abnormal operating conditions respectively. Neural Computation, 13, 1443-1471.
Schölkopf B. and A.J. Smola (2002). Learning with
Using one-class SVMs, improved decision boundaries Kernels. MIT Press, Cambridge, MA.
around a specific class are obtained, as shown for the Shawe-Taylor, J. and N. Cristianini (2002). Kernel
normal class in Figure 7. An important property of the Pattern Analysis. Cambridge University Press,
generalized one-class SVM is the exclusion of Cambridge, UK.
spurious decision regions resulting from lack of Sorsa, T. and H.N. Koivo (1991). Neural networks in
sufficient data or empty regions as in the case for process fault diagnosis. IEEE Transactions on
MLPs and k-NN classifiers. Furthermore, tighter Systems, Man, and Cybernetics, 21, 815-825.
boundaries are also useful in detecting novel Stephanopoulos, G. and C. Han (1996). Intelligent
operating regimes not previously encountered, systems in process engineering: a review.
limiting the risk of improper diagnostic decisions. Computers and Chemical Engineering, 20:743-
791.
7. CONCLUSIONS Tax, D.M.J. and Duin, R.P.W. (1999). Support vector
A framework for using one-class SVMs in fault domain description. Pattern Recognition Letters,
diagnosis was discussed and critically analysed using 20, 11-1-1199
a simple two-dimensional system. In spite of its Venkatasubramanian, V. and K Chan (1989). A
simplicity, the system is representative of a number of neural network methodology for process fault
industrial reactor systems. Compared to previously diagnosis, AIChE Journal, 35, 1993-2002.
proposed nonlinear methods of fault diagnosis using
artificial neural networks, the one-class SVM
approach was shown to be robust to process changes,