IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 32, NO.

9, SEPTEMBER 2010 1553

Coupled Prediction Classification

for Robust Visual Tracking
Ioannis Patras, Member, IEEE, and Edwin R. Hancock

Abstract—This paper addresses the problem of robust template tracking in image sequences. Our work falls within the discriminative
framework in which the observations at each frame yield direct probabilistic predictions of the state of the target. Our primary
contribution is that we explicitly address the problem that the prediction accuracy for different observations varies, and in some cases,
can be very low. To this end, we couple the predictor to a probabilistic classifier which, when trained, can determine the probability that
a new observation can accurately predict the state of the target (that is, determine the “relevance” or “reliability” of the observation in
question). In the particle filtering framework, we derive a recursive scheme for maintaining an approximation of the posterior probability
of the state in which multiple observations can be used and their predictions moderated by their corresponding relevance. In this way,
the predictions of the “relevant” observations are emphasized, while the predictions of the “irrelevant” observations are suppressed.
We apply the algorithm to the problem of 2D template tracking and demonstrate that the proposed scheme outperforms classical
methods for discriminative tracking both in the case of motions which are large in magnitude and also for partial occlusions.

Index Terms—Regression, tracking, state estimation, relevance determination, probabilistic tracking.

1 INTRODUCTION
V ISION-BASED tracking is one of the fundamental and
most challenging low-level problems in computer
vision. Formally, it is defined as the problem of estimating
the state x (e.g., position, scale, rotation, or 3D pose) of a
target, given a set of noisy image observations Y ¼
f. . . ; y ; yg up to the current frame.1 Usually, an estimate
of the state at each frame is the location of a minimum of a
cost function, or, in the probabilistic framework, the
location of a maximum of the posterior pðx j Y Þ. Alterna-
tively, a representation of the posterior pðx j Y Þ can be
maintained for each frame of the image sequence.
1.1 Literature Review
The large number of methods that have been proposed in
the last decades for maintaining a representation or for
finding a maximum of the posterior pðx j Y Þ fall into two
main categories. In the first category belong the generative
methods (e.g., [11], [9]). Such methods require the inver-
sion of the posterior pðx j Y Þ using the Bayes rule and the
evaluation of the likelihood pðy j xÞ at certain sample states
x. An important drawback is that at least some evaluations
need to be performed at sample points in the state space that
1. In this work, we denote with a an observation or (an instantiation of)
a random variable a in the previous time instant. For example, we denote
with y the observation at the current frame and with y the observation in
the frame before.
. I. Patras is with the School of Electronic Engineering and Computer
Science, Queen Mary University of London, Mile End Road, E1 4NS
London, UK. E-mail: i.patras@elec.qmul.ac.uk.
. E.R. Hancock is with the Department of Computer Science, University of
York, YO10 5DD York, UK. E-mail: erh@cs.york.ac.uk.
Manuscript received 16 May 2008; revised 21 Oct. 2008; accepted 20 Aug.
2009; published online 6 Oct. 2009.
Recommended for acceptance by P. Perez.
For information on obtaining reprints of this article, please send e-mail to:
tpami@computer.org, and reference IEEECS Log Number
TPAMI-2008-05-0291.
Digital Object Identifier no. 10.1109/TPAMI.2009.175.
0162-8828/10/$26.00 ß 2010 IEEE
are close to the “true” state, and therefore, a large number of
solutions need to be examined. Detection-based methods [27]
that exhaustively search all image locations for the presence
of a target fall into this category.
In order to reduce the computational complexity and to
deal with likelihoods with multiple modes, a number of
methods have been proposed for selecting candidate
solutions in the generative tracking framework. A common
choice utilizes a motion model (e.g., [11]) for the proposal
distribution that is for the distribution from which the
candidate solutions will be sampled. Typical choices for the
motion model range from general constant velocity/
acceleration models to higher order models whose para-
meters can be learned from training data [11], [12] or online
[5]. A motion model assists the estimation process so long
as the temporal evolution of the state follows it. However,
the residual between the model-based temporal prediction
and the true target state can be significant in the general
case of irregular motion, novel motion, or a moving camera.
Other methods utilize the fact that in certain domains, the
true target state may lay on a manifold in the state space
and therefore (probabilistic) priors may exist on the state of
the target. This is the case when the state encodes the
position of multiple interacting targets (such as facial points
[21]) or the position of the components of a constrained
articulated structure such as the human body [23]. Alter-
native methods utilize the observations in the current frame
in order to sample from areas where the likelihood is
expected to be higher. This may be done by performing a
two-stage propagation [22], or by using mixtures of learned
detectors and dynamic models (e.g., [18]). In the latter case,
a target detector needs to be applied at every image location
and various scales.
The second category consists of the discriminative (or
prediction-based) methods. In contrast to generative meth-
ods, in discriminative tracking, an observation y delivers a
direct prediction of the hidden state x. This alleviates the
need for a good proposal distribution and multiple
Published by the IEEE Computer Society
1554 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,
evaluations. The predictor can be obtained in two ways.
First, it can be derived analytically from modeling assump-
tions. This is the case with classical motion estimation
schemes that utilize the optical flow equation, such as the
method of Lucas and Kanade [16] and the work of
Simoncelli et al. [24]. Many motion estimation schemes are
posed as an optimization problem which is solved using the
@y
gradient @x of the observation y with respect to the state x,
for example, within a gradient descent approach. While a
point estimation is usually obtained [16], Simoncelli et al.
[24] derive an estimate of the distribution of the posterior
pðx j yÞ by explicitly modeling the distribution of the noise
in the various terms that appear in the optical flow equation.
Second, the predictor may be learned in a supervised way
from training data. In the learning framework, a number of
researchers have recently proposed methods for deformable
motion estimation, 2D template tracking, and 3D human
pose estimation [8], [25], [2]. One of the first learning-based
approaches is that of Cootes et al. [8], which estimates the
parameters that optimally warp an image to an appearance
model in an iterative way. The state update x at each
iteration is estimated from the intensity differences y
between the warped image and the appearance model; note
that the intensity residual y depends on the current state x.
Instead of estimating x using the gradient of the observa-
@y
tion (i.e., @x ), they learn in a supervised way a linear relation
between them, that is, they learn a matrix A such that
x ¼ Ay. For 3D human pose tracking, Sminchisescu et al.
[25], [6] train a Bayesian Mixture of Experts in order to learn
a multimodal posterior pðx j yÞ. Agarwal and Triggs [2] use
Relevance Vector Machines (RVMs) in order to learn
mappings between vectors of image descriptors and the
3D pose of a human body. In [1], they use Nonnegative
Matrix Factorization in order to remove parts of the
observation vector that are due to noise or occlusion. For
2D tracking, Williams et al. [30] use RVMs in order to learn
the posterior of the location of a visual target (e.g., a human
face) given an observation at a certain image location.
Finally, for 2D tracking, Jurie and Dhome [15] learn in a
supervised way a linear relation between the intensity
differences between two templates and the corresponding
motion transformation.
In order to deal with possibly large prediction errors,
most of the previous methods mainly rely on temporal
filtering. Sminchisescu et al. [25] and Agarwal and Triggs
[2] use as observations features that are extracted from a
single-object silhouette. They address prediction errors by
adopting a multiple hypotheses tracking framework that
performs temporal filtering. On the other hand, Williams
et al. [30] couple the regression-based tracking to a
detection-based scheme that is employed to validate that
the target is at the predicted position or pose. In the case of
a validation failure, a full-scale detection phase is initiated.
A Kalman filter is used for temporal filtering and leads to
a reduction of the error by an order of magnitude.
However, none of these methods addresses explicitly the
problem of assessing in advance how well the observation y
can predict the state x. Nor do they use multiple observations
in order to increase robustness. Regression-based methods
are known to be sensitive to observations that do not belong
to the space that is sampled by the training data set.
Therefore, the accuracy of the prediction of the posterior
VOL. 32, NO. 9, SEPTEMBER 2010
Fig. 1. Prediction error as a function of the true horizontal and the true
vertical displacement. The performance deteriorates sharply outside the
training area. Here, a Bayesian Mixture of Experts was trained for
displacements in the interval ½11 . . . 11.
pðx j yÞ can deteriorate sharply for observations y that are
contaminated with noise or come from areas that are
uninformative concerning the state of the visual target
(e.g., occluded areas). In particular, in the case of the 2D
tracking, when the motion magnitude is larger than in the
training set, then the prediction error is likely to be large and
tracking is likely to fail. In Fig. 1, we illustrate this effect by
plotting the prediction error as a function of the true
displacement in an artificial example for a regression-based
predictor. Similar observations are reported in [15] and [30]
for regression-based schemes and also hold true for
gradient-based predictors such as [16].
1.2 Contribution
In this paper, we propose a coupled prediction-classification
scheme for prediction-based 2D visual tracking. The method
allows the use of multiple observations in a way such that
each observation yðrÞ ðr 2 fr1 ; . . . rR gÞ contributes to the
prediction of the state of the target according to its relevance
(or reliability). In our scheme, the corresponding reliability is
determined by a probabilistic classifier. In this way, the
contribution of predictions that originate from reliable
observations is the most significant, while the contribution
of predictions of observations that originate from occluded
areas or of unreliable observations is largely suppressed.
In order to achieve this goal within a discriminative
particle filtering framework [25], we introduce an addi-
tional random variable r that is used to obtain (or, in
general, utilize) multiple observations denoted by yðrÞ,
together with a binary random variable z that is related to
the relevance of the observation. We use a probabilistic
classifier [26] in order to model the conditional probability
pðz ¼ 1 j yðrÞÞ (the probability that the observation yðrÞ is
relevant/reliable) and use a probabilistic predictor [29] in
order to model pðx j yðrÞÞ (the posterior probability of the
state x given an observation yðrÞ). Both the predictor and
the classifier are trained in a supervised way using data that
are generated by applying synthetic transformations to the
target template from the first frame. Alternatively, both may
be trained using data from an annotated database. During
tracking (Fig. 2), multiple observations are generated by
sampling r, and the prediction of each observation (as given
by the probabilistic predictor) is moderated by the
corresponding relevance/reliability weight (as given by
the probabilistic classifier).
Our overall contributions in this paper can be summar-
ized as follows:
PATRAS AND HANCOCK: COUPLED PREDICTION CLASSIFICATION FOR ROBUST VISUAL TRACKING
Fig. 2. Overview of the proposed method.
. We explicitly address the problem of the determina-
tion of the relevance/reliability of an observation to
the state estimation process by learning in a
supervised way the underlying conditional prob-
ability distribution.
. We devise a probabilistic framework that allows
multiple observations yðrÞ to contribute to the
prediction of the state of the target according to
their corresponding relevance/reliability.
. We make explicit the relation between our frame-
work and alternative discriminative and generative
estimation/tracking schemes. More specifically, we
show that under certain modeling assumptions
(simplifications), our estimation scheme is practi-
cally equivalent to classical generative and discrimi-
native estimation schemes.
The remainder of the paper is organized as follows: In
Section 2, we provide an outline of the proposed discrimi-
native tracking framework with data relevance determina-
tion. In Section 2.1, we briefly describe the Bayesian Mixture
of Experts predictor, and in Section 2.2, we present our
method for observation relevance determination. Section 2.3
presents a procedure which, given a predictor, selects an
appropriate classifier, and in Section 2.4, we show the
relation of the proposed scheme with alternative generative
and discriminative tracking methods. Section 3 presents
experimental results. Finally, in Section 4, we give some
conclusions and directions for future work. An early
version of the proposed scheme appears in [20].
2 PREDICTION-BASED TRACKING WITH RELEVANCE
DETERMINATION
Filtering, such as Kalman filtering or particle filtering, has
been the dominant framework for recursive estimation of
the conditional probability of the unknown state x given a
set of observed random variables Y ¼ f. . . ; y ; yg. In the
discriminative filtering framework (Fig. 3a), the filtered
density can be derived as [25]:
Z template tracking where x 2 R2 , the random variable r 2 R2
pðx j Y Þ ¼ dx pðx j Y  Þpðx j x ; yÞ; ð1Þ
where y (y ) is the observation at the current (previous)
frame and x (x ) is the state in the current (previous)
frame, respectively. Similarly, Y is the set of observations
up to the current frame and Y  the set observations up to
the previous frame.
The derivation of (1) ignores the fact that for certain
problems, different parts of the observation y can give
different predictions of the state of the target. For example,
in [30], for 2D tracking where the evidence y is an image
frame, the prediction of the state of the target (e.g., its 2D
location) is based on the data yðrÞ extracted from a single
1555
Fig. 3. Graphical models (a) for classical discriminative tracking and
(b) for regression tracking with relevance determination.
window centered at a position r. In the absence of a motion
model, r is the estimated position of the target in the
previous frame, that is, r ¼ x^ . However, using data from a
single window disregards the information that is available
at other positions r. Similarly, for 3D tracking [2], [25], a
single feature vector is extracted from the object silhouette.
On the other hand, in the generative particle filtering
framework for 2D tracking, it is common practice that
several parts of the observation are examined. This is
achieved by using multiple samples (particles) r and by
assigning to each particle a weight ðy; rÞ proportional to
the likelihood pðy j rÞ. The particles r are sampled from pðx j
Y  Þ using the transition probability pðx j x Þ and, most
usually, the sampled r determines how the observation y
will be utilized. The latter means that the likelihood pðy j rÞ
is modeled as a function of yðrÞ, that is, pðy j rÞ ¼ pðyðrÞ; cÞ,
where c are some model parameters. In the simplest case, a
number of measurements yðrÞ at positions rðr 2 fr1 ; . . . rR gÞ
around the location of the target in the previous frame are
utilized.2 Given the above, the posterior is empirically
approximated using a set of weighted particles, that is, a set
of pairs fððy; r1 Þ; r1 Þ; . . . ððy; rR Þ; rR Þg. Formally,
1X rR
pðx j Y Þ  ðy; rÞðx  rÞ; ð2Þ
Z r¼r1
where Z is a scaling parameter and ð:Þ is the Kronecker
function.
Here, we propose a discriminative particle filtering
method that utilizes the fact that several parts of the
observation can yield predictions of the state of the target.
We do so by introducing a random variable r that determines
which parts, or in general how, the observation y will be used.
Without loss of generality, in the derivations that follow, we
will assume that r has the dimensionality and the physical
meaning of the hidden state x. For example, in the case of 2D
will determine the centers of the windows/patches at which
we will extract observations yðrÞ that will give predictions of
x. In general, r will be used for obtaining a set of candidate
observations yðrÞ and does not need to have the dimension-
ality of x. We will also condition r on x as we expect that the
previous state can be sufficiently informative on how
candidate observations can be obtained. Subsequently, we
introduce a binary variable z and denote with pðz ¼ 1 j y; rÞ
the probability that the observation yðrÞ is relevant for
the prediction of the unknown state x. The dependencies of
the variables are depicted in Fig. 3b where y is observed and
2. In general, in the case that the state x is not only a 2D displacement,
obtaining yðrÞ requires warping.
1556 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,
TABLE 1
Discriminative Filtering with Data Relevance Determination
the remainder are hidden. In this graphical model, the filtered
density can be derived as
Z
pðx j Y Þ ¼ dx pðx j Y  Þ
Z Z  ð3Þ
drpðr j x Þ dzpðx j z; x ; y; rÞpðz j y; rÞ :
In what follows, we will describe our modeling choices
and a computational scheme for maintaining a representa-
tion that approximates the above posterior. We will show
that an approximation of the posterior pðxjY Þ is
1 XrR
pðx j Y Þ  ðy; rÞpðx j z ¼ 1; x ; y; rÞ
R r¼r1
! ð4Þ
X
rR
þ ð1  ðy; rÞÞpðx j z ¼ 0; x ; y; rÞ ;
r¼r1
4
where ðy; rÞ ¼ pðz ¼ 1 j y; rÞ is the relevance of the ob-
servation yðrÞ, pðx j z ¼ 1; x ; y; rÞ is the probabilistic pre-
diction for the state x given that the observation yðrÞ is
relevant, and pðx j z ¼ 0; x ; y; rÞ is a probabilistic prediction
of the state x given that yðrÞ is not relevant. The r1 ; . . . ; rR are
samples of the hidden variable r and need to be sampled
properly (in the way that is described below) so that (4)
indeed becomes an approximation of the posterior. Finally,
notice the similarity in form between our approximation (4)
and the approximation in the generative framework (2). We
will make the relationship explicit in Section 2.4.
In order to complete the specification of our framework,
we need to define its three main formal components, that is,
ðy; rÞ, pðx j z ¼ 1; x ; y; rÞ and pðx j z ¼ 0; x ; y; rÞ. We
assume that these probability distributions are either
derived from modeling assumptions or learned in a training
phase. For example, modeled probabilistic distributions
have been used in the context of motion estimation [24].
VOL. 32, NO. 9, SEPTEMBER 2010
TABLE 2
Modeling Choices
Here, we opt for a learning approach (as explained in
Sections 2.1 and 2.2) in which a probabilistic classifier
determines the observation relevance/reliability ðy; rÞ and
a Bayesian Mixture of Experts (BMEs) determines the
prediction pðx j z ¼ 1; x ; y; rÞ. In the tracking phase given
a triple ðy; r; x Þ, the trained BME yields a mixture of
Gaussians that is our approximation of pðx j z ¼ 1; x ; y; rÞ.
For pðx j z ¼ 0; x ; y; rÞ, that is, for the probabilistic predic-
tions given that the observation is not relevant, we use a
modeling approach and approximate it using a Gaussian
with a large covariance matrix S0 (alternatively, we could
have used a uniform distribution). Our modeling choices
lead to an approximation of pðx j Y Þ using a mixture of
Gaussians. This allows us to deal with posteriors with
multiple modes and also to recover from tracking failures.
In order to keep the number of mixture components
constant (equal to M), we devise, in the Appendix, a
method for approximating an L-component Gaussian
mixture with an M-component Gaussian mixture (M  L).
In Table 1, we describe the computational scheme that,
given an approximation of the posterior pðx j Y  Þ of the
state at the previous frame by an M-component mixture of
Gaussians, yields an approximation of the state posterior
pðx j Y Þ at the current frame by an M-component mixture of
Gaussians. In Table 2, we summarize our modeling choices.
Note that in (5), the integral is approximated using K þ 1
Gaussian components. In practice, in order to reduce the
number of the components, we use the approximation:
Z
dzpðx j z; x ; y; rÞpðz j y; rÞ
( P ð6Þ
 K i¼1 i N ði þ r; Si Þ; if  > z ;
¼
N ðx ; S0 Þ; otherwise:
R R
As a result, we approximate the term drpðr j x Þ dzpðx j z;
x ; y; rÞpðz j y; rÞ with a mixture of unnormalized L
(R  L  RK) Gaussians. We reduce it to an M-component
mixture in step 4.
The computational complexity of this scheme is
OðRK þ RJ þ RKMÞ, where OðRKÞ is the complexity of
the BME predictor and OðRKMÞ the complexity of the EM
algorithm for the reduction of the RK-component mixture
PATRAS AND HANCOCK: COUPLED PREDICTION CLASSIFICATION FOR ROBUST VISUAL TRACKING
of Gaussians to an M-component mixture of Gaussians.
OðRJÞ is the complexity of the kernel-based Relevance
Vector Machine classifier (Section 2.2), where J is the
number of the support vectors.
2.1 Bayesian Mixture of Experts for Regression
In what follows, we will describe a method that, given an
observation yðrÞ and the target state at the previous frame x ,
yields a probabilistic prediction of the state x at the current
frame. For notational simplicity, let us here denote with y
the Cartesian pair ðyðrÞ; x Þ.
Our method follows the work of Sminchisescu et al. [25]
and uses the Bayesian Mixtures of Experts for regression.
Given an observation y, the BME delivers a probabilistic
prediction of x as a mixture of Gaussians. The rationale
behind this choice over alternative regression methods (e.g.,
RVMs [26]) is that the BME can successfully model
predictive distributions that are multimodal in character.
Such distributions often arise in the case of 3D tracking due
to, for example, front/back and left/right ambiguity [25],
[2], [23]. They are also expected to arise in the case of 2D
tracking due to the aperture problem [10]. However, this
choice is not restrictive and any linear [32] or nonlinear
regression method [8] could be used as an alternative. More
generally, any method that can deliver a prediction of the
state x, given an observation y can be used. In the case of 2D
tracking, the Lucas-Kanade method [16] could be used to
make an estimate of the 2D target location x by delivering
an estimate of the displacement vector x with respect to
the position at which the observation yðrÞ was extracted.
Similarly, the method of Simoncelli et al. [24] could deliver
a probabilistic prediction (a Gaussian) for x.
The (Hierarchical) Mixtures of Experts, which was
introduced by Jordan and Jacobs [14], is a method for
regression and classification that relies on a soft probabil-
istic partitioning of the input space. This is determined by
gating coefficients gi ðyÞ (one for each expert i) that are input
dependent and have a probabilistic interpretation; that is,
the coefficients of the siblings at each level of the hierarchy
sum up to one. The prediction of each expert i is then
moderated by the corresponding gating coefficient. For-
mally, in the simple case of a flat hierarchy for regression,
X
K
pðx j yÞ ¼ gi ðyÞfi ðx j yÞ; ð7Þ
i¼1
where fi ðx j yÞ is a probability density function, usually a
Gaussian centered around the prediction of the expert i. In
the simple case that linear experts are used:
T positive examples that yield good predictions and with a set
e i y
gi ðyÞ ¼ P T y ; ð8Þ
je
j
and
 
fi ðx j yÞ ¼ N wTi y; Si ; ð9Þ
where the wi and i are the unknowns to be estimated. Jordan
and Jacobs [14] proposed a Maximum Likelihood method for
the estimation of wi and i , while in [29], a Bayesian approach
is used. We adopt the approach in [29] in which a set of
hyperparameters models the prior distributions of wi and i ,
1557
and follow a variational approach for the estimation of their
posterior distributions. As in [29], we make a Laplace
approximation and estimate the mode and the variance of
the posteriors, which (with a slight abuse of notation) we
denote here as ðwi ; wi Þ and ði ; i Þ. In the process, we also
estimate the optimal value for the hyperparameters i that
are associated with the noise covariance Si of the prediction
of expert i (see [29] for details).
In [29], a procedure is described for scalar regression. In
the case where the target is a vector x with dimensionality D,
we may train D different Mixtures of Experts. Here, we have
extended the methodology to experts that have multidimen-
sional output (i.e., fi ðx j yÞ is a multidimensional Gaussian
with diagonal noise covariance Si ). In this case, wi is a matrix
with the number of rows equal to the dimensionality of y and
number of columns equal to the dimensionality of x.
For prediction, we marginalize over the parameters and
hyperparameters as in [29]. For a new observation y, the
predictive distribution is a mixture of Gaussians given by
X
K  
p^ðx j yÞ ¼ gi ðyÞN wTi y; Si0 ; ð10Þ
i¼1
where the kth element of the diagonal covariance matrix Si0 ,
0
denoted with Sik , is given by
0
Sik ¼ yT wik y þ Sik ; ð11Þ
where Sik is the corresponding element in the covariance
matrix of the ith expert. Alternatively, we may straightfor-
wardly use (7), or use only the prediction of the expert with
the highest gating coefficient gi , or approximate (10) with a
single Gaussian.
For the problem of 2D visual tracking, we aim to
estimate the transformation x (e.g., translation, rotation,
and scaling) that a visual target undergoes in an image
sequence. We train the BME in a supervised way with pairs
ðyðxÞ; xÞ in which the observations yðxÞ are produced by
synthetically transforming (e.g., translating) the visual
target with the transformation x. In this case, we choose
to ignore the state x at the previous frame when training
the BME. Subsequently, in the test phase, an observation
will give a probabilistic prediction according to (10).
2.2 Data Relevance Determination
For the determination of the relevance/reliability pðz j y; rÞ
of an observation yðrÞ, we use a classification scheme with
the RVMs. The goal is to obtain an a priori assessment of
whether the probabilistic prediction pðx j yðrÞÞ (10) of the
state of the target is expected to be good. To this end, we
train an RVM classifier in a supervised way with a set of
of negative examples that yield bad predictions. Let us
denote with sigm the sigmoid function, with f~ yi g the
training set of the classifier, and with ðyi ; yj Þ a kernel
function (e.g., a Gaussian, or a linear one).
Then, after training and when presented with a novel
observation yðrÞ, the RVM yields a prediction of the
relevance of the observation yðrÞ as
!
X
rvm
pðz ¼ 1jyðrÞÞ ¼ sigm wi ðyðrÞ; y~i Þ ; ð12Þ
i
1558 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,
Fig. 4. Positions r of the vectors selected by the RVM-based classifier.
The inner (outer) window illustrates the range from which the training set
of the BME-based predictor (RVM-based classifier) is constructed.
where wrvm is a sparse weight vector that is learned in the
training phase.
The training set f~ yðrÞg is constructed as follows: A
candidate observation y~ðrÞ is generated by artificially
transforming (e.g., translating) the visual target with a
transformation which we denote here with r. Then, for each
of the candidate observations, a probabilistic prediction is
made using (10). We place in the set of positive examples
candidate observations for which an appropriate norm of
the difference between the true transformation r and the
expected value (i.e., the mean) of the prediction pðx j yðrÞÞ is
less than a threshold. That is,
kr  E pðxjyðrÞÞ ðxÞk < r :
As pðx j yðrÞÞ is a mixture of Gaussians, the expected value
of x in the above equation can be obtained in closed form.
Alternative schemes for constructing the positive training
set, such as thresholding the distance between the true
transformation r and the mode of pðx j yðrÞÞ, or by thresh-
olding the probability of the ground truth transformation r
(i.e., pðr j yðrÞÞ > r ), are also possible. The set of the
negative examples is composed of the observations for
which (13) is not satisfied. Other examples, such as
observations from regions in the background, could also
be added in the negative training set. Clearly, the
transformations r that generate the candidate training set
need to explore larger parts of the state space than the ones
used to construct the training set of the BME.
In Fig. 4, we illustrate the 2D case in which yðrÞ is an
observation taken at 2D windows around each position r. In
Fig. 4, superimposed on the original frame we depict the
range from which the r is taken for constructing: 1) the
training set of the BME-based predictor (inner window) and
2) the training set of the RVM-based classifier (outer
window). In the same figure, we superimpose the r for
the vectors that have been chosen by the RVM classifier.
In Fig. 5, and for the toy example that we used in Fig. 1,
we illustrate the true prediction error of an BME with eight
experts that have been trained to predict 2D displacements
in the interval ½11 . . . 11. Also shown is the corresponding
2D plot of pðz j y; rÞ. Note that we test with observations
that result from displacements from both inside and outside
the training interval. The RVM has been trained on positive
examples that have been selected by thresholding the L1
error norm. It is clear that we can predict reasonably well
which observations are associated with a low prediction
error. Note that not all observations that fall outside the
training range give high prediction errors. This indicates
that, in this case, the BME is capable of extrapolating.
VOL. 32, NO. 9, SEPTEMBER 2010
Fig. 5. (a) Prediction error and (b) pðz j y; rÞ as functions of the true
displacement. Here, a BME was trained to predict displacements in the
interval ½11 . . . 11 for a template of size 11  11. An RVM was trained to
classify as “relevant” observations in the interval ½22 . . . 22 that can
deliver accurate (þ= 2 pixels) predictions of the true displacement.
2.3 Classifier Selection
The classification scheme of Section 2.2 depends on the
choice of the parameter r (13) and a number of internal
parameters that need to be set. The former can be
interpreted as the desired level of prediction accuracy. Its
selection has direct implications to the complexity and
accuracy of the classifier since certain levels of prediction
accuracy might be difficult or impossible to achieve. The
first consideration is the ability of the regression scheme to
ð13Þ learn sufficiently well the true posterior pðx j z ¼ 1; y; x ; rÞ,
that is, to learn a pdf with most of its mass concentrated
around the true target state x . This depends on the size of
the template, the range of x in the training set, and a
number of parameters (such as the number of linear
predictors and the data y itself). A second consideration is
that the classification problem at a certain level of accuracy
(i.e., for a certain r ) might be very difficult to solve, while,
for a different value of r , it might be considerably easier. A
small r can lead to an empty positive training set, while a
large value of r can lead to low accuracy.
The threshold r and the remaining parameters, such as
the internal parameters of the classifier, can be selected in a
cross-validation scheme. In what follows, we propose an
alternative way of determining the classification scheme.
The main idea is that ideally, the probabilistic classifier
should rank the observations according to their prediction
accuracy. In other words, the rank order of 1  pðz ¼ 1 j y; rÞ
should coincide with the rank order of the error of the
corresponding prediction. The divergence between the ideal
ranking and the ranking obtained by a specific probabilistic
classifier can therefore give a measure of the goodness of
the classifier in question.
Formally, let us assume that the observations are ranked
in order according to some criterion. Let q vary between 0
and 1, and let eðqÞ be the mean error of the predictions of
the observations that are on the upper fraction q according
to the ranking used. Let eo be the mean error under the
“ideal” ranking, i.e., under the ranking according to the
error itself, and e be the mean error under the ranking of a
certain probabilistic classifier. That is, eðqÞ is the mean
error of the top q fraction of observations ranked according
to the learned pðz ¼ 1 j x; yÞ. We vary the threshold r and
the size of the kernel of the RVM classifiers ( phi ) and, in
Fig. 6, we depict eo ðqÞ and eðqÞ for some of them, where the
differences in the performance are clearly visible.
PATRAS AND HANCOCK: COUPLED PREDICTION CLASSIFICATION FOR ROBUST VISUAL TRACKING
Fig. 6. Mean error versus the fraction q of positives for different
classifiers. The curve eo corresponds to the ideal classifier.
We make the selection according to the 2 test between eo
and e, that is, we select a curve that lies on the lower right
part of the plot. Such a classifier generates a large number of
positives for a given error level, a property that is important
in our estimation scheme which might rely on few
observations only. As Fig. 6 reveals that a number of
classifiers (solid lines) have similar ranking properties.
Among classifiers with similar ranking properties (with a
5 percent margin), we favor the one that delivers, on
average, the lower weighted error under the weighing of the
positives according to the probabilistic weighing scheme.
2.4 Relation to Generative and Discriminative
Tracking
In this section, we will make explicit the relation between
the proposed tracking framework on the one hand, and
both discriminative and generative tracking methods on the
other hand. More specifically, we will show that under
certain modeling assumptions, we derive estimation
schemes that are practically equivalent to classical gen-
erative and discriminative estimation schemes.
The relation to classical discriminative methods (e.g.,
[25]) is rather straightforward. In the case that a single
observation yðrÞ is used and the data relevance  ¼ pðz ¼
1 j y; rÞ is set to one (i.e., the single observation is considered
relevant/reliable), our framework reduces to the discrimi-
native tracking framework of [25]. Formally, (1) can be
derived from (3) when the following three modeling choices
are made:
. First, pðr j x Þ ¼ ðr  x Þ. That is, the auxiliary
variable r that controls multiple observations coin-
cides with x , and therefore, effectively is not used.
. Second, pðz ¼ 1 j y; rÞ ¼ 1 (and therefore, pðz ¼
0 j y; rÞ ¼ 0). That is, the single observation that is
utilized is considered relevant/reliable.
. Third, pðx j z; x ; y; rÞ ¼ pðx j x ; yÞ. That is, the
prediction does not dependent on the auxiliary
variables z; r.
We now show that under suitable modeling choices, the
proposed framework reduces to an estimation scheme that
is equivalent to particle filtering in the generative frame-
work. To commence, we note that, using the Bayes rule, the
posterior is given by
1559
pðy j xÞ
pðx j Y Þ ¼ pðx j Y  Þ; ð14Þ
pðy j Y  Þ
Z
pðy j xÞ
¼ dx pðx j x Þpðx j Y  Þ: ð15Þ
pðy j Y  Þ
In the generative framework, a number of candidate
solutions ri are sampled from pðx j Y  Þ and they are
subsequently weighted using the likelihood3 pðy j ri Þ. If
we denote with ðy; rÞ the weight that is assigned to sample
r, then an approximation of the posterior is given by (2).
Recall that in our framework, the approximation of the
posterior is given by (4). Also, observe the relation between
(2) and (4): In the generative case, the mass of the posterior is
on the samples ri , while, in the discriminative case, the mass
of the posterior is on the predictions pðx j z ¼ 1; x ; y; ri Þ (let
us for the moment ignore the “outlier predictions”
pðx j z ¼ 0; x ; y; ri Þ). Therefore, if we choose the predictors
such that pðx j z ¼ 1; x ; y; ri Þ is equal to ðx  rÞ, and let r be
sampled from pðx j Y  Þ, then the estimation schemes of the
two frameworks will be equivalent. There are just two
differences in the methods. First, in our framework, once a
sample r is obtained in this way, it should be assigned a
weight  equal to pðz ¼ 1 j r; yÞ. By contrast, in the generative
framework, the sample r should be assigned a weight equal
pðyjrÞ 
to pðyjY  Þ , or a weight equal to pðy j rÞ since the term pðy j Y Þ
is independent of r and therefore is canceled in the normal-
ization. Second, in our case, the prediction of the outliers (i.e.,
observations that are irrelevant/unreliable) is made explicit
in the form of pðx j z ¼ 0; x ; y; ri Þ.
Formally, under the modeling choice pðx j z ¼ 1; x ;
y; rÞ ¼ ðx  rÞ, (3) becomes
pðx j Y Þ ð16Þ
Z Z
¼ 
dx pðx j Y Þ 
drpðr j x Þ
Z ð17Þ
dzpðx j z; x ; y; rÞpðz j y; rÞ
Z
¼ pðz ¼ 1 j y; xÞ dx pðx j x Þpðx j Y  Þþ
Z Z
dx pðx j Y  Þ drpðr j x Þpðx j z ¼ 0; x ; y; rÞpðz ¼ 0 j y; rÞ
ð18Þ
¼ pðz ¼ 1 j y; xÞpðx j Y  Þþ
Z Z
dx pðx j Y Þ drpðr j x Þpðx j z ¼ 0; x ; y; rÞpðz ¼ 0 j y; rÞ:
  
ð19Þ
Let us now comment on the main differences between (18)
and the filtering equation in the generative framework (15).
The first difference is that instead of using the likelihood
pðy j xÞ, our framework uses the term pðz ¼ 1 j y; xÞ in order
to weight samples r that are sampled from pðx j Y  Þ. In our
3. In general, a number of candidate solutions r 2 fr1 ; . . . ; rR g are
sampled from a proposal distribution gðrÞ and they are subsequently
Þ
weighted by pðy j rÞ pðrjY
gðrÞ .
1560 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,
framework, the variable z takes the interpretation of the class
of the observation y and pðz ¼ 1 j y; xÞ is the probability that
the observation y belongs to the target class. Essentially, a
classification scheme is used within the particle filtering
framework. This is similar to other works in classification-
based tracking [3], [7] and to more classical target detectors
[27]. Therefore, our scheme offers a formal framework in
which classification-based approaches can be used for
recursive estimation of the posterior density.
The second main difference of the degenerate case of our
scheme with the generative particle filtering tracking (15) is
that in our case, the prediction of the outliers becomes
explicit in the form of pðx j z ¼ 0; x ; y; ri Þ, that is, the
second term of (18). This bears similarities to generative
methods that introduce an occlusion process and condition
the likelihood on it. In [31], two likelihood models are
defined, one given that the target is occluded and one given
that the target is visible. However, while in [31] the
occlusion state is inferred, in our case the relevance of the
observation is determined by the probabilistic classifier.
Also, note that in the general case of our method, the
relevance of an observation is not necessarily related to the
degree of occlusion but rather to the degree at which a
reliable prediction can be obtained from the observation in
question. Three possible choices for the prediction given that
the observation is irrelevant/unreliable are the following.
The first is to use a Gaussian with large variance around x .
A second choice is to use a uniform distribution , where
 << 1. The third is to use a prior pðxÞ in case that it is
available [19]. Since pðx j z ¼ 0; x ; y; ri Þ depends on y, we
can easily derive classical outlier processes based on robust
statistics [17], or use a Gaussian with large variance as in
[13]. Formally, the second term which models the predic-
tion density of the “unreliable observations” under two
reasonable models of pðx j z ¼ 0; x ; y; rÞ becomes
1
pðz ¼ 0 j y; xÞ pðxÞpðx j Y  Þ
Z ð20Þ
1
for pðx j z ¼ 0; x ; y; rÞ ¼ pðxÞðx  rÞ;
Z 3.1 Artificial Displacements and Noise
and
pðz ¼ 0 j y; xÞpðx j Y  Þ
ð21Þ
for pðx j z ¼ 0; x ; y; rÞ ¼ ðx  rÞ;
where Z is a scaling parameter and  << 1.
3 EXPERIMENTAL RESULTS
We have performed a number of experiments in order to
illustrate the performance of the proposed method under
different conditions, including occlusions, fast motion, and
moderate deformations. Here, we present both quantitative
and qualitative results for image sequences that are
annotated by hand as well as comparative results with
alternative state-of-the-art methods. More specifically, we
compare our Coupled Prediction-Classification algorithm
(CPredC) to discriminative tracking when a single ob-
servation is used (e.g., [25], [30]) and to the simplified
version of the proposed algorithm in which the data
relevance determination mechanism is discarded. We do
not use any dynamic model, nor temporal filtering in order
to judge the performance when large deviations from the
VOL. 32, NO. 9, SEPTEMBER 2010
motion model are present. In addition, we have compared
our method with a particle filtering algorithm in the
generative framework.
For each of our experiments and in order to reduce the
computational complexity during training, we reduce the
data dimensionality by applying Principal Component
Analysis (PCA) to the data with which the BMEs are trained.
PCA gives some marginal improvements, and omitting it
does not lead to a significant degradation in the performance.
Before being used, the training data for the BME, the training
data for the RVM, and all test data are projected to the new
space that is spanned by the leading N eigenvectors that
were extracted using PCA. We choose N such that 95 percent
of the variance is retained, a choice that leads to values of N
between 40 and 50 for the “CD” and the “Head” sequences.
In order to deal with illumination changes, we normalize
each observation by the average intensity of the window at
which it was extracted. The normalization is performed
before learning the PCA transform (i.e., during training) and
before applying it (i.e., during tracking). In our experiments,
we tracked windows of size ranging from 15  15 to 25  25
pixels. For training the BME, we used 900 Cartesian pairs
ðyðxÞ; xÞ in which the observations yðxÞ are produced by
artificially transforming (e.g., translating) the visual target
with the transformation x. The examples that were used for
training the RVM were generated using transformations
with a range two-three times that used for training the BME.
In order to reduce the complexity of RVM learning, we apply
k-means clustering to the set of candidate observations and
train the RVM using the cluster centers. The class identity of a
cluster (positive or negative) is determined by the class
identity of the majority of the examples that belong to it. The
number of clusters is considered as a parameter of the
classification scheme and is determined automatically using
the criterion of Section 2.3. For all the experiments, we track
five Gaussians (i.e., M ¼ 5) and use 50 samples r (i.e.,
R ¼ 50), unless stated otherwise.
We first present results for estimating the location of a facial
feature (the corner of an eye) under artificially generated
displacements. Both the regressor and the classifier were
trained on data from the first frame of the sequence. Here,
1) a 17  17 target window was used, 2) the BME was trained
with displacements of up to 17 pixels, and 3) the RVM was
trained with displacements of up to 34 pixels. The tests were
performed on the frames 157 and 358 (Fig. 11). In frame 157,
some degree of both deformation and illumination change
are present. In frame 358, the target is partially occluded. For
these frames, artificial displacements were simulated by
sampling x at a distance equal to the “true displacement”
from the true target location. During the estimation phase,
50 samples of r are sampled from a 2D uniform distribution
with mean x and width S at each dimension. As we have
demonstrated in Section 2.4, for a sampling range of width
S ¼ 1 and when a single r is sampled, we are effectively
using classical regression-based tracking methods (e.g., [30],
[2]) that use a single observation.
In Fig. 7, we summarize the experimental results for
artificially generated motions of different magnitudes by
plotting the Root Mean Square (RMS) error as a function of
both the true displacement and of the sampling range for
PATRAS AND HANCOCK: COUPLED PREDICTION CLASSIFICATION FOR ROBUST VISUAL TRACKING
Fig. 7. “Head” sequence: Error (RMS) as a function of both the true
displacement and the sampling range. (a) Frame 157. (b) Frame 358.
various frames of the “Head” sequence. In Fig. 7a, we show
the error for frame 157, a frame in which the target is
slightly deformed. As expected, for the successful estima-
tion of larger displacements, r needs to be sampled using a
2D distribution with larger width S. Also, there exists a
range of values of S for which the true displacement can be
reliably estimated. In Fig. 8a, for the same frame (157), the
solid line depicts the error for S ¼ 36. This demonstrates
that we are able to reliably estimate motions larger in
magnitude than the range over which the regressor was
trained. In the same figure, we demonstrate the importance
of using both multiple observations and observation
relevance determination by presenting comparative results
for two simplified versions of our method. The cases
considered are: 1) a simplified version (nPred) in which the
relevance determination is not used (i.e.,  ¼ 1) and 2) a
simplified version (sPred) in which only a single observa-
tion is used (i.e.,  ¼ 1; R ¼ 1). It is clear that when
compared with the proposed CPredC algorithm, both
simplifications lead to reduced performance in terms of
the range of motions (displacements) that can be reliably
estimated. This difference in the performance becomes even
larger in the case of partial occlusions, as evidenced by the
results for frame 358 (Fig. 11, row 4) that are depicted in
Figs. 7b and 8b. In this frame, for some displacements and
sampling ranges S, it is the case that x and some of the
samples r are likely to fall within the occluded area. This
results in fewer relevant observations that can deliver good
predictions. However, while the performance is worse than
for frame 157 under large displacements, there seems to be
little difference in the performance for displacements of up
to 15 pixels. This indicates that the RVM-based classification
scheme manages to detect observations that fall within the
Fig. 8. “Head” sequence. Error as a function of the true displacement.
Blue line: proposed method (CPredC), red line: multiple observations
without relevance determination (nPred), black line: Single observation
without relevance determination (sPred). (a) Frame 157. (b) Frame 358.
1561
Fig. 9. “Head” sequence: Estimation error at different noise levels for
frames 1 and 157. (a) Frame 1. (b) Frame 157.
occlusions. The importance of relevance determination is
clearly illustrated in Fig. 8a, where we show the RMS error
for the case in which the relevance is not used (i.e.,  ¼ 1)
and for the sampling range with S ¼ 36. Similarly, when a
single observation is used the performance deteriorates
rapidly as the magnitude of the displacement increases.
In Fig. 9, we present results that illustrate the estimation
accuracy at different levels of additive Gaussian noise with
zero mean. Here, n denotes the noise standard deviation.
In Fig. 9a, we present results for the first frame of the
sequence at which the training in performed. In Fig. 9b, we
show results for the 157th frame, where the target
appearance is slightly affected by lighting and there is also
some mild deformation and compression. The performance
seems to degrade gracefully as the standard deviation of the
noise increases and the technique is quite robust especially
for low displacements.
3.2 2D Template Tracking
Here, we present results for tracking 2D templates in a
number of image sequences under changes in the illumina-
tion, large motion, rotation, and nonrigid deformations. In
Fig. 11, we present some characteristic frames for the “Head”
image sequence. A 17  17 window is tracked using 50
observations at a rate of 3.1 frames per second using an
unoptimized Matlab implementation (excluding time for
input/output). The tracking is consistently good throughout
the image sequence, even in the presence of large motion (as a
fraction of the window size), occlusion, and some deforma-
tions. In Fig. 10a, we show the horizontal and vertical
components of the error in pixels, Fig. 10c shows the ground
truth velocity for a T S ¼ 3, and Fig. 10b shows the ground
truth displacement of the target from its position in the first
frame. In the first column of Fig. 11, we depict in red (blue) the
location at which relevant (unreliable) observations were
extracted, in the second column, we give the corresponding
probabilistic predictions (each ellipse represents a 2D
Fig. 10. “Head” sequence (SS ¼ 1; T S ¼ 3): (a) Error, (b) true target
displacement, and (c) true velocity (pixs per frame).
1562 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 32, NO. 9, SEPTEMBER 2010

Fig. 11. Tracking results for frames 70, 118, 154, 202, 358, and 580 of the “Head” sequence (SS ¼ 1 and ST ¼ 3). (a) Location of relevant (red)
and unreliable (blue) observations. (b) Relevant (red) and unreliable (blue) predictions. (c) Probabilistic prediction (red) and point estimation
(black box).

Gaussian), and in the third column, we give the probabil-
istic prediction (each ellipse represents one of the M 2D
Gaussians) as well as the final estimate of the target location
(black box). It is clear that the classification scheme reduces
(or completely discards) the influence of inaccurate predic-
tions. This is apparent both in the case of larger motion
(rows 1-3) in which the prediction of observations further
away from the target (in the direction opposite from that of
the motion) is discarded, as well as in the case of occlusions
(row 4).
In order to illustrate the benefits of using both multiple
observations and data relevance determination, we present
comparative results with the two simplified versions of our
algorithm (sPred and nPred), together with two alternative
methods reported in the literature, namely, the CONDEN-
SATION algorithm and [32] when a single target is tracked.
The simplified version sPred is similar to classical regres-
sion-based tracking methods (e.g., [30], [2]) that use a single
observation. Recall that we arrive at this simplification by
setting the number of samples of r equal to 1 (i.e., R ¼ 1) and
the width of the sampling range very small (i.e., S ¼ 1). The
PATRAS AND HANCOCK: COUPLED PREDICTION CLASSIFICATION FOR ROBUST VISUAL TRACKING
TABLE 3
RMS Error and Percentage of Target Losses (in Parentheses)
at Various Spatial (SS) and Temporal (TS) Subsamplings
for Various Sequences
For “Towel,” the RMedS error is reported.
algorithm nPred is a simplified version of our method in
which relevance determination is not used. Instead, the
probabilistic predictions for all candidate observations yðrÞ
are weighted equally (i.e., ðy; rÞ ¼ 1). For the CONDENSA-
TION algorithm, we 1) use 500 particles, 2) use a Gaussian
observation model (for the CD sequence, we use an
observation model that relies on the Normalized Cross
Correlation), and 3) use as a state transition model pðxjx Þ of
the 2D state x a Gaussian with mean equal to x . In Table 3,
we present results for the “Head,” “CD,” and “Towel”
sequences. In the case of the “Head” and “CD” sequences,
the location of the tracked template is manually annotated
every six frames. In the case of the “Towel” sequence, we
have, for each frame, a manual annotation of four points of a
large planar object of which the tracked target (a 21  21
template) is part. The ground truth location of the target is
estimated using an affine transform. We report the averages
over five runs.
For the proposed CPredC scheme and the Sequence of
Linear Predictors (SLiPs) [32], we report the RMS Error and
the percentage of frames for which the error is larger than the
dimensions of the tracked template (in the case of [32], the
percentage of frames for which the error is larger than
the training range). In the case of large errors, all other
algorithms are also reinitialized and, for them, we report the
number of reinitializations as a percentage of the tracked
frames. Note that the number of reinitializations reported for
sPred (i.e., a scheme that utilizes a single observation) is the
number of times that the error was larger than the template
size. It is very likely that a scheme that validates a prediction,
for example, by classifying an observation extracted at the
predicted location [30], would signal that the target has been
lost. In comparison to SLiPs, both the error (RMS or RMedS
for the Towel) and the number of frames at which the target
is lost are consistently lower for the CPredC. For the Towel
sequence and for SS ¼ 4, while the tracking is significantly
more accurate for CPredC (as reflected in the very low
RMedS), at frames where the target is significantly rotated
Fig. 12. Frames 175, 346, and 494 of the Towel sequence.
1563
Fig. 13. Examples of tracking under different Occlusion Types (OTs).
From left to right: OT = 4, OT = 2, OT = 1.
such as frame 384 at which the rotation is around 80 degrees
(Fig. 12), the tracking becomes locked on clutter. This is
largely expected and illustrates the need to obtain observa-
tions at scales and orientations that are similar to the ones
used in the training set. In such cases, maintaining a
(multimodal) estimate of the scale and rotation components,
for example, by tracking multiple points and compensating
for it in order to obtain warped observations, is needed. For
all sequences tested, both sPred and nPred perform sig-
nificantly worse than CPredC, and, in general, sPred is
consistently worse than nPred. Let us also note that all of the
algorithms deteriorate at higher temporal subsampling, that
is, as TS increases. Finally, note that in the second row of
Fig. 11, a multimodal posterior is maintained by CPredC.
This is due to the fact that some predictions are locked on the
other eye as some observations fall in its vicinity. This is a
point where the other trackers occasionally fail, and is a point
where CPredC also fails occasionally in the case of larger
temporal subsampling (i.e., larger apparent motion).
3.3 Occlusions and Motion Large in Magnitude
In what follows, we present results for experiments
performed in order to test the behavior of our algorithm
under large and persistent occlusions. In order to do so, we
have artificially occluded part of the target at frames in
which the ground truth is known. We have used different
Occlusion Types and patterns (denoted by OT), examples of
which are presented in Figs. 13, 14, and 15. Both uniform
and textured patterns are used and either half (for odd OT)
or a quarter (for even OT) of the target is occluded. Again,
different spatial and temporal subsamplings are examined.
Since for the given sequences, the ground truth is available
every sixth frame, subsampling with TS ¼ 3 results in the
target being occluded every second frame, and subsampling
with TS ¼ 2 results in the target being occluded every third
frame. For the CD sequence, we used R ¼ 100 observations
and tracked a 15  15 template at a rate of 5.8 frames/sec in
an unoptimized Matlab implementation (excluding time for
input/output).
The results are summarized in Table 4, in which the RMS
error is reported. While the performance of the alternative
algorithms clearly deteriorates, the proposed CPredC
method is capable of tracking robustly under partial
occlusions. In some cases, the occlusions are severe (e.g., in
the last two rows of Table 4, half of the target is occluded)
and in the case of Occlusion Type 5 (OT ¼ 5) the appearance
of the occluding area changing. The results of CPredC in
comparison to sPred and nPred indicate the significance of
using multiple observations as well as weighing their
predictions in terms of their expected relevance. For these
sequences, the difference in the performance with SLiPs is
more pronounced in terms of the RMS, especially in the case
1564 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,
Fig. 14. Tracking results for the “Head” sequence. OT = 5, SS = 2, TS = 3 (see Table 4).
Fig. 15. Tracking results for frames 49, 115, and 253 of the “CD cover” sequence (SS ¼ 2 and T S ¼ 3). A quarter (QRT) of the target is artificially
occluded every other frame.
of larger occlusions, such as the last two rows of Table 4 in
which half of the target is occluded. The results in
comparison to generative tracking (i.e., CONDENSATION)
illustrate the ability of prediction-based methods to utilize an
observation near the target, rather than on the target itself in
order to predict the target location. This is the case even
when the target is partially occluded, so long as the
observation is relevant. In the case of partial occlusions, it
is also the case that methods that rely on the validation of a
prediction, for example, by classifying the observation that is
extracted at the predicted location [30], are likely to fail even
when the prediction is accurate.
In Fig. 16, we give some insight into the ability of the
algorithm to deal with partial occlusions. More specifi-
cally, in the case of partial occlusions, it is those
observations that are extracted in regions neighboring
the true target position that are deemed relevant by the
classifier. These observations are therefore used to deliver
reliable predictions of the target state. In Fig. 16a, we show
the locations at which relevant (unreliable) observations
were extracted in red (blue), in Fig. 16b, we show the
corresponding probabilistic predictions (each ellipse re-
presents a Gaussian), and in Fig. 16c, we show the final
TABLE 4
Errors for Various Occlusion Types (OTs) at
Various Spatial (SS) and Temporal (TS) Subsamplings
VOL. 32, NO. 9, SEPTEMBER 2010
probabilistic prediction using five Gaussians. Note that
our relevance determination scheme suppressed most of
the observations that were close to the true target location.
This indicates that a validation scheme using the trained
RVM classifier would also be likely to fail. Similar results
are obtained for alternative types of occlusions.
3.4 4D Template Tracking
Finally, we present results for the sequential estimation of
the location, scale, and rotation of visual targets in image
sequences. In Figs. 17 and 18, we show some representative
frames for two image sequences in which the 4D state
(translation, scale, and rotation) of a primary target is
estimated and the position of a secondary target is inferred.
For each of the sequences, a single Bayesian Mixture of
Experts is trained for predicting the 4D state vector. Here, r
has the dimensionality of the state and determines the
location, scale, and orientation parameters of the warping
required in order to obtain the observation yðrÞ. For these
experiments 1,500 examples were used for training the
BME-based predictor. The results for the “Soda can” [4]
(Fig. 17) and the “Hand Held” (Fig. 18) image sequences
Fig. 16. Coupled prediction classification in occlusions (frame 469,
Fig. 14, last column). (a) Location of relevant/unreliable observations (red/
blue). (b) Corresponding probabilistic predictions. (c) Final prediction.
PATRAS AND HANCOCK: COUPLED PREDICTION CLASSIFICATION FOR ROBUST VISUAL TRACKING
Fig. 17. Soda Sequence [4] (ST ¼ 3): Estimate of the translation/scale
and rotation of an 25  25 window. The position of the lower corner of the
can is inferred from the estimation of the state of the target (central
square).
give an illustration of the accuracy at which the scale and
the rotation of the primary target are estimated. In order to
create larger motions, we have temporally subsampled the
“Soda can” sequence by a factor of 3. The “Hand Held”
image sequence (Fig. 18) contains significant motion blur at
certain frames (e.g., third row and second column) as well
as occasional partial occlusions of the target. Quantitative
results are presented in Table 5 which give the estimation
error of the translation, scale, and rotation components.
The results in the first row of Table 5 refer to the “Head”
image sequence where the secondary target is the right eye
of the depicted person.
3.5 Synopsis
In summary, the experimental results demonstrate the
efficiency of the proposed scheme in both 2D and 4D
tracking. In the case of 2D template tracking, a clear
improvement is demonstrated in comparison to: 1) alter-
native methods that use a single observation (sPred and [32]),
2) alternative methods that use multiple observations with-
out relevance determination (nPred), and 3) a generative
particle filtering method [11]. The improvements were more
pronounced in the case of partial occlusions and rapid
motion. This is due to the ability of the proposed scheme to
discard the predictions of observations that originate at
occlusions or from areas too far from the target. The results in
sequences with large rotations and scale changes indicate the
need to maintain an estimate of the transform so that warped
Fig. 18. “Hand Held” sequence: Estimate of the translation/scale and
rotation of a window centered around the center of the picture. The
position of the lower corner of the picture is inferred from the estimate of
the state of the target (central square).
1565
TABLE 5
Mean Absolute and RMS (in Parentheses) Errors
for the “Head” and the “Soda Can” Sequence (SS ¼ 4)
observations are obtained at scales and orientations similar
to those used in the training phase.
4 CONCLUSIONS
In this paper, we have presented a method for efficient and
robust visual tracking. We propose a discriminative frame-
work in which multiple observations provide predictions of
the state of the target. Each prediction is moderated by the
relevance of the corresponding observation as this is
determined by a probabilistic classification scheme. This is
the first work that utilizes multiple observations for
discriminative tracking or uses a classification scheme to
assess in advance the relevance of an observation (as
opposed to the a posteriori validation of the prediction). We
have illustrated the efficiency of our approach in a number
of image sequences for the problem of 2D tracking and in
particular its ability to deal with large motion and with
partial occlusions. For future work, we intend to extend the
proposed scheme for tracking in higher dimensional spaces,
for example, for tracking 3D human pose under occlusions
and background clutter.
The estimation scheme that we propose is rather general
and neither limited to the problem of tracking nor tied to
the specific regression-based state prediction or RVM-based
observation relevance determination. In principle, any
estimation scheme that employs a regression-based pre-
dictor can be tied to an observation relevance/reliability
estimator in order to utilize multiple observations and
moderate their predictions according to their expected
accuracy (the latter being determined by the observation
relevance/reliability estimator). It would therefore be
interesting to investigate the properties of the proposed
framework in domains other than visual tracking.
Finally, in order to focus on the properties of the coupled
regressor-classifier scheme, we focused our analysis on the
problem of a single target tracking without a motion model.
In future work, we intend to extend the scheme on the
problem of multiple interacting target tracking [21], and
using our scheme with learned motion models.
APPENDIX A
FROM AN L-COMPONENT TO AN M-COMPONENT
MIXTURE OF GAUSSIANS
In this Appendix, we will briefly outline a method for
approximating a mixture of L Gaussians with a reduced
M-component mixture. Our derivation builds on the
method of Vlassis and Verbeek [28] for learning a Gaussian
Mixture from noisy data.
1566 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 32, NO. 9, SEPTEMBER 2010

Let us denote with REFERENCES

[1] A. Agarwal and B. Triggs, “A Local Basis Representation for
X
L
Estimating Human Pose from Cluttered Images,” Proc. Asian Conf.
fðxÞ ¼ pl fðx j lÞ Computer Vision, pp. 50-59, 2006.
l¼1 [2] A. Agarwal and B. Triggs, “Recovering 3D Human Pose from
Monocular Images,” IEEE Trans. Pattern Analysis and Machine
the given L-component mixture, where fðx j lÞ ¼ N ðxl ; Cl Þ,
Intelligence, vol. 28, no. 1, pp. 44-58, Jan. 2006.
for l ¼ 1 . . . L, is a Gaussian with known mean xl and [3] S. Avidan, “Support Vector Tracking,” IEEE Trans. Pattern Analysis
covariance Cl . Let us also denote with and Machine Intelligence, vol. 26, no. 8, pp. 1064-1072, Aug. 2004.
[4] M. Black and A. Jepson, “Eigentracking: Robust Matching and
X
M Tracking of Articulated Objects Using a View-Based Representa-
pðxÞ ¼ m pðx j mÞ tion,” Proc. European Conf. Computer Vision, pp. 329-342, Apr. 1996.
m¼1
[5] A.M. Buchanan and A.W. Fitzgibbon, “Combining Local and
Global Motion Models for Feature Point Tracking,” Proc. IEEE
the unknown M-component mixture, with pðxjmÞ ¼ N ðm ; Conf. Computer Vision and Pattern Recognition, 2007.
[6] D.M.C. Sminchisescu and A. Kanaujia, “BMA3 E : Discriminative
Sm Þ, for m ¼ 1 . . . M, is a Gaussian whose mean m , Density Propagation for Visual Tracking,” IEEE Trans. Pattern
covariance Sm , and mixture coefficient m we seek to Analysis and Machine Intelligence, vol. 29, no. 11, pp. 2030-2044,
estimate. Nov. 2007.
[7] R. Collins, Y. Liu, and M. Leordeanu, “Online Selection of
As in [28], we minimize the Kullback-Leibler divergence Discriminative Tracking Features,” IEEE Trans. Pattern Analysis
between pðxÞ and fðxÞ by maximizing an objective function and Machine Intelligence, vol. 27, no. 10, pp. 1631-1643, Oct. 2005.
that is a lower bound of the negative of the KL-divergence. [8] T. Cootes, G. Edwards, and C. Taylor, “Active Appearance
Models,” IEEE Trans. Pattern Analysis and Machine Intelligence,
Formally, we maximize vol. 23, no. 6, pp. 681-685, June 2001.
[9] J. Deutscher, A. Davison, and I. Reid, “Automatic Partitioning of
L Z
X High Dimensional Search Spaces Associated with Articulated
F ¼ dxfðx j lÞflog pðxÞ  KLm ½ql ðmÞkpðm j xÞg; ð22Þ Body Motion Capture,” Proc. Int’l Conf. Computer Vision and
l¼1 Pattern Recognition, Dec. 2001.
[10] B. Horn and B. Schunck, “Determining Optical Flow,” Artificial
where ql ðmÞ, for l ¼ 1 . . . L, are the auxiliary variational Intelligence, vol. 17, nos. 1-3, pp. 185-203, Aug. 1981.
distributions that are introduced for bounding from below [11] M. Isard and A. Blake, “Condensation—Conditional Density
Propagation for Visual Tracking,” Int’l J. Computer Vision,
the negative of the KL-divergence between pðxÞ and fðxÞ. vol. 29, no. 1, pp. 5-28, 1998.
The update equations are very similar to the update [12] M. Isard and A. Blake, “A Mixed-State Condensation Tracker with
equations of the EM algorithm. More specifically, the Automatic Model-Switching,” Proc. IEEE Int’l Conf. Computer
Vision, pp. 107-112, 1998.
variational distributions ql are updated as [13] A.D. Jepson, D.J. Fleet, and T.F. El-Maraghi, “Robust Online
 Appearance Models for Visual Tracking,” IEEE Trans. Pattern
1  1  Analysis and Machine Intelligence, vol. 25, no. 10, pp. 1296-1311, Oct.
ql ðmÞ / m pðxl j mÞ exp  T r Sm Cl ; ð23Þ 2003.
2
[14] M. Jordan and R.A. Jacobs, “Hierarchical Mixtures of Experts and
while the mixture components are updated as the Em Algorithm,” Neural Computation, vol. 6, pp. 181-214, 1994.
[15] F. Jurie and M. Dhome, “Hyperplane Approximation for Template
PL Matching,” IEEE Trans. Pattern Analysis and Machine Intelligence,
pl ql ðmÞ vol. 24, no. 7, pp. 996-1000, July 2002.
m ¼ PM l¼1PL ; ð24Þ
[16] B. Lucas and T. Kanade, “An Iterative Image Registration
m¼1 l¼1 pl ql ðmÞ
Technique with an Application to Stereo Vision,” Proc. Int’l Joint
Conf. Artificial Intelligence, pp. 121-130, 1981.
PL [17] P. Meer, D. Mintz, and A. Rosenfeld, “Robust Regression Methods
pl ql ðmÞxl
m ¼ Pl¼1
L
; ð25Þ for Computer Vision: A Review,” Int’l J. Computer Vision, vol. 6,
l¼1 pl ql ðmÞ no. 1, pp. 59-70, 1991.
[18] K. Okuma, A. Taleghani, N.D. Freitas, J.J. Little, and D.G. Lowe,
PL  T  “A Boosted Particle Filter: Multitarget Detection and Tracking,”
l¼1 pl ql ðmÞ xl xl þ Cl Proc. European Conf. Computer Vision, pp. 28-39, May 2004.
Sm ¼ PL  m Tm : ð26Þ [19] M. Pantic and I. Patras, “Dynamics of Facial Expression:
p q
l¼1 l l ðmÞ Recognition of Facial Actions and Their Temporal Segments from
Face Profile Image Sequences,” IEEE Trans. Systems, Man, and
In the case that the Gaussians have zero covariance Cybernetics, Part B, vol. 36, no. 2, pp. 433-449, Apr. 2006.
matrices (i.e., Cl ¼ 0) and pl ¼ L1 , we obtain the update [20] I. Patras and E. Hancock, “Regression Tracking with Data
equations of the classical EM algorithm for learning Relevance Determination,” Proc. IEEE Conf. Computer Vision and
Pattern Recognition, June 2007.
the mixture parameters from L data points l . For arbitrary [21] I. Patras and M. Pantic, “Particle Filtering with Factorized
Cl and pl ¼ L1 , we obtain the update equations of [28] for Likelihoods for Tracking Facial Features,” Proc. IEEE Int’l Conf.
learning the mixture parameters from l noisy data points Face and Gesture Recognition, pp. 97-102, May 2004.
[22] M. Pitt and N. Shephard, “Filtering via Simulation: Auxiliary
(with Cl being the covariance matrix of the lth point). Particle Filtering,” J. Am. Statistical Assoc., vol. 94, no. 446, pp. 590-
599, 1999.
[23] L. Sigal, S. Bhatia, S. Roth, M.J. Black, and M. Isard, “Tracking
ACKNOWLEDGMENTS Loose-Limbed People,” Proc. Int’l Conf. Computer Vision and
Pattern Recognition, 2004.
The work of Ioannis Patras is partially supported by the [24] E. Simoncelli, E. Adelson, and D. Heeger, “Probability Distribu-
Engineering and Physical Sciences Research Council, tions of Optical Flow,” Proc. IEEE Int’l Conf. Computer Vision and
research grant EP/G033935/1. The work of Edwin Hancock Pattern Recognition, pp. 310-315, June 1991.
[25] C. Sminchisescu, A. Kanaujia, Z. Li, and D. Metaxas, “Discrimi-
was supported by a Royal Society Wolfson Research Merit native Density Propagation for 3D Human Motion Estimation,”
Award and the EU FET project SIMBAD (213250). Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2005.
PATRAS AND HANCOCK: COUPLED PREDICTION CLASSIFICATION FOR ROBUST VISUAL TRACKING
[26] M. Tipping, “The Relevance Vector Machine,” Advances in Neural
Information Processing Systems, Morgan Kaufmann, 2000.
[27] P. Viola and M. Jones, “Rapid Object Detection Using a Boosted
Cascade of Simple Features,” Proc. IEEE Conf. Computer Vision and
Pattern Recognition, vol. 1, pp. 511-518, 2001.
[28] N. Vlassis and J. Verbeek, “Gaussian Mixture Learning from
Noisy Data,” Technical Report IAS-UVA-04-01, Informatics Inst.,
Univ. of Amsterdam, 2004.
[29] S. Waterhouse, D. MacKay, and T. Robinson, “Bayesian Methods
for Mixtures of Experts,” Advances in Neural Information Processing
Systems, vol. 8, pp. 351-357, MIT Press, 1996.
[30] O. Williams, A. Blake, and R. Cipolla, “Sparse Bayesian Regres-
sion for Efficient Visual Tracking,” IEEE Trans. Pattern Analysis
and Machine Intelligence, vol. 27, no. 8, pp. 1292-1304, Aug. 2005.
[31] Y. Wu, G. Hua, and T. Yu, “Switching Observation Models for
Contour Tracking in Clutter,” Proc. IEEE Conf. Computer Vision and
Pattern Recognition, pp. 295-302, June 2003.
[32] K. Zimmermann, J. Matas, and T. Svoboda, “Tracking by an
Optimal Sequence of Linear Predictors,” IEEE Trans. Pattern
Analysis and Machine Intelligence, vol. 31, no. 4, pp. 677-692, Apr.
2009.
Ioannis (Yiannis) Patras received the BSc and
MSc degrees in computer science from the
Computer Science Department, University of
Crete, Heraklion, Greece, in 1994 and in 1997,
respectively, and the PhD degree from the
Department of Electrical Engineering, Delft
University of Technology, The Netherlands, in
2001. He has been a postdoctoral researcher in
the area of multimedia analysis at the University
of Amsterdam, and a postdoctoral researcher in
the area of vision-based human machine interaction at TU Delft.
Between 2005 and 2007, he was a lecturer in computer vision in the
Department of Computer Science, University of York, United Kingdom.
Since 2007, he has been a lecturer in computer vision in the Department
of Electronic Engineering, Queen Mary, University of London. He is/has
been on the organizing committees of IEEE SMC ’04 and Face and
Gesture Recognition ’08, and was the general chair of WIAMIS ’09. He
is an associate editor of the Image and Vision Computing Journal and
the Journal of Multimedia. His research interests lie in the areas of
computer vision and pattern recognition, with emphasis on motion
analysis and their applications in multimedia data management, multi-
modal human computer interaction, and visual communications.
Currently, he is interested in the analysis of human motion, including
the detection, tracking, and understanding of facial and body gestures.
He is a member of the IEEE and the IEEE Computer Society.
1567
Edwin R. Hancock received the BSc degree in
physics, the PhD degree in high-energy physics,
and the DSc degree from the University of
Durham in 1977, 1981, and 2008, respectively.
From 1981 to 1991, he worked as a researcher
in the fields of high-energy nuclear physics and
pattern recognition at the Rutherford-Appleton
Laboratory (now the Central Research Labora-
tory of the Research Councils). During this
period, he also held adjunct teaching posts at
the University of Surrey and the Open University. In 1991, he moved to
the University of York as a lecturer in the Department of Computer
Science, where he has been held a chair in computer vision since 1998.
He leads a group of some 25 faculty, research staff, and PhD students
working in the areas of computer vision and pattern recognition. His
main research interests are in the use of optimization and probabilistic
methods for high and intermediate-level vision. He is also interested in
the methodology of structural and statistical pattern recognition. He is
currently working on graph matching, shape-from-X, image databases,
and statistical learning theory. His work has found applications in areas
such as radar terrain analysis, seismic section analysis, remote sensing,
and medical imaging. He has published about 135 journal papers and
500 refereed conference publications. He was awarded the Pattern
Recognition Society Medal in 1991 and an Outstanding Paper Award in
1997 by the journal Pattern Recognition. He has also received the best
paper prizes at CAIP ’01, ACCV ’02, ICPR ’06, BMVC ’07, and ICIAP
’09. In 1998, he became a fellow of the International Association for
Pattern Recognition. He is also a fellow of the Institute of Physics, the
Institute of Engineering and Technology, and the British Computer
Society. He has been a member of the editorial boards of the journals
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Pattern Recognition, Computer Vision and Image Understanding, and
Image and Vision Computing. In 2006, he was appointed the founding
editor-in-chief of the IET Computer Vision Journal. He was the
conference chair for BMVC ’94, the track chair for ICPR ’04, and
the area chair at ECCV ’06 and CVPR ’08, and in 1997, he established
the EMMCVPR workshop series. In 2009, he was awarded a Royal
Society Wolfson Research Merit Award.
. For more information on this or any other computing topic,
please visit our Digital Library at www.computer.org/publications/dlib.