Quantitative Color Optical Flow

John Barron
Department of Computer Science, University of Western Ontario
London, Ontario, Canada, N6A 5B7, email: barron@csd.uwo.ca

Reinhard Klette
CITR, University of Auckland, Tamaki Campus, Building 731
Auckland, New Zealand, email: r.klette@auckland.ac.nz

Abstract field smoothing technique described by Schunck [11]. The

We perform a qualitative and quantitative analysis of
various multi-frame color optical flow methods for synthetic
and real panning and zooming image sequences. We show
that optical flow accuracy improvement can be slightly im-
proved if color images are available instead of gray value or
saturation images. We show the usefulness of a directional
regularization constraint for computing optical flow when
the camera motion is known to be panning or zooming.
1. Introduction
$ 
 
This paper is concerned with the qualitative and quanti-
condition number of the integration matrix for the system of
linear equations was used for estimating numerical stability.
Golland and Bruckstein [3] follow the same algebraic
method. They compare a straightforward 3-channel ap-
proach using RGB data with two 2-channel methods, the
first based on normalized RGB values and the second based
on a special hue-saturation definition. The standard opti-
cal flow constraint may be applied to each one of the RGB
quantities, providing an overdetermined system of linear
equations:
# # #
   
% % %
 & 
 '
 
tative performance of multi-frame color optical flow. ( ( (
The literature reporting on optical flow from color im-    $
 '
 !
ages is quite sparse, despite the fact that multispectral ap-
proaches have been suggested already for about twenty
The solution for the optical flow may be found by using a
years. Markandey and Flinchbaugh [8] have proposed a
pseudo-inverse computation:
multispectral approach for optical flow computation. Their
two-sensor proposal is based on solving a system of two lin-  *)+  +-,/.0 + 21
ear equations having both optical flow components as un-
knowns. The equations are the standard optical flow con- where:
straints: 45 #$7 #$8 9 : 45 #$= 9 :




 
 
 % 7 % 8 % =
+3 6 ;  1  6*< ;
( 7 ( 8 ( =

<
where is one image channel, formulated for each channel <
separately and  
 is the image velocity or opti- The matrix + must be non-singular. The smallest eigen-
cal flow. Their experiments use color TV camera data and a value of +>?+ or the condition number of +>@+ can be
combination of infrared and visible imagery. In each experi- used to measure numerical stability, i.e. if the smallest
ment the data was smoothed by convolution with a Gaussian eigenvalue is below a threshold or the condition number is
of   !" before calculation of the derivatives and the cal- above a threshold, then we set  to be undefined at this im-
culated vector fields were post-processed using 10 iterations age location. We implemented Golland and Bruckstein’s
of median filtering, followed by 100 iterations of the vector method with weights for each channel in a standard least

1
squares framework. If we assume a constant flow vector in
a /ACBD/EF/ACBD neighbourhood then +>?+ becomes the
least squares Lucas and Kanade optical flow method [1, 7].
We used AGHB for the Lucas and Kanade results reported
here. If all weights were 0 except for the Y channel (with
value 1) then we would have the standard gray value Lucas
and Kanade calculation [1, 7]. If the R, G and B weights are
1 with all other weights 0 we have color Lucas and Kanade.
Lastly, if AIJ , we have the standard Golland and Bruck-
stein method, where the neighbourhood size is 1.
We recast the standard Horn and Schunck regularization
[1, 4, 5] as the minimization of:
K M P M M MRQTS S S S S S
LMON
7
8
= 7 8 7 8
&
$ GU  V W
G
X!
M
N M
The weights N importance
indicate the attached M of each Figure 1. Panning flow using the Golland and
channel of image information, e.g.,
Y N J
  means channel Bruckstein method on the RGB images.
has absolutely no importance. If all are 0 except the
one corresponding to the Y channel (with value 1) then this S S S S S Sihij
minimization is exactly Horn and Schunck on a gray value 7 8 7 8
image [1, 5]. U  G V
G
egf kl
/m!
M M
MOhta [10] derives optical flow in a least squares
frame-7 Again, as N for the Horn and Schunck regularization, the
work

8 on channel derivatives in color images. Let
 Y and weights indicate the attached importance of each channel
hij
denote the

7[Z and derivatives
 in color channel . For of image information and can be set so as to include/exclude
example could be the derivatives in the hue image. any color channel. kl
/ is a directional constraint that
Ohta assumes that there are AW\] optical flow constraints depends on one knowing whether the camera is panning or
S channels.
for the different S For A^ , it follows S that S

F8 _
=
F=
_8
F=
F7
_7
_= zooming. Because the 3D translational direction is known,
S S S S the direction of flow can be computed everywhere using the
C
7 0
8
8 0
7 and

7 0
8
8 0
7 !
< S < S` standard optical flow equations [6] and this can be used to
0 0
F7
_8 M 0 F
8 <
F7 0
<
Both equations are solvable if   . Let- constrain the minimization. The directional constraint can
Y
ting the error in the -th equation0 be a < , taking 0 the sum of hij j S S j S
be expressed as:
S S
all squared errors for all A equations, calculating its partial k
n  IoX    Or!
derivatives with respect to  and
, and letting these formu-
M we M obtainM [10]M SM M M <qpRp pRp pRp pRp
lae be equal to zero,

7
8 S M
8 S
M =
8M M
S 7
=
  b M
M7b
M 8 M < b
F7 S M
Fb 8
  M M

7 b
8 S M b
7
S M = < b
7 M M
S 8
=

 b
7b
8 < b
7
b8 
 
b b b
where summations are from 1 to< A . This approach and the
generalized inverse method [3, 8] assume constant weights
for all equations. Our implementation allows us to set arbi-
trary weights for each of the nine RGB, HSI and YIQ color
channels. Note that if all weights are zero except for the
Y channel weight being 1, then in a /AcdBDEW/AcdBD
neighbourhood we again have a standard Lucas and Kanade
[1, 7] least squares calculation.
Barron and Klette [2] recast the standard Horn and
Schunck regularization as the minimization of:
K M P M M MRQ S Figure 2. Panning flow using the Ohta
L M N
7
8
= method on the RGB images.

$ 
 Figure 3. Panning flow using the Lucas and Figure 4. Zooming flow using the Lucas and
Kanade method on the RGB images. Kanade method on the RGB images.

j
7
Because sensor motion is only due to translation,  can be and !RBDŽ/ŽŠ/ . For example, to compute
• , we first con-
written as [6]: volve the prefiltering kernel in the dimension, that con-
j
j h S volve the prefiltering kernel on that result in the dimen- 
j Wt   7/w 8yxe{ z sion and finally convolve the differentiation kernel in the
 s  u o     o +cv
V 

dimension on that result.
pRp pRp
 
where    is an image location, u is the relative depth, i.e. 3. Error Measurement
uG v~ |"w | €}w e|"| x , { is the (unknown) sensor translation
|"| z |"| ‚
magnitude, { is the pRp (known)
pƒp sensor direction of translation We report on quantitative errors for optical flow com-
and T„…‡†ˆ‰ are the 3D  coordinates of some 3D moving puted from color synthetic image sequences using relative
j
point projected onto    . Note that since we do not know magnitude error (as a percentage) and angle error (in de-
—˜ —'™
u ,  is only in the direction of the true optical flow grees). – is the correct optical flow and – is the estimated
7w 8Dvector
x
and scaled by the u value used. We can write +-v as a optical flow in the equations below. For magnitude error we
E‹Š matrix: report: š›
h S
h  t — ˜ — ™
7w 8Dx – –S
+-v  s    —'< ˜ EgBDœ^
<  pRp – pRp
h
< š pRp pƒp
where is the effective focal length (determined empiri- while for angle error we report:
cally to be about 40mm for the used digital video camera). —˜ '— ™
Ož”Ÿl m ¡/¢Dr£ o £ m!

2. Differentiation
4. Experimental Results
The choice of the differential kernel is essential to the
accuracy of both image and range flow calculations. We We generated pan and zoom synthetic image sequences
P
used the balanced/matched Q filters for prefiltering and dif- where the correct flow is known. For panning, we simply
ferentiation proposed S by Simoncelli [12]. A simple av- translated an image of the Tamaki campus Computer Sci-
eraging filter Π0  0  Π0 was used to slightly blur the im- ence building to the left by 3 pixels to make each new frame.
ages before prefiltering/differentiation. The prefiltering ker- For zooming, we imaged a poster of downtown Auckland
nel’s coefficients were !"Š/Ž/Ž , r!"i‘“’/’”‘“Ž , r! ‘“Š/’/ , (the surface structure is planar) and then downscaled it with 
!"i‘’’”‘“Ž and r! /Š/Ž/Ž while the differential kernel’s the ImageMagick program  [9] from Ž/’$E¤rBy by ‘ in the
coefficients were  r!ƒBD/ŽŽŠ , !"’/ŽrB , !" , !"’/Ž/B dimension and Š in the dimension for each frame interval
< <
 Synthetic Panning Sequence x S x S
Synthetic Zooming Sequence x x
vR¥¦ ¨v §i© Z  ¨v §i¦ v¸§i©
Method Data Magnitude Error
ŒŒm­® ­ Angle Error
Œm°®Œ ° Method Data Magnitude Error
­ ® S Œ Z¹ Angle Error
Œ °® S °
Horn-Schunck RGB
Œ 0«ªm¬ ­® 0«ªm¬ ­Zªª ¬ ¯ ° ® ¬ Œ±‡²Œ °
Horn-Schunck RGB
0¬ ²­® 0«¯X¬ Œº­ Œ¯m¬ ² °Z ® ªX¬ ²l² °
¬ ³l± Œm­® ³X¬ ´D0 Œm­ Z±X0¬¬"±0«Œ±¯ °° ® ´DS¬ ° ²D¬"0«Œª ­® 0 Œ¬ ± ­ ¬ ±X0 °
Lucas-Kanade RGB
0 ¬"0´
Lucas-Kanade RGB

0«S¯m¬"0 ­® 0«ªm¬ ­ 0«ŒªX¬ ´Œm­® S0 ¬ ªl³ ­ ±m¬ Z±± ° ® Œ °

Barron-Klette RGB Barron-Klette RGB

00l¬ ´² ­® 0«ªm¬ ´m¯ ­ ¬ ± Œ ° ® 00S ¬ ²m± ° 0 ¬ ª ­® S0«³X¬ ´ºSª ­ ªm¬ ¯ °«®k° 0 S ¬ ²mª
Golland-Bruckstein RGB Golland-Bruckstein RGB

0 Z ¬ ´Œm² ­® 0²y¬ ³XŒm0 ­ Œ¯X¬ ± ° ® 0 Z ¬ ²m°± 0²D¬ S¯l³ ­® ªX¬ ²ºª ­ ³mZ ¬ ª‡²S °® S °
Ohta RGB Ohta RGB

0²y¬ Œm³ ­® 0«¯m¬ ´ ­ ¬ ³ª ° ®Œ¯X¬ ²Œ ° ´S ´D¬"S0µ¯ ­® S ´DŒ ¬ ± ­ 0l0¬"0²°® 0 ¬ ³X°0
Horn-Schunck S+Y Horn-Schunck S+Y

¯X¬ Z ­® 0´DSS¬"0µ³ Z ­ 0¬ ´mZ³ ° S ¬ S0«±X¬ 0 ­® S0 ¬ ¯l¯ ­ S¬ ´ Œm° 0S´y¬ ¯
Lucas-Kanade S Lucas-Kanade S
Barron-Klette S+Y
0²y¬ ¯ ­® 0²y¬"Œ 0 ­ S±X¬ ± Œm°® ° Barron-Klette S+Y
SSªX¬ S0 ­® Œ 0¬"0´Z ­ ±m¬ ±S± ° ® °
0«¯m¬ ³ª ­® ¬ Œm0 ­ ³X¬ ´ °® ZZ ±XS ¬ ²m±° ±X¬ ±«² ­® ¬± ­ Z00µ±X¬¬ ³Z ³³° ° ® ® D´ ¬ ³‡² °
Golland-Bruckstein S Golland-Bruckstein S

0«¯m¬ Œmª‡S ²­® 0²y¬ Z³Z ­ ¬ ³ª °® ¬"0² ° S Œ ¬ ª ­® ŒS0«ªX¬ ŒºS±m­0 00¬ ¯ª°
Horn-Schunck Y Horn-Schunck Y
Lucas-Kanade Y
S¯XZ¬"0 ­® 0S0¬ ³ ­ 0S¬ ¯‡´Z ° S¬ ´ Lucas-Kanade Y
³X¬ ¯ ­® 0 S ¬± S ­ ¬± Z ° 0S´y¬ ¯X0
0 ¬ 0 ­® 0²y¬ ´ ­ ±X¬"0² ° ® Œ ° 0¬"0² ­® ±X¬ ª ­ ±m¬ ±Œ± ° ® Œ°
Barron-Klette Y Barron-Klette Y

0l¬ ±‡² ²y¬ ±ª 0 ¬ ³ ¬ ³³

Golland-Bruckstein Y
0²D¬ ³m0 ´D¬ ª 0l0¬ ¯ 0¬ ²
Golland-Bruckstein Y

Table 1. Panning sequence results. Table 2. Zooming sequence results.

to generate each new image. Thus, the second image was and Klette method produces slightly better flows than Horn
the Ž/E…”‘ central block of the first image, expanded to and Schunck.
fit the original size of Ž/’&EcBy . Similarly, the third image We obtained diverging flows for the real panning and
was the central ŽŽ”‘…E¶‘“’ block of the first image, again zooming sequences using Lucas and Kanade, shown in Fig-
expanded to fit the original size of Ž/’·EIBD . The correct ure 6, and for Horn and Schunck, shown in Figure 7. The
diverging flow field has corner velocities having a magni- Horn and Schunck flow is slightly worse than the Lucas and
 
tude of ‘ in the dimension and a magnitude of Š in the Kanade flow. These flows are poor but in the correct general
dimension. Our real image sequence was at Point England direction. Our directional constraint proved to be useful to
on the Tamaki river in Auckland. The scene consists of a obtaining a more feasible flow field for these images, see the
flat lawn in the foreground with water in the background. flow shown in Figure 8. In this case, there is no (or little)
We used UGJŠ!" , fG3B!" and 100 iterations for the Horn angle error, although clearly there is magnitude error.
and Schunck and Barron and Klette results reported here.
Tables 1 and 2 show some quantitative magnitude and 5. Conclusions
angle errors for the synthetic pan and zoom sequences.
Figures 1, 2 and 3 show synthetic panning flow fields while Color optical flow computed via the three color chan-
Figures 4 and 5 show synthetic zooming flow fields. Be- nels (with weights 1.0) seems better than gray value opti-
cause good directional information is available the Barron

Figure 5. Zooming flow using the Barron and Figure 6. Zooming flow using the Lucas and
Klette method on the RGB images. Kanade method on the RGB images.

Figure 7. Zooming flow using the Horn and
Schunck method on the RGB images.
cal flow. The grayvalue results can be obtained from the
colour images using the YIQ weights of 0.299, 0.587 and
0.114. Similarly, weighting the RGB channels equally with
weights of 0.333, 0.333 and 0.333 gives the same flow as
for the I image. The saturation channel (which we origi-
nally thought was the best candidate for adding extra mo-
tion information) actually has a negative effect on optical
flow. Our directional constraint is useful for computing op-
tical flow when the camera motion is known to be panning
or zooming. Overall, Lucas and Kanade RGB flows were
the best.
[10] N. Ohta. Optical flow detection by color images. NEC
[11] B. G. Schunck. Image flow: fundamentals and algo-
[12] E.P. Simoncelli, ”Design of multi-dimensional deriva-
Figure 8. Zooming flow using the Barron and
Klette method on the RGB images.
References
[1] J. L. Barron, D. J. Fleet and S. S. Beauchemin. Per-
formance of optical flow techniques. Int. Journal of
Computer Vision, 12(1), 1994, pp. 43–77.
[2] J.L. Barron and R. Klette, “Experience with Optical
Flow in Color Video Image Sequences”, Image and
Vision Computing’2001 (ICVNZ2001), pp. 195-200.
[3] P. Golland and A. M. Bruckstein. Motion from color.
Center for Intelligent Systems TR #9513, Computer
Science Department, Technion, Haifa, August 1997.
[4] P. Handschack and R. Klette. Quantitative compar-
isons of differential methods for measuring of image
velocity. In Proc. Aspects of Visual Form Processing,
Capri, Italy, World Scientific, 1994, pp. 241 - 250.
[5] B. K. P. Horn and B. G. Schunck. Determining optical
flow. Artificial Intelligence, 17, 1981, pp. 185–204.
[6] H.-C. Longuet-Higgins and K. Prazdny. The interpre-
tation of a moving retinal image. Proc. R. Soc. Lond.,
B 208, 1980, pp. 385–397.
[7] B. D. Lucas and T. Kanade. An iterative image-
registration technique with an application to stereo vi-
sion. DARPA Image Understanding Workshop, 1981,
pp. 121–130 (see also IJCAI’81, pp. 674–679).
[8] V. Markandey, B. E. Flinchbaugh. Multispectral con-
straints for optical flow computation. Proc. 3rd ICCV,
Osaka, Japan, December 1990, pp. 38–41.
[9] D. P. Mitchell and A. N. Netravali, ”Reconstruc-
tion filters in computer graphics:, Computer Graphics,
22(4), 1998, pp:221-228.
Research and Development, (97), 1990, pp. 78–84
(preliminary version: N. Ohta. Optical flow detection
by color images. IEEE Int. Conf. on Image Processing,
Pan Pacific Singapore, pages 801 –805, 1989.)
rithms. In Motion Understanding - Robot and Hu-
man Vision (W.N. Martin and J.K. Aggarwal, eds.),
Kluwer, Amsterdam 1988.
tive filters”, IEEE Int. Conf. Image Processing”, Vol.
1, pp. 790-793, 1994