Efficient Content Extraction In Compressed Images
W. Brent Seales C. J. Yuan
Computer Science Department
University of Kentucky
Lexington, Kentucky 40506
Abstract
I n many contexts it is desireable for users to be able
to extract content directly f r o m image and video data
via queries formed dynamically on-line. Queries over
precomputed features of an image set restrict the user’s
access to what has been computed by an editor or su-
pervisor of the data. This work presents an approach
to extracting patterns from images and video eficiently
in order to support on-line queries directly o n the orig-
inal image data. Our technique relies on compressed-
domain processing, where we perform content extrac-
tion o n image or video data that are compressed with
standard algorithms such as JPEG, M P E G or wavelet
schemes.
1 Introduction
The problem of searching for patterns in images
and video is central to providing users intelligent ac-
cess to large image archives. In the context of on-line
querying of the data, the computational efficiency of
the search is a primary concern, and is traded off with
the accuracy of the search. In this paper we describe
a method for matching patterns over large sets of im-
age data. There are two key features of our approach.
First, we search for patterns in the compressed domain
without attempting to impose a special-purpose (or
domain-dependent) compression scheme. This allows
integration with existing types of standards for com-
pression of large data archives, which are supported by
general-purpose and widely-available hardware. Sec-
ond, we include the parameters of lossy compression
in the search process. These parameters can be con-
trolled by an archive editor or the user, and allows a
measure of control over the tradeoff between search
time and the accuracy of the results. It is often possi-
ble to obtain very accurate query results with signifi-
cantly degraded data.
Researchers have laid an important foundation for
extracting low-level features, performing visual ef-
fects, and doing low-level segmentation (cut detection
$10.00 0 1997 IEEE
0-8186-7981-6/97
Authorized licensed use limited to: K J Somaiya Inst of Engg and Information Tech. Downloaded on March 22,2010 at 02:36:55 EDT from IEEE Xplore. Restrictions apply.
Michael Brown
and bandwidth allocation) in compressed video [18].
The idea of processing data without decompressing
it is attractive and has been explored by researchers
interested in real-time visual effects such as fading
and warping. Smith and Rowe [19] outline the ba-
sic approach for applying image operators to JPEG-
compressed images. They show large speed gains for
certain operations by avoiding decompression. Rao [3]
formulates the equivalent of the convolution theorem
for the DCT. He shows that image operations such as
high-pass filtering can be done by manipulating the
DCT values directly. Shen and Sethi [17] perform edge
detection in the compressed domain, showing that a
coarse edge map can be obtained up to 20 times more
efficiently. In terms of video segmentation and con-
tent analysis, the problem of scene-change, or “cut”
detection, has been solved in the compressed domain
[2]. There are also algorithms for resource bandwidth
allocation [7] which rely on predictions using cues ex-
tracted from compressed video.
The work of the Photobook project by Pentland et
al. [12, 111 uses compression schemes that preserve
semantic content and are thus searchable. The com-
pression algorithms are devised for particular kinds of
imagery (e.g., faces), and within this context the Pho-
tobook system demonstrates excellent performance.
The Photobook project’s central idea of extracting
content from data that has been compressed is a pow-
erful one and follows the trend of operating as much
as possible in the compressed domain. The idea of
“semantic preserving compression” distinguishes our
work and the Photobook project from other image
database schemes which use extracted cues but do not
preserve all the semantics of the image data [9]. Our
work seeks to extract content from data that have been
compressed using standard algorithms such as JPEG
and MPEG, which are lossy but preserve image se-
mantics. This differentiates our formulation from that
of Photobook, where the compression algorithms are
tailored toward the specific content of the image. The
52
 distinction is similar to compression algorithms them-
selves, where greater compression can sometimes be
gained by using a special-purpose algorithm that per-
forms very well for a particular class of images.
In Section 2 we formulate the mathematics of our
approach, relate that to the JPEG/MPEG compres-
sion schemes, and summarize the implied requirements
and constraints. Section 3 brieflly describes the imple-
mentation of a testbed system and then shows exper-
imental results. The results at tempt to quantify the
positive and negative aspects of using our approach in
practice. Finally, Section 4 provides a summary of our
contributions.
2 Mathematical Formulation
The goal is to perform efficient pattern matching in
an archive of data that is compressed. There are two
obvious approaches. First the archive can be stored
in a form that is convenient for the search, either in
a custom format or as a secondary archive where key
features have already been extracted. Second, the raw,
uncompressed pixel data can simply be searched with
classical techniques. We attempt to avoid the first ap-
proach because it may be important to preserve the
data in a canonical form independent of the types of
queries to be performed or the specific content of the
data. It may also be considered restrictive to limit
searches only to a secondary, precomputed set of ex-
tracted features because this potentially restricts the
user’s ability to query the primary data directly. The
second approach, using raw data directly, is sure to be
computationally expensive and may be unsuitable for
on-line querying.
The key idea of this paper is to exploit a reasonable
assumption about the form of the archived data: we
assume that the archived data will be stored in com-
pressed form, using a transform-coded compression
scheme. MPEG, JPEG and wavelet methods are all
transform-coding schemes. This, assumption, in most
cases, is very reasonable for large image and video
archives. Given image data that are already com-
pressed, it is advantageous to process it directly with-
out needlessly expending resources to recover pixel in-
formation. Transforming image (databack to the pixel
domain expands the data size and forfeits some of
advantages that are only present when dealing with
transform-coded data. Note that it is necessary, even-
tually, to recover the pixel information for a small
number images for the purpose of end-user display.
Even so, many more images must be examined in a
typical search, and over that large data set there is
a substantial savings gain in avoiding or delaying the
final decoding until search results are finalized.
53
Authorized licensed use limited to: K J Somaiya Inst of Engg and Information Tech. Downloaded on March 22,2010 at 02:36:55 EDT from IEEE Xplore. Restrictions apply.
Our purpose is to search for patterns which can be
accurately and efficiently extracted from the data in
ils compressed form. We derive a measure of match
from the set of principal components computed from
an image set. It is interesting to apply the princi-
pal component method at a point after part of the
compression transformation, rather than in the pixel
domain. It is straightforward to show that the mea-
sure of match using the “distance in eigenspace” mea-
sure for pattern matching with images is preserved
under linear, orthogonal transformations. This im-
plies that the principal component method gives ex-
actly the same measure of match on transformed data
as on pixel-domain data. Perturbation of the mea-
sure of match occurs as a result of the quantization
of the transformed data, which is the key part of the
loss that is introduced in transform-coded compression
schemes.
We use the principal component method on
transform-coded data and expect exactly the same
quality of classification results as when performing the
principal component method on the pixel data. This
will be true if the transformation is orthogonal and
there is no subsequent quantization of the transformed
data. Lossy coding methods, however, will also intro-
duce perturbations from quantization in the principal
components which are difficult to quantify. But we
expect matching measurements to be much more effi-
cient to perform on the transform-coded data because
(1) the decompression costs are skipped and (2) the
transform vectors are usually sparse as a result of the
quantization and coefficient reordering.
2.1 Principal Components and JPEG
We will use the JPEG standard as an example. We
have applied our approach to wavelet transformations
[6] and to index frames (I-frames) of MPEG video as
well [13]. The JPEG standard consists of the follow-
ing key steps: (1) Pixel range shifting; (2) Discrete
Cosine transform (DCT) on 8 x 8 pixel blocks; (3)
DCT quantization; (4) DCT “zig-zag” ordering; ( 5 )
Run-length coding; and (6) Entropy coding. Applying
all the steps gives a totally-compressed JPEG image.
Motion JPEG is essentially a continuous set of JPEG
frames, and MPEG I-frames are similarly encoded.
Our algorithms operate on run-length-encoded
(RLE) vectors, which is the data after step (5). Very
little can be done with the data after step (6) since
it is no longer byte-aligned. Refer to [20, 81 for more
detail regarding the JPEG and MPEG compression
algorithms.
We develop the method of principal components
[5] by defining the input as a set of IC images {fl,
 f2,. . . ,fk}, represented as vectors, where each image The distance between two vectors XI and xz is
of dimension n x m has N = n x m pixels. We treat an 2
image as a vector by placing the pixel values in scan- 11x1 - x2ll = ( x 1 - X2)'(Xl - x 2 ) (8)
line order. First we subtract from each input vector which becomes
the average value
- xi11 = llX11l2 + llX21l2 - 2 X I . X 2
2
. k
11x1 (9)
Eq. (9) relates the Euclidean distance between two
vectors x1 and xz to their cross-correlation, which is
of the input vector set. the term x 1 - x 2 . When llxlII = IIx211 = 1, Eq. (9)
Now the covariance matrix C of the mean-adjusted becomes
2
input vectors f is 11x1 - x 2 l l = 2(1 - X l . X 2 ) (10)
This shows that the squared distance between two in-
c N ~ N= U U ~ (2)
put vectors is a measure of the correlation between
where the elements of U are written as those vectors as defined by Eq. (7). Vectors that are
close to each other are highly correlated.
[fl -m, f2 -m, ..., fk -mINxk (3) Now suppose that the input to the eigenspance
method is a set of JPEG-compressed images {fl,
The principal components are the eigenvectors and
f2, . . . ,f k } generated from a set of intensity images
eigenvalues of C:
{fl, f 2 , . . . , f k } . w e assume that we have vectors at
&Ai = C Ai (4) step ( 5 ) in the JPEG algorithm. Thus the discrete co-
sine transform (DCT) coefficients represented by the
Although the dimension of C is N x N , its rank data between steps ( 5 ) and ( 6 ) are quantized, are in
is k and hence this decomposition produces IC non- zig-zag (frequency) order, and are run-length encoded
negative eigenvectors {AI,A2, . . . ,A,} and their as- (RLE vector). We perform the inverse quantization
.
sociated eigenvalues {A,, x 2 , . . , &}. w e let A be a directly to the RLE vector, which can be done with-
matrix whose rows are formed from the eigenvectors out run-length decoding or reordering the coefficients.
A%of C ,and we order the rows so that the first row Note that the particular coefficient order is not im-
is the eigenvector corresponding to the largest eigen- portant, only that a consistent choice be made. Our
value and the last row is the eigenvector corresponding performance results in Section 3 show the substantial
to the smallest eigenvalue:

["'I]
A=
Ak
Because C is real and symmetric, the eigenvectors are
complete and orthogonal, forming a basis that spans
the k-dimensional space.
The matrix of ordered eigenvectors A can be viewed
as a matrix that projects input vectors, adjusted by
the mean vector m, into eigenspace:
gains obtained by using dequantized RLE vectors.
The steps in computing the eigenspace for this set of
input vectors are the same as before. Using Eqs. (1)-
(5) ( 5 ) on the set of input vectors {?I, ?2, . .. ,&} we obtain
..
a set of eigenvectors { 9 1 ,9 2 , . ,*k} and their asso-
ciated eigenvalues {$I, $ 2 , . ,$k}. ..
As before (Eq. 5 ) , we form the transformation ma-
trix \-k from the eigenvectors of the covariance matrix:

p = A(f - m) (6) The key observation in dealing with vectors that

Given two vectors that are projected into have been transformed with the DCT is that the DCT
eigenspace, the closer the projections in eigenspace, is a distance-preserving linear transformation:
the more highly correlated the original vectors [lo].
In particular, the normalized cross-correlation ~ 1 be-
2 llfl - fill = Ilfl - 211 (12)
tween two vectors of the same dimension is given by This means that we should be able to match pat-
terns in the transformed data, expecting the same
(7) measure of match to hold.

54

Authorized licensed use limited to: K J Somaiya Inst of Engg and Information Tech. Downloaded on March 22,2010 at 02:36:55 EDT from IEEE Xplore. Restrictions apply.
 2.2 Quantization When IIE112/11C1/2is small we can bound the change in
The quantization step of mlost compression algo- the eigenvectors and eigenvalues of C. In particular,
rithms gives the largest reductions in data size. But since both C and E are symmetric, according to the
quantization causes data loss anld introduces errors be- Weilandt-Hoffman theorem [4]
tween a compressed image and the original. As noted
N
earlier, in the JPEG algorithm the DCT values are
quantized to compress the data. This quantization
i=l
is lossy and the question we rnust address is “HOW
does progressively larger, lossy quantization affect the where
N
structure of the eigenspace thait will be used to clas-
sify images?”. We want the structure of eigenspace IIEII=
F~ Ieij12 (21)
i,j=l
to be perturbed slowly and smoothly as a function of
increasing loss due to quantization. In this section, we and Xi is the ith eigenvalue of C and X i is the ith eigen-
show that there are tight bounds on how the eigenval- value of C. Thus the difference between the eigenval-
ues and eigenvectors of the covariance matrix C will ues of C and C is bounded by the order of the largest
change when we introduce perturbations due to quan- element of E, while the elements of E are bounded
tization. We have verified these bounds with empirical directly by the JPEG quantization matrix and scale
results [13]. factor.
Let the input to the classification method be a set The change in the eigenvectors of C is also bounded
of compressed, mean-adjusted images {fl, f2, . . . fk} by the magnitude of E. In particular, the angular
generated from a set of intensity images { f1, f2, . . . ,fk} difference between the perturbed unit eigenvector Ai
of dimension N = n x m. Quantization introduces from C and the eigenvector Ai from C satisfies the
errors into the vectors { f ~f,2 1 . .. ,f k } , and we model expression [11
the error in each element of f as
fqAi,Ai) 5 c
I1E112
-
SaPi
where c is a small constant, 1!9(Ai,Ai) is the angle be-
tween & and Ai in radians, and

gap; = min IXi - X j I (23)

Now we write the covariance matrix as i#j

is the absolute value of the gap between X i from C

and its nearest neighboring eigenvalue.
where U is written as When the case of duplicity of eigenvalues arises,
the expression for the bound on the perturbation in
eigenvectors is similar, and measures the change be-
tween subspaces rather than the angle between single
and each vector 6i represents the perturbation intro- vectors.
&
duced into as a result of quantization. We now let In summary, the change in the structure of
eigenspace as a function of the perturbations intro-
ANxk =[61 62 6k] (16) duced by quantization is strongly bounded and well-
behaved. The bounds are a result of the form of C,
so that which is a symmetric covariance matrix. These error
bounds guarantee that distance in eigenspace, which
C = UUT = (U + A)(UT+ A T ) (17) is the basis for classification, will be perturbed slowly
and will yield good classification results over a signif-
This expands to
icant range of quantizations.
C = UUT + UAT + AUT + AAT (18)
These results imply that adjusting the quantization
factor of compressed images allows the data size to be
which can be rewritten as reduced to a point that optimizes compression ratio
and classification rate. The ability to tune the data
c + ( U A +~ A U +~ AA,*) = c +E (19) loss versus classification performance is potentially

55

Authorized licensed use limited to: K J Somaiya Inst of Engg and Information Tech. Downloaded on March 22,2010 at 02:36:55 EDT from IEEE Xplore. Restrictions apply.
 Figure 1: (top) This page of the Beowulf manuscript (folio 133 recto) was used as a training set for the Old
English letter forms “d” “P, “ea”, and “m”. Using these training images we detect letter forms on other pages
of the manuscript. (bottom) The result of the detection of the word “dream” as a combination of its individual
letters (folio 130 verso).

56

Authorized licensed use limited to: K J Somaiya Inst of Engg and Information Tech. Downloaded on March 22,2010 at 02:36:55 EDT from IEEE Xplore. Restrictions apply.
 very valuable. As a result, data stored in archives
at fairly high quantization rates is likely to contain
enough information to allow very efficient searching
and matching with large gains in performance over
the same operations in the pixel domain. In addition,
even at low compression rates it is still more efficient
to operate in the compressed domain because classi-
fication distances are equivalent,, decompression can
be avoided, and sparse data consumes less disk and
memory.
3 Experimental Analysis
A prototype tool has been developed which imple-
ments the methods described in this paper. The tool
reads JPEG images or MPEG frames and partially de-
compresses them, quantizes as dlesired and allows the
principal component method to be applied at various
places in the decompression pipeline. We are using
the tool to examine a variety of ,applications [16, 151.
In the experiment shown in Fig. 1, we use high
resolution images of the Beowulf manuscript supplied
by the British Library [14]. The goal is to detect letter
forms and use the combination of letter forms in order
to extract arbitrary words and phrases across the 77-
page manuscript. Each page of the manuscript has
been digitized to a high resolution of detail, creating
a very large digital archive (approximately 14 MB per
page).
The example in Fig. 1 creates a training set for the
letter forms “d” “r”, “ea”, and “m” in order to search
for the word “dream”. The top image was used as a
template base for constructing the training sets. The
bottom image shows the result of the search. Twenty
eigenvectors were used for each template. Fig. 2 gives
the time savings as a function of the JPEG quality
factor of the data. The recognition task runs twice
as fast, giving the same classificaiion results as when
applied to the raw (uncompressed) data. In particu-
lar for the quality values shown in Fig. 2 classifica-
tion results were identical. This supports the mathe-
matical discussion in Section 2 which argues that the
eigenspace distance measures are robust over a large
range of perturbations from quantization.
In the case of this example, decompression time
is only a small part of the total computation time,
which is dominated by the search and projection into
eigenspace. In cases where a much smaller number
of templates from the image are to be projected into
eigenspace, such as tracking in video frames, the de-
compression time plays a much larger role and begins
to dominate the cost. We have found in such cases
that the computation times can be up to five times
faster when operating on compressed data.
57
Authorized licensed use limited to: K J Somaiya Inst of Engg and Information Tech. Downloaded on March 22,2010 at 02:36:55 EDT from IEEE Xplore. Restrictions apply.
4 Summary
In this paper we have shown that significant
speedups are possible in solving the problem of match-
ing patterns in compressed image data. The speedups
are available by operating on the data while it is com-
pressed. Without errors from quantization, which is
involved in most lossy-coding schemes, the classifi-
cation results are exact. But even with significant
levels of quantization, the eigenspace which serves
as the classifier remains robust and performs well.
When the eigenspace projection dominates the com-
putation time, as is the case when a small template is
matched at all locations in a large image, we achieve a
speedup of a factor of 2 . In other experiments where
a small number of image templates from each com-
pressed image are projected to eigenspace, the decom-
pression time plays a significant role as well and leads
to speedups on the order of a factor of five.
Acknowledgments
We gratefully acknowledge the support of the Na-
tional Science Foundation for this research under
grants IRI-9308415, CDA-9320179 and CDA-9502645.
References
[l] E. Anderson, Z. Bai, C. Bischof, J. Demmel,
J. Dongarra, J. Du Croz, A. Greenbaum, S. Ham-
marling, A. McKenney, S. Ostrouchov, and
D. Sorensen. LAPACK Users’ Guide, Second
Edition. Society for Industrial and Applied Math-
ematics (SIAM), Philadelphia, PA, 1995.
[2] F. Arman, A. Hsu, and M.-Y. Chiu. Image
processing on compressed data for large video
databases. In Proceedings of the First ACM Inter-
national Conference on Multimedia, August 1993.
[3] B. Chitprasert and K. Rao. Discrete cosine trans-
form filtering. Signal Processing, 19(3):233-245,
1990.
[4] G. Golub and C. Van Loan. Matrix Computation.
The Johns Hopkins University Press, 1984.
[5] R.C. Gonzales and R.E. Woods. Digital Image
Processing. Addison-Wesley, 1993.
[6] W. Hu and W. B. Seales. Biorthogonal wavelets
and object recognition in the compressed domain.
Technical report, Computer Science Dept., Uni-
versity of Kentucky, Lexington, Kentucky, 1997.
[7] K. Lakshman and R. Yavatkar. An empirical eval-
uation of adaptive qos renegotiation in atm net-
works. In Proc. 6th Intl. Workshop on Network
 Decompression Searching JPEG Quality Method Size
Time (sec) Time (sec) Factor (bytes)
5 6190 2 RLE 9393
6 6207 3 RLE 10672
6 6273 5 RLE 15084
7 6383 10 RLE 26741
8 6534 15 RLE 36612
9 6669 20 RLE 48171
10 6797 25 RLE 63328
19 7705 50 RLE 129601
24 8103 75 RLE 174351
66 11732 75 RAW 1571328

Figure 2: The timing results for the letter-form search of a page of the Beowulf manuscript show that search
times are up to twice as fast using the compressed (RLE) data.

and Operating System Support for Digital Audio
and Video, April 1996.
[8] D. Le Gall. MPEG: A video compression stan-
dard for multimedia applications. Communica-
tions of the ACM, 34(4):47-58, April 1991.
[9] D. Lee, M. Flickner, R. Barber, J. Hafner,
W. Niblack, and D. Petkovic. Indexing for
complex queries on a query-by-content image
database. In Proc. Int. Conf. Pattern Recogni-
tion, pages 142-146, 1994.
[lo] H. Murase and S. Nayar. Visual learning and
recognition of 3-D objects from appearance. In-
ternational Journal of Computer Vision, 14:5-24,
1995.
[Ill A. Pentland, B. Moghaddam, and T. Starner.
View-based and modular eigenspaces for face
recognition. In Proc. IEEE Computer Vision and
Pattern Recog., 1994.
[12] A. Pentland, R.W. Picard, and S. Scarloff. Pho-
tobook: Content-based manipulation of image
databases. Int. Journal of Computer Vision, to
appear, 1996.
[13] W.B. Seales, M.D. Cutts, and W. Hu. Content
analysis of compressed video. Technical Report
265-96, Computer Science Dept., University of
Kentucky, Lexington, Kentucky, 1996.
[14] W.B. Seales, J. Griffioen, and R. Yavatkar.
Content-based multimedia data management and
remote access. Journal of the Association for
Computers and the Humanities, 1997.
[15] W.B. Seales, J. Lumpp, and M. Brown. Dis-
tributed wireless stereo reconstruction. In IEEE
Aerospace Applicataons Conference, 1997.
[16] W.B. Seales, J. Lumpp, and C.J. Yuan. Model-
based tracking in compressed video. In IEEE
Aerospace Applications Conference, 1997.
[17] B. Shen and I.K. Sethi. Direct feature extrac-
tion from compressed images. In Proc. Interna-
tional Society f o r Optical Engr. (SPIE), Storage
and Retrieval for Image and Video Databases IV,
October 1996.
[18] B. Smith. Survey of compressed domain process-
ing techniques. In Reconnecting Science and Hu-
manities in Digital Libraries, a Digital Library
Symposium at the Universitg of Kentucky, Oct
1995.
[19] B. Smith and L. Rowe. Algorithms for manipulat-
ing compressed images. IEEE Computer Graph-
ics and Applications, 13(5):34-42, Sept 1993.
[20] G. Wallace. The JPEG still-picture compres-
sion standard. Communications of the ACM,
34(4):31-44, April 1991.

58

Authorized licensed use limited to: K J Somaiya Inst of Engg and Information Tech. Downloaded on March 22,2010 at 02:36:55 EDT from IEEE Xplore. Restrictions apply.