Matrix Principal Component Analysis for Image Compression and Recognition
Kohei Inoue, Kenji Hara and Kiichi Urahama
Department of Visual Communication Design
Kyushu University
Fukuoka-shi, 815-8540 Japan
Abstract
Vector data are mapped into low-dimensional vector spaces
with one orthonormal matrix in the principal component
analysis (PCA), which has been extended to 2DPCA in
which matrix data are mapped into low-dimensional ma-
trix spaces with one orthonormal matrix. We present a ma-
trix principal component analysis (MPCA) method in which
matrix data are mapped into low-dimensional matrix spaces
with two orthonormal matrices, which are derived from the
higher-order singular value decomposition of a third-order
tensor as an approximate solution of simultaneous dimen-
sionality reduction of matrices. The MPCA method is ap-
plied to image compression and recognition of face images
where it outperforms the PCA and the 2DPCA in compres-
sion accuracy, recognition speed and classification rates. In
addition to improvement in recognition, this method is also
useful for speeding up dimensionality reduction of learning
data. We examine this by using it as a pre-processor for the
PCA together with its extension to third-order tensors for
similarity search of images with their color histograms.
1 Introduction
This paper is addressed to dimensionality reduction of data
given by a form of matrices, of which typical example is
image data. A popular technique used for it is the prin-
cipal component analysis (PCA) as is used in the eigen-
face method[1]. The PCA, however, deals with data given
by a form of vectors, hence its application to matrix data
needs unfolding of matrices into vector forms which are
then mapped into low-dimensional vector spaces with one
orthonormal matrix which minimizes the errors of recon-
structed vectors from the original ones. Unfolding of ma-
trices into vectors, unfortunately, disorders neighborhood
relations in their elements, hence in the PCA the correlation
between data vectors are utilized for compression, but that
between elements in each data is not exploited.
We should deal with data given by a matrix form in-
tactly in order for reducing redundancies from correlations
between elements as well as those between data. Yang et
al.[2] have presented the two-dimensional PCA (2DPCA)
which reduces data matrices into those with smaller size by
multiplying one orthonormal matrix to them, which mini-
mizes the errors of reconstructed matrices from the original
data. Its computational time for mapping data is shorter
than the PCA. The 2DPCA, however, exploits correlations
only between columns of data matrices, hence its compres-
sion capability is insufficient.
If we extend the 2DPCA to exploiting both of correla-
tions between columns and those between rows in data ma-
trices, we arrive at a mapping with two orthonormal ma-
trices multiplied to data matrices from both sides in simi-
lar form of the singular value decomposition. One of such
forms of simultaneous low-rank approximation of matri-
ces has been presented by Pesquet-Popescu et al.[3] and
Shashua et al.[4]. Their algorithms, however, do not nec-
essarily ensure orthogonality between basis vectors in map-
ping matrices, hence cannot be used for image recognition
except for use only for image compression. An algorithm
which gives orthonormal mapping matrices has been pre-
sented by Ye et al.[5]. Their algorithm is, however, an
iterative solution method, hence demands long computa-
tional time especially for large size of data. They called
their algorithm the generalized principal component analy-
sis (GPCA)[5], however, this term is confusing because it
has also been used for other types of generalization of the
PCA such as [6]. We call it Ye’s-GPCA in this paper.
We present, as an alternative to Ye’s-GPCA, a direct so-
lution method for the simultaneous low-rank approximation
of multiple matrices. Our method is based on an analyti-
cal formulation of the higher-order singular value decom-
position for dimensionality reduction of a single third-order
tensor. Our algorithm is a direct solution method in con-
trast to an iterative one by Ye et al., hence is faster than
their algorithm. We call this algorithm the matrix principal
component analysis (MPCA). Though the term MPCA has
also been used as the multi-way PCA which is an extension
of the PCA to tensors[7], this terminology appears rarely,
hence the MPCA is less confusing than the GPCA.
 We apply the MPCA to compression and recognition 2.2 Two-Dimensional PCA
of face images and show that the MPCA outperforms the
2DPCA which outperforms the PCA in the compression ac- In the PCA described above, data matrices are transformed
curacy, recognition speed and classification rates. to vectors. We next treat

nthem in their intact matrix forms.
In addition to its high recognition speed, the MPCA At first, the mean Ā = k=1 Ak /n is subtracted from each
is also useful for speeding up dimensionality reduction of Ak as Bk = Ak − Ā. This centered matix is projected to
learning data. We show this by using it as a pre-processor Ck = Bk V with a q × r matrix V which is orthonormal:
for the PCA together with it extension to third-order tensor V T V = Ir . Reconstruction from this projection becomes
(rec)
data for similarity search of images with color histograms. Ak = Ā + Ck V T = Ā + Bk V V T . Hence the projection
(rec)
matrix V minimizing the reconstruction error
Ak −Ak

is given by
2 Simultaneous Low-Dimensional
n

Approximation of Multiple Matri- V∗ = arg min
Bk − Bk V V T
2F (4)
V
ces k=1

where
•
F is the Frobenius norm of a matrix. Since

Let us be given n p × q matrices Ak (k = 1, ..., n). We con-
sider linear dimensionality reduction methods which mini- n

mize the errors of matrices reconstructed from compressed
Bk − Bk V V T
2F
ones from their original data. k=1
n

= tr[(Bk − Bk V V T )(BkT − V V T BkT )]
2.1 Principal Component Analysis
k=1
It is needed in the PCA that each data matrix is unfolded n n

into a vector ak of length pq. They are preliminarily cen- = tr(Bk BkT ) − tr(V T BkT Bk V ), (5)

n
tered by subtraction of their mean ā = k=1 ak /n from
k=1 k=1

them as bk = ak − ā. The centered vector bk is pro-

eq.(4) is equivalent to
jected to ck = U T bk with a pq × r matrix U which is
orthonormal: U T U = Ir where Ir is the r × r identity n

matrix. Reconstruction from this projection is given by V∗ = arg max tr(V T BkT Bk V ) (6)
(rec) V
ak = ā + U ck = ā + U U T bk . Hence the projection k=1
(rec)
matrix U minimizing the reconstruction error
ak − ak
The solution V∗ is the matrix whose columns

n are the first r
is given by eigenvectors vj of the covariance matrix k=1 BkT Bk , i.e.
n
 V∗ = [v1 , ..., vr ].
U∗ = arg min
bk − U U T bk
2 (1) Yang et al. at first called this method IMPCA (im-
U
k=1 age PCA)[8], but next renamed it 2DPCA (2-dimensional
where
•
is the Euclidean norm of a vector. Since PCA)[2].
n


bk − U U T bk
2 2.3 Simultaneous Singular Value Decomposi-
k=1
n
tion of Matrices

= tr[(bk − U U T bk )(bTk − bTk U U T )] In the above 2DPCA, only the number of columns in data
k=1 matrices is reduced while their rows keep intact. This sug-
n n
 gests another projection of the form Ck = U T Bk which re-
= tr(bk bTk ) − tr(U T bk bTk U ) (2)
duces only rows leaving columns intact. Thus the 2DPCA
k=1 k=1
is partial and this observation leads to combination of these
where tr(•) is the trace of a matrix, eq.(1) is equivalent to two projections to reduce rows and columns at once with
n
 the form U T Bk V .

U∗ = arg max tr(U T bk bTk U ) (3) We firstly subtract from Ak their mean Ā = nk=1 Ak /n
U
k=1 as Bk = Ak − Ā and this centered matrices are projected to
The solution U∗ is the matrix whose columns Ck = U T Bk V with a p × r1 matrix U and a q × r2 matrix

are the first
r eigenvectors uj of the covariance matrix nk=1 bk bTk , i.e. V where U and V are orthonormal: U T U = Ir1 , V T V =
(rec)
U∗ = [u1 , ..., ur ]. Ir2 . Reconstruction from this projection is Ak = Ā +
U Ck V T hence the projection matrices U and V minimizing
(rec)
the reconstruction error
Ak − Ak
are given by
n
 in X except for the first r1 ones and denoting the remain-
{U∗ , V∗ } = arg min
Bk − U Ck V T
2F (7)
U,V
k=1
This is an extension of the singular value decomposition
(SVD) to multiple matrices. In the SVD, center matrix Ck
is diagonal, while in eq.(7), it is a dense matrix. In con-
trast to the PCA and the 2DPCA, we cannot obtain the so-
lution of eq.(7) analytically. Ye et al.[5] presented a method
solving eq.(7) numerically with an iterative scheme which,
however, converges to only a local optimal solution and de-
mands long computational time especially for large size of
data. Ye et al. called this iterative algorithm the GPCA
which is confusing as is described above, hence we refer it
Ye’s-GPCA in this paper.
3 Approximate Solution Based on
Higher-Order SVD
We derive an approximate solution of eq.(7) analytically
from the higher-order SVD (HOSVD)[9]. Before it, we
summarize the HOSVD briefly.
3.1 Higher-Order SVD
The higher-order SVD (HOSVD) is an extension of the
SVD to tensors and is defined for any order of tensors, in
which the third-order is relevant to eq.(7).
If we pile up n p × q matrices Bk , we get a third-order
tensor B = [bijk ] (i = 1, ..., p; j = 1, ..., q; k = 1, ..., n).
Before we define the HOSVD for this tensor, we define
some terminologies. At first
[Definition 1] A tensor composed of piled-up matrices
can be decomposed into constituent matrices which are
then laid end to end to construct a matrix long sideways.
This oblong matrix and also this operation is called un-
folding and denoted by B(1) = [B1 , ..., Bn ], B(2) =
[B1T , ..., BnT ], B(3) = [H1T , ..., HpT ]; Hi = [bijk ](j =
1, ..., q; k = 1, ..., n). Conversely the operation restoring
B from B(1) , B(2) , B(3) is called folding.
[Definition 2] A tensor obtained by folding of XS(1) which
is the product of two matrices X and S(1) which is an un-
folding of a tensor S is denoted by S ×1 X and is called the
product of the tensor S and the matrix X. This multiplica-
tion is commutative: S ×1 X ×2 Y = S ×2 Y ×1 X.
The HOSVD is defined by using this notation as:
[Property 1] An arbitrary third-order tensor with size p×q×
n can be decomposed as B = T ×1 X ×2 Y ×3 Z where T is
a third-order core tensor with size p × q × n and X, Y, Z are
matrices with size p × p, q × q, n × n. This decomposition
is called the HOSVD[9].
We can reduce the rank of tensors by discarding columns
ing p × r1 matrix as U , similarly denoting the first q × r2
submatrix of Y as V , and also denoting the first n × r3
submatrix of Z as W . Then the tensor B is shrunk to
B̃ = S ×1 U ×2 V ×3 W where S is a third-order core
tensor of the size r1 × r2 × r3 . This B̃ is called a truncated
HOSVD which is called the HOSVD shortly in the sequel.
This (truncated) HOSVD B̃ satisfies the following property:
[Property

p 2] There
qholds the
n
B −
inequality B̃
2 ≤
2 2 2
i=r1 +1 σi(1) + j=r2 +1 σj(2) + k=r3 +1 σk(3) where
σi(l) (l = 1, 2, 3) are three-mode singular values[9].
This property 2 is formally an extension of the error
bound of the SVD. The SVD is, however, the minimum er-
ror solution, while the HOSVD is not necessarily so. Nev-
ertheless it is practically a good approximation of the min-
imum error solution and coincides with it in many well-
posed tensors[9]. Thus the HOSVD is a good approximate
solution of
min
B − S ×1 U ×2 V ×3 W
2F
U,V,W,S
subj.to U T U = Ir1 , V T V = Ir2 , W T W = Ir3 (8)
A practical procedure for computation of the HOSVD of
a tensor B is
(1) We unfold B into three matrices B(1) , B(2) , B(3) .
(2) We compute their first r1 , r2 , r3 left singular vectors and
array them into matrices U, V, W .
(3) We compute the core tensor S by S = B ×1 U T ×2
V T ×3 W T .
3.2 Matrix PCA
We can derive an analytical approximate solution of eq.(7)
from the above HOSVD. Our derivation is based on the fol-
lowing relation:

B − S ×1 U ×2 V ×3 W
2F
=
B − C ×1 U ×2 V
2F
n
=
Bk − U Ck V T
2F (9)
k=1
where C = S ×3 W and Ck = [ci,j,k ](i = 1, ..., r1 , j =
1, ..., r2 ) with C = [ci,j,k ] (i = 1, ..., r1 ; j = 1, ..., r2 ; k =
1, ..., n).
The last term in eq.(9) coincides with the objective func-
tion in eq.(7). From this observation, we know
[Property 3] Two matrices U and V derived from the
HOSVD are approximate solutions of eq.(7).
Hence we can approximately solve eq.(7) analytically
by:

20
Ye's-GPCA
MPCA
time (sec)
10
0 0
302 602 902 1202
image size
Figure 1: Time for computation of U and V .
(1) unfolding B to B(1) , computing its first r1 left singular
vectors and arraying them to a matrix U , and next
(2) unfolding B to B(2) , computing its first r2 left singular
vectors and arraying them to a matrix V .
This is a direct solution method in contrast to an iterative
algorithm Ye’s-GPCA. We call this direct method the matrix
PCA (MPCA).
3.3 Comparison of Speed with Ye’s-GPCA
The direct solution method MPCA is expected to be faster
than the iterative algorithm Ye’s-GPCA. We measured the
time for computing U and V for 200 monochromatic square
images, i.e. p = q in this example. We set the lower dimen-
sions as r1 = r2 = p. The computational time is illustrated
in Fig.1 where the horizontal axis is the image size p2 . Our
MPCA is faster than Ye’s-GPCA. Obtained U and V almost
coincide in both methods MPCA and Ye’s-GPCA, i.e. in
this example, almost optimal solution is obtained with both
of Ye’s-GPCA and MPCA.
4 Face Image Compression
Recognition
We examined the performance of the MPCA in compari-
son with the PCA and the 2DPCA for face image compres-
sion and recognition. We used the ORL database of faces
(http://www.uk.research.att.com/facedatabase.html) which
is composed of 40 persons with 10 images each, totally 400
face images with various illumination and expressions. Im-
age size is 92 × 75.
4.1 Compression of Face Images
The relationship between the compression rates and recon-
struction errors is plotted in Fig.2 for the PCA, the 2DPCA
and the MPCA. The compression rate is 1 − (pq + pqr +
nr)/npq in the PCA, 1−(pq+qr+npr)/npq in the 2DPCA
and 1 − (pq + pr1 + qr2 + nr1 r2 )/npq in the MPCA (in this
30
PCA
2DPCA
MPCA
20
error
10
1502 0 0.2 0.4 0.6 0.8 1
compression ratio
Figure 2: Relation between compression rate and recon-
struction error
Figure 3: Reconstructed images
definition, the rate is high as data-size after compression is
small). If the compression rate is the same, the reconstruc-
tion error is small in the order MPCA<2DPCA<PCA, i.e.
and the MPCA can compress images highly under a same error
tolerance.
We confirm this result with reconstructed images shown
in Fig.3 where five images in the top row are the im-
ages reconstructed from the PCA compression with r =
5, 10, 15, 20, 25. The numbers in the parentheses are the
compression rates (nearer to one in higher compression).
Five images in the middle row are images reconstructed
from the 2DPCA with r = 5, 10, 15, 20, 25 and those in the
bottom row are images reconstructed from the MPCA with
r1 = r2 = 5, 10, 15, 20, 25. As is shown in Fig.3, good
reconstruction is obtained even if the compression rate is
high in the MPCA. In the 2DPCA, as was reported in [2, 8],
horizontal blur is remarkable owing to its compression only
along rows as is seen in the left image in the middle row
in Fig.3. In contrast, the MPCA compresses images along
both directions in rows and columns hence such directional
blur is unseen in its reconstructed images.

0.8
recognition rate 0.6
0.4 PCA
2DPCA
MPCA
0.2
0.005 0.01 0.015
time (sec)
Figure 4: Relation between time and rate of classification
4.2 Face Recognition
The capability of high compression rates in the MPCA is
expected to be valuable for face recognition with reduction
of computational cost of classification. We examined face
recognition by the MPCA by using the same 400 face im-
ages as above, which are divided in half into 200 learning
data and 200 test data. The relationship between the classifi-
cation time and the recognition rate is shown in Fig.4 where
the horizontal axis is an average classification time per one
datum and the vertical axis is the average recognition rate.
The PCA is plotted with the dotted line, the broken line
denotes the 2DPCA and the solid line is the MPCA. The
breadth of the lines is narrow in the 2DPCA and the MPCA
because their classification time is short even at their lowest
compression rates. The recognition rates are high in the or-
der PCA<2DPCA<MPCA with the computational time in
the order PCA>2DPCA>MPCA.
5 Use as Preprocessor for PCA and
Extension to Third-Order Tensors
We showed in the above section that the MPCA is useful for
speeding up the computation in the recognition phase. It is
also useful for speeding up the computation in the learning
phase. In the PCA, data are unfolded into vectors whose
high dimensionality demands long computational time in
the eigenvector decomposition and their projection. We can
compress this computational time by reducing preliminar-
ily the dimension of data with the MPCA and applying the
PCA to those medium dimensionality data. We examine
this use of the MPCA as a pre-processor for the PCA to-
gether with its extension to third-order tensor data.
5.1 Tensor PCA
Let us be given n third-order tensors Bk (k = 1, ..., n) of
size p1 × p2 × p3 with their elements Bk = [bi1 i2 i3 k ](i1 =
1, ..., p1 , i2 = 1, ..., p2 , i3 = 1, ..., p3 ). We project each
Bk into a lower-rank third-order tensor Ck as Ck = Bk ×1
U T ×2 V T ×3 W T from which Bk is approximately recon-
structed as B̃k = Ck ×1 U ×2 V ×3 W where U, V, W are
p1 × r1 , p2 × r2 , p3 × r3 matrices and Ck is a r1 × r2 × r3
third-order core tensor. The simultaneous low-rank approx-
imation problem similar to eqs.(1) and (7) is written in this
case as
 n
min
Bk − Ck ×1 U ×2 V ×3 W
2F
U,V,W,Ck
k=1
subj.to U U = Ir1 , V T V = Ir2 , W T W = Ir3 (10)
T
Similarly to eq.(7), we can get an approximate solution
of eq.(10) from the HOSVD of the fourth-order tensor B =
[bi1 i2 i3 k ](i1 = 1, ..., p1 , i2 = 1, ..., p2 , i3 = 1, ..., p3 , k =
1, ..., n), whose HOSVD is an approximate solution of
min
B − S ×1 U ×2 V ×3 W ×4 X
2F
U,V,W,X,S
subj.to U T U = Ir1 , V T V = Ir2 ,
W T W = Ir3 X T X = Ir4 (11)
where S is a r1 × r2 × r3 × r4 core tensor. If we set r4 = n,
then the objective function of eq.(11) is transformed as

B − S ×1 U ×2 V ×3 W ×4 X
2F
=
B − C ×1 U ×2 V ×3 W
2F
n

=
Bk − Ck ×1 U ×2 V ×3 W
2F (12)
k=1
where C = S ×4 X is a fourth-order core tensor of size
r1 × r2 × r3 × n. If we denote C = [ci1 i2 i3 k ](i1 =
1, ..., r1 , i2 = 1, ..., r2 , i3 = 1, ..., r3 , k = 1, ..., n), then
Ck = [ci1 i2 i3 k ](i1 = 1, ..., r1 , i2 = 1, ..., r2 , i3 = 1, ..., r3 ).
The last term in eq.(12) is the same as the objective func-
tion of eq.(10). Hence we can get an approximate solution
of eq.(10) by the following procedure:
(1) We pile up Bk into a fourth-order tensor B.
(2) We unfold B into B(1) and compute its first r1 singular
vectors and array them to a matrix U .
(3) We unfold B into B(2) and compute its first r2 singular
vectors and array them to a matrix V .
(4) We unfold B into B(3) and compute its first r3 singular
vectors and array them to a matrix W .
(5) We compute Ck = Bk ×1 U T ×2 V T ×3 W T (k =
1, ..., n).
We call this algorithm a third-order tensor PCA (TPCA).
The first-order TPCA is a vector PCA which is the basic
PCA in section 2.1 and the second-order TPCA is the matrix
PCA (MPCA) is section 3.2.
5.2 Image Retrieval
We examined an application of the TPCA as a pre-processor
for speeding up the reduction of dimensions of database

0.8 PCA
TPCA+PCA
time (sec)
0.6
0.4
0 20 40
dimension
Figure 5: Computational time for dimensionality reduction
of data
1
0.8
precision
PCA
TPCA+PCA
0.6
0.4
0 20 40
dimension
Figure 6: Precision of retrieval of color images
data in an image retrieval method using the PCA[10]. We
experimented similarity search in a database of 512 color
images whose RGB color histograms are represented by
8 × 8 × 8 third-order tensors. We reduced preliminarily
those tensors to 4 × 4 × 4 tensors by the TPCA, which are
then unfolded into 64 dimensionality vectors and projected
into lower dimensionality vectors with the PCA. Computa-
tional times for dimensionality reduction of entire data is
plotted in Fig.5 where the x-axis denotes the final dimen-
sion of data. The broken line denotes direct application of
the PCA and the solid line denotes the combination of the
TPCA and the PCA. The precision of the retrieval with vec-
tors reduced with TPCA+PCA was almost the same as that
with PCA as is shown in Fig.6. The precision was evaluated
by the proportion of the number of retrieved images by us-
ing compressed histograms coinciding with those retrieved
with the original color histograms.
6 Conclusion
We have presented a direct solution method MPCA for si-
multaneously reducing the dimensionality of multiple ma-
trices. We have shown that the MPCA is faster than the
iterative algorithm Ye’s-GPCA, and its rates of image com-
pression and classification are higher than those of the PCA
and the 2DPCA. We extended the MPCA to third-order ten-
sor data and used it as a pre-processor for the dimensionality
reduction with the PCA and applied it to similarity search
of images with their color histograms.
References
60
[1] M. Turk and A. Pentland, “Eigenfaces for recognition,” J.
Cog. Neurosci., vol. 3, pp. 71-86, 1991.
[2] J. Yang, D. Zhang, A. F. Frangi and J.-Y. Yang, “Two-
dimensional PCA: a new approach to appearance-based face
representation and recognition,” IEEE Trans. PAMI, vol. 26,
no. 1, pp. 131-137, Jan. 2004.
[3] B. Pesquet-Popescu, J. Pequet and A. P. Petroupulu, “Joint
singular value decomposition - A new tool for separable
representation of images,” Proc. ICIP, vol. 2, pp. 569-572,
Greece, Oct. 2001.
[4] A. Shashua and A. Levin, “Linear image coding for regres-
sion and classification using tensor-rank principle,” Proc.
CVPR, vol. 1, pp. 42-49, Hawaii, Dec.2001.
60
[5] J. Ye, R. Janardan and Q. Li, “GPCA: An efficient dimen-
sion reduction scheme for image compression and retrieval,”
ACM Conf. KDD, 2004.
[6] R. Vidal, Y. Ma and S. Sastry, “Generalized principal com-
ponent analysis (GPCA),” IEEE Conf. CVPR, 2003.
[7] A. Smilde, R. Bro and P. Geladi, “Multi-way analysis: Ap-
plications in the chemical sciences,” Wiley, Oct. 2004.
[8] J. Yang and J.-Y. Yang, “From image vector to matrix:
a straightforward image projection technique–IMPCA vs.
PCA,” Patt. Recogn., vol. 35, no. 9, pp. 1997-1999, Sept.
2002.
[9] L. D. Lathauwer, B. D. Moor and J. Vandewalle, “A multi-
linear singular value decomposition,” SIAM J. Matrix Anal.
Appl., vol. 21, no. 4, pp. 1253-1278, Apr. 2000.
[10] R. T. Ng and A. Sedishe, “Evaluating multidimensional in-
dexing structures for images transformed by principal com-
ponent analysis,” Storage and Retrieval for Images and
Video Databases (SPIE), pp. 50-61, 1996.