Face Recognition Using Eigenfaces Matthew A. Turk and Alex P, Pentland ision and Modeling Gronp, The Media Laboratory “Massachusetts Institute of Technology Abstract We present an approach to the deteetion and iMentifention of hunnan faces and describe x work: Ing, near-real-time face recognition system which Tracks a subject's head and then recognizes the per son by comparing cheractersies of the face to thowe ff Known individuals. Oar approach treats ace Fecognition us a wordimensional recognition probe Tem, taking advantage ofthe fact that faces are are sormally Upright and thus may be described by stnall att of 2D characteristic ews, Face inages fie projected onto a Venture space ("ace space") that best encodes the vation among known face lnmages. ‘The face space is defied by the “igen faces", which ate the eigenvectors of the set of Faces they do not necessarily coreespond to isolated es tres much ar yen, ears and noses, The femmework provider the ability to earn to recognize new fers Ivan snsupervsed manner. 1 Introduction Developing « computational model of face recosn- tion is quite diffealt, because fee ate complex, Inultidimensional, and meaning. vial stimu They ate a natural case of objets, nd stand stark contrast to sine wave aratings, the “blocks trork and other ariel stimall wed in human nd computer vision resarch(I] Thus unlike most arly visual functions, for which we may constrict ‘etsiled models of retinal or sninte actnity, face ‘recognition is very high level task for which com: putational approaches an currently only siggest broad constants on the rotresponding, nearal Ac ‘We therefore focused our research towards devel coping 3 sort of early. peeattentive pattern recog tion capability that dors not depend upon having {all threedimensional models or detailed geometry ‘Our aim wae to develop « computational model of face tecogition ‘which is fast, reasonably” simple, And accurate in constrained environments sich as Sh office ors hoveehold ‘Although face recognition a high level visual problem, there is quite « bit of structure imposed on the task) We take advantage of some of this struc tue by proposing a scheme for recognition which is Trsed on an information theory approach, seeking tovencode the most relevant information ia group of faces which will best distinguish them fom one (¢H2963-5/91/0000/05861S01.00 © 1991 IEEE smother. ‘The approach transforms face images into {Smal se of characteristic feature images, called “cigenfaces", which are the principal components of the initial ehining set of Face images. Recogaition i performed by projecting a new image into the sub- Space spanned by the eigenfaes [face space") and ‘hen clasifying the face by comparing its position in face space withthe positions of krow individual. Automatically learning and later recognizing new faces i practical within this framework. Recog. sion under reasonably varying conditions is achieved by training on a limited qumber of churactoristic Views (eg a “steulght on” view, a 45" view, and profile view). ‘The approach has advantages over ‘other face recognition schemes in ts speed and smn ply, earning capacity, and relative insensitivity {small oF gradual changes in the face image, 1.1 Background and related work Muck of the work in computer recognition of faces has focused on detecting individual featanes atch as the eyes, nose, mouth, and head outline, and defn ing 3 face model by the position, size, and relation ships among these features. Beginning with Ble oes [2] and Kanade’ [3] early systems, xumber fof antomnted or semenntomated face recognition Stmtegics hnve modcied and clasifed faces based fon normalized distances snd ration among feature points." Recently this general approach has been Continued and improved by the rent aork of Yoile etal [ Such approaches have proven difcult to extend to multiple views, and have often been quite fag ie. "Reseach in human strategies of fase recog tion, moreover, hax shown that individual Features tnd their immediate relationships comprise an insu cient representation to account for the performance Gf adult human face Mentifeation[5}. Nonetheless, thi approach to fare recogition remains the most popular one in the computer viion iteratare Connections approaches to face identification seck to capture the configurational, or gestalt Iike nature ofthe task. Fleming and Cotte (6, build ing on easier work by Kehoven and Labtio {7}, se nonlinear units to train network via back proper ation to classify face images. Stonkam’s WISARD tern (has been applied with sme success tb sary face imagen recogniring both identity aad ex pression. Most connectionist systems dealing with foes treat the input image as» general 2D pattern,tnd can make no explicit use of the confgerstions! properties ofa face. Only very simple systems have been explored 10 date, and it i onclear how they will sale to larger problems Recent work by Burt otal aes “emart sco” spproach based on multiescution template match ing [9]. This conrae-to-Bne strategy wes 0 special [rpose compater built to calculate emaltieslution Pyramid images quickly, aad has been demonstrated lenifying people in near-real-time. ‘The Tace mod cls are but by hand from face images. 2 Eigenfaces for Recognition Muck of the previous wotk on astomated fare recog nition hus ignosed the howe of just what aspects of the face stimulus are important for Wdentieaton ‘assuming that predefined mensuremenls were tele Nant and sufcent. This suggested to vs that an Information theory approach of coding and decod ‘al and global “festures™, Suck features may oF may not be directly related to our iauitive notion offace features suchas the eyes, nose, lps, and ha In the language of information theory, we want to extract the relevant information ina fae image, ‘encode ita ficiently as possible, and compare one face encoding with & database of models encoded similarly. A simple approach to extrecting the infor ration contained in an image af faces to somehow faptare the variation in a cllection of face images, independent of any judgement of features, and ase this information to encode and compare individual face images Tn mathematical terms, we wish to find the prin- cipal components of the distribution of faces, oF the tdgenvector of the covariance nintsix of the set of ace images, ‘These eigenvectors can be thought of fn s art of features which together characterize the ‘ariaton betmeen face images. Each image lation Contributes more of les to each eigenvector, 20 that wwe cum display the eigenvector as sort of ghostly face which we call an eigenface, Some ofthese faces ate shown in Figote 2, Tach face image inthe training set can be repre sented exactly in terms of « linear combination of the eigenfaces, The number of possible cigenfaees i ‘equal to the number of face images in the traning ft. However the fares can nla be approximated ti ing only the "bes egentaces~ thoce that have the largest eigenvalues, and which therefore account for the most variance within the set of face images. The primary reson for ahing fewer cgenfaces comps {ational efficiency. The best A” egenfacescpan an 'sdimensional subspace “face space” of al possible images. As sinusoids of varying frequency ed pluse are the bars fonctions of = fourier dee ‘composition and are infact eigenfonctons of linear ystems), Uke eigentaces are the basis vectors of the Cigenface decomposition ‘The iden of using cigentices was motivated by technique developel by Sitovich and Kirby (10} for ficiently representing pictures of Spal component analysis They lection of face images can be approximately recon structed by storing a stall collection of weights for ach face and a small art of standard pictues, 1 opcurred to te that if a mtitude of fae im- ages can be reconstructed by weighted sums of Sina collection of eharsctesintic mages, then a ef Fesent way to learn and recognize facce might be {to build the characteristic features from known face mages and to recognize particular face by compar ifthe Teature weights needed to (approximately) reconstruct them with the weights associated with the known individuals, "The following steps summarize the recognition 1. Initialization: Acquire the training set of face images and caleulate the eigenfaces, which de Fin the fice space. 2, When anew face image is encountered caleu- Inte a eet of weights based on the Hep image and the Af eigenfaces by projecting the input image onto each of the eigenfnes 4. Determine if the image i a face at all (whether Known of unknown) by checking to see if the inmage i sulfcently close to "face space, 4. IC is w face, classify the wright pattern as ‘ithera known person of as unknown, 5. (Optional) Ifthe same wnknown face is seen several times, calculate its characteristic weit [pattern and incorporate into the known faces fie Tearn to recognize). 2.1 Calculating Eigenfaces Tet a face image I(2,9) be two-dimensional N by Nanay of intensity values, ora vector of dimension NA" R typical image of sie 256 by 258 describes & ‘vector of dimension 65,536 or, equivalently, «point jn 65,396-dimensional space. Am ensemble of iat tages, ther, mape to a collection of poiats ia this ge space Trager of faces, bring similar in overall configura tion, will not be randomly, distributed i this huge image space_and thus ean be described. by arch ively low dimensional subspace. The Of the principal component analysis (or Karhunen Loewe expanrion) is to find the vectors which best scout for the distribution of face images within the eatie image space. Those vectors define the sabspace of face images, which we ell Each vector is of length NV, describes an by ¥ linage, and is near combination of the original face images, Because these Yeetors ate the eget ‘estore of the covariance matrix coresponding to the original face images, and because they are face like im appearance, we refer to them a8 “egenfaces” Sone examples of cigenfaces ate shown in Figure 2ae o) Figure 1: (a) Pace images used as the traning ‘rib The averuge face Let the training set of face images be 1 .TTyaP ur, The average face of the se sd fined by Pom dp SNL T, Bach face difers fom the average ty the ctor =T)—W_ An example hining ct is shows in Figure Ha} wit the average re W shown in Figute 1(b). This tof very Irae ‘rctore is then subject to principal component aa poe which seks aset of Merthonormal ctor ty find theie associate! eigenvalues he sehich best d ibes the ustribation of the data, The vectors Mc find scalars Ay are the eigenvectors and eigenvals, respective, of The covariance mataie c= DM eat = aa? whose the matrix A= [4y dh «a. The matrig C'however, is A by N¥yand determining the N° ‘ogenvectore snd eigenvalues is an intractable task for typical image sizes We needa computationally feasible method t0 find there eigenvectors Fort tntely we can determine the eigenvectors bs fst ‘Giving a mach smaller M by AP mate problem and taking linear combinations of the rvaling ve tors, (See {11} for the details) ‘With this analyse the esleultions are greatly te duced, rom the order ofthe mombe of pal in the images CX") to the onder of the munber so 8: Figure 2: hive images and their projections nto the face space defied by the eigentaers of Figure 2 in the training set |. In practice, the teaning et of face images will be relatively small ( and the cleuations become quite manageable ‘Ssociated eigenvalues allow ys to tank the eigen for according to ele usefulness in characterizing the variation among the images. Figure ® shows tp seven eigentaces derived from the inpat imager St Figore 1. Normally the background i= temosed by cropping teaning images. 0 that the eigenfaces bve sero values utside of the face ae, The 22 Using Eigenfaces to classify a face image Once the cigenfaces are create. identification be comes a patters recognition task. The cigenaces Span an f"dimeasional subspace ofthe orginal? image space. The MT” signifeant rigenscctors of the Tanateic arc chosen as thove with the Ingest ss ciated eigenvalues, In any of Ov fest eases, bused fon = I8 face unages, A" = 1 eigenfaces were thed. ‘The number of egenfaces to be wad chosen heuristically baced om the igenalucs, Anew face image (T) i transformed into ite igenfare components (projected nto. “face spare") by simple operation DLW This deseriber 4 set of point-by- point image multiplications sac summations Figured shows thece inmeee and their projections sato the The weights foro 2 ret that desebee the contribution of wiT— 9) fork