Computer Methods and Programs in Biomedicine 71 (2003) 141 /147

www.elsevier.com/locate/cmpb

Principal component regression analysis with SPSS

R.X. Liu a,*, J. Kuang b, Q. Gong a, X.L. Hou c

a
Medical College of Jinan University, Guangzhou 510632, People’s Republic of China
b
Guangdong Provincial People’s Hospital, Guangzhou 510080, People’s Republic of China
c
Jinan University Library, Guangzhou 510632, People’s Republic of China

Received 30 March 2001; received in revised form 10 April 2002; accepted 11 April 2002

Abstract

The paper introduces all indices of multicollinearity diagnoses, the basic principle of principal component regression
and determination of ‘best’ equation method. The paper uses an example to describe how to do principal component
regression analysis with SPSS 10.0: including all calculating processes of the principal component regression and all
operations of linear regression, factor analysis, descriptives, compute variable and bivariate correclations procedures in
SPSS 10.0. The principal component regression analysis can be used to overcome disturbance of the multicollinearity.
The simplified, speeded up and accurate statistical effect is reached through the principal component regression analysis
with SPSS.
# 2002 Elsevier Science Ireland Ltd. All rights reserved.

Keywords: Multicollinearity diagnosis; Principal component regression analysis; SPSS

1. Introduction that symbol and value of actual regression coeffi-

In multivariate analysis, the least-squares
method is generally adopted in fitting a multiple
linear regression model, but estimation of the
least-squares is sometimes far from being perfect.
One of important causes leading to the result is
column vectors of matrix X is close to linear
correlation. Approximate linear relationship
among independent variables is called multicolli-
nearity. That there exists multicollinearity among
independent variables tends to lead to the result
* Corresponding author. Tel.: /86-20-8522-0259; fax: /86-
20-8522-1343.
E-mail address: trxliu@263.net (R.X. Liu).
cient are not consistent with the expected ones.
The often-used index to justify collinearity is
simple correlation coefficient. When simple corre-
lation coefficient between two independent vari-
ables is large, the collinearity is considered. Apart
from simple correlation coefficient, SPSS provides
collinearity statistics ([1], pp. 221): tolerance and
variance inflation factor (VIF). Tolerance
/1/
R2i , where R2i is squared multiple correlation of
ith variable with other independent variables.
When its value is small (close to 0), the variable
is almost a linear combination of the other
independent variables. VIF is reciprocal of toler-
ance. Variables with low tolerance tend to have
large VIF, so variables with low tolerance and

0169-2607/03/$ - see front matter # 2002 Elsevier Science Ireland Ltd. All rights reserved.
doi:10.1016/S0169-2607(02)00058-5
142 R.X. Liu et al. / Computer Methods and Programs in Biomedicine 71 (2003) 141 /147
large VIF suggest that they have a collinearity.
Eigenvalue, condition index and variance propor-
tion are also indices of collinearity diagnosis ([1],
pp. 229 /230). Eigenvalues provide an indication
of how many distinct dimensions there are among
independent variables. When several eigenvalues
are close to 0, the variables are highly intercorre-
lated and the matrix is said to be ill-conditioned.
Condition indices are square roots of ratios of the
largest eigenvalue to each successive eigenvalue. A
condition index greater than 15 indicates a possi-
ble problem and an index greater than 30 suggests
a serious problem with collinearity. Variance
proportions are proportions of variance of esti-
mate accounted for by each principal component
associated with each of eigenvalues. A component
associated with a high condition index contributes
substantially to variance of two or more variables,
so independent variables with large variances are
the ones being highly intercorrelated. The princi-
pal component regression is the method of com-
bining linear regression with principal component
analysis ([2], pp. 327/332). The principal compo-
nent analysis can gather highly correlated inde-
pendent variables into a principal component, and
all principal components are independent of each
other, so that all it does is to transform a set of
correlated variables to a set of uncorrelated
principal components. Then we built the regres-
sion equations with a set of uncorrelated principal
components and get the ‘best’ equation according
to the principle of the maximum adjusted R2 and
minimum standard error of estimate. At last the
‘best’ equation is transformed into the general
linear regression equation. The present paper will
demonstrate how multicollinearity problem is
solved by using SPSS 10.0 to do the principal
component regression [3].
2. Basic principle and formulas
(1) Proceed a stepwise regression with a depen-
dent variable Y and all independent variables X
for getting the p independent variables with
statistical significances (P B/0.05) and revealing
whether the p independent variables have a multi-
collinearity or not.
(2) Proceed a principal component analysis with
the p independent variables for transforming a set
of correlated variables to a set of uncorrelated
principal components and indicating information
quantities of different set of principal components.
(3) Compute the standardized dependent vari-
able, the p standardized independent variables and
the values of the p principal components respec-
tively according to Eqs. (1) /(3) for making a
preparation of setting up p standardized principal
component regression equations.
Y ?
(Y  Ȳ )=SY (1)
X ?
(X
i i  X̄ i )=SXi (i
1; . . .; p) (2)
Ci
ai1 X 1? ai2 X 2? . . .aip X p?
(3)
(i
1; . . .; p)
where Y ? stands for the standardized dependent
variable, Y the dependent variable, SY the stan-
dard deviation of dependent variable, Ȳ the mean
of dependent variable, Xi? the ith standardized
independent variable, Xi the ith independent
variable, X̄ i the mean of the ith independent
variable, SXi the standard deviation of the ith
independent variable, Ci the ith principal compo-
nent, aij the coefficient of principal component
matrix (the matrix consists of Ci and Xi?).
(4) Built the standardized principal component
regression equation with the first principal com-
ponent, then add principal component backwards
one by one to get the m standardized principal
component regression equations, as shown in Eq.
(4), meanwhile check whether all principal compo-
nents are independent of each other or not, then
determine the ‘best’ standardized principal com-
ponent regression equation in Eq. (4) on the basis
of the maximum adjusted R2 and minimum
standard error of estimate by comparing the
adjusted R2 and standard error of estimate of
each standardized principal component regression
equation.
X
ŷ?j
B?i Ci (j
1; :::; m 5p; i
1; :::; K 5 p)
(4)
where ŷ?j is the estimate of the j th standardized
principal component regression equation, Bi? the
 R.X. Liu et al. / Computer Methods and Programs in Biomedicine 71 (2003) 141 /147
ith standardized partial regression coefficient of
the standardized principal component regression
equation.
(5) Applying Eq. (3) to the ‘best’ standardized
principal component regression equation yields the
standardized linear regression equation after sort-
ing it out, as shown in Eq. (5).
X
ŷ?
b?i X i? (i
1; :::; K 5 p) (5)
where ŷ? is the estimate of the standardized linear
regression equation, b?i the ith standardized partial
regression coefficient of the standardized linear
regression equation.
(6) Compute partial regression coefficients and
constant, as shown in Eqs. (6) and (7), at last,
transform the standardized linear regression equa-
tion into the general linear regression equation, as
shown in Eq. (8).
bi
b?(Lyy=Lx
i x )1=2 (i
1; . . .; K 5p)
X i i
b0
Ȳ bi X̄ i (i
1; . . .; K 5p)
X linear regression procedure, obtain the results of
ŷ
b0  bi Xi (i
1; . . .; K 5 p)
where bi is the ith partial regression coefficient of
the general linear regression equation, Lyy the
sum of squares of dependent variable Y , Lxi xi the
sum of squares of the ith independent variable Xi ,
b0 the constant of the general linear regression
equation.
3. Example
Between the years of 1951 and 1998 (in which
the data of 1969 and 1986 are not indexed), the
mortality (1/100000) of traffic accident in the
mainland of China each year, the quantity
(10000 vehicles) of motors, the quantity (10000
tons) of freight transport, the quantity (10000
Table 1
The mean and S.D. of all variables
Y X1 X2
Mean 2.3504 730.1911 324111.65
SD 2.0428 1196.0913 347739.798
143
persons) of passenger transport, the mileage
(10000 km) of motor’s running on formal highway,
and the mileage (10000 km) of motor’s running on
informal highway are respectively expressed as the
dependent variable Y and independent variables
X1, X2, X3, X4 and X5. Table 1 displays the mean
and standard deviation (S.D.) of all variables.
(1) Select the significant independent variables
(P B/0.05) with SPSS backward and diagnose the
multicollinearity for each independent variable
([3], pp. 299/308).
In the SPSS linear regression dialog box, enter ‘y ’
(the dependent variable) into the dependent box
and ‘x1, x2, x3, x4 and x5’ (all independent
variables) into the independent box, select the
backward in the method selection control. In the
SPSS linear regression: statistics dialog box, click
on Descriptives, Covariance matrix and
(6) Collinearity diagnostics, while the others are the
(7) items assumed by SPSS. After running the SPSS
(8) Tables 2 and 3.
Table 2 displays that partial regression coeffi-
cients b1, b3 and b4 of three independent variables
(X1, X3 and X4) are highly significant (P B/0.0005)
and b1 is equal to /7.52 /104, which indicates
there is a negative correlation between the mortal-
ity of traffic accident and the quantity of motors.
The result is contrary to the common sense. We
will check whether there are multicollinearities
among the independent variables. It also displays
that toleranceX1 and toleranceX3 are small (0.04
and 0.022), VIFX1 and VIFX3 are large (25.233 and
46.402). Table 3 shows that RX1,X3 is large (/
0.950), the 4th eigenvalue is close to 0 (0.007352),
condition index is more than 15 (21.362) and the
variance proportions of the independent variables
X1 and X3 are large (0.88 and 0.99). These facts
X3 X4 X5
313836.61 54.1220 18.5461
372520.677 31.5176 5.6636
144 R.X. Liu et al. / Computer Methods and Programs in Biomedicine 71 (2003) 141 /147

Table 2
Partial regression coefficient and collinearity statistics of the linear regression equation

Model bi t P Collinearity statistics

Tolerance VIF

Constant /8.94/10 2 /0.773 0.444

X1 /7.52/10 4 /3.972 0.000 0.040 25.233
X3 6.132/10 6 7.439 0.000 0.022 46.402
X4 1.967/10 2 4.891 0.000 0.126 7.906

Table 3
Indices of collinearity diagnosis and simple correlation coefficient RXi Xj of the linear regression equation

Dimension Eigenvalue Condition index % of variance

Constant X1 X3 X4

1 3.355 1.000 0.01 0.00 0.00 0.00

2 0.572 2.422 0.14 0.02 0.00 0.00
3 0.06533 7.166 0.56 0.10 0.01 0.24
4 0.007352 21.362 0.29 0.88 0.99 0.76

Table 4
The Eigenvalue, % of variance and coefficients for each principal component

Principal component Eigenvalue % of variance % of cumulative variance Standardized independent variable

X1? X3? X4?

1 2.746 91.523 91.523 0.954 0.993 0.922

2 0.241 8.033 99.556 /0.292 /0.0787 0.387
3 0.01333 0.444 100.000 0.06481 /0.0906 0.03044

clearly indicate that there is a collinearity between
X1 and X3.
(2) Use the SPSS factor analysis procedure to
obtain the principal component matrix of the
independent variables X1, X3 and X4 and the
cumulative variance proportion of different prin-
cipal components ([3], pp. 323-334).
In the SPSS factor analysis dialog box, enter ‘x1,
x3 and x4’ (the independent variables X1, X3 and
X4) into the variable box. In the factor analysis
extraction dialog box, click on Number of
factors, and type ‘3’ into the box, whereas the
others are the items assumed by SPSS. All results
are shown in Table 4 after running the factor
analysis.
Table 4 displays that the cumulative variance
proportion of one principal component (the 1st
principal component C1) is 91.523%, the one of
two principal components (C1 and C2) is 99.556%
and the one of three principal components (C1, C2
and C3) is 100.000%. In Table 4, obtain coeffi-
cients (aij ) related the three standardized indepen-
dent variables to three principal components to
create expressions of three principal components:
C1
/0.954X1?/0.993X3?/0.922X4?, C2
//
0.292X1/0.0787X3/0.387X4, C3
/0.06481X1?/
? ? ?
0.0906X3?/0.03044X4?.
(3) Obtain the standardized dependent variable
Y ?, the standardized independent variables X1?, X3?
and X4? by using the SPSS descriptives procedure
 R.X. Liu et al. / Computer Methods and Programs in Biomedicine 71 (2003) 141 /147
([3], pp. 219 /222) and the value of each principal
component Ci according to expression of each
principal component and by using the SPSS com-
pute variable procedure ([3], pp. 89 /92).
In the SPSS descriptives dialog box, enter ‘y, x1,
x3 and x4’ (the dependent variable Y and the
independent variables X1, X3 and X4) into the
variable[s]: box, click on Save standardized
values as variables and click on OK button to
run the SPSS descriptives procedure to create the
standardized dependent variable zy and the stan-
dardized independent variables zx1, zx3 and zx4 in
the current working data file.
In the SPSS compute variable dialog box, type
‘c1’ into the target box as the variable name of the
first principal component C1, type ‘0.954+zx1/
0.993+zx3/0.922+zx4’ into the numeric expression
box, click on OK button to run the SPSS compute
variable procedure to create a new variable c1 and
its value in the current working data file. When the
second and third principal components C2 and C3
are computed, type ‘c2’ and ‘c3’ into the target
boxes, and type ‘/0.292+zx1/0.0787+zx3/
0.387+zx4’ and ‘0.06481+zx1/0.0906+zx3/
0.03044+zx4’ into the numeric expression box
respectively. After running the SPSS compute
variable procedure one by one, generate the new
variables c2 and c3 and their values in the current
working data file.
(4) Use the SPSS linear regression procedure to
do the principal component regression analysis:
includes to build each standardized principal
component regression equation, check whether
Table 5
Standardized partial regression coefficients, collinearity statistics and correlation coefficients RCi Cj of three standardized principal
component regression equations
Model Bi? t P
1 C1 0.971 26.935
2 C1 0.970 33.907
C2 0.148 5.189
RC1,C2
//0.009
3 C1 0.969 43.925
C2 0.148 6.726
C3 /0.121 /5.496
RC1,C2
//0.009 RC1,C3
/0.002
145
all principal components are independent of each
other or not and determine the ‘best’ standardized
principal component regression equation ([3], pp.
299/308).
In the SPSS linear regression dialog box, enter
‘zy and c1’ into the dependent and the independent
boxes respectively. In the SPSS linear regression:
statistics dialog box, click on Covariance matrix
and Collinearity diagnostics, while the others are
the items assumed by SPSS. Thus, generate the 1st
standardized principal component regression
equation: ŷ?1/
/B1?C1. Following the same steps, fit
the equations: ŷ?2/
/B1?C1/B2?C2 and ŷ?3/
/B1?C1/
B2?C2/B3?C3. The differences of their operations
are that in the independent box, the former is
entered into ‘c1 and c2’, and the latter is entered
into ‘c1, c2 and c3’. After running the SPSS linear
regression procedure, all results are shown in
Tables 5 /7, respectively.
In Table 5, obtain all standardized partial
regression coefficients Bi? with high significance
(P B/0.0005) of all principal components Ci in all
models (equations) to generate three standardized
principal component regression equations: ŷ?1/
/
0.971C1, ŷ?2/
/0.970C1/0.148C2 and ŷ?3/
/
0.969C1/0.148C2/0.121C3. Table 5 displays all
simple correlation coefficients RCi Cj of all princi-
pal components Ci are close to 0 and their
tolerances and VIFs are equal to 1. Table 6 shows
that their eigenvalues and condition indices are
close to 1. These suggest that all principal compo-
nents are independent of each other.
Collinearity statistics
Tolerance VIF
0.000 1.000 1.000
0.000 1.000 1.000
0.000 1.000 1.000
0.000 1.000 1.000
0.000 1.000 1.000
0.000 1.000 1.000
RC2,C3
/0.000
 146 R.X. Liu et al. / Computer Methods and Programs in Biomedicine 71 (2003) 141 /147

Table 6
Indices of collinearity diagnosis of three standardized principal component regression equations

Dimension Eigenvalue Condition index % of variance

C1 C2 C3

1 1.000 1.000 0.00

2 1.000 1.000 1.00
1 1.009 1.000 0.50 0.50
2 1.000 1.005 0.00 0.00
3 0.991 1.009 0.50 0.50
1 1.009 1.000 0.50 0.47 0.03
2 1.000 1.005 0.00 0.07 0.93
3 1.000 1.005 0.00 0.00 0.00
4 0.990 1.010 0.50 0.47 0.04

R2 is a measure of goodness of fit of a linear
model and tends to be an overestimate of popula-
tion parameter ([1], pp. 197 /198, 208). R2 ranges
from 0 to 1. The closer to 1, the better on goodness
of fit of a linear model. As R2 is affected by the
number of independent variables in model and
sample size, we usually use the adjusted R2 when
comparing the goodnesses of fit between different
linear models. Adjusted R2 is designed to compen-
sate for the optimistic bias of R2. Standard error of
estimate is the square root of the residual mean
square and measures the spread of the residuals
about the fitted line ([1], p. 198), so it is also a
measure of goodness of fit of a linear model. The
closer to 0, the better on goodness of fit of a linear
model. ŷ?3/
/0.969C1/0.148C2/0.121C3 is deter-
mined as the ‘best’ equation, as Table 7 shows that
adjusted R2 (0.978) and standard error of estimate
(0.1480) of the third standardized principal com-
ponent regression equation is the largest and the
smallest in the three equations respectively, and its
F value is equal to 670.541 and it is also highly
significant (P B/0.0005).
(5) Using the SPSS bivariate correlations proce-
dure to compute Lyy and Lxi xi ([3], pp. 285/290).
In the SPSS bivariate correlations dialog box, enter
‘y , x1, x3 and x4’ (the dependent variable Y and
the independent variables X1, X3 and X4) into the
variables box. In the bivariate correlation: options
dialog box, click on Cross-product deviation and
covariance. Run the bivariate correlations proce-
dure to get Lyy
/187.788, Lx1x1
/64378550,
Lx3x3
/6.245 /1012 and Lx4x4
/44701.175.
(6) Transform the ‘best’ standardized principal
component regression equation into the standar-
dized linear regression equation and then into the
general linear regression equation.
From Table 4, get C1
/0.954X1?/0.993X3?/
0.922X4?, C2
//0.292X1?/0.0787X3?/0.387X4?
and C3
/0.06481X1?/0.0906X3?/0.03044X4?, and
apply them to the ‘best’ standardized principal
component regression equation: ŷ?/
/0.969C1/
0.148C2/0.121C3
/0.969 (0.954X1?/0.993X3?/
0.922X4?)/0.148(/0.292X1?/0.0787X3?/0.387X4?)
/0.121(0.06481X1?/0.0906X3?/0.03044X4?), then
after having sorted it out, obtain the standardized

Table 7
The expression, adjusted R2, standard error of estimate, F value and P value in each equation

Standardized principal component regression equation Adjusted R2 Standard error of estimate F P

/ŷ?1/
/0.971C1 0.942 0.2418 725.490 B/0.0005

/ŷ?2/
/0.970C1/0.148C2 0.963 0.1918 589.972 B/0.0005
/ŷ?3/
/0.969C1/0.148C2/0.121C3 0.978 0.1480 670.541 B/0.0005
 R.X. Liu et al. / Computer Methods and Programs in Biomedicine 71 (2003) 141 /147
linear regression equation: /ŷ?/
/0.8734X1?/
0.9615X3?/0.9470X4?.
Calculate the general partial regression
coefficients bi with b1?
/0.8734, b3?
/0.9615
and b4?
/0.9470 in the light of Eq. (6), b1
/b1?
(Lyy /Lx1x1)1/2
/0.8734(187.788/64378550)1/2
/
0.00149, b3
/b3?(Lyy /Lx3x3)1/2
/0.9615(187.788/
6.245 /1012)1/2
/0.0000053, b4
/b 4?(Lyy /
Lx4x4)1/2
/0.9470(187.788/44701.175)1/2
/0.0648,
and the constant b0 in accordance with Eq. (7),
b0
/Ȳ /S bi X̄ i
/2.3504/(0.00149/730.191/
0.0000053 / 313836.61 / 0.0648 / 54.1220)
/
/3.908 and finally, obtain the general linear
regression equation: ŷ/
//3.908/0.00149X1/
0.0000053X3/0.0648X4.
4. Discussion
Not only can the principal component regres-
sion analysis overcome disturbance of collinearity
and real face of the fact is exposed (e.g. that b1
/
/7.52 /104 is corrected to b1
/0.00149 through
principal component regression analysis indicates
there is a positive correlation between the mortal-
ity of traffic accidents and the quantity of motors,
as is in accordance with the fact), while original
information is not lost yet (Table 4 shows that the
cumulative variance proportion with three princi-
pal components goes to 100%, and namely the
‘best’ principal component regression equation ŷ?3/

/0.969C1/0.148C2/0.121C3’ uses all original
information).
The B1?, B2? and B3? in the ‘best’ principal
component regression equation are highly signifi-
cant (P B/0.0005). It indirectly proves that the b1?,
b3? and b4? in the standardized linear regression
equation also are highly significant since the each
principal component includes the information of
the standardized independent variables X1?, X3? and
147
X4?. It is inferred that the b1, b3 and b4 in the
general linear equation are highly significant by
the same principle. Hence, we can do a factor
analysis by the standardized partial regression
coefficients bi?, and also do a prediction by using
the general linear regression equation: ŷ/
/
/3.908/0.00149X1/0.0000053X3/0.0648X4.
We should use standardized independent vari-
ables Xi? when computing value of each principal
component Ci as numeric expression is Ci
/
ai 1X1?/ai 2X2?/. . ./aip Xp? , and should not use
raw independent variables Xi . If value of principal
component Ci is computed with the raw indepen-
dent variables Xi , it will result in the complete
correlation among principal components Ci
(RCi Cj
/1 or /1 i "/j ).
In multiple linear regression analysis, when
there is a phenomenon in which results differ
from the fact, it will usually be suspected there are
multicollinearities among independent variables.
At that time, you can use the above method to
analyze. The principal component regression ana-
lysis with SPSS is an effective method. Not only can
it diagnose collinearity for each independent vari-
able, but also solve the collinearity problem. At
the same time, the majority of computation
procedures are completed with help of computer,
which greatly reduces the complicated manual,
and simplified, speeded up and accurate statistical
effect is reached at last.
References
[1] SPSS Inc., SPSS Base 10.0 Applications Guide, SPSS Inc.,
USA, 1999.
[2] N.R. Draper, H. Smith, Applied Regression Analysis, 2nd
ed., John Wiley & Sons Inc, New York, 1981.
[3] SPSS Inc., SPSS Base 10.0 User’s Guide, SPSS Inc., USA,
1999.