CHAPTER 9:

Factor Analysis and Inference

FOR Structured Covariance
Matrices
Review
 In order not to omit information and to be accurate, we often
get a lot of indicators to analyze. For example, in order to study
the influencing factors of a certain disease, we may collect
dozens of indicators such as patients‘ demographic data,
medical history, physical signs, laboratory tests, etc. If these
indicators are directly included in the multivariate statistical
analysis, not only the model will become complex and unstable,
but also the multiple collinearity between variables may cause
large errors. Is there a way to condense information, reduce the
number of variables, and eliminate multicollinearity? To deal
with the problems, principal component analysis is on the stage.
Principle of principal component analysis
 The essence of principal component analysis is the rotation transformation of
coordinates. The original n variables are combined linearly again to generate n new
variables. They are not related to each other, so they are called n "components". At
the same time, according to the principle of variance maximization, the variance of
the first component is guaranteed to be the maximum, and then the variance
decreases in turn. These n components are arranged according to the variance
from large to small, and the first m components may contain most of the variance
(and variation information) of the original variables.
 Note that the principal component is not the residual variable after the original
variable is screened, but the "comprehensive variable" after the original variable is
recombined.
 We use the simplest two-dimensional data to intuitively explain the principle of
PCA. Suppose there are two variables X1 and X2, and draw the scatter diagram on
the coordinates as follows:
X1 and X2 are correlated. Y1 and Y2 are uncorrelated.
 Factor analysis is an extension of principal component analysis.
 In the process of principal component analysis, the new variable is a linear
combination of the original variables, that is to say, multiple original
variables are obtained by linear (coordinate) transformation.
 In factor analysis, it is to group the internal correlation structure among the
original variables. The ones with strong correlation are in a group, while the
groups are weak correlated, so each group of variables represents a basic
element (common factor). The original variables are decomposed to obtain
common factors and special factors. The original variable is expressed as a
linear combination of common factors. The common factor is the common
feature of all the original variables, while the special factor is the unique part
of the original variables. Factor analysis emphasizes the interpretation of the
practical significance of new variables (factors).
Basically, the factor model is motivated by the following argument: Suppose
variables can be grouped by their correlations. That is, suppose all variables
within a particular group are highly correlated among themselves, but have
relatively small correlations with variables in a different group. Then it is
conceivable that each group of variables represents a single underlying
construct, or factor, that is responsible for the observed correlation.
For example, the scores of 100 students in mathematics, physics, chemistry,
Chinese, history and English are obtained. The correlations from the group
of test scores in mathematics, physics, chemistry suggested an underlying "
science " factor. A second group of variables, representing “Liberal arts”
scores. It is this type of structure that factor analysis seeks to confirm.
 The software SPSS does not provide a separate principal
component analysis method, but is mixed with factor analysis.
Later we will discuss how to use SPSS to analyze the data by
principle component analysis and factor analysis. Firstly, we
will learn the principle of factor analysis.
9.2 The Orthogonal Factor Model
The observable random vector X, with p components, has mean μ and
covariance matrix Σ. The factor model postulates that X is linearly
dependent upon a few unobservable random variables F1,F2,...,Fm, called
common factors, and p additional sources of variation ɛ1,ɛ2, …,ɛ p , called
errors or, sometimes, specific factors.
In particular, the factor analysis model is

or, in matrix notation,

The coefficient lij is called the loading of the ith variable on the jth factor, so the
matrix L is the matrix of factor loadings. Note that the ith specific factor ɛi is
associated only with the ith response Xi. The p deviations X 1  1 , X 2  2 , , X p   p
are expressed in terms of p + m random variables F1,F2,...,Fm, ɛ1,ɛ2, …,ɛ p
which are unobservable.
 We assume that

 and that F and ɛ are independent, so

 These assumptions and the relation in (9-2) constitute the orthogonal factor
model?
Orthogonal Factor Model with m Common
Factors

The unobservable random vectors F and ɛ satisfy the following conditions:

 The orthogonal factor model implies a covariance structure for
X. From the model in (9-4),
Covariance Structure for the Orthogonal Factor
Model
The ith communality is the sum of squares of the loadings of the ith variable on the m
common factors. That portion of the variance of the ith variable contributed by the m
common factors is called the ith communality. That portion of Var (Xi) =σii, due to the
specific factor is often called the uniqueness, or specific variance.
Example 9.1 (Verifying the relation   LL  
for two factors)
 Consider the covariance matrix
 Unfortunately for the factor analyst, most covariance matrices cannot be factored
as LL' +Ψ, where the number of factors m is much less than p (see Example 9.2).
 When m > 1, there is always some inherent ambiguity associated with the factor

model. To see this, let T be any m×m orthogonal matrix, so that TT' =T'T =I.
Then the expression in (9-2) can be written
 where
 Since
 then
 That is,

Factor loadings L are determined only up to an orthogonal matrix T. Thus, the

Loadings L* =LT and L (9-9) both give the same representation. The
communalities, given by the diagonal elements of LL' =(L*) (L*)' are also
9.3 Methods of Estimation
 The Principal Component (and Principal Factor) Method:
 Let Σ have eigenvalue-eigenvector pairs
 Then
 Although the factor analysis representation of Σ in (9-11) is exact, it is not
particularly useful: It employs as many common factors as there are
variables and does not allow for any variation in the specific factors ɛ in (9-
4). We prefer models that explain the covariance structure in terms of just a
few common factors. One approach, when the last p-m eigenvalues are
small, is to neglect the contribution of
 Neglecting this contribution, we obtain the approximation

 The approximate representation in (9-12) assumes that the specific factors ɛ

in (9-4) are of minor importance and can also be ignored in the factoring of

 Σ.
 Allowing for specific factors, we find that the approximation becomes

 To apply this approach to a data set

 X1, X2,…Xn. In cases in which the units of the
 variables are not commensurate,
 it is usually desirable to work with
 the standardized variables:
 The representation in (9-13), when applied to the sample covariance matrix
S or the sample correlation matrix R, is known as the principal component
solution.
Principal Component Solution of the Factor Model
 For the principal component solution, the estimated loadings for a given factor
do not change as the number of factors is increased. For example, if m =1,

 then, how do we select the number of factors m?

 If the number of common factors is not determined by a priori considerations,
 such as by theory or the work of other researchers, the choice of m can be

based on the estimated eigenvalues in much the same manner as with principal
components.
 Consider the residual matrix
 The contribution to the total sample variance, s11 +s22 + …+ spp =tr(S), from
the first common factor is then
Example 9.3 (Factor analysis of consumer-preference data)

 In a consumer-preference study, a random sample of customers were asked

to rate several attributes of a new product. The responses, on a 7-point
semantic differential scale, were tabulated and the attribute correlation
matrix constructed. The correlation matrix is presented next:
 The first two eigenvalues, ̂1 = 2.85 and ̂2 =1.81, of R are the only
eigenvalues greater than unity. Moreover, m =2 common factors will account
for a cumulative proportion

 of the total (standardized) sample variance. The estimated factor loadings,

communalities, and specific variances, obtained using (9-15), (9-16), and (9-
17), are given in Table 9.1.
 The communalities (.98, .88, .98, .89, .93) in Table 9.1 indicate that the two factors account
for a large percentage of the sample variance of each variable.
A Modified Approach—the Principal Factor Solution
The Maximum Likelihood Method
9.4 Factor Rotation
 From matrix algebra, we know that an orthogonal
transformation corresponds to a rigid rotation (or reflection)
of the coordinate axes. For this reason, an orthogonal
transformation of the factor loadings, as well as the implied
orthogonal transformation of the factors, is called factor
rotation.
9.4 Factor Rotation
 Some common factors also have a significant impact on many variables,
indicating that they have a significant impact on multiple variables. This
factor model is actually not conducive to highlighting the main
contradictions and their main aspects, it is also difficult to provide a
reasonable explanation for the actual background of the factors. At this
point, it is necessary to use factor rotation to ensure that each variable
has a relatively large load on only one common factor, while the load on
the remaining common factors is relatively small. At this point, the
connection between each common factor and the variables with higher
loads is highlighted, and the main contradiction is revealed. The meaning
of this common factor can be reasonably explained through these
variables with higher loads, which also demonstrates the main properties
of this common factor
9.4 Factor Rotation
 Example 9.8 (A first look at factor rotation) Lawley and Maxwell present the
sample correlation matrix of examination scores in p = 6 subject areas for
 n =220 male students. The correlation matrix is

 and a maximum likelihood solution for m =2 common factors yields the

estimates in Table 9.5.
We point out that Figure 9.1 suggests an oblique rotation of the coordinates.
One new axis would pass through the cluster {1,2,3} and the other through the {4,5,6}
group. Oblique rotations are so named because they correspond to a nonrigid rotation
of coordinate axes leading to new axes that are not perpendicular.
9.5 Factor Scores
 In factor analysis, interest is usually centered on the parameters in the
factor model.
 However, the estimated values of the common factors, called factor scores,

may also be required. These quantities are often used for diagnostic
purposes, as well as inputs to a subsequent analysis.
The weighted least squares method
CFA
 Confirmatory factor analysis is a statistical analysis of social
survey data. It tests whether the relationship between a factor
and its corresponding measurement term conforms to the
theoretical relationship designed by the researcher.
Confirmatory factor analysis is often tested through structural
equation modeling （ SEM ） . In practical scientific research,
the process of confirmatory factor analysis is also the process
of testing measurement models.