PERSONNELPSYCHOLOGY

1980, 33

DIMENSIONS OF WORK OUTCOMES: A

MULTIDIMENSIONAL SCALING APPROACH
ROBERT S. BILLINGS AND EDWIN T. CORNELIUS, I11
Ohio State University

There exists a need for basic research on the nature of work out-
comes, to facilitate integration of theory and results and to aid in the
sampling of outcomes for research and evaluation. Existing re-
search, primarily using factor analysis, has led to a hierarchical
model, with numerous sub-factors grouped under the major factors
of “intrinsic” and “extrinsic.” Recent research, however, has shown
that industrial/organizational psychologists do not agree on the def-
inition of intrinsic and extrinsic. It is argued that a more useful ap-
proach is a dimensional model, which allows each work outcome to
be described by a set of values on multiple dimensions. A multi-
dimensional scaling analysis of 21 outcomes yields a latent structure
with three dimensions: value attached to the outcome by society,
level of psychological need met, and extent to which the outcome is
inherent in the work itself. Implications for the sampling of out-
comes and integration of different literature are discussed. The re-
sults suggest that a multidimensional model of work outcomes is
more useful than a categorical approach built on the intrinsic/ex-
trinsic dichotomy.

I N their recent review of the literature on work motivation, Camp-

bell and Pritchard (1976) argue that there is a need for basic research
on the nature of work outcomes. A better understanding of outcomes
would aid in the communication and synthesis of research findings
and integration of those theoretical models which rely on some per-
ception of or attitude toward work outcomes. Further, selecting an
adequate sample of work outcomes is often important in research
(e.g., testing expectancy theory) and evaluation (e.g., obtaining a com-
prehensive profile of satisfaction with job outcomes). The more that is
known about the structure of outcomes, the easier it is to obtain a rep-
resentative sample.

The authors wish to thank Joan Brett for her help in the development of stimulus
materials and the data collection phase of this study. Requests for reprints should be
sent to Robert S. Billings, Department of Psychology, The Ohio State University, 404-C
West 17th Avenue, Columbus, Ohio, 43210.
Copyright 0 1980 by Personnel Psychology, Inc.

151
152 PERSONNEL PSYCHOLOGY

The current model of outcomes which is most widely used attempts

to classify outcomes as intrinsic versus extrinsic. Reviewing the factor
analytic studies that have been done, Campbell and Pritchard (1976)
conclude that a hierarchical structure emerges. Specific factors (e.g.,
compensation, working conditions) are clustered within two general
factors-intrinsic versus extrinsic.
However, the adequacy of this model of outcomes can be ques-
tioned. Researchers working within different paradigms use different
definitions of intrinsic and extrinsic (Billings and Cornelius, Note 1).
Also, in a survey of industrial/organizational psychologists, Dyer and
Parker (1975) found the intrinsic/extrinsic distinction to be highly
ambiguous. The survey respondents showed little agreement on the
labeling of specific outcomes as intrinsic or extrinsic, and also gave a
variety of different definitions of the terms. If intrinsic and extrinsic
mean different things to different psychologists, then it is a poor basis
for understanding work outcomes.
One solution might be to try to agree on the “right” definition of
these terms. However, a more basic question involves the structure
people actually use to organize outcomes. Do individuals think of
work outcomes as either intrinsic or extrinsic, or do they use multiple
dimensions? Empirical evidence that subjects perceptually organize
work outcomes in a multidimensional fashion would further question
the usefulness of the intrinsic/extrinsic distinction.
A technique that is useful in uncovering the underlying dimensions
in perceptual data is multidimensional scaling (MDS). MDS actually
refers to a class of analytic models that are used to uncover the latent
spatial structure inherent in the perceptual judgments that subjects
give to a set of objects (see Kruskal, 1964; Shepard, 1962; Torgerson,
1958). Input for a MDS analysis typically consists of ratings of “simi-
larity” between all possible pairs of objects. The analytic routine then
seeks a parsimonious set of dimensions that can best reproduce the in-
put judgments given by the subjects. The locations of the objects on
the dimensions of the resulting solution are used to “interpret” the di-
mensions and attach psychological meaningfulness to the results.
Although MDS has not been used previously in the study of work
outcomes, one clear advantage of this technique is the unstructured
nature of the rating task required of the subjects. Most previous stud-
ies of work outcomes have used factor analysis to analyze satisfaction
judgments, although a few have used importance judgments (e.g.,
Quinn and Cobb, 1971) or expectancy and instrumentality ratings
(e.g., Lawler and Suttle, 1973). Thus, it is not certain that the factor
structure of outcomes is generalizable across the characteristic being
rated. With MDS, however, subjects are not told what characteristic
 BILLINGS AND CORNELIUS 153

of objects to attend to in making their similarity judgments. Instead,

the underlying characteristics themselves are inferred from the MDS
results.
The conceptual model underlying MDS techniques is also an at-
tractive one. Existing studies of work outcomes have resulted in clus-
ters of outcomes which are then placed within the “intrinsic” or “ex-
trinsic” category. It may be, however, that work outcomes can be
described better by underlying continuous dimensions, with each out-
come having a specific value on each dimension. This approach has
the advantage of more fully specifying the relationships among out-
comes; an outcome can be thought of as either similar or dissimilar to
other outcomes depending upon the dimension used, instead of al-
ways being clustered with the same set of outcomes. MDS is consis-
tent with this assumption that underlying dimensions can be used to
describe the structure of outcomes.
The purpose of this study is to examine the perceptual structure of
work outcomes using multidimensional scaling, and to demonstrate
the usefulness of this methodology. Our general hypothesis is that the
desired dimensional structure will be interpretable, and will be fairly
complex in the sense that more than one dimension will be needed to
account for the data; the intrinsic/extrinsic distinction will not be suf-
ficient to account for the empirical structure.

Method
Description of the Questionnaire
For purposes of establishing convergence across somewhat different
methods, two separate questionnaires were developed for the present
study: a similarity judgment questionnaire and a likelihood judgment
questionnaire. Both questionnaires contained all possible different
paired comparisons of the work outcomes used in the Dyer and
Parker (1975) survey. In addition, 11 of the paired comparisons were
randomly selected as repeated pairs and were included a second time
in the questionnaire for reliability purposes.
The similarity judgment questionnaire contained 22 1 paired com-
parisons of the following type: “To what extent is the work outcome
high salary similar to the work outcome prestige?” The response scale
contained seven discrete points and ranged from “Very Dissimilar” to
“Very Similar.” The likelihood judgment questionnaire contained
likelihood estimates of the following type: “Imagine a work situation
that provides high salary; how likely would the work situation also
provide prestige?” The response scale in this case contained seven dis-
crete points and ranged from “Extremely Unlikely” to “Extremely
I54 PERSONNEL PSYCHOLOGY

Likely.” All seven points on both types of rating scales were anchored
by adjective descriptions. On both questionnaires subjects were asked
to make an additional set of ratings on the 21 outcomes using eight
unidimensional scales that were hypothesized as potential underlying
dimensions of the space (see explanation below).
Subjects
There were 182 participants in this study. Ninety-one subjects filled
out the similarity judgment questionnaire and 91 subjects filled out
the likelihood judgment questionnaire. All subjects were under-
graduates enrolled in introductory psychology courses at The Ohio
State University who participated in the study for partial fulfillment
of course credit requirements. University students were selected for
use in this study primarily on the basis of convenience for an initial
test of a dimensional model of work outcomes. Further, many studies
involving work outcomes have used university subjects, making the
perceptual structure of outcomes for this population of individuals
important in its own right.
Procedure
Subjects were randomly assigned either the similarity judgment
questionnaire or the likelihood judgment questionnaire in one of two
group sessions. Before starting each session, the purpose of the study
was explained in general terms to all participants. Subjects in both
sessions were then given a standardized set of instructions and al-
lowed to begin. Subjects proceeded at their own rate, and most sub-
jects completed the questionnaire within 75 minutes.
Multidimensional Scaling Model
The particular multidimensional scaling model used to derive the
spatial configuration from the judgments in this study was the individ-
ual differences weighted Euclidian model that is incorporated in the
ALSCAL computer program. For a discussion of the ALSCAL pro-
gram, its advantages, and the computational method, the reader is re-
ferred to Takane, Young, and deLeeuw (1977).
Projection of Hypothesis Vectors into the Space
Each outcome was rated on eight unidimensional scales that were
hypothesized as possible underlying dimensions. Five of these a priori
dimensions were derived from the definitions of intrinsic/extrinsic
given by industrial/organizational psychologists in Dyer and Parker’s
(1975) survey: (1) extent outcome inherent in work itselc (2) extent
outcome internally mediated; (3) level of underlying psychological
 BILLINGS AND CORNELIUS 155

need being met; (4) degree of concreteness of the outcome; (5) extent
outcome valued as a means to an end. A subsequent review of four re-
search areas (two-factor theory, job design, intrinsic motivation/ex-
trinsic reward, and expectancy theory) indicate that these five dimen-
sions capture most of the distinctions made between intrinsic and
extrinsic outcomes (Billings and Cornelius, Note 1). However, one ad-
ditional dimension is suggested by the literature on two-factor theory
and expectancy theory: extent to which the individual can control the
outcome through his/her behavior. The final two a priori dimensions
were the value the individual places on the outcome and the value
placed on it by society.
As an interpretational aid, the ratings of outcomes on these a priori
dimensions were inserted into the obtained space using the least
squares procedure outlined by Cliff and Young (1968). Briefly, this
method involves regressing each a priori scale on the various dimen-
sions of the MDS space, using the scale values for the 21 stimuli as the
“observations.” The Multiple R2 is thus a measure of variance in the
hypothesis vector accounted for by the MDS solution, while the stan-
dardized beta weights may be used as coordinates to locate the hy-
pothesis vector in the resulting MDS configuration.

Results
Criteria for Selecting Subjects for the Analysis
The questionnaires for five subjects were eliminated due to in-
complete data, leaving a potentially usable sample of 177. For two
reasons we elected not to use the data from all these subjects. First, we
wanted only those subjects who had sufficient work experience so that
the stimulus items would have some meaning to them. Therefore, we
included in the analysis only those subjects who were working at the
time of this study (either full-time or part-time) or who had at least
moderate work experience for this age group of subjects (ie., one year
full-time work experience or two years of part-time experience).
Secondly, we wanted only those subjects who reliably completed
the paired comparisons judgments. It is well known that MDS models
are not robust in recovering underlying dimensional structure with
large amounts of error in the data (Isaac and Poor, 1974; Sherman,
1972; Spence, 1972; Young, 1970; others). Anecdotal accounts from
our experimenter indicated that the difficult nature of the judgments
and the unusual length of the questionnaire (35 pages) contributed to
possible low motivational levels of the university students who served
as subjects in this study. For our purposes we therefore analyzed only
156 PERSONNEL PSYCHOLOGY

those questionnaires that met two of the following three additional

criteria:
1. The correlation between 11 repeated pairs on the questionnaire
had to be statistically significant (for 9 degrees of freedom, the one-tail
critical value for r is .52, p < .05).
2. Eight of the 11 repeated judgments had to be the same or within
one scale point of each other, as calculated by the Sanson-Fisher and
Mulligan (1977) index.
3 . The within person standard deviation of ratings on the 221 pairs
had to be equal to or greater than 1.0.
Using both the reliability criterion and the job experience criterion
we identified 95 questionnaires (54%) for further analysis. The dele-
tion of cases not meeting our criteria was deemed appropriate since
the purpose of this paper was not to generalize the substantive find-
ings to other samples, but rather to demonstrate the multidimensional
nature of perceptions of outcomes. We felt that an initial illustration
of this phenomenon should be free from as much error variance in the
input data as possible. A similar rationale for dropping subjects in a
scaling study has been offered elsewhere (Nygren and Jones, 1977).
The resulting sample sizes of 47 for the Similarity questionnaire and
48 for the Likelihood questionnaire were more than adequate for use
with the individual differences weighted Euclidean scaling model (see
MacCallum and Cornelius, 1977).
Determination of Dimensionality
The paired comparisons from the similarity judgment questionnaire
were entered into the ALSCAL version 4D computer program using
the following options: interval measurement level, individual differ-
ences model, negative weights permitted, and matrix conditional data.
A maximum of 30 iterations were specified for one-, two-, three-,
four-, and five-dimensional solutions.’ The judgments from the likeli-
hood questionnaire were entered in a separate analysis using the same
options.
The determination of underlying dimensionality was made on the
basis of visually inspecting plots of the stress values for various di-
mensional solutions, the interpretability of various solutions, and the
degree of convergence of results from the two separate analyses (Like-
lihood data vs. Similarity data). The “convergence” analysis was car-
ried out by correlating the underlying dimensions in the various s d -

‘The one dimensional solution cannot be obtained with the individual differences 11
model, therefore the solution was obtained using the standard Euclidean MDS model I’
(The Torgerson Method)
 BILLINGS AND CORNELIUS 157

TABLE 1
Multimethod Convergence: Correlations of Dimensions in the Similarity Space with
Dimensions in the Likelihood Space
(N = 21 Stimuli)

Likelihood Judgment Dimensions"

I I1 111
Similarity I .94 -.90 .22
Judgment I1 -.13 .81 .so
Dimensions I11 .06 -.42 .68
The signs of the projections of the 21 stimuli on Dimension I1 of the Likelihood space were arbitrarily reflected
in order to have a positive correlation in the diagonal.

larity spaces with the underlying dimensions in the corresponding

likelihood spaces. The results of this analysis revealed that the best
convergence was obtained by comparing the three-dimensional likeli-
hood space with the three-dimensional similarity space. Table 1 pre-
sents the cross-correlations for the three-dimensional solutions. In a
Campbell and Fiske (1959) sense, Table 1 is heterotrait-heteromethod
matrix, and it can be shown that the results of this analysis meet the
requirements for convergent and divergent validity.
On the basis of both the interpretability and the degree of con-
vergence across two paired comparisons judgments (similarity judg-
ments and likelihood judgments) the three-dimensional ALSCAL so-
lution was selected to be the one most likely to represent the correct
underlying dimensionality of the data.
Interpretation of the Underlying Dimensions
The normalized projections of the 21 work outcomes in the three-
dimensional ALSCAL solution for the similarities questionnaire data
are presented in Table 2. Graphical representations of the multi-
dimensional space are presented in Figures 1 and 2. Figure 1 is a plot
of Dimension 1 against Dimension 111, and Figure 2 is a plot of I1
against 111. These comparisons were chosen to highlight the position
of certain outcomes, as discussed below. The data from the likelihood
judgment questionnaire, which will not be presented here, exhibited a
similar structure, except that the third dimension was somewhat less
interpretable.
To aid in the interpretation of the space, vectors representing the
eight hypothesized dimensions were inserted in the three-dimensional
space as described earlier in the paper. The fit of these a priori vectors
in the multidimensional space was good (multiple R's ranged from .66
to 32). For each of the three recovered dimensions, the a priori vector
with the best fit was selected and located in Figures 1 and 2. The loca-
tions of these inserted scales as well as the projections of the stimuli
 158 PERSONNEL PSYCHOLOGY

TABLE 2
Locations of 21 Work Outcomes in the Three-Dimensional ALSCAL Soluiion
( N = 47)

I1 111
I Under- Extent
Societal lying Inherent
Symbol Outcome Values Needs in Work
IND Opportunity for independent thought and action -1.14 .93 .a
SAL High salary .99 -.07 1.49
REC Recognition 1.08 .49 -1.18
FRD Opportunities to develop friendships -.88 -1.51 -33
SLF Feelings of self-fulfillment -1.00 1.03 -.47
ACC Feelings of worthwhile accomplishment -.I9 1.42 -.71
RES Responsibility .92 .33 1.33
CON Good working conditions -.90 -1.71 .43
ENJ Enjoyment of the work itself -1.53 -.38 .02
ADV Opportunity for advancement 1.42 .72 -.12
SEC Job security .87 -1.67 -.08
PRS Prestige 1.51 -.I2 -.68
GRO Opportunity for personal growth and development -1.26 -.39 -.71
RSP Respect of fellow workers .07 -1.46 -1.19
PRD Feelings of pride in work -.61 1.20 -.94
ACH Feelings of achievement .00 1.30 -32
VAR Variety on job -1.32 -.11 1.44
STR Stress or pressure .62 -.54 1.92
ABL Opportunity to use special abilities -.66 1.35 .58
ATH A lot of authority 1.29 -.I3 1.27
SUP Support and consideration from supervisor .7 1 -.68 -1.38

themselves were used as the basis for interpreting the dimensions as

follows:
I. Societal Values, anchored on the high end by prestige, opportu-
nity for advancement, and a lot of authority, and on the low end by
enjoyment of the work itself, variety, and opportunity for personal
growth and development.
11. Underlying Needs, anchored on the high end by feelings of
worthwhile accomplishment, opportunity to use special abilities, and
feelings of achievement, and on the low end by good working condi-
tions, job security, and opportunity to develop friendships.
111. Extent Inherent in Work, anchored on the high side by stress,
high salary, and variety, and on the low side by support and consid-
eration from supervisor, respect of fellow workers, and recognition.
Discussion
The results of this study support the general hypothesis that an in-
terpretable, multidimensional structure underlies the work outcomes
examined. For this particular list of outcomes and sample of subjects,
the latent spatial structure consisted of three dimensions. This pro-
 BILLINGS AND CORNELIUS I59

*STR

WAR

ORSP .REC

*SUP

FIGURE1 . Plot of Dimension I against Dimension I11 in the three-dimensional AL-

SCAL solution (stimuli symbols explained in Table 2).

vides further evidence that the intrinsic/extrinsic dichotomy is not

useful, in that no single dimension can account for how these subjects
perceive the similarities among this set of work outcomes.
Although different dimensions might emerge in research with other
samples of subjects, there are some intersting things about these spe-
cific findings. One dimension is interpretable as a valence or social de-
sirability dimension, which is a concept common to many areas of
psychology. The other two dimensions (underlying need and inherent
in the work itself) are the most commonly used definitions of “in-
trinsic” and “extrinsic.” The recent literature on job design (e.g.,
Hackman and Oldham, 1976) focuses on the underlying need as the
important aspect of intrinsic outcomes, while the two-factor literature
generally focuses on the extent to which the outcome is inherent in the
work versus being an external, “hygiene” factor (see Billings & Corn-
elius, Note 1). The MDS results suggest that these two dimensions are
not synonymous. This reinforces the argument that comparisons of
findings among separate bodies of literature are apt to be confusing
because of the use of different intrinsic/extrinsic dimensions.
 160 PERSONNEL PSYCHOLOGY

*STR
SAL.
VAR.

I
ORES
ATH
INHEREN1
IN
WORK

OIND
.ABL

*SEC
UNDERLYING
NEEDS

OFRD

ORSP

.SUP
'GRO

t Pf

*REC
WACH
PRD
*ACC

FIGURE2. Plot of Dimension I1 against Dimension 111 in the three-dimensional AL-

SCAL solution (stimuli symbols explained in Table 2).

The potential usefulness of MDS studies of work outcomes can be

further demonstrated. If a series of MDS analyses found similar di-
mensions across samples of subjects and stimuli, then the empirical
structure would be an aid in selecting a good sample of psychologi-
cally meaningful outcomes for research or evaluation. Taking these
specific results as an example, Figure 1 shows that outcomes high on
one dimension, but low on the other (e.g., recognition and variety)
have been sampled. However, Figure 2 suggests that one off-quadrant
(high level needs, independent of work itself) is oversampled, while
the other (low level needs, inherent in the work itself) is poorly repre-
sented.
While evidence for the multidimensionality of work outcomes has
been found, this study is limited in its generalizability. Since only sub-
jects meeting certain criteria were used, it may not be appropriate to
generalize to subjects with different characteristics. For example, only
subjects with moderate work experience were used; those with no
work experience might use fewer dimensions, while those with more
 BILLINGS AND CORNELIUS 161

experience might use more. Likewise, subjects who gave unreliable

data were excluded. We believe that the length of the questionnaire
and conditions under which the data were collected led to low motiva-
tion for some subjects, resulting in large amounts of error. An alterna-
tive explanation is that some subjects had no firm structure around
which to organize work outcomes. In any event, future MDS studies
of work outcomes should attempt to obtain reliable judgments from
full-time workers, so that more generalizable results will be obtained.
In summary, within the limits of the samples of stimuli and subjects
used, this study has provided empirical evidence that multiple dimen-
sions are necessary to account for the perceptual structure of work
outcomes. If individuals do not use a simple dichotomy to organize
their perceptions of outcomes, then the overly simple “intrinsic/ex-
trinsic” distinction is a poor basis for understanding the nature of
work outcomes.

REFERENCE NOTES
I. Billings, R. S. and Cornelius, E. T. Dimensions underlying the intrinsic/extrinsic di-
chotomy: A literature review and conceptual analysis. Industrial/Organizational Psy-
chology Working Paper No. 78-1, September, 1978.

REFERENCES
Campbell, D. T. and Fiske, D. W. Convergent and discriminant validation by the mul-
titrait-multimethod matrix. Psychological Bulletin. 1959, 56. 8 l - 105.
Campbell, J . P. and Pritchard, R. D. Motivation theory in industrial and organizational
psychology. In M. D. Dunnette (Ed.), Handbook of indu.wial and organizational psy-
chology. Chicago: Rand McNally, 1976.
Cliff, N. and Young. F. W. On the relation between unidimensional judgments and
multidimensional scaling. Organizational Behavior and Human Peformance, 1968, 3 ,
269-285.
Dyer, L. and Parker, D. F. Classifying outcomes in work motivation research: An ex-
amination of the intrinsic-extrinsic dichotomy. Journal of Applied Psychology, 1975,
60, 455-458.
Hackman, J. R. and Oldham, G. R. Motivation through the design of work: Test of a
theory. Organizational Behavior and Human Performance, 1976, 16, 250-279.
Isaac, P. D. and Poor D. 0. S. On the determination of appropriate dimensionality in
data with error. Psychometrika, 1974, 34, 319-330.
Kruskal, J. B. Multidimensional scaling by optimizing goodness of fit to a nonmetric
hypothesis. Psychometrika, 1964, 29( I), 1-27.
Lawler, E. E. and Suttle, J . L. Expectancy theory and job behavior. Organizational Be-
havior and Human Performance. 1973, 9, 482-503.
MacCallum, R. C. and Cornelius, E. T. A monte carlo investigation of recovery of
structure by ALSCAL. Psychometrika, 1977, 42, 40 1-428.
Nygren, T. E. Individual differences in perceptions and political candidates. Journal of
Experimental Social Psychology, 1977, 13, 182-197.
Quinn, R. and Cobb, W. What workers want: Factor analyses ofimportance ratings of
jobfacets. Ann Arbor, Mich.: Institute of Social Research, 1971.
Sanson-Fisher, R. W. and Mulligan, B. The validity of a behavioral rating scale: Appli-
162 PERSONNEL PSYCHOLOGY

cation of a psychophysical technique. Multivariate Behavioral Research, 1911, 12,

351-312.
Shepard, R. N. The analysis of proximities: Multidimensional scaling with an unknown
distance function. Psychometrika, 1962, 21, 125-139.
Sherman, C. R. Nonmetric multidimensional scaling: A monte carlo study of the basic
parameters. Psychometrika, 1912, 31, 323-355.
Spence, I. A monte carlo evaluation of three nonmetric multidimensional scaling al-
gorithms. Psychometrika, 1912, 31, 461-486.
Takane, Y., Young, F. W., and deleeuw, J. Nonmetric individual differences multi-
dimensional scaling: An alternating least squares method with optimal scaling
featrues. Psychometrika, 1917, 42, 7-61.
Torgerson, W. S. Theory and Methods of scaling. New York: John Wiley, 1958.
Young, F. W . Nonmetric multidimensional scaling: Recovery of metric information.
Psychometrika, 1910, 35, 455-413.