Professional Documents
Culture Documents
A Research Note On The Incremental Validity of Job Knowledge and Integrity Tests For Predicting Maximal Performance
A Research Note On The Incremental Validity of Job Knowledge and Integrity Tests For Predicting Maximal Performance
Human Performance
Publication details, including instructions for
authors and subscription information:
http://www.tandfonline.com/loi/hhup20
University of Minnesota
To cite this article: Deniz S. Ones & Chockalingam Viswesvaran (2007) A Research
Note on the Incremental Validity of Job Knowledge and Integrity Tests for
Predicting Maximal Performance, Human Performance, 20:3, 293-303, DOI:
10.1080/08959280701333461
To link to this article: http://dx.doi.org/10.1080/08959280701333461
This article may be used for research, teaching, and private study purposes.
Any substantial or systematic reproduction, redistribution, reselling, loan,
sub-licensing, systematic supply, or distribution in any form to anyone is
expressly forbidden. Terms & Conditions of access and use can be found at
http://www.tandfonline.com/page/terms-and-conditions
Chockalingam Viswesvaran
Florida International University
This study examines the relationship between job knowledge measures and integrity tests in predicting maximal work performance. Our goal is to present data that
support the notion that integrity tests can predict maximal performance, in addition
to their previously meta-analytically established predictive validity for typical performance (Ones, Viswesvaran, & Schmidt, 1993). We also aim to show that integrity tests have incremental validity over cognitively loaded job knowledge measures in the prediction of maximal performance. In the following paragraphs, we
first highlight the conceptual relations and distinctions between typical and maximal performance, making a case for the importance of investigating maximal performance. We next focus on work samples as measures of maximal performance.
Correspondence should be sent to Deniz S. Ones, Department of Psychology, University of Minnesota, 75 East River Road, Minneapolis, MN 554550344. E-mail: Deniz.S.Ones-1@tc.umn.edu
294
We then review the conceptual bases for studying integrity and job knowledge as
determinants of work sample performance and offer specific hypotheses.
MAXIMAL PERFORMANCE:
CONCEPTUAL BASIS AND IMPORTANCE
In recent years, personnel selection researchers have distinguished between typical
and maximal performance (see Sackett, Zedeck, & Fogli, 1988, for the introduction of the concepts to the domain of criterion measurement and Klehe & Anderson, 2005, for a summary of recent research). Typical performance refers to the
day-to-day performance on the job over extended periods of time and comprises of
situations where the performers are not continually being monitored and are not
consciously performing at their optimal level of performance (Sackett et al., 1988).
On the other hand, employees perform maximally when they are continually monitored, bear the consequences of their performance, and therefore consciously try to
perform at their peak capacity. Typical performance refers to what people will do,
and maximal performance refers to what people can do. Thus, typical performance
and maximal performance are theoretically distinct.
Conceptually, maximal performance sets the ceiling on job performance. As
DuBois, Sackett, Zedeck, and Fogli (1993) noted, the distinction between typical
and maximal performance lies in the role that motivation plays in each. Typical
performance differs from maximal performance due to greater variability in motivation among employees when they have choices in three motivational elements:
to perform or not to perform, the level of effort to expend, and to persist or not to
persist. Conversely, it is possible to view maximal performance as a motivationally
charged-up version of typical performance, whereby external monitoring and consequences act to equalize motivational differences among employees. Either way,
ability to perform maximally is a necessary but not a sufficient precursor to typical
performance.
In support of the theoretical distinctions between maximal and typical performance, empirical studies report low to moderate correlations between the two
types of measures. For example, Sackett et al. (1988) found correlations ranging
from .11 to .32 between the two types of measures in a sample of 1,370 supermarket cashiers. Similarly, Klehe and Latham (2003) found that typical and maximal
performance correlated .30. Findings such as these raise the need for validating
predictors with both types of measures (or clearly delineating what criteria will be
predicted by a particular measure). Predictors may have differential usefulness for
maximal and typical criteria, and investigations with each are essential in providing a full understanding of job performance. Describing and explaining how a
given predictor is linked to job performance is essential for building models of
performance.
295
296
297
for maximal performance than for the prediction of typical performance due to the
motivational constraints in maximal performance.
Perhaps more centrally, the construct of integrity may be expected to predict
maximal performance through procedural knowledge. If individuals high on integrity behave according to societal and organizational expectations, they are likely to
know better what the expected behaviors are. In maximal performance situations,
they may have a better sense of how to behave to perform the tasks at hand, to restrain undesirable behaviors, to keep anxiety under control, and to potentially support their performance by additional socially approved behaviors. Integrity is
likely to influence how individuals go about performing tasks even in monitored
settings and even when all individuals are attempting to perform at their peak performance. Given these conceptual reasons, it is important to examine the validity
of integrity tests for measures of maximal performance. This was a unique question answered by our research. We expected integrity test scores to be related to
maximal performance.
298
In recent years, research has concentrated on intercorrelations between cognitive ability and other predictors. The primary reason for this is that cognitive abilities have been found to yield the highest validity across situations and settings
(Schmidt, Ones, & Hunter, 1992). In this domain, prior studies have reported correlations between personality and cognitive ability (e.g., Day & Silverman, 1989).
Based on previous meta-analyses, Schmidt and Hunter (1998) were able to estimate the correlation between cognitive ability tests and alternate predictors.
Schmitt, Rogers, Chan, Sheppard, and Jennings (1997) and Bobko, Roth, and
Potosky (1999) were also able to find intercorrelations between cognitive ability
and other predictors. However, relationships between other cognitive and noncognitive predictors remain unreported. In fact, Bobko et al. (1999), Schmitt et al.
(1997), and Cortina, Goldstein, Payne, Davison, and Gilliland (2000) were unable
to find any correlations between popular noncognitive predictors in job-applicant
samples. In this study, another goal was to estimate the correlation between a job
knowledge test and an integrity test. We then used this information to estimate incremental validities for job knowledge and integrity measures each in the prediction of maximal performance.
In sum, previous research in personnel selection has compared cognitive and
noncognitive measures for predicting typical performance measures (e.g., supervisory ratings of job performance). However, direct comparisons of cognitive and
personality-based measures for predicting maximal performance have been lacking. We offer first such comparisons using job knowledge as our focal cognitively
based predictor and integrity as our focal noncognitive predictor. We assess the validity of an integrity test and a job knowledge measure, both separately and in combination, for maximal performance. We also examine incremental validity of these
predictors over each other for predicting maximal performance.
METHOD
Database and Measures
Data were collected from job applicants to skilled manufacturing jobs. The organization was a Fortune 500 company specializing in the manufacture of many defense-related electrical and mechanical products. The data were collected when
one of the authors was serving as an external consultant to the organization. The
plant for which the hiring was intended was located in a midwestern state. There
were 110 job applicants. All but 2 applicants were male.
The jobs for which the organization was hiring were mechanics, machine shop
operators, machine setters, machine builders, and machine shop supervisors. Most
job duties for all jobs were similar in that they included setting up and adjusting
machine tools, fitting and assembling machine components according to blue-
299
prints and sketches, repairing and maintaining machines and equipment, verifying
alignment and tolerances of parts, and applying knowledge of machine shop procedures and techniques. The Dictionary of Occupational Titles codes for these jobs
were 600130010, 600281022, and 600360014.
All applicants completed a personality-based integrity testthe Personnel Reaction Blank (Gough, 1954). The Personnel Reaction Blank is a well-respected
personality-based integrity test (Ones et al., 1993), with substantial data supporting its construct and criterion-related validity (Viswesvaran & Ones, 1997). Part 1
of the instruments asks about occupational preferences and Part 2 about views and
experiences. An overall score of integrity is calculated based on scored items from
both parts. It was initially created in 1950s based on what was then called the Delinquency scale of the California Psychological Inventory, which has since developed into the Socialization scale. The more recently revised version of the Personnel Reaction Blank (PRB; Gough, Arvey, & Bradley, 2004) was used in this
research. Detailed reliability, factor structure, and validity information on the PRB
is provided in the technical manuals for the test (Gough, 1971; Gough et al., 2004)
and published reviews (Hough, 1990; Viswesvaran & Ones, 1997).
Following the completion of the integrity test, the applicants took a job knowledge test. The job knowledge measure was a custom-made multiple-choice instrument composed of 40 items, scored right or wrong. It was constructed and validated for the organization by a consultant industrialorganizational psychologist
in the late 1970s. The test measured knowledge of mechanical principles and mechanical movements as well as knowledge of reading prints and drawings. The job
knowledge test scores could range from 1 to 200 and had a KR20 reliability of .90
in this sample.
The maximal performance criterion was assessed using a hands-on work sample measure. Job applicants were asked to perform the job they were a candidate
for at three different workstations. Applicants performed the work sample to demonstrate that they could actually do the job. Tasks included setting up and adjusting
machine tools, fitting and assembling machine components according to blueprints, verifying alignment and tolerances of parts, and applying knowledge of machine shop procedures. The work sample took about 3 hours to complete. As such,
this measure captured the can do aspects of job performance. The scores on the
work sample measure were assigned by supervisors who observed applicants performing and could range from 1 to 100. Each rater assigned an overall score to
work sample performance separately at each of the three workstations. Two raters
observed each applicant. The interrater reliability of scores on the composite of the
three work sample measures (in this particular sample) was .74.
Unfortunately, not all applicants completed the job knowledge and the work
sample measures. Eighty-six of the 110 applicants took the job knowledge tests,
and 67 of the 86 completed the maximal performance measures. The reduction in
sample size was based on, primarily, applicants choosing not to continue with the
300
selection process upon realizing that they would have to progressively take job
knowledge tests and that they would then have to actually perform work under observation. Note also that there was no selection done by the organization on the basis of integrity scores prior to the applicants being tested for job knowledge or
work sample performance. Therefore, we do not make any direct range restriction
corrections when reporting correlations.
RESULTS
Observed and corrected intercorrelations among the measures are reported in Table 1. Job knowledge tests had an observed validity of .36 for predicting maximal
performance. Compared to job knowledge tests, integrity test scores had a somewhat lower validity of .27. The criterion unreliability corrected operational validities were .42 and .31 for the job knowledge and integrity measures, respectively.
The job knowledge measure and the integrity test correlated .14.
Regression analyses using both integrity and job knowledge resulted in a multiple R of .49. The operational variance accounted for by job knowledge over the integrity measure for predicting maximal performance was .14. Conversely, operational variance accounted for by the integrity test over the job knowledge measure
was .07.
TABLE 1
Intercorrelations Among Variables
Correlations
Predictors (1, 2)/
Criterion (3)
1 Integrity (PRB)
.80
2 Job knowledge
.14
(.03 to .31)
.27
(.08 to .46)
3 Maximal performance
Regression
.26
.07
.90
.31
(.13 to .50)
.42
(.25 to .59)
.74
.38
.14
.36
(.18 to .54)
R / R2
R2
.49 / .24
Note. Observed correlations are presented below the diagonal; operational validities (corrected
for unreliability in the criterion measure only) are in bold and presented above the diagonal. Values on
the diagonal represent reliability estimates. Confidence intervals (90%, two-tailed) are presented in parentheses. Total sample sizes (Ns) were 110 for the integrity test and 86 for the job knowledge test.
Pairwise N with the criterion measure was 67. The regression is based on operational validities and the
observed predictor intercorrelation. Regression results are for including both integrity and job knowledge measures in the prediction of maximal performance. R2 indicates incremental variance accounted for by adding the predictor represented in the row over the other predictor. PRB = Personnel
Reaction Blank.
301
DISCUSSION
There are several unique contributions of this study. First, this was the first investigation of the validity of integrity tests for predicting maximal performance. Second, it provided a direct comparison of a cognitively based (job knowledge) and a
personality-based (integrity) measure for predicting maximal performance. Both
measures were predictive of the maximal performance criterion and their confidence intervals overlapped. Furthermore, of interest, the confidence intervals of
the validities for both predictor measures in the prediction of maximal performance included their meta-analytic point estimates reported in previous research
for typical performance (Schmidt & Hunter, 1998). Third, we assessed the relationship between two predictors of maximal performance and examined the incremental value of these predictors over each other.
Of course, the sample employed in this study came from one organization and
was comprised of medium-complexity skilled jobs. Future research is needed with
larger samples, different jobs, and different organizations to test the
generalizability of the findings. Given the importance of assessing intercorrelations among predictors in job-applicant samples, we hope this study generates
more research along these lines. Results reported here should facilitate future
meta-analytic efforts to estimate the intercorrelations among predictors in applicant samples. The results reported here should also be useful for practitioners in
designing selection systems as well as being useful to researchers in facilitating the
development of comprehensive theories of work behavior by providing estimates
of how different predictors correlate in applicant samples and how they relate to
measures of maximal performance.
In this research note, we examined the independent and combined effects of integrity and job knowledge on maximum performance. Previous papers on typical
and maximal performance have suggested that cognitively based predictors should
predict maximal performance and noncognitive measures should have limited usefulness for the same criterion. By offering evidence that integrity tests can predict
maximal performance, we provide a justification for future studies to further examine the mechanisms through which the construct of integrity influences maximal performance (i.e., procedural knowledge and motivation). Such future research can also establish the relative importance of motivational processes versus
procedural knowledge in both typical and maximal performance. Another avenue
for future research may entail partitioning the explanatory variance in predictors
into maximal and typical components. Such investigations of interactive effects
are likely to enrich our understanding of both the predictor (e.g., Digmans factor
alpha) and criterion constructs. Finally, future research is needed to assess the
intercorrelations in other applicant samples so as to facilitate constructions of the
selection models and integrated selection systems.
302
ACKNOWLEDGMENT
Both authors contributed equally; the order of authorship is arbitrary.
REFERENCES
Bobko, P., Roth, P. L., & Potosky, D. (1999). Derivations and implications of a meta-analytic matrix incorporating cognitive ability, alternative predictors, and job performance. Personnel Psychology, 52,
561589.
Borman, W. C., White, L. A., Pulakos, E. D., & Oppler, S. H. (1991). Models of supervisory job performance ratings. Journal of Applied Psychology, 76, 863872.
Campbell, J. P. (1990). Modeling the performance prediction problem in industrial and organizational
psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational
psychology (Vol. 1, pp. 687732). Palo Alto, CA: Consulting Psychologists Press.
Campbell, J. P., & Knapp, D. J. (Eds.). (2001). Exploring the limits in personnel selection and classification. Mahwah, NJ: Lawrence Erlbaum Associates.
Cortina, J. M., Goldstein, N. B., Payne, S. C., Davison, K. H., & Gilliland, S. W. (2000). The incremental validity of interview scores over and above cognitive ability and conscientiousness scores. Personnel Psychology, 53, 325352.
Cronbach, L. J. (1960). Essentials of psychological testing (2nd ed.). Oxford, England: Harper.
Day, D. V., & Silverman, S. B. (1989). Personality and job performance: Evidence of incremental validity. Personnel Psychology, 42, 2536.
Digman, J. M. (1997). Higher-order factors of the Big Five. Journal of Personality and Social Psychology, 73, 1246-1256.
DuBois, C. L., Sackett, P. R., Zedeck, S., & Fogli, L. (1993). Further exploration of typical and maximum performance criteria: Definitional issues, prediction, and white-black differences. Journal of
Applied Psychology, 78, 205211.
Gough, H. G. (1954). Personnel Reaction Blank. Palo Alto, CA: Consulting Psychologists Press.
Gough, H. G. (1971). The assessment of wayward impulse by means of the Personnel Reaction Blank.
Personnel Psychology, 24, 669677.
Gough, H. G., Arvey, R. D., & Bradley, P. (2004). Manual for the Personnel Reaction Blank. Champaign, IL: IPAT.
Hough, L. M. (1990). Personnel Reaction Blank (1988 edition). In J. Hogan & R. Hogan (Eds.), Business and industry testing (pp. 434440). Austin, TX: Pro-Ed.
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternate predictors of job performance.
Psychological Bulletin, 96, 7298.
Kirk, A. K., & Brown, D. F. (2003). Latent constructs of proximal and distal motivation predicting performance under maximum test conditions. Journal of Applied Psychology, 88, 4049.
Klehe, U., & Anderson, N. (2005). The prediction of typical and maxim um performance in employee
selection. In A. Evers, N. Anderson, & O. Voskuijl (Eds.), Handbook of personnel selection (pp.
331353). Malden, MA: Blackwell.
Klehe, U., & Latham, G. (2003, April). Towards an understanding of the constructs underlying situational and patterned behavior description interviews in predicting typical versus maximal performance. Paper presented at the 18th annual conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
McCloy, R. A., Campbell, J. P., & Cudeck, R. (1994). A confirmatory test of a model of performance
determinants. Journal of Applied Psychology, 79, 493505.
303
Ones, D. S. (1993). The construct validity of Integrity tests. Unpublished doctoral dissertation, University of Iowa, Iowa City.
Ones, D. S., & Viswesvaran, C. (1998). Integrity testing in organizations. In S. B. Baracharach, A.
OLeary-Kelly, J. M. Collins, & R. W. Griffin (Eds.), Dysfunctional behavior in organizations: Violent and deviant behavior (pp. 243276). Stanford, CT: Elsevier Science/JAI Press.
Ones, D. S., Viswesvaran, C., & Dilchert, S. (2005a). Cognitive ability in personnel selection decisions.
In A. Evers, O. Voskuijl, & N. Anderson (Eds.), Handbook of selection (pp. 143173). Oxford, UK:
Blackwell.
Ones, D. S., Viswesvaran, C., & Dilchert, S. (2005b). Personality at work: Raising awareness and correcting misconceptions. Human Performance, 18, 389404.
Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (1993). Comprehensive meta-analysis of integrity test
validities: Findings and implications for personnel selection and theories of job performance. Journal of Applied Psychology, 78, 679703.
Ployhart, R., Lim, B., & Chan, K. (2001). Exploring relations between typical and maximal performance ratings and the five-factor model of personality, Personnel Psychology, 54, 809843.
Sackett, P. R., Zedeck, S., & Fogli, L. (1988). Relations between measures of typical and maximum job
performance. Journal of Applied Psychology, 73, 482486.
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262274.
Schmidt, F. L., Hunter, J. E., & Outerbridge, A. N. (1986). Impact of job experience and ability on job
knowledge, work sample performance, and supervisory ratings of job performance. Journal of Applied Psychology, 71, 432439.
Schmidt, F. L., Ones, D. S., & Hunter, J. E. (1992). Personnel selection. Annual Review of Psychology,
43, 627670.
Schmitt, N., Rogers, W., Chan, D., Sheppard, L., & Jennings, D. (1997). Adverse impact and predictive
efficiency of various predictor combinations. Journal of Applied Psychology, 82, 719730.
Viswesvaran, C., & Ones, D. S. (1997). A review of the Personnel Reaction Blank (PRB). Security
Journal, 8, 129132.