APPLIED COGNITIVE PSYCHOLOGY

Appl. Cognit. Psychol. 24: 27–40 (2010)

Published online 23 December 2008 in Wiley InterScience
(www.interscience.wiley.com) DOI: 10.1002/acp.1541

Cognitive Processes in Evaluating Reference Letters

KIN FAI ELLICK WONG1*,y and JAMES D. WERBEL2z

1
Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong, SAR, China
2
Iowa State University, Ames, Iowa, USA

SUMMARY
This study simultaneously evaluates confirmatory information search theory and dual-process theory
in a selection process that uses reference letters as the evaluation tool. Confirmatory information
search suggests that evaluators give attention to information that is congruent with first impressions.
Dual-processes theory suggests that evaluators give attention to information that is incongruent with
first impressions. Using a sample of undergraduate students, in two separate experiments, we found
that a poor impression from the résumé led to more positive information and less negative
information searching in reference letters than did a good impression. These results of both
experiments suggest that reference letter evaluators are likely to use cognitive processes associated
with dual-process theory. Copyright # 2008 John Wiley & Sons, Ltd.

The selection process is a multiple hurdle process with the sequential use of different
selection tools. The process normally begins with evaluation of job applications and is
followed by other selection tools such as interviews, tests and reviews of reference letters.
The sequential evaluation process is open to evaluators’ impressions formed at an early
stage (e.g. from the applicant’s résumé) influencing subsequent evaluations in later stages
such as reference letter evaluation and interview (Dipboye, 1982).
The current study seeks to increase the understanding of the effect of initial impressions
on reference letter evaluations by simultaneously testing two different theories of
sequential evaluation processes in the selection context of reference letter evaluations.
According to confirmatory search theory (Binning, Goldstein, Garcia, & Scattaregia, 1988;
Dougherty, Turban, & Callender, 1994), evaluators using post-résumé selection tools, such
as reference letters, are inclined to gather and attend to information that is congruent with
initial impressions (Dipboye, 1982). In contrast, the dual-process theory of social judgment
(Chaiken, 1980; Fiske, 1988) suggests that evaluators first form categories (Feldman, 1981;
Kulik & Ambrose, 1993) or prototypes of job applicants from their résumés. When
subsequently presented information is inconsistent with that prototype, evaluators will
deeply process the information. This theory suggests that evaluators using post-résumé

*Correspondence to: Kin Fai Ellick Wong, Hong Kong University of Science and Technology, Clear Water Bay,
Hong Kong, SAR, China. E-mail: mnewong@ust.hk
y
Associate Professor.
z
Professor of Management.

Copyright # 2008 John Wiley & Sons, Ltd.

28 K. F. E. Wong and J. D. Werbel

selection tools, such as reference letters, are inclined to gather and attend to information
that is incongruent with initial impressions.
In the next section, we compare these two theories in relation to the evaluation of
reference letters in a sequential selection process. Then, we report on two experiments
designed to simultaneously test if more cognitive attention is paid to congruent or
incongruent information in the reference letters. We used reference letters containing both
positive and negative information in Experiment 1 and used letters containing only positive
information in Experiment 2. Converging results from the two experiments could then be
used to support one of these two theories.

COGNITIVE PROCESSES AND IMPRESSION EFFECTS

Confirmatory information search

The confirmatory information search explains the impression effects within a motivational
framework. The underlying motive for a confirmatory information search is linked to
dissonance theory (Festinger, 1957), which assumes people are uncomfortable with
information that challenges an established belief (Aronson, 1969). Accordingly, a person is
motivated to look for consistent evidence and minimizes inconsistent evidence with initial
impressions. This biased information search in turn escalates their commitment to their
prior belief, resulting in decision biases such as confirmation bias (Jonas, Schulz-Hardt,
Frey, & Thelen, 2001; Schulz-Hardt, Frey, Lüthgens, & Moscovici, 2000) and escalation of
commitment (Staw, 1976; Wong, Kwong, & Ng, 2008).
Using the confirmatory search explanation, Dipboye (1982) presented a process model
of interviewing, suggesting that both cognitive and behavioural processes mediate the
effects of an applicant’s pre-interview impression on subsequent evaluations. The pre-
interview impressions cause evaluators to maintain a similar evaluation of an applicant
throughout different stages of the selection process. From a cognitive bias, interviewers are
likely to selectively attend to positive information while ignoring negative information
when evaluating job applicants with favourable initial impressions from résumés. From a
behavioural bias, interviewers are likely to ask supportive or confirming questions of
candidates who made favourable initial impressions while asking more difficult questions
of candidates who made unfavourable initial impressions. Research supports that the
behavioural bias aspects of the confirmatory search process in application selection is
largely due to the types of questions asked during interviews (Cable & Gilovich, 1998;
Dipboye, Stramler, & Fontenelle, 1984; Dougherty et al., 1994).
Dipboye (1982) hypothesizes that cognitive processes associated with initial
impressions influence attention paid to information gathered from subsequent selection
hurdles. According to the dissonance theory (Aronson, 1969; Festinger, 1957), an evaluator
is more likely to process impression-congruent information and ignores impression-
incongruent information to reduce challenges to their beliefs (i.e. the impression).
Therefore, the evaluator would collect evidence that reinforces the initial impression. That
is, the evaluator is likely to attend to consistent information that confirms the initial
impression from a résumé and ignore inconsistent information in order to maintain or
strengthen initial perceptions from the résumé.
Hypothesis 1. Reference letter evaluators will more likely attend to impression-con-
gruent information than to impression-incongruent information in reference letters.

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp
 Reference letters evaluation 29

Dual-process approach
Dual-process approach explains the impression within a cognitive capacity framework by
suggesting that there are two independent and related modes of information processing.
The first mode suggests that judgments of others rely heavily on heuristics (Maheswaran &
Chaiken, 1991) or social categories (Fiske & Pavelchak, 1986). This process is automatic
and rapid because individuals apply heuristics or simple decision rules to process a limited
subset of available information. The categories may be relatively simple dichotomous
categories or more complex multidimensional categories. This mode has been called
‘heuristic processing’ (Maheswaran & Chaiken, 1991) or ‘category-based processing’
(Fiske & Pavelchak, 1986). The second mode suggests that people rely on comprehensive
information to make judgment. This mode has been called ‘systematic processing’
(Maheswaran & Chaiken, 1991) or ‘feature-based processing’ (Fiske & Pavelchak, 1986),
is relatively under individual conscious control and more demanding in mental resources.
The relative influences of the category-based mode are particularly pronounced when
insufficient mental resources are given to the feature-based mode. The use of the category-
based processing leads evaluators to interpret neutral information to be consistent with
their existing categories, resulting in impression effects (Feldman, 1981). However, when
evaluators encounter counter-category information, which contradicts prototypic beliefs,
the feature-based processing mode is triggered. This forces the evaluator to scrutinize the
available information more carefully to resolve inconsistencies with social categories.
Maheswaran and Chaiken (1991) reported that impression-incongruent information
undermines individuals’ confidence of a categorical judgment. This increases the amount
of information processing needed to resolve the discrepancies (Maheswaran & Chaiken,
1991; Stangor & McMillan, 1992).
Applying this approach to the selection process, the dual-process model suggests that
evaluators initially use category-based processing (i.e. initial impressions) to evaluate early
selection steps. Then, the evaluators would rely on feature-based processing when
encountering impression-incongruent information in later selection steps such as reference
letters. Consequently, the impression-incongruent information would be deeply processed
to resolve discrepancies between initial impressions and subsequent information. Thus,
Hypothesis 2. Reference letter evaluators will more likely attend to impression-incon-
gruent information than to impression-congruent information in reference letters.
In summary, the two approaches root from different psychological motives. The primary
motive for confirmatory search approach is self-image protection. Evaluators use the
information to reinforce or to affirm the impression formed at an earlier stage. The dual-
process approach describes a cognitive mechanism that seeks to have a more rational
understanding of the reality, suggesting that evaluators use the information to verify the
accuracy of an impression. The collected information is used to counterbalance and alter
their initial impressions.

Overview
The two experiments reported here required participants to read and evaluate applicants’
reference letters after reviewing their résumés. They read reference letters using a self-
paced mouse-tracking program. The self-paced mouse-tracking method is useful for
revealing cognitive processes (Haberlandt & Graesser, 1985, 1989). This method enables
recording of the real-time information collection pattern during different cognitive tasks. In

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp
30 K. F. E. Wong and J. D. Werbel

particular, Just, Carpenter, and Woolley (1982) reported a correlation of .57 between data
collected from eye-tracking and self-paced reading methods, suggesting that the self-paced
reading method captures a large common variance with collecting information through eye
movement.

EXPERIMENT 1

In this experiment, we asked participants to evaluate applicants’ qualifications for an entry-

level banking position using multiple hurdles. Participants first read a target applicant’s
résumé, which had been manipulated to be high or low in quality. Next, participants read
reference letters using a self-paced mouse-tracking program. Note that following previous
studies of confirmatory search and information gathering in managerial decision making
and social judgment (Conlon & Park, 1987; Jonas et al., 2001), we intentionally masked the
words until the evaluators pointed at them. This ensured that attention to information was
directly tracked. Masking the words not pointed to in the mouse-tracking procedure was
deemed necessary to test the two theoretical approaches unobtrusively.

Method
Participants
One hundred and thirty-eight undergraduates (46 men and 92 women) enrolled in
Organizational Behaviour classes at Hong Kong University of Science and Technology
volunteered for the study in exchange for course credit. Over 90 per cent of the students in
these classes agreed to participate in the study. They were Chinese-English bilinguals who
had been taught in an English language educational environment for more than 15 years.
Their average age was 19.52, ranging from 19 to 22.

Materials and measures

Mouse tracking program. The program was written with Delphi and was compatible with
major windows-based operating systems such as Windows 98 and Windows XP. The
outlook of the program was displayed in Figure 1. Before participants started reading a
reference letter, the words of the reference letter were masked by rectangular boxes.
Participants then moved the cursor on the screen by controlling the computer mouse at their
own speed. The rectangular box disappeared when the cursor was being pointed to the
word. The word was masked again when the cursor was moved away. Thus, only a single
word was displayed at any given time. Participants could backtrack to read any word in the
reference letter. The cursor did not move when there were mouse lifts for repositioning.
The program automatically recorded the cursor movements by the x,y coordinates on the
computer screen. To measure the amount of attention paid to information in the reference
letter, the software program kept track of the number of times each word was unmasked
(number of pointings) and the duration of time each word was unmasked. The current
research used mechanical mouse.

Résumé. Three résumés were developed: two for filler applicants and one for the target
applicant. We introduced the filler applicants as an attempt to make the task more realistic
by increasing the variety of applicants. The quality of the target applicant’s résumé was
manipulated to have relative degrees of favourability by changing information such as

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp
 Reference letters evaluation 31

Figure 1. The outlook of the program display

public examination results (i.e. 1A, 3Bs and 3Cs vs. 2Bs, 4Ds and 2Ds in HKCEE), GPA
(i.e. 3.55 vs. 2.75), school of graduation (a first-tier university vs. a third-tier university) and
work experience (i.e. part-time bank teller in a large bank vs. no working experience).

Reference letter. Three reference letters were developed, each with 119 words (i.e. Version 1,
Version 2 and Version 3). Half of the participants read Version 1 for the target applicant
and Versions 2 and 3 for the filler applicants. Half read Version 2 for the target
applicant and Versions 1 and 3 for the filler applicants. Thus, we replicated the same
manipulation and procedure with two different versions of the reference letters to see if a
similar pattern of results could be found.
The reference letters contained two paragraphs. The first paragraph described the
applicant’s skills, motivation and performance. The second paragraph described the
applicant’s hobbies and other general information that was less relevant to the selection
context. Therefore, we analysed only the information search pattern of the first paragraph.
This paragraph consisted of six sentences. As the cursor began by pointing at the first word
of the letter, the reading time for the first sentence was dramatically large though evaluators
in fact had not started to read. Accordingly, we treated the first sentence as the ‘practice
trial’ and excluded it from further analyses. To determine the quality of the information in
the remaining five sentences, we asked ten judges to rate each sentence on a seven-point
scale (1 ¼ very negative, 4 ¼ neutral and 7 ¼ very positive). According to the ratings, we
grouped the first three sentences to be the General Information Region (M ¼ 4.63,
SD ¼ 0.40 for Version 1; M ¼ 4.57, SD ¼ 0.22 for Version 2), the fourth sentence to be the
Positive Region (M ¼ 6.7, SD ¼ 0.48 for Version 1; M ¼ 6.5, SD ¼ 0.71 for Version 2) and
the fifth sentence to be the Negative Region (M ¼ 2.4, SD ¼ 0.51 for Version 1; M ¼ 2.3,
SD ¼ 0.48 for Version 2). All differences across the three conditions for the two versions
were significant, p < .001.

Résumé evaluation. The measures consisted of four items. The first three items asked the
evaluator to assess the skills, knowledge and abilities (e.g. ‘The skills needed to be a loan
officer’) on the basis of the information from résumé. The scale responses ranged from

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp
32 K. F. E. Wong and J. D. Werbel

1 (not having the attribute) to 10 (having a superior attribute). The fourth item asked the
evaluator to give an overall assessment of the applicant (e.g. ‘Overall how would you assess
the résumé for a loan officer position?’). The scale responses ranged from 1 (unqualified for
the job) to 10 (extremely qualified for the job). The reliability a of the evaluation of the
target person’s résumé was .93.

Reference letter evaluation. The measures consisted of four items using a ten-point scale
with the scale responses like those of the résumé evaluation. The items asked the evaluator
to assess the skills, knowledge and abilities and to give an overall assessment of the
applicant on the basis of the information from reference letters. The scale was anchored by
not having the attribute and having a superior attribute. The reliability a of the evaluation
of the target person’s reference letter was .91.

Procedure, apparatus and design

Participants were tested in a large computer laboratory, with the group size averaging 20.
The experiment involved three phases. In phase 1, participants read two short passages
using the mouse-tracking program to familiarize them with the skill of controlling the
mouse for reading. In phase 2, participants were given a personnel selection case. In this
case, participants were asked to assume the role of a manager of the loan department of a
large international bank who was now recruiting new loan officers. There were three
applicants fulfiling the minimum requirement and the participants were asked to evaluate
all applicants’ résumés and reference letters. After reading all of the information, they
were to decide which two applicants they would select for job interviews and complete a
résumé evaluation form for all applicants. In phase 3, participants were asked to read the
reference letters displayed on the screen by using the mouse-tracking program. They read
one letter and then completed the reference evaluation form of that applicant before reading
the second letter. The target applicant’s reference letter was always presented second.

Data treatment of the on-line information search data

Information search pattern was reflected in reading time duration and number of pointings.
Reading time duration was the total amount of time spent on reading a region of
information. The number of pointings was the number of times a set of words was pointed
to with a cursor. For example, when reading a sentence ‘My name is John’, a participant
may read it as the following sequence: My ! name ! is ! John ! name ! John. The
reading time duration was the length of time it took to read these six words. The number of
pointings was six in total.
Given that people read at different speeds, it is important to standardize individual
differences in reading times and pointings. We used the proportion of reading times and the
proportion of pointings in a region to indicate the relative amount of attention paid to a
particular region to standardize individual differences in reading abilities. Specifically, for
each evaluator, a high proportion of reading time or pointings in a particular region
indicates that the evaluators paid relatively more attention to this region.

Results
Preliminary analyses
Manipulation check. The mean résumé evaluation for the poor impression condition
(M ¼ 5.31, SD ¼ 1.12) was significantly lower than that for the good impression condition

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp
 Reference letters evaluation 33

(M ¼ 7.57, SD ¼ 1.15), F(1, 136) ¼ 124.06, MSE ¼ 1.42, p < .001, h2p ¼ .48. This finding
indicates that the manipulation of the initial impression was successful.

Impression effects. A 2 (Impression: Good vs. Poor)
2 (Version: 1 vs. 2) ANOVA

revealed that only the main effect of the initial impression was significant, F(1, 134) ¼
10.18, MSE ¼ 1.42, p < .005, h2p ¼ .07, indicating that the mean reference letter evaluation
for the poor impression condition (M ¼ 6.14, SD ¼ 1.39) was significantly lower than that
for the good impression condition (M ¼ 6.76, SD ¼ 0.98). This provides evidence that
initial impression effect exists in reference letter evaluation.

Impression effects on information search

Table 1 summarizes the information search data as a function of impression (Poor vs.
Good). We first conducted a 3 (Region: Neutral, Positive or Negative)
2 (Impression:
Good vs. Poor) repeated measure ANOVA to examine whether the impression effects in the
three regions were different. Results showed a significant Region
Impression interaction
when the proportion of reading time was the dependent variable, F(2, 272) ¼ 4.14,
MSE ¼ 31.73, p < .005, h2p ¼ .03 and when the proportion of pointings was the dependent
variable, F(2, 272) ¼ 4.26, MSE ¼ 31.53, p < .05, h2p ¼ .03. To have a better understanding
of these interaction patterns, we conducted univariate ANOVA to examine the impression
effects on each region.
We found no significant effects for the neutral region. For the positive region,
participants in the poor impression condition had significantly higher proportion of reading
time in this region (M ¼ 14.98, SD ¼ 5.57) than did those in the good impression condition
(M ¼ 12.96, SD ¼ 4.03), F(1, 136) ¼ 6.81, MSE ¼ 20.70, p < .05, h2p ¼ .05. Similarly,
participants in the poor impression condition had significantly higher proportion of
pointings in this region (M ¼ 16.33, SD ¼ 5.24) than did those in the good impression
condition (M ¼ 13.95, SD ¼ 4.11), F(1, 136) ¼ 8.89, MSE ¼ 21.97, p < .01, h2p ¼ .06.
For the negative region, participants in the poor impression condition had significantly
lower proportion of reading time in this region (M ¼ 37.93, SD ¼ 4.96) than did those in the
good impression condition (M ¼ 39.81, SD ¼ 6.70), F(1, 136) ¼ 3.44, MSE ¼ 35.24,
p ¼ .07, h2p ¼ .03. No significant results were found for the pointing data in the negative
region.

Table 1. The mean of proportions of reading time and pointings as a function of Region and
Impression in Experiment 1
Proportion of reading time (%) Proportion of pointings (%)

Poor impression Good impression Poor impression Good impression

Neutral
Sentence 1 25.63 (6.44) 25.08 (5.48) 26.15 (5.09) 26.29 (5.66)
Sentence 2 11.25 (2.25) 11.81 (3.39) 11.88 (3.16) 12.04 (3.00)
Sentence 3 10.20 (4.21) 10.35 (4.43) 9.53 (3.46) 10.60 (3.23)
Positive
Sentence 4 14.98 (5.57) 12.96 (4.03) 16.33 (5.24) 13.95 (4.11)
Negative
Sentence 5 37.93 (4.96) 39.81 (6.70) 36.11 (6.35) 37.12 (6.85)
Note. Numbers in parentheses represent standard deviations.

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp
34 K. F. E. Wong and J. D. Werbel

We also constructed two information search indices to indicate the amount of positive
information sought from the reference letter. The reading time index was the proportion of
reading time in the positive region/proportion of reading time in the negative region. The
pointing index was the proportion of pointings in the positive region/proportion of
pointings in the negative region. A high value on these indices indicates that individuals
pay relatively more attention to positive information but tend to ignore negative
information.
Results showed that participants in the poor impression condition had a significantly
higher mean reading time index (M ¼ 0.40, SD ¼ 0.13) than did those in the good impression
condition (M ¼ 0.34, SD ¼ 0.12), F(1, 136) ¼ 8.2, MSE ¼ .02, p < .01, h2p ¼ .06. Similarly,
participants in the poor impression condition had a significantly higher mean pointing index
(M ¼ 0.47, SD ¼ 0.18) than did those in the good impression condition (M ¼ 0.39,
SD ¼ 0.13), F(1, 136) ¼ 9.29, MSE ¼ .02, p < .005, h2p ¼ .06). Version did not interact with
impression on both reading time and pointing indices, indicating that results associated with
the two versions are comparable.
In sum, these results indicated that participants paid more attention to impression-
incongruent information. Thus, the information search pattern was more consistent with
the dual-process approach than with the confirmatory search approach.

Discussion
These results are more consistent with the predictions derived from the dual-process theory
than the confirmatory search theory. Evaluators who had a poor initial impression sought
more positive information from reference letters than did those who had a good initial
impression. Reference evaluators who form a good initial impression sought more negative
information with reference letters than did those with a poor initial impression.
In reality, however, reference letters seldom contain negative information (Heneman &
Judge, 2003). Because people tend to be particularly sensitive to negative information
(Dunegan, 1993; Kuvaas & Selart, 2005; Wofford & Goodwin, 1990), there is a concern
about the generality of the findings of Experiment 1. To address this concern, we replicated
Experiment 1 with a manipulation of impression-congruence embedded in reference letters
containing positive information only.

EXPERIMENT 2

Method
Participants
One hundred and twenty undergraduates (46 men and 74 women) from the same
population pool described in Experiment 1 participated in this experiment. Their average
age was 19.63, ranging from 19 to 22.

Materials and measures

Résumé. Three résumés were developed; two for filler applicants and one for the target
applicant. There were two versions of résumés for the target applicant. In the academic
performance dominant version, the résumé indicated that the applicant had a 3.5 GPA and
was a committee member of only one student club. In the social skill dominant version, the

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp
 Reference letters evaluation 35

résumé indicated that the applicant had a 2.7 GPA and was an officer in three student clubs
and a member of another three clubs.

Reference letter. Three reference letters were developed, each with 131 words. One was
for the target applicant and the other two were for the filler applicants. Participants were
told that all letters were written by instructors of the applicants’ undergraduate courses.
The target letter had one paragraph with 11 sentences. The fourth sentence conveyed
positive information about the applicants’ academic performance, stating that the applicant
got an ‘A-’ from the course and developed good writing skills by writing an excellent term
paper. The fifth and sixth sentences conveyed positive information about the applicants’
social skills, stating that the applicant had very good social skills and showed leadership in
class.

Résumé evaluation. The measures consisted of six items using a ten-point scale (1 ¼ very
poor to 10 ¼ very excellent). Two items asked the evaluator to assess the applicants’
academic performance (e.g. ‘How would you rate the applicant’s academic performance as
revealed from his/her résumé?’ and ‘How would you rate the applicant’s academic
achievement as revealed from his/her résumé?’). Two items asked about social skills (e.g.
‘How would you rate the applicant’s social experience as revealed from his/her résumé?’
and ‘How would you rate the applicant’s social skills?’. Two items asked about the overall
impression (e.g. ‘How would you rate the applicant’s overall quality?’ and ‘What is your
general impression on this applicant?’). The reliability a for academic performance, social
skills and overall ratings were .73, .78 and .73, respectively.

Reference letter evaluation. All materials were the same as those for résumé evaluation,
except that participants were asked to evaluate on the basis of the information from
reference letters. The reliability a for academic performance, social skills and overall
ratings were .82, .87 and .70, respectively.

Procedure, apparatus and design

All aspects were the same as those in Experiment 1, except that there were one between-
subject factor, Version (academic performance dominant vs. social skills dominant), and
one within-subject factor, reference region (academic performance vs. social experience).

Data treatment of the on-line information search data

All aspects were the same as those in Experiment 1.

Results
Preliminary analyses
Manipulation check. In résumé evaluation, the mean rating of academic performance of
those who read the academic performance dominant version (M ¼ 6.3, SD ¼ 1.17) was
significantly higher than the mean rating of those who read the social skills dominant
version (M ¼ 5.13, SD ¼ 1.35), F(1, 118) ¼ 25.69, MSE ¼ 1.59, p < .01, h2p ¼ .18. The
mean rating of social skills of those who read the social skills dominant version (M ¼ 6.03,
SD ¼ 1.12) was significantly higher than the mean rating of those who read the academic
performance dominant version (M ¼ 5.40, SD ¼ 1.34), F(1, 118) ¼ 7.87, MSE ¼ 1.53,

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp
36 K. F. E. Wong and J. D. Werbel

p < .01, h2p ¼ .06. This suggested that the manipulation was successful. In addition, the
mean rating on overall résumé quality in the academic performance dominant condition
(M ¼ 5.38, SD ¼ 1.33) was not statistically different from that in the social skills dominant
condition (M ¼ 5.08, SD ¼ 1.21), F(1, 118) ¼ 1.67, p > .1.

Impression effects. In reference letter evaluation, the mean rating of academic

performance of those who read the résumé with the academic performance dominant
version (M ¼ 7.57, SD ¼ 0.93) tended to be higher than the mean rating of those who read
the résumé with the social skills dominant version (M ¼ 7.28, SD ¼ 1.09), though the
difference was not statistically significant, F(1, 118) ¼ 2.35, MSE ¼ 1.03, p ¼ .13,
h2p ¼ .02. The mean rating of social skills of those who read the résumé with the social skills
dominant version (M ¼ 6.88, SD ¼ 1.21) was significantly higher than the mean rating of
those who read the résumé with the academic performance dominant version (M ¼ 6.27,
SD ¼ 1.10), F(1, 118) ¼ 8.53, MSE ¼ 1.34, p < .01, h2p ¼ .07. The mean rating on overall
résumé quality in the academic performance dominant condition (M ¼ 8.17, SD ¼ 0.72)
was not statistically different from that in the social skills dominant condition (M ¼ 8.28,
SD ¼ 0.67), F(1, 118) ¼ 1.67, p > .1. These results indicated that reference evaluations
were generally shifted towards the impressions created by résumés.

Impression effects on information search

Table 2 summarizes the information search data as a function of impression (academic
performance dominant vs. social skills dominant). We first conducted a 2 (Region:
academic information vs. social information)
2 (Impression: academic experience
dominant vs. social skills dominant) repeated measure ANOVA to examine whether the
impression effects in the two regions were different. Results showed a significant
Region
Impression interaction when the proportion of pointings was the dependent
variable, F(1, 118) ¼ 17.52, MSE ¼ 13.33, p < .01, h2p ¼ .13 and when the proportion of
reading time was the dependent variable, F(1, 118) ¼ 25.03, MSE ¼ 12.97, p < .01,
h2p ¼ .18. To have a better understanding of these interaction patterns, we conducted
univariate ANOVAs to examine the impression effects on each region.
For the academic information region, participants in the academic performance
dominant condition had significantly less proportions of pointings in this region (M ¼ 8.8,
SD ¼ 3.67) than did those in the social skills dominant condition (M ¼ 10.68, SD ¼ 3.44),
F(1, 118) ¼ 8.41, MSE ¼ 12.67, p < .05, h2p ¼ .07. Similarly, participants in the academic
performance dominant condition had significantly less proportions of reading time in this

Table 2. The mean of proportions of reading time and pointings as a function of Region and
Impression in Experiment 2
Proportion of pointings (%) Proportion of reading times (%)

Academic Social Academic Social

performance skills performance skills
dominant dominant dominant dominant
Academic region 8.80 (3.67) 10.68 (3.44) 7.40 (3.86) 9.08 (3.98)
Social region 12.81 (4.46) 10.22 (3.73) 12.34 (3.92) 9.32 (3.57)
Note. Numbers in parentheses represent standard deviations.

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp
 Reference letters evaluation 37

region (M ¼ 7.40, SD ¼ 3.86) than did those in the social skills dominant condition
(M ¼ 9.08, SD ¼ 3.98), F(1, 118) ¼ 5.46, MSE ¼ 15.4, p < .05, h2p ¼ .04.
An opposite pattern was found in the results pertaining to the social information region.
Participants in the academic performance dominant condition had significantly more
proportions of pointings in this region (M ¼ 12.81, SD ¼ 4.46) than did those in the social
skills dominant condition (M ¼ 10.22, SD ¼ 3.73), F(1, 118) ¼ 7.55, MSE ¼ 16.9, p < .05,
h2p ¼ .06. Similarly, participants in the academic performance dominant condition had
significantly more proportions of reading time in this region (M ¼ 12.34, SD ¼ 3.92) than
did those in the social skills dominant condition (M ¼ 9.32, SD ¼ 3.57), F(1, 118) ¼ 19.5,
MSE ¼ 14.04, p < .05, h2p ¼ .14.
In sum, consistent with results of Experiment 1, these results indicated that participants
paid more attention to impression-incongruent information. Thus, the information search
pattern was more consistent with the dual-process approach than with the confirmatory
search approach.

GENERAL DISCUSSION

The two experiments from the present study consistently showed that evaluators sought
impression-incongruent information more than impression-congruent information from
reference letters. The contribution of this research is that it offers the insight into the
cognitive processes in reference letter evaluation. This study shows that people evaluate
reference letters by following a process described by dual-process approach, while
provides evidence against the evaluation process the confirmatory search approach
suggests. That is, reference letter evaluations are jointly determined by two independent
processes. The impression formed after reviewing an applicant’s résumé determines the
mental category for this person as either ‘good’ or ‘poor’ (i.e. the category-based process;
Feldman, 1981). At the same time, evaluators may encounter information that is
inconsistent with the initial impressions. The impression-incongruent information then
triggers a feature-based process leading to a comprehensive information search.
Another contribution of this study is that it suggests that reference letters as a selection
source might be more useful than previously believed. Reference letters are used to provide
qualitative information about external job applicants (Muchinsky, 1979) and significantly
influence evaluators’ assessments of job applicants (Levy-Leboyer, 1994). Furthermore,
both personal and employment references are widely used in many countries as a
supplemental selection tool (Ryan, McFarland, Baron, & Page, 1999). In spite of the wide
use of reference letters in selection, some researchers question their use because they are
biased, unreliable and poor predictors of job performance (Aamodt, Bryan, & Whitcomb,
1993; Baxter, Brock, Hill, & Rozelle, 1981). In contrast to these commonly held views, this
study demonstrates that the impression incongruent information in reference letters leads
to comprehensive information search, which should reduce but not reinforce impression
effects.
An interesting question concerns why the cognitive processes underlying impression
effects from interviews (i.e. the confirmatory search mechanism, Dipboye, 1982) fail to
operate in the context of reference letter evaluations. There may be two different issues that
create the difference. First, interviewers, especially those who conduct unstructured
interviews, have a high degree of freedom to frame questions that elicit expected responses.
Given that they have the opportunity to frame questions (Dougherty et al., 1994), they are

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp
38 K. F. E. Wong and J. D. Werbel

likely to control the type of information that is provided to them by the interviewees. On the
other hand, reference writers have no knowledge of the evaluator’s initial impressions. The
lack of control by the evaluators increases the chances that the information provided is
impression incongruent or different from the evaluators’ initial impressions based on the
résumé. This feature forces evaluators to encode impression-incongruent information,
which in turn triggers the more comprehensive feature-based processing mode. A second
element may be the mode of communication. In comparison to oral communication, the
written mode of communication may be more open to careful evaluations required by the
dual-processes approach. There is more time to examine the document and the evaluator
can re-read information if inconsistencies appear.

Practical implications
Although reference letters could be considered biased, unreliable and poor predictors of
job performance (Baxter, Brock, Hill, & Rozelle, 1981), they are still commonly used and
have significant impact on personnel selection (Ryan et al., 1999). An important practical
issue of any selection tool is the need to maximize validity and reliability. The present study
suggests that reference letters may be less biased than expected from the evaluator’s side.
That is, given that the typical procedure is to review applicants’ résumés before reviewing
their reference letters, the impression effect might have contaminated reference ratings,
constituting errors and minimizing validity. However, the present research shows that
evaluators pay more attention to impression-incongruent information than to impression-
congruent information. This indicates that information in reference letter reduces, but not
reinforces, such contamination on reference letter evaluation.
To some extent, this research suggests that reference letters should be used in tandem
rather than as an independent selection tool. In addition to initial impressions, performance
evaluations in early stages of selection processes are likely to have different kinds of biases
(e.g. Wong & Kwong, 2005a,b). When used in tandem, a reference provider may provide
information that would be inconsistent with early evaluations. This inconsistency forces
evaluators to manage incongruent impressions and to modify the biased mental
categorizations of job applicants. This may be why reference letters are valued. In many
cases, reference letters are used as a separate hurdle and evaluators commonly seek
negative information that could be used in a reference letter that contradicts initially
favourable impressions (Tommasi, Williams, & Nordstrom, 1998).
However, this current research suggests that reference letters could also provide a better
evaluation of the second tier of applicants in that some from this tier may be seen more
favourably if reference letters are used and evaluators focus on the positively incongruent
information from second-tier applicants. Thus, when used in tandem with résumés, both
tools may facilitate a process that requires evaluators to assess more carefully applicants
with incongruent characteristics from the first impression and so to reduce impression
effects. As a separate and independent hurdle when only strong applicants are considered,
evaluators are likely to emphasize perceived negative information only, which may limit
the utility of reference letters.

Limitations and future research

There are two limitations to this study that may warrant additional research. One limitation
is the short time lag between résumé and reference reviewing. In formal selection

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp
 Reference letters evaluation 39

processes, the lag between initial résumé impressions and reading of reference letters is
usually longer than the current research time frame. Although previous research on
interviews reveals that impression effect persists over time (Cable & Gilovich, 1998),
future research may attempt to address whether the dual-process underlying reference
evaluations occurs when there is a longer time lag.
Another limitation of the present study is the use of undergraduate business students as a
sample, which may limit the generalizability to HR professionals who screen applicants.
Although using business undergraduates in a highly controlled experiment like the present
study can increase the overall internal validity of the study and some of them may evaluate
reference letters in their early careers right after graduation, future research may seek
further evidence for external validity for those who have extensive experience in reading
reference letters.
In summary, reference letters should be considered a useful supplemental hurdle in the
selection process. The current study suggests that reference letters are useful in helping
evaluators to more carefully assess and modify impressions created from reviewing
résumés.

ACKNOWLEDGEMENTS

This research was partially supported by a grant from the Research Grants Council of the
Hong Kong Special Administrative Region, China (HKUST6641/08). The authors thank
Michael S. Humphrey and an anonymous reviewer for their useful comments and
suggestions on an earlier version of this paper.

REFERENCES

Aamodt, M., Bryan, D., & Whitcomb, A. (1993). Predicting performance with letters of recom-
mendation. Public Personnel Management, 22, 81–90.
Aronson, E. (1969). The theory of cognitive dissonance: A current perspective. Advances in
Experimental Social Psychology, 4, 1–34.
Baxter, J. C., Brock, B., Hill, P. C., & Rozelle, R. M. (1981). Letters of recommendation: A question
of value. Journal of Applied Psychology, 66, 296–301.
Binning, J. F., Goldstein, M. A., Garcia, M. F., & Scattaregia, J. H. (1988). Effects of preinterview
impressions on questioning strategies in same- and opposite-sex employment interviews. Journal
of Applied Psychology, 73, 30–37.
Cable, D. M., & Gilovich, T. (1998). Looked over or overlooked? Prescreening decisions and
postinterview evaluations. Journal of Applied Psychology, 83, 501–508.
Chaiken, S. (1980). Heuristic versus systematic information processing and the use of source versus
message cues in persuasion. Journal of Personality and Social Psychology, 39, 752–766.
Conlon, E. J., & Park, J. M. (1987). Information requests in the context of escalation. Journal of
Applied Psychology, 72, 344–350.
Dipboye, R. L. (1982). Self-fulfilling prophecies in the selection-recruitment interview. Academy of
Management Review, 7, 579–586.
Dipboye, R. L., Stramler, C. S., & Fontenelle, G. A. (1984). The effects of the application on recall of
information from the interview. Academy of Management Journal, 27, 561–575.
Dougherty, T. W., Turban, D. B., & Callender, J. C. (1994). Confirming first impressions in the
employment interview: A field study of interviewer behavior. Journal of Applied Psychology, 79,
659–665.

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp
40 K. F. E. Wong and J. D. Werbel

Dunegan, K. J. (1993). Framing, cognitive modes, and image theory: Toward an understanding of a
glass half full. Journal of Applied Psychology, 78, 491–503.
Feldman, J. M. (1981). Beyond attribution theory: Cognitive processes in performance appraisal.
Journal of Applied Psychology, 66, 127–148.
Festinger, L. (1957). A theory of cognitive dissonance. Evanston, IL: Row, Peterson.
Fiske, S. T. (1988). Compare and contrast: Brewer’s dual process model and Fiske et al.’s continuum
model. In T. K. Srull, & R. S. Wyer, Jr. (Eds.), Advances in social cognition (Vol. 1, pp. 65–76).
Hillsdale, NJ: Erlbaum.
Fiske, S. T., & Pavelchak, M. A. (1986). Category-based versus piecemeal-based affective responses:
Developments in schema-triggered affect. In R. M. Sorrentino, & E. T. Higgins (Eds.), Handbook of
motivation and cognition: Foundations of social behavior (pp. 167–203). New York: Guildford Press.
Haberlandt, K., & Graesser, A. C. (1985). Component processes in text comprehension and some of
their introductions. Journal of Experimental Psychology: General, 114, 357–374.
Haberlandt, K., & Graesser, A. C. (1989). Processing of new arguments at clause boundaries. Memory
& Cognition, 17, 186–193.
Heneman, H. G., III, & Judge, T. A. (2003). Staffing organizations (4th ed.). Boston: McGraw-Hill
Irwin.
Jonas, E., Schulz-Hardt, S., Frey, D., & Thelen, N. (2001). Confirmation bias in sequential
information search after preliminary decisions: An expansion of dissonance theoretical research
on selective exposure to information. Journal of Personality and Social Psychology, 80, 557–571.
Just, M. A., Carpenter, P. A., & Woolley, J. D. (1982). Paradigms and processes in reading
comprehension. Journal of Experimental Psychology: General, 111, 228–238.
Kulik, C. T., & Ambrose, M. L. (1993). Category-based and feature-based processes in performance
appraisal: Integrating visual and computerized sources of performance data. Journal of Applied
Psychology, 78, 821–830.
Kuvaas, B., & Selart, M. (2005). Effects of attribute framing on cognitive processing and evaluation.
Organizational Behavior and Human Decision Processes, 95, 198–207.
Levy-Leboyer, C. (1994). Selection and assessment in Europe. In H. C. Triandis, M. D. Dunnette, &
L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 4, 2nd ed., pp.
173–190). Palo Alto, CA: Consulting Psychologists Press.
Maheswaran, D., & Chaiken, S. (1991). Promoting systematic processing in low-motivated settings:
Effect of incongruent information on processing and judgment. Journal of Personality and Social
Psychology, 61, 13–25.
Muchinsky, P. M. (1979). The use of reference report in personnel selection: A review and evaluation.
Journal of Occupational Psychology, 52, 287–297.
Ryan, A. M., McFarland, L., Baron, H., & Page, R. (1999). An international look at selection practices:
Nation and culture as explanations for variability in practice. Personnel Psychology, 52, 359–392.
Schulz-Hardt, S., Frey, D., Lüthgens, C., & Moscovici, S. (2000). Biased information search in group
decision making. Journal of Personality and Social Psychology, 78, 655–669.
Stangor, C., & McMillan, D. (1992). Memory for expectancy-congruent and expectancy-incongruent
information: A review of the social and social development literatures. Psychological Bulletin,
111, 42–61.
Staw, B. M. (1976). Keeping-deep in big muddy – Study of escalating commitment to a chosen course
of action. Organizational Behavior and Human Performance, 16, 27–44.
Tommasi, G. W., Williams, K. B., & Nordstrom, C. R. (1998). Letters of recommendation: What
information captures HR professional attention? Journal of Business & Psychology, 13, 5–18.
Wofford, J. C., & Goodwin, V. L. (1990). Effects of feedback on cognitive processing and choice of
decision style. Journal of Applied Psychology, 75, 603–612.
Wong, K. F. E., & Kwong, J. Y. Y. (2005a). Between-individual comparisons in performance
evaluation: A perspective from prospect theory. Journal of Applied Psychology, 90, 284–294.
Wong, K. F. E., & Kwong, J. Y. Y. (2005b). Comparing two tiny giants or two huge dwarfs?
Preference reversals owing to number size framing. Organizational Behavior and Human Decision
Processes, 98, 54–65.
Wong, K. F. E., Kwong, J. Y. Y., & Ng, C. K. (2008). When thinking rationally increases biases: The
role of rational thinking style in escalation of commitment. Applied Psychology: An International
Review, 57, 246–271.

Copyright # 2008 John Wiley & Sons, Ltd. Appl. Cognit. Psychol. 24: 27–40 (2010)
DOI: 10.1002/acp