Learning and Instruction 21 (2011) 220e231

www.elsevier.com/locate/learninstruc

Measuring spontaneous and instructed evaluation processes during

Web search: Integrating concurrent thinking-aloud protocols and
eye-tracking data
Peter Gerjets*, Yvonne Kammerer, Benita Werner
Knowledge Media Research Center, Konrad-Adenauer-Str. 40, 72072 Tuebingen, Germany

Abstract

Web searching for complex information requires to appropriately evaluating diverse sources of information. Information science studies
identified different criteria applied by searchers to evaluate Web information. However, the explicit evaluation instructions used in these studies
might have resulted in a distortion of spontaneous evaluation processes. Accordingly, the present study compared explicit evaluation instructions
and neutral thinking-aloud instructions. Data from thinking-aloud protocols, eye tracking, and information problem-solving were collected from
30 participants equally distributed to two experimental conditions, that is, the Instructed Evaluation condition and the Spontaneous Evaluation
condition. Instructed evaluation, as compared to spontaneous evaluation, resulted in more verbal utterances of quality-related evaluation criteria,
in an increased attention focus on user ratings displayed on Web pages, and in better quality of decision making, although participants in the
Instructed Evaluation condition were not able to better justify their decision as compared to participants in the Spontaneous Evaluation
condition.
Ó 2010 Elsevier Ltd. All rights reserved.

Keywords: Web search; Evaluation criteria; Thinking aloud; Eye tracking

1. Introduction information. There is hardly any quality assurance on the Web

For the public, the World Wide Web (WWW) has
increasingly evolved into one of the most important informa-
tion sources, especially when it comes to complex contents of
personal concern like medical, environmental, or technical
issues (Stadtler & Bromme, 2007). More and more, searching
for complex and science-related information on the Web
supplements the interaction with experts, for instance, when
a diagnosis or a treatment for a medical or technical problem
is needed. However, on the Web not only scientific and other
institutions, but also journalists, companies, and lay-people
provide information on complex domains, yielding a substan-
tial variability in terms of the quality and reliability of Web
* Corresponding author. Tel.: þ49 7071 979 219; fax: þ49 7071 979 126.
E-mail address: p.gerjets@iwm-kmrc.de (P. Gerjets).
as it used to be when expert knowledge was obtained from
academic professionals or publishers (e.g., certificates of
professional qualifications).
As a result, Web users are required to appropriately eval-
uate diverse, potentially diffuse, or even contradictory sources
of information. Hence, Web search has turned into an activity
that goes far beyond simple fact finding or browsing, although
these tasks are still standard paradigms used in scientific
studies on Web search (Brumby & Howes, 2008; Cutrell &
Guan, 2007; Joachims, Granka, Pan, Hembrooke, & Gay,
2005; Miller & Remington, 2004). Instead, Web search is
often related to personal decisions under uncertainty in
domains characterized by fragile and conflicting evidence
where goals of high personal relevance are set up and pursued
in a self-directed way. Accordingly, adequate strategies for the
critical evaluation of information quality while searching for
complex contents on the Web become a crucial factor for the

0959-4752/$ - see front matter Ó 2010 Elsevier Ltd. All rights reserved.
doi:10.1016/j.learninstruc.2010.02.005
 P. Gerjets et al. / Learning and Instruction 21 (2011) 220e231
searcher’s success in terms of gaining important and valid
information as well as in terms of solving his or her infor-
mation problems. However, it has been shown that searchers
usually face difficulties in appropriately evaluating informa-
tion during Web search (Brand-Gruwel, Wopereis, &
Vermetten, 2005; Gerjets & Hellenthal-Schorr, 2008).
From these considerations, it should become clear that
there are normative as well as descriptive questions to be
answered with regard to the evaluation of Web information.
Normative issues concern the questions of what evaluation
processes searchers should engage in and how they should be
supported to do so. Descriptive issues address the questions of
what evaluation processes searchers actually do display
spontaneously during Web search and how different
constraints of their search situation actually do influence their
engagement in evaluation processes. In this paper we focus on
a descriptive perspective by investigating what type of spon-
taneous evaluation processes occur when users solve
a complex search task related to a medical problem by using
a standard search-engine design. In particular, we are inter-
ested in the role of different evaluation criteria in different
stages of the Web search process.
1.1. Evaluation processes during information search
on the WWW
Several models have described the process of Web search
as an information problem-solving process that can be
segmented into several sub-processes (Brand-Gruwel et al.,
2005; Gerjets & Hellenthal-Schorr, 2008; Marchionini, 1995;
Wilson, 1999). A compilation from different existing models
yields five important stages of the Web search process:
(a) An information problem is recognized and a search goal is
defined.
(b) Mostly, as a means to access Web information a search
engine (e.g., Google) is selected, where a query with
search terms is entered and sent off. Subsequently, a so-
called ‘‘search engine result page’’ (SERP) with a list of
search result links is returned to the user.
(c) Available search results are scanned and evaluated with
regard to their significance for the search goal and links have
to be selected for further inspection. This evaluation of
search results is based on sparse information about the
corresponding Web pages and the information they contain
(e.g., titles, summaries, and uniform resource locators,
URLs). Clicking on a link can be regarded as the central
(observable) action in the information search process as it
reflects which information will be subsequently accessed.
(d) After accessing a selected Web page, its information is
scanned, evaluated with regard to its relevance for the
search goal, and in case of relevance information is
extracted for further processing.
(e) The information from different pages is compared and
integrated towards a solution of the information problem.
Here again evaluations are required to resolve conflicts and
incoherences within the document collection retrieved.
221
Accordingly, three different types of evaluation processes
are pivotal during Web search, namely the evaluation (a) of
search results, (b) of Web pages, and (c) of document
collections. Several studies have focused on the third evalua-
tion phase, that is, on the evaluation of an incoherent docu-
ment collection in order to construct an integrated mental
representation of the search domain (Rouet, 2006; Rouet,
Favart, Britt, & Perfetti, 1997; Stadtler & Bromme, 2007).
In the present article, however, the focus will be on the
cognitive processes occurring during the first two evaluation
phases, that is, the search-result evaluation phase and the Web-
page evaluation phase. Specifically, for the search-result
evaluation phase it will be analyzed how search results are
scanned and evaluated with regard to their significance for the
current search goal. For the Web-page evaluation phase it will
be investigated how users subsequently evaluate the Web
pages that they actually selected from a SERP. Nevertheless,
the third evaluation phase will not be neglected, as also
searchers’ solutions to their information problem that results
from integrating different sources will be analyzed.
1.2. Judging the ‘‘relevance’’ of Web information
Theoretical accounts for the evaluation of search results
and Web pages usually refer to the concept of relevance.
However, the definition of this concept varies between
different scientific disciplines concerned with Web search,
with a rather narrow definition of relevance in information
retrieval (IR) research and a broader definition in information
science.
In research on IR, the term relevance denotes how well
a retrieved set of documents meets the information needs of
a user (Maglaughlin & Sonnenwald, 2002). In this literature,
relevance thus refers to topical relevance (or topicality), that
is, the match between the topic of the information need or
query and the topic of a search result (Pirolli & Card, 1999).
Several Web search studies have shown that users rely heavily
on the topicality ranking offered by search engines and
predominantly select the first few links on the top of a SERP
(Cutrell & Guan, 2007; Granka, Joachims, & Gay, 2004; Guan
& Cutrell, 2007; Joachims et al., 2005; Pan et al., 2007). In
accordance with the IR tradition to identify relevance with
topicality, cognitive Web-search models derived from research
at the intersection of IR, humanecomputer interaction (HCI),
and cognitive science are usually restricted to the evaluation of
topicality (Brumby & Howes, 2008; Miller & Remington,
2004; Pirolli & Card, 1999; Pirolli & Fu, 2003).
Based on the empirical success of these models to explain
and predict the selection of search results, it seems at first sight
that the concept of topical relevance is sufficient to under-
standing users’ evaluation of Web information. It has to be
noted, however, that this conclusion might be a bit premature,
because these models have been mainly applied to rather
simple fact-finding tasks or to search tasks for which a selec-
tion of Web sources containing uncontroversial and consistent
information of high quality was provided. Such tasks demand
that users focus their attention mainly on the topical fit of
222 P. Gerjets et al. / Learning and Instruction 21 (2011) 220e231
available information. However, in contrast to traditional
search tasks like fact finding, searching for complex infor-
mation on the Web is characterized by a much larger vari-
ability with regard to search goals pursued and information
quality encountered. It can thus be assumed that Web
searchers will also evaluate other aspects of Web information
beyond topicality, at least when certain preconditions are given
(e.g., the search task is sufficiently complex, the available
information is highly variable with regard to its quality, the
user has the cognitive prerequisites to engage in these
processes at his or her disposal).
Furthermore, it has to be considered that different stages of
the search process might differ with regard to the evaluation
processes users engage in. It can be assumed that evaluating
links on a SERP differs from evaluating the corresponding
Web pages in several respects. For instance, SERPs are very
often confined to topical information in terms of the title and
the description of each search result. Contrary to that, Web
pages usually also include quality-related cues like user
ratings, author information, or publication information.
Accordingly, users might be more focused on the evaluation of
topicality when selecting links from a SERP, whereas they
might be more open to other quality-related evaluation criteria
when scanning the corresponding Web pages.
One way of addressing the assumption that evaluating Web
information should not be confined to judgments of topicality,
is to broaden the concept of relevance beyond topicality. For
instance, in information science relevance is introduced as
a concept that cannot be reduced to topical fit, but is based on
a set of different evaluation criteria (Borlund, 2003; Schamber,
1994) also reflecting the quality of information. In the last
decade several studies which addressed the evaluation of Web
search results and Web pages (Crystal & Greenberg, 2006;
Rieh, 2002; Savolainen & Kari, 2005; Tombros, Ruthven, &
Jose, 2005) yielded that although topical fit (e.g., topicality,
topical interest, scope) is the evaluation criteria most often
applied, searchers nevertheless used other evaluation criteria
as well which refer to the quality of information (e.g., credi-
bility, up-to-dateness, design). Moreover, in these studies,
publishing information (e.g., publication date, name and
profession of the author) and source information (e.g., refer-
ences, contact information, affiliation) turned out to be
important Web page characteristics that were used to evaluate
the quality of Web information. According to social navigation
research it can be additionally assumed that social information
like user ratings (e.g., rating stars, comments) will also be used
by searchers to derive quality judgments when navigating and
exploring information spaces (Höök, Benyon, & Munro,
2003). However, as we will argue in the next section, the
results on evaluation criteria obtained in information science
are by no means conclusive yet due to two methodological
shortcomings of current research paradigms. Beyond this
methodological critique, we also have the impression that the
idea of including quality aspects into the concept of relevance
might be rather confusing for readers not familiar with this
terminology from information science. Thus, in the remainder
of this paper we will avoid using the concept of relevance and
will instead directly refer to topicality and quality as two
important evaluation criteria.
1.3. Methodological considerations: measuring
evaluation processes during Web search
Concerning the measurement of evaluation processes
occurring during Web search, the scope of most studies on
evaluation criteria conducted in information science is limited
by two specific methodological shortcomings.
First, many studies investigating the use of different eval-
uation criteria in the evaluation of Web information seem to
fail in addressing spontaneous evaluation processes as partic-
ipants are explicitly instructed to evaluate search results and
Web pages. All of these studies have in common e indepen-
dent of the methods of data collection applied, e.g., screen
capturing, thinking-aloud protocols, highlighting of important
aspects of Web pages, or retrospective reports e that partici-
pants are instructed beforehand to mention or mark important
factors or criteria which might influence their evaluation of
information (Crystal & Greenberg, 2006; Rieh, 2002;
Savolainen & Kari, 2005; Tombros et al., 2005). This kind
of task does not seem to be suitable to investigate spontaneous
evaluation processes, as it is likely to yield artifacts. Partici-
pants may become much more aware of their evaluation and
selection processes due to the instruction used, resulting in
a distortion of their spontaneous evaluation processes. In the
present study the aim was to provide evidence that e as
criticized e a thinking-aloud procedure including explicit
instructions to apply evaluation criteria during Web search (as
it is done, for instance, by Tombros et al., 2005) significantly
increases the number of evaluation criteria mentioned
compared to standard neutral thinking-aloud instructions (in
line with Ericsson & Simon, 1993), which should reveal more
spontaneous and undirected evaluation processes.
A second concern with regard to the methods currently
used to investigate evaluation criteria in information science is
the strong focus on consciously accessible, verbalized criteria
and decisions that lead to overt interactions with the search
environment (e.g., mouse movements and clicks). Hence,
evaluation processes that go beyond overt actions remain
largely undiscovered, and so do quick and unconscious eval-
uation processes. Moreover, the evaluation processes leading
to the decision to not select a search result are as important as
the processes leading to the selection of a search result. To
unravel these processes that might not show up in overt
behaviour, eye-tracking methodologies seem particularly
promising because they provide more detailed insights in
participants’ cognitive processing (Rayner, 1998). For
instance, eye tracking allows to reconstruct every search result
and every Web page characteristic that was looked at irre-
spective from its selection or rejection or from being verbally
addressed or not. Moreover, Van Gog, Paas, and Van
Merriënboer (2005) argue that especially a combined use of
thinking-aloud protocols and eye-tracking data can provide
deeper insights into implicit and fine-grained aspects of
cognitive processes. Therefore, in the study outlined in this
 P. Gerjets et al. / Learning and Instruction 21 (2011) 220e231
paper concurrent thinking-aloud data was supplemented by
participants’ eye-tracking data during Web search. This
method also allows us to test whether explicit evaluation
instructions alter the way users pay attention to different types
of information.
1.4. The present study
The present study tested the general hypothesis that explicit
evaluation instructions as used in information science studies
(Rieh, 2002; Tombros et al., 2005) might result in a severe
distortion of users’ spontaneous evaluation processes. As
control condition neutral thinking-aloud instructions were
chosen in order to detect this distortion in participants’ verbal
utterances. Additionally, eye-tracking data were used in
combination with thinking-aloud protocols to reveal fine
grained or implicit cognitive processes that do not show up in
users’ verbal utterances.
It was expected that explicit evaluation instructions
(compared to neutral thinking-aloud instructions) will influence
users engaged in a complex search task with regard to different
aspects of their Web search behaviour, namely their verbal
utterances and eye movements with respect to SERPs and
selected Web pages, their search result selection on SERPs, and
the result of their information problem solving. The basic
assumption that explicit evaluation instructions will elicit
evaluation processes that would not take place spontaneously
thus results in the following more specific expectations.
1.4.1. Verbal utterances
Explicit evaluation instructions should result in an
increased number of verbal utterances addressing different
evaluation criteria during both the evaluation of SERPs and of
selected Web pages. In particular, for criteria that go beyond
topicality and concern the quality of Web information (e.g.,
credibility, up-to-dateness, design) due to the explicit evalua-
tion instructions a strong increase was expected, as compared
to spontaneously produced verbal utterances because, in the
latter case, searchers are probably less used in evaluating the
quality of information than its topicality (Hypothesis 1).
1.4.2. Eye-tracking data
The assumption that explicit evaluation instructions will
elicit evaluation processes that would not take place spon-
taneously renders the expectation that users’ eye-tracking
data will differ on parameters that might indicate evalua-
tion processes. Potential indicators for processes of search
result evaluation are a higher number of attended search
results or a longer total fixation time on search results.
Additionally, it was expected that explicit evaluation
instructions, as compared to neutral thinking-aloud instruc-
tions, would result in a longer total fixation time or a higher
number of gazes on specific pieces of information (e.g.,
publishing information, source information, user ratings) on
a SERP or a Web page, indicating that users try to evaluate
the quality of search results and of retrieved Web pages
(Hypothesis 2).
223
1.4.3. Search result selection
Spontaneously, users tend to predominantly select the first
few search results on a SERP and neglect the remaining ones
(Guan & Cutrell, 2007; Joachims et al., 2005). Explicit evalu-
ation instructions, as compared to neutral thinking-aloud
instructions, should increase the number of search results with
lower rankings chosen because users will be stimulated to
evaluate all search results on their own with regard to infor-
mation quality instead of selecting links only according to the
topicality ranking provided by the search engine (Hypothesis 3).
1.4.4. Quality of information problem solving
If explicit evaluation instructions stimulated users to eval-
uate search results or Web pages in the course of their infor-
mation problem solving, then the quality of the resulting
problem solution should improve (e.g., the quality of an
informed decision users are asked for or the quality of
a statement to justify an informed decision) as compared to
spontaneous evaluation processes (Hypothesis 4).
2. Method
2.1. Design e participants
The independent variable of the study was the type of
instructions given to the participants. Participants received
either neutral thinking-aloud instructions (in line with Ericsson
& Simon, 1993) without any instructions to evaluate (Sponta-
neous Evaluation condition) or thinking-aloud instructions
including instructions to evaluate (Instructed Evaluation
condition; cf. Tombros et al., 2005). In the latter case partici-
pants were asked to mention the evaluation criteria that they
applied while selecting search results and assessing Web pages.
Participants in the study were 30 students (10 male, 20
female; M ¼ 25.40 years, SD ¼ 3.95) from social and natural
sciences and humanities (no computer science students) at the
University of Tuebingen, Germany; participation was rewar-
ded with either course credit or payment. Participants had
normal or corrected to normal vision.
Participants were randomly assigned to the two experi-
mental conditions with 15 participants serving in each of the
conditions.
2.2. Material and apparatus
2.2.1. Task
A complex and controversial domain characterized by
fragile and conflicting evidence was chosen that provided
sufficient affordances for searchers to engage in evaluation
processes beyond topicality. The task was to achieve an
informed decision between a low fat and a low carb (i.e.,
carbohydrates) diet with regard to which of the two diet
methods better promotes a healthy and long-lasting weight
loss. In line with the method used by Stadtler and Bromme
(2007) participants were confronted with a request from
a fictitious overweight friend, who wants to lose weight by
changing her diet and asks for advice. Participants were asked
224 P. Gerjets et al. / Learning and Instruction 21 (2011) 220e231
to conduct a 20-min Web search regarding this controversial
topic in order to decide which of the two diet methods they
would recommend to their friend.
2.2.2. Web stimulus materials
For their Web search, participants were provided with three
preselected Google-like SERPs. All three SERPs were
accessible by means of a start Web page presenting three
hyperlinks with the search terms used to generate these
SERPs. The three search terms were ‘‘low fat’’, ‘‘low carb’’,
and ‘‘low carb þ low fat’’ whereby each of the search terms
was used to generate one SERP containing 10 search results
(for an example screenshot of the SERP ‘‘low carb þ low fat’’
see Fig. 1). The set of search results and Web pages for each of
the three SERPs reflected the given heterogeneity of infor-
mation sources and their different perspectives and interests
with regard to this controversial search topic. All three SERPs
included Web pages provided by scientific institutions (e.g.,
universities), journalists (e.g., online magazines), industry and
companies (e.g., online shops for nutrition or pharmaceutics),
and lay-people (e.g., discussion pages). There was an
approximately equal distribution of the type of Web pages for
each SERP. Accordingly, for each SERP the available infor-
mation varied largely with regard to contents and quality. The
three SERPs and the 30 Web pages (with 845 words per Web
page on average and a range from 241 words to 3191 words
per Web page) corresponding to the search results were put
offline to guarantee a standardized and controlled experi-
mental setting. Hyperlinks within the 30 Web pages could be
used whereby in that case the corresponding Web pages were
accessed online. The layout of the Google-like SERPs was set
up close to original, but ads and the hyperlinks ‘‘in cache’’ and
‘‘similar pages’’ were removed. The Web materials were dis-
played on a standard 17-inch computer screen and were pre-
sented with Microsoft Internet Explorer 6.
2.2.3. Eye tracking equipment
For eye tracking during task processing a 50 Hz Tobii 1750
remote eye-tracking system with infrared-cameras built into
a 17-inch monitor (www.tobii.com) was used. The Web stimulus
recording mode of the ClearView 2.7.1 analysis software was
used that captures not only the eye movements, but the entire
task performance process (including mouse operations). The
minimum fixation duration was set to 100 ms with a fixation
radius of 30 pixels (cf. Cutrell & Guan, 2007). Participants’
thinking-aloud protocols were recorded digitally by Camtasia
3.0 software using a standard microphone attached to the PC.
2.2.4. Thinking-aloud instructions
In the Spontaneous Evaluation condition instructions to
think aloud were worded in line with the standards described
by Ericsson and Simon (1993). The instructions were:
Please think aloud during your Web search, that is,
verbalize everything that comes to your mind.
Please keep constantly talking from beginning till the end
of the task.
Act as if you were alone, with no one listening, and just
keep talking.
In contrast, in the Instructed Evaluation condition the
instructions were similar to the instructions used, for instance,
by Tombros et al. (2005) or Rieh (2002):
Please think aloud during your Web search, that is, mention
the evaluation criteria you apply to select search results and
to assess Web pages.
Please keep constantly talking from beginning till the end
of the task.
Act as if you were alone, with no one listening, and just
keep talking.
During task performance, whenever participants stopped
verbalizing their thoughts, the experimenter reminded them
(after 5 s) to think aloud.
2.2.5. Demographics, computer use and Web search skills,
prior knowledge
Before task processing, participants were asked to fill in
a short computer-based questionnaire about demographics
(gender, age) as well as about computer use and their skills in
Web search (5 items; Cronbach’s a ¼ .82). Moreover, the
questionnaire included nine items about participants’ prior
knowledge on diets and nutrition; example item is ‘‘I never
heard about the low carb versus low fat controversy’’. The
items had to be rated on five-point Likert-type response scales
ranging from 1 (highly disagree) to 5 (highly agree). Cron-
bach’s alpha was .86 for the nine items. Analyses of the
respective data revealed no differences between the two
experimental conditions, that is, for gender, c2(1,
N ¼ 29) ¼ 0.84, ns, for age, t(27) ¼ 0.07, ns, for computer use
and Web search skills, t(27) ¼ 0.96, ns, and for prior knowl-
edge on diets and nutrition, t(27) ¼ 0.03, ns.
2.2.6. Decision making and decision justification
At the end of task processing, participants were required to
decide which of the two diet methods they would recommend
to their friend. Additionally, participants had to write a short
statement to justify their decision.
2.3. Procedure
Participants were tested in individual sessions of approxi-
mately 1 h. First, they filled in the questionnaire on demo-
graphics, computer use and Web search skills, and their prior
knowledge on diets and nutrition. They also received
instructions about Web search as well as the thinking-aloud
instructions according to their experimental condition.
Participants were then calibrated on the eye-tracking system
using a 9-point calibration.
Before working on the main task, participants underwent
a training task for approximately 5 min to get acquainted with
the thinking-aloud method and the Web search environment.
In this training task, they had to conduct a Web search on
 P. Gerjets et al. / Learning and Instruction 21 (2011) 220e231 225

Fig. 1. Screenshot of the SERP with low carb þ low fat as keywords.

possible causes and treatments of backache. They were pre-
sented three search terms (‘‘backache’’, ‘‘back gym’’, and
‘‘backache þ back gym’’) leading to three Google-like SERPs
(with 10 search results each) linked to websites. During the
training task, participants’ thinking aloud was practiced
together with the experimenter. In the case that participants
did not verbalize their thoughts according to the instructions
received the experimenter repeated the instructions and
encouraged them to think aloud freely. When the experimenter
had the impression that the participants were able to think
aloud freely and that they felt comfortable enough with the
procedure the training task was finished.
When participants had finished the training task, they were
given the instructions for the main task including the fictitious
request of their friend to give a recommendation about low
carb or low fat diets. Furthermore, participants were again
reminded to think aloud during their task performance and to
use all three search terms. Eye movements, screen recordings,
and concurrent verbalizations were captured during the entire
20 min task performance. If necessary the experimenter
226 P. Gerjets et al. / Learning and Instruction 21 (2011) 220e231
Table 1
Evaluation criteria.
Evaluation criteria Short description
Topicality Is the search
result or the Web page
about the topic
in general?
Scope Is the information
focused enough on the specific
request of the fictitious friend?
Credibility Is the information
valid and is the source
providing information trustworthy
and reliable?
Design Is the design
of the Web page
good and is the information
clearly displayed and easy
to find?
Up-to-dateness Is the information
recent or up-to-date?
reminded the participants to keep thinking aloud and informed
them after 10 min that half of the available time was over.
Participants were asked to use all three SERPs for their Web
search, but were not allowed to generate new SERPs by
changing the search terms. Participants could access all Web
pages corresponding to the list of search results.
Subsequent to the search task participants were required to
decide which of the two diet methods they would recommend to
their friend and to write a short statement about their decision.
2.4. Data analyses and dependent variables
2.4.1. Coding scheme for thinking-aloud protocols
For the analysis of participants’ thinking-aloud protocols
a coding scheme was developed that was based on the evaluation
criteria found in the information science literature (Savolainen
& Kari, 2005). This scheme was refined by analyzing data
from thinking-aloud protocols of a pilot study. It included the
following five evaluation criteria: (a) topicality, (b) scope, (c)
credibility, (d) up-to-dateness, and (e) design. The first two
criteria were topic-related criteria, whereas the latter three were
quality-related criteria. Short descriptions of these five evalua-
tion criteria are provided in Table 1. Two raters familiar with the
search task and the materials as well as with the coding scheme
scored 30% of the protocols. Interrater reliability computed on
this subsample of protocols yielded a Cohen’s kappa of 0.72.
One rater scored the remaining protocols. As dependent vari-
ables the number of verbal utterances referring to the five
different types of evaluation criteria was analyzed for both the
search result evaluation and the Web page evaluation.
2.4.2. Eye-tracking data
For the analysis of participants’ eye-tracking data so-
called ‘‘areas of interest’’ (AOIs)1 were defined manually
1
‘‘Areas of interest’’ (AOIs) are precisely specified areas of an object (e.g.,
a search result on a SERP or an author name on a Web page).
on the SERPs and the Web pages. It was determined for all
AOIs whether and for which amount of time a participant was
looking at this area. On the three SERPs each of the ten
search results (including title, summary, and URL) was
defined as a single rectangular AOI. For all 30 Web pages
AOIs were defined on specific Web page characteristics,
namely areas that provided publishing information (publi-
cation date, name and profession of the author), source
information (references, contact information, affiliation), and
user ratings (e.g., rating stars, comments). All AOIs corre-
sponding to the same Web page characteristic (i.e.,
publishing information, source information, and user ratings,
respectively) were aggregated for data analyses across the 30
Web pages. Similar AOIs were defined on the three SERPs.
However, as the SERPs contained no user ratings and only
a very limited amount of publishing information we aggre-
gated these AOIs into a single variable indicating publishing
and source information.
As a first eye-tracking variable during the search-result
evaluation phase the number of visually inspected Google
search results was counted. A visual inspection was defined
as at least one fixation (100 ms) within a search-result AOI.
The second eye-tracking variable was the total fixation time
for which participants visually inspected Google search
results during the search-result evaluation phase. The total
fixation time on search results was defined as the sum of
fixation durations for all search results. As a third eye-
tracking variable the number of gazes on publishing and
source information within the SERPs was registered. A single
gaze was defined as all subsequent fixations on an AOI. In
case that an AOI was re-inspected after fixations on another
AOI, this was counted as a second gaze. As a fourth eye-
tracking variable during the search-result evaluation phase
the total fixation time on publishing and source information
within the SERPs was measured by summing up all respective
fixation durations. Furthermore, three eye-tracking variables
during the Web-page evaluation phase were formed based on
the number of gazes on the aggregated AOIs of each of the
three Web page characteristics, namely publishing informa-
tion, source information, and user ratings. Finally, three more
eye-tracking variables were formed based on the total fixation
time for each of the three Web page characteristics during the
Web-page evaluation phase. The total fixation time for
a single Web page characteristic (e.g., publishing informa-
tion) was defined as the sum of all fixation durations on this
Web page characteristic across all Web pages. All fixation-
time data were transformed into seconds for ease of
interpretation.
2.4.3. Log file data (mouse clicks)
As a first selection variable the number of search results
selected per SERP was registered. Additionally as a second
and third selection variable the number of selections of the top
five search results per SERP (i.e., positions 1e5) and the
number of selections of the bottom five search results per
SERP (i.e., positions 6e10) were registered (cf. Pan et al.,
2007).
 P. Gerjets et al. / Learning and Instruction 21 (2011) 220e231 227

2.4.4. Quality of decision making and decision justification Table 2

Participants’ solution to the information problem, that is, Means (and standard deviations) for the number of verbal utterances as
a function of experimental condition in the two evaluation phases for the five
the decision to either recommend low carb diets or low fat evaluation criteria.
diets, was analyzed by counting the frequency with which the
Evaluation criteria Condition
two diet methods were recommended in each experimental
condition. Additionally, participants’ statements to justify their Instructed evaluation Spontaneous evaluation Overall
decision were rated with respect to their quality on a 3-point Search-result evaluation phase
rating scale, ranging from 0 (false or no statements), 1 Topicality 4.60 (4.09) 3.50 (2.77) 4.07 (3.49)
Scope 0.67 (0.90) 0.14 (0.36) 0.41 (0.73)
(personal opinions or likes and dislikes without any further Credibilitya 4.60 (4.55) 1.79 (1.72) 3.24 (3.71)
argumentation), 2 (fuzzy statements mentioning risks and Design 0.07 (0.26) 0.07 (0.27) 0.07 (0.26)
benefits of one diet method), to 3 points (detailed statement Up-to-dateness 0.00 (0.00) 0.00 (0.00) 0.00 (0.00)
with arguments in favour and against both diet methods). Web-page evaluation phase
Topicality 1.93 (2.76) 0.93 (1.00) 1.45 (2.13)
3. Results Scope 3.33 (2.97) 2.07 (2.17) 2.72 (2.64)
Credibilitya 9.13 (5.78) 3.57 (2.31) 6.45 (5.21)
Designa 5.53 (4.69) 0.93 (1.07) 3.31 (4.12)
An alpha level of .05 was used for the statistical tests
Up-to-datenessa 0.47 (0.83) 0.00 (0.00) 0.24 (0.64)
reported. Due to technical problems one participant from the
Spontaneous Evaluation condition was excluded from data Overall
Topicality 3.27 (2.72) 2.21 (1.41) 2.76 (2.21)
analyses; consequently, statistical analyses were conducted for Scope 2.00 (1.49) 1.11 (1.08) 1.57 (1.36)
29 participants. For all analyses partial eta-squared is reported Credibilitya 6.87 (4.72) 2.68 (1.20) 4.85 (4.04)
as a measure of effect size. Designa 2.80 (2.40) 0.50 (0.52) 1.69 (2.09)
Up-to-datenessa 0.23 (0.42) 0.00 (0.00) 0.12 (0.32)
3.1. Verbal utterances a
Significant differences between the two experimental conditions.

Data from thinking-aloud protocols were analyzed with
a 2(conditions: instructed evaluation vs. spontaneous eval-
uation)  2(evaluation phases: search result evaluation vs. Web
page evaluation) mixed model repeated measures MANOVA
with the number of verbal utterances addressing the five eval-
uation criteria as five dependent variables; conditions served as
the between subjects factor and the evaluation phases as within
subjects factor (for means and standard deviations see Table 2).
The MANOVA showed a significant overall main effect of
conditions on the number of evaluative verbal utterances, Pil-
lai’s trace ¼ 0.43, F(5, 23) ¼ 3.43, p < .05, partial h2 ¼ .43.
Univariate ANOVAs showed that participants in the Instructed
Evaluation condition showed significantly more verbal utter-
ances than participants in the Spontaneous Evaluation condition
only for the three quality-related evaluation criteria: credibility,
F(1, 27) ¼ 10.36, p < .01, partial h2 ¼ .28; design, F(1,
27) ¼ 12.33, p < .01, partial h2 ¼ .31; and up-to-dateness, F(1,
27) ¼ 4.38, p ¼ .05, partial h2 ¼ .14. For topicality and scope the
difference was not significant, F(1, 27) ¼ 1.68, ns, and F(1,
27) ¼ 3.38, p ¼ .08, respectively.
There was also a significant overall main effect of the eval-
uation phase, Pillai’s trace ¼ 0.11, F(5, 23) ¼ 15.37, p < .001,
partial h2 ¼ .11. Univariate ANOVAs showed that for the
following four evaluation criteria participants showed signifi-
cantly more evaluative verbal utterances during the Web-page
evaluation phase than during the search-result evaluation
phase: F(1, 27) ¼ 19.64, p < .001, partial h2 ¼ .42 for scope; F(1,
27) ¼ 19.23, p < .001, partial h2 ¼ .42 for credibility; F(1,
27) ¼ 24.95, p < .001, partial h2 ¼ .48 for design; and F(1,
27) ¼ 4.38, p ¼ .05, partial h2 ¼ .14 for up-to-dateness. In
contrast, for topicality, that is, the fifth criterion, a reversed
pattern was shown with significantly more verbal utterances
during the search-result evaluation phase than during the Web-
page evaluation phase, F(1, 27) ¼ 13.80, p < .01, partial
h2 ¼ .34.
Furthermore, there was a significant interaction between
conditions and evaluation phases, Pillai’s trace ¼ 0.49, F(5,
23) ¼ 4.43, p ¼ .01, partial h2 ¼ .49. Univariate ANOVAs
showed that this interaction was significant only for design and
up-to-dateness, F(1, 27) ¼ 13.25, p < .01, partial h2 ¼ .33,
and F(1, 27) ¼ 4.38, p ¼ .05, partial h2 ¼ .14, respectively.
For topicality, scope, and credibility the interaction was not
significant, F < 1, ns, F < 1, ns, F(1, 27) ¼ 3.64, p ¼ .07,
respectively. Bonferroni post hoc tests revealed that during the
Web-page evaluation phase participants in the Instructed
Evaluation condition showed significantly more verbal utter-
ances with respect to the criteria design and up-to-dateness
than participants in the Spontaneous Evaluation condition
( p < .01 and p ¼ .05, respectively), whereas during the
search-result evaluation phase the number of verbal utterances
with regard to these criteria did not differ between participants
(in both cases p > .90). This was not surprising as there are
hardly any hints present in the search results concerning the
design of the Web page and how up-to-date the information on
the Web page is. With regard to the criterion credibility, during
both the search result and Web-page evaluation phases,
participants in the Instructed Evaluation condition showed
more verbal utterances than participants in the Spontaneous
Evaluation condition ( p <. 05 and p <. 01, respectively).
3.2. Eye-tracking data
Eye-tracking data were analyzed separately for the search-
result evaluation phase and the Web-page evaluation phase as
AOIs were not comparable between the two phases (for means
and standard deviations see Table 3).
228 P. Gerjets et al. / Learning and Instruction 21 (2011) 220e231

Table 3
Means (and standard deviations) for the eye-tracking data as a function of experimental condition in the two evaluation phases.
Eye-tracking data Condition
Instructed Spontaneous Overall
evaluation evaluation
Search-result evaluation phase (sum of all 30 search results)
Number of visually inspected search results 27.07 (3.20) 28.64 (1.65) 27.83 (2.65)
Total fixation time on search results (in seconds) 103.63 (46.84) 100.68 (51.48) 102.20 (48.27)
Number of gazes on publishing and source information 47.07 (21.16) 47.36 (20.25) 47.21 (20.36)
Total fixation time on publishing and source 24.37 (15.91) 21.94 (16.33) 23.20 (15.87)
information (in seconds)
Web-page evaluation phase (sum of all 30 Web pages)
Number of gazes on publishing information 14.00 (7.22) 13.93 (6.82) 13.97 (6.90)
Number of gazes on source information 3.20 (2.43) 2.14 (1.75) 2.69 (2.16)
Number of gazes on user ratingsa 1.60 (1.99) 0.36 (0.63) 1.00 (1.60)
Total fixation time on publishing information (in seconds) 8.85 (7.11) 6.51 (3.13) 7.72 (5.59)
Total fixation time on source information (in seconds) 1.87 (1.18) 1.01 (1.72) 1.45 (1.52)
Total fixation time on user ratingsa (in seconds) 2.08 (3.50) 0.12 (0.29) 1.13 (2.67)
a
Significant differences between the two experimental conditions.

For the search-result evaluation phase the ANOVAs with
conditions as between subjects factor showed no significant
main effect of conditions on the number of visually inspected
search results, F(1, 27) ¼ 2.73, ns, and on the total fixation
time on search results, F < 1, ns. Similarly, the ANOVAs
showed no significant main effect of conditions on the
number of gazes, F < 1, ns, and on the total fixation time,
F < 1, ns, on publishing and source information of the search
results.
For the Web-page evaluation phase results of a MANOVA
with conditions as between subjects factor and the number of
gazes on the three Web page characteristics (i.e., publishing
information, source information, and user ratings) as three
dependent variables showed a significant overall main effect
of conditions on the number of gazes on these Web page
characteristics, Pillai’s trace ¼ 0.30, F(3, 25) ¼ 3.48,
p < .05, partial h2 ¼ .30. However, univariate ANOVAs
revealed that only for user ratings the number of gazes was
significantly higher in the Instructed Evaluation condition
than in the Spontaneous Evaluation condition, F(1,
27) ¼ 11.19, p < .05, partial h2 ¼ .16. For the two other
Web page characteristics there were no significant differ-
ences found between the two conditions, that is, for source
information, F(1, 27) ¼ 1.79, ns, and for publishing infor-
mation, F < 1, ns.
Furthermore, regarding the total fixation time on the three
Web page characteristics similar patterns were found. Results
of the respective MANOVA showed a nonsignificant overall
main effect of conditions on the total fixation time on the Web
page characteristics, F(3, 25) ¼ 2.40, p ¼ .09. However,
univariate ANOVAs showed that only for user ratings the total
fixation time was significantly longer in the Instructed Eval-
uation condition than in the Spontaneous Evaluation condi-
tion, F(1, 27) ¼ 4.34, p ¼ .05, partial h2 ¼ .14. For the two
other Web page characteristics there were no significant
differences found between the two conditions, that is, for
source information, F(1, 27) ¼ 2.42, ns, and for publishing
information, F(1, 27) ¼ 1.28, ns.
3.3. Search result selection
During the 20 min Web search participants on average
selected 4.98 (SD ¼ 1.98) search results per SERP. The number
of selected search results did not differ significantly, F(1,
27) ¼ 1.20, ns, between the two experimental conditions, with
M ¼ 4.57 (SD ¼ 2.08) per SERP in the Instructed Evaluation
condition and M ¼ 5.41 (SD ¼ 1.98) in the Spontaneous
Evaluation condition. Furthermore, between the two conditions
there were neither significant differences with regard to the
number of selections of the top five search results per SERP,
F(1, 27) < 1, ns, nor with regard to the number of selections of
the bottom five ones, F(1, 27) ¼ 1.77, ns. Participants in the
Instructed Evaluation condition on average selected 2.76
(SD ¼ 0.93) of the top five search results per SERP and 1.82
(SD ¼ 1.32) of the remaining ones. Participants in the Spon-
taneous Evaluation condition on average selected 2.95
(SD ¼ 1.00) of the top five search results per SERP and 2.45
(SD ¼ 1.22) of the bottom five ones. A repeated measures
ANOVA showed that participants selected the top five search
results of a SERP significantly more often than the bottom five
ones, F(1, 27) ¼ 15.07, p < .01, partial h2 ¼ .36.
3.4. Quality of information problem solving
A c2-test showed that there was a significant effect of
conditions on participants’ decision with regard to the infor-
mation problem, c2(1, N ¼ 29) ¼ 4.97, p < .05. In the
Spontaneous Evaluation condition 10 participants recom-
mended low fat diets and 4 participants recommended low
carb diets, whereas in the Instructed Evaluation condition all
of the 15 participants recommended low fat diets. However,
with regard to the quality of the statements justifying the
decision the results of an ANOVA showed no significant
differences between the two experimental conditions, F(1,
27) ¼ 2.42, ns, with M ¼ 1.29 (SD ¼ 0.92) for the Sponta-
neous Evaluation condition and M ¼ 1.73 (SD ¼ 0.96) for the
Instructed Evaluation condition.
 P. Gerjets et al. / Learning and Instruction 21 (2011) 220e231
4. Discussion
Appropriately evaluating Web search results and Web pages
is a crucial component when searching for complex and
science-related information due to the diverse and often
contradictory sources of information that can be found on the
Web. To investigate these evaluation processes, information
science studies (Rieh, 2002) have used explicit evaluation
instructions that yielded sets of different criteria used by
searchers to evaluate Web information. However, it can be
criticized that these studies fail to address spontaneous eval-
uation processes due to the instructions used. Thus, the study
reported in the present article compared standard thinking-
aloud instructions to explicit evaluation instructions to test
whether the latter lead to a distortion of users’ spontaneous
evaluation processes. As expected, differences were found for
thinking-aloud data (Hypothesis 1) and eye-tracking data
(Hypothesis 2) as well as on participants’ information problem
solving (Hypothesis 4), but not on search result selection
(Hypothesis 3). We will first discuss the results for the eval-
uation of search results on the SERPs and subsequently
elaborate on the findings on Web page evaluation.
4.1. Evaluation of search results
In line with Hypothesis 1, during the search-result evaluation
phase users in the Instructed Evaluation condition differed in
their verbal utterances from users in the Spontaneous Evalua-
tion condition in that they showed more credibility-related
verbal utterances when they inspected the SERPs. However,
contrary to the assumptions of Hypothesis 2 and Hypothesis 3
no differences could be found between experimental conditions
with regard to users’ eye-tracking data and search result
selection on the SERPs. More precisely, explicit evaluation
instructions did neither affect the number of attended search
results or the fixation time on search results nor the processing
of specific information on search results (publishing and source
information). Second, experimental groups did not differ with
regard to the overall number of selected search results or the
number of selected search results with lower rankings as would
have been expected if users had been stimulated to evaluate all
search results on their own with regard to information quality
instead of selecting search results only according to the topi-
cality ranking provided by the search engine.
In sum, it was found for the search result evaluation on
SERPs that due to the evaluation instructions participants
uttered their considerations with regard to the credibility of
different search results, but the effects of the instructions on
verbal utterances were not reflected in eye-tracking data and
log file data. One reason for the pattern of results found in this
study for the search-result evaluation might be that users spent
more time on the SERPs than they would spontaneously do
because we provided them with 20 min time for their Web
search that was confined to three SERPs with ten search
results each. Accordingly, the number of attended search
results as well as the number of selected search results per
SERP was uncommonly high. Both groups attended about
229
nine of the ten search results per SERP and selected about half
of the available search results. It is known from previous
research, however, that users normally attend to only three to
four search results per SERPdat least when solving simple
fact finding tasks (cf. Granka, Feusner, & Lorigo, 2008). Thus,
the lacking time pressure might have lowered the importance
of participants’ selection decisions. In future studies the time
available for Web search could be manipulated to clarify
whether a lack of time pressure may have reduced the pro-
cessing differences between the experimental groups in the
present study.
4.2. Evaluation of Web pages
As Web pages more often than SERPs contain character-
istics like publishing information, source information, or user
ratings that can be used to evaluate the quality of Web infor-
mation it was not unexpected that participants showed more
quality-related evaluative verbal utterances during the Web-
page evaluation phase than during the search-result evalua-
tion phase. Interestingly, the pattern for topicality was
reversed; this is in line with the idea that searchers focus on
topicality judgments when they evaluate Web search results.
During the Web-page evaluation phase there were several
differences between participants in the Instructed Evaluation
condition and participants in the Spontaneous Evaluation condi-
tion with regard to their thinking-aloud data (Hypothesis 1) and
their eye-tracking data (Hypothesis 2) as well as with regard to
the results of their information problem solving (Hypothesis 4).
First, in line with Hypothesis 1, evaluation instructions
increased the rate of verbal utterances with respect to quality-
related criteria like up-to-dateness, design, and credibility.
Second, eye-tracking data showed that evaluation instructions
increased searchers’ attention for user ratings, which is quality-
related information. Therefore, Hypothesis 2 was at least partly
confirmed. Third, evaluation instructions influenced the results
of participants’ information problem solving. In this study, all
participants in the Instructed Evaluation condition decided to
recommend low fat diets, whereas in the Spontaneous Evalua-
tion condition 10 participants recommended low fat diets and 4
participants recommended low carb diets. As the sub-collection
of sources in the present study that cited scientific evidence
mainly favoured low fat diets over low carb diets the decision
quality of the instructed evaluation group can be seen as supe-
rior. However, as participants in the Instructed Evaluation
condition were not able to better justify their decision than
participants in the Spontaneous Evaluation condition, we
consider Hypothesis 4 as only partly confirmed.
In sum, it was found for Web page evaluation that due to
the evaluation instructions, participants not only uttered more
quality-related evaluation criteria but also changed their
processing strategies and improved their information
problem solving. Thus, it becomes clear from our results that
findings in information science studies (Rieh, 2002) by no
means reflect the evaluation criteria users spontaneously
apply during Web search. It might be even criticized that the
standard thinking-aloud instructions that were used as
230 P. Gerjets et al. / Learning and Instruction 21 (2011) 220e231
a control condition for the explicit evaluation instruction is
still not very close to a natural search situation. First, users’
search space was restricted to three SERPs with given search
terms and was linked to 30 Web pages. Additionally,
participants were required to study these materials for
20 min. So they might have elaborated the materials more
thoroughly than they might have done under natural search
conditions. Second, the procedure of the experiment in both
conditions required participants to verbalize their thoughts
concurrently to their search process, which might have
interfered with participants’ cognitive processes. The use of
obtrusive measurements like concurrent thinking aloud may
have reactively influenced the search process itself (e.g.,
stimulated participants to process the materials more elabo-
rately). To investigate Web users’ natural search behaviour
more unobtrusively in future studies one might consider
combining eye tracking with cued retrospective verbal
utterances by presenting them with a screen recording of
their task processing superimposed with their eye movements
and mouse operations subsequent to the search task and
asking them to report retrospectively what they were thinking
during information seeking (cf. De Koning, Tabbers, Rikers,
& Paas, 2010; Jarodzka, Scheiter, Gerjets, & Van Gog, 2010;
Van Gog, Paas, Van Merriënboer, & Witte, 2005).
With regard to the practical implications of the present study it
can be concluded that stimulating learners to engage in quality-
related evaluation processes might be useful to improve their
Web search performance. This is in line with findings by Stadtler
and Bromme (2007) on the effects of the metacognitive tool
met.a.ware on users’ Web search. Results of their study showed
that participants’ knowledge about information sources and their
justification of credibility judgments could be significantly
enhanced by providing them with evaluation prompts from the
tool. Other manipulations beyond explicit evaluation instructions
that might serve this purpose could be to provide searchers with
a specific training (cf. Brand-Gruwel & Gerjets, 2008) or with
additional quality-related information on SERPs and Web pages.
For instance, a recent study by Walraven (2008) on a training
program for evaluation skills on the Web revealed that the
program indeed improved the evaluation behaviour of high
school students. With regard to the latter approach to provide
additional quality-related information on SERPs, Kammerer,
Wollny, Gerjets, and Scheiter (2009) could show that an
augmented search engine interface containing additional infor-
mation beyond topicality (source category labels) influenced
participants’ viewing and selection behaviour concerning the
search results compared to a standard search engine interface.
Similarly, Sundar, Knobloch-Westerwick, and Hastall (2007)
found that the availability of quality-related cues for news items
presented by the online news service Google News (i.e., infor-
mation on the source and the recency of a story, and on the number
of related articles) affected the subjective evaluation of news
leads. To conclude, the provision of both a specific training on
how to use different criteria to evaluate the quality of Web
information as well as an augmented search engine interface
design seem to be promising approaches in order to improve
searchers’ evaluation behaviour during Web search.
We expect that the methodological critique addressed in the
present article and the possible improvements outlined, finally
result in future research that will gain deeper insights into
evaluation processes during Web search. These insights will
also be useful to inform concrete training programs and design
guidelines for search interfaces in order to support users in an
adequate evaluation of Web information.
References
Borlund, P. (2003). The concept of relevance in IR. Journal of the American
Society for Information Science and Technology, 54, 913e925.
Brand-Gruwel, S., & Gerjets, P. (2008). Instructional support for enhancing
students’ information problem solving ability. Computers in Human
Behavior, 24, 615e622.
Brand-Gruwel, S., Wopereis, I., & Vermetten, Y. (2005). Information problem
solving by experts and novices: analysis of a complex cognitive skill.
Computers in Human Behavior, 21, 487e508.
Brumby, D. P., & Howes, A. (2008). Strategies for guiding interactive search:
an empirical investigation into the consequences of label relevance for
assessment and selection. Human-Computer Interaction, 23, 1e46.
Crystal, A., & Greenberg, J. (2006). Relevance criteria identified by health
information users during web searches. Journal of the American Society for
Information Science and Technology, 57, 1368e1382.
Cutrell, E., & Guan, Z. (2007). What are you looking for? An eye-tracking
study of information usage in Web search. In M. B. Rosson & D. J. Gil-
more (Eds.), Proceedings of the SIGCHI conference on human factors in
computing systems (pp. 407e416). New York: ACM Press.
De Koning, B. B., Tabbers, H. K., Rikers, R. M. J. P., & Paas, F. (2010).
Attention guidance in learning from complex animation: seeing is under-
standing? Learning and Instruction, 20, 111e122.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as
data. Cambridge, MA: MIT Press.
Gerjets, P., & Hellenthal-Schorr, T. (2008). Competent information search in
the World Wide Web: development and evaluation of a web training for
pupils. Computers in Human Behavior, 24, 693e715.
Granka, L., Feusner, M., & Lorigo, L. (2008). Eye monitoring in online search.
In R. I. Hammoud (Ed.), Passive eye monitoring: Algorithms, applications
and experiments (pp. 283e304). Berlin: Springer.
Granka, L., Joachims, T., & Gay, G. (2004). Eye-tracking analysis of user behavior
in WWW search. In M. Sanderson, K. Järvelin, J. Allan, & P. Bruza (Eds.),
Proceedings of the 27th annual ACM SIGIR conference on research and
development in information retrieval (pp. 478e479). New York: ACM Press.
Guan, Z., & Cutrell, E. (2007). An eye tracking study of the effect of target
rank on Web search. In M. B. Rosson & D. J. Gilmore (Eds.),
Proceedings of the SIGCHI conference on human factors in computing
systems (pp. 417e420). New York: ACM Press.
Höök, K., Benyon, D., & Munro, A. J. (Eds.). (2003). Designing information
spaces: The social navigation approach. London: Springer.
Jarodzka, H., Scheiter, K., Gerjets, P., & Van Gog, T. (2010). In the eyes of the
beholder: how experts and novices interpret dynamic stimuli. Learning and
Instruction, 20, 146e154.
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005). Accurately
interpreting click through data as implicit feedback. In R. Baeza-Yates, N.
Ziviani, G. Marchionini, A. Moffat, & J. Tait (Eds.), Proceedings of the 28th
annual international ACM SIGIR conference on research and development in
information retrieval (pp. 154e161). New York: ACM Press.
Kammerer, Y., Wollny, E., Gerjets, P., & Scheiter, K. (2009). How authority-related
epistemological beliefs and salience of source information influence the eval-
uation of web search results e an eye tracking study. In N. A. Taatgen & H. van
Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science
society (pp. 2158e2163). Austin, TX: Cognitive Science Society.
Maglaughlin, K. L., & Sonnenwald, D. H. (2002). User perspectives on
relevance criteria: a comparison among relevant, partially relevant, and
not-relevant judgments. Journal of the American Society for Information
Science and Technology, 53, 327e342.
 P. Gerjets et al. / Learning and Instruction 21 (2011) 220e231
Marchionini, G. (1995). Information seeking in electronic environments.
Cambridge, UK: Cambridge University Press.
Miller, C. S., & Remington, R. W. (2004). Modeling information navigation:
implications for information architecture. Human-Computer Interaction,
19, 225e271.
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L.
(2007). In Google we trust: users’ decisions on rank, position, and rele-
vance. Journal of Computer-Mediated Communication, 12, 801e823.
Pirolli, P., & Card, S. K. (1999). Information foraging. Psychological Review,
106, 643e675.
Pirolli, P., & Fu, W.-T. F. (2003). SNIF-ACT: a model of information foraging
on the World Wide Web. In P. Brusilovsky, A. T. Corbett, & F. De Rosis
(Eds.), Proceedings of the 9th international conference on user modeling
(pp. 45e54). Johnstown, PA: Springer.
Rayner, K. (1998). Eye movements in reading and information processing: 20
years of research. Psychological Bulletin, 124, 372e422.
Rieh, S. Y. (2002). Judgment of information quality and cognitive authority in
the web. Journal of the American Society for Information Science and
Technology, 53, 145e161.
Rouet, J.-F. (2006). The skills of document use: From text comprehension to
Web-based learning. Mahwah, NJ: Erlbaum.
Rouet, J.-F., Favart, M., Britt, M. A., & Perfetti, C. A. (1997). Studying and
using multiple documents in history: effects of discipline expertise.
Cognition and Instruction, 15, 85e106.
Savolainen, R., & Kari, J. (2005). User-defined relevance criteria in web
searching. Journal of Documentation, 62, 685e707.
231
Schamber, L. (1994). Relevance and information behaviour. In M. E. Wiliams
(Ed.), Annual review of information science and technology (ARIST) (pp.
3e48). Medford, NJ: Learned Information.
Stadtler, M., & Bromme, R. (2007). Dealing with multiple documents on the
WWW: the role of metacognition in the formation of documents models.
Computer-Supported Collaborative Learning, 2, 191e210.
Sundar, S. S., Knobloch-Westerwick, S., & Hastall, M. R. (2007). News cues:
information scent and cognitive heuristics. Journal of the American Society
for Information Science and Technology, 58, 366e378.
Tombros, A., Ruthven, I., & Jose, J. M. (2005). How users assess web pages
for information-seeking. Journal of the American Society for Information
Science and Technology, 56, 327e344.
Van Gog, T., Paas, F., & Van Merriënboer, J. J. G. (2005). Uncovering
expertise-related differences in troubleshooting performance: combining
eye movement and concurrent verbal protocol data. Applied Cognitive
Psychology, 19, 205e221.
Van Gog, T., Paas, F., Van Merriënboer, J. J. G., & Witte, P. (2005). Uncov-
ering the problem solving process: cued retrospective reporting versus
concurrent and retrospective reporting. Journal of Experimental
Psychology: Applied, 11, 237e244.
Walraven, A. (2008). Becoming a critical websearcher: Effects of instruction to
foster transfer. Unpublished doctoral dissertation. Heerlen, The
Netherlands: Open University of the Netherlands. http://hdl.handle.net/
1820/1734 Retrieved from.
Wilson, T. D. (1999). Models in information behaviour research. Journal of
Documentation, 55, 249e270.