CS Exam GuidelinesV1.2

Search
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
IP Lu 22
o m Le from
f r iK
22 Kuo uo IP
2 0 i
16, u Le
Satisfaction
e
b yLr
e cem 4.6 b
d a y, D 16.25
i .
Fr 172
Guidelines
A guide to providing satisfaction ratings for search results
Version 1.2
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172
 
Introduction 4 Flights 17
Search Needs and Satisfaction 4 Movies/TV Shows/Books/Music ida
Fr 18
y
The Query 5 How to Assign Ratings 17 , Dec 19
2.1 em
6.2 be
54 r 1
Steps in the Grading Process 5 When to Grade Highly Satisfying (HS) .6
by 6, 20 19
IP Lu 22
Definitions from 6 When to Grade Satisfying (S) Le from 22
iK
uo IP
022 i Kuo
Result Validation
, 2
16 Le 8 When to Grade Somewhat Satisfying (SS) 24
m ber by Lu
eWrong
c e .6 Language 8 When to Grade Not Satisfying (NS) 25
a y , D 6.254
id .1
Fr 172 Content Unavailable 8 Grading Specific Situations & Result Types 27
Inappropriate 9 Ambiguous Queries (Multiple Interpretations) 28
Satisfaction Principles 11 Locale Sensitivity 30
Satisfaction Scale 11 English Results in Non-English Locales 31
Degrees of Separation 12 Redirected Pages 31
Think About the Meaning, Not Just Matching Words 13 Apps 32
Consider User Effort 13 News 33
Consider Source Quality 13 Maps 34
Overview of Result Types 14 Web Video 35
Web Results 14 Dictionary, Stocks, Weather, Knowledge / Answers , Sports 36
Apps 14 Web Results (also called Suggested Web Sites) 36
Maps 14 Web Images 36
Fr
y Stocks 15 Common Grading Mistakes 39
ida
17 , Dec
2.1 Dictionary 15 Failing to Use Web Search 39
6.2 embe
54 r 1
.6
by 6, 20
Weather 15 Failing to Visit Destination Page 40
Lu 22 IP
L fro m
Sports ei Kuo m IP 15 Ignoring Time and Place 2 2 fro o 40
u
6 , 20 Lei K
News 16 Ignoring Conceptual Distance 1
ber by Lu
40
c e m
Web Images 16 Ignoring Relevance Grading Principles y, De .254.6 41
ida .16
Web Video 16 Examples: Satisfaction Rating Fr 172 43
Answers and Knowledge 17 Highly Satisfying 43

Satisfying Examples 45
Somewhat Satisfying Examples 48 Fr
ida
y
Not Satisfying Examples 50 17 , Dec
2.1 em
6.2 be
54 r 1
Other Aspects Related to Search Satisfaction Grading 51 .6
by 6, 20
IP Lu 22
Overall Preference
m Rating (OPR) 51 Le from
fro iK
uo IP
022 i Kuo
Writing 16 Comments
, 2
Le 52
b er y Lu
OPRDe&cem Comment
4 .6 b Examples 53
y , 6.2 5
id a .1
Fr 172
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

Introduction ida
Fr
y
17 , Dec
2.1 em
6.2 be
54 r 1
Search Needs and Satisfaction 4 Search Needs and Satisfaction .6
by 6, 20
IP Lu 22
The Query m 5 Le from
2 fro o iK
uo IP
2 u
, 20 Lei K Process Search engine users are trying to accomplish a task (or achieve a goal)
Steps in the16Grading 5
m ber by Lu that requires some information or quick access to some other
Definitionse c e .6 6
y , D 6.254 resource, such as an app.
id a .1
Fr 172
A user s information need or search need is de ned as the
information or resource that the user needs in order to accomplish
A search service may return many di erent types of results. How are
their task. The user's query is an attempt to express that need to the
these graded? What is a satisfying search result? In these guidelines
search engine. If the search results enable the user to accomplish their
we talk about what constitutes a search query, the di erent types of
task, we say that the search need is satis ed.
results, and how to grade them. In addition we describe some typical
grading tasks that use the principles learned in satisfaction grading.
We say that a result is satisfying if it satis es the search need of a
query. Results can be more satisfying or less satisfying depending on
how well or how completely they satisfy the need.
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP A search need → a search query 2f o
A search query → results returned , 202 i Ku
16 Le
er y Lu
You may assume all searches are made on an Apple iOS mobile b
m b
Dece 54.6
device. y ,
ida 72.16
.2
r

F 1
What is Search Need and Relevance Page 4 of 58

 

ff

fi
fi
 
fi
ff

Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
IP Lu 22
o m Le from
f r iK
22 Kuo uo IP
2 0 i
r 16, u Le
e
b yL A query and its associated information in the grading interface.
e cem 4.6 b
y, D 16.25
FrThe
i 72. Query
d a 1. Click on the Google and Bing web search links and scan the results
1
to make sure you understand what the query is about. Keep in mind
The grading interface displays each query together with additional queries can have more than one meaning.
information that provides useful context. As shown in the gure above,
2. Validate the result to make sure it can be graded, as explained in the
this includes the following components:
Result Validation section. Following step (1) is crucial for correct
• The query itself validation.
• Web Search links you will use to research the possible intents and 3. Assign the satisfaction rating per the guidelines outlined in
interpretations of the query
• Relevance Principles
• The language of the user. We do not want to return results in other • Assigning a Satisfaction Rating
languages • Special Situations
• The location of the user. We want to return results appropriate for

When assigning your grade, be on the lookout for common mistakes!
their area (e.g. locations of business).
Fr Details can be found in Common Mistakes made.
iday,
•1 DateDe of query. We want to return results that are relevant in time.
72
.16 cemb
.25 er ⚠ Search engines often correct query spelling errors and/or predict
4.6 16
by , 20 (“autocomplete”) what a partially typed query was intended to be. If the web search
⚠ Unless Lu you
Le fhave
22
rom been speci cally instructed otherwise, skip to the next results show results for a corrected or autocompleted version ofothe
m
IPquery, you
i
task if any
K uoof Ithe
P above information about the query is missing. should grade your result as if the user typed the corrected
fr o
022 orKucompleted
2 i
query.
r 16, u Le
e L
Examples:
mb 6 by
Steps in the Grading Process • Query is “fac,” result is “facebook.com”. Grade
c e .
De if.25the
, as 4 query was “facebook.”
a y 1 6
• Query is “ted cruise,” result is a wikipedia F rid 7about
page . U.S. senator Ted Cruz.  
1 2
The grading of results consists of the following steps. Grade as if the query was “ted cruz.”

fi

 
fi
Definitions
Fr
ida
The following terms are used throughout these guidelines: y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
IP Lu 22
o m Le from
f r iK
22 Kuo uo IP
16, u Le Term Definition Examples
2 0 i
e
b yLr
e cem 4.6 b • Stephen Curry
d a y, D 16.25
i
Fr 172
. • Yellowstone National Park
• Jupiter
• Médecins Sans Frontières
A person, place, organization, business, product, service, or • Starbucks
Named Entity event whose name would normally be capitalized in English. • Post-It Notes
(This includes ctional entities.) • Skype
• Super Bowl LI
• Boxer Rebellion
• Frodo Baggins
• photosynthesis
• elephant
A word or phrase describing a concept or object of study • ROC curve
Fr (other than a named entity) that users may wish to learn • linear algebra
ida
y
17 , Dec
more about. Knowledge terms may come from any eld of • cancer
2.1 em
6.2 be Knowledge Term study, including: science, technology, mathematics, medicine, • oligarchy
54 r 1
.6
by 6, 20 history, philosophy, literature, art, economics, etc. They are • veto
Lu 22 IP
Le from m
iK
uo IP most often noun phrases, but may also be other parts of • existentialism r o
2 f uo
0 2
speech. • metaphor 2 iK
r 16, u Le
e L
• impressionism c e mb 6 by
e .
y , D 6.254
• interest rate rid 72.1
a
F 1

fi
fi
Term Definition Examples
Fr
ida
y
17 , Dec
• Microsoft (company):
2 .16 emwww.microsoft.com
.25 ber
• U.S. Internal Revenue4.6Service1
by 6, 20 (government
IP L 22
r o m A website provided by a named entity (or their employer or organization): www.irs.govu Lei from
f Ku
22 Kuo o IP
6, Lei cial
O Site
2 0 organization) that represents how they want to be presented • Taylor Swift (performer): www.taylorswift.com
1
er Lu to the world online. • Henry Louis Gates Jr. (professor at Harvard
c emb 6 by
e .
a y , D 6.254 University): https://aaas.fas.harvard.edu/
id .1
Fr 172
people/henry-louis-gates-jr
A generalization of o cial site that includes not just o cial

sites but also other online homes provided by an entity and • https://twitter.com/StephenKing
O cial Online Presence existing on commercial services such as social networks. This • https://www.youtube.com/user/therock
may include: a Twitter feed, Facebook page, YouTube • https://www.instagram.com/badbunnypr/
channel, Instagram feed, or other similar platform.
A business (or organization) that consists of many locations • Starbucks

that all provide basically the same product or service, AND • Taco Bell
Chain Business
where its customers (or users ) primary interaction with the • Party City
business happens in person at those locations. • California Department of Motor Vehicles
Fr
ida
y
17 , Dec
2.1 em • Jacinda Ardern
6.2 be
54 r 1
.6 • Taj Mahal
by 6, 20
Lu 22 IP
Le from Anything whose concept or identity can be usefully conveyed • ball-peen hammer o m
iK r
uo IP
0 2 2 f uo
by a visual image. People and places are visually distinctive • dodecahedron 2
16, u Le
iK
Visually Distinctive Entity e r L
entities, but so are certain tools, geometric gures, • mesa c e mb 6 by
De 54 .
geological or architectural features, and visual artworks. • ying buttress iday, .16.2
Fr 172
• The Thinker (sculpture by Rodin)

fl
ffi
ffi
ffi
fi
ffi
Result Validation ida
Fr
y
17 , Dec
2.1 em
6.2 be
54 r 1
Wrong Language 8 4. Query is in a foreign language and result is .6 in locale
by 6, 20 language, but
IP Lu 22
Content Unavailable m 8 query is also the name of a popular song, movie, Le business,
fr
i K om I
etc. in
fro uo P
2 022 i Kuo the current locale (e.g. viva la vida query in en-US).
Inappropriate ,
16 Le 9
b er y Lu
m b
D ece 54.6
y,
Before
ida 72.16 you can grade the satisfaction of a result, you ll be asked to
.2
r
F 1
indicate whether there are any problems that would prevent you from ⚠ English results are never considered Wrong Language
judging it. There are three types of result problems you ll be asked to
identify: wrong language, content unavailable, and inappropriate.
Content Unavailable
Wrong Language
Flag result as content unavailable in any of these situations:
A result is in the wrong language if it is neither in English nor in the
language of the user s locale. • A result is a web/news or videos result but does not show a page
when clicked.
However, there are a few exceptions that are NOT considered wrong
language results: • Result requires log-in or subscription to access, speci cally where the
user would be able to see the content of the page by logging in, but
1. Result (e.g. amazon.co.jp) is the same country-speci c site as you cannot.
requested by the query ( amazon.co.jp ), even if the requested site
Fr • The browser presents a dialog box warning of a privacy or security
ida is not in your locale.
y
17 , Dec issue on the page.
2.1 em
6.2 be
2. Query54 rand result are in the same language, even though it s not the
.6 16,
by 20
primaryLulanguage
22 for this locale. • Required information for this result type is missing (e.g.Pno distance
Le from I
iK shown for Maps result). f rom
uo IP 2 o
3. User is visiting another country, query is for a local business or 6 , 202 ei Ku
1 L
attraction, result is in the language of the visited country (i.e. where b er y Lu
em .6 b
ecrating
query was submitted), and there is no equivalent result in the user s ⚠ Even if there is enough content to provide y , Da .25
4 but the page is behind
a 6
Fr Content Unavailable ag
a pay-wall/log-in, please check the id 72. 1
own locale language. 1
Result Validation Page 8 of 58

 

fl
fi
fi
 

Inappropriate
Fr
ida
A result is considered inappropriate if it has any of the following: y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
pornography,
o m
IP adult advertising/services, sex toys, illegal drugs, hate speech, gambling, spam/phishing, Lu 22
Le from
f r iK
, 2
6 Le
022 i Kuo pirated content(including those posing as free video streaming services), or gore/shock uo IP
1
m ber by Lu
D ece 54.6
In ,
y 16.2
r ida general,
. we want to connect users with useful content for their topic attempt to arti cially boost their relevance (e.g., link farming,
F 172
of interest while protecting them from being exposed to harmful keyword stu ng, etc).
information summarized below.
• Results that do not contain original and useful content. Examples:
• Hateful: the result should not advocate discriminatory content that pages with content scraped from Wikipedia or otherwise
intentionally attacks someone s dignity. This can include references automatically-created content.
or commentary about religion, race, sexual orientation, gender,
national/ethnic origin, or other targeted groups. • Illegal: We also manually remove reported results in those
circumstances that are required by law in the corresponding locale
• Violent or harmful: the result should not intentionally incite imminent (e.g., images of child abuse, content related to sex tra cking,
violent, physically dangerous, or illegal activities, nor provide copyright infringement, etc.) and when action is required to keep
information that leads to immediate harm. people safe (e.g., involuntary posting of sensitive personal
information, etc). Movie streaming sites such as those posing as free
• Sexually explicit: the result should not have overtly sexual or movies are also part of this category
pornographic material, de ned by Webster s Dictionary as "explicit
descriptions or displays of sexual organs or activities that are ⚠ Content that might otherwise be considered inappropriate is acceptable
Fr principally intended to stimulate erotic without su cient
ida if it occurs in a medical, educational, ne art, or journalistic context, and
y, D
17 aesthetic
2. ece or emotional feelings. should not be agged (e.g Wikipedia).
16 mb
.25 er
4.6 16
• by , 20
Contradicting
Lu 22 expert consensus on public interest topics: the Examples IP
Le from m
result should i K not Icontradict well-established or expert consensus on r o
uo P
0 2 2 f uo
a popular topic or issue. This includes misleading or inaccurate • User searched for [tinyzone] and the result is
r
2 ei K
16,https://
L
b e y Lu
information. tinyzonetv.to/ which contains pirated content. m b
Dece 54.6
y , .2
ida 72.16
• Spam Results that are malicious, deceptive, or manipulative. • r
User searched for [sdc.com] and Fresult1 is http://sdc.com/, or user
Examples: pages that contain phishing schemes, install viruses, or searched [olga 24k gold] and the result is https://www.lelo.com/

ffi
fl
fi

fi
fi
ffi
ffi
 
blog/olga-24k-gold-review/. Both results contain adult advertising
and should be agged. Fr
ida
Irrespective of whether the user
17 , Decwas searching for
y
2.1 em
6 be
this, these results need to.25be
4.6 r 16agged.
, by 2
IP Lu 022
m Le from
2 fro o iK
uo IP
6 , 202 ei Ku
1 L
m ber by Lu
D ece 54.6
y , .2
r ida 72.16
F 1
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

fl
fl
Satisfaction Principles  Fr
ida
y
17 , Dec
2.1 em
Satisfaction Scale 6.2 be
54 r 1 11
.6
by 6, 20
Degrees of Separation
IP Lu 22 12
m Le from
Think About the 2 fro o
Meaning, Not Just Matching Words
iK
uo IP 13
, 2 02 i Ku
6 Le
ConsiderbeUser r 1 LuEffort 13
m by
ece 54.6
Consider
y, D .2 Source Quality 13
r ida 72.16
F 1
Satisfaction Scale
When judging how satisfying each result is, you ll use the following scale
Highly Satisfying Satisfying Somewhat Satisfying Not Satisfying
Almost all users would want to see this result. Many users would be interested in seeing this Some users may nd this result useful, but it s This result has nothing to do with the query, or
It s authoritative, accurate, up-to-date, and result. Satisfying results often provide probably not what most searchers were looking provides incorrect information, and should not
addresses the most likely search need(s). If the supplementary information that is one step for. It s often only indirectly related to the be shown.
Fr
user
ida is asking a speci c question, the result away from the query topic. search need or assumes an uncommon
y,
72 Dethe
1gives correct answer clearly and concisely. For example, if the query is a restaurant, it interpretation of the query. All results agged as Inappropriate ,
.16 cemb
.25 er might be a review of the restaurant; if the Content Unavailable , or Wrong Language
4.6 16
by , 20 query is a company, it might be the current should be rated as Not Satisfying.
Lu 22 IP
Le from stock price, or news about the company. o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
Satisfaction Scale c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

fl
fi

fi

 

Degrees of Separation
Fr
ida
Results are often associated with concepts in the real world, and di erent concepts are connected by their relationships.y
17 , Dec
2.1 em
6.2 be
54 r 1
For example, the concept of the singer Beyoncé .6
by 6, 20
IP Lu 22
m Le from
f ro iK
• is related to02the
2 uoconcept of her album Lemonade, uo IP
K
1 6, 2 Lei
er y Lu
• whichc emb 6in
b turn is related to a review of the album in Rolling Stone magazine,
D e 54.
y, .2
r ida 72.16
F 1• which is related to the author of the review, Rob She eld.
Each time we pass through one of these relationships, we increase the distance from the original concept
Query : Beyoncé Query: Rolling Stone Lemonade album review
Beyoncé's o cial website. The review of the album.

Highly Satisfying
Her Lemonade album on iTunes. The album.

Satisfying
A Rolling Stone magazine review of the album. The singer's o cial site and Rob She eld's Twitter.
Somewhat Satisfying
The reviewer Rob She eld's Twitter. Random article from same issue of Rolling Stone
Not Satisfying
Fr
ida
y Degrees of Separation
17 , Dec
2.1 em
6.2 be
We can 54 think
.6 16, of these relationships as degrees of separation so in this example, the review of the Lemonade album is two degrees of separation
r
by 2
from Beyoncé. Lu 022 IP
Le from o m
iK r
uo IP
0 2 2 f uo
When Grading results, each degree of separation from the concept mentioned in the query, that is, the number of relationships 1you ei K
6, 2 Lhave to traverse to
b r
e yL u
get to the result, lowers the grade by one level. See table above. m b
ece 54.6
y , D .2
r ida 72.16
F 1

ffi
ffi
ffi

ffi
ffi
ff
Think About the Meaning, Not Just Matching Words Consider Source Quality
Fr
ida
Note that some highly satisfying results may not contain all (or even Sources of results, including web sites 1andy,
72 Denews providers, can have
.16 cemb
any) of the query words; what matters is the meaning. For example: large di erences in quality. When you are grading
.25 er a result, particularly
4.6 16
by , 20 ̶ pay attention to
if the user s query is looking for speci c information
I P Lu 22
• The result www.premierleague.com/home
rom
is highly satisfying for the L
the quality of the source(see table Source Quality ).ei KFor
fro
m example, if
2 f o uo IP
query english, 202 ei Kpremier
u league soccer even though that result you are interested in getting news about an event that happened in a
1 6 L
doesn er y Lu
mbt containb
the words english or soccer. certain city, a story in that city s newspaper is generally more reliable
D ece 54.6
y, 16.2 than a blog post by a random person who doesn t live there. If the
idaThe
F 172 result https://music.apple.com/us/album/25/1544494115 is
•
r .
highly satisfying for the query adele s third album, even though it source of a result is low quality, you should assign a lower grade than
doesn t contain the word third. you would have otherwise.
It's also possible for a result to contain all the query words and not be High Quality Low Quality
satisfying. For example:

Professionally written, clear and Unclear, hard to read, lled with
Writing understandable. grammatical and spelling errors.
• The result https://en.wikipedia.org/wiki/My_Girl_Has_Gone (a web
page about a song from the 1960s) is not satisfying for the query
gone girl, even though the result contains both query words. Gone Has "hidden agenda," such as
Neutral point of view, or makes point of view
Motivation pretending to o er information while
clear.
Girl is the title of a book and movie from the 2010s, and the song actually trying to sell its services.
result is clearly not what the user intended.
Well-known and well-respected among those Unknown (or known to be unreliable
Reputation who provide this kind of service. and untrustworthy).
Consider User Effort
Fr
ayid Use of If o ering scienti c or medical information, Makes medical or scienti c claims
When
17 , Dec the user is looking for speci c information, a result that displays cites sources. without citations or evidence.
2 em Citations
this.16information
.25 ber directly is preferable to a regular web result. For
4.6 16
example,by if ,the
2
Lu 022
query is how old is Obama , then a Knowledge card
Source Quality IP
ei K from
that directly Ldisplays his age without requiring any user action is better r o m
uo IP
0 2 2 f uo
than a web result that the user needs to click on, wait for it to load, and 2 iK
r 16, u Le
scroll through to nd the desired information. e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

ff
ff
ff
fi
fi
fi
fi

fi
fi
 

Overview of Result Types ida

Fr
y
17 , Dec
2.1 em
There are many types of search results. Some results, when clicked, take you to a web page. Some others reveal rich user6.2experiences
b
54 er 1
when clicked.
.6 6
Others are self contained (not clickable) and answer search needs directly in the information presented, without the need for further
by , 20 user action.
Lu 22
I P
Rating advice isfrgiven o m sections How to Assign Ratings and Special Advice for Result Types. Le from
iK
2 2 u o uo IP
, 2 0 iK
16 Le
b er y Lu
m b
ece 54.6
Web y, D
ida 72.16
Results
.2
r
F 1
By far the most common result types. These cards usually have an
icon with a brief title of the webpage and are designed to be clicked by
the user and taken to the corresponding website.
Maps
These results help the user navigate to a place. Usually they have
address and distance from the user. If it s a business it often has hours
of operation.
Fr
ida
y
17 , Dec
Apps
2 .16 emb
.25 er
4.6 16
by , 20
22
Cards thatLutake Le fthe
r user to the Apple app store (or open an app on the m
IP
i K om I r o
device). Usually uthey o P have an icon of the app and the star ratings. 0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172
 
Stocks Weather
Fr
ida
This card provides nancial information related to stocks. They should This card that shows the temperature of y
17 ,aDelocation (and sometimes
2.1 cem
show the ticker symbol, the company name and the stock price. When other weather conditions). When the user 6taps
.25 bethis card, they are
4.6 r 16
the user interacts with this card detailed stock information such shown detailed multi day weather forecasts. by L 2022 ,
IP uL
historic price graphs
ro m are displayed. ei K from
2f 02 i Kuo
uo IP
, 2
16 Le
b er y Lu
m b
D ece 54.6
y, .2
r ida 72.16
F 1
Sports
These cards are meant to display sports scores, or latest scores for a
Dictionary team (and dates of upcoming matches). Some examples
This card shows the de nition of word. When the user interacts with
this card it provides detailed usage.
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172
fi
fi
News Web Video
Fr
ida
These are often types of web results that are restricted to news sites The user can click on these results which y D
17 , play a video (usually taken
2.1 ecem
(sports, fashion, political and so on). The usually have age of news from video channels such as YouTube and6.Vimeo.
25 ber
4.6 1
indicator at the bottom. They are designed to be clicked on and take by 6, 20
IP Lu 22
m Le from
the user to the2 fdestination
ro news site. iK
uo IP
, 2 02 i Kuo
16 Le
b er y Lu
m b
D ece 54.6
y, .2
r ida 72.16
F 1
Web Images
Groups of images clustered together. Usually the user doesn t interact

with the images and they provide visual information about the search
query.
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172
Answers and Knowledge Flights

Fr
ida
Users ask questions (implicit, explicit, grammatically incorrect) about a This will display ight status such arrival y, D
17 time, departure time and
2.1 ecem
concept or knowledge term or general knowledge question. Knowledge destinations. When the user taps on this result,
6.2 be detailed information
54 r 1
.6 6
cards can return exact answers or rich experiences about knowledge about arrival/departure gates, baggage claims bare y L , 2displayed.
02
IP 2
uL
m
concepts and entities.
ro ei K from
2f 02 i Kuo
uo IP
, 2
16 u Le
(Note, mthe ber byterm
L Knowledge might not appear)
e ce 4.6
d a y, D 16.25
i .
Fr 172
Query: Where is Olympics 2024 Query: macron
Query: Bubonic plague Query: haiku
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172
fl

Movies/TV Shows/Books/Music
Fr
ida
Cards that provide the user a very rich experience for example to y
17 , Dec
2.1 em
watch movies/tv show, learn about the cast, social media links, links to 6.2 be
54 r 1
.6
media related sites P(e.g IMDB), listen to music, get lyrics for songs, by 6, 20
I Lu 22
Le from
read books. They f romusually show a picture, popularity ratings etc. Some iK
uo IP
022 uo
examples:16, 2 Lei K
er Lu
c emb 6 by
e .
a y , D 6.254
id .1
Fr 172
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

How to Assign Ratings Fr

ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
When to Grade IHighly
P Satisfying (HS) Lu 22
Le from
m
2 fro o iK
uo IP
6 , 202 ei Ku
1 L
ber by Lu
m
ece 54.6 ⚠ Note that some types of results can never be HS.
Almost
y , D .2 all users would want to see this result. It s authoritative,
r ida 72.16
F accurate,
1 up-to-date, and addresses the most likely search need(s). • News results can never be HS, because people have di erent preferences for where they get their
news, so we can’t say that almost all users would want to see a given story
If the user is asking a speci c question, the result gives the correct • Results for advice or recommendation queries (e.g.,“how to lose weight”, “chicken parmesan recipe”,
answer clearly and concisely. “best beatles song”, “thai restaurant”) can never be HS, because we don’t know if almost all users
would agree with the recommendation.
When to grade Highly Satisfying
Rule If query is And result is Description Examples
Query is the name of a well-known app; result is the a. Query is “facebook”, result is the Facebook app.
1 App Query Official App
app with that name b. Query is “calculator,” result is the built-in Calculator app.
App Regularly Used Query is the name of a business; result is an app a. Query is “b of a,” result is the Bank of America mobile banking app.
2 Business to Interact with regularly used to interact with that business. See b. Query is “dominos,” result is the Domino’s Pizza app, which allows
Business details under “Apps” in “Additional Guidance”. users to place orders.
Fr
ida Query is looking for a specific location / business /
y
17 , Dec a. Query is “1234 market street sf”; result is a Map for that exact address
2.1 em institution / point of interest, or the closest example
6.2 be
54 r 1 b. Query is “new york public library”; result is a Map to that location
.6 of a chain business / type of business, and the
by 6, 20 c. Query is “larry and joe’s”; result is a Map to a restaurant with that name
Lu 22 result showed that location on a map. IP
Le from
i Maps in the same town where user is located o m
3 Ku Query Closest Map r
2 f the
o IP d. Query is “closest lowe’s”; result is a Map showing o
2 ei Ku Lowe’s store
0 2
Queries with a map intent often have a distance 6 ,
location closest to the user’s location. er 1 Lu L
qualifier e.g. "nearest", "closest", "near me". Also m b by
e. Query is “starbucks”; result is a Map c e showing
6 the closest Starbucks
such queries often relate to business where one y , De .254.
branch. ida .16
must physically go to e.g. gas stations, cinema halls Fr 172
How to Assign Ratings Page 19 of 58

fi

ff


Fr
i a
a. Query is “facebook,” result is dFacebook’s
y official website,
17 , Dec
facebook.com. 2 .16 mbe
.25 er
b. Query is “taylor swift,” result is the singer’s
4.6 16official website,
by , 20
IP taylorswift.com. Lu 22
o m Le from
f r c. iK
Query is “charli d’amelio” (social media personality/vlogger), result is
22 Kuo uo IP
2 0
16, u Le
i her TikTok channel.
e r Official Online Query is a named entity; result is an official online
4 emb by L Named Entity d. Query is “joe biden,” result is his Twitter profile https://twitter.com/
ec 54.6 Presence presence for that entity if it has one.
y , D .2 JoeBiden.
r ida 72.16
F 1 e. Query is “empire falls book,” result is publisher’s official page for the
book, https://www.penguinrandomhouse.com/books/159148/empire-
falls-by-richard-russo/9780375726408/.
f. Query is “captain fantastic,” result is official web site for the
movie, https://bleeckerstreetmedia.com/captainfantastic.
a. Query is “taylor swift” (singer), result is https://en.wikipedia.org/wiki/

Taylor_Swift.
b. Query is “nope” (2022 movie), result is https://en.wikipedia.org/wiki/
Nope_(film).
c. Query is “iliad” (ancient epic poem), result is https://en.wikipedia.org/
wiki/Iliad
d. Query is “the school of athens” (Renaissance painting by Raphael),
result is https://en.wikipedia.org/wiki/The_School_of_Athens
Query is a named entity; result is the wikipedia
Wikipedia or Other e. Query is “marie curie” (Nobel-prize-winning scientist); result is https://
page for that entity, a page from another
5 Named Entity Authoritative en.wikipedia.org/wiki/Marie_Curie
authoritative reference, or a knowledge card about
Reference f. Query is “angkor wat” (ancient temple complex in Cambodia); result is
that entity.
Fr
ida https://en.wikipedia.org/wiki/Angkor_Wat
y g. Query is “aristotle,” result is a page about the philosopher from the
17 , Dec
2.1 em
6.2 be Stanford Encyclopedia of Philosophy
54 r 1
.6
by 6, 20 h. Query is “jurassic world dominion,” result is https://www.imdb.com/
Lu 22 IP
Le from title/tt8041270/, IMDB page about that movie. o m
iK r
uo IP i. Query is “mike trout,” result is page of this player’s 0 2 2 f uofficial
o statistics in
6 , 2 ei K
the Baseball Reference, https://www.baseball-reference.com/players/t/
1
er y Lu
L
b
m b
troutmi01.shtml. ece 54.6
y , D .2
r ida 72.16
F 1

Fr
ida
Query is a knowledge term or general request to y
a. 17 , Dec
Query is “linguistics”; result is https://en.wikipedia.org/wiki/Linguistics
learn about a subject; result is the wikipedia page 2.1 em
b. Query is “what causes diabetes,”6result
.25 beisr a page about that disease
for that term, a page from another authoritative 4.6 16
from the Mayo Clinic website (https://www.mayoclinic.org/diseases-
by , 20
IP Wikipedia or Other reference, or a knowledge card. Common for Lu 22
Knowledgeo m Term or conditions/diabetes/symptoms-causes/syc-20371444). Le from
iK
6 r
f o Authoritative medical queries.
“Learn 2
202 ei KAbout”
u Query c. o IPgiving the
Query is “utilitarianism,” result is a Dictionary infoucard
1 6 , L Reference
ber by Lu definition of the term.
m Note that if “X” is a knowledge term, queries such
ece 54.6 d. Query is “challenger disaster” (historical event); result is https://
y , D .2 as “what is X?” or “tell me about X” still count as a
r ida 72.16 en.wikipedia.org/wiki/Space_Shuttle_Challenger_disaster
F 1 knowledge term queries.
a. Query is “when did wwi end,” result is a direct answer or info card that
says “November 11, 1918”
b. Query is “dodgers score,” result is a sports info card that shows the
current score of the Dodgers’ baseball game in progress, or (if no
game is in progress), the final score of the most recent game they
Query is asking for a specific piece of information played.
Explicit Correct that has a simple right answer, and the result c. Query is “msft quote,” result is an info card showing the latest stock
7 Exact Question
Answer showed that information directly without the need price for Microsoft (which has the stock symbol MSFT).
for further user action. d. Query is “jet blue 334,” result is an info card showing the current
status of that airline flight.
e. Query is “define attenuated,” result is an info card showing the
definition of that word.
f. Query is “weather boston", result is an info card showing current
weather for that city.
Fr a. Query is “nelson mandela,” result is the following set of images:

ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6Visually
,2
Distinctive
Query is (or asks about) a visually distinctive entity,
Lu 022 IP
8 Le from Web Image and result is a high quality web image set showing m
i K Entity r o
uo IP that entity. 0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172
When to Grade Satisfying (S)

Fr
ida
Many users would be interested in seeing this result. Satisfying results often provide supplementary information that is y, D step away from the
17 one
2.1 ecem
query topic. For example, if the query is a restaurant, it might be a review of the restaurant; if the query is a company, it might
6.2 be be the current stock
54 r 1
.6
price, or news about the company. Here are some common situations where a result is Satisfying: by 6, 20
P I L 22 uL
f rom ei K from
uo IP
, 2 022 i Kuo When to grade Satisfying
16 Le
b er y Lu
m b
RuleD ece 54.6 If query is And result is Description Examples
a y , 6.2
id .1
Fr 172
Query is the name of an app, result is a variant version (e.g.,
a. Query is “candy crush saga,” result is app store result for
1 App Name Variant of App “Pro” or “Lite”) of or sequel to that app, or another
“candy crush friends,” a newer game in the same series.
complementary app from the same vendor.
a. Query is “currency converter,” result is “My Currency

Query is a description of a type of app or function that app
App Performing Converter” app.
2 App Description needs to perform; result is an app (or web app) that performs
That Function b. Query is “time in different countries,” result is https://
that function.
www.timeanddate.com/worldclock/ .
Query is the name of a performer (singer, actor, etc.) or

a. Query is “taylor swift,” result is Apple Music result for singer’s
Performer’s/ creator (author, composer, artist, etc.); result is a
3 Performer/Creator recent album “Lover,” https://music.apple.com/us/album/lover/
Creator’s Work representation of their work (album, song, movie, book, etc.),
1468058165.
where user can view/hear/download/stream/learn about it.
Query is the name of a creative work (music album, movie, a. Query is “fleabag,” result is https://en.wikipedia.org/wiki/
Fr 4 Creative Work Performer/Creator etc.); result is a representation of the creator/performer (e.g., Phoebe_Waller-Bridge, the wikipedia page about the creator and
ida
y artist’s official site). star of that television series.
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172


Fr
ida
a. Query is “jbl bluetooth y speaker,” result is page of matching
17 , Dec
items from electronics.1retailer
2
6.2 embeBest Buy.
b. Query is “empire falls book,” r
54 result
.6 16, is Amazon’s detail page
by 2
IP Lu 022
for that book, https://www.amazon.com/Empire-Falls-
o m Query is the name of a product (which may be media item Le from
f r Richard-Russo/dp/0375726403 iK
22 Kuo such as a book, movie, song, etc.); result is a page from a uo IP
2 0
5 16, u Le Product
i Reputable Vendor c. Query is “captain fantastic,” result is iTunes store page for
e
b yLr well-known site where the item can be purchased,
that movie, https://itunes.apple.com/us/movie/captain-
e cem 4.6 b downloaded, or streamed.
y, D 16.25 fantastic/id1127934488
i d a .
Fr 172 d. Query is “taylor swift lover album,” result is Spotify page to
stream that album, https://open.spotify.com/album/
3rYkgtFOo9AlPaeKTtn6pM
Query is a named entity, result is an authoritative page (other a. Query is “facebook,” result is news story “Facebook agrees to
6 Named Entity News than official online presence) providing news about that pay FTC $5 billion fine for various privacy violations,” dated
entity. the same day the search was performed.
Query is asking for specific piece of information with a simple a. Query is “barack obama age,” result is https://
Embedded Correct right answer, and the result contains that answer, but the en.wikipedia.org/wiki/Barack_Obama.
7 Exact Question
Answer user has to take an action (e.g., follow link to destination b. Query is “cambridge library hours,” result is https://
page and read it) to get the answer. www.cambridgema.gov/cpl/hoursandlocations.
a. Query is “ebola,” result is New York Times news story “Ebola

Knowledge Term or Query is a knowledge term or request to learn about a
8 News Outbreak in Congo Is Declared a Global Health Emergency,”
“Learn About” Query subject, result is relevant and timely news about that subject.
published the same day search was performed.
Fr Query is the name of a chain business; result is a Map

ida
y Secondary Maps a. Query is "dunkin", [in location Sunnyvale, CA], map result
17 ,9Dec Chain Business showing a nearby branch of business, but not the closest
2.1 em Result presents San Jose, CA location, 6.8 miles from the user.
6.2 be
54 r 1 one.
.6
by 6, 20
Lu 22 IP
Le from Query is a type of business, or a product or service; result is a. Query is “thai food” [in location Cambridge, fr o MA], result
iK o m
uo IP 2 2
Maps or Multiple map entry or an official website for a business of that type or is http://www.thesimilans.com, official 0 site Ku for local Thai
10 Type of Business 1 6, 2 Lei
Official Websites that offers that product/service. In the Maps case, business restaurant. ber by Lu
e m
must be nearby. b. Query is “thai restaurant”; De .result
c 4.6 is a nearby thai restaurant.
5
,
a .16 2
y
i d
Fr 172
When to Grade Somewhat Satisfying (SS)

Fr
ida
y
Some users may nd this result useful, but it s probably not what most searchers were looking for. It s often only indirectly
17 , Decrelated to the search need
2.1 em
or assumes an uncommon interpretation of the query. 6.2 be
54 r 1
.6
by 6, 20
IP Lu 22
m Le from
f ro When to grade Somewhat Satisfying iK
22 uo uo IP
0 K
6, 2 Lei
Rule 1 If query is And result is Description Examples
m ber by Lu
D ece 54.6
y , .2 Query is the name of a chain business or a type of
r ida 72.16
F 1 Chain Business/Type of Moderately Distant business; result is a Map showing a branch of a. Query is "starbucks", user is in San Jose, CA, result is a map result for
1
Business Maps Result business that is not nearby, but still accessible starbucks, 17 miles away in Fremont, CA.
(perhaps up to an hour’s drive away)
Query is a type of business or organization; result is

Official Website of a. Query is “vietnamese restaurant” [in Cupertino, CA]; result is https://
Type of Business/ the official website of an instance of this business
2 More Distant www.slanteddoor.com, the official site of a particular vietnamese
Organization or organization that is not nearby, but is still
Instance restaurant in San Francisco, CA, 50 miles from the user.
accessible.
a. Query is “zillow”, result is the video “Living Large in a Tiny Home” from
Query is the name of the entity; result is not their
Zillow’s YouTube channel.
official website, but is a site, page, video, or app
Company/Product/ Related Site/Video/ b. Query is “sonicare” (brand of electric toothbrush), result is website for
3 related to their business. For example, this might be
Named Entity App Oral-B (a competing brand of electric toothbrush).
a 3rd party site about that company or its products,
c. Query is “billy idol” (singer), result is wikipedia page for Generation X, a
or a site for a competing product or service.
band from the 1970s he was in before he became famous.
Query is the name of an event or named entity; a. Query is “super bowl news,” result is a news story “Patriots Come from.
Stale but Valid News result is a news story about an earlier event or early Behind to Defeat Falcons in Super Bowl LI.” The story is still accurate,
ida 4 Named Entity or Event
Fr
y, D Story news about the entity. The news story must still be but it describes something that happened in 2017, not in the most
17
2.1 ecem valid. recent or upcoming Super Bowl.
6.2 be
54 r 1
.6
by 6, 20 Query is the name of a general concept or event a. Query is “dogs”, result is wikipedia page for the dog breed Beagle.
Lu 22 IP
Le from Overly Specific (such as a TV show); result is about a specific b. Query is “suits” (a TV show that ran for 9 seasons), result o m is https://
5 iK
General fr o
uo IPQuery 2 2
Result instance of that concept or event (such as a www.peacocktv.com/watch-online/tv/suits/8003089882869075112/
0 Ku
1 6, 2 Lei
particular episode of that show). seasons/5, a page where viewers can stream ber by Luthe 5th season.
c e m 6
y , De .254.
ida .16
Fr 172

fi


Fr
ida
Query is the name of an app; result is that app on y
17 , Dec
the Google Play store website. Since users are 2.1 em
a. Query is “slickdeals”, result is https://play.google.com/store/apps/
6.2 be
54 r 1
6 App Name Google Play Result conducting their search on an Apple iOS device, we .6
developer?id=Slickdeals&hl=en. by 6, 20
IP can assume most of them do not want an android Lu 22
o m Le from
f r app as a result. iK
22 Kuo uo IP
2 0 i
r 16, u Le
e
b yL
e cem 4.6 b
y, D 16.25
FrWhen to Grade Not Satisfying (NS)
d a
i 72.
1
This result has nothing to do with the query, provides incorrect information, or fails the validation step, and should not be shown.
When to grade Not Satisfying
Result was flagged as Wrong Language, Content

Flagged During a. Query is “uniqlo”; user is in en-US; result is “https://www.uniqlo.com/jp/
1 Any Query Unavailable, or Inappropriate during validation
Validation Step ja/“ which is in Japanese and was flagged as Wrong Language.
step.
a. Query is “samsung tv”, result is web page for Samsung washing

Result that is not about the query topic. Note that in machine.
some cases the URL may appear to be about the b. Query is “obama age”, result gives the age of Joe Biden.
2 Any Query Off-Topic Result
query, but clicking through shows that the c. Query is “Messi goals”, (Messi is a soccer player) result is total goals by
destination page is not related. Barcelona (his team)
Fr d. Query is “target stores”, result is about an Ace Hardware store location.
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1 a. Query is “starbucks” [in San Francisco, CA], result is a Maps result for
.6
by 6, 20 Query indicates or assumes nearby location, result
LuLocal22 Unreasonably a Starbucks in San Diego, CA, 500 miles away.
3 Le frIntent Query is so geographically distant that it makes no sense m
IP
i K om I Distant Result b. Query is “airport” [in Boston, MA], result is official owebsite
fr o for
uo P to show it. 2 2
Heathrow Airport in London, UK. 0 Ku
1 6, 2 Lei
m ber by Lu
Query explicitly seeks result from a specific locale; e
ec 54.6result is https://
Explicitly Locale- a. Locale is en_US, query is “kit kat
y , Djapan,”
.2
4 Wrong Locale Result result pertains to a locale different from the one ida 72.16
Sensitive Query www.hersheys.com/kitkat/en_us/home.html
r
F 1
specified.

Fr
ida
a. Locale is en_US, query is “ticketmaster,”
y result is UK-specific
17 , Dec
Query does not mention a locale, but the user need Ticketmaster app 2 .16 mbe
.2 er
Implicitly Locale- implicitly requires results from the user's locale; b. Locale is en_IN, query is “do I need 5a4visa
.6 1to visit japan,” result is US
5 Wrong Locale Result by 6, 20
Sensitive IP Query result pertains to a locale different from the user's government page https://travel.state.gov/content/travel/en/
Lu 22
o m Le from
f r locale. iK
international-travel/International-Travel-Country-Information-Pages/
22 Kuo uo IP
2 0
16, u Le
i Japan.html
e
b yLr
e cem 4.6 b Query is asking for a specific answer; result is an
a y, D 16.25 Missing or Incorrect a. Query is “dmx real name,” result is an info card that says “dmx birth
Fr 1762
i d . Exact Answer Query info card that correctly identifies what the query is
Answer name: dmx” (which is incorrect).
asking, but then fails to give that answer.
Result is a blank page, a parked domain, a 404

a. Query is “bisq restaurant cambridge”, result is http://
Result Fails to Load / error, something unavailable in user’s country, or
7 Any Query www.bisqcambridge.com
Inaccessible anything else where the content has been removed
b. Query is “brokerbot”; result is http://brokerbot.com
or is inaccessible.
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

Grading Speci c Situations & Result Types Fr

ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
Ambiguous Queries (Multiple Interpretations) by 6, 20 28
IP Lu 22
m Le from
Locale Sensitivity 2 fro o iK
uo IP 30
6 , 202 ei Ku
1 L
English Resultsber by Lu in Non-English Locales 31
m
ece 54.6
Redirected
y , D .2 Pages 31
r ida 72.16
F 1
Apps 32
News 33
Maps 34
Web Video 35
Dictionary, Stocks, Weather, Knowledge / Answers , Sports 36
Web Results (also called Suggested Web Sites) 36
Web Images 36
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172
Speci c Situations & Result Types Page 27 of 58

fi

fi

Ambiguous Queries (Multiple Interpretations)

Fr
ida
While most queries express several di erent user intents, some queries are also ambiguous in what they refer to (e.g.,17 apple y, D could be a company or a
2.1 ecem
fruit). In this case you should still grade the result, using the following additional guidelines. 6.2 be
54 r 1
.6
by 6, 20
IP Lu 22
If you're not sure ro m whether there is a dominant interpretation, look at the web search results for the query. If most of the highly ranked
Le fromresults on the
iK
2 f o uo IP
rst page are, 20 for
2
i K one interpretation, then you should consider that to be the dominant interpretation.
u
16 Le
b er y Lu
m b
D ece 54.6 Multiple Interpretatons
y, .2
r ida 72.16
F 1
Type Description Examples
1. The query is "allegiant", result is the o cial website

for the airline. Grade as HS, since the dominant
Dominant Interpretation Exists.  Dominant Interpretation: If a result is for the dominant interpretation of the query is the airline.
When one interpretation is much more popular than the interpretation, you should grade using the normal 2. The query is "apple", result is a map result for the
others. guidelines. apple store near the user, but not the closest.
Grade as S, since the dominant interpretation of the
query is the technology company.
1. Query is “michael jordan”, result is IMDB page for

actor Michael B. Jordan. Grade as SS, since
dominant interpretation of query is for a di erent
person, the former NBA basketball player.
Fr
ida 2. Query is “american eagle”, result is home page of
y,
72 Dece
1Dominant web developer americaneagle.com. Grade as SS
.16 mb Interpretation Exists.  Secondary Interpretation: If a result would be relevant
(rather than HS), since the dominant interpretation of
When .25oneer interpretation is much more popular than the (HS/S/SS) for a secondary interpretation, you should
4.6 16
by , 20 the query is clothing retailer American Eagle
others (cont’d) grade it as “SS”.
Lu 22
Le from Out tters.
m
IP
iK r o
uo IP 3. Query is “golden retriever”, result 2 2 f is oa song titled
, 2 ei Ku
0
Golden Retriever. Grade as 1 SSL(rather than S/HS),
6
b er y Lu
since the the song isecnot em .6theb dominant interpretation
4
of the query. The ydog .breed
, D 2 5 is the dominant
ida 72.16
interpretationFfor 1this query.
r

fi
fi
fi
ffi
ff
ff

Type Description Examples

Fr
ida
y
17 , Dec
1. Query is “um athletics,”
2.1 em (location is Texas) result is
6.2 be
home page for the University
54 r 1 of Miami athletics
Sometimes there are several reasonable interpretations .6
by 6,than
2
program. Grade as S (rather Lu 022 HS), because “um
IP but none of them are dominant. In that case you should Le frto
m
fro o athletics” could equally well refer i K omthe University of
2 grade normally for all of them, except that results that uo IP
Multiple Interpretations,
, 202 ei Ku None Dominant.  Michigan or University of Maryland athletics
1 6 L would have been HS if there were only one (or one
When there er are Lu two or more interpretations of similar programs, among others.
emb 6 by dominant) interpretation should be graded S instead.  

popularity.
e c
, D 6.254
. 2. Query is “um athletics,” result is a photo gallery
a y  
id
Fr 172
.1 showing some athletic facilities under construction
That’s because if we can’t say which interpretation is
at the University of Michigan. Grade normally: it’s
one that nearly all users would want to see.
SS, because although it relates to the query, it’s not
what most users doing that search are looking for.
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

fi
Locale Sensitivity
Fr
ida
y
Locale Sensitivity 17 , Dec
2.1 em
6.2 be
54 r 1
Scenario Grade Examples .6
by 6, 20
IP Lu 22
m Le from
2 fro o iK
uo IP
Explicitly Locale-Sensitive.
6 , 202 ei Ku Query is “amazon france”. The user is in EN-GB locale.
1 L
Query explicitly m ber by Lu speci es that user is seeking results Results that do not pertain to the locale speci ed in the The result is https://amazon.co.uk. Grade as NS, since
from ecae locale
.6 that di ers from their current location.
query should be automatically graded as “NS”. the Amazon page in the UK is not what the user is
a y , D
6. 254
i d
Fr 172
. 1 searching for.
Implicitly Locale-Sensitive.
Query does not explicitly ask for results in a particular Any results from a di erent locale (even if they’re in the Query is “ticketmaster”; user is located in US. Result is
locale, but the user need is inherently locale- correct language) should be automatically graded as ticketmaster.co.uk. Grade as NS, since user did not
speci c (e.g., local law information, country-speci c “NS”. express any interest in UK events.
merchant sites, nearby real-world business).
Foreign results (as long as they’re in the correct

Query is “vaccine recommendations”. User’s locale is
language) should be SLIGHTLY penalized by assigning
en-US, and the result is https://www.nhs.uk. The NHS
a grade one level lower than you would normally give. 
Mildly Locale-Sensitive. is the UK's National Health Service that provides health
Query does not explicitly ask for results in a particular care to all British residents. Since di erent countries
• “HS” results should be downgraded to “S”
locale, but those in other locales may be somewhat less • “S” results should be downgraded to “SS”
provide di erent medical advice for their residents, the
useful. UK's advice would be less useful to a US resident than
• “SS” results should be downgraded to “NS”
advice from a US medical agency. The result should be

• “NS” results should remain as “NS”
SLIGHTLY penalized from S, down to SS.

Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
Not Locale-Sensitive.
.6
by 6, 20 Query is “tennis news.” User is in en-US; result is news
L 22 Grade result without regard to locale.
IP the
Results fromu any Le flocale
r would be equally useful for this from the BBC about the latest resultsofrom
m
i K om I r
query. u o P Wimbledon tennis tournament. 022 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

fi
fi
ff
ff
ff
fi

ff
fi
fi
English Results in Non-English Locales Fr
ida
y
17 , Dec
2.1 em
6.2 be
English is a widely-understood second language in many countries, and all our international graders are uent in it. For this 5reason,
4.6 r 16 rather than simply
marking an EnglishIPresult in a non-English locale as wrong language, graders should go ahead and grade the result, with the bfollowing y L , 202
u L 2 fr locale-speci c
ro m ei K om
considerations. 2 fYou o will need to use your own knowledge of the locale to decide which guideline to apply. uo IP
, 2 02 i Ku
16 Le
b er y Lu
m b
ece 54.6 English Results in Non English Locales
y , D . 2
ida .16
Fr 172 Scenario Grade
The user’s locale is one where most users understand English uently (i.e. ES-US)
Grade the result normally, the same way you would if it were in the locale language.
and would likely be interested in English-language results.
Grade the result one level lower than you would if it were in the locale language.  
The user’s locale is one where many users understand English uently (i.e. Western
⚠ Results that would have been NS should still be graded as NS
Europe) and would possibly be interested in English-language results.
The user’s locale is one where relatively few users understand English uently and
Grade the result as NS.
would be unlikely to be interested in English-language results.
Redirected Pages
Fr
id
If1 athe
y, D result displayed URL gets redirected to a di erent URL, then you should grade the page you re redirected to as if that were the result.
72 e
.16 cemb
.25 er
4.6 16
by , 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

fi

ff
fl
fl
fl
fl
 
fi
Apps
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
When a user clicks these results it takes them to app store (usually Apple .6
by 6, 20
IP the app if present on the device. Lu 22
app store) or opens
rom
Le from
iK
f uo IP
2 0 22 Kuo
i
r 16, u Le
e
b yL
e cem 4.6 b
a y, D 16.25
i d . App Rating Guidance
Fr 172
Rule Additional Details
Rule 1 under HS refers to cases where the query is the name of a well-known app —
a service that is best known as an app.
Examples: Instagram, Spotify, and Candy Crush
⚠ A well-known app is not the same thing as a well-known company!
Rule 3 under HS refers to cases where the query is a business and the result is an
app “regularly used to interact with that business.” Meaning, the app is a common
way that customers or clients perform the ordinary tasks they need to do business
with that company.
1. If the query is the name of a bank, then the app should allow the user to
  perform mobile banking tasks.
⚠ Just because a company has an app does not mean that it’s regularly used 2. If the query is the name of a restaurant chain, then the app should allow the
user to order food at that restaurant.
to interact with that business. For example, the query “dell” refers to the name
3. If the query is the name of an airline, then the app should allow the user to
of a computer company. But their app “Dell@Retail 2019” is described as “a
Fr make reservations, choose their seat assignment, and check ight status.
ida chance for our global retail partners to immerse themselves in the design,
y, D
17 performance, 4. If the query is the name of a retail chain, then the app should allow the user
2.1 ecem and vision driving Dell’s innovation.” This app is NOT used
6.2 be to browse and purchase items sold by that chain.
regularly
54 r 1 by Dell’s customers and should NOT be graded HS.
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

fi

fl
News Fr
ida
y
17 , Dec
2.1 em
6.2 be
News articles usually have the word News prepended to them. The are speci c web 54 r 1
.6
by 6, 20
results that link to Inews
P websites. Lu 22
m Le from
2 fro o iK
uo IP
• The relevance 6 , 202 ei Kugrade for a news article depends in part on the amount of time
1 uL
between ber by L the date the search was done and the date of the article.
m
ce 4.6 A news item result with the recency below the title
e
d a y, D 16.25
i 72. search date is shown in the result preview itself.
Fr• 1The
• Keep in mind validity ags (Inappropriate, Wrong language, and Content Unavailable).
Grading time Sensitive News Articles
Type Scenario Grade
Timely Article: up to 3 months older than the search date Either S or SS if it's about the query topic.
Current Event
May never be graded better than SS even if
Stale Article: more than 3 months older than the search date
it's about the query topic.
Time sensitivity does not impact the relevance grade of the results for these types of queries. Examples of historical events are
Historical Events
Notre Dame re, Harry and Meghan wedding, Sandy Hook shooting, Pope Benedict resigns, etc.
Fr
ida
y
⚠ You might see articles with dates in the future! For these rare occurrences, grade it the same way as a timely article,
17 , Dec
2.1 em
6.2 be as long as the date is not more than 3 months newer than the search date.
54 r 1
.6
by 6, 20
Lu 22 ⚠ News items are never HS. Why? one news organization – even one reporter – may actually write several stories IP
Le from o m
iK fr
uo IP
about the same event. Maybe one person wants to get an overview of an event while another wants the latest updates. , 2022 i Kuo
16 Le
Or one person only likes stories from Fox News while another prefers MSNBC. For these reasons, we can't say that m aber y Lu
b
given news story is one that almost everyone wants to see. So it is mistake to rate a news result as Highly Satisfying.
Dece 54.6
y , .2
r ida 72.16
F 1

 
fi

fi
fl
fi
Maps
Fr
ida
y,
The relevance of Maps results depends in part on the distance from the user. You should check to see if the info card 1has
72 Dedistance displayed. If not,
.16 cemb
this result cannot be judged. .25 er
4.6 16
by , 20
I P Lu 22
m Le from
f ro iK
2 2 uo uo IP
20 K i
r 16, u Le
e
b yL
e cem 4.6 b
d a y, D 16.25
i .
Fr 172
Maps Results
Grading Maps
Type Scenario Grade

Maps result is correct and is the closest one. HS
Maps result is correct and near the user, but is not the closest one S
Business
Maps result is correct, and is still accessible to the user but is not close. SS
Maps result is correct but is too far away. NS

Fr
ida
17 , DePoint of Interest (e.g., cities, parks,
y
2.1 cem landmarks, monuments) Maps result is correct. HS
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
fro
1. Grade onLewhat iK m is visible: Only use what is in the title and description to grade. Do not grade NS just because clicking the result takes
fr o you
o m
uo IP 2 2
nowhere or the wrong place. 0 Ku
1 6, 2 Lei
m ber by Lu
c e 6
2. Distant results are not always NS. For example: y , De .254.
ida .16
Fr 172

fi

• People looking for expensive, rarely purchased items (cars, furniture, etc.) are generally willing to travel longer distances to nd the right one than
people looking for inexpensive, common items (e.g., a cup of co ee). So if the query is Lexus dealer, a result 30 Fr miles away might be S (or even
ida
HS if it's the closest match), while if the query is donuts, it would be NS. y
1 , De 72
.16 cemb
.25 er
• People living in sparsely populated rural areas are generally willing to travel longer distances than people in cities. If the 4.query
6 b 16, 2 restaurants is
y L 02
issued in Wilsall, IP MT (population 237), then a result 39 miles away in Bozeman (population 39,860) might be S. But if the same 2f
u L query rom were issued
r o m e i
f K uo IP
in New 2York
, 022 i KuCity,
o a result 36 miles away in Greenwich, CT would be NS
16 Le
b er y Lu
3. Keep m b
D ece 54.6in mind Intent and Distance! For some queries, users are looking for a Maps result. For other queries, they aren't. If a Maps result is shown
y, 16.2
r ida 7for
. a non-Maps intent query, then grade it as NS. Use the distance to guide you. If a Maps result is very far away, that s often a sign that the user
F 1 2
was not looking for a map.
• Query is "prime video" and result description is: "prime time video, 2511 springs rd ne, hickory, nc 28601- distance: 529 mi
• Query is "Lakers" and result description is: "great lakes brewing company, 2516 market ave, cleveland, oh 44113 - distance: 2,165 miles
Web Video
• If a query speci cally refers to a particular video (e.g., lemonade o cial video,
stepanov elements of programming lecture ), the desired result should be
graded as Highly Satisfying regardless of its popularity.
• For other results, and for more general queries where many di erent video results
could satisfy the user's need (e.g., guitar lesson ), then popularity may factor into
Fr
idayour decision; you may want to grade a video with millions of views higher than a
y, D
17 similar
2.1 ecem one with only a handful.
6.2 be
54 r 1
.6 6
y L , 202
• When bdeciding
u L 2 fr on your grade, think about whether video results are what user is IP
o m
looking foreiwhen Ku m IPtyping the query. r o
2 f uo
o 0 2
2 iK
r 16, u Le
⚠ You are not required to watch the entire video to arrive at a rating
e
e
mb 6 by
L
e c .
a y , D 6.254
id .1
Fr 172

fi

fi
ff
ff
ffi
fi
Dictionary, Stocks, Weather, Knowledge / Answers , Sports
Fr
ida
Grade these cards based on what is visible. Thee grader cannot click on them but a user is provided self contained snippets y
17 , Dec of information and which
2.1 em
can often be interacted with to learn more (e.g. the Stock card opens up to show historic prince graphs) 6.2 be
54 r 1
.6
by 6, 20
• Dictionary: Is the u L 22 fr it must be the
IPuser seeking a de nition or a concept? If the card precisely answers the need, this is Highly Satisfying. In all Lcases
m ei K om
2 fro o uo IP
correct interpretation
2
, 20 i K
u for that word
16 Le
b er y Lu
m b
• Stocks:
D ece 54.6 check for correct stock symbol and presence of price.
y, .2
r ida 72.16
F 1
• Weather: the result s location should match the location speci ed in the query (e.g. weather boston ), or the user s location if location is not
mentioned in query.
• Answers: If the query is an explicit question, see HS7. Grade on what is visible.
Web Results (also called Suggested Web Sites)
Please click on the thumbnail and grade the destination page(after redirects).
Web Images
A group of web images should be graded as a single result. Check to see if all the images have the
Ffollowing properties:
rid
ay,
1 De
1.72.1Imagec displays correct subject. The image must actually show the subject of the query. For
6.2 embe
54 r 1 if the query is dodecahedron, the image must actually show that geometric gure and
example,
.6
by 6, 20
not some 22
Lu other
Le from one. Missing images (or ones that do not load) do not have this property. o m
IP
iK r
uo IP
0 2 2 f uo
2. Subject clearly shown. All images in the set must clearly show the subject of the query. The 2 ei K
Query: eMenr 16, inu Black
L
b y L
subject should not be blocked, out of focus, too far away, or otherwise di cult to see clearly. m b
ece 54.6
y , D .2
r ida 72.16
3. Subject is focus of image. In cases where the image includes multiple people or objects, it F 1
should be clear who or what is the subject of the query. (For example, if the query is Joe Biden,
Query: David Beckham

fi

fi

fi

ffi
fi
it s ne to have people in the background of a picture of President Biden giving a speech, but it s not ne to have a picture of Presidents Biden and
Macron shaking hands.) Fr
ida
y
17 , Dec
4. Image shows representative version of subject. For example, if the query is the name of a currently popular actor, .the 2 16 eimage
m should show that
.25 ber
person as they look today (or how their character looks in a currently popular movie), not how they looked many years ago. 4.6 If
by 6,the
1
20 query is the
name of a famous I P person from the past who is no longer alive, the image should show them as they were best known. For
L u L 22 fr if the query is
example,
m ei K om
2 fro o u I
02 i Ku a picture should show him during the time he was U.S. president, not 20 years later when he was near the end of ohis Plife.
Richard 2Nixon,
r 16, u Le
e
b by L
5. No e cemduplicates.
4.6 The images in the set should all be di erent.
y , D . 2 5
ida .16
Fr 172
If ALL the images have all of the above properties, grade the result Highly Satisfying. Otherwise, downgrade the results as shown in the table below.:
If… … Then
All images exhibit all properties Grade as Highly Satisfying
All but 1 or 2 images in the set exhibit all properties Grade as Satisfying
Up to half of the images exhibit all properties Grade as Somewhat Satisfying
Property #1 violated for any image Grade as Not Satisfying
Examples:
• Query is David Beckham, result is set shown above. It has all the desired properties, so you would grade as Highly Satisfying.
Fr
ida
y, D
•17Query
2.1 ecem is dodacahedron (a geometric shape); result set is shown on the left below. Neither the second image nor the last image in this set are
6.2 be
dodecahedrons,
54 r 1
.6 so they violate property #1. Therefore you would grade this Not Satisfying.
by 6, 20
Lu 22 IP
Le from m
• Query is ta i Kyu brodesser-akner (an author); result set is on the right below. Two of the images in the set are problematic; one shows 2 f uopart of a
r o
o IP 0 2
poster for an event featuring the author, and another shows her with another person, both partly cut o . Neither of these violates 2 iK
r 16, u Leproperty #1
e yL
because both attempt to represent the author and not something else that would confuse or mislead the user, like a picture c e mb ofb a di erent author. But
De 54.6
y , .2
r ida 72.16
F 1

fi
fi
ff
ff
fi
ff
ff
each violates at least one of properties 2-4. Overall you would grade this Satisfying because all but two images have all the desired properties.
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
IP Lu 22
o m Le from
f r iK
22 Kuo uo IP
2 0 i
r 16, u Le
e
b yL
e cem 4.6 b
d a y, D 16.25
i .
Fr 172
Web image results for query “taffy brodesser-akner” Web image results for query “dodecahedron”
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

fi
 
Common Grading Mistakes  Fr
ida
y
17 , Dec
2.1 em
6.2 be
Failing to Use Web Search 54 r 1
.6 39
by 6, 20
P Lu 22
Failing to Visit Destination
rom
I Page Le from
iK
40
f uo IP
22 Kuo
Ignoring Time 2andi Place
0 40
r 16, u Le
e
b by L
Ignoring cem Conceptual Distance 40
, D e
2 5 4.6
y .
ida 72.16 Relevance Grading Principles
FrIgnoring
1
41
Failing to Use Web Search 3. Falsely Assuming Dominant Interpretation. If you have heard of a
result, you may assume that it's the dominant interpretation. But this
1. Misunderstanding Query Meaning. The query may be a common is not always true.
word that you think you know. But the web search may show that
• Example: Query is "u of m scholarships," result is a page about
the primary meaning is something entirely di erent.
scholarships at the University of Michigan. A grader who knew
• Example: Query is "canada goose"; result is the wikipedia page nothing about the subject might conclude that this is a great
about that kind of bird. If you had not heard of the Canada Goose result, and rate it Highly Satisfying. But looking at the web results
clothing brand, you might assume that the bird page is what shows that the query has no dominant intent. It might be referring
almost all users would want to see. But by looking at the web to the University of Minnesota, or the University of Manitoba, or
search results, you can tell that this is not the case. many other things. Therefore the grade cannot be HS.
2. Misunderstanding Dominant Interpretation. This is a slight

Fr
ida variation of the previous error. Based on your personal experience, ⚠ Do not use web search ranking to determine grade!
y
17 , Dec
2.1youemmay know that there is more than one interpretation of the
6.2 be The only purpose of looking at the web search (Google and Bing) results is to
54 r 1
query,.6 but
by 6, 2 you may not realize that one is dominant. make sure you understand the possible meaning(s) of the query, and which
Lu 022 IP
Le from meaning is dominant.
om
• Example:i KQuery
uo IP is "jaguar"; result is the home page for the car
0 2
r
2 f uo
You should never use the ranking on the search result page 2 iK
company. If you believe the animal is the dominant interpretation, r 16, utoLedecide your
e L
grade. In other words, you should never think (for example) mb 6 by "Google says this
you would downgrade the car company result. But by doing the c e
De "Bing 4 .
is the #1 result, so it must be Highly Satisfying,"a y , or 6 .25 puts this at the
web search, you can see that the car company is actually the id
bottom of the page, so it must not be that good." .1
Fr 172 Once you understand the
dominant interpretation, accounting for all but one of the results query, only these guidelines and your judgment should determine the grade.
on the rst page of both Google and Bing results.
Common Grading Mistakes Page 39 of 58

fi

ff

1. Mismatched Location. Graders usually notice when the user is in

one location and the result is a Map
Fr to a very distant location. But
Failing to Visit Destination Page id
they frequently miss the case where1 athe
y, result is a web result for a
72 Dece
very distant location. .16 mb
Another class of mistakes can occur when the grader fails to visit the .25 er
1 4.6
by 6, 20
destination page ofIPa web/news result, and in particular, if they try to L 22
m • Example: User is in Virginia (state in Easternu U.S.),
Le fromquery is
grade a web/news fro o result based only on the URL and/or snippet. iK
2 2
20 ei Ku o IP
"harold's kitchen menu." Result is home page for uHarold's Kitchen
6 ,
1 uL
1. Missing ber by LError Condition. The URL and/or snippet may make this and Bar. At rst glance, this looks like a Highly Satisfying result.
m
ce 4.6
e
D .25 like a perfect result ‒ perhaps the home page of a company.
y, look
It's a restaurant with a matching name, and the page shows their
a 6
i d
Fr 17But 2.1 menu. But a closer look shows that this restaurant is actually in
if you actually clicked on it, you'd discover that the page does
not load, or redirects to some entirely unrelated page. Richmond, British Columbia, Canada ‒ nearly 3000 miles (5000
km) away from the user. It is extremely unlikely that this was the
• Example: Query vallco shopping center, result is result the user was looking for (especially since there is a di erent
www.vallcoshoppingcenter.com. If you click no the result, you ll be restaurant named Harold's Kitchen close to the user's location).
taken to an advertising page that has nothing to do with the
shopping center (which is out of business). 2. Mismatched Date. Graders may notice the date of a news story,
but forget to notice the date of the search. Or they may not notice
2. Incorrect Page Owner Assumption. The URL may be a perfect an implicit date in the content of a web result.
match for the name of a company or product you're familiar with.
But if you visited the destination page, you'd see that it's actually for • Example: Query dated 2022 is "presidential election results";
an entirely di erent company with a similar name. result is a page showing the results of the 2016 U.S. presidential
election. The user was almost certainly looking for the most recent
• Example: Query "american eagle," result is presidential election results, not one from six years earlier.
www.americaneagle.com. Since American Eagle is a well-known
ida clothing brand, you assume the page is the home page of that
Fr
y
17 , Dcompany.
2.1 ecem
But it isn't. Clicking on the result would have shown that Ignoring Conceptual Distance
6it's
.25 the
4.6 r 16home page of a web design company, which is not what
be
,2
mostbysearchers
Lu 022 are looking for. Some mistakes involve the conceptual distance between Ithe
P result and
Le from
iK what the user was looking for. rom
uo IP 2f o
6 , 202 ei Ku
1 L
1. Too Speci c or Too General. Graders sometimes er y Lu incorrectly give a
Ignoring Time and Place m bb
ece 54.6
result a high grade without realizing athat
y 16.2 is too speci c or too
, D it
i d .
general. Fr 172
Many grading mistakes happen when the grader doesn't pay attention
to the time or place of the query and/or result.

fi
fi
ff

fi
ff
• Example: Query is "dog," result is wikipedia page about the welsh
corgi, a particular breed of dog. This is too speci c.
Ignoring Relevance Grading Principles
Fr
ida
y
17 , Dec
• Example: Query is "new england patriots news," result is home 2.1 em
1. Matching Words Instead of Meaning.6.Graders
25 ber
4.6 16 sometimes forget
page for a regional sports news network that covers many by , 20
IP teams in New England, not just the New England the principle "Think about meaning, not just Lmatching
u L 22 fr words."
di erent sportso m ei K m o
r
2 f uo is too general. Just because the query words appear in the result udoes
o IP
not mean
Patriots.
202This
iK
r 16, u Le the result is a good one, and just because the query words are
e
b yL
2. Wrong e cem 4.6 b Level of Web Page. Pages on a given web site often form a missing does not mean the result is a bad one.
y, D 16.25
i d a hierarchy,
. with a home page for the site, subpages for di erent
Fr 172 • Example: Query is "far alone," result is a page containing the
topics, sub-sub-pages, and so on. A common mistake is not to
inspirational quote "If you want to go quickly, go alone. If you want
notice that a page is too high or too low in the hierarchy, compared
to go far, go together." The result contains both query words, but
to what the user is looking for.
they match only incidentally. It's clear that this is not what the user
• Example: Query is "us passport information"; result is was looking for, and in fact the web search results show that "Far
www.state.gov. This page is too high in the hierarchy of this web Alone" is the name of a song.
site. It is about everything the U.S. State Department does
2. Rating News Results Highly Satisfying. When a news event
(diplomatic relations, trade policies, etc.), not just passports.
happens, it is often reported by many di erent news organizations,
• Example: Query is "us passport information"; result is a page from whether it's local TV stations, newspapers, or major news networks.
the U.S. State Department about what to do if your passport is Furthermore, one news organization ‒ even one reporter ‒ may
lost or stolen. This page is too low in the hierarchy of the site. The actually write several stories about the same event. Maybe one
user never said anything about their passport being lost or stolen person wants to get an overview of an event while another wants
‒ in fact, we don't even know if the user already has a passport. the latest updates. Or one person only likes stories from Fox News
Fr while another prefers MSNBC. For these reasons, we can't say that
3.
ida Ignoring Degrees of Separation. Graders often ignore the principle
y a given news story is one that almost everyone wants to see. So it is
17 , Dec
2.1of degrees of separation. A result that's associated with the thing
6.2 embe mistake to rate a news result as Highly Satisfying.
5 r
the4user
.6 16,is looking for is not the same as the thing the user is
by 2
Lu 022 • Example: Query is brittney greiner sentencing and Iresult
P is a
looking for. Le fro m
iK m f r o
uo IP timely news article about the event on the news 2 022 iwebsite
Ku
o
• Example: Query is "chez panisse," result is Yelp's page of reviews 6 , e
theguardian.com. Although this result is about
b er y Lu the topic, it should
1 L
for that restaurant. This is a very useful result, but it is not Highly not be Highly Satisfying because it is a
m 6b
ecenews
. result.
, D 254 y .
Satisfying, because it is one degree of separation from what the r ida 72.16
F 1
user was looking for. 3. Ignoring Basic De nitions of Grading Scale. A common mistake is
to ignore the basic de nitions of each grade and only look at the

ff
fi
fi
ff

fi
ff
individual rules. The rules are meant to illustrate the de nitions in
di erent situations, not to replace them. If you're faced with a Fr
ida
grading situation where you don't see a rule that applies, just go y
17 , Dec
2.1 em
back to the de nitions: Is this a result most users would want to 6.2 be
54 r 1
.6
see? Etc. by 6, 20
IP Lu 22
o m Le from
r
f o iK
• Example: 22Query u is el pais (name of several newspapers, including uo IP
, 2 0 iK
6
1 uL e
one berin y Cali,
L Colombia and one in Madrid, Spain); user is in
c em b
e 54.6
a y , D Colombia
6.2 but result is for a more popular one in Madrid,
i d . 1
Fr 172 elpais.com. There s no rule about matching similarly-named results
in di erent countries, and the guidance about locale-sensitivity
doesn t exactly address this example. It s clear that the Spain
result is not what most Colombian users are looking for, but it
might be useful to some. By de nition, that means it s Slightly
Satisfying.
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

ff
ff
fi
fi
fi
Examples: Satisfaction Rating Fr
ida
y
17 , Dec
2.1 em
Note: defn in Rule column means that the grade follows from the grading scale de nitions. 6.2 be
54 r 1
.6
by 6, 20
IP Lu 22
o m Le from
f r iK
22 Kuo uo IP
Highly Satisfying 2 0
16, u Le
i
e
b yLr
e cem 4.6 b
d a y, D 16.25 Query Result(s) Rating Explanation
i .
Fr 172
Instagram is best known as an app, so result is what
instagram Instagram app HS
almost all users would want to see. (Rule HS1)
Almost all users searching for a celebrity would want
olivia rodrigo O cial website for the pop star, oliviarodrigo.com HS
to see that person's o cial web site. (HS4)
Wikipedia is a high quality source of information

olivia rodrigo Wikipedia entry for Olivia Rodrigo HS
about the artist (HS5)
Almost all users searching for a company or

microsoft Their o cial website, microsoft.com HS organization would want to see its o cial web site.
(HS4)
Wikipedia is a highly satisfying result for any named

jane austen Wikipedia page about the early 1800s author HS
entity. (HS5)
Fr
ida Since it's both a company and an app, both of these
facebook
y
17 , Dec facebook.com, Facebook app HS are "o cial" results that most users would want to
2.1 em
6.2 be
54 r 1 see. (HS4 & HS1)
.6
by 6, 20
Lu 22
Le from The Premier League is the top englishomsoccer IP league.
iK fr o
uo IP Note that this is a result most users 2 would want to see
202 ei Ku
top english soccer league Home page of the Premier League, premierleague.com HS ,
even though it doesn't use the er ywords
1 u L "English" or
6
L
b
“Soccer." (HS4) e c e m 6b
4 .
D 5
i d ay, .16.2
Fr 172
Satisfaction Rating Examples Page 43 of 58

ffi
ffi
ffi
ffi

ffi
fi
Query Result(s) Rating Explanation
Fr
ida
y, D
The result (knowledge
17 card
2.1 ecem
with the answer)
how many stomachs does a
HS immediately gives the user6.2 all
54 erthe
b information they
cow have .6 16,
asked for. (HS6) by 2
L 02
IP u L 2 fr
r o m ei K om
f uo IP
22 Kuo
6, 2 0
ei Almost all users searching for a business or service
beat thebebomb r 1 Lu L o cial website : https://beatthebomb.com HS
m by would want to see its o cial web site. (HS4)
ece 54.6
y, D .2 Result is the o cial Roland Garros (French Open)
ida 72.16
r
F 1 YouTube channel. Although there is no speci c rule for
french open highlights https://www.youtube.com/channel/UCF3K1Jf8hjFW8qliei8fQ3A HS
this case, it clearly satis es the de nition of Highly
Satisfying.
mountain mike's pizza

Result provides authoritative map information to the
HS
[user is in Berkeley, California] closest location of a chain business. (HS3)
The info card immediately gives the user all

how tall is gwen stefani HS
information they asked for. (HS6)
This info card provides relevant and accurate

iphone 11 HS information, even though it is not the o cial site for
Fr the product. (HS5)
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22
Le from All of the images satisfy the properties described
IP in
eric stonestreet iK HS o
fr o
m
uo IP the section on how to grade Web20Image
2 Ku results. (HS8)
2
i
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

ffi
ffi
ffi
fi
fi
ffi
fi
Fr
ida
y,
The o cial page for 1the
72 Dmovie.
e
.16 cemb Contains streaming
saw HS
links and descriptions about.25 ethe
4.6 r 16 movie. (HS4)
by , 20
IP Lu 22
o m Le from
f r iK
22 Kuo uo IP
2 0 i
16, u Le
e r
b by L The wikipedia page for a named entity is Highly
Wonder cem 4Woman
.6 HS
y
e
, D 6.25 Satisfying. (HS5)
id a .1
Fr 172
The wikipedia page for a named entity is Highly

gilmore girls HS
Satisfying. (HS5)
saw
A knowledge card for a named entity is Highly

HS
Satisfying. (HS5)
Satisfying Examples
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6 The query is asking an implicit question (how to change
by 6, 20
instagram.com Lu 22change Instagram password. This web page has the authoritativeIP
Le from O cial instructions on how to change instagram password S m
pass iK
uo IP answer, but the user has to click on the result
2 f uo to visit the
r o
2
page in order to see the answer. (S7)16, 2 Lei K
0
m ber by Lu
c e 6
y , De .254.
ida .16
Fr 172

ffi
ffi


F
The Google/Bing results from
rid Step 1 show that there is no
ay,
D
dominant meaning of the query.
1 7 2.1 eceThe user might have
u m sociology (location: 6.2 mbe
wanted University of Montana, or 54 University
r of Miami,
texas)
Home page for University of Michigan sociology department S .6 16,
IP among others(and the user is located b y L far02away from both
u
2
2
m
fro o states). So we can't say that almost all Lusers ei K fromwould have
2 uo IP
6 , 202 ei Ku wanted this result.
1 L
m ber by Lu
ece 54.6 The page contains the answer, but the user has to do some
howy many
, D .2 stomachs does
ida 72.16
F a cow
Wikipedia page about cows. S extra work to nd it -- clicking on the result, reading and
r
1 have
scrolling through it. (S7)
We don’t really know what user wanted. Maybe it’s a video

warriors vs lakers https://www.youtube.com/watch?v=p478C35sgzA (highlight of recent game highlights, but could also be a schedule of
S
(searched on 06/01/2021) video of most recent game on o cial NBA channel). upcoming games between these teams, or an info card with
the latest score.
We do not know exactly which taxes the user has in mind
and there are other websites (including an o cial one from
web page containing an Indiana tax calculator, from a nancial
indiana tax calculator S the state government) that o er similar information, so we
services company
can't say that almost all users would have wanted this result.
(S2)
The link is to a highly rated QR reader app, however there
qr reader An app to read QR codes S are other highly rated QR reader apps and we do not know if
the result would entirely meet the user's search needs. (S2)
premier league news
A BBC News article, “Why Premier League teams are ocking
Fr[searched on 29 July S The news article is timely and about the query topic.
ida back to Asia” dated 28 July 2022.
2022]
1 ,D
y
72 e
.16 cemb
.25 er
4.6 16 Since there are several possible results for popular BTS
by , 20 O cial video of a recent song by the band BTS, https://
bts Lu 22 S songs, and the user didn’t express a preference IP for a
Le from www.youtube.com/watch?v=WMweEpGlu_U m
iK
uo IP particular song, this is at best Satisfying.22(S3)
o
fr o
0 Ku
1 6, 2 Lei
ber by Lu
User could be searching for aDsuite ec 54.6in the Plaza Hotel, but
e m
O cial website for the Plaza, a hotel in New York City that has , 2
plaza suite new york S "Plaza Suite" is also a famous
ida 72.play,
y 16. often performed on
rooms and suites r
F 1
Broadway in New York. There is no dominant meaning.

ffi
ffi
fi
ff
ffi
ffi
fl
fi
F
rid
There are several GPA calculators
ay and though this site is
17 , Dec
credible, users might want to2.1see alternatives. It is
6.2 embe
gpa calculator https://gpacalculator.net S
impossible to conclude that almost 54 all
.6 16,users would wish to
r
by 2
IP see this result. (S2) Lu 022
f
m Le r
f r o i K om I
22 Kuo uo P
2 0 i The result is from a trusted website and has a description
of
16, u Le
b e r L the experience and user submitted reviews. This is a good
em .6 by
beat ethe4bomb
c
y, D 16.25
reviews page for the experience S example of a result that is "one step away" -- it isn't the
a
i d
Fr 172
. o cial site for the service, but it gives the user helpful
information about that service. (S6)
Query is a product (a movie) and result is a site where user

Wonder Woman S
can buy/rent the movie. (S5)
Query is a product (a movie) and result is a site where user

gilmore girls S
can buy/rent the movie. (S5)
Frsaw Query is a product (a movie) and result is a site where user

S
ida
y can buy/rent the movie. (S5)
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

ffi
Somewhat Satisfying Examples ida
Fr
y
17 , Dec
2.1 em
6.2 be
Query Result(s) Rating Explanation 54 r 1
.6
by 6, 20
IP The Google/Bing results from Step 1 showLthat u L 22the dominant
r o m ei K from
f
22 Kuo IMDB page about the director of the 2013 movie meaning of the query is a di erent person, an actor uo Ifrom
P the
steve mcqueen 2 0 i SS
e r 16, u Le 12 Years a Slave 1960s & 70s with the same name. So this result is not what most
b yL
e cem 4.6 b users are looking for.
d a y, D 16.25
i .
Fr 172
This result, for a restaurant in San Francisco, is 43 miles from the
vietnamese restaurant [user is
SS user’s location in San Jose, and there are dozens of closer
in San Jose, California]
Vietnamese restaurants. (SS1)
Probably not what most users were looking for. (If they had
camden county college Home page for library at the college SS
wanted the library, they would have mentioned it in the query.)
Though the result is from Farmers Insurance, it has information

farmers insurance
farmers hawaii SS about a di erent state, so is not likely what most users would
[user is in Texas]
want to see.
A very popular interview with BTS. and tv show host, but not very
bts
2018 video of interview with the band SS relevant given that it is several years old, and several newer
[searched in 2022]
interviews are available.
Frcao
Irish website about applying to undergraduate
SS
There is a grocery chain in Florida called CAO, so it's unlikely that
[user
ida
y is in Florida] programs in Ireland. the user had the Irish website in mind.
17 , Dec
2.1 em
6.2 be
54 r 1
.6
Query is about a German track and eld star, so the most
by 6, 20 satisfying results will be about her competitions, herPathletic
Lu 22 I
alica schmidtLei K from I https://hotsportsgirls.com/alica-schmidt/ SS achievements, etc. In contrast, this result is solely f romabout her
uo P 2 uo
physical appearance, which will be of interest , 2 02toiK only some
16 u Le
searchers. e
mb y
r L
b
Dece 54.6
y , .2
r ida 72.16
F 1

ff
ff
fi


The dominant interpretation is Fthe
rid singer. Furthermore, the dog
ay,
breed is correctly spelled as two17words
2.1 ecem(“pit bull”), while the
D
Pitbull SS
singer is spelled as one. So these dog 6.2 pictures
b
54 er 1 are not likely to be
.6 6
of interest to most searchers. by ,
L 022
2
IP uL
f r o m ei K from
22 Kuo uo IP
2 0 i
r 16, u Le Most users who do this search are looking for the Apple CEO, not
Tim Cook e
b yL SS
e cem 4.6 b the historian and author.
d a y, D 16.25
i .
Fr 172
eeting meaning SS De nition of a related word but not the word the user asked for
Fr
ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

fl
fi
Not Satisfying Examples
Fr
ida
y
Query Result(s) Rating Explanation 172 , Dece
.16 mb
.25 er
4.6 16
by , 20
IP Lu 22
o m User may either be looking for Le public
fr
i K om I
nearest subway 22 [user
r
f o is in
0 iK
u NS transportation or the restaurant.uoIn either
case, a
P
Seattle, WA] 6 , 2 e
1
ber by Lu
L result 710 miles away is not satisfying. (NS3)
m
D ece 54.6
y , .2
r ida 72.16
F 1
Despite the similar name, this result is for a
harold's kitchen menu [user is restaurant 3000 miles away from the user. (And
Home page for Harold's Kitchen & Bar in British Columbia, Canada NS
in Virginia, US] there is a di erent Harold's Kitchen near the
user.) (NS3)
how many weeks has it been Despite matching some words in the query, this
https://www.answers.com/Q/
since march 25th
NS result is for a totally di erent year and does not
How_many_weeks_has_it_been_since_April_27_2009
[query issued in April 2021] give the user any useful information. (NS6)
Poorly written website and talks about resetting

instagram.com change pass Low-quality website describing instructions NS password when it has been forgotten (which is a
di erent meaning of the query)
James Watt did not invent the steam engine,

which already existed by 1712, before he was
Frwhat year did james watt invent born. He did make some important
ida NS
the y steam engine improvements to it in the 1760s and 1770s. This
17 , Dec
2.1 em
6.2 be result contains only incorrect or misleading
54 r 1
.6
by 6, 20
information. (NS6)
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 ei K
tour de france stage 1 (queried Result is for a previous year’s 16, u LTour de France,
NBC video of stage 18 of 2021 Tour de France. NS r
e yL
on 29 July 2022) and is not even the estage cem 4.6 bthe user asked for.
b
, D 5
i d ay .16.2
Fr 172

ff
ff
ff

Other Aspects Related to Search Satisfaction Grading Fr

ida
y
17 , Dec
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Overall Preference
IP Rating (OPR) Lu 22
Le from
m
2 fro o iK
uo IP
6 , 202 ei Ku
1 L
m ber by Lu
ece .6
, D 6.254grading tasks you will be presented with two sets of results presented
Inaysome
id 72.1
Frside
1 by side for the same query, as shown on the right
After providing satisfaction ratings for every result, you will be asked to choose
which side you prefer. This is called the Overall Preference Rating (OPR).
The rating scale is About the Same, Slightly Better, Better and Much Better.
OPR Criteria:
Use the following criteria to decide on the OPR:
1. Prefer the side whose results have higher satisfaction grades.
2. If there are multiple results, prefer the side where results with higher
satisfaction are ranked higher.
3. If there are multiple results, prefer the side with a more varied result set. This
might be a variety of result types (maps, apps, web pages, etc.), satisfying a
F variety of meanings of the query.
rid
ay
17 ,Note
4. D that the side with more results is not necessarily better.
2.1 ecem
6.2 be
5. If you 54 rer
.6 16,having trouble deciding which side is better, choose About the Same.
by 2
Lu 022 IP
Le from o m
iK r
How much these IP
uo criteria a ect OPR also depend on the position of the result. For example, 0 2 2 f uo
2 iK
if the satisfaction rating of the results in position 1 are di erent, that should have a bigger r 16, u Le
e L
impact on OPR than if the satisfaction rating of results in position 4 are di erent. c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172
Overall Preference Rating

Examples Page 51 of 58
ff

ff
ff
Writing Comments Fr
ida
y
17 , Dec
2.1 em
6.2 be
You might be asked to leave a comment (written in English) for why you chose the OPR. These are very helpful to the clients 54 ofr 1the grading task. It
.6
by 6, 20
helps understand the IP reasoning behind the rating for complex grading tasks and especially in locales the clients doesn t understand. Lu 22
Le from
ro m iK
2 f o uo IP
2 2
0 iK u
,
16 Le
b er y Lu
m b
D ece 54.6
r
y,
ida 72.16
.2 The query intent is Yahoo News and is most likely
F 1
to visit the main page of headlines of the queried
website. The 1st and 2nd results are the same on
the both sides. The rest of the results are similar
I came to the conclusion that the left side on both sides showing some speci c pages from
o ers more suitable results and therefore sports, entertainment and weather categories on
should be rated as better Yahoo News website and there is a little better
news among them (R5) on the right than the left
which is a breaking news from domestic news
category. Thus the right side is slightly better due
to better relevance and freshness.
Fr
ida
y
17 , Dec Poor Comment Excellent Comment
2.1 em
6.2 be
54 r 1
.6
by 6, 20
Lu 22 IP
Le from o m
The comment i on Ku the
o IP left can be improved by providing reasons why the left is more suitable . 2
r
2 f uo
2 0 iK
r 16, u Le
For the comment on the right, the writer states presumed search need and then goes on to describe how the results help meet e yL
c e mb 6 bthat and ultimately
D e 5 4 .
why they chose one over the other. d ay, .16.2
i
Fr 172
Examples Page 52 of 58
ff

fi
OPR & Comment Examples  ida
Fr
y
17 , Dec
2.1 em
6.2 be
54 r 1
Query 1: tdecu addresses four (the main app, the mortgage app, .6
by 6the
, 2 web page, and
IP Lu 022
o m the Twitter feed). So the right has a slightly more diverse Le from result set.
f r iK
Location: Richwood, 0 22 Kuo TX uo IP
, 2 i However, the user gave no indication that they were interested in the
16 Le
b er y Lu Twitter feed, so this is a very unlikely intent.
m b
D ece 54.6 LEFT RIGHT
a y , 6.2
id .1 Since we don t know whether more people are interested in the map or
Fr 172Official TDECU Digital Banking App Official TDECU Digital Banking App
the o cial site, the two sides are About the Same.
TDECU Mortgage Simplified App TDECU Mortgage Simplified App
Maps info card with directions to TDECU

TDECU.org official website
branch, 3 miles away Query 2: diesel
Maps Info Card with directions to a Location: Cambridge, MA

TDECU.org "About Us" page
TDECU branch 4 miles away
LEFT RIGHT
@TDEC twitter page
Diesel Online Store (shop.diesel.com/en/ Diesel Online Store (shop.diesel.com/en/
Slightly Slightly Much homepage) homepage)
Much Better Better About the Same Better
Better Better Better
DIESEL（ディーゼル）公式オンライン Diesel Fuel - Wikipedia
ストア(diesel.co.jp) (en.wikipedia.org/wiki/Diesel_fuel)
Diesel Fuel - Wikipedia Diesel [Maps result], 339 Newbury St.,

OPR Explanation: The query refers to a credit union (essentially, a (en.wikipedia.org/wiki/Diesel_fuel) Boston (2 miles)
Fbank)
rid with two branches near the user. We can assume the user wants
ay,
to
17 either
D do a bank transaction, go to the bank, or get information Much Better Better
Slightly
About the Same
Slightly
Better
Much
2.1 ecem Better Better Better
about 6.2 the
5 er bank.
b
4.6
1
by 6, 20
L 22 IP
The o cial uapp,
Le frthe
i K om I
o cial website, and the map results for the r o m
uo P 2 f uo
nearest locations are all Highly Satisfying. The map results appear on OPR Explanation: The query could refer to a clothing
0 2
iK
16, u Le store or a kind of
2
the left but not the right, while the o cial website appears on the right fuel.
e r
mb 6 by
L
e c e .
but not the left. y , D 6.254
a
id 72.1
• Fr 1on
Two out of three results are the same both sides, so they aren t
The left side addresses three search needs (it satis es people looking that di erent.
for the main app, the mortgage app, and the map) while the right
ffi
ffi
ff
ffi
ffi
fi
• The left side has a wrong language result, which is Not Satisfying to • The rst two results are the same on both sides.
users. F
• Both result sets have three types of rsearch
ida
y, D results.
17
• The right side ranks the diesel fuel result higher, showing both likely 2.1 ecem
6.2 be
interpretations near the top. • The third result on the left is only vaguely 5related
4.6 r 16 to the Apollo space
by , 20
IP program. It seems unlikely that someone searching Lu 22 for apollo
o m Le from
• The right side 0 22 K
r
f has
uo more diversity of result types (web pages and project would nd an obscure artist s ambient music i K useful
uo IP in
, 2 i
maps, instead 16 Lof
er y Lu
e only web pages). satisfying their search need.
b
m b
D ece 54.6
Since
y,
ida 72.16
.the
2 are multiple reasons to prefer the right side, that side should • The third result on the right is not at all related to the Apollo space
r
F 1
be more than Slightly Better. But since the lists aren t that di erent, it s program; it has something to do with a project of the Apollo Theater.
not Much Better. So we choose Better. Based on the web results, it s extremely unlikely that this was the
user s intended interpretation of the query.
Since only the last result is di erent, and the last result on the left is
Query 3: apollo project
less bad than the one on the right, we conclude that the left side is
Location: Cincinnati, OH on Feb. 13, 2020. Slightly Better.
LEFT RIGHT
Apollo Space Program wikipedia article Apollo Space Program wikipedia article
(en.wikipedia.org/wiki/Apollo_program) (en.wikipedia.org/wiki/Apollo_program)
Project Apollo documentary [Movie] Project Apollo documentary [Movie]
FrProject Apollo — Moonlight Richards 50 Apollo Global Video Project: Les Twins
ida
songs
y, D to the moon, an Apollo 11 space of Sarcelles by Apollo Theater, Harlem
17 mission
2.1 ecemtribute [Apple Music result] [YouTube video]
6.2 be
54 r 1
.6
by 6, 20 Slightly Slightly Much
Much Better Lu LBetter 22 About the Same Better IP
f
ei K omr Better Better Better m
r o
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
OPR Explanation: The query refers to the space program from the a y , D 6.254
id .1
Fr 172
1960s that rst put a human on the moon.
OPR Examples Page 54 of 58

fi
fi
fi
ff
ff
Query: best actor winner • Result #3 on the right tells us about another recent best actor
award ̶ the Golden Globes, rather than
Fr the Oscars ̶ which had the
Location: Bellevue, WA on Feb. 13, 2020. ida
same winner, Joaquin Phoenix. Even though
17 , Dec we assume the user was
y
2.1 em
looking for the Oscar winner, they might 6also.25 bebe interested in other
LEFT RIGHT 4.6 r 16
awards won by the same actor for the same role. by , 20
2
Academy Awards Best Actor IP and Best L
u L 2 fr
m Winners Joaquin Phoenix — Academy Award for ei K om
Supporting Actor r —
o
2f o Best Actor — Winner [Info card] o IPis better
Since all of these observations suggest that the right uside
202 i Ku
(filmsite.org/bestactor2.html)
16, u Le Academy Awards Best Actor and Best
Andy Serkis e r
b bfor y LBest Actor [YouTube than the left, you would conclude that the right side is Much Better
cem 4video
. 6 from 2011]
Supporting Actor — Winners
e
y, D 16.25 (filmsite.org/bestactor2.html) than the left.
d a .
i 72 Best Actors Who Won Oscars for
Fr The1 Joaquin Phoenix: Best Actor, Motion
Their First Movie (www.ranker.com/list/
Picture, Drama: 2020 Golden Globes
actors-who-won-oscars-for-their-first-
(YouTube video)
movie/ranker-film) Query: anthony ramos
Much Better Better

Slightly
About the Same
Slightly
Better
Much Location: Fairfax, VA on April 17, 2021.
LEFT RIGHT
Anthony Ramos wikipedia page Anthony Ramos official site

OPR Explanation: The query very likely refers to the winner of the Official video for Ramos' 2021 song Official video for Ramos' 2021 song
Academy Award (aka Oscar ) in the best actor category. Since the "Lose My Mind" "Lose My Mind"
query was on Feb. 13, 2020, we assume the user wanted the most Official video for Ramos' 2021 song
NBC News article from February 2021
"Blessings"
recent award winner at the time, announced at the ceremony on Official video for Ramos' 2021 song
Anthony Ramos instagram page
February 8, 2020. “Say Less"
Slightly Slightly Much
• Result #1 on the left (same as #2 on right) contains the answer, but Better Better Better
Fr requires visiting the page and scrolling all the way to the bottom to
ida
y, D
17 nd
2.1 ecit. Result #1 on the right gives us the answer right away, without
6.2 embe
even54having r1 to click on it. OPR Explanation: The query refers to an actor and singer who
.6
by 6, 20
Lu 22 appeared in the original cast of the musical Hamilton. IP
• Result #2Lonei K fthe
rom left is a YouTube video from a non-authoritative m
fro o
uo IP 2 2
source (a random fan), and it s very outdated ̶ from 2011. 0 i Ku
• Results L1, R1, and R4 all all Highly Satisfying. 1 6, 2 LAll
e the rest of the
b er y Lu
results on both sides are Satisfying. cem .6 b
• Result #3 on the left is related to best actor winners, but doesn t e
, D 6.254
a y
actually contain the answer the user is looking for. id .1
• Fr 172 providing more di erent
The set on the right is more diverse,
types of results.

fi
ff
Since the only di erences favor the right side, it is Better. Query: tina turner movie
Fr
ida
Location: Kansas City, MO on 2021-08-17.y, D
17
2.1 ecem
Query: dana 6.2 be
54 r 1
LEFT .6
by 6, 20 RIGHT
IP Lu 22
Location: Hampton,
rom
VA on 2021-08-17. 1985 movie "Mad Max: Beyond
Le from
i
f Web page forK2021
uo Idocumentary "Tina"
2 0 22 Kuo Thunderdome" (which co-starred Tina
P
16, u Le
i on HBO
e r LEFT RIGHT Turner)
b yL
e cem 4.6 b
y, D 16.25 Home page for Dana Inc. 1993 movie "What's Love Got to Do 1993 movie "What's Love Got to Do
d a
i 72. (www.dana.com), a company that With It," about the life of Tina Turner With It," about the life of Tina Turner
Fr 1Dana (Indonesian digital wallet) app
makes drivetrain parts for passenger
vehicles 1985 movie "Mad Max: Beyond
Web page for 2021 documentary "Tina"
Thunderdome" (which co-starred Tina
Video of Israeli singer Dana International on HBO
Turner)
Home page for Nigerian airline Dana Air performing the winning song at the
1998 Eurovision contest
Video of 2021 song "Dana Dana" by Wikipedia page for South Korean singer
Now United Dana Slightly Slightly Much

OPR Explanation: Both sides have the same results, but they are
ranked di erently. Since the search was done in 2021, it s most likely
OPR Explanation: The query can refer to many di erent things or that the new 2021 documentary about Tina Turner ( Tina ) is what the
people, and the web search results make it clear that none of them is a user was looking for. Since the only di erence is the ranking, and the
dominant interpretation. Furthermore, these results all seem to be only right side ranking is clearly better than the left side (moving the best
FSomewhat
rid Satisfying, since it isn t likely that most users in the United result into position #1), it s Better.
ay,
States
17 Dec were searching for (say) an Indonesian app or an Israeli Singer
2.1 em
from 6.2thebe1990s. Therefore the two sides are About the Same.
5 r
4.6 1
by 6, 20
Lu 22 IP
Le from o m
iK r
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
e .
a y , D 6.254
id .1
Fr 172

ff
ff
ff
ff
Query: hannah waddingham would have needed some additional content that added diversity, such
as thelink to o cial page. Fr
Location: Dickinson, TX on 2021-09-22. ida
y
17 , Dec
2.1 em
6.2 be
LEFT RIGHT 54 r 1
.6
Query: audra mcdonald by 6, 20
A news article on her winning IP an Emmy Lu 22
min the tv series The IMDB page for the actor Hannah Le from
award for her character r o iK
2 f uo Waddingham uo IP
202Lasso
Ted iK Location: Bergen, NJ on 2021-09-22.
r 16, u Le A different news article on her wining an
A website e y L the Emmy 2021
b listing
e cem 4.6 b winners Emmy award for her character in the tv
LEFT RIGHT
d a y, D 16.25 series Ted Lasso
i .
Fr 172Much Better Better
Slightly
About the Same
Slightly
Better
Much A Knowledge Card A Knowledge Card
Better Better Better describing the singer/actor including describing the singer/actor including
links to her official site and Twitter links to her official site and Twitter
handle handle
A web video of a lesser well known song
Official website
"My Man's Gone Now" from 2007
OPR Explanation: Both sides have a fresh and relevant news article
A web video of another song "Rainbow
but the second result on the left doesn't add any additional value. On Twitter handle
High"
the right, we have an excellent ranking, the rst result is a professional
page about the actor and her experience and the second a fresh news Slightly Slightly Much
article. Better Better Better
Query: monster hunter stories 2 OPR Explanation: Both sides have the brief Knowledge card describing
the person (with links to her o cial website and twitter feed). The left
Location: Miami, FL on 2021-08-10. side also has web videos for two of her songs, while the right side also
Fr has her o cial website and Twitter feedResults R2 and R3 are more
ida LEFT RIGHT
y, D
17 Wikipedia
valuable than L2 and L3, but the lack of any videos makes the right
2.1 ecem entry for the video game Wikipedia link to Monster Hunter Stories
6.2 Monster
b Hunter Stories 2: Wings of Ruin side only Slightly Better.
54 er 1
.6 6, Slightly Slightly Much
Much Betterby L 2Better
02 About the Same Better
u L 2 fr Better Better Better IP
ei K om r o m
uo IP
0 2 2 f uo
2 iK
r 16, u Le
e L
c e mb 6 by
OPR Explanation: The user speci cally asked for Monster Hunter e
, D 6.254
.
a y
Stories 2 . The left side has a more general result (it s about the entire id .1
Fr 172
video game series), while the right is about the exact thing the user
asked about, so the right is Better. To be Much Better, the right side

ffi

ffi
 
ffi
fi
fi
OPR Explanation: The user is looking for the news site Hu ngton Post.
O cial website,app, and Twitter feed Fare all Highly Satisfying. The UK
Query: sunrise rid
ay
site is Somewhat Satisfying. Left is better
17 , Ddue to more satisfying
2.1 ecem
Location: West Melbourne, FL on 2021-09-01 results. 6.2 be
5 r4.6 1
by 6, 20
IP Lu 22
LEFT m RIGHT Le from
fro 2 uo
iK
uo IP
Weather Info card 02for
6 Lei West Melbourne A website selling the domain name
, 2 K
(with 1
esunrise/sunset
r Lu times) http://www.sunrise.am
c e mb 6 by Weather Info card for West Melbourne
App e 54link. for sunrise/sunset times
a y , Dstore
6. 2 (with sunrise/sunset times)
id 72.1
Fr Knowledge
1 Info card about the topic Knowledge Info card about the topic
Sunrise Sunrise
OPR Explanation: Both have same third result. Both have the same
Highly Satisfying info card, but it s ranked better on the left. Of the
remaining results, the one on the left might be useful, while the one on
the right is Not Satisfying. Both of these di erences favor the left side,
so it is Better.
Query: huffington post
FLocation:
rid Paxtonia, PA 2021-09-22.
ay,
17 Dec
2.1 em
6.2 be
54 r 1 LEFT RIGHT
.6
by 6, 20
Lu 2website
Official 2 Official UK website IP
Le from o m
iK r
uo IP
0 2 2 f uo
Twitter handle Huffington Post News App 2 iK
r 16, u Le
e L
e mb 6 by
Slightly Slightly Much e c .
Much Better Better
Better
About the Same
Better
Better
Better a y , D 6.254
id .1
Fr 172

ffi
ff
ffi

CS Exam GuidelinesV1.2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CS Exam GuidelinesV1.2

Uploaded by

Copyright:

Available Formats

Search

What is Search Need and Relevance Page 4 of 58

• The location of the user. We want to return results appropriate for

What is Search Need and Relevance Page 5 of 58

What is Search Need and Relevance Page 6 of 58

A generalization of o cial site that includes not just o cial

A business (or organization) that consists of many locations • Starbucks

What is Search Need and Relevance Page 7 of 58

Result Validation Page 8 of 58

Result Validation Page 9 of 58

Result Validation Page 10 of 58

Highly Satisfying Satisfying Somewhat Satisfying Not Satisfying

Result Validation Page 11 of 58

Query : Beyoncé Query: Rolling Stone Lemonade album review

Beyoncé's o cial website. The review of the album.

Her Lemonade album on iTunes. The album.

Result Validation Page 12 of 58

satisfying. For example:

Result Validation Page 13 of 58

Overview of Result Types ida

Result Validation Page 14 of 58

Result Validation Page 15 of 58

Groups of images clustered together. Usually the user doesn t interact

Result Validation Page 16 of 58

Answers and Knowledge Flights

Query: Where is Olympics 2024 Query: macron

Query: Bubonic plague Query: haiku

Result Validation Page 17 of 58

Result Validation Page 18 of 58

How to Assign Ratings Fr

When to grade Highly Satisfying

Rule If query is And result is Description Examples

How to Assign Ratings Page 19 of 58

Rule If query is And result is Description Examples

a. Query is “taylor swift” (singer), result is https://en.wikipedia.org/wiki/

How to Assign Ratings Page 20 of 58

Rule If query is And result is Description Examples

Fr a. Query is “nelson mandela,” result is the following set of images:

How to Assign Ratings Page 21 of 58

When to Grade Satisfying (S)

a. Query is “currency converter,” result is “My Currency

Query is the name of a performer (singer, actor, etc.) or

How to Assign Ratings Page 22 of 58

Rule If query is And result is Description Examples

a. Query is “ebola,” result is New York Times news story “Ebola

Fr Query is the name of a chain business; result is a Map

How to Assign Ratings Page 23 of 58

When to Grade Somewhat Satisfying (SS)

Query is a type of business or organization; result is

How to Assign Ratings Page 24 of 58

Rule If query is And result is Description Examples

When to grade Not Satisfying

Rule If query is And result is Description Examples

Result was flagged as Wrong Language, Content

a. Query is “samsung tv”, result is web page for Samsung washing

How to Assign Ratings Page 25 of 58

Rule If query is And result is Description Examples

Result is a blank page, a parked domain, a 404

How to Assign Ratings Page 26 of 58

Grading Speci c Situations & Result Types Fr

Speci c Situations & Result Types Page 27 of 58

Ambiguous Queries (Multiple Interpretations)

1. The query is "allegiant", result is the o cial website

emb 6 by dominant) interpretation should be graded S instead.  