Page Recommendations v2.

1 Guidelines
Last updated Decembe r 2, 202 0

Task Overview
In this task, we will ask you to compare two web pages side by side. The left page will be the clicked
page, which we assume the user to have visited, and the right page will be a recommended page. Your
task is to determine if a user would be interested in visiting the second web page given they visited the
clicked page and additional search context.

Search Context
Query:

Each hit will provide a search query the user issued that led them to clicking on the clicked page. The
search query will help inform you on the likely intent of the user, which can in turn inform you on how
useful the recommended page may be to the user. Keep in mind the query should not become the focus
of the hit and the goal is not to determine whether the recommended page is useful for the query
alone. Consider the query and the clicked page together as clues to what the user may be interested in
recommended pages.

If you are not sure how to interpret the query and/or content of the clicked page, feel free to do side
searches or use other resources to get an idea before making your final judgment.

Location:

Each hit will also have a location of the user when they issued the search query and clicked on the
clicked page. This can be helpful since some search intents are only useful in certain locations, and as a
result, some recommendations are as well. Not all user intents involve location, so use your best
judgment to determine if a hypothetical user would be interested in the recommended page, or if the
location would make it useless.

Judging Guidelines
Not Useful
Recommended pages can be considered Not Useful if they fall into one of the following areas:

Situation Positive Example Negative Example

Unrelated to the A recommended page about the best
clicked page waterparks in the state and a clicked
page about the income taxes are
completely unrelated.
For a query like {franklin county
auditor} where the clicked page is the
homepage for the Franklin Country
auditor and the recommended page
is the homepage for the Franklin
County recorder would be considered
Somewhat Useful – Focus Mismatch.
While the auditor isn’t directly in the
office of the auditor, these two pages
are still related at a topical level
enough to be useful.
 Has content that is A query like {apply for
heavily dependent unemployment} with a location of
on location, and the Miami, Florida and a clicked page
location that is being the government website for
completely different submitting unemployment
from the one given applications for Florida could have
by the search context multiple outcomes. If the
recommended page is an article
about step by step instructions on
applying for unemployment in
Alabama, it has a location mismatch.
However, if the article is specific to
Florida, or applying on some national
level, it does not.
Please check the box that says
Location mismatch for any hit that
has this issue.
Is the same as the The two pages have the same URL, or
clicked page with very minor differences in the
URL. A recommendation with
https://www.amazon.com/SYLVANIA
-Dimmable-Equivalent-Google-
Assistant/dp/B088G2B48D/ref=sr_1_
17?m=ATXPNO3SPGCJ4
as the clicked URL and as the
https://www.amazon.com/SYLVANIA
-Dimmable-Equivalent-Google-
Assistant/dp/B088G2B48D as the
recommended URL would be
considered duplicates, even with
some differences in the URL.
If the recommended page falls into one of these categories, please mark it as Not Useful.
Somewhat Useful - Repetitive
Recommended pages can be considered to be Somewhat Useful – Repetitive if they fall into one or
more of the following areas:
Situation Positive Example
Has a high For the query {Benadryl side effects}
intent/content where the clicked page is a
overlap with the Drugs.com page listing the side
clicked page effects, and the recommended page
is a WebMD article listing the side
effects, those may be marked
Somewhat Useful - Repetitive
because the content will be very
A query like {us embassy jobs} from
the location of Washington D.C., USA
with the clicked page being a
government page for embassy jobs in
Thailand and a recommended page
for embassy jobs in Uganda does not
have location issues and is highly
relevant. U.S. embassies are mostly
in other countries so someone
looking for a job in one is likely
looking for one in another country.
The two pages have very similar
products but not exactly the same. A
recommendation with
https://www.lowes.com/pd/Top-
Choice-2-in-x-4-in-x-14-ft-Fir-Lumber-
Common-1-5-in-x-3-5-in-x-14-ft-
Actual/4082900 as the clicked URL
and https://www.lowes.com/pd/Top-
Choice-2-in-x-4-in-x-16-ft-Fir-Lumber-
Common-1-5-in-x-3-5-in-x-16-ft-
Actual/4082902 as the
recommended URL are not
duplicates, as these are different
items and may have different
purposes, although they look very
similar.
Negative Example
A query {sugar cookies recipe} where
the clicked page is a recipe for baking
sugar cookies one way and the
recommended page is a different
recipe for baking sugar cookies would
be considered Highly Useful. Even
though the recipes may contain some
similar steps and ingredients, this is
 similar between the two pages. It is true for almost all recipes. It is likely
unlikely someone would be this user is looking to explore
interested in reading both unless the different recipes before picking one,
recommended page contained some and the differences may be
other supplemental information. significant. See the last row in this
section for further explanation.
If the recommended page falls into one of these categories, please mark it as Somewhat Useful -
Repetitive.

Somewhat Useful – Focus Mismatch

Recommended pages can be considered Somewhat Useful – Focus Mismatch if they fall into one of the
following areas:

Situation Positive Example Negative Example

Too broad: related to For a query like {Minecraft A query like {republic online bill pay}
the higher-level topic download} where the clicked page is where the clicked page is the home
of the clicked page the homepage for downloading page for Republic car parking bill
but does not Minecraft and the recommended payment and the recommended page
consider one or more page is the download page for is the bill payment page for Comcast
key intents League of Legends, we would internet would be considered Not
consider this Somewhat Useful – Useful. Although both sites are bill
Focus Mismatch. A Highly Useful pay sites, they are for totally
recommendation would ideally key different and unrelated services, and
into the Minecraft part of the search there is no reason to connect the two
intent, but in this case, the sites in any other way. In this case,
recommendation is not useless the connecting topic is too broad for
because it was still connected to the there to be a useful connection.
topicality of the clicked page
(recommending popular games).
Too specific: highly A query like {Burlington free press} A query like {cheap flights} for a user
useful for only some with a clicked page of the Burlington in a different location where the
search intents of Free Press newspaper homepage and clicked page is a travel portal for
users who issue this a recommended page for obituaries finding cheap flights and the
query and clicked from the Burlington Free Press would recommended page is a page listing
this page be Somewhat Useful – Focus cheap flights to Grand Rapids, this is
Mismatch. While the main intent can considered to be too specific of an
be inferred to be reading news in the intent to even be considered
Burlington area, some people may Somewhat Useful. The slice of people
find the relevant obituaries page who issue this query who are
useful, since it is usually published in
interested in flights to Grand Rapids
the area newspaper. is likely too small to be of general
use, so this would be Not Useful.
If the recommended page falls into one of these categories, please mark it as Somewhat Useful – Focus
Mismatch.

In general, Somewhat Useful pages are pages that are almost Highly Useful but miss on a key aspect of
the search intent based on the search context and clicked page. They differ from Not Useful pages
because they are still acceptable to show a user without causing dissatisfaction, and some slice of users
may find them useful.
Highly Useful
Recommended pages are pages that are very relevant matches for the intent of the clicked page and the
query context and meet one or more of the following:
Situation Positive Example
Provides a different A query {what does the color purple
perspective on the mean} where the clicked page is an
user intent article discussing the meanings of the
color purple and the recommended
page is a different article explaining
some different meanings can be
Highly Useful, since it provides some
additional perspective that might be
useful for the user.
Encourages the user For a query like {IRA taxes} where the
to continue their clicked page is the IRS page on IRA
exploration of a topic taxes and the recommended page is
in a coherent manner an article from a financial services
company on how to minimize taxes
on your IRA distributions is Highly
Useful. Someone who is interested in
learning more about IRA taxes is likely
to be very interested in how to
minimize taxes for these kinds of
investments as well.
Reasonable next step A query like {best floating water mats}
in the most likely where the clicked page is an article
task a user had in reviewing the top 10 best floating
mind following a click water mats to buy, and a
on the clicked page recommended page that is a shopping
site page for buying floating water
mats would be Highly Useful. The user
is likely interested in possibly
purchasing a floating water mat, and
the shopping site is the logical next
step for someone who is researching
the best one to buy.
Negative Example
Pages where there is only one
generally accepted perspective or
viewpoint would not qualify for this
category. For example, a query {how
far around is the earth} where the
clicked page is an article from
Space.com on the measurements
and the recommended page is a
page on the flat earth theory saying
there is not circumference would not
be considered to give a different
perspective. The second page is
obviously wrong as there is not
much room for disagreement, so it
would be marked Not Useful.
For example, take a query like {tom
cruise starring movies} and a clicked
page listing the movies featuring
Tom Cruise. A recommended page
detailing the life of Tom Cruise’s
cousin would be Not Useful because
although he is related to Tom Cruise,
he has nothing to do with his movies
and does not expand on the intent in
a logical manner. It is too big a jump
to suggest his extended relatives
when the intent is around the actor
and his movies.
For a query like {install lap pool}
where the clicked page is an article
about the costs and requirements
for installing lap pool and the
recommended page is the best open
water swimming races in the area
would get a label of Not Useful.
While it is conceivable that someone
who swims might be interested in
both, it’s too far of a leap to assume
someone who is interested in
installing a pool would also be
 interested in nearby outdoor
swimming races.
Provides another For a query like {honey mustard A query {des moines area
option to complete recipes} where the clicked page is an community college} where the
the task the user has Allrecipes page for honey mustard, clicked page is the homepage for Des
in mind and the recommended page is a Moines Community College and the
honey mustard recipe from a cooking recommended page is for the
blog, the rating should be Highly University of Iowa. Both are higher
Useful because someone who is education in Iowa, but they are not a
interested in making honey mustard full match on intent because
will likely find another option on how community college is very different
to go about it useful. from the 4-year University of Iowa. It
also does not fall under Somewhat
Useful – Repetitive because one site
is not easily accessible from another,
and there isn’t a high content
overlap between the two pages. This
is a better match for Somewhat
Useful - too broad.
Highly Useful pages for location-sensitive search contexts always need to be have content that is
relevant to the location listed in the hit.

If the recommended page falls into one of these categories, please mark it as Highly Useful.

Can’t Judge
Some hits may be impossible to judge due to one of the following reasons:

Situation Positive Example Negative Example

Recommended or Page contains content that is illegal,
clicked page contains like drugs or piracy
inappropriate/illegal Page contains pornographic or adult
content content
Page contains malware or otherwise
malicious content
You cannot view the Pages with 404 or other loading
content, even after errors
opening it in a new Pages with content behind privacy
tab or incognito warnings from your browser
window Blockers due to being served from
another country
You can’t understand One or more of the pages are in a
the content of one or foreign language
both pages Spam or excessive advertisements
make it so you cannot access the
content
If you see a hit that falls into these categories, please mark it as Can’t Judge and report it to the HitApp
using the “Report a technical issue” button.
Pages that are informational about
potentially illegal/inappropriate
information, like a legal help page on
the legality of growing marijuana or
an interview with a former adult
actress
Pages behind soft paywalls or logins
where you can exit the popup easily
Pages where the understanding of
the content might require some extra
thought but are still written in plain
English
Using Tags
Once you select your answer to the core HitApp question from the above guidelines, you may also see
some tags appear that ask you to provide a more specific label for your choice:

If you see these appear, please select the tag that matches the reason you chose for your response. If no
tag appears or matches your decision, please leave a comment instead.