Sign OutRating HomeRater HubManage Account

arrow1483230evitanora@gmail.com
0

You are looking at the STAGING site!

This page is deprecated. Try the new UI here.

 User Guide
 Program Introduction
 Using the EWOQ System
 Task Instructions and Concepts
 >Common Task Instructions
 >Reporting a Problem
 Rating Mobile Ads
 General Information
 >Work Requirements
 >System Information
 >Glossary
 >FAQ
 >Contact Ads Eval

Common Task Instructions

This section contains the task instructions for the main rating tasks. For
each of the main tasks you will be able to see the full task instructions as
well as a screenshot of the task itself. All task instructions and task
screenshots are available in this document. You will be able to search for
terms or tasks by using CTRL+F (on Windows) or Command+F (on Mac) to
search for text in this document.

Task 1 Search Ads

Search Ad Instructions
 
During your time as an Ad Quality Rater, you should expect to encounter several
different task types in the ad rating system. Instructions are always accessible at the
top of all tasks. The most common task type is the search ad evaluation and
instructions for this task type are included below. These instructions are identical to
those found on Search Ads Evaluation tasks.
Search Ads Evaluation: General Guidelines
Version: 2015-08-20
 
Search ad rating involves interpreting a user query. A user query is the set of
keywords that a user enters into the Google search engine. When rating a search
ad, perform the following steps:
 1. Review the Google search results page,
try to understand the user query, and
form an opinion about what the user
hopes to accomplish by using a search
engine.
2. Use the evaluation criteria found in the
following instructions to analyze an
advertisement and the advertising
experience the user will have if he or she
clicks on the ad.
 

User Intent
An understanding of the user intent is necessary to accurately rate a search ad. The
user intent is what the user hopes to accomplish by using the Google search engine.
Note that users use the search engine to look for a variety things, and there are
many user intents.
 
Some queries are very easy to understand, others are more difficult, and some may
seem impossible to understand.  Regardless of its meaning, you must research the
query and form an opinion about the user intent.  We strongly advise you to review
the Google search results page to determine user intent.  In order to objectively
determine how promising or unpromising an advertiser offering is for a particular
user query, it is important to form an opinion about the user intent before beginning
an analysis of the advertisement.
 

Queries with Multiple Meanings

If a query has multiple meanings, please consider that all meanings can be placed
on a spectrum between plausible meanings and highly implausible meanings. When
analyzing an advertiser offering, consider what meaning the advertiser uses and
where it falls along this spectrum.  This will help you determine the appropriate
search ad rating.
Plausible Meanings
If a query has several plausible meanings, it is important to consider them all.  If an
advertiser assumes a particular meaning in an ad or on a landing page, and it is a
reasonable meaning, assume that is the meaning that the user intended.
 
Refer to the following example to better understand plausible meanings:
 
User query:  [ java ]
 
This query could refer to an
island, coffee, or a computer
language.  With no additional
information available, it is
impossible to say which meaning
the user intended.  Ads that
respond to any of these meanings
are acceptable since all three
 meanings are reasonably
plausible.
 

Possible but Unlikely Meanings

If an ad or landing page assumes a meaning that is possible but not very likely, this a
secondary interpretation.  An ad or landing page that addresses only a secondary
interpretation of the query is given a lower rating than an ad that addresses a
plausible meaning.  Use the Secondary Interpretation of Query flag in this case.
Generally, rate an ad or landing page that responds to a secondary interpretation
negatively.
 
Refer to the following example to better understand possible but unlikely meanings:
 
User query: [ paris ]
 
While there are a number of cities
called Paris, unless there is some
reason to think that one of the
smaller cities is meant, a query
mentioning Paris is probably
referring to Paris, France. So, an
ad for hotels in Paris, Texas
instead of Paris, France is
probably incorrectly
comprehending the user intent.
Even if the ad is otherwise a good
one, rate it on the negative side of
the scale and use the Secondary
Interpretation of Query flag.
Implausible Meanings
If an ad or landing page assumes a meaning that is completely implausible, treat it
as completely wrong and choose a very negative rating. Do not use the Secondary
Interpretation of Query flag if the meaning is clearly implausible.
 
Refer to the following example to better understand implausible meanings:
 
User query: [ paris ]
 
The query probably refers to the
city of Paris, France.  If an
advertiser interprets the meaning
to be plaster of paris, it is almost
certainly not addressing the user
query. Use a very negative rating.
 

Misspelled Queries
Users often misspell queries. When evaluating a query, if it is clear what the user
means, and the misspelled version of the query has no meaning, ignore the
misspelling.
 
Analysis is more difficult if the query appears to be a misspelling, but the misspelled
version has a unique meaning.  First consider the query as the user entered it, and
then consider if it may be misspelled.  If advertisers respond to misspellings, ratings
may need to be adjusted.
 
Refer to the following example to better understand misspelled queries:
 
User query: [ goodnight moom ]
 
There is a famous children’s book
called Goodnight Moon. It is very
possible that the user means to
type [ goodnight moon ] but types
[ goodnight moom ] instead.
However, there is actually a novel
titled Goodnight Moom. While the
novel is quite obscure, it might be
what the user wants.
 

Advertiser Responds to Actual Spelling in Query

If  the advertiser assumes that the query is correct as it stands (in the example
above, assumes the user meant [ goodnight moom ]), treat the advertiser’s query
interpretation as acceptable.  You would then need to decide separately how
promising the ad or landing page are.
 

Advertiser Responds to a Plausible Correction of Spelling in Query

If the advertiser assumes that the query is misspelled and addresses a corrected
version of the query (in the example above, assumes that the user mistyped and
meant [goodnight moon]), judge for yourself whether this was a good assumption. If
you think it was a reasonable assumption, the ad and landing page are treated as if
this were the user intent.  Don’t modify your scores to account for the spelling
correction, and don’t use the Secondary Interpretation of Query flag.
 

Advertiser Responds to a Possible but Unlikely Correction of Spelling in Query

If the advertiser assumes a corrected spelling that you think is possible but not very
likely, this is a secondary interpretation.  An ad or landing page that addresses
only a secondary interpretation of the query is given a lower rating than if it had
responded to a likely or plausible meaning.  The Secondary Interpretation of Query
flag must be used.  An ad or landing page that responds to a secondary
interpretation is generally rated negatively.
 

Implausible Spelling Correction

If the advertiser’s interpretation of the query is based on a completely unreasonable
assumption, treat it as completely wrong and give it a very negative score.  The
Secondary Interpretation of Query flag is not used in this case.
 
Continuing with the previous [ goodnight moom ] example, both query interpretations
are reasonable. The only way to know this is to research the query and analyze how
the advertiser interprets it.
Queries for which a Reasonable Ad is Impossible
Sometimes a query is either so hard to interpret or so non–commercial in nature that
no ad will be a good match. Be careful in these cases—rate the ad and landing page
according to how well they actually respond to the query, and do not worry about
how hard it would be to show an appropriate ad for that query. Do not give an ad
positive ratings if a better ad for the query cannot be determined or if it seems like it
was a good try. Rate it positively only if it addresses the query intent. If the query
intent cannot be determined, the ad must be rated negatively. For example, a query
of [ www ] or [ when did ] is not complete enough to serve a proper ad.
Unrateable Queries
In some rare cases, a query may appear that is the result of an error in how the task
was added to the evaluation system. For example, a query may appear in the
incorrect rating language, or a query of jumbled characters may appear that, after
research, has no discernible meaning.  Do not attempt to provide AC or LP ratings
for queries like this. Instead, use one of the flags provided in the Query Flag section.
These flags are only present on the first page of a Search Ads task, the Ad Creative
rating section.
 

Approximate Query Location

Many tasks will include a map indicating the geographical region where the query
was entered. If you are unfamiliar with the location in the map, you can zoom in and
out of the map to familiarize yourself with it. Knowing the approximate location where
the query was entered may help you decide how relevant the Ad Creative or Landing
Page is to a user. Some ads will be more relevant given the location, some will be
less relevant. For some ads, knowing the location will not make any difference.
When Location Matters
Sometimes the query, the ad, or both may refer to a specific geographic location.
Even when the approximate query location is available, it can sometimes be difficult
to determine how to handle a location-specific advertisement.
 
Location Mismatch Between Query and Approximate Query Location
If the query contains geographical information, you should use the information
available in the query instead of the map, especially if the two conflict.  For example,
if the query is [pizza in new york], but the map indicates Los Angeles, pizza
restaurants in Los Angeles are a bad result.
Query Specifies a Location but Ad Does Not
If a query specifies a location, take it into account when evaluating the ad.
Sometimes research is required to determine whether the product or service is
compatible with the location. This research is required before you can choose an Ad
Creative rating score.
 
User query: [ pizza Santa
Monica ]
 
If an ad pointed to the main
Round Table Pizza chain
homepage, but didn’t mention this
California city, this might initially
seem useful. However, if, upon
using the location finder on the
site, there are no locations within
a reasonable distance of Santa
Monica, this ad is probably not
useful.
Ad Specifies Location but Query Does Not
If the query does not specify a location and the approximate query location is also
not available, then evaluate the ad as if the user were in an appropriate location. For
example, if the query is [ pizza ] and the ad is for a pizza restaurant in Barstow,
California, assume that the user was in Barstow, California.
Neither Query Nor Ad Specify a Location
If the query does not specify a location and the ad does not either, then assume that
the user can be anywhere in the target country of the rating language. Ignore the
approximate query location, if one is provided.
Location Mismatch Between Query and Ad
Assuming there is a match between the product or service and the query and ad,
take the location proximity into account in evaluating an ad.  If the ad for a given
query specifies a different and incompatible location, this makes it a worse ad.
 
User query: [ pizza Santa
Monica ]
 
If the ad is for a pizza restaurant
in Manhattan, this is very
unpromising. However, an ad
returned for pizza in a different
but nearby location, like a
neighboring town, could be useful,
and this ad might not be as bad
as the previous example.  Use
common sense to determine if the
ad exceeds a reasonable distance
for the user, and an acceptable
distance may vary depending on
the query. For example, a user
may be willing to travel farther to
buy a new car than to get a
 haircut or go to the supermarket.
For certain queries, serving an ad
with a completely different
location may still be promising.
For example, if a user in the
United States is looking for
[ vacation in Australia ], then an
ad for “vacation in New Zealand”
is not necessarily a bad ad since
the user is likely to be willing to
travel a long distance for a
vacation.
 

Search Ad Rating: How Promising is this Ad?

Evaluate the Ad Creative.  A promising ad is one that seems like it will give the user
what he or she wants. An unpromising ad looks like it will be disappointing,
unhelpful, dangerous, or irrelevant.
 
Use the slider bar to select from four possible ratings:
 Very Promising
 Somewhat Promising
 Somewhat Unpromising
 Very Unpromising
Very Promising and Somewhat Promising are positive ratings:  use them for ads that
look like they’re good ads for the user to see that would be worth clicking on.
Somewhat Unpromising and Very Unpromising are negative ratings: use them for
ads that look like they’re bad ads for the user that aren’t worth clicking on.  Consider
the following factors while evaluating an ad:
 

Satisfying the User

Does the ad offer something that will satisfy the user? An ad that has nothing to do
with what the user wants is always very unpromising. An ad cannot be promising if it
is not relevant. However, being relevant is not enough to make an ad good.
 

Correct Meaning of the Query

Does the ad address the correct meaning of the query? An ad for a car dealership
does not address the query [ cars 2 ]—that is the name of a movie. Even if it would
be a good ad for some other car-related query, it is a completely terrible and
unpromising ad for that query.
 

Clarity and General Appeal

Is the ad written in a clear, appealing way? An ad that makes sense and does not
have mistakes, hard-to-understand language, or awkward phrasing can be good;
and, an ad that looks stupid, looks like it was written by a machine, is unintentionally
funny, or just does not make sense is usually bad.  Does the ad clearly state what
the advertiser offers?  A good ad is easy to understand.  A bad ad may be overly
vague or may not communicate enough information to conclude the ad will lead to a
positive advertising experience.
 

Potentially Scammy Ad
Does the ad look like a scam? An ad that seems too good to be true, sleazy, or
deceptive to users is usually bad.
 

Promise of Additional Links

Some ads contain multiple links to different sections of the website. You do not need
to click on these links, and when you are rating the creative, you cannot click on
them. However, if these links look promising or useful, this may be a reason to
increase your ad creative score. If links look unpromising, confusing, or useless, this
may be a reason to decrease the ad creative score. See the Ads with Additional
Features section for more detailed guidance.
 

Advertiser is Different Merchant or Provider from Query

Sometimes a query will specify both a product or service and a particular merchant
or provider. If the ad offers the desired product or service from a merchant or
provider that is not the one specified in the query, it should not be considered a
negative user experience unless there is another issue with the ad.
 

Analyzing the Advertiser Visible URL

The web address (also called the visible URL) displayed in the ad can provide clues
about how promising or unpromising an ad may be.  The visible url can affect your
evaluation in the following three ways:
 If you are familiar with the advertiser
based on the URL displayed in the
ad, you may use your background
knowledge when rating the ad. Just
use the “Used Prior Knowledge In
Judging Advertiser” flag.   
 If you aren't familiar with the
merchant, assume the merchant is
legitimate, even if that's not how you
behave in your own online activities.
Important note: this only applies if
there's nothing in the URL that looks
suspicious (see next bullet point).
 If the URL itself makes you
suspicious, don’t hesitate to mark
the ad bad. For example, an ad for
online book shopping that looks very
good except that the URL of the
merchant is www.amazom.com is
pretty suspicious--that looks like the
merchant is trying to trick you into
thinking you’re going to
 amazon.com. This is likely a
scammy ad, and if you think an ad is
scammy, it deserves a bad rating.
(Other tricks of this sort in addition
to misspellings in URLs include
adding random numbers,
unexpected extensions, or
subdomains to create URLs like
www.amazon22.com,
windows7onet.in, or
windows7.customersupport-us.net)
 
 
 
 

Do not assume an ad is promising just because it contains the same words as

the query. Do more than note that the words match—machines can tell us this. We
need human judgment:  tell us whether a human being will find an ad appealing. If a
user is looking for [ blue pants ] an ad that says “PANTS BLUE BLUE PANTS
www.bargainautoparts.com” is likely a bad ad even though it has the words “blue
pants.”

 
 

Distinguishing Between Very Promising and Somewhat Promising

If an ad looks like it will lead to a page that satisfies the user intent, it deserves a
rating of Somewhat Promising or Very Promising. A good ad deserves one of the
ratings described in the following two sections.
Very Promising
A Very Promising ad should look like it points to a page where a user can find almost
exactly what is described in the query. If the user is looking for a particular product,
the ad should appear to point to a page for that product. If the user seeks a particular
kind of store, the ad should appear to point to a store of that kind. If the landing page
of a Very Promising ad does not satisfy the user intent, it will be a surprise and a
disappointment to the user.
Somewhat Promising
A Somewhat Promising ad should also appear to take the user to a page where the
product he or she is looking for can be found; however, rather than appearing to
point to a page where the user can find exactly what he or she wants, a Somewhat
Promising ad might do one of the following things:
 It might look like it points in the right
direction but not exactly at the
target. For example, if the user
seeks a specific model of camera,
 an ad that looks like it will point to a
reputable camera store’s main page
is Somewhat Promising.
 It might look like it points to
something that might satisfy the
user intent but is not exactly what he
or she wanted. For example, if the
user seeks a particular model of
camera, a Somewhat Promising ad
might point to a slightly different but
reasonably similar model of camera.
Sometimes it is just not possible to be confident about what the user seeks. If an ad
seems to point in the right general direction but there is no way to tell exactly what
the user wanted, Somewhat Promising is the highest possible rating.
Distinguishing Between Positive and Negative Ad Creative Rating
Evaluate the Ad Creative and weigh the criteria for Very Promising/Somewhat
Promising against the criteria for Somewhat Unpromising/Very Unpromising.  If you
find that positive elements and negative elements both seem applicable to the
creative you’re evaluating, ask yourself which side of the positive/negative division
seems to be a more reasonable fit and choose a rating on that side.
Distinguishing Between Somewhat Unpromising and Very Unpromising
It is especially important to distinguish between ads that are simply bad and ads that
are very bad for the user entering the query. The following section provides
additional guidance to distinguish between the Somewhat Unpromising and Very
Unpromising ad creative rating.
Somewhat Unpromising
A Somewhat Unpromising ad generally isn’t a great ad to show the user, but it is
likely that some subset of users may find it useful.  
 Even if the creative is not of the
exact same topic as the query, as
long as there is some clearly related
task or intent, there are some users
who may find the creative
appealing.  One example is if the
user seeks [ weight loss pills ] and
the ad is for “diet tips” or “exercise
machines”.  These types of ads
should be rated as Somewhat
Unpromising.  They don’t directly
provide what the user is looking for,
but could be somewhat useful to the
user so don’t deserve the lowest
ratings.
 If it is not really clear whether users
will find the ad useful, rate it as
Somewhat Unpromising.  One
example is if a user is searching for
some information and the ad asks
 the user to search for the same
information again elsewhere.  It is
hard to know in these cases
whether the ad will be able to
provide anything useful to the user,
since he/she is being asked to
repeat the same action again
possibly just to get similar results.
Please view the Google search
results for the query to get an
understanding of what the user
currently sees and what information
he/she currently has access to.  If
you believe that the ad won’t
provide any additional information
from what is already presented to
the user, rate it as Somewhat
Unpromising. However, if you
believe that clicking the ad will
provide additional useful information
to the user, don’t rate it as
Somewhat Unpromising-- give it a
higher rating.  One example that
would deserve a higher rating than
Somewhat Unpromising is if the
user is searching to buy a particular
item and the ad is asking the user to
search for that particular item across
numerous stores and merchants.
Another example that would again
deserve a higher rating is a query
for some specific industrial
machinery part, and an ad inviting
the user to repeat the search on a
search engine devoted to machine
part sales and manufacture.
 Sometimes a query specifies a
location, and the ad targets a
different location. For these specific
examples, please refer to the When
Location Matters section.
 

Very Unpromising
There are several cases where Very Unpromising is the only appropriate rating.
 Very Unpromising ads have no
reasonable chance of satisfying the
user. Try to put yourself in the user's
mindset - is it possible at all that the
creative offers something useful to
 the user?  If there is no reason at all
to think that the user will find the
creative useful, rate it Very
Unpromising. (Note: you might think
“It’s always possible that someone
might find anything useful, even
though it has nothing to do with the
query.” Don’t go that far!)
 If the creative looks like a scam, or
leads the user to harm, rate it as
Very Unpromising.
 If the creative falls into one of the
categories listed in the Machine-
Generated Ads section, rate it as
Very Unpromising.
 If the creative promises to do the
impossible, such as selling a person
or city, rate it Very Unpromising.
 Just because there is a strong term
overlap between the query and
creative does not mean the ad is a
good match for the query.  If the
user is searching for
[ homeowners insurance ] and the
ad is for “medical insurance,” the
user will very likely not find the
creative useful and you should rate
it as Very Unpromising.
 

Machine-generated Ads
 
Some ads are partially auto-generated to take words from the query and place them
in the creative text.  There is nothing wrong with this in itself. For example, if the
query is [ xbox 360 used ] and the creative says “Buy a used xBox 360 on eBay,”
that’s a good ad. Unfortunately, sometimes these machine-generated ads turn out
very badly. Very Unpromising ad creatives may have some of the following issues:
 
Things offered that cannot be bought
 
User query: [ san diego, ca ]
 
An ad that says “Buy San Diego
cheap on eBay” is ridiculous--you
can’t buy a city. Ads that are
unintentionally ridiculous, horrible,
or offensive, by suggesting that
you can buy concepts, human
beings, body parts, criminal acts,
 or similar things are Very
Unpromising.
 
Part of the query removed, substantially changing the meaning
 
User query: [ roses lime juice ]
 
An ad that offers the action, “Buy
roses,” has left out so much of the
query that the entire meaning has
changed.  By taking only part of
the text of the query what remains
substantially changes the
meaning.
 
Part of the query removed, resulting in overly general ad
 
User query: [ how do i remove
gum from satin ]
 
An ad that offers “Get information
on how to remove,” is nearly
meaningless:  too much has been
removed from the query.  By
taking only part of the text of the
query, the result is far too general
to be promising for the user
query.
 
Nonsensical, jumbled, or ungrammatical ad creative
 
User query: [ how do i remove
gum from satin ]
 
An ad that says “Search for how
do I remove gum” or “Find how do
I remove gum from satin” is
awkward and ungrammatical. Ads
that end up nonsensical, jumbled,
or ungrammatical because a
query has been crammed into a
space where it doesn’t really
belong is Very Unpromising.
 
Be on the lookout for these. If you’re not paying close attention to how the ads
actually look and sound, it can be easy to think these look fine—but to a user who is
actually reading the text, they can look laughable, annoying, or foolish, and in some
cases, deeply offensive or hurtful. Even those that just look sort of silly or awkward
are very bad.
 
Ads with Additional Features
Some ad creatives are just text and a single link to the advertiser page. Other pages
contain additional features that may or may not provide something valuable to the
user. Ad creatives may contain maps, videos, images, star ratings from customers,
multiple links to specific pages on the advertiser site, and a variety of other features.
 
These additional features may affect the quality of an ad creative. If the special
features add to or detract from the appeal, informativeness, or usefulness of an ad,
the Ad Creative score can be raised or lowered.
 
For an ad that contains only text and a single link to an advertiser page, use only the
previous criteria in making a decision. For ads that contain anything in addition to
text and a one link, consider the following factors, and decide whether to raise or
lower your score:
 If an ad does not deserve a score of
Somewhat Promising or Very
Promising based on the previous
criteria, be cautious about giving it a
positive rating just because of
additional features.
 An ad that deserves a score of
Somewhat Promising or Very
Promising based on the previous
criteria can be given a negative
rating if additional features detract
from it severely.
 Use common sense when deciding
whether additional features improve
or detract from an ad enough to
move it between Somewhat
Promising and Very Promising
scores in each category.
 An ad that is scammy or harmful
can never be improved by additional
features.
 Additional feature should relate to
the user intent in a sensible way.  If
the user is looking for information
about a current movie, a movie
trailer in the ad creative relates to
the user in a sensible way, but a
map to the movie studio where the
film was made does not. The trailer
probably improves the experience,
but the map detracts from it.
 Where an additional feature is
relevant to the user intent, it should
be informative, easy to use, and
clear. If it is confusing, boring, or
hard to figure out, it may either
 detract from the experience or just
fail to improve it.
 An ad may have multiple additional
features. Consider all of them
together when determining your ad
creative score.
 Raise or lower your rating by a small
amount if the additional feature has
little impact on the ad creative.
Raise or lower it a large amount,
according to the previous criteria, if
the additional feature has a big
impact.
 Do not consider the physical size of
an ad creative rating it.  If the size of
an ad creative causes it to display
incorrectly in the ads rating
interface, alert an administrator but
ignore the issue while rating it.
 

Search Ad Rating: Does Landing Page Satisfy User Intent?

Use the slider bar to select from the following four rating categories while
determining how likely it is that a landing page will satisfy the user intent.  Only
consider the user query and the landing page, and ignore the ad creative completely.
 Satisfaction Likely
 Satisfaction Possible
 Dissatisfaction Possible
 Dissatisfaction Likely
Satisfaction Likely and Satisfaction Possible are positive ratings that satisfy the user
query.  Dissatisfaction Possible and Dissatisfaction Likely are negative ratings that
do not satisfy the user query.
 
 
 
 

Only rate the landing page that opens after clicking on the Visit Landing Page
button. Do not base your score on pages that are accessible by clicking on links in
the body of the ad creative. NEVER copy and paste a link to visit the page, and
NEVER manually change the URL.

 
 
The fundamental principle of landing page evaluation is this: the user starts a search
on Google.com with a goal in mind. The user then enters a query and reviews
Google’s search results and ads. The user then clicks on the ad currently being
reviewed, and that ad takes the user to the landing page. Keep in mind that in order
for a user to have a positive experience with an advertiser landing page, he or she
should be closer to the goal expressed in the query, otherwise it is a negative
experience.  The section below helps frame how distance from the user’s goal helps
determine a landing page rating score.
 
Distance from the User’s Goal
Carefully review the Google search results page to determine the distance from the
user’s goal.  Does the Landing Page move user closer to his or her goal, further from
the goal, or neither closer nor further from the goal?
 
If the user is closer to the goal, the landing page deserves a positive rating.  For
example, if the user is hoping to buy a specific camera, and the landing page is a
store offering that camera for sale, the user has come closer to accomplishing his or
her goal.
 
If the user is further from the goal, the landing page deserves a negative rating.  If
the user is hoping to buy a specific camera, and the landing page is a store offering
pet food, this is a dead end. The user will need to go back to the search page or start
a new search, so he or she is actually further from the goal than before clicking on
the ad.
 
If the user is neither closer to nor further from the goal, the landing page deserves
a negative rating.  If the user is on a Google search results page and clicks on an ad
that just takes them to a page of similar search results, which overall did not provide
any additional value, no progress has been made; the user is no closer to or further
from the goal than before clicking the ad.
 
Deciding this is not an exact science. Rely on good judgment. The following
guidelines more deeply explain how to generally rate landing pages, but they do not
explain how to rate a landing page in every situation.
 

Distinguishing Between Satisfaction Likely and Satisfaction Possible

Satisfaction Possible and Satisfaction Likely are positive ratings. If the landing page
offers the user exactly or very nearly what he or she wants, use a Satisfaction Likely
or Satisfaction Possible rating.
Satisfaction Likely
To receive this rating, a landing page must offer just what the user looked for. If the
user wants car reviews, it should offer car reviews. If the user wants car reviews
about a specific model, it should offer car reviews about exactly that model. If the
user wants a category of product, the landing page should be devoted to or include
that exact category of product. For a Satisfaction Likely rating, what the user is
looking for should be apparent with no additional action needed by the user. It is
permissible, however, to click on a link to get detailed information.
Satisfaction Possible
Use this rating if the page is satisfactory but does not immediately present exactly
what the user seeks. If the product or service is for sale on the site, but a search or
straightforward navigation is required to find the item, select a rating of Satisfaction
Possible rather than Satisfaction Likely. If the site offers a very plausible substitute
for a particular product specified in the query, it may receive a rating of Satisfaction
Possible or lower. If the query is a search for information, and this information can be
found without too much trouble on the advertiser site but is not on the landing page,
use Satisfaction Possible.  The one exception here being if the user could have
found that same information on the search results page before clicking on the ad. If
that is the case, the landing page does not deserve a positive rating.
 

Considering Trustworthiness
Do not give a landing page a Satisfaction Possible or Satisfaction Likely rating if you
do not trust the information found on that landing page or if you would not make a
purchase from the advertiser site. A page that offers the exact product that a user is
looking for is useless unless the user trusts it enough to actually make a purchase
there. A seemingly trustworthy merchant selling a particular camera at a particular
price might deserve a better rating than a page that clumsily aggregates a random
set of products, even if the same camera at the same price is offered on that page
too. Similarly, a page offering the exact information that the user is looking for is not
useful if there is no reason to think that the information is correct. For example, if the
user seeks some medical information, a site belonging to a medical school is a good
source of trustworthy information while a blog post by an unknown person is a much
more doubtful source.  Never use a rating of Satisfaction Likely or Satisfaction
Possible if the page appears scammy or harmful.
 

Specific Versus General: Mismatch Between Queries and Landing Pages

Sometimes when the query is for a specific product, the landing page is basically on
target but much broader or much more specific than the query.
 
If the landing page has the product specified in the query, but a huge number of
other products too, this may be a decent experience, but probably is not good
enough to get into the Satisfaction Likely range in most cases.
 
If the query is for something general, like [ camera ], the landing page might be
extremely specific. For example, a product page for a particular model of camera
from a particular manufacturer with a particular set of options. In this case, too, it
might appear to be a decent experience, but it probably is not good enough to get
rated as Satisfaction Likely in most cases.
 
You may judge that in particular cases, the experience is better or worse than the
guidelines above would suggest. For example, if the page has a huge number of
different products, but the product in the query is clearly the most prominent and the
first thing you see, you might decide it deserves Satisfaction Likely; if it’s so buried in
the other products that you don’t even realize it’s there, you might decide it deserves
a negative rating. Similarly, if the query looks general and the landing page is for a
very specific product, you might think that the product is so obviously the best
possible thing to offer for that query that it deserves Satisfaction Likely; on the other
hand, if the product is technically in the right category but very very unlikely to be
what the user wants (for example, an expensive antique camera requiring glass
plates instead of film for the query [ camera ]), you might decide that it deserves a
negative rating.
 
Distinguishing Between Positive and Negative Landing Page Rating
 
Evaluate the query and the Landing Page and weigh the criteria for Satisfaction
Likely/Possible against the criteria for Dissatisfaction Possible/Likely.  If you find that
positive elements and negative elements both seem applicable to the landing page
you’re evaluating, ask yourself which side of the positive/negative division seems to
be a more reasonable fit and choose a rating on that side.
Distinguishing Between Dissatisfaction Possible and Dissatisfaction Likely
 
Dissatisfaction Likely and Dissatisfaction Possible are negative ratings.  If you think
that the user will have a negative experience, always use either Dissatisfaction
Possible or Dissatisfaction Likely. If you have no particular reason to think a page will
interest the user, always use either Dissatisfaction Possible or Dissatisfaction Likely.
 

Dissatisfaction Possible
 If the page is marginally related to
the query and you think that there’s
a small chance the user would be
interested, use Dissatisfaction
Possible.
 If the page can eventually lead to
what the user wants, but only
through many clicks or through
clicks that lead to an entirely
different website, use Dissatisfaction
Possible.
 If the page offers something that
you think the user might be
interested in, but not what the user
was looking for and not especially
close to it, use Dissatisfaction
Possible. For example, if the user is
looking for baseball gloves, and the
landing page offers athletic socks,
there’s probably some chance that
the user might be interested.
However, it’s not what the user was
looking for, and not all that close to
it, so it deserves Dissatisfaction
Possible.
 If the page can eventually give the
user what he or she is looking for,
but the process is protracted and
difficult, use Dissatisfaction
Possible.
 

Dissatisfaction Likely
  If the page has nothing to do with
the query, use Dissatisfaction Likely.
 If the query is for a product or
service, and neither the
product/service nor anything close
to it can be purchased from the
page, use Dissatisfaction Likely.
 If the query or a word in the query
has two meanings, it is clear which
meaning is intended by the user,
and the advertiser responds to the
wrong meaning, use Dissatisfaction
Likely. For example, [ cars 2 ] refers
to a movie. A page for a car
dealership is clearly a bad landing
page for this query, even if it might
be a good result for [ car sales ].
 If the page looks like a scam, you
think users could be harmed by it, or
it either attempts to trick the user
into downloading something by
labeling a download button in a
confusing way or tries to download a
file without action by the user, use
Dissatisfaction Likely.
 If the page loads but is completely
unusable (for example, because
some content does not load, or
page doesn’t display properly) use
Dissatisfaction Likely.  If enough of
the page does not load at all (for
example, you encounter a 404
error), use the Error Did Not Load
flag instead of a rating.
 If the page is very bad for any other
reason, use Dissatisfaction Likely.
  

Query Flags
Use these flags to indicate that a query is unrateable. This means that it, and the AC
and LP paired with it, are not eligible to be assigned ratings. A Search Ads query is
unrateable if it has one of the following problems:
 
 it is in a language other than the task
language (Foreign Language)
 it is unambiguously pornographic or
about sexual services (Porn)
  it is complete nonsense; research
reveals no plausible meaning
(Nonsense)
 it was transcribed incorrectly, using an
English rather than Cyrillic keyboard for
Russian words (Russian Transcription
Error)
 
If you use one of these flags, all of the later questions will turn gray and don’t need to
be answered. Note that these flags are only on the first page of a Search Ads task,
the Ad Creative rating section.
Foreign Language Query
Use this flag when the query is in a language other than the language of the task. If
the query contains words or phrases in another language, but there is enough
content in the task’s language that it is understandable, do not use this rating. If the
query appears to be in a foreign language, but research reveals that the query term
may be commonly used in your rating language or is the name of a specific group,
product, or person, do not use this rating.
 
Remember to check the language of the task, especially if you work in multiple
languages. If your rating language is English, you rate ads in English for English
queries. If you rate in another language, you will rate some tasks in that language
and some tasks in English. Your rating language is always designated at the top of
the task page. Even if you speak the language of the query, if the task is supposed
to be for a different language, use this rating.
 

Porn Query
Use this flag only for queries that are unambiguously for pornographic content or
sexual services. Queries for racy or suggestive content, medical information, or art
photos generally shouldn’t get this rating. Queries for dating services generally
shouldn’t get this rating, unless those dating services depict nudity or specifically
identify themselves as sexual rendezvous services.
 

Nonsense Query
Use this flag for queries that are complete nonsense, where research reveals no
plausible meaning. As you research, take into consideration that queries that may
look like nonsense might actually turn out to be meaningful. The following are
examples of queries that do have meaning and should not receive the Nonsense
Query flag:
 
 a misspelling
 a product code or model number
 technical specifications
 a partial web address or YouTube video
ID
 a specific username or Twitter handle
 an uncommon acronym or abbreviation
 
Don’t assume that a query is nonsensical just because you do not immediately know
what it means. Encountering a completely nonsensical query is rare. Most queries
mean something, so you should always research the query, even if at first it seems
like nonsense. Only use this rating when there is no way for you to reasonably guess
about user intent, even after researching the query.
 

Russian Transcription Error

This flag applies only to raters working in Russian. If you are working in a language
other than Russian, this flag will never be applicable to your tasks, and you should
not use it. If you are working in Russian, you will receive separate instructions for
determining when queries should be considered transcription errors.
 
While you will not be able to assign AC, LP, or AC to LP ratings after using one of
these Query Flags, you will still need to submit the task for your answers to be
recorded. You will submit your responses directly from the first page of the task by
clicking the Submit button at the bottom left of the task.
 

Ad Creative Flags

If an Ad Creative meets the criteria for using one of the following flags, please use
that flag. If criteria are not met for a flag, do not use the flag.
Navigational Bullseye
 
Use the Navigational Bullseye flag when both these things are true:
1. The query appears to be a search for a particular website, section of a website, or
web page.
2. The creative looks like it will point to the corresponding website, section of a
website, or web page.
 
Not every query is a search for a particular website--in fact, the vast majority are not.
 
The Navigational Bullseye flag should only be used where the frame of reference is
similar or compatible between query and creative.  For example, with the query,
[ ford explorer ], the Navigational Bullseye would be used for creative that appears to
take the user to the Ford Explorer section of the Ford website; however, the flag
would not be used if the creative appeared to take the user to a different page on the
Ford site (a page devoted to the Ford Focus) or a general page on the Ford site
(their homepage, for example).
 

Foreign Language
 
Use this flag when the creative is in a language other than the language of the task.
Remember to check the language of the task, especially if you work in multiple
languages. Even if you speak the language of the creative, if the task is supposed to
be for a different language, use this flag.
 
A creative should be legible in your rating language:  if the creative contains words or
phrases in another language, but there is enough content in the task’s language that
it is understandable, do not use this flag and proceed with the normal creative rating.
 
If you use this flag, some of the later questions will turn gray and don’t need to be
answered.
 

Unexpected Porn
 
Use this flag when both these things are true:
 
1. The query is not a search for pornographic content or sexual services. If the query
has both a pornographic interpretation and a non-pornographic interpretation,
assume that the non-pornographic interpretation is the actual user intent.
2. The creative appears to offer pornographic content or sexual services.
 
Use this flag only for unambiguously pornographic content or sexual services. Racy
or suggestive content with no nudity, nudity in a medical context, or art photos
generally shouldn’t get this flag. Dating services generally shouldn’t get this flag
unless they depict nudity or specifically identify themselves as sexual rendezvous
services. A regular dating service may deserve a bad rating if it doesn’t match what
the query appears to be looking for, but it would not get the flag.
 
If you use this flag, some of the later questions will turn gray and don’t need to be
answered.
 

Error / Did Not Load

 
Please use the Error / Did Not Load flag for the ad creative in case the ad creative
does not load or is blank. You should use this flag instead of reporting a problem if
the ad creative does not load.
 

Used Prior Knowledge In Judging Advertiser

 
Use this flag when knowledge you already had about the advertiser influenced your
ratings, either for good or bad. Use this only when your rating is different from what
you think you would have given seeing the ad for the first time with no prior
experience. If a creative is clearly bad, don’t use the flag even if you already happen
to have confirmation that a bad rating is deserved.
 

Secondary Interpretation of Query

 
Use this flag when the creative text indicates that the advertiser is targeting a clearly
secondary  interpretation of the query. An interpretation is secondary if it’s
reasonable, but there is some other interpretation of the query that you consider
much more likely. Don’t use this flag with interpretations that are wrong or
unreasonable.  Don’t use this flag if you think that the query has multiple, equally
likely meanings, and the advertiser is targeting one of those meanings. Do use the
flag where the query has multiple, equally likely meanings and the advertiser targets
an obscure or less-likely meaning.  Please review the main guidelines for instructions
on how to select a scale rating when you use this flag.
 

Landing Page Flags

If a Landing Page meets the criteria for using one of the following flags, please use
that flag. If criteria are not met for a flag, do not use the flag.

Navigational Bullseye
 
Use the Navigational Bullseye flag when both these things are true:
 
1. The query appears to be a search for a particular website.
2. The landing page is that site.
 
Not every query is a search for a particular website--in fact, the vast majority are not.
 

Foreign Language
 
Use this flag when the landing page is in a language other than the language of the
task, with no obvious way of getting to a version in the language of the task.
Remember to check the language of the task, especially if you work in multiple
languages. Even if you speak the language of the page, if the task is supposed to be
for a different language, use this flag.
 
Don’t use this flag if there is some clear way to get to a version in the target
language. For example, if you are rating a Japanese task, a landing page in English
with a Japanese flag in the corner pointing to a Japanese version of the site should
not get this flag.
 
If you use this flag, some of the later questions will turn gray and don’t need to be
answered.
 

Unexpected Porn
 
Use this flag when both these things are true:
 
1. The query is not a search for pornographic content or sexual services. If the query
has both a pornographic interpretation and a non-pornographic interpretation,
assume that the non-pornographic interpretation is the actual user intent.
2. The landing page offers pornographic content or sexual services.
 
Use this flag only for unambiguously pornographic content or sexual services. Racy
content with no nudity, nudity in a medical context, or art photos generally shouldn’t
get this flag. Dating services generally don’t get this flag unless they depict nudity or
specifically identify themselves as sexual rendezvous services. A page with racy
content, nudity in an art or medical context, or dating services may deserve a
negative rating if it doesn’t match what the query appears to be looking for, but it
shouldn’t get the flag.
 
If you use this flag, some of the later questions will turn gray and don’t need to be
answered.
 

Unexpected Download
 
Use this flag when any of the following happens:
 
1. Clicking on the Visit Landing Page button initiates an attempt to download a file.
2. Some link, button, or graphic on the landing page initiates a download when
clicked, but does not clearly indicate that it will do so. For example, a big red button
that says “Enter site” or “Check the weather,” but starts a download when clicked,
deserves the flag. A similar button that says “Get It Now” or “Click here to download”
does not.
 
Never install downloads that a site tries to initiate in this way:  it is not part of the
rating process.
 
If you use this flag, some of the later questions will turn gray and don’t need to be
answered.
 

Error / Did Not Load

 
Your job when evaluating a search ads task is to evaluate content provided by the
advertiser (the ad creative and landing page).  Use the Error/Did Not Load (EDNL)
flag to indicate that you cannot evaluate the landing page because there is no
landing site content provided by the advertiser.  There are several reasons why you
might not be able to access landing site content provided by the advertiser,
including:
 the page or site no longer exist
 the page or site are under
construction
 your browser is not able to find or
access the page we provided you
 your virus/malware protection
software blocks you from accessing
the site
 the landing page opens using a 3rd-
party program (such as iTunes) that
you do not have installed
 
It’s not always easy to immediately determine if the EDNL flag should be used
because different things can happen when a landing page is not working properly.
Here are some examples of what you might see when no landing site content is
available to evaluate:
  a completely blank page
 a generic Not Found message
generated by your web browser
(example: https://www.google-
news.com/default.html)
 a generic error message generated
by the advertiser’s server
(example:
http://www.centraldopolidor.com.br/
enceradeiras.htm)
 a generic webpage (often filled with
affiliate links) shown by the hosting
service in place of the actual landing
page
(example:
http://genealogywise.com/?
reqp=1&reqr=)
 a search results page shown by
your internet service provider
because the actual landing page
cannot be accessed
(example:
http://www.dnsrsearch.com/index_re
sults.php?
querybox=sdiwfkdis.com&submit=S
earch)
 a notice that the site or page is
under construction with no way to
access any other part of the landing
site
(example:
http://www.reidknorr.com/demos/vint
a_ss/)
 
In all these cases you should use the EDNL flag because you cannot access any
content from the actual landing site to evaluate.  In the first two examples, above,
there is little or no content to evaluate.  In the last four examples, there may be
content you can see, but it is either not content from the landing page advertiser (e.g.
the hosting service, browser, ISP), or the entire advertiser site is inaccessible.
 
Note that a landing page could have an error on it but still have landing site content
or a way to access landing site content on the page.  Here are some examples of
things you might see when there is an error on the page but advertiser content is still
available to evaluate:
 a page which partially loads
 an error saying that the page could
not be found, but linking to another
part of the landing site
  an error stating that the product
could not be found, but page
provides alternatives or a way to
search the landing site for other
products
 a blank page or an error page that
still has site navigation tools (usually
on the top or side)
 an error page which automatically
redirects to and loads a working
page on the landing page advertiser
site
 a landing page which is blocked by
a registration form
 
If an advertiser landing page provides enough content to rate, don’t use the EDNL
flag.  In the cases above, the flag is not used because there is at least some
advertiser content on the LP upon which to base your evaluation on.
      
If you use this flag, some of the later questions will turn gray and won’t need to be
answered.
 

Secondary Interpretation of Query

 
Use this flag when the landing page content indicates that the advertiser is targeting
a clearly secondary interpretation of the query. An interpretation is secondary if it’s
reasonable, but there is some other interpretation of the query that you consider
much more likely. Don’t use this flag with interpretations that are wrong or
unreasonable.  Don’t use this flag if you think that the query has multiple, equally
likely meanings, and the advertiser is targeting one of those meanings. Do use the
flag where the query has multiple, equally likely meanings and the advertiser targets
an obscure or less-likely meaning.  Please review the main guidelines for instructions
on how to approach the scale rating when you use this flag. 

If you have questions about this project that are not answered by the instructions
above, please review the Rater Hub, which contains additional content about Search
Ads rating.  If you encounter a technical problem with this rating task, use the
“Report A Problem" link in the lower-right hand corner of the rating page.

Task 1 Screenshot:
 

Task 2 Ad Creative and Landing Page multidimensional features

This task has short instructions at the top of the task, and 14 questions.
Each question has an instructions section with examples. Each instruction
section is shown below its relevant question. At the end, you will see a few
screenshots of the actual task. 

Version: 2017-04-24

Task: You will be presented with the following in sequential order: a

query, an ad creative and a landing page. You will have to:

1. Research the query and try to

understand the user intent. You will
then have to assess how clear the
user intent is to you.
2. Next, you will be presented with an
ad and you will have to assess a few
different features of the ad and the
relationship of the ad to the query. 
3. Finally, you will be shown the landing
page for the ad, and you assess a few
different features of the landing page
and the relationship of the landing
page to the query as well as the ad. 
4. Each question you will be asked is
contained within a box. At the bottom
of the box, there are examples and
instructions. Please make sure to
read through these instructions
and examples in detail and refer
to them as frequently as
necessary while you are
completing these tasks.

Query: If the query is absolutely incomprehensible to you, choose the

"Unrateable query" flag. You will not have to answer any of these
questions in the task after this and can go on to task submission. If you
can comprehend the query to some degree, go ahead with the
rest of the task.

Question 1: How clear is the user intent to you?

In this section we want to know whether the user's intent is clear to you or
not. After researching the query, if you understood none of the words,
then select the "unrateable query" flag. Assuming you do understand at
least some of the words in the query and broadly what the user might be
trying to achieve, try to think of the possible things the user might be
looking for or trying to do. 

If you can only think of a single thing that the user might be looking to do,
then the user intent should be "Very Clear".  On the other hand,  if you
can think of a few distinct things  (2 - 5 or so) that might be valid user
intents, then the rating should be "Somewhat clear". If there can be so
many possible user intents that it is impossible to enumerate them, then
the rating should be "Very unclear".

Rating Description Examples

Very
unclear There can be no Query: [life]
reasonable user
intents or many Although the query is comprehensible,
reasonable distinct it is absolutely not clear what the user
user intents. was looking for. There are just too
many possible user intents to
enumerate. 

Query: [cash]

The user can be looking for ways to

make money. Or the user could be
looking to borrow money. Or the user
could be looking to store cash at a
bank. The possibilities are endless and
since there can be many
interpretations, it is not clear what the
user is looking for.

Query: [www]

It is really not clear what the user is

looking for and there are no reasonable
user intents that can be attributed to
this query.

Somewh
at clear There can be a few Query: [chocolate chip cookies]
 (2- 5 or so) It is a bit unclear what the user is
reasonable distinct looking for. The user might be looking
user intents. for recipes on how to make chocolate
chip cookies. Or the user might be
looking for online stores to purchase
chocolate chip cookies. It's possible
that the user is looking for information
about different kinds of chocolate chip
cookies. Since there can be a few
different valid user intents,  the user's
intent is not very clear.

Query: [how to fix roof]

The user intends to fix some problem

with a roof. It is unclear what the
problem is and hence what targeted
information to show the user to satisfy
their intent. The user might want
information about roofing services, or
about roofing supplies or even tutorials
on how to fix roofs.

Query: [best buy santa monica]

There could a few different user intents

behind this query. The user could be
looking for directions to the Best Buy in
Santa Monica. Or the user might want
to determine if there is a Best Buy in
Santa Monica or they could be looking
for the website for Best Buy with the
store location set to Santa Monica.

Very There is only a single

clear reasonable user Query: [amazon.com]
intent.
The user intends to visit amazon.com. 

Query: [best budgeting app iphone]

The user is interested in finding good

financial budgeting apps for the
 iphone.

Question 2: What is the scope of the concepts covered by the ad

in relation to the concepts covered by the query?

What is a concept?

A concept is a broad term for an idea that represents something, like an

activity (shopping), a thing (an iPhone, a discount) or a class of things
(blue sweaters, US-based companies).

Although an ad or a query can refer to multiple concepts, focus on the

concepts which are critical to defining what the ad or query is about.

For example, for the following ad:

the important concepts are those of "Amazon.com" and "Books" since

they define what the ad is about. Less important concepts for this ad are
those like "Amazon Prime", "Shipping", "Deals", "Electronics", "Toys" etc.

How do I compare scope?

In this question, you will have to assess whether the scope of the ad is the
same as the scope of the query and if they differ, in what way they
differ. Consider the important concepts that are associated with the query,
and those that are associated with the ad. When you compare scope, you
look at the overlap between these groups.

Rating Description Examples

No
Overla There is Query: [Android marshmallow update]
p absolutely no
overlap
between the
concepts
covered by
the ad and
 the query. The ad is for actual candy while the query is for
the android software update named
Marshmallow.

Partial
overla There is some Query: [new american university chemistry
p degree of building]
overlap
between the
scopes of the
ad and the
query, but
their scopes The query and the ad overlap to the extent that
cover they are both about universities, but they are
fundamentall about fundamentally different concepts. The
y different query is about the Chemistry building within New
sets of topics American University and the user is likely looking
and one is not for its location, but the ad is for degree programs
contained at the University of Phoenix.
within the
other.
Query: [bases for beds]

The query and the ad are both about sleeping

products, but the query is about bases while the
ad is about mattresses. Since both concepts are
about bedding products, they overlap partially
but they are also each about different, unique
products so the overlap is only partial.

Query: [recipes]

The query and the ad overlap because they are

both broadly about cooking but the user is
searching for recipes, while the ad is about
cooking ware.
 Query: [plymouth nc]

The query is for Plymouth, a city in North

Carolina, and the ad is for a very specific news
article about North Carolina's governor. There is
some degree of overlap because both the query
and the ad are about North Carolina, but the
query and the ad are about fundamentally
different concepts, so this is a case of a "Partial
Overlap".

Ad
Narro The range of Query: [jewellery macys ]
wer concepts
covered by
the ad is
narrower than
those covered
by the query.

The user is looking for jewellery at Macy's. The ad

is for earrings and hence is narrower in scope.

Ad
Broad The range of Query: [Women's sweaters JCPenny]
er concepts cove
red by the
ad is broader
than those
covered by
the query. The query is specifically about women's sweaters
at JCPenny but the ad is about all sweaters. So,
the scope of the ad is broader than that of the
query.

Query [amazon.com books]

 The query is specifically for books at amazon, but
the ad is for the broader amazon website.

Query: [frozen poster]

The query is looking specifically for frozen

posters but the ad is for Disney posters in
general.

Same
The range of Query: [amazon.com books] 
concepts
represented
by the ad and
query are the
same.
The ad is looking for books on amazon.com and
The ad does the ad is for books on amazon.com. The ad also
not cover any describes other concepts like "Amazon Prime",
concepts not "Deals" etc.,  but the important concepts for both
present in the the ad and the query are the same.
query and
vice-versa
Query: [Women's sweaters JCPenny]

Question 3: Does the ad misinterpret the concept the user

specified in the query and target a completely different concept
with a similar spelling or pronunciation?

Sometimes the same or similar phrases can refer to two entirely different
concepts. Determine the concept that the user is likely referring to and
then assess whether the ad targets a completely different concept with
the same or similar spelling or pronunciation.
Rat Description Examples
ing
Yes
The Query: [paris]
ad misinterprets t
he concept the user
specified in the
query and targets
a completely  diff
erent concept with The ad is for the purchase of plaster of paris
a similar spelling or which is very likely an incorrect interpretation
pronunciation. of the query. The user probably want to see
information about Paris, France.

Query: [transformers 2]

The ad is for electrical transformers while the

query probably refers to the Transformers
series of movies.

No The ad does not

target a completely Query: [hair laser treatment growth]
different concept
with a similar
spelling or
pronunciation to
what the user
specified in the The user is looking for laser treatment for
query. hair growth.  The ad is showing treatments for
hair removal. These are related concepts and
so even though this is a bad ad, the ad still
targets the same concept the user is looking
for. If instead, the ad targeted "Hair" the
movie, then the response to this question
would have been "Yes".

Query: [Pizza Santa Monica]

 The user is looking for Pizza in Santa Monica,
California. The ad is for Chinese food delivery
in New York City. While the ad targets a
different concept, it is not similar sounding in
any way to "Pizza Santa Monica", so the rating
for this example is "No".

Question 4: Does the ad miss a critical requirement for satisfying

the user intent?

Think about all the concepts  and requirements the user has specified in
the query. Now for each requirement, ask yourself how likely the user is to
consider an alternative. If it is very likely that the user will consider an
alternative, then the requirement is non-critical. If missing this
requirement or providing an alternative would be unacceptable to most
users, then it is critical. If an ad drops both a critical and a non-critical
requirement select "Misses critical requirement" since that leads to
worse ad experience for the user.

Rating Descript Examples

ion

Misses The ad
critical misses Query: [hair laser treatment growth]
require one or
ment more
critical
requirem
ents that
the user
The user is looking for laser treatment for
specified
hair growth. But the ad is showing treatments
in the
for hair removal. The ad completely missed the
query.
key requirement of hair growth instead of hair
removal  from the query and it is very unlikely
that the user can be swayed to accept this ad as
  satisfying their intent.

Query: [womens sweaters]

The user is looking for womens' sweaters but the

ad is for childrens' sweaters. This ad completely
missed the requirement of  "women" and hence
misses a key requirement since the user is
unlikely to be satisfied by children's sweaters.

Query: [wow stutter]

The user is looking for information about a

specific problem while rendering graphics in the
game World of Warcraft, called stuttering. The
ad, instead, provides information about the
stuttering, the speech impediment. The ad has
thus completely missed the key requirement that
information about stuttering should be in the
context of the video game World of Warcraft.

Query: [Pizza Santa Monica] 

The query is for Pizza in Santa Monica while the

 ad is for Pizza Delivery in San Francisco. The ad
completely misses the critical requirement that
the pizza delivery should be in Santa Monica
since San Francisco is too far away from Santa
Monica for a pizza delivery.

Misses The ad
non- misses Query: [leather sofas]
critical one or
require more
ment non-
critical
requirem
ents that
the user The user is searching for leather sofas but the ad
specified is for fabric sofas. While this misses the "leather"
in the requirement for the sofa, it is likely that the user
query. might also be willing to consider fabric sofas.

Query: [macy's women's shoes]

The user is looking for women's shoes at Macy's.

The ad is for shoes at Zappos. Although this is
not a perfect match and the store is different
from what is specified in the query, the exact
store is not such a critical requirement to
significantly affect user satisfaction. The user will
likely find equally good shoes at Zappos.

Doesn't The ad
miss does not Query: [nike men's shoes]
any miss any
require requirem
ments ents that
the user
specified
in the
 query.

Question 5: Is the ad awkwardly phrased or difficult to make

sense of?

Is the ad text awkwardly phrased or difficult to make sense

of? One possible way this can happen is when the ad is "machine
generated" and terms from the query are stuffed into the ad creative
resulting in a nonsensical or ungrammatical ad. Machine generated ads
aren't always bad, just the ones which are nonsensical or
ungrammatical. Please read the full ad creative text.

Rati Descripti Examples

ng on
Yes The ad
text is Query: [del mar food fair]
awkwardl
y phrased
or difficult
to make
sense of.

It is obvious that the word "food" has been taken from

the query and inserted into the ad creative in an
awkward manner resulting in an ad which says "Map
Food". This doesn't quite make sense and this ad is
likely to have been machine generated.

No
The ad Query: [amazon.com books]
text is not
awkwardl
y phrased
or difficult
to make
 sense of. The ad creative is not nonsensical or ungrammatical.

Question 6: What is the relationship between the products,

services or information advertised by the ad to what is specified
in the query?

In this section, we want to know whether the Ad Creative offers what

is specified in the query, whether it offers a viable substitute or if it
doesn't offer a viable substitute or related product at all. 

Rating Descriptio Examples

n
No The
Relatio information, Query: [Google pixel]
nship products or
services
advertised
by the ad
and those
specified in The query is for a Google Pixel phone, but the ad
the query interprets it to be the movie "Pixels" released in
are not 2016. 
related in
any way.
Related The
products/ser Query: [HP printers]
vices or
information
advertised
in ad does
not perform
the same
function as
The ad is for ink for printers. Ink does not perform
what is
the same function as printers but it is necessary
specified in
for printers to work.
the query,
but it
performs a
related Query: [HP printers]
 Query:  [Beach Cities Car Wash Venice]
function.  
They might
be
tangentially
relevant to Car wash detergent is related to car washes
the user. because it is used in car washes.

Viable The Query: [HP printers]

substit products/ser
ute vices or
information
advertised
in the ad
performs
the same
function as Both the ad and the query are for printers but
what is while the query is for HP printers, the ad is for
specified in printers from Epson which is a competitor.
the query
and hence it
can be used Query: [Beach Cities Car Wash Venice]
as a viable
substitute
by the user.

The ad is for a mobile car wash service which

services the Venice, Los Angeles area while the
query is for another service within the Venice, Los
Angeles area.

Query: [macy's shoes]

 Query: [job listings monster]

The query is for job listings on the site

Monster.com, but the ad is for the competitor site
Indeed.com. 

Same
The Query: [plumbing service venice]
product/serv
ice or
information
advertised
in the ad is
exactly for
what is
specified in The ad is exactly for what the query specified.
the query.

Query: [Sony playstation]

 
Question 7: How good is the ad relative to the top 3 search
results?

In this section, we want to know how good the ad is relative to the search
results. Look at the top 3-5 search results to form your judgment. Please
ignore Product Listing Ads (the sliding panel of ads at the top of
the results page) when making your decision.  Below,  the panel of
ads displaying different Nikon digital cameras are Product Listing Ads. 

Some examples for each of the rating categories:

Rati Descrip Examples

ng tion
Both Both the Query: [what is the phone number for Company X]
Ad ad and
and the Company X does not have an online presence. The ad
Sear search and the search results are for another unrelated
ch results company.
Bad are
unlikely
to satisfy
the user
intent.
Ad The
wors search Query: [molecular mass of sulphur]
e results
than are more The ad is for a company that sells industrial sulphur,
while the search results point to resources containing
Sear likely to
ch satisfy information about the chemistry of sulphur.
the user
intent
than the Query: [Food Spain]
ad.
The ad is for a travel company that organizes tours in
Spain. The search results point to webpages containing
information about Spanish cuisine.

Ad The ad Query: [HP printer]

and either
Sear exactly The ad is for an online store that sells HP printers and
ch matches the search results point to pages which seem to contain
good the HP printers.
search
results,
or adds Query: [Ethiopian coffee]
useful
variety The ad is for a company that sells coffee while the
over search results point to webpages that contain different
them. kinds of coffee including Ethiopian.
Both are
likely to
satisfy
the users
intent.

Ad The Ad is
bett likely to Query: [new egg promo]
er be more
than relevant
Sear to the
ch user
than the
search
results. 
Search results:
 The search results are from some other websites while
the ad is from  newegg.com. So, the ad is likely to be
more relevant to the user than the search results.

The Remaining Questions are about the Landing Page (LP)

Question 8: Does the LP have any issues that make it unusable /

infeasible? Select all that apply.

In this section, we want to determine if there are any serious issues that
make it infeasible to satisfy the user's intent. Select all the options that
apply.

Rating Description Examples

Location The landing page is for
mismatc a product/service in a Query: [pizza Santa Monica]
h location too far away
from the user to be Landing page: The LP is for a
feasible. pizzeria in Santa Barbara which is 85
miles away from Santa Monica. It is
unreasonable to expect the user to
travel that far to get pizza.

Languag
e Parts of the landing
mismatc page are in a foreign
h language.
Query: [hotels paris]
Landing page: While large parts of
the LP are in English, the hotel
descriptions within the LP are in
French. This affects the user's ability
to understand content on the page
relevant to their intent.
 Out of The product / service
stock the user is looking for is Query: [NIKE FREE RN FLYKNIT 2017]
out of stock / no longer
available. Landing page: The LP has other Nike
shoes available but not the model
specified in the query. The image /
listing for this specific model is
marked as Out of Stock.

Query: [tickets superbowl 2016]

Landing page: The LP has tickets to

other football games but not the
2016 SuperBowl since it is already
over.

Question 9: Does the LP have any issues that affect the user's
interaction with the page? Select all that apply.

Think about the user's interaction with the landing page when they visit it.
Do they have any negative experiences on any of the following
dimensions? Select all that apply.

Rating Description

Cluttere The LP is cluttered. Text and images are not well formatted
d and the LP is not aesthetically appealing. It is not clear
whether the information the user is looking for is on the page
and the user has to make an effort to locate it.

Interstiti
al / form When the user visits the LP,  they first see an interstitial
(content that prevents the user from accessing the LP) or
a form that is difficult to dismiss or requires extra steps. The
screenshot below shows an example of an interstitial.
 Distracti The LP has ads which attract the user's attention away from
ng ads the main content. For example, flashing ads or too many ads
or ads which are shocking and surprising.
Prods The LP prods the user to stay / makes it difficult for them to
the user leave the page through repeated messages or alerts.
to stay
Bad The LP consists of content which has bad grammar or
grammar awkward phrasing

Question 10: Does the LP have any issues that might undermine
the trust or safety of the user? Select all that apply.

Think about whether the ad is trying to deceive or mislead the user. A

landing page should be considered deceptive even if it technically has
disclaimers/clarifying text, but the text is hard to find, and likely to
confuse the user.

Rating Description Examples

LP-AC
mismatch The LP does not Query: [NIKE FREE RN FLYKNIT 2017]
provide what the ad
creative promises. Ad Creative: The ad creative promises
 Nike shoes. 

Landing page:  The LP doesn't

actually have any Nike shoes.

Pretends
to be The LP falsely claims Query: [tickets LA Opera]
official to be the official
page when it's not, Landing page: The LP falsely claims
or has a disclaimer that they are the official site to buy
on the page but the tickets to the Los Angeles Opera, but
disclaimer is hard to it is not. The official site is
find. actually that of the LA opera.

Query: [iphone repair]

Landing page: The LP falsely claims

that they are the official site for
repairing iphones, but the official site
is actually within the apple.com
domain.

Query: [United Airlines]

Landing page: The LP looks very

much like the official united airlines
page, but has small print at the
bottom of the page indicating that
they are not affiliated with United
Airlines. This could mislead/confuse
users into believing that it is the
official page

Provides The LP provides false

false or misleading Query: [vitamin C]
informatio information based on
n common sense. Landing page: The LP sells vitamin C
supplements and says that these
supplements can help the consumer
lose weight. There is no medical
evidence supporting this claim.

Unexpecte The LP leads to an

d unexpected Query: [songs by band x]
download download without
 explicit permission
from the user.  Landing page: When the LP loads, it
starts downloading an .exe file which
is supposed to play music available
on the site. The user never explicitly
gave permission to download this
.exe file.

Question 11: What is the scope of the concepts covered by the LP

in relation to the concepts covered by the query?

What is a concept?

A concept is a broad term for an idea that represents something, like an

activity (shopping), a thing (an iPhone, a discount) or a class of things
(blue sweaters, US-based companies).

Although an query or a landing page can refer to multiple concepts, focus

on the concepts which are critical to defining what the landing page or
query is about.

For landing pages, this means that you should focus on the main content
of the webpage and not on concepts introduced by interstitials, web
forms, product reviews on the landing page or comments by users on
content on the web page. For example, consider the landing
page https://www.amazon.com/s?field-keywords=Nike+shoes which has
Nike shoes. The main concept that should be considered is that of Nike
Shoes. The other content such as product ratings, comments and reviews,
other ads on the page, links to other products etc. aren't very important.

How do I compare scope?

In this question, you will have to assess whether the scope of the landing
page is the same as the scope of the query and if they differ, in what way
they differ. Consider the important concepts that are associated with the
query, and those that are associated with the landing page. When you
compare scope, look at the overlap between these groups. You
answered this question for the ad creative above, but the answer
need not be the same for the landing page.

Rating Description Examples

No There is absolutely no
overlap overlap between the Query: [Android Jellybean update ]
concepts covered by the
LP and the query. Landing page: The LP offers actual
jellybeans.

Partial There is some degree of

overlap overlap between the Query: [Nike Men's shoes]
scopes of the LP and the
query, but their scopes Landing page: LP contains shoes by
cover fundamentally different companies but not Nike. 
different sets of topics
and one is not contained
within the other. Query: [Frozen poster]

Landing page: The LP contains

posters about Disney movies but
not about Frozen.

There is some degree of overlap

between the LP and query since
both of them cover Disney
movies,  but the LP doesn't contain
a poster specifically for the Frozen
movie. Note that this rating for
the LP is different from the ad
creative rating (for the same
example in the ad creative
section).

Query: [women's sweaters]

Landing page: The LP contains

children's sweaters. 

LP
narrower The range of concepts Query: [Nike Men's shoes]
covered by the LP is
narrower than those Landing page: The LP points to a
covered by the query. page with nike.com and contains
information about returning
purchased nike shoes.

While the LP is about Nike shoes, it

is specifically about returning them
and hence much narrower in scope
 that the query.

Query: [drive thru]

Landing page: The LP points to a

page about services for repair drive
through buildings/machinery.

While this is relevant, it still

focuses on a very small aspect of
drive thru's and hence the scope of
the LP is narrower than that of the
query.

LP The range of Query: [Nike Men's shoes]

broader concepts covered by the
LP is broader than those Landing page: LP contains different
covered by the query. kinds of men's shoes, including
some by Nike. But it also has shoes
by other companies.

Query: [Plumbing services]

Landing page: The LP points to a

page where the user can request
plumbing services, purchase
plumbing related products and
read articles about plumbing.

Same
The range of concept(s) Query: [Nike Men's shoes]
covered by the LP are
exactly the same as Landing page: The LP contains only
those covered by the nike men's shoes. It doesn't
query. contain shoes by other companies
or shoes for women and children.
The LP doesn't cover any
concepts that the query
doesn't cover and vice- Query: [Plumbing services]
versa.
Landing page: LP points to the
home page for a company offering
plumbing services. 
 

Question 12: Does the LP miss a critical requirement for

satisfying the user intent?

Think about all the concepts  and requirements the user has specified in
the query. Now for each requirement, ask yourself how likely the user is to
consider an alternative. If it is very likely that the user will consider an
alternative, then the requirement is non-critical. If missing this
requirement or providing an alternative would be unacceptable to most
users, then it is critical. If the landing page drops both a critical and a
non-critical requirement select "Misses critical requirement" since that
leads to a worse ad experience for the user.  You answered this
question for the ad creative above, but the answer need not be
the same for the landing page.

Rating Description Examples

Misses The LP misses one
critical or more critical Query: [catalonia bicycle tours]
requiremen requirements that
t the user specified Landing page: The LP is about biking
in the query. in British Columbia, Canada.

The user is looking for biking tours in

Catalonia, Spain. While the LP
correctly identifies the biking
requirement, it misses the key
requirement that the biking tour has
to be located in the vicinity of
Catalonia, Spain.

Query: [zipcode mesquite tx]

Landing page: The LP is

sells mesquite smoked wood for grills.

The user is looking for the zipcode for

the location Mesquite, TX. The LP
instead is for mesquite smoked wood.
It therefore missed the key
requirement that the user was
looking for a zipcode and not a
product with the same name.

Misses non- The LP misses one

Query: [macy's womens shoes]
 critical or more non-critical
requiremen requirements that
t the user specified
in the query. Landing page: The LP is for Zappos
and contains women's shoes. 

This LP misses the “macy’s”

requirement specified by the user.
But, this is likely non-critical since the
user can find womens shoes at
zappos as well.

Query: [leather sofas]

Landing page: The LP has fabric

sofas. This misses the non-critical
requirement that the sofas should be
leather. However, it is possible that
the user might willing to accept fabric
sofas.

Doesn't The LP doesn't miss

miss any any requirements Query: [amazon.com]
requiremen that the user
ts specified in the Landing page: amazon.com
query. homepage.

Query: [nike men's shoes]

Landing page: The LP sells Nike Mens

shoes.

Question 13: What is the relationship between the products,

services or information of the LP to what is specified in the
query?

In this section, we want to know whether the landing page offers what

is specified in the query, whether it offers a viable substitute or if it
doesn't offer a viable substitute or related product at all. This is very
similar to the same question above for the ad creative, but consider the
landing page instead. You answered this question for the ad
creative above, but the answer need not be the same for the
landing page.
Rating Description
No The information, products or
Relations services available on the LP
hip and those specified in the
query are not they related in
any way.
Related The product/service or
information in the LP does not
perform the same function as
what is specified in the query,
but it performs a related
function.
Viable The product/service or
substitut information in the LP performs
e the same function as what is
specified in the query and
hence it can be used as a
viable substitute by the user.
Examples
Query: [Sony playstation]
Landing page: The LP is
about the history of Sony the
company and hence does not
offer a viable substitute or a
related product/service for
what is specified in the
query.
Query: [Sony playstation]
Landing page: The LP is
about playstation game
reviews. It does not actually
offer any consoles or games
for sale. 
Query: [zumba classes]
Landing page: The LP is for
fitness classes. A user who is
interested in zumba classes
could be interested in other
fitness classes as well.
Query: [Sony playstation]
Landing page: The LP has
Nintendo Wii consoles and
video games. It does not
contain any Sony
playstations.
Query: [HP printer]
Landing page: The LP has
Epson printers. Epson
 printers are a viable
substitute for HP printers.

Same The product/service or

information in the LP is exactly Query: [plumbing service
what is specified in the query. venice]

Landing page: The LP is for a

plumber servicing the
Venice, Los Angeles area.

Query: [Sony playstation]

Landing page: The LP

contains different video
game consoles such as the
Sony Playstation, the XBox
360 and the Nintendo Wii.

Question 14: Does the LP help the user make progress towards
their goal?

Imagine that the user has entered a query into Google, and is presented
with a page of search results (SERP, Search Results Page) and some ads,
including the ad being rated in this task. The user then clicks on this ad
and visits the landing page. Think about whether the user is substantially
closer to achieving their goal than they were before clicking on the ad.
(i.e. on the google search results page). 

Please click on the query link provided above and look at the Google
search results before answering this question. When you are considering
the Google search results page, please ignore any other ads from this
same advertiser (e.g. Product Listing Ads, these are the “Shop for X”
pictures that sometimes show on top of the page).

Ratin Description Examples

g

Nega Visiting the LP Query: [buy fondant molds]

tive takes the user
Progr farther away
ess from their Before clicking LP:
goal,
compared to
the Google
search results
page. The user
could get to
their
destination
from the
Google search
results page
with fewer
clicks / less
effort than
from this page.

The user will

have to go
back to the
search results
page, or
restart their
task in order to
make
meaningful
progress. 

After clicking LP:

 The user's query was for a baking mold while the
LP is for removing toxic mold from homes.

Query: [Nike shoes]

Landing page: The LP contains a history of Nike

as a company and doesn't contain any Nike shoes
for purchase. There are no links on the LP to
webpages where the user might be able to
purchase Nike shoes.

No Visiting the LP Query: [almay coupons]

Progr takes the user
ess neither closer Before clicking LP:
to, nor further
from their
goal, i.e. it
offers little to
no additional
information,
functionality or
variety that
helps the user
in their task,
beyond what
was already
available on
the Google
search results
page.

It would take
the user the
same amount
of effort to get
to their goal
from the
Google search
results page
that it would
take to get
there from this
page.
  After clicking LP:
 Visiting this LP takes the user neither closer to,
nor further from, their task, since most of the
information on it was already available to the
user pre-click. It adds some new information (on
the right), but the new information added by the
page does not directly help the user in their task. 

Some
progr Visiting the LP Query: [Students part time job Palo Alto]
ess takes the user
somewhat Before clicking LP:
closer to their
goal,  i.e. it
offers some to
moderate
additional
information,
functionality or
variety that
helps the user
in their task,
but the user
has to take
additional
steps (say, use
a menu to
navigate to a
different part
of the website
etc.) to find
what they
were looking
for.

It would take
the user a little
more effort to
get to their
goal from the
Google search
results page
than it would
take to get
there from this
page.
 

After clicking (LP):

 Visiting this LP takes the user closer to their task
by (i) showing additional new job listings beyond
those available before they clicked on the ad and,
(ii) providing added functionality like filtering by
company, distance etc.

Query [Nike shoes]

The LP is the Nike-specific page of an athletic-
clothing retailer. The user would have to click on
the “shoes” subsection to see a list of Nike
shoes. 

Subs
tanti Visiting the LP Query: [buy cannon 70d]
al helps the user
progr make Before clicking:
ess significant
progress
towards their
goal,
compared to
the Google
search results
page i.e. it
offers
substantial
additional
information,
functionality,
or variety that
help the user
accomplish
their task,
beyond what
was already
available on
the Google
search results
page. 

It would take
the user more
effort to get to
their goal from
the Google
search results
page than it After clicking (LP):
would take to
get there from
this page. (For
ex. the
product/servic
e/information
that the user is
looking for is
directly
available on
the current
LP).
 
  

Task 2 Screenshots
 
 

Task 3.1 Query App Match (Google Searches)

Mobile App and Query Evaluation for Google Searches

Version: 2016-11-08

In this task, you will research a mobile app and a query entered in Google
Search. You will then answer a question about how well the app fulfills
what the user is looking for.

Terms

 You already know what a query is:

it is a word or phrase that a real
person typed into a Google search.
 A mobile device is a small,
handheld computing device, such
as a smartphone or tablet.
 A mobile app is a smartphone or
tablet application.

How to rate this task

Step 1: Review and understand the user query.

You will see a query that a user entered. Just as in normal search ads
rating, it will be your responsibility to do your best to understand what the
user was looking for. You will be asked to assume that the user
entered the query in a regular search on Google.com on a mobile
device. Although this evaluation is concerning the relevance of mobile
apps, do not assume that if a user entered a search into Google that they
are searching for an app. Instead, consider the user to be an average
person entering the query into Google search. For more information on
how this is applicable to the task at hand, please see examples in the
table below Step 3.

Unrateable Query

If the query is in a foreign language, is incomprehensible, or otherwise

can’t be rated, select the “Unrateable query” flag and submit the task.

 
Step 2: Familiarize yourself with the mobile app.

Click the button labeled “research mobile app” to open the app’s Google
Play page, then review its content. You should consider all the information
available to you in order to inform your understanding of the app,
including the text that describes the app, ratings, any relevant comments
from users, and even your overall impression of how professional and
trustworthy the app looks. Some additional notes:

 Do not download the app.

 The Play Store page may include
some additional content, such as
links to other parts of the Play
Store, or lists of “Similar” mobile
apps. Please ignore this and only
focus on the content that is related
to the main app on the page.
 You may, however, follow links and
do additional research outside of
the Play Store if it helps you
understand the app.

Unrateable Apps

If the Play Store page or the app it describes is in a foreign language, is

incomprehensible, or otherwise can’t be rated, select the “Unrateable
app” flag and submit the task.

Personal Knowledge of Apps

If you've personally installed and used the app, or believe you have
additional knowledge about the app that would affect your ratings in this
task, select the flag labeled "I am personally familiar with this app." You
will still be required to complete the task.

Step 3: Answer the following question: Would installing and using

the app satisfy the user intent?

Think about what the mobile app is used for and what the user was
looking for. For example, if the app’s purpose is to deliver food and the
provided query contains the word “fruit ninja”, then the user intent is
unlikely to be satisfied by the user installing and using the app. Pay
attention to the app’s quality as well. A good quality app looks
trustworthy, has a well-written description that makes the app’s purpose
clear, and gets positive reviews from users. The user intent is less likely to
be satisfied by a poor quality app.

Taking all of these factors into consideration, select a rating from [-100] to
[100] to indicate the likelihood that a specific user would be satisfied by
installing and using the app.

Rating Description Examples

Satisfaction The user is looking for an app, User query: [flashlight app]

Likely and this one provides exactly
the functionality that the user
is looking for. The app seems
to be of high quality (i.e., it's
not scammy).
If the user is looking for a
specific app but is presented
with a plausible substitute, you
should select this option.
App: Super-Bright LED
Flashlight
The user can turn their phone
into a flashlight with this app.
User query: [caviar food
delivery]
App: DoorDash

While the user is looking for a

specific food delivery app -
Caviar - the DoorDash food
delivery app is a plausible
substitute.

Satisfaction The user might install and use User query: [flashlight]

Possible the app because the query
has multiple equally likely user
intents, namely the user could
be looking for a physical good
or app. The app can only
satisfy one of the plausible
intents.
App: Super-Bright LED
Flashlight
The user is equally likely
looking for a flashlight app as
s/he is looking for a physical
flashlight for sale.

User query: [airline]

App: Kayak

There is no app that could


Dissatisfaction The user probably won't install
Possible the app because the user
intent is unlikely to be met by
the app,  and/or because the
app appears to be of low
quality.
possibly receive "Satisfaction
Likely" for the query [alrline] - it
is simply too vague of a query.
However, a subset of users
might find Kayak, a flight deal
app, useful. (Note: other airline
apps
like Southwest or Delta would
also be Satisfaction Possible).
User query: [facebook]
App: Facebook
The user may be looking for the
Facebook app, but it's also
likely that the just want to log in
through Facebook's website as
quickly as possible.
User query: [install facebook]
App: Snapchat
The user is looking for a
specific app. In some ways,
Snapchat is similar to
Facebook, but it's not close
enough to be a true plausible
substitute.
User query: [board games]
App: Checkers Elite
The user intent is much more
likely to find physical board
games rather than the app-
related version, but it is still
possible that the user might
download the app.
User query: [veterinarian game
app]
App: Vet Clinic
The user likely won’t install this
 app because it is very low
quality: it has bad reviews in
the Play store and the app
description is riddled with
spelling and grammatical
errors.

Dissatisfaction The user won’t install the app User query: [bicycle]

Likely because the app is unrelated
to what they are looking
for, because the app is of
obviously low quality, or
because the query has
multiple possible user intents
and the query is unlikely
referring to an app.
App: Bike Mayhem Free
If a user enters [bicycle] into a
Google search query, it is
highly unlikely that the user is
searching for an app.

If you have questions about this project that are not answered by the
instructions above, please review the Rater Hub first, which contains
additional content about rating. If you encounter a technical problem with
this rating task, use the “Report A Problem" link in the lower-right hand
corner of the rating page.

Task 3.1 Screenshot

Task 3.2 Query App Match (Play Store Searches)

Mobile App and Query Evaluation for Play Store Searches

Version: 2017-01-19

In this task, you will research a mobile app and a query entered on the
Google Play Store. You will then answer a question about how well the app
fulfills what the user is looking for.

Terms

 You already know what a query is:

it is a word or phrase that a real
person typed into a Google search.
However, in this evaluation, we will
 be talking about queries within
the Google Play store.
 A mobile device is a small,
handheld computing device, such
as a smartphone or tablet.
 A mobile app is a smartphone or
tablet application.

How to rate this task

Step 1: Review and understand the user query.

You will see a query that a user entered. Just as in normal search ads
rating, it will be your responsibility to do your best to understand what the
user was looking for. However, unlike the normal search ads rating, it is
important to keep in mind that the query was entered on the Google Play
store. Assume that if a user entered a search into the Google Play
store, that they are searching for apps.

Click the query to open the Google Play store results page, and research
to understand what the user was looking for. You may perform additional
research outside of the Play store well. For example, if the user searches
for [watch mr robot], they are probably looking for a way to watch the USA
Network TV show Mr. Robot, even though the apps that appear in the Play
Store search results are mostly for robot games.

Unrateable Query

If the query is in a foreign language, is incomprehensible, or otherwise

can’t be rated, select the “Unrateable query” flag and submit the task.

Step 2: Familiarize yourself with the mobile app.

Click the button labeled “research mobile app” to open the app’s Google
Play page, then review its content. You should consider all the information
available to you in order to inform your understanding of the app,
including the text that describes the app, ratings, any relevant comments
from users, and even your overall impression of how professional and
trustworthy the app looks. Some additional notes:

 
  Do not download the app.
 The Play Store page may include
some additional content, such as
links to other parts of the Play
Store, or lists of “Similar” mobile
apps. Please ignore this and only
focus on the content that is related
to the main app on the page.
 You may, however, follow links and
do additional research outside of
the Play Store if it helps you
understand the app.

Unrateable Apps

If the Play Store page or the app it describes is in a foreign language, is

incomprehensible, or otherwise can’t be rated, select the “Unrateable
app” flag and submit the task.

Personal Knowledge of Apps

If you've personally installed and used the app, or believe you have
additional knowledge about the app that would affect your ratings in this
task, select the flag labeled "I am personally familiar with this app." You
will still be required to complete the task.

Step 3: Answer the following question: Would installing and using

the app satisfy the user intent?

Think about what the mobile app is used for and what the user was
looking for. For example, if the app’s purpose is to deliver food and the
provided query contains the word “fruit ninja”, then the user intent is
unlikely to be satisfied by the user installing and using the app. Pay
attention to the app’s quality as well. A good quality app looks
trustworthy, has a well-written description that makes the app’s purpose
clear, and gets positive reviews from users. The user intent is less likely
be satisfied by a poor quality app.

Taking all of these factors into consideration, select a rating from [-100] to
[100] to indicate the likelihood that a specific user would be satisfied by
installing and using the app.
Rating Description
Satisfaction The app provides exactly the
Likely functionality that the user is
looking for, even if the app is a
competitor app. The app is of
good quality, and the user
intent would be satisfied by
installing and using it.
Satisfaction The app provides similar
Possible functionality to what the user is
looking for, even if the app is a
competitor app. The user
intent could be satisfied by
installing and using the app
because it is of generally good
quality, even though some of
its features don’t match what
they are looking for.
Examples
User query: [install facebook]
App: Facebook
The user is looking for this
specific app, so this is a perfect
match. It deserves a very high
"Satisfaction Likely" score.
User query: [free games]
App: Snood Free
The user's query is very vague
and therefore any type of free
game would fully satisfy the
user's intent.
User query: [caviar food
delivery]
App: DoorDash
While the user is looking for a
specific food delivery app -
Caviar - the DoorDash food
delivery app is a plausible
substitute.
User query: [clothes]
App: Amazon Shopping
The user can buy clothes via
the Amazon app, but the user
would probably prefer
something more specific to
fashion or clothes shopping.
User query: [airline]
App: Kayak
There is no app that could
possibly receive "Satisfaction
Likely" for the query [alrline] - it
is simply too vague of a query.
However, a subset of users

Dissatisfaction The app provides features that
Possible are related to what the user is
looking for, but they’re unlikely
to be satisfied by installing and
using it because it’s not a very
good match and/or because
the app appears to be of low
quality.
might find Kayak, a flight deal
app, useful. (Note: other airline
apps
like Southwest or Delta would
also be "Satisfaction Possible").
User query: [facebook]
App: Snapchat
The app is of good quality and
offers many of the same
features the user searched for.
User query: [youtube]
App: YouTube Music
The YouTube app or other
similar video apps would
deserve "Satisfaction Likely" for
this query, but because this
app offers similar functionality
that's more limited to music, it
deserves "Satisfaction
Possible."
User query: [taxi]
App: Taxis of Houston
The app would fulfill the user
intent if the user were in
Houston, but without
approximate query location
information to ensure that's
true, the app deserves
"Satisfaction Possible" instead.
User query: [3d games]
App: Hill Climb Racing
The user is looking for a 3d
game, but the Hill Climb Racing
app, while a game, is not 3d
and is therefore not what the
user wanted.
 User query: [gmail]

App: Hangouts

The user is looking for a

specific email app. While Gmail
and Hangouts are both used for
communication, they have very
different feature sets.

Dissatisfaction The app is unrelated to the User query: [games]

Likely user intent and/or the app is of
obviously bad quality. The
user intent will definitely not be
satisfied by installing and
using the app.
App: NYTimes - Latest News
The user is looking for a game
app, not a newspaper app like
the NYTimes.

User query: [free game app]

App: Spotify

The user is looking for a free

game app. While Spotify - a
music streaming service - might
be entertaining for the user, it's
unlikely that installing and using
the app would satisfy the user's
intent. The user is looking for a
gaming app, not a music
streaming service.

If you have questions about this project that are not answered by the
instructions above, please review the Rater Hub first, which contains
additional content about rating. If you encounter a technical problem with
this rating task, use the “Report A Problem" link in the lower-right hand
corner of the rating page.

Task 3.2 Screenshot

Task 3.3 Query App Match (YouTube Searches)

Mobile App and Query Evaluation for YouTube Searches

Version: 2017-04-04

In this task, you will research a mobile app and a query entered in
YouTube. You will then answer a question about how well the app fulfills
what the user is looking for.

Terms

 You already know what a query is:

it is a word or phrase that a real
person typed into a Google search.
However, in this evaluation, we will
be talking about queries
within YouTube.
 A mobile device is a small,
handheld computing device, such
as a smartphone or tablet.
 A mobile app is a smartphone or
tablet application.

How to rate this task

Step 1: Review and understand the user query.

You will see a query that a user entered. Just as in normal search ads
rating, it will be your responsibility to do your best to understand what the
user was looking for. However, unlike the normal search ads rating, it is
important to keep in mind that the query was entered on YouTube.

Click the query to open the YouTube results page, and research to
understand what the user was looking for. You may perform additional
research outside of the YouTube as well.

Unrateable Query

If the query is in a foreign language, is incomprehensible, or otherwise

can’t be rated, select the “Unrateable query” flag and submit the task.

Step 2: Familiarize yourself with the mobile app.

Click the button labeled “research mobile app” to open the app’s Google
Play page, then review its content. You should consider all the information
available to you in order to inform your understanding of the app,
including the text that describes the app, ratings, any relevant comments
from users, and even your overall impression of how professional and
trustworthy the app looks. Some additional notes:

 Do not download the app.

 The Play Store page may include
some additional content, such as
links to other parts of the Play
Store, or lists of “Similar” mobile
apps. Please ignore this and only
focus on the content that is related
to the main app on the page.
 You may, however, follow links and
do additional research outside of
the Play Store if it helps you
understand the app.

Unrateable Apps

If the Play Store page or the app it describes is in a foreign language, is

incomprehensible, or otherwise can’t be rated, select the “Unrateable
app” flag and submit the task.

Personal Knowledge of Apps

If you've personally installed and used the app, or believe you have
additional knowledge about the app that would affect your ratings in this
task, select the flag labeled "I am personally familiar with this app." You
will still be required to complete the task.

Step 3: Answer the following question: Would installing and using

the app satisfy the user intent?

Think about what the mobile app is used for and what the user was
looking for. There will inherently be a mismatch between searching on a
video site and what apps can offer. For example, if the user is searching
for a [baked alaska recipe], the user wants to see video tutorials on
making Baked Alaska. An app of dessert recipes would probably be
Satisfaction Possible/Satisfaction Likely, whether or not it includes videos.
Pay attention to the app’s quality as well. A good quality app looks
trustworthy, has a well-written description that makes the app’s purpose
clear, and gets positive reviews from users. The user intent is less likely
be satisfied by a poor quality app.

Taking all of these factors into consideration, select a rating from [-100] to
[100] to indicate the likelihood that a specific user would be satisfied by
installing and using the app.

Rating Description Examples

Satisfaction The app provides exactly the User query: [how to file taxes]

Likely functionality that the user is
looking for, even if the app is a
competitor app. The app is of
good quality, and the user
intent would be satisfied by
installing and using it.
App: TurboTax
The user is looking for a
tutorial/information on how to
file taxes, and is provided an
app that guides a user through
filing taxes.

User query: [formation
beyonce]

App: Spotify

The user's YouTube search is

ambiguous because the user
may or may not care about
watching the music video. If
the user is trying to find a way
to listen to the song, the
Spotify app can satisfy that
need.

User query: [learn spanish]

App: Duolingo

The user is trying to find a

Spanish language tutorial and
the app provides resources to
learn Spanish.

Satisfaction The app provides similar User query: [full episode of

Possible functionality to what the user is
looking for. The user intent
could be satisfied by installing
and using the app because it is
of generally good quality, even
though some of its features
don’t match what they are
looking for.
Dissatisfaction The app provides features that
Possible are related to what the user is
looking for, but they’re unlikely
to be satisfied by installing and
using it because it’s not a very
good match and/or because
the app appears to be of low
quality.
fargo s1e4]
App: Hulu Plus
There is no app that could
possibly receive "Satisfaction
Likely" for this query since the
user is likely looking for
illegal free content on
YouTube. Even though the
user's query cannot fully be
satisfied in a legal way,
installing and using the Hulu
Plus app will mostly satisfy the
intent.
User query: [weight loss tips]
App: Lose It!
The user is looking for a video
that contains weight loss tips
and the app has strategies for
setting and accomplishing
weight loss goals, but is more
of a fitness/calorie tracker than
a repository of tips.
User query: [macaroni and
cheese recipe]
App: Whole Foods
The user is looking for visual
tutorial of how to make
macaroni and cheese. The
Whole Foods app has some
recipes and coupons for the
grocery chain that could help
the user move closer to his/her
goal, but only a small subset of
users would be satisfied with
this app recommendation.
User query: [vinyasa yoga
class]
App: MindBody

Dissatisfaction The app is unrelated to the
Likely user intent and/or the app is of
obviously bad quality. The user
intent will definitely not be
satisfied by installing and using
the app.
If you have questions about this project that are not answered by the
instructions above, please review the Rater Hub first, which contains
additional content about rating. If you encounter a technical problem with
The user is looking for a yoga
class to follow along with, and
the app is a yoga class
reservation system. While the
yoga reservation system can
help the user find a yoga class,
only a small subset of users
would be satisfied with this
app.
User query: [eyebrow shaping
tutorial]
App: Makeup Genius
The user is looking for a
tutorial on shaping eyebrows
and the app allows the user to
apply different makeup looks
on him/herself. Only a small
subset of users would be
satisfied with this app.
User query: [dogfight plane]
App: Modern Dog Fighting
The user is looking for videos
of aerial combat, not a gaming
app.
User query: [donald trump
press conference]
App: Driving President Trump
The user is looking for a press
conference video of Donald
Trump, not a gaming app.
this rating task, use the “Report A Problem" link in the lower-right hand
corner of the rating page.

Task 3.3 Screenshot

Task 4 Best Match Keyword Evaluation (User to Keyword)

User to Keyword Evaluation

In this project, you will evaluate whether a particular user is likely to be
interested in things related to a keyword. You will assess the user’s
current interests by evaluating the cached content of a page they are
visiting. You will then evaluate a keyword and determine whether this
user is or might be interested in things related to the keyword.

The Task

Evaluating the User's Current Page

Open the user’s current page using the Open cached page HTML button.
You should review the textual content on the cached page until you have
a sense of the main ideas and concepts on it.

Because you are reviewing the cached page HTML, the user’s page will
have most of the non-textual content stripped out. So, the layout may be
affected, and you may have to scroll to thoroughly evaluate the page.

You may also see an Open cached page image button. You can use the
cached page image for context or in order to help you understand how the
page is intended to be displayed, which will help you find the main
content. However, you should base your evaluation of the page on the
cached page HTML.

Boilerplate text like the navigation menu and any text not related to the
main content should be considered as contributing to minor topics of the
page.

If the main content of the content page is an image or video, please look
for textual information associated with the image or video that explains
what it is about. This text could be a title, caption, or short description. Do
not consider comments when viewing YouTube pages.

NOTE: If a cached page is not available for this task, please use the live
page for rating. There will be a note below the live page button if this is
the case.

Part 1: Page Classification

First, select one of the following classifiers that best describes the page’s
content:

Classifie Definition: Examples:

r:

 A news article
Topical The main content of the page is  A sports forum
Content primarily focused on a single topic or a  A blog mostly
small number of topics.  Additional
 about cooking
secondary, minor, or sidebar topics are
okay, as long as the page is mostly
about one or a few topics.

 A portal page
Diverse The main content of the page is  The homepage
Content divided among several or many of a newspaper
different topics.  A list of links or
articles that are
not focused
around a single
subject area
 A list of
products for sale
that are not
focused

 Song lyrics
Fictional The main content of the page consists  Excerpts from a
Content of fictional (non-real) material. novel
 Fan fiction
 Video game
walk-through

 Videos
Low- The main content of the page contains  Images with
Text very little or no text. minimal captions
Content  Comics

Part 2: User Assessment

Now that you have thoroughly reviewed the user’s current page, you can
develop a profile of the user’s current interests. Think about the types of
products, services, ideas, and activities a user who is visiting this page is
seeking or might be interested in (i.e. current content of the page, as well
as anything related to the content on the page).

Evaluating the Keyword

A keyword consists of one or more words that represent one or more

concepts. Review the keyword and consider its meaning and scope. What
things does the keyword represent? You can research the keyword to
augment your understanding, but remember that the keyword is not a
query; the keyword represents concepts and not a specific user intent.

Now that you have a good understanding of the keyword’s scope and
meanings, recall your profile of the user’s current interests. If the keyword
contains a specific location, you can assume the user belongs to that
location:

Given the user’s current interests, is the user likely to be

interested in things related to the keyword?

Choose your rating based on these categories and descriptions:

Very Likely

The keyword represents or captures products, concepts, or services the

user is currently interested in.

Somewhat Likely

The keyword represents or captures products, concepts, or services

related to the user’s current interests, or that may appeal to the user
given their current interests; and/or: It is plausible that the user might
become interested in things related to the keyword, perhaps impulsively,
even though the connection with the keyword is somewhat weak.

Unlikely

The keyword captures products, concepts, or services weakly related to

the user’s current interests. It is unlikely the user is interested or would
become interested in things related to the keyword, but there is still a
weak connection.

Very Unlikely

There is no reason to believe that the user might be interested in things

related to the keyword more than any random internet user. There is no
connection at all between the user’s current interests and things related
to the keyword.

Please carefully review the below examples to calibrate your ratings:

User's Current Keyword: Rating: Explanation:

Page:
healthcare facilities healthcare Very Likely The user’s
expo/conference info technology page shows
information
about a
healthcare
conference for
 the hospital
and medical
equipment
industry, so we
already know
that the user is
currently
interested in
things related
to the
keyword,
“healthcare
technology”.
reverse phone lookup cellphone directory Very Likely The user is
lookup interested in
reverse phone
lookup and the
keyword
captures very
similar
services, so we
already know
that the user is
interested in
things related
to the
keyword.
table of prayer times muslim fashion Somewhat The user is
for Muslims in Likely viewing Muslim
Chicago prayer times,
so we know
that the user
may be
interested in
certain Muslim-
related things
in general,
given their
current
interests. The
keyword
"muslim
fashion" falls
into this
category; it is
related to the
user's current
 interests and
represents
things that
may appeal to
this user.
hotmail signup page business email Somewhat The keyword
Likely represents
products/servic
es related to
the user's
current
interests. The
keyword
captures
products and
services that
may appeal to
this user, who
we know is
interested in
creating an
email account.
math worksheet for multiplication Somewhat The user is
kids about 3D shapes Likely currently
interested in a
kids' math
lesson about
shapes, so it is
reasonable to
interpret that
things related
to
"multiplication"
might appeal
to this user.
listicle "10 movies mobile phone Unlikely The user is
that couldn't have advertising reading a fun
happened without article about
cellphones" movies, so
their current
interests are
not primarily
related to
"mobile phone
advertising",
and it is not
plausible that
 they might
become
interested in
things related
to that
keyword given
their current
interests.
However, since
there is still a
weak
connection
since the
article
mentions
cellphones, so
we use
'Unlikely'
rather than
'Very Unlikely'.
question on an exam Aspen chartered Unlikely There is a
for aircraft personnel flight weak
connection
between the
scope of the
keyword and
the user's
current
interests
(flights,
aircraft, etc.)
but it is
unlikely that
things related
to "Aspen
chartered
flight" would
appeal to this
user, who is
interested in
an
employment
exam.
job listings page jcpenney Very The user is
(keyword not on Unlikely interested in
page) job listings.
There is no
 reason to
believe that
the user would
be interested
in things
related to a
department
store
(shopping,
clothing, etc.)
more than any
random
internet user.
There is no
connection
between the
scope of the
keyword and
the user's
current
interests.
article about rennet free online Very The user is
investing Unlikely reading an
article about
rennet in
cheese
making. There
is no reason to
believe the
user would
interested in
things related
to "free online
investing"
more than any
random
internet user.
The scope of
the keyword
does not
overlap with
what we can
deduce about
the user's
current and
possible
interests, and
there is not
 even a weak
connection.
youtube music video plus size tops Very The scope of
(drake bell) women Unlikely the keyword
and the user's
current
interests don't
overlap at all;
there is no
reason to
believe the
user would be
interested in
things related
to this keyword
more than any
random
internet user.

Task 4 Screenshot

Task 5 User Interest Evaluation

User Interest Evaluation

In this project, you will evaluate whether a business would want to

advertise to a particular user. You will first assess a business’ products,
services, and customer base by reviewing a business page, and then you
will assess the user visiting a separate webpage. You will be able to
determine whether the business would want to advertise to the user by
comparing the user’s current interests and state of mind to the business’
products, services, and target customers.

The Task

Business Assessment

First, visit the business page and assess the following:

(1) What products or services is this business selling?

(2) Who purchases these products or services (i.e. who are the customers
of this business)?

Think carefully about the customer base. Consider whether the business
appeals to a broad set of customers, or a very narrow set (such as a
business-to-business seller). Consider also whether customers need to
spend a lot of money to purchase these products or services, or whether
the prices are more broadly accessible.

You may or may not see a list of the top search queries for the business
page, depending on whether this data is available. These are typical
search queries by which users have chosen to navigate to the business
page. You can use these to supplement your understanding of the
business, if doing so is helpful.

User Assessment

Next, visit a user’s current webpage. You should develop a profile of the
user’s current interests and state of mind by reviewing the page the user
is visiting. Try to think about the types of products, services, and activities
a user who is currently visiting this page is seeking or might be interested
in. Think about the user’s state of mind and the various things they are
interested in, and even the kinds of things they might become interested
in impulsively while in this current state of mind.

Now that you have a good understanding of the user’s current interests
and state of mind, recall your profile of the business’ customer base and
its products or services. If the business is targeting customers in a specific
location, you can assume the user belongs to that location:

Given the user’s current interests and state of mind, how

likely is it that the business would want to advertise to this
user?

Choose your rating based on these categories and descriptions:

Likely
There are good reasons to believe the user is researching the type of
products or services offered by the business. The user appears to match
the type of customer targeted by the business.

Possible

It is plausible that the user might become interested in the products or

services offered by the business, perhaps impulsively, even though the
connection with the business is weak.

Unlikely

There is no reason to believe that the user might be interested in the

business more than any random internet user. It is unlikely that the user is
the type of customer targeted by the business, or that the user is in the
correct state of mind to consider the products or services of the business
as an impulse buy.

Please carefully review the below examples:

Business Page: Business’ Us

Top
Search
Queries:
http://www.cargurus.com/ [new cars], htt
[search oul
cars for
sale], [buy
used cars]

http://www.discoveryacademy.com/admissions/assessment/ [boarding htt

schools], m/
[military
school for
troubled
youth],
 [free boot
camp for
teens],
[child
behaviour
problems]

http://producttestingusa.com/13/freebies-samples free stuff], htt

[freebies],
[free
samples],
[free
product
samples]

http://www.scorpionwindowfilm.com [residential htt

window lite
tinting], win
[residential
window
tint], [home
window
tinting],
 [window
tint for
home],
[privacy tint
for home
windows]

http://www.fitness19offers.com/CALIFORNIA/CITRUSHEIGHTS/0enroll [fitness 19], htt

ment.htm [fitness 19 ren
citrus cal
heights], spe
[crunch
fitness
citrus
heights],
[cheap gym
membershi
p], [gym
membershi
p deals]
http://creditcards.citicards.com/usc/Hilton/HHonors/dual/2015/Oct/P [hilton htt
S/dual.html hhonors cre
credit card],
[hilton
credit card],
[hilton visa
card]

https://www.247lendinggroup.com/apply.htm [online cash htt

advance], 9_s
[bad credit
loans],
[installment
loans],
[quick loans
online]
http://www.powdersize.com [hammer htt
mill for mil
sale],
[hammer
mill],
[hammer
mill rental],
[custom
particle
reduction],
[pin mill]
http://www.gelatoproducts.com [frozen htt
yogurt], bee
[gelato
supplies],
[how to
make
frozen
yogurt],
[ice cream
games],
[gelato
supply]

http://thefez.com/ [halls for htt

wedding jus
 receptions], sur
[venues in
saxonburg
pennsylvani
a], [barn
wedding
venues in
pittsburgh]

Task 5 Screenshot

Task 6 Close Match Evaluation

In this project, you will be looking at keywords and queries, making

judgments about how their intents are related, and deciding how close in
meaning the keywords are to the queries.

What are keywords and queries?

You already know what queries are: a user query is the set of words that
a user enters into the Google search engine. User intent is what the user
hopes to accomplish by using the Google search engine.

Keywords are words and phrases that advertisers choose in order to

have their advertisements to show up alongside particular Google search
results. Despite the name, keywords don't have to be single words. Most
of the keywords you encounter in this project will contain multiple
words. Advertiser intent is what the advertiser hopes to accomplish by
placing their ad alongside a query that matches the keyword they’ve
chosen. Advertisers often formulate keywords very similarly to how users
enter search queries, and expect their advertisements to show on queries
that have a very similar meaning to their specified keyword.

You should research the keyword and query to form an opinion about
what the advertiser and user meant by them, and what each may have
been looking for.

Task Description

In each task, you will be presented with a keyword and a query. You will
be asked to decide what these terms mean, and how their user and
advertiser intents are related to each other. We want to know whether
advertisements that are relevant to the keyword will also be relevant to
the query.

Before you begin, research each term by clicking on it to look at the

Google search results page for that term. If a keyword or query has
several plausible meanings, focus on the one where the intent makes the
most sense within the context of the keyword-query pair as you answer
the questions in the task.

When you are ready,  you will be asked to define how the intent of the
keyword and the intent of the query are related.

If you are not able to answer this question because the keyword or query
is unclear, ungrammatical, missing or in a foreign language,  select the
'unrateable' checkbox next to the keyword and query. You will then be
asked to identify whether the problem lies with the keyword, the query, or
both.

How are these two terms related?

Your task is to determine how the keyword and query are related. There
are five options:

Rating Rating Description

Same intent The keyword and query have the same intent. They refer to
the same product, service or concept possibly using
 Rating Rating Description
synonyms, paraphrases, or implied terms.
The query is a more specific case of the keyword. The query
Query more adds a more specific, non-implied, aspect, product sub-type,
specific location, or brand, but preserves the full intent of the
keyword. Nothing is more specific in the keyword.
The keyword is a more specific case of the query. Some
Keyword aspect of the keyword is not reflected in the query: the query
more may be missing a non-implied brand, location, or product
specific specification that is present in the keyword. Nothing is more
specific in the query.
The keyword and the query have related but different intents.
Related but For example, they are about related but different products.
different The query is at the same time more and less specific than the
intents keyword. The query refers to an accessory or service for the
keyword (or the other way around).
The keyword and the query have completely unrelated
Unrelated
intents. For example, they represent items from completely
intents
different non-overlapping product areas.

Below you will find additional guidance and specific examples that will
help you correctly categorize tricky and unclear cases.

Option 1: Same intent

Select this option if the keyword and the query have the same intent. For
this task, this does not mean that the terms need to contain all the same
words or be phrased in the same way: as long at the keyword and
query refer to the same product, service, or concept they should
be rated as having the same intent.

You should pick this option if you think the keyword and query pick out
the same things as depicted in the picture below. The query, fruit, and
the keyword, fruit, pick out the same set of items.

Not every pair you rate will be as easy as the pair above. Below we review
several common types of Query-Keyword pairs that have the same intent.
Sometimes you will encounter a mix of the cases covered below. In this
case, you should still select this option.
1) Synonyms: the keyword and query express the same intent
using different words, for example, synonyms or paraphrases.

Keyword Query Explanation

hazard The query may use synonymous words to refer
safety shoes
shoes to the same.
doctors for
pediatrician The query is a paraphrase of the keyword.
children
diaper bag for men diaper The keyword paraphrases the query with
a man bag additional words.
The keyword and query are synonyms; they
climate global
use different words but target the same
change warming
concept.

2) Identical Search Intent: the keyword and query target the

same information and express essentially the same search intent.

In the table below are some examples of keywords and queries that have
very similar search intents. These keywords and queries target the same
information even though they are not exact paraphrases of each other.

Keyword Query
male infertility low sperm how to get pregnant with low sperm
count count
romantic music online love songs online playlist
global map turkey turkey's location on the world map

Note: You can usually tell that the keyword and query express the
same search intent by checking the actual search results for
each. However, you should be careful since sometimes queries
and keywords that have different search intents show similar
search results. For example, 'flights la ny' shows similar results to
'flights ny la' even though the search intents are different.

3) Implied Terms: the keyword contains an additional word or

words that are always or almost always implied by other terms in
the query (or the other way around).

In the table below you can see keywords or queries that should be rated
as having the same intent even though the keyword or query has
additional words that are not included in the other.

Keyword Query Explanation

toyota camry camry
houses for houses for rent in
The Toyota Camry is a specific car
model. It is fine to drop the brand
"toyota" here since it is implied by the
model name "camry".
Since Colville is in WA (Washington
 Keyword Query Explanation
rent in Colville Colville wa State), "wa" is implied by Colville.
Since all daydream headsets are vr
daydream daydream vr heads
(virtual reality) headsets, the term "vr"
headset et
is implied in the keyword.

4) Implied Intent: the keyword has the same implied intent as

specified by the query (or the other way around).

A keyword or query has an implied intent if a user searching for a concept

does not need to specify that intent to achieve his or her goal.  

Keyword Query Explanation

Since "hydrographics" is a complex
topic, the query, which asks for an
hydrographics ho
hydrographics explanation of hydrographics, should be
w it works
treated as having same intent as the
keyword.
A user searching for "health insurance
what is health marketplace" probably wants to find out
health insurance
insurance details about it, so the keyword which
marketplace
marketplace offers a definition should be treated as
expressing the same intent.
Since vacation homes are almost always
vacation
vacation homes rented, the keyword and query have the
home rentals
same intent.
A user looking for "games online" likely
play games wants to play games online, so the
games online
online keyword and query have the same
intent.
When a user searches for gifts (in this
christmas case 'christmas gifts'), they are likely to
christmas gifts
gift ideas want gift ideas,  so this query has the
same intent as the keyword.

 5) Commercial Intent: the query expresses the intent to purchase

the product or service referred to by the keyword (or the other
way around).

Keyword Query
buy pro tools 11
pro tools 11 download
download
where to
rx bars
purchase rx bars
hyundai accent hyundai accent price
adidas shop adidas
Note: if the keyword or query expresses a specific price point, for
example "cheap" or "discount", that is not present in the other
you should not select this option. For example the keyword or query
"handbags" includes expensive handbags like Louis Vuitton handbags, but
the keyword or query "discount handbags" would exclude expensive
handbags. Therefore, you should treat "discount handbags" as more
specific than "handbags".

Examples that don't fall into this option (same intent).

Below you will find some additional tricky examples that seem like they
might belong in this category, but don't.

Keyword Query Correct Rating Explanation

Although the addition of the
The query is word 'used' implies commercial
used acer more specific intent, it also implies that the
acer laptop
laptop than the user does not seek new laptops,
keyword. which is included in the
keyword.
Although both the keyword and
query both refer to trains
The keyword
train from train sj  to between Tahoe and sj (San
and query
tahoe to sj tahoe Jose), the keyword and query
are related.
have different directions of
travel.
The keyword and query are
about different products: the
The keyword
iphone with c keyword is about iphone cases
iphone  case and query
ase while the query is primarily
are related.
about iphones (that come with
cases).

Option 2: Query more specific

This option applies if the query adds a more specific aspect (for example,
a product sub-type, location, or brand) to the keyword, but preserves the
main intent of the keyword.

You should pick this option if you think the query picks out a narrower set
of things than the keyword as depicted in the picture below. The
query, apples, picks out a narrow set of items than the keyword, fruit.
Again, the case depicted above is simpler than many cases you will see.
Below we review several common types of queries and keyword that
should fall in this category.

1) Product Subtype: the query is a specific product subtype of the

keyword.

Keyword Query Explanation

office Since an office desk is a type of office furniture, the
office desk
furniture query is more specific than the keyword.
women's
shoes "women's boots" belongs to the category "shoes".
boots

2) Additional Aspect: the query has an additional non-implied

aspect to its intent, but still preserves the main intent of the
keyword.

Keyword Query
aftermarket auto cheap after market auto
parts parts
wineries in long best long island
island vineyards
lunch recipes easy lunch recipes
buy printer color printer
micro sd 64gb micro sd card
buying coffee
coffee beans
beans online
play games play games for free

3) Specific Product Intent: the query adds specific intent directly

related to the product or service referred to by the keyword. 

Common additional intents include users looking for reviews,

comparisons, coupons, or discounts on a specific product or service.
 Keyword Query
hp hp
notebook notebook reviews
credit credit
cards card comparisons
oil change oil change coupon

Note: if the keyword describes a product and the query intent

shifts from the product to a service or accessory for that product,
you should treat them as having related but different intents
(option 4). For example ,"iphone" and "iphone case" (accessory), "tesla"
and "tesla charging station" (service), and "laptop" and "laptop scratched
screen" (problem) are related but have different intents. 

4) Specific Product Information: the query requests specific

information about the properties of the product, service or topic
that the keyword refers to.

Keyword Query Explanation

how long does itThe user query asks for specific
learn
take to learn the
information about the aspect of the
piano
piano service offered by the keyword.
The keyword is about a specific product,
hp hp notebook screen
and the query requests information about
notebook  size
a specific aspect of that product.

Note: if the keyword describes a product or service and the

information requested by the query shifts the intent  from the
product or service to a problem with that product or service, you
should treat them as having related but different intents (option
4). For example, an advertiser keyword "hp notebook" intends to offer "hp
notebooks" while a user query "hp notebook won't turn on" requests help
with a specific problem with an "hp notebook". The problem the user has
is related to product, but they are distinct enough that they should be
treated as related.

Similarly, if the query shifts the intent from a product to an

accessory or service related to the product, you should treat
them as having related but different intents (option 4). For
example, if the keyword is "hp notebook" and the query is "hp notebook
power cord" (an accessory) or "hp notebook repair" (a related service) ,
you should mark the keyword and query as having related but different
intents.

5) Request Additional Information: the query formulates a specific

request or question about the problem, topic, or question that
the keyword refers to.
If the keyword offers a general problem, topic, or question and the query
asks a specific question or makes a specific request about the
keyword, you should select this option.

Keyword Query
child do i have enough to gain full child
custody custody
what do you do for fleas on a 2 week old
puppy fleas
puppy

 Note: this is only applicable if the keyword describes a problem,

topic, or question. If the keyword describes a product or
service, this does not apply.

6) Specific Location: the query adds a location to the keyword or

specifies a more specific location than the keyword.

Keyword Query Explanation

doctors near The query specifies doctors in a location
doctors
me while the keyword is for doctors generally. 
Since Dallas is a city in Texas, the query is
texas apartme dallas apartme
asking for a more specific location than
nts nts
the keyword.

7) Specific Brand: the query adds a specific brand, business, or

merchant to the keyword or specifies a more specific brand,
business, or merchant than the keyword.

Note: You should treat a query for a brand, business, or merchant to be

more specific than the a keyword for the main product or service the
brand, business, or merchant is primarily associated with. However, if a
brand or merchant is associated with many types of products (for
example, Amazon is associated with a wide range of products),
you should not consider it as more specific than any of its particular
products (for example,  keyword 'books').

Keyword Query Explanation

epson printe
printer repair The query specifies the brand of printer to fix.
r fix
general chevrolet ca Chevrolet is a specific division of General
motors cars rs Motors.
The query is about the merchant 'rockauto'
auto parts rockauto
that mainly sells autoparts.

Examples that don't fall into this option (query more specific).

Below you will find some additional tricky examples that seem like they
might belong in this category, but don't.
Keyword Query Correct Rating Explanation
Although one could complete the
keyword with "chromecast" to get
set the query, there is no way to know
The keyword is
set up up chromec that the advertiser intended this.
unrateable.
ast We can't know what the advertiser
meant because the keyword is
incomplete.
Although, the query only adds
The keyword words to the keyword, the query
cast
cast iron and query are shifts the intent from a product
iron repair
related. (cast iron) to a service (repair)
related to the product.
Although the query adds words to
The keyword
2015 the keyword, the additional words
2015 and query have
truck for only specify a commercial intent,
truck the same
sale which can be assumed in the
intent.
keyword.
The query "target" picks out a
The keyword
merchant that sells many products,
and query have
laundry including laundry baskets.
target related but
basket However, since Target does not
different
primarily sell laundry baskets, you
intents.
should chose Related.

Option 3: Keyword more specific

This option applies if an aspect of the keyword is not reflected in the

query. The missing aspect may be a brand, a location, a product
specification or other additional information.

You should select this option if the query picks out a broader set of things
than the keyword as depicted in the picture below. The keyword, apples,
picks out fewer items than the query, fruit.
When rating, you should consider the same aspects of the keyword and
query that you would take into account when deciding whether the query
is more specific than the keyword (option 2) above.

If the keyword does not preserve the main intent of the query, do not
choose this option: instead mark the pair as having related but different
intents (Option 4) or unrelated intents (Option 5).

Option 4: Related but different intents

This is the case for example if the keyword and query are about related
but different products, or if the query is at the same time more and less
specific than the keyword.

One way for a keyword and query to be related is if they pick out
overlapping sets of items. For example, the query, cats, and
keyword, happy animals, pick out different but overlapping items in
the picture below.

Another way a keyword and query can be related is if they pick out

entirely different sets of items that still have something to do with one
another. For instance the query, orange juice, and the keyword, orange,
should be treated as related because orange juice is made
from oranges. 

Below we review several types of queries and keywords that should fall
into this category.
1) Different Products: the keyword and query specify related but
different products, services, problems, or topics.

Keyword Query Explanation

Keyword and query are different models of
macbook pro macbook air
the same brand.
The keyword asks for the service "printing"
large format used wide
while the query asks for the product
printing format printers
"printer".
super size
sofa recliner One product is a sofa, the other a chair.
recliner

2) Accessory/Service Shift: the keyword shifts the intent

expressed in the query, for example, the keyword refers to an
accessory or a service for the query's product (or the other way
around).

Keyword Query Explanation

The query is an accessory for the product
macbook macbook pro
referred to by the keyword, so these have
pro keyboard
related but different intents.
The query "toyota" implies that the user is
toyota car looking for toyota cars. The keyword "toyota
toyota
parts car parts" picks out accessories for Toyota
cars.
The keyword refers to a service related to the
desktop pc
desktop pc  product referred to by the query, so these
repair
have related but have different intents.
The keyword offers the a hairdressing service
training in
hairdressers in NYC (New York City). The query is for a
hairdressing
in nyc related service, namely training to become a
nyc
hairdresser in NYC.
The keyword refers to granite countertops
granite
granite while the query specifies that the user is
countertops
countertops looking for images of granite countertops. The
images
intents are related but different.

Note: If the query shows that the user's main intent is focused on
the product in the keyword itself, then the query is more specific
than the keyword (option 2). For example, if the user is looking
for reviews of a product or service, wants to compare the product
or service to other alternatives, or is looking for coupons,
discounts, or delivery of the product, the query is more specific
than the keyword.

3) Incompatible Aspects: the keyword and query refer to the

same product, service, problem, or topic but include incompatible
locations, brands, merchants, or other properties.
 Keyword Query
italian restaurants
italian restaurants sf
LA
nike shoes reebok shoes
refugee crisis refugee crisis
solutions causes

4) More and Less Specific: the keyword is at the same time more
and less specific than the query.

Keyword Query Comment

arthritis of
arthritis of the "cervical" is not reflected in the query.
the spine
cervical spine "surgery" is not reflected in the keyword.
surgery
first aid first aid
"children" is not reflected in the keyword.
certification children"certification" is not reflected in the query.
The keyword offers used campers that may or
campers for
used campers may not be in Virginia, while the query is for
sale in
for sale campers in Virginia that may or may not be
virginia
used. 

5) Direction Switching: the keyword and query refer to the same

action, but switch directions.

The instructions for Option 1 (same intent) discussed cases in which the
keyword and query can use different words or different orders of words to
express the same intent, but sometimes you will encounter keywords and
queries that use the same or similar words, but express different intents.
This is often the case when the order of the words in the keyword or query
imply a direction.

Keyword Query Explanation

Both the keyword and query are looking for flights
turkey to between Toronto and Turkey, the keyword picks
toronto
toronto out flights from Turkey to Toronto, but the query
turkey flight
flight picks out flights from Toronto to Turkey, so the
intents are different.
rupee to today's The keyword and query specify different
dollar rate dollar rate in directions of the exchange rate between Rupees
today rupees and Dollars.
translate translation The keyword and query specify different
french to german directions for the translation between French and
german french German.

Option 5: Unrelated intents

Select this option if the keyword and query have completely unrelated

intents.
Select this option if the keyword and query represent non-
overlapping items that come from entirely different non-overlapping
domains or product areas. For example, the query, orange, and the
keyword, orange cat, should be considered unrelated since they have no
items in common, and the items they pick out don't really have anything
to do with one another. 

Below are some examples of keywords and queries that are unrelated.

Keyword Query
lord of the rings wedding rings
las vegas entertainment stock option
options quotes
international year of the
the international
potato

Note: If the keyword and query have partially overlapping intent,

you should choose the option related but different instead. For
example, the query "1984 symbolism essay" is related to the
keyword "1984 book", because they are both related to the novel
1984. 

Advice for ambiguous queries and keywords

1. Start by clicking on the keyword or query and looking at the search

results for each. These may reveal the correct interpretation of a keyword
or query.

2. If a keyword or query is unclear, has many interpretations, expresses

only partial concepts, or includes non-sensical or ungrammatical text
(usually keywords), that you cannot fully interpret you should treat the
keyword or query as unrateable. Some examples include keywords or
queries like "blue", "cakes in", and "inthe word".

3. If a keyword or query has several reasonable and clear meanings that

are all equally likely you should consider interpretation of the keyword or
query that is mostly likely and best matches the other. Below are some
examples. However, try not to stretch the meaning of the keyword or
query or look for one that isn't clearly present.

Keyword Query Rating

luxury Since Jaguar is a brand of luxury cars, the query is
jaguar
cars more specific than the keyword.
Since jaguars are big cats, the query is more specific
big cats jaguar
than the keyword. 
contact contact Since contact lenses are often called 'contacts', the
lenses s keyword and query have the same intent.
Since contacts can also refer to a list of people who's
address contact
contact information you have, the keyword and the
book s
query have the same intent.

4. If a keyword or query has several reasonable interpretations and one is

clearly more plausible than the others, you should consider only that
interpretation. For example, the keyword or query [paris] could refer to
plaster of paris or Paris, France. Because the interpretation Paris, France is
much more plausible, you should provide a rating as if Paris, France were
the only interpretation. 

The Unrateable Flag

If either the keyword, the query, or both are unclear, nonsensical, missing,

or in a foreign language, select the unrateable flag. If you select the
unrateable flag, you will be asked to specify whether the problem is with
the keyword, the query, or both. Choose the appropriate option and
submit the task.

Note: If the keyword or query is in a foreign language,  you

should select the unrateable flag only if the keyword or query is
not something you would expect a typical speaker in the locale to
understand. For example, a query that was a mix of German and
English should not be marked unrateable in a German language
task, since many people who primarily speak German also know
at least some English. 

Note: Porn-related terms are expected for this task, and should
be rated normally.

If either the keyword or the query is so ambiguous or so general that it is

difficult to understand its intent, it may not be rateable. If it’s not clear
what the advertiser wanted when they chose a keyword, or what the user
was looking for when they entered a query, it may be impossible to
provide a rating. Below are some examples of keywords or queries that
you should rate as unclear or nonsensical.
Keyword or
Explanation
Query
The keyword or query "take" could refer to anything from
'taking pills' to 'take a job'; it's too hard to rate this
take
keyword/query because we cannot say what the advertiser or
user really wanted.
The keyword/query "blue" is too general for have a clear intent.
blue The user or advertiser could be interested in anything from
blue dresses to the blue man group.
cakes in The keyword or query "cakes in" is nonsensical. 
banda The keyword or query is in Spanish while English ratings were
larga requested. 

Keep in mind that not all keywords or queries that seem nonsensical
actually are. It is important to research the keyword and query to make
sure that you are not misunderstanding its intent.

If you encounter a technical problem with this rating task, use the "Report
A Problem" link in the lower-right hand corner of the rating page.

Task 6 Screenshot

Task 7 Religion Page Classification

Rating guidelines for Religion

NOTE: This is a training project in which you will be asked to solve

example tasks from the Religion category. After each of the tasks you
will be shown the answers recommended by linguists. The goal of this
training is to help you understand what the category is about. There may
be borderline or debatable cases where your answer differs from the
recommended labels, and you're not expected to match the linguists'
judgments in every case. However, you should consider this a way to get
better calibrated to the ratings that the linguists are applying.

1. Project Overview

In this project you will review webpages and decide if they are related to
the topic of Religion. The purpose of this work is to collect data for
building classifiers which can automatically detect such content,
ultimately allowing advertisers to prevent their ads from appearing on
such pages.
2. Process

Step 1: Review the webpage in the central frame.

Please scroll through the page, if necessary, until you have a strong sense
for what the page is about.

 What you see in the central frame

is a cached page, which is a
stored snapshot of the original
webpage.
 The original URL of the page is
shown below the central frame as it
may provide additional context.
 Please don't attempt to open up
the live webpage during your
evaluation, or browse to other
pages on the site, as the contents
of the live pages may have
changed since the page was
cached.

The default assumption is that pages are rateable. However, you may
come across unrateable content, which you can flag by ticking the
following checkbox:

☐   This page is unrateable.

Cached pages often show imperfect formatting and may even be missing
images, but please still do your best to complete the task, and only tick
the checkbox if the page:

 is not primarily in your rating

language;
 does not load;
 has no text, or contains too little
content to understand whether or
not the page is related to Religion;
 does not render well enough to
interpret the content that is
present;
 returns an error preventing you
from viewing its content.

Please note: when considering the language of the page, don't limit
yourself to the perspective of users in any one country. E.g., if you are
working on Portuguese tasks and you imagine the typical viewers of the
page are from Portugal, or from Brazil, try to consider the task from their
perspective (as best you can).
If you mark the page as unrateable, please leave a brief explanatory
note in the comment box.

Step 2: Rate the page.

If the page is rateable, please assess whether the page is related to the
topic of Religion (see Section "Detailed Guidelines" below for guidance),
and select the most suitable label from the following four options:

 Definitely Religion: This

page definitely belongs to the
category Religion.
 Probably Religion: This
page probably belongs to the
category Religion but it's not
clear-cut.
 Probably NOT Religion:
This page is probably NOT a
member of the category
Religion but it's not clear-cut.
 Definitely NOT Religion:
This page definitely does NOT
belong to the category
Religion.

Final Step: Review answers

Once you've chosen your answers, click on "Show answers" to view

project administrator's recommended labels. After reviewing the labels
and rationale, please click on "Submit" and move to the next task.

3. Detailed Guidelines

What falls in the Religion category?

Content that is related to major religions, as well as ethnic religions and

traditions (e.g. Chinese folk religion), and atheism; it does not include
mythology and superstition. The category includes opinions either
supporting or criticizing religion.

Content best labeled as Religion:

 religious reference and

resources
o religious customs,
practices, rituals, and
ceremonies [link]
 o religious teaching,
sermons, devotions and
prayers [link] [link]
o overview of a religion
[link]
o religious quotes [link]
o religious cartoons and
jokes [link]
 religious media
o religious literature and
publications [link]
o religious movies and tv
series [link] [link]
o religious music and
song lyrics [link]
 religious principles and
positions on social issues,
such as abortion [link]

 religious places
o places of worship (e.g.
churches, temples)
[link]
o holy sites [link]
 religious hobbies, special
occasions and events
o religious festivals and
holidays (e.g. passover,
hanukkah) [link]
o religious gifts and
merchandise
[link] [link]
 Other religious content
o news stories related to
religious institutions or
clergy [link]
o religious dating and
social networking sites
[link]

Content best labeled as NOT Religion:

 non-religious spirituality
o new age, inspirational
and self-help [link]
o astrology and divination
[link] [link]
 o occult and paranormal
practices [link]
o yoga practice [link]
o neutral meditation (e.g.
acme meditation) [link]
o feng shui [link]
 religious holidays
mentioned or discussed in
a non religious context
o top Christmas gifts
[link]
o Easter egg arts & crafts
[link]
o St. Patrick's day pub
crawls [link]
o Hannukah gifts for
everyone [link] 
o Pastry recipe for Easter
& Eid  [link]  
 Other non-religious
content
o mythology [link]
o psychological tests
[link]

Borderline cases

It is likely that you will encounter borderline cases where you are unsure
of the page’s membership in the category, and where you could easily
imagine other reasonable people disagreeing with your decision. To allow
you to indicate that certain pages are not obviously either “Definitely
Religion” or “Definitely NOT Religion”, we provide the “borderline”
labels “Probably Religion” and “Probably NOT Religion”. You will
need to use your gut instinct when deciding which of these to use, but
still try to follow the spirit of the guidelines by carefully considering which
side of the spectrum would be more suitable for the given page in the
target language/countries under consideration. It may also help to
consider a hypothetical advertiser who has opted out of having their ads
appear on pages with content about Religion. Would that advertiser be
likely to consider the page you’re rating as too related to Religion to show
their ads on it? If so, you should label the page as “Probably Religion”
rather than “Probably NOT Religion”.

Task 7 Screenshot