BANE CURATION

April 2020
Contents
Introduction ................................................................................................................................. 2
First Steps ................................................................................................................................... 2
Instructions for Part 1 – Step 1: Validity of Query ............................................................................. 2
Step 1: Understand the Query .................................................................................................. 2
Important! Non English (United States) Locales: Queries that are in ENGLISH should be
judged accordingly and can count as valid queries. ................................................................. 2
Step 2: Determine Whether the Query is Valid ............................................................................ 2
Instructions for Part 1 – Step 2 Find Top 5 URLs ............................................................................... 4
Step 1: Provide the Top 5 URLs for the Query Based on the Given User Location ............................. 4
IMPORTANT! NEVER INCLUDE THESE URLS – THEY ARE NOT ACCEPTED AS INPUTS! ........... 4
Important Rules & Tips .................................................................................................................. 6
Tips for acquiring the URLs: ...................................................................................................... 6
Instructions for Part 2 Top Three URLs............................................................................................. 6
Step 1: Review all 10 URLs for the Query Based on the Given User Location ................................... 6
For the given Query and User Location, find the best 3 URLs that match the user's intent based on
their specific location. Prior to doing this, you must click on all 10 URLs and determine
which three are the best match for the query. You MUST click on all ten first to review them
carefully (each should open in a different browser tab). ...................................................... 6
Step 2: Rank Top 3 URLs.......................................................................................................... 6
Introduction
The goal of this task is to identify the top three results by a given search query and the user location where the
query originated. In preparation for the task, please read the below instructions before you access the job. This
is three-part task with individual instructions for each task. Your attention to detail is EXTREMELY
IMPORTANT.

First Steps
You must complete this task on a desktop or laptop. Never use a mobile phone. We expect you to rate search
results from the view of the user’s specific city location. Please ensure your computer is configured to be
in the language of the project you are working on.
Each query will have a specific location listed underneath it. You will find results targeted to each
query’s location by searching with the geolocation button in the F8 workpage.

Instructions for Part 1 – Step 1: Validity of Query

Step 1: Understand the Query
The first step is to understand the query and decide if it is valid. DETERMINE WHAT THE USER COULD BE
SEARCHING FOR. WHAT IS THEIR INTENT?

Read the query, review the live search results page based on that query, and think about the possible user
intents it could represent. For Example:

• Query [Tom Cruise] Possible user intents are information about Tom Cruise, watch a movie that
Tom Cruise is in, or the latest celebrity gossip about the actor, etc.
• Query [Firestone] Possible user intents are locations of tire stores, information on a beer
brewery, an album of that name by Kygo, etc.
• Query [Samsung] Possible user intents are to shop for Samsung products, technical support for
a Samsung phone, homepage of the company, etc.

Important! Non English (United States) Locales: Queries that are

in ENGLISH should be judged accordingly and can count as valid queries.
Step 2: Determine Whether the Query is Valid
Use the answer dropdown to select whether the query is valid, inappropriate, or too vague to answer.

Misspelled queries – If you search the query, and find clear intent based on a corrected spelling, then mark
VALID.

IS THE QUERY TOO VAGUE?

If you have no idea what the user could be looking for, you must research the query. A query that may be
quite vague to you, may in fact not be vague at all. If you find there is a dominance of one intent when
searching the query, then it is NOT vague and can be rated for relevance. If search results show the query
term doesn't really seem to refer to anything, and is basically gibberish or an incomplete query, only then
should you select the Query is Too Vague option and move on to the next query. Note this option should be
used sparingly!

Common examples of Vague queries:

• The query consists of only special characters: [@#$]

 • The query is a random word for which we cannot determine an intent: [just] or [the]
• The query is incomplete: [where can i find] or [best price for]

There are different levels of vague. Some queries like [how to] [what] [aaaaaaaaa] are Vague because
we don’t know what the user is asking about, so there’s no intent we can retrieve.

For some queries, like [manners anita], although there are many people called by this name, and none very
popular, the intent is still clear. The user wants to search for anyone called "manners anita". We can find URL
results about a person by that name. For example: the query [my best friend in the whole world] the
intent is also clear. The user wants to search for anything about this sentence. It could be an image, a video, a
song, a book or anything, as long as it contains that sentence, find the best URL results for this query.

It may be helpful to view the knowledge graph on the right side of the search engine to assist in determining if
a query is truly vague. For example, the query [addons] may seem vague, but researching the query, the
intent is clear. The user is searching for addons for browsers. The knowledge graph will suggest what this result
might be for:

Important! Queries that are clear but have multiple plausible intents, are NOT Vague queries, as we can still
judge the relevance of a result. For example, [apple] could refer to an Apple product, Apple stock, Apple news,
wiki page, and/or apple the fruit. We do not consider these Vague queries.

IS THE QUERY INAPPROPRIATE?

A query is Inappropriate if it falls into any of the following categories:

• It encourages illegal behavior (e.g. piracy, child pornography)

• It includes words generally considered profane
• It uses derogatory language to refer to certain groups (racial slurs), or otherwise constitutes hate
speech
• It is concerned with non-medical sexual practices or situations. This includes queries that are
clearly related to pornography.
• It describes graphic violence and/or would likely retrieve grisly content

If you select the Inappropriate option, then no further ratings are necessary.

WHAT IF THE QUERY IS IN A FOREIGN LANGUAGE?

If the query is in a language you do not understand, you MUST research or use a translation site such as
Google Translate to determine whether it is valid, vague, or inappropriate.
Instructions for Part 1 – Step 2 Find Top 5 URLs
Step 1: Provide the Top 5 URLs for the Query Based on the Given User Location
For the given Query and User Location, find the top 5 URLs for Google, using the geolocation search button
for each query!

Please note that the result sometimes doesn’t need to be related to the location. For example, if the query is
[cnn] most users’ intent is to go to CNN’s website to look for local news, national news or news around the
world. So the best results are https://www.cnn.com/, https://www.facebook.com/cnn/ and
https://twitter.com/CNN. Only a small portion of users want to look for a special news page that happened
locally, like https://www.cnn.com/2020/02/21/us/idaho-missing-children-timeline/index.html

You must be sufficiently knowledgeable about the query to know what most users’ interest is
regarding the query. For example, if you are not a Twitter user, you may not know that
https://twitter.com/CNN is a very popular source to look for CNN news. In major search engines, it lists all
URLs that relate to a query sorted by its popularity. Researching will show you the best results for the query.

[Amazon] is a company, but also a forest in South America. However, the most common interpretation of
the query is for the company.

If the given query is about a chain store, or its name is shared by two or more objects, then the one closest
to the given location is a better choice than the one that is further away. For example, the query
[Home Depot] or [Starbucks] the best result would be the location that is closest to the user.

If the given query is formatted as a URL, search the URL to see if you can find other related URLs.

Searching Queries with Location

1. Click on "Google search via geolocation" to see Google search results for the given query based on the
location shown under the query.

Provide the top five (5) URLs that appear in results into the required text boxes provided by clicking on the
result, copying the URL in the address bar, and pasting it into the task answer box. NEVER right click on the
Google result title and select "Copy Link Address" to obtain the URL UNLESS you are using the
CHROME BROWSER ONLY. All other browsers will give you a Google redirect URL with this method
and these are NOT acceptable inputs!

IMPORTANT! NEVER INCLUDE THESE URLS – THEY ARE NOT ACCEPTED AS INPUTS!
YouTube links (UNLESS they are the top SINGLE result that satisfies the query). What is NOT
accepted are pages that show a full list of many video results. Google redirect links, Google image
or video search result pages, and Ads that appear on the very top of results, as they appear in the
same format as results. These tend to appear as part of the top 5 results. DO NOT CHOOSE these
and go to the next website landing page result. If necessary, scroll down until you do not see Ads results:
YouTube Results: Below is a VALID URL result for the query [thriller]

Below is NOT an accepted YouTube result!

Important Rules & Tips
NEVER INCLUDE these types of result links! These types of URLs will NOT be accepted!

➢ Ads shown at the top of the page

➢ YouTube video search results links (UNLESS they are the top SINGLE result that satisfies the
query) What is NOT accepted are pages that show a full list of many video results.
➢ Google redirect URLs (see above for example)
➢ Google image search result links
➢ Google video search result links
➢ Google map result links

Tips for acquiring the URLs:

➢ Open each URL in a new tab by right clicking the result and hitting "open in new tab" to review the URL.
Then, copy and paste the URLs from each tab's address bar.
➢ Right click each URL directly from the search results page and select "copy link address". Then, paste
the link into the text boxes in the F8 task page.
➢ NEVER copy the text from the search results. This is not a valid URL.
➢ You must ALWAYS enter 5 top URLs for both Google. NEVER leave empty fields and NEVER
duplicate entries. If the first search page does not show 5 URLs, then click to page 2 or
further pages of search results to find the remaining links.

Instructions for Part 2 Top Three URLs

Step 1: Review all 10 URLs for the Query Based on the Given User Location

For the given Query and User Location, find the best 3 URLs that match the user's intent based on
their specific location. Prior to doing this, you must click on all 10 URLs and determine which three
are the best match for the query. You MUST click on all ten first to review them carefully (each
should open in a different browser tab).

Step 2: Rank Top 3 URLs

After you have reviewed all 10 URLs by clicking on each one, choose the top 3 web site links that best match
the user's intent from the dropdown for each Rank 1, Rank 2, and Rank 3 fields appropriately.