Search Results Comparison and Query Categorization

Task:
Every query in this hitapp represents a user internet search query where users are searching for Local
Businesses. For this task, Local Business means a business the user can visit or directly contact a
business. This is explained more below in section 1.

Hitapp Practice: from the dashboard, select the Preview icon to open practice hits with explanations
before you begin the Training sets. Preview hits are not graded, and do not affect your Training scores.

For a given user internet search (query) (including the location where the query was issued from), you
will make the following judgment:

For each [Query, Location] pair, indicate if the user expects a single precise result to be returned.

Here are three quick examples:

• {restaurants} for a user in Seattle, WA – The user is clearly looking for multiple local businesses.
They are not expecting a single precise result to be returned.
• {ray’s boathouse} for a user in Seattle, WA – The user is searching for a local business by name,
and there is only one such business by that name. In this case they do expect a single precise result
to be returned.
• {olive garden} for a user in Seattle, WA – The user is searching for a local business by name,
however it is a chain with multiple nearby locations. In this case the user is not expecting a single
precise result to be returned.

Research Links:
In the hitapp, up to two different map results for this search will be displayed. This is to help give
you context for the query. If there is only one, or there are no search results displayed, or if there
are no relevant results shown, you will have to do a side search using the Research Links or
entering the query and location information into a separate search browser.
Research Links have been provided to perform further research on the Businesses relevant to the user-
search query. Shown below is a snapshot of the Research links in the HitApp. Start with the Archive
option which is a saved view of a search done from the user’s location. Internet browsers customize
your experience based on your own location and search history, so the archive can help give you context
for the kinds of results that would make the most sense to someone in the same area as the original
user.

Please note that while Results presented for Goal / Task (I) provide context on the type of Businesses
that are potentially relevant to the query issued, the Research links provide more in-depth information
and a broader context. The Research links could be especially helpful in cases where no Local Results
are returned for a particular query. In such cases, the Research links could help in the annotation
process.

Guidelines:
Do YOU expect a single result if you searched this query from this user location (Indicate if the
user-expectation is for a single precise result to be returned)
Based on the results displayed, and / or research done using the Research links, if there are
multiple businesses that match the user's (your) search criteria, then select the “No” option (you
do not (the user does not) expect a single business result to be returned for the query issued).
When dealing with adjectives (best, newest, etc.) which are a part of the query, because
those results can change over time, and we do not know the user’s specific requirements for
terms like “best,” we select “No” and let the user decide what they think is “best” from the
options given.

1. Assume the query has Local Business intent

For every query you judge, assume that the user is looking for some Local Business. Local business is
defined as a retail customer facing business that you or I could call or visit for products or services. Some
businesses come to the customer instead of the customer visiting them, like plumbers, and are still
considered Local Businesses. Please note that some queries may have multiple contexts or
interpretations. If at least one of these interpretations is a Local Business, then we assume that is what
the user intended. Make sure you search for probable intents of ambiguous queries, rather than assuming
you already know. For example, the query {atlas} could mean:
✓ Find a book containing maps
✓ Find information about a character in Greek mythology named Atlas
✓ Find businesses:
▪ US wide: Moving service ({Atlas} is nationwide moving service)
▪ St Louis, MI: There could be restaurants named {Atlas}
▪ Many cities: There could be Bars named {Atlas}
For your judgment, always assume the last intent: “find business”.
2. Consider if the query contains location information. Also consider where the query was issued from
Sometimes the user wants results in a specific location, which they provide in the query. This is called an
“Explicit Location.” If the query contains an Explicit Location, that is the area we must search for results.
Note however, that location terms that are part of a business name are not considered Explicit Location
terms. For Example, “Seattle’s Best Coffee” is a business with many branches outside the city of Seattle.
If you are unsure, look at the business logo and see if the location word is included, if so, it’s not an Explicit
Location.
If the query does not specify an Explicit location, (the location is Implicit) we must keep looking for the
location context. If there is no Explicit Location, consider if the business is a unique or famous business
that people would travel to visit. For Example, for the query [Taj Mahal] issued from <San Francisco,
California> it is safe to assume the user expects the monument [Taj Mahal] in India to be returned.
If the query is for a unique or famous business that does have Explicit Location information that
contradicts with the location of the business, search for results in the Explicit Location. For example, the
query [Eiffel tower salt lake city] has an Explicit location different from the famous tower in Paris, France.
When we search for “Eiffel tower Salt Lake City,” we do find a catering business (Eiffel Tower Fine Catering)
and so we assume this is the result the user intended to find.
If there is no Explicit location, and the business is not unique or famous, then we use the location the
query was issued from to determine the location context. Consider the level of detail included in the user
location. If it includes a post code or neighborhood, use that as your first search area. If there are no
results in the post code or neighborhood, look at the city level. City level will usually be the implicit location
context, if there is more specific information and a result close to the user we can consider that first, but
if there are no results in the city we don’t keep expanding the location context until we find one.
The Table below summarizes the material discussed above through examples.

Examples:
Query Where query Location Note
was issued from Intent
Le Cirque Salt lake city, Implicit No explicit location in query. However, Le Cirque is an
Utah 84105 iconic restaurant and people around the US are
expecting to find the appropriate business by simply
searching by name, without specifying location.
Therefore in this case, user location need not be
considered while making relevance judgments.
Eiffel tower Bellevue, Explicit Explicit location in query. Explicit location in the query
Salt Lake City Washington in conflict with the location of a renowned business
(Eiffel tower is a renowned monument in France).
Since the user provided explicit location information,
in this case further research is required to check if
there is a business named {Eiffel tower} in Salt Lake
City.
 Plumber los Medford, Explicit Explicit location in query. Explicit location differs from
Angeles Oregon 97504 location from where the query was issued from. First
we check to see if there is a plumber around
Medford, OR called “Los Angeles Plumbing” or
“Plumber Los Angeles.” If there is not, it is safe to
assume that the user is looking for {plumber} within
the location context of <los angles>
Pediatrician Newnan, Implicit No explicit location in the query. The query is generic
Georgia and does not specify any unique business. Treat this
as if the user is looking for {pediatrician} within the
context of their location, Newnan, Georgia.
Kentucky Fried Seattle, Implicit The word “Kentucky” is part of the business name
Chicken Washington and should not be considered explicit location.
Assume that the user is seeking Kentucky Fried
Chicken results in the context of their location,
Seattle, WA.
Seattle’s Best Seattle, Explicit Business name contains Seattle, but there is also an
coffee, Seattle Washington explicit location in query. The explicit location
happens to match the user location in this case.
Italian Los Angeles, Explicit Business name contains Explicit location (zipcode).
restaurants California Please note that the specificity of the Explicit location
98005 may vary. For example, Explicit location information
may be present at the street-level, in the form of a
zipcode or post code, town, city, district, county,
state, country, etc.

Open vs. Closed Businesses

Upon researching (using the Research links), if it is found that some of the Businesses (entities) returned
are “Closed”, please make judgments assuming that the Businesses were still “Open”. That is, please judge
the relevance of the entity without taking into account whether the Business is currently “Open” or
“Closed”.
Phone number
For queries that only include a Phone Number, please perform research using the Research links to
determine what business may be associated with that number.

Address queries
For queries that include only an Address, judge whether that address could reference a single business or
multiple businesses. For example, if the address is a building that has many clinics or businesses, we would
say a single business was Not expected. If the address for this building included a suite number for a
specific business, we would say Yes, a single result is expected.

If you are unable to find any businesses that the query could be referencing, judge it Not Confident.
Queries for businesses in other countries are ok, and if the query is in a language not spoken in the hitapp
market, but you can judge it, feel free to. But if the query is in a language you do not understand, or the
only search results are in a language you do not understand, rate it Foreign Language. However, please
make sure you do a search for the query to see if the query is the name of, for example, a point of interest
or national monument or other business name that has business information that you can understand.
For example, many restaurants will have names in other languages, but with a search it’s clear we don’t
need to know what “l’Orangerie” means to see that there are restaurants with that name