Answer Usefulness

Guideline Version 1.0, 10/27/2021

HitApp
Task Description
When a person uses a Search Engine such as Bing, they will type text in the search bar to try to
achieve whatever their goal is – we call this user input a Query. The Search Engine will then
return information to the user in the form of a Search Results Page. We have taken text from
one of the websites in the Search Results and presented it as a short Answer to the user to try
and address their Query. In this HitApp you will be given a user Query and an Answer.

Your task will be to identify the intent of the user’s query (what they are trying to do) by looking
at the Search Results and determine if the Answer provided to them is Useful and to what
degree the user would be satisfied.

Note: To view search results ensure you are using Chrome Browser and install the W5P plugin

Instructions
Task Process
1.) Identify and Understand the User Query:
• Use the Bing Search Results found in the tabs of the HitApp to determine likely
intents of the User’s query
• If you don’t have enough knowledge on the query to make a judgment, use the
Search Results to understand the context and intent of the query
2.) Make and Submit a Judgment
• Ensure that the Answer is relevant to the query and that it does not just have
similar keywords that make it appear to be about the same topic
• After understanding the likely intent of the query, determine to what degree the
Answer satisfies the user’s Query
• Submit a Judgment
Ratings
o Useful
o The Answer COMPLETELY satisfies the user’s intent. By reading the answer
alone, the user can walk away with all the information he or she needs. No other
effort is required on the part of the user
o The Answer is a partial Table or List with useful information, where the
complete list on the Source Page fully satisfies the User’s intent (Example)
o Partially Useful
o The Answer is a Passage that provides some useful information to the user, but
doesn’t completely satisfy the User’s Intent
o [Passage Answers Only] The user needs extra work beyond reading the Answer
text to satisfy his or her need, such as:
▪ Clicking on the article to find more information
▪ Seeking more information elsewhere on the Source Page
▪ Performing more Searches on the Topic
o Not Useful
o The Answer does not help the user at all; reading it is a waste of the user’s time
o The Source Page returns an error (404, “server did not respond”, etc.) (Example)
o The Answer is one or more of the following:
▪ Incorrect – Presents the user with wrong information
▪ Irrelevant – Not about the topic of the user’s intent
o The User is intending to do something other than look at a text passage, list, or
table such as:
▪ Buy something
▪ Find a business or use a map
▪ Navigate to a website
▪ View a Video / Image
▪ Download content / play a game
o In Wrong Language
o *Note*: English Queries with English Answers should always be completed and
Judged as normal, do not mark these queries as “In Wrong Language”
o The Query and/or Answer are in a language other that the market language or
English
o The Query and Answer are in different languages

Additional Considerations
Judging Truncated or Incomplete Tables / Lists in Answer Area
When working with tables and lists, many times the table/list will be truncated and not
completely visible in the answer section due to it being too long. If the table / list provided in
the Answer section contains some useful information to the user (something of value), then
the entire table / list on the source page should be used when determining the final rating.
In the below example, the intent of the user is to find out how to use different paper sizes in
Google Docs. The Answer given provides some value to the user, and indicates that there’s
more information in the list in the Source Page. Because the list in the Answer section had
some useful information for the user, the entire list on the Source page should be used when
rating – leading to a final Judgment of “Useful”.

Partially Useful List Completely Useful List on Source Page

However, in this example below, there’s nothing of value to the user in the Answer and the
complete table on the source page should not be used (even if there is useful information on
the source page). The reasoning for this is that if a User didn’t see useful information in the
Answer, they would have no reason to open the source page. The final rating: “Not Useful”

Not Useful List Completely Useful List on Source Page

Beware of Answers Containing Keywords That Are Not Actually Relevant to Query
Keywords can be a great way to quickly identify if an Answer is Useful, however it’s important
to confirm that the passage provided is in fact relevant to the query. In some situations, an
answer will contain keywords, but will not actually be relevant to the user’s query.

Here is an example of a situation where an answer at first glance looks like it is Useful to the
user’s query. The main intent of the user’s query is to find out how many controllers a
Playstation 4 comes with when purchased. The answer contains many keywords found in the
query such as “ps4” and “controllers”, however the Answer is not actually relevant to the user’s
query. In this case, the Answer mentions the maximum amount of controllers that a PS4 can
use. This would result in a final rating of Not Useful as the answer is not relevant to the User
Query of how many come with a purchase. It is necessary to always ensure that the Answer is
actually Useful to the Query, regardless of Keywords. Keywords in the Answer Text, Title, or
URL does not automatically make an Answer Useful.

HitApp Answer

Examples
Final Answer of “In Wrong Language” Due to the Query and Answer Being in Different Languages

HitApp Query / Answer

In this Hit, the Query is in English and the Answer is in French. As Query and the Answer are not
in the same language, the final rating would be “In Wrong Language”