Search Result Satisfaction (Simplified)

System Requirements
In order to optimize your judging experience, this Hitapp has some task specific technical requirements
that you will need to meet in order to start judging pages. They are listed below. For this task, you will
be required to work in Google Chrome and install a chrome extension. Please also make sure to disable
any ad-blockers.

1. Google Chrome
2. UHRS Extension

After you have installed Google Chrome and the UHRS Extension, the first time your start Chrome after
installing you must follow this procedure:

1. chrome://extensions/
2. Click the details button on the UHRS Extension.
3. Ensure that the extension can run in incognito mode

If a page does not open in the hitapp, please open the URL in a new tab and judge the page that
appears.

Overview
For this HitApp, you will look at a given query and evaluate how well the page would satisfy the user.
Satisfaction is how happy a user would be if they got the page after entering the given query.

Choose a label between Perfect, Excellent, Good, Fair or Bad based on how well the query would satisfy
the user. Consider things such as:

 Intent of the Query: What might the user be looking for? There can be multiple possibilities.
Some options are more expected than others.
 Page Purpose: What is the goal of this page? Ex: Is it to provide information on a specific topic?
 Value: Does the page provide value to the user searching this query? How well does the page
match the needs of the user? For example, if the goal of the page is to provide information, does
the topic match what the user is asking for? If it doesn’t answer the query, it is Bad.
 Scope: Does the scope of the page match the scope of the query? Is the page scope more broad
or more specific than the query? Is it too broad or too specific?
 Location: Is the user’s search location important for this query?
 Date: Is the information is out-of-date for the user?
 Any other potential flaws or positives

HitApp Interface
You will evaluate the web page you are given with the user’s search information: query, location, and
time. Location and time are very important with some search queries, but not for all of them.

1. Search Query: This is the search issued by the user. Please consider what the user is looking for.
 2. Search Location: This is where the user was located when they issued the search query. Please
take this into consideration if the search query is location-sensitive.
3. Search Time: This is the data and time the search query was issued. Please take this into
consideration if the query is time-sensitive.
4. URL/Landing Page: This is the page you will evaluate. Please open the URL in a new tab if it does
not load in the hitapp and judge the page that appears.
5. Hit packs: A Search Query and URL pair is called a hit. Hits are often served in packs. This means
that the same Search Query will have more than one URL. If a Search Query has 6 URLs in the
left column to evaluate, you are working 6 hits.
6. Rating: Evaluate the landing page based on the search information.
7. Done: Save the rating for that hit. Rate any other hits.
8. Submit: Submit the ratings.

Missing Key Concepts

In order to satisfy the query, a web page MUST address all key terms in the query. If any part of the
query is missing from the web page, is CANNOT satisfy the query. However, it is possible that the page
might cover the concept of the query while using similar words that are not exactly the same, such as a
query looking for “best restaurants” and a page that lists “top restaurants”.

Example 1
Query: Walmart.com Nike socks
Web page: Walmart.com listings for Nike socks
 Excellent: The web page would satisfy the query for Nike socks at Walmart.com
Example 2
Query: Walmart.com Nike socks
Web page: Target.com listings for Nike socks
Bad (Type 2): The web page would not satisfy the query as it specifically is looking for
Walmart.com not target.com
Example 3
Query: Walmart.com Nike socks
Web page: Walmart.com listings all socks with some Nike socks
Good: The web page could still satisfy the query as it has Nike socks, just to a lesser extent.

Example 4
Query: Walmart.com Nike socks
Web page: Wikipedia page on socks
Bad (Type-1): The web page is completely irrelevant.

Search Location
Look at the query to determine if the location the user made the search from matters. If yes, look at
Search Location and use that to help evaluate the page.

· Location-sensitive query: If the someone searches for {restaurants near me} you should
compare the locations on the page to the search location of the user.
· Search Location does not matter: If someone searches for {Microsoft word} you can mostl likely
search location. If the query lists a location directly in such as {new york pizza place, the user’s
search location does not matter since only “New York” would satisfy the query.
· Depends on the landing page: Location does not matter if someone searches for {blueberry
muffins} and gets a recipe. Location does matter if the page is a bakery that sells blueberry
muffins.

Search Time
Look at the query to determine if the time the user made the search matters. If yes, look at Search Time
and use that to help evaluate the page.

· Time-sensitive query: If someone searches for {chicago apartment fire}, they are probably
looking for an event that happened recently and you should look at the search time.
· Search Time does not matter: If someone searches for {2016 Olympics}, the results will be the
same no matter when the user issued their query and you can ignore Search Time.
· Depends on the landing page: If someone searches for {facebook}, time would not matter if you
got the Facebook homepage. However, it would be important if the URL was a news article and
you should compare the article with the Search Time.