Page Recommendations: Judge a webpage abstract

Last updated May 29, 2022

Task Overview
In this task, you are given a source/viewed webpage and two recommended webpage abstracts, you are
required to determine which recommended webpage is better and more interesting for user visiting.

You may visit the recommended webpages too, if needed, by clicking on the link to get more context
about the recommended pages. But please don't forget that it is the abstract you are judging, not the
second page in full.

To help complete this task, please use Chrome browser, and install UHRS Chrome Extension. It will help display
some pages in the HitApp properly.

If you see a blank screen, please open the page in a new tab by clicking the url. If it does not open in the
new tab either, choose 'Can't judge'.

Judgement Values
There are three levels of interestingness for recommended webpages as judging criteria reference:

The top level (very interesting):

• Provides a different perspective on the user intent

• Encourages the user to continue their exploration of a topic in a coherent manner
• Reasonable next step in the most likely task a user had in mind following a click on the viewed
page
• Provides another option to complete the task the user has in mind.
• User will love the recommended page and click it to explore more.

The second level (fair):

In general, fair recommendations are pages that are relevant but miss on a key aspect of the viewed
page. They differ from NOT interesting pages because they are still acceptable to show a user without
causing dissatisfaction, and some slice of users may find them useful.

• Too broad: Connection between viewed page and recommended page is too general: The
recommended page is related to the higher-level topic of the viewed page but does not consider
one or more key intents.
• Too specific: Only useful for some specific search intents of users who viewed this page.
• Somehow parts of users may click the recommended page.

The last level (not interesting):

• Is unrelated to the source page, the subject is completely different.

• Has usefulness/freshness/authority/credibility issue. (Highly repetitive to the source page falls
into usefulness issue bucket.)
• User would never click it.
Reference above judging criteria, and choose corresponding value:

• EquallyGood. Both recommendations are equally interesting.

• EquallyBad. Both recommendations are equally not interesting.
• Reco1Better. The recommendation1 is more interesting than recommendation2.
• Reco2Better. The recommendation2 is more interesting than recommendation1.
• Can'tJudge. If you see a blank screen, please open the page in a new tab by clicking the url. If it
does not open in the new tab either, choose 'Can't judge'.

Judging Examples
Examples

Example 1
Source url: https://www.msn.com/en-us/money/realestate/cities-with-the-fastest-growing-home-prices-in-
spokane-metro-area/ss-AAXArhz?ocid=BingNews&li=BBnbfcL
Webpage1: https://www.zillow.com/spokane-wa/sold/
Webpage2: https://www.msn.com/en-us/news/us/high-inflation-leaves-food-banks-struggling-to-meet-needs/ar-
AAX9PN2?ocid=BingNews

Label: Webpage1 better

Comment: The source url is about real estate. Webpage1 is relevant and useful. Webpage2 is irrelevant.
Example 2
Source url: https://en.m.wikipedia.org/wiki/World_Turtle_Day
Webpage1: https://www.msn.com/en-us/travel/news/things-to-do-in-baltimore-may-20-26/ar-
AAXsBKh?ocid=BingNews
Webpage2: https://www.firstpost.com/world/world-turtle-day-2022-history-significance-quotes-and-how-the-
day-is-celebrated-10695231.html

Label: Webpage2 better

Comment: The source url is about World Turtle Day. Webpage2 is relevant and useful. Webpage1 is irrelevant.
Example 3
Source url: https://www.bing.com/search?q=fiscal+policy+in+2008+recession
Webpage1: https://www.ukessays.com/essays/economics/monetary-policies-implemented-2008-recession-
5263.php
Webpage2: https://en.wikipedia.org/wiki/National_fiscal_policy_response_to_the_Great_Recession

Label: Webpage2 better

Comment: Both recommended pages are relevant to the source url. However, webpage1, which is a research site,
is not useful to the user viewing the given source url.
Example 4
Source url: https://www.rotowire.com/basketball/injury-report.php?team=OKC
Webpage1: https://www.espn.com/nba/injuries
Webpage2: https://www.msn.com/en-us/sports/nba/pistons-injury-report-against-the-bucks/ar-
AAW0Avl?ocid=BingNews

Label: Webpage1 better

Comment: Both recommended pages are relevant to the source url about “NBA injuries”. However, webpage2,
which is a news site, is not as useful as webpage1 to the user viewing the given source url.