Professional Documents
Culture Documents
Ad History Guidelines Upstream
Ad History Guidelines Upstream
These guidelines contain confidential and proprietary information belonging to Microsoft Corporation. The recipient
understands and agrees that these materials and the information contained herein may not be used or disclosed without the
prior written consent of Microsoft Corporation.
Overview
In this project we will look at a user’s online activity and an advertisement that is shown to the user. You
will help us compare the ad to the user history item to determine how closely the two are related. In the
labeling interface, we show an item from the user’s history on the top and an ad on the bottom. A
history item is an action taken by the user, such as visiting a website, performing a search etc. As a
judge, you will choose the best label option to describe the relationship between the user’s history item
and the ad.
These guidelines describe how to categorize the relationship between the ad to a history item into one
of the following buckets: 1. Excellent, 2. Fair, 3. Bad, 4. Not Sure.
Note: Some of the links in the user’s history may not be accessible due to several reasons. However, the
displayed text of the URL would often contain salient information such as name of the website,
particular section the user visited etc., which can help you to judge with more accuracy. For example, in
the screenshot below, just by looking at the displayed URL, you can infer that the user is interested in
jobs at Accenture located in India.
Judging Interface
The judging interface will look like the image below. The annotations are further explained below:
1. User History: This section provides either the title of a webpage and a link to that page
(shown in the screenshot) or a simple search query.
2. The Ad: This is a blob of text comprising a title, a link, and a description of the Ad.
3. Rating Options: The options to label the relevance of this item to the ad shown on top as
Excellent, Fair, Bad or Not Sure. The rules to determine this label are explained in the next
section.
1
2
Rating Options
Excellent ad for this history item
The history item is about the exact same thing (product/service) as the ad.
The history item is about a variation of the same product/service in a different color (black
evening dress/History: red evening dress), size, specification, material, style, or format (print vs
Kindle) as the ad.
If the ad (it will be part of the displayed title, description or URL ) contains a location and
the history item contains (In case of search history, it will be part of the displayed string.
In the case of a visited website, it can be part of the website’s title or URL or can be
known after opening the URL) a different location with the same/similar product,
it cannot be excellent. You should mark this as Fair.
If only one among the ad or the history item contains a specific location, while the other does
not contain any location, the rating cannot be excellent. You should mark this as Fair.
A history item referring to the same product in a different target age or gender as the
ad cannot be excellent. Such a webpage should be rated Bad. For example, women’s trench
coats vs men’s trench coats or women’s shoes vs kid’s shoes.
Not sure
The judge is not sure if this page is relevant or not. In this case, please leave a comment
explaining why you could not select a rating.
Examples