Stanford Guidelines Raters 11.22.19 PDF

Stanford Guidelines
Updated November 22, 2019
Objective
Determine the level of sameness between two highlighted products by comparing

attributes.
Project Overview
For this task you will evaluate the level of sameness between two products. To compare
sameness you will compare the products names, and also the attributes that are present in
the highlighted text and within the content of the document. By “sameness” we mean the
degree to which two products resemble each other
Attributes refer to characteristics that describe the product such as the brand, style,
material, etc. For example, if someone says “I bought the same shirt at the mall for $5 less,”
do they mean that they bought exactly the same shirt with the same brand, style and
material for $5 less, or do they mean they bought a similar shirt but with a different brand,
style, material, for $5 less. By comparing the attributes, we can determine how close two
shirts are to being the same shirt.
Instructions:
Compare two products by evaluating the product’s name and attributes. There is no
minimum view time required to complete the task. You may use your favorite search
engine to side search attributes or product names you are unfamiliar with. If possible,
please do not exceed 60 seconds per task.
CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

Tool Diagram
Before you begin the task, please familiarize yourself with the components of the tool
below.
• Document 1 and Document 2 are the Documents that each product appear in.
• Document Text 1 and Document Text 2 are the text of each Document.
• Product 1 and Product 2 are the highlighted text that includes the product name
and attributes that you will evaluate for this task.
• Product Category Comparison check this box to compare whether the two
products belong to the same product category.
• Product Category Evaluation determine which product category is the best fit for
the two products.
• Attribute Comparison compare and evaluate the attributes by clicking the best fit
rating option.
Rating Options
Rate the products names and attributes as either “Match”, “Partial Match”, “Different”, or
“Absent”. Refer to the definitions below to generate accurate ratings.
• Match = attributes for each product are both present and exactly the same
• Partial Match = attributes for each product are present and almost the same
• Different = attributes for each product are present but not the same; Attribute is
present only for one product
• Both Absent = both of the product’s attributes are missing
Definitions
• Attributes: are characteristics that describes a specific product that distinguishes it
from similar products and may affect a consumer's purchase decision. Examples of

attributes are brand, style, model and specification (please see Attributes to Include
in Evaluation below)
• Product: a product is a specific, physical good that a person can use and retain
(Example: a purse). Products are items that can be bought/sold and may be offered
as a free trial or as a sample.
• Product Category: describes how products are organized by type. Please refer to
the 15 product categories listed below for this task. Example: the Nike Air Max 270
Blowfin Running Shoe belongs to the product category “Clothing & Apparel”.
• Product Name: is what the product actually is. In the example above, the product
name is “Shoe”.
Product Product 1 Product Product Product 2 Product Product

Category Name 1 Attributes 1 Name 2 Attributes 2
1 Household recycled bags recycled, large ivory bags large, ivory,
Supplies Kraft Kraft, paper, paper paper,
paper food food parcel shopping shopping,
parcel bags case
bags of 250
2 Animals & golden puppies golden akc puppies akc certified
Pets retriever retriever certified half American
puppies half half English
American golden
half retriever
English
golden
retriever
puppies
3 Food & golden honey golden, natural honey natural, raw,
Drink wildflower wildflower raw wildflower,
honey wildflower 500g
honey
500g
4 Automobiles 2018, car 2018, Ford, 1968 Ford truck 1968, Ford, f
Ford, Taurus f 100 100 ascender,
Taurus limited ascender 4wd rc
limited 4wd rc
truck
Before You Begin
As you begin the task, if any of the following issues occur, please skip the rest of the task by
selecting the correct option at the bottom. If none of the issues occur, move on to Steps.
• Page Won't Load: Image fails to load. SRT (The review tool) is throwing a widget
error. If the text is present and the image is not fully rendered, please try to do
the task. Only when its impossible to do the task, select this button.

• Wrong Language: If the content is not fully comprehensible because it is in a
language that was not intended for the given workflow.
o Note: If the text on the document was auto-translated (usually a link that will
say 'Show original'), then select 'Wrong Language.'
• Sensitive: In instances where the Document is sensitive, please mark as 'Sensitive'.
'Sensitive' is defined as containing or potentially containing any of the following:
o Child Exploitation and/or Child Nudity; Self Injury and Suicidal Content;
Credible Threats, Violence or Calls to Violence; Sexual Content and/or
Nudity; Hate Speech; Acts of Terrorism; Human Trafficking; Bullying and
Harassment; Profanity
o Instructions for Immediate Escalation — If potential Child Exploitation
and/or Child Nudity, Self-Injury/Suicidal or Time-Sensitive Credible Threat is
the reason for marking as “Sensitive”, escalate the Job ID immediately to your
manager.
Steps:
1. Review each document carefully to understand the context of the featured products.
2. Is the Product Comparison Box pre-checked “Yes”?
a. Yes, the Product Comparison Box is checked “Yes” = Move on to step 5
b. No, the Product Comparison Box is not checked = Move on to step 3
3. Do the products belong to the same product category?
a. Yes, the products do belong to the same product category, check Yes. Move
on to step 4.
b. No, the products do not belong to the same product category, check No. Click
Submit. You are done with this task.
4. What product category do the two products belong to? Check the best-fit product
category that describes both products and move on to step 5.
5. Compare and rate the attributes of Product 1 and Product 2. Choose from the
following rating options:
a. Are the attributes for each product present and the exactly same? If yes, click
“Match”.
b. Are the attributes for each product present and partially the same? If yes,
click “Partial Match”.
c. Are the attributes for each product present but different? If yes, click
“Different”.
d. Is the Attribute present only for one product? If yes, click “Different”.
e. Are the attributes for one or both of the products missing? If yes, click “Both
Absent”.
6. Click submit. You have completed the task!
Definitions:
• Match: When the two attributes or product names means the same. They can mean
the same thing without having exact string matches.
o Ex: “Moyen” poodle vs “Medium” Poodle. Please select they are the same
• Partial Match: When only part of the attributes or product name is the same.

o Ex: 24k gold vs gold, Goldendoodle vs Sheepadoodle. Some part of the
attributes matches.
o Ex: Shampoo vs shampoo and conditioner. This is a partial match because
shampoo is part of a shampoo and conditioner set.
• Different: when two attributes or product names have nothing in common. Also,
when attribute or product name is present in one product and absent on other one
please select “Different”.
o Ex: 4k LED ray vs smart
o Ex: “Katespade” bag vs bag. If brand can’t be found for product name 2,
please select different.
• Both Absent: When attributes or product names are absent in both products, please
select both absent.
Attributes to Include in Evaluation

1. Brand: is an identity used to distinguish a specific product from all others in order
to have a competitive edge in the marketplace.
a. Example: There are billions of products for sale on the market. The Nike
brand distinguishes Nike products (Example: Nike shirts, shoes, socks) from
all other shirts, shoes and socks not sold by Nike.
2. Purpose: is the primary function of the Product. To determine the purpose, ask
“what is the product used for?”
a. Example: A rain jacket’s purpose is to protect the wearer from rain. The
purpose of a rain jacket is for “rain”.
3. Specification: describe specific features of the product. To determine the
specification, ask “what feature does this product include that stands out from other,
similar products?”
a. Example: “200mm lens”, “200mm” is the specification.
b. Example: “Retractable dog leash”, “Retractable” is the specification.
4. Model: is a unique number or name given to a line of products to identify it from
other products from the same manufacturer. The model helps manufacturers to
keep track of each product and identify or replace the proper part when needed.
a. Example: “Michael Kors Isla Ring Shoulder Tote“, ”Isla Ring“ is the model
b. Example: “Dyson dc17 replacement filer” “dc17” is the model.
5. Persona: is the intended user of the product. Who is the product made for?
a. Example: “Men’s shirt”, “Men’s” is the persona.
b. Example: “Organic baby food”, “Baby” is the persona.
6. Material: is the matter from which the product is made from.
a. Example: “wooden bar stools”, “wooden” is the material.
b. Example: “porcelain serving bowl”, “porcelain” is the material.
7. Style: is the distinctive appearance of the product, typically determined by how the
product is designed. Style pertains mostly to clothing & apparel.
a. Example: In “Fit and Flare dress”, “Fit and Flare” is the style of dress.
b. Example: In “Textured coated leather frame bag”, “Frame” is the style of bag.
8. Year: is numeric date the product was manufactured or released.
a. Example: In “Subaru Outback 2017”, “2017” is the year.
9. Shape: describes the form of the product.

a. Example: In “Round coffee table”, “Round” is the shape.
Attributes to Not Include in Evaluation

1. Marketing descriptor: are descriptive language based on opinion rather than fact.
a. Examples: “amazing”, “best”, “easy to use”
2. Color: is the visual quality such as red, blue, green, yellow, etc., that is seen when
you look at the product.
a. Example: In “Blue sweater”, “blue” is the color of the sweater.
b. Example: In “rose gold watch”, “rose gold” is the color of the watch.
3. Size: is the dimensions of the product in height, width and bulk.
a. Example: In “wedge heels 7.5”, “7.5” is the size.
b. Example: "In "small skirt", "small" is the size.
c. Example: In “Dozen apples”, “dozen” is size. Please always ignore quantity.
Cases
1. The product name is not included in the text, please use your best judgment to
infer what the product is.
a. For example, if the document features a “GoldenDoodle”, you can reasonably
infer that the product is a dog.
b. For example, if the document features “Air Jordan”, it is ok to infer that the
product is a shoe.
2. Matching attributes are not present in the highlighted product name but is
present in the same sentence as highlighted product name. Evaluate matching
attributes if they are present, either in the same sentence as the highlighted product
name or in the product name.
a. Exception: Brand and Purpose are two attributes which can be evaluated
from the entire text of the document and not restricted to just the sentence of
the highlighted products.
3. Attributes are not included in the text. Do not consider attributes that are not
present in the text in your evaluation. For example, do not assume the material of a
handbag is “leather” by looking at the image. “Leather” must be stated in the text,
“leather handbag” to be considered in the evaluation.
4. One product name is plural, the other is singular. Singular and plural forms of a
product names are a match. Please never consider quantities while comparing.
a. Example: car vs cars is a match.
5. Synonymous. For products or attributes that are synonyms, meaning they are
referred to by different words that have the same meaning, consider it a match. For
example, “waterproof boots” and “rain boots” are synonymous.
6. Products have different color and/or size. Please ignore color and size, do not
consider color and size in specification of the product.
7. Products have different marketing descriptors. Please ignore marketing
descriptors, do not consider marketing descriptors in your evaluation of the two
products.
8. When the product category pre-selected is wrong. Sometimes the prediction can
be wrong but please ignore it and complete the task with the given list of attributes.
9. The highlighted product is a generic product in a list of products. Do not
consider the attributes unless it is directly associated with highlighted product.

a. Ex: In the following example it is ok to consider Nike as a brand for gears but
do not consider women as persona

10. When dashes or commas are present in the highlighted product’s sentence, do
not consider them as sentence separators. ex: The highlighted product sentence
here is “Buy HRX active by Hrithik Roshan Green Joggers - Track pants for Men
1866566 | Myntra”.
Category
1. Animals:
Description:
This category includes all the animals like crickets, cats, dogs, birds, etc.
Product name is the name of the animal.

Examples: Dog, Cat, Fish
Attributes: Breed, Characteristics, Breeder

• Breed/Species (Model): is the breed or the variety of the animals.
o Ex: romuender “German shepherds”
• (Brand)Breeder is the seller of the animals.
o Ex: “romuender” German shepherds, “PetSmart” cats
• Characteristics: is the characteristics of the animals.
o Ex: “Long coat“ German shepherds
2. Apparel and accessories
Description:
This category includes apparel like dress, shirts, jackets etc.; accessories like hair band,
scarf, handbags, shoes, luggage, wallets, etc.

Product name:
Examples: Dress, leggings, shoes, handbags, hats, t-shirt, tank-top
Attributes:
• Brand: is the designer or name of the company.
o Ex: “Nike” running shoes
• Purpose: is the primary function of the product. What is the product used for?
o Ex: “Wedding” dress, “Hiking” sandals, “Rain” Jacket
• Persona: describes the gender for which the product is intended.
o Ex: “women’s” hat, “unisex” tee
• Style: refers to the design of the product.
o Ex: “Fit and Flare” dress, “Midi” skirts,
“tote” handbags, “windprotecting” rain jacket, “quick dry” socks, “short
sleeve” tshirt, “V-neck” tee, “backless” dress
• Model: is the specific type of product from a range of similar products.
o Ex: Kate Spade “margaux” medium satchel, Givenchy “sylvia” large
crossbody peach
o Tip: The difference between a style and model is that, model is most likely a
proper noun.
• Material: is the matter from which the product is made of.
o Ex: “Chiffon” blouse, “Soft silk” sarees
3. Baby & Toddler
Description:
This category includes products for newborns up to age 4.
Product name:
Examples: romper, crib, baby monitor, car seat, stroller, playpen, etc.
Attributes:
• Purpose: is the primary function of the product. What is it being used for?
o Ex: baby “Waist” carrier
• Persona: is the user of the product.
o Ex: “baby” monitor
• Specification: is the feature of the product.
o Ex: “foldable” car seat, “organic” baby food
4. Business & Industrial
Description:
This category includes machinery and materials used in industries and businesses.
Product name:
Examples: rods, digger, excavators, etc.

Attributes:
o Ex: “Welding” rods
o Ex: “electric” pallet jack
5. Celebration and Games
Description:
This category includes party supplies, toys, games, puzzles and products needed for
religious ceremonies.
Product name:
Examples: props, robot, scramble
Attributes:
o Ex: “bachelorette” props
o Ex: “Marble” playing board
o Ex: “stuffed” animal
6. Electronics, accessories, and vehicle parts
Description:
This category includes electronics devices such mobile phones, cameras and accessories
such as phone cover.
Product name:
Examples: phone, camera, headphone, tablet
Attributes:
• Brand: is the seller or the manufacturing company. Brand distinguishes a specific
product from all others in order to have a competitive edge in the marketplace.
o Ex: “Google” Pixel
o Ex: “Mobile” phone case
• Model: is the name of a collection or a range of products.
o Ex: iPhone “7”
o Ex: “200mm” lens Nikon
7. Food, Beverages & Tobacco
Description:

This category includes fast food items such as burgers, tacos; off the shelf food products
like chips, salsa, coke, juice; restaurant food such meals, buffets and tobacco products such
as cigarette.
Product name:
Examples: chocolate, milk, bread, butter, etc.
Attributes:
• Brand: is the seller or the manufacturing company or the restaurant.
o Ex: “Starbucks” coffee
• Specification: is the type of the product.
o Ex: “Organic” rice, "frozen" beans
8. Health & Beauty
Description:
This category includes health care such as vitamin tablets, protein supplements and
personal care such as moisturizers, shampoo, etc.
Product name:
Examples: conditioner, tablet, soap, etc.
Attributes:

o Ex: “Waldo” contact lenses
o Ex: “Anti-dandruff” shampoo
• Specification: is the feature of the product .
o Ex: “SPF 50” sunscreen lotion
9. Home improvements and furniture
Description:
This category includes furniture such as couch, table, stools; home and garden products
such as showers, kitchen cabinet, etc.
Product name:
Examples: stool, chair, coasters, etc.
Attributes:
o Ex: “"Garden" hose
• Shape: is the form of the object.

o Ex: “Round” coffee table
o Ex: “Wooden” bar stool
o Ex: “wall mounted” toilet brush
10. Jewelry
Description:
This category includes jewelry such as a chain, ring, and bracelet.
Product name:
Examples: earring, brooch, etc.
Attributes:
product from all others in order to have a competitive edge in the marketplace
o Ex: “swarovski” chain
o Ex: “Wedding” ring
• Persona: is the user of the product
o Ex: “women’s” charms
• Style: is the design of the product
o “Princess cut” ring, “heart shaped” pendant, “braided” bracelet
o Ex: Zales “Gemy” emerald ring
o Tip: The difference between a style and model is that, model is most likely a
proper noun.
o Ex: “gold” bracelet
11. Musical instruments
Description:
This category includes musical instruments such as keyboard, flute, etc.
Product name:
Examples: piano, guitar, strings, etc.
Attributes:
o Ex: “guitar” stand
o Ex: “Yamaha” keyboard

o Ex: “Nylon” strings
• Specification: is the feature of the product
o Ex: “electric” guitar
12. Pet Supplies
Description:
This category includes pet food, pet toys, cages, beds, etc.
Product name:
Examples: food, bed, leash, carriers, etc.
Attributes:
• Persona: is the user of the product.
o Ex: “Dog” food, “Pet” carriers
o Ex: “Raw” cat food
13.Sporting Goods
Description:
This category includes sports equipment and supplies such as volleyball, throwball,
snowboards, etc.
Product name:
Examples: bat, ball, net, etc.
Attributes:
product from all others in order to have a competitive edge in the marketplace
o Ex: “Fitbit” Alta
o Ex: “jogging” belt
o Ex: “inflatable” stand up board
14. Tools, parts and office supplies
Description:
This category includes tools such as screwdrivers and office supplies such as stapler, pens,
etc.
Product name:
Examples: Pen, mat, lock, etc.
Attributes:

o Ex: “car” mat
o Ex: “oak” cabinet
o Ex: “cordless” screwdriver
15.Vehicles
Description:
This category includes automobiles such as cars, trucks, vans and motorbikes.
Product name:
Examples: car, bike, van, etc.
Attributes:
o Ex: “Ford” Fusion
o Ex: BMW “Mini Cooper”
o Ex: “V6” Mustang
• Year: is the year the product was built or released.
o Ex: “2014” Subaru
16.Others
If both documents are of same category, please use your best judgment to select a category.
EXAMPLES
1. Here both the Product name is gum.
a. The brand is “run gum” which is absent in both the highlighted product.
b. Energy is the specification in product 2, In product’1 highlighted sentence the
words “pick me up” means providing energy. So they semantically mean the
same.
c. Original is a specification in Product 2 but absent in Product 1. Since only
part of specification matches it is a partial match.

2. Here the product name can be inferred as perfumes in both products.
a. Purpose and specification are not mentioned in both highlighted products.
b. Brand in both cases is “Juliette has a gun” so its a match.

3. Here the product names are different, Product name 1 is holder and Product name 2
is puzzles.
a. Purpose 1 is playing cards, Purpose 2 is absent so select different
b. Brand 1 is imperial home, Brand 2 is The Legend of Zelda (extracted from the
sentence of highlighted product) so select different.
c. Material 1 is plastic, Material 2 is absent so select different.
d. Specification 1 are set of 2, hands free and Specification 2 are only set of 2 so
select partial match.

4. Here the Product Name 1 is keyboard and mouse, Product Name 2 is mouse so select
partial match.
a. Purpose is absent in both products.
b. Brand is Microsoft in both products.
c. Model 1 is 600 and Model 2 is 3600. This is different and please do not select
partial match. They refer to different models.
d. Specification 1 is wired, Specification 2 is Bluetooth so select different.
i. Never consider color as part of specification.
5. Here two highlighted products do not fall under same category so please select “No“
for ”Are the of same product category?“
a. Note: The image is only for additional context, please use the text to complete
the task.

6. Here the Product Names are different (bodysuit and leggings).
a. Purpose is absent.
b. Persona is “Plus size” in both.
i. Plus size is not a size like US 7, it is a target audience like petite.
c. Brand is absent in both products.
i. Only look for attributes beyond the highlighted product if it’s present
in at least one product.
d. Model is absent.
e. Style 1 is floral, Style 2 is trim so select different.
f. Material 1 is velvet and Material 2 is faux leather so select different.

7. Here both product names are “Futons”.
a. Purpose is absent in both cases.
b. Shape is absent in both.
c. Material is absent.
d. Specification is also both absent. Do not consider color as part of
specification. Always ignore color.
e. Even when images don’t load, please do the task with the given text. You can
use your favorite side search to get more contexts.

Stanford Guidelines Raters 11.22.19 PDF

Uploaded by

Copyright:

Available Formats

You might also like

Stanford Guidelines Raters 11.22.19 PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Stanford Guidelines Raters 11.22.19 PDF

Uploaded by

Copyright:

Available Formats

Stanford Guidelines

Updated November 22, 2019

Determine the level of sameness between two highlighted products by comparing

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

Product Product 1 Product Product Product 2 Product Product

Before You Begin

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

Attributes to Include in Evaluation

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

Attributes to Not Include in Evaluation

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

Product name is the name of the animal.

Attributes: Breed, Characteristics, Breeder

2. Apparel and accessories

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

3. Baby & Toddler

4. Business & Industrial

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

5. Celebration and Games

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

Examples: conditioner, tablet, soap, etc.

• Brand: is the seller or the manufacturing company. Brand distinguishes a specific

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

11. Musical instruments

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

12. Pet Supplies

14. Tools, parts and office supplies

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

CONFIDENTIAL TO APPEN-DO NOT FORWARD Last Updated 11/22/2019

You might also like