WPX Search Relevance

Search Relevance: Standard Operating Procedures
WPX Search Relevance SOP (Version 3.1)
Contents
WPX Search Relevance SOP (Version 3.1) .......................................................................................................................... 1
Change Log: ........................................................................................................................................................................ 2
Task Overview .................................................................................................................................................................... 3
Illustrative Examples ...................................................................................................................................................... 4
Assessing the Relevance Match Labels .............................................................................................................................. 5
Product Attributes and Impact of Mismatches .............................................................................................................. 6
Additional Important Considerations .......................................................................................................................... 12
Product Variations from Product Detail Page .......................................................................................................... 12
Ambiguous Queries.................................................................................................................................................. 13
Misspelled Queries .................................................................................................................................................. 14
Broad Queries .......................................................................................................................................................... 14
In-Category Search ................................................................................................................................................... 14
Very Vague Queries ................................................................................................................................................. 15
Products Not Sold on Amazon ................................................................................................................................. 15
Labeled Examples ............................................................................................................................................................. 16
Appendix .......................................................................................................................................................................... 20
Examples of Applying the Labeling Process ................................................................................................................. 20
Example: pickup tents for camping short bed.......................................................................................................... 20
Example: bobs shoes with dogs on them ................................................................................................................. 22
Working with Categories.............................................................................................................................................. 24
Category specific examples .......................................................................................................................................... 26
1. Electronics .................................................................................................................................................... 26
2. Media and Games ........................................................................................................................................ 26
3. Fashion ......................................................................................................................................................... 28
4. Luggage and bags ......................................................................................................................................... 31
5. DIY & Tools ................................................................................................................................................... 32
6. Beauty .......................................................................................................................................................... 33
7. Medicines ..................................................................................................................................................... 34
8. Grocery......................................................................................................................................................... 34
1|Page
9. Fresh……………………………………………………………………………………………………………………………………………………….38
10. Pets…………………………………………………………………………………………………………………………………………………………39
11. Toys .............................................................................................................................................................. 40
12. Home ............................................................................................................................................................ 40
13. Kitchen ......................................................................................................................................................... 41
14. Automobiles ................................................................................................................................................. 41
15. Shoes ............................................................................................................................................................ 41
16. Theme .......................................................................................................................................................... 41
17. Adult ............................................................................................................................................................. 42
18. Books ............................................................................................................................................................ 42
19. Brand affinity ............................................................................................................................................... 45
20. Subscription ................................................................................................................................................. 46
More Labeling Nuances.................................................................................................................................................... 46
Change Log:
Change description Page Date
Size: Rule updated 10
Genre: Updated example 10
Other Attributes: Added example 11
Ambiguous queries: Explanation updated 13
In category Search: Example updated 14

2023-01-12
Very Vague queries: Added example 15
Unjudgeable: Explanation added for foreign language 18
Category specific examples-Media & Games-Video games: Added example 27
Category specific examples-Media & Games- Movies/Prime video: Complementary section 27

updated
Category specific examples-Fashion- Softlines: Complementary section updated 28
2|Page
Category specific examples-Beauty- Body care & products: Added example 33
Category specific examples-Pets: Updated example 39
Category specific examples-Toys: Updated example 40
New category addition of Brand affinity 45
New category addition of Subscription with revised examples and decisions 46

2023-03-20
’More labelling nuances’ section updated with examples for Irrelevant 46
New example under grocery 34
Task Overview
The goal of the task is to assess the relevance of the search results returned to the Amazon customers. Each task will have a
search query and an e-commerce product as the input, and the annotator will provide their judgement on the type of a relevance
match between the query and the product. The context of the search query (SQ) is specified as
1. Search query/keywords, e.g., red nike shoes

2. Search category, e.g., aps (stands for All Products Search)
3. Marketplace where the search takes place, e.g., amazon.com.
In addition to the above, the judges will be provided links to the Google and Amazon search engine results for the queries as well
as Google search engine results restricted to Amazon retail site.
The information about the offered product (OP) consists of
1. Product title,
2. Product image,
3. Current/live product detail page (PDP) accessible via a link in the user interface (UI). (This is usually done by providing
the Amazon Standard Identification Number, or ASIN, for the offered product.)
The annotators will determine and mark the most likely type of the relevance match for the query-ASIN pair from the perspective
of a reasonable Amazon customer. The high level description of the types of Relevance Match is provided below.
Match Type Description
Exact The offered product fully satisfies the intent of the query, matches on all intended product attributes.
The offered product does not fully match the intent of the query, but serves the same purpose as the intended
Substitute
product and could be used instead to address the customer’s need.
3|Page
The offered product does not serve the purpose of the intended product, but could be of interest to the customer
Complement either because it could be used or worn together with the intended product, or because the intended and the
offered products are related by brand, genre, theme, occasion, or a similar non-functional characteristic.
The offered product is either entirely unrelated to the intended product or is unsuitable for the customer due to a
Irrelevant
critical to the intent of the query mismatch on a product attribute.
The relevance match relationship cannot be assessed because the query cannot be understood, the query has a
Unjudgeable non-product intent, or the information about the offered product is not available or not sufficient to determine if it
matches the query.
Illustrative Examples
Query and Offered Product Product Image Judgement
Query: red hanes cotton tshirt Exact
Category: aps (All Products) The offered product matches the query on
Marketplace: amazon.com all intended product attributes: product
type (shirt), color (red), brand (Hanes),
Product Title: Hanes T-Shirt, material (cotton), shirt type (t-shirt).
Beefy Classic Crewneck Cotton
Tee
ASIN: B00WQMX5G6
Query: panasonic waterproof Substitute

electric shaver
The offered product serves the same
Category: aps (All Products) purpose as the intended product (shaves),
Marketplace: amazon.com matches on the product type (shaver),
feature (waterproof), another feature
Product Title: Braun Electric (electric), but mismatches on brand
Razor for Men, Waterproof Foil, (Panasonic intended, but Braun
Shaver, Series 7 7027cs, Wet & suggested).
Dry Shave, With Beard Trimmer,
Rechargeable, Charging Stand
and Travel Case Included, Silver
ASIN: B07Z8P11Y5
4|Page
Query: samsung galaxy z fold 4 Complement
Category: aps (All Products) The offered product is designed to be

Marketplace: amazon.com used together with the intended product,
has a product type (smartphone case)
Product Title: SUPCASE Unicorn which is complementary to the intended
Beetle Pro Case for Samsung product type (smartphone) and is
Galaxy Z Fold 4 5G (2022), Full- compatible with the intended product
Body Dual Layer Rugged Case model (Samsung Galaxy Z Fold 4).
with Built-In Screen Protector &
Kickstand & S Pen Slot (Black)
ASIN: B0B7MC26MS
Query: nike sneakers for men Irrelevant
Category: aps (All Products) While the offered product matches the
Marketplace: amazon.com intended product on the product type
(shoes), shoe type (sneaker), and brand
Product Title: Nike Women’s Flex (nike), the offered product fails to fulfill
Trainer 4 Sneaker the purpose of the intended product due
ASIN: B07HDS7TVT to a gender mismatch (as shoes designed
for women typically have a different width
from those designed for men).
Query: my past purchases Unjudgeable
Category: aps (All Products) The query has a help intent, not a product
Marketplace: amazon.com intent.
Product Title: JUST MY SIZE

Women’s Apparel Women’s Plus
Size 2 Pocket Pull on Short,
Indigo, 2X
ASIN: B075LSDMDW
Assessing the Relevance Match Labels

While providing the labels for many of the tasks will be intuitive and straightforward, it is strongly suggested to follow the
following basic steps when assessing the relevance, especially when encountering non-trivial instances or when training.
5|Page
1. Understand the query: research and identify the intended product type, the broader purpose of the product, and the
desired product attributes; use Google or Amazon search results for the provided queries, and collect additional
information from third party sites if needed. If the query cannot be understood or interpreted or does not have a
product intent, then assign Unjudgeable label and stop.
2. Understand the ASIN: research and identify the offered product’s attributes corresponding to those identified in the
query; use the product title, image, information on the PDP (product detail page) including the product description, and
in the rare cases of inconsistency among the product images, title, and description, use the information in the product
reviews and Q&As. If the product cannot be understood, then assign Unjudgeable label and stop.
3. Identify the product attribute matches and mismatches: for each of the understood product attributes in the query,
determine whether query and the offered product agree, and take a mental inventory of the attribute matches and the
attribute mismatches (the latter also referred to as defects).
4. Assign the label based on the identified defects: If all of the attributes match, then assign Exact label. If the intended
purpose is preserved and none of the defects make the product unsuitable, then assign Substitute label. If the offered
and intended products can be used together or are of the same brand/genre/theme/occasion and product category, and
if none of the defects make the offered product unsuitable, then assign Complement label. Otherwise, assign Irrelevant
label. See Product Attributes and Impact of Mismatches.
Examples of Applying the Labeling Process contains a few illustrations of applying the process above.
Product Attributes and Impact of Mismatches

E-commerce search queries contain mainly the information about the attributes of the desired product (referred to as DP
throughout this section). The table below lists the most common types of product attributes that the judges are likely to
encounter together with how the mismatch on the attribute would impact the assigned relevance match label.
Important note: any critical attribute defect makes the offered product unsuitable for the intent and leads to Irrelevant label. It
would take only one such defect among the attributes to lead to Irrelevant label, regardless of the matches on other
attributes. For example, for a query nike size 9 black shoes for men, a pair of size 9 black nike women’s shoes would be Irrelevant
despite matching on the product type, brand, color, and size.
Relationship
Attribute Definition Mismatch Examples
Type
Purpose is the minimum viable SQ: mens belt
shopper’s need as described or implied OP: belt sander
by the query. Explanation: The purpose
is to keep the pants in Irrelevant
When assessing whether the OP serves place, not to shape wood.
the same purpose as the DP, consider Exception: OP
The OP is unrelated
whether the task which the DP is SQ: office desk and DP are of
to the purpose of DP.
designed for can be accomplished by OP: coffee table the same brand
This implies that OP
the OP. neither fulfills the Explanation: The purpose (see Brand
is to acquire a surface Affinity related
Purpose purpose of DP nor
Examples: can be used as an capable of supporting the products and
Brand Affinity
- [coffee grinder] refers to a product accessory to DP (see activities for office work
(e.g., writing, reading, unrelated
with the utility of grinding coffee Lead product vs
computer work). A coffee products),
beans into a coarse powder. Other accessory).
table’s purpose is to genre, or
devices able to do this, e.g., theme/occasio
support beverages,
multipurpose crusher, would serve n.
remote controls,
the same purpose.
decorative objects, and
- [acne cream] refers to a product small items (wiki), not
with the utility of reducing the designed for office work.
6|Page
symptoms of acne. There are

products not specializing in acne
treatments but which nonetheless
can be used for that purpose, e.g.,
tea tree oil.
SQ: Migrain relief pill
What is a product type?
OP: Headache & Migraine
OP does not match relief hat
In SQ, product type is the term or the product type of
phrase that captures the idea of the DP but can be used to Substitute
product the customer is looking for. In SQ: treadmill
satisfy the same
an item, it is a class of common generic utility/purpose as DP. OP: Elliptical Machine
goods or services. Amazon catalog Explanation: Both are
would contain hundreds or thousands cardio fitness equipment
of products of the same product type. If SQ: 13-inch MacBook Air
a term in the query or a product is laptop
covered by another product attribute, it OP is an accessory to
OP: Carrying case Complement
should not be a part of the product DP.
compatible with a 13-inch
type. MacBook Air laptop
OP is either an SQ: office desk
Examples in queries:
independent or an OP: office chair
- Simple: [popcorn], [4 ft rug square]
interdependent Complement
- Keeping it minimal: [dri fit t shirts for
complementary SQ: shower curtain
men], [mens black loafers dress
product to DP. OP: shower curtain rod
shoes], but [raised garden bed]
- Can refer to an accessory to another DP is an accessory,
product: [charger for dell laptop] and OP is a different SQ: iphone 13 case
Irrelevant
accessory to the lead OP: iphone Charger
Product product.
Type SQ: iphone 13 case
Lead product vs accessory OP is the lead
OP: iphone 13 Irrelevant
product to the DP.
smartphone
Some products are commonly used SQ: Electric Guitar
together but can be either sold OP is a bundle, and
OP: Electric Guitar, Exact
separately or together, e.g., a Quest 2 DP is its lead product.
amplifier, stand, cover
Pro Bundle (ASIN: B09DCM6CNZ)
consists of a virtual reality (VR) headset, OP is a bundle, and
an improved head strap, and a carrying DP is part of this
bundle with another SQ: Shampoo
case.
either inter- or OP: Shampoo + Substitute
- If a product can fulfill its basic
independent Conditioner
purpose on its own without
complementary
depending on other product, it is product.
considered a lead product (the VR
OP is a bundle with a SQ: pastel paper pad
headset).
lot of items, and DP is OP: 54-piece drawing and Irrelevant
- An accessory is a product used to
just one of the items. sketching art set
enhance the lead product, but
OP is a bundle, and
cannot function on its own (the head SQ: ipad cover
DP is an accessory in Irrelevant
strap and the carrying case). OP: ipad with a cover
this bundle.
- Sometimes the products are
DP is a bundle, and SQ: scanner with scanner
interdependent in order to function
OP product is the cover Substitute
(e.g., camera body and lens) in which
lead product in it. OP: scanner
7|Page
case the items are complementary,

but neither is a lead product or an
SQ: printer and scanner
accessory.
OP: scanner
- Similarly, products can function DP is a bundle, and
independently, (e.g., shampoo and OP is in it but not a Complement
SQ: scanner with scanner
conditioner combo in separate lead product.
cover
bottles) in which case they are also
OP: scanner cover
complementary without either
being an accessory or a lead product.
The brand of OP is
not the same as that Query: Nike shoes
of DP, but the Offered Product: Puma Substitute
By brand we refer to a proprietary products have the Shoes
language, licensed trademark, same functionality.
manufacturer, or a retailer associated SQ: Dove shampoo
with the intended product. Brands can Brand Affinity, OP: Dove deodorant
also market series of products under a related products Explanation: Both are
different name referred to as a product The brand of OP is personal care items
line. We won’t distinguish the two for the same as that of sharing the brand
the purpose of defects. DP, and the products
are neither SQ: yogalicious biker Complement
Examples of brands (and possibly, functionally shorts
Brand or product lines) in queries: [nike jordans equivalent nor OP: yogalicious fleece
Trademark shoes men], [garmin fenix 5], [almay complementary, but robe
concealer] are from the same Explanation: Both are
product category. clothing items sharing the
Shoppers build a connection based on brand
their experience with the branded Brand Affinity,
products and develop an affinity unrelated products
towards them. This affinity can The brand of OP is
potentially make the offered product the same as that of
relevant to the shopper even if the SQ: Samsung laptop
DP, the products are
product does not have the intended OP: Samsung tv remote Irrelevant
neither functionally
functionality. control
equivalent nor
complementary, and
do not have a related
purpose.
Model usually refers to a smaller class
of products within a product line. The
class could be further restricted to a The specific model of
Model/Mo specific product with the model OP is not the same as
SQ: Samsung S7
del number. that of DP, but the Substitute
OP: Samsung S10
Number products offer the
Examples of models and model same functionality.
numbers in queries: [logitech k860],
[samsung galaxy z flip 4]
DP is an accessory to SQ: Charging cable pixel
Because of the specific design of the
Compatibil a lead product. OP is phone
products, the accessories for them Irrelevant
ity the same type of OP: mini-usb charging
sometimes need to be compatible, i.e.,
accessory, but is not cable
8|Page
specialized to be used with the lead compatible with the Explanation: Pixel phones
product. that lead product. use USB-C cables
Note: sometimes the

products are marked
as universal meaning
that they are
compatible with all of
the products of a
particular type/class.
SQ: books for a 6-year old
OP: book for a first-grader
OP overlaps on the
Explanation: 1st grade Exact
age range with DP.
typically means 6-7 years
old
OP is +/-1 age range
from DP. If the size is
specified by age, e.g.,
3-6 months baby
shoes, then one
range apart, e.g., 0-3
months or 6-9
months baby.
The age attribute indicates an age range Note: +/-1 age range
intended or suitable for a product. is a high-level
guidance as age SQ: shoes for an infant
Examples in queries: [board games for a ranges are
starting to walk Substitute
3-year old], [outfits for a 6-month old] sometimes difficult to
Age OP: shoes for a toddler
establish. Try
Note: when the age group is used in a answering a
clothing related query, it refers to the question: is OP
clothing size. Same mismatch guidance potentially suitable
applies. for the age range of
DP? If the age range
is not matching
perfectly, but the
product is still likely
to be suitable, then
the product is a
Substitute.
OP is unlikely to be
suitable for the age
range of DP (general SQ: toys for toddlers
guidance: more than OP: toys for a 9-14 year Irrelevant
one age range old boy
difference from the
intended).
Gender attribute refers to the gender SQ: Women’s pants
OP is not designed
Gender that the intended product is designed or OP: Men’s pants Irrelevant
for the explicitly
styled for, one of male, female, or
9|Page
unisex (suitable for either female or expressed gender in

male shopper). DP.
Note: intended gender is limited to

human beings only.
The intended species attribute captures
what species of pets/pests the product
OP is not designed to SQ: Dry food for dogs
Intended is designed for/against.
work for the species OP: Food pellets for Irrelevant
Species
in DP. rabbits
Examples in queries: [flea medicine for
cats], [dog leash], [mosquito repellant]
The colors of OP do
Color attribute refers to a significant SQ: Red bag
Color not match the color Substitute
color in product appearance. OP: Black bag
description of DP.
OP is not the same
size as DP but differs
in up to +/-1 size unit.
This applies to
SQ: Boots size 6
clothing and shoes
OP: Boots size 7 Substitute
only. (no longer
Size refers to physical dimensions of the applies to table,
product, be it in values with or without towel, blanket as
units, adjectives, or product specific these of any size can
descriptions. be used)
OP differs by more
Examples in queries: [large litter box], than +/-1 size unit
Size
[32” tv], [mens dress shoes size 9] from DP.
Alternatively, OP is of
Since different brands may size label a different size than SQ: 34” mens levis jeans
fluctuations (primarily for clothing), we DP, and DP is OP: 32” mens jeans
consider providing some leeway in the designed to fit
Irrelevant
labeling. another product SQ: king size bedsheet set
(e.g., cushion cover, OP: queen size bedsheet
pillow cover, fitted set
sheet, duvet cover,
bed sheet, mattress
or mattress cover,
curtain).
OP does not refer to
the same entity but is
in the same or
SQ: batman tshirt
Genre refers to classes of creative work overlapping genre
OP: superman tshirt/ Plain
in media. Other non-media items can with the entity for
tshirt
be themed with entities who would DP. This rule can Substitute
Genre (overlapping genres:
have genre associated with them, e.g., refer not just to
superhero)
harry potter action figures, frozen- media item, but to
themed merchandize. items which use the
likeness of media
characters.
Genre affinity SQ: jazz music Complement
10 | P a g e
OP is a media OP: Jazz Dance Level 1

product of the same video
genre as DP, but is of
a different media
type.
Theme is an idea or point central to the SQ: Halloween wreath
appearance of or association with the OP does not match OP: Christmas wreath
product. Theme could be associated the broad theme or Irrelevant
with media characters or franchises occasion of DP. SQ: Cricket jersey
(e.g., Avengers), sports (e.g., cricket), OP: Football jersey
visuals (e.g., images of cats), or
Occasion/T occasion/event (e.g., wedding,
heme Christmas). Theme provides a Occasion/Theme
connection between the products, with affinity SQ: 4th of July mugs
customer potentially interested in other OP is not a functional OP: July 4th American Flag Complement
products with the same theme. Not substitute for DP but Banners Garlands
matching the desired theme is a critical matches its theme.
defect, making the offered product
unsuitable.
Likeness of the product to religious SQ: Buddhist necklace
symbols or references to religious texts, OP: Love Cross necklace
OP does not match
Sensitive and medical or dietary supplements
DP on sensitive Irrelevant
Content context of use of the product is sensitive SQ: back pain
content.
to the customer. Not matching on this OP: Medicine primarily for
content makes OP unsuitable. headache
SQ: waterproof shoes
OP: shoes (not
waterproof) Substitute
Label: Substitute
Explanation: customers
might still consider other
sturdy shoes which are
not designed as
waterproof.
This is a catch all rule for the remaining
attributes, e.g., style, feature, material. SQ: cotton shirt
OP has a different OP: shirt (made of less
If the mismatch in the attribute changes
Other style, feature, than 50% cotton)
the function of the product, the
Attributes material, etc. than
resulting defect would be critical, and Label: Substitute
that of DP. Substitute
the resulting label would be Irrelevant. Explanation: customers
Otherwise, it would be Substitute might still consider the
product if other
characteristics are
satisfied the product is
otherwise attractive (e.g.,
price).
SQ: Kettle for 110v

OP: Kettle 210v
Label: Irrelevant
11 | P a g e
Explanation: Customer is
specific for a
compatibility/feature of
product searched. Voltage
difference could result in
short circuits or result in
Irrelevant
damaging of products.
Voltages are also very
specific to countries and
states. Thus, the defect in
the queries is to be
treated as incompatibility
differences and rated as
Irrelevant.
Additional Important Considerations
Product Variations from Product Detail Page

When customers search for products which have variants of the same model or product line, instead of returning all of such
variants in the results, Amazon product search UI returns only one representative product for all such variants. The customer is
given a choice to select the desired variant either from the ASIN impression in the search results or from the product detail page.
The judges are expected to check the product detail page for available variants when making their assessment and to select the
variant which matches the query the closest.
Example
Keywords: brooks running shoes black
Category: aps (All Products)
Marketplace: amazon.com
Product Title: Brooks Ghost 14 Men’s Neutral Running Shoe
ASIN: B08QVGSNCR
The search result page lets the customer select some of the color variants for each returned model of shoes. The product detail
page allows the customer to select both the color pattern and shoe size. The product image and other information available in the
task UI is unlikely to provide the information about all of the product variant choices available to the customer. The judges are
expected to check the product detail page for available variants when making their assessment and to select the variant which
matches the query the closest following the sequence Exact, Substitute, Complement, Irrelevant in that order. For this search
query, the faceout variant matches on all attributes and is an Exact match. Searches for brooks running shoes grey or brooks
running shoes orange would both be Exact matches for the ASIN below since both colors can be selected from the PDP, but for a
search for brooks running shoes gold it would be a Substitute since no variant selections have gold as the primary color.
12 | P a g e
Ambiguous Queries
A query is considered ambiguous if it has multiple possible interpretations. For example, a query with keywords glasses could
refer to sunglasses or reading glasses, or to drinking glasses. The annotators can mark all these types of glasses as Exact. For
example, SQ is query cat pajamas and OP is pajamas with cat images on them, pajama costume of a cat, or pajamas for cats
would all be marked Exact. For example, SQ is reading glasses and OP is sunglasses then it would be marked Irrelevant as the
feature is crucial and is defeated.
13 | P a g e
“Annotators is required to mark the query as ‘Ambiguous’ by adding additional comment in SDAT without fail when they notice
the query/ASIN is ambiguous in nature”.
Misspelled Queries
For misspelled queries, use the intended spelling when assessing the relevance match type. For example, query bumber sticker
very likely refers to bumper sticker. If there is a product interpretation for the query without a spelling correction, give the
customer the benefit of the doubt and use the unmodified spelling. For example, cotreau square drinking glasses orren ellis
refers to a product not sold on Amazon as of this writing (link), and the query should not be spell corrected despite Amazon’s
search engine doing so (as indicated by the preamble below shown on the search page).
If there are several plausible spelling corrections depending upon which letters are substituted, pick one that will result in the
closest relevance match label, following the same guideline as for Ambiguous Queries. If the query is unclear, consider all
common misspellings as different interpretations for the query, you can use Google or Amazon search to identify common spell
corrections.
Broad Queries
Some of the queries would be exploratory in nature, providing only a basic information about the product, typically, either just a
category, a product type, or a brand. These broad types of queries could potentially have thousands of relevant matches, and any
of the products matching the intent are considered Exact matches. Some broad queries may not admit Substitute or
Complement labels since the function is very broad to establish Substitutes, and not enough information about the product to
establish compatibility relationship.
For example, any jewelry item would be considered an exact match for query with query jewelry. Any product made by Reebok
would be considered an Exact match for a query reebok.
In-Category Search
Most queries will reference All Products search category, meaning that the customer issuing the query did not specify a product
search category. (All Products is the default, and unless otherwise mentioned, assume that the queries refer to All Products.)
When a search category is specified, it can be utilized to disambiguate and focus the intent of the query.
Example: Marvel is a broad query referring to products related to the media franchise spawned by a series of Marvel comics.
Absent additional information, the query can refer to comic books and books, movies, games, toys, clothing, and other items
branded with the images and likeness of the characters and the items from the franchise. However, if the query references Toys
& Games product category, the query intends the product type to be toys or figurines (e.g., a spider-man lego toy – exact,
Captain America lego toy – Substitute), and products of other types would be considered as Irrelevant (e.g., a captain america t-
shirt, a green lantern t-shirt)
Exception: When the intent based on the search keywords are inconsistent with the provided category, ignore the provided
category and assume All Products, e.g., jackets with category Electronics.
14 | P a g e
Very Vague Queries

Occasionally, a customer would enter either an incomplete query or one so ambiguous that it is not clear what product it might
be referring to. These queries would typically be with All Products Category as knowing the category can sometimes be used to
disambiguate such queries. If it is not possible to infer a likely interpretation, mark it as Unjudgeable as the labels collected
under unlikely interpretations are not useful in the downstream applications.
Examples:
Keywords: waterproof
Judgement: Unjudgeable
Explanation: Waterproof very likely refers to a property of the product, but it is not clear what product is being referred to: bag,
jacket, casing for cameras or mobile devices, shoes, tarps, etc. Rather than to guess the intent, mark it as unjudgeable.
Keywords: metal
Judgement: Unjudgeable
Explanation: Metal could refer to the material of the product and to genre of music, or even to media or videogame titles. There
is not enough information to judge the intent, so mark as unjudgeable.
Keywords: metal
Category: music
OP: Metallica LP
Judgement: Exact
Explanation: Category is specifically mentioned and any music album of the genre Metal will be rated Exact, overlapping genre to
be rated as Substitute and different genre to be rated as Irrelevant.
Products Not Sold on Amazon

If the query is for products not sold on Amazon like cat, dog, gun and car etc., and the ASIN shows product related to the same
query, which can be used for/buy the product not sold on Amazon to be marked as Exact. However, merchandise which only
depicts items that are not sold (printed posters, pet printed tshirt/mugs, poster of cars, toy or airsoft gun, Image of gun etc.)
would be considered as Irrelevant.
Keywords: cat
Product: Cat food
Judgement: Exact
Explanation: The object referenced in the query is not sold on Amazon, any product related to the query to be marked as “Exact”.
Keywords: Cat
Product: A figurine of a dog
Judgement: Irrelevant
Explanation: The product is not related to any potential interpretation of the query.
Keywords: Car
Product: A poster with a car
Judgement: Irrelevant
Explanation: Poster cannot be of any use to the queries product (car). Hence should be marked as “Irrelevant”. However, if the
ASIN shown is “car repair book” then judgement would be “Exact”.
15 | P a g e
Labeled Examples
Exact: An offered product an Exact Match when the offered product is exactly the same as the search query. A very specific query
will only be satisfied if the product matches all of the characteristics included in the query.
Substitute: An offered product is a Substitute when it differs from the intended product, but it is functionally equivalent. This will
be the same type of product, but may differ by brand, model, color, title, or other details. The offered product may also meet the
purpose or the utility of the desired product even though it is not strictly of the same product type.
Note: Any critical matching defect/attribute mismatch which makes the recommended ASIN unsuitable in the context of the query
should lead to the ASIN judged as Irrelevant. (See Product Attributes and Impact of Mismatches.)
Complement: A product is complementary to the SQ when it could be useful to the customer if they did buy the intended product.
This could be an accessory, a component, replacement parts, or even another item designed to or could reasonably be used
together with the product. More broadly, the product can be considered a complement if the customer could be interested in it
because of the brand affinity (same brand) or because the offered product follows the same theme, genre, or occasion as the
intended product.
Note: Any critical matching defect/attribute mismatch which makes the recommended ASIN unsuitable in the context of the query
should lead to the intended product judged as Irrelevant. (See Product Attributes and Impact of Mismatches.)
Irrelevant: A product is Irrelevant to the SQ when it does not satisfy the criteria for either Exact, Substitute, or Complement label,
but when the query and the product information are clear enough to evaluate the type of the match. The offered product has
failed to match the purpose like: utility, size, age, gender, compatibility of the desired product, or the offered product fails to
create interest of a purchase by not aligning with the desired product on brand, genre, theme. (See Product Attributes and Impact
of Mismatches.)
Assume All Products category and amazon.com marketplace for all examples in the table below
Query Exact Substitute Complementary Irrelevant

Clorox bleach Clorox Performance Lysol Disinfectant Spray Clorox Bathroom Foamer Glad ForceFlex Tall
Bleach, 121 Oz, Pack of 3 Refillable Cleaner Kitchen Drawstring
Trash Bags
Brand affinity, related

products (home care Products do not
category) serve functionally
the same purpose.
Same purpose
(disinfectant)
Coleman 225 Coleman RoadTrip 225 Char-Griller E1515 Patio Bernzinatuc Disposable GasOne Camping
portable grill Portable Stand-Up Pro Charcoal Grill, Black Propane Cylinder, 16.4- Fuel Blend Isobutane
Propane Grill Oz Fuel Canister
16 | P a g e
Same purpose/function, Not compatible with

different brand/model The grill takes 16.4-Oz DP -- defect
propane cylinders – an
accessory to DP.
Navy blazer for Mens Casual Sport Coat 1 Men’s Standard-Fit Stretch Haggar Men’s Dress Blazer Jacket for
men Button Suit Blazer Twill Blazer Women
Can be used as an
independent product
with DP as part of an
outfit Designed for
Does not come in
navy/blue women, won’t fit
men – defect wrt DP
Otterbox case OtterBox Defender Spigen Liquid Air Google Pixel 7-5G
for pixel 7 Series case for Google Designed for Pixel 7 Case Android Phone
Pixel 7
Compatible with DP, Lead product when

different brand DP is an accessory
Unicorn Kids Girls Backpack Unicorn Kids Luggage, Robo Pets Unicorn Toy Disney Princess
backpack Preschool Unicorn Girls carry on suitcase Blanket
with backpack
17 | P a g e
OP is a bundle with DP as Product does not

Same theme, different
one of independent match the theme.
product.
products.
PS5 Console PlayStation 5 Console – Sony Playstation PS4 1TB Call of Duty: Modern SIKEMAY PS5
Horizon Forbidden West Black Console Warfare II – PlayStation Controller Skin
Bundle 5
Bundle with the console

Same purpose (gaming
as the lead product
console), different model
The product
neither serves the
purpose of nor is it
Game compatible with
an accessory to the
the DP
DP. (It is an
accessory to an
accessory.)
Unjudgable: This judgment is only to be used if the product cannot be judged, either because the query cannot be understood, or
because the product is lacking enough information to determine if it matches the query. An Unjudgable query is one that
a. is complete gibberish,
b. is seeking help,
c. is attempting to navigate to a website,
d. is in a non-market/foreign language (Primary/target language to be considered for the particular marketplace
unless the annotation language is specified clearly. Any language other than primary/ target to be marked
unjudgeable. Ex: French marketplace – primary language is FR and queries or offered products in any other
language like English/ Dutch to be marked Unjudgeable. If it’s a combination of French and English that is written
due to Brand name, adjectives or nouns which can be interpreted may be judged accordingly. Ex: Puma des
chaussures (Puma Shoes)- can be judged.
e. is a string of ASINs, numbers/symbols that does not correspond to an ISBN, ASIN or model number in Google
searches. *A single ASIN or ISBN number, with corresponding results on Google, is judgeable,
f. is very vague without a clear intent (see Very Vague Queries).
18 | P a g e
The query-ASIN pair is also unjudgable if the product/ASIN information is insufficient to render a judgement due to not having
access to all of title/picture/product detail page link, or due to having ASIN information in a non-market/foreign language.
S. No. Search Query ASIN/ISBN Category Judgement Reason

1. 823^%0* Socks (B0895LL44V) aps Unjudgable Gibberish
2. Flipkart.com Pen (B082PN4X5J) aps Unjudgable Navigational Query
3. i) 博朗 Chocolate aps Unjudgable Foreign Language

ii) chocolate寒服 (B08J37H3QK) A query, half in
target language/
English and half in
foreign language
will also be
unjudgeable.
Exception: If query
has noun, adjective
or Brand names
then to be judged.
4. How to cancel Women’s black Top aps Unjudgable Service-related
my prime (B07SN3KMS6) query
membership?
5. Volume of The Ghost and the Hemispheres aps Unjudgable Non-product type
hemisphere with Kindle edition query
radius 15cm? (B08KNSWQ5G)
6. Gift for mother Notebook aps Unjudgable Product detail page

(searched in (B08TYJYD71) in other language
.com EN)
19 | P a g e
Appendix
Examples of Applying the Labeling Process
Example: pickup tents for camping short bed
Query: pickup tents for camping short bed

Product Title: Napier Backroadz Truck Tent, Grey/Green, Full

Size Regular Bed (6.4’-6.7’)
ASIN: B07HXC4HBD
Step 1: Understand the query: Searches on both Amazon and Google (note: the UI will typically provide a link) return tents
mounted in a bed of a pickup truck, broadly used for sleeping while camping. There is no other obvious interpretation for this
query. The query implies product type tent, with camping as the intended use, compatible with a pickup truck with a short bed.
20 | P a g e
Step 2: Understand the ASIN: From the title and the picture, the product is a tent compatible with the full size regular bed pickup
truck. To get more information on the intended use of the product and its compatibility, head over to the product detail page (the
UI will typically provide a link). The product detail page indicates car camping as the recommended use for product.
Step 3: Identify the product attribute matches and mismatches: The ASIN matches the query on the product type (tent), use of
product (camping), overall compatibility (pickup truck, mentioned on the product detail page as Design: Vehicle Mounted Tent).
There is a pickup truck bed size compatibility defect. The keywords mention short bed while the ASIN is for full size regular bed
(6.4’-6.7’). How bad is this defect?
One of the available images (4th from the top on the left) is a model compatibility sizing chart (see below) which indicates that
short bed has 5’-5.8’ length rather than 6.4’-6.7’ length which this model of the tent is designed for. The mismatch is two sizes
(ASIN model is 19022, the one matching the query is either 19033 or 19066, and there is an intermediate size model, 19044, for
the bed length 6’-6.3’).
21 | P a g e
Step 4: Assign the label based on the identified defects: There is only one defect, compatible bed size length. Based on the
information available to us, the defect is significant (two model sizes, unlikely to fit the vehicle) likely making the ASIN
incompatible with the intent of the query leading to Irrelevant label.
Example: bobs shoes with dogs on them
Query: bobs shoes with dogs on them

Product Title: Skechers Women’s Slipper

ASIN: B08559LHNW
Step 1: Understand the query: The query likely refers to shoes with images of dogs on them. What is bobs referring to? A search
for the keywords on Google indicates that Skechers likely has a Bobs line of shoes (see below). An additional search for skechers
bobs turns up a page for the product line (see below). Therefore, the query can be interpreted as having a shoe product type, with
brand Sketchers, product line BOBS (rather, BOBS for Dogs), and images of dogs for theme.
22 | P a g e
23 | P a g e
Step 2: Understand the ASIN: The product title and the image already suggest that the ASIN product type is shoes, brand is
Sketchers, and theme is images of dogs. To product line is unclear from the image, but going to the product detail page and
examining the third image from the top left (see picture below) indicates that the product line is indeed BOBS.
Step 3: Identify the product attribute matches and mismatches: All attributes identified in the query match those in the ASIN.
Step 4: Assign the label based on the identified defects: The label is Exact match.
Working with Categories

Product type mentioned
Category: APS Category: Specified (other than APS)
24 | P a g e
When the search query clearly mentions the product type which When the search query mentions the product type and a
clarifies the customer’s intent and the availability of the product specific category is also selected. In this case the query and
on Amazon. The category selected is APS, then we only have to category together can give you a better understanding of
mark the searched product as Exact. the specific need of the customer. Then only the intended
For example: if the search query is basketball (APS) , now we result should be marked as exact.
know Amazon sells basketball in this case we will mark only the For example: if the search query is basketball (Electronics) ,
basketball should be marked as exact. now due to the specific category, one can find phone
covers, pendrives or airpods cover in shape of basketball
(changing the intend and focusing on the specific product),
thus such products shall be marked as exact and not the
basketball.
Product type not mentioned

Category: APS Category: Specified (Other than APS)
In many cases the search query doesn’t have any product type When the search query doesn’t have any product type
mentioned. And the category selected here is APS. We can mentioned and consider it is a theme, the search alias can
consider our search query as a theme. In such cases we can give us a fair idea to understand the intent of the query. In
provide all the possible products related to the theme as SPT. such cases we can consider the selected category as a
For example: the search query is Marvel (APS) , now we know product type.
Amazon sells a lot of products related to marvel , in this case we For example: If the search query is Marvel (alias : toys) in
will mark all the products as Exact such as t-shirts , books , this case we will consider the category as a product type
DVD’s, figurines, etc. and we will mark only marvel Lego toys/figurines etc. as
Exact. Other products related to marvel like movies, books,
posters, mugs will be Complement.
Exception: When the product type is clearly understandable, but the category selected doesn't specify a different intend of the
customer. In such cases, we will not take the category into consideration while taking an enforcement action. For example: If the
search query is ‘Jacket’ and the search alias is ‘Shoes & bags’, in this case we will ignore the category and consider only Jackets as
the Exact product.
25 | P a g e
Category specific examples

In this section we have bifurcated the search queries on a category basis, to understand how different queries pertaining to certain
categories need to be judged.
**All the Queries are in APS store unless otherwise mentioned. **
1. Electronics
Headphones: Earbuds/Headphones/Earphones all are Exact except in JP where they are substitute
The extended definition of substitute for ASINs which meet the purpose/utility of the intended products apply to Electronics as
well.
Sub category Query Exact Substitute Complementary Irrelevant
Samsung note Samsung note 10+ note 10+ cover + Samsung note 10
Cell phones iPhone XR
10+ phone protective glass lite Cover
Computers &
Motherboard Motherboard - Processor PC screen
peripherals
laptop cover 13-15 laptop charger
Asus Zen book Asus Zen book 14 HP Elite book
inches for Lenovo
Gaming HyperX gaming headphone charging
Headphones Apple ear pods+ case keyboard
headphones headphones support
TV support 55 TV support Samsung console table with
TV screws TV
inches 45-75 inches universal TV stand
Amazon
Echo dot Echo dot Echo dot + smart bulb Echo dot cover JBL speaker
devices
All Alexa compatible
devices
Non Alexa/Non
Alexa
Smart devices
like Alexa compatible
Plugs/bulbs
Canon EOS Canon EOS 2000D + SDXC memory card canon printing
Camera Nikon cool pix
2000D lens + tripod 16 GB cartridge
men's
screen protector for
Watches intelligent Fitbit versa 3 (unisex) men's analog watch Boys’ Fitbit
Fitbit versa 3
watch
Electronics Gadget All devices/machines
2. Media and Games

Media primarily includes films and music. Under films, we have formats like DVD, Blu-Ray and Video cassette. Under songs we have
CD, Vinyl, cassette, and streaming. Genres in movies are the primary judgeability criterion. If genres don't match, then movies are
irrelevant.
Games here mainly refer to the video games including all PS models, Nintendo, all X boxes and Windows games. The category also
includes game accessories like controllers, consoles, game codes etc. Model compatibility is paramount in the videogames.
For Books, that doesn’t match the query, we have to mark as per the Genres.
Substitutability in Media → When the query is a topic or media title such as music, book, or movie, then other media that are in
the same general category, the judgment is Substitute. If a media product can be considered to be in multiple categories, then
either could be Substitute. Also, if any of the same contributors to one media product are also in another media product, then they
are Substitute.
Infinity war (Action) Infinity war i)Spider-man Trailer of the movie Titanic (genre
Movies different, romance)
26 | P a g e
ii)Infinity war
themed products
iii) Music score and
Songs
Complements in Media → Media products that are about other media products are considered to be secondary material and are
judged Complement so long as they match the category included in the query. This includes reviews, analyses, study guides,
anthologies, or commentary, where the book/video is meant as a companion to the central product they are about.
COD PS4 game
and all COD Games CD stand
COD ps4 PS4 GTA game COD XBOX ONE
themed product Guidebook for COD PS4
for PS4
ps5
Charging station Cables for charging Charging station for PS5
charging
for PS 5 console station for console Controllers
Video Games station
Video Video game CD, Joystick, Controller
game DVD, console Gamepad
Cases/Accessories for
Only XBOX
Cases/Accessories/Game Xbox Controllers
Xbox Consoles are the PS4 Console
s for Xbox Console (Accessories to
main PT
Accessories)
Harry
Movies/Prime Harry Potter Fantastic Beasts Harry Potter books or Titanic (Genre Different)
Potter
video Movie movie Merchandise
Movie
Endgame prime
video/Endgame
Endgame Marvel kindle
DVD format Endgame Blu-ray DVDs, CDs, Blu-ray holder
Blu-ray version/vice-versa
SACDs and DVD
audio discs
One Direction
One Linkin Park One Direction themed
music and brand Spiderman movie
Direction music, products
products
Purpose CD
Justin Bieber - Fine Line Vinyl Vinyl cleaning kit

Purpose Vinyl
Justin
Bieber Vinyl Player
Vinyl
Music/Vinyl/Audio
Walls CD Reputation CD Walls Vinyl
Walls CD
CD Player
27 | P a g e
Guide to use snapchat, Temple Run app,

Snapchat Snapchat Instagram
Gift card to buy Snapchat General Gift cards
Apps/Software
Windows Windows 10 Windows server Laptop with inbuilt
Windows 10 guide
10 Home 2019 window 10
3. Fashion
Sub
Query Exact Substitute Complementary Irrelevant Exceptions
category
If the query
Pink Cushion was “cushion
covers, size Blue Cushion covers, Cushion cover cover 30 x 40”
50*50 size 50*50 Any decorative to of different then even if we
Pink Cushion size 30*50
decorate the cushion get cushion
cover 50 x
covers/straps/zips for cover of 40 x
50
cushion covers 30 size, it will
be marked
Exact.
Superman t shirt
Softlines Batman pants/
innerwear
Plain inner
Batman t shirt
wears
Batman T-
shirt Plain t shirt
Plain pant will be

considered
complement.
Polyester T-shirt.
Cotton T- T-shirt made of T-shirt made of other
Cotton shorts/pants Woolen pants
shirt >=50% cotton material or with less
than 50% cotton
28 | P a g e
Cotton fabric
(pure or >=50%) Cotton T-shirt
Cotton <50% or
Material Cotton
Leather, Polyester,
(fabric)
Silk (Fabric)
In DEFRITES_JP
Tunic/ blouse/ Pants/ jeans Sweatshirt/Pull and IN
Tank top over marketplace T-
Any Shirts
shirt will be
Shirt substitute
Any Inner
wears
Apparels
Tunic
Pants
Top Jacket
(anything
from neck to
waistline) Coat
A one-piece
garment from Jumpsuit or Saree, Salwar
top till ankles. 2 –piece dress
Top till knees
Buttons/Accessories
Clothing Dress
for a dress
29 | P a g e
Ready made
Lehanga.
Unstitched materials Dress

for lehenga/ salwar
suit.
Salwar suit/ Ready to wear Semi-stitched
Lehenga Salwar suits. Lehenga/ salwar suit.
choli
Any pant/set of
clothes, which
Pajama Top
can be worn
while sleeping Any normal
Pajama/PJs/ or lounging. pant/shirts not
Pyjama used as sleep
wear.
2-Piece Bikini One-piece Bikini

Bikini Cover ups
Bikini
Any Necklace Necklace set with Pendant Anything other

bracelet and ring than necklace
Necklace like only
rings/bracelets
/anklet
Jewelry Different gemstone

Ruby Stone
Ruby Any jewelry made out

of ruby stone
30 | P a g e
Watch strap, screen

Any Wrist protector and watch
watch/ Smart tool kit
watch Watch with bracelet
Wrist Watch
Wearabl Any sneakers,

es sandals, pumps,
sliders, training
shoes Shoelace, socks and
insoles
Only in IN
marketplace
Shoes sandals/sliders
will be
irrelevant.
4. Luggage and bags

Luggage: By definition, luggage refers to empty bags or suitcases, used for carrying a person’s clothes and things on a journey.
Anything which is used to carry personal stuff for the purpose of travelling can be considered as a luggage.
Bags: a strong container made from cloth, plastic, leather, etc., usually with one or two handles, used to carry things in when
travelling, shopping, etc.
Subcateg
Query Exact Substitute Complementary Irrelevant Exceptions
ory
Rolling Suitcases,
Backpacks, Duffel Bags,
Softside Luggage,
Hardside Luggage, Two-
Wheeled Luggage,
Luggage Luggage Spinner Suitcases
31 | P a g e
Skybags Suitcase cover,

Suitcase Skybags suitcase Aristocrat suitcase Backpack
Suitcase wheels, handle
Handbag, Backpacks,
tote, slings, Lunch bag,
carry bags, Gift bags,
etc. Bags for
Bags Bags A bag used for any carrying pets
purpose.
A bundled
Bag Strap, Chain,
item of 3
bag Charms
bags can be
Handbag Caprese Caprese Tote bag/sling Lavie Tote bag/sling Suitcase/
given as
s Handbag bag/purse/wallet bag/purse/wallet backpack
exact, as all
are
handbags
Gucci Suitcase/Bac
Purses MK Purse MK Purse Purse straps
Purse/Handbags kpack
Tommy Hilfiger
Allen Solly Zips/chains, Wallet Suitcase/Bac
Wallets Allen Solly Wallet/clutch wallet/clutch/handb
Wallet charms kpack
ags
5. DIY & Tools

Subcategory Query Exact Substitute Complementary Irrelevant
If alias is DIY,
chair/bedsheet
having patchwork is
Irrelevant. However,
Patchwork Patches and scale for if the category is
patchwork Beads or other Needle/ thread "aps" the query can
decorating for sewing the be considered as
Art/Craft/DIY
(If alias aps, products for patchwork on "style" and mark
chair/bedsheet having textile the textile chair with
patchwork) patchwork as
"Exact"
Cloud decoration 1. Star shaped

Store: DIY DIY decoration one accessory/ Non-deco products,
DIY, Tools Cloud deco DIY kit
part of cloud utility objects like
and Garden
2. cloud deco deco DIY kit chair, table etc.
32 | P a g e
Nail & hammer

Nail paint or
(bundle)
Nail (Tools) Bundle of different artificial nail (as
Store: DIY sizes of nails alias is DIY/
industrial)
Solar Light DIY 1.Normal light

DIY kit (not
solar) Battery or any
solar light DIY kit
2. Solar light set other single part
Store: DIY
(FINISHED for solar light
PRODUCT)
1.tools/machines for
working with wood
Industrial/
accessory of the
Research and Wooden product
wood working 2. book on the theme. wood-working
development like chair, table etc.
tool
products
6. Beauty

Hair care & Shampoo Shampoo Shampoo + Conditioner Conditioner
products
Wigs Hair wigs Hair Extensions (both are Clips/caps to attach

Body care & used to change person’s a wig
products hair style – same purpose).
Perfume Perfume Deodorant
Moisturizer Moisturizers body cream Skin tightening

creams
Face care & Foundation Foundation in BB cream/ CC cream/ DD Brushes/pad to apply Lip gloss
products liquid form and cream foundation to face;
Powder form directly related
products like primers
and concealers/
setting powder
33 | P a g e
Eye liner Pencil/liquid/gel kajal Eyeliner brush contact lenses/ lip

liner color
Lipstick liquid lip color lip gloss/ lip balm lip primer eyeliner
face mask Sheet mask mask Scrub/ toner/ mascara
sheet moisturizer/cleanser
Blush Blush cheek tint highlighter/ blush eye shadow

brush
7. Medicines
For any formula foods we need to ensure that the age of the baby/kids matches.
Dietary Supplement
Fat Burner for weight loss Weight gain
Book on Weight Loss
powder or just
supplements
Supplements Weight Loss
Cold and runny Nose

Suspension.
Here, it also can be
used for fever as per
the product Fertility Support
Fever Reducer Pill description. Thermometer
Fever
Drug store
8. Grocery
Sub
category
34 | P a g e
Tomato puree, ketchup, all veggies Vegetable wash

spray Tomato shaped
Tomato earrings
Food Tomato
Tropicana orange Juice Juicer
Beverage
Orange juice
s
Apple juice
Whisky - The Orgain organic

Bourbon Whiskey
Rum Islay Edition nutritional shake
(documentary)
Alcohol Whiskey
Book on
Potato (raw), Frozen consumption of
or potatoes Potato wine
dried potato
Amazon
Potato
Fresh
Potato chips
Wheat flour 5Kg X 3 Wheat flour 2.5Kg X 2 Wheat bran

Wheat drinks
Amazon Wheat flour

Pantry 5Kg
35 | P a g e
Almond flavour
chocolate/
Almonds Other dry fruits/almond baking essesnce
[salted/unsalted] flour/almond butter/
almond milk/almond oil
Nuts Almonds
Ketchup/mustard Fritters with

Mayo mayo filling
Condime
Mayo
nts
Peanut Butter Chocolate spread/Almond Ketchup/Mustar

butter/Nutella spread d sauce
Peanut
Butter
Cake Pancake cake mix

Chips
Gourmet Cake
Honey/Dryfruits
Any Spices
Oats
Oats milk/ oats biscuit/oats cake

oats (primary ingredient is oats)/any
breakfast cereal
36 | P a g e
Chips of another brand salsa dip/any Biscuits

Pringles other dip
Pringles
Anything vegan
Anything non
Vegan
vegan
Non
vegetarian
Comment: Meat Vegetarian food/vegan food /plant
Non based meat
vegetarians
may
consume
vegetarian/v
egan food,
not vice
versa
Lactation
Other milk [buffalo/cow’s] or milk
products like
alternatives;
pumps,
https://www.amazon.com/dp/B00 Pregnancy
Beverage Mother’s lactation
Breast milk LBGOW2Y detection kit/
s milk improvement
any item with the exact title match Nursing clothes.
products or
https://www.amazon.com/dp/B00
book/videos on
09F3POY
lactation
37 | P a g e
9. Fresh
Sub Query Exact Substitute Complementary Irrelevant
category
Fruits Apple Apple apple juice/apple jam/ Fruit basket; apple

dried fruit/Apple puree/ apple peelers chocolate/apple
mixed fruit juice yogurt/
[Note - Flavors
will be irrelevant
but when added
as an ingredient
then substitute]
Lemon Lemon any fruit/lemon lemon mint/pie

pickle/lime squeezer/lemon
[Note - Flavors zest
will be irrelevant
but when added
as an ingredient
then substitute]
Coconut Fresh & dry coconut Coconut milk/ coconut Coconut ice
oil (for consumption)/ cream
grated coconut flakes
[Note - Flavors
will be irrelevant
but when added
as an ingredient
then substitute]
38 | P a g e
ginger ginger garlic/ginger tea with ginger

paste/ginger powder and lemon
[Note - Flavors
will be irrelevant
but when added
as an ingredient
then substitute]
Fish Raw fish Cooked fish/Frozen Fish shaped

Meat/Processed snack/chocolate
10. Pets
Domesticated Animals - Do not treat them as theme, however Wild Animals can be considered as theme
Sub
Query Store Exact Substitute Complementary Irrelevant
Category
39 | P a g e
Dog food, Dog

accessory, Dog toy, Product with their
Stuff toy, Figurines, Image (Print),
miniature themed, Products
DOG
of CAT.
pets/aps
(domestic e.g.
cat/ dog)
Pet
Product with their

Image (Print),
themed
Wild animals pets/aps
11. Toys
Sub
Category
Board games Ludo/Catan Card games like Uno Dices, pawns, chips Soft toys/figurines
Any other character
costume from Harry
Harry potter
Harry Potter Potter like Dumbledore
costume Harry Potter wand Batman mask
costume or Hagrid/ Fantastic
Toys beasts costume or
other theme costume.
Game for 0-4 years or 6
Games for 5 year Games for 0-5 year
years (we allow -1 and Guide to the game Game for adults
old old
+1 age only)
Accessory for plush Figurine, Plush toy
Plush toys Any Plush toys
toys for pets, Adult toy
12. Home
Sub Category Query Exact Substitute Complementary Irrelevant
Furniture Bed Single bed/ Double bed Sofa cum bed Mattress/Bed Sheet Chair
Wall Art Wall paper Wall paper Poster Glue Stick Photo frame
Bathroom Bath Towel Towel Hand Towel Hook/Rack Bath Mat
Storage and Organization Wardrobe Almirah Clothes Stand Wardrobe light Bed
Plant/ Real Plant/ Flower Artificial Plant/ Planter
Garden Planter/ Fertilizer
Flower (including seeds) Flower stand
40 | P a g e
13. Kitchen
Sub Category Query Exact Substitute Complementary Irrelevant
Dinning Plate Plate Thali/Bowl Spoon Cooker
Cookware Induction Induction Gas Stove Induction cleaner Cylinder
Appliance Chimney Chimney Exhaust Fan Chimney pipe Celling Fan
Microwave Microwave OTG (Oven Toaster Grill ) Microwave plate Sandwich Toaster
14. Automobiles
Sub Category Query Store Exact Substitute Complementary Irrelevant
Plastic model,
Car (same model) miniature, figure (of
Car Accessory (of the same the same model)
(Automobile) model)
Vehicle aps
(name of the
car model)
Automobiles
Tire/Headlights/ for same model

aps Bulbs
Battery (compatible)
Car
Plastic model, miniature,
(Automobile) toys
figure
Vehicle
15. Shoes
Sub
Category Query Store Exact Substitute Complementary Irrelevant Exceptions
Clogs aps Clogs for any Bathroom Sole for sandals Sport shoes for
gender Slippers any gender
Sports aps any laces for sports high heels/formal
shoes for sports/running shoes shoes for
women shoes women/sports
shoes for kids or
men
Shoes Tommy shoes shoes , wallet, shoes, wallet t shirt from any The shoes
Hilfiger and shoes laces, and other brand category/store
bags etc. from accessories consists of small
Tommy from other bags and wallets,
Hilfiger brand hence to be
marked exact in
this category.
16. Theme
Theme Query Exact Substitute Complementary Irrelevant
41 | P a g e
Cricket All cricket - Book on cricket Normal shoes

equipment/gears.
Note: unlike SQ - Football
which is a product type in
itself (the ball is called
football), cricket is treated
as a theme
SQ: Football
OP: Football shoes-
complement
Profession Anything for that - Toy sets for
profession children
like stethoscope for SQ associated with
Doctor that profession
or on that profession like Doctor set
for SQ-doctor
17. Adult
Sub Query Exact Substitute Complementary Irrelevant
category
Condoms condoms XL condom L Lubricant 1.condom S

Adult XL 2. Sex toy
sex toy sex toy for men Lubricant 1. sex toy for women
for men 2. condom
18. Books
Key Points:
• Price, adjectives, services – to be considered

• Overlapping ages to be considered Exact, adjacent ages to be considered Substitute
• Sub-genres of the books to be deep-dived via available decision-tree
• Merchandise to be rated as Substitute
Further elaboration:
Important points to consider in specificity to ‘books’ queries, divergent to the above/usual guidelines:
(1) Price/Free: any book (in any format) matching the price constraints chosen by the customer are to be taken into consideration
If customer is looking for “free books”, price must be 0 in the local currency
Any books which is non-free would thus be rated Substitute.
(2) Bestsellers/Validated Adjectives: query constraints must be respected (e.g. “romance bestseller” will need to satisfy both
bestseller and genre guidelines to be judged Exact).
While adjectives such as ‘cute’, ‘best book’ may be subjective and can continue to be disregarded at time of relevance
judgement, however ‘top seller’, ‘high rated’, ‘bestsellers’ and the like may be valuable for judgement.
42 | P a g e
>>The criteria for researching and dive deep here may include pointers and hints such as:
* any Book with the Amazon Charts OR Editor’s Pick OR Bestseller badges
* any Book with “New York Times bestseller” reported on the cover
* any book ranked within Books/Kindle Store Zeitgeist top 2,000
* any book with more than 1,000 reviews
>>Below books would be rated Substitutes:
* any book ranked between 2,000 and 10,000 in Books/Kindle Store
* any book having between 500 and 1000 reviews
(3) Service queries: while service or membership seeking queries such as “cancel kindle unlimited” continue to be disregarded,
however, keywords (and functional equivalents): “kindle unlimited”, “prime reading”, “audible plus”, “kids plus”, “comixology
unlimited” all contextually hint towards a customer searching for books and thus must be considered for judgement accordingly.
E.g. while the SQ: ‘kindle unlimited’ could entail all books to be marked as Exact, ‘audible plus’ hints at a particular format of book
and thus must be provided the relevant judgement, distinguishing format.
(4) Age range/reading level: a children book for an age range adjacent to the one the customer was looking for (e.g. a book suitable
for 3-4 yrs. old when customer is looking for “children books 5-6” would be rated Substitute.
Overlapping age range, e.g. books for 3-4-year-old and resulting product with 4-5-year-old book may be rated Exact.
Grade Age
Pre-kindergarten 3-5
Kindergarten 5-6
1st grade 6-7
2nd grade 7-8
3rd grade 8-9
4th grade 9 - 10
5th grade 10 - 11
6th grade 11 - 12
7th grade 12 - 13
8th grade 13 - 14
43 | P a g e
9th grade (High school) 14 - 15
10th grade (High 15 - 16

school)

school)

school)
(5) Distinguishing between genre and subject: Genre refers to categories of works that are characterized by similar plots, themes,
settings, situations, and characters. However, the Subject describes what a book is about, or its topic. Example of subjects are:
Architecture, Photography, Sports (in general or specific, e.g. football), etc.
*fiction and non-fiction books are to be rated Irrelevant
(6) Sub-genres: it is important to take note that while sub-genres such as Sci-Fi-thriller and a usual action-thriller may be rated
as Substitute products, however a significant movement from sub-genre, e.g. Sci-Fi-thriller and historical-thriller will be
rated Irrelevant.
(7) Media Vs non-media/merchandise Products: all the media products that exactly match the keywords and are either the main
work(s) in the original medium or other media derived directly from the original medium are Exacts. Conversely, non-media
products about the original work(s) are Substitutes. E.g. all Harry Potter movies and books are Exacts as they are either the primary
work (the books) or direct media derivations of the primary work (the film transpositions).
However, all non-media results, e.g. merchandising (https://www.amazon.com/s?k=harry+potter), are Substitutes.

Similarly for people names (historical figures, celebrities, authors) and fictional characters – any media product authored by or
about the person is an Exact. Whereas, merchandising and related products are Substitutes
a) SQ: Harry Potter
ASIN: Harry Potter coffee mug (merchandise)
Judgement: Substitute (all Harry Potter movies and books are Exacts as they are either the primary work (the books) or direct
media derivations of the primary work (the film transpositions).
All non-media results, e.g. merchandise is Substitute.)
b) SQ: Harry Potter book
ASIN: HP coffee mug
Judgement: Complement (This would be the same for SQ- "harry potter mug", the mug is exact, the book Complement)
(8) Broadly, books on the same topic are considered to be a Substitute. The reader (customer) might be interested in other
books on the same topic. Therefore, in books related queries any book related to the same topic irrespective of the target
audience would be marked as substitute. There can be new-released/ books released earlier (that the shopper might not have
seen), which might intrigue their desire to buy other books related to the same topic. This is different from the search relevance’s
book SOP which emphasizes importance to the target audience before marking it as substitute.
Examples:
44 | P a g e
Keywords: The Cancer Code

Recommended Title: Everyday Strength: A Cancer’s Patient’s Guide to Spiritual Survival
Explanation: The search query is a medical book to understand medical conditions during cancer, treatment etc. The search
result however is a guide to cancer patients on spiritual survival. Hence marked substitute.
Keywords: dieters gourmet cookbook
Recommended Title: The Korean Vegan Cookbook: Reflections and Recipes from Omma’s Kitchen
Explanation: Query is dieter’s gourmet cookbook and offer product is Korean Vegan cookbook. Irrespective of target audience, it
is still considered as cookbook and hence marking Substitute.
Keywords: cookbook
Recommended Title: Fatty Liver Diet Cookbook: A Liver Cleanse and Detox Guide
Explanation: Query is just cook book and the ASIN is cook book for fatty liver, it is considered as Exact. But if the query is “Cook
book for thyroid” and the result is “Cook book for fatty liver”, then it is Irrelevant.
For further detailed insight, please refer to the books SOP that is shared separately.
19. Brand affinity

Brand affinity is a personal or emotional connection forged between customer and a brand. From a customer’s perspective, they
might be interested in seeing similar category product offerings from same brand. Therefore, considering this emotional
connection between customer and brand and labelling as Complement to the customer query.
Two different lead products from the same brand can be marked as Complement, if they fall under same product category. Brand
affinity is to be considered in irrelevant cases, where query and shown product are not exact match or cannot substitute each
other or are not used together products. But we would still consider shown product as Complementary as both products are
from same brand and fall under same category. Here we consider query and shown product as complementary not because both
are used together products, but considering the possibility that customer who likes a product from a particular brand might have
a curiosity/interest to buy a different product of the same brand, which fall under the same product category. Thus, when it
comes to brand affinity, we are defying the purpose of products, but the product category will act as a proxy for purpose and will
be a guard rail to check the products are not too far off.
Sub
Category Query Store Exact Substitute Complement Irrelevant Exception
Maybelline APS Maybelline Mac cosmetics Maybelline Hair brush -
-
eye-liner eyeliner eye liner lipstick
swisse APS swisse Vitafusion - Swisse hair
multivitami multivitamin multivitamins nutrition for
n men men https://www.a men
https://www.a mazon.com.dp https://www.a
mazon.com/dp /B007L0DON mazon.sg/dp/B
/B07B8NF7C9 W 07BF67B9C
Dove APS Dove Shampoo Loreal Dove
-
Shampoo Shampoo Deodorant
Benadryl APS Benadryl fever Ibuprofen Benadryl Anti- Brand affinity
fever tablets fever tablets Itch Gel cannot be
-
tablets’ considered for
medicines
45 | P a g e
20. Subscription
Prime subscription related queries, annotator to do an Organic search and check for products/ videos under prime. Mark those as
Exact and any other product which is not purchased under prime to be Irrelevant.
Sub
Categor
y Query Store Exact Substitute Complement Irrelevant Exception
hbo max ALL All products or All products or - All non-prime -
subscription videos under videos under products
for prime prime & HBO MAX prime in
members in organic search organic search
-
sign-up
https://www.ama https://www.a
zon.com/dp/B0B6 mazon.com/dp
6PQZRB /B0916PSH18
Ebt-snap All All products that All products Explanation: EBT
are eligible for that do not SNAP – Electronic
either ‘SNAP’, say ‘SNAP’, Benefit transfer
‘EBT’ or both on ‘EBT’ or both that allows
the PDP on PDP Supplement
(mentioned in Nutrition
PDP) Assistance
https://www.ama Program.
zon.com/dp/B00
WI0T5AQ
More Labeling Nuances

Truncated queries Some of the most difficult queries are partial words or partial titles. If they are issued within an alias
such as books or instant-video, then it’s easier to judge based upon products that begin with that
string. Use Amazon or Google search to see what the auto-complete query options are, and consider
each of them as valid interpretations for the query. Each of those interpretations may be in different
categories, so ASINs can be judged Substitutes or Complements for any of the interpretations. Do not
consider truncations if a word is a product by itself. The aforementioned to be followed only when
the query is an incomplete product title.
Topic Queries for a product sometimes also return ASINs which are books or other media about that
product as a topic. These are secondary material to be judged as Complement that
includes documentaries, video reviews, how-to manuals, and educational materials.
Products which display an image of another product or miniatures are considered Irrelevant unless
the customer has selected an alias like toys or apparel, or added additional terms to the query
indicating they want a shirt, mug, sticker or other product displaying an image.
Quantity/ Quality/ • If any of these terms appear in a query, they are not used when determining ESCI relevance,
Price/Shipping/Condition and the query is judged as if those terms were removed. Subjective terms such as ‘cute’
‘cool’ ‘nice’, refurbished, best seller, best, renewed, cheap, free, sexy, awesome, new can be
ignored.
46 | P a g e
• Query is 5kg rice and Offered product is 2kg Rice, the judgement would be : Substitute as
the weightage or unit of the product is considered whereas 2 pack/3 pack rice/book/shirt
(any other product) should be marked as “Exact” as quantity should be ignored.
Name When a query consists of a person’s name, it is judged differently depending upon what relationship
that person has to the ASIN:
• if they are a primary contributor towards a book, video, piece of music or art, then that
ASIN is Exact, and any ASINs in the same category are Substitute, in case of author name
we check for genres.
• If the ASIN is about them or depicts them, then it is Complement
For people names any media product authored by or about the person is an
(historical figures, Exact
celebrities, authors) and
fictional characters: merchandising and related products are Substitutes
merchandise of another person is Irrelevant
Media format [DVD, Vinyl etc.] may match but if the genre
of the asked person is different then the OP is Irrelevant
Name of a music band merchandise of that band- substitute, as the OP is a related

product and can be expected to be bought by fans of the
music bands
music of band of different genre is Irrelevant. Merchandise
of a different band is also Irrelevant.
Name of a singer ( For ex Music album/songs of the singer- Exact

Beyonce)
Multiple artists in the same song- (Beyonce + Shakira
"Beautiful liar)- Exact, in case of multiple contributors, we
consider all as primary contributors
Megan Thee Stallion - Savage Remix (feat. Beyoncé)- Exact

Merchandise of the singer- Substitute
Merchandise of Miley Cyrus [another singer] is Irrelevant

Music of a different genre is Irrelevant
William Morris (Name of William Morris was a designer, who designed many types of
a designer) products- furniture, fabrics, stained glass, wallpaper etc., so
any product bearing his designs can be exact. Similarly, if SQ
is an apparel designer and OP is an apparel from that
designer (Exact), but if OP is an apparel from a different
designer or an apparel without designer name then
(substitute). Product type designed by the asked designer
will be narrowed down, for ex jewelry designer, interior
designer etc.
If the asked product type is not designed by the designer
then Irrelevant. Example: SQ - William Morris Furniture and
OP is Apparel from a different designer.
47 | P a g e
For a query for a fictional character-
• the primary book/movie/game that character came from is to be judged Exact

• costume & merchandise- such as posters, toys, clothing- Substitute.
• Other characters from the same book/movie would be Substitute,
• and books of analysis/commentary on the subject of the fictional character would
be Complement.
John Snow OP- GOT season 1 Exact- If the query is

OP- character’s character’s name, the
figurines or primary
merchandise book/movie/game that
character came from is to
be judged Exact, unless are
also represented in many
types of products, such
as posters, toys, clothing,
any of which can be
Substitute. Other
characters from the same
book/movie would be
Substitute, and books of
analysis/commentary on
the subject of the fictional
character would be
Complement.
Characters from a different
book/movie where the
asked character Is absent is
Irrelevant. Other media
items of the actor who
played the asked character
is also Irrelevant
Kit Harington Game of Game of throne series Exact

Throne
Search Category Offered Judgement Rationale

Query product
Deals related queries Deals/Offers aps T-shirt Unjudgable This is an ambiguous query as
of the day the product type is not
mentioned and the customer is
looking for offers/deals. So, we
would mark this as Unjudgable.
Deals/Offers apparel Ladies skirt Exact Due to the selected category, we
of the day would not mark this search
query as unjudgable. Instead, we
would mark all the products
belonging to the selected
48 | P a g e
category (apparel in this case) as

Exact.
Iphone aps Iphone 13 Exact As the product type is clearly
deals of the mentioned in the search query,
day we would not mark it as
unjudgable. Judgment would be
given according to the product
type mentioned in the query.
49 | P a g e

WPX Search Relevance

Uploaded by

Copyright:

Available Formats

You might also like

WPX Search Relevance

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

WPX Search Relevance

Uploaded by

Copyright:

Available Formats

Search Relevance: Standard Operating Procedures

WPX Search Relevance SOP (Version 3.1)

Size: Rule updated 10

Genre: Updated example 10

Other Attributes: Added example 11

Ambiguous queries: Explanation updated 13

In category Search: Example updated 14

Unjudgeable: Explanation added for foreign language 18

Category specific examples-Media & Games-Video games: Added example 27

Category specific examples-Media & Games- Movies/Prime video: Complementary section 27

Category specific examples-Fashion- Softlines: Complementary section updated 28

Category specific examples-Beauty- Body care & products: Added example 33

Category specific examples-Pets: Updated example 39

Category specific examples-Toys: Updated example 40

New category addition of Brand affinity 45

New category addition of Subscription with revised examples and decisions 46

New example under grocery 34

1. Search query/keywords, e.g., red nike shoes

The information about the offered product (OP) consists of

Match Type Description

Query and Offered Product Product Image Judgement

Query: red hanes cotton tshirt Exact

Query: panasonic waterproof Substitute

Query: samsung galaxy z fold 4 Complement

Category: aps (All Products) The offered product is designed to be

Query: nike sneakers for men Irrelevant

Query: my past purchases Unjudgeable

Product Title: JUST MY SIZE

Assessing the Relevance Match Labels

Product Attributes and Impact of Mismatches

symptoms of acne. There are

case the items are complementary,

Note: sometimes the

unisex (suitable for either female or expressed gender in

Note: intended gender is limited to

OP is a media OP: Jazz Dance Level 1

SQ: Kettle for 110v

Additional Important Considerations

Product Variations from Product Detail Page

Very Vague Queries

Products Not Sold on Amazon

Query Exact Substitute Complementary Irrelevant

Brand affinity, related

Same purpose/function, Not compatible with

Compatible with DP, Lead product when

OP is a bundle with DP as Product does not

Bundle with the console

S. No. Search Query ASIN/ISBN Category Judgement Reason

2. Flipkart.com Pen (B082PN4X5J) aps Unjudgable Navigational Query

3. i) 博朗 Chocolate aps Unjudgable Foreign Language

6. Gift for mother Notebook aps Unjudgable Product detail page

Query: pickup tents for camping short bed

Category: aps (All Products)

Product Title: Napier Backroadz Truck Tent, Grey/Green, Full

Example: bobs shoes with dogs on them

Query: bobs shoes with dogs on them

Category: aps (All Products)

Product Title: Skechers Women’s Slipper

Working with Categories