Image Relevance Quick Guide

Page 1: General Information

In this HitApp we are determining the relevance of image results for search queries
issued on the Images tab of a search engine (such as Bing and Google). Your goal is to
determine if the image makes a good result for the query.

Search query: text used for search on web, for example {cute black kitten}
Search Image result(s): any image(s) returned by the search
Search relevance: how well does the image in the result satisfy the user query

Query: {cute black kitten}

Start
Let’s help the internet searchers find the best images on web!
Image is clean, free from
Looking at the Image and the
pornography, offensive text
Image loaded correctly and Page, I can determine if the
or other unacceptable Yes
can be judged Image is relevant to the
Yes Yes content.
query
No (Image is broken) No (it contains porn, or
No (I don’t understand this any of the cases covered
Image or content on Page) in Detrimental section)

Did Not Load Cannot

Judge Detrimental

An image where a
Image prominently shows Yes cute black kitten is
exactly what the user was Relevant the focal point is
If the
looking for exactly what the user
image
asked for, Relevant
returned
No contains
what the
There is a cute black user asked
Image shows exactly what kitten in this image for, one of
the user wants BUT which would suggest these will
Yes Relevant but
the queried topic is not the Relevant BUT the dog be the
only/main one, or there is a Imperfect in the image takes away final rating
the focus from kitten,
distraction Relevant-But-Imperfect
No (Image does not contain
exactly what the user asked for)
An image of a cute
Image has some topical If the
kitten that is not black,
connection to the query, but Yes it is not exactly what
image
there is a mismatch. Related user asked for but
returned
(e.g. Part of the query is not does not
could still provide value
contain
satisfied) to user, it is Related.
what the
user asked
No (Image is completely off topic) An image of a sock for, one of
does not satisfy these will
The image is useless to Yes the query, it is be the
Not Relevant useless to user, final rating
the user query
Not Relevant.
 Image Relevance Quick Guide
Page 2: Navigating UHRS, Helpful Features, and Using Source Pages

You can see the user’s search query You can research or translate the query in
and the user’s location, and the Bing or Google search using the links on
Query Interpretation top or CTRL + [hotkey letter]

If images or pages do not load in

HitApp, click the URLs provided
to load them onto another page

Use this button if image is detrimental

https://i.pinimg.com/736x/cc/1a/c3/cc1ac3c8a710cc261bc66d7ae64de523--cute-kittens-cat-lovers.jpg

Choose a rating by clicking on the

rating button.

Quality of the image is

Click “Submit” evaluated in another HitApp,
Expand and to submit your You can work faster on this HitApp please, do not downgrade
Collapse final rating. using the keyboard shortcuts provided based on image being
GUIDELINES at the bottom. Use the spacebar to watermarked, ugly, dark, or
using arrow expand the Image or Page. Use the tab blurry!
key to move from Image to Page.

Purpose of the Source Page

The Source Page is the page where the image was found. It may be useful to better understand what the image
contains. While information on the page may be useful in understanding the image, remember, you are judging
relevance of the Image and NOT the page. If page does not load (even after checking link provided) try to judge
the Image Result without the page. If you cannot make a judgement without the page, use “Cannot Judge”.

Unless you are familiar with the

architecture in Albany, you
cannot know if this Image Result comes
from Albany.

The Source Page

can explain what
exactly is in the
image. The information on the Source
Page confirms that the Image
Result is of the city of Albany.
The correct Rating is Relevant.
 Image Relevance Quick Guide
Page 3: Image Relevance Ratings Cheat Sheet

DID NOT LOAD

The Image does not load (even after using direct link provided) and prevents working the Hit

CANNOT JUDGE
There is some issue preventing you from Judging the Hit
• The Query is in a language you don’t understand
• The Query Intent is not clear after reading the Interpretation and viewing the source page
• The Query is about a technical/extremely specific intent that you do not have knowledge of
(medical, scientific, etc.)
• The Source Page does not load, and you need the Source Page in order to judge the Image Result.

DETRIMENTAL
The Image shows pornography, visible nudity (breasts with visible nipples, sexual organs), or gore
(dead/mutilated bodies of people) and/or contains offensive text (hate speech, swear words, etc.)
Image would not be shown on Western news television broadcasting. Reminder: If Page is detrimental but
Image Result is not, we do not use “detrimental” label, we only judge the Image.

RELEVANT
The Image Result is exactly what the user asked for, this is a great result
• The Image Result completely satisfies the user’s intent – the correct object, entity, or idea is prominently
displayed in the image (Examples)
• The Image Result would make a great result for the User’s search query

RELEVANT-BUT-IMPERFECT
The Image Result contains the right object, entity, or person BUT there are some flaws
The result is Relevant, BUT also has one or more of these problems:
• There is distracting text or other elements in the image; the queried object is not the focus of the image, or it
is the secondary topic (Example)
• The image is showing the wrong side, only a small part of the object asked for, partially visible (Example)
• The Image shows an unexpected version of the query intent or User asks for a famous/known individual, but
gets another person with the same name
• The Image resolution is larger than the specific size the user asked for (Example)
• The user asks for content from a specific website but gets an image from a different website.
• There is foreign or unrelated text in the image, but the image can still be judged

RELATED
The Image Result is not what the user asked for, but is related or could still provide value
• The Image does NOT contain the object/entity/idea the user asked for, but is either not too far off or would
potentially provide some value to users (Example)
• Some parts of the query are not satisfied (Example)
• User asks for a specific number of entities/objects, but the image provides fewer than requested
• The Image is smaller than the specific size the user asked for (Example)

NOT RELEVANT
The Image Result is not useful to the user issuing the query
• The Image does not satisfy the intent of the query at all and would be useless to the user (Example)
• The Image is unreadable or so hard to read that it would be useless to the user regardless of query intent
(Example)
 Image Relevance Quick Guide
Page 4: Ratings Breakdown with Examples

Query Result Comment

Examples and exceptions for RELEVANT

{heart- Images of heart-shaped clouds are RELEVANT;
shaped the images contain exactly what the user is
cloud} asking for (even if Image is photoshopped).
ALL images provide exactly what the user asks
for in their query, they are RELEVANT.
Each of these images prominently display the
{samurai}
correct subject of the query: “samurai”.
NOTE: Some images may have quality issues BUT
that does not affect relevance.

Both images are considered RELEVANT. (Left)

Leonardo Da Vinci, the painter; (right) painting by
{Leonardo Leonardo Da Vinci, “Mona Lisa”. The focal points of
davinci} both these image results would satisfy the user
query. Both pictures of artist and his works are
expected results for the query “Leonardo Da Vinci”.

Query Result Comment

Examples and exceptions for RELEVANT but IMPERFECT

A relevant result would ideally be focused on

the query, the Eiffel Tower. BUT since the
{Eiffel
focus is on tourist and not the Eiffel Tower,
tower}
this Image Result is RELEVANT BUT
IMPERFECT.

This shows the logo of the football club in

query, Liverpool, BUT because the opponent’s
{liverpool} team logo is also shown, this is a distraction
from the Liverpool logo, RELEVANT BUT
IMPERFECT.

The image is of a toy car version of a Toyota

Corolla, it is an unexpected representation
of the entity searched for, thus it is
{Toyota RELEVANT BUT IMPERFECT.
corolla} NOTE: An animated or cartoon version of
the car would also be considered Relevant
But Imperfect (UNLESS the user explicitly
asks for that in their Query).
 Image Relevance Quick Guide
Page 5: Ratings Breakdown with Examples Continued

Query Result Comment

Examples and exceptions for RELEVANT but IMPERFECT (continued)

The queried subject, Lebron James, is in the

image which makes it RELEVANT, BUT it is
{lebron
IMPERFECT because he is turned away and
james
we only see his back. The subject/entity of
picture} search queries should be the focal point of
the Image, displayed prominently in order to
be Relevant.

We always assume that a user is looking for a

photograph of the actual object or concept of
their query where possible. Artwork, clipart,
drawings, or other versions of the objects should
be seen as unexpected unless the user specifically
{banana leaf} asks for that content in their query: e.g., {tomato
clipart}; {Toyota Corolla drawing}
NOTE: Such cases should be considered
unusual/unexpected representation and
downgraded by one level: RELEVANT BUT
IMPERFECT.

Examples and exceptions for RELATED

{June 2021 The query was issued in the USA, image result
calendar with shows June 2021 calendar with German
holidays} holidays. Based on the user’s location, we
assume the user wanted U.S. holidays. A
USER calendar with German holidays is not what the
LOCATON: user asked for but similar. Could be of some
USA interest to user, RELATED.

The user query is very specific, they want Image

Results relevant to the US Navy’s “Blue Angels”
{Navy blue
first female pilot. This Image Result is of Blue
angels first
Angel Hornets jets flying, it is not of the female
female pilot}
pilot, or even of her flying, the user may be
interested, RELATED.

This image result is of Diamond Head in Hawaii,

{Hawaii but it is not a drawing as the user query
diamond head requests, it is a photograph. The query is not
drawings} completely satisfied, nonetheless this image
may provide value to the user, RELATED.
 Image Relevance Quick Guide
Page 6: Ratings Breakdown continued and Specific Image Size Queries

Query Result Comment

Examples and exceptions for NOT RELEVANT
The user most likely wants to see or
{save download a background for PowerPoint
environment Presentation about saving the environment.
background This image is not relevant to the user intent,
for ppt} it is a living room stock photo, NOT
RELEVANT.

We do not judge quality in this HitApp but

this image result is so dark, we cannot see
what's in it, we cannot judge it. An Image
Result like this would be NOT RELEVANT
because we can be certain that the user
would not want to see this image, we know
{car inside
this would not be a good result for the query.
seat rain pic}
NOTE: The “CANNOT JUDGE” rating would
not apply here because we know the user
would not be happy with this image. (Use
“Cannot Judge” when you do not understand
the Query or the Page provided and thus
don't know if result is good or bad.)

Ratings for Queries Where Users Asked for Specific Image Sizes
This table is an example of how to rate Images where the User specifically asks for an image size in the
query. This table assumes that the Image is relevant to the query. If an image is not “Relevant”, then
we rate the query as normal. The Image size is always displayed above the image in the Hit.

Query: {1080 x 1080 bear images}

Image Result Size Final Rating
Smaller Than 1080x1080
Related
Exactly 1080x1080
Relevant
Bigger Than 1080x1080
Relevant But Imperfect
RULE: If a user asks for “HD” or “high definition” in their query, the image should be at least
1920x1080 in size or larger (larger than 1920x1080 would be seen as “Relevant”.)

Thank you for helping make search results better!