Relevance Measurement Labeling Guidelines

Cold Start

80 l
Section Overview

63 tia
A. Escalation Process
B. Program Overview
C. Tooling
D. Contributor Steps
1. Account Setup and Screen Recording

45 n2. Switching Apps and Tracking Session Numbers

3. Watching Videos
e
4. ADAP Data Entry
E. Contributor Notes
14 id
1. Important Reminders
F. Appendix
1. How to access videos on Facebook, Instagram and TikTok
2. How to access the correct work link
nf

3. ADAP UI
4. What to do when you’ve found an error with your ADAP task
5. Identify sponsored content, live content and advertisements
6. How to enter # of video likes
Co

7. Genre Definitions
8. Metric Definitions
9. How to Screen Record
i. iPhone or iPad
ii. Android
A. Escalation Process

Report if you see any content relating to:

1. Terrorism (endorsement or active practice) or any dangerous organizations
2. Acts of mass death
3. Self-injury or suicidal content

80 l
4. Child exploitation and child nudity
5. Human trafficking

63 tia
The following are procedures for reporting said content:
a. TikTok:
1. Tap the arrow at the bottom-right corner of the video
2. Tap the Report icon
3. Select the category that it relates to

45 n
b. Instagram Reels:
e1. Tap the three-dot icon on the bottom-right (or top-right) corner of the video
2. Select “Report…”
3. Select the category that it relates to
14 id
c. Facebook Reels
1. Click on the 3 dots in the bottom-right corner of the video
2. Click “Find support or report reel”
3. Select the category that it relates to
nf

To immediately escalate content with credible threats of self-injury or suicidal content:

a. TikTok:
1. Tap the arrow at the bottom-right corner of the video
2. Tap the Report icon
3. Select "Suicide and self harm”
Co

b. Instagram Reels:
1. Tap the three-dot icon on the bottom-right (or top-right) corner of the video
2. Select "Report…"
3. Select “Suicide or self-injury"

c. Facebook Reels:
1. Click on the 3 dots in the bottom-right corner of the video
2. Click “Find support or report reel”
3. Select “Suicide or self-injury”

For escalating content that matches the above criteria on Facebook or Instagram, please
submit Meta’s Safety Escalation Form. For content depicting self-harm, self-injury, or suicide on
Facebook or Instagram, email outsourcing-suicidealertesc@fb.com. For escalating content
found on TikTok, please submit TikTok’s feedback form or email info@tiktok.com. CC your Team
Lead on any emails, who will also escalate to the Project Manager.
B. Program Overview
Each contributor should create new accounts on Instagram Reels, TikTok, and Facebook
Reels to evaluate how well the platforms recommend relevant content.

80 l
C. Tooling

63 tia
Mobile Devices – Contributors will conduct the exercise on their mobile devices while
screen-recording the session for subsequent ADAP data entry.

Apps – Contributors will create new accounts and conduct the exercise on three different
apps - Facebook (Facebook Reels), Instagram (Instagram Reels), and TikTok.

45 n
ADAP Tooling – Contributors will enter video data utilizing their screen recording for each
ADAP job.
e
D. Contributor Steps
Please thoroughly read through all of the steps in this section before creating your accounts.
14 id
1. Account Setup and Screen Recording
a. Information to create a throwaway email will be assigned to you for this project.
i. Ex: ColdTurkey3213
b. Please create a throwaway email on ProtonMail to use for the exercise.
i. Ex: ColdTurkey3213@protonmail.com
nf

ii. Reminders on email address creation:

1. ONLY use this email for the project, NOT for personal use for the
duration of the study.
2. Ensure that the email address ends with “@protonmail.com”.
c. Using that email, create an account on Instagram, TikTok, and Facebook. Please
Co

download each app from the app store on your phone; do not use the
browser to access these sites.
i. Reminders on username creation:
1. Create your username using a combination of a random first name and
last name and a number. Ex: @denisesmith99
2. DO NOT use your real name.
3. DO NOT include any PII (personally identifiable information, such as
your name, exact birthday, etc.).
4. DO NOT include the words Appen, TikTok, Instagram, Facebook,
Touchet, “throwaway”, “project”, or anything related to the project.
5. Good Examples:
@denisesmith999 (This is only an example; DO NOT use)
@robertjones123 (This is only an example; DO NOT use)
6. Bad Examples (DO NOT USE):
@appentiktokproject
 @janetreels714199
ii. Please DO NOT upload a profile picture.
iii. When prompted to provide demographic info on your test account, please
input your actual gender and birth year (you can provide any month and
day for the birthdate section on the app). Do not provide any other
demographic information (e.g., location), and DO NOT tell the app what your

80 l
interests are.
iv. Accept the terms of service for creating new accounts on each app.

63 tia
d. When signing up on all apps:
i. DO NOT enter your mobile phone number. DO sign up with the email
address you created for this project.
ii. DO NOT upload your contact info
e. When signing up on Instagram:
i. DO NOT log in using the throwaway Facebook account or click yes on

45 n
f.
e
g.
h.
getting Facebook Suggestions.
ii. DO NOT click on any accounts on the “Discover People” screen.
Before entering each app (once your account has been created), start screen
recording your session (please see How to Screen Record in the FAQ for
additional information).
Once the countdown starts, take note of your start time for your recording session.
Once your session ends and you stop recording, take note of the end time for
14 id
your recording session.
i. Reminders on account usage:
i. These accounts should be used only for the tasks stated in the contributor
exercise below for the full duration of the exercise. Do not interact with any
other parts of the app or user experience unless explicitly instructed to do
nf

so.

2. Switching Apps and Tracking Session Numbers

a. For Switching Apps - You will receive instructions and a prescribed sequence that
dictates the order in which the contributor should watch videos on the three apps.
Co

After each session, you’ll proceed to the next app in the sequence until you’ve
watched 250 videos per app.
i. Example App Sequence - FB Reels IG Reels TikTok
1. Start screen recording and navigate to Facebook Reels Watch
Facebook Reels Stop recording and enter data into ADAP for this
Facebook Reels session.
2. Take a break! This can be for however long you want, but be mindful of
how long it will take to complete the full exercise.
3. Once the break is over, start screen recording and navigate to Instagram
Reels Watch Instagram Reels Stop recording and enter data
into ADAP for this Instagram Reels session. Take a break!
4. Once the break is over, start screen recording and navigate to TikTok
Watch TikTok Videos Stop recording and enter data into ADAP
for this TikTok session. Take a break!
 b. For Tracking Sessions - Following the steps above, the contributor will have
completed one (1) session per app. Complete as many sessions as needed until
you’ve watched 250 videos per app.
i. Let’s see how the number of sessions works on our App Sequence example,
FB Reels IG Reels TikTok
1. Day 1: Facebook session 1 (Videos 1 to 47) Break Instagram

80 l
session 1 (Videos 1 to 51)

63 tia
2. Day 2: TikTok session 1 (Videos 1 to 62) Break Facebook
session 2 (Videos 48 to 112) Break Instagram session 2 (Videos
52 to 124
3. Day 3: TikTok session 2 (Videos 63 to 142) Facebook session 3
(Videos 113 to 178) Break Instagram session 3 (Videos 125 to
189) Break

45 ne ii.
4. Continue until you reach 250 videos on each app and ensure that you
record your data after each session.
Note: It is entirely up to the contributor to correctly track their session
numbers. For this reason, we highly encourage the contributor to use some
of the following tips and methods to help them on this vital task. Note these
are just suggestions, contributors can also be creative in finding solutions
that work for them to keep track of their session number. Some suggestions
14 id
on how to do so:
1. Use a notebook/paper and pen or pencil to manually write a record of
the session number.
2. Create a Word or Excel or similar sheet/document to keep track of the
nf

session numbers.​
3. Just keep in mind to not watch all the 250 videos in one sitting.
4. The best practice is to also include the date, start and end time of each
session in these notes to keep on hand while entering data on ADAP.
Co

3. Watching Videos
a. Contributors will watch 250 unique videos on each of the three apps for a total of
750 videos. For each app:
i. To access the videos:
1. Open the Instagram app → tap the Reels icon
2. Open the Facebook app → tap "Menu" → scroll down and tap the
"Reels" icon. Note: If you don’t see the Reels icon click “See More” to
expand the window.
3. Open the TikTok app → tap the "For You" tab
ii. Use the app as you normally would in real life - watching any videos that
seem interesting to you personally and skipping those that don’t for multiple
sessions.
1. You can click the “like” button if you feel inclined to do so.
 2. DO NOT comment on or share any videos, follow, or interact with any
other accounts on each of the apps. Do not click on the comment or
share buttons.
3. DO NOT access the messaging function on any platforms.
4. DO NOT use search for your friends / people you may know.
5. DO NOT follow any accounts.

80 l
6. DO NOT search for specific content creators.

63 tia
7. SKIP sponsored content (usually indicated as “Sponsored” in overlay
text on the video) and advertisements.
8. SKIP live videos.
a. Note: Sponsored content, advertisements and live videos do not
count as a video for data collection purposes.

45 n
9. DO NOT watch all 250 videos in one session. Please treat this as you

ewould in your day-to-day life. Watch a number of videos and take a

break (20 mins, 30 mins, an hour) then come back and watch some
additional videos. Repeat this until you reach 250 videos. Please
proceed to ADAP to record your entries after each session.
10. Each session should last between 20 min - 2 hours max.
14 id
a. Aim for whatever duration is good for you.
b. You don’t have to watch a video in its entirety (unless you
choose to do so) for it to count for data collection. If a video is
not interesting initially, feel free to move on to the next video
directly.
nf

11. DO NOT watch the same video more than 3 times.

a. Additional Info:
1. If you find a video interesting and want to keep watching,
up to 2 full re-watches of the same video are acceptable
and this still counts as 1 video for data collection
Co

purposes.
2. Note on counting loops (rewatches): If you watch a video
completely on the 1st run and then watch it for half the
duration on the 2nd run, then that counts as (1) loop. If you
watch the video in its entirety twice then this is considered
as (2) loops. It is only counted as a loop if the video is
watched in its entirety.
12. Once the session is over, stop recording, close the app, and move on
to step D4. ADAP Data Entry.
13. Repeat the instructions until you have scrolled through 250 total
videos on each app.
14. Once you have scrolled through 250 total videos on each app, stop
the exercise.
 4. ADAP Data Entry
a. Look at the “How to access the correct work link” FAQ entry.
b. Use your screen recordings to help enter the relevant information into ADAP for
each video watched during a session. For each video, record the following
information in the log:

80 l
i. Enter the session number (e.g., 1, 2, 3, etc.)
ii. Answer the “Have you entered information for this session before?” prompt.

63 tia
1. If this is the 1st video in your session, select ‘No’ and go through the
following steps.
a. Enter the throwaway email associated with the profile used for the
video scrolling
b. Select the date of your scrolling session
c. Enter the time you started your screen recording

45 ne d. Enter the time you ended your screen recording

2. Otherwise, select ‘Yes’ and continue on to the next step.
iii. Before entering data on a particular video, contributors should only enter
data for accepted videos. Data from content that is not accepted as a video
should not be entered into ADAP and the contributor should skip this
immediately when shown. Please take note of the following:
14 id
1. WHAT COUNTS AS 1 VIDEO FOR DATA ENTRY:
a. Watching a video briefly that was then skipped
b. Watching any portion of a video, even if you didn’t watch the full
length
c. Watching a video in its entirety
nf

d. Watching a video more than once

e. Watching a split screen video under the previously mentioned
conditions (content where more than one video is playing once)
Co

2. WHAT DOES NOT COUNT AS 1 VIDEO FOR DATA ENTRY: (see

appendix for how to identify these videos)
a. Sponsored content
b. Advertisements
c. Live Videos

iv. Comments
1. This comment box is optional, but please feel free to comment here if
you want to notify the management team of any blockers you
experienced while labeling this job.
a. See “What to do when you’ve found an error or have difficulties
with your ADAP task”
 v. How many likes does the video have?

80 l
63 tia
45 ne 1. IMPORTANT NOTES:
a. Follow the instructions in the “How many likes does this video
have?” and enter the appropriate response.
b. For Like counts greater than 1:
1. Enter exactly as displayed in the app (Ex: 4.7K)
14 id
2. Do not convert numbers. Input exactly as shown in the app.
(Ex: always insert 14.5K instead of 14500)
3. The letters in the number of likes for condensed numbers
should always be capitalized.
4. Keep in mind that Android shows a decimal point, while
nf

iPhone shows a comma between numbers. Please enter

exactly as shown on your screen.

vi. Did you press the ‘like’ button for the video?
1. Yes - if “Yes” proceed to question vii below.
2. No - if “No” proceed to question viii below.
Co

a. IMPORTANT NOTE:
1. Continue using the app naturally, but make a note of what
the first video you liked on each app was about.

vii. Is this the first time you liked a video on this app?
1. Yes - if “Yes” proceed to question ix below.
2. No - if “No” proceed to question viii below.

viii. Was this video similar to the one you first liked on this app?
1. Yes
2. No
3. N/A (I have not liked a video yet)

ix. What was this video about? Limit your answer to 1-3 words.
 x. Which of the following best describes this video? (Please see the section.
Genre definitions for additional information)
1. IMPORTANT NOTE: If a video fits within multiple genres, select the
genre that defines the main purpose/theme of the video. See genre
definitions for how to better interpret each genre and how to tackle split
screen videos.

80 l
xi. How many times did you watch the video to completion?

63 tia
1. REMINDER: If you watch a video completely on the 1st run and then
watch it for half the duration on the 2nd run, then that counts as (1)
loop. If you watch the video in its entirety twice then this is considered
as (2) loops. It is only counted as a rewatch if the video is watched in
its entirety.
a. 0 – Did not watch completely
b. 1 – Watched completely (the video restarted)

45 n
xii.
e c. 2 – Watched twice (the video restarted twice)
d. 3 – Watched three times (the video restarted three times)

How well does the video match your interests?

1. Extremely well
2. Very well
3. Moderately well
14 id
4. Slightly well
5. Not at all well

xiii. Was the video worth your time?

1. Completely
nf

2. Mostly
3. Somewhat
4. Barely
5. Not at all

xiv. Were you pleasantly surprised to see this video?

Co

1. Extremely
2. Very
3. Somewhat
4. A little
5. Not at all

xv. Do you want to see more videos like this?

1. Yes
2. No

xvi. Does this video fall into one of the following categories? Select all that apply.
Note: If you select one of the categories, then “This video does not fall into
any of the above categories” does not apply.
1. The video was offensive or inappropriate.
2. The video used misleading language or images to get people to watch
longer or click a link.
3. The video used images that were low resolution and/or audio that had
 poor sound quality.
4. The video showed content that I’ve previously seen multiple times.
5. The video showed content that was in a language that I don’t speak.
6. This video does not fall into any of the above categories.

xvii. Is this the last video for your current session?

1. Yes - if yes, answer the following questions in section c, looking at the

80 l
overall experience for your last session.

63 tia
2. No - if no, continue to the next video for data entry.

Question Reference

45 ne
14 id
nf
Co

c. Once you’ve recorded the information for each video in a session, record the
following information in ADAP.
i. How varied were the topics shown to you in this last session?
1. Extremely diverse
2. Very diverse
3. Moderately diverse
4. Slightly diverse
5. Not diverse

ii. Can you explain why? Limit your response to no more than 50 words.

Please see metric definitions for how to better interpret these questions.
E. CONTRIBUTOR NOTES

1. Important Reminders
a. After logging is finalized, delete the screen recording video
immediately.

80 l
b. Each account email should have AT MOST 750 video views associated with
it: 250 for each of the 3 apps.

63 tia
c. If invited for another round of testing, please DO NOT use the same
accounts or throwaway email.
d. DO NOT comment on or share any videos, follow, or interact with any other
accounts on each of the apps. Do not click on the comment or share buttons.
e. DO NOT access the messaging function on any platforms.
f. DO NOT use search for your friends / people you may know.

45 ne g. DO NOT follow any accounts.

h. DO NOT search for specific content creators.
i. DO NOT interact with any other parts of the app or user experience unless
explicitly instructed to do so.
14 id
nf
Co
F. Appendix / FAQ

1. How to access videos on Facebook, Instagram and TikTok

Facebook Reels Guide

a. Click on your profile picture to access the menu.

80 l
63 tia
b. Click on the Reels icon. If it is not showing on the Menu, tap “See More” at the
bottom.

45 ne
14 id
nf
Co
 Instagram Reels Guide
a. Click on the Reels icon at the bottom.

80 l
63 tia
45 ne
14 id
nf
Co
 b. To verify that you are really on Reels, you will see in the upper left a text that
shows “Reels”.

80 l
63 tia
45 ne
14 id
nf
Co
 TikTok Guide
a. Ensure that you’re accessing the “For You” page as shown below.

80 l
63 tia
45 ne
14 id
nf
Co
 2. How to access the correct work link
a. To avoid inserting data in the wrong link, before starting to enter the data,
contributors should double check the ADAP task link title to ensure the data will
be entered in the correct link of the prescribed app sequence. In this example we
will use the sequence of Facebok Reels Instagram Reels TikTok.

80 l
63 tia
45 ne
14 id
nf

b. After the contributor is finished submitting the data from all the videos in the
session for each ADAP work link, the contributor should click on “Give up” in
ADAP top menu, to properly end and save the work done on that link, as well as
Co

to avoid being redirected for the same work link when clicking on a link for a
different app.
 3. What to expect on the ADAP UI

80 l
63 tia
45 ne
14 id
nf
Co
 4. What to do when you’ve found an error or have difficulties with your ADAP task

a. Contributors should use the comment box when they detect that something is wrong
on their ADAP task. Contributors are highly encouraged to insert flags, comments, or
suggestions within the comment box on ADAP with the session and video number
where the error was detected. This helps the support team to resolve and revert any

80 l
errors when possible.
b. Contributors should also use the comment box to: flag any missed/wrong information

63 tia
from the previous sessions about session date, session start and end time, flag the
use of a new email due account being banned, etc.

45 ne
14 id
5. How to identify Sponsored content, Live content, and Advertisements?
Note: The data from the contents below should not be inserted on ADAP, and
these contents should be skipped immediately.

a. Sponsored content often has an overlay text that identifies the content in the
nf

bottom left of the image. The appearance and the text itself can vary according to
the app but should be skipped and does not count as a video. Examples on
each app shown below:
Co
 Facebook Instagram TikTok

80 l
63 tia
45 ne
14 id
nf

b. Advertisement and Ads content often have a translucent or colored rectangle

with a sales call to action at the bottom of the image, which indicates the video is
part of a paid ads campaign. The text in the call to action can vary, such as learn
Co

more, download now, subscribe, order now, etc. The appearance of the
advertisement button also can vary; however, the purpose of the content is often
promoting a brand, product, and/or service, so does not count as a video for the
scope of the project and should be skipped. Examples on each app shown
below:
 Facebook Instagram TikTok

80 l
63 tia
45 ne
14 id
c. Live content often has an indicator that the video shown is to call the viewer to a live
nf

session; thus, are not considered as a video to the scope of the project and
should be skipped. Live videos on Facebook and Instagram can be a little more
difficult to identify than on TikTok app, as on FB and IG the signal for live videos is just
a “Live” tag text near the account/creator profile picture. TikTok shows an easy to
identify call to action “Tap to watch live” on the screen. Examples on each app shown
below:
Co
 Facebook and Instagram TikTok

80 l
63 tia
45 ne
14 id
nf
Co

6. Genre Definitions

Note: For the videos with split screen content (2 panes of content in one video) where
there is a main subject video and a random video (often times ASMR or a video game),
the contributor should use the main subject video as the source of information for data
entry and avoid using the random content. Often, we have this type of content split with
podcasts in the TikTok app, in these cases, the podcast is the source of information to
determine the video genre. Examples:
a. Life Experience Genre with random ASMR
b. Comedy Genre with random game play
c. Information Genre podcast with random game play
 Genre Description
Humor & Comedy This genre is applicable when the primary perceived intent of
the video is to make people laugh, generally in the form of
scripted content whereby the comedic aspect of the short-form
video is planned, and unscripted content that can consist in
unplanned humorous situations or improv. It includes skits,

80 l
funny fails, stand-up comedy, situational humor, wordplay,
funny videos, etc.

63 tia
Example 1; Example 2; Example 3
Life Experience This genre is applicable when the primary perceived intent of
a short-form video is to invite viewers to observe or take part
in an event or process in one’s life, a collection of memories,
routines, special events or something interesting that

45 ne
Material Interest
happened. It includes video diaries, photo dumps, “day in my
life” mini vlogs, and videos with weddings, networking events,
or vacations. Example 1; Example 2; Example 3

This genre is applicable when the primary perceived intent of a

short-form video is to generate interest about, promote, or
14 id
showcase products, people, artifacts, and places. Videos of
this genre can range from content where a person discusses a
fashion piece, home décor, tech gadgets, pets, paintings /
sculptures, to tattoos, hairstyles, and make-up tutorials. These
videos often take the form of haul videos, unboxing videos,
nf

outfit of the day

videos, tours, and tutorials. Example 1; Example 2; Example 3;
Example 4

Performance This genre is applicable when the primary perceived intent of a

Co

short-form video is to share a talent or skill to impress or

engage. This includes short-form videos that show someone
who's dancing, performing a physical feat, playing sports,
painting, singing, trained animal activities, etc. Example 1;
Example 2; Example 3

Oddly Satisfying This genre is applicable when the primary perceived intent of a
short-form video is to share something beautiful, cute, and
visually and/or audibly pleasant. This includes short-form
videos with beautiful sceneries and things, cute animals,
ASMR, calligraphy, cleaning and pressure washing,
photography - generally videos that revolve around
aesthetically pleasing subjects.Example 1; Example 2;
Example 3; Example 4
 Informational This genre is applicable when the primary perceived intent of
a short-form video is to share instructions, information, or
recommendations that help the audience learn how to do
something, learn something new, or get guidance on a topic.
It includes how-tos, DIYs, lifehacks, etc. Example 1; Example
2; Example 3; Example 4

80 l
63 tia
None of the above If after repeated attempts, the main theme of the video
doesn’t map to any of the genres mentioned, categorize the
video genre as None of the above.

45 n
7. Metric Definitions
a. How well does this video match your interest?
e i. Think of generally what you are interested in viewing. What type of content do
you want to see when you go on these platforms? Does this content match that?
b. Was the video worth your time?
i. Based on what you just viewed, do you z think the viewing time was well spent,
in a way that you didn’t regret watching it?
14 id
c. Were you pleasantly surprised to see this video?
i. Did this have content that you haven’t seen before? Was the content about a
topic that you didn’t expect to see / expect the app to show you? It can mean the
content is interesting to see, you learned something new, an eye-opening
content, etc.
nf

d. How varied were the topics shown to you in this last session?
i. When evaluating topic diversity in your last session, consider whether the videos
spanned a wide range of subjects or focused narrowly on specific themes. Was
there excessive repetition of content? or a good balance between depth and
breadth of topics?
Co

e. How quickly did the app adjust based on how you interacted with the content?
i. Assess how quickly the app tailored the videos it displayed based on your
interests and previous interactions. Did the app seem to learn and adapt from
your activity from previous sessions? How fast did the app provide a
personalized experience that aligns with your preferences based on your
previous behavior?

8. How to screen record

a. For iPhone and iPad:
i. Open Control Center (swipe down on the top right-hand side of the screen),
tap the Screen record icon, then wait for the three-second countdown.
ii. If you don’t see Screen record:
1. Go to Settings Control Center, then tap + next to Screen Recording
(please note, you may already have this enabled in your control center).
 iii. Once the countdown starts, take note of the start time for your recording
session.
iv. To stop recording, open Control Center, tap. or the red status bar at the top
of the screen, then tap Stop.
v. Once you stop recording, take note of the end time for your recording
session.

80 l
b. For Android:

63 tia
i. Swipe down twice from the top of your screen.
ii. Tap Screen record. You might need to swipe right to find it.
iii. If you don’t see Screen record:
1. Click the pencil icon in the left bottom corner of the active tiles and
scroll down until you see the screen record tile.

45 ne
iv.
2. Hold and drag the Screen record tile to the top of the active tiles.
3. Press back to return to the previous screen where you should see the
words Screen record.
4. If you don’t see Screen record, swipe right on the active tiles.
Choose what you want to record, and tap Start. The recording begins after
14 id
the countdown.
v. Once the countdown starts, take note of the start time for your recording
session.
vi. To stop recording, swipe down from the top of the screen and tap the
nf

Screen recorder notification.

vii. Once you stop recording, take note of the end time for your recording
session.
Co