Guidelines: Competitive Ad Load

A. Tooling:

l
ADAP (Appen’s Annotation Tool) - Raters will record responses in the ADAP tool and Appen team lead will

04 ia
upload aggregated metrics to a shared OneDrive folder weekly.

B. Annotation Process Overview:

Initial Set Up

91 nt
1. Each rater should use their main existing accounts on each platform (Facebook Reels, Instagram
Reels, TikTok, YouTube Shorts) with at least 3+ months of active usage history. This ensures that
data gathered is representative of real user ad load experience for mature accounts. NOTE: Raters
should not use a YouTube Premium account, as this will remove all ads.

7
2. Raters should commit to completing a full session consisting of 100 video labels.
12 ide
3. For this project, raters are expected to complete 2 full sessions on each app (Facebook Reels,
Instagram Reels, TikTok, YouTube Shorts) for a total of 8 sessions.
4. The rating process involves the following steps:
1. Record mobile device screen as the rater scrolls through the app on FB Reels, IG Reels,
TikTok, or YouTube Shorts and enters information into ADAP about each video, including:
■ Whether the video is an ad or non-ad
1. If ad, what type of ad
2. Other subjective questions about the rater’s perception of each ad
2. Upload each screen recording into Appen’s cloud storage folder for QA auditor review
nf

immediately upon session completion.

■ NOTE: Link to cloud storage folder will be provided via email to raters during project
onboarding.
5. Guidance for violating content: If the content contains imagery, text, or audio indicating any of the
below (non-exhaustive) that the raters are not comfortable viewing, the rater can skip that video and
move to the next video.
Co

● Child Exploitive Imagery (CEI), Child (Sexual) Solicitation, Child Abuse

● Human Trafficking
● Non-Consensual Intimate Imagery
● Credible Threats of Terrorism
● Potential mass killing or violence
● Self-injury or suicidal content
● Credible Threats of Violence
6. Raters will not be able to access any of the annotated data once entered into ADAP

Rater Steps

1. Sessions should be completed using the full version of each app (i.e. TikTok Lite is not allowed) on
mobile device.
■ NOTE: Before starting the screen recording, check your phone’s available
storage to ensure that there is space available to save the ~60-minute
 session recording (approximately 8-10 GB) and check to ensure the
phone battery life will last the full duration of the ~60-minute session.
Charge the phone before initiating work if necessary.
2. Enable “Do Not Disturb” mode on the mobile device to prevent interruption during your session.
3. Turn on screen recording for the session. Each session should be ~60 minutes of uninterrupted
recording (100 total video labels should require approximately 60 minutes per session with an
additional 15 minutes expected for screen recording upload).

l
■ NOTE: Notifications that do not automatically stop your scrolling session
or bring you into other apps are NOT considered interruptions and you

04 ia
should continue the session.
4. Open the chosen app for your session, navigate to the appropriate surface within the app (“For You
Page” on TikTok, “Shorts” tab on YouTube, “Reels” tab on Facebook and Instagram), and begin
scrolling through videos.
■ TikTok – For you Page

91 nt
■ YouTube – Shorts tab
■ Facebook – Reels tab
■ Instagram – Reel tab

TikTok ad classification will be determined by whether or not the video has one several tags, including

7
12 ide
“Sponsored,” “Paid partnership,” “Eligible for commission,” “Promotional content,”
“Targeted Ad,” or “Ad Box” at the bottom left of the screen (screenshot below).

Appropriate place on Sample TikTok Ad

TikTok to begin scrolling.
classification
nf
Co
 Other TikTok sample Ad classifications
“Banner Ad” & “Ad Box”

l
04 ia
91 nt
7
12 ide
nf

YouTube ad classification will be determined by whether or not the video has an “Ad” or “Sponsored” tag
at the bottom left of the screen (screenshot below).
Appropriate place on
YouTube to begin scrolling. Sample Youtube Ad
Co

classification
Facebook ad classification will be determined by whether or not the video has a “Sponsored”, “Banner
Ads” or “Post-loop Ads” tag at the bottom left of the screen(screenshot below).
Appropriate place on
Facebook to begin scrolling. Sample Facebook Ad
classification

l
04 ia
91 nt
7
12 ide
nf

FB Banner ad FB Post-loop ad
Co
Instagram ad classification will be determined by whether or not the video has a “Sponsored” or “Banner
ad” tag at the bottom left of the screen (screenshot below).
Appropriate place on
Instagram to begin scrolling. Sample Instagram Ad
classification

l
04 ia
91 nt
7
12 ide
nf

IG Banner ad
Co
5. For each video shown, enter into ADAP whether it is an ad or not (Y/N). If the video is not an ad,
do not watch it to the end—simply mark that it is not an ad and continue scrolling.
Each non-ad label should take 10 seconds or less. If the video is an ad, watch the entire
video and respond to the additional questions in ADAP.

6. For each video that is labeled an ad in ADAP, an additional set of questions will appear. Respond to

l
each of them. These questions include:
■ “Is this an ad?”

04 ia
● Yes
● No
■ “What type of ad is this?”
● [Responses vary by app—different options for TikTok, YouTube Shorts, FB
Reels & IG Reels]

91 nt
■ “How relevant or irrelevant to you is this ad?”
● Not at all relevant
● Slightly relevant
● Moderately relevant
● Very relevant
● Extremely relevant

7
12 ide
■ “How much do you feel this ad is offensive, misleading or inappropriate?”
● A great deal
● A lot
● A moderate amount
● A little
● Not at all
■ “How entertaining did you find this ad?”
● Not at all entertaining
● Slightly entertaining
nf

● Moderately entertaining
● Very entertaining
● Extremely entertaining
■ “How would you categorize the type of ad this is?”
● Static single-image ad (not a video)
● Slideshow ad with multiple images / animations (not a video)
Co

● Video ad that feels professionally done

● Video ad that feels homemade
● I don't know
■ “What is the size of the media for this ad?”
● Full screen image or video
● Square or half screen with image or video
● I’m not sure
■ “How visually pleasing is this ad?”
● Not at all pleasing
● Slightly pleasing
● Moderately pleasing
● Very pleasing
● Extremely pleasing
■ “Why did you select your previous answer regarding whether or not it is visually
pleasing? Select all that apply”
● The human presence in the media
● The position of the text
 ● The wording used in the text
● The stickers and/or emojis in the media
● The color scheme
● Image or video quality
● Image or video was full screen
● Other
■ “Is there sound in this ad? Select all that apply”

l
● Yes, there is background music
● Yes, there is talking by people seen in the ad

04 ia
● Yes, there is voiceover by someone not seen in the ad
● No, there is no sound
■ “Are humans used in the media? Select all that apply”
● Yes, they were using or demoing the product
● Yes, they were talking to the camera

91 nt
● Yes, they were in the media, but not doing any of the above
● No, there were no humans in the media
■ “Did the ad catch your attention within the first three seconds?”
● Yes
● No
● I don't know

7
12 ide
■ “Which helps you the most to understand what this ad is about?”
● Background music or voiceover
● Visual (image or video)
● Text in the media
● Caption / description at the bottom
● This ad is difficult to understand
■ “How well does this ad blend in with non-ad content on this app?”
● It blends in well
● It blends in somewhat
nf

● It doesn't blend in
■ “Which of the following best describes this ad? Select all that apply”
● Intriguing / Inspiring
● Amusing / Funny
● Touching / Heart-Warming
● Thought provoking / Insightful
Co

● Neutral (providing information without triggering emotions)

● Annoying / Displeasing
● Boring / Mundane
■ “Are you more or less interested in learning about the product/service after seeing
this ad?”
● More interested
● Less interested
● No Change
7. ADAP will display the current video number for each entry so that the rater knows where they are in
relation to completing 100 labels.
■ After 100 labels have been completed in ADAP, end your scrolling session and stop
screen recording.
8. Upload the screen recording into Appen’s cloud storage folder for QA auditor review.
■ Appen will provide detailed instructions on uploading, reducing file size of screen
recording, etc. in a separate document.

END