Guide to Content

Moderation & AI:

Everything You
Need to Know

Human 280 seconds

AI 200 milliseconds
Guide to Content Moderation & AI: Everything You Need to Know Introduction

Contents Introduction

2
Introduction
When user-generated content (UGC) is often discussed, its

definition is narrowed to purely social media shares.

Considerations for Developing a Moderation Strategy


UGC is everything from the profiles people build on dating apps to products

4 Common types of moderation


listed for sale on online marketplaces to reviews on travel websites. It's all of

5 Techniques for moderating content

the digital content and interactions created by everyday people. From dating

apps to marketplaces, moderation is used almost everywhere UGC is shared or

uploaded.


6
 How It Works

Businesses face concerns and challenges when trying to institute or evolve

Benefits of AI-Automated Content Moderation

their content moderation strategy. Your strategy should comply with your

7 Scale operations
 corporate terms and conditions of what defines appropriate.

8 Increase accuracy

This guide covers the following:

8 Reduce costs

1
Considerations when developing a content moderation strategy

9
Content Moderation AI with Clarifai
Content moderation techniques

3
New technologies to consider

4 Benefits of adopting AI

2
Guide to Content Moderation & AI: Everything You Need to Know
Considerations for
Developing a 
 Cultural context
Moderation Strategy
Businesses face concerns and challenges when trying to institute
or evolve their content moderation strategy.
Your strategy should comply with your corporate terms and conditions of what
defines appropriate. Having a clear understanding of what content requires
moderation helps to organize work internally and builds trust with your
customers, users and community members.
Developing a Moderation Strategy
Let’s take a look at a few things to consider when developing your
moderation strategy.
W hen a business operates across different countries, it’s easy to overlook
cultural nuances. User habits, types of online communications and the tone of
communications can vary greatly between individuals — let alone nations.
Consider humor: what’s socially acceptable in one country can be considered
treason in another. Understanding local customs, laws and languages is
critical, and country manager moderators come into play here.
Semantics
Technologies often lack the contextual resources to understand a piece of
content. NLP can analyze text posts based on tone or content but can miss
the nuances in meaning as it relates to a particular phrase or structure of a
sentence and word placement. For example, "I saw him on the hill with a
telescope." Even this simple short sentence has a variety of interpretations.
Business-specific topics
Each business has different categories of content they need to moderate, and
often this content cannot be found by common moderation standards, like
GARM. Companies can develop specific taxonomies to create granular
business rules for curating content that meet their terms and conditions.
3
Guide to Content Moderation & AI: Everything You Need to Know Developing a Moderation Strategy

Common Types of Moderation

Post-moderation
Reactive Moderation

Moderation occurs after the content is posted. The drawback to this method is as The responsibility to report inappropriate content to the website owner is with
communities grow, resources needed to grow and costs become a factor. the community users. The advantage of this moderation method is that you can
Keeping up with what to moderate and unposting content quickly can become an scale with your community growth without putting extra strain on your
issue. From a legal standpoint, the website owner becomes the publisher of the moderation resources or increasing costs.

content, which exposes them to liability risk.

Distributed moderation
Automated moderation

This method combines community self-moderation with a rating system. Content This method is the most common type of content moderation method. It involves
is published directly to the website, and users vote on whether the submissions using advanced artificial intelligence, computer vision and natural language
are appropriate and within community guidelines or rules. Users are in control of processing to review content with minimal human involvement. This method can
the comments or posts with some guidance from human moderators.

moderate images, video, audio and text content, including text within images, and
AI automation allows for content to be reviewed faster and with greater accuracy.

Should this text be shown to our community?

Pre-moderation

Moderation is completed before content is posted. This method provides higher

control of what's seen but delays content from being published. It's ideal for
communities with a high level of legal risk, like children's communities, where Toxic Obscene Threat Insult Identity Hate
protection is vital.

4
Content
Guide to Moderation
Content Moderation
& AI: Everything
& AI: Everything
You NeedYou
to Know
Need to Know Developing a Moderation Strategy

Techniques for Moderating Using different AI technologies, many types of content can be
moderated. Let’s break this down by data types.
Content in any Format Image Classification

AI and computer vision use algorithms and models to distinguish Detect inappropriate image content using visual search and tag and categorize
between features in images, videos, text and audio—from obvious it based on your business criteria.

things like pornography to more under-the-radar details like hate

Image Detection

speech embedded in photos and videos.

Use classification technology to detect harmful image content and then
identify its exact placement within a scene.

Video Moderation

Review video footage frame-by-frame by breaking it into parts and marking

each detected label and video timestamp with confidence scores.

Text Classification

Analyze and categorize text using NLP and assign predefined tags to it based
on its contents and sentiment.

Object Character Recognition (OCR)

Combine text classification with OCR to identify inappropriate text within an

otherwise harmless image.

Audio Moderation

Convert speech in videos to text and check it for the use of profanities, hate
speech, or other inappropriate language.

5
Guide to Content Moderation & AI: Everything You Need to Know How it works

Step 1

How it works
User generated content is uploaded to a community or marketplace.

Step 2

Using AI automation, a set of predictions are made as to whether the

content is undesirable.

Step 3

Pre-trained and custom models identify the inappropriate visuals or

Here is an example of how AI automated content moderation text, and these are used to predict when other data is likely to have
works to improve operational efficiency for reviewing and posting similar inappropriate content.

user generated content: Step 4

When model confidence meets a predetermined threshold of the low

probability that the content is unsafe, it is automatically posted without
human oversight.

AI automation tags and

categorizes content.
Approved content is
automatically posted.

User uploads
 Clarifai AI

content Questionable content is sent
to human moderators for
further review.

6
Guide to Content Moderation & AI: Everything You Need to Know Benefits of AI-automated content moderation

Benefits of
 1. Scale operations

AI-Automated
 Whether you're an online marketplace or social application, as your company
grows, so will your user-generated content. A lack of a scalable moderation strategy

Content Moderation
The gold standard for content moderation has always been human
moderators. However, there are critical problems with this manual
approach: scalability, effectiveness and cost.
inhibits growth. Companies that rely solely on human moderation teams are limited to
moderating only content samples. On average, moderators can screen ~200 pieces
of content daily, which does not include video streams. AI automation speeds review
times up to 100x. By automating the classification and detection of inappropriate
content, only a select number of questionable content routes to humans for further
review, increasing the number of images moderated daily and reducing the time to
post content significantly.

Another increasingly important consideration driving AI adoption is

human moderator burnout from the stream of toxic, offensive and Op en ab e business is based on aggregating user-generated content
T l ’s ;

utterly inappropriate material they see daily. Not only are however, it poses significant trust and safety risks if illegal or unwanted
moderators affected but also those working to source the data, content is posted to its website. ith over
W ma e u oaded da ,
2M i g s pl ily

develop taxonomies and label it.

their team of five moderators can t catch everything that goes against
'

their terms of service. On average, moderators re e ed vi w ma e

300 i g s

da , only 1 of the 2 images, creating the potential of missing of

By building AI into their content moderation systems, companies are ily % M 99%

unwanted content.
taking steps to put the mental health of their employees first by



reducing their exposure to this material. Not only is that improving
job satisfaction, but it also offers employees the opportunity to
focus on higher-value, more rewarding tasks.
Clarifai helped them build a custom AI moderation model to curate their
restaurant photos specifically to their guidelines. After training the model,
the content was automatically reviewed, flagged if inappropriate and sent
only a sub-set to humans for review. As a result, Open able n rea ed the
T i c s

ima e moderated da rom

g s ily f to
300 .
5000

7
Guide to Content Moderation & AI: Everything You Need to Know
2. Increase accuracy
Some websites and marketplaces have millions of pieces of content uploaded
daily, and it can take moderators several minutes to thoroughly review only a few
for quality and appropriateness. With so much content to review, there is a high
margin for error and a strong chance of posting content that violates company
policy. With AI automation, content is reviewed based on consistent taxonomy
and is not subject to human bias and opinion. AI can categorize content based on
popular taxonomies, like GARM, to recognize sexually explicit images, threatening
language, and other inappropriate material faster and more accurately than
humans. As moderation models ingest more data, they get smarter. As a result,
moderators have fewer "false positives" to clear.
9GAG is a social media platform with over 100M daily users. They rely on
four moderators to manually process the 1000s of user uploads received
reduction in
daily. Relying solely on moderators made it easy for content to slip through
the gaps. Shifting from manual moderation to AI automation and a pre-built
Not Safe for Work (NSFW) model, they increased the amount of content
reviewed to 20K daily. They quickly classified images against NSFW
categories like gore, explicit and suggested nudity with 95% accuracy over
their manual process. On top of that, they reduced person-hours by over
90%, saving costs and allowing their team to reinvest time in more
impactful work, like growing their user community.
Benefits of AI-automated content moderation
3. Reduce costs
Recruiting, training and replacing human moderators can be costly.
Average salaries for content moderators in the United States can range
from $40k to $65k a year. By reducing just a fraction of the human
workforce, companies can experience significant savings.
On average, moderators can screen ~200 pieces of content daily, which
does not include video streams. AI automation speeds review times up to
100x. By automating the classification and detection of undesirable content,
fewer images need human review, resulting in greater efficiencies without
hiring an extra workforce.
700 to 20 70% 90%
35x reduction
 reduction in the
in moderator
 workforce needed
 moderation hours,
team size. to moderate UGC. saving 145 person-
hours weekly.
8
Guide to Content Moderation & AI: Everything You Need to Know

Content Moderation
AI with Clarifai
Clarifai’s AI-powered content moderation solutions enable
companies to moderate content at scale across multiple channels Drugs
with greater speed and improved accuracy. Our solutions provide
operational efficiency gains through quick and predictable
approval processes.

Get up and running quickly with pre-trained image, text, and audio models.

Create your own moderation taxonomy specific to your business needs.

Automate moderation to speed review times by 100x.

Build human-in-the-loop workflows for content that requires further review.

Detect and extract undesirable text within images with multimodal workflows.

For more information visit: www.clarifai.com/solutions/content-moderation

9 9
www.clarifai.com