QA Content

Moderation
Training
November 22
Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved
Overview

1. Quality Performance Thus Far

2. QC Review & Escalation Process

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Performance Thus Far- SPI

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

QC Review & Escalation Workflow

Rating
No Abuse
QC Review
Strike 1

Rating Remove
Strike 2
+Escalation Abusive Content

Remove
Strike 3: Revoke
Abusive Content
Abuse

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

 QC Review & Escalation Workflow

I. List of Experts provided to QC Team

I. We have an algorithm that monitors CF score, # ratings, average TTA, # of edits
per answer, previous quality scores.
II. QC Team Reviews Answers provided by Experts
I. This determines what kind of review & feedback to provide
III. If Expert showed no abusive behavior, simple rating of work done.
IV. If Expert showed abusive behavior- escalation is needed.
V. Escalation Process: 3 Strike Policy
I. Strike 1: “Advice” Warning
II. Strike 2: Warning
III. Strike 3: Revoke Access (Repeated instances/Flagrant abuse)
VI. Remove Abusive Content

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

“Advice” Warning Email Example

Subject line for warning would

be “Violation of Chegg Q&A
Guidelines”

expertwarning@chegg.com will
send warnings.

We recommend reaching out to VMT rather than submitting a support ticket.

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

 Additional Warning Email Examples

Abuse type - Need more info

Abuse type - False Report

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Revoke Access Email Example
chegg@alerts.chegg.com will
send this revoke email.

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

 Revoke Rules

∙ Stringent 3 Strike Policy- action taking on Strike 3

∙ We rarely remove someone immediately without review of their work & behavior.
∙ We have a zero tolerance policy for abusive behavior:
o Vulgar language
o Harassment
o Adult content
o Racism
o Other ethically questionable behavior

▪ Reviews of Abuse utilizes the same criteria as the initial quality/abuse monitoring algorithm
▪ CF Score – student rating

▪ Fraud Rating

▪ Authoring Time (Avg time, avg TTA under 2 min)

▪ Edit %

▪ Skip %

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Violating Expert ”Code of Conduct"
Abusive Behavior that breaks our Code of Conduct

• Illegal: Defamatory, libelous, copyright, privacy, patent, trademark, or contract rights violations

• Violent: Abusive, harmful, harassing, threatening

• Inappropriate: Sex, indecency, bodily function, obscenities, death/trauma/grief, socially

offensive topics

• Prejudiced: Bigotry, racism, xenophobia, sexist, ageist, disability, religious discrimination

• Malicious: Security threats, exploitative

• Fraudulent: Commercial activities, contests, sweepstakes, unsolicited advertising, spam

• Misappropriating: Modifying, copying, distributing, downloading, scraping, or transmitting

content

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Examples of Abusive Behavior & Honor Code Violations:

∙ Solicitation
∙ False Report
∙ Incorrect Skip Reason
∙ Copyright
∙ Profanity/Spamming
∙ Expert answering HCV (Exam/Test/Quiz) question
o Examination/ Test Questions
o Opinion/ Perception- Based Questions
o Essay Questions
∙ Incorrect, Incomplete, irrelevant solution- submitted on purpose.
∙ Illegible/Blurred answer
∙ Third Party Url
∙ Plagiarism

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Re-instatement Workflow

1. Expert reaches out with Re-instatement request – This request will generate
by vendor management team.
1. Vendor leadership to VMT
2. A disciplinary revoke (abusive behavior or solicitation) will not be reinstated.
3. An expert who was removed for inactivity (90+ days) will automatically be
reinstated –
4. For all other revoke reasons- evaluation of past performance conducted
1. General timeline- 1 week
2. Review Parameters- Student ratings, tenure on platform, general
QC/fraud ratings.

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Next Steps

13

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Soft Launch of Moderation System

• Timeline
• Training week of 11/1
• 2 week trial – “silent moderation” until 11/14/2022
• Then “active moderation” for rest of November
• Folks will get strikes & revoked
• It is up to vendor leadership to review & actively address pain points.
• Chegg VMT will review quality ratings and warnings weekly.

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Abusive Behavior
Examples

15

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

 Copyright Violation More examples of questions you shouldn’t solve (skip under “This is
abusive or spam” reason):

Experts should not answer

copyrighted material.

Experts should not use

Dealing with: material in
copyrighted
their answer.

X Copyrighted Content

X Exam / Copyrighted Content

Confidential Material / © 2022 Chegg, Inc. / All Rights Reserved

False Report & Incorrect Skip Reason

• We monitor skip rate and skip reasoning by expert, as well as

compare the skip reasons used on a given question.

• Experts who skip too frequently in a given timeslot are reviewed

closely.

• Experts should not purposely use the wrong skip reason or falsely
report questions.

• Those who consistently use the wrong skip reason or report

questions that are benign will be removed.

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Solicitation

• Students may reach out providing links to third party sites where they’d like you to do
their work, do not work off platform with students.

• Asking students to work off the platform or providing links or contact information to
do so, is prohibited.

• Sharing personal identifying information (PII) is prohibited. PII such as personal email
id, phone number, etc.

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Profanity & Spamming

• Experts are prohibited from using vulgar language, harassment, adult content, racism,
other ethically questionable behavior
• Experts should not send multiple responses to students or “spam” them.

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

 Incorrect, Incomplete, irrelevant solution
• Incorrect Solution- Experts will be marked
negatively if they have provided not only an
incorrect solution, but If their answer has
more than 2 dislikes to an answer

• Incomplete Solution – We ask that experts

answer the whole question supplied by
student.
• If the expert believes they cannot answer
the full question in the given time they
should skip. Refer to answering guidelines
for multi-part questions)
• If an expert answers only a few parts of a
question they will be marked down.

• Irrelevant solution- Unrelated answer

X
provided will negatively impact quality
score.

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Minimal or Blank Explanation

• Expert giving one-word

answer or very less
information in the answer

• This also includes providing an

answer that simply states
“Need more information.”
• If you have insufficient
information, skip the question
and use corresponding skip
reason.

X Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Third Party URLs
• Students will sometimes provide
URLS to third party sites with their
assignment.
• For your own safety and the safety of
the student’s personal information,
do not follow links provided by the
student.

• A link indicates that it may also be

copyrighted material- meaning even
a screenshot of the page cannot be
worked on.

• Skip these questions and mark

“abusive or spam”

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Illegible/Blurred Answer
• In Live Q&A, Experts should not submit
blurry or hard to read answers.
• This includes messy handwriting or
formatting.
• Experts who consistently submit low rated
answers due to illegibility will be negatively
rated.

• If a student submits a question photo that is

hard to read, skip the question rather than
guessing at the content.

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Plagiarism: Direct
Direct Plagiarism - Copying/pasting or
screenshotting exact work from another
source is direct plagiarism.

This includes:
• Answers/portions of answers, written
passages, excerpts
• Diagrams or images

• Our internal review team holds a

zero-tolerance policy against plagiarism.

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Plagiarism: Self Plagiarism
Self Plagiarism - When an expert has submitted an answer that they have written as
original content more than once. Full duplication is not permitted; experts are welcome to
revise and improve their previous answer.

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Honor Code
Violation
Examples
26

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

 Examination/ Test Questions
Examination/ Test Questions
Do not solve questions with terms like “ACS Exams”, “Exams Institute”, “Div.
Ched”, “ACS”, “Sapling Learning”, as these are examination / copyrighted
questions.
- Look at the top or bottom of the page
You must skip these under “This is abusive or spam” reason.
for indication of due date/exam title
- In this example you can see the
“Exam” name at the top of the first
page, as well we “End of test” on the
last screenshot.
X Exam Question X Exam Question

- Sometimes students are given

practice exams - so look for dates.

- You may see points next to

questions as well.

Confidential Material / © 2022 Chegg, Inc. / All Rights Reserved

Examination/ Test Questions Do not solve questions with terms (without any date) like “Final Exam”, “Test”,
“Exam”, “Quiz” etc. We do not support cheating/academic dishonesty.
Please skip them under “This is abusive or spam” reason.

• Again, here you can see at the top &

bottom of certain pages “Exam”

• Experts should always report & skip

Dealing with:
questions with exams, tests,
X Exam Question

quizzes, etc.
X Exam Question
Do not solve questions which require you to perform an activity or experiment
to arrive at the final solution. Skip such questions should under “This is
abusive or spam” reason.

Note: Questions belonging to any previous years’ examination, quiz or test are not
spammed and can be attempted.

Questions with terms like “Exam Review”, “Practice Test”, ”Practice Exam” are not
abusive and can be attempted.

Confidential Material / © 2022 Chegg, Inc. / All Rights Reserved

Opinion/ Perception- Based Questions

• This is a trickier issue to watch out

for.
• These questions usually ask for a
long answer or essay.

• Their primary focus is to get the

student’s personal opinion on an
highlevel idea or concept- this is
something an expert cannot assist
with. X Opinion Based Question

Do not solve questions that explicitly ask for your opinion or personal inputs.
You must skip such questions under “This is abusive or spam” reason.

Confidential Material / © 2022 Chegg, Inc. / All Rights Reserved

 Essay Questions

Solve questions that explicitly ask for “an essay writing” to be solved by :
1. providing detailed instructions on how to structure the essay.
2. including key points or examples.
3. explaining concepts used to formulate the essay.

Do not provide a complete explanation, thesis or write an essay.

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Resources
• Plagiarism Guidelines
• Live Authoring Guidelines
• 9.2022 SQNA Authoring Guidelines

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Performance Thus Far- TBD
• What areas are they currently struggling in?
• Pattern in behavior?- specific metrics
• Subjects that need the greatest intervention?

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Questions?

33

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved

Thank You.

34

Confidential Material / © 2021 Chegg, Inc. / All Rights Reserved