Supporting Material: Factuality and

Rating Justifications
This guideline goes through the different criteria you will need to identify when going
through the Rating section in depth. We encourage you to go through it, complete the
quizzes without initially using the answer keys and have it as a reference throughout
the course.

Factuality
How to determine the factuality of the responses?
The first thing we want to assess when checking the accuracy of an answer is to
know if the answer have verifiable statements.
Some common things that appear as verifiable statements include:

 Names of People or Places

 Dates of Historical Events
 Statistics
 Scientific Facts
 Laws
 Geographical Information
 Mathematical Statements

These statements break out into explicit statements and implicit statements:
Explicit statements: The statement itself is about the fact. These are easier to spot.
Implicit statements: The statement itself assumes the fact to be true. These can be
more difficult to detect.

Explicit Statements
Examples of these explicit statements are:

 The Berlin Wall came down in 1997

 The Miami Heat were the NBA’s most prolific scoring team last year
 The largest waterfall on earth is called Niagara Falls in Buffalo, NY

You can see that the statements above have a primarily goal of expressing the
verifiable statement.
You can also pretty easily determine whether these statements are true by searching
online:

 Question #1: "What year did the Berlin Wall come down?"
 Question #2: "What NBA team scored the most points in the 2022-2023
season?"
 Question #3: "What is the largest waterfall on earth by volume?" and "What is
the largest waterfall on earth by height?"

In each case, 30 seconds spent typing into Google and verifying with the first few
links will show you that:
  The Berlin Wall actually came down in 1989
 The Washington Wizards were actually the NBA's highest scoring team last
season; the Miami Heat were the lowest scoring
 Angel Falls (by height) and Inga Falls (by volume) are the largest waterfalls in
the world

Now you have to be careful because Google Spotlight is not always completely
accurate (the first thing that shows up when you type into Google). It is best practice
to scan the first 3-4 links below the Spotlight to see if they’re giving the same
information.

Quiz: Can You Identify All the Explicit Statements?

Becoming an expert in chess, defined by the United States Chess Federation
(USCF) as having a rating of 2000, and by the International Chess Federation (FIDE)
as having a rating of 1800-2000, requires a significant amount of study, practice, and
competitive play.

. Learn the Basics: The game of chess has 6 different types of pieces (pawn,
rook, knight, bishop, queen, king), each with its own movement rules. The
World Chess Federation (FIDE) provides a comprehensive guide to the rules
of chess.

. Study Opening Principles: Recognized openings have been studied

extensively and categorized into 'open', 'semi-open', 'closed' and 'semi-
closed' games. The Encyclopedia of Chess Openings (ECO) is a significant
resource containing around 1000 different opening variations.

. Master Tactics and Middle Game Strategies: Tactics involve short-term

attacks or defenses, while strategies involve longer-term plans. A survey by
Chess.com analyzed 2 million games and found that the most common
tactical themes were forks and pins.

. Endgame Knowledge: The endgame is when there are few pieces left on the
board. According to GM Dejan Bojkov, the ability to convert a pawn into a
queen (pawn promotion) is one of the critical skills in the endgame.

. Play and Practice: Regular play helps put learning into practice. According to
a study published in Acta Psychologica in 2007, more experienced chess
players can recall significantly more pieces from a game position than
novices.

Remember, the journey to becoming an expert in chess is a marathon, not a sprint. It

requires dedication, patience, and a love for the game.
Identify all the explicit statements:

 “The game of chess has 6 different types of pieces (pawn, rook, knight,
bishop, queen, king)”

 “The World Chess Federation (FIDE) provides a comprehensive guide to the

rules of chess”
  “Recognized openings have been studied extensively and categorized into
‘open’. ‘semi-open’, ‘closed’, and ‘semi-closed’ games”

 The Encyclopedia of Chess Openings (ECO) is a significant resource

containing around 1000 different opening variations"

 “Tactics involve short-term attacks or defenses, while strategies involve

longer-term plans”

 “A survey by Chess.com analyzed 2 million games and found that the most
common tactical themes were forks and pins”

 “The endgame is when there are few pieces left on the board”

 “According to GM Dejan Bojkov, the ability to convert a pawn into a queen

(pawn promotion) is one of the critical skills in the endgame”

 “Regular play helps put learning into practice.”

 “According to a study published in Acta Psychologica in 2007, more

experienced chess players can recall significantly more pieces from a game
position than novices”

 “Remember, the journey to becoming an expert in chess is a marathon, not a

sprint”

 “It requires dedication, patience, and a love for the game”

Answer key:

 “The game of chess has 6 different types of pieces (pawn, rook, knight,
bishop, queen, king)”

 “The World Chess Federation (FIDE) provides a comprehensive guide to the

rules of chess”

 “Recognized openings have been studied extensively and categorized into

‘open’. ‘semi-open’, ‘closed’, and ‘semi-closed’ games”

 The Encyclopedia of Chess Openings (ECO) is a significant resource

containing around 1000 different opening variations"

 “Tactics involve short-term attacks or defenses, while strategies involve

longer-term plans”

 “A survey by Chess.com analyzed 2 million games and found that the most
common tactical themes were forks and pins”

 “The endgame is when there are few pieces left on the board”

 “According to GM Dejan Bojkov, the ability to convert a pawn into a queen

(pawn promotion) is one of the critical skills in the endgame”
  “Regular play helps put learning into practice.”

 “According to a study published in Acta Psychologica in 2007, more

experienced chess players can recall significantly more pieces from a game
position than novices”

 “Remember, the journey to becoming an expert in chess is a marathon, not a

sprint”

 “It requires dedication, patience, and a love for the game”

🌟 "Regular play helps put learning into practice" is NOT an explicit verifiable
statement. In fact, it is a claim. "the journey to becoming an expert in chess is a
marathon, not a sprint" and "it requires dedication, patience, and a love for the game"
are both also claims. They are not verifiable facts - someone could reasonably
(although it might be rare) disagree. All the other statements are explicit and
verifiable -- they are either true or not true. The game of chess either has 6 different
types of pieces or it doesn't, there is no debate. FIDE either does provide a guide to
the rules of chess or it doesn't. Chess.com either ran a survey on 2 million games
and found the most common tactical themes were forks and pins, or they didn't.
When you come across these kinds of statements - you should IMMEDIATELY
recognize that you should look up the information online to confirm it's true :)

Implicit Statements
Remember: Implicit statements are where the statement itself assumes the fact to be
true. These can be more difficult to detect.
Examples of these implicit statements are:

 One of the most stunning landmarks in Rome, the Eiffel Tower has attracted
tourists from all over the world
 On one of the episodes of his show, The Barbeque, Lebron James and his
guests discussed the challenge of celebrity knowing that you can’t walk
around in most cities living a normal life
 Game of Thrones is a popular fantasy show. It follows Ned Stark and his 8
children throughout their various battles.

Within each of these implicit statements is an embedded explicit statement. Can you
spot them? They, in essence, are:

 The Eiffel Tower is in Rome

 Lebron James has a show called The Barbeque
 Ned Stark has 8 children in the show

Additionally, the statements have their core purpose which may or may not be an
explicit statement in its own right:

 The Eiffel Tower has attracted tourists from all over the world
 Lebron James and his guests discussed the challenge of celebrity on one of
his episodes
 Ned Stark and his children are the protagonists of Game of Thrones

You must validate all of these facts as best you can - but a good heuristic is to first
Some you will be able to do very easily (most of the above), but others might be a bit
more challenging (i.e., Did Lebron James and his guests discuss the challenge of
celebrity?) and require more research.
As a general rule, don't spend more than 5 minutes trying to verify a single statement
- at that stage use your best judgement from your research and move on.

Implicit Verifiable Statement: An Example

Statement: “While traveling frequently back to DC and leaving his wife Sandra home
in Los Alamos during the Manhattan Project, Robert Oppenheimer eventually had an
extra-marital affair.”
All the verifiable statements you would need to look up:

 Robert Oppenheimer’s wife’s name is Sandra

 Robert Oppenheimer and his wife lived in Los Alamos during the Manhattan
Project
 Robert Oppenheimer traveled frequently to DC during the Manhattan Project
 Robert Oppenheimer had an extra-marital affair during the Manhattan Project

You would then need to determine what would be best to look up online:

 Robert Oppenheimer’s wife’s name is Sandra —> “what was Robert

Oppenheimer’s wife’s name?
 Robert Oppenheimer and his wife lived in Los Alamos during the Manhattan
Project —> “where did Robert Oppenheimer and his wife live during the
Manhattan Project?”
 Robert Oppenheimer traveled frequently to DC during the Manhattan Project
—> “did Robert Oppenheimer travel frequently to DC during the Manhattan
Project?”
 Robert Oppenheimer had an extra-marital affair during the Manhattan Project
—> “did Robert Oppenheimer have an extra-marital affair during the
Manhattan Project?”

Quiz: Implicit vs. Explicit Statements

Identify all the IMPLICIT statements below.

 Ever since its first publication in 1611, the Iliad has been one of the most
influential texts in Western literature.

 The speed of light in a vacuum is approximately 299,792 kilometers per

second.

 The hit musical Hamilton, written by J.K. Rowling, has won numerous Tony
Awards and has been praised for its innovative blend of hip-hop and
traditional musical theatre styles.

 Known for its iconic ring system, Mercury is a popular subject of study among
astronomers.

 The Great Fire of London occurred in 1666.

  The highest peak on Earth, Mount Everest, stands at approximately 8,848.86
meters above sea level.

Answer Key:

 Ever since its first publication in 1611, the Iliad has been one of the most
influential texts in Western literature.

 The speed of light in a vacuum is approximately 299,792 kilometers per

second.

 The hit musical Hamilton, written by J.K. Rowling, has won numerous Tony
Awards and has been praised for its innovative blend of hip-hop and
traditional musical theatre styles.

 Known for its iconic ring system, Mercury is a popular subject of study among
astronomers.

 The Great Fire of London occurred in 1666.

 The highest peak on Earth, Mount Everest, stands at approximately 8,848.86

meters above sea level.

🌟 For the Iliad, the implicit verifiable statement is that the Iliad was published in 1611
(false); for Hamilton, it is that it was written by JK Rowling (false), for Mercury it is
that it has a ring system (false), and for Mount Everest it is that it is the highest peak
on Earth (true)

Let’s do some Live! Video!

Rating Justifications
What Makes Up a Good Justification?
Now, let's talk about your justification.
Here are the four most important dimensions of quality for justifications:

 Conclusion: The overall claim that the justification makes as to which

response is better
 Supporting Claims: The key supporting points that the justification makes to
defend its conclusion.
 Specific Evidence: The precise examples or evidence in the text used to
support each supporting point.
 Analysis: The explanation(s) of how the evidence defends the supporting
claim

Consider this justification:

“Response A is the better answer given Response B includes both an unsafe and
factually inaccurate remark. While Response B is likely formatted and structured
more effectively, it does not compensate for its more egregious issue.”
  Factuality: Response B claims “speed limits are more like guidelines”, which
is factually incorrect. Speed limits are laws, and if you are caught breaking
them you are subject to fines, license suspension, and/or even jail time.
 Safety: Response B encourages the user to drive faster and break speed
limits in order to arrive at their destination more quickly. This is directly
promoting illegal and unsafe behavior, as breaking speed limits can get you in
trouble with the law and lead to more physically severe incidents such as a
car crash.
 Formatting & Coherence: Response B is structured as a numbered list with
bullet-pointed suggestions and opening and concluding sentences for each of
its sections. This is more readable and digestible than Response A’s
paragraph format.
 While Response B is a bit easier to read and follow, its flagrant
recommendation to break speed limits renders it the worst of the two
responses.

Let's Break it Down:

The conclusion is "Response A is the better answer..."
The supporting claims are:

 Response B includes an unsafe remark

 Response B includes a factually inaccurate remark
 Response B is likely formatted better
 The errors in Response B outweigh the issues in Response A

The specific evidence is:

 Response B claims “speed limits are more like guidelines” [factuality]

 Response B encourages the user to drive faster and break speed limits in
order to arrive at their destination more quickly [safety]
 Response B is structured as a numbered list with bullet-pointed suggestions
and opening and concluding sentences for each of its sections [formatting]

The analysis is:

 Speed limits are laws, and if you are caught breaking them you are subject
to...
 This is directly promoting illegal and unsafe behavior, as breaking speed
limits can get you in trouble with the law...
 This is more readable and digestible than Response A’s paragraph format…

Quiz: Conclusions
Which of the following are valid conclusions?

 Response A is better than Response B…

 Both responses are similar quality, but Response A is slightly better…

 Response A makes a factuality error

 Response B does not uphold safety guidelines

Answer Key:

 Response A is better than Response B…

 Both responses are similar quality, but Response A is slightly better…

 Response A makes a factuality error

 Response B does not uphold safety guidelines

🌟 A conclusion is the single and clear statement indicating the final judgement of
which response is better. "Response A makes a factuality error" and "Response B
does not uphold safety guidelines" are both supporting claims that would defend a
conclusion for one response being better than another.

Supporting Claims
It’s really important to make these clear and articulate in your justifications.
What is a Good Supporting Claim?
A supporting claim should either be:

 Focused --> each claim focuses on one dimension or underlying reason.

 Supportive of the conclusion OR acknowledging of the nuance --> each claim
should either support the main conclusion (which response is better) or
provide nuance as to why the other response has some qualities that it might
spike on.
 Clear and coherent --> each claim should be stated directly and
unambiguously.
 Defensible --> each claim should have supporting evidence, so make sure to
think about why you believe the claim to be true and how you can prove it.

Some Examples
"Response A is more factually accurate than Response B" --> this claim is focused,
supports the conclusion, and clear. For it to be defensible, there would need to be
particular evidence showing how Response B makes factuality errors to a greater
degree than Response A
"Response B does not follow the instructions of the prompt" --> similarly this claim is
focused, would support the conclusion that Response B is the better response, and is
clear.

Quiz: Supporting Claims

Please select which of the following are NOT supporting claims

 Response B is wordier and less verbose than Response A

 Response B specifically says “Albert Einstein worked at Columbia University”

 Response A has less sophisticated writing than Response B

 Response A is overly verbose

  Response B fails to respond to the prompt’s request for 3 reasons NOT to
drink alcohol

Answer key:

 Response B is wordier and less verbose than Response A

 Response B specifically says “Albert Einstein worked at Columbia University”

 Response A has less sophisticated writing than Response B

 Response A is overly verbose

 Response B fails to respond to the prompt’s request for 3 reasons NOT to

drink alcohol

🌟 Claims are not indisputable facts, they are assertions that can be argued one way
or the other (even if one side of the argument has very little, if any, evidence to
support it). The comments about Albert Einstein and failing to respond to the
prompt's request are facts, not claims

Evidence
This is where most people trip up!
Evidence is essential in your justifications. If you can't provide evidence to back up
one of your claims, then the claim is likely weak.
What Does Good Evidence Look Like?
A good piece / set of evidence should be:

 Relevant --> each piece of evidence should directly serve to validate its claim.
It should clearly and effectively illustrate the point being made
 Verifiable --> the evidence should be present in the response texts
themselves - anyone reading your justification should be able to verify the
evidence by reading the responses
 Sufficient --> There must be enough evidence to support the claim. A single
piece of evidence may not be enough to convincingly support a claim.

Examples of Good Evidence

 Claim #1: Response A is factually less accurate than Response B

 Evidence #1: While Response B inaccurately mentioned that Italy won the
World Cup in 2010, Response A correctly identified that Spain was the
winner.
 Claim #2: Response B is overly verbose and redundant
 Evidence #2: The 2nd paragraph mentions the "internet being vast and full of
information" multiple times within a few sentences.

Quiz: Evidence
Please select which of the following are GOOD pieces of evidence:
  Claim: Response A fails to address the main explicit constraint of the prompt
|| Evidence: The prompt specifically asks to create a meal plan without dairy,
but the response includes yogurt as part of lunch.

 Claim: Response B is not formatted according to guidelines || Evidence: The

response hops around between multiple different ideas sporadically without
clear cohesion.

 Claim: Response B strikes an inappropriate tone for the prompt || Evidence:

The prompt asks to write a cover letter for a job, but the response uses terms
like “What’s Up!” and “It was wild”.

 Claim: Response A is categorically unsafe || Evidence: In the 3rd paragraph,

Response A says “isnt the world small”.

Answer Key:

 Claim: Response A fails to address the main explicit constraint of the prompt
|| Evidence: The prompt specifically asks to create a meal plan without dairy,
but the response includes yogurt as part of lunch.

 Claim: Response B is not formatted according to guidelines || Evidence: The

response hops around between multiple different ideas sporadically without
clear cohesion.

 Claim: Response B strikes an inappropriate tone for the prompt || Evidence:

The prompt asks to write a cover letter for a job, but the response uses terms
like “What’s Up!” and “It was wild”.

 Claim: Response A is categorically unsafe || Evidence: In the 3rd paragraph,

Response A says “isnt the world small”.

🌟 Saying "the response hops around between multiple different ideas sporadically
without clear cohesion" is actually a claim, NOT a verifiable fact. And the evidence
for Response A being unsafe is not actually relevant to the particular support claim.

Analysis
People also trip up here :)
Analysis, which often can be grouped into evidence depending on the type of error,
can be important in your justifications.
When Do You Need Analysis?
Analysis is particularly important for claims that are not obviously proven / disproven
with one fact (i.e., Verbosity, Writing Style, etc.) It should serve to clarify the
connection between the evidence and the claim.
Examples of Good Analysis

 Claim #1: Response A is better formatted than Response B

 Evidence #1: It is structured as a bulleted list with suggestions and an
opening and closing statement, whereas Response B is a paragraph.
 Analysis #1: Given the prompt is asking for a number of athletes who live in
LA, it is ideally suited for a list.
  Claim #2: Response B uses awkward grammar and is less readable than
Response A
 Evidence #2: Its opening line “Today, I had a conversation. A paradoxical one
at that, with a flower, no less!” is split into two fragmented sentences.
 Analysis #2: The latter fragment is particularly poorly phrased, and this idea
could have been much more directly conveyed by saying “Today I had a
paradoxical conversation with a flower!”

Putting It All Together

So you have everything you need to start writing excellent justifications :)
The Essentials
A good justification must have:

 A clearly stated conclusion

 Supporting claims that logically defend the conclusion AND/OR acknowledge
the relative strengths of the other response
 Sufficient evidence to back up each supporting claim
 Analysis that explains and/or illustrates how the evidence backs up the claim

Always Remember

 Someone reading your justification should know within the first 5 seconds
which response (if any) you think is better.
 You should mention the dimensions where the two responses vary in quality.
 Your evidence should cite or reference specifics in the response texts.
 You don't need your justification to be redundant, but it should contain each of
these elements.

Response A is better because it follows all the instructions of the prompt.

Both responses are good, and although Response B mentions outdoor adventures, and
team building activities for the weekend getaway, it doesn't talk about accommodations
for the trip as asked for in the prompt.

Completeness: Response B doesn't complete all the asks of the prompt, by not
providing more information about where people will stay. "there are cozy
accommodations nearby, be it campgrounds or comfier lodgings." Instead of a general
suggestion, Response A states "YMCA of the Rockies" and describes its amenities,
which will be more helpful for those planning the trip.

Though both responses gave interesting and insightful ideas for the weekend getaway,
Response A is better because it gives more specific information around
accommodations.