Download as docx, pdf, or txt
Download as docx, pdf, or txt
You are on page 1of 2

the requirement specifies that the response must be in Vietnamese

How to compare responses


A vs B Response Rating Cheat Sheet
When to strongly prefer one response over the other:

 One response has accuracy issues and the other does not
 One response has major grammatical errors and the other does not
 One response completely responds to the prompt and the other does not

When to weakly prefer one response over the other:

 One response has minor errors that the other does not, in any dimension
 When the errors in one response, when compared to errors in the other response, produce a
significantly different user experience

 For example, a user prompt asking about the inspirations of Monet’s paintings:

 One response may give the correct inspiration but misidentify where Monet lived

 The other may respond with incorrect inspiration but correctly identify where Monet lived

 In this case, the first should be weakly preferred

When to mark one of the two middle (close to neutral) ratings:

 When the difference between the two responses comes down to preference and is subjective
 If the only difference between A and B is based on personal opinion, rather than an objective
metric, choose one of the two neutral ratings.

The 8-point scale


Think of the scale like this:

1. A is so much better than B, and B is almost useless.


2. A is much better than B. B is clearly nowhere near as useful as A.
3. A is slighly better than B.
4. Almost neutral, but I'd choose A.
5. Almost neutral, but I'd choose B.
6. B is slightly better than A
7. B is much better than A. A is clearly nowhere near as useful as B.
8. B is so much better than A, and A is almost useless.
How to write a justification
You will write two types of justifications in this project. The first one is justifying why you chose
Response A over Response B, or vice versa. The second one is justifying how you rate your selected
response (A or B). Here are the guidelines:

1. Always refer to Response A or Response B directly, never as "the selected response" or "the
first/second response" or "the other response".
2. Always be VERY specific. If you choose A over B because B has grammatical errors, point out
where those errors are.
3. Ensure the grammar and reasoning in your justification is logical and clear, and mention as many
details as possible.

You might also like