3

Yukon Qualification Exam

Part
At a Glance
Where is it?
Access the exam on your computer at https://www.raterhub.com/evaluation/

rater

Log in with the Gmail account

you provided to Appen

To view results, click on the

result title in the rating tool on

your computer, then navigate to

https://www.raterhub.com/url

on your mobile phone, and log

in with the same Gmail account

What's in it?
7 Needs Met tasks, each consisting of a query and multiple results

Determine user intent based on the query and location, review each result,

assign a Page Quality rating, then assign a Needs Met rating based on how

well the result meets the user intent

You must enter both PQ and NM ratings, but only NM will be scored

How do I pass it?

This is an open book exam, so keep the General Guidelines open

Quick Rating Guide for Needs Met:

Start at MM, then move the slider based on the following:

Is the result close enough to be helpful for a user in this location?

Yes, the result is close to the user's location: + 1 (MM+ to HM)

Yes, but closer results would be better: +05 . (MM+)

Location not important for this query: No change (MM)

Sort of, the result is near the user but too far to be convenient: - 1 (SM to SM+)

No, the result is nowhere near the intended location: FailsM (skip next question)

Does the result meet the query intent?

Yes, the result is highly relevant, accurate, up-to-date, highly credible, and gives the user

exactly what they are looking for and more: +2 (HM to FullyM*)

Yes, the result is highly relevant, accurate, up-to-date, and highly credible: +1 (MM+ to HM)

Yes, the result is relevant and meets the bare minimum of the query intent, but no more: + 0
to 05
. (MM to MM+)

Sort of, the result is relevant but slightly out-of-date or has minor inaccuracies: - 1 (SM to

SM+)

Not really, the result is on-topic but too broad, inaccurate, or outdated to be useful: - 2
(FailsM+ to SM)

No, the result isn't what the user is looking for and would satisfy no one: FailsM

*WSRBs are typically not rated above HM unless they satisfy a website intent or dual intent

Tips and reminders:

When unsure between two ratings, choose the lower rating

News or encyclopedia information must be accurate, up-to-date, and highly

credible

Medical or science information must align with well-established medical or

scientific consensus

Page Quality + Needs Met:

Useless results should always be rated FailsM, even if they have a high PQ rating;

useless is useless

On-topic, helpful results with low PQ should get a lower NM rating than on-topic,

helpful results with high PQ; page quality is part of helpfulness

The HM rating is appropriate for on-topic, helpful results with high PQ, or for on-

topic, very helpful results with medium PQ and other desirable characteristics

The HM rating is not appropriate for results with low PQ or any other undesirable

characteristics such as outdated or inaccurate information

SM is often appropriate for low quality but on-topic pages, but sometimes pages

have such low quality that they are useless for nearly all queries and should be

rated FailsM (e.g. gibberish pages, unless they are an exact match for a query with

clear website intent, in which case they should be rated FullyM)

Pages that lack a beneficial purpose should be rated Lowest PQ, regardless of

NM rating or page design (see Section 4.1)

Always rate the following as Fails to Meet:

Harmful to Self or Other Individuals (Section 4.2)

Harmful to Specified Groups (Section 4.3)
Harmfully Misleading Information (Section 4.4)
Untrustworthy (Section 4.5)
Spammy (Section 4.6)
Porn (Section 15.1)

Special Content Results Blocks (SCRB) vs. Web Search Result Blocks (WSRB):

SCRBs provide "at a glance" results without WSRBs include only a title, url, and brief

needing to go to a website. These should be snippet. These should be rated based solely on

rated based on block content and can be the landing page content (ignore the snippet)

rated as high as HM+ or FullyM so long as they and are typically not rated above HM unless

meet all requirements for the rating. they satisfy a website or dual intent.