Is the Answer Credible?

Task Definition
The authority of an answer’s source is very important to a user, especially when
the query is about a critical topic such as health issues, legal matters, finances, etc.
In this task, we provide:

• A User Query: A search query that has been submitted by a user into a
search engine
• An Answer: A text response pulled from a website that is displayed to the
user in an attempt to satisfy their query
• The Answer URL: A link to the Source Page where the Answer was extracted

Your task is to identify how much credibility a user would expect a website to have
in order to trust it to answer their query. Then you will decide whether the shown
answer is extracted from a credible source that can be trusted by the user.

Instructions
There are 5 steps in this process:

1. Credibility level: Determine how important is the credibility of the answer

to the user:
o High-Risk/Impact: The query is about a sensitive topic for which the
credibility of the answer is important and could have a profound
impact on an individual. E.g.: health, taxes, finances...
o Low-Risk/Impact: The query represents a common man’s everyday
question that can be answered by a reasonably authoritative source.
There are also queries on topics which are not actively researched,
documented, or written about by any institutions or authors having
reputation in the topic (such as gossip, opinion pieces, etc.). If the
Credibility Level of the answer is not High-Risk, it will automatically
be Low-Risk
2. Check the source page: You need to spend time looking at the source page
of the answer to determine the credibility of the page.
o You can check the source page by clicking on the Answer URL or the
“Web Source” tab on the top-right of your screen.
o Make sure the page is still available. If the page is unavailable, select
"Not Credible".
3. Select the credibility of the source: Use the table below to guide your
decision about the credibility of the page:

4. Compare the credibility of the Source Page to the Search Results:

o Once the credibility of the source page has been determined, look at
the Google results tab and determine if there are any URLs with
higher credibility in the search results.
o Note: Remember that even if two results are “Credible” according to
the above table, one URL can still be more credible than the other.
For example, for the query “How to Install Microsoft 10” both
Quora.com and Microsoft.com would be seen as “Credible” in the
table above, but Microsoft.com would be the highest authority on
the topic and would be seen as more credible
5. Make a Judgment
o After comparing the Credibility of the source page to other URLs in
the search engine results, make a final Judgement based on:
1. The credibility level of the query
2. If there are more credible sources available in the Search
Results
Please consider the factors below in making your judgment:

• How credible is the Answer URL?

Oftentimes you can use prior knowledge to tell if the URL is credible based
on its domain name. For example, www.irs.gov is a highly credible domain
for tax matters, while www.nih.gov and www.mayoclinic.org are highly
credible domains for health related issues.

If you cannot tell by the domain name, click the URL, and look for signs of
whether it is credible. For example, does the writing appear to be
professional? What is the educational background and credential of the
author(s)? Does the URL have valid citation links? Is it cluttered with junk,
ads, or other spammy content? When in doubt, follow common sense.

• Does the Answer URL have credibility on the intent of the User’s Query?

Just because a source page has a credible Answer URL, author, or

references does not automatically make it a credible source for the user’s
Query. For example, if a user is looking for the “Top 10 Restaurants in
Seattle”, www.irs.gov will never be a credible source on the topic – even
though it is an extremely trustworthy site.

Making A Judgment
How you come to the final rating of the Hit and what Judgments are available to
you depend on the credibility level. Below are each Judgement available
depending on the credibility level and situation:

Credibility Level: High-Risk/Impact

• Credible
o The Source Page is credible and there are no pages in the Search
results that are more credible on the query topic
• Not Credible
o The Source Page is not credible enough for the query given
 o There are pages in the search results that are more credible than the
Source Page on the query topic
Credibility Level: Low-Risk/Impact

• Credible
o Answer is from a highly credible or fairly credible source for the query
topic
• Less Credible
o Answer is from any source that is not spammy/junky/misinformation
site
• Not Credible
o Answer is from a Source Page that is actively spammy, junky,
misinformative, or otherwise harmful to the user
Rating Detrimental Queries and 404/Unavailable Source Pages

There are two cases where the answer will be Not Credible regardless of the
credibility level of the query.

• Detrimental Queries
o Situations where a query may be controversial, untrustworthy,
misleading, biased, or embarrassing to show to the users.
• 404 / Unavailable Source Pages
o Situations where the source page is broken and will not load
o Cases where the page loads, but the information on the page is no
longer available
Examples

Query- is autism a handicap?

Answer- Autism is not a handicap or a disability; it is a sign of the awakening of

very powerful intellectual abilities, and it is a gift that needs intense stimuli and
encouragement.

Guideline: The query is High-Risk as it involves health-related content and the

user would expect a high level of credibility from the source page. Because of
this we would mark the first question as “High-Risk/Impact”. After clicking on
the source URL, the source page appears to be a personal blog on Jesus and
would not be authoritative on this health-related topic. Due to this being a high-
risk query and the authority of the source page not being high enough for the
query topic, the final rating would be Not Credible.

Query: Where to find topaz in terraria?

Answer: Rarely, Topaz can be found in bulk in its own special gemstone cave, or
in one that contains all of the other gems as well (besides Amber). Topaz is the
second-least valuable gem available, being able to craft the Sandgun, a type of
ranged weapon, along with the rest of the items that usually all gems have
counterparts of.
https://www.terraria.gamepedia.com/Topaz

Guideline: The query is Low-Risk as it does not have a high impact on the user
and would not need a higher level of credibility. Because of this we would mark
the first question as “Low-Risk/Impact”. After clicking on the source URL, the
source page appears to be a wiki on the game Terraria and would be a high
enough credibility for the Query. Due to this being a Low-Risk query and the
authority of the source page being high enough for the query topic, the final
rating would be Credible.

Query: Isaac newton’s discovery

Answer: Sir Isaac Newton (Dec. 25, 1642 - March 31, 1727) was an English
mathematician and physicist who invented calculus (simultaneously but
independently of Leibniz), formulated the law of gravitation and discovered the
laws of motion. He also investigated the nature of light

https://www.enchantedlearning.com/subjects/astronomy/glossary/Newton.sht
ml

Guideline: This query is Low-Risk as it does not have a high impact on the user.
We would mark the first question as “Low-Risk/Impact”. The source page has no
author, citations, or affiliation so we would judge the page itself as less credible
on the credibility scale. Looking at the search engine results, there are more
credible pages available. Due to this being a Low-Risk query with a not very
credible source page and better sources available in the search results, the final
rating should be Less Credible.

Query: Growing Plants

Answer: Seed-starting containers should be clean, measure at least 2-3 inches

deep and have drainage holes. They can be plastic pots, cell packs, peat pots,
plastic flats, yogurt cups, even eggshells. As long as they are clean (soak in a 9
parts water to one part household bleach for 10 minutes), the options are
endless.

https://www.natria.com/learn/rose-flower/10-steps-growing-plants

Guideline: This query is Low-Risk as it does not have a high impact on the user.
We would mark the first question as “Low-Risk/Impact”. The source page in this
case results in a 404 error. Due to the Source Page resulting in a 404 error, the
final rating should be Not Credible.