Guidance: Understanding the

Credibility of Information v2.1

Why are we doing this?

We are doing this to better understand the Facebook ecosystem as far as how
individuals conducting this task assess the credibility of information

Big points to keep in mind:

This document contains ever-evolving and ever-improving guidance on how to

think about the questions and what the response options mean. This isn't intended
to be a set of guidelines because this isn't intended to be a purely objective task.
There will be some variability between people in how they approach the task and
how they respond to the questions, and that is by design. However, try as much as
possible to follow the instructions laid out below and answer each question to the
best of your ability.

Starting Point

This set of tasks will ask you to evaluate information from a claim in an article
related to the core premise of that article.

By core premise, we mean the main point(s) or take-away(s) in the article that the
authors expects you to understand. Usually the core premise is described in the
article's headline, though it may be related to the unstated reason why the article
was written in the first place (e.g., breaking news).

A claim is a statement about the core premise that could be falsifiable: with
evidence, you could consider the information in the claim to be “true” or “false” (or
somewhere in between). An article may contain many separate, distinct claims
related to the core premise. Note: in some cases, the claim may be interpreted to be
the same as the core premise.

Example
Rolling Stone reported that “Anthony Bourdain Dead at 61 of Apparent Suicide.”
https://www.rollingstone.com/culture/culture-news/anthony-bourdain-dead-at-
61-of-apparent-suicide-629356/
The core premise is that Anthony Bourdain has perhaps recently died, and this is
novel news to the general population.

You might consider two claims in this article: 1) the claim that Anthony Bordain
died (perhaps you did not know this, as it is a recent development), or 2)
the claim is that Anthony Bourdain specifically died of a suicide (perhaps you
already knew that he died but not by what cause).

Part 1: Context

PURPOSE: The purpose of the following question is to gauge how bad the impact
might be if the core premise – the piece of information you're evaluating – turned
out to be false.

Guidance for the following question:

• You should be able to answer the following question relatively fast, looking
only at the page title and perhaps the first few lines of text.

Q1.1: To what degree would it matter to someone's

knowledge, livelihood, or safety if the information in this
core premise was wrong or misleading?
• Not a lot - It would not matter if the information
were wrong or misleading.
• Somewhat - It may matter if the information were wrong
or misleading.
 o e.g., sports, local events, celebrity gossip or
entertainment stories.
o Includes “trivial news”
• A Great Deal - It would matter if the information were
wrong or misleading. The claim could impact one's
life, such as:
o Medical, legal, financial, political, commercial,
or other categories where being wrong might
matter.
o News articles that are important for fostering an
informed group of people, community, or nation.
This could include information related to
government processes, natural disasters,
international relations, etc.

PURPOSE: The purpose of the following two questions is to gauge the type of
content you're evaluating.

Guidance for the following two questions:

• If your answer is “Yes, and it's clear/obvious” to the satire/parody question,
then you do not need to answer remaining questions.
Q1.2: Would a reasonable person conclude that the core
premise is satire or parody? Parody here means humorous
imitations of serious literature, media, etc.; satire here
means information that uses irony, sarcasm, or ridicule to
expose or denounce human foolishness, immorality, or
degrading behavior.
• Yes, and it is clear or obvious that the core premise
is satire or parody.
• Yes, but it is not obvious that the core premise is
satire or parody.
• No, it does not appear to be or is definitely not
satire or parody.
• Unable to Determine

Q1.3: Would a reasonable person conclude that the core

premise is opinion? Opinion here means both the Opinion
section of professional news coverage (e.g., an "op-ed" or
editorial article) or someone's personal opinion as written
in a blog, social media post, personal webpage, or related
non-journalistic coverage.
• Yes, it is opinion, and it is an "op-ed" or editorial
article
• Yes, it is opinion, and it is an article, social media
post, or blog post expressing a personal opinion
• No, it does not appear to be opinion
 • Unable to Determine

Part 2: Content

PURPOSE: The purpose of the following questions is to gauge whether a source of

information that you trust supports or rejects the claim that you're evaluating.

What is the claim of the article?

• Q3.0: Claim:
o Paste the exact text of the claim from the
original article, or summarize the claim in your
own words. The claim could be the headline or
another statement from the article.
• Q3.1: Search:
o If you used a search engine, type the text of the
search keywords you first used to begin to find
evidence for or against the claim. If you
searched multiple times, still only include the
first set of keywords you used.

Guidance for the following questions:

• We ask you to find “evidence” for or against a claim, where evidence means a
relevant statement about the claim. It does not mean that the statement
always supports the claim; in other words, it could be evidence against the
claim.
• To find evidence for the claim, try a web search on a search engine of your
choice for the primary keywords in the core premise or claim. To find an
evidence link, you can use the following methods:
o (using the Anthony Bourdain example)
▪ Within a specific search engine of your choosing, across
multiple sources:
• for example: “Anthony Bourdain dead suicide”
at http://duckduckgo.com, then clicking into specific
pages in the search results
▪ Within a specific search engine of your choosing, within a
specific source:
• for example: “Anthony Bourdain dead in:npr.com”
at http://google.com, then clicking into specific pages in
the search results
Can you find a URL from a source you trust that seems to
either agree with or dispute the claim?
• Q3.2: Did you find evidence?
o No Evidence - I could not find any evidence about
the claim I'm evaluating
o Weak Evidence - I found evidence that somewhat
agrees with or disputes the claim
 o Strong Evidence - I found evidence that clearly
or entirely agree with or disputes the claim
• Q3.3: Is the source you used one you trust?
o Mostly Trust - I generally trust the source where
I found the evidence
o Somewhat Trust - I maybe trust the source where I
found the evidence, but I'm uncertain
o Do Not Trust - I do not trust the source where I
found the evidence
• Q3.4: If you found evidence, which link was it from?
o Paste only the text of the URL/link here. Do not
include additional text from the webpage,
article, etc. Make sure that the URL/link is from
the article of the source and not the URL/link of
the search engine results.
• Q3.5: What statement related to the claim did the link
you found make? Copy and paste one statement from the
link you found.
o Copy and paste 1-3 text statements (phrase,
sentence, or paragraph) directly from the
evidence page you found (NOT including the
headline) that support or reject the claim. If
the statements come from different sections of
the link you found, include them in separate
boxes below.
o If the statements appear in different
sections/paragraphs of the page, paste them
separately into Boxes 2 and 3 below. If the
evidence is not in text format on the page and is
instead in an image, video, or other format, type
out the evidence, paste it into statement
box(es), and check the appropriate additional
box(es) below. If the evidence is in both text
and another format, include both the text below
and check off the appropriate boxes.
▪ Statement 1 text: (Required) <text box>
▪ Second Statement, copy and paste
below: (Optional) <text box>
▪ Third Statement, copy and paste
below: (Optional) <text box>
▪ If the statement is in another non-text
format, it is included in: [] image(s) []
video(s) [] other format(s) <explanation
box>
• Q3.6: What is your decision about the claim?
 o False - The claim is without merit. It is false
information, a hoax, or a scam.
o Misleading - The claim has flaws or biases that
make it misleading. A source that I trust has an
alternative angle that is not fully represented
in the original article's core premise or claim.
o True - The claim is confirmed. It is true.

Examples
Example 1 - True

Rolling Stone reported that “Anthony Bourdain Dead at 61 of Apparent Suicide.”

https://www.rollingstone.com/culture/culture-news/anthony-bourdain-dead-at-
61-of-apparent-suicide-629356/

The core premise is that Anthony Bourdain has perhaps recently died, and this is
novel news to the general population. You decide that the claim is that he
specifically died of a suicide. You are trying to find evidence to support or reject if
Anthony Bourdain died of a suicide.

CNN might be a source that you trust, and one link from CNN – which hosted
Anthony Bordain's TV show – supports the claim that Anthony Bourdain did commit
suicide. https://www.cnn.com/2018/06/08/us/anthony-bourdain-
obit/index.html A statement in the CNN article is “His good friend Michael Ruhlman
said he was stunned by news of the suicide,” which supports the original claim.
Example 2 - False

Kypo6.com reported that “Pope Francis Shocks World, Endorses Hillary Clinton for
President.”
https://web.archive.org/web/20160725221648/http://kypo6.com/breaking/pope
-francis-shocks-world-endorses-hillary-clinton-for-president-releases-
statement/ (the webpage is now deleted, but a version is viewable at this link)

The core premise is that the Pope has perhaps endorsed Hillary Clinton for as a
Presidential candidate leading up to the 2016 US Presidential election. You decide
the claim is the same: that the Pope endorsed Clinton. You are trying to find
evidence to support or reject if the Pope endorsed Clinton.

Hoax-Slayer.net might be a source that you trust, and one link from this website
rejects the claim that the Pope endorsed Hillary Clinton. https://www.hoax-
slayer.net/no-the-pope-has-not-endorsed-hillary-clinton-for-president/ A
statement in the article is “The claims in the story are untrue. The Pope has not
released any such statement and he has not publicly endorsed Hillary Clinton for
President. There are no credible news reports that support the claims in the story. It
comes from a network of fake-news websites that publish an ongoing stream of
clickbait nonsense tricked up as news,” which rejects the original claim.