Uolo v4.

0 - FAM Guidelines for Ad

Content
These guidelines are intended to be applied when reviewing content in all CR queues related to
“FAM” content.

“FAM” queues contain advertising content (i.e. ad content). While much of these guidelines are
consistent with the Guidelines specific to images, videos, and links (i.e. that used in other
content type-specific queues), there are elements that pertain specifically to ad content.

Here are the key take-aways for reviewing Ad Content:

• Ad content is unique because it often contains multiple different types of component parts
(i.e. links, images, videos, and text)
• You need to review all components of any given ad to successfully complete both Task A
and Task B.
• Your answers to the questions in each task should always represent the totality of the ad
content you are reviewing, not just any one component.
• Many ads contain both opinions and statements of fact. Whenever there is a statement of
fact in the content used to support a central claim you will be expected to evaluate that ad
content (i.e. not disqualify).

Overview of the task:

In this task you will be asked to research and evaluate claims stated in ad content.

This task is split into two parts - Task A: Content Description and Task B: Claim Assessment. Each
task will have its own rating queue.
• Task A Queue will focus on providing some information about the context of various pieces
of ad content
• Task B Queue will focus on identifying, researching, and evaluating the central claim stated
in ad content

Why are we doing this?

We are doing this to better understand the information shared and promoted on Facebook, in
terms of how individuals assess its credibility after doing some research.
Big points to keep in mind:

This document contains ever-evolving and ever-improving guidelines on how to think about the
task's questions and what the response options mean for evaluating ad content. One piece of ad
content may contain many media components such as text, photo, video, and links. To refer to any
one component in a piece of ad content (e.g. one video, or one photo), this guideline uses the
term “ad component.”

This isn't intended to be a strict set of guidelines, because this isn't intended to be a purely
objective task. There will be some variability between people in how they approach the task and
how they respond to the questions, and that is by design. However, try as much as possible to
follow the instructions laid out below and to answer each question to the best of your ability.

Starting Point

This set of tasks will ask you to evaluate the central claim presented in a piece of content.

A central claim is a statement of fact related to the content's main point or purpose. By a statement
of fact, we mean a sentence or main idea that the content is conveying as a fact (regardless of
whether or not it is factually accurate). An important characteristic of a statement of fact is that
it could be supported or contradicted by evidence (regardless of whether or not it would
be possible to find that evidence via an internet search).

The central claim is often described in one or more of these locations in ad content:
• image component: the image or overlaid text on the image
• link or article component: in the headline or title, or body text
• video component: the video, overlaid text, or audio
• text component: the caption for the ad (typically accompanying an image, link, or video)
[placeholder for new rendering visual of Ad compared to SRT components]

• Usually, a piece of content will have one primary central claim or a single multi-part claim,
where the parts are closely related. You should focus on identifying and evaluating this
central claim whenever possible.
• In some cases, however, a piece of content may contain multiple distinct central claims (i.e.,
equally central claims that are distinct enough that they would require multiple evidence
searches and evidence sources, for example, in a photo whose main purpose is to convey a
list of facts, like a photo meme about “5 things to know from today’s news”).
o In these cases, we ask you to focus on identifying and evaluating the most
severe claim (i.e., the highest impact claim, you evaluated in the Potential Impact of
the Claim section)
o For more guidelines on how to evaluate contents with multiple claims, see the
Claim Identification section in Part B.

How to review Ad content:

When reviewing ad content it is essential to review all of the text, image, video, and link
components for each ad. Different ads will be made up of different combinations of ad
components.

For example:
• This ad contains four components:
o (1) Text (“Video Message”)
o (2) Image (“Video Thumbnail”)
o (3) Video
o (4) Link (“Call To Action”)

• Another ad contains two components:

o (1) Image
o (2) Link (“URL")
o Note: there might have been a text component for this ad (“Caption”), but in this
case it is blank
• Another ad contains four components:
o (1) Image
o (2) Link (“Display URL”)
o (3) Link (“URL”)
o (4) Shared via app - this can always be ignored during review
o Note: there also might have been a text component for this ad (“Caption”), but it was
left blank
Important Instructions Unique to Ad Content:
• Once you are done reviewing all components of the ad content, you should consider
whether or not there is a central claim across the total set of components, holistically.
• If there is a central claim contained in any one component, then the ad content is
considered to be making a claim and should not be disqualified.
• If there are multiple links for the same ad, please click on all links to review for claims.
o In many instances the links will lead to the same article or website. When this is the
case, take a quick glance to verify they are indeed the same. Once you have
confirmed they contain the same information, you only need to review one link for
claims.

Example
The ad content and the link component are reporting information about a specific motorcycle
company, Harley-Davidson.
The main purpose of the content is to share news about the Harley-Davidson company’s finances
and production plans.

The central claim in this case is contained in two of the ad components: the text and the link.

In this case, the central claim has two parts: 1) that Harley-Davidson is increasing motorcycle
production in Thailand and 2) the company’s quarterly earnings fell.
To identify and evaluate the central claim in this ad content, you should review the photo, text, and
link article holistically.
A given piece of ad content may contain many separate, distinct claims, but we want you focus on
identifying and evaluating the central claim (which is related to the main point of the content).

For example, although the above article about Harley-Davidson’s business also contains many
statements about sales in the EU and the impact of tariffs on the company (e.g., stating that the
company “seeks to avoid a 31% EU tax on motorcycles made in the U.S.”), in this task you should
focus on evaluating the central claim only.

Some general instructions to keep in mind:

• To evaluate the central claim, you will be asked to search for evidence about the claim
using an internet search engine of your choice, and to provide some information about the
evidence you found (the steps of this process are described in detail below).
• It's possible that in some cases a central claim may be a statement of fact, but there may be
no publicly available evidence about that statement of fact. We will ask you to separately
assess: 1) whether a piece of content contains a statement of fact and 2) whether there is
available evidence about that statement of fact.
• The amount of time it will take to evaluate a piece of content will vary from item to item.
o Some ad content may be very easy to research and evaluate and will take only 1-2
minutes to complete. Other ad content with videos, links, or difficult to assess claims
or multi-part claims will require more time to research and enter your responses
(up to 20 minutes). Most items will fall somewhere in between.
o You will have up to 15 minutes of research time for each item.
o If you encounter an item that is very difficult to evaluate, and you do not feel you’ve
come to a satisfactory conclusion with your evidence search after 15 minutes of
research time, you should still answer the task’s questions to the best of your ability
based on the evidence you were able to find. You will have the option to specify that
you needed more time for the item at the end of the task.

Task A: Content Description

In the Task A queue, you will be asked to provide some information about the context of various
pieces of advertising content.

Disqualifying Issues

There are five reasons why ad content should be disqualified from being rated:
• (1) You are not able to review the content due to a bug or technical issue in SRT. This may
be because the SRT preview is broken and/or the ad content or caption does not render
properly. (Note that for links, only the title and URL will render - not an accompanying
image thumbnail.)
 • (2) You are not able to review the content because it is in a different language from what
you are assigned to rate.
• (3) You are not able to review the content because it was not possible to access the content
using the link provided. This may be because:
o the link is broken
o the content is missing or you encounter a 'page does not exist' message
o your web browser blocks the link or identifies it as unsafe
• (4) You are not able to review the content because the content is paywalled or requires a
subscription to view the full content. The website asks you to pay for content, or it requires
a subscription to access content and you do not have one.
• (5) The content does not contain a central claim that is based on a statement of fact. The
content does not contain a central statement of fact that can be supported or contradicted
(or somewhere in between) based on evidence. This may be because:
o The main purpose of the content is solely to convey an opinion or values
statement and it does not contain any statements of facts as rationale/support for
the opinion/value statement; or
o The main purpose of the content is not related to conveying a statement of fact
(see Guiding Examples for examples of this type of item).
Remember: You need to consider all components of the ad during your review and answer
questions with all components in mind.

If the item you are reviewing contains any statement(s) of fact related to the content's main
point or purpose, then you should not select this option.
• For example, if an ad contains a broken link, but the ad text and image contain a claim, then
you should not disqualify the ad content.

Additional Guidance for 'The content does not contain a central claim that is based on a
statement of fact' Label

PURPOSE: The purpose of this option is only to indicate that the item you are reviewing
contains no central statement(s) of fact.

Guidance for this option:

• You should not select this item if the ad content you are reviewing
contains any statement(s) of fact (i.e., a sentence or main idea that the content is
conveying as a fact) related to the content's main point or purpose.
• Remember: You should consider all components of a given ad (i.e. any images, image
captions, video, audio, video captions, overlaid text, headlines/titles of articles/links, and
body text of articles/links) to identify the central claim of the content as a whole. See the
claim identification section below for more details.
• To identify whether a video component contains a central claim based on a statement of
fact, you should review all additional components that accompany the video (for
example text caption and/or link accompanies the video), and watch the first 30 seconds of
video content (you may need to watch for a bit longer or need to skip to watch a few
 seconds of other portions in the middle or end of the video in some cases. You do not need
to watch the full video). It may be helpful to skim through the video transcript (when
available) to identify relevant portions of the video to watch.

• You SHOULD NOT use the 'The content does not contain a central claim that is based
on a statement of fact' label in any of the following instances:
o The item contains an unimportant, humorous, or trivial claim
§ Even if you think a claim is unimportant or you think it wouldn't matter if the
claim is wrong, if the content contains a claim that is a statement of fact, then
you should not select this option
§ For example, “Teen tweets from LG smart fridge, goes viral" is a funny claim,
but is a claim that was presented as a fact in news stories. This type of claim
should not be disqualified.
o The item contains a claim that you do not think there will be evidence about
or that you think that it would be difficult to find evidence for/or against
§ Even if you think that it may be difficult to find evidence about a claim, if the
content contains a claim that is a statement of fact, then you should not select
this option
§ For example, “Jane Smith lost her dog ” may seem like a claim that it will be
difficult to find evidence for/against (if Jane is not a celebrity or public
figure). However, this claim is stating a fact (i.e., that Jane lost her dog), even
if it may be a difficult to verify one. This type of claim should not be
disqualified.
o The item contains both statements of fact and statements of personal values
or opinions
§ Even if you think some of the information in the content is conveying
statements of personal values or opinions, if the content contains a central
claim that is a statement of fact, then you should not select this option.
• Example statement: “Bengal tigers are the most physically
overwhelming animals on Earth.”
o This is an opinion statement, and it does NOT use a
statement of fact to support its claim.
o This would be disqualified from review, unless there is a
statement of fact in the content to support this claim.
• Example statement: “Weighing up to 500 pounds, Bengal tigers are
the most physically overwhelming animals on Earth.”
o This is an opinion statement, but it does use a statement of
fact to support its claim. The statement of fact is that "[Bengal
tigers weigh] up to 500 pounds." The statement of fact is used
to support the opinion statement.
o Note that it does not matter whether this statement is true or
false, only that it is stated as a fact to support an opinion
statement.
o This would qualify for review.
 § Even if you think the central claim is a personal values or opinion statement,
if the content includes a statement of fact to support the central value or
opinion statement, then you should not select this option.
• Example statement: “The Wizard of Oz is the most important movie of
all time.”
o This is an opinion statement, and it does NOT use a
statement of fact to support its claim.
o This would be disqualified from review, unless there is a
statement of fact in the content to support this claim.
• Example statements: “The Wizard of Oz is the most important movie
of all time. A study found that The Wizard of Oz has been more
frequently referenced in subsequent movies and TV shows than any
other movie ever made.”
o The second sentence (“A study found...”) is used here as the
main supporting statement of fact for the opinion statement.
o This type of claim would qualify for review.
o As the central claim, focus on evaluating this supporting
statement of fact (i.e., that a study found this was the most
frequently referenced movie by subsequent movies/TV
shows).
• See the accompanying Guiding Examples document for additional
examples on how to identify opinion statements.

Q1.0: Are you able to review this item?

• Yes - I can review this item.
• No - I'm unable to review the content due to a bug or technical
issue in SRT.
• No - I'm not assigned to rate content in this language.
• No - I’m unable to review the content because it was not
possible to access the content using the link provided and there
were no claims elsewhere in the content.
• No - I'm unable to review the content because the content is
paywalled or requires a subscription to view the full
content and there were no claims elsewhere in the content.
• No - The content does not contain a central claim that is based
on a statement of fact.

For additional guidance on identifying disqualified ad content types, please reference the
Guidelines Examples document for Ad Content.
Part 1: Context
Potential Impact of the CLaim

PURPOSE: The purpose of the following question is to gauge how much it would matter if the
central claim turned out to be false.

Guidelines for the following question:

• Your answer to this question should represent the whole piece of ad content you are
evaluating. To answer the question, you should consider all components of the ad content
you are evaluating, including any text captions in or on media content (link/photo/video),
any video audio or transcript, and the content caption (displayed above the photo in the
review tool, if present).
o For content with a video component, this includes watching through at least the
first 30 seconds of video content (you may need to watch for a bit longer or need to
skip to watch a few seconds of other portions in the middle or end of the video in
some cases. It may also be be helpful to skim over the video transcript [when
available]).
• To answer this question, you may consider the potential impacts on the lives and well-
being of the general public or specific audiences who see the content, as well as on the
individuals, groups, or organizations described in the content.
o Although we provide some examples of types of claims that often fall into each
impact-level category, these are only meant to serve as illustration and are not
meant to be strict guidelines for which topics should be assigned to which impact
levels.
o You should use your best judgment to determine the impact level of each piece of
content you evaluate.
§ e.g., while medical claims that might impact a person’s healthcare choices
would matter A Great Deal (e.g., a claim about a new cancer cure), other types
of medical claims (e.g., a claim that eating too much chocolate causes acne)
might matter Somewhat or Not a Lot
• If you are rating a multi-part claim, you should provide the impact rating for the claim or
part of the claim that seems to matter the most (i.e., the highest impact or most
severe claim, according to the impact level descriptions outlined below).

Q1.1: To what degree would it matter to someone's livelihood,

decision-making, safety, political and social worldview, or well-
being if the information in this central claim was wrong or
misleading?
• Not a lot - It would not matter if the information were wrong or
misleading. It is unlikely that this information would have any
impact on those who see it or on the individuals or groups
described in the claim.
o e.g., entertainment stories, posts about extraordinary
animals, news about fashion or food trends
 • Somewhat - It may matter if the information were wrong or
misleading. It is possible that this information could have some
impact on those who see it or on the individuals or groups
described in the claim.
o e.g., life tips, reporting about non-violent local news
events, celebrity or sports gossip
• A Great Deal - It would matter if the information were wrong or
misleading. It could impact the lives or well-being of those who
see it or of individuals or groups described in the claim.
o e.g., medical, legal, financial, or political
information that could impact someone's livelihood, safety,
or well-being
o Content about subject matter that is important for
fostering an informed group of people, community, or
nation. This could include information related to
government processes, natural disasters, international
relations, violent crimes/crime rates, etc.

Content Type

PURPOSE: The purpose of the following two questions is to provide context of the type of content
you're evaluating.

Guidelines for the following two questions:

• Your answer to the following questions should represent the whole piece of ad content you
are evaluating.To answer the questions you should look over and consider all components
of the ad content you are evaluating, including any text caption, image, overlaid text, video,
and link.
o For links: Read through the title and first few lines of text (or watch a bit of the
video if one is included in the link).
o For videos: look over any text caption that accompanies the video, and watch
through the first 30 seconds of video content (you may need to watch for a bit
longer or need to skip to watch a few seconds of other portions in the middle or end
of the video in some cases. It may also be helpful to reference the video transcript,
when available).
o For images: Consider both what the image is portraying as well as any overlaid text.
• We ask you to tell us whether the content you are evaluating is Parody or Satire:
o Parody refers to humorous imitations of serious literature, media, etc.
o Satire refers to humorous content that uses irony, sarcasm, or ridicule to expose,
denounce, or make fun of human folly, immorality, and/or shortcomings.
o The main purpose of satire and parody is humor and entertainment, not to convey
facts.
• We also ask whether the content you are evaluating is Opinion:
 o Opinion here means someone's personal opinion as written in any of the ad
components.
o Some content may contain both opinion and factual statements (e.g., content may
contain a factual central claim to support an opinion.)
§ Therefore, you will still be asked to identify and evaluate any central
claims present for content labeled as opinion.
§ Additionally, not all personal webpages, blog posts or social media posts
contain opinion. You should only label content from these sources as opinion
if it is sharing an opinion.

Q1.2: Is this content satire or parody?

• Yes, and it is clear or obvious that the main purpose is satire
or parody. Most people would be able to tell that the content is
meant to entertain and not to convey serious information or
reporting.
• Yes, but it is not obvious that the main purpose is satire or
parody. Some people may mistake the content for serious
information or reporting.
• No, it does not appear to be or is definitely not satire or
parody.
• Unable to Determine

Q1.3: Is this content opinion?

• Yes, it is opinion, and it is an "op-ed" or editorial article
• Yes, it is opinion, and it is a blog post, social media post, or
personal webpage expressing a personal opinion
• Yes, it is an opinion, but it is NOT an “op-ed”, editorial
article, blog post, social media post, or personal webpage.
• No, it does not appear to be opinion
• Unable to Determine

For additional guidelines on identifying Satire/Parody, please reference the Examples Appendix to this
document.

Content Publication Date

PURPOSE: The purpose of the following question is identify how long ago the content was
published, to help identify claims that may be out-dated. This is only relevant for ad content
that contain links.

Guidelines for the following question:

• To answer the following question you should locate the publication date on the article/link
you are evaluating.
 • Usually this information can be found near the top of the page/below the headline or title
• Some websites (especially blogs sites) place the date and the end of the content/post, so
you should also check there if there is no date at the top of the page

Q1.4: How long ago was the linked content published?

• Less than 3 months ago
• Over 3 months, but less than 1 year ago
• Over 1 year ago
• The content does not contain a publication date
• N/A - The content does not contain a link/article

Task B: Claim Assessment

In the Task B queue, you will be asked to identify the central claim, search for evidence about that
claim, and make an evaluation about whether or not it is supported by evidence from trustworthy
sources.

At the start of Task B you will also have to opportunity to indicate whether the content should be
disqualified from review (using same question as Q1.0 above).

HOWEVER, items in this queue are items we have identified as very likely rate-able, and so
you should very rarely need to mark items in this queue as disqualified!

• It’s possible that items in this queue may contain opinions, but those items will likely
contain a central claim that is a statement of fact or provide statements of fact to support a
main opinion or value statement. Therefore, you should only need to rarely mark those
items as disqualified from review.

Q2.0: Are you able to review this item?

• Yes - I can review this item.
• No - I'm unable to review the content due to a bug or technical
issue in SRT.
• No - I'm not assigned to rate content in this language.
• No - I’m unable to review the content because it was not
possible to access the content using the link provided and there
were no claims elsewhere in the content.
• No - I'm unable to review the content because the content is
paywalled or requires a subscription to view the full
content and there were no claims elsewhere in the content.
• No - The content does not contain a central claim that is based
on a statement of fact.
Part 2: Claim Identification and Evidence Search

PURPOSE: The purpose of the following questions is for you to: (1) identify the central claim
made by the content that you're evaluating, and (2) find evidence from another source regarding
that central claim, that would allow you to assess whether it is is true, false, or somewhere in
between.

Claim Identification

Guidelines for the following questions:

• As discussed at the start of these instructions, the central claim should be one sentence or
idea from the content you are evaluating, that is a statement of a fact related to the content's
main point or purpose.
• Usually, a piece of content will have one primary central claim or a single multi-part claim,
where the parts are closely related. You should focus on identifying and evaluating this
central claim whenever possible.
o For example, if a post says: “Lactose intolerance is more common than any other
allergy worldwide. The WHO estimates that up to 2 billion people experience
symptoms related to lactose intolerance annually.” You may identify these claims:
§ Claim 1: Lactose intolerance is more common than any other allergy
worldwide.
§ Claim 2: The WHO estimates that up to 2 billion people experience symptoms
related to lactose intolerance annually.
o In this case, the second claim builds upon the first claim, so you would synthesize a
unified central claim: Lactose intolerance is the most common allergy in the world
and up to 2 billion people experience related symptoms each year.
• In some cases, however, a piece of content may contain multiple distinct central claims (i.e.,
equally central claims that are distinct enough that they would require multiple evidence
searches and evidence sources, for example, in a photo whose main purpose is to convey a
list of facts, like a photo meme about “5 things to know from today’s news”).
o In these cases, we ask you to focus on identifying and evaluating the most
severe claim (i.e., the highest impact claim, you evaluated in the Potential Impact of
the Claim section)
o If all the claims are equally impactful and relevant to the main purpose of the
content, you should evaluate the claim that is most apparent or otherwise obvious
as the central claim (for example, a claim in a bold font, or a claim that is circled in
red).
§ For example a post says: “Fun facts! 1. The world is over 25,000 miles in
circumference. 2. There are over 50 words for snow in the Inuit language. 3.
Drinking 4 cups of coffee a day reduces a woman’s risk of endometrial
(uterine) cancer by 22%.” You may identify these claims:
• Claim 1: The world is over 25,000 miles in circumference.
• Claim 2: There are over 50 words for snow in the Inuit language.
 • Claim 3: Drinking 4 cups of coffee a day reduces a woman’s risk of
endometrial (uterine) cancer by 22%.
§ In this case, you may decide that Claim 3 may affect reader’s choices about
their health, so it has the highest potential impact. You may choose to
evaluate Claim 3 as the central claim.
§ If the content only listed Claim 1 and Claim 2, and if Claim 2 was in larger,
bolded text, you may decide that the main purpose of the content was to
share information in Claim 2, and you may choose to evaluate Claim 2 as the
central claim.
• However, if the content still included Claim 3, then Claim 3 should still
be the central claim because it is more severe, even though Claim 2
might be in larger, bolder text.
• A statement of fact, is a sentence or main idea that the content is asserting as a fact
(regardless of whether or not the statement is factually accurate).
• An important characteristic of a statement of fact is that it could be supported or
contradicted by evidence (regardless of whether or not it would be possible to find that
evidence via an internet search).
o In ad content containing images, the central claim may be described in the content
text (i.e., the text caption of the photo and/or in text that is overlaid on the photo).
You should consider both types of text as well as the photo itself and all other
components when identifying the central claim for a piece of ad content.

Ad Examples for Identifying Claims

In addition to the examples below, please reference both of the following documents for examples
that apply to the review process for ad content:
• Guiding Examples - Understanding the Credibility of Information v4.0 - FAM Guidelines for
Ad Content
• Guiding Examples - Understanding the Credibility of Information v4.0 - Photos
o You likely already have access to this document. If you do not, please reach out to
your team lead.

When identifying claims in ad content, it is important to review all components to identify

any potential claims.
• Sometimes the headline or title of the ad content is a call-to-action (e.g. a link titled “Sign
the Kiwi petition today!”), but there is a central claim in the content body or another
component (e.g. the link leads to a webpage that states, “The Australian Kiwi is
endangered! Sign this petition to tell your representatives you want to save the Kiwi bird
today!”)

• Sometimes an ad will not contain a claim in the text or image component, but will contain a
claim in the link.
o In this case, the image and text for the ad content do not contain a statement of fact.
o When we review the link, we see it is asking people to complete a survey about
vaping and addiction. Additionally, there is a claim about the health consequences of
vaping devices: “vaping devices may cause respiratory illnesses, cardiovascular issues,
and seizures”. https://inhalation-complications.medical-recovery.com/
o Therefore the central claim you would evaluate for the ad content as a whole is the
claim found in the linked website.
• Sometimes ad content will contain a mixture of opinion and factual statements.

• In this case, the central claim is the statement of fact that can be supported or contradicted
by evidence.
o The central claim is making a factual claim about the number of people who hold a
certain opinion.
• This example:
o Central Claim: “81% of likely US voters agree it’s important to require businesses to
use E-Verify.”
o Search Terms: “voter survey results e-verify”
• Sometimes ad content text is long and contains many, distinct claims.
o In this case you should focus on identifying and evaluating the overarching central
claim (which is related to the main point of the content).
o In this example: “RVs are unsafe for children, because there is no safe way to install
a car seat in an RV”.
IDENTIFYING CLAIMS BY COMPONENT TYPE
• For Links:
o Sometimes part of a claim may be omitted from a headline or title (e.g., “Why
'getting lost in a book' is so good for you, according to science”), and needs to be
found in the body of the content.
o Sometimes the headline or title of the content is a question (e.g., “Is Austin, TX
getting a new name?”). In these cases, the central claim is the answer to that
question provided by content (e.g., “Austin, TX is not getting a new name”).
o Sometimes the headline or title of the content contains a mixture of fact and opinion
(e.g., “Heartbreaking: baby bear who survived hunting wound cries while he's left to
die”). In these cases, the central claim is the factual portion of the statement (e.g.,
“baby bear who survived hunting wound cries while he's left to die.”).
o Sometimes the headline or title of the content is phrased as an opinion statement or
doesn't contain a factual claim, but the content itself is rooted in a factual central
claim (e.g., an article titled “You should be worried about black mold” that reports
that black mold causes serious respiratory health problems). For these cases, you
should submit the central factual claim that the content is supported by or based on.

For image and video components, there can be claims within the content (e.g. an image contains a
quote, which is attributed to a particular person) and there are claims about content (i.e. that the
image or video is what they claim it is).
• For Images:
o No matter where the claim is located in the content, if the central claim you
identify is a claim about what is being shown in the photo, the claim you should
evaluate has two parts: 1) whether the thing described in the claim actually
happened, and 2) whether the photo is actually showing the thing described in
the claim.
• For Videos:
o Although video content often contain multiple, distinct claims that are statements of
fact, you should identify and evaluate the claim that is most central to the
video content/its main purpose.
§ Usually the central claim in a video is the most prominently displayed
claim in the content — i.e. the claim that a viewer would be most likely to
observe (or to understand that they will be told more about in the video), if
they looked at/watched the content briefly.
§ You should consider any audio, visual, and text-based elements in the video
content together to decide what the central claim is.
o In many cases, the central claim will be directly stated in the video text that
would be immediately visible to viewers (e.g., title, caption, or subtitles).
o In other cases, the central claim will be implied or only partially stated in the
text that would be immediately visible to viewers (e.g., text, title, or subtitles on the
video), with the implication that the viewer should watch the video to see the
full claim or learn more.
o Sometimes the video text states a question, indicating that the viewer should
watch the video to learn the answer to that question.
§ In these cases, the central claim is the answer that the video gives to the
question that is posed in the text.
§ You should look through the video to identify the answer the video gives to
the question, and evaluate that answer as the central claim.
§ For example:
• In this video
( https://www.facebook.com/soccerstories.KeliNetwork/videos/196
3488203716494/), the status message states “Why did Edgar Davids
wear glasses while playing on the pitch?”. A voiceover in the video
(0:54) explains that he wore the glasses due to glaucoma.

§ The central claim you should identify and evaluate in this case is “Edgar
Davids wore glasses while playing football [soccer] due to glaucoma.”
o Sometimes the video text partially states the claim but withhold
details, indicating that the viewer should watch the video to learn the full
claim.
§ In these cases, you should look through the video to identify the full claim
that’s partially stated/implied in the video text.
§ For example:
• In this video
(https://www.facebook.com/NowThisEntertainment/posts/2329954
810399401) the status message states “Jason Momoa shaved his
beard for the first time in 7 years — and it was all for a good cause,”
but the text doesn’t state the specific cause. Captions displayed on the
 video (0:14-0:16) state that he did it to raise awareness about the
benefits of using aluminum over plastic.

§ The central claim you should identify and evaluate in this case is “Jason
Momoa shaved his beard for the first time in 7 years to raise awareness
about the environmental benefits of using aluminum rather than plastic.”

Q2.1: Central Claim:

o What is the central claim in this piece of content?
o REQUIRED [text box]

Q2.1b: Where is the central claim located in this piece of content?

(If multiple parts of the content need to be considered together to
identify the central claim or the central claim is contained in
multiple componenents of the ad, select all options that apply.)
• The Caption/Status Message text above the Video/Photo/Link - The
claim is written in the text that accompanies the media (i.e.,
 in the "Status Message" or caption text that is
displayed separately, above the media).
• Text/Captions Displayed on the Video or Photo Itself - The claim
is written in text that is overlaid on the video/photo itself
(i.e., in captions or a title within the video/photo).
• Spoken Audio in the Video - A speaker in the video says the
claim verbally.
• Visual Imagery or Live Action in the Video or Photo - There is a
claim that is implied in the actions, events, or imagery shown
in the video/photo.
• Link Headline - The claim is in the title or headline of the
link/article (i.e. in the title of the URL preview).
• Link Body Text - The claim is in the body text of the
link/article.
For additional guidelines on identifying central claims in difficult cases, please reference the
Examples Appendix.

Evidence Search

PURPOSE: The purpose of the following step is to find evidence from a trustworthy source that
would allow you to assess whether the central claim you’re evaluating is supported or
contradicted by evidence (or somewhere in between).

Guidelines for the following question:

• We ask you to find evidence for or against the central claim you've identified from a
trustworthy source.
o By evidence, we mean a statement, photo, video, audio, or statistic relevant to the
central claim.
o This evidence should be reported by a different source than the one whose content
you are investigating.
o This evidence can either support the initial claim or go against it.

• To find evidence about the claim, please use the following approaches:
o 1) Try a reverse image search on the image or the video thumbnail image to
locate sources that provide additional information about the video/image and claim
you are evaluating.
§ Please refer to the Appendix on “How to Conduct a Reverse Image Search” for
step-by-step instructions on how to do this (link here). *Note: You must
delete any images saved for reverse image search after you are done
searching. Please save all images to a single folder that you delete
regularly (at least daily).
§ After you have saved the image, go to the reverse image search engine of
your choosing (e.g., https://images.google.com/) and upload the image to
search.
 § After searching, you should click into specific pages in the search results to
see what those sources say about the video/photo/its claim.
§ Finally, select the source with the highest quality evidence to evaluate the
claim in the next steps. *Note: please make sure the source is discussing
the same video/photo as the one you’re evaluating, not another
similar video/image.
§ You should look for sources that provide additional information or
reporting about the video/photo and claim (for example, sources that
describe what is shown in the video/photo, when or where the video/photo
was taken, whether the video/photo has been altered/digitally manipulated,
and/or whether the claim is supported)
§ *Note: You should not use a link to an identical copy of the video/photo
content you are evaluating that includes no additional
information/reporting as an evidence source.

o 2) Try a general web search on a search engine of your choice, using some
keywords from the central claim as your search terms.
§ for example, in the case of the baseball game protestors example we
described earlier:
• you might search: “rick monday protestors flag burn april 25 1976”
using your preferred search engine.
§ After searching, you should click into specific pages in the search results to
see what those sources say about the claim.
§ Finally, select the source with the highest quality evidence to evaluate the
claim in the next steps.
§ *Note: If you typically search using terms in a language that is different
from the content you are evaluating (e.g., you are rating Hindi content
but you typically search using English search terms): If you are
struggling to find evidence, you should try searching using the same
language used in the content.
o 3) Try a search within the website of a trustworthy source (e.g., an authority on
information for topics related to the claim you're evaluating).
§ for example, in the case of the claim “New study finds you can starve cancer
cells to death by cutting sugar from your diet”, you might choose to search
for information from Mayo Clinic.
• You could search “cut sugar starve cancer cells” in the search bar of
the Mayo Clinic's website
• You could also search “cut sugar starve cancer
cells site:mayoclinic.org“ at google.com
• The following search tips may help you to find relevant evidence:
o DO NOT directly copy and paste the caption, overlaid text, or title of the content you
are evaluating into the search engine as your search terms.
§ Directly copying content text as your search term increases your chance of
finding only ineligible evidence (the same piece of content you are
 evaluating, content that is a copy/paste version of the content you are
evaluating).
§ Directly copying content text as your search term may also make you more
likely to only encounter sources that agree with the content you are
evaluating (vs more diverse perspectives or reporting on the claim).
o DO select key terms related to the central claim to enter into the search engine as
your search terms.
§ To locate relevant evidence, try to also include in your search terms the
names of key people, locations, actions and events described in the
content, when possible.
§ NOTE: It is possible that this information may not be directly stated in
the caption or other text on the video/photo. You should also review
the content of the video/photo to try to find names of key people,
locations, or events related to the claim to use as keywords in your
evidence search.
§ For claims about current events, you should also include search terms
related to when the event occurred or the ad content you are evaluating was
published (e.g., the month and/or year) to help locate relevant
evidence. (This information may be found in the text, link, or video itself, or
in the “Post Creation Time” field in the rating tool).
o DO try using synonyms for different key terms if you’re having trouble finding
evidence from sources you trust.
§ For example, if a piece of content uses extreme language in the way it states
its claim, searching using those same extreme terms may make you more
likely to only encounter sources that agree with the content you are
evaluating (vs more diverse perspectives or reporting on the claim).
§ Therefore, when evaluating content that uses extreme language, you should
try searching using more neutral terms.
o DO search for evidence that describes what is being shown in the video/what is the
original context of the video.
§ It is acceptable to use evidence that describes what is shown in the original
video content (e.g., the person, place, event, or context that is shown), even if
the information provided by the evidence does not directly speak to the
specific version of the claim you’re evaluating.
• For example, for a claim “This is a video of flooding in Texas last
week”, you may find an original news article about that same video
that demonstrates that the video actually shows flooding in Florida
two years ago. (i.e., You do not need to find evidence that directly says
“This video is not of Texas” or “This video is of Texas,” as long as you
find evidence of what the video is actually showing).
§ Note: You should always check to be sure the evidence you provide about the
original context/content of a video is discussing the same video you are
evaluating.
o DO check the “images” search results tab for your query if you are having a hard
time determining if search results are relevant from the link previews.
 § Image search results may contain a video preview photo, which will help you
to find an evidence link that discusses the same video you are evaluating.

• Q2.1: Search:
o What are the keywords you used to research this claim? If
you searched multiple times, enter just the keywords you
used on your final/successful search. If you used a reverse
image search, please enter "reverse image search" in the
text box.
o REQUIRED [text box]

For additional guidelines on selecting search terms for photo claims, please see the
Examples Appendix of search term selection for different claims.

Part 3: Evidence Description

Evidence Description

PURPOSE: The purpose of the following questions is for you to describe the evidence that you
found.

Guidelines for the following questions:

• In the following questions, we ask you to describe the highest quality piece of evidence you
found about the central claim in your search. High quality evidence is evidence that is:
o from a trustworthy source (you should try to find evidence from a source you
know is trustworthy, whenever possible). A trustworthy source might be:
§ A source that you are familiar with and that you have found to provide
reliable information in the past
§ A source that is a widely accepted authority on the topic (this may be
based on your prior knowledge or you may choose to do a quick search
engine query for the author or source to see if this is the case)
§ A source that cites other widely accepted authorities or sources you
trust to support their claims
§ A source that generally provides high quality, professional content (this
may be based on your prior knowledge or you may choose to look through
additional content or articles from the source to see if their other content is
generally credible and professionally written)
o provides clear, credible facts/evidence about the central claim
o provides evidence that is relevant to the central claim in the content you are
evaluating, For example:
§ you should check to be sure evidence you use is not describing a different
place, person, or event than the content you are evaluating (e.g., for a claim
warning about a series or crimes in a community, check to see whether the
 evidence speaks to the same community and period of time as the claim you
are evaluating)
§ you should check to be sure evidence you use does not contain facts or
statistics pertaining to a different time period than the content you are
evaluating (e.g., for a central claim about the amount of money a municipal
government has spent in the past month, check to see whether the evidence
source speaks to the same time-period as the claim you are evaluating)
o speaks directly to the same event, time period, or video/image as the original
claim (Unless the evidence you find is showing that the video/image is of a different
time, place or person than described by the claim).
§ You should check to be sure evidence you use is not outdated or about
previous events, especially for claims about current events.

• If you are rating a multi-part claim and you feel that you need different sources of evidence
to support or disprove different parts of the claim, please paste each source into a separate
entry field.
o If you feel that one piece of evidence properly speaks to all parts of the central
claim, you only need to paste one evidence link.

• Finally, the following ARE NOT appropriate evidence sources:

o The same content you are currently evaluating—either from the same source (if the
source is identifiable) or an identical version of the content you are evaluating from
another source (e.g., you may not use a tweet or Facebook post of the same photo
and message you are evaluating as an evidence source). Your evidence must include
additional reporting/information about the ad content and its central claim.
§ However, it is o.k. to use an evidence link that includes the same video/image
you are evaluating, along with additional reporting, facts, or information to
support or contradict the claim, beyond what was in the content you are
evaluating.
o Other content (articles, links, posts, etc.) from the same source as the content you
are evaluating (if the source is identifiable, e.g., from a logo on the photo or URL in
the content text)
§ You may be able to identify the source of the video/image through logos or
watermarks on the video/image.
o The url for your full search engine results (e.g.,
https://www.google.com/search?q=ice+disk+Westbrook+River)
o Social media posts, unless:
§ 1) you are rating a claim about whether a person said something on social
media and you are providing a link to the post referenced in the claim
(however, because people may delete social media posts, a claim shouldn't be
labeled false only because the tweet/post is no longer present)
§ 2) the post is from the official account for a person or organization who is an
authority on the claim's subject (e.g., a post from local emergency
responders' office would be an appropriate source for a claim about a local
emergency in their district)
 § For additional guidelines on social media sources are and are not appropriate,
please see the Examples Appendix to this document.

• Q3.1 Which of the following best describes the highest quality

evidence you found about the claim in your search?
o Note: You should try to find evidence from a source you
know is trustworthy, whenever possible. If you cannot find
evidence from a source that you know is trustworthy, you
should try your best to find evidence from a source that
seems trustworthy, before selecting evidence from a source
that seems untrustworthy or a source you distrust.
o I found evidence from a source that I know and is
trustworthy - I'm familiar with the source and know it
usually provides reliable information.
o I found evidence from a source that I don’t know but seems
trustworthy - I'm not familiar with the source, but I have
researched the source and it seems to be an accepted
authority and/or the source generally provides high
quality, professional content and cites trustworthy
sources.
o I found evidence from a source that I don’t know but seems
untrustworthy - I'm not familiar with the source, but I
have researched the source and it seems to have a negative
reputation and/or the source generally provides low
quality, unprofessional, false, or misleading content.
o I found evidence, but it's from a source that I know and
is untrustworthy - I'm familiar with the source and know it
provides unreliable or false information.
o I did not find any evidence about the claim. I have tried
using reverse image search AND keyword search (including a
search using terms in the same language as the content I'm
evaluating), but I was unable to find any evidence about
the claim.

• Q3.2a Evidence Link:

o Please paste the link for the highest quality evidence you
found. (Paste only the text of the URL/link here. Do not
include additional text from the webpage, article, etc. If
you are evaluating a multi-part claim and found different
evidence sources for different parts of the claim, please
paste each additional source into a separate box.)
o You MUST enter the full URL/link to the evidence you found
(e.g., https://time.com/5305838/anthony-bourdain-dead/) (Do not
enter just the just domain or name of website domain where
you found the evidence e.g., do not enter “time.com” or
“TIME”)
 o REQUIRED - [text box] o
OPTIONAL - [text box] o
OPTIONAL - [text box]

• Q3.3 What evidence related to the claim does this link provide?
o If the evidence you found was an article/written in text,
please copy and paste the relevant evidence from the link
into the box below (evidence statement should be no longer
than 3-5 sentences). If the evidence you found was an
image, video or audio, please summarize the evidence in the
text box.
o REQUIRED - [text box]
o OPTIONAL - [text box]
o OPTIONAL - [text box]

Part 4: Evaluation
Central Claim evaluation

PURPOSE: The purpose of the following questions is for you to make an assessment about the
central claim you are evaluating based on the evidence you described above, and to provide some
context for that decision.

Guidelines for the following questions:

• Considering the evidence you found about the central claim, we would like you to provide
your best judgement about whether the claim is fully supported, partially supported
or partially contradicted, contradicted, or documented as unverified by the evidence
from a trustworthy source
o You should use the category descriptions provided in Q4.1 below to make your
judgment.
• Even if you are not 100% sure about which category fits best, please make your best
judgment based on the most trustworthy evidence you found. You will be given space
at the end of the task to indicate if you are were uncertain about your assessment and to
provide some additional context.
• After you make your evaluation, we will also ask you to indicate if any of the following
apply to the ad content that isn’t Fully Supported:
o The ad content has been taken out of context (the photo/video shows a different
person/place/object/event/time than what is described by the claim)
o The ad content has been digitally manipulated (the photo/video has been digitally
manipulated/altered/morphed to show something different than the original
photo/video).
o This ad content has contradicted or unverified text claims (the text caption, status
message, or overlaid text of the photo/video contains a factually inaccurate claim).
 o The ad’s video contains contradicted or unsupported spoken claims (i.e.- a speaker in
the video says a factually inaccurate claim)
o The ad content is recycled news - The content mis-represents the dates of the
event(s) described in the claim as happening recently (i.e., the content has a recent
publication date or no publication date, and doesn't describe when the event(s)
described in the central claim took place), but evidence shows the event(s) actually
occurred much farther in the past.
o The ad content is a scam - The content is a link to coupon, giveaway, offer, or
promotion that the evidence you found indicates is fake or fraudulent (especially
one that requests that people provide information to receive the offer).
o The ad content has a misleading headline/title - The headline/title of the content you
are evaluating states an exaggerated version of the claim that is not supported by
evidence, but a less extreme version of the claim stated in the body of the content is
supported by the evidence that you found.

• After the final evaluation, there is also a space for you to provide optional additional
information about your evaluation. You can use this space to share:
o issues you are having in general with the rating task
o specific issues you encountered when rating this claim
o additional context for why you did or didn't trust a source (or why it was too
difficult to decide if you should do either)
o any additional information you'd like to share about your evaluation of this claim
o anything else you think would be useful for us to know

• Q4.1 Based on the evidence you found, which of the following

best describes the central claim in this piece of content?
o Fully supported- I found evidence from a trustworthy source
that supports the major aspects of the claim.
o Partially supported or partially contradicted- Only some
aspects of the claim were supported by evidence from a
trustworthy source, while others major aspects were
contradicted or not supported by evidence.
§ This may be because:
• the claim was partially but not fully
contradicted;
• the claim is about something someone said and the
quote is correct, but is taken out of context or
has been significantly misrepresented;
• the claim is represented in significantly
different ways in the title, caption, or other
written text of the link/video/image vs. in the
link/video/image itself you are evaluating, and
not all aspects of the claim are supported by
evidence from a trustworthy source.
o Not supported - There is no support for this claim from
trustworthy sources. I found evidence from one or more
 trustworthy sources that contradict major aspects of the
claim OR that conclude there’s no evidence to support the
claim.
§ If not supported is selected, choose which of the
following option best describes the claim:
§ Contradicted - The claim is fully contradicted by
evidence from one or more trustworthy sources. I found
evidence from a trustworthy source that directly
debunks or disproves the major aspects of the
claim (e.g., you find evidence that shows that the
central claim as stated in
the link/video/image content/its text is factually
inaccurate, the media is taken out of context, or that
the media has been digitally manipulated).
§ Documented as Unverified - A trustworthy source
concludes that there’s no evidence that
directly contradicts the claim, but also that there’s
also no evidence to support the claim.

• Q4.2: [if any option other than Fully Supported selected

for Q4.1]: Based on evidence you found, do any of the following
describe the content you are are evaluating?
o Taken out of Context Photo/Video - the photo/video shows a
different person/place/object/event/time than what is
described by the claim
o Digitally Manipulated Photo/Video - the photo/video has been
digitally manipulated/altered/morphed/edited to show
something different than the original photo/video. For
video, this may include:
§ digitally editing or splicing multiple videos together
to make a factually inaccurate claim
§ voice editing or dubbing to make it appear as though
someone in the video said something they actually have
not (e.g., by editing together un-related
words/statements from that person, adding a voice
over, or speeding up or slowing down audio)
§ visual editing or manipulation to present something
other than what was originally shown in the video
(e.g., by adding or removing visual elements from the
video, or speeding up or slowing down the video)
§ ‘Deepfake’ Videos - realistic computer generated videos,
which use advanced digital manipulation combined with
real video to make it appear as though a person in the
video says and/or does something they actually have
not.
 o Contradicted or Unverified Text Claims - the text caption, overlaid
text, and/or other written text of the content contains a
factually inaccurate claim.
o Contradicted or Unverified Spoken Claims - a speaker in the video
says a factually inaccurate claim
o Recycled News - The content mis-represents the dates of the
event(s) described in the claim as happening recently
(i.e., the content has a recent publication date or no
publication date, and doesn't describe when the event(s) in
the central claim took place), but evidence shows the
event(s) actually occurred much farther in the past.
o Scam - The content is a link to coupon, giveaway, offer, or
promotion that the evidence you found indicates is fake or
fraudulent (especially one that requests that people
provide information to receive the offer).
o Misleading Headline/Title - The headline/title of the content
states an exaggerated or otherwise misleading version of
the claim that is not supported by evidence, but the claim
as stated in the body of the content is supported.
o None of the above apply
§ [Optional] - You may use this box to enter a brief
description (no more than 3-4 words) about why the
claim wasn't Fully Supported.

Part 5: Additional Information

PURPOSE: The purpose of the following questions is to provide you with space to provide us with
additional information about your evaluation of this content. We will use this information to
better understand what parts of the task are difficult and to identify future improvements.

Guidelines for the following questions:

• In the central claim evaluation above, we ask that you provide your best judgement about
each claim, even if you are not 100% sure about which which of the category in Q4.1 fits
best.
• In this section, we would like you to indicate if you were very uncertain about an item, and
to provide us with a bit of information about why that item was difficult to evaluate.

• At the end of the task, there is also a space for you to provide optional additional
information about your evaluation. You can use this space to share:
o additional information about why an item was difficult to evaluate or you were
uncertain about an evaluation
o issues you are having in general with the rating task
o additional context for why you did or didn't trust a source (or why it was too
difficult to decide whether you should trust the source)
o any additional information you'd like to share about your evaluation of this claim
 o anything else you think would be useful for us to know

• Q5.1a: OPTIONAL: Select this button below if this item was very
difficult to evaluate, and you believe it may need further
review.
o [check box] Item may need further review

• Q5.1b: OPTIONAL: Select the option(s) below that best describes

why this item was difficult to evaluate. (Select all that
apply)
o I needed more time - I needed more time to search
for/evaluate evidence
o It was difficult to identify one central claim - It was
difficult to identify the central claim(s), among multiple
possible claims
o I’m not sure if the central claim was a statement of
fact- I’m not sure if this item should have been rated or
disqualified for not containing any central statement(s) of
fact
o I had a difficult time choosing a final evaluation
category - I had a difficult time deciding between two
final evaluation categories (i.e., fully supported,
partially supported/contradicted, documented unverified)
o The central claim was about a technical subject matter
- The subject matter of this claim is very technical or
unfamiliar to me; specialized knowledge may be needed to
evaluate this claim
o The evidence I found was weak or inconclusive - Evidence
was weak or inconclusive. This may be because: I found
evidence both for AND against the claim, I am not sure if
the evidence I found is trustworthy, I am not sure if the
evidence I found is eligible, I am not sure if the evidence
I found is relevant
o I identified a central claim, but could not review all ad
components - I identified a central claim in one of the ad
components, but another component was inaccessible for
review (for example, link component was behind a paywall or
the image was broken.)
o Something else (please describe below)

• Q5.2: OPTIONAL: Provide any other relevant details about your

evaluation that you'd like to share:
o [text box]