Guiding Examples - Understanding the

Credibility of Information
Table of Contents
Rating UI Overview 2
Task A Queues 2
Task B Queues 8
Starting Point Examples 15
What is the main purpose? 15
P-I-E Persuade-Inform-Entertain 15
Main purpose examples 16
Does This Content Have a Central Claim? 18
What kind of jobs get the “No, the content does not have a central claim label” 20
Ads, recipes, quizzes, and events 20
Jokes and funny memes/videos/articles that do not make factual claims 22
Opinion only posts that do not make factual claims 23
More Notes on Satire and Opinion 25
Satire 25
Examples of the ‘likely to mislead’ label when there is a claim 25
Reminder about what is NOT satire 27
Opinion labeling 27
Important Notes about Identifying the Central Claim 30
Multiple claim examples: 30
Reminder about selecting the most serious claim 31
Keyword and Evidence Search Labeling 32
Notes about the authenticity of the photo or video as part of the claim 33
Tip: For many contents that include media, always consider whether the best central claim is that the media shows real
people, events, or actions. 34
Claim Location Labeling 35
Examples of the “Which parts of the media contribute to the central claim” label. 36
Evaluating Trustworthiness Walkthrough 37
Evaluation Section 38
Example- Fully Supported 39
Examples – Partly Supported-Partly Contradicted 40
Example Partly Supported- Not Contradicted 43
Example Not Supported- Fully Contradicted 45
Example Not Supported -Not Contradicted 47
Difficult Job Types 49
Page | 1
 Political accusations/opinion 49
Twitter posts 50
Sales posts with and without claims 51

This document provides a number of guiding examples to help illustrate the task instructions.

Please note that while this document presents many images examples, the same broad logic applies
to any content type. Photos tend to be easier to present on a document.

Happy rating!

Rating UI Overview

Task A Queues


Page | 2
Please note that the job is labeled as No-This content does not contain a central claim or Cannot Determine, SRT
automatically moves to the optional space to provide further information and the job is submitted and does not flow into
Task B queues for further evaluation.

Next is the claim impact section

Page | 3
Page | 4
Now, you are asked some further questions about content type.

Page | 5
If you answer Yes- The content is satire or parody, you are then asked which type

If your answer to the satire or parody question was No or after you identify which type in the above, you are then asked
about opinion.

Page | 6
Next is an optional space to provide any notes to pass along

Finally, the job is submitted.

Page | 7
Task B Queues
 Task B Queues automatically open to the question asking -"What is the content’s central claim”?

Plese note that you can navigate back to the first question if necessary by simply clicking on it, which brings the UI back to
the prior question.

Page | 8
Then, after entering the answer with the identified central claim (as the first image demonstrates), the labeling continues.

Page | 9
If you indicate that at least part of the claim is located in the media, the following question opens

Please note that if you answer No to the above question, you are then able to submit the job. However, if you did find
evidence, the rating continues.

Page | 10
Please note that if you indicate the source DID NOT seem trustworthy, the wrap up questions regarding details that made
the job difficult to review open and the evaluation stops.

However, if you indicated the source seems trustworthy, the labeling continues.

Page | 11
Then comes the details about the final evaluation.

If the content is labeled as fully supported, the optional wrap up questions open, but if partly supported or not supported is
indicated, further questions that collect more details open.

Or

Page | 12
Finally, Partly Supported or Not Supported labeling asks if any of the following scenarios apply.

Regardless of the point in labeling that completes the job, before the final submission, you will be given a chance to indicate
that the job was difficult to review.

Page | 13
If you answer yes, you are asked for further details.

You also have the option to add any notes with the final job submission.

Page | 14
Starting Point Examples
What is the main purpose?

** The Main Purpose of the content is the main thing the content is trying to get across to the
audience. Is it trying to persuade the audience to agree with an opinion or to buy a product? Is it
trying to inform the audience of some idea, event, or thing? ? Is it trying to entertain us with a
joke or satire? Usually content is trying to achieve one (or a combination) of those three purposes.

P-I-E Persuade-Inform-Entertain

Page | 15
While we sometimes find a central claim based on a statement of fact in entertainment/persuasive
content, there must also be an element of “to inform”. Thus, if the content’s main purpose is to
persuade us to agree with an opinion, it often also informs us of something to prompt that persuasion.
Same concept with content meant to entertain: does it inform us of something that can be researched
and shown to be supported or not supported to set up the entertainment?

**Content that has the purpose of informing us is suitable for our evaluation. The caveat for content
meant to persuade or entertain is that it should also be informing us of something related to its main
purpose.

Main purpose examples

Page | 16
Page | 17
Does This Content Have a Central Claim?

**Tip Remember, the central claim you identify should ALWAYS be relevant to the
main purpose of the post.

Page | 18
Page | 19
What kind of jobs get the “No, the content does not have a central claim label”

Ads, recipes, quizzes, and events

Page | 20
Values statements and wishes
Sometimes posts are just sentimental, feel good, or general life advice statements, or perhaps a wish
to have a nice day or a happy birthday. These types of posts typically fall somewhere in between
intents to persuade (express an opinion) or entertain (by inspiring feelings of goodwill) but unless there
is an element of informing us of something related to their core main purpose or intent that can be
evaluated as factually supported or not supported (or somewhere in between) these can safely be
labeled as No Claim.

**Tip Be sure to check the status update with these images to make sure there is not a
claim.

Page | 21
Jokes and funny memes/videos/articles that do not make factual claims

Jokes and other types of humorous content are often posted with an intent/main purpose to entertain.
While these still do need to be checked for any inclusion of an intent to inform with a fact that can be
evaluated, most likely posts exclusively meant to entertain will be labeled as no claim.

**Tip Remember that humorous content meant to entertain is sometimes sarcastic or

uses ridicule, but, posts that make no claims at all, even an exaggerated or seemingly
satirical one will get the no claim label, which immediately ends the labeling, and NOT
one of the satire labels further down on the rating UI.

Page | 22


Opinion only posts that do not make factual claims

Sometimes posts contain opinion only (intent to persuade) without including any factual statements
relating to the main purpose that can be evaluated as supported or not supported.

**Tip Remember that ONLY type of opinion posts that get the Yes, it is opinion label
are those that have some factual claims to back the opinion, otherwise, the job will have
already been submitted when you selected the “No, this content does not have a central
claim” label.

Page | 23
Page | 24
More Notes on Satire and Opinion
Satire

Examples of the ‘likely to mislead’ label when there is a claim

Satire often has the intent to entertain, making the main purpose humor; however, it is important to
note that sometimes something that turns out to be satirical (you learn after research) is probably
Page | 25
appropriate to label as Yes, but not obvious and likely to mislead. Or, maybe you did recognize it as
satire right away because you were familiar with the site from previous experience with it; or maybe
you just are a wiz at picking up on satire. Whichever the case, if the sarcastically funny or absurdly
exaggerated post is satire and still has an intent of informing with a factual claim mixed in with the
intent to entertain, then it should be evaluated.

Page | 26
Reminder about what is NOT satire

**Tip Remember that just because a post is funny or seems sarcastic, it does not
mean it is always satire or parody.

Opinion labeling
Opinion posts often have the intent to persuade. Those that ONLY set out to persuade us to the
poster’s way of thinking and that do not have an intent to inform us to sway our opinions are
submitted before this point in the labeling, so the types of posts we must choose between the yes, this
is opinion and the no, it does not appear to be opinion are those that have relevant factual claims that
Page | 27
attempt to INFORM us to persuade us. In other words, there must be factual claims that we can
evaluate as supported or not supported related to the main purpose of the post.

**Tip Remember that ONLY type of opinion posts that get the Yes, it is opinion label
are those that have some factual claims to support the opinion, otherwise, the job will
be submitted when you selected the “No, this content does not have a central claim”
label.

Page | 28
**Tip Keep in mind that it is not only politics or social issues that are appropriate for
labeling as opinion.



Page | 29
Important Notes about Identifying the Central Claim
The claim you select should be related to the main purpose of the content. So, the way to approach
claim identification is from the perspective of the purpose of the content, or, more simply put, what is
the content mainly trying to tell us?

We will go through a few challenging examples where it may be tough to identify a claim. Please note
that our goal is always to give guidance on potential approaches to jobs, but there is always an
intended degree of subjectivity in this project, and often, there are multiple appropriate approaches
to the same job.

Multiple claim examples:

Identifying the claim(s) should be approached by determining the main statement of fact, whether true
or false, by pinpointing the overarching or big point (Main Purpose) of the content. If there is more
than one claim, can they be condensed into one single claim?

Page | 30
Reminder about selecting the most serious claim

**Tip Remember that when we suggest considering the most serious claim as the
central claim, it still MUST relate to the main purpose of the post.

Page | 31
Keyword and Evidence Search Labeling

Page | 32
Notes about the authenticity of the photo or video as part of the claim

Page | 33
Tip: For many contents that include media, always consider whether the best
central claim is that the media shows real people, events, or actions.
One example is this video post that showed U.S. House Speaker Nancy Pelosi speaking. It’s possible to interpret the post
text two ways:
1. It’s an opinion content not based on a statement of fact (opinion is that the post author believes Speaker Pelosi’s
comments are so wild they think she is “out of her mind”). While this is understandable, it’s not the best approach
for this content.
Page | 34
 2. The best option is considering that the post contains media and the author says the media (video) is accurately
showing that Pelosi is “blowed out of her mind” (intoxicated). Note: You should ALWAYS consider, when a post
contains an image or video, whether the claim might be that the media accurately represents people, actions, or
events as they exist/have occurred.
When taking the second approach for this job, you may find this article from Reuters fact-check
(https://www.reuters.com/article/uk-factcheck-nancypelosi-manipulated/fact-check-drunk-nancy-pelosi-video-is-
manipulated-idUSKCN24Z2BI) about this content’s claim. If Reuters is a source you trust, you can then rate the claim “Fully
Contradicted” and label the content as “Altered Media”.

Claim Location Labeling

Page | 35
Examples of the “Which parts of the media contribute to the central claim”
label.

Page | 36
Evaluating Trustworthiness Walkthrough

Page | 37
Evaluation Section
In this section we will work through a few jobs to show possible scenarios of the final evaluation labels.

Page | 38
Example- Fully Supported

The main purpose of this content is to share who Dennis Kucinich has endorsed in the US Democratic
Primaries for the 2020 election.

After reviewing the text, image, and video components, you decide the central claim is that Dennis
Kucinich supports Tulsi Gabbard as the 2020 Democratic Nominee for President.

To find evidence, you enter the search terms “dennis kucinich endorsement 2020 election” in a search
engine.

The Hill might be a source that you trust, and one link from The Hill supports the claim that Kucinich
has endorsed Tulsi Gabbard as the 2020 Democratic Nominee.
Page | 39
Because the Hill is a source you trust and it provides supporting evidence for the claim, you decide to
rate the claim as Supported.

Examples – Partly Supported-Partly Contradicted

Page | 40
The main purpose of this content is to inform us that a study showed eating fermented foods lowers
Covid death rates.

After reviewing the text post by author and the article, you decide the central claim is that a study
shows that eating fermented foods lowered deaths in South Korea from Covid-19.

To find evidence, you enter the search terms “fermented foods lower Covid deaths study” in a search
engine.

Perhaps you think Lead Stories is a trustworthy site, and they provide a relevant fact check for the
claim you identified.

Page | 41
You find the following from reading the article-
Main aspects of the claim are that a scientific study suggested kimchi reduced South Korea’s total
COVID-19 deaths. This article supports two main aspects (that kimchi was associated with reduced
COVID19 deaths), contradicts one (the study actually showed this in Europe, not South Korea) and adds
missing context on one (the study was pre-print).

So, if you do find Lead Stories to be a trustworthy source, an appropriate label is Partly Supported and
Partly Contradicted.

Page | 42
Example Partly Supported- Not Contradicted

The main purpose of this content is to inform us that California has used contact tracing in the past to
combat the spread of diseases like tuberculosis and measles.

After reviewing the text post by author and viewing the video, you decide the central claim is
California has used contact-tracing in the past to combat the spread of infectious illnesses, including
tuberculosis and measles.

To find evidence, you enter the search terms “California contact tracing to fight tuberculosis and
measles” in a search engine.

In this case, you might find that a relevant article in a medical journal publication is the most
trustworthy source available, which you find talks about the state using the tracing for measles, but
there is no mention of tuberculosis.

Page | 43
Some people might not consider “tuberculosis and measles” main aspects, and that’s acceptable; in
that case you would rate this as Supported.

However, if you believe that the tuberculosis and measles examples are important aspects, and the
evidence only supports the measles assertion (and in this case does not discuss tuberculosis), you may
rate this as Partly supported and Not contradicted.
Please note- In this case, the content is “partly supported, not contradicted” because it’s partly
supported, but it is not partly unverified - it does not explicitly discuss tuberculosis and/or say there is
no available evidence about tuberculosis.

Page | 44
Example Not Supported- Fully Contradicted

The main purpose of this content is to inform us people are leaving bricks around protest sites to incite
people to use them.

After reviewing the content, you decide the central claim is that Piles of bricks are being left near
protests to bait people into causing chaos and destruction.

To find evidence, you enter the search terms “People planting piles of bricks at protest sites” in a
search engine.

You might find Snopes to be a trustworthy source and there is a relevant fact check available on the
site.

Page | 45
The main aspects of the claim are that bricks are being left intentionally near protests to encourage
protesters to throw them or to cause general violence and chaos. The evidence article says that they
found no evidence of staging. Instead, in many cases, the bricks were delivered for construction
projects. In this case, appropriate labeling would be Not supported and Fully contradicted.

Page | 46
Example Not Supported -Not Contradicted

The main purpose of this content is to inform us that people who have already had Covid can get
infected with it again.

You may decide here that the central claim should be “People can catch Covid again after recovering
from it the first time”.

To find evidence, you enter the search terms “Can you catch Covid more than once”?

You might find Snopes to be a trustworthy source and there is a relevant fact check available on the
site. Alternately, there is also information on the claim from the CDC.

Page | 47
The claim explicitly states that it is possible to become infected with Covid more than once and does
not qualify that with language like “it might be possible”. The evidence articles from the CDC and
Snopes explicitly state there is no evidence to prove this or say that the claim is supported, but neither
site can debunk the claim and say that it is not a true statement. The claim is explicitly stated on both
accounts to be unconfirmed, unproven or unverified. Since, the claim is not supported, nor even
partially supported by the evidence, but there is also documentation that there is nothing to
contradict the claim, in this case, appropriate labeling would be Not supported and Not Contradicted.

Please note in this type of case, an additional SRT question will open before you can submit the job and
this example would also be appropriately labeled as “Explicitly unverified”.

Page | 48
Difficult Job Types
Political accusations/opinion

Page | 49
Twitter posts

It is fairly commonplace to find results of Twitter posts in these workflows, the following is a quick
guide on how to approach these types of results (including when to consider the authenticity of the
tweet part of the claim).

Page | 50
Sales posts with and without claims

Page | 51