Uolo | Guideline Snapshots – Task A, Task B, Custom Search

Purpose of Uolo

To develop a system that identifies and matches evidence to Misinformation, the rating process is split
into 2 tasks, Task A and Task B. Task A is identifying if the job has a claim, Task B is where you state the
claim, find evidence relating to the claim and evaluate the truthfulness of the claim.

Task A is the first step of the rating process where you make key decisions about the content such as
determining the main purpose of the post, is it satire, and should it be passed on to Task B.

Task A
In Task A, you will be given a video, picture, post, text, link, etc. and provided the following answer
choices:

 Yes – This content has a central claim

 Yes, but obviously satire or parody - This satire or parody content is not likely to mislead most
people.
o NOTE: This answer choice is not available on all workflows – don’t be alarmed if you
do not have this answer choice
 No – This content does not have a central claim
 Cannot Determine

What is a central claim? The central claim is a statement of fact relating to the content’s main point or
purpose that can be supported or contradicted (or somewhere in between) using an evidence source.
Some jobs will only have one statement of fact being made, while others might have multiple.

In the context of determining if there is a claim, we are speaking of the main purpose in a broad and
overall sense, referring to a broad category of posting:

 Is the main purpose to inform the reader with a fact or facts?

 Is it to persuade the reader (to agree with an opinion, purchase a product, sign a petition, etc.)?
 Is it to entertain the reader (a joke, a well wishes post, etc.)?

The post has a central claim when:

 The main purpose of the post is strictly to inform the reader of a fact or facts that can be
evaluated.
 The content is opinion/persuasion based on facts.
 The entertainment is based on facts.

Selecting a Response
In Task A, you will be given a video, picture, post, text, link, etc. and provided the following answer
choices:

 Yes – This content has a central claim

 Yes, but obviously satire or parody - This satire or parody content is not likely to mislead most
people.
o NOTE: This answer choice is not available on all workflows – don’t be alarmed if you
do not have this answer choice
 No – This content does not have a central claim
 Cannot Determine

If you choose Yes – This content has a central claim, you will be given a follow-up question, “If this
claim turns out to be wrong, would it impact someone's:” followed by 6 choices. Please note that you
are able to select all choice that apply to the content you are reviewing.

 Health, safety, or well-being

 Views of other groups (race, ideology, nationality, etc.)
 Political beliefs or behavior
 Views of institutions (government, news media, etc.)
 Something else
 No Impact

If you choose, Yes, but obviously satire or parody - This satire or parody content is not likely to mislead
most people., you will be given the following choices:

 Yes - This item may need further review. After choosing this option, you will be given the
following question, What made it difficult to complete this job?:
o I needed more time.
o It was difficult to identify just one central claim.
o I’m not sure if the central claim was based on a statement of fact.
o I had a difficult time deciding if the central claim was supported or contradicted.
o The central claim was about a complex topic.
o The evidence I found was weak or inconclusive.
o I identified a central claim but could not review all components.
o Something else: [Enter your explanation in the free text box]
 No - This item was not difficult to review. After choosing this option, you will be given a free
text box where you will manually enter your response

If you choose No – This content does not have a central claim, you will be given a follow-up question,
“Please provide any other information you’d like to share:” followed by a free text box where you will
manually enter your response.

In some workflows, if you select, No – This content does not have a central claim, you will be given the
following question, Was this content difficult to review?:
  I needed more time.
 It was difficult to identify just one central claim.
 I’m not sure if the central claim was based on a statement of fact.
 I had a difficult time deciding if the central claim was supported or contradicted.
 The central claim was about a complex topic.
 The evidence I found was weak or inconclusive.
 I identified a central claim but could not review all components.
 Something else: [Enter your explanation in the free text box]

When might the content NOT have a central claim:

 The content does not contain a central statement of fact that relates to the MAIN PURPOSE of
the post that can be supported or contradicted (or somewhere in between) by evidence.
 This may be because:
o The main purpose of the content is solely to convey an opinion or values statement and
it does not contain any statements of facts as rationale/support for the opinion/value
statement; or
o The main purpose of the content is not related to conveying a statement of fact.
 An entertainment example:
 The post is an entertaining video of a dog playing fetch during the 2020
quarantine for Covid and the caption is “he sure misses when I was
home playing fetch all day during the Covid quarantine.” – In this case,
the main purpose of the post is to entertain you, not inform you there
was a Covid quarantine in 2020. It would be marked as having no central
claim.
 A persuasion/opinion example:
 The country has been going downhill for decades now. Every election,
each administration gets worse and worse. I am convinced we have not
had a good president since Teddy Roosevelt in the early years of the
20th century. - Here, the main purpose is not to inform that Roosevelt
was the president in the early 1900s. This post would also be labeled as
no central claim.

If you choose, Cannot Determine, you will be given a follow-up question, “Why are you unable to
determine if this content has a central claim?” followed by 4 choices:

 SRT Technical Issue

 Content Inaccessible
 Why are you unable to determine if this content has a central claim?
 Some other malfunction

After choosing your reason, you will be given a free text box to give a brief explanation of the issue.
Task B

Task B The second step of the rating process where you identify and select or write the claim, find
evidence, assess the quality of the evidence, and make the final evaluation.

Selecting a Response
The first question in Task B is, What is the content’s central claim?, followed by a free text box where
you will manually type your response.

 **Please use the guidance outlined in the project guidelines to formulate your response.

Once you have entered your response and select, “Continue”, you are given the next question, “Where
is the central claim located in this content?”, and the following answer choices:

 Text from post author only. After selecting this choice, you are given the following question:
o Did you find evidence about the claim in your search?
 Please see below for more information about this question**
 Media only
 Text from post author and media separately
 Text from post author and media together

The 3 labels above all lead to the same question, “Which parts of the media contribute to a central
claim?” with the following choices:

 Which parts of the media contribute to the central claim?

 Link – Headline
 Link - Body text on article or landing page
 Photo - Visuals, actions, or events depicted
 Photo - Text inside image
 Video - Visuals, actions, or events depicted
 Video - Text inside video
 Video – Audio
  Other - Other media component
 Please describe the media part(s) in a few words: [Free text response]
o Did you find evidence about the claim in your search?
 Please see below for more information about this
question**

Did you find evidence about the claim in your search?

Once you reach the question, “Did you find evidence about the claim in your search?”, the following
questions and answer choices are the same regardless of the preceding question.

“Did you find evidence about the claim in your search?”,

o Yes – I found evidence about the claim.

o No – I was not able to find any evidence about this claim

--

Yes – I found evidence about the claim.

 “Please enter the URL of the highest-quality evidence you found:” Then enter your URL into
the free text box. Next question is:
o Is this evidence from a source that you know?
 Yes, I am familiar with this source. Next question:
 Does this evidence come from a source that seems trustworthy?
o Yes – this source seems trustworthy
 What evidence related to the claim does the link
provide? [Enter free text response]. Next question:
 Based on the evidence you found, is the
central claim:
o Fully supported
o Partly Supported
o Not Supported

At this point in the decision tree, selecting Partly Supported and Not Supported leads to an extra question not included
when you select Fully Supported. We’ll cover that question first.

o Based on the same evidence, is the “partially/not supported” claim also:

 Partially contradicted
 Not contradicted

Both selections above all lead to the following questions and choices:

o Based on the evidence you found, do any of the following describe this content?
 Misleading headline
 Missing context
 Altered media
 Explicitly unverified
 Scam
 None of these apply
  Was this content difficult to review?
o Yes – this item may need further review
 What made it difficult to complete
this job?
 I needed more time.
 It was difficult to identify just
one central claim.
 I’m not sure if the central claim
was based on a statement of
fact.
 I had a difficult time deciding if
the central claim was supported
or contradicted.
 The central claim was about a
complex topic.
 The evidence I found was weak
or inconclusive.
 I identified a central claim but
could not review all
components.
 Something else: [Free Text]
o No – this item was not difficult to
review
 Please provide any other
information you’d like to
share: [Free Text]

o No – I am not familiar with this source. Next question:

 Was this content difficult to review?
 Yes – This item may need further review. Next question:
o What made it difficult to complete this job?
 I needed more time.
 It was difficult to identify just one central claim.
 I’m not sure if the central claim was based on a
statement of fact.
 I had a difficult time deciding if the central claim was
supported or contradicted.
  The central claim was about a complex topic.
 The evidence I found was weak or inconclusive.
 I identified a central claim but could not review all
components.
 Something else: [Free text response]
 No – This item was not difficult to review
o Next Question, Please provide any other information you’d like
to share: [Free text response]

No – I was unable to find any evidence about the claim

 Was this content difficult to review?

o Yes – this item may need further review
 What made it difficult to complete this job?
 I needed more time.
 It was difficult to identify just one central claim.
 I’m not sure if the central claim was based on a statement of fact.
 I had a difficult time deciding if the central claim was supported or
contradicted.
 The central claim was about a complex topic.
 The evidence I found was weak or inconclusive.
 I identified a central claim but could not review all components.
 Something else: [Free Text]
o No – this item was not difficult to review
 Please provide any other information you’d like to share: [Free Text]

Custom Search

Part A

Disqualifying Issues - Are you able to review this item?

 Yes - There is a central claim based on a statement of fact.

o To what degree would it matter to someone's livelihood, decision-making, safety,
political and social worldview, or well-being if the information in this central claim
was wrong or misleading?
 Not a lot
 Somewhat
 A great deal
 **The three labels above all lead to the following question:

 Is this content satire or parody?

o Yes, and **it is clear or obvious** that the main purpose is
satire or parody. Most people would be able to tell that the
content is meant to entertain and not to convey serious
information or reporting.
o Yes, but it is **not obvious** that the main purpose is satire or
parody. Some people may mistake the content for serious
information or reporting.
o No, it does not appear to be or is definitely not satire or parody.
o Unable to Determine

**The labels above all lead to the following question:

 Is this content opinion?
 **Yes**, it is opinion, and it is an "op-ed" or
editorial article.
 **Yes**, it is opinion, and it is a blog post,
social media post, or personal webpage
expressing a personal opinion.
 **Yes**, it is opinion, but it is NOT an “op-ed”,
editorial article, blog post, social media post, or
personal webpage.
 **No**, it does not appear to be opinion.
 Unable to determine
**All responses above lead to the following question:
 How long ago was the linked content
published?
o Less than 3 months ago
o Over 3 months, but less than 1 year
ago
o Over 1 year ago
o The link/article does not contain a
publication date
o N/A - The content does not contain
a link/article
 OPTIONAL: Provide any
other relevant details about
your evaluation that you'd
like to share: [Free Text
Response]
 No - I accessed all content parts and there was no central claim.
 No - I can't review due to an SRT technical issue.
 No - I'm not assigned to rate in this language.
  No - Link issues block some content - what's visible has no central claim.

**All responses above lead to the following option:

o OPTIONAL: Provide any other relevant details about your evaluation that you'd like to
share: [Free text]
 Submit your Task (then you are moved to the next task)

Part B

Disqualifying Issues - Are you able to review this item? Note: Items in this queue have been identified
as very likely rate-able, so you should very rarely need to disqualify these contents!

 Yes - There is a central claim based on a statement of fact.

o What is the central claim you have identified from the original piece of content? [Free
text]
 Where is the central claim located in this piece of content?
 Post text only
**Please see below for more about the following question:
o What are the keywords you used to research this claim? If you
searched multiple times, enter just the keywords you used on
your final/successful search. [Free Text]
 Media only
 Post Text and Media, Each
 Post Text and Media, Together

The 3 labels above all lead to the same question, “Which parts of the media
contribute to a central claim?” with the following choices:

 Which parts of the media contribute to the central claim?

o Link – Headline
o Link - Body text on article or landing page
o Photo - Visuals, actions, or events depicted
o Photo – Overlaid text
o Video - Visuals, actions, or events depicted
o Video – Overlaid text
o Video - Audio or Subtitles (Closed Captioning)
o Other - Other media component

**Please see below for more about the following question:

 What are the keywords you used to research this

claim? If you searched multiple times, enter just the
keywords you used on your final/successful search.
[Free Text]

 No - I accessed all content parts and there was no central claim.

  No - I can't review due to an SRT technical issue.
 No - I'm not assigned to rate in this language.
 No - Link issues block some content - what's visible has no central claim.
o OPTIONAL: Provide any other relevant details about your evaluation that you'd like to
share:

Once you reach the question, “What are the keywords you used to research this claim?”, the remaining
questions and label choices are the same

 Which of the following best describes the evidence you found about the claim in your search?
o I **found at least one fact-check article** about the content claim.
 Evidence Link: Please paste the link for the most relevant fact-check article you
found. [Enter Link]
 Evidence Article Title: What is the title of the fact-check article? [Enter Title]
 Evidence Claim: What claim does the fact-check article evaluate? [Enter Claim]
 Evidence Rating: Please copy and paste the evaluation rating the fact-check
article gave (e.g. “true”, “mostly false”, "pants on fire", etc.).
 Based on the fact-check article you found, what would this fact-
checker say about this content’s central claim? Remember that the
content’s central claim may vary from the claim evaluated in the fact-
check article.
o True or Supported
o Misleading, Mixture, or Partly False/True
o False or Contradicted
 Based on evidence you found, do any of the following
describe the content you are evaluating? (Select all
that apply)
 Taken out of Context Video/Photo
 Digitally Manipulated Video/Photo
 Contradicted or Unsupported Text Claim
 Contradicted or Unsupported Spoken Claims
 Recycled News
 Scam
 Misleading Headline/Title
 Other
o Unsupported, Unverified, or Undocumented
o Unsupported, Unverified, or Undocumented
o No Conclusion or Unknown
o Other
o I **did not find any evidence from a fact-checker** about the content claim.
 The search provided results, but there were no relevant fact-check articles
 OPTIONAL: Provide any other relevant details about your evaluation
that you'd like to share: [Free Text]
 The search provided 0 results

 OPTIONAL: Provide any other relevant details about your evaluation
that you'd like to share: