Zeya Images Guidelines: Understanding the Credibility of Information

Table of Contents
Topic Page
Project Overview 1
Disqualifying Issues 2
Part 1: Context 3
Part 2: Claim Identification and Evidence Search 5
Part 3: Evidence Description 6
Part 4: Evaluation 8

Project Overview:

In this task you will be asked to research and evaluate claims stated in photo content.

Why are we doing this?

We are doing this to better understand the information shared on Facebook, in terms of how individuals
assess its credibility after doing some research.

Big points to keep in mind:

This document contains ever-evolving and ever-improving guidance on how to think about the task's
questions and what the response options mean for evaluating photo content.

This isn't intended to be a strict set of guidelines, because this isn't intended to be a purely objective
task. There will be some variability between people in how they approach the task and how they
respond to the questions, and that is by design. However, try as much as possible to follow the
instructions laid out below and to answer each question to the best of your ability.

Starting Point:

This set of tasks will ask you to evaluate the central claim presented in a piece of content. A central
claim is a factual statement related to the content's main point or purpose that is falsifiable. By
falsifiable, we mean that you could assess the claim to be “true” or “false” (or somewhere in between)
based on evidence.

Example

Photos show a giant floating ice photographed in Presumpscot River in Westbrook.

1
Zeya Images Guidelines: Understanding the Credibility of Information

For example, for the content above you should focus on evaluating whether there is evidence that
indicates that these photos are actually photos of the giant ice disk described in the text of the status
update (e.g., rather than being photos of another place or thing). The claim is not only whether there
was “a giant ice disk is floating in the Presumpscot River Westbrook” as stated in the caption.

Disqualifying Issues

There are two reasons why photo content is disqualified from being rated:

• It was not possible to access the content in SRT. This may be because: o the SRT preview is
broken and the photo or caption does not render properly o the content is in a different
language from what you are assigned to review
• The content does not contain a falsifiable central claim (i.e., the content does not contain a
central claim that can be assessed as “true” or “false” [or somewhere in between] based on
evidence.)
• Below are some examples of photo content types that often do not include a falsifiable claim.
*However, if you are able to identify a factual central claim in an item of these types, then you
should complete the rating task for that item rather than marking it as disqualifying § quizzes §

2
Zeya Images Guidelines: Understanding the Credibility of Information

event or sales flyers (e.g., an ad for a car dealership sales events posted on that dealer's
website, a sales flyer for a department store sale on that store's website, a promotional poster
for stating the date and time of a show) § song lyrics § recipes § humorous memes or joke posts
that do not make statements about specific people, entities, or facts/statistics

Part 1: Context

POTENTIAL IMPACT OF THE CLAIM

PURPOSE: The purpose of the following question is to gauge how much it would matter if the central
claim turned out to be false.

Guidance for the following question:

• To answer the following question, you should consider all components of the piece of content
you are evaluating, including the photo itself, any text/caption in the photo, and the content
caption (displayed above the photo in the review tool, if present).
• To answer this question, you may consider the potential impacts on the general public or
specific audiences who see the content, as well as on the individuals, groups, or organizations
described in the content.
• If you are rating a multi-part claim, you should provide the impact rating for part of the claim
that seems to matter the most (i.e., the highest impact claim)

Question 1.1 To what degree would it matter to someone's livelihood, decision-making, safety,
political and social worldview, or well-being if the information in this central claim was wrong or
misleading?

• Not a lot - It would not matter if the information were wrong or misleading.
• Somewhat - It may matter if the information were wrong or misleading. o e.g., sports, local
events, celebrity gossip or entertainment related content.
• A Great Deal - It would matter if the information were wrong or misleading. It could impact
the lives of those who see it or of individuals described in the claim.
o e.g., medical, legal, financial, political, commercial, or other topics where being
wrong might matter.
o Content containing information, facts, or statistics that are important for fostering
an informed group of people, community, or nation. This could include information
related to government processes, natural disasters, international relations, etc.

CONTENT TYPE

PURPOSE: The purpose of the following two questions is to provide context of the type of content you're
evaluating.

Guidance for the following two questions:

3
Zeya Images Guidelines: Understanding the Credibility of Information

• We ask you to tell us whether the content you are evaluating is Parody or Satire:
o Parody refers to humorous imitations of serious literature, media, etc.
o Satire refers to humorous content that uses irony, sarcasm, or ridicule to expose,
denounce, or make fun of human stupidity, immorality, and/or shortcomings.
• We also ask whether the content you are evaluating is Opinion:
o Opinion here means someone's personal opinion as written in the photo caption or text.
o However, some content may contain both opinion and factual statements (e.g., content
may contain a factual central claim to support an opinion.)
o Therefore, you will still be asked to identify and evaluate any central claims present for
content labeled as opinion

Question 1.2 Would a reasonable person conclude that the linked content is satire or parody?

• Yes, it is clear or obvious that the main purpose is satire or parody

• Yes, but it is not obvious that the main purpose is satire or parody
• No, it does not appear to be or is definitely not satire or parody
• Unable to determine.

Question 1.3 Would a reasonable person conclude that the linked content is opinion?

• Yes, it is opinion
• No, it does not appear to be opinion
• Unable to determine

Part 2: Claim Identification and Evidence Search

CLAIM IDENTIFICATION

PURPOSE: The purpose of the following questions is for you to: (1) identify the central claim made by
the content that you're evaluating, and (2) find evidence from another source regarding that central
claim, that would allow you to assess whether it is true, false, or somewhere in between

Guidance for the following question:

• The central claim should be one sentence or idea from the content you are evaluating, that is a
factual, falsifiable statement related to that content's main point or purpose.
o In photo content, the central claim is usually described in the content text (i.e., the text
caption of the photo and/or in text that is overlaid on the photo). You should consider
both types of text as well as the photo itself when identifying the central claim for a
piece of content.

4
Zeya Images Guidelines: Understanding the Credibility of Information

5
Zeya Images Guidelines: Understanding the Credibility of Information

6
Zeya Images Guidelines: Understanding the Credibility of Information

7
Zeya Images Guidelines: Understanding the Credibility of Information

8
Zeya Images Guidelines: Understanding the Credibility of Information

Question 2.1 Central Claim: What is the central claim you have identified from the original piece of
content?

EVIDENCE SEARCH

PURPOSE: The purpose of the following step is to find evidence from another source regarding the
central claim that you're evaluating, that would allow you to assess whether the central claim is true,
false, or somewhere in between.

Guidance for the following question:

• We ask you to find evidence for or against the central claim you've identified. By evidence, we
mean a statement, photo, video, audio, or statistic relevant to the central claim. This evidence
can either support the initial claim or go against it.
• To find evidence about the claim, please use the following approaches:
o Try a reverse image search to locate sources that provide additional information about
the image and claim you are evaluating.
▪ To do a reverse image search you will need to save a copy of the image you are
evaluating. To do this, right click the image, select “save image as”, and save the
image. *Note: You must delete any images saved for reverse image search after
you are done searching. Please save all images to a single folder that you delete
regularly (at least daily).

9
Zeya Images Guidelines: Understanding the Credibility of Information

▪ After you have saved the image, go to the reverse image search engine of your
choosing (e.g., https://images.google.com/) and upload the image to search.
▪ After searching, you should click into specific pages in the search results to see
what those sources say about the photo/its claim.
▪ Finally, select the source with the highest quality evidence to evaluate the claim
in the next steps. *Note: please make sure the source is discussing the same
photo as the one your evaluating, not another similar image.
▪ You should look for sources that provide additional information or reporting
about the photo and claim (for example, sources that describe what is shown in
the photo, when or where the photo was taken, whether the photo has been
altered/digitally manipulated, and/or whether the claim is true)
▪ *Note: You should not use a link to an identical copy of the photo content you
are evaluating that includes no additional information/reporting as an evidence
source.
o Try a general web search on a search engine of your choice, using some keywords from
the central claim as your search terms.
▪ After searching, you should click into specific pages in the search results to see
what those sources say about the claim.
▪ Finally, select the source with the highest quality evidence to evaluate the claim
in the next steps.
▪ *Note: If you typically search using terms in a language that is different from the
content you are evaluating (e.g., you are rating Hindi content but you typically
search using English search terms): If you are struggling to find evidence, you
should try searching using the same language used in the content.

Question 2.2 Search: What are the keywords you used to research this claim? If you searched multiple
times, enter just the keywords you used on your final/successful search. If you used a reverse image
search, please enter “reverse image search” in the text box.

Part 3: Evidence Description

EVIDENCE DESCRIPTION

PURPOSE: The purpose of the following questions is for you to describe the evidence that you found.

Guidance for the following question:

• In the following questions, we ask you to describe the highest quality piece of evidence you
found about the central claim in your search. High quality evidence is evidence that is:
o From a source you trust (you should try to find evidence from a source you trust,
whenever possible). A trusted source might be:
▪ A source that you are familiar with and that you have found to provide reliable
information in the past

10
Zeya Images Guidelines: Understanding the Credibility of Information

▪ A source that is a widely accepted authority on the topic (this may be based on
your prior knowledge or you may choose to do a quick search engine query for
the author or source to see if this is the case)
▪ A source that cites other widely accepted authorities or sources you trust to
support their claims
▪ A source that generally provides high quality, professional content (this may be
based on your prior knowledge or you may choose to look through additional
content or articles from the source to see if their other content is generally
credible and professionally written)
o Provides clear, credible facts/evidence for or against the claim
o speaks directly to the same event, time period, or photo as the original claim (Unless the
evidence you find is showing that the photo is of a different time, place or person than
described by the claim).
▪ You should check to be sure evidence you use is not outdated or about previous
events, especially for claims about current events.
• If you cannot find evidence about the claim from a source that you trust, you should try to find
the most relevant evidence about the claim you can find from any source, even one you don't
trust.
o In the task, you will be asked to note whether you do not trust the evidence provided by
the source because:
▪ You distrust the source (i.e., you are familiar with the source and know it
typically provides unreliable information)
▪ You do not know enough about the source to feel you can either trust or
distrust it
• If you are rating a multi-part claim and you feel that you need different sources of evidence to
support or disprove different parts of the claim, please paste each source into a separate entry
field.
o If you feel that one piece of evidence properly speaks to all parts of the central claim,
you only need to paste one evidence link.
• Finally, the following are not appropriate evidence sources:
o The same content you are currently evaluating—either from the same source (if the
source is identifiable) or an identical version of the content you are evaluating from
another source (e.g., you may not use a tweet or Facebook post of the same photo and
message you are evaluating as an evidence source).
o Other content from the same source as the content you are evaluating (if the source is
identifiable, e.g., from a logo on the photo or URL in the content text)
o The url for your full search engine results (e.g.,
https://www.google.com/search?q=ice+disk+Westbrook+River)
o Social media posts, unless:
▪ You are rating a claim about whether a person said something on social media
and you are providing a link to the post referenced in the claim (however,
because people may delete social media posts, a claim shouldn't be labeled
false only because the tweet/post is no longer present)he post is from the
official account for a person or organization who is an authority on the claim's

11
Zeya Images Guidelines: Understanding the Credibility of Information

subject (e.g., a post from local emergency responders' office would be an

appropriate source for a claim about a local emergency in their district)

Question 3.1 Which of the following best describes the highest quality evidence you found about the
claim in your search?

• I found evidence from a source that I trust - I'm familiar with the source and know it usually
provides reliable information; OR I have researched the source and it seems to be an accepted
authority that generally provides high quality, professional content.
• I found evidence, but it's from a source I don't know enough about to trust or distrust - I'm
unfamiliar with the source and couldn't determine whether I should trust the information it
provides.
• I found evidence, but it's from a source I distrust - I'm familiar with the source and know it
provides unreliable or false information; OR I have researched the source and it seems to have a
negative reputation and/or generally provides low quality, false, or misleading content.
• I did not find any evidence about the claim - I have tried using reverse image search AND
keyword search (including a search using terms in the same language as the content I'm
evaluating), but I was unable to find any evidence about the claim.

Question 3.2A Evidence Link:

• Please paste the link for the highest quality evidence you found (Paste only the text of the
URL/link here. Do not include additional text from the webpage, article, etc.)

Question 3.2B Optional Additional Evidence Links:

• [OPTIONAL] If you are evaluating a multipart claim and found different evidence sources for
different parts of the claim, please paste the additional sources here
• [TEXT BOX]
• [TEXT BOX]

Question 3.3 Evidence Statement

• What evidence related to the claim does this link provide? For articles, please copy and paste 1
to 3 statements below. For images, videos or audio, please summarize the evidence in the first
text box.
• [TEXT BOX]
• [TEXT BOX]

Part 4: Evaluation

CENTRAL CLAIM EVAULATION

PURPOSE: The purpose of the following questions is for you to make an assessment about the central
claim you are evaluating based on the evidence you described above, and to provide some context for
that decision.

Guidance for the following question:

12
Zeya Images Guidelines: Understanding the Credibility of Information

• Considering the research, you did about the central claim, we would like you to provide your
best judgement about whether the claim is True, False, or Misleading.
• The following definitions may help your decision making:
o True - based on your research, the central claim you are evaluating is factually accurate.
You found facts/evidence that support the claim.
o False - based on your research, the central claim is factually inaccurate. You found
facts/evidence that do not support the claim. This includes evidence that shows:
▪ The photo has been taken out of context (i.e. - you find evidence that shows
that the photo is of a different person/place/object/time than described in the
central claim)
▪ The photo has been digitally manipulated (i.e. - you find evidence that shows
the photo has been digitally manipulated/altered/morphed to show something
different than the original photo)
▪ The photo contains a fake quote (i.e. - the claim is about something someone
said, and you find evidence that shows the quote was never said by the person
described in the claim).
▪ The claim as stated in the content/its text is factually inaccurate
o Misleading- based on your research, the central claim you are evaluating is:
▪ Only partially accurate (e.g., it contains significant inaccuracies alongside some
accurate information, or it contains multiple photos and some are
authentic/accurately represented while others are misappropriated or altered)
▪ Only partially supported by evidence (e.g., only part of the claim is confirmed,
but there is no evidence about the other part(s) of the claim)
▪ About something someone said or claimed, and the quote is correct but is taken
out of context or has been significantly misrepresented
▪ Represented in significantly different ways in the text of the photo caption vs.
the text on the photo/image, and not all aspects of the claim are supported by
evidence based on your research (e.g., the aspect of the claim presented in the
caption is not supported by evidence, but the aspect of the claim presented in
the text on the photo/image is).
o Not Sure - based on the evidence you found in your research (or lack thereof), you do not
feel you can judge whether the central claim is true, false, or misleading.
• For photo content evaluated as False or Misleading, we will also ask you to indicate, if
applicable, if the content is False/Misleading because:
o The photo has been taken out of context (the photo shows a different
person/place/object/event/time than what is described by the claim)
o The photo has been digitally manipulated (the photo has been digitally
manipulated/altered/morphed to show something different than the original photo).
• After the final evaluation, there is also a space for you to provide optional additional
information about your evaluation. You can use this space to share:
o Issues you are having in general with the rating task
o Specific issues you encountered when rating this claim o additional context for why you
did or didn't trust a source (or why it was too difficult to decide if you should do either)

13
Zeya Images Guidelines: Understanding the Credibility of Information

o Any additional information you'd like to share about your evaluation of this claim o
anything else you think would be useful for us to know

Q4.1 Based on the research you did, what is your assessment of the central claim in the original
content?

• True - based on your research, the central claim you are evaluating is factually accurate. You
found facts/evidence that support the claim.
• Misleading - based on your research, the central claim: is only partially accurate (e.g., contains
some significant inaccuracies alongside some accurate information), is only partially supported
by evidence, takes out of context or significantly misrepresents something someone said or
claimed, or is represented in significantly different ways in the photo caption vs. the text of the
image you are evaluating (and only one of these is supported by evidence).
• False - based on your research, the central claim is factually inaccurate. You found
facts/evidence that indicate the claim is without merit (e.g., you find evidence that the photo is
taken out of context, the photo has been digitally manipulated, the photo contains a fake quote,
or that claim as stated in the content/its text is factually inaccurate).
• Not Sure - based on the evidence (or lack thereof) you found in your research, you do not feel
you can judge whether the central claim is true, false, or misleading.

Q4.2: [if Misleading or False is selected for Q4.1]: Based on the research you did, do either of the
following describe the content you are are evaluating? [This question is optional - select only if
applicable]

• Taken out of Context Photo - the photo shows a different person/place/object/event/time than
what is described by the claim
• Digitally Manipulated Photo - the photo has been digitally manipulated/altered/morphed to
show something different than the original photo

Q4.3: OPTIONAL: You may use the space below to briefly state any additional information or details
about your evaluation that you'd like to share:

• [TEXT BOX]

14