Clickbait Labeling Guidelines v2

(Approved by 8/25/20)

7 l
63 tia
Please see the addendum for additional clarification on confusion areas:
Clickbait-V1-V2-Guideline-Addendum-Confusion-Area-Clarifications

Objective

Our objective for this work is to understand the prevalence of clickbait-y content in
News Feed. We know from research that people do not like to see clickbait in their

39 n
News Feeds. If we can understand the prevalence and nature of such posts, we can
work to improve the News Feed experience for everyone.

The purpose of this document is to provide instructions on how to evaluate link

e
titles for potential Clickbait and label content in SRT along the different subtypes of
Withholding & Sensationalism
21 id
VERY IMPORTANT: This labeling work should be done objectively. In other words,
this labeling work isn’t subjectively asking if something ’feels clickbait-y’ to you, but
rather, for example, whether the link title accurately captures and reflects what’s
included in the article it links to.

General Instructions
nf

• The Big Picture: Our goal is to identify posts with links on Facebook where
the text is trying to ’lure’ people to click on them by either 1) withholding
information that’s needed to understand the main take-away(s) of an article
(also called ’curiosity gaps’) or 2) using sensationalist tactics to make the
content of the article seem more important/exciting than it actually is.
Thoroughness/accuracy is the most important factor when you are labeling
Co

posts, so please focus on ensuring each piece of content is labeled to the best
of your ability
• Definitions:
o Link title = the title of the URL’s article as it’s shared on Facebook
▪ Note: this is not the headline of the article on the website;
these can sometimes be different and here we want to focus on
link title in the post itself, which will be rendered in SRT.
o Link description = the text underneath the link title that
describes/previews the content of the article
o Body text / composer box = the text of the FB post itself in the
“composer box” on Facebook
o Preview photo = the image that appears as part of the link preview in
the post
 o Main content = the article / content itself on the website
• Compare FB Post to Webpage Article: We are using the FB Post (including
link title, link description, body text, and preview photo/video) as it
compares to the content on the site to classify posts (see image below)
o You will evaluate a post for withholding or sensationalizing

7 l
information; to know if it withholds/sensationalizes information, you

63 tia
need to click through on the link and review the content of the article
to be sure.
o To be clear, you should not be using the title as it reads on the website
directly in your assessment of whether the title withholds information
- you should be relying on the post in SRT and the main take-aways of
the article on the site in your assessment
• Exceptions to Clicking the Link: In most cases, you will need to click
through the link to accurately evaluate for Clickbait. Exceptions include...

39 ne o If you attempt to click through and you get an interstitial (a warning

message) saying the site is suspected to be malicious (or similar), DO
NOT proceed on. Instead, Respond to the final question in SRT with
the “Link broken or malicious or site content will not load” option and
move on to the next job.
o If you have a bad feeling about the URL itself (e.g., you suspect it’s
pornographic based on the wording in the URL), you DO NOT have to
21 id
click through on it. Instead, respond to the final question in SRT with
the “Site contains objectionable content” option and move on to the
next job.
o If the site contains a blocker that you cannot “X” out of or cannot
otherwise dismiss and that requires you to take some action
nf

(liking/following/etc) to get through, DO NOT proceed on. Instead,

respond to the final question in SRT with the “Site paywalled &
requires log-in, subscription, or other action to access content” and
move on to the next job.
• Response Options: You will be asked two main questions: “Does this link’s
title withhold information needed to understand the full story?” and “Does
Co

this link’s title use sensationalist phrasing or tactics?”. If you respond “yes” to
either of these questions, you will be asked to select which subtype of
Withholding or Sensationalism applies (subtypes explained in the detailed
guidelines sections below) and where the Withholding or Sensationalism
occurs in the post (link title, link description, body text, or preview
photo/video).
• SRT UI: Below is a screenshot of what the Clickbait labeling UI will look like
in SRT.
 7 l
63 tia
39 n
▪
eQuestion 1: Does any part of this post withhold
information needed to understand the full story?
21 id
• Yes
o If yes, please select which subtype of withholding
applies:
▪ Key Detail [A key piece of information is
referenced but not disclosed]
▪ Ellipses [Use of ellipses to create a
nf

curiosity gap]
▪ Listicle [Reference to listicle articles and
call out of a particular, unnamed item]
▪ Celebrities [Reference to celebrities/
famous people without naming them]
Co

o If yes, please select where the withholding

occurs in the post
▪ Body Text [The text of the FB post itself in
the “composer box”]
▪ Link Title [The title of the URL’s article as
it’s shared on FB]
▪ Link Description [The text underneath the
link title that describes/previews the
content of the article]
▪ Preview Photo/Video [The image that
appears as part of the link preview in the
post]
o If yes, is the withheld information revealed in a
different part of the post?
 ▪ Yes
• If yes, please select where the
information is revealed
o Body Text [The text of the
FB post itself in the

7 l
“composer box”]

63 tia
o Link Title [The title of the
URL’s article as it’s shared
on FB]
o Link Description [The text
underneath the link title
that describes/previews the
content of the article]
o Preview Photo/Video [The

39 n
▪
e • No
▪ No
image that appears as part
of the link preview in the
post]

Question 2: Does any part of this post use sensationalist

phrasing or tactics?
21 id
• Yes
o If yes, please select which subtype of
sensationalism applies:
▪ Exaggeration [Exaggerated or made to
seem more important than the article
nf

content really is]

▪ Extreme Language [Language that seems
extreme compared to most other link titles
and article headlines]
▪ Formatting Cues [Unnecessary or excessive
capitalization or punctuation, like multiple
Co

exclamation marks]
▪ Audience Reaction [Attempt to predict the
audience’s reaction]
▪ Behavioral Directive [Link titles that tell
the reader to consume the article]
o If yes, please select where the withholding
occurs in the post
▪ Body Text [The text of the FB post itself in
the “composer box”]
▪ Link Title [The title of the URL’s article as
it’s shared on FB]
 ▪ Link Description [The text underneath the
link title that describes/previews the
content of the article]
▪ Preview Photo/Video [The image that
appears as part of the link preview in the

7 l
post]

63 tia
• No
▪ Question 3: If you are unable to evaluate the post, please
select the reason why:
• Site contains objectionable content
• Link broken or malicious or site content will not load
• Not in correct queue language
• No title listed in SRT or title does not render in readable

39 n
•
e way
• Site paywalled & requires log-in, subscription, or other
action to access content
• Site blocked with non-dismissible pop-up (e.g., an ad)
Immediate Escalation: If the content (Link title or article) contains imagery
or text/voice indicating or soliciting Child Exploitation or Child Nudity,
escalate the Job ID immediately to your project lead.
21 id
o Child Exploitive Imagery (CEI) refers to imagery (images, videos)
depicting the sexual exploitation of a child.
o If you are not comfortable reviewing the content, you can skip the job
after escalating.
▪ Project leads should use appropriate version of the contact
form [Externally Facing] to route job ids into appropriate
nf

queues (CEI should never be screenshot or replicated in any

way as it only further exacerbates the issue. Please always look
to use task/job number for issue identification.)

Withholding Guidelines
Co

We will evaluate whether any part of a post ( link title, link description, body text, or
preview photo/video) withholds information needed to understand key details of
the article to which they link. Sometimes entities will post content where the post
doesn’t tell the full story or withholds key details about the story in order to
encourage people to click through to find out the missing details. This is oftentimes
indicative of what people call “clickbait.”

Withholding Subtypes
Key Detail

Key Detail: The link title, link description, body text, or preview photo/video
 suggests there is a key point about the article needed to understand the main
message or story, but it does not disclose that detail. In other words, the post makes
you question what the main take-away detail is of the article, when that detail could
have easily been included.
• Withholding Examples

7 l
o The one reason you can’t sleep well → key detail needed to

63 tia
understand the story withheld; article explains that ”the one
reason” you can’t sleep well is your diet.
o You’ll never clean with chemicals again after you read about this
natural homecleaning solution! → key detail needed to understand
the story withheld; article clarifies that “this natural home-cleaning
solution” is baking soda & water
o Blake Lively just donated how much to charity?!?!” key detail needed
to understand the story withheld; article shares that Blake Lively

39 ne donated $1,000,000.
o What is the little hole next to your camera on your cell phone? → if
this is not a crowd sourced Q&A site like Quora and it does link to an
article where it states the 1 purpose (i.e. it’s a mini microphone), then
it withholds information
o This city ranked #1 for homicides last year → key detail needed to
understand the story withheld; article is all about Chicago and its
21 id
position as the city with the most homicides
• Not Withholding Examples
o Diet may be causing poor sleep → key detail (cause of poor sleep)
included
o Ryan Reynolds and Blake Lively pledge $1 million to Feeding America
nf

→ key detail (donation amount) included

o The many health benefits of cacao nibs → Not withholding because we
don’t except all of the “many health benefits” to be listed
o Circadian rhythms of neural genes change with aging → no key details
referenced or withheld
o French Fry Tour: Brugge Brasserie → no key details referenced or
Co

withheld
o One injured in crash with train → no key details referenced or
withheld
o Chicago ranked #1 for homicides last year → key detail (which city)
included
o The little hole next to your camera on your phone actually does
several important things → not withholding because we don’t expect
all of the “important things” to be listed
o An Alaskan Mining Town Has Revitalized Its Industry Through Moving
to Solar Power → not withholding because the name of the Alaskan
mining town, which is not widely known, is not needed to understand
the story
Note: Questions as titles depend on how they are used:
 • Questions that lead to thought pieces or essays that include comprehensive
explanations or answers from several standpoints → do not withhold
information.
• Questions from crowd sourced Q&A sites (such as Quora) where the link
title is the question that was asked → do not withhold information

7 l
• Questions that link to a site where a single, short answer is given to the

63 tia
question → withhold information
Ellipses

Ellipses: The link title, link description, body text, or preview photo/video uses
ellipses to denote there’s more to the link title and to create a curiosity gap about
what the rest of the text might say.

39 n
Note: Ellipses used to denote a pause or a formatting issue (e.g. the link title is too
long) ARE NOT indicative of withholding information
• Withholding Examples
e o A person walked across the street, then this happened… → ellipses
creates a curiosity gap around “what happened” (i.e. the man was
shot)
o This woman was deployed for 5 years. When she got home, her
21 id
dog… → ellipses creates a curiosity gap around what “her dog” did
(i.e. he leaps on her once he recognizes her)
• Not Withholding Examples
o A person walked across the street...then they found their pet they had
lost weeks previously. → ellipses used only to denote a pause
nf

o Woman deployed for five years receives ecstatic welcome from her
dog → no ellipses used & main details included in link title
o New postal policies that are slowing service may affect 2020 mail-in
voting, uni... → ellipses used as a formatting function to shorten the
title
Co

Listicles

Listicles are posts that provide content typically in a numbered or listed format
(often within an article or slideshow). Typically, listicles are not considered to
withhold information as long as the post explains what the listicle is about, since it’s
not possible to list out all of the reasons in the post itself. However, when a listicle
post calls out a particular item in the list as noteworthy but doesn’t say why, it can
be withholding.
• Withholding Examples
o These 7 super fruits are associated with better health. #4 will
surprise you! → calls out a particular item from the listicle and
withholds why “it will surprise you”
o Two people from the show are dating in real life and look less than
put together... 15 Pictures That The Cast Of 'Shameless" Wants
 Deleted From The Internet → calls out a particular picture from the
listicle and withholds who "look less than put together" in that picture
• Not Withholding Examples
o These 20 super fruits are associated with better health → We know
we’re going to read about fruits linked to health benefits and it’s not

7 l
possible to list out all 20 in the post

63 tia
o 5 Health Benefits of Cacao Nibs → We know we’re going to read about
health benefits and we don’t expect all 5 to be listed in the post
Celebrities

Celebrities: When a link title, link description, body text, or preview

photo/video explicitly notes that someone is ”famous,” ”well-known,” ”legendary” or
otherwise calls attention to someone’s widely-recognized public status AND then

39 n
withholds their name, this should be considered withholding. The question raised
here is ”OK, which famous person are they referring to?” Otherwise, if the link title,
link description, body text, or preview photo/video withholds the individual’s name
but does not point out that they are well-known, a celebrity, or similar, then it
e
should be tagged as “does not withhold.”

Note: If the link title, link description, body text, or preview photo/video says or
21 id
implies that the individual person may be well-known/ famous and when you click
through on the article you’ve never heard of the individual, do not let this sway your
judgment - this should still be considered withholding. And vice versa - if the link
title, link description, body text, or preview photo/video does not imply that a
person is famous/well-known, but the article reveals that they are, this should still
be considered not withholding.
nf

• Withholding Examples
o Legendary race car driver dies after crash → race car driver
referenced as “legendary” and name is withheld - even if we don’t
recognize upon clicking the article, this is still withholding
Co

o Famous cricket player donated half of his income to charity → cricket

player is referenced as “famous” and name is withheld
o Top model determined to be most beautiful woman → woman
referenced as “top” model and name is withheld
o French politician, known worldwide, leads moment of silence for
fallen law enforcement →politician referenced as “known worldwide”
and name is withheld
o Rugby star volunteers at children’s hospital → rugby player
referenced as a “star” and name is withheld
o Celebrity donates car to neighbor → Subject referenced as “celebrity”
and name is withheld
• Not Withholding Examples
 o Race car driver dies after crash → race car driver is not referenced as
famous, therefore we don’t consider the name to be an important
detail to include in the title
o Amateur race car driver injured in crash → race car driver referenced
as “amateur” i.e. not famous, therefore we don’t consider the name to

7 l
be an important detail to include in the title

63 tia
o Little-known congressman upended US immigration → Congressman
referenced as “little-known” i.e. not famous, therefore we don’t
consider the name to be an important detail to include in the title
o Brazilian president takes office → does not withhold because there’s 1
Brazilian president
o Celebrities react to claims of toxic work environment on The Ellen
Show → The article lists many celebrities (>3), therefore it is not
reasonable for the celebrities’ names to be listed in the link title.

39 ne o Tennis player arrested while on vacation → tennis player not

referenced as famous, therefore we don’t consider the name to be an
important detail to include in the title
o US cabinet member trips and falls, resulting in injury → US cabinet
member not referenced as famous, therefore we don’t consider the
name to be an important detail to include in the title
o Nigerian politician requests investigation into funds
21 id
misappropriation → Politician not referenced as famous, therefore we
don’t consider the name to be an important detail to include in the
title

Withholds Then Answers Scenario

nf

Scenario: If a post withholds information in one place (link title, link description,
body text, or preview photo/video) but clearly answers or reveals that information
in another part of the post (link title, link description, body text, or preview
photo/video), please label accordingly. We want to identify where in the post the
initial withholding takes place as well as where in the post the withheld information
Co

is revealed.

How to Label: If any part of the post (link title, link description, body text, or
preview photo/video) uses one of the withholding tactics described above, but then
reveals the withheld information elsewhere in the post, please select “yes” to
withholding, then select which type of withholding applies and where the
withholding occurs in the post, then select where in the post the information is
revealed.

Examples:

Withholding Exceptions
• Quotes : If the link title, link description, body text, or preview photo/video
is quoting someone, generally, the quoted portion should not be used when
 labeling the below-explained dimensions, because the quoted text is not the
publisher’s words. Exceptions include when a post relies on the quoted
portion to create a curiosity gap...
o Withholding Examples
▪ “I can’t believe one simple ingredient could make my skin so

7 l
much better” is what Celebrity Jane Doe said about our skin

63 tia
cream! → quote is used to create a curiosity gap by withholding
information about the skin care ingredient
▪ “Wow, I can’t believe the results” is what this celebrity said
about our skin cream! → does withhold because it creates a
curiosity gap by not naming the celebrity
o Not Withholding Examples
▪ “I can’t believe one simple ingredient could make my skin so
much better” is what Celebrity Jane Doe said about the

•
39 ne strawberry extract in our face cream → quote creates a
curiosity gap but rest of link title clarifies it
Spoilers: When a link title, link description, body text, or preview
photo/video leads to an article about a TV show, movie, book, podcast, etc.,
(such as an episode re-cap) and withholds information, we will label these as
“does not withhold information” because one motivation for avoiding
spoilers is to practice good “etiquette” by not sharing spoilers.
21 id
o Withholding Examples
▪ Pablo Escobar’s Last Days on Earth Included a Surprise Visit
→ an article that’s not about a show/book/movie/etc. and is
about Pablo Escobar’s last days on earth, withholds
information
nf

o Not Withholding Examples

▪ The season finale of The Walking Dead aired last night. It had a
surprising ending. → does not withhold info because post is
avoiding a spoiler
▪ Walking Dead: Episode III - does not withhold even though it
only mentions the episode title
Co

▪ Narcos Spoiler: Pablo Escobar’s Shocking Last Days on Earth →

does not withhold info
▪ Who Killed Andrea on The Night Of? →does not withhold info
▪ Ming’s Dilemma →if this title is the name of an episode of a
show, it does not withhold
▪ ‘Vikings’ Season 4 Episode 15: The Shocking Death of a Major
Character → not withholding because post is avoiding a spoiler
• Niche Stories: Importantly, some posts will accompany articles/content that
have to do with niche interests. Sometimes these posts tend to use
“shorthand” or abbreviations to convey information. It’s important to ’give
the benefit of the doubt’ in these instances and assume the target audience
has at least a working level of knowledge in a given niche interest area.
o Withholding Examples
 ▪Lebron commits to a team move → If this post links to an
article that lists the team that Lebron is considering moving to,
it WOULD withhold
▪ NBA star makes a team move to the Lakers → withholds
because it doesn’t mention the NBA star (and alludes to the fact

7 l
that the individual is well-known enough to call a star)

63 tia
o Not Withholding Examples
▪ Lebron is considering a team move to the Lakers → wouldn’t
withhold information because we should assume that people
interested in sports would have awareness of who Lebron is
and know that he’s a basketball player.
▪ Lebron is considering a team move → If this post links to an
article that lists out more than 3 teams Lebron is considering
moving to, it WOULD NOT withhold (as this would be similar to

39 ne a listicle and the link title couldn’t list them all out reasonably)
▪ Lebron is considering a team move → If this post links to an
article that does not list the teams that Lebron is considering
(e.g., if he hasn’t named them), then this WOULD NOT withhold

Withholding Response Options

21 id
A B
1 Response Option Use When...
2 Yes, does withhold link title, link description, body text, or preview
photo/video clearly leaves out key details
necessary to understand the main content (article)
nf

one way to think of this is ”can I relatively easily

come up with phrasing that would have been more
straightforward and not withhold key details?” (so,
you can come up with another title with minimal
Co

effort that reflects the information in the article

without leaving out the primary take-away from
the article)
3 If you determine that
part of the post is
withholding, please
select what subtype of
withholding applies
4 Key Detail A key piece of information is referenced but not
disclosed
5 Ellipses Ellipses are used to create a curiosity gap
6 Listicles Reference to listicle articles and call out of a
particular, unnamed item
 7 Celebrities Reference to celebrities/ famous people without
naming them
8 If you determine that
part of the post is
withholding, please

7 l
select where the

63 tia
withholding occurs:
9 Body Text The text of the FB post itself in the “composer box”
on Facebook
10 Link Title The title of the URL’s article as it’s shared on
Facebook
11 Lin Description The text underneath the link title that
describes/previews the content of the article

39 n
12 Preview Photo/Video

13 If you determine that

the part of the post
e
withholds information,
but another part of the
post reveals that
information, please
The image that appears as part of the link preview
in the post
21 id
select where the
information is
answered/revealed:
14 Body Text The text of the FB post itself in the “composer box”
on Facebook
nf

15 Link Title The title of the URL’s article as it’s shared on

16 Lin Description
17 Preview Photo/Video
Co
Facebook
The text underneath the link title that
describes/previews the content of the article
The image that appears as part of the link preview
in the post

Sensationalism Guidelines
Sensationalist post create the wrong expectation in the reader. One of the main
ways to create the wrong expectation is to create an expectation that the
information in the link is somehow more significant or surprising than it actually is
(so, the post might exaggerate or overstate the content included in the article itself
or it might deceive by mentioning something that is not true upon reading the
article itself). Sensationalist posts often over-promise and under-deliver, and upon
reading the article for a sensationalist post, it’s common to feel let down or think,
“well, that was lame” or “ugh, I wish I hadn’t clicked on that. What a waste of time.”
 One way of thinking about whether a news link is sensationalist is to imagine
replacing the actual text of the post with a one sentence summary of the article. If
the text is exaggerated or misleading relative to the one sentence summary, that’s a
clue that the post uses sensationalist tactics.

7 l
63 tia
Sensationalism Subtypes
Exaggeration

Exaggeration: Link titles, link descriptions, body texts, or preview photos/videos

that say or imply something about the nature the story that, upon reading the story,
is exaggerated, misleading, or made to seem more important than it really is. One
way to test this is to evaluate whether the post agrees with the main conclusions of

39 n
the article.
• Sensational Examples
e o Ginger gene is the secret to everlasting youth → exaggerates the
benefits of the ginger gene, saying it results in “everlasting youth”
while the article cites that it only makes you look “two years younger”
o A really unbelievable thing happened to this woman on the highway
→ exaggerates that what happened to the woman as “unbelievable”
21 id
when the article states that what happened was a man stopped to
change her flat tire
• Not Sensational Examples
o Potential ginger tea benefits include improved digestion and clearer
skin link → Frames the specific benefits of lemon tea as potential and
specific
nf

o Man changes woman’s flat tire in viral social media post → articulates
what happened to the woman without exaggerating
Extreme Language

Extreme Language: Link titles, link descriptions, body texts, or preview

Co

photos/videos that contain language that seems extreme in nature, compared to

most other link titles and article headlines. Example language: “shocking,” “never
believe,” “never guess,” or “mind-blowing.” Extreme language is a clue that a post
might be sensational, but doesn’t always necessarily mean a post IS sensational,
since extreme language can be justified in certain contexts (i.e. truly extreme
circumstances). Ultimately, you need to compare the post to the article context to
determine if the language is unnecessarily extreme.
• Sensational Examples
o Shocking weather phenomenon could upend all of your summer
plans → use of extreme language (“shocking phenomenon”) not
warranted given the content of the article is about incoming
thunderstorms
 o Absolutely mind-blowing details in the latest version of Xbox’s NHL
→ use of extreme language (“absolutely mind-blowing”) not
warranted given the content of the article is about minor updates to
an Xbox game
• Not Sensational Examples

7 l
o Incoming thunderstorms are causing folks to move their backyard

63 tia
BBQs indoors → no extreme language used
o Xbox improves gameplay precision in latest version of NHL ’20 → no
extreme language used
o The wartime hell that continues to unfold in Aleppo → use of “hell” is
warranted given the content of the article is about dangerous
conditions in a war-torn city
Note: Cursing or swear words can be considered extreme language when used to
sensationalize the link title.

39 n
• If the link title uses curses / swearing in the course of quoting another
person, this should not be counted as sensational
• For our purposes, we do not consider “hell” to be a curse word
e
Formatting Cues

Formatting Cues: Link titles, link descriptions, body texts, or preview

21 id
photos/videos that use cues like CAPITALIZATION or exclamation points (!!!!!), in an
unnecessary or excessive way, to draw the eye to the post. When we talk about
capitalization here, this often (but not always) happens when capitalizing the
extreme or exaggerated wording (such as “SHOCKING”)
nf

Note: Capitalization of acronyms or proper nouns that are always capitalized should
not be considered sensationalism.
• Sensational Examples
o He Holds the Patent that Could DESTROY Monsanto and Change the
World!!!! → capitalization of an extreme word + excessive use of
exclamation points
Co

o A Rescuer Turned this Pup Over Because of What They Saw on Her
Belly. But Watch THIS... → capitalization of “THIS” used to draw the
eye to the link title
o Soccer player makes ASTONISHING claim about FC Barcelona →
capitalization of extreme word
• Not Sensational Examples
o DON’T LOOK NOW → this is an episode name so it would not be
sensationalized
o How to Wear an LBD like JLaw → We expect acronyms like “LBD” to
be capitalized, so it is not considered sensational
o BREAKING: Local robbery suspect on the loose in downtown
neighborhood → this is not sensationalized because it is a legitimate
use of capitalization for an urgent local news story
 Audience Reaction

Audience Reaction: Link titles, link descriptions, body texts, or preview

photos/videos that make assumptions or predictions about how you, the target
audience, or another group of individuals, will react to the content.

7 l
• Sensational Examples

63 tia
o You’ll be astounded by how soap bubbles form → predicts that you
will be astounded by the content
o You won’t believe how many purses are in Kim Kardashian’s closet
→ assumes that you will not believe the content
o Women will rejoice when they follow this advice about negotiating!
→ predicts how women will react to the content
o Dog-owners’ lives will change once they learn about this grooming
trick → predicts that dog-owners’ lives will change based on the

39 ne content
• Not Sensational Examples
o The Ancient Comet Atlas Arrives Again After 4,400 Years - Info to
Worry About → presents the information as generally worrying, but
does not assume how the audience will react to it
o Kim Kardashian’s closet is home to over 200 purses → does not make
any predictions or suppositions about the audience reaction to the
21 id
content
o My life changed when I learned how to properly groom my dog → first
person description of the “life-changing” experience does not predict
how others will react
nf

Behavioral Directive

Behavioral Directive: Link titles, link descriptions, body texts, or preview

photos/videos that gives a forceful behavioral directives to or tell the reader to
consume the content.
• Sensational Examples
Co

o Read all the way to the end to learn what happens. → instructs the
audience to “read all the way to the end” without sharing any context
as to what the link contains; this would also be considered
withholding (key detail)
o You have to watch this → tells the readers that they “have to watch”
the content without sharing any context; this would also be
considered withholding (key detail)
o A Photographer Spotted a Rare (and Indescribable) Natural Event ...
It’s a Must See → describes the content as a “must see”; this would
also be considered withholding (key detail)
o Everyone should watch this to the end → asserts that “everyone
should watch” the content; this would also be considered withholding
(key detail)
 o Must see: Gigi Hadid's new apartment → describes the content as a
“must see”
• Not Sensational Examples
o This video provides a powerful, 60-second lesson on feminism → link
title does not instruct anyone to watch the video, instead it explains

7 l
what the video contains

63 tia
o Photographer spotted a rare natural event - giant snowballs in Siberia
→ link title describes the content without directing anyone to
consume it
o Gigi Hadid’s new apartment exhibits colorful and eclectic interior
design → link title describes the content without directing anyone to
consume it
o Visit our website to book your dream vacation → Link title shares an
action (e.g. visit a website) that readers can take to achieve a goal (e.g.

39 ne book a dream vacation)

Sensationalism Exceptions
• Words like BREAKING/UPDATED/DEVELOPING: sometimes non-
sensational content legitimately uses this wording and capitalization to draw
attention to the fact that there’s a newly developing story or an ongoing story
with new developments. You might notice these being used by, for example,
21 id
local news outlets when reporting a developing story about a crime. Just
because a post uses these words and/or uses them in caps, does not
necessarily mean the link title is sensationalist.
o Sensational Example
▪ BREAKING: Matt Damon’s unbelievable new campaign to get
nf

people clean water → uses exaggeration “unbelievable” and

formatting cues (capitalization of “BREAKING”) in a context
that is not legitimate (this is not truly a breaking news story)
o Not Sensational Example
▪ BREAKING: Local robbery suspect on the loose in downtown
neighborhood → does not use sensationalist tactics because the
Co

capitalization is justified
• Quotes: If the post is quoting someone, the quoted portion should not be
used in your assessment.
o Not Sensational Example
▪ “It was the MOST INSANE game I’ve ever played” says soccer
star → does not use sensationalist tactics

Sensationalism Response Options

A B
1 Response Options Use When...
2 No, not sensationalized post does not meet any of the criteria
outlined in this document for
sensationalism
 3 Yes, is sensationalized post meets some/most/all of the criteria
outlined in this document for
sensationalism
4 If you determine that the
link title, link description,

7 l
body text, or preview

63 tia
photo/video is
sensationalized, please
select what subtype of
sensationalism applies
5 Exaggeration Link titles that are exaggerated or made
to seem more important than the article
content really is
6

8
39 n
Extreme Language

Formatting Cues
e
Audience Reaction
Link titles that include language that
seems extreme compared to most other
link titles and article headlines
Link titles that include unnecessary or
excessive capitalization or punctuation,
like multiple exclamation marks
Link titles that try to predict the
audience’s reaction
21 id
9 Behavioral Directive Link titles that tell the reader to consume
the article
10 Other Most link titles should fall under one of
the subtypes above. However, if there is a
link title that does not apply to any
nf

subtype, select "other" and explain what

you would call the link title category
11 If you determine that part
of the post is
sensationalist, please
select where the
Co

sensationalism occurs:
12 Link Title
13 Lin Description
14 Body Text
15 Preview Photo/Video
The title of the URL’s article as it’s shared
on Facebook
The text underneath the link title that
describes/previews the content of the
article
The text of the FB post itself in the
“composer box” on Facebook
The image that appears as part of the link
preview in the post
 General Exceptions
• Missing People Posts: Posts that seek help to find missing/abducted people,
missing/stolen pets, or missing/stolen property should automatically be
classified as not withholding and not sensational given the serious nature of

7 l
the content. Note: this exception does not apply to satire, figures of speech,

63 tia
trolling or posts that rely on the missing person reference to create a
curiosity gap.
o Violating Examples
▪ You’ll never guess why Hillary Clinton has gone missing from
politics! → Figure of speech used to withhold a key detail;
sensational due to predicting of the audience reaction (“you’ll
never guess”)
▪ You’ll never guess where this missing child was found! → This

39 ne is not about seeking help finding a missing child but rather a

post that withholds information about where a missing child
was found; sensational due to predicting of the audience
reaction (“you’ll never guess”)
▪ These thieves stole a church’s property and police found them
in a really funny way! → Not about seeking help finding stolen
property but rather a post that withholds information about
21 id
how police found the thieves; not sensational
o Not Violating Examples
▪ Help us find this missing child → not withholding, not
sensational
▪ Police searching for missing person → not withholding, not
nf

sensational
▪ Person requests help searching for missing pet → not
withholding, not sensational
▪ This famous man went missing and police need your help
→ not withholding, not sensational
▪ Citizen seeks help with finding stolen property → not
Co

withholding, not sensational

• Proper Nouns: If the link title, link description, body text, or preview
photo/video is a proper noun such as a song name (or song name + artist
name), film name, podcast series name, proper noun blog series names, or
proper noun photo series / photo galleries names, title of podcast episode,
book title, etc., AND does not include other verbiage, they should not be
tagged as withholding information or sensationalist
o Not Violating Examples
▪ “This American Life” by NPR → not withholding, not
sensational (title of series)
▪ “The Tim Ferriss Show” → not withholding, not
sensational (podcast series name)
▪ “Kristin’s Life Musings” → not withholding, not
sensational (blog title)
 • Oftentimes, blog titles or series names or similar will be
offset with a vertical line in the title such as, “Kristin’s
Life Musings | Trip to Germany Was Amazing”
• Social Media/Sharing Sites:
o Spotify, Soundcloud, and Pandora (and similar) shares that include

7 l
song titles / artist names → does not withhold information or

63 tia
sensationalize
o Giphy, imgur, and similar photo sharing link titles typically do not
withhold info or sensationalize
▪ However, if a photo sharing site’s title does explicitly create a
curiosity gap (or use sensationalist tactics), then it should be
categorized as such
o Twitter titles, Instagram titles → typically does not withhold
information or sensationalize

39 ne o Kickstarter, GoFundMe, and similar → typically does not withhold

information (titles or names for fundraising campaigns) or
sensationalize
o Pinterest pinboards should be handled on a case-by-case basis - some
entities use Pinterest (and similar platforms) to gain traffic via
clickbait-y titles.
o YouTube titles should also be handled on a case-by-case basis - some
21 id
entities use YouTube (and similar platforms like Vimeo, etc.) to gain
traffic via clickbait-y titles.

Cannot Evaluate
nf

If you are unable to evaluate the post/webpage, please indicate the reason why in
the final question.

A B
Co

1 Response Option Use When

2 site contains Webpage includes content that depicts
objectionable pornography or sexually suggestive
content images/videos (note: nudity that is not
(hatefulness, sexualized should not be tagged here - e.g.,
graphic / violent breastfeeding photos, photos of people after
content, nudity / reconstructive surgery)
pornography)

Webpage includes content that is violent or

macabre (e.g., deceased people) or depicts
content that is disturbing in terms of being
”gross” (e.g., pictures of infections)
 Webpage includes content (text/images/videos)
that is hateful, hate-speech, or geared towards
inciting hateful attitudes towards a group of
people or a person (i.e., this could include racism,

7 l
sexism, ageism, or any other hate speech directed

63 tia
at specific people or groups but can be more
broadly defined hatefulness that doesn’t fit into
particular ’hate speech’ classifications). This
includes ads

This includes ads

3 link broken or Use when the link is broken or takes you to a

39 n
malicious or site

e page where there is no content or the content

content won’t load won’t fully load (for example, the site itself loads,
but an embedded video on the site won’t load for
any reason)

**You may also use this when the link is blocked

21 id
for suspicion of being malicious

4 not in correct Use this when the link title or article/content is

nf

queue language not in the language for the queue you’re working
in
5 no title in SRT or Select this either when SRT does not render a
title not readable title or when there is something written in the
title area in SRT but it is not readable
Co

Here, ”not readable” means that the title is in

weird characters and not letters or title renders
as a url/link addres

6 site paywalled or Select this when the site requires you to

requires action to subscribe or complete a survey or ”Like” their
access content Page (or otherwise requires you to take some
action before you can access the content)
7 site blocked with Select this when the site’s content is blocked by a
non-dismissible pop-up that can’t be dismissed (but doesn’t ask
 pop-up (e.g., an you to take a specific action - and example here is
ad) an ad that can’t be closed out of)

FAQs

7 l
1. What if I get an error or nothing renders at all in SRT?

63 tia
a. Leave the responses and text box blank. Click the advance arrow at
the very bottom of the page to go to the next item to see if it’s an
isolated error.
i. If the next item renders in SRT, then please resume your work.
ii. If the next post does not render, please click the bottom right
advance arrow one more time to see if it’s just a couple of
errors.

39 ne 1. If the next item renders in SRT, then please resume your

work.
2. If the next post is also an error, please escalate this
issue to your project lead
2. What if the link is broken or blocked because it might be malicious?
a. Please select the appropriate checkbox to reflect this (e.g. “Link
broken or malicious or site content will not load")
21 id
b. If you are >90% certain that the link title clearly withholds
information or uses sensationalist tactics and you don’t need to
confirm via the article that this is the case, then please classify as
you’re able. However, if you need to be able to see the full article to
know if the link title withholds/sensationalizes information and the
link is broken/malicious, please leave the first two questions blank
nf

and select the appropriate checkbox (e.g. “Link broken or malicious or

site content will not load")
3. What if the link requires a log-in to access or is behind a paywall (or wants
me to take a survey or is otherwise blocked and not accessible)?
a. Please select the appropriate checkbox to reflect the situation you’re
Co

running into (e.g. “Site paywalled & requires log-in, subscription, or

other action to access content”
b. If you are >90% certain that the link title clearly withholds
information or uses sensationalist tactics and you don’t need to
confirm via the article that this is the case, then please classify as
you’re able. However, if you need to be able to see the full article to
know if the link title withholds/sensationalizes information and the
link is broken/malicious, please leave the first two questions blank
and select the appropriate checkbox (e.g. “Site paywalled & requires
log-in, subscription, or other action to access content”
4. What if the link I’m directed to is gated / blocked (but not by log-in or
paywall - for example, a pop up that can’t be exited)?
a. Please select the appropriate checkbox to reflect this (e.g. “Site
blocked with non-dismissible pop-up (e.g., an ad”))
 b. If you are >90% certain that the link title clearly
withholds/sensationalizes information and you don’t need to confirm
via the article that this is the case, then please classify as you’re able.
However, if you need to be able to see the full article to know if the
link title withholds/sensationalizes information and the link is

7 l
broken/malicious, please leave the first two questions blank and

63 tia
select the appropriate checkbox (e.g. “Site blocked with non-
dismissible pop-up (e.g., an ad”)

39 ne
21 id
nf
Co