Ads NPNC Relevance Rating

Guidelines v3

5 l
00 tia
Revision Date 12/11/2019

Workflow Goals + Context

The purpose of this workflow is to identify relevant and high-quality ads for search

56 n
queries on Facebook SERP (All search tab). For each ad to be rated, raters will see
a search query containing the terms used to search and a sponsored post.
This sponsored post includes the name of the advertiser, visuals such as images or
video, and text commentary. Then, based on the content, raters will rate the ad for
e
relevance if there is no defined product intent.

The convention of {query} → [the main topic of result] will be used to provide
33 id
examples in these guidelines.
Definitions
o Query is the text used to perform a search for specific results. The
word(s) or phrase entered into the search engine (e.g. “cat
videos,” “pancake recipes,” “Michelle Obama”.)
nf

o Result is an item returned by the search engine as an outcome of the

search. Landing page is considered to be a part of result for all Ads
workflows.
o Product is the item that is the subject of the resulting post. Note that
for our purposes a “product” can encompass multiple items for sale in
the same result.
Co

o Searcher is a supposed person who conducts a search on Facebook All

(SERP workflow) or Marketplace (MP workflow) tabs using the given
query and receives the given result.
o Searcher's intent is what the searcher intended or wanted to find
when they typed their query into search bar considering the query
itself, the intention behind those words and the context. The most
traditional query intent classification includes informational (to learn
about something), navigational (to find a specific site) and
transactional (i.e. to buy a car or to download an app) intents.
o SERP (Search Engine Result Page) is the page of search results that a
search engine (e.g. Google or Facebook) returns in response to a
query, i.e. Facebook All Tab SERP or Facebook Marketplace SERP.
 o Facebook Marketplace is a digital marketplace where users can
arrange to buy, sell, and trade items with other people in their area.
o Workflow refers to the way we group together a particular type of
search results, depending on what kind of search process we are
interested in examining. For example, in the Ads SERP “workflow,” we

5 l
look at only Ads showing in response for queries on Facebook All Tab

00 tia
SERP, and assess them for relevance.
o NPNC: no people no commerce.

Search Ads NPNC Labels

Skip/Do Not Skip

• 0 - Skip
o Gibberish Query

56 ne o Foreign Language
o Rendering Issue
o No Landing Page
o Brand Policy Violation
• 1 - Rate
Product Intent/ No Defined Product Intent
• 0 - Product Intent
33 id
• 1 - No Defined Product Intent
Query-Product Relevance
• 1 - Off-Topic
o Poor Quality
o Not Topically Related
nf

• 2 - Somewhat Relevant
o Not the main topic
o Location sensitive
• 3 - Relevant
o Complements
o Substitutes
Co

o Meet most needs

0 - Skip

SKIP is used when the job cannot be evaluated in Ads Rating workflows because of
one of the following reasons:
• The job cannot be rated consistently with other jobs, i.e. because of technical
issues.
• The job does not belong to Ads Rating workflows, i.e. the query or the result
is not compliant with Facebook policies.
A B C
1 Skip sub-label Brief explanation
 2 Job cannot Gibberish Rater cannot understand the query
be rated
consistently
with other
jobs

5 l
3 Foreign If the result including the landing page

00 tia
Language contains foreign language
4 Rendering Error message shows up in SRT instead of a
Issues job
5 No Landing The result does not include a link to the
Page landing page
6 Other Tool All the other tool issues preventing rater
Issues from understanding the query or the result in

8
56 n
Job does not
belong to
Ads Rating
e
workflows
Brand Policy
Violation
full
The query or the result do not comply with
Facebook policies

SKIP/Gibberish: Skip if you cannot understand what the query means after
conducting a thorough side search and considering possible search intentions. This
33 id
sub-label includes queries in a foreign language and queries that have a very broad
interpretation in side search or are not understood by side search at all.
nf
Co

• Examples: {m}, {123879324509y14}, {семантика}

SKIP/Rendering Issues: Sometimes an error message shows up in SRT instead of a

job. It happens for various reasons; for example, when an ad has been removed by
the advertiser after the job was created. Skipping is the only way to remove a job
 from the workflow so it wouldn't show up again. If a job has both rendering issue
and no landing page, please skip under Rendering Issues.

Please note: It is important to copy the error to the text box when possible, so that we
can track these errors.

5 l
00 tia
56 ne
33 id
nf
Co
Co
nf
33 id
56 ne
00 tia
5 l
 5 l
00 tia
SKIP/No Landing Page: Please SKIP the jobs that render but do not have links to a
landing page or have non-functional links under No Landing Page sub-label. We take
the landing page into consideration when rating, thus these jobs cannot be rated
consistently with the others.

56 n
SKIP/Foreign Language:
e
If the result including the landing page contains foreign language, and it
prevents you or can prevent your colleagues from understanding the result
consistently and in full, please label the job as skip/foreign language, for example:
33 id
nf
Co

SKIP/Other Tool Issues: SKIP the job if you cannot rate it because you cannot
understand the result from what you see in SRT, i.e. visuals of the result are cropped
too badly and you cannot comprehend the text on it. If a tool issue can be classified
as a Rendering Issue or No Landing page scenario, use the corresponding sub-labels.
If it can't be, choose Other Tool Issues sub-label and explain your selection in the
text box.
 SKIP/No landing page: SKIP the job if the link doesn’t load when you click on it so
you are not able to rate the job. Also, if the external link does not have a working
website or present a 404 error or non functional landing page it should be SKIP.

5 l
00 tia
56 ne
33 id
@
nf
Co
 5 l
00 tia
56 ne
33 id
SKIP/Brand Policy Violation: Skip the jobs where either query is asking for
Prohibited Content/Sensitive results, or the result can be classified as one of those,
nf

or if you discover it at any moment of further rating.

Prohibited Content: Query asks for products from the list below or the resulting
post falls under one of the Prohibited Content categories.
• Prohibited Products:
Co

o Tobacco Products
o Drugs & Drug-Related Products
o Personal Health items
o Unsafe Supplements
o Unauthorized Streaming Devices
o Surveillance Equipment
o Spyware or Malware
o Circumventing Systems
o Counterfeit Documents
o Weapons, including parts, accessories and related uncommon products
o Any Illegal Products and Services
• Prohibited Types of Content in the Ad:
o Profanity or Bad Grammar
 o Sensational Content, including shocking, sensational, disrespectful or
excessively violent content
o Misleading or False Content: deceptive, false, or misleading content,
including deceptive claims, offers, or methods
o Controversial Content: content that exploits controversial political or

5 l
social issues for commercial purposes.

00 tia
o Third-Party Infringement: content that infringes upon or violates the
rights of any third party, including copyright, trademark, privacy,
publicity, or other personal or proprietary rights.
o Personal Attributes and Discriminatory Practices: content that
discriminates or encourages discrimination against people based on
personal attributes such as race, ethnicity, color, national origin,
religion, age, sex, sexual orientation, gender identity, family status,
disability, medical or genetic condition or content that asserts or

56 n
• Financial:
e implies personal attributes.

o Payday Loans, Cash Advances and Bail Bonds

o Multilevel Marketing
o Penny Auctions
o Prohibited Financial Products and Services such as initial coin offerings
and binary options
33 id
• Sensitive Content: Query is related to one of the topics from the list below
or the topic is present in the resulting post:
o Violence or Calls to Violence: any weapons, including weapon parts
and accessories, display/antique weapons (e.g. display/antique
swords)
nf

o Sexual Content and/or Nudity

o Child Exploitation and/or Nudity
o Human Trafficking
o Acts of Terrorism
o Self-Injury
o Suicidal Content
Co

o Hate Speech
o Bullying and Harassment
Please note, if the category is contains Sensitive Content, escalate the Job ID to
your lead immediately.

The full list of Prohibited Content items and detailed explanations can be found
here: https://www.facebook.com/policies/ads/prohibited_content.

Note: #15 of the advertising policy (non-functional landing page) should be rated as
SKIP/No Landing Page
 5 l
00 tia
56 ne
33 id
Product Intent vs. No Defined Product Intent
nf

In order to determine if the query has no product intent (or vague product
intent, VPI, as opposed to defined product intent, DPI), please conduct the side
search and check products on Google Shopping Tab or bing.com/shop. Relevance
rating will require a deeper understanding of the query and side search in “All” Tab,
but possible product intents are the key to determine if the query is VPI or DPI.
Co

product intent

NPNC workflow doesn’t rate Product Intent jobs. A good way to determine if a query
has product intent is to ask yourself if any searcher would look for the results
displayed in the side search using the given query. If the answer is “Yes”, and the
side search reveals an intent for specific product or product category, please label
the job as “0 - Defined Product Intent”. No further relevance rating is required for
DPI queries. Here are some examples:
 5 l
00 tia
56 ne
33 id
Product query {iPhone XS} has product intent for [iPhone XS], search query {285
75 15} has strong product intent for [tires.] If all the products in the side search
belong to 1 or 2 products or product categories, the query has product intent for
these product(s)/product categories. If the query mentions features or
characteristics of the product, i.e. {iPhone XS 128GB}, it should be taken into
nf

consideration, and it also narrows the intent. Chances that [iPhone] is an expected
result for query {iPhone} are very high.
#productintent
Co
 5 l
00 tia
56 ne
33 id
nf

Brand queries like {Apple}, {because cosmetics} can be matched with

the products and product categories associated with the queried
brand and have product intent. Side search results make sense and are expected
considering the query.
#productintent
Co
 5 l
00 tia
56 ne
33 id
Query for specialized retailer {footlocker} has product intent for [active shoes].
Side search results are focused on one product category and this product category is
nf

expected considering the query. There are other search terms to look for shoes, but
at least some searchers can use this query to get these results.
#productintent

no defined product intent

Co

If you can easily think of much better queries to search for the results that you see in
the side search and the side search results are not focused on a specific product or
product category, rate the job as “1 - No Defined Product Intent” and proceed to
relevance rating. For example, the query is {Seattle}, the result is [a t-shirt with
depiction of the Space Needle]. The most reasonable product query to find this
result is {space needle t-shirt}, and no product result including [a t-shirt with
depiction of the Space Needle] cannot fully meet the searcher’s intent.
 5 l
00 tia
56 ne
33 id
nf

Broad retailers typically carry a wide range of products and product categories,
and cannot be clearly associated with only one product category. Query for a broad
retailer {walmart} has navigational intent for [www.walmart.com] or [walmart
page/group on Facebook.] Chances that any searcher would intentionally search for
a [tent] or an [Apple iPhone 6] using the query {walmart} are very slim.
Co

#nodefinedproductintent
 5 l
00 tia
56 ne
33 id
nf

Search query {Chicago} primarily has informational intent to learn more about the
city. Product interpretation of this query is vague and side search returns topically
related souvenir products at the top of shopping results which typically is a good
indicator of no product intent.
#nodefinedproductintent
Co
 5 l
00 tia
56 ne
33 id
nf

Similar to {Chicago}, query {nicki} primarily has informational intent to learn more
about the rapper and no specific product intent. Side search returns t-
shirts. #nodefinedproductintent
Co
 5 l
00 tia
56 ne
33 id
Broad product categories: Queries {hair}, {Christmas}, {moving} can be associated
with some products but do not make the expected product clear and may have other
interpretations than the product intent one.
nf

A searcher might expect the products returned by the side search, but the query is
not strongly focused on one of them, and there are much better search terms to look
up any of the returned results (for example, “hair extensions”, “blond wig”, “dry hair
treatment”, “hair regrowth treatment” etc.)
#nodefinedproductintent
Co
 5 l
00 tia
56 ne
33 id
Food and drinks: query {Aztecan Mexican}, the query’s intent is for a restaurant
with this title, likely located near the searcher. Food and drink establishments have
no product intent, as they are informational queries.
Food and drink establishment have no defined product intent, as they are
informational queries.
nf

#nodefinedproductintent

Please note: Most queries in this workflow should not have defined product intent
(DPI). If there is defined product intent, raters should select “0 - Defined Product
Co

Intent”, and the job is considered complete with no relevance decision needed. Also,
note that if a company/brand offer both services and products, it should be
considered as No Defined Product Intent.

Side Search for not defined product intent queries

Please conduct a thorough side search on “All” Search Tab. It is important to

conduct the side search with and without searcher’s location as the same search
terms may refer to different businesses based on location. For example,
• {golden boys} is primarily related to novel by Australian author Sonya
Hartnett,
• {golden boys Austin} refers to a musical band,
• {golden boys San Francisco} refers to a pizza restaurant,
 • {golden boys Fitzgerald GA} refers to a food wholesale company

Relevance

5 l
1 - Off-Topic

00 tia
Off-Topic / poor quality

First, please note that if the quality of the displayed job in SRT prevents you from
understanding the result, SKIP this job under SKIP/Other Tool
Issues or SKIP/Rendering Issues whichever seems to be more appropriate.

Job can be rated as Off-Topic/Poor Quality based on poor visuals in the ad or

56 n
because the ad may create misleading experience for the searcher.

Each ad is accompanied by visuals like photos or videos. Some of the visuals may
employ disruptive techniques like flashing screens in videos to attract more
e
attention, which goes against Facebook policies and should labeled Off-Topic / Poor
Quality. Most common factors creating poor viewing experience
of photos include:
• Lighting effect is too light or too dark
33 id
• Has oppressive text, masks, excessive filters that distract from the viewing
experience
Most common factors creating poor viewing experience of videos include:
• Poor stability and wildly moving video (unless the context of the video
justifies it)
nf

• Subtitles are overwhelming and take up the entire screen

• Video of a video
The following factors are not considered to be Poor Quality indicators:
• Photo: Resolution is grainy or blurry,
• Photo: Aspect ratio is skewed causing distorted images,
• Photo: Image is cropped badly and makes it visually jarring,
Co

• Video: Video does not have the correct sharpness, contrast, size, and aspect
ratio for a good visual experience,
• Video: Poor audio or static (unless the context of the video justifies it.)
SRT is aimed to convey the main idea of the ad to rater, but it simplifies the imagery.
Thus, these cases are really tool issues rather than bad quality ads.

Please find some examples below:

 5 l
00 tia
56 ne
33 id
nf
Co

Adfarm.mediaplex.com is used by brands to track where visits are coming from, but
it is perceived as misleading AND does not return an expected page for this ad,
hence this result should be rated Off-Topic / Poor Quality.
 5 l
00 tia
56 ne
33 id
The paragraph states that 5 million people like this product, but in reality there are
only 5 likes. This should be rated as Off-Topic / Poor Quality for containing
misleading/false content.

If the major topics of 1) the ad commentary, 2) ads visuals and 3) the landing page
nf

outside of Facebook are not quite coherent and do not correspond with each other,
the ad may create misleading experience for the searcher.
Co
 Example of a misleading job, SRT UI screenshot. The post commentary and the
visuals are about furniture, and at first the ad seems to be relevant to the query.

5 l
00 tia
56 ne
33 id
The landing page is completely on different topic. Considering searcher’s intent, this
nf

result can create bad searcher experience, and should be rated as Off-Topic / Poor
Quality
Job can be also considered misleading and rated as Off-Topic / Poor Quality if the
ad visuals do not match the ad commentary. An unrealistic price or 50%+ price
difference between the ad and the landing page are also indicators of Off-Topic /
Poor Quality. If the price difference is less than 50%, disregard the price and rate
Co

normally.

Kindly note that if the external link on SRT job leads to a landing page
within Facebook domains (fb.me, facebook.com), in some cases you won’t be able to
access the landing page content at the moment of rating. These jobs are not
considered misleading.

Off-topic / not related

The ad in the result is not related to query’s primary intent and does not look like a
natural or reasonable response to the query. The result may or may not match
keywords, but if it does, the match is in a random or unintelligible way. In other
 words, the result should be rated as Off-Topic when the result does not make sense
considering the query and there are no or very slim chances that any searcher
would be interested to learn more about this result.
• Random word match:
o {Fox News} → [domesticated foxes]

5 l
• Random word match:

00 tia
o {land} → [informational article on land rover vehicles]
• Based on the side search, the query has navigational intent for the popular
page https://tasty.co/, example is not topically related:
o {Tasty Videos} → [A mouth-watering video posted by someone of their
gigantic burrito with the caption “tasty”]
• Navigational intent for Facebook
page https://www.facebook.com/AmandaEllersMissing/, not related:

56 ne o {findamanda} → [MCO RodBuckle]

• Category “Cities of the U.S.” matches, but there are no other common topics and
there are no chances that any searcher’s need could be met:
o {city of dallas georgia – government} → [promotion by ‘Visit Las
Vegas!’ Facebook page]
• The result is clearly not expected for given search query
o {Housekeeping services} → [Hotel cart housekeeping sold by
Walmart]
33 id
Note: word match is not always necessary for topical relevance, especially for vague
product intent queries. For example:
• Topically related:
o {bad credit} → [Get Your Next Used SUV Today Regardless of Credit
Score. Warranty Included and a Carfax History Report].
nf

While the product intent of the query {bad credit} is broad, the result is a
meaningful response to the query and the searcher might be interested to learn
more about it. “Bad credit” and “regardless of credit score” are not strictly
synonyms, but “regardless of credit score” implies the idea of “bad credit” in given
context.
Co

2 - Somewhat Relevant

Somewhat Relevant is used when the result is loosely related to the query, for
example, the result may satisfy the same need, but we cannot assume it is a perfect
match to the query. Somewhat Relevant result is of interest to the searcher
considering the query, but does not fully meet searcher’s need, for example, it can
possibly satisfy the same need with another product or service, or it can be helpful
to the searcher in any other quality, but it might be not the exact match and not fully
meet most searchers’ intent.
Somewhat Relevant/Location sensitive

If user is explicitly looking for a service (restaurant, grocery, cinema, etc.) in a

 specific location and the ad is in a different location > 100 miles radius from the
specified location, rate as Somewhat Relevant. If the ad is an online service, then
location does not matter.
o {sushi restaurants in Mountain View} → [Top 5 restaurants in New
York]

5 l
o {tugwell roofing California} → [roofing service in Texas]

00 tia
o {motes pools Arizona} → [Solstice pools Houston]
Note:
• If the user is looking for a service in a specific location but it’s a brand chain
that has multiple branches and the ad is from a different location, in this case,
the distance doesn’t matter, thus rate as relevant.
o {Trader Joe’s San Jose} → [New product launch in Trader Joe’s San
Diego] → relevant
• If user is looking for a specific school but the ad is an online program of a

56 ne similar education, in this case location should not matter (education

category).
o {MBA Berkeley} → [Online MBA in Columbia] → relevant
somewhat relevant / Not the main topic

The query and the result (including the landing page) have a topic in common, but
33 id
this topic is not the main focus of the ad and/or the landing page.
• Topic is not the main focus of the ad and/or the landing page. The query and
the result (including the landing page) have a topic in common, but this topic
is not the main focus of the ad and/or the landing page. Somewhat Relevant
includes other search engines as focused on searcher’s intent clarification,
nf

pages that require sign in or sign up as focused on collecting data, and all
the other scenarios when the queried topic or reasonable substitutes are
not the main focus of the ad and the landing page.
o {Las Vegas} → [Las Vegas Getaway]
User is looking for Las Vegas, though it’s a vague query the landing page seems to be
relevant to the query, but the searcher is required to introduce the contact
Co

information in order to see the result so this job is somewhat relevant/not the main
topic.
o {The Handmaid’s tale} → [Hulu- subscription free trial]
In this example the user is looking for a specific TV show and the ad is a landing
page to enter the contact information in order to try the free trial. This LP doesn’t
allow us to rate the job but it’s topically related.

• Resulting ad’s topic is not the primary intent of the query: query and result
have word match, but the result is not the most common product/service
that user would search for with the given query, but could be of some
interest.
o {John Mayer} → [John Mayer electric guitar]
• Query and result are under the same category but it’s not exactly what the
user is looking for but it’s still related.
 o {Trader Joe’s} → [Hello fresh black Friday offer meal prep delivery]
Query is for a public figure/celebrity, in this case a singer, songwriter. The primary
intent would be information, articles, and videos of this celebrity. The resulting ad is
an item (a guitar) from the celebrity’s product line, which would be a secondary
intent and may not be exactly what a searcher is looking for, but could be of some

5 l
interest.

00 tia
o {emily king} → [an ad for a concert subscription]: intent for an
American singer.
▪ category is ‘Music & Audio’, there is topical connection and
shared topic, but the topic of the ad is not directly related to
Emily King.
o {kopkey manure management} → [Manna Irrigation ad: “A new
technology in the irrigation industry, Manna provides a software only,
sensor-free irrigation recommendations solution that is based on

56 ne
3 - Relevant
satellite technology“]:
▪ Navigational intent for Facebook page Kopkey Manure
Management, category is ‘Home & Garden’.
▪ The query and the result are acceptably topically related, but
are not directly related to each other.
33 id
The Relevant label is reserved for results that meet most searchers' need
considering the query. There should be a high chance that the searcher would
have continued interest in the result and learning more detail about ad's main
topic.
nf

relevant / substitutes

Query is for a specific service, result is other services that satisfy a similar need or
desire. This category may include proper name/location mismatch, but still has the
same general operation.
o {dean smith photography} → [erickmarquez_photography]
Co

o {lion's tail brewing co.} - [McFleshman's Brewing Co.]

o {joes pizza ann arbor} → [Guido's Pizza]
o {national life group} - [Colonial Penn Life Insurance]
Note: If the service location is specified in the query, ad shows substitute service
more than 100 miles away, then rate as Somewhat Relevant as instructed in
Somewhat Relevant section.

relevant / COMPLEMENTS

User’s query refers to a service and the shown product/service does not completely
satisfy the user's need but can be used for that service.
o {Dean Smith photography} → [picture frame]
o {Yoga Classes} → [Yoga Mat]
 o {Blue Horsehoe Tattoo Hampton} → [Revitalize Tattoo Aftercare]
o {Ski Resort in Tahoe} → [Daily equipment rental to ski]
relevant / mEET MOST SEARCHER’S NEED

5 l
The Relevant label is reserved for results that meet most searchers' need
considering the query. There should be a high chance that the searcher would

00 tia
have continued interest in the result and learning more detail about ad's main
topic.
• Result that responds directly to the query:
o {disney} → [Disney cartoon theme event]: Category is ‘TV &
Movies’, the results looks very natural considering the query.
o {coffee} → Seattle's Best Coffee‘s post [We’re celebrating
#NationalDonutDay on June 7 with free samples of Seattle’s Best

56 n Coffee® and Walmart Freshness Guaranteed donuts! Click to find

events at your nearest Walmart]: category is ‘Food & Drink’, the result
naturally responds to the query.
• Common products reasonably related to the query can all be 4 - Relevant.
e o {hair} → [hair extension/shampoo/hair iron/hair color]: category is
‘Fashion & Style’, the result is very reasonable considering the query.
o {garden} → [garden plants/soil/water hose/pruner]: category is
33 id
‘Home & Garden’, the result belongs to broad category pointed by the
query.
o {Nicki Minaj} → [Nicki Minaj music CD]: category is ‘Music & Audio’,
the result is a natural response to the query and can be helpful for
many searchers.
nf

FAQ/Troubleshooting
Q: I can't get into the queue!
Co

A: There are many reasons you might not be able to enqueue. First, check the group
chat to make sure that your workflow is still using that queue. Try to enqueue a few
times, and make sure your internet connection is steady. If you still can't get
in, screenshot the error and send it to your lead.

Q: I got a maxed out error.

A: This might mean that either you've done all the necessary work OR there's an error
with the queue. Screenshot the error message and send this to your lead to see what
you can do next.

Q: I accidentally submitted a job before I was done. Can I go back in SRT and
fix it, or pick up where I left off?
 A: Unfortunately, you can't go back and change your answer in SRT. The flow will
also not pick up where you left off last, so it's best to finish the post you're on before
closing the browser.

5 l
00 tia
56 ne
33 id
nf
Co