Search Ads NPNC Relevance Rating

Guidelines v5.1

6 l
72 tia
Change log
Revision — 12/12/2022

New: Guideline Updates – Please Do Not Skip Messenger Ads

There has been a high skip rate of Skip: No Landing Page for Santiago/NPNC. Messenger
Ads do not have external links associated with them. While we do want contributors to
skip jobs that don’t render, we do not want the contributors to skip these Ads. See

27 n
examples in Skip section below

Change log
Revision — 09/01/2021
e
• Skip/BPV Label
• Removed invalid BPV labels
• Added many updated BPD labels (Removed prescription drugs as this has
29 id
been removed from FB prohibited. Also, Personal Health, Unauthorized
Streaming Devices, Counterfeit documents, and Surveillance Equipment.
(They might consider some of these illegal products and decided to take it
out as a separate issue.) Added illicit or recreational drugs, body parts/fluid,
adult products. Added Vaccine discouragement, Adult Content,
nf

Misinformation.
• Added Hierarchy examples
• Skip/Ambiguous
• Added extra context to use before choosing ambiguous.
• Correct two job examples incorrectly labeled as MMN and Complements
to PI.
Co

• Added an updated Decision Tree

• Product Intent
• Added real estate agencies as well as rental products due to confusion over
this topic.
• NDPI
• Added non branded food query examples.
• Added information on digital content
• NTMT
• Added many examples for education levels and food queries.
• Substitutes
• Clarified education levels.
 Workflow Goals + Context

The purpose of this workflow is to identify relevant and high-quality ads for search
queries on Facebook SERP (All search tab). For each ad to be rated, raters will see
a search query containing the terms used to search and a sponsored post.

6 l
This sponsored post includes the name of the advertiser, visuals such as images or

72 tia
video, and text commentary. Then, based on the content, raters will rate the ad for
relevance if there is no defined product intent.

The convention of {query} → [the main topic of result] will be used to provide
examples in these guidelines.

27 n
Definitions

The following terms and their meanings will be used throughout this guideline:
e o Query is the text used to perform a search for specific results. The
word(s) or phrase entered into the search engine (e.g. “cat
videos,” “pancake recipes,” “Michelle Obama”.)
29 id
o Result is an item returned by the search engine as an outcome of the
search. Landing page is considered to be a part of result for all Ads
workflows.
o Product is the item that is the subject of the resulting post. Note that
for our purposes a “product” can encompass multiple items for sale in
nf

the same result.

o Searcher is a supposed person who conducts a search on Facebook All
(SERP workflow) or Marketplace (MP workflow) tabs using the given
query and receives the given result.
o Searcher's intent is what the searcher intended or wanted to find
when they typed their query into search bar considering the query
Co

itself, the intention behind those words and the context. The most
traditional query intent classification includes informational (to learn
about something), navigational (to find a specific site) and
transactional (i.e. to buy a car or to download an app) intents.
o SERP (Search Engine Result Page) is the page of search results that a
search engine (e.g. Google or Facebook) returns in response to a
query, i.e. Facebook All Tab SERP or Facebook Marketplace SERP.
o Facebook Marketplace is a digital marketplace where users can
arrange to buy, sell, and trade items with other people in their area.
o Workflow refers to the way we group together a particular type of
search results, depending on what kind of search process we are
interested in examining. For example, in the Ads SERP “workflow,” we
look at only Ads showing in response for queries on Facebook All Tab
SERP, and assess them for relevance.
 o NPNC: no people no commerce.
o DPI: Defined Product Intent
o PI: Product Intent

6 l
72 tia
27 ne
29 id
nf
Co
 Search Ads NPNC Decision Tree

6 l
72 tia
27 ne
29 id
nf
Co
 0 - Skip

SKIP is used when the job cannot be evaluated in Ads Rating workflows because of
one of the following reasons:
• The job cannot be rated consistently with other jobs, i.e. because of technical

6 l
issues.

72 tia
• The job does not belong to Ads Rating workflows, i.e. the query or the result
is not compliant with Facebook policies.

Note: All Skip jobs have a hierarchy, if more than one rating applies we would rate
based on hierarchy. For example, if a job is in a foreign language and the landing
page does not work, we would choose Skip – Foreign Language.
If a job was ambiguous but also had a non-working landing page, we would choose

27 n
Skip - Ambiguous.

e
Job cannot be
Skip sub-label
Gibberish
Ambiguous Intent
Foreign Language
Brief explanation
Rater cannot understand the meaning of the query
Rater cannot understand the intent of the query
If the query or result including the landing page
contains foreign language
29 id
rated consistently
with other jobs Rendering Issues Error message shows up in SRT instead of a job
No Landing Page The result does not include a link to the landing page
All the other tool issues preventing rater from
Other Tool Issues
understanding the query or the result in full
Brand Policy The query or the result do not comply with Facebook
nf

Violation policies
Job does not
The ad has poor quality and may lead to a misleading
belong to Ads Poor Quality Ad
experience for the user
Rating workflows
Poor Quality Landing The ad's landing page has poor quality and may lead
Page to a misleading experience for the user
Co

SKIP/Gibberish Query: Skip if you cannot understand what the query means after
conducting a thorough side search and considering possible search intentions. This
sub-label includes names (not relating to brands, companies, etc.) and queries
that have a very broad interpretation in side search or are not understood by side
search at all.
• Examples: {m}, {123879324509y14}, {Greg},{Chris}
 6 l
72 tia
27 n
SKIP/Ambiguous Intent: Skip if you can understand the meaning of the query but
e
lack enough context to determine an appropriate rating even after you have
thoroughly conducted side searches. This sub-label includes those that direct to
private groups that do not provide enough information in the description, or other
29 id
vague search queries.

If a query seems ambiguous, add the searcher’s location before deciding on a rating.
There might be a specific business/service in the searchers location that is a
dominant intent.
nf

Query Image Reason

Co

This query is vague; it

can be a misspelling too
for “sun coast Spain” but
suncoastspin
we cannot be sure. A side
search reveals no helpful
information
 The query is vague; it
can refer to a popular

6 l
song by Katy Perry, or
making conversational
small talk

72 tia
discourse with someone.
We would need
additional context to
make a accurate decision.

27 ne
I wish for
you
Query might be a song
but side search results are
vague and we need more
context to know user's
29 id
intent.
nf

SKIP/Foreign Language:

For workflows intended for the English language, rate SKIP / Foreign Language if
the query or ad contains any other language AND prevents you from understanding
Co

the content.

Example: {El Farolito} → Farolito is originally derived from the Spanish language
that refers to a small paper lamp, but is also the name of a popular Mexican
restaurant in San Francisco. This would would be OK to be rated and it is expected
that you use side searchers to correctly determine searcher intent.

For international languages, rate SKIP / Foreign Language if the query or ad

contains any other language other than the intended language of the workflow AND
prevents you from understanding the content. For international languages, it is OK
to rate content that contains English.
 If the result including the landing page contains foreign language, and it
prevents you or can prevent your colleagues from understanding the result
consistently and in full, please label the job as skip/foreign language, for example:

6 l
72 tia
27 ne
29 id
SKIP/Rendering Issues: Sometimes an error message shows up in SRT instead of a
job. It happens for various reasons; for example, when an ad has been removed by
the advertiser after the job was created. Skipping is the only way to remove a job
from the workflow so it wouldn't show up again. If a job has both rendering issue
and no landing page, please skip under Rendering Issues.
nf

Please note: It is important to copy the error to the text box when possible, so that we
can track these errors.
Co

Note: If the advertisement contains partially rendered content, continue to rate

normally unless the advertisement is critical to evaluating the job and a rating
decision is not possible otherwise.

Example:

The following ad is missing a product image on the left side. However, the
description “2-Burner” and landing page provides enough information for us to
determine that the ad is selling a propane gas grill/burner. However, since the query
is searching for a product, we would rate this as Defined Product Intent.
 Note: We would still need to check the landing page to ensure it works and does not
require any of the Skip ratings before rating Product Intent.

6 l
72 tia
27 ne
29 id
nf
Co
 6 l
72 tia
27 ne
29 id
nf
Co

SKIP/No Landing Page: Please SKIP the jobs that render but have one of the
following issues:
• Have no link to a landing page as in the screenshot below
Co
nf
29 id
27 ne
72 tia
6 l
 • Have link that does not load when you click on it, so you are not able to rate
the job. We take the landing page into consideration when rating, thus these
jobs cannot be rated consistently with the others.

6 l
72 tia
27 ne
29 id
• Have link that leads to landing page that is not functional. For example: “Page
nf

not found”, “404 error”, etc.

Co
 6 l
72 tia
27 ne
29 id
Note: In some cases you won’t be able to access the landing page content at the
moment of rating. If the external link on a SRT job leads to any landing page
nf

within Facebook domains (fb.me, facebook.com, Facebook newsfeed), these jobs are
not considered misleading and can be rated normally.

Example:
Co

The follow job contains an ad, which has a landing page on Facebook that displays
the error message “Sorry, this content isn’t available right now”. Please disregard
this landing page and continue to rate the job normally. In this example, the
advertisement is selling a screen protector for a Samsung S9 and would be rated as
Defined Product Intent - PI.
Co
nf
29 id
27 ne
72 tia
6 l
Co
nf
29 id
27 ne
72 tia
6 l
 6 l
72 tia
New: Guideline Updates – Please Do Not Skip Messenger Ads

27 n
There has been a high skip rate of skip_reason: landing_page for
Santiago/NPNC. Messenger Ads do not have external links associated with
them. While we do want contributors to skip jobs that don’t render, we do
e
not want the contributors to skip these Ads.

This is an example of a job to skip:

29 id
nf
Co

This is an example of a job to rate- Please do not skip Messenger Ads

 6 l
72 tia
27 ne
29 id
nf

Note: “Messenger” in bottom left corner of image

SKIP/Other Tool Issues: SKIP the job if you cannot rate it because you cannot
Co

understand the result from what you see in SRT, i.e. visuals of the result are cropped
too badly and you cannot comprehend the text on it. If a tool issue can be classified
as a Rendering Issue or No Landing page scenario, use the corresponding sub-labels.
If it can't be, choose Other Tool Issues sub-label and explain your selection in the
text box.

SKIP/ poor quality Ad

First, please note that if the quality of the displayed job in SRT prevents you from
understanding the result, SKIP this job under Skip/Other Tool
Issues or Skip/Rendering Issues whichever seems to be more appropriate.
 Job can be rated as Skip/Poor Quality based on poor visuals in the ad or because
the ad may create misleading experience for the searcher.

Each ad is accompanied by visuals like photos or videos. Some of the visuals may
employ disruptive techniques like flashing screens in videos to attract more

6 l
attention, which goes against Facebook policies and should labeled Skip / Poor

72 tia
Quality Ad. Most common factors creating poor viewing experience
of photos include:
• Lighting effect is too light or too dark
• Has oppressive text, masks, excessive filters that distract from the viewing
experience
Most common factors creating poor viewing experience of videos include:
• Poor stability and wildly moving video (unless the context of the video

27 n justifies it)
• Subtitles are overwhelming and take up the entire screen
• Video of a video
The following factors are not considered to be Poor Quality indicators:
e
• Photo: Resolution is grainy or blurry,
• Photo: Aspect ratio is skewed causing distorted images,
• Photo: Image is cropped badly and makes it visually jarring,
29 id
• Video: Video does not have the correct sharpness, contrast, size, and aspect
ratio for a good visual experience,
• Video: Poor audio or static (unless the context of the video justifies it.)
SRT is aimed to convey the main idea of the ad to rater, but it simplifies the imagery.
Thus, these cases are really tool issues rather than bad quality ads.
nf

Please find some examples below:

Co
 6 l
72 tia
27 ne
29 id
nf
Co

Adfarm.mediaplex.com is used by brands to track where visits are coming from, but
it is perceived as misleading AND does not return an expected page for this ad,
hence this result should be rated Skip / Poor Quality Ad.

Note: Adfarm.mediaplex.com has since been taken down, but the concept still
applies.
 6 l
72 tia
27 ne
29 id
The paragraph states that 5 million people like this product, but in reality, there are
only 5 likes. This should be rated as Skip/ Poor Quality Ad for containing
misleading/false content.
nf

SKIP/ Poor Quality Landing Page

First, please note that if there is no landing page for the ad of the of the displayed job
in SRT, please rate as Skip / Landing Page.

If the landing page of the ad provides a misleading experience for the searcher,
Co

then rate as Skip / Poor Quality Landing Page.

 6 l
72 tia
27 ne
Example of a misleading job, SRT UI screenshot. The post commentary and the
visuals are about Bob Barker, and at first the ad seems to be relevant to the
29 id
query. The landing page is completely on different topic and do not say anything
about Bob Barker. Considering searcher’s intent, this result can create bad searcher
experience, and should be rated as Skip / Poor Quality Landing Page

Job can be also considered misleading and rated as Skip / Poor Quality Landing
nf

Page if the ad visuals do not match the ad commentary. An unrealistic price or

50%+ price difference between the ad and the landing page are also indicators of
Skip/ Poor Quality Landing Page. If the price difference is less than 50%,
disregard the price and rate normally.
Co

SKIP/Brand Policy Violation: Skip the jobs where either query is asking for
Prohibited Content/Sensitive results, or the result can be classified as one of those.

Prohibited Content: Query asks for products from the list below or the resulting
post falls under one of the Prohibited Content categories.
• Prohibited Products:
o Tobacco Products
o Illicit or Recreational Drugs & Related Paraphernalia
o Unsafe Supplements
o Body Parts/Fluids
o Adult Products
o Spyware or Malware
o Circumventing Systems
 o Weapons, including parts, accessories and related uncommon products
o Any Illegal Products and Services
• Prohibited Types of Content in the Ad:
o Profanity or Bad Grammar
o Vaccine Discouragement

6 l
o Circumventing Systems

72 tia
o Adult Content: Ads must not contain adult content. This includes
nudity, depictions of people in explicit or suggestive positions, or
activities that are overly suggestive or sexually provocative. Ads that
assert or imply the ability to meet someone, connect with them or
view content created by them must not be positioned in a sexual way
or with an intent to sexualize the person featured in the ad.
o Sensational Content, including shocking, sensational, disrespectful, or
excessively violent content

27 ne o Misinformation: Facebook prohibits ads that include claims debunked

by third-party fact-checkers or, in certain circumstances, claims
debunked by organizations with particular expertise. Advertisers that
repeatedly post information deemed to be false may have restrictions
placed on their ability to advertise on Facebook.
o Misleading or False Content: deceptive, false, or misleading content,
including deceptive claims, offers, or methods
29 id
o Controversial Content: content that exploits controversial political or
social issues for commercial purposes.
o Third-Party Infringement: content that infringes upon or violates the
rights of any third party, including copyright, trademark, privacy,
publicity, or other personal or proprietary rights.
nf

o Personal Attributes and Discriminatory Practices: content that

discriminates or encourages discrimination against people based on
personal attributes such as race, ethnicity, color, national origin,
religion, age, sex, sexual orientation, gender identity, family status,
disability, medical or genetic condition or content that asserts or
implies personal attributes.
Co

• Financial:
o Payday Loans, Cash Advances and Bail Bonds
o Multilevel Marketing
o Penny Auctions
o Prohibited Financial Products and Services such as initial coin offerings
and binary options
• Sensitive Content: Query is related to one of the topics from the list below
or the topic is present in the resulting post:
o Violence or Calls to Violence: any weapons, including weapon parts
and accessories, display/antique weapons (e.g. display/antique
swords)
o Sexual Content and/or Nudity
o Child Exploitation and/or Nudity
 o Human Trafficking
o Acts of Terrorism
o Self-Injury
o Suicidal Content
o Hate Speech

6 l
o Bullying and Harassment

72 tia
Please note, if the category is Child Exploitation and/or Nudity, escalate the
Job ID to your lead immediately.

The full list of Prohibited Content (except Sensitive Content) items and detailed
explanations can be found
here: https://www.facebook.com/policies/ads/prohibited_content. Please note, for
the purposes of Ads Rating workflows we do not use policies related to Restricted

27 n
Content.

Note: #15 of the advertising policy (non-functional landing page) should be rated as
e
SKIP/No Landing Page as instructed in the Skip section
29 id
nf
Co
 Product Intent vs. No Defined Product Intent

In order to determine if the query has no defined product intent (NDPI), as

opposed to product intent (PI), please conduct the side search and check products
on Google Shopping Tab or bing.com/shop. Relevance rating will require a deeper

6 l
understanding of the query and side search in “All” Tab, but possible product intents

72 tia
are the key to determine if the query is NDPI or PI.
product intent

NPNC workflow doesn’t rate Product Intent jobs. A good way to determine if a query
has product intent is to ask yourself if any searcher would look for the results
displayed in the side search using the given query. If the answer is “Yes”, and the
side search reveals an intent for specific product or product category, please label

27 n
the job as “0 - Defined Product Intent”. No further relevance rating is required for
DPI queries. Here are some examples:
e
29 id
nf
Co

Product query {iPhone XS} has product intent for [iPhone XS], search query {285
75 15} has strong product intent for [tires.] If all the products in the side search
belong to 1 or 2 products or product categories, the query has product intent for
these product(s)/product categories. If the query mentions features or
characteristics of the product, i.e. {iPhone XS 128GB}, it should be taken into
consideration, and it also narrows the intent. Chances that [iPhone] is an expected
result for query {iPhone} are very high.
#productintent
 6 l
72 tia
27 ne
29 id
nf

Brand queries like {Apple}, {because cosmetics} can be matched with

the products and product categories associated with the queried
brand and have product intent. Side search results make sense and are expected
considering the query.
#productintent
Co
 6 l
72 tia
27 ne
29 id
Query for specialized retailer {footlocker} has product intent for [active shoes].
Side search results are focused on one product category and this product category is
nf

expected considering the query. There are other search terms to look for shoes, but
at least some searchers can use this query to get these results.
#productintent
Co

Query is for “get yarded kosciusko”, according to a side search, the query's intent is
for a company that sells yard signs. Because this is a specialized retailer that sells
 products which fit into a narrow category, the Product Intent label should be
selected.

Additional Examples of specialized retailers: {David’s Bridal} as a retailer that

specializes in bridal and formal clothing. {sports wear} has product intent for a

6 l
specific category of products despite being broad. {Bass Pro Shops} or {Cabela’s}

72 tia
would also have product intent for a sporting goods store.
{car dealerships} have product intent for the primary intent is to purchase or lease a
vehicle. While some may include service departments, this is not the primary intent
of the query unless the query includes services.

27 ne
29 id
nf

The query is for a flower shop located in Nebraska. Because this is a specialized
retailer that sells items which fit into a narrow category, the Product Intent label
should be selected.
Co
 6 l
72 tia
27 ne
The query is {hunter's ridge} that according to side search the user is looking for an
29 id
apartment complex and it seems that it's looking for apartments to rent. Similar to
queries like {House for rent} or {House to buy} it has a commerce/product intent
and it should be rated as PI.

Queries for rentals such as cars, snowmobiles, and ATVs would also be considered
nf

Product Intent.

Real Estate Agencies are considered Product Intent. They have a commercial
intent.
Co

Note: Items such as books, games, apps etc. that can be downloaded are considered
digital content and should be rated NDPI.
 6 l
72 tia
27 ne
Query is for specialized retailer {buy buy baby} has product intent for baby and
kids items and this should be rated as #product intent
29 id
Query is for {numa demi grow} side search results are focused in a product category
nf

for stroller. This should be rated as Product intent.

Co
Co
nf
29 id
27 ne
72 tia
6 l
 6 l
72 tia
27 ne
29 id
nf
Co

Query is for {the bookery} and side search result is for a used book store and it
should be rated as product intent.

Query is for {journeys} and the side search for shopping is showing a specialized
retailer that is selling a type of shoes. This should be rated as Product intent.
 6 l
72 tia
27 ne
29 id
nf
Co

Please note:
• Most queries in this workflow should not have product intent (PI). If there is
defined product intent, raters should select “0 - Defined Product Intent”, and
the job is considered complete with no relevance decision needed.
 • If a company/brand offers both services and products, it should be
considered as No Defined Product Intent.
o {the sewcial club}: query intent is for a shop which offers products as
well as services (sewing lessons/classes/alterations)
o Exception: When the query is seeking a specialized retailer that offers

6 l
services (installation, repair, etc.) of their product, rate as PI because

72 tia
the services are specifically tied to the product of the retailer. For
example, a query for a car dealership and the dealership also provides
repair services.
▪ Example: {mckee automotive llc} – [Branded Bills] – query is
seeking an Auto repair shop in Monmouth, Illinois.
• Queries for rentals such as cars, snowmobiles and ATVs would also be
considered Product Intent.
•

27 n
•
e Real Estate Agencies are considered Product Intent. They have a
commercial intent.
Items such as books, games, apps etc. that can be downloaded are considered
digital content and should be rated NDPI.

No defined product intent

29 id
If you can easily think of much better queries to search for the results that you see in
the side search and the side search results are not focused on a specific product or
product category, rate the job as “1 - No Defined Product Intent” and proceed to
relevance rating. For example, the query is {Seattle}, the result is [a t-shirt with
depiction of the Space Needle]. The most reasonable product query to find this
nf

result is {space needle t-shirt}, and no product result including [a t-shirt with a
depiction of the Space Needle] can fully meet searcher’s intent. This should be rated
as Somewhat Relevant - Not the Main Topic. Products posted by the queried retailer
would all be Relevant/Meets most Needs
Co
 6 l
72 tia
27 ne
29 id
Broad retailers typically carry a wide range of products and product categories,
nf

and cannot be clearly associated with only one product category. Query for a broad
retailer {Walmart} has navigational intent for [www.walmart.com] or [Walmart
page/group on Facebook.] Chances that any searcher would intentionally search for
a [tent] or an [Apple iPhone 6] using the query {Walmart} are very slim.
#nodefinedproductintent
Co
 6 l
72 tia
27 ne
29 id
Search query {Chicago} primarily has informational intent to learn more about the
city. Product interpretation of this query is vague and side search returns topically
nf

related souvenir products at the top of shopping results which typically is a good
indicator of no product intent. In this case, if the suggested as result was topically
related souvenir products, you should rate that as Somewhat Relevant /Not the
Main Topic.
#nodefinedproductintent. Additional examples of broad retailers: {Aldi}, {CVS},
{Target} would have no defined product intent.
Co
 6 l
72 tia
27 ne
29 id
Similar to {Chicago}, query {nicki} primarily has informational intent to learn more
about the rapper and no specific product intent. Side search returns t-
nf

shirts. #nodefinedproductintent
Co
 6 l
72 tia
27 ne
29 id
Broad product categories: Queries {hair}, {Christmas}, {moving} can be associated
with some products but do not make the expected product clear and may have other
interpretations than the product intent one.
nf

A searcher might expect the products returned by the side search, but the query is
not strongly focused on one of them, and there are much better search terms to look
up any of the returned results (for example, “hair extensions”, “blond wig”, “dry hair
treatment”, “hair regrowth treatment” etc.)
Co

#nodefinedproductintent
 6 l
72 tia
27 ne
29 id
Food and Drink Establishments: query {Aztecan Mexican}, the query’s intent is for
a restaurant with this title, likely located near the searcher. Food and drink
establishments have no product intent, as they are informational queries.
#nodefinedproductintent
nf

Note: Specific food and drink queries such as {skittles}, {folgers}, {blue buffalo} to
have Product Intent as those brands cannot truly be associated with a service, but
refer to purchasable/tangible products

Non branded food queries such as “No Bake Cookies” or “Coffee” are not considered
Co

Product Intent. These are informational queries and as such are rated NDPI.
 6 l
72 tia
27 ne
29 id
Authors and public figure/celebrity: query is for a specific author (Nicholas
Sparks) and result is for a book of that author. The primary intent would be
nf

information, articles, etc. Even though the result is a book from the Author, there is
no product intent and would be rated as Relevant - Meets Most Searcher’s
Needs. #nodefinedproductintent

Please Note:
Co

• Most queries in this workflow should not have product intent (PI). If there is
defined product intent, raters should select “0 - Defined Product Intent”, and
the job is considered complete with no relevance decision needed.

• If a company/brand offers both services and products, it should be

considered No Defined Product Intent.

• {the sewcial club}: query intent is for a shop which offers products as well as
services (sewing lessons/classes/alterations)
 6 l
72 tia
27 ne
Animals: Query for {dogs}, {cats}, {tigers} as well as dog breeders has an intent for
information rather than an intent to purchase the actual animal. This would fall
under NDPI because an animal is not be a defined product., but a living being.
29 id
Side Search for not defined product intent queries

Please conduct a thorough side search on “All” Search Tab. It is important to

nf

conduct the side search with and without searcher’s location as the same search
terms may refer to different businesses based on location. For example:
• {golden boys} is primarily related to novel by Australian author Sonya
Hartnett
• {golden boys Austin} refers to a musical band
• {golden boys San Francisco} refers to a pizza restaurant
Co

• {golden boys Fitzgerald GA} refers to a food wholesale company

Note: Location only queries such as {Las Vegas} is vague and depending on the
result, different approaches would be applied.
• {las vegas} -[services in las vegas] - Relevant/Meets Most Needs
• {las vegas} - [services in miami] - Off Topic/Not Related
• {las vegas} - [poster of las vegas] - Somewhat Relevant/Not The Main Topic
• {las vegas} - [poster of miami] - Off Topic/Not Related
 Relevance

Generally, if you are on the fence between choosing Off-Topic / Not Related or
Somewhat Relevant / Not The Main Topic for a query and ad pair, the guidance
is to select Somewhat Relevant / Not the Main Topic.

6 l
72 tia
1 - Off-Topic
Off-topic / not Topically related

The ad in the result is not related to query’s intent at all and does not look like a
natural or reasonable response to the query. The result may or may not match
keywords, but if it does, the match is in a random or unintelligible way. In other
words, the result should be rated as Off-Topic when the result does not make sense

27 n
considering the query and there are no chance that any searcher would be
interested to learn more about this result.

Showing this ad within the top search results for this particular query would
e
provide a negative search experience. In other words, if you were to place yourself
in the shoes of the searcher, the suggested ad would not serve any useful purpose
and you would rather not see an ad at all.
29 id
• Random word match:
o {Fox News} → [domesticated foxes]
o {land} → [informational article on land rover vehicles]
o {leanne morgan} - [Bed Bath & Beyond - Leanne 800-Thread-Count
Cotton Blend King Sheet Set in Beige] - query is informational for a
comedian and the result is a keyword match to the comedian’s first
nf

name only and not related to the sheets.

• Complete not related:
o {findamanda} → [MCO RodBuckle]
o {cedar creek} - [abcmouse.com]
o {tanners custom} – [Coco & Eve - Sunny Honey Bali Bronzing Foam] –
query is informational for Tanner's Customs & Collision an auto body
Co

shop in Kansas City, Missouri and the result is for a bronzing foam.
o {masterchef Australia} – [Zebra] – query is seeking information for a
reality TV series and the result is for Zebra, a solutions provider for
businesses.
o {mountain view vintage market 2020} – [Blissy pillowcase] – query is
seeking information for a market/festival and the result is for a
pillowcase.
• Category “Cities of the U.S.” matches, but there are no other common topics and
there are no chances that any searcher’s need could be met:
o {city of dallas georgia – government} → [promotion by ‘Visit Las
Vegas!’ Facebook page]
 o {pollock louisiana} - [[Visit Renton] - query is for a town in Louisiana
and the result is for a city in King County, Washington. The locations
are not related to each other and are not substitutes or mismatches.
o {vevay indiana} - [Explore Wayne County Huntington Area CVB] -
query is for a town located in Jefferson Township, Switzerland County,

6 l
Indiana and the result is to Explore Wayne County Huntington Area.

72 tia
This query/ad pair are not topically related.
• Generally, if you are on the fence between choosing Off-Topic / Not Related or
Somewhat Relevant / Not The Main Topic for a query and ad pair, the guidance
is to select Somewhat Relevant / Not the Main Topic.

Note: word match is not always necessary for topical relevance, especially for no
defined product intent queries. For example:

27 n
• Topically related:

e o {bad credit} → [Get Your Next Used SUV Today Regardless of Credit
Score. Warranty Included and a Carfax History Report]: While the
product intent of the query {bad credit} is broad, the result is a
meaningful response to the query and the searcher might be
interested to learn more about it. “Bad credit” and “regardless of
credit score” are not strictly synonyms, but “regardless of credit
29 id
score” implies the idea of “bad credit” in given context.

2 - Somewhat Relevant
nf

Somewhat Relevant is used when the ad result is loosely related to the query, if
the ad result can satisfy at least one intent of the query, or if the query and ad
result pairing makes any loose logical sense.

In some cases, a Somewhat Relevant ad result is of interest to the searcher

Co

considering the query, but does not fully meet the searcher’s need. For example, it
can possibly satisfy the same need with another product or service, or it can be
helpful to the searcher in any other quality, but it might be not the exact match and
not fully meet most of the searchers’ intent.
Somewhat relevant / Not the main topic

The query and the result (including the landing page) have a topic in common, but
this topic is not the main focus of the ad and/or the landing page. If the ad result
can satisfy even one possible intent for the searcher, the query and ad pair
should be rated somewhat relevant at minimum.

• Topic is not the main focus of the ad (including the landing page): The
query and the result (including the landing page) have a topic in common,
 but this topic is not the main focus of the ad and/or the landing page.
Somewhat Relevant includes:
o Other search engines that focus on searcher’s intent clarification
o Pages that require sign in/sign up or enter contact information
that focus on collecting data. This also includes audible books where

6 l
you need to log in in order to see product description.

72 tia
▪ {healthy instant pot recipes} → [audible subscription including
a book on healthy instant pot recipes]
▪ {Las Vegas} → [Las Vegas Getaway ad - which requires user to
enter personal information in order to see more information]
▪ {Buffalo Wild Wings} → [Get $12 off your first Buffalo Wild
Wings delivery order - Enter your email to get started]
▪ {mcdonald's} → [free mcndonald's meal - Enter your email]
• Note: If the ad shows a landing page with an

•
27 ne app download, it should be rated as normal
o {color by number} → [Color by number app
download] - Relevant/Meets Most Needs
o All the other scenarios when the queried topic or reasonable
substitutes are not the main focus of the ad and the landing page.
Resulting ad’s topic is not the primary intent of the query: Query and
result have word match, but the result is not the most common
29 id
product/service that user would search for with the given query, but could
be of some interest.
o {John Mayer} → [John Mayer electric guitar]: Query is for a public
figure/celebrity, in this case a singer, songwriter. The primary intent
would be information, articles, and videos of this celebrity. The
nf

resulting ad is an item (a guitar) from the celebrity’s product line,

which would be a secondary intent and may not be exactly what a
searcher is looking for, but could be of some interest.
o {Morgan Jones - a fictional character} → [Morgan Jones (The Walking
Dead) Standup/Cardboard]: Information related directly to the
queried character Morgan Jones would be the primary intent, thus, a
Co

result depicting the queried character like a standup/cardboard

would be considered of potential interest but not a direct match.
o {Housekeeping services} → [Hotel cart housekeeping sold by
Walmart]
• Query and result are under the Same Category but it’s not exactly what
the user is looking for but it’s still related. Here are some examples of
category that query and ad can both fall into, thus should be rated as
Somewhat Relevant:
o Education: Please note that for the Somewhat Relevant- Not The
Main Topic label, we do not take education level differences into
consideration.
▪ {MBA Berkeley} → [Graystone elementary school]
▪ {Miami school district} – [Texas University]
 ▪ {Greystone Elementary} – [Drivers Education Course]
o Sport & Fitness: If a query related to sports returns an ad that is
related to sports, but is not the same sport/level of play, rate as
Somewhat Relevant.
▪ {triway girls basketball} → [girls soccer tram club]

6 l
▪ {nmu wildcat volleyball} - [Northeast Wiffle Ball League]

72 tia
o TV & Movies: Query is for a TV show and ad is about a children TV
show, they are in the same category.
▪ {ball in the family} → [They're a Modern Stone-Age Family! The
Flintstones Quiz!]:
▪ {reality show} → [cooking show]
o Food & Drink
▪ {brothers pizza parlor} → [2-Ingredient Dough Pizza by Tasty]
▪ {Tasty Videos} → [A mouth-watering video posted by someone

27 ne of their gigantic burrito with the caption “tasty”]

▪ {McDonalds} – [Pizza Ovens]
▪ {Denny’s} – [Mexican Restaurant]
▪ {Japanese Steakhouse} – [Lipton Tea]

Note: If a query and an ad are both for restaurants but have different cuisines, we
would rate this Not The Main Topic.
29 id
▪
o Music & Audio
▪ {Emily king} → [an ad for a concert subscription]: query intent
is for an American singer, the topic of the ad is not directly
related to Emily King but they are in the same category.
nf

o Home & Garden

▪ {Kopkey manure management} → [Manna Irrigation ad: “A
new technology in the irrigation industry, Manna provides a
software only, sensor-free irrigation recommendations
solution that is based on satellite technology“]: The query and
the result are not directly related to each other but are
Co

acceptably topically related and in the same category.

o Music/Comedy/Entertainment: If a query is looking for a specific
comedian, musician, or other performer, any ad for a ticket distributor
should be considered at least Somewhat Relevant - Not the Main
Topic. If the ad has information related to that exact individual/event,
you should rate that as Relevant - Meets Most Needs.
▪ {Kip Moore} → [Seat Geek Ad that shows John Elton tickets and
Silver Snakes Tickets.]
Query Category Ad Rating Reason
The query is ambiguous, but
the seacher is most likely
looking for a specific gym of
Somewhat the same name in Horsham,
universal pro Services//Sports & Relevant - Pennsylvania. This ad is still

6 l
gym Recreation//Gyms Not the related to gyms (marketing for
Main Topic someone creating their own

72 tia
gym), but not an ad for that
specific gym nor a similar
substitute.
The searcher is likely looking
for a specific tv show based
Content//Movies, off of the popular children's
Somewhat
Television & novel. This ad is related to tv
the last kids Relevant -

27 n
on earth

e Web//TV Shows &

Programs//TV Sci-Fi
& Fantasy Shows
Not the
Main Topic
shows (a list of the top 10 tv
shows of the decade) but is
not for that specific TV show
nor for similar tv shows (eg.
children's shows).
The searcher is likely looking
for a specific tattoo shop in
29 id
Lakewood, NY. This ad is
wild side Somewhat related to tattoo supplies for
tattoo and Relevant - professionals, which is
Tattoos
piercing Not the related, but not an exact
lakewood ny Main Topic match. An exact match would
be an ad for that specific tatoo
nf

shop or a similar tatoo

shop/service.
The searcher is likely looking
for information to learn more
about the specific
Co

actor/comedian. The ad is for

an article about acting roles
Somewhat that famous actors regret
Famous Relevant - taking. This is related since
ronny chieng
comedians/actors Not the the topic is the same (famous
Main Topic actors), but there are other ads
that would be more relevant
(eg. an Ad about Ronny
Chieng's upcoming tv show,
comedy skits, merchandise,
etc.)
 The searcher is looking for a
specific financial advice
website that provides
information on personal

6 l
finances. The searcher could
be interested in learning more

72 tia
Somewhat about accounting or looking
the penny Relevant - for account services. The
Finance Advice
hoarder Not the suggested ad is about a
Main Topic company that provides
accounting services to both
individuals and businesses,
which is somewhat relevant.

27 ne A relevant ad might be one

that displays other personal
finance websites or services

The searcher is looking for a

specific school in Dominican
29 id
Republic and the ad is for the
Somewhat
University of Mississippi.
carol morgan Relevant -
Education This is not considered a good
school Not the
substitute but it's still under
Main Topic
the same category so it should
be rated as SR/Not the main
nf

topic
Co

The searcher is likely looking

for information to learn more
about a specific musician
Somewhat
Peter Murphy. The ad is for a
Relevant -
peter murphy Music & Audio song by a rock band The
Not the
Velvet Crowns. Even though
Main Topic
ad is not a match, it's still
related since the topic is the
same - Music & Audio
 6 l
The searcher is looking for a
mobileapp puzzle game. Ad

72 tia
shows a website called Skillz,
Somewhat
which focuses on mobile
Relevant -
two dots Games gaming tournaments. Even
Not the
though the ad does not focus
Main Topic
on downloading mobile game
apps, is still related to the
query on the topic Games

27 ne The searcher is likely looking

for information about the
29 id
music artist Dobey Dobe, who
sings rap and hip hop. The ad
Somewhat
is for a song by a different
Relevant -
dobey dobe Music & Audio artist - 2Four, who also sings
Not the
rap and hip hop. While query
Main Topic
and ad are not an exact match,
nf

they belong to the same

musical genres thus should be
rated as SR
Co

The searcher is looking for a

restaurant which sells subs
Somewhat and sandwiches. Ad shows an
obee's sub Relevant - online bakery which sells
Food & Drink
shoppe Not the variety of sweet treats. Even
Main Topic though they are not a match,
but still related on the topic
Food
 6 l
The searcher is looking for a
Rock band called nickelback

72 tia
and the result is the instagram
Somewhat
account from another musical
Relevant -
nickelback Music & Audio group. It's not consider a good
Not the
substitute but it's under the
Main Topic
same category and in some
way it can satisfy the user
needs.

27 ne
29 id
The searcher is likely looking
for information about the rock
Somewhat
band New Found Glory. The
new found Relevant -
Music & Audio ad is about a duo musical
glory Not the
nf

artist FARR. They are not a

Main Topic
match but in the same
category Music & Audio.
Co

The searcher is looking for a

personal trainer called Kathy
Somewhat Smith and the ad is about an
Relevant- article of Novartis telling a
kathy smith Sport & Fitness
Not the real story of a son and her
main topic mother and how having a
personal trainer helped her to
keep his mother’s mind limber
 The query is looking for The

6 l
Entrepreneurship Center at
Washtenaw Community

72 tia
College which is a resource
Somewhat hub that supports individuals
entrepreneurs
Relevant- in developing their
hip center at Entrepreneurship
Not the entrepreneurial capacity. Ad
wcc
main topic shows a training for business
owner on how to monetize
online. They are not an exact

27 ne match but still related on the

topic Entrepreneurship.
29 id
User is looking for the street
racing channel that it covers
everything related to Street
Somewhat
Racing, interviews, videos etc.
street racing Media (Videos, Relevant -
nf

The ad is for the Official

channel books, etc) Not the
Facebook Page of the FIM
main topic
Motocross World
Championship that it's under
the same category.
Co
 The searcher is looking for
Brian Mcknigh that is a

6 l
Somewhat singer- songwriter. Ad is for a
brian Relevant - list of pop music to be used

72 tia
Music & Audio
mcknight live Not the while workout. It should be
main topic rated as somwhat relevant/Not
the main topic as they're both
under the same category.

27 ne
Entertainment
The searcher is looking for a
festival called the lights fest
Somewhat and the result is for a different
29 id
Relevant - type of event (concert) but
the lights fest (concerts, festivals,
Not the they're both under the
events etc)
main topic entertainment category so it
should be rated as SR/Not the
main topic
nf
Co

The searcher is looking for a

specific venue called "The
Somewhat
Party Room" in Chicago and
the party Relevant -
Event Planning we can likely assume that the
room chicago Not the
searcher is interested in
main topic
booking a room for event
planning.
 6 l
The searcher is looking for a
specific school district in New

72 tia
niskayuna Somewhat York and the ad is for the
central Relevant - Baylor University. This is not
Education
school Not the considered a good substitute
district main topic but it's still under the same
category so it should be rated
as SR/Not the main topic

27 ne The query is ambiguous, but

29 id
the seacher is most likely
Somewhat looking the horror movie or
Media (Videos, Relevant - the game. The Ad is an
dead matter
books, etc) Not the audiobook of horror theme, so
main topic it's under the same category
but not a good substitute for
nf

the searcher.
Co
 6 l
The searcher is looking for
cooking videos and the ad
Somewhat

72 tia
shows a recipe for a fruit
Cooking Media (Videos, Relevant-
drink.Even though it's a recipe
videos books, etc) Not the
specifically it seems at least
main topic
tangentially related to cooking
videos.

27 ne
The searcher is looking for a
29 id
Sandblasting service
(painting) for different type
Somewhat (pools, boats, cars) and the
cape cod
Relevant- result is painter home
dustless Services
Not the improvement. It's under the
blasting
nf

main topic same category or type of

service so it should be rated as
Somewhat relevant not the
main topic.
Co
 6 l
User is looking for an

72 tia
American disc jockey and the
Somewhat
ad is for a youtube video
Relevant-
romanthony Music & Audio sonmg from Rockie Brown.
Not the
Both are under the same
main topic
category and it should be rated
as SR-NTMT

27 ne
29 id
The searcher is looking for the
region sports network that it's
a channel to watch live video
Somewhat broadcast games and the result
nf

region sports Relevant- is about an article talking

Sports
network Not the about FanDuel's sportsbook
main topic that it's an online sports
betting center. They're both
under the same category and it
should be rated as SR-NTMT.
Co
 6 l
72 tia
The searcher is likely looking
Somewhat for dance lessons. The ad is an
Relevant- online course for dance
dance lessons Sports
Not the teachers, it's still related to
main topic dancing but not a match or a
substitute.

27 ne
29 id
nf

User is likely looking for

facebook page name Shelby
Somewhat Lynne Art, which specializes
shelby lynne Relevant- in oil painting and pencil
Co

Art
art Not the drawings. Ad shows digital art
main topic of Khalif Kelly, which is not a
good substitute but still related
regarding the topic Art.
 6 l
The seacher is most likely
looking for a specific pizza

72 tia
Somewhat restaurant. This ad is related to
Relevant- pizza (marketing for a pizza
lilys pizza Food & Drink
Not the franchise opportunity), but not
main topic an ad for that specific
restaurant nor a similar
substitute.

27 ne
Query likely looks for
29 id
information, music albums or
concert tickets of the rock
Somewhat band Nickelback. Ad shows a
Entertainment
Relevant- website which sells
nickelback (concerts, festivals,
Not the concerts/events tickets which
events etc)
nf

main topic also include concert tickets of

this band. It's not a match but
still related on the topic
entertainment/concerts/events
Co

The query has a unfamiliar

spelling of "dog" and could
Somewhat refer to other meanings such
Relevant- as slag for "male companion."
dawg Animals & Pets
Not the Thus, we cannot infer that pets
main topic in the exact intent of the user,
but this does satisfy one
possible intent.
 The query is likely looking to
Somewhat
buy something from that
Relevant-
hvyee Grocery Store specific grocery store, which

6 l
Not the
can possibly also be brought
main topic
on Ebay.

72 tia
Somewhat Relevant/Location sensitive

27 ne If user is explicitly looking for a service (restaurant, grocery, cinema, etc.) in a

specific location and the ad is in a different location > 100 miles radius from the
specified location, rate as Somewhat Relevant. If the ad is an online service, then
location does not matter.
• {sushi restaurants in Mountain View} → [Top 5 restaurants in New York]
• {tugwell roofing California} → [roofing service in Texas]
• {motes pools Arizona} → [Solstice pools Houston]
29 id
• {first steps education lakeland} → [early childhood center in Weston, Florida
- which is more than 100 miles from the user location]
Note:
• If the user is looking for a service in a specific location but it’s a brand chain
that has multiple branches and the ad is from a different location, in this case,
nf

the distance doesn’t matter, thus rate as relevant.

o {Trader Joe’s San Jose} → [New product launch in Trader Joe’s San
Diego] → relevant
• If user is looking for a specific school but the ad is an online program of a
similar education, in this case location should not matter (education
category).
Co

o {MBA Berkeley} → [Online MBA in Columbia] → relevant

• If user is looking for a specific photography service, ad shows a substitute
service more than 100 miles away, rate as somewhat relevant - location
sensitive.
o If the substitute service is more than 100 miles away and the
photographer is willing to travel worldwide, please rate as Somewhat
Relevant - Substitutes.

Somewhat relevant / COMPLEMENTS

The query is looking for a specific service/games/videos/informational websites

etc. and the shown ad is not an exact match or does not completely satisfy the user's
 need, but can still be used for that service. The ad shows a product or service that
can used jointly and improve overall experience.
o {Dean Smith photography} → [picture frame]
o {Yoga Classes} → [Yoga Mat]
o {Blue Horsehoe Tattoo Hampton} → [Revitalize Tattoo Aftercare]

6 l
o {Ski Resort in Tahoe} → [Daily equipment rental to ski]

72 tia
o {ads lawn & landscaping llc} → [cutting lawn chemicals]

27 ne
29 id
nf
Co
 6 l
72 tia
27 ne
29 id
nf

Somewhat relevant / Substitutes

The query is looking for a specific service/games/videos/informational websites

etc., and the ad result is not an exact match, but can still satisfy a similar need or
desire. This category may include proper name/location mismatch, but still has the
same general operation.
Co

o {lion's tail brewing co.} - [McFleshman's Brewing Co.]

o {joes pizza ann arbor} → [Guido's Pizza]
o {national life group} - [Colonial Penn Life Insurance]
o {Trader Joe’s} → [Hello fresh black Friday offer meal prep delivery]
o {pappadeaux seafood restaurant} → [Hancock Gourmet Lobster -
delivered to your doorstep]
o {mosaic coffee company} → [Grounds & Hounds Coffee & Co.]
o {this is us tv show} → [outlander tv show]
Note: If the service location is specified in the query, ad shows substitute service
more than 100 miles away, then rate as Somewhat Relevant as instructed in
Somewhat Relevant section.
 Note: For content related to TV Shows/Movie/Entertainment, raters should
consider substitutes within the same genre as relevant.
Examples include:
• {Fortnite} → [Any third-person shooter game]
• {Golden State Warriors vs. Rapters} → [Celtics training camp]

6 l
• {Hawaii Five-O} → [Law and Order]

72 tia
Exception: Only content related to entertainment as mentioned above can be
treated as Substitutes.

Name queries for artists/celebrities/etc., would not have substitute results, even if
in the same category.
• {angelina jolie} - [nicole kidman article] would be rated as Somewhat
Relevant/Not The Main Topic due to the category relation to actors but is not

27 ne a substitute to query’s primary intent

• {angelina jolie comedy movies} - [nicole kidman comedy movie] would be
rated as Somewhat Relevant/Substitutes due to being a query for
entertainment content {comedy movies} and belonging to the same genres.

Note: Queries for a specific local newspaper or for a specific local radio station
would not have Substitute results but can have a categorical connection and be
rated as Somewhat Relevant/Not The Main Topic.
29 id
• {tampa bay times} - [shreveport times] - while the query and result are both
local newspapers, they would not be reasonable substitutes as the topics
between the queried and returned newspaper focus on different locations
and are mismatched. These query-result pairs would not be of much interest
to a user, but due to the categorical connection (newspapers) would be
nf

Somewhat Relevant/Not The Main Topic.“

Education queries: If a query is searching for a specific education range such as a

K-12 school and the returned Ad is for a school offering grades 1st through 5th, we
would rate this as Not The Main Topic. Education levels must match to be
considered a good Substitute. The Ad can offer more education such as a query for
Co

grades 1st – 5th but the ad offers 1st – 8th, this would be considered a good substitute
as it covers the queried range.

3 - Relevant

The Relevant label is reserved for ad results which have a high logical connection
with the searcher’s query and is very likely to match the searcher’s intent. In some
cases, the ad result may be the exact or perfect fit to the searcher’s query.
Relevant / MEET MOST SEARCHER’S NEED

The ad result would likely match most of the searcher’s intent for a given query.
 There is a high chance that the searcher would have a continued interest in the
result and learning more detail about ad's main topic. In many cases, the ad result
matches the exact business/location/service that the searcher is looking for, is a
synonym or near synonym.
• Result that responds directly to the query:

6 l
o {disney} → [Disney cartoon theme event]: Category TV & Movies, the

72 tia
results looks very natural considering the query.
o {coffee} → Seattle's Best Coffee‘s post [We’re
celebrating #NationalDonutDay on June 7 with free samples of
Seattle’s Best Coffee® and Walmart Freshness Guaranteed donuts!
Click to find events at your nearest Walmart]: category Food & Drink,
the result naturally responds to the query.
• Common products reasonably related to the query can all be Relevant.
o {hair} → [hair extension/shampoo/hair iron/hair color]:

27 ne category Fashion & Style, the result is very reasonable considering

the query.
o {garden} → [garden plants/soil/water hose/pruner]: category Home
& Garden, the result belongs to broad category pointed by the query.
o {Nicki Minaj} → [Nicki Minaj music CD]: category Music & Audio, the
result is a natural response to the query and can be helpful for many
searchers.
29 id
o {Charlotte NC} → [Real Estate Listings in Charlotte, NC]

Clarifications
nf

1. Always check to see if there is an abbreviation in the query, which can

provide more insight into the searcher’s intent. Conducting a side search can
usually provide some guidance on what the abbreviation is referring to.
a. Example: Query {city reia}
Co
 6 l
72 tia
27 ne
29 id
nf
Co

b. A quick side search reveals that the term “reia” is an abbreviation for
“Real Estate Investors Association.” Furthermore, the searcher is
likely using the term “city” to refer to his or her present location (in
this case, St. Louis, Missouri). The likely intent is that the searcher is
looking to get information on the St. Louis Real Estate Investors
Association. The ad by virtual partners for a webinar “Opportunity
Zones” provides information on a specific type of real estate investing.
This ad would be rated Somewhat Relevant - Substitute.
 2. Quick Reference Guide
a. Skip - Gibberish Query: The meaning of the user’s query
b.

6 l
72 tia
FAQ/Troubleshooting
Q: I can't get into the queue!

A: There are many reasons you might not be able to enqueue. First, check the group

27 n
chat to make sure that your workflow is still using that queue. Try to enqueue a few
times, and make sure your internet connection is steady. If you still can't get
in, screenshot the error and send it to your lead.
e
Q: I got a maxed out error.

A: This might mean that either you've done all the necessary work OR there's an error
with the queue. Screenshot the error message and send this to your lead to see what
29 id
you can do next.

Q: I accidentally submitted a job before I was done. Can I go back in SRT and fix
it, or pick up where I left off?
nf

A: Unfortunately, you can't go back and change your answer in SRT. The flow will
also not pick up where you left off last, so it's best to finish the post you're on before
closing the browser.
Co