FBE Relevance PRP Guideline [v3]

Context
The survey is designed to measure the off-topic prevalence for FBE content which contains organic business posts
and sponsored ads. FBE is triggered when a user shows interest in the current organic post or the current ad, the
system recommends a set of related content to the user. We expect that the recommended content is contextually
relevant to the original post.

Each post or ad has:

● the name of a business/advertiser

● images or video
● text

Based on the content, raters will decide if the suggested post/ad and given organic post/ad are related. More
specifically, we will ask the rater to rate 4 possible post / ads pairs:

● <organic business post, organic business post>

● <organic business post, ad>
● <ad, ad>
● <ad, organic business post>

The first item in the pair is called original post/ad, and the second suggested post/ad. The goal is to measure how
related the suggested post or ad is to the original post.

● Example of original post is organic business post, and suggested one is organic business post
● Example of original post is organic business post, and suggested post is an ad:
Besides, we ask the raters to rate the image quality of the second/suggested post based on the guideline in Q2 —
Image quality question.

Clarifying instructions:

● For business posts, raters will rely on:

■ the business page name
■ text and images and videos from the post to extract the intent (like
product/service/content)
● For ads, raters will rely on:
■ the advertiser name
■ text, images and videos from the ad (to extract intent/product/service).
● For posts with videos, we ask raters to view the video at least for 10 seconds to interpret the content
in the video.
● Extra contextual information: Raters can load business pages or search advertisers on FB to
understand unfamiliar business/advertisers, but raters will mainly rely on posts for rating
● For jobs where the post media content has been removed, please select “Next” and do not rate the
job
■ NOTE: Above the original post and suggested post, you will find their respective
category names. Please refer to this detail within the UI when evaluating for relevancy
between the posts.
■ It should not be considered the ground truth for determining relevance, but instead
use this as a reference point to determine the actual business category. The reason
this is being mentioned is because the automated categories may not always be
correct
■ If something doesn’t seem right, please click the link to find the category on the
business’s Meta page or if the link isn’t working, please copy the business name and
paste it into Meta in another tab to side search the business and determine the
correct category
Definitions

● Business: the owner of a post or ad with certain brand: advertiser or business page
● Business page: the Facebook page created by the business to connect with Facebook users; Raters
can leverage it to understand unfamiliar business, but raters will mainly rely on posts for rating.
Please note the difference between Level 0 and Level 3 is that Level 3 has an original business page.
● Commercial Post: a post that contains products/services/themes offered by the business; a non-
commercial post would not contain any of these items (ie. auction post, job listing, unrelated event
etc…)
● For a post to be considered commercial, something must be being sold or there must be a shop
section of the blog/article/website associated with the post
● Product/Services/Content: The item that is the subject of the post/ad
● Category: A broad type of category class that the product/service/content can fall into: footwear,
clothing, restaurant, car
● Specific category: A more granular category under a broad category: running shoes, leather jacket,
pizza restaurant, SUV.
● Related category: Categories that have some product/service/content in common but not fully
overlap: sport equipment vs famous athletes, Farm vs grocery store, hotel vs restaurant, etc.
● Original post is on the left, while the suggested post is on the right.

Directions:

● Click on the green “Link to business page” button on the Original Post and Suggested Post to
determine what type of business is making the ad/post for both. (If there isn’t enough information,
click on the “about” section of the business page.)
● Read and determine the intent (product/service/content) of the ad/post for the Original post and
Suggested Post (PLEASE READ CAREFULLY)
● Compare the Original Post to the Suggested Post using the information you’ve gathered. (The
business type and the ad/post.) Use the following guidelines to determine the best label.

Rating Process Flowchart

 (only applies to Q1 which comes right after Q0)

Firstly, raters will have to answer whether or not there is sensitive content on the page, then proceed to this flow:
Rater Guideline:

Q0 — Integrity Screening (Sensitive/Prohibited Content)

Prior to starting Question 1, please review the text and image/video content for sensitive/prohibited material. If
sensitive/prohibited material is present within the job page, please select of the options:

No → Continue

Yes → Original Post

Yes → Suggested Post

Sensitive/Prohibited Material Criteria:

● Post contains content that would go against Facebook Community Standards such as nudity, partial
nudity, strong sexually suggestive content, sexy click-bait images, unverified scientific claims, illegal
drugs/paraphernalia, offensive language or swearing, weapons (guns, bullets, knives, blades, other
purposefully dangerous objects), illegal fireworks/explosives, human trafficking, child exploitation,
terrorism, suicidal or self-harm verbiage and deceased animals.

Q1 — Contextual Relevance
How relevant are any of these pairs?

<original post/OP, suggested post/SP>

<original post, suggested ad>

<original ad, suggested post>

<original ad, suggested ad>

● Can’t decide
● Non-commercial
○ Level 0 NC (Non-Commercial): Original post has no commercial content AND comes from a non-
commercial page
● Relevant
○ Level 1 R (Relevant): Same category of business/advertisers AND same specific category of
posts/ads (original post has commercial intent and is relevant to original page category)
● Somewhat relevant
○ Level 2 SR (Somewhat Relevant): Related category of both businesses/advertisers AND
posts/ads (original post has commercial intent and is relevant to original page category)
○ Level 3 SR NC Post (Non-Commercial Post): OP has no commercial content, but comes from a
commercial page. Original page category is related to suggested post/page category
○ Level 4 SR NC Page (Non-Commercial Page): OP has business commercial content, but comes
from a non-commercial page. OP category is related to suggested post/page category
● Complements
○ Level 5 C (Complements): OP and SP are complements
● Irrelevant
○ Level 6 IR Post (Irrelevant Posts): Related category of business/advertisers, but irrelevant
posts/ads
○ Level 7 IR Pages (Irrelevant Pages): Irrelevant businesses/advertisers

Can’t Decide

● If the rater is not able to identify the product/service/content due to wrong unfamiliar language is
shown in the original OR suggested posts, even if the language is translated automatically by the
system (foreign language translation may still be unclear or not precise enough to determine
relevance

0. No trigger
Level 0: The trigger (original) post has no commercial content and comes from a non-commercial entity
We should trigger FBE only if a post is from a commercial entity or has commercial content. Thus, if a trigger post is
from a non-commercial entity AND the post has no commercial content, we create a new relevance level to
differentiate this case, called Level 0.

Defining Commercial Content

Type 1: Awareness
Guidance for Human Raters: Type 1 commercial content attempts to build brand awareness or engagement with
its viewer. This content will always include some (positive) reference to the business or its brand in an attempt to
ensure the viewer perceives the page posting it favorably.

● Type 1 Content might have

○ References or hyperlinks to the businesses page or website (example - Moss & Pine)
○ References to the brand, especially as a tag or hashtag (example - Peloton)
○ Content related to the business category, for example references to furniture in the post if the
business sells furniture, or cheese if the business is a cheese shop (example - Alithea Travel)
○ Media content containing the brand logo (example - Peloton)

Type 2: Consideration
Guidance for Human Raters: Type 2 commercial content attempts to drive the user to consider it more deeply or
convert. This content will usually reference specific products or services, providing users a reason to want to click
on the content (in order to learn more about it or to purchase a specific offering).

● Type 2 Content might have

○ References and/or details regarding a specific product or service (example - ModCloth, example -
Rasoi IV Authentic Indian Cuisine)
■ Specific products are often Camel Case in the post’s text (example - Reformation,
example - Starbucks)
■ Listing the prices of items ($ or other currency symbols in the text)
○ “Reasons to believe” in the brand’s message such as customer testimonials or videos/images
featuring the business or its products (example)
○ Calls to action with hyperlinks, with action verbs like
■ “Check out {insert brand material} here!” (example - Joybird)
■ References to “sales” or special/exclusive “offers” i.e. ‘Memorial Day Sale’ (example -
Joybird, example - Joybird)

TIPS:

● Side search is needed to determine if a page is commercial.

○ Click on the business page and find the website.
 ○ If there is a shopping link or merchandise for sale by the page, we consider this a commercial
page.
● The description on the page and/or widget isn’t always correct.
○ For instance, sometimes the page will be described as a company such as “food company”.
However, when the page is searched, we will see that it’s a recipe page that doesn’t sell any
merchandise. In this case, it’s not a commercial page.
● To determine if an ad/post has commercial content:
○ There must be links to the website, references to the brand, a logo, testimonials, reasons to
believe in the brand, etc.

Examples:

Level 0: The trigger (original) post has no commercial content and comes from a non-commercial entity
Relevant

The relevant label is used when the suggested post shows closely matching content. There should be a high chance
that users would have continued interest in the suggested post and learning more details about it. This means that
the suggested post has the product/service/content and the related specific category as the original post does.

Level 1: Same category of business/advertisers and same specific category of posts/ads (original post has
commercial intent and is relevant to original page category)

● with the same business and same specific category of Product/service/content

○ {black 2021 Nissan altima} → [blue 2020 Subaru Forester]

○ {Crate and Barrel Round Dining table} → [Crate and Barrel Rectangle Dining Table]
○ {Iphone 7} → [Iphone 7 Plus]
○ {HM Women's Suit Pants} → [HM Women's Casual Pants]
○ {Salty Pet Grooming with cat} → [Salty Pet Grooming with dog]
○ {homemade cake (any flavor)}→[wedding cake (any flavor)]
○ with bundle: {Ikea INGO Pine table} → [Ikea LINNMON / ADILS Table, white with chairs]
○ Note: The approach is to think broadly, but not too broadly. For instance, women’s suit
pants and women’s casual pants would both be considered types of pants women could
wear every day. If the SP was women’s snow pants, it would fit better under the
substitutes label because those are pants worn at specific times only.
● with same category of business and the same specific category of Product/services/content

○ {Six string guitar lesson} → {Newman guitar lesson}

○ {xbox one} → [Nintendo Switch], [Sony PS4]
○ With bundle: {Ikea Dining table} → [Crate and Barrel Dining Table with 2 sets of white
chairs]
○ {women’s jeans} → [women’s pants] These are different materials but are both
specifically pants.
○ {fishing boat} → [pontoon boat] These are different types of boats but are both
specifically boats.
○ More examples from services:

Note: We should focus on high-level relevance of business commercial content. If both OP and SP share relevant
business products at a high level, we should consider them as relevant. For example, 1) OP and SP are both related
to martial arts 2) OP and SP are both related to car repair service etc (We should rate such cases as Level 1 instead
of Level 4 given the original post category has commercial intent and is relevant to the original page category)

TIPS:

● Notice that the description states that the category of ads/posts must be the same specific category.
 ○ This can be tricky because there is a broad approach to the labels. For instance, cupcakes and
cakes seem like they are a little bit different, but they are both specifically cakes. However,
flavors don’t matter. Rum cake vs. chocolate cake would not be a substitute because they’re
both kinds of cake

Note: For the examples below the Original Post will be noted as “OP” and the Suggested post will be noted as “SP”.
Somewhat Relevant

Somewhat relevant is used when the suggested content is about appropriate related products, services, events, or
other types of related content. These posts should be marked Somewhat Relevant when they are compared to the
category of the product in the original post. Somewhat Relevant is reserved for results that might be helpful for a
user considering the original post’s category but the suggested post does not fully meet user’s intent. Related
goods or services generally are classified as substitutes

Level 2: Related category of both businesses/advertisers and posts/ads (original post has commercial intent and is
relevant to original page category)

Substitutes are products or services that satisfy a similar need or desire.

● {Ikea Dining table} → [Ikea Sofa] A sofa is not a replacement for a dining table, but a person shopping
for sofas might also be shopping for other types of furniture. So, an ad about a sofa could be of
interest to a person shopping for furniture.
● {Columbia Men's Hiking Shirt} → [Northface Men's Jacket] A shirt and a jacket are different products
but they’re both in the clothing category. A person shopping for a shirt might also be interested in a
jacket. (Or pants, socks, tshirts, etc.)

○ If result was for women’s clothing, then it would be Level 4 since query was for men’s

● {Disney+ Loki TV series} → [Netflix movie] A tv series and a movie are similar types of entertainment,
but a tv series will have many seasons and episodes where a movie is one story told over a couple of
hours.
● {Nike Womens Running Shoes} → [Adidas Womens Sandals Shoes] While sandals and running shoes
serve different purposes, when we approach it broadly, we can see that if a user was searching for
running shoes they might be willing to look at other types of shoes as well. The idea is to think of it as
perhaps an additional purchase and not a replacement for the running shoes.
● {window trim and gutters replacement} → [Roofing replacement] This is a great example to show the
broad approach used in this project. In general, these are home improvement services. The same
could be said for {home anti-theft vehicle installation}→[solar panel installation] or {lawn care}→[pool
maintenance]
● {Mexican restaurant promoting items from their menu}→[italian restaurant promoting items from
their menu] When both businesses are restaurants but they have different cuisines on their menu,
this will be labeled as a substitute.
● {ice cream}→[pizza] These are not replacements for each other but ice cream and pizza are both food
items.
● {beauty}→[nail salon, makeup retailer, eyebrow stencils, skin lotion, hair salon, hair styling products]
● {food}→[grocery stores, farms, specialty food shops (like a cheese shop), bakeries, wine stores, juice
bars, restaurants]
● {clothing and accessories}→[hat shop, t shirt store, dress shop, shoe store, handbags and wallets,
belts, undergarment store]
● {hot dogs}→[cookies] These are both foods.
● {women’s pants}→[women’s coat] These are both clothing items.
Note: For the examples below the Original Post will be noted as “OP” and the Suggested post will be noted as “SP”.

Note: OP and SP can be labeled as “Somewhat Relevant” even if the SP isn’t entirely similar with the OP (recipes
most commonly) but goes give out additional information on how a product can be used more effectively provided
that they are in the same category

TIPS:

● The business category of the SP can be the same or similar to the business of the OP.
○ In other words, it doesn’t have to be the exact same as it does for Level 1. For instance, there are
different types of beauty services. If one business is a make-up service and the other is a hair
salon, these both are in the overall category of beauty.
■ They’re not the same thing, but they are similar.
● The ads/posts will also be similar but not the exact same.
Level 3: If OP (Original Post) has no commercial content, but comes from a commercial page, we should retrieve
posts, which do have commercial and relevant content, based on the OP’s real business page categories. To be
more specific, for this case, if the SP (suggested post) comes from a related business category with OP’s business
page categories and SP has commercial and relevant content, then it should be considered as Level 3.

Pair Notes Label

OP has no business commercial Level 3

content, but comes from a business
page of Fruit and Vegetable Store, SP
has fruit and comes from a business
page of Fruit and Vegetable Store,
thus it's a relevant recommendation,
we label it as Level 3

OP has no business commercial Level 3

content, but comes from a business
page of Sports Boxing, SP has boxing
related content and comes from a
business page of Sports Boxing, thus
it's a relevant recommendation, we
label it as Level 3

OP has no business commercial Level 3

content, but comes from a business
page of Tea Room, SP has food
related content and comes from a
business page of Tea Room, thus it's a
relevant recommendation, we label it
as Level 3
 OP has no business commercial Level 7
content, but comes from a business
page of Tea Room, SP has clothing
related content and comes from a
business page of Clothing Store, thus
it's a irrelevant recommendation, we
label it as Level 5

OP has no commercial content, but Level 6

comes from a business page of
Bakery Cafe, SP comes from a
relevant business page of Restaurant,
but SP's content is about clothing, we
label it as Level 4

TIPS:

● This is a variation of Level 0

○ For Level 0, both the page and the ad/post are non-commercial as discussed in slide 2. For Level
3, the business is commercial, but the ad/post is not commercial.
● Remember for the ad/post to be considered commercial it must have:
○ Links to the website, references to the brand, a logo, testimonials, reasons to believe in the
brand, etc.
○ If none of these are present in the ad/post, but the business is commercial this will be labeled
Level 3 as long as the category of the SP is related.

Examples:

{hair salon with an ad/post about jobs available}→[hair salon promoting hair services]

{restaurant with an ad/post of a funny meme}→[food truck advertising tacos]

Level 4: If OP has business commercial content, but OP’s page has no business commercial content or has quite
different business commercial content, we should prioritize OP’s content over its business page categories. To be
more specific, for this case, if the SP comes from a related business category with OP’s commercial content and SP
has commercial and relevant content as OP, then it should be considered as Level 3. Otherwise, label as Level 4 or
Level 5 depending on if OP and SP business pages categories match. For example,
 OP has clothing content, but comes Level 4
from a business page of Restaurant,
SP has clothing content and comes
from a business page of Clothing
Store, thus it's a relevant
recommendation, we label it as Level
3

OP has clothing content, but comes Level 6

from a business page of Restaurant,
SP has food content and comes from
a business page of Restaurant, thus
it's a irrelevant recommendation, we
label it as Level 4

OP has clothing content, but comes Level 7

from a business page of Restaurant,
SP has home-inside content and
comes from a business page of
Property Management, thus it's a
irrelevant recommendation, we label
it as Level 5

TIPS:

● Level 4 isn’t often found in the queue

○ There is not an example that follows, but it’s good to know what to recognize when you come
across it.
● Level 4 is the opposite of Level 3
○ If the page is not commercial BUT the ad/post is commercial, and the SP is a related category we
label this Level 4
○ For example, if a recipe site that doesn’t sell its own merchandise (so not commercial business)
was advertising a meal delivery service such as “hello fresh” (which is commercial) and the SP
category was also food related, this would be Level 4.

Level 5: If OP has business commercial content and OP’s content is relevant to its business page categories, and
OP and SP are complements
Complements are products and/or services that are used jointly and improve overall experience, or categories that
are connected together because they’re commonly associated with one another.

● {gym subscription} → [Nike running shoes]

● {computer} → [screen protector]

● {pool} → [swimming lessons]

● {piano} → [piano lessons]

● {make up product} → [skin care product]

● {restaurant}→[recipe]

● {handbags}→[clothing]

● {dress}→[shoes]

● A complement is different from a substitute in that it’s products or services that work together to
satisfy the user’s needs.
● Example 1: {bell pepper}→[stuffed peppers meal from a restaurant] While it could be argued that a
bell pepper can be consumed on its own (and therefore satisfy a similar need) in most cases the
average person would not sit down to a meal of bell peppers only. In this scenario we would consider
them an “ingredient” that could be used with other ingredients to make the meal of stuffed bell
peppers. This would be a complement.
● Example 2: {handbag}→[hat] Handbags and hats are both accessories to clothing, however they serve
different purposes and therefore satisfy different needs. A handbag fulfills the need to carry personal
items while a hat fulfills the need to warm the head. So these can’t fulfill a similar need to one
another. However, they could both be used for the broader purpose of accessorizing clothing.These
are complements to one another.
● {motorcycles for sale}→[motorcycle repair services]
● {tires for sale}→[trucks for sale]
● More examples:
TIPS:

● Deciding between Substitutes and Complements

○ Substitutes (Level 2)
■ Satisfy a similar need
■ Items such as pants and shirts are in the same category of clothing, but they serve
different purposes and are substitutes.
○ Complements (Level 5)
■ Frequently bought/used together
■ Handbags and shoes are not clothing, but they are commonly associated with clothing
and are used to accessorize an outfit. Therefore, pants and boots are complements. (Or
shirts and handbags)

Not Relevant [Off-Topic]

Product/service/content including its category are not related between original posts and suggested
posts/suggested ads. Category should be rated as Not Relevant when product and the category tagged are NOT
reasonably related to each other and chances that users would be interested in the suggested content are low.

Note: If the rater is not able to identify the product/service/content due to wrong unfamiliar language shown in
the original and suggested posts please rate “Can’t decide”.

Foreign Language - If the post has any foreign language, choose Can’t Decide.

Image Quality Tip:

If you choose Can’t Decide for the relevance label, then the Image Quality label will always be
Unacceptable-Other – write in the reason
Example: Can't Decide/ Unacceptable-Other – foreign language

Level 6: Same/related category of business/advertisers, but irrelevant posts/ads

Level 6

Both posts come from business

pages of Cargo Freight, but the
SP's content is about new
recruited people

Level 6

Both posts come from business

pages of Restaurant, but the SP's
content is about clothing
Additional Level 6 EXAMPLE:

In this example, both pages are pizzerias, but they have unrelated posts
TIPS:
 ● In order to label Level 6, we must eliminate the possibility that it could be Level 3
○ Please make sure that the OP ad/post has commercial intent.
○ If it doesn’t, it might be Level 3, so we’d need to evaluate that.
■ To be considered commercial the ad/post must have links to the website, references to
the brand, a logo, testimonials, reasons to believe in the brand
■ If the OP ad/post has commercial intent, but is unrelated to the ad/post of the SP, this is
Level 6, as long as the OP is in a related category to the SP

Level 7: Irrelevant businesses

Note: If the business in the original post and the suggested post are in different categories, the level 7 label should
be selected without considering the ad/post. The only exception is if there is a tool error in the post, then can’t
decide/unacceptable -other should be chosen.

Example of tooling issue rating: Can't Decide/ Unacceptable-Other – Tooling Issue

Image Quality Tip: If you choose Can’t Decide for the relevance label, then the Image Quality label will always be
Unacceptable-Other – write in the reason

TIPS:

● In order to label Level 7, we must eliminate the possibility that it could be Level 0.
○ First, we need to make sure that both the business and the ad/post of the OP have commercial
intent
○ If neither does, meaning there is no commercial intent, then it will be a Level 0, regardless of the
relation of the business of the OP and SP.
● Next, we will look at the business category of the OP and SP. If they are not related, this will be Level 7,
regardless of what the ads/posts are promoting.
Q2 — Image Quality

How do you rate the quality of the image from the second post/ad being rated from Q1?

● Acceptable
● Not acceptable, and choose any of the reasons:

○ Low resolution
○ Very inadequate lighting (Low light is not automatically inadequate - if a photo has low
light, but all the details are still clear and the quality is fine, it should still be rated as
acceptable. However, if the low light creates black spots or shadows that damage the
quality, it is unacceptable)
○ Very unprofessional background (Busy, no-contrast, etc)
○ No product/service focus - The photo should represent either a product that the business
in the Suggested Post offers, or represent a service that they provide (For a restaurant, a
waitress/waiter would represent a service that the restaurant provides.) If there is a
random product shown in the photo, but it’s not a product that the business offers for
sale, this will still be labeled no product.
○ Others (specify) For prohibited content please write in “Prohibited”.

From our UXR study, a visually appealing image represents good resolution, adequate lighting, professional
background, clear and clean product/service focused (e.g, no too many products). If the image contains ALL the
above features, please rate as Acceptable; Otherwise, please rate as Not acceptable with the corresponding
reason. If multiple issues are seen in the photo, please choose the issue that appears first in SRT. (Example, if the
photo has both low resolution and inadequate lighting, you would choose low resolution because that option
appears first in SRT.)

If there are multiple images in the Suggested post/ad, please consider only the 1st image.

Please use zoom on Chrome to view images. Videos should not be magnified.

Image Quality Tip:

If you choose Can’t Decide for the relevance label, then the Image Quality label will always be Unacceptable-Other
– write in the reason

If an image qualifies for two labels, choose the one that appears first in SRT. We see a lot of "no product" labels
that should be "low resolution" because the image is both but low resolution comes first on the list.

•A product in the SP image must be sold by the business in the SP. If the business is a clothing store and it shows a
plate of food, this is still no product, because a clothing store doesn't sell plates of food.

•ONLY judge the first photo of the SP. We don't evaluate any photos in the OP, or any other photos in the SP
besides the first one (if more than one is included).

Here are some examples:

“RECIPE” JOB EXAMPLES:
In both of the following examples, it was found through side-search that these original pages
were not just simply informational pages, but also sell something on their website in addition to
providing free recipes. Please use the same guidance when determining commercial intent for
these common recipe pages

“CAN’T DECIDE” JOB EXAMPLE:

In the following example, the suggested post did not generate at all. In these types of cases,
please select “Can’t Decide”