Professional Documents
Culture Documents
Table of contents
Project Goal......................................................................................................................................... 1
User Interface ..................................................................................................................................... 2
Questions ............................................................................................................................................ 3
Sensitive Content ............................................................................................................................ 3
Rating Labels .................................................................................................................................. 4
Relevant .......................................................................................................................................... 5
Somewhat Relevant ........................................................................................................................ 5
Not Relevant ................................................................................................................................... 5
Rating Process ..................................................................................................................................... 6
Decision flowchart .......................................................................................................................... 7
Definitions ........................................................................................................................................... 7
Specific Category............................................................................................................................. 7
Broad Category ............................................................................................................................... 8
Complementary Products ............................................................................................................. 11
Substitutes .................................................................................................................................... 12
Related Products and services...................................................................................................... 12
Same brand ................................................................................................................................... 12
Special Cases ..................................................................................................................................... 12
Additional Instructions ..................................................................................................................... 17
Examples ........................................................................................................................................... 25
Project Goal
This workflow aims to accurately identify relevant and irrelevant Sponsored Items (ads) against
Marketplace Listings (products), which are pages containing detailed information for a given listing.
The Marketplace Listing contains the title, image, description, and category of a listing. The Sponsored
Item includes the name of the advertiser, images or video, and text.
As an Ad Relevance Rater, your role is to help rate the relevance of the Sponsored Item (the ad) against
the Marketplace Listing (the product).
User Interface
Marketplace Listing: the product that is being sold by the seller
Sponsored Item: the advertisement that is shown on the same page as the Marketplace Listing
Advertiser name: brand name of the advertiser
No – if the job does not contain any sensitive content and it can be labeled
Yes (Product) – if the ‘product item’ contains sensitive content (see definition and the list below).
Yes (Ad) – if the sponsored item contains sensitive content (see definition and the list below).
Sensitive Content
Below are the sample scenarios on when to tag the job as sensitive:
• Violence or Calls to Violence: any weapons including weapon parts and accessories,
display/antique weapons (e.g., display/antique swords)
• Sexual Content and/or Nudity
• Child Exploitation and/or Nudity
• Human Trafficking
• Act of Terrorism
• Self-Injury
• Suicidal Content
• Hate Speech
• Bullying and Harassment
Rating Labels
You see this interface if you answer ‘yes’ to the first question.
Cannot Decide
Provide a rating of Cannot Decide if any of the following conditions are met for either the Marketplace
Listing or the Sponsored Item, and select the reason why you could not decide:
• Product is in a foreign language (non-English) that cannot be properly translated using Google
Translate, or the foreign language is embedded in the text/video/audio (options 1 or 2 above)
• Audio is in a foreign language (non-English) - (options 1 or 2)
• The product/service/content cannot be determined because:
o The text description is unclear, or not sure what the picture/video is trying to show
(options 3 or 4)
o The text description is clear, but the image shows otherwise, in other words, the image
suggests that the ad could be providing something different from what the text
suggests (options 3 or 4)
o Image/video/text is not rendering (options 5 or 6)
• The ad video is longer than 102 seconds: Right-click the video ad, check 'Show All Control", click
the kebab menu, and choose Full Screen to see the video duration
Relevant
Provide a rating of Relevant when the Marketplace Listing and the Sponsored Item are in the same
specific category (= they are functionally equivalent). See more information on specific categories.
Somewhat Relevant
Provide a rating of Somewhat Relevant when the Marketplace Listing and the Sponsored Item are not
in the same specific category, but satisfy one of the following:
Not Relevant
Provide a rating of Not Relevant when the marketplace Listing and the Sponsored Item are neither
Relevant, Somewhat Relevant, nor Cannot Decide.
Rating Process
The first step is to determine whether the job can be rated.
o Remember to use Google Translate to try and understand foreign content if possible.
o If you can understand the language of the ad and marketplace listing but still do not know what
either of them is providing (due to lack of knowledge), you may do a quick side search (<30
seconds).
o If you still are not able to assess the content, choose Cannot Decide
o More information on side searching in Additional Instructions below
Decision flowchart
We will be using the following flowchart to assess the relevance of the Sponsored Item against the
Marketplace Listing:
Definitions
Specific Category
Two products are considered to be in the same specific category when they are functionally
equivalent, meaning: they can be used to achieve the same functional purpose, but may have
differences in brand, appearance, color, and price. A person who is interested in purchasing one
product is likely to also consider the other product in their purchase decision.
• Examples of not functionally equivalent but still in the same broad category (Somewhat
Relevant)
o [Broad Category: Clothing, Shoes & Accessories] A pair of black Nike running shoes
and a pair of black The North Face hiking boots do not serve the same functional
purpose because running shoes are for running while hiking boots are for hiking. A
shopper looking to buy a pair of running shoes for running will not consider buying a
pair of hiking boots instead.
o [Broad Category: Clothing, Shoes & Accessories] A Men’s Hoodie (Hooded sweatshirt)
and a Men’s Dress Shirt are not functionally equivalent because they are worn on
different occasions and provide different levels of warmth. A shopper looking to buy a
hoodie will not consider buying a dress shirt because they are worn on completely
different occasions.
o [Broad Category: Electronics] A silver Apple MacBook Pro laptop and a silver Apple
iMac desktop computer do not serve the same functional purpose because the
MacBook Pro is a portable computer meant to be carried around while the iMac is a
stationary computer meant to be placed in a fixed position in the Home or Office. A
shopper is looking to buy a laptop because the portable aspect is important, and they
will not consider a stationary desktop computer.
o [Broad Category: Vehicles] A red Toyota 4Runner SUV and a red Toyota Corolla sedan
are not functionally equivalent because one is an SUV while the other is a sedan. A
person who is interested in purchasing an SUV is likely not going to consider a sedan in
their purchase decision because it does not fit their lifestyle needs.
o [Broad Category: Furniture] An IKEA dining table and an IKEA office table are not
functionally equivalent because one is for dining while the other is for working. A
shopper looking to buy a dining table in the living room will not consider buying an
office table.
Broad Category
A broad category is a way to group items that are similar to each other at a high level. Products that are
not in the same specific category can still be in the same broad category. For example, a Sedan and a
Motorcycle are under the same broad category of Vehicles. We can also say a Sedan and an SUV are
under the same broad category of Vehicles as well.
Below is the full list of Broad Categories, each Marketplace Listing or Sponsored Item must fit under
one of these Broad Categories. If you are not able to find an appropriate Broad Category, you may
categorize it under the Miscellaneous category:
Remember that products that belong to the same Broad Category are considered Somewhat Relevant.
Complementary Products
Complementary Products are two products that are often purchased together. One product could be an
accessory to the main product and can be added to make it more useful, versatile, or attractive.
If the Sponsored Item is a related product or service, it will be labeled Somewhat Relevant.
The brand is considered when relating a specific car brand (Vehicle) vs. brand-specific services
(Services).
• Honda Mobilio (product) and Nissan Intelligent Car Service (service) – Not Relevant
• Nissan Altima (product) and Hyundai Service (service) – Not Relevant
If the product/service is brand specific and it does not match the brand of the other product, the pair is
not considered a related product/service and it will be labeled Not Relevant
Same brand
Some Marketplace Listings and Sponsored Items have identifiable brands. If both the listing and the
sponsored item have a brand, then we can determine if they are the same brand.
Products from the same brand are considered Somewhat Relevant, regardless of their categories.
• Water bottle with the BMW logo and a BMW SUV both have the same brand of BMW
• A gift card for Nike and a pair of Nike running shoes both have the same brand of Nike
• An Apple Watch and an Apple MacBook pro both have the same brand of Apple
Special Cases
Product vs store, or product vs product?
• If the ad shows the image of a product, then it is advertising this specific product:
__________________________ __________________________________________________
• Sometimes, the sponsored item will be an ad for a store, with no image for a specific product.
In this case, assess whether the store sells similar products to the listing:
The ad does not show any specific item, but just a logo
(Renault) with the message “Current Offers” and the ad’s
caption is not referring to any specific item.
Listing
Additional Instructions
Some listings can display multiple images (example below). If these images are about different items,
only the first item will be evaluated.
Never choose the ‘Skip’ action, always choose the ‘Label’ action:
Side Searching:
• If you are unsure of what the ad is promoting, you will need to do a side search. However,
sometimes, zooming in on the ad can help determine its content:
• Sometimes, the ad is unclear, and the Advertiser’s name is cut off, follow the below steps to
conduct a side search to help view the Advertiser’s full name.
The Advertiser Name does not Triple-click the Advertiser Name to highlight the whole text. A
display fully right-click will enable you to search for the full Advertiser name
on Google
• If the ad is unclear and the Advertiser’s name does not help, you can try performing a side
search from the image (if available).
The Sponsored Item is from a car dealership, but the Ad itself is about their UTEs, as indicated by the
description and the image.
UTEs and Wagons belong to different specific categories, but belong to the same Broad Category, so
the rating is Somewhat Relevant. There is no need to consider the other Vehicles/products sold by the
company.
Same approach for this pair:
The text is very clear about the product advertised: the Roomba robot vacuum. For this sample, a side
search is not needed as it had enough data.
The vacuum is not complementary to the bed; the pair is Not Relevant since they are in a different
broad category (Furniture vs. Home & Kitchen).
Note: Reading the caption as supporting details about the ad being promoted should always be
performed to know what the ad is promoting. Like this example, if you won’t read the caption, you’ll
think that the ad is promoting the sideboard or the dining set which will make your decision label
incorrect
• If the ad displays a clear product but the text and the ad suggest a different thing, a side search
may clarify what the ad is sponsoring.
The listed product is a couch, and the sponsored item is showing multiple furniture items with a sofa as
a focus. However, the advertiser’s name and the text do not suggest that the actual product belongs to
the Furniture category since it is mentioning ’hospitality’.
In this case, we can use the advertiser’s name for a side search (Casa Cipriani) but if the right click does
not work or the results may be different due to the locale settings, performing a side search from the
logo should help.
The pair is a couch and hotel. Furniture vs. Services but hotel service is not related to the furniture
piece in Marketplace Listing and the pair is Not Relevant.
o Another example:
Sponsored items is showing a car on the beach and the marketplace listing is for a car.
We need to be sure that the car in the sponsored item is the actual product being promoted. The
advertiser’s name and ad caption are about travel in Peru, so a side search is needed for confirmation
Sponsored item is a Peru travel service, and it is NOT related to the car, even though it displays a car.
This is why a side search is done for confirmation.
• Side search from the text only when the advertiser’s name may be different due to locale
settings.
The listed product is an office chair and sponsored item does not show a specific product on the ad
image and does not give helpful insights about a specific product or service.
We use the advertiser’s name and the description for quick side searching
Watson’s is a store that sells multiple products, including Furniture.
The office chair is part of the Furniture category as well as the Furniture store. The pair should be
labeled as Somewhat Relevant.
Examples
Cannot Decide
Marketplace Listing Sponsored Item Reasons for label
There is Foreign Language
embedded in the image,
which cannot be translated.
This is a Cannot decide label.
Relevant
Marketplace Listing Sponsored Item Reasons for label
Same Specific category? Yes
Specific category:
Smartphone/Cell Phone
Broad Category: Cell Phones &
Accessories
Specific category:
Smartphone/Cell Phone
Broad Category: Cell Phones &
Accessories
Smartwatches belong to
both Electronics and Jewelry
& Watch categories.
Smartwatches and regular
watches are also reasonable
substitutes as they have the
Specific category: Smartwatch same basic features.
Broad category: Specific category: Watch This pair is labeled
Electronics/Jewelry & Watches Broad category: Jewelry & Somewhat Relevant.
Watches
Not Relevant
Marketplace Listing Sponsored Item Reasons for label
Same Specific category?
No
Marketplace Listing is a
Volkswagen Van while
the sponsored item is a
summer trip package
Specific category: Van service for Kia cars.
Broad Category: Vehicles Specific category: Car check
Broad Category: Services
The service is brand
specific and does not
work for vehicles of other
brands. It is not
considered a related
service and therefore is
labeled as not relevant.