Professional Documents
Culture Documents
Table of contents
l
Project Goal......................................................................................................................................... 1
User Interface ..................................................................................................................................... 2
79 ia
Questions ............................................................................................................................................ 3
Sensitive Content ............................................................................................................................ 3
Rating Labels .................................................................................................................................. 4
Relevant .......................................................................................................................................... 5
48 nt
Somewhat Relevant ........................................................................................................................ 5
42
Not Relevant ................................................................................................................................... 5
Rating Process ..................................................................................................................................... 6
Decision flowchart .......................................................................................................................... 7
13 ide
Definitions ........................................................................................................................................... 7
Specific Category............................................................................................................................. 7
Broad Category ............................................................................................................................... 8
Complementary Products ............................................................................................................. 11
Substitutes .................................................................................................................................... 12
Related Products and services...................................................................................................... 12
Same brand ................................................................................................................................... 12
nf
Project Goal
This workflow aims to accurately identify relevant and irrelevant Sponsored Items (ads) against
Marketplace Listings (products), which are pages containing detailed information for a given listing.
The Marketplace Listing contains the title, image, description, and category of a listing. The Sponsored
Item includes the name of the advertiser, images or video, and text.
As an Ad Relevance Rater, your role is to help rate the relevance of the Sponsored Item (the ad) against
the Marketplace Listing (the product).
User Interface
Marketplace Listing: the product that is being sold by the seller
l
79 ia
Description: description of the product added by
the seller
48 nt
seller (DO NOT solely rely upon the category
listed here, please use your best judgment when
42
assigning a Broad Category)
13 ide
Sponsored Item: the advertisement that is shown on the same page as the Marketplace Listing
Advertiser name: brand name of the advertiser
l
79 ia
48 nt
42
13 ide
No – if the job does not contain any sensitive content and it can be labeled
Yes (Product) – if the ‘product item’ contains sensitive content (see definition and the list below).
Yes (Ad) – if the sponsored item contains sensitive content (see definition and the list below).
Sensitive Content
nf
Below are the sample scenarios on when to tag the job as sensitive:
• Violence or Calls to Violence: any weapons including weapon parts and accessories,
display/antique weapons (e.g., display/antique swords)
• Sexual Content and/or Nudity
• Child Exploitation and/or Nudity
Co
• Human Trafficking
• Act of Terrorism
• Self-Injury
• Suicidal Content
• Hate Speech
• Bullying and Harassment
Rating Labels
You see this interface if you answer ‘yes’ to the first question.
l
79 ia
48 nt
42
13 ide
Cannot Decide
nf
Co
Provide a rating of Cannot Decide if any of the following conditions are met for either the Marketplace
Listing or the Sponsored Item, and select the reason why you could not decide:
• Product is in a foreign language (non-English) that cannot be properly translated using Google
Translate, or the foreign language is embedded in the text/video/audio (options 1 or 2 above)
• Audio is in a foreign language (non-English) - (options 1 or 2)
• The product/service/content cannot be determined because:
o The text description is unclear, or not sure what the picture/video is trying to show
l
(options 3 or 4)
o The text description is clear, but the image shows otherwise, in other words, the image
79 ia
suggests that the ad could be providing something different from what the text
suggests (options 3 or 4)
o Image/video/text is not rendering (options 5 or 6)
• The ad video is longer than 102 seconds: Right-click the video ad, check 'Show All Control", click
the kebab menu, and choose Full Screen to see the video duration
48 nt
42
Relevant
Provide a rating of Relevant when the Marketplace Listing and the Sponsored Item are in the same
specific category (= they are functionally equivalent). See more information on specific categories.
13 ide
Somewhat Relevant
Provide a rating of Somewhat Relevant when the Marketplace Listing and the Sponsored Item are not
in the same specific category, but satisfy one of the following:
Not Relevant
Provide a rating of Not Relevant when the marketplace Listing and the Sponsored Item are neither
Relevant, Somewhat Relevant, nor Cannot Decide.
Co
Rating Process
The first step is to determine whether the job can be rated.
o Remember to use Google Translate to try and understand foreign content if possible.
o If you can understand the language of the ad and marketplace listing but still do not know what
either of them is providing (due to lack of knowledge), you may do a quick side search (<30
seconds).
o If you still are not able to assess the content, choose Cannot Decide
l
o More information on side searching in Additional Instructions below
79 ia
48 nt
42
13 ide
nf
Co
Decision flowchart
We will be using the following flowchart to assess the relevance of the Sponsored Item against the
Marketplace Listing:
l
79 ia
48 nt
42
13 ide
Definitions
Specific Category
Two products are considered to be in the same specific category when they are functionally
nf
equivalent, meaning: they can be used to achieve the same functional purpose, but may have
differences in brand, appearance, color, and price. A person who is interested in purchasing one
product is likely to also consider the other product in their purchase decision.
Co
• Examples of not functionally equivalent but still in the same broad category (Somewhat
l
Relevant)
o [Broad Category: Clothing, Shoes & Accessories] A pair of black Nike running shoes
79 ia
and a pair of black The North Face hiking boots do not serve the same functional
purpose because running shoes are for running while hiking boots are for hiking. A
shopper looking to buy a pair of running shoes for running will not consider buying a
pair of hiking boots instead.
o [Broad Category: Clothing, Shoes & Accessories] A Men’s Hoodie (Hooded sweatshirt)
48 nt
and a Men’s Dress Shirt are not functionally equivalent because they are worn on
different occasions and provide different levels of warmth. A shopper looking to buy a
42
hoodie will not consider buying a dress shirt because they are worn on completely
different occasions.
o [Broad Category: Electronics] A silver Apple MacBook Pro laptop and a silver Apple
iMac desktop computer do not serve the same functional purpose because the
13 ide
MacBook Pro is a portable computer meant to be carried around while the iMac is a
stationary computer meant to be placed in a fixed position in the Home or Office. A
shopper is looking to buy a laptop because the portable aspect is important, and they
will not consider a stationary desktop computer.
o [Broad Category: Vehicles] A red Toyota 4Runner SUV and a red Toyota Corolla sedan
are not functionally equivalent because one is an SUV while the other is a sedan. A
person who is interested in purchasing an SUV is likely not going to consider a sedan in
their purchase decision because it does not fit their lifestyle needs.
o [Broad Category: Furniture] An IKEA dining table and an IKEA office table are not
nf
functionally equivalent because one is for dining while the other is for working. A
shopper looking to buy a dining table in the living room will not consider buying an
office table.
Broad Category
Co
A broad category is a way to group items that are similar to each other at a high level. Products that are
not in the same specific category can still be in the same broad category. For example, a Sedan and a
Motorcycle are under the same broad category of Vehicles. We can also say a Sedan and an SUV are
under the same broad category of Vehicles as well.
Below is the full list of Broad Categories, each Marketplace Listing or Sponsored Item must fit under
one of these Broad Categories. If you are not able to find an appropriate Broad Category, you may
categorize it under the Miscellaneous category:
l
o This category includes all cell phones/smartphones (Android phones and iPhones) but
NOT the accessories.
79 ia
▪ iPhones and Androids are in the same broad categories of Cell Phones
o Excluded 1: All Tablets (Android and Apple) and all their related pens and accessories
like tablet chargers, tempered glass, cases, Kindle, and accessories, tracking devices
(Samsung Smart Tag and Apple Air Tag), and earbuds.
▪ All these products fall into Electronics
48 nt
o Excluded 2: Cellphone Accessories ((batteries, phone cases, tempered glasses, chargers
in different shapes, cables, etc.)
42
▪ All these products fall into Cellphone Accessories
• Cell Phone Accessories (updated)
o This category includes all cellphone accessories (chargers, batteries, cables, glasses,
cases, protectors, and so on).
13 ide
o Excluded: Cell Phones and Smartphones. Smartwatches, Tablets, and Earbuds.
• Clothing, Shoes & Accessories
o This broad category includes all the clothing-related things, including shoes (any type of
shoes, any gender but NOT any age*), wedding dresses/suits, prom dresses, specific
regional dresses/suits, beach wear, swimsuits, fitness wear, scarves, hats, gloves, kids
clothing, underwear, sleepwear and so on.
o Hiking Boots and Running Shoes are different specific categories under the same broad
category of Clothing, Shoes & Accessories
o A Saree (traditional Indian women’s-wear) and a bra (women's undergarment) are different
nf
specific categories but fall under the same broad category of Clothing, Shoes, and Accessories
o Excluded: Jewelry and Watches | Bags and Luggage | *Baby Products (which belong to
other broad categories)
• Electronics
o In this category, we have all the electronic devices like computers (laptops and
desktops), all types of TVs, smartwatches, keyboards, mouse, DVD players, earbuds,
Co
l
rent, apartments, detached houses, home listings, garages, commercial
places/buildings, condos, etc.
79 ia
o Detached Single Houses and Apartment are different specific categories under
the same broad category of Property
o Land Plots and Office Building are specific categories under the same broad
category of Property
o Excluded: Hotels and Resorts (Services
48 nt
• Home & Kitchen (note: “Home” here refers to products/services related to the home, not
homes for sale/rent)
42
o This category includes all the things you can find in the Home & Kitchen, including
appliances. curtains, rugs, cake stands, blinds, sofa covers, bedding, kitchen wear,
wallpaper, cushions, refrigerators, ovens, stoves, microwave ovens, cutting boards, and
so on.
13 ide
o Appliances and Bedding are different specific categories under the same broad
category of Home & Kitchen
o Excluded: Property, Furniture, Garden appliances such as outdoor pizza oven, BBQ |
and all Patio and Garden items that are listed in the Patio and Garden
• Jewelry & Watches
o Bracelets, rings, earrings, wedding bands, engagement rings, necklaces, body jewelry,
regional wedding sets, and watches
o Smartwatch is also a specific category in Jewelry & Watches
• Miscellaneous
nf
rugs, outdoor blinds, plants, plant pots, garden heating, garden ornaments and
accessories, lawn mowers, seeds planting tray. Could also include BBQs and
accessories, outdoor appliances, outdoor cushions, and many others.
o Garden Sofas and Outdoor Ovens are different specific categories under the
same broad category of Patio & Garden
o Plants and Lawn Mower are different specific categories under the same broad
category of Patio & Garden
o Excluded: indoor Furniture (Furniture) and Home & Kitchen items
• Pet Supplies
• Services (e.g., fitness classes, auto insurance, etc.) - Services DO NOT offer a physical product. It
is a service if you must pay someone to do something.
o Services include fitness classes, workshops, product delivery (food, furniture, etc.),
swimming lessons, piano lessons, and insurance (auto insurance, home, life, travel,
etc.). Mobile service provider. Real Estate Agent (a person who helps you find and
finalize a rent or a sale).
o Real Estate Agent/Agencies are specific categories in Services
o Hotels and Resorts are specific categories of Services.
o Car repair service, after-sale service for vehicles (disregarding brand and
product model), and product warranties.
o Hotels and Resorts.
o Electronics and smartphone services (for example, AppleCare)
• Sporting Goods
o Hockey equipment, horse gear, golf bag, tennis racket, camping gear, non-motorized
l
bicycles
o Bicycles constitute a specific category in Sporting Goods
79 ia
o Electric Bikes are excluded from the Sporting Goods category
o Non-Motorized vehicles are part of the Sporting Goods category
o Excluded: e-bike (Vehicles)
• Tools & Home Improvement
o Drill, screwdriver, pliers, stepladder, floor, bath remodeling product
48 nt
o Excluded: Products in Home & Kitchen → like wood flooring is for Home Improvement
while wood floor cleaning product is for Home & Kitchen
42
• Toys & Games
• Bags & Luggage
• Vehicles
o All motorized vehicles like cars, motorbikes, scooters, campers, RVs, ATVs, UTVs,
13 ide
commercial trucks, snowmobiles, flatbed trucks, dump trucks, tractors, bulldozers,
concrete mixers, trailers, airplanes, trains, boats (all types), e-bikes and e-scooters and
other motorized vehicles.
o Pickup Truck and Sedans are different specific categories under the same
broad category of Vehicles.
o Excluded: bicycles and those non-motorized like mountain bikes, off-road bikes, etc.
These types of bikes fall under Sporting Goods
• Video Games & Consoles
nf
Remember that products that belong to the same Broad Category are considered Somewhat Relevant.
Complementary Products
Complementary Products are two products that are often purchased together. One product could be an
accessory to the main product and can be added to make it more useful, versatile, or attractive.
l
Related Products and Services
79 ia
In general, the Marketplace Listing is a product, and the Sponsored Item can be a product, service, or
neither. When the Sponsored Item is a service that is related to the Marketplace Listing, we call this a
Related Product service. In this scenario, one must be a product and the other must be a service related
to the product. Examples of Related Products or Services.
48 nt
• Pool (product) and swimming lessons (service)
• Piano (product) and piano lessons (service)
42
• Computer (product) and computer repair (service)
• Makeup (product) and makeup service (service)
• Food (product) and food delivery (service)
• House (product) and mortgage loans (service)
13 ide
• Car (product) and automotive insurance (service) [e.g., Geico]
• Apartment rental (product) and moving services (service)
• Boat (product) and Boat License (service) [e.g., Are you wondering how to get your Captain’s
License? Request our comprehensive guide today! If you’re thinking of becoming a boat
captain, this guide will provide insight and information to help you easily navigate the licensing
process. Turn your dream into a reality today!]
If the Sponsored Item is a related product or service, it will be labeled Somewhat Relevant.
The brand is considered when relating a specific car brand (Vehicle) vs. brand-specific services
nf
(Services).
• Honda Mobilio (product) and Nissan Intelligent Car Service (service) – Not Relevant
• Nissan Altima (product) and Hyundai Service (service) – Not Relevant
Co
If the product/service is brand specific and it does not match the brand of the other product, the pair is
not considered a related product/service and it will be labeled Not Relevant
Same brand
Some Marketplace Listings and Sponsored Items have identifiable brands. If both the listing and the
sponsored item have a brand, then we can determine if they are the same brand.
Products from the same brand are considered Somewhat Relevant, regardless of their categories.
• Water bottle with the BMW logo and a BMW SUV both have the same brand of BMW
• A gift card for Nike and a pair of Nike running shoes both have the same brand of Nike
• An Apple Watch and an Apple MacBook pro both have the same brand of Apple
Special Cases
Product vs store, or product vs product?
• If the ad shows the image of a product, then it is advertising this specific product:
l
79 ia
48 nt
42
The image shows multiple products, but the caption advertises
the set of chairs specifically, so this is the product we are
assessing.
13 ide
nf
Co
The Sponsored Item shows multiple products that belong to the
Broad Category of ‘Furniture’ (table, bench, chairs, sideboard).
In this case, check if one of the products matches the
Marketplace Listing to assess relevancy.
l
Relevant at best).
79 ia
__________________________ __________________________________________________
42
caption clarifies that the actual product is scooter decal stickers
in ‘Auto Parts and Accessories’.
If the Listing is a scooter, the pair is Somewhat Relevant as the
ad is sponsoring the stickers, not the scooter.
13 ide
• Sometimes, the sponsored item will be an ad for a store, with no image for a specific product.
nf
In this case, assess whether the store sells similar products to the listing:
l
79 ia
48 nt
The ad does not show any specific item, but just a logo
(Renault) with the message “Current Offers” and the ad’s
42
caption is not referring to any specific item.
l
Accessories.
79 ia
The Sponsored item is a
clothing store.
Since Jerseys can be a
piece of clothing, we can
consider that both ads
48 nt
belong to the broad
category of Clothing,
42
Shoes, and Accessories,
Sponsored Item and consider the
Sponsored Item Somewhat
Relevant.
13 ide
Listing
nf
Co
Additional Instructions
Some listings can display multiple images (example below). If these images are about different items,
only the first item will be evaluated.
l
79 ia
48 nt
42
13 ide
The Skip option:
nf
Never choose the ‘Skip’ action, always choose the ‘Label’ action:
Co
Side Searching:
• If you are unsure of what the ad is promoting, you will need to do a side search. However,
sometimes, zooming in on the ad can help determine its content:
l
79 ia
48 nt
42
The Sponsored Listing’s video is
small, so it is difficult to
determine its content
13 ide
Simple zoom-in allows seeing that it is for a Kids’ book and
avoids having to side search.
• Sometimes, the ad is unclear, and the Advertiser’s name is cut off, follow the below steps to
conduct a side search to help view the Advertiser’s full name.
nf
Co
The Advertiser Name does not Triple-click the Advertiser Name to highlight the whole text. A
display fully right-click will enable you to search for the full Advertiser name
on Google
• If the ad is unclear and the Advertiser’s name does not help, you can try performing a side
search from the image (if available).
l
79 ia
48 nt
42
The advertiser’s name is for
Mercedes-Benz Canada. Right/left click on the image and choose ‘search image with
13 ide
Google’. A separate tab will open, and you will find more info.
nf
Co
• If the Ad displays a product or indicates clearly what products are advertised, there is no need
to side search: use the Advertiser name, image, and description to decide:
l
79 ia
48 nt
42
13 ide
nf
The Sponsored Item is from a car dealership, but the Ad itself is about their UTEs, as indicated by the
description and the image.
Co
UTEs and Wagons belong to different specific categories, but belong to the same Broad Category, so
the rating is Somewhat Relevant. There is no need to consider the other Vehicles/products sold by the
company.
Same approach for this pair:
l
79 ia
48 nt
42
13 ide
The listed product is a bed frame.
The sponsored item is showing a sideboard and a dining set which some raters may conclude that the
sponsored ad is probably under Furniture, but we must be mindful of the robot vacuum beside the
sideboard.
The text is very clear about the product advertised: the Roomba robot vacuum. For this sample, a side
search is not needed as it had enough data.
nf
The vacuum is not complementary to the bed; the pair is Not Relevant since they are in a different
broad category (Furniture vs. Home & Kitchen).
Note: Reading the caption as supporting details about the ad being promoted should always be
performed to know what the ad is promoting. Like this example, if you won’t read the caption, you’ll
think that the ad is promoting the sideboard or the dining set which will make your decision label
incorrect
Co
• If the ad displays a clear product but the text and the ad suggest a different thing, a side search
may clarify what the ad is sponsoring.
l
79 ia
48 nt
42
13 ide
The listed product is a couch, and the sponsored item is showing multiple furniture items with a sofa as
a focus. However, the advertiser’s name and the text do not suggest that the actual product belongs to
the Furniture category since it is mentioning ’hospitality’.
In this case, we can use the advertiser’s name for a side search (Casa Cipriani) but if the right click does
not work or the results may be different due to the locale settings, performing a side search from the
logo should help.
The pair is a couch and hotel. Furniture vs. Services but hotel service is not related to the furniture
piece in Marketplace Listing and the pair is Not Relevant.
13 ide
o Another example:
nf
Co
Sponsored items is showing a car on the beach and the marketplace listing is for a car.
We need to be sure that the car in the sponsored item is the actual product being promoted. The
advertiser’s name and ad caption are about travel in Peru, so a side search is needed for confirmation
l
79 ia
48 nt
Sponsored item is a Peru travel service, and it is NOT related to the car, even though it displays a car.
This is why a side search is done for confirmation.
42
• Side search from the text only when the advertiser’s name may be different due to locale
settings.
13 ide
nf
Co
The listed product is an office chair and sponsored item does not show a specific product on the ad
image and does not give helpful insights about a specific product or service.
We use the advertiser’s name and the description for quick side searching
l
79 ia
Watson’s is a store that sells multiple products, including Furniture.
The office chair is part of the Furniture category as well as the Furniture store. The pair should be
labeled as Somewhat Relevant.
48 nt
42
Examples
Cannot Decide
13 ide
Marketplace Listing Sponsored Item Reasons for label
There is Foreign Language
embedded in the image,
which cannot be translated.
This is a Cannot decide label.
nf
Co
l
Relevant
79 ia
Marketplace Listing Sponsored Item Reasons for label
Same Specific category? Yes
48 nt
because they are both
photography art that can be
42
hung on a wall for house
decoration purposes.
Specific category: Photography
art
13 ide
Broad Category: Arts & Crafts
Specific category: Photography
art
Broad Category: Arts & Crafts
Same Specific category? Yes
trucks
l
the Sponsored Item are
Functionally Equivalent
79 ia
because they are both
smartphones.
48 nt
Specific category:
Smartphone/Cell Phone
42
Broad Category: Cell Phones &
Accessories
Specific category:
Smartphone/Cell Phone
13 ide
Broad Category: Cell Phones &
Accessories
women’s hoodies.
& Accessories
l
to the same broad category,
they are automatically
79 ia
Specific category: Recliners Specific category: Dining Set
Somewhat Relevant.
Broad Category: Furniture Broad Category: Furniture
42
the Sponsored Item are not
functionally equivalent,
because the polo shirt is for
Specific category: Men’s polo shirts more formal occasions and
13 ide
Broad Category: Clothing, Shoes & the T-shirt is for casual
Accessories occasions.
Since the pair still belongs to
the same Broad Category,
Somewhat Relevant applies.
Specific category: T-shirt
Broad Category: Clothing,
Shoes & Accessories
to different broad
Specific category: Off-Road Vehicle categories.
Broad Category: Vehicles However, the car cleaning
product can be used to clean
the Off-road vehicle, so the
item is a complement to the
product.
Specific category: Car
cleaning product
Broad Category: Auto Parts
& Accessories
l
insurance to the SUV,
making it financially safer to
79 ia
Specific category: SUV
own and drive the SUV, so
Broad Category: Vehicles
the service is directly related
Specific category: Auto to the product.
Insurance
Broad Category: Services
48 nt
(Insurance)
42
Same Specific category? No
l
to different broad
categories.
79 ia
The auto repair service is
directly related to the
product (the sedan), as
someone owning this type of
vehicle could want to use
Specific Category: Sedan
48 nt
Specific Category: Auto this service, so it can be
Broad Category: Vehicles
Repair considered Somewhat
Relevant
42
Broad Category: Services
Smartwatches belong to
both Electronics and Jewelry
& Watch categories.
Smartwatches and regular
watches are also reasonable
substitutes as they have the
Specific category: Smartwatch same basic features.
Broad category: Specific category: Watch This pair is labeled
Electronics/Jewelry & Watches Broad category: Jewelry & Somewhat Relevant.
Watches
Not Relevant
l
Marketplace Listing Sponsored Item Reasons for label
79 ia
Same Specific category?
No
48 nt Same Brand? No
42
The Marketplace listing
and the Sponsored Item
belong to different broad
categories, and they are
Specific category: Motorcycle
13 ide
Specific category: House not directly related
Broad Category: Vehicles Renovation & Construction (house renovation has no
Broad Category: Services direct link to
motorcycles).
Same Specific category?
No
Same Brand? No
l
either.
79 ia
Specific category: Computer
Monitor
Broad Category: Electronics
42
Same Broad category? No
Complementary item? No
Related Service? No
13 ide
Same Brand? No
Related Service? No
Same Brand? No
l
The Marketplace Listing is
a bumper for Dodge
79 ia
RAM, while the
sponsored item is
showing a 2023 Chevy
Bolt EV.
48 nt
The bumper is brand-
Specific category: Bumper Specific Category: Hatchback specific for Dodge RAM
Broad Category: Auto Parts & Broad Category: Vehicles and cannot be placed on
42
Accessories Chevrolet Bolt. It cannot
be considered a
complementary product;
they are not the same
13 ide
brand either.
Same Specific Category:
No
Marketplace Listing is a
Volkswagen Van while
the sponsored item is a
summer trip package
Specific category: Van service for Kia cars.
Broad Category: Vehicles Specific category: Car check
Broad Category: Services
The service is brand
Co