These guidelines contain confidential and proprietary information belonging to Microsoft Corporation.

The recipient understands and

agrees that these materials and the information contained herein may not be used or disclosed without the prior written consent of
Microsoft Corporation.

Project Overview: When a user performs a search, they expect the search engine to understand what they want and
when appropriate return Ads that satisfy their requirements. Your responses will help us improve the quality and
relevance of the Ads displayed to users.

Task: Is this Ad Not Bad, Bad or Junk?

In the tool you will see a query (a word or phrase typed by a user into the search engine) and a text Ad.
Query: Cheap Flights

1. Use the search links in the tool to research the query and ensure you understand what product or service the user is
looking for (the query or user intent) and what the Ad is selling.

• You should research any aspects of the Ad and/or query that are unfamiliar to you (such as terms for
products, models, brands, and locations) to ensure you fully understand both what the user wants and what
the Ad is selling.

➢ Locality – Places in the same state/county/country may be miles apart, so always research
unfamiliar locations. {See Location & Locality Section}

2. Once you have a clear understanding of both what the user wants and what the Ad is selling, you then need to
decide if the Ad is:

Not Bad - the Ad might be useful and/or of interest to the user. This may be because it is: what they are looking
for; is somehow relevant to their query; or is an alternative, substitute, or related product to what they want.
or
Bad - the Ad would be of no use or interest to the user.
or
Junk – the Ad is impossible to understand.

User Query: iPhone 12 Pro

iPhone 12 pro iPhone 12 pro Galaxy s21 iPhone pro 12 Samsung Galaxy iPhone 8
max case case
Not Bad Not Bad Not Bad Not Bad Bad Bad
What the user Alternative phone Alternative phone Related product – Unrelated Model is too old to be
wants. – same brand – competitor user buying phone product – wrong reasonable alternative.
brand may want a case model case. Recent model is OK.
for that phone.

Page 1 of 10
 User Query: hyatt hotels in Seattle
Booking.com Grand Hyatt Seattle Marriot Seattle restaurant Hyatt Regency Port Townsend, WA hotels
Seattle Waterfront reservations Bellevue
Not Bad Not Bad Not Bad Not Bad Not Bad Bad
Will include One of the Hyatt Alternative brand A user visiting Bellevue is less Port Townsend is over 50
Hyatt hotels in brand hotels in of hotel in correct Seattle might than 25 mi. miles away from Seattle.
Seattle. Seattle. location. need reservations. from Seattle. That’s above the 25 mi. cap
for nearby locations.
{See Section on Locations
& Locality}

But wait! Please review the following information carefully before selecting your rating:

General Rules/Guidelines
Select Not Bad if:
• The Ad offers the product/service the user is looking for.
• The Ad is a partial match to what the user is looking for.
o For example, where a user is looking to compare two products, the Ad is for one of those products.
• The Ad is a competitive offer, alternative option, or product related to what the user is looking for (see sections
below for more details on Competitive/Alternative/Related products).
• The user has specified a location and the Ad offers the product/service within a 25-mile (41km radius) of that
location. [Or is from a national provider with services accessible from anywhere.]

Select Bad if:

• The Ad is for a product/service completely unrelated to the query intent.
• The user has specified a location and the Ad is for a product/service outside of a 25-mile (41km radius) of that
location.
• The Ad seems malicious or trying to trick the user.

Select Junk if:

• The Ad and/or query is not in English or is impossible to fully understand due to extreme misspellings or
extreme junk characters.

Examples
➢ Below you will find 2 sections of examples to help you understand the guidelines:
1. General Examples – a selection of common scenarios from our task.
2. Focus Areas - more information and examples on areas judges may not be familiar with – such as
competitor/alternative/related products, login queries, location specific query/ads, misleading ads etc.

We strongly recommend you review the additional examples and instructions provided below to help build your
knowledge and ensure your success on the project.

Page 2 of 10
 General Examples

Query Ad Comment
Shoes Shop The Largest Vans Selection The user is looking for shoes. While the Ad is for one single brand
Find New Styles & Classic Designs. of shoes and so narrower than the broad query, this may still be of
www.Vans.com interest to the user. Not Bad
Anniversary Fresh Flowers Delivery By FTD® | 15% The user is looking for a specific type of flower arrangement. The
flowers Off Flower Delivery ad is broader to all types of flower arrangements and could
Get 15% Off FTD® Flower Delivery reasonably include anniversary arrangements. Not Bad
Today!
www.ftd.com
Walmart.com Target: Expect More. Pay Less A user looking for Walmart may be interested in offers from
For everything from groceries and Target, as it is a competitive retailer offering a similarly wide range
essentials to clothing and electronics. of products. Not Bad
Choose contactless pickup or delivery
today.
www.target.com
adidas Men's Adidas Shoes A user wanting to shop for men’s Adidas shoes on Overstock may
barricade Up to 40% Off Men's Adidas Tennis be interested in viewing offers on Adidas shoes from this
overstock.com Shoes. Free Shipping. competing website. Not Bad
www.Midwest Sports.com
Puma football Nike football cleats Ad offers correct product from an alternative brand. Not Bad
cleats Find your perfect pair of Nike Football
Cleats. Fast delivery and Free shipping!
www.nike.com/footballcleats
charter one Home Mortgage Refinance A user looking for Charter One Bank may be interested in a
bank Take Advantage Of Low Rates Now! common banking product from a competitor bank. Not Bad
Rates Dropped To 2.88% (3.04% APR).
TheEasyLoanSite.com
Men’s suits Men’s ties | Tie Bar A user purchasing a new suit may also be interested in related
Create your perfect look for any accessories such as ties, cufflinks, etc. Not Bad.
occasion.
www.thetiebar.com
Is the S21 Best Samsung S21 Deals The user is likely researching this phone with a view to purchase.
waterproof 36 months 0% finance. Free next day They may therefore be interested in seeing offers for the phone.
delivery Not Bad
www.buymobiles.com
Galaxy Gear Samsung Galaxy Gear A user looking for product reviews or to compare product (ex.
review Samsung Galaxy Gear At Best Buy® + iPhone vs. Galaxy) may also be interested in purchasing the
Free Shipping. Shop Now product. Not Bad
www.bestbuy.com/Samsung
ecu vs unc ECU Pirates Apparel A user looking for scores for this team may be interested in
Get ready for Pirates football season purchasing fan merchandise. Not Bad
with officially license ECU apparel &
merchandise.
www.fanatics.com/ecu
arcadia ca Top Hotels in Arcadia, CA A user looking for information about the Arcadia, CA airport may
airport also be interested in lodging near the airport. Not Bad

Page 3 of 10
 Earn free nights and get our price
guarantee – book now!
www.hotels.com/arcadia
online photo Photo Editing Software The user most likely wants an online photo editor, but in this case,
editor Download our award-winning photo they may consider downloading a software product for photo
editing software for PC or Mac free editing. Not Bad
www.nchsoftware.com/photopad
Adopt a pet Adopt Virtual Pets The user is looking for information on adopting a real pet in
Bellevue Pick a Pet, Feed, Play, Care for It & Bellevue. The Ad is about adopting a virtual pet, which is not what
More. Fun, Easy & 100% Free! the user is looking for. Bad
www.Guffins.com (Note: that the same words ‘Pet’ and ‘Adopt’ appear in a different
context between the Ad and Query).
Nike Shoes Nike Golf Club The user is looking for Nike shoes. The Ad is for Nike golf clubs.
Save on Top Rated Nike Golf Clubs — There is no reason to think the user would be interested in golf
Guaranteed Low Prices at Golfsmith! clubs. Bad
www.Golfsmith.com/Nike
Ford F-250 Honda Accord The Ford F-250 is a truck, while the Accord is a sedan. The two
Buy a brand-new Honda Accord Today. products are not similar, so would not be considered competitive.
Honda.com/accord-sedan Bad
Audible.com Audible Official Site – FREE TRIAL! A user looking to login to the site will already have an account, so
Login Unleash the power of the spoken word. this ad aimed at new users will not be of interest. Bad
Sign up for a 3-month free trial.
www.audible.com/signup
4th July pictures Art Canvases – 4th July Sale While the ad includes part of the query term, it is in the wrong
20% off with code 4THGET20OOF context. The user is looking for pictures of the 4th of July, whereas
www.afflictionclothing.com/artcanvases the ad for a 4th July sale of art canvases (unrelated to the 4th of
July). This is not what the user wants. Bad

Context & Research

Never assume that an ad is useful just because it contains the query (or part of it), as details can often have different
meanings depending upon the context in which they appear. Where necessary, research both the ad and query to
determine whether or not the ad matches what the user is looking for.

da talent IBM Talent Management
management Reimagine Ways of Working. Enhance
Engagement and Productivity. Reskill
your Workforce.
www.ibm.com/talent-management
Research shows that the user is looking for an entertainment
industry talent agency. This ad relates to corporate talent
management. While the ad includes the query term, the ad is
for a totally different service to what the user is looking for, so
will not be of interest. Bad

Location & Locality

• Ads offering what the user wants within a 25-mile (41km) radius of a specified location can also be considered of
interest. [Use a service such as Bing Maps to measure the distance between locations.]
• Where a location is mentioned in the query but not in the Ad, the product/service offered may still be of
interest if it is from a national provider and so accessible from anywhere.
• When a query is a location, it's reasonable to think that the user may intend to travel to that place. Therefore,
ads for hotels, restaurants/bars, theme parks, museums, historical sites, tourist attractions, guided tours, cruise
lines, travel agents, etc. in the area can be considered of interest.
Page 4 of 10
Query Ad Comment
Buy books Buy Books at Amazon The query intent is to buy books in Seattle. While the Ad is not
Seattle Low Prices on Millions of Books. Free specific to Seattle, the user could still buy books from here. Not Bad
Shipping on Qualified Orders!
www.Amazon.com/books
Chicago, il 100 Hotels in Chicago The user entering this query might be intending to visit Chicago, so
Book your hotel in Chicago online. this Ad for hotels could be of interest. Not Bad
Save up to 50% on your reservation.
www.booking.com/Chicago
statue of The Ritz-Carlton Hotels The query is for a popular tourist attraction and the user may be
liberty Experience Club Level Comfort At The interested in staying in the area. Not Bad
Ritz-Carlton Hotels of NY.
RitzCarlton.com/NewYork
auto air Stay Cool This Summer Though the Ad does not mention the specified location, it is from a
conditioner Schedule an AC Performance Check at national provider. The user can access the information regardless of
repair ann Your Local Firestone. Act Fast. where they are, so this is Not Bad.
arbor FirestoneCompleteAutoCare.com
Mechanic Mechanic, Clayton Idaho Research shows that Clayton is 24 miles from Challis. As the
in Challis Experience mechanic, serving the local locations are less than 25 miles apart, this Ad may be of interest to
Idaho community for over 30 years. the user. Not Bad
claytoncarrepairs.com
Mechanic Mechanic, Boise Idaho Research shows that while both Boise and Challis are in Idaho, the
in Challis Repairs, servicing & parts. Friendly locations are over 190 miles apart. Bad.
Idaho service with a smile!
lhmusedcarsboise.com/mechanic
Audi Knopf Audi The user is looking for Audi dealers in Pittsburgh, PA. The Ad is for
dealers Knoft Automotive - search our new an Audi dealer in Allentown, PA, which is more than 25 miles from
pittsburgh inventory. Pittsburgh. Bad
Knoftaudi.com
Pool 100 Hotels in Chicago The user is looking for a pool cleaning service in Chicago. The Ad is
Cleaning Book your hotel in Chicago online. for hotels in Chicago. Although Chicago is mentioned both in the
Chicago IL Save up to 50% on your reservation. query and the Ad, there is no relationship between the Ad and what
www.booking.com/Chicago the user is looking for. Bad

Plumbers Plumbers The user is looking for plumbers in Kent, Washington. The Ad
Kent WA Since 1976, Top Plumbing Services. displays a phone number from Los Angeles, California so there is a
Call 818-273-1842. location mismatch. (Note it may be necessary to research phone
www.sicariplumbing.ybsites2.com numbers to determine service area.) Bad

Competitive Products & Alternative Options

• Users looking for a specific brand or retailer may be interested in seeing useful Ads from other brands or retailers so
they can see what competitive offers or alternative options are available.
• Users looking for a particular product model may also be interested in seeing Ads for similar products (either of the
same or different brand).
o Rate Ads for different brands/retailers/product models according to how well the product or service
matches with what the user is looking for.
Page 5 of 10
 Query Ad
Taco stop, Dallas Velvet Taco Knox Dallas
Quick Bites, Mexican! No. 1
rated on TripAdvisor.
www.velvettaco.com/knox
adp ezlabor Paychex® Payroll Services
Payroll services and software.
Fast and accurate.
www.Paychex.com/Payroll
usoutdoor.com Back Country Gear
Huge Clearance Sale! Save up
to 45%. Free Shipping.
www.backcountrygear.com
Surface Pro 7 Surface Pro X
Take your business anywhere
with our ultra-thin 2-in1.
www.microsoft.com/surfacex
Taco stop, Dallas Urchi – Fine dining
Exquisite Japanese seafood
and sushi – let us transport
you from Dallas to Tokyo.
www.uchidallas.com
adp ezlabor Credit Karma Tax®
$0 to file both state and
federal tax returns.
www.creditkarma/tax
usoutdoor.com Macys – Fashion for All
Extra 20% off selected lines.
www.macys.com
Surface Pro 7 Surface Duo
Dual-screen Android phone
from the makers of Windows
– Shipping Now!
www.microsoft.com/duo
Comment
The user is looking for Taco Stop in Dallas. The Ad is for a
competitor taco outlet in the same location. It is an alternative
option, which may interest the user. NOT BAD
Research shows that ADP ezLabor is software for managing
payroll. This Ad is for a competitor software also for managing
payroll. It is an alternative option, which may interest the user.
NOT BAD
Research shows that both these retailers specialise in selling
outdoor, camping, ski gear, etc. The user may be interested in
seeing offers from this competitor to compare prices, product
range, etc. NOT BAD
While this Ad is not for the Surface model the user is looking for,
it is from the same product range and a comparable product with
similar features. NOT BAD
While this restaurant is in the specified location, it offers a totally
different cuisine and dining experience to Taco Stop (Japanese
fine dining vs Mexican fast food). It is not at all similar to what
the user is looking for. BAD
Research shows that ADP ezLabor is a software for managing
payroll. The Ad is for a free tax filing service for individuals, which
is a completely different service to what the user is looking for.
BAD
Macy’s is best known as a fashion and home retailer. They are
not especially known for selling outdoor, camping, ski gear etc.,
so is not similar to the retailer the user is looking for. BAD
The user is looking for a Surface Pro 7, which is a Windows based
laptop/tablet. The Ad is for the Surface Duo, which is an Android
based phone. It is a completely different product to what the
user is looking for. BAD

Related Products & Services

• Users may be interested in seeing Ads for related products and services (i.e., a user interested in ‘X’ may also be
interested in ‘Y’).
• This includes, but is not limited to, products commonly purchased together, query is to complete a task and
the ad would help complete that task, query is for a topic and ad covers a related topic etc.
• These relationships are not always reciprocal. For example: a user shopping for a new car might buy car
insurance, but a user shopping for car insurance probably isn’t also looking to buy a new car.

Query Product/Service Comment

2021 Ford Explorer All State Car Insurance A user looking to purchase a new car may also be
Get a quote in seconds. Full Cover. interested in what options are available for insuring a
new car. Not Bad
Page 6 of 10
 www.allstate.com

Hotels Vancouver, Eat out in Vancouver A user looking for hotels in Vancouver may also be
Canada Exclusive offers and discounts for interested in booking/finding out about restaurants
Vancouver’s top restaurants. Book they could visit during their stay. Not Bad
your top table online today!
www.eater.com/vancouver
How to grow roses Rose Bushes For Sale -- Huge A user looking for how to grow roses may be
Selection Of Plants -- Shop Now interested in purchasing rose bushes.
America's Largest Online Plant
Nursery. Shop Trees, Shrubs, Bulbs &
More.
naturehills.com
Creamed honey recipe Buy Creamed Honey on Amazon A user looking to make creamed honey might also be
Read Customer Reviews & Find Best interested in purchasing the product pre-made.
Sellers.
Amazon.com/Grocery
Car Insurance Discover the Full Ford Range A user looking for car insurance probably already has
Small city cars, family cars & SUVs – the car for which they are seeking insurance, so is
we have the new car to suit you! unlikely to be interested in this Ad for a new car. Bad
www.edmunds.com/ford
Restaurants Vancouver, Hotels in Vancouver A user looking for restaurants in a specific location
Canada Same room, different price. most likely lives or already has accommodation in the
Compare and save. area, so is unlikely to be interested in hotels. Bad
www.trivago.com/vancouverca

Ad is Too Broad
You will see generic ads for sellers. If the query is for a specific product/service and isn’t for that specific seller you
should evaluate if the Ad is too broad.
• Consider if the seller carries a very broad range of products (Amazon, Walmart, Target, etc.) or carries a more
focused range of products related to the query. Ask yourself - would the user feel confident clicking the Ad could
lead to their desired product?

Query Title Comment

2-person Amazon.com Official Site - Fast Free User wants to find a tent and isn’t specific to Amazon. Ad
tent Delivery with Prime doesn’t mention tents or related products. Because Amazon
Earth's biggest selection of books, sells thousands of unrelated products the user wouldn’t be
electronics, apparel & more at low prices. confident this generic Amazon Ad would lead to a tent.
www.amazon.com Bad
2-person Amazon.com Official Site - Fast Free In this query, looking at the same Ad, the user has specified a
tent Delivery with Prime desire to shop at Amazon. This makes even a generic Amazon
amazon Earth's biggest selection of books, Ad relevant. Not Bad
electronics, apparel & more at low prices.
www.amazon.com
2-person Cabela's Official Website - Hunting, While this Ad is a generic Ad for Cabela’s, research or reading
tent Fishing, Camping the Ad shows that this is a store specializing in outdoor gear and
the user would be confident they could buy a tent here. Not
Bad
Page 7 of 10
 Cabela's is your home for quality hunting,
fishing, camping, recreational shooting,
and outdoor gear at competitive prices.
www.cabelas.com

Login & Account Queries

For queries where users are looking to login or access information related to existing accounts, consider whether or not
the ad would be of interest to a user who has already signed up to a service.

lifelock.com LifeLock Official Site – Enroll Today!
login Let Us Help Protect Your Identity.
Save up to 25% on your first year.
www.lifelock.com/signup
csaa- Compare Auto Insurance Quotes.
insurance- Get a free online quote today. Find
aaa- your perfect policy in minutes!
com/mypolicy carinsurance.net/quote
A user looking to login to the site will already have an account,
so this ad aimed at new users will not be of interest. Bad
The user wants to go to the specified URL to access
information on their existing car insurance policy. A user
looking to access a current, existing policy will not be
interested in an ad for comparing quotes for a new policy. Bad

Downloads
Unless the user is specifically looking for or might reasonably expect downloadable content to answer their query, ads
that require the download of a toolbar or software should be rated Bad. For example:

Query Title Comment

Fox News Fox Business News The user is required to download a toolbar to answer their
Read About Breaking News & Current query. There is no indication of an intent or expectation of
Events – Download Free Toolbar. download in the query. Bad
www.Headline Alley.com
Driving Free Online Maps The user is required to download a toolbar to answer their
directions Travel Easier With Easy Directions 100 % query. There is no indication of an intent or expectation of
Free & Easy Toolbar Download. download in the query. Bad
Quick-Directions.TranslatorBar.com
evernote EverNote Download The query intent is for a downloadable software, and this
Download the free version of EverNote. download is for the relevant software. Not Bad
download89.com/evernote
Free photo Free photo editing While the user hasn’t directly stated an intent to download
editing Download photo editing software – Free! software, they might reasonably expect it for this query
software Fast and Easy! and this download matches what they are looking for. Not
Download.CNet.com/photoediting Bad
Games to Awesome Puzzle Games Users looking for online games will often consider
play online Download tons of games -- check 'em out downloadable games as an alternative to playing online.
www.egames. com/puzzle/ This download may therefore be of interest to the user.
Not Bad.

Page 8 of 10
 Misleading & Inconsistent Ads
You may come across ads that are either deliberately or unintentionally misleading and inconsistent. For example, an
advertiser may intentionally try to trick the user into visiting a site other than that advertised or may unintentionally
enter a URL with a subdomain that doesn't match the product or service they are offering. In such a case, use your
judgment to consider whether the user would trust the Ad enough to be confident that clicking upon it would lead them
to the expected content. For example:

Query Ad Comment
broken Fractured tailbone The Ad title is relevant to the query, but both the description
tail bone We have all the information you need. Learn and URL refer to an unrelated topic. The user would not be
everything about the endocrine system. confident of reaching the correct topic. Bad
www.everydayhealth.com/en docrine
hsbc hsbc.com The Ad title is relevant to the query, but the URL is for a
online Open an Online Bank Account Free Checks, totally different site which is misleading and inconsistent. Bad
banking ATM & More.
www.BankTech.com
Labels Labels This Ad is relevant to the query and consistent overall, but the
Save On Labels at Grainger. Helping You Get advertiser has entered an incorrect subdomain. This is likely
it Done Since 1927. an unintentional error (remember URLs are provided for
www.grainger.com/Lubrication display only) and the user might reasonably click the link to
access the correct content. Not Bad

Language
Content from all English-speaking markets (United Kingdom, Canada, New Zealand, South Africa, etc.) should be rated
per the guidelines regardless of different currencies, market-specific URLs, and such:

Query Title Comment

The The Amazing Spiderman 2 The Ad addresses the query intent to stream the requested film.
Amazing Rent Online Instantly For £4.49. No The use of UK currency does not change the label. Not Bad
Spider- Subscription. Join & Watch Now.
Man 2 skystore.com/Amazing-Spiderman-2
amazon Amazon.com.au – Amazon Official Site The Ad matches what the user is looking for, it does not matter
Huge Selection and Low Prices Free UK that this is the Australian site. Not Bad
Delivery on Eligible Orders
www.amazon.com.au

Junk & Non-English Content

You may come across ads and queries containing misspellings, junk characters, transposed letters, misused punctuation,
or truncated information. In these cases:

• Small mistakes like missing or misused punctuation (commas, hyphens, semicolons, missing or extra periods,
etc.) and misspellings that do not prevent the understanding of the Ad can be ignored. If a user could
reasonably understand the Ad, it can be rated Bad or Not Bad as appropriate.
• If the errors and/or omissions are so extreme it is not possible to understand the Ad at all, apply the Junk rating.
• Also apply the Junk rating if the query and/or Ad is not in English (unless it is for a commonly understood term in
the market, for example, “Les Misérables” (movie/book), “salsa” (popular food), “Aldi” (grocery store), etc.
Page 9 of 10
Query Ad Comment
book a flight Save 70% on flights The Ad title and description appear related to the query, but the URL is
calling united Book Your Flight Today! Use totally missing making it impossible to fully understand the Ad. Junk
airlines code: CallCenter15

?????? ???? Severe Skin Itching The query is just a string of question marks and makes no sense. The Ad
???? Find Facts, symptoms & therefore can’t be fully understood. Junk
Treatments. Severe Skin
Itching Help.
Lifescript.com/Health
snow atv piezas Snow Atv The query includes non-English language terms that cannot be
barata Great deals on parts from understood. Junk
top online merchants.
www.exava.com
Juegos.com Juegos Gratis en Flash The query is for a non-English (Spanish) website and the Ad is non-
Chatea y haz amigos jugando English. Junk
online. Registro Gratis!
www.mundiju egos.com
Puerco pibil Tasty Mexican Recipes Though the query is non-English (Spanish), a search reveals that it is a
Scrumptious Mexican dish for which many English language recipes are available. The Ad
Recipes To Make Any Meal A matches the query intent to find recipes. Not Bad
Success
Kraft!KraftRecipes.com
michael Are you Michael? While the URL is missing the top-level domain (.com) it is still possible
slossberger See what others will see to understand this Ad and see that it is relevant to the query intent to
about you. Get security find out about a person called Michael. Not Bad
monitoring alerts.
www.identity
cnet.com Work At Home Moms Though the description contains an incomplete sentence, we can still
Gain Access To A Work At understand what the Ad offers and see that it has no relevance to the
Home Job Paying As Much query. Bad
As $87 an
www.consumersearch.com
Driving Free Online Maps Both the query and Ad description contain errors, but they are not so
directions Travel Easier?”” With Easy severe that the content can’t be understood. We can see that the user
Direction 100 % Free & Easy is required to download a toolbar to answer their query (when there is
Toolbar Downlaod. no intent for a download). Bad
Quick-
directions.TranslatorBar.com

Page 10 of 10