These guidelines contain confidential and proprietary information belonging to Microsoft Corporation.

The recipient understands and

agrees that these materials and the information contained herein may not be used or disclosed without the prior written consent of
Microsoft Corporation.

In this task you will review pairs of terms. One is a search query and the other is an advertiser keyword. You
will identify if a query/keyword pair is "bad". A "bad" pair is one where they aren't related to each other. This
will help us improve the usefulness of the ads offered and prevent the display of unrelated ads to users.

Rating Tips
➢ In these instructions QY = query and KW = keyword.

➢ You may see "nn" in front of numbers which can be ignored. You may see a “d” in place of a decimal
point. ("new apple ipad nn10d2 inch wi fi 32gb". “nn2006 chrysler nn300 touring nn2d6 maintenance
manual).

➢ Please Research as Needed! This task is for terms coming from the United States, keep that in mind
for research purposes. You will see many unfamiliar terms that you will need to research. You can
click on they query and keyword to see search results to assist you.

Rating Definitions
This section contains an overview of each rating. To help you be more successful on the project we also provide
a review of common areas of confusion and more detailed examples in the sections below.

For each pair you will:

1. Review the QY. Performing any research needed to help your understanding.
2. Review the KW. Performing any research needed to help your understanding.
3. Select your rating based on the relationship between the QY and KW,

➢ If the KW is for a relevant product, service, topic etc. to a user searching for the QY choose: NOT BAD
o Not Bad categories include:
▪ The KW is what the user wants.
▪ The KW is a broader product category or topic that would reasonably include the QY.
▪ KW is partially what the user wants, offering a narrower product selection or topic.
▪ KW offers a competitive/alternative product, brand, or service to the one in the QY; or
KW offers an alternative product from the same brand.
▪ KW is not what the user is looking for, but offers other products, services or information
that can be used together with the QY.
▪ QY is for a product/service in a specific location and the KW is for the correct
product/service in a nearby location within a 25 mi./40 km. radius.

➢ If the KW is unrelated to the QY choose: BAD

o Bad categories include:
▪ KW and QY have no relationship to each other. It includes but is not limited to:
• product mismatch, topic mismatch, location mismatch etc.

➢ If both the QY and KW are sexually explicit choose ADULT.

➢ If the KW and/or QY is not in English or consists of junk/nonsense text choose JUNK.

 Common Areas of Confusion

Not Bad
The Keyword is relevant to the user’s search query.
This can include:
broader or narrower product selection, competitor or alternative products, accessories, products
often purchased together and related topics as well as direct matches to the search term.
QY KW Comment

jcpenney online shopping bedding Not Bad (narrower product selection) - JCPenney sells home
goods, clothing etc. KW is one category of product sold here.
spyder men polar jacket review backcountry Not Bad (broader product selection) – Backcountry is a store
that sells outdoor clothing like the jacket the user is looking for.
salomon skis skis Not Bad (narrower product selection) – Skis are one product
made by the Salomon brand.
ripped jeans forever 21 Not Bad (broader product selection) – Jeans are one type of
clothing sold by Forever 21.
best travel apps travel Not Bad (related product) – A user looking for an app to plan a
insurance trip might purchase insurance.
secretary day unique gifts Not Bad (related product) – This user may be interested in
shopping for a gift for their employee to celebrate this day.
Target online shopping Forever 21 Not Bad (competitors) – These are two competitors that both
sell clothing among other products.
raised twin bed headboard Not Bad (accessory) – A user purchasing a bed might also
purchase a headboard for that bed.
secluded private owned cabins booking Not Bad (alternative) – The user might be interested in seeing
nc com hotel other type of lodging options in this location on the KW site.
animal kingdom resort hotel disney Not Bad (direct match) – Search results show these both refer to
animal the same hotel.
kingdom

Bad
The Keyword is not relevant to the user’s search query.
QY KW Comment

large elk antlers sale holiday Bad - There is no relationship between the terms.
pcr test near 32829
document stand holder
high back ergo chair
spider web back wood chairs
Buy snow tires online
c peptide Bad - Research shows these two tests are for completely unrelated
testing medical conditions (COVID 19 vs. Diabetes type)
blueprint Bad - Research shows these are very different products and one
storage shelf cannot be used in place of the other nor are these the type of
products commonly bought together.
lenovo com Bad - Research shows KW site sells computers. There is no
reason to think a user shopping for an office chair is also planning
to buy a computer.
game chairs Bad - Gaming chairs aren’t relevant to the search term – which is
for formal chairs such as those used in a dining room.
US Bad - KW is too broad – it could represent thousands of different
things. We cannot see a relationship between QY and KW.
 We strongly encourage reviewing the full examples with explanations provided below in
order to improve your chance of success on the project.

Additional Examples & Explanations

Not Bad

As long as the KW is for a relevant product, service, topic etc. to a user searching for the QY, it should be
labeled NOT BAD. Below are common QY/KW scenarios for the NOT BAD rating. Please review the given
topics and examples to better understand the task.

1. The KW is what the user wants.

Query Keyword Comment

Nike.com Nike Direct match to user intent
Shoes Shoes eBay eBay sells many types of shoes. Direct match to user
intent.
Plane tickets Flight tickets Direct match to user intent
Hilton hotel Hilton Direct match to user intent
Pants ski Winter ski pants Skiing happens in winter
London Edition London Edition Same hotel
Fitzrovia London
Pumpkin bread Best pumpkin bread Research shows this is what most users want.
recipe
Douglas airport clt Charlotte intl airport Both are names for the same airport.
Students rating colleges Niche.com KW is a website based on student ratings.
Hotels Fairfield Fairfield hotels There are multiple locations called “Fairfield”. We can
assume both reference the same “Fairfield.
www.aaa.com www aaa Research shows KW may return QY.

2. The KW is a broader product category or topic that would reasonably include the QY.

Query Keyword Comment

Maybelline cat eye mascara
www.shutterfly
Tag Heuer watches reviews
Hotels NYC
Flights London
Santo Domingo Hotel
Dog Grooming Long Beach
CA
Sea World San Diego
Ralph Lauren bernadine
riding boots
2011 Ford F 250 specs
best prices d batteries
Ford F-250
Maybelline cosmetics QY is for one specific make-up product. KW is broader
as it covers all make-up from this brand.
Picture printing service QY is for a picture printing site. KW is for all printing
service sites.
Tag Heuer watches KW is broader for all information on Tag Heuer
watches, including reviews.
Hotels KW is broader and would include hotels in all locations.
London KW is broader and would include information on travel
to London.
Discount hotels KW is broader as it offers hotels in all locations.
Dog Grooming KW is broader for dog grooming in all locations.
San Diego KW is broader for all information on San Diego,
including points of interest.
ladies shoe boots QY is for a specific pair of women’s boots. KW would
encompass all women’s shoes.
Ford F-250 KW would include all information on this vehicle,
including specs.
cheap alkaline batteries KW is broader. Information on D battery prices would
be part of information on Alkaline batteries. “Best”
and “Cheap” are marketing words that can be
ignored.
Ford.com The F-250 model would be included in the broader Ford
website.

3. KW is partially what the user wants, offering a narrower product selection or topic.

Query Keyword Comment

H&M H & M shirts KW is narrower, offering one product type from H & M
Brooks sneakers Brooks Bedlam KW is one model of Brooks brand sneakers.
Adiovox SIRHK1 Research shows KW is a model number for an
Audiovox product.
Men’s shoes Allis Comfortech Cap Research shows KW is one pair of men's shoes.
Toe Oxford
Chicken Chicken recipe KW offers one topic related to the QY.
Game of Thrones Game of Thrones KW offers one topic related to the QY.
episodes
Coffee near me Starbucks coffee KW offers one of many coffee shops.
Dating site Match.com KW offers one of many dating sites.
Nike Basketball Basketball is one type of gear sold by Nike.
Liberty Travel Airline ticket price Research shows QY is a travel agency. KW is one
Poughkeepsie NY service we can reasonably expect to be offered.
Starbucks.com Starbucks store locater Store location is one type of information on the
Starbucks site.
Cancer.org Cancer treatments QY is site for American Cancer Society. Information on
cancer treatments is one type of information founds on
the site.
Hotels Hotels NYC KW is narrower as New York City is one of many
locations.
Campgrounds northwest Destin camping Research shows Destin is located in NW Florida – it is
Florida one possible camping location.
Santo Domingo Hotel Occidental El Embajor Research shows the KW is the name of one hotel in this
location. This is not considered a Non-English KW
because it is the proper name of the hotel.
Frequent small voidings Tolterodine QY is for information on this topic, which includes
suggest treatment options. KW is one treatment option.
Nerve pain after hip surgery Sciatic nerve pain QY is for information related to nerve pain after hip
exercises surgery - such as general information, diagnosis, and
treatments. The KW offers a treatment option for nerve
pain for one of the nerves found in the hip.

4. KW offers a competitive/alternative product, brand, or service to the one in the QY; or KW offers an
alternative product from the same brand.
• In order to be competitive/alternative the KW must offer similar usability.
• In order to be competitive/alternative there must be an overlap in services/products offered between QY
and KW.
Query Keyword Comment
Delta airlines reservations British Airways Competitor companies.
Metoprolol Coreg Research shows these are competitor medications for
high blood pressure.
USB 128gb flash drive 256gb flash drive USB KW offers an alternative product.
Dell laptop HP laptop Competitor products.
Dell laptop HP tablet Alternative product – tablets offer similar capabilities to
laptops.
Adidas samba Nike Competitors – Nike offers similar products to the
Adidas Samba.
Gucci sunglasses Amazon Gucci sunglasses eBay Amazon and eBay are competitor merchants.
Frog bracelet Magnetic rubber KW offers an alternative style of bracelet.
bracelets
Most popular acne face Acne creams KW offers an alternative product that could be used for
wash the same purpose.
Free drum kits Used drum kits KW offers an alternative.
 5. The KW is not what the user is looking for, but offers other products, services or information that can be used
together with the QY.
• Think of this as: If you like "X" you may be interested in "Y".

Query Keyword Comment

Tobacco pipes Buy pipe tobacco A user shopping for a pipe might be interested in also
purchasing tobacco for it.
iPhone iPhone case A user shopping for a phone might also purchase a case
for it.
Asus Vivobook Asus Vivobook battery KW user shopping for a laptop might purchase
accessories like a spare battery.
reasons swollen lymph lymph nodes locations KW offers a related topic that could be of interest to the
nodes user.
men’s dress shirts men’s ties Ties are often purchased to be worn with suits
Ugg boots Ugg coupons User might be interested in a coupon for this product
Six Flags Atlanta Atlanta hotels QY is a tourist attraction, a visitor might also need a
hotel.
Austin hotels Austin restaurants A user booking a hotel may need information on
restaurants in the area they are visiting.
Space Needle Pike Place Market QY is a popular tourist attraction. Pike Place Market is
another popular tourist attraction in the area. Users
planning to visit the Space Needle may also be interested
in other local attractions.
Matthew McConaughey True Detective season 1 QY is for information on the latest movies from
latest movie McConaughey. A user interested in his movies may also
be interested in his TV shows.
Weight loss soup diet Walmart weight loss Users looking for diet plans may also be interested in diet
supplements supplements.

6. QY may be for a specific location and the KW is for a different, nearby location.
• Locations within (25 mi./40 km) of the requested location are acceptable. Locations further away are not
acceptable.
• Radius should be determined by measuring the distance from location to location using a mapping site.
• Mapping should start at a city level - for example if the QY and KW are for two hotels, you can map
their distance from the city, not street address level.
Query Keyword Comment
Seattle hotels Bellevue hotels The KW offers the same product (hotels) in a nearby
location. Bellevue is within a 25 mi./40 km. radius of
Seattle.
Boston apartments Newton apartments The KW offers the same product (apartments) in a
nearby location. Newton is within a 25 mi./40 km. radius
of Boston.

7. QY is only a geographic location name (ex. "London")

• The KW may be for travel intents (flights, hotels, car rentals, points of interest, tours etc.)
• The KW may be for real estate intent (purchase home, rent apartment, buy property etc.)
• The KW may not be for any other type of business. It is very unlikely a user looking for a specific
service/company (ex. plumber, lawyer etc.) would search for it by only entering a general location term.
 Query Keyword Comment
London Tower of London tour QY is location name and KW offers a travel/tourism
intent.
Santa Barbara Kimpton Canary Hotel QY is a location name and KW offers a travel/tourism
Santa Barbara intent.
Springfield homes for sale QY is a location name and KW offers a real estate intent
with no conflicting location.

Bad

➢ KW and QY have no relationship.

o It includes but is not limited to: product mismatch, topic mismatch, location mismatch etc.

Below are common QY/KW scenarios for the BAD rating. Please review the given topics and examples to
better understand the task.

1. It is very unlikely the user searching for the query term, would be interested in advertising related to the
KW.

Query Keyword Comment

iPhone case iPhone x A user shopping for a phone case is very unlikely to want
to purchase a phone.
Car insurance New cars It is very unlikely a user looking for car insurance will
want to purchase a car.

However - a user shopping for a car may also need car

rv insurance
Get revenge your ex
Austin restaurants
insurance
much does it cost rent rv A user looking to buy insurance usually owns the vehicle,
san diego it is very unlikely this user wants to rent an RV when they
are most likely already an owner.
Getting ex-girlfriend It is very unlikely a user looking to get revenge on an ex
back would also be interested in information on reconciling
with an ex. The QY & KW are unrelated.
Austin hotels It is very unlikely a user searching for restaurants will also
need lodging, many users will be local to the area.
However, a user looking for hotels might be interested in
restaurants too.

2. Important Attribute/Feature Mismatch

• While the general product may be the same, there is a mismatch on an important feature – for example
size, gender, accessory/part for wrong model, cannot be used for the same purpose etc.

Query Keyword Comment

Mariners tickets Red Sox vs. Angels KW is the wrong team. A user is very unlikely to be
tickets interested in unrelated teams.
Timberland shoes men Bria Chukka sneaker The QY is for men's shoes. Research shows KW is a pair
of women's shoes.
size 7 wide shoes Womens Size 5 Shoe Wrong size. The user won’t want shoes in a different size
than they searched.
HP Envy 5055 ink HP Officejet Pro 8035 Not the same model of printer. A user won’t want to shop
ink for ink for a different printer.

3. Similar looking terms or text string with unrelated meanings.

Query Keyword Comment

New Jersey warranty
mazda oil change
Manakin
Bench Warrant Terms look similar but have unrelated meanings.
mazda cx9 change oil Text string looks similar but a user wanting an oil change
press sensor is not related to replacing an oil press sensor.
Mannequin “Manakin” is a type of bird. This is unrelated to the KW.
Since the query is a legitimate word, we cannot assume
the QY is a misspelling of the KW.

4. KW offers an incorrect product or service without the same usability as the QY.

Query Keyword Comment

www face cbook com wwwyahoo com Don’t offer similar content or services.
Dell laptop HP desktop The KW offers a product that is not similar enough to a
laptop. Though both are computers; portability is a key
function of a laptop that a desktop does not share.
Pole lamps Antique street lighting Pole floor lamps are used indoors. The KW is for a type
of lighting primarily used outside.
Toyota HondaJet While Toyota and Honda offer many competitive
products, research shows that Toyota does not offer
planes.

5. KW term is too broad.

• KW term is so broad it represents thousands of unrelated products, locations, topics etc. We cannot
determine any relationship between the KW and QY.

Query Keyword Comment

lake ouachita KW is too broad. It would cover every location/topic for
us
campground the US. We cannot determine a relationship.
Pool covers The query is for a specific product. The KW offers an
online shopping site that sells every type of product. The
QY & KW are too far apart to be related.
Amazon Note: If the KW had been a store more targeted to the QY
(ex. pool supplies, hardware/home supplies etc.) it would
be Not Bad as there would be a clearer relationship
between the two.

6. Location Mismatch
• The QY and KW contain unrelated locations.
• The QY and KW contain locations that are more than 25 mi. or 40 km. apart.

Query Keyword Comment

Car dealerships Pittsburgh Car dealerships LA Location Mismatch.
 UCSD Marshall college Marshall edu Location Mismatch. QY is located in California and the
KW is located in West Virginia. Location is important
for businesses that serve a specific geographic location.
Seattle WA hotels Everett WA hotels Everett is not within a 25 mi./40 km. radius of Seattle.

7. Adult vs. Non-Adult mismatch

• When QY is adult and KW is non-adult
• When KW is adult and QY is non-adult

Query Keyword Comment

Massage Voyeur sex massage The query isn’t inherently adult in nature. The KW is for
pornography. Non-Adult/Adult mismatch is always Bad.

Adult
➢ Choose this rating when BOTH the QY and the KW are adult in nature.
o A QY and/or KW is categorized as “Adult” when it offers or refers to products/services/sites that
are sexually explicit or are intended to cause sexual excitement.
➢ The QY & KW do not have to match in intent to qualify for the Adult label; they only must share the
quality of being adult.
➢ If only the QY or only the KW is adult in nature then it should be labeled Bad rather than Adult.
Adult/Non-Adult mismatches are always Bad.

Query Keyword Comment

www freeextremecams Free sex cams Webcam site focused on adult acts and activities. The KW,
com for sexcam sites, is also adult in nature.
Real homemade porn Homemade videos The URL in the query does not exist. We can assume the
com query intent is to find adult sites. Research shows the KW
term is most commonly associated with homemade
pornographic videos.
Mini upskirts Swinger porn “Upskirt” photos or videos are adult in nature. The KW is
for pornographic videos. Note: the QY and KW do not
have to match in intent for the Adult label.

Junk
➢ The QY and/or KW are in a language other than English.
➢ The QY and/or KW are for a Non-English website.
➢ The QY and/or KW is junk/nonsense text and can't be understood.
➢ This does not include: proper names/titles (ex. Les Misérables), brand names (ex. L'Autre Chose) or
terms that are commonly used in English speaking locations and have many English language results
(ex. salsa).

Query Keyword Comment

Casas de venta en el Hotel en el centro precio QY and KW are not in English.
Salvador

Breckenridge CO summer Hotel buen precio en KW is not in English.

activities Aspen
hert9ya45 Hertz Research shows no results for the QY, it is junk/nonsense.