Ad Keyword - Guidelines

Last Updated: 8/8/2017

Last Updated By: Tracy Ortman (v-traort)

These guidelines contain confidential and proprietary information belonging to Microsoft Corporation.

The recipient understands and agrees that these materials and the information contained herein may not be used
or disclosed without the prior written consent of Microsoft Corporation.

1
Table of Contents
1 Overview ................................................................................................................................................................................... 3
2 Understanding the query ........................................................................................................................................................... 4
3 Evaluate the Query .................................................................................................................................................................... 5
3.1 Query Meaning .......................................................................................................................................................................... 5
3.2 Determine the query intent ....................................................................................................................................................... 5
3.3 Evaluate the Level of Specificity ................................................................................................................................................ 5
4 Understanding the Ad Keyword ................................................................................................................................................. 6
5 Rating Options ........................................................................................................................................................................... 7
5.1 Excellent .................................................................................................................................................................................... 7
5.2 Good .......................................................................................................................................................................................... 8
5.3 Fair ............................................................................................................................................................................................. 9
5.4 Bad........................................................................................................................................................................................... 10
5.5 Adult ........................................................................................................................................................................................ 11
5.6 Not Sure ................................................................................................................................................................................... 12
6 Competitive/Complementary .................................................................................................................................................. 14
7 URL Queries/URL KWs ............................................................................................................................................................. 15
8 Location ................................................................................................................................................................................... 16
9 Nearby Locations ..................................................................................................................................................................... 17
10 Mixed Rating Examples ............................................................................................................................................................ 18
11 Glossary ................................................................................................................................................................................... 22

2
1 Overview
As a judge working on the Ad Keyword project, you will see queries that users type into search engines, and keywords that the
advertisers have purchased to address the query. You will evaluate how satisfied users would be with each Ad Keyword based on the
query they entered, and so help us evaluate and improve the quality of the Ad Keywords that our advertisers purchase.

These guidelines describe how to accurately determine user intent and assess the relevance of the Ad Keyword to that intent. While
the task may seem daunting, there are really only four steps:

1. Consider what the user could have had in mind (their intent) when they typed in the query; allow for misspellings or other
ambiguity.
2. Complete research to best understand the meaning of the query.
3. Determine whether the Ad Keyword covers any of these intents, and if so, how well it satisfies them.
4. Give the Ad Keyword a rating on a six-point scale to indicate how useful the Ad Keyword would be to most users who type in
the query.

The rest of the document explains how to perform each of these steps. Keep these basic tips in mind:

• When first considering the query and Ad Keyword, complete research on both items to provide better understanding.
• You will often need to research possible intents associated with the query.
• The rating you are providing is based on the Ad Keyword. Consider if the query and Ad Keyword are a perfect match. Is the Ad
Keyword broader than the query? Is the Ad Keyword narrower than the query?
• You should select your label based on the relationship between the query & Ad Keyword.
o The query and Ad Keyword may contain very different words, but still represent related concepts.
o You are not labeling how similar two text strings are, but how closely the query and Ad Keyword are related.

We have included examples in these guidelines; you should use these to understand the concepts, and not expect an example for
every possible type of query or Ad Keyword. The example intents we’ve listed for queries are not exhaustive; you will probably be able
to identify more.

3
2 Understanding the query
A query is a search term entered by a user into a search engine. A query can be entered in many formats and can provide you with
information to get you started on what the user is searching for. When considering a query, keep in mind the following:

• A query can be one word or a set of words.

o If a query is a set of words, read the query as one entity e.g. “Susan Jones” (name of a person), “Curtain call” (single
entity), and “Yellow Rose of Texas” (song title).
• Check each query for misspellings – Remember to complete research before deciding if a word is misspelled.
o For minor misspelled words, you are permitted to correct the spelling mistakes.
• Disregard capitalization and punctuation.
• Normalization
o Numbers may be preceded with the characters “nn”. Judges should ignore these characters.
▪ Example: “youtube just dance nn4”
o You may see unrelated strings of text proceeded by an asterisk. Judges should ignore these characters.
▪ Examples: *RJNS, *RBNR
• You are responsible for understanding the query intent. Even if you think you know what the user intended, be sure to
perform complete research.

4
3 Evaluate the Query
Before you review the Ad Keyword, you must determine three aspects of the query to help you understand what the user was thinking
when they typed the query into the search engine. Users with identical queries may not be looking for exactly the same thing and
some user intents may be more popular than others.

3.1 Query Meaning

To begin your evaluation, you should understand the meaning of the query. To assist you in understanding, complete the following:

• Is the query ambiguous?

• Complete research by copying and pasting the query into one or more search engines.
• View the generic result list. Is there one result that is predominant? Is one meaning more likely than the others? (If you cannot
determine the meaning from reviewing the generic result list, you may click on any of the links).
• What is the core concept?
• What are the attributes of the query?
• What would most people think the query means?
• Within the task application, you may be provided with the top 5 results from a search engine. You should not limit yourself to
these results and also research the query outside of the task application.

3.2 Determine the query intent

To determine the intent of the query, you need to consider why the user entered this query, and what results the user would be
hoping to see. Most query intents can be classified into categories. In this task, there is no categorization required that would change
the relevance of the Ad Keyword to the query.

3.3 Evaluate the Level of Specificity

When reviewing the query, consider how broad or how narrow the Ad Keyword is, based on what the user was searching for:

• Do the query and the Ad Keyword match?

• Is the Ad Keyword broader than the query?
• Is the Ad Keyword narrower than the query?
• Disregard marketing terms like “cheap”, “best”, “highest quality”, “superior”, “great”, and “greatest”. The meaning of these
words is arbitrary. What may seem “cheap” to one user may not seem cheap to another. This list is not exhaustive; there can
be more marketing terms to add to this list.
• “New”, “Used”, “Free” and “Wholesale” are not marketing terms. These terms are essential attributes in defining a product
type. For example “Used bicycle”, “New car”, “Used clothing”, “Free download”, “Wholesale furniture”

Query Ad Keyword Specificity Comment

Shoes Shoes Same The QY and KW match exactly in terms of specificity.

Shoes clothes Broader The KW is broader than the QY. Shoes are one type of clothing. Shoes are part of the broader product
group “clothes”.
Shoes Macys.com Broader The KW is broader than the QY. Macy’s sells a broad selection of products, including shoes.

Shoes Men’s shoes Narrower The KW is narrower than the QY. The product group “men’s shoes” is part of the broader product group
“shoes”.
Shoes Nike Shoes Narrower The KW is narrower than the QY. Nike shoes are one brand of shoe and are part of the broader product
group “shoes”.
Shoes Nike Air Max Narrower The KW is narrower than the QY. “Nike Air Max” is one style of athletic shoe and part of the broader
product group of “shoes”.

5
4 Understanding the Ad Keyword
This task involves reviewing an Ad Keyword and providing a rating based on a rating scale. When reviewing the Ad Keyword, you will
rate how relevant the Ad Keyword is based on the query intent of the user.

• An Ad Keyword can be one word or a set of words.

• If an Ad Keyword is a set of words, read the Ad Keyword as one entity e.g. “Susan Jones” (name of a person), “Curtain call”
(single entity), and “Yellow Rose of Texas” (song title).
• Check each Ad Keyword for misspellings – Remember to complete research before deciding if a word is misspelled. For minor
misspelled words, you are permitted to correct the spelling mistakes. You should determine if the misspelled keyword would
still pull relevant results.
• Disregard capitalization and punctuation.

Remember!
• You must give the Ad Keyword a rating based on a six point rating scale: Excellent, Good, Fair, Bad, Adult and Not Sure.
• You are responsible for understanding the Ad Keyword (even if you think you know what the Ad Keyword means, do complete
research).
• Research should be done to determine the meaning of the Ad Keyword.
o Judges should not base their rating on the Ads pulled by this Ad Keyword that they may see displayed during this
research. Judges should base their rating on the criteria outlined in the rating guidelines below.

6
5 Rating Options
5.1 Excellent
The Excellent Ad keyword has the same meaning, intent and specificity as the query.

• A user is most likely to be interested in advertisements related to the Ad Keyword.

• The Ad Keyword matches the most likely intent of the query.
• The query and the Ad Keyword are an exact match in meaning and intent.
• The specificity of the Ad Keyword matches the specificity of the query, that is, the Ad Keyword is neither broader nor
narrower than the query.
• If an Ad Keyword is misspelled but is still an exact match in meaning, intent and specificity, it can be given a rating of
Excellent.
• If an Ad Keyword is a spelling alternative or synonym but is still an exact match in meaning, intent and specificity it can be
given a rating of Excellent.

Meaning Intent Specificity

Query is clear KW is an exact match to the intent of KW is an exact match in specificity to

the query the query; is not broader or narrower
than the query

Query Ad Keyword Comment

mariners tickets mariners ticket The intent of the query is to purchase Mariners tickets. The change from
plural to singular equally captures the intent of the query.
mariners tickets seattle mariners tickets The intent of the query is to purchase Mariners tickets. Research shows
“Mariners” and “Seattle Mariners” refer to the same team.
mariners tickets seattle baseball tickets The intent of the query is to purchase Mariners tickets. Research shows
that the Mariners are Seattle’s baseball team.
mariners tickets cheap mariners tickets The intent of the query is to purchase Mariners tickets. “Cheap” is a
marketing term and can be ignored, it doesn’t narrow the selection of
tickets.
Nike.com Nike The QY & KW are an exact match. Both encompass all Nike products and
services.
shoes Macy’s shoes The most likely query intent is to shop for shoes. Macy’s has a national
and online presence covering most locations and they offer a broad range
of types of shoes from many brands. The KW and QY are an exact match.
plane tickets flight tickets The QY & KW are an exact match. The terms “plane ticket” and “flight
ticket” are synonyms.
sheraton hotel sheraton The QY & KW are an exact match. Sheraton is a hotel chain so the QY &
KW match in terms of meaning, intent and specificity.
pants ski winter ski pants The QY & KW are an exact match. Since skiing is a snow sport, the
addition of the term “winter” doesn’t make the KW narrower than the
QY.
london edition hotel london edition fitzrovia london Research shows that the QY refers to one specific hotel named “The
London Edition” located in Fitzrovia. The QY & KW are an exact match.
pumpkin bread best pumpkin bread recipe The most likely query intent is to find recipes for pumpkin bread. “Best” is
a marketing term that can be ignored, so the QY and KW exactly match in
specificity.
mailman mailmen The QY & KW are an exact match in terms of concept. In this case the fact
that the KW is plural doesn’t change the fact that this is an exact match,
because they both represent the exact same concept.

7
5.2 Good
The Good Ad Keyword does not match the specificity of the query and/or addresses one intent of a multiple intent query.

• A user is likely to be interested in advertisements related to the Ad Keyword.

• The Ad addresses one intent of a multiple intent query.
• The Ad Keyword is broader or narrower than the query.
o The Ad Keyword is about the right product/service/site, but without the correct level of specificity.
o The Ad Keyword may not mention the exact product/service/site, but offers a broader or narrower product
category.

Meaning Intent Specificity

Query may have multiple meanings Addresses a possible query intent Broader or narrower in specificity than
the query

Query Ad Keyword Comment

mariners tickets seattle mariners baseball The intent of the query is to purchase Mariners tickets. The KW is
broader than the QY – information on the team in general would include
ticket sales.
mariners tickets Mariners The intent of the query is to purchase Mariners tickets. The KW is broader
than the QY - information on the team in general would include ticket
sales.
mariners tickets seattle tickets The intent of the query is to purchase Mariners tickets. The KW is
broader than the QY – tickets to the Mariners would be a part of the
broader product group of tickets for events in Seattle.
mariners tickets red sox vs. mariners tickets The intent of the query is to purchase Mariners tickets. The KW is
narrower than the QY – offering tickets for one specific match-up.
forever ever 21 forever 21 shirts The KW is narrower than the QY offering one type of clothing (shirts) of
the many types of clothing offered by this store.
maybelline cat eye maybelline cosmetics The KW is broader than the QY. The QY is one product offered in the
mascara broader product group of Maybelline cosmetics.
chicken chicken recipes The query has multiple intents; the user may be looking on information
on chickens, chicken recipes, raising chickens etc. The KW addresses one
intent of this multiple intent query.
mentalist cbs mentalist episodes The query has multiple intents, such as finding information, news and
gossip or episodes of this show. The KW addresses one intent of this
multiple intent query.
www snapfish com picture printing service KW is broader than QY and would include more vendors than the one
specified in the QY.
donuts dunkin donut Research shows the most likely intent is to find stores selling doughnuts.
The KW is one option, but may not cover all the types of donuts or
locations.
Car insurance All state car insurance The KW is narrower than the QY. Though Allstate offers car insurance, it
is just one of many car insurance options.
Dating site Match.com The KW is narrower than the QY. Though match.com is an online dating
site for a general population, it is just one of many options.
akribos watches akribos xxiv watches The QY intent is to find reviews on Akribos watches. Research shows that
review Akribos and Akribos XXIV are interchangeable terms. The KW is broader
than the QY, but reviews would be found within the broader information
on these watches.
parfum vs eau de eau de toilette The QY intent is to find comparisons of perfume v. eau de toilette. The
toilette KW is narrower than the QY, but addresses a possible intent of the query,
such as finding eau de toilette for purchase.
Nike Golf The KW is narrower than the QY. Golf products (shoes, apparel, clubs,
8
 etc.) are just one type of product Nike carries.
Printable 4th grade education The KW is broader than the QY. These types of worksheets are just one of
writing sheets many types of educational tools.

5.3 Fair
The Fair Ad Keyword does not address the query intent, but still has a weak relationship to the query.

• A user is less likely to be interested in advertisements related to the Ad Keyword.

• The Ad Keyword may be for an alternative or competitive product/service/site.
• The Ad Keyword may be for a complementary product/service/site
• The Ad addresses a similar intent to that of the query.
• The Query and Keyword are not Disjoint/Unrelated.

Meaning Intent Specificity

Query may have multiple intents Does not address the query intent, weak Does not align in specificity, weak
relationship to the query relationship

Query Ad Keyword Comment

mariners tickets seattle mariners schedule The intent of the query is to purchase Mariners tickets. The KW doesn’t
address that intent, but offers complementary information. Users looking
to purchase tickets may also be interested in viewing the schedule.
tobacco pipes buy pipe tobacco KW does not address the QY intent, but users looking to purchase a pipe
may also be interested in purchasing tobacco to use with it.
plymouth scamp plymouth scamp covers KW does not address the QY intent, but users looking for the Scamp may
be interested in accessories or parts.
most popular acne acne creams KW does not address the QY intent, but users looking for this information
face wash may be interested in alternative acne treatments to face wash.
air tran airlines american airlines QY intent for Air Train airlines reservations. The KW is for a competitor,
reservations American Airlines.
reasons swollen lymph nodes locations KW does not address the QY intent to find information on why lymph
lymph nodes nodes might become swollen, but users may be interested in other
information related to lymph nodes.
hydralazine coreg KW does not address the QY intent, but offers a different medication also
used to treat high blood pressure.
usb 128gb flash drive 8gb flash drive usb KW does not address the QY intent, but offers a similar product.

9
5.4 Bad
The Bad Ad keyword is unrelated to the intent of the query.

• A user is very unlikely to be interested in advertisements related to the Ad Keyword. The ad is very unlikely to receive any
clicks.
• In some cases, the Ad Keyword mentions the search terms, but has a totally different meaning than what the user intended.
• There is no relationship between the query and Ad Keyword.

Meaning Intent Specificity

Ad Keyword has a totally different There is no relationship between the There is no relationship between the
meaning to that of the query query and the Ad Keyword query and the Ad Keyword

Query Ad Keyword Comment

mariners tickets seattle seahawks tickets The intent of the query is to purchase Mariners tickets. The keyword is
providing tickets for the incorrect sporting event.
mariners tickets lion king seattle The intent of the query is to purchase Mariners tickets. The keyword is
providing tickets for the wrong type of event.
mariners tickets cheap indians tickets The intent of the query is to purchase Mariners tickets. The keyword will
return ticket options for the wrong baseball team.
mariners tickets red sox angels tickets The intent of the query is to purchase Mariners tickets. The keyword will
return ticket options for the wrong baseball team.
ucsd marshall college marshall edu The QY is for Thurgood Marshall College at the University of California
San Diego. The KW corresponds to Marshall University in West Virginia.
Additionally, while other universities or colleges in and around San Diego
may be weakly related, these two have a location mismatch. The KW
mentions the search terms, but has a totally different meaning than what
the user intended.
new jersey warranty bench warrant The QY is for a warranty and the KW is for a warrant. Though the terms
look similar, they are for very different and unrelated documents.
toby mac concerts casting crowns concerts The QY & KW offer concerts for different artists. Even if they offer the
same genre, given the wide number of artists it is very unlikely a user
looking for one artist will be interested in a different, random artist.
www face cbook com wwwyahoo com These sites do not offer similar services or content. The QY & KW are
unrelated.
manakin Mannequin The query intent is to find information about “manakin”, which is a type
of bird. This is unrelated to the KW. Since the query is a legitimate search
term, we cannot assume the QY is a misspelling of the KW.

10
5.5 Adult
An Ad Keyword is rated Adult when both the query and the Ad Keyword are categorized as “Adult”.

• A query and/or Ad keyword is categorized as “Adult” when it offers or refers to products/services/sites that are sexually
explicit or are intended to cause sexual excitement.

If only the query or the Ad keyword are categorized as “Adult”, but not both, then Bad is the correct rating.

• In this case, the general nature of the query and Ad keyword do not match each other and the potential audience is greatly
conflicting.
o An example of this category is when the query is G-rated but the Ad Keyword is X-rated, or vice versa.

Query Ad Keyword Rating Comment

www freesexcams Adult The query intent is to visit a specific site that provides webcams focused on adult acts and activities. The
cams4free com KW, for sexcam sites, is also adult in nature. Both the QY & KW are Adult in nature.

real homade homemade Adult The URL in the query does not exist. We can assume the query intent is to find sites offering homemade
porn com videos pornographic videos. Research shows the KW term is most commonly associated with homemade
pornographic videos. Both the QY & KW are Adult in nature.
mini upskirts swinger porn Adult The query intent is to find “upskirt” photos or videos which are adult in nature. The KW is for
pornographic videos specific to the swinger genre. Both the QY & KW are adult in nature. Note: the QY
& KW do not have to match in intent to qualify for the Adult label; they only must share the quality of
being adult.
t4m dating dating Bad The query uses an abbreviation common in arranging for sexual encounters, often on adult dating or
classified sites. The KW is likely to return Ads for dating sites that are not specifically adult in nature.
There is an Adult/Non-Adult intent mismatch.
massage voyeur sex Bad The query intent is to find information on or services for massage. There is nothing in this query that is
massage inherently adult in nature. The KW will most likely return Ads for a specific type of pornography. We
would not want to return adult Ads for a non-adult query. There is a Non-Adult/Adult intent mismatch.
2 Kids 1 sandboxes Bad The query intent is related to a sexually graphic “shock” video. The KW is for children’s play equipment.
Sandbox There is an Adult/Non-Adult intent mismatch.

Cream Pie creampies Bad The query intent is to find recipes for cream pies. There is nothing in this query that is inherently adult in
recipes nature. Research shows the KW term, as written, is most likely for a specific category of pornography.
We would not want to return adult Ads for a non-adult query. There is an Adult/Non-Adult intent
mismatch.
Hentai Movies Teen movies Bad The query intent is for a type of sexually graphic animated film. Research shows that in one search
free engine the KW returns mostly sexually graphic content while in another it return many non-adult
results. Since research indicates that the KW term is not inherently adult and many users may be
searching for non-adult teen themed movies we should consider this an Adult/Non-Adult intent
mismatch.

11
5.6 Not Sure
This category is used for any Ad Keyword that does not belong to any of the above-mentioned ratings in the following cases:

• The entire query and/or Ad Keyword case cannot be understood.

• The query and/or Ad Keyword are in a foreign language.
o Foreign language is defined as a non-target language for your project.
▪ For the EN-US task the only target language is English.
▪ For International markets – English is treated as a special case. English is always considered a target
language – even in non-English markets (DE-DE, FR-FR etc.) English is never considered a foreign language.
• Note: there is an exception for proper names & titles. These should not be treated as being in a foreign language.
o Examples:
▪ Les Misérables (title of a book & musical)
▪ Vie de France (a chain of gourmet food stores)
• Please note examples below are judged from the EN-US perspective

Checkboxes
Once you have selected Not Sure – please apply one of the following checkboxes if appropriate:

• [Query] Foreign language

o Query must be in a foreign language. KW may be in either foreign language or target market language.
• [Query] Junk
o Query must be junk. KW may be either junk or not junk.
• [Keyword Only] Foreign language
o Keyword must be foreign language. Query must be target language.
• [Keyword Only] Junk
o Keyword must be junk. Query must be not junk.

Query Ad Keyword Rating Comment

casas de venta hotel en el centro precio Not Sure QY and KW are in a foreign language. Research shows that these phrases aren’t
en el Salvador proper names or titles, nor are they phrases commonly associated with English
language content. Apply “[Query] Foreign language” checkbox.

breckenridge co hotel buen precio en Not Sure KW is in a foreign language. QY is in the target language. Apply “[Keyword Only]
summer aspen Foreign language” checkbox.
activities

hert9ya45 Hertz Not Sure The QY cannot be understood. Research returns no matches for the term and we
cannot determine user intent. Apply “[Query] Junk” checkbox.

12
under armour cygnusplaceholderkeyw Not Sure KW cannot be understood. Even though the KW has some recognizable terms within
shoes ord013 it – research returns no close matches for the term. KW is meaningless. QY can be
understood. Apply “[Keyword Only] Junk” checkbox.
Le Monde France news Good Research shows the QY is the proper name of a famous French newspaper. The KW is
broader than the QY; Le Monde is just one option for news from France.
puerco pibil puerco pibil recipe Excellent Research shows that though “puerco pibil” is a Spanish phrase – it is the name of a
pork dish and many English language sites offer recipes for this dish. The QY & KW
are an exact match.

13
6 Competitive/Complementary

➢ Competitive relationships
o This is when the KW offers a similar product/service/site to the one specified in the query from a competitor

Query Ad Keyword Rating Comment

Dell laptop HP latop Fair The KW offers the same product (laptops) from a competitor to Dell.

Dell laptop HP tablet Fair The KW offers a similar product with similar capabilities from a competitor. Tablets are portable and
can be used for many of the same functions as a laptop.

Dell laptop HP desktop Bad The KW offers a product that is not similar enough to a laptop, though both are computers; portability
is a key function of a laptop that a desktop does not share.

Adidas Samba Nike Fair The KW is for a company that produces products which are competitive to the Adidas Samba.

toyota HondaJet Bad While Toyota and Honda offer many competitive products, research shows that Toyota does not offer
planes.

➢ Complementary relationships

o This is when the KW offers a product/service/site which is not what the user is looking for, but which may still be of
interest.
▪ You may think of this as “If you are interested in X, you may be interested in Y”

Query Ad Keyword Rating Comment

men’s dress men’s ties Fair A user interested in shopping for dress shirts may also be interested in purchasing ties to go with
shirts them.

Ugg boots Ugg coupons Fair A user shopping for Ugg boots may also be interested in coupons for products from this company.

Six Flags Atlanta hotels Fair Six Flags is a popular tourist destination and users looking to visit this attraction may be in need of
Atlanta lodging.

Six Flags hotels Fair Six Flags is a popular tourist destination and users looking to visit this attraction may be in need of
Atlanta lodging. The KW would include hotels in the specified location.

Six Flags Atlanta hotels Fair Six Flags is a popular tourist destination and users looking to visit this attraction may be in need of
lodging. Research shows there is a Six Flags located in Atlanta and Atlanta hotels are weakly related to
the QY.
Austin hotels Austin Fair Users booking hotels in Austin may also be interested in nearby dining options.
restaurants

iPhone5s iPhone5 cases Fair The QY intent is for the IPhone 5s. The KW doesn’t address that intent, but users looking for this
product may also be interested in accessories and parts.
Toyota prius Toyota engine Fair The QY intent is for the Toyota Prius. The KW doesn’t address that intent, but users looking for this
product may also be interested in accessories and parts.

14
Austin Austin hotels Bad It is very unlikely a user searching for restaurants will also be in need of lodging, many users will be
restaurants local to the area.

Car insurance New cars Bad While a user shopping for a car may be interested in car insurance for that car, it is very unlikely a
user looking for car insurance will be looking to also purchase a car.
iPhone5 cases iPhone5s Bad The user is looking for cases for the iPhone 5. A user looking for a case most likely already owns this
product and it is very unlikely they will be interested in the phone itself.
Toyota engine Toyota prius Bad The user is looking for information on or to purchase an engine compatible with Toyota vehicles. It is
very unlikely that user would be looking to find information on or purchase a Toyota vehicle as a
whole.

7 URL Queries/URL KWs

Query Ad Keyword Rating Comment

www yahoo yahoo com Excellent The QY & KW intend to refer the same URL. They are an exact match.
com

www.aaa.com www aaa Excellent The QY & KW intend to refer the same URL. They are an exact match. Research shows that the URL
fragment “www aaa” most likely refers to www.aaa.com.
www.aaa.org www aaa Good The KW is a partial URL that could refer to www.aaa.org among other URLs. However, research shows
that the KW most likely refers to www.aaa.com.
www.Nike Nike Excellent Both the QY & KW encompass the entirety of Nike’s products and services. They are an exact match.

www starbucks Good The store locator is a part of the information found on the official Starbucks site. The KW is narrower
starbucks com store locator than the QY.

Cancer.org Cancer Good The user wishes to visit the American Cancer Society website. The KW is narrower than the QY.
treatments Information on cancer treatments is just one type of information found here.

basketball www.basketb Fair When we research this partial URL KW we see that most results are related to informational
shoes all basketball sites. These types of sites would be complementary to the QY as a user looking for
basketball equipment is likely to be interested in the sport in general.
basketball www.basketb Fair When we research this URL KW we find it leads to a parked domain site hosting advertising. While this
shoes alls.com isn’t a strong match – the site does contain links to information related to the topic and so it is weakly
related to the QY rather than unrelated or disjoint.
www fcebook www Fair The user wishes to visit Facebook. The KW does not address this intent, but offers a competitive or
com myspace com alternative social networking site.

www hulu com www hula Bad The user wishes to visit the Hulu TV & movie streaming site. The KW is the URL for a Hawaiian ISP. The
com QY & KW are unrelated.

15
8 Location
Query Ad Keyword Rating Comment
hotels Fairfield hotels Excellent The QY has an ambiguous location, as Fairfield is a city found in multiple locations. The KW offers an
Fairfield, explicit location that is a possible match. When presented with this scenario we should assume that
Connecticut the QY & KW reference the same “Fairfield”. The QY & KW are an exact match as they offer hotels in
the same location.
Hotels Fairfield Fairfield Excellent The QY & KW have an ambiguous location as Fairfield is a city found in multiple locations. When
hotels presented with this scenario we should assume that the QY & KW reference the same “Fairfield”. The
QY & KW are an exact match as they offer hotels in the same location.
hotels Fairfield hotels Bad The QY & KW are a location mismatch.
CA Fairfield,
Connecticut

Hotels NYC Hotels Good The KW is broader than the QY. The broader product group of “hotels” will include hotels in the
specified location.
Dog Grooming Dog Good The KW is broader than the QY. The broader service group of “dog grooming” will include dog
Long Beach CA Grooming groomers in the specified location.

Hotels Hotels NYC Good The KW is narrower than the QY. Hotels located in NYC are a part of the broader product group of
“hotels”.
Trade show Trade show Good The KW is narrower than the QY. Trade Show Displays for sale in Durham are a part of the broader
Displays displays product group of “trade show displays”.
Durham NC

NYC Hotels NYC Good The KW is narrower than the QY. Lodging information is a subset of the available information on NYC.

Hotels NYC NYC Good The KW is broader than the QY. Information on hotels in NYC would be one part of the larger category
of information on NYC.
San Diego Sea World Good The KW is narrower than the QY. Sea World is one attraction in San Diego.
San Diego

Sea World San San Diego Good The KW is broader than the QY. Information on Sea World would be one part of the larger category of
Diego information on San Diego.

Car Car Bad The QY & KW are a location mismatch.

Dealerships Dealerships
Pittsburgh LA

16
9 Nearby Locations

When users specify a location, Ad Keywords offering the sought after product, business or content in nearby locations are weakly
related to the query.

• A location within a 25 mile radius of the requested location should be considered a “nearby” location and weakly related to
the query.

o Radius should be determined by measuring the distance from location to location using a mapping site, such as Bing
maps

For the example below, if we map these two locations we can see Bellevue is within a 25 mile radius of Seattle

• For projects in International Markets, please consult your PM for guidance on market specific treatment of location

Query Ad Keyword Rating Comment

Seattle hotels Bellevue Fair The KW offers the same product (hotels) in a nearby location. Bellevue is within a 25 mile radius of
hotels Seattle. Bellevue hotels are weakly related to Seattle hotels.

Seattle hotels Everett hotels Bad The KW offers the same product, but the location is not considered a nearby location. Everett is not
within a 25 mile radius of Seattle.

17
10 Mixed Rating Examples
Query Ad Keyword Rating Comment

douglas airport charlotte intl Excellent Research shows that both the QY & KW refer to the same airport – Charlotte Douglas International
clt airport airport. The QY & KW are an exact match.
students rating www Excellent The user is looking for student reviews of institutions of higher learning. The KW is for a site which
colleges collegeprowle provides the exact service the user wants and the content isn’t limited to any certain type of school,
r com location, etc.
acer inspire acer aspire 1 Excellent The QY & KW refer to the exact same product.
one netbook
stress reliever stress release Excellent Release and relief/relieve are synonyms. The KW & QY are an exact match in meaning, intent &
games games specificity.
justice girls www.shopjus Excellent The most likely query intent is to find information on the girl’s clothing brand Justice. The URL is for
tice.com the Justice homepage. The QY & KW are an exact match.
att Atlanta Bellsouth Excellent QY intent is to find info about AT&T in Atlanta; the KW is for Bellsouth, which merged with AT&T.
Atlanta The QY & KW are the same company therefore QY and KW are an exact match.
plasma protein total protein Good QY intent is to find information about protein in the blood; the KW is referring to a specific test for
measuring protein levels in the blood. The KW is narrower than the QY. It addresses one possible
intent of the query: to find information on protein testing.
frequent small Tolterodine Good QY intent is to find information on frequent urination. The KW is narrower than the QY as it provides
voidings information specifically on one treatment for this condition.
suggest
brooks brooks Good QY intent is for a brand of shoes. The KW is narrower than the QY as it is for one specific style of the
sneakers glycerin brand.
Adiovox smt nn5600 Good QY intent is to find information about the brand audiovox. The KW is narrower than the QY as it
offers one Audiovox product.
map sunken sunken Good QY intent is to find maps for the sunken temple in WOW. KW is broader than the query as
temple temple quests information on quests would include maps.
charlie one ladies shoe Good QY intent is for a specific style of women’s boots made by Lucchese. The KW is broader than the QY.
horse boots boots This style of boot is a subset of the broader product group of ladies’ shoes & boots.
campgrounds destin Good QY intent is to find information on campgrounds in Northwest, FL. KW is narrower than the QY as it
northwest camping offers camping information for one city in Northwest FL.
florida
bluebook Mechanical Good The query intent is for the Bluebook site which has services for users looking to bid out construction
construction estimating projects and users looking to bid for construction projects. The KW is narrower than the QY as
mechanical estimating is just one type of information the user would expect to find on this site.
liberty travel airlines ticket Good Research shows that the queried business has a strong online presence and serves a national
poughkeepsie price customer base online. The KW is narrower than the QY, airline ticket prices are one type of travel
ny information this company would provide.
2002 dodge dodge Good The KW is broader than the QY. Specs for the 2002 Dodge Durango would be part of the broader
durango specs durango information on the Dodge Durango.
cheap Good The user is looking for pricing information for D batteries. The KW is broader than the QY, but
best prices d alkaline information on D battery prices would be found within the broader category of information on
batteries batteries Alkaline batteries. “Best” and “Cheap” are marketing words that can be ignored.
timberland timberland Good The QY and KW do not match in specificity, but Timberland causal shoes would include men’s shoes,
shoes mens casual shoes though most likely not all the styles of shoe and boot that Timberland makes for men.
nerve pain sciatic nerve Good QY intent for information related to nerve pain after hip surgery - such as general information,
after hip pain exercises diagnosis and treatments. The KW offers a treatment option for nerve pain for one of the nerves
surgery found in the hip. It is important to note that though the QY & KW contain very different terms, they
are closely related. We are labeling QY/KW pairs based on their relationship to each other, not on
how closely the text matches.
prepare my printable pre- Good QY intent for information, products etc. to prepare children for kindergarten. KW offers one type of
child k activities preparation.
kindergarten
halo pocket lipstick Good QY intent for a specific product. The KW offers the broader product category.
charger 2800 charger
18
cowan systems trucking cost Good The KW is narrower than the QY. The KW represents one type of information or service that the
baltimore md per mile queried company offers.
Dehumidifier air cleaners Fair QY intent is for dehumidifiers. The KW is offering air cleaners which do not address the query intent,
but users might be interested in this similar product.
free Free Fair QY intent is for cartoons featuring educational themes. The KW does not address this intent, but
educational education users may be interested in this similar content.
cartoons clipart
play station play station Fair QY intent is for games for the PS3 system. The KW doesn’t address the intent, but is weakly related
unrivalled one games as users interested in PS3 games may also be interested in PS1 games.
games
free drum kits used drum Fair QY Intent is to find free drum kits. We should not ignore the term “free”. The KW does not address
kits the intent, but is weakly related as some users looking for free kits may be interested in purchasing
used drum kits.
sony vaio Fair The QY intent is for the Sony Vaio. The KW doesn’t address that intent, but users looking for this
laptop product may also be interested in accessories and parts.
sony vaio battery
frog bracelet magnetic Fair The user is looking for a specific type of bracelet. The KW does not address that intent, but the user
rubber may be interested in additional types of bracelets.
bracelet
tyler perry the haves and Fair The user is looking for information related to the latest movies Perry has released into theaters. The
latest movies have nots tv KW does not address this intent, but a user interested in Perry’s movies may also be interested in his
show TV shows.
weight loss walmart Fair QY intent for information on a specific diet. KW does not address this intent, but users interested in
soup diet weight loss dieting information may be interested in weight loss products.
supplements
tv into smart tv panasonic Fair QY intent for information on how to transform a non- smart TV into a smart TV. KW doesn't address
st60 60 inch intent, but users may be interested in purchasing a smart TV.
avg computer Norton Free Fair QY intent for a specific product - a registry cleaner from AVG. KW offers AV/Security software from a
tune up Trial 30 Days competitor. While these types of software are different, a user looking to clean or repair their
registry may be interested in additional security software.
real estate sample offer Fair QY intent for forms for a specific real estate transaction. KW offers forms for a different, but similar
transfer forms to purchase type of transaction. Users in the business of transferring real estate may also be in the business of
real estate buying/selling real estate.
samsung ht vizio 39 inch Bad The user wishes to visit the Best Buy site to purchase a specific home theater system. The KW is for a
d5500 best buy smart tv smart TV which is not similar, competitive or complementary to the query. A user looking to
purchase a home theater system most likely already owns the TV for which they are purchasing the
system.
timberland timberland Bad A user looking for men’s shoes would be very unlikely to be interested in women’s shoes. The QY &
shoes mens shoes KW are unrelated.
womens
size 7 wide Womens Size Bad Though both QY/KW offer shoes - it is very unlikely a user looking for size 7 wide shoes will find
shoes 5 Shoe wrong size shoes of interest
hp photosmart ink for hp Bad KW offers ink for the wrong model of printer. QY & KW are unrelated.
7510 ink officejet 6700
cartridge
aa car www aa co uk Bad Research shows the user is most likely looking for AA car insurance which is a UK based insurance
insurance company. If we enter the KW into a browser with the appropriate punctuation we see that this URL
is for the UK homepage of American Airlines. The QY & KW are unrelated.
pole lamps antique street Bad QY intent is for pole floor lamps which are used indoors. The KW is for a type of lighting primarily
lighting used outside. It is very unlikely a user looking for indoor lamps will want to purchase external street
lighting.
Johnny Depp Bad The query intent is to find information on the actor Johnny Depp. The KW is a different actor. It is
Orlando
very unlikely a user searching for one specific actor would be interested in information on a
Bloom
different, random actor.
pool covers Bad The query is for a specific product. The KW offers an online shopping site with no reference to pool
beso
accessories, pool covers etc. The QY & KW are too far apart to be related.
get revenge getting ex Bad It is very unlikely a user looking to get revenge on an ex would also be interested in information on
your ex girlfriend reconciling with an ex. The QY & KW are unrelated.
back

19
bing toolbar Google Fair The QY is to install the Bing Toolbar. The KW offers a competitor toolbar.
install toolbar
bing toolbar restore Bad A user looking to install the Bing Toolbar is very unlikely to need information on restoring the Google
install google Toolbar.
toolbar
Space Needle space needle Excellent The most likely query intent is the Space Needle in Seattle. The KW matches the QY exactly since
seattle center adding the location does not change the meaning, intent or specificity of the KW.
Space Needle space needle Good The most likely query intent is the Space Needle in Seattle. The KW is narrower than the QY.
tickets Purchasing tickets is one possible intent of the query.
Space Needle space needle Good The most likely query intent is the Space Needle in Seattle. The KW is narrower than the QY. Finding
restaurant information on this restaurant is one possible intent of the query.
Space Needle Seattle Good The most likely query intent is the Space Needle in Seattle. . The KW is broader than the QY – the
Center Seattle Center features a number of tourist attractions, including the Space Needle. The user can
expect information on the Space Needle to be found within broader information on the Seattle
Center.
Space Needle Seattle Good Since the Space Needle is a well-known tourist attraction in Seattle, the location is implicit – as if the
user entered Space Needle, Seattle. The KW is broader than the QY as information on the Space
Needle is just one part of the larger category of information on Seattle.
Space Needle Pike Place Fair The most likely query intent is the Space Needle in Seattle. The KW does not address this intent, but
Market Pike Place Market is another popular tourist attraction in Seattle and users planning to visit the
Space Needle may also be interested in other local attractions.
Santo Domingo Hotels Santo Excellent The user is looking for hotels in Santo Domingo and the KW matches that intent exactly. Per the
Hotel Domingo Location section – we can assume both the QY & KW are referring to the same location - even
though there are multiple locations with this name.
Santo Domingo santo Excellent The user is looking for hotels in Santo Domingo and the KW matches that intent exactly. Per the
Hotel domingo Location section – we can assume both the QY & KW are referring to the same location - even
hotel, though there are multiple locations with this name.
dominican
republic
Santo Domingo santo Good The user is looking for hotels in Santo Domingo and the KW is narrower than the QY as it offers one
Hotel domingo bed type of hotel.
and breakfast
Santo Domingo Dominican Good The user is looking for hotels in Santo Domingo and the KW is broader than the QY as it offers hotels
Hotel Republic in the broader Dominican Republic. The user can reasonably expect to find hotels in Santo Domingo
hotel in this broader selection of hotels.
Santo Domingo occidental el Good The user is looking for hotels in Santo Domingo and the KW is narrower than the QY as it offers a
Hotel embajor single hotel in this area. Please note the KW is a proper name and shouldn’t be considered foreign
language.
Santo Domingo Discount Good The user is looking for hotels in Santo Domingo and the KW is broader than the QY as it offers hotels
Hotel hotels in all locations. Hotels in Santo Domingo are a part of this broader selection of hotels.
Santo Domingo boca chica Fair The user is looking for hotels in Santo Domingo. The KW is for hotels in a different town. Research
Hotel hotels shows that Boca Chica is listed as 21 miles away on Bing maps and 28 miles away from Santo
Domingo on Google Maps. In the case you find close, conflicting distances as long as one of them is
within 25 miles you can consider the location nearby.
Santo Domingo Hotels Madrid Bad The user is looking for hotels in Santo Domingo. The KW is for hotels in Madrid, Spain. While there is
Hotel Spain a town in Spain named Santo Domingo de la Calzada, it is very unlikely the user was searching for
that town. Additionally, that town is very far from Madrid. This is a location mismatch and the QY &
KW are unrelated.
Chocolate chocolate Excellent QY intent is to find candy cigarettes made of chocolate. Though the KW is misspelled it still is an
Cigarettes cigaretes exact match to the query
Chocolate candy Good QY intent is to find candy cigarettes made of chocolate. The KW is broader than the query. Chocolate
Cigarettes cigarettes candy cigarettes are part of this broader product group.
Chocolate bubble gum Fair QY intent is to find candy cigarettes made of chocolate. The KW is a candy that is similar to the
Cigarettes cigars query.
Chocolate Fair QY intent is to find candy cigarettes made of chocolate. The KW does not address the intent, but a
Toblerone
Cigarettes user looking for one type of chocolate candy may be interested in other types of chocolate candy.
Chocolate Bad QY intent is to find candy cigarettes made of chocolate. The KW offers a candy that is too far from
bubble gum
Cigarettes the QY to be related, it is not similar in form or flavor.
Chocolate Bad QY intent is to find candy cigarettes made of chocolate. It is very unlikely the user will be interested
Marboro
Cigarettes in real cigarettes. The QY & KW are unrelated.
20
Dodge Razor dodge razor Excellent
The query intent is to find the Dodge Razor Hybrid car. The KW is an exact match to the query.
car
Dodge Razor dodge Good The query intent is to find the Dodge Razor Hybrid car. The KW is broader than the QY. It is for all
concept cars Dodge concept cars which includes the Razor.

Dodge Razor dodge Good The query intent is to find the Dodge Razor Hybrid car. The KW is broader than the QY. It is for all
vehicles Dodge vehicles, which includes the Razor.

Dodge Razor concept cars Good The query intent is to find the Dodge Razor Hybrid car. The KW is broader than the QY. It is for all
concept cars, which includes the Razor.
Dodge Razor www.dodge.c Good The query intent is to find the Dodge Razor Hybrid car. The KW is broader than the QY. It is for the
om official Dodge site, which would include information on the Razor.

Dodge Razor Bad The query intent is to find the Dodge Razor Hybrid car. The KW offers Dodge pick-up trucks. Pick-up
dodge ram
trucks are not similar to the Razor, which is a hybrid sports car. While the user may be interested in
trucks
alternative or competitive hybrids or sports cars, they are very unlikely to be interested in a pick-up
truck which is a very different type of vehicle.
Dodge Razor razor blades Bad The query intent is to find the Dodge Razor Hybrid car. The KW is for razor blades, such as are used
in shaving. Though the QY & KW contain similar terms, they are completely unrelated.
Dodge Razor razor pocket Bad The query intent is to find the Dodge Razor Hybrid car. The KW is for a mini electric bike. Though the
rocket QY & KW contain similar terms, they are completely unrelated.

Dodge Razor dodge Bad The query intent is to find the Dodge Razor Hybrid car. The KW is for a specific car part. Though the
transmission QY & KW contain similar terms, it is very unlikely a user looking for information on this concept car
would want to purchase car parts as this car has yet to be put into production.

21
11 Glossary
Query (QY) A search term (a word or group of words entered by a user)

Ad Keyword (KW) Keywords that advertisers have purchased to address the query
Intent What the user was looking for when they entered the query for example a website to receive information
or a website to purchase a product.
Core Concept The main idea being searched for within a multiple word QY; for example - “men’s track shoes”. The core
concept is “shoes”.
Attribute This is a word that modifies the QY by limiting its meaning; for example - if a QY is “gloves” and you add
“baseball” (baseball gloves), it narrows the definition of the QY and helps define user intent.

22