Map Deep Relevance: Judge Guidelines

Updated 2021-08-07 (v1.4)

If you have read previous versions of these guidelines, you should read Section 4 (“Version History”) to
learn what has changed. That way, you will not have to re-read the entire guideline. We also recommend
watching the video guidelines (in the HIT app). If they don’t load, refresh your browser.

1. Overview
This HIT app shows user queries made on a map search website (e.g. bing.com/maps). Your task is to
read the query and understand the user’s search “intent” (what they are looking for). Then, for each
part of the query that is outlined, you will rate the relevance of the result as EXCELLENT, GOOD, BAD, or
MISSING. You will do this for each important (outlined) part of the query, one by one.

The HIT app will try to help by outlining possible matches in the result. However, you cannot completely
rely on these hints. Not all outlined words will be relevant. Also, the HIT app will NOT outline words that
are relevant even if the only differences are due to abbreviations, alternate names, or misspellings.
Therefore, you may need to look for any relevant words that are not outlined.

In the screenshot below, the Query section shows outlined words. The “Type” section describes what
kind of word / phrase is outlined in the query (e.g. Street Name / Street Number / City / Country, etc.).

In this example, the city “Southern Pines” is the part of the query being judged. Its “Type” is City, shown
below the rating section. “Southern Pines” also appears in the result, and thus is outlined.

The CITY portion (“Southern Pines”) of the query is perfectly matched by the CITY portion (“Southern
Pines”) of the result. Thus, this part of the query should be rated as EXCELLENT.

Caution: A match in the result does not always mean EXCELLENT relevance. It is only a hint. In the
example below, “Southern Pines” appears in the result as the STREET NAME. However, the intent of the
outlined part of the query is for the CITY of “Southern Pines”. The result has a completely different CITY
(“Raleigh”). Therefore, the relevance of the result to this part of the query should be rated as BAD.

2. How to judge
2.1 Judgment steps
Step 1: Read and understand the query and result.

The Query Type and Original Query will appear in the top left of the HIT app. Below that, you will see the
simplified Query that you are rating for relevance. They both represent the same query, but punctuation
has been removed from the Simplified Query to simplify judging.

Sometimes, the punctuation will help you understand search intent. Please read and understand both
the Original Query and the Simplified Query.

The Query Type will help you understand the user’s search intent, especially when it could be
interpreted in different ways. The queries were made by users on a map search website (e.g.
bing.com/maps), so you must interpret them in a MAP context. Query Types are:
• Address: User is searching for a street address, like “123 Main Street, San Francisco CA” or
“Interstate 80 Nevada”.
• Place: User is searching for a region, district, or political entity (city/village/town, state/province,
country, etc.), such as “San Francisco CA”
• Business: User is searching for a business/organization by name, chain, product, category (type
of product, service, or POI), or by phone number. Includes (but not limited to) hotels, K-12
schools, official buildings (not famous), business parks / industrial estates, and organizations of
all kinds. Note: some businesses have location words in their proper name, e.g. “Kentucky Fried
Chicken”.
• POI (Point of Interest): User is searching for a specific, significant man-made "Point of Interest",
often with important cultural, historical, tourist, or other major public interest.
• Natural Feature: User is looking for natural area (e.g. parks, forests, ski areas etc.) or
geographical feature (e.g. mountains, rivers, lakes, beaches, deserts, islands etc.).
Step 2: Use research tools to understand the query and result.

Select “Query” or “Result” to choose between

researching Query or Result.

Select default search engine for right research

panel or disable autoload of research.
Enable Google research link. Select if
research is blocked by Google Captcha.

Step 3: For each outlined part in the query, rate the relevance of the result. The first section of the
query that you will judge will automatically be outlined. After you rate the relevance of the result for
that part of the query, the HIT app will automatically move on to the next part of the query for you to
judge

Compare only the part of the result that corresponds to the outlined part of the query that you are
judging, but compare all relevant words. (Remember, the outlined part of the result is only a hint and
may not correspond to the outlined part of the query). For example, if the query is for a business name,
compare all of the result that is also a business name, but do not compare other parts of the result (such
as the address or city). If the business name in the result is more specific, less specific, or different from
the search intent, then penalize.

You can use the left and right arrow buttons (circled in red below) or use the arrow keys of your
keyboard to move to different sections of the query. You can also simply click on the part of the query
you would like to rate next.

To select a rating, click on 1 of the 4 ratings, or press one of the number keys shown in square brackets
(e.g. number key 2 to select "[2] Good").

For each outlined part of the query, rate the relevance of the result as EXCELLENT, GOOD, BAD, or
MISSING (see Section 2.2 below for definitions).

You will not always have to judge every single word in the query! You will know you are done when
appears in the rating section. At that point, you can either change any rating, or you can
submit your judgment (by clicking the Submit button or pressing the ENTER key).

At any time, you can clear all your judgments for the query and start over (click the [R]eset button or
press ‘R’ on your keyboard).
2.2 Judgment ratings

RATING DESCRIPTION
Excellent Result is a perfect (or almost perfect) match for the outlined part of the query. Very minor
differences are OK. Do NOT penalize for differences due to abbreviations or valid
alternate names. Do NOT penalize for valid corrections to the query, such as fixing
spelling errors or showing the full and proper name. Do NOT penalize extra, non-
conflicting information in the result (but penalize extra information if it conflicts).
Good Result is NOT excellent, but it mostly or partially satisfies the outlined part of the query.
Reasonable differences are OK. Also, if result is excellent but has weird format or bad
characters, then penalize slightly to GOOD.
Bad Result is a terrible match compared to the outlined part of the query. It conflicts with the
query, has major differences, is completely wrong, or has little (or even nothing) in
common with the query intent.
Missing Result shows nothing for the outlined part of the query that you are judging, e.g. if
outlined part of query is a city but there is NO city in result. This is different from BAD,
which has some kind of match (but of poor quality, e.g. different city in result than in
query). MISSING means the result has no match for that part of the query at all.

EXCELLENT: Result is a perfect (or almost perfect) match for the outlined part of the query. The result
has (at most) minor differences compared to the search intent for that part of the query.

In the example below, the search intent of the outlined part of the query (“Cold Lake”) is for the city of
Cold Lake. The city of Cold Lake is also in the result, so relevance for this part of the query is EXCELLENT.

GOOD: Result is NOT a perfect match for the outlined part of the query, but it is a reasonable match
that mostly or partially satisfies the search intent. The result may have differences from the user’s
search intent, but the differences are minor and the result is NOT completely wrong.

Also, if the result is EXCELLENT but has weird formatting or bad characters, slightly penalize to GOOD.
This includes results with non-standard capitalization, extra commas, and non-standard abbreviations.

The image below shows a query for the “San Jose – Edenvale – North” branch of a hotel chain named
“Extended Stay America”. However, the result has a different branch of this hotel chain: “San Jose –
Airport”. The result is not a perfect match to the query, but it is quite similar. It is the right business, but
wrong branch. Thus, the relevance of the result for the outlined part of the query is GOOD.
BAD: Result is terrible match for the outlined part of the query that you are judging. Also select BAD if
the match is completely wrong, conflicts with the query, has major differences, or has little or nothing in
common with the query intent.

In the example below, the outlined part of the query is for a STREET named “Holly Springs”. Holly
Springs does not appear in the result, and the result has a different street name (“Tanglewood”). The
street name in the result is completely wrong, so “Holly Springs” in the query should be rated as BAD.
Because a street name is present in the result, “Holly Springs” should NOT be marked as MISSING.

MISSING: Result is missing a match for the outlined part of the query that you are judging. Please note
the difference from a BAD rating. A BAD rating means the result at least has some kind of match for that
part of the query (even if it is a poor quality match).

A MISSING rating means the result has no match for that part of the query at all.

In the example below, “china” should be marked as MISSING because no country is given in the result.

In the example below, “raleigh” should be marked as MISSING. Although there is a text match in the
result for “raleigh”, the search intent of “raleigh” in the query is for the CITY of Raleigh ( in North
Carolina, USA). The “Raleigh” shown in the result is clearly a STREET NAME.

CITY is missing from the result completely, so this part of the query (“Raleigh”) should be rated MISSING.

2.3 Judging concepts

2.3.1 EXCELLENT vs. GOOD
Select EXCELLENT if the result is a perfect (or nearly perfect) match for the search intent for the outlined
part of the query. Very minor differences with the query are okay.

But if the result is not EXCELLENT but mostly or partially satisfies the user’s search intent, select GOOD.
 Example Result Rating Reason
123 ocean front dr 123 Ocean Front Drive, EXCELLENT Result Street Type (“Drive”) is a
key largo Key Largo, FL 33037 perfect match for query Street
Type (“dr”). Abbreviation is OK.
123 ocean front dr 123 Ocean Front Drive, EXCELLENT Result Street Name (“Ocean
key largo Key Largo, FL 33037 Front”) is an exact match for
query Street Name.
123 ocean front dr 123 Oceanic Front Dr, GOOD Result Street Name (“Oceanic
key largo Key Largo, FL 33034 Front”) is NOT an exact match for
query Street Name (“ocean
front”), but it is very similar.

2.3.2 GOOD vs. BAD

Select GOOD if the result mostly or partially satisfies the user’s search intent for the outlined part of the
query.

But if the result completely misses the search intent of the query, conflicts with the query, or has other
major differences, select BAD.

Query Result Rating Reason

123 ocean front dr 124 Ocean Front Dr, Result Street Number (124) is not
seattle Seattle, WA, 98199 GOOD an exact match to query Street
Number (123) but is very similar.
123 Ocean Dr, Seattle, 123 Oceana Dr, Result Street Name (“Oceana”) is
WA 98199 Seattle, WA 98199 GOOD very similar to query Street Name
(“Ocean”), but not exact.
123 ocean front dr 123 Ocean Dr, Seattle, Result Street Name (“Ocean”) is
seattle WA 98199 GOOD similar to query Street Name.
123 Ocean Dr, Seattle, 123 Ocean Front dr Result Street Name is similar to
WA 98199 seattle GOOD query Street Name.
123 ocean front dr 123 Ocean View Dr, Result Street Name is similar to
seattle Seattle, WA 98199 BAD query Street Name but (“View”)
conflicts with “Front”).
123 ocean front dr 123 Oceanside Dr, Result Street Name is similar to
seattle Seattle, WA 98199 BAD query Street Name but (“side”)
conflicts with “front”).
123 ocean front dr 123 Meadow Dr, Result Street Name (“Meadow”)
seattle Seattle, WA 98199 BAD is completely different than
query Street Name.
414 1st Street 414 2nd St, Snohomish, Result Street Name (“2nd”) is
WA 98290 BAD completely different than query
Street Name (“1st”). It is
obviously a different street.
46 Four Limes, 46 Four Limes, St Query has 2 parts to its town/city
Wheathampstead, St Albans, AL4 8JN, UK GOOD info (common in the UK):
Albans, AL4 8JN “Wheathampstead” (Locality)
and “St Albans” (Post Town). But
result has only “St Albans”.
2.3.3 BAD vs. MISSING
Select MISSING only if the result has nothing to match the outlined part of the query. Applies only to
similar query word types.

For example, if the outlined part of the query is a street name, but no street name appears in the result
at all, the street name in the query should be rated as MISSING. However, if a street name appears in
the result, but the street name is wrong, then that part of the query should be marked as BAD.

Query Result Rating Reason

123 ocean front dr Ocean Front Dr, MISSING Query search intent is for a Street
seattle Seattle, WA 98199 Number (123). But Result has no
Street Number.
123 ocean front dr 10595 Ocean Front Dr, BAD Query search intent is for a Street
seattle Seattle, WA 98199 Number (123). Result has a Street
Number (10595), but it is
completely different.
123 ocean front dr 123 Ocean Front Dr MISSING Query search intent is for a CITY
seattle (Seattle), but result has no CITY.
123 ocean front dr 123 Seattle Dr MISSING Query search intent is for a CITY
seattle (Seattle). Result has a text match
for “seattle”, but it is a road.
123 ocean front dr 123 Seattle Dr, BAD Query search intent is for a CITY
seattle Kirkland, WA 98199 (Seattle). Result has a City
(Kirkland), but it is completely
different.

2.3.4 Spelling errors

If a query contains an obvious typing or spelling error, do NOT penalize if the result corrects the error.
But if result has spelling error, then you should penalize it.

Query Result Rating Reason

123 oceann front dr 123 Ocean Front Dr, EXCELLENT Query Street Name contains an
Seattle, WA 98199 obvious typo in the word
“Ocean”. Result Street Name is
an EXCELLENT match for the
search intent of the query.
123 ocean fromt dr 123 Ocean Front Dr, EXCELLENT Query Street Name contains an
Seattle, WA 98199 obvious typo in the word
“Front”. Result Street Name is an
EXCELLENT match for the search
intent of the query.
123 ocean front dr 123 Ocean Front Dr, EXCELLENT Query City contains an obvious
Seatle WA Seattle, WA 98199 spelling error in the word
“Seattle”. Result City is an
EXCELLENT match for the search
intent of the query.
123 ocean front dr 123 Ocean Front Dr, GOOD Result City closely matches but
Seattle WA Sattle, WA 98199 has an obvious spelling error.
2.3.5 Category searches
If a query contains a search for a CATEGORY (e.g.. “restaurants”, “parks”, “clothing stores”, or “homes
for sale”), then the result should be a specific match for the general CATEGORY.

Compare all of the words in the result that are related to the category query words. Usually, category
words do NOT exactly match words in the result, so normally no words will be outlined in the result. But
the result may still match the search intent in the query.

For example, if the query is for a business CATEGORY, compare all of the result that is also a business,
but do not compare other parts of the result (such as the address or city). If the business name in the
result is more specific, less specific, or different from the search intent of the CATEGORY, then penalize.

Special case: in CATEGORY queries for real estate, like “homes for sale” or “property in Ann Arbor”, the
CATEGORY portion should be rated as EXCELLENT if the rest of the result satisfies the query. This may
include results of only a specific address (with street NUMBER but without any words like “for sale”),
because the property at that specific address may be a home that is for sale.

Query Result Rating Reason

chinese food in seattle Panda Express, 12513 EXCELLENT
wa Lake City Way NE,
Seattle, WA 98125
chinese food in seattle Moderna Pizza, 8014
wa Lake City Way NE,
Seattle, WA 98115
parks near seattle Discovery Park, 1801 EXCELLENT
Discovery Park Blvd,
Seattle, WA 98199
hotels seattle wa Hillside Motel, 2451
Aurora Ave N, Seattle,
WA 98109
hotels seattle wa 99 Union St, Seattle,
WA 98101
Homes for sale Seattle Keller Williams Realty, EXCELLENT
1100 Dexter Ave N Ste
275, Seattle, WA 98109
“Chinese food” is category
search. Any business selling
Chinese food in the result (like
“Panda Express”, “Din Tai Fung”,
or “Pink Pearl Chinese Seafood
Restaurant”) is an EXCELLENT
match for the search intent of
the category “chinese food”.
BAD “Chinese food” is a category
search. Result business
(“Moderna Pizza”) is a
restaurant, but it has Pizza, not
Chinese food.
“Parks” is category search. Any
park in the result is an EXCELLENT
match..
GOOD “hotels” is a category search.
“Hillside Motel” is a motel (not a
hotel) and thus an imperfect, but
reasonable result.
MISSING “hotels” is a category search.
Research shows that the address
in the result is for a hotel (“Four
Seasons Hotel Seattle”), but the
result is MISSING the Business
Name. In this case, the business
category “hotels” should be
marked as MISSING from Result.
“Homes for sale” is a category
search. Any result with a real
estate company OR address of a
home for sale is an EXCELLENT
match for the search intent.
 Homes for sale Seattle 8828 42nd Ave SW,
Seattle, WA 98136
Real estate agent Keller Williams Realty,
Seattle 1100 Dexter Ave N Ste
275, Seattle, WA 98109
Real estate agent 8828 42nd Ave SW,
Seattle Seattle, WA 98136
EXCELLENT “Homes for sale” is a category
search. Any result with a real
estate company OR address of a
home for sale is an EXCELLENT
match for the search intent.
EXCELLENT “Real estate agent” is a category
search. Any result with a real
estate agent or company is an
EXCELLENT match.
MISSING “Real estate agent” is a category
search for a business or agent
that sells real estate. But result is
a specific home (that is for sale).

2.3.6 Result specificity

In most cases, results should be as specific as the query. If the result is less (or more) specific, then
penalize to GOOD or BAD. Any penalty will depend on the strength of the relationship of the result to
the query.

Example: for query “Long Island, NY”, you should penalize to BAD any results that are too specific (or
don’t match) such as “Huntingdon, Long Island, NY”, “Long Island City, NY”, “Long Island Railroad, Long
Island, NY”, “Long Island Airport”, “Long Island Expressway”, or “Long Island Aquarium”. None of those
results are relevant to the user’s search intent.

Remember to compare only the part of the result that corresponds to the portion of the query that
you are judging, but compare all relevant words. (Also remember that the outlined part of the result is
only a hint and may not correspond to the outlined part of the query). For example, if the query is for a
business name, compare all of the result that is also a business name, but do not compare other parts of
the result (such as the address or city). If the business name in the result is more specific, less specific, or
different from the search intent, then penalize.

As always, do not penalize extra, non-conflicting information (unless it weakens the strength of the
relationship). Extra, non-conflicting information does NOT make the result more specific. Only penalize if
the extra information conflicts with the query and/or makes it more specific.

For example, do NOT penalize a strongly relevant result if the query is incomplete but the result is
complete, especially if the result has the full and proper name. Do not penalize for use of valid alternate
names, valid abbreviations, or any other reasonable alternate forms unless they affect specificity or
correctness of the result.

2.3.6.2 Result LESS specific than query

If the result is slightly less specific (STRONG relationship to the query), then you should penalize it
slightly to GOOD. If it is much less specific (WEAK relationship), then penalize to BAD.
Query Result Rating Reason
Anthropology Dept, Department of EXCELLENT Perfect match. Query has
UW, Seattle, WA Anthropology, abbreviation, result has full name
University of (either is OK).
 Washington, Seattle
WA 98105
Anthropology Dept, University of
UW, Seattle, WA Washington, Seattle
WA 98105
Anthropology Dept, Seattle WA 98105
UW, Seattle, WA
Andare Kitchen Hyatt Hyatt Regency, 808
Seattle Howell Street, Seattle,
WA 98101
GOOD Slightly less specific. There is still
a strong relationship between
the result and the query.
MISSING No part of the result corresponds
to this part of the query.
BAD Restaurant is inside this Hyatt
hotel, but is a separate business.
It has a WEAK relationship to it.

2.3.6.3 Result MORE specific than query

On the other hand, results should NOT be more specific than the search intent of the query. If the
result is more specific, then you should penalize. The penalty will depend on the strength of the
relationship of the result to the query. Caution: a CATEGORY query should obviously have a result that is
more specific than the category because a category is general, but it should NOT be more specific than
the intent of that category.

If the result is slightly too specific (STRONG relationship to the query), then penalize slightly to GOOD.
But if it is much too specific (WEAK relationship), then penalize to BAD.

Query Result Rating Reason

london UK London, UK EXCELLENT Perfect match
london UK SoHo, London UK GOOD “SoHo” is a neighborhood in
London. It has a STRONG
relationship to London, but is too
specific.
151 W 34th St, NY Macy’s, 151 W 34th St, EXCELLENT “151” is a perfect match. Do NOT
New York, NY 10001 penalize result for including
extra, non-conflicting
information (“Macy’s” is at the
address and does not conflict).
restaurants in Seattle The Harvest Vine, 2701 EXCELLENT As expected, this restaurant is
E Madison St, Seattle, more specific than the
WA 98112 CATEGORY word “restaurant”,
but it matches the search intent
of the CATEGORY.
hotel in seattle Parking service, Hyatt BAD Result is part of a hotel, but it is
Hotel, Seattle not itself the hotel. It is more
specific than the CATEGORY.
U Washington, Seattle, University of EXCELLENT Perfect match
WA Washington, Seattle
WA 98105
U Washington, Seattle, Dept. of Anthropology, GOOD Compare the full relevant portion
WA University of of result. It is slightly too specific
Washington, Seattle (department is one small part of
WA 98105 the whole).
U Washington, Seattle, Prof. J. Danzburg, Dept. BAD Compare the full relevant portion
WA of Anthropology, of result. It is much too specific.
University of
Washington, Seattle
WA 98105
Walmart Walmart Vision & GOOD Result is a department of the
Glasses, 11400 Hwy 99, store that user was looking for. It
Everett, WA 98204 has a STRONG relationship to the
store.
Mercy Hospital Muckerman Garage, GOOD Compare the full relevant portion
Mercy Hospital, of result. This parking garage is
McAuley Dr, Creve part of the hospital that user was
Coeur, MO 63141 looking for. It has a STRONG
relationship to the hospital.
Hyatt Seattle Hyatt Room Service, GOOD This kitchen is inside the Hyatt
808 Howell Street, hotel that the user was looking
Seattle, WA 98101 for. It is owned by the hotel and
has a STRONG relationship to it.
Hyatt Seattle Andare Kitchen 808 BAD Restaurant is inside this Hyatt
Howell Street, Seattle, hotel, but is a separate business.
WA 98101 It has a WEAK relationship to it.
NY Tourist Attractions Empire State Building, EXCELLENT Perfect match for the search
20 W 34th St, New York, intent of the category.
NY 10001
Empire State Building Empire State Building, EXCELLENT Perfect match.
NY, NY 20 W 34th St, New York,
NY 10001
Empire State Building Empire State Building EXCELLENT The result might seem too
NY, NY Ticket Office, 20 W specific, but it is actually exactly
34th St, New York, NY what many users are searching
10001 for.
Empire State Building Empire State Building GOOD The result has a STRONG
NY, NY Gift Shop, 20 W 34th St, relationship to the query, but it is
New York, NY 10001 slightly too specific.
Empire State Building LinkedIn, Empire State BAD The result has a WEAK
NY, NY Building, 26th Floor, 20 relationship to the query (the
W 34th St, New York, business is in the building but has
NY 10001 no other relationship). Thus, it is
much too specific.
Empire State Building LinkedIn, Empire State EXCELLENT The outlined part of the query is
NY, NY Building, 26th Floor, 20 a city (“NY”). Compare only the
W 34th St, New York, relevant portion of result (city of
NY 10001 “New York”). The result is a
perfect match.

3. Examples
Query Result Rating Reason
Seattle Seattle, WA EXCELLENT Perfect match
Siattle Seattle, WA EXCELLENT Do not penalize result for
correcting spelling error.
Seattle Siattle, WA GOOD Penalize spelling error in RESULT.
Seattle SeaTac, WA GOOD Small city with similar name has
STRONG relationship to query
(“Sea” in “SeaTac” is Seattle).
Seattle Seabeck, WA BAD Poor match.
Seattle Spanaway, WA BAD Poor match.
Tacoma Convention Tacoma, WA 98402 MISSING Query is for a POI (Point of
Center Interest). Result is a city. The POI
is missing from the result.
Tacoma WA 98402 Tacoma Convention EXCELLENT The outlined part of the query is
Center, 1500 a Post/ZIP code (“98402”).
Commerce St, Tacoma, Compare only the relevant
WA 98402 portion of result. The result is a
perfect match.
mt baker WA Baker, MT 59313 MISSING Query is for a mountain. Result
has similar name but has nothing
to do with search intent. The
mountain is missing from result.
Baker, MT Mt Baker WA MISSING Query for a city (Place), but result
is for a mountain (Natural
Feature). Result has a similar
name but it has nothing to do
with the intended city.
mt baker WA Mt Baker-Snoqualmie BAD Result is much less specific than
National Forest, WA the query. Query is for a
mountain. Result is for a national
forest with a similar name, that
contains Mt Baker and many
other mountains. Both are
natural features, but they are not
the same thing.
94105 San Francisco 94105 EXCELLENT Perfect match
94105 San FRANCISCO 94105 GOOD Weird capitalization in RESULT.
94105 San Francisco### GOOD Weird characters in RESULT.
94105
94105 San Francisco 94103 GOOD Slightly different ZIP (Post) code
94105 San Francisco 94005 BAD Very different ZIP (Post) code
123 dune st seattle wa 123 Dune St, Seattle, EXCELLENT Result Street Name (“Dune”) is
98199 WA 98199 an exact match for the query
Street Name (“Dune”).
123 dune st seattle wa 123 Dunes St, Seattle, EXCELLENT Result Street Name (“Dunes”) is
98199 WA 98199 an ALMOST perfect match for the
query Street Name (“Dune”).
123 dune st seattle wa 123 Dune Street, EXCELLENT Result Street Type (“Street”) is an
98199 Seattle, WA 98199 abbreviation for the query Street
Type (“St”), thus a perfect match.
123 dune st seattle wa 123 Dunn St, Seattle, GOOD Result Street Name (“Dunn”) has
98199 WA 98199 a minor difference with the query
Street Name (“Dune”).
123 dune st seattle wa 123 Dane St, Seattle, GOOD Result Street Name (“Dane”) has
98199 WA 98199 a minor difference with the query
Street Name (“Dune”).
123 dune st seattle wa 123 Dune Hill St, GOOD Result Street Name (“Dune Hill”)
98199 Seattle, WA 98199 has a non-conflicting difference
from query Street Name.
123 dune st seattle wa 123 Dunedin St, BAD Result Street Name (“Dunedin”)
98199 Seattle, WA 98199 has a major difference from
query Street Name (“Dune”).
123 dune st seattle wa 123 Daffodil St, Seattle, BAD Result Street Name (“Daffodil”) is
98199 WA 98199 completely different from query
Street Name (“Dune”).
123 dune st seattle wa Seattle, WA 98199 MISSING Street Name is MISSING from
98199 result.
123 dune st seattle wa 123 Daffodil St, Seattle, EXCELLENT Result Post Code (98199) is an
98199 WA 98199 EXCELLENT match for the query
Post Code (98199).
123 dune st seattle wa 123 Daffodil St, Seattle, GOOD Result Post Code (98109) has
98199 WA 98109 minor differences to the query
Post Code (98199).
123 dune st seattle wa 123 Dune St, Seattle, BAD Result Post Code (98269) has
98199 WA 98269 major differences from query
Post Code (98199).
123 dune st seattle wa 123 Dune St, Seattle, BAD Result Post Code (58769) is
98199 WA 98769 completely different from Query
Post Code (98199).
123 dune st seattle wa 123 Dune St, Seattle, MISSING Post Code in query is MISSING
98199 WA from result.
DSW DSW Designer Shoe EXCELLENT Compare the full relevant portion
Warehouse, 40 E 14th of result. Do not penalize result
St, New York, NY 10003 for giving full proper name of
business.
John’s Pizza NY John's of Bleecker St., EXCELLENT Do not penalize result for
278 Bleecker St, New correcting query with full proper
York, NY 10014 name of business.
Kentucky Fried Chicken KFC, 4401 Shawnee Dr, EXCELLENT Do not penalize result for giving
Kansas City, KS 66106 current proper name of business.
Andinion Rugs Andonian Rugs Sales, EXCELLENT Do not penalize result for
Services & Repair, 92 correcting spelling error or giving
1st St, Omak WA 98005 full proper name of business.
Building A, Vancouver Bldg A, Vancouver EXCELLENT Perfect match
General Hospital General Hospital, 899
W 12th Ave, Vancouver,
BC V5Z 1M9, Canada
Building A, Vancouver Vancouver General GOOD Slightly less specific. But there is
General Hospital Hospital, 899 W 12th still a STRONG relationship
 Ave, Vancouver, BC V5Z between the result and the
1M9, Canada query.

Vancouver General Vancouver General EXCELLENT Perfect match

Hospital Hospital, 899 W 12th
Ave, Vancouver, BC V5Z
1M9, Canada
Vancouver General Bldg A, Vancouver
Hospital General Hospital, 899
W 12th Ave,
Vancouver, BC V5Z
1M9, Canada
GOOD Compare the full relevant portion
of result. Slightly too specific
(there is a STRONG relationship,
but the building is just one part
of the whole).

Nordstrom, Northgate Nordstrom, Northgate EXCELLENT Perfect match

Mall Mall
Nordstrom, Northgate Women’s shoe dept, GOOD Compare the full relevant portion
Mall Nordstrom, Northgate of result. Slightly too specific
Mall (there is a STRONG relationship,
but the shoe department is just
one part of the whole store).
Nordstrom, Northgate Northgate Mall BAD Much less specific. Unlike a
Mall building in a hospital complex,
Nordstrom store in the mall does
NOT have a strong relationship
with the mall itself.
Northgate Mall Nordstrom, Northgate BAD Compare the full relevant portion
Mall of result. “Northgate Mall” is in
the result, but result is a different
business. Though contained by
the mall, it does not have a
STRONG relationship other than
location. It is much too specific.
Northgate Mall Nordstrom BAD Compare the full relevant portion
of result. Result is a different
business than the one in query.

4. Version history
2021-08-07 (v1.4):

- It is recommended that you re-read this entire guideline because there are so many changes
(including many small ones not listed below).
- Section 1 Overview (and elsewhere): emphasis that outlined words in result are only hints. They
may not indicate a match to the query intent for the outlined part of the query.
- Section 2.1 Judgment steps:
▪ The Query Type will help you understand the user’s search intent, especially when it
could be interpreted in different ways
 ▪ Compare only the part of the result that corresponds to the outlined part of the query
that you are judging. (Remember, the outlined part of the result is only a hint and may
not correspond to the outlined part of the query). For example, if the query is for a
business name, compare all of the result that is also a business name, but do not
compare other parts of the result (such as the address or city). If the business name in
the result is more specific, less specific, or different from the search intent, then
penalize.
- Section 2.2 Judgment Ratings. Changes to all ratings definitions. MUST RE-READ full table!
- Section 2.3.2 GOOD vs BAD: Please re-read. Many changes: clarified definitions, more examples.
- Section 2.3.3 BAD vs MISSING: Another update (highlighted): Select MISSING only if the result
has nothing to match the outlined part of the query. Applies only to similar query word types.
For example, if the outlined part of the query is a street name, but no street name appears in
the result at all, the street name in the query should be rated as MISSING. If a street name
appears in the result, but the street name is wrong, then that part of the query should be
marked as BAD.
- Section 2.3.5 Category searches:
▪ Many changes, please re-read.
▪ No longer consider entire result. Compare only the parts of result that relate to the
category word(s) but be sure to include all those parts in your rating decision.
▪ Added special case for Real estate, homes for sale, property for sale.
- Section 2.3.6 Result specificity:
▪ Many changes, please re-read.
▪ No longer consider entire result for some queries. Compare only the parts of result that
relate to the outlined part of the query but be sure to consider all those parts.
▪ More examples.
- Section 3 Examples. Add more examples, removed a few confusing ones, improved
explanations.

2021-07-30 (v1.3):

- Section 2.2 Judgment Ratings:

▪ EXCELLENT: Do NOT penalize for differences due to abbreviations, valid alternate
names, or corrections to spelling errors in query. Do NOT penalize extra, non-conflicting
information in the result (but be sure to penalize if the extra information weakens the
strength of the relationship of result to query).
▪ GOOD: Penalize for weird format or bad characters.
- Section 2.3.3 BAD vs MISSING: Updated rule (highlighted): Select MISSING only if the search
intent for that part of the query is not included in the result at all. Applies only to similar query
word types. For example, if a query includes a street name, but no street name appears in the
result at all, the street name in the query should be rated as MISSING. If a street name appears
in the result, but it is completely incorrect, then the street name in the query should be labeled
as BAD.
- Section 2.3.6 Result specificity. Completely rewritten. Please re-read entire section. Highlights:
▪ To understand difference between slightly vs much less/more “specific”, emphasize
consideration of STRONG vs WEAK relationship between query and result.
▪ Consider the entire part of the result that corresponds to the portion of the query that
you are judging. For example, if the query is for a Business name, consider all of the
result that is also a business name.
 ▪ If query portion is MAIN THING user is looking for and it is a Place, Natural Feature, or
POI, then compare the entire result and penalize if result is too specific.
▪ Changed and added several examples.
▪ Improved presentation to show query portion vs corresponding result portion.
- Section 3 Examples. Changed / added several examples.

2021-07-22 (v1.2):

- Section 2.1. Judgment steps:

o Added query types (e.g. Address, Place, POI, etc.)
o Updated screenshot for Query vs Judgment search subject selection
- Section 2.2. Judgment ratings: Added underlining for BAD and MISSING definitions, added new
example for MISSING
- Section 2.3.3. BAD vs MISSING: added additional examples
- Section 2.3.6 (NEW!): Result specificity.
- Section 3: Added more EXCELLENT and GOOD examples. Added Place, POI, Natural Feature, and
business examples, including judgments affected by specificity (see Section 2.3.6).

2021-06-30 (v1.1):

- Section 2.2. Judgment ratings: clarified wording: “GOOD: Select this rating if the result is not a
perfect match for the part of the query you are judging, but a reasonable match. The result may
have small but clear differences from the user’s search intent for that part of the query, but the
differences are NOT major and the result is NOT completely wrong.
- Added Section 2.3.1: EXCELLENT vs GOOD
- Expanded Section 2.3.4 (SPELLING ERRORS): “If a query contains an obvious typing or spelling
error, do NOT penalize if the result corrects the error. But if result has spelling error, then you
should penalize.” Also added an example to table.
- Added Section 2.3.5: BUSINESS CATEGORY SEARCHES
- Added Section 3: Examples