In this task you will see user searches and ads for hotels, vacation rentals, or other types

of lodging we
might show with the results for these searches. Your task is to rate how closely the advertised lodging
matches what the user wants based on their search.

The following are the criteria to consider:

o User Intent
• User intent refers to the intent of the user based on their search.
• Judgment criteria: If the user has no direct or indirect intent of looking for places to stay,
then select the corresponding label.
o Ad Intent
• Ad intent refers to the information shared in the ad.
• Judgment criteria: If the ad is not about a hotel, vacation rental, or other types of
lodging, then select “Ad not related to lodging”.
o User specified location
• This refers to the location specified by the user in their search terms.
• The intent of looking for places to stay may be implied through a location search:
▪ Searching for “Palau Island” implies the user is interested in lodging in this
location
• Judgment criteria: Search on map to determine how far away the hotel shown in the ad
is from the location mentioned in user's search. If the distance is more than 30 miles (48
km) apart, the ad is considered 'far'.
o User specified requirements
• These are the terms included by the user to specify exactly what type of lodging or
features they want. These may include explicit things such as:
▪ star ratings (5-star, 3-star, etc.)
▪ descriptive terms (“cheap,” “luxury” etc.)
▪ amenities (pool, spa, gym, pets allowed, free breakfast etc.)
▪ type of lodging (hotel, vacation rental, etc.)
• These terms may be implied:
▪ searching for “Ritz-Carlton Hotel” implies the user is interested in a 5-star hotel
• Judgment criteria: If the user specified or implied any such terms, the ad should not be
hugely different from user's requirements. Ex. if the user searched for '5-star' hotel,
showing an ad for a 1-star hotel would be 'Bad', but showing an ad for a 4-star hotel or a
5-star vacation rental would be 'Fair'. Ex2. If the user searched for St. Regis Hotel,
showing an ad for a 1-star hotel would be ‘Bad’.
o Ad Image
• This refers to the images for the ad that would be shown to the user.
• Judgment criteria: Image should show a relevant item, such as a hotel, amenity (ex.
pool) etc., and the image should not be of poor quality (blurry, overly zoomed in or
zoomed out, etc.)

Rating Options

The user has no direct Search is not about staying at hotels, vacation rentals, or other types of
or indirect intent of lodging.
looking for places to
stay
 Ad not related to Ad is not about hotels, vacation rentals, or other types of lodging.
lodging

Bad Ad matches one or more of the following criteria:

• Far from the location specified by user.
• Hugely different from user's requirements.
• Image is not relevant and/or is of poor quality.
Fair Ad does not match all of the user’s requirements but is similar enough to
be a reasonable recommendation.
Good Ad matches all of the user’s requirements and is not far from the
location specified by user.
Excellent Ad matches all of the user’s requirements and is the exact same hotel
that the user searched for.

Examples

Ad Rating Comment

The user has User search does

no direct or not indicate a
indirect requirement for
intent of hotels, vacation
looking for rentals, or other
places to types of lodging.
stay
Bad Far from the
location specified
by user - user
searched for
hotels in Brazil
and is very
unlikely to choose
a hotel located in
Peru.

Bad Far from the

location specified
by user and
hugely different
from user's
requirements.

The user searched

for vacation rental
in Hawaii,
therefore, a 5-star
hotel located in
New York is a Bad
match.
Bad Hugely different
from user's
requirements.

The user searched

for Four Seasons,
a 5-star hotel,
which implies the
desire for a 5-star
hotel. The user is
unlikely to choose
Red Roof Inn, a 3-
star hotel,
therefore this ad
is a Bad match.

Bad Hugely different

from user’s
requirements.

The user searched

for a 2-star hotel.
Showing a 5-star
hotel is a Bad
match.
Bad Image is not
relevant and/or is
of poor quality.

Fair Ad does not

match all of the
user’s
requirements but
is similar enough
to be a reasonable
recommendation.

The user searched

for Comfort Inn, a
3-star hotel, which
implies the desire
for a 3-star hotel.
The ad is of a
different 3-star
hotel in the same
area and
therefore a Fair
match.
Fair Ad does not
match all of the
user’s
requirements but
is similar enough
to be a reasonable
recommendation.

The user searched

for a 5-star hotel
in Miami Beach.
Showing a 4-star
hotel in Miami
Beach is a Fair
match.

Fair Ad does not

match all of the
user’s
requirements but
is similar enough
to be a reasonable
recommendation.

The user searched

for hotels in New
York. Showing a
vacation rental in
New York is a Fair
match.
Good Ad matches all of
the user’s
requirements and
is not far from the
location specified
by user.

Excellent Ad matches all of

the user’s
requirements and
is the exact same
hotel that the user
searched for.