USER HISTORY TO AD RELEVANCE JUDGING GUIDELINES

LAST UPDATED: 03/09/2021

UPDATED BY: NSARANGI

These guidelines contain confidential and proprietary information belonging to Microsoft Corporation. The recipient
understands and agrees that these materials and the information contained herein may not be used or disclosed without the
prior written consent of Microsoft Corporation.

Overview
In this project we will look at a user’s online activity and an advertisement that is shown to the user. You
will help us compare the ad to the user history item to determine how closely the two are related. In the
labeling interface, we show an item from the user’s history on the top and an ad on the bottom. A
history item is an action taken by the user, such as visiting a website, performing a search etc. As a
judge, you will choose the best label option to describe the relationship between the user’s history item
and the ad.

These guidelines describe how to categorize the relationship between the ad to a history item into one
of the following buckets: 1. Excellent, 2. Fair, 3. Bad, 4. Not Sure.

Note: Some of the links in the user’s history may not be accessible due to several reasons. However, the
displayed text of the URL would often contain salient information such as name of the website,
particular section the user visited etc., which can help you to judge with more accuracy. For example, in
the screenshot below, just by looking at the displayed URL, you can infer that the user is interested in
jobs at Accenture located in India.

Judging Interface
The judging interface will look like the image below. The annotations are further explained below:

1. User History: This section provides either the title of a webpage and a link to that page
(shown in the screenshot) or a simple search query.
2. The Ad: This is a blob of text comprising a title, a link, and a description of the Ad.
3. Rating Options: The options to label the relevance of this item to the ad shown on top as
Excellent, Fair, Bad or Not Sure. The rules to determine this label are explained in the next
section.
 1
2

Tips for Selecting Your Rating

This chart provides guidance on determining how closely a product matches the user query. Use it to
help select the most appropriate rating option.

Attributes Excellent Fair Bad

Seller, Color, Size, None of these attributes does not impact rating does not impact
Specification, need to match for an rating
Material, Style, example to be labeled as
Format, Excellent.
Quantities

Brand If present in both ad and If present in both ad and history, NA

history, then
all these attributes have to
match for the example to be
labeled as Excellent.
Location : physical If present in both ad and
location mentioned in history, then
query or history all these attributes have to
Age : defined for two match for the example to be
categories only: labeled as Excellent.
Children and Adults
Gender: defined for
two categories only:
Men and Women
Product category If both ad and history have a
product, then
Product needs to match
except for attributes listed in
AND
attributes do NOT match
OR
If present in only one of ad or
history
If present in only one of ad or If present in both
history, the example should be ad and history
labeled as Fair.
and the attributes
do not match,
label as Bad.
If both ad and history have a If the product
product AND both products are categories from
part of same product category the ad and history
(example: women’s clothing, cell are different, label
phones) as Bad.
 above three columns
OR

If either or both ad or history is a

category or search page AND both
ad and history map to the SAME
category

Rating Options
Excellent ad for this history item
 The history item is about the exact same thing (product/service) as the ad.
 The history item is about a variation of the same product/service in a different color (black
evening dress/History: red evening dress), size, specification, material, style, or format (print vs
Kindle) as the ad.

 If the ad (it will be part of the displayed title, description or URL ) contains a location and
the history item contains (In case of search history, it will be part of the displayed string.
In the case of a visited website, it can be part of the website’s title or URL or can be
known after opening the URL) a different location with the same/similar product,
it cannot be excellent. You should mark this as Fair.
 If only one among the ad or the history item contains a specific location, while the other does
not contain any location, the rating cannot be excellent.  You should mark this as Fair.
 A history item referring to the same product in a different target age or gender as the
ad cannot be excellent. Such a webpage should be rated Bad. For example, women’s trench
coats vs men’s trench coats or women’s shoes vs kid’s shoes.

Fair ad for this history item

 The ad and the history item target the same category of viewers (of the same age, gender, or
location) and belong to the same category or a related topic. The webpage should target the
same audience. For example, Women’s down jacket vs Women’s fleece jacket. Both products
target the same audience, they are related topics since they are both variations of jackets.
However, Women's down jacket and women’s shoes or women’s party dresses are different
categories altogether.
 The product in the ad and the product in the history item belong to the same category, but the
query is about a different brand of the product. For example, Ryobi impact driver vs Dewalt
impact driver.
 The ad is about the same category but is from a different location.
 The history item covers a more general category of the product/service, while the ad covers a
subcategory within the category. For example, kitchen small appliances vs toaster.

Bad Query for this Page

 The ad and the history item are completely unrelated.
  This could happen if the users targeted by the ad and the history item belong to
different segments such as different age, gender etc.
 This could also happen if the product/service in the ad and the history item are completely
unrelated.

Note: When determining if the ad and the history item are unrelated consider what the user wants to
accomplish or learn, and the main content offered by the history item.

Not sure
 The judge is not sure if this page is relevant or not. In this case, please leave a comment
explaining why you could not select a rating.

Examples

Ad: best beach hotels in Florida

Excellent Fair Bad
Florida Beach Resort (exact Hotel Booking Site Homepage Beach Body Tanning Salon
match) (same category, more general) Miami Beach
(unrelated)

Ad: black Northface jacket

Excellent
Red Northface Jacket
(exact product, different color)
Fair
Winter gloves
(Targets same group of users –
other winter outerwear)
Bad
Ski vacation
(Unrelated – a user shopping for
a winter jacket doesn’t indicate
vacation planning)