Judgment Guideline for Related Pages

Introduction
When users search on Bing, they are often have a intent that trys to achieve a task, and
need additional information to do so. The first page they click on (green one) in the search
results isn't always comprehensive, and additional pages can retrieve information that
complements the first page. These additional pages (red ones) that help in this regard is
what we mean by "related pages".

In the following labeling task, we need annotators have some “thought reading” ability that
only read the green one page to guess users’ searching intent and further judge whether the
red one is or isn’t related to the green one page.
Annotators can use these possible directions to judge the “relatedness”: (1) provides a different
perspective on the user possible intent. (2) Or, reasonable next step in the most likely task a
user had in mind following a click on the viewed page. (3) Or, provides another option to
complete the task the user has in mind.

Preliminaries
Source Document: the original document user clicked.
Target Document: The additional pages can retrieve information that complements the source
document.
Label: Used to define the correlation between Source Document and Target Document
(Related, Unrelated, Similar or Unknown).
Principles
You need to keep the following two principles in mind all the time for judging whether the Target
Document is related to the Source Document.
(1) Topic Consistency: Does the Target Document cover the same topic as the Source Document?
(2) Content Extensibility: Does the Target Document provide a significant amount of information that
the Source Document does not?
Guideline
You would be provided with two web pages. The left-hand side (we call it Source Document) is the
article that user has just read and on the right-hand side (we call it Target Document) is another article.
We need you to judge whether the Target Document is related to the Source Document.
Steps
Please keep the above two principles in mind and follow the guidelines below to help you decide
“relatedness”:
1. Step1: Completely understand the Source Document. Open the page and have a good grasp of
what it is about. You might also refer to the provided title and snippet (they are not always
accurate) if that helps.
2. Step 2: Repeat Step 1 for Target Document.
3. Step 3: After completely understanding both documents, select one of the following labels that
best describes the pair of documents; if you don’t understand one or both documents, or just not
sure about their relatedness, click skip.
Labels
 Related: Target Document complements or extends the topic in the Source Document while
remaining in the same topic. It is very likely that readers would want to read Target Document
after reading Source Document, e.g. (1) provides a different perspective on the user possible
intent. (2) Or, reasonable next step in the most likely task a user had in mind following a click on
the viewed page. (3) Or, provides another option to complete the task the user has in mind.
 Unrelated: Target Document is not relevant, not accurate, not useful for Source Document.
 Similar: (1) Target Document's content is very similar or is identical to Source Document and
can't provide more information about Source Document Topic; (2) Or,
as long as the Target Document comes from different tabs of the same website as Source
Document.
 Unknown: It is truly hard to distinguish whether the Target Document and the Source Document
is Related or Unrelated, e.g. (1) the page’s language isn’t English (2) Or, the url can’t be opened.
(3) Or, don’t have a conclusion.
(Choose this will also be included in calculation for preventing garbled data.)
Examples

Source Document Target Document Explanation

Both documents answer
Related Example1(click here)
the same question, but
How to do so with baking soda
in different ways
Both documents answer
Related Example2(click here)
Source Document(click here) the same question, but
How to do so with baking soda
How to remove mildew from in different ways
Unrelated Example(click here)
clothes with vinegar Violate Topic
How to clean the grout between
Consistency Principle
bathroom tiles
Similar Example(click here)
Violate Content
How to remove mildew from
Extensibility Principle
clothes with vinegar

Related Example(click here)
An article about the history of the
Source Document(click here)
Taliban
A news article about the Taliban
takeover of Kabul
Unrelated Example(click here)
The ancient history of the Middle
East
Related Example1(click here)
A news article about Elon Musk &
the companies he owns
Related Example2(click here)
Source Document(click here)
An article about how different
Elon Musk tweet about selling
billionaires invest their money
Tesla stock
Unrelated Example(click here)
An article about Waymo's latest
developments
Related Example1(click here)
A product page for Nintendo
Switch controllers
Source Document(click here)
A product page for the Nintendo
Related Example2(click here)
Switch
A product page for an
Xbox Series S
Unrelated Example(click here)
A product page for a laptop
charger
Source Document(click here) Similar Example(click here)
Florida Driving Records Another page for
Florida Driving Records
Similar Example(click here)
Source Document(click here)
CIBC Official website’s
CIBC Official website
Another page
The Target Document
gives context that covers
the main topic of the
Source Document more
comprehensively
Violate Topic
Consistency Principle
The Target Document
focuses on a different
aspect of the same
general topic
The Target Document
focuses on a different
aspect of the same
general topic
Topic connection is too
tenuous; could be a good
recommendation but not
a good related page in a
search context
The Target Document
presents
products/services that
complement a
product/service presented
in the Source Document
The Target Document
presents
products/services that
complement a
product/service presented
in the Source Document
Violate Topic
Consistency Principle
Provide almost
the same content,
Violate Content
Extensibility Principle
Both Document come
from the same domain,
Violate Content
Extensibility Principle