Search Topicality Guidelines and Changes

In App Guidelines – condensed version of Downloadable Guidelines.

Downloadable Guidelines – Contain all details of the Hitapp, as well as examples of all label types and
edge cases.

Terms to Understand

Query – a question, especially one addressed to a search engine.

Keyword – a word or concept of significance.

Label – the answer selected for each question within the Hitapp.

About – on the subject of, concerning.

Focus – the center of interest or activity.

Scope – the extent of the area or subject matter that something deals with or to which it is relevant.

Result Page – the webpage given in the hitapp to be evaluated.

Mostly – as regards the greater part or number.

Somewhat – to a moderate extent or by a moderate amount.

Barely – only just; almost not.

Broad – covering a large number and wide scope of subjects or areas.

Narrow – limited in extent, amount, or scope; restricted.

Similar – resembling without being identical.

Task Overview

When a user issues a query on a search engine, their goal is to find a webpage that’s about the query.
Topicality is a measurement of how well a result page meets that goal. Your task is to determine how
much of the content of the results page is focused on the query and how the scope of the result page
matches the scope of the query.

Steps for Judging

Understand the query – Read the query. Use the Query Intent Description, Bing Results and Google
results to ensure you understand what the user is searching for. Think about what the keywords of the
query are.

Understand the Result Page – Read through the main content of the Result Page and determine how
much of that content is about the query and what its scope is.

Understanding the Questions:

 1. Is the Result Page about the query? How much does the Result Page focus on the query?
a. Content focused on the query is content that discusses or provides information
about the query.

The Result Page

Content focused on the Query

Slightly

Somewhat

Completely Mostly
Not at All

Understanding the Labels:

1. Is the Result Page about the query?

a. Completely – the result page is almost exclusively about the query and there is little
to no content that is not focused on the query.
b. Mostly – the result page is mostly about the query. It is not completely about the
query as it has some content about other topics/subjects beyond the query.
c. Somewhat – the result page is somewhat about the query. It still has some content
about the query but much of its content is about other topics or it only focuses on a
part of the query.
d. Slightly – the result page is only slightly about the query. It may contain some
related keywords, links, or thin content about the query, but it is hard to see a
useful connection, however it is not completely unrelated.
e. Not at All – the result page is not about the query. It is off topic or is missing a key
part of query.
 Labeling the Hitapp – select the label that best answer the hitapp questions.

Questions to consider:

Does the Result Page address all of the keywords of the query?

Does the Result Page focus only on the query? Is it also about other topics as well?

Does the scope of the Result Page match the scope of the query? Is one more narrow than the
other? Is one more broad?

Examples

Query Result Page Label Explanation

411 River Rest https://www.tripadvisor.com/Hotel_Review-g55026- Not at all The query is for a specific
Campground d10546990-Reviews-Watauga_Dam_Campground- campground in
Elizabethton_Tennessee.html Tennessee and the
Result Page is for a
different campground.
This Result Page is Not at
All about the query.
Xfinity Login https://my.xfinity.com/?cid=cust Slightly The page is the Xfinity
homepage with links to
login to the user’s
account. This would be
Slightly about the query.

George https://www.history.com/topics/us-presidents/george- Somewhat The page is a biography

Washington’s washington about George
birthday Washington and while it
does give his date of
birth, it also covers many
other topics related to
George Washington.
George https://www.history.com/topics/holidays/presidents-day Somewhat The page’s main focus is
Washington’s President’s Day, While it
birthday is still about the query,
the scope of the page is
very limited.
Blueberry shorturl.at/pE569 Mostly The page is Mostly about
Muffin recipe a blueberry muffin
recipe, however it also
provides information
about how to use extra
muffins.
Florida lotto http://flalottery.com/lotto Completely This page is completely
about the query. All
content on the page is
about the query.
Can’t judge the hit? – some hits can’t be judged for reasons that are outside of your control. If that
occurs, please select the appropriate reason and submit.

• In Wrong Language – if the query or result page is in a different language from the one
marked in the title (Search Intent [EN]). English queries with English Result Pages should
always be judged normally.
• Does Not Load – If the page doesn’t load.
• Adult Content – If the page has adult content.
• Requires Login – If you cannot view the content of the page without logging in to the
site.
• Harmful Content – If you suspect the page might download something harmful to your
computer (spyware or malware).