Answer Relevance

Guideline Version 1.1, 08/31/2021

HitApp
Guideline Updates:
• 1.1 – Included example to emphasize keywords do not automatically make an Answer relevant

Task Description
When a person uses a Search Engine such as Bing, they will type text in the search bar to try to
achieve whatever their goal is – we call this user input a Query. The Search Engine will then
return information to the user in the form of a Search Results Page. We have taken text from
one of the websites in the Search Results and presented it as a short Answer to the user to try
and address their Query. In this HitApp you will be given a user Query and an Answer.

Your task will be to identify the intent of the user’s query (what they are trying to do) by looking
at the Search Results and determine if the Answer provided is relevant to what the user is
trying to do.

Note: To view search results ensure you are using Chrome Browser and install the W5P plugin

HitApp Layout

1.) User Query – A collection of text that a user has typed in

2.) Answer – A short collection of text pulled from a web site that is attempting to address
the intent of the User’s Query
 3.) Source Page Tab – Tab that will display the web site that the Answer was taken from
4.) Search Results Tab – Tab that will display the Search Results from Bing
5.) Judgement Area – Area where you will select your Judgment once you’ve finished
working through the Hit

Instructions
Task Process
1.) Identify and Understand the User Query:
• Use the Bing Search Results found in the tabs of the HitApp to determine likely
intents of the User’s query
• If you don’t have enough knowledge on the query to make a judgment, use the
Search Results to understand the context and intent of the query and use your
best judgment to judge
2.) Confirm Source Page is Available and On-Topic
• Check that both the Query and Answer are in the same language and are in the
Market Language or English
• Check the source page by clicking the “Website” tab and check if the page loads
correctly and that the content is still available
• Ensure that the source page is relevant to the query and not using Keywords that
make it seem related
3.) Make and Submit a Judgment
• After understanding the likely intent(s) of the query, determine if the Answer is
relevant to the same topic as the Query
• Ensure that the Answer is relevant to the query and that it does not just have
similar keywords that make it appear to be about the same topic
• Determine if the Scope of the answer is similar to the Scope of the query
• Submit a Judgment

Ratings
o Good
o The Answer addresses the most likely intent of the user’s query
o The Answer’s Scope isn’t too broad or narrow compared to the Query (Example)
o Fair
 o The Answer is still relevant to the user’s Query, but the scope is moderately
broad or narrow compared to the user’s Query (Example)
o The user is looking for multiple things but only receives information on one
o The Answer is related to an unlikely user intent for the query (but still a plausible
intent)
o Bad
o The Answer is irrelevant to the user’s query (Example)
o The Source Page returns an error (404, “server did not respond”, etc.) (Example)
o The Answer (or a slight edit of the Answer) is not present on the Source Page
o The Answer is extremely broad or narrow compared to the Query to the point
where it is no longer relevant
o The Answer or Source Page contains detrimental or harmful content to the user
o The Location of the User makes the Answer not relevant to the Query
o In Wrong Language
o *Note*: English Queries with English Answers should always be completed and
Judged as normal, do not mark these queries as “In Wrong Language”
o The Query and/or Answer are in a language other that the market language or
English
o The Query and Answer are in different languages (Example)

Additional Considerations
Beware of Answers Containing Keywords That Are Not Actually Relevant to Query
Keywords can be a great way to quickly identify if an Answer is relevant, however it’s important
to confirm that the passage provided is in fact relevant to the query. In some situations, an
answer will contain keywords, but will not actually be relevant to the user’s query.

Here is an example of a situation where an answer at first glance looks like it is relevant to the
user’s query. The main intent of the user’s query is to find out how many controllers a
Playstation 4 comes with when purchased. The answer contains many keywords found in the
query such as “ps4” and “controllers”, however the Answer is not actually relevant to the user’s
query. In this case, the Answer mentions the maximum amount of controllers that a PS4 can
use. This would result in a final rating of Bad as the answer is not relevant to the User Query of
how many come with a purchase. It is necessary to always ensure that the Answer is actually
relevant to the Query, regardless of Keywords. Keywords in the Answer Text, Title, or URL
does not automatically make an Answer Relevant.
HitApp Answer

Here is an example of a situation where an answer at first glance looks like it is relevant to the
user’s query. However we can see that the URL and the Source Page both show that the Answer
is giving the user information on a Nickel-Plated Faucet – not a brass faucet. This would result in
a final rating of Bad as the answer is not relevant to the User Query. It is necessary to always
check the source page to ensure the answer is relevant.

HitApp Answer Source Page

Incomplete Lists / Steps Can Be Relevant

An incomplete list of items, steps, etc. can still be relevant if the full list on the source page is
relevant. For example, the step-by-step guide in the below Answer is not showing all of the
steps, however the complete list is on the source page. The final rating for this Hit would be
Good.

HitApp Answer Source Page

Scope of Query vs. Answer Affecting Final Rating

Users generally expect to get answers of similar scope to their query intent. Individuals who
type in a broad query (“Dog Breeds”, “Businesses”, “Illnesses”) would not be satisfied with
overly-specific answers to those queries. Conversely, users who type in narrow queries would
not find overly-broad answers as relevant. These sorts of situations should be labeled as Fair.
In extreme cases, a difference in scope can be so large that it would not be seen as relevant to
the user’s query. Below is an example of a scope difference that is so large that the final rating
would be Bad. In this Hit, the user’s query is “Styles of Art” – a very broad query.

o A Good answer would have been a high-level list/descriptions of different types of art.
o A Fair answer might focus on a single high-level type of Art (impressionism, cubism,
modern art, etc.)
o This Bad answer is extremely narrow in scope compared to the query as it is a very
specific style of art

Examples
Final Answer of “Good”

HitApp Query/Answer Source Page

In this Hit, the intent of the User’s query is to get information on how to become a barber
without going to school. By checking the souce page we can determine that the page still loads
and the content of the Answer is on the same topic that the query is asking about. As the
Answer is relevant and on the same scope as the Query, the final rating would be “Good”.
Final Answer of “Fair” Due to Scope Mismatch

HitApp Query / Answer Source Page

In this Hit, the intent of the user is to find places in the video game “Grand Theft Auto: 5”. The
answer provides specifically liquor shops in the game which is quite narrow in scope compared
to the query. The Answer would be much more relevant to the user if it included different
locations beyond the narrow scope of just liquor stores in the game. Due to the narrow scope
of the answer compared to the broad scope of the User Query, the final rating would be “Fair”.

Final Answer of “Bad” Due to Answer Not Being Relevant

HitApp Query / Answer Search Results Page

In this Hit, the user is most likely looking for information related to the program Movie Maker
10. We find this out by checking the Search Results page to determine the most likely intent of
the query. The Answer does not provide any relevant information to Movie Maker 10 – instead
it mentions other types of free alternatives to Movie Maker 10. As this would not be relevant to
the user’s query, the final rating would be “Bad”.

Final Answer of “Source Page Issues” Due to Source Page Error / 404

HitApp Query / Answer 404 Error on Source page

In this Hit, the User is looking for information on the best paying engineer jobs. However, when
the source page is clicked, it leads to a 404 Error as seen above. Due to the Source Page
containing an error like this, the final rating would be “Bad”.

Final Answer of “Bad” Due to Answer / Source Page Being Harmful to the User

Example of Scam Site

If the Answer or Source Page contains any of the following content:

• Adult Themes
• Excessive Ads
• Malicious Content that could harm the user
The final rating should be marked as “Bad”.
Final Answer of “In Wrong Language” Due to the Query and Answer Being in Different Languages

HitApp Query / Answer

In this Hit, the Query is in English and the Answer is in French. As Query and the Answer are not
in the same language, the final rating would be “In Wrong Language”