SEARCH RESULT SATISFACTION

GUIDELINES
Who is this task for?
• Have you ever used a search engine like Google or Bing in your life?
• Do you consider yourself smart, thorough, and attentive?
• Are you curious about different topics and would like to learn more
about things while you earn money?

If the answer to the above questions is Yes, then this task is for You! Read the guidelines to become a
master of Search Result Satisfaction.

Task Overview
When a user issues a query on a search engine, their goal is to find a webpage
that satisfies their needs. While we cannot know exactly what the user was
looking for when they made the query, we can still evaluate each possible
page and determine if and how well it could satisfy the user’s expectations.
Your task will be to consider how well the user would be satisfied by the result
page.

Terms to Understand
Term Definition
Query a question, especially one addressed to a search engine
Keyword a word or concept of significance
Label the answer selected for each question within the Hitapp
Result Page the webpage given in the hitapp to be evaluated
Satisfaction fulfillment of one’s wishes, expectations, or needs
Expectations the belief about, or mental picture of, the outcome
HitApp Layout
9 10
1 12
2 11 13
3
a
b

4
c
14
5
c
6

7
8

1 Query The question the user has entered into the search engine.
2 Query Location The user’s location at the time they submitted the query
3 Query Intent Description information provided to help you understand the query
a Query Intent information about the possible intent the user may have had when
submitting the query
b Expected Results possible web pages or information that could satisfy the user
4 Unable to Judge options for when you are unable to judge the hit
c Learn More additional details about the options available
5 Hitapp Question what you are trying to determine in the hitapp
6 Labels all possible responses for the hitapp question
7 Comments a free text area for judges to provide additional feedback
8 Submit Button used to submit your judgment
9 Result Page Tab a tab to display the web page to be evaluated with the query
10 Research Tabs tabs for search results from Google and Bing for the query to aid in
understanding the search
11 Open in new Tab can be used to open the URL in a new browser tab if the page does
not load in the application
12 Guidelines a link to view the hitapp guidelines while working the hitapp
13 Report empty page use to report a hit with no URL in the address bar
14 Result Page The web page to be evaluated
Steps for Judging
Understand the query
Workflow

Read the query and query location. Use the Bing and
Google Results to ensure you understand what the user
is searching for. Think about what the keywords of the
query are.

Understand the Query Intent Description

(Query Intent Description may not be available on all hits)

Read the Query Intent.

Read the Expected Results May Include. Is the Result

Page included or similar to the sites listed?

Understand the Result Page

Read through the main content of the Result Page to

understand how much of that content is related to the
query. Determine if the page can be judged.
Determining a Label:

User’s Purpose
Expecta- of the
tions Page

Does not Meet

User’s Purpose
Expecta- of the
tions Page

Read the question. How well does the Result Page Somewhat Meets
meet the intent and expected results?

Choose the appropriate label.

a. Does Not Meet – The result page does not meet the User’s
Purpose
expectations of even a few users. The result page may be Expecta-
of the
unrelated to the query.
b. Somewhat Meets – The result page would meet the
tions
Page
expectations of only a few users. Either not many users
would look for this information, or the page has several
issues or flaws and would not fully satisfy the user’s Meets
expectations.
c. Meets – The result page fully or largely meets the user’s
expectations.

Be sure to add any needed comments before clicking

on Submit.
Tips for Judging
• If the Result Page does not open in the hitapp, use the ‘Open in new tab’ button to view
the Result Page in a new tab in your browser.
o If the Result Page still does not load, or you receive a 404 or similar error, use the
Unable to Judge option ‘Page did not Load’.
• If the page requires you to be logged in to view the content, use the Unable to Judge
option ‘Login Required’.
• If the Result Page is in a different language from the hitapp, use the Unable to Judge
option ‘In wrong language’.
• Keyboard shortcuts can be used to select labels and submit the hit. These will appear
next to the label in brackets [] to indicate which key to press.

Examples

Query Result Page Label Explanation

Iflagler https://iflagler.org Meets The result page meets the user’s
virtual expectations as it is the
school homepage of the Iflagler School.

New York https://www.usnews.com/t... Somewhat The result page somewhat meets

Times Meets the user’s expectations as it
provides headlines from the New
York Times, however the user
would prefer the official website
instead of an aggregator site.

Craigslist https://orangecounty.craigslist.org/ Does not The result page does not meet
orange Meet the user’s expectations as the
county fl user is looking for Craigslist in
Orange County, FL while the
result page is Craigslist for
Orange County, CA.