Search Engine Relevance

Result Preference

Before you start

IMPORTANT: This HitApp requires the UHRS Extension. This extension helps with loading pages in
frames inside the app. This improves your experience, speeds up judging and leads to better quality
labels.

Please install the extension from the Chrome Web Store before launching starting this task. If you do
not install the extension, you’ll see an error (see below) which also contains the download link.

THIS INSTALLATION WILL TAKE NO MORE THAN A MINUTE.

Here is the link and screenshot of the extension:

Overview
The goal of this task is to judge the relative relevance of results from a search engine. You will be
performing 2 main tasks:

1) Understand the query

2) Compare a few pairs of search results and say which result is better

Please familiarize yourself with the keyboard shortcuts for each step

Step 1: Understand the intent of the query

You will be shown a query issued to a search engine and the location of the user. Please try to put
yourself in the mindset of the user issuing the query from that location.

Screenshot of this step

What were they hoping to find? There might be a range of intents. For example, most people issuing
the query “facebook” are probably looking to go to Facebook’s home page, but there will be some
people interested in learning about the company through Wikipedia’s page about Facebook. There
might be other people interested in downloading the Facebook app from either iTunes or Google Play as
well.
Common intents are Navigation (people looking to go to a specific site), Information (people looking to
understand more about a topic), Shopping (people looking to buy a product), News (people wanting the
latest information), or Images/Videos (people looking for visual results).

Action: Review intent description created by an earlier judge

For many queries we asked another judge to think about the most likely intent of the query and what
useful results would look like. If this information is available, you’ll see it as well.

Please consider this information carefully as it may provide useful hints and make your judging task
easier.

To assist you, you can open Bing or Google to see what their search results pages look like. There is also
a link showing the user’s location on a map. (You can also access the map by clicking on the location
name.)

Shortcuts
[g] = Google
[b] = Bing
[m] = view location on map

Note on location: Think about whether location would matter. Most people issuing queries like
“museum” are probably looking for museums in their general area, not ones across the country or far
away.

Note on spelling: in this task you are seeing queries as they were typed by users. They sometimes
contain misspellings. If the misspelling is obvious, please, ignore it and judge the page as if the query
was spelled correctly. But please, be careful, some queries might be researching misspellings, or they
might be about product/brand names which might be misspelled on purpose. When not sure, use the
Google/Bing search links to see how the engines would interpret the query.

Note on URL queries: sometimes queries are web sites addresses, (e.g., “comcast.net”). If you don’t
recognize the address right way, please, use the Google/Bing research links, or simply visit that page
yourself. When doing that, please, note whether the page redirects to a different location (e.g.,
https://my.xfinity.com/ for comcast.net) and check that the page you are reviewing matches it.
Step 2: Give an overall preference for two results
You’ll be shown pairs of results and be asked to judge each. There will be at most seven pairs.

Each of these results was returned by a search engine for the query and location. Please try to
understand the content of each result and whether the page would be helpful for the intents you
thought about in Step 1.

Please consider the following aspects when reviewing each result

• Intent – what is the page’s purpose?

• Content – does the page provide enough information to meet the user’s need?
• Trustworthiness – can the user trust this web page or site?
• Location – is this page targeting people in a location near the user?
• Freshness – is this page’s content outdated or still recent/current enough?
• Usability – is the information presented logically or it the page hard to use?

Note on information: Some pages have very superficial content which won’t satisfy users either because
the content is incorrect or because there isn’t enough meaningful content. Other pages have much
more information than the users need making it harder to satisfy their intent. Ideally the page would
provide just the right level of content to satisfy the user for that query.

Note on page quality: Some pages will try to deceive users into clicking an ad or downloading some
malware onto their machine. Be on the lookout for pages whose primary purpose seems like it is trying
to fool users.

Note on location: Some pages, like news sites, might be more appropriate to show closer to the user’s
location for a query like “news”. Other pages, like Facebook’s home page, might always be appropriate
to show for a query like “facebook”. Be aware that sometimes the results should be different for
different users in different locations.

Note on freshness: Some pages might have stale information either because the page hasn’t been
updated in a while or because the page was about an old topic. For example, if somebody issues the
query “Manchester United” they are probably not looking for a page about an old match from a couple
of years ago. Instead, they would either be looking for the home page (which is always fresh/current) or
looking for the most recent match results or news.

Note on usability: Some pages are hard to use because they are laid out poorly or because too much
unnecessary content (ads, popups) interferes with the usability of the page.

The web pages should load below the results. However, many sites (even great sites) block the
embedding of the web page within our HitApp. When this happens (because it will happen), you must
open the search results and can do this by (1) clicking on the search result or (2) pressing the “Open
page” button or using the shortcuts. This is what a real user would see if they clicked the search result.

IMPORTANT: Sometimes the embedded web page will try to take the focus from our HitApp. This is
normal and expected. You might occasionally need to click inside the HitApp window to bring focus to
it, before the hotkeys start working. If you hit any accessibility issues, please, let us know.
Screenshot of this step

We see two results shown in columns (L – left, R – right) shown next to each other. We mark our
preference ranging from “L much more” to “About the same” to finally “R much more”.

If neither of the results is satisfactory, please use “Neither is likely to satisfy”.

Please take advantage of the hotkeys (1 to 7, 0 for “neither”, and “Enter” to move forward).

Shortcuts
[1] = Left if much better
[2] = Somewhat meets
[3] = Left is slightly better
[4] = Neutral
[5] = Right is slightly better
[6] = Right is better
[7] = Right is much better
[0] = Neither when both results are bad
[ENTER] = Next (to move to the next pair)
Considerations:

• Intent – what is the page’s purpose?

• Content – does the page provide enough information to meet the user’s need?
• Trustworthiness – can the user trust this web page or site?
• Location – is this page targeting people in a location near the user?
• Freshness – is this page’s content outdated or still recent/current enough?
• Usability – is the information presented logically or it the page hard to use?

Choose “much better” if one page is clearly superior either on multiple aspects or because the other
page is clearly lacking.

Choose “better” is one page would be preferred by users, but that preference isn’t quite as obvious.

Choose “similar” if the pages would be roughly equivalent for users. Maybe one page wins on a couple
of aspects but loses on a couple so there isn’t a clear winner.

If neither page would likely satisfy, perhaps because their intents are unrelated or too removed from the
query, or the pages themselves are “bad”, please choose “Neither is likely to satisfy”.

Note on large websites: Some websites are large and can satisfy a range of intents. Carefully review the
query and the pages (including their titles and URLs) to make the appropriate determination. Consider,
for example, the query “amazon” and the Amazon homepage and Amazon Music page. Both of the
pages might look very similar, but the user would likely prefer the homepage over the music page.

Note on pages that are defective: If the page does not load or work properly, is pornographic or
otherwise offensive to you or if it’s in a language you do not understand, please ignore it, and indicate it
by choosing the appropriate flag next to the “Open page” button. Consider the “bad” page as losing to
the other one. If both are “bad”, choose “Neither”.

Note on pages that are or look very similar: while we do our best to avoid these cases, sometimes you
might see results that differ only slightly and lead to the same page (e.g., due to redirects). An example
is shown below. In these cases, please, use “about the same” or at most “slightly better”— if, for
example, one of the URLs is much shorter. No matter your preference, please be consistent with your
approach so that we can get the best signal.