Change Log: Map SBS Rendering – V2

May 03, 2022

• Added example number 8 on page number 12 for a type of broken hit. Both maps should always
display the same geographic area. If not, the hit should be rated as broken.
May 15, 2022
• Added an example of a different rating scale without “About Same” option used in this study on
page 17. Note that all hits may not have “About Same” option. Carefully consider both the
images and choose the appropriate rating irrespective of which scale is displayed.
November 6, 2022
• Changed “Location Labels/Names” to “Place Names” on Page 3 and added a note to
differentiate between place names and business/poi labels on page 4. This change is made
because judges are confusing between names of places and names of businesses/POIs.

i
Map SBS Rendering – V2
1. What is SBS Rendering?
You will be shown two map images, both showing the same region on the map. Considering all shown
details, you will decide which image (left or right) presents a “Much Better”, “Better”, or “Slightly
Better” view of the area shown on the map. Before you provide your rating, you will have to consider
many elements of the map such as transportation network completeness/styling/labels,
businesses/POIs, location names, water bodies, buildings, readability, colors and decide which side wins
overall. No search results are shown, but your judgment should reflect your preference of the
displayed image.

2. Instructions for using the Map SBS Rendering – V2 HIT app

A. Setting up
Login with your credentials and look for {language}_Map SBS Rendering – V2 HIT app to start judging
where {language} refers to your language. For example, English language judges should look for
en_Map SBS Rendering – V2 whereas Japanese language judges should look for ja_Map SBS
Rendering – V2 to Start Judging.

B. HIT app Step 1: Compare map images carefully

To make both screenshots visible in the side-by-side view, the images have been shrunk. Because of
this, it can be difficult to compare images properly.

1
However, you can (and should!) see the images at full size: click on an image or use the left and right
arrow keys. You can then flip back and forth between the zoomed image of each side to compare
them. To do this, click on the “Left | Right” navigator above the zoomed image, or use the left and
right arrow keys. To shrink the images back to normal and return to the main view, click on or outside
the zoomed image, or repeat the same arrow key twice, or press the ESC key. You should do this for
most HITs.
It will look something like this:

2
Carefully review the images shown above. Consider both the map images and indicate which side you
prefer and how much you prefer it. Each image shows the user’s view when they were searching for
information on maps, but before any results were returned by the maps engine. Your goal is to tell us
which image you prefer, based only on map data and its presentation.

Imagining you are the user who is looking at the map, consider the following questions:

Which side do you think most users would prefer and how much better is that side
than the other?

C. Judgment Criteria

Consider the following two major criteria while judging.

a) Level of detail
Consider the zoom level of the map i.e. how zoomed out or in it is. At the shown zoom level, think
about how much level of detail is appropriate. For example, at certain zoom levels showing even the
smallest/local roads can blur the location names displayed on the map and hence will result in a less
readable map. Consider the following dimensions for level of detail while judging:

i. Transportation Network Detail

o Transport network includes roads, railways, trails, ferry lines etc.
o Completeness: Which side displays the appropriate transport network? Consider
the zoom level and decide whether missing/additional roads on one side make a
difference.
o Color: Does one side clearly identify major roads (e.g. highways) with yellow/other
color styling?
o Labels: Which side shows appropriate labels for the transport network? Both,
showing no labels or displaying too many labels could be problematic.

ii. Businesses or Points of Interest (POIs):

o Businesses or POIs refer to entities such as restaurants, hotels, schools, tourist
points etc. which are identified with icons or pins on the map.
o Does one side show many businesses/POIs compared to the other side?
o Again, too many businesses/POIs can be distracting and too few will not help the
map users. So, consider the appropriate level of detail that will be helpful to users at
the zoom level before choosing a side.

iii. Place Names:

o Place names refer to names of entities like countries, regions, states, cities, villages,
and neighborhoods.
o Does one side show many place name labels compared to the other side?

3
 o Again, too many labels can be distracting and too few labels will not help the map
users. So, consider the appropriate level of detail that will be helpful to users at the
zoom level before choosing a side.
o NOTE: Text labels beside businesses/POIs are not place names.

iv. Parks & Forests:

o Which side provides an appropriate detail for parks & forests?
o Does the winning side show more parks and forests?
o Are they labeled?

v. Water Bodies:
o Water bodies would include lakes, ponds, rivers, seas, and oceans.
o Which side displays water bodies at an appropriate level of detail?
o Are labels provided for the water bodies?

vi. Building Footprints:

o Do both sides show the building footprints?
o Which side’s display of building footprints is better?

vii. Land Terrain:

o Does one side allow you to easily differentiate plain areas from
rugged/mountainous regions?

b) Overall presentation
Consider the overall view of the map with a particular focus on the following elements:

viii. Map Readability

o Readability refers to the quality of being legible or decipherable.
o For example, does one side display the labels with an appropriate font and size?
o Are labels easier to see and read on one side?
o Are labels displayed fully or are they cropped?
o Are labels in your language or use a foreign language?

ix. Overall Presentation/Colors

o Refers to the style of displayed image.
o Are appropriate colors used on both sides? For example, blue colored water bodies
on one side look better than grey colored water bodies in the other image.
o Maps on winning side generally do not look cluttered and use visually appealing
color schemes.

4
D. HIT app Step 2: Judgement
After considering the above questions, decide which image (left or right) you think is better and
indicate whether that image is “much better”, “better”, or “slightly better”.

a) Rating Categories
i. Slightly Better
o Both images are very similar in terms of either data or presentation, but one side has a
slight edge.
o Once you decide which side is better, think about how much additional value or
information the winning side provides to the user. If the additional details on winning
side is only of little value to the user, then it wins with a “Slightly Better” rating.
• Example 1:
Map A

Map B

Verdict and Reasoning: Map B wins with “Slightly Better” rating. Map B displays few additional
roads while Map A displays additional Parks & Forests. In our opinion, missing roads on the map
at this level of zoom is more problematic than missing green spaces. So, Map B wins with
“Slightly Better” rating as additional information on “Side B” adds only a little bit of value.

5
• Example 2:
Map A

Map B

Verdict and Reasoning: Map A wins with “Slightly Better” rating. Both sides display limited
information, however "Map A" has a slight edge as it displays additional parking lot roads and
couple of more businesses/POIs which add slight additional value to the user.

6
 ii. Better
o One side is clearly better in terms of either data or presentation.
o In these cases
▪ You can easily identify the winning side but the losing side also provides enough
detail.
▪ Even if both sides show similar detail, presentation or readability on losing side
is bad, hence winning side gets a “Better” rating.
• Example 3:
Map A

Map B

Verdict and Reasoning: Map A wins with “Better” rating. Clearly, “Map A” provides more detail
in the form of more number of businesses/POIs spread throughout the map. “Map B” also
provides businesses/POIs although we can easily see that they are less in number and don’t cover
the entire map. Additionally, “Map B” displays a green area.

7
• Example 4:
Map A

Map B

Verdict and Reasoning: Map B wins with “Better” rating. Clearly, “Map B” provides more detail
in the form of more number of businesses/POIs and additional green areas. But, “Map A” also
provides quite a number of businesses/POIs along with more place names and water bodies.

8
 iii. Much Better
o One side is unarguably better in terms of either data or presentation. By comparison,
the other side is terrible.
o In these cases
▪ The winning side presents a good amount of detail whereas the losing side looks
quite empty with almost no or very little detail.
▪ Losing side may have comparable or slightly lesser detail but map presentation
and readability is extremely bad, hence winning side gets a “Much Better”
rating.
• Example 5:
Map A

Map B

Verdict and Reasoning: Map B wins with “Much Better” rating. Clearly, “Map B” displays many
more businesses/POIs compared to “Map A” and hence is “Much Better” at providing
information to the users.

9
• Example 6:
Map A

Map B

Verdict and Reasoning: Map A wins with “Much Better” rating. "Map A" is much better because
it displays significantly more businesses/POIs compared to "Map B".

10
 iv. About Same
o Both sides look almost the same with little or no difference in data, presentation, and
colors.
• Example 7:
Map A

Map B

Verdict and Reasoning: Map A and B look exactly same and hence this hit can be rated “About
Same”.

11
 v. Broken
o If one or more result images are incomplete or too blurry to read, or show different map
areas on each side then please select “broken image”.
o If you see “LOADING IMAGE”, please wait. If after 30 seconds any of the images are still
blank, please mark as broken, and select “ok” in the pop up to move onto the next hit.
o You will get the below prompt before rating hit as BROKEN.

• Example 8:
Map A

Map B

Verdict and Reasoning: BROKEN hit as both maps show different areas. Both maps should
display the same geography to be rated.

12
• Example 9:
Map A

Map B

Verdict and Reasoning: BROKEN hit as “Map B” did not load fully.

13
• Example 10:
Map A

Map B

Verdict and Reasoning: BROKEN hit as “Map A” loaded improperly.

14
*Please note:

• A blank map is not the same as a broken image. For example, if one side is blank while other
side provides detail then it is not a broken hit as seen in below example.
• However, if both images are blank then you may mark that hit as BROKEN. Please make sure
that both sides are indeed blank before rating hit as BROKEN.
• Example 11: This is not a BROKEN hit
Map A

Map B

Verdict and Reasoning: “Map A” wins with “Much Better” as “Map B” displays almost a blank
canvas with no detail.

15
• Example 12: This hit can be rated BROKEN
Map A

Map B

Verdict and Reasoning: BROKEN hit since both maps display an empty canvas and there is
nothing to judge.

b) Rating Scale
Below is the rating scale used in this study. Based on your judgment, you need to choose one of the
buckets shown in the below figure.

16
Note: For cases where we know that images cannot be rated “About Same”, we display the rating scale
without the “About Same” option as shown below. Presence or absence of “About Same” rating option
should not influence your decision. Please carefully consider both the images in accordance with the
judgment guidelines before providing your rating.

c) Important Note
Occasionally, you will come across scenarios where each side has some missing data. Please use the
following thumb rule to pick the winner in such situations:

Think of yourself as the user of this map and ask yourself: given the current map view, what type
of information is most important to me?

Let’s consider the scenarios below:

1. One image is missing a few inner streets whereas the other image is missing an exit on a
highway/freeway. In our opinion, “freeway exit” is very critical information; if a side is
missing this data point, then it should get penalized more.
2. One image is missing trails inside a park whereas the other image is
missing some streets inside the same park. In this case, you should choose the first image
over the second image as streets are more important than trails, even within a park.
However, if this park happens to be very popular, then you may still prefer the second
image.

You may not agree with the above, and you should tell us your own opinion as if you were the user
yourself. Our guidance is always that you should think yourself as an actual user of this map, and
consider "What type of information is more important to me?"

E. Comments
Below HIT app step 2, you will find a few important comment tags as shown below. Please provide
the loss reasons for a hit using these tags.

17
 To comment, please click on the tags. For example, if you think “map readability” and “parks &
forests” are the loss reasons for a hit, then simply click on these two tags and your view will
change as shown below.

If you want to remove a comment tag, then simply click on the selected tag. For example, if you
want to remove “parks & forests” then click on the highlighted/selected “parks & forests” tag
and it will be removed.

If you think the tags provided do not sufficiently describe the loss reasons for a hit, then simply
start typing in the comment box below the tags and press enter to create a new tag.

Please make sure your comments are consistent with the judgements given in step 2.

F. Troubleshooting
1. Images of the map may load slowly sometimes. Please be patient. If the problem persists, there
may be a short-term problem with your connection or with one of our servers. We’re very sorry
for the inconvenience, but please try again later. If the problem is with image loading, the problem
usually is fixed within a few minutes. If the problem is with loading the task, please contact your
vendor contact if it lasts for more than a few hours.
2. If images are incomplete, or blurry, or display different map areas on each side then please select
“Broken image” checkbox.
3. If images don’t ever load, skip the HIT or reload the browser page.
4. If the HIT app buttons are all squished together or are difficult to see, it may be necessary to adjust
the zoom/magnification level in your browser. For example, click on the page and use the CTRL
and ‘-‘ or ‘+’ keys to adjust. Also, some Internet Explorer versions have a zoom level adjuster on
the lower right corner of the browser window.

Thank you for reading the guidelines; you should now be ready to take the qualification test.

18