Task Pane Relevance Guidelines – Crowd – v10.

Task Goal & Terms

Use a Query + paired Search Results (w/ brief side research as needed) to answer 2 questions about how well an Info Card satisfies a user’s
Intent. You may do side research to understand these queries and their intent.

Sometimes, a specific Location should be considered when making your judgment.

Question 1: How many Search Results match the Info Card topic?
Tips: How do I know if a given Search Result matches the Info Card topic?
Question 2: How well does the Info Card match & satisfy the Query Intent?
Reference Table: Scenarios and How to choose?
Optional Comments & Submit
Technical Difficulty / Adult
Thank you!

QUERY: A question/idea someone types or speaks into a search engine. You likely issue “queries” multiple times a day like weather or
latest tech news
LOCATION: If provided, the physical location when a query is issued. e.g. Redmond, Washington for a given query
SEARCH RESULTS: Main set results in center of page. Includes various answers (Images, Videos, web results, etc.)

INFO CARD + Topic: Vertical “card” besides the main search results,
with information about a specific topic (an object, person, place,
business, movie, fictional character, etc.)
INTENT: The user’s end goal, often expected results, answers,
actions or information in answer to a query.
Example topics include Angelina Jolie (actress), Paris, France
(place), HP Inc. (business), Forza Motorsport 5 (video game)
User goals/expected results vary in obviousness for outside
observers from quite clear ( Inception movie cast) to very ambiguous
( avs)
 Question 1: How many Search Results match the Info Card topic?
Please scroll to the bottom of the Search Results in answering this question.

All or nearly all

Most/All (1-2 may not match)

Most of top 5 3+ of top 5 match

Few Only 1+ match anywhere

None 0 results match

Most/All All or most Search Results match the Info Card topic

(For example, 1 or 2 results may not match, while the rest do)
Most of top 5 3+ of top 5 Search Results match the Info Card’s topic
Few Fails first two options, but still has 1+ matches in the Search Results.

None The Info Card topic does not match any of the search results
Tips: How do I know if a given Search Result matches the Info Card topic?

Search Result Info Card topic Do they Comments

(Web page, Videos, Images, etc.) match?

✔Yes The blue web link,

green hint text and
snippet clearly
indicate a match.

✔Yes The search result is a

news article about a
few subjects; The
target topic is one of
a few core subjects.

✔Yes The search result is a

news article about a
few subjects; The
target topic is one of
a few core subjects.
❌No The search result is a
news article about a
few core individuals;
This topic is of much
less overall
importance to the
article.

✔Yes All OR majority of the

videos and titles clearly
match the target topic.

✔Yes All OR majority of the

images are of the
topic.
❌No Only a minority of
images are about he
target topic.

✔Yes The topic matches all

of the videos well,
even though they are
performed by various
artists.
❌No The Info Card topic
only slightly matches
the result – it is
overspecifc.

❌No The topic shares a

name with the
subject of the search
result, but they are
different towers.
 Question 2: How well does the Info Card match & satisfy the Query Intent?
This is approximated with two major considerations. Your decision should be informed by the Query, the search results and side searches, as needed.

Matches Query Satisfies Query Intent

😠 Poor / Non-Match A poor, non- or incomplete match with Query OR Satisfies little to none of the Query Intent

A Poor / Non-match either is a strong miss on matching the query OR does little or nothing to satisfy the user intent.

😐 Partial / Ambiguous Match Only partially, indirectly or ambiguously satisfies

Only a partial or ambiguous match with Query OR
Query Intent

A Partial / Ambiguous Match partially or ambiguously matches a Query or only partially satisfies a user’s intent.

😄 Excellent Match An excellent match with Query AND Directly satisfies Query Intent

An Excellent Match matches the query well (despite some established misspellings, aliases, etc.) and directly satisfies the
intent of the user’s query.

Reference Table: Scenarios and How to choose?

NOTE: You do not need to fully read this now. Use this reference where helpful in making judgment calls.

😄 Excellent Match & Value 😐 Partial / Ambiguous Match or 😠 Very poor / Non-Match or
Value Value
Query Type Example queries
Name/Alias queries adele ✔ Adele (Singer) California (US State)
No reasonable interpretation could relate the
for 1 topic Query strongly relates to the famous singer, even
Topic (California) with the query (“Adele”)
without further specification.

 We show the target A book A (The letter)

topic No reasonable interpretation could relate the
Topic (The letter, A) with the query (“A book”)

A Book (By Desi Arnaz)

Based upon search results
ciracha sauce ✔ Sriracha (sauce)
Query is misspelled, but still an alias for the topic
utube ✔ YouTube
Although query seems to be a misspelling, the search
results and side searches quickly confirm that most
users are looking for the extremely well known
“YouTube” and not the uncommon equipment
company.
Red fly agaric ✔ Amanita Muscaria (Fungus)
Query is an alias of the topic, and so directly fulfills
intent, even though initially non-obvious
Shawn Corey Carter ✔ Jay-Z (rapper, businessman)
Query is an alias of the topic, and so directly fulfills
intent, even thdough initially non-obvious
stormblessed ✔ Kaladin (stormlight archive
character)
Query is an alias of the topic, and so directly fulfills
intent, even thdough initially non-obvious
columbine shoorting ✔ Columbine high school massacre
Query is misspelled and incomplete, but still a strong
match for topic name
Al pacino movie in alaska ✔ Insomnia (Movie)
Query directly fulfills the user’s query and intent of
trying to find the movie starring al pacino in Alaska.
john smith the singer ✔ Johnsmith (American Singer)
‘the singer’ specifies that the user is not looking for
the well-known explorer, so this topic directly fulfills
the user intent
Cuckoo calling author ✔ J. K. Rowling
Though J.K. Rowling is a relevant topic, the Info Card
may not make that obvious. All the same, as the
author, J.K. Rowling is the primary intent.
Sauce
Incomplete query match and entirely misses
the user’s intent at learning more about a
specific sauce.
Si Racha (City in Thailand)
Though historically related, the sauce is both
significantly well known and the query
explicitly refers to it. This misses query match
and intent fulfillment.
Universal Tube & Rollform
Equipment Corp
Although this query seems to perfectly match
this equipment company’s website,
utube.com, other results and side searches
quickly disprove this likelihood.
Shawn Carter
Query appears to be a partial match, but
based on web results and well known nature
of Jay-Z, brief investigation would show: this is
a strong mismatch of clear intent for the artist.
Alaska (State), Al Pacino (Actor)
Misses nearly all of the user’s intent to find a
specific movie.
John Smith (Explorer)
‘the singer’ specifies that this is not the well-
known explorer
Queries with Rayburn elementary san ✔ Rayburn Elementary School (School San Antonio, TX
qualifiers antonio in San Antonio, TX) San Antonio is intended to clarify a specific
elementary school – showing the city entirely
“San Antonio” helps to clarify that the user is looking
misses the user’s main intent of finding the
for a very specific school. This is an excellent match.
 We show the target school.
topic, not the
qualifier or a loose Rayburn Elementary School
name match (School in McAllen, TX)
Though the name is a perfect match, the user
has given an explicit location and so the
location in McAllen is a strong mis-match.

Tom brady nfl ✔ Tom brady (Football player) NFL

“nfl” clarifies that the user is explicitly looking for the The user’s intent is to find out about a specific
well-known football player (National Football sports player – NFL is only acting as a clarifier /
league), rather than any other Tom Brady. qualifier here and so does not fulfill the user’s
intent.

Tom Brady (American writer)

“NFL” clarifies that the user is explicitly looking
for a football player. The query, search results
and side searches all make it clear that the
writer is a strong mis-match.

Chicago floord 2013 ✔ Chicago Flood (2013) Chicago flood (1992)

User has explicitly specified a flood taking place in Bad match with the query as the qualifier
2013 – this is a direct fulfillment of the user’s intent. explicitly states a specific year and this Info
Card is about a different year.

Michael collins nlp ✔ Michael collins (Computer science
researcher)
With the help of side searches, the computer science
researcher quickly emerges as being the target of the
phrase “NLP”, referring to a specific field of study.
NLP (Natural Language
Processing, a field of study)
NLP is used as a qualifier here – by itself, an
NLP info card would entirely miss the intent of
trying to find out information about a specific
professor.

Michael Collins (Astronaut)

Side searches, the explicit addition of “NLP”
and the search results make it obvious that the
astronaut has no relationship to the query
intent.
Ambiguous / Unclear srs Savannah River Site airport
(after brief Research) Match with query is obvious, but also ambiguous
(web results make this clear), so it may or may
 No Excellent not fulfill the user intent.
matches, except for
well-known Matt jones Matt Jones (writer)
celebrities, historical Clearly related to Query, but unclear whether
this is the exact desired Info Card, even after
figures, landmarks, investigation. Possibly the user’s intent, but it’s
etc. an ambiguous match.

Tara macantosh Tara MacIntosh (BBA, HBA)

Potentially related to the query, but the
confidence is not strong based on very diverse
web results and additional side searches: there
are multiple other people that the user could be
searching for.

jungle Jungle (Movie), Jungle (geographic

biome)
Some web results match, but it’s unclear from
the web results, query or side searches what the
user is explicitly referring to.

thule board shuttle 811xt Thule board shuttle

Clearly related to Query, but unclear whether
this is the exact desired topic without very
specific understanding, even after investigation.

John smith ✔ John Smith (Explorer) Johnsmith (American Singer)

Even though other people show in the web results, Without additional clarification, most users
the explorer is a well-known historical figure and likely would be searching for the well-known
likely directly fulfills the intent. historical explorer

paris ✔ Paris (city) Paris (song by the chainsmokers)

Even though other subjects exist (songs, other cities, Without additional clarification, most users
etc.), most users likely intend to search for the well- likely would be searching for the well-known
known city. city.
Group or Category of Movies starring tom ✔ Tom Hanks
Topics hanks Tom Hanks is a meaningful parent to the intended
category of subjects (his movies)
 We show close,
meaningful parent
topics
 We don’t show
single, overspecific
members of the
group
Action movies with tom Tom Hanks
hanks Query intent is to browse a specific set of
action movies. Info Card provides little to
fulfillment of this intent and topic is becoming
distant.

Action (film genre)

Partial match with query, but does not fulfill
the user intent to find action movies starring a
particular actor. Over-generic.

Types of eggs ✔ Eggs Bird eggs

A clear match that meaningfully covers the intent of Only distantly relevant, the intent is to learn
different types of eggs. about various kinds of eggs, not just bird eggs
specifically – this Info Card is overspecific.

High heel shoes for sale High Heel shoes

near me Though query is obviously related, the Info
Card unlikely to fulfill the user’s intent to buy
high heel shoes, unless specific meaningful
content is present.

Stiletto Heel
Overspecific class of shoes that the user has
not specified. Will not fulfill users intent to
purchase and query is only distantly related.

stainless steel ice cream Whynter ICM-15LS Ice Cream

maker Maker, Stainless Steel, Black
Query intent is to browse multiple items, but
showing just one overspecific Info Card

Ice cream maker

Info Card fulfills almost none of the intent to
browse stainless steel ice cream makers
Attribute of / Action of tom hanks height Tom hanks
/ Means to reach a Although topic is an exact match, the Info Card is
only a partial match, but is mostly tangential,
Topic providing more information beyond the specific
requested information.

 At best Partial / Brandon Sanderson Brandon Sanderson

Incomplete match books Although topic is an exact match, the Info Card is
only a partial match, but is mostly tangential,
providing more information beyond the specific
requested information.

www.microsoft.com Microsoft
Info Card provides additional info and context,
but user’s primary intent is explicitly to navigate
to microsoft.com so the Task Pane only partially
fulfills intent.

youtube glory to glory Glory to glory (Song by Bethel

bethel Music)
Info Card provides additional info, but user’s
primary intent is to navigate to YouTube to watch
the song.

trailer park boys reddit Trailer park boys

Intent is to navigate to a relevant page on Reddit.
Info Card is a partial match with the intent and
query.

2 UnEqual Intent adele simon konecki ✔ Simon konecki Adele

Topics Adele is a well-known musician, so the intent is most Likely negligibly fulfills intent (to learn more
likely to learn about her lesser known ex-husband about the lesser known Simon Konecki), or not
 We show the topic at all.
with more intent
Creative work + The way of kings brandon ✔ The Way of Kings Brandon Sanderson
Participant/Creator Sanderson Intent is to find out more about the book The Info Card fulfills little to none of the user
intent to find out more about the book.
 We generally show
the Creative Work joe pags show ✔ The Joe Pags Show Joe Pagliarulo
Query about the show (creative work), not the show As the show is specifically named after its
creator creator, it may partially fulfill intent for users that
are looking this up for the first time (for instance,
having just heard about the show). However, it
only partially fulfills intent.

Tom hanks the post ✔ The Post (Movie) Tom Hanks

The core intent is to find out more about the movie Misses the core intent which involves finding
(with Tom Hanks acting as a qualifier) or about his out more about the movie, or Tom Hanks role
role in the movie. In general, information about the in it, but not just generic info about Tom
movie will best satisfy this intent. Hanks.
Creative work + Cinderella stepmother ✔ Lady Tremaine Step-Mother (ballet composition)
Character Directly fulfills the intent to find out the name, Web results make it clear that the overall
background information about a particular character intent is for the character, not a related
 We generally show in a well known film. musical piece. This Info Card will do little to
the character nothing to fulfill intent.

Directions Flights from slc to Tegucigalpa (City) Salt Lake City (City)
(Driving, flying) tegucigalpa Showing destination, about which user is more Showing place of origin
likely to have deeper interest to keep browsing,
We may show the learning more.
destination, but it is
not the primary intent
Find cheap flights slc Salt Lake City (City)
tegucigalpa Showing place of origin

Tegucigalpa (City)
Showing destination, but user intent is more
specifically focused on finding cheap tickets –
the user is likely not interested in generic
information about the city.

Waco tx to anaheim Anaheim, CA (City) Waco, TX (City)

Showing destination Showing place of origin

2 Equal Intent Tampa Bay Buccaneers vs ✔ Tampa Bay Buccaneers AND Tampa Bay Buccaneers
Topics Pittsburgh Steelers Pittsburgh Steelers Showing only one entity, unlikely to fulfill
much of the user’s intent without additional
Showing both entities
searches.
 We show both
entities or neither.
Tom hanks rita wilson ✔ Tom Hanks AND Rita Wilson Rita Wilson
Showing both entities Showing only one entity, is a partial match, but
incomplete match with the query and the Info
Card is unlikely to fulfill much intent.

s20 vs. iphone 11 pro ✔ Samsung Galaxy S20 AND iPhone iPhone 11 Pro
comparison 11 Pro Showing only one entity, unlikely to fulfill
much of the user’s intent without additional
Showing both entities
searches.
Over-specific OR 1967 mercury cougar xr7 Mercury cougar
Info Card is only loosely related to the Query
distant/over-generic and provides little to no satisfaction of the
Topics user’s intent of finding out about a very
specific model and trim of the car.

Yahoo calendars ✔ Yahoo! Calendar (Online software) Yahoo!

A nearly perfect query match that completely fulfills Only loosely related to the desired topic. Very
user’s intent to navigate to or learn more about this unlikely to fulfill the user’s intent or provide
specific software offering. meaningful information.

Calendar
Extremely distant from the user’s intent to go
to or find out more about Yahoo calendar,
with no provided intent satisfaction.

Turmeric tea Tumeric

Incomplete match with query, unlikely to fulfill
the user’s intent to find out more about
turmeric tea specifically (as opposed to
turmeric generally)

Tea
Incomplete, very distant match with query,
highly unlikely to fulfill the user’s intent

Columbo candidate for a ✔ “Candidate for a Crime” (TV show Columbo (TV Show)
crime Episode) Incomplete match with query, provides almost
no satisfaction of intent as user is looking for a
User is looking specifically for an episode of a TV
specific episode. Distant and over-generic
Show. This info card directly fulfills the user’s intent
match.
to find out more about the episode.

Trump Russia sanctions Donald Trump, Russia, Sanction

(Economic policy)
All are incomplete matches with the query,
highly unlikely to fulfill the user’s intent main
intent to learn about trump’s discussed
sanctions against Russia, not just any one
piece.

2019 lincoln navigator 2019 lincoln navigator (Specific car lincoln navigator (car make and
colors year, make, model) model)
Very good match with the query, but the Info Incomplete match with query, highly unlikely
Card only indirectly fulfills the intent of the user. to fulfill the user’s intent
Optional Comments & Submit
After answering Question 1 and Question 2, you may leave optional comments before hitting Submit.

We do appreciate your crisp, specific feedback and try to act on it!

Technical Difficulty / Adult

Please mark a judgment as “Technical Difficulty / Adult” if:
- Technical Difficulty:
o No Info Card is shown
o The hit fails to display or is blank
o You have issues using the HitApp
- Adult:
o The hit is explicitly adult in nature or contains adult content
o If you’re unsure if it’s Adult, but would prefer not to closely investigate, just use Skip.

Thank you!
We appreciate your work! Know that you truly are helping to make Search better for everyone!