UHRS 

Hitapp Name: Entity-aspect association

Hitapp terminology: 
QUERY
Text used to search for information on a search engine such as Bing or Google.
ENTITY
A thing with distinct and independent existence. An entity can exist in real-life or be hypothetical or fictional.
Examples of entities include: Katy Perry (singer), Paris (place), Microsoft (business), Games of Throne (tv
series), Star Wars (movie), Pikachu (fictional character), etc. 
Aspect
For the context of this hitapp, we use the term aspect to refer to a particular part of an Entity or an
exploration path the user might want to do related to the entity.
From the screenshot below, "Whale" is the entity and "lower classifications", "Facts", “Habitat”, …,
"Adaptations", “Reproduction” are aspects of the "Whale" entity.
TASK PANE
The vertical region on the right side of the search results that shows-up the relevant entity for some queries.
SEARCH RESULTS
We are referring to the Bing "search results" for the “Entity-aspect” query. See screenshot below.
 
Hitapp judgement instructions:
You will be shown the search results page for an “entity-aspect” query. Your job is to evaluate whether
the search results help support the aspect in association with the entity. One of the purposes for this
hitapp is to help-us add new insightful entity exploration paths to the Bing users (screenshot below):

Before you judge: 

1. Take time to understand the entity and its aspect. Please click on their respective link to go to
Bing to learn more about it. 
2. Have a look at the search results;
Judgement question:
Question 1: 
Example: “Size” (which is the aspect) is a relevant aspect of “Chihuahua” (which is the entity)
Do the search results support this statement?
Note: you are not expected to look at more than 3 non-advertising search results to answer this
question.
 Yes
 You can find a complete or partial answer mentioning the aspect in relation to its entity.
 The aspect is about the entity, an attribute of the entity or a topic of the entity
o Examples:
 Attribute of an entity
Whale size
Whale weight
 Topic of an entity
Whale habitat
Whale predators
o Look at the screenshot below; start by looking, on the right side, at
which entity the query is referring to, then you can judge the first search
results that highlight that humans are hunting whales and thus are
predators. So the “predators” is a relevant aspect to the “Whale” entity.
 No:
 The aspect is not relevant to this Entity.
 Examples:
Snake spine
 The example below shows that the search results talk about a “snake spine” as
an item in a video game called “old school RuneScape” while the taskpane, on
the right side, clearly show we are talking about a real snake, not a video game.
So the “spine” aspect of the entity “snake” is not relevant here.

Whale eggs
 Whales are mammals and thus do not have eggs.
 You can see on the following screenshot that even though there is a search
result that seems to talk about “Whale egg”, when you carefully read the
description you can find that it’s not talking about Eggs in association with
Whales; instead it’s specifically talking about eggs in Anime, Manga, Comics and
in a video game movie. Furthermore, the taskpane, which is on the right side of
the screenshot, shows that the entity refers to a real-life Whale and not a comic,
anime or video game whale character that could have eggs in a fantasy world.
 The aspect is not mentioned in association with the entity
 Examples:
 Jeff Denham quotes
 The below screenshot shows “Quotes” for “Jeff Dunham” but the entity
mentioned in the right-side taskpane is “Jeff Denham” which is a different
person.

First air history

 In the example below, the taskpane on the right side shows that the entity the
query is referring to is an airline company called “First Air”; so the query “First
Air history” should have search results that talk about the history of that airline.
Unfortunately, when you look carefully at the search results, the search results
talk about the history of the first airplanes or airlines.

 You see a partial search result that does mention the aspect and the entity but you don’t believe
the result is relevant.
 The relevance of the entity-aspect is hard to judge based on the search results.
 Outdated answers
 Example:
 United states current president
 Showing a “Barack Obama” search result would not be relevant here as
he’s not the current president.
 Detrimental or inappropriate content;
 Note: if you feel an entity-aspect search result page has inappropriate content, please
judge “No”; if you feel uncomfortable judging, please use “Skip”.
 Inappropriate aspect or entity or the entity-aspect could lead to inappropriate
content:
o Racist
o Adult, sexually suggestive/explicit, etc.
“naked”
“porn”
“explicit” (context of movies)
o Spam / Scam
o Wild / unsubstantial claim
o Guide the users toward extreme political views
o Violence
 Terrorist or extremist propaganda /activities
1. Suggestions that may steer or help users find
extremist/terrorist/violent content
2. Note: If we do/could return gory results then treat as bad (as
it’s sensitive) ex: ww2 deaths. Else if the suggestion will just
result in informative content then do not mark as bad
(sensitive). Example - WW2 countries, WW2,WW2 Movies,
WW2 Summary
3. Examples:
Benghazi Attack
Holocaust
Insider Attack Afghanistan
Acid Attack Victims
Muslim Attacks in America
Saudi Arabia Supports Terrorism
WW2 deaths
WW2 countries
WW2
WW2 Movies
WW2 Summary
 Other Hate Driven Violence/Intimidation
1. Suggestions that incites/promotes violence by threatening a
person or group and intending to make the target fear for their
safety
2. Suggestions that indicate any violence against individuals or
groups, eg: kkk, Nazi, serial killer.
3. suggestions that may steer users towards knowledge about how
to act violently or that aim to intimidate a certain individual or
group
4. Examples:
burn all Jews
Muslims should be killed
how to take revenge on Israelies
 black beatings video
blacks men attack white
Kill trump family
Chain the Clintons
scalp a person
how to acid attack
Black Men Lynched Video
Black crime in Alabama (also ‘offensive’)
Nancy Kerrigan Attack
 Hurting or killing humans &animals
1. Including self-harm and suicidal content
2. Including Gruesome medical conditions
3. Examples:
How to Hang Yourself Suicide
Die pain free
 Gore / Graphic
1. Intent to look at pictures of dead people, mutilated people, or
disturbing content.
2. Examples:
Greenlake park shooting video
911 attack bodies
o Offensive
 Derogatory/insulting
1. Suggestions that dehumanize, insult, ridicule, intimidate an
individual or group.
Mexicans are crazy
Did Obama Eat Dog
Foolish Democrats
Nancy Pelosi Stupid Quotes
Racist African Jokes
Democrats Are Morons
Republicans Are Hypocrites
Police are evil
Adele is fat
Crazy White people
Ban immigrants
Funny Acting People
Men Jokes
Strangest People Ever
Funny Drunk People
 Defamation
1. Suggestions that defame someone by suggesting/questioning
certain negative associations like dishonest or illegal activities
unless it is a historical fact.
Chris Heisser Realestate Fraud
jerry watson killer
Tommy Robinson Jailed
gerald connolly armed robbery
 Is Hillary Clinton corrupt
Democrats Are Destroying America
Paul Ryan kkk member
2. Example cases where some defamation is not offensive:
Hitler is a dictator
 historical fact
Gary Ridgway serial killer
 Has been considered factual over a decade.
 Biased
1. Suggestions that show known or existing bias, prejudice, or
intolerance, negative stereotypes, or generally emphasize the
negative traits of group and/or encourage feelings or fear or
disgust towards them.
2. Non-neutral suggestions about individuals or groups promoting
or demoting an individual/group
1. Note – “White supremacy” is problematic since it promotes one
group as higher. But "Black lives matter", "Gay pride Month" are
not problematic since they do not promote the group over
others.
2. Examples:
women are emotionally weak
why are blacks poor
Immigrants should not be allowed
crimes by blacks in 2018 (also ‘violence’)
Men are mean
Aryan Brotherhood of Texas
Blacks are targeted by police
o Group examples:
 Promote one group as higher
White supremacy
 Do not promote the group over others:
Black lives matter
Gay pride month
 Sensitive:
1. Suggestions that could trigger painful memories of some
traumatic event, or trigger users in some other negative way
2. Suggestions that could be considered in some way sensitive or
emotionally charged for certain groups due to historic or
cultural, religious, ethnic, nationality reasons.
3. Suggestions about that question, ridicule or endorse religious
matters, ethnic group.
4. Suggestion that may be associated to a negative connotation.
5. Suggestions that are not targeting specific individuals or groups
but contain hate/offensive/abusive intent.
6. Suggestions that may reveal information that might be
construed as sensitive personal information.
 7. Suggestions that are not sexualized but referring to individual
body parts with a dominant intent of insulting the individual
should be treated as offensive.
8. Examples:
Paul Lynde Gay
Neil Patrick Harris Gay Marriage
Benjamin Bratt Daughter Disability
Layne Staley Suicide Note (also ‘violence’)
Robert Muller Jaw
Mussolini Dragged to Death (also ‘violence’)
Hitler is a hero
Islam is cult
Christians are fake
Allah is supreme god
Foolish People Quotes
Famous Funny Idiots
Funny Swear Words
Katy Perry naked
 Profanity
1. Suggestions that include any sort of slurs, expletives, and swear
or curse words
2. Examples
shit with republicans
 Offensive to a large group of people
Ethnic minorities
Gays and Lesbians
Feminists
Christians
o Illegal
1. Illicit topic or facilitates Illicit activities
o Any suggestion that suggests illicit activity/content as according
to federal law. If the query seems odd and possibly leaning
towards illegal content, lean on the more conservative side on
your judgement; you can fail it and label/comment it as
“illegal”. Including:
 Suggestions that may facilitates/ nudge users towards
illicit activities like child porn, drugs, incest, bestiality
 Suggestions that may infringe on intellectual
property/copyright rights or trademark agreements.
o Examples:
Fake Gucci Shirt
Fake passport
Fake Louis Vuitton
video to consume meth
12 you naked
4chan Incest Father and Daughter
How to smoke heroin
how to attack DDoS
 how to smoke weed
Credit Card Codes Free
o Examples of [Video game names here] with added keywords:
1. Should not be treated as problematic:
Mod
Cheats
Cheat sheet
2. Illicit
Hack
o Note: Suggestion containing video games with “mod”, “cheats”
“cheat sheets” should not be treated problematic. But any
containing “hack” should be labeled problematic under the
category of illicit.
o Misleading:
1. suggestions that promote information or associations that are factually
incorrect or factually questionable, or that reinforce or nudge users
towards conspiracy theories
2. Examples:
climate change is a hoax
vaccines are harmful
911 was staged
Obama is born in Africa
Muslims trying to takeover west
o Controversial:
1. Suggestions that appear to endorse one side of a known controversial
topics
2. Examples:
abortion should be illegal
marijuana must be banned
2. Other examples (note that those examples are not applicable to
fictional characters and locations):
“Controversy”
“Scandals”, “romantic interests”
 Romantic interests for fictional characters is ok
 Romantic interests for a real person is risky
“boyfriend”, “sexual orientations”
o Not authoritative or trustworthy
o Fake news/content:
1. If an Entity-aspect surface fake news, please judge it as bad.
2. When you see fact verification domains such as “snopes.com,
trusthorfiction, hoax-slayer, factcheck, politifact, etc.” that gives you a
good clue that the query may be guiding the user to fake news or fake
content. You can click on one of those domains to see whether the
relevant fact checking website judged the content as factual or not.
o Conspiracy theory
o May offend or upset the subject/owner
o Embarrassing
 o The entity-aspect doesn’t appear to be factually correct or appears to be a
rumor
o Profane
 Risky aspect:
 The aspect has a risk to mislead the Bing users to inappropriate contents 
 Examples: 
A “[movie title here] adult rating" usually shows a search results
that mention for whom the movie is appropriate to watch but
sometimes, such query could lead to adult content and movies.
 High-risk of non-existence
 These are aspects that may look factual but could provide the user with
non-factual information.
“sister”
o If a person search for a “[Person] + sister”, it doesn’t
mean that this person has a sister.
"boyfriend", “sexual orientation”
o This is leading the user toward gossip information
which may provide wrong information.
 High risk of non-existence or aspect that don’t make any sense to an entity:
 Examples:
Taylor Swift “children” (she doesn’t have children)
Donald Trump boyfriend (he doesn’t have a boyfriend)
Whale egg (Whale do not come from eggs)
 Branded tabs
 Tab should not include the name of a branded website as we want to give a
broad choice of websites to our users when available.
 Examples:
Having a “Tracker stats” tab for entities “Fornite” or “Destiny” mean we
are sending our users directly to the “Tracker network” website for their
game stats. Instead, the tab should be called “stats” or “profile stats”
which should return a more comprehensive set of stats website that the
user can then chose from.
“Gamestop” for “NBA 2k16” would lead people to buy the game from a
specific retailer instead of giving them a proper choice. “Buy” is a better
tab name for this intent.
“TV Tropes” tab for “Dragon Quest VIII” would bring the users directly
to the tvtropes.com website while “Tropes” can also bring people to
fandom.com for example so it’s better.
“Microsoft store” tab for a video game entity mean the user want to
try/buy/download the game which can happen on many websites
across the web; not just on Microsoft store so we should call this intent
“try” or “buy” or “download” which overall can be summarized as “get”
or “play”.
“For Nintendo”, “For Playstation”, “For Windows” tabs can be branded
because the user is explicitly looking to play the games “for” a specific
system.
 Formatting, spelling, grammatical or wrong-language issues:
  The tab has formatting issues
We don't want "_" (underscore) between words such as
 Grammatical issues
 Spelling mistakes
 Wrong language: The entity and/or the aspect text is unreadable or written in a
language we don’t expect the user to understand for the market
For English markets: having a “Familia” aspect which the Spanish word
for “Family”.
"寿命"  shouldn't show-up as an Aspect in the “en-us” (English United
States) market.

Finally, after answering the question, you will see 3 additional controls:
1- The comment box:

 Please share with us any thoughts, concerns or issues you may have.
2- The submit button:

 Once you are done judging “Yes” or “No” for the question, you can click submit. If you forgot
to answer the question, then the button will not be clickable.
3- The skip button:

 You can use this if for some reason you don’t want to judge this item. For example, if you
don’t feel comfortable judging something for any reasons, please skip.