Apple Music Hints / Top Hits Guidelines

Updated: 10/8/2020

3 l
Introduction

25 tia
In this document we will explain how to evaluate Search Hints (=text hints) and Top Hits (=entity
hints) for Apple Music. You will use the BaseLine tool to make these judgments. The purpose of this
evaluation is to evaluate the relevance of a Hint and a Top Hit result for a given string. We want to
ensure that the Hints and Top Hit suggestions we generate add value to the users. Use your best
judgment when considering the popularity and recency of a suggestion. There are no hard rules or
guidelines around popular suggestions. Take into account the length of the string and how strong
the intent is.

09 n
In the screenshot below you can see the two types of hints you’ll be evaluating:

e 1. Top Hits = entities, which specify the type of result it is, i.e. “Taylor Swift” - Artist, “Only
The Young” - Song by Taylor Swift, “Lover” - Album by Taylor Swift. Top hits will also
have a thumbnail image.
2. Hints = text hints, i.e. “taylor”, “taylor swift”, “taylor swift essentials”.
20 id
nf
Co
 Rating Scale
The grades on the relevance scale are as follows:

Perfect
Good
Acceptable
Unacceptable: Extra Language

3 l
Unacceptable: Spelling
Unacceptable: Other

25 tia
Problem: Other

Relevance Rating
Researching Query Intent:

09 n
We expect you to use search engines such as Google, YouTube etc. to help you understand the
likely intents of the inputs.
If you have an iOS device, we recommend checking out the suggested hints/top hits in Apple Music
for the input you're evaluating for a better understanding of how the product works.
e
Primary and Secondary intent
The primary intent of a query is the most likely intent based on recency, quality and popularity in
the market you’re evaluating i.e. the intent of most users who enter that query on Apple Music.
20 id
A secondary intent, on the other hand, is less likely, less popular and/or less recent intent
compared to a primary one.

Perfect
A suggestion that very likely satisfies a common primary intent of the input, aka query.
nf

Perfect suggestions complete the query, are mainstream popular, add value, and 'make sense' given
the length of the string.

Examples of Perfect hints/top hits:

• t → taylor swift (popular and very likely to satisfy a primary intent)
• tay → taylor swift lover (popular and recent album by a very likely primary intent artist)
• big → big (repeats the input, and is a valid search query. Likely points to Young M.A's song
Co

"BIG" (49M views on Youtube, 8 months ago))

• big → big sean (popular and very likely to satisfy a primary intent)
• bey → beyoncé (popular and very likely to satisfy a primary intent)
• bey → Beyoncé essentials (playlist)
• pop → A-list pop (playlist)
• pink → p!nk (the artist)
• pink → pink floyd
• billie → billie eilish (a very likely primary intent)
• bt → bts
• dra → drake
• 25 → 25 by Adele (a likely primary intent)
• 19 + → 19 You + Me by Dan + Shay (the most likely intent of this input)
• yummy → yummy justin bieber (popular and very likely to satisfy a primary intent)
• fro → frozen
• fro → frozen 2
• fro → frozen soundtrack
• sico → sicko mode (Sicko Mode by Travis Scott is a likely primary intent given that the query
is very likely misspelled)
 • canib → cannibal corpse (the band Cannibal Corpse is a likely primary intent given that the
query is misspelled)
• canib → cannibal (the song Cannibal by Kesha is a likely primary intent given that the query
is misspelled)
• shau → shaun frank (a very likely primary intent of this query)

Good

3 l
A suggestion that likely satisfies a secondary intent of the input. These suggestions are less recent,
less popular and/or slightly specific, for example a popular cover of the primary intent song.

25 tia
Less popular/less likely
Less popular suggestions might be “billie jean“ for the query “billie“. The primary intent is likely Billie
Eilish. Use your best judgment when considering the popularity of a suggestion and evaluating how
strong the intent is.

Slightly Specific
Some suggestions start veering into specificity. For example, if your query is “cou” and the
suggestion is “Country love songs playlist 2010”. While it fits the basic intent of the input, it is slightly

09 n
too specific for the input “cou”.

Examples of Good hints/top hits:

• tay → taylor swift change (a decade old and lesser known song by Taylor Swift; likely to only
e satisfy a secondary intent of the query. Note that most people are looking for the most
recent and/or popular songs by the artist.)
• b → Big Moochie Grape (a less popular artist)
• big → Biggie Smalls (this is a nickname for The Notorious B.I.G → slightly too specific).
20 id
• big → Big Green Tractor by Jason Aldean (this song is less recent, from 2009, and likely to
satisfy a secondary intent)
• billie → billie jean (a likely secondary intent in today’s market where Billie Eilish is the
dominant intent)
• billie → billie holiday (a likely secondary intent in today’s market where Billie Eilish is the
dominant intent))
nf

• 25 → 25 by BOL4 (Less popular. The primary intent is Adele’s album 25)

• 19 + → 19 You + Me (feat. Peter Hollens) by Home Free (a popular cover of the intended
song)
• dont → don't touch my truck skrrt, skrrt (this is a hint pointing to the memorable lyrics of a
TikTok viral song called My Truck by BRELAND)
• yummy → Justin Bieber - Yummy (Music Video) (the music video is slightly too specific a
result, and likely to satisfy a secondary intent)
Co

• shau → shawn mendes (this artist is very popular and the suggestion could satisfy a
secondary intent if the user misspelled the query)
• 6 → ’60s Hits Essentials (a playlist featuring music from the '60s is likely to only satisfy a
secondary intent of the query. It's more likely that the users are looking for highly popular
and recent content matching the query such as the artists 6ix9ine, and 6lack)
• we → the weekend (This suggestion points to the song The Weekend by SZA from 2017,
and is likely to satisfy a secondary intent considering the partial match and the length of
the query. This suggestion is less recent and less popular compared to likely primary
intents The Weeknd, and the song We Paid by Lil Baby & 42 Dugg.
• goo → 6ix9ine (the artist) (The song GOOBA by 6ix9ine would be rated Perfect for this
query, the artist page for 6ix9ine should be rated as Good.)
• jada → Jaydayoungan (this artist is quite popular and the suggestion could satisfy a
secondary intent if the user misspelled the query. More likely intents of this query would
be Jada Kingdom or Jadakiss.)
 Acceptable
A suggestion that may be incomplete, too vague, too specific (and often also unpopular), or points to
a piece of content that does not exist in Apple Music at this time. Also unpopular covers of the
intended song.

Too Specific/Unpopular
Suggestions that are unpopular, incomplete, or too specific.

3 l
Also, suggestions that have a lyrics match that are not very memorable/likely to be the intent of the
query.

25 tia
Examples of Acceptable hints/top hits:
• bey → bey (the hint is likely incomplete and leads to an unpopular artist called Bey. The
primary intent is Beyoncé)
• 25 → 25 blues traveler album (Unpopular. The primary intent is Adele’s album 25)
• t → t (the hint is likely incomplete and leads to an unpopular artist named “t”. This is a very
unlikely intent.)
• big → little big town (Popular country band, but it's very unlikely that the user would look for
this artist with the query "big".)

09 n• red → lady in red

• 19 + → 19 You + Me by Twinkle Twinkle Little Rock Star (unpopular cover of the intended
e song)
• yummy → Screaming Yummy by Yummy (Unpopular and unknown, likely to only poorly
satisfy a secondary intent.)
• pop → BLACKOUT by Lil Kapow Matching Lyrics: Pop a Perc' and I blackout, f**k it, I'm
blowing (The lyrics contain the search term “pop” but this is not a particularly memorable
part of the lyrics, nor is it likely that the user would query for this particular song with this
20 id
query, hence the rating of Acceptable.)
• dua → dua lipa madonna (This is an incomplete suggestion, and hence rated Acceptable. A
Perfect suggestion would be "dua lipa levitating", which is the song ft. Madonna.)
• grea → grateful dead (the artist) (There is a slight possibility that the query is misspelled and
the user intended to look for the artist Grateful Dead. It is much more likely that the user
intent is for LANCO's song Greatest Love Story, or the The Greatest Showman
nf

soundtrack.)
• plug → plug kodak (The suggestion is incomplete. The complete suggestion would be 'plug
kodak black'.)
• nle␣ → nle choppa walk em (The suggestion is incomplete. The complete suggestion would
be 'nle choppa walk em down'.)
Co

Unacceptable: Extra Language

A suggestion that includes extra language that is obvious or redundant. For example, if your query is
‘rave’, a suggestion for “rave music”' contains extra language.

• rave → rave music (in the context of Apple Music, it’s obvious that the user is looking
for music, and not rave events, etc.)

Unacceptable: Spelling
A suggestion that contains spelling errors, spacing errors, punctuation errors, etc. Please note that
capitalization does not matter in the context of this evaluation.
Should you be unsure if a suggestion should fall into this category, it is better to be more critical and
use this rating than not.

Examples of Unacceptable: Spelling hints:

• 19+ → 19+ (this hint does not match any song/artist/album title. The query is likely
misspelled and points to the song "19 You + Me".)
 • bli → blink 182 (should be "blink-182")
• u → u-2 (should be "u2")
• mu → mussic (correct spelling: "music")
• rih → rihana (correct spelling: "rihanna")
• jhe → jhené aiko bs (Artist name is spelled correctly, but the song title should be spelled as
"b.s.")
• tik␣ → tik tok playlist (correct spelling is "tiktok playlist")
• bey → beyonce (A missing accent is considered a misspelling. Correct spelling: beyoncé.)

3 l
25 tia
Unacceptable: Other

Examples of Unacceptable: Other hints:

• nba → one shot (The most likely primary intent is the artist YoungBoy Never Broke Again
aka "nba". The hint is a song by the artist, but the connection between the query and the
suggestion is not clear to the user, and hence this suggestion will be rated as
Unacceptable: Other.)

09 n• post → take what you want (Likely primary intent is the artist Post Malone. The hint is a song
by the artist, but the connection between the query and the suggestion is not clear to the
user, and hence this suggestion will be rated as Unacceptable: Other.)
• taylor → Taylor Swift 1980 (this is a hint for a non-existent song/album)
• car → scars (Irrelevant hint)
e
Problem: Other
20 id
Use this rating only if there is a problem with the task itself in BaseLine. For example, there's no
output to judge. If such cases should exist, contact your support team immediately.

Rating Non-English Queries

nf

Please note that Apple Music users query for content in multiple languages, not only in English.
All queries in all languages are valid search queries, and the results should be rated based on
relevance, popularity, recency, and how strong the intent is. Utilize translation tools and conduct
thorough side searches to understand likely user intent.

• echa → Échame La Culpa (Perfect; a highly popular and recent song by Luis Fonsi and
Demi Lovato, very likely to satisfy a primary intent of the user)
Co

• echa → Échame a Mi la Culpa (Good, this is a song from 2005 by the famous singer Julio
Iglesias, likely to satisfy a secondary intent of the query)
• echa → Échame La Culpa by Reggaeton Acústico (Acceptable; an unpopular remix/cover of
the intended song)