Please ensure that you are using the latest version of the guidelines from BaseLine, which

are found in the upper right corner of every task.

Guidelines for Search - Podcast Hints

25 l
65 tia
Table of Contents
Please ensure that you are using the latest version of the guidelines from BaseLine, which are found in the upper
right corner of every task. ............................................................................................................................................. 1

Guidelines for Search - Podcast Hints ........................................................................... 1

Introduction ................................................................................................................................................................ 2

37 n
What is a search hint? ............................................................................................................................................. 3
Hint types ................................................................................................................................................................. 3
e
The purpose of podcast search hints evaluation ............................................................................................. 4
Mandatory Comment.............................................................................................................................................. 4
Rating Scale ................................................................................................................................................................ 5
14 id
Ratings Overview ...................................................................................................................................................... 5

Examples of different ratings................................................................................................................................. 7

Perfect ...................................................................................................................................................................... 7

Good ......................................................................................................................................................................... 8
nf

Acceptable ............................................................................................................................................................... 9

Unacceptable: Spelling ........................................................................................................................................ 10

Unacceptable: Other ............................................................................................................................................ 10

Unacceptable: Extra Language [Removed] ..................................................................................................... 11

Co

Problem: Other ...................................................................................................................................................... 11

 Update:

02/14/23 - Add Unacceptable: Extra Language rating guidance.

25 l
Introduction

65 tia
In this document, we will explain how to evaluate podcast search hints. You will use
the BaseLine tool to make these judgments.

37 ne
14 id
nf
Co

Podcasts Hints on iOS

 25 l
65 tia
What is a search hint?
A hint is a suggestion that the search engine is offering to the user as they start typing their query to
help the user submit a query to the search faster.
The hint is not a search result, it is a query by itself.

Examples:
• Input: 'ser' → Hint: 'serial'
• Input: 'npr' → Hint: 'npr news now'

37 n• Input: 'the' → Hint: 'the daily'

• Input: 'sar' → Hint: 'sarah silverman'
e
Hint types
For podcast search we offer several types of hints:
14 id
Content title
A search hint that suggests a specific podcast show.
• Input: 'mur' → Hint: 'murder & magnolias'
• Input: 'cr' → Hint: 'crime junkie'
nf

A search hint that suggests a channel.

• Input: 'ted' → Hint: 'ted audio collective'
• Input: 'slate' → Hint: 'slate podcasts'

A search hint that suggests a specific podcast episode.

• Input: ‘jordan peterson huber’ → Hint: ‘andrew huberman jordan peterson'
Co

• Input: 'lex joe rogan' → Hint: 'lex fridman joe rogan'

Person
A search hint that suggests a person. It could be a podcast host or another person.
• Input: 'jo' → Hint: 'jon stewart'
• Input: 'anna' → Hint: 'anna faris'

Other hint types

A search hint that suggests podcast-related terms like genre, provider, category, topic, etc.
• Input: 'co' → Hint: 'comedy'
 • Input: 'te' → Hint: 'team coco'
• Input: ‘cent’ → Hint: ‘comedy central’

A search hint that suggests a combination of podcast-related terms like: channel + genre, channel +
content type, genre + content type, channel + genre + content type.

25 l
• Input: 'wond' → Hint: 'wondery true crime'
• Input: 'ki' → Hint: 'kids stories'

65 tia
The purpose of podcast search hints evaluation
The purpose of this evaluation is to evaluate the relevance of a search hint for a given input.
We want to ensure that the hints add value to the users in their search journey and the hint is
relevant to the input given by the user.
A search hint should help the user find a podcast, podcast channel, episode, host, or any other

37 n
podcast-related content.

Use your best judgment and try to assess if the hint can be used as the primary or secondary intent
for the given input, and consider the popularity, relevancy, and recency of a suggestion. Some
inputs might be more ambiguous than others (like inputs of only a few letters) and therefore have a
e
bigger number of matching relevant hints. Take into account the length of the string and how strong
the intent is.
For example, the input ‘p’ has many more possible hints compared to the input ‘pard’, and ‘t’ has
many more possible hints than ‘tre’. You should adjust your popularity/recency judgment
14 id
according to the set of content which is available given an input.

To help you find out if a hint has podcast context and can be the primary or secondary intent of the
input some links on the task page will point you to popular search websites (e.g., Google, Spotify). It
is always good to check more than one source when deciding on the rating. Other good resources
would be Apple Podcasts, YouTube, Podchaser, Listennotes, Soundcloud, Stitcher, TuneIn, etc.
nf

Mandatory Comment
Each rating must be explained in the comment box, even if “optional” is indicated
— rating comments are always mandatory. The comment should be concise and must only
Co

explain why the rating chosen is the correct one in application of the guidelines.
 Rating Scale
The grades on the relevance scale are as follows:
Perfect
Good

25 l
Acceptable
Unacceptable: Spelling

65 tia
Unacceptable: Other
Problem: Other

When choosing your rating take into consideration the following:

Relevance to the input
Recency of the content hinted
Popularity of the content hinted

37 n
Ratings Overview

e
Primary and Secondary Intent
The primary intent of a query is based primarily on relevance, and you should place strong
emphasis on the recency and popularity of the content you are evaluating.
A secondary intent, on the other hand, may be less relevant, less popular and/or less recent intent
14 id
that is relevant to the input. This can be different than the primary intent, e.g., if the primary intent is
clearly a podcast, but the same query could have a secondary intent for example for person
category.

Relevance
nf

Relevance captures the relationship between the literal text of the query and the hint.
Please consider how you would search for content as a user of podcasts. Typically, a user’s query
matches the start of a title, but not always. Hint is only relevant for notable/memorable parts of the
title, how someone would refer to the content. Not all secondary titles (after colons), or random
matching keyword to middle of string are considered relevant.
For example, a user may start their query with ‘spo’ when the intent is ‘Snap Judgement Present:
Spooked.’
Co

Input: ‘obam’ → Hint: ‘the michelle obama podcast’

Popularity and Recency

Please consider popularity and recency in relation to other hints possible for a given
query. Popularity of content should be evaluated in the context of today — typically newer content is
more popular today.
However, you may encounter hints that point to an old memorable and popular podcast episode. In
this case, the hint might fully satisfy the intent.
The popularity of very new content will often be somewhat unclear or appear unpopular (as it is a
new content with growing popularity), specially when the hint points to an episode. Please consider
some of following factors when considering popularity of new content.
• Is the content part of a popular/legit channel/podcast show?
• Is the content well-known?
• How popular is other content produced by the same channel/provider?
 • Consider how popular a guest (in the case of an episode) is. For instance, content with Elon
Musk will be popular as he is mainstream popular.

Example:
• Input: ‘lex elon’ → Hint: ‘lex fridman elon musk’

25 l
The hint points to an episode from Lex Fridman Podcast: ‘#252 – Elon Musk: SpaceX, Mars, Tesla

65 tia
Autopilot, Self-Driving, Robotics, and AI’. That would be a typical example of popular and dated
content. As you can see from the screenshot below, the episode aired in December 2021. Also,
there is no popularity data available. However, a quick search on Google would return the YouTube
(video cast) version of the episode, which has received over 6.5 million views since its release,
hence very popular.

37 ne
14 id
Podcasts - Lex Fridman Episode
nf

Market
Local market should be considered in ratings, a hint may point to something that is not that popular
Co

worldwide but very popular in the local market. Leverage your understanding of the language to
identify misspellings, translations, or aliases related to the query.
• For example, for the query ‘flo’, the hint ‘floorball’ would likely be Acceptable in the US
storefront as the sport is less popular, however, in the Swedish storefront it would be
Perfect as the sport is popular and users would want related content.

Other Languages
Please note that Apple Podcasts users query for content in multiple languages.
All queries in all languages are valid search queries, and the results should be rated based on
relevance, popularity, recency, and how strong the intent is. Utilize translation tools and conduct
thorough side searches to understand likely user intent.
 Articles
You may choose to ignore ‘the’ at the start of the hint. For example, the input query ‘day’ is highly
relevant to the hint ‘the daily’.
However, when the query input is like ‘t’ , ‘th’, ‘the’, ‘a’, etc., this is important to determine relevance.

25 l
Misspelled query

65 tia
When rating, take into consideration that the search query might be misspelled. In cases where the
intent of the query is clear also if the query is misspelled, a suggestion that points to the
primary/secondary intent content should be rated based on the guidelines.
• For example, for the search query ‘tevo’, the correctly spelled suggestion ‘trevor noah’
should be rated as Perfect.
• Similarly for the query ‘rhe’, hints starting with ‘the’ could be considered as the primary intent

37 n
Examples of different ratings
e
Perfect
A suggestion that very likely satisfies a common primary intent of the input.
14 id
Perfect suggestions complete the query, are mainstream popular, add value, and “make sense”
given the length of the string.

Keep in mind that incomplete queries can have more than one possible primary intent. All the
hints that are complete and/or are strongly related to the query and that point to possible primary
intents should be rated as Perfect.
nf

• For example, for the query ‘lex’, possible hints that should be rated as Perfect are for
example ‘lex fridman’ and ‘@lexie’ that are both recent and popular and would both satisfy
the primary intent for multiple users.

Examples of Perfect hints:

• ‘adam grant’ → ‘worklife with adam grant’: The hint points to a popular podcast show hosted
Co

by Adam Grant.
• ‘all’ → ‘all-in with chamath, jason, sacks & friedberg’ : The hint points to a popular and well-
know Podcast show.
• ‘better’ → ‘you better you bet’ : The hint points to a popular Podcast show that satisfies the
primary intent.
• ‘brain health journey’ →‘the brain health journey podcast’: The input is very long and there
are not a lot of matching results, would most likely satisfy the primary intent. Since the
‘podcast’ is part of the name, it should not be considered to include extra language.
• ‘cr’ → ‘crime’ : The hint surfaces a variety of crime type of content.
• ‘cr’ → ‘crime junkie’ : The hint points to a popular and well-known Podcast Show: Crime
Junkie.
• ‘gle’ → ‘glennon doyle’: Person results for a popular author with a popular podcast.
• ‘lex’ → ‘@lexie’ : The hint points to a Podcast Show that is popular and it would satisfy the
intent given the length of the input.
• ‘lex f’ → ‘lex fridman’ : The hint points to a popular Podcast Host and it would satisfy the
primary intent of the input.
 • ‘lex frid’ → ‘lex fridman podcast’ : The hint points to a popular Podcast Show and it would
satisfy the primary intent.
• ‘lex fridman dan’ → ‘lex fridman dan carlin’: The hint would satisfy the primarily intent.
• ‘n’ → ‘npr’ : The hint points to a well-known provider and it would satisfy the primary intent.
• ‘ser’ → ‘serial’: The hint points to a very popular podcast show.

25 l
You may encounter cases where the hint is identical to the input, and complete. The rating for these

65 tia
cases, where the input/hint is directing to a clear primary intent should be Perfect.
• 'dateline' → 'dateline': Popular and very likely to satisfy a primary intent.
• ‘bbc podcasts’ → ‘bbc podcasts’: Hint points to a well-know channel that is identical to the
search query and likely to satisfy the primary intent.

Good

37 n
A suggestion that is relevant and likely satisfies a secondary intent of the input.

A secondary intent is less likely intent compared to a primary one. A secondary intent hint could be:
• Less popular: Hint that points to lesser known content.
e o ‘the daily’ → ‘the daily dad’: The primary intent of the query is the podcast show
‘the daily’ by The New York Times. The suggestion points to a less popular
content therefore it might satisfy a secondary intent.
o ‘se’ → ‘self-improvement daily’: The primary intent is likely the podcast ‘serial’’,
14 id
or the topic ‘serial killer’, and this hint points to a less popular podcast.
• Less recent: Hints that points to a less recent content.
o ‘al’ → ‘alone: a love story’: The primary intents are likely the more recent
podcasts, such as ‘all there is with anderson cooper’, ‘all-in with chamath, jason,
sacks & friedberg’, and ‘the ben and ashley | almost famous podcast’, and the
hint points to the less recent podcast ‘Alone: A Love Story’ that is popular, but
nf

has not been updated since 2019.

• Slightly Specific: Hint that points to content that is relevant to a smaller group of users than
for the primary intent or that restricts the intent.
o A slightly specific suggestion might be ‘audio drama spanish’ for the query is
‘au’, where one of the primary intents could be ‘audio drama’, and ‘audio drama
spanish’ would define the hint to be slightly more specific.
Co

Examples of Good hints:

• ‘au’ → ‘audio drama spanish’: The suggestion points to a specific content provider for the
‘audio drama’ genre that may not be applicable for all users, but would still be popular in
the US market.
• ‘bbc’ → ‘bbc football’: The hint is slightly specific and it would satisfy a secondary intent.
• ‘cr’ → ‘crash dummies podcast with pat and mike’: This is recent and somewhat popular
podcast. However, there would be more popular content available for this ambiguous
query, hence this is rated ‘Good’.
• ‘espn’→ ‘espn 1000’: The hint points to podcast show relevant to Chicago. It is popular but
slightly specific given the length go the input, hence it satisfies a secondary intent.
• ‘gle’ → ‘glee’: Popular TV show topic that is still popular, not as recent. There are new
episodes in podcasts for this topic, hence this would be likely to satisfy a secondary intent.
• ‘hotel’ → ‘the hotel florist’: Relevant to the query, the article can be ignored. Recent (2021-
2022), not that popular.
 • ‘lex’ → ‘lex fridman dan carli’ : The hint points to a popular episode from Lex Fridman
Podcast. Given the length of the input, it would satisfy a secondary intent.
• ‘lex f’ → ‘lex fridman elon musk’: The hint points to a popular episode from Lex Fridman
Podcast. Given the length of the input, it would satisfy a secondary intent.
• ‘ser’ → ‘serial chillers podcast’: The podcast is recent, but not that popular.

25 l
65 tia
Acceptable
A suggestion that may be incomplete, too vague, too specific, or unpopular, or it doesn't have a
strong relevance to the input query.
• Poorly related: A suggestion is consider poorly related to the search query if it points to a
content that unlikely users would look for with that search query.
o ‘th’ → ‘ain't that the tooth’: It is unlikely that the users would search this content
with ‘th’. More likely the user was looking for podcasts shows such as ‘this
american life’.

37 ne• Incomplete: A suggestion is considered incomplete when it does not contain the full content
title/name.
o ‘hub’ → ‘huber’: The title is incomplete, the complete hint would be ‘huberman
lab.’
• Unpopular: A suggestion that is related to the search query but points to an unpopular piece
of content should be rated as Acceptable. Unpopular content can be an unpopular
podcast, channel, genre, host/person etc.
o ‘the’ → ‘there's something about mary carey’’: In this case, there would be
14 id
multiple more popular intents for the query, such as ‘the daily’, ‘game theory
podcast’, ‘the joe rogan experience’, etc.
• Too vague/too specific: A suggestion that is overly vague or specific.
o ‘e’ → ‘episode’: The query is overly vague, the intent would likely be podcasts
like ‘the economist’, or popular channels such as ‘espn’.
o ‘au’ → ‘audio drama by pulp audio’: The suggestion would be overly specific for
nf

a query ‘au’, when less specific suggestions for ‘audio drama’ or ‘audiochuck’
would be more popular suggestions.

Examples of Acceptable hints:

• ‘art’ → ‘arthur’: Several unpopular podcasts available with this name, that do not have many
episodes published, and/or have not been updated recently.
Co

• ‘cr’ → ‘the crown: podcast lilibet’: Not that popular, has not been updated recently. Multiple
other more popular content available for the input ‘cr’ available.
• ‘book’ → ‘the sleepy bookshelf’: Low relevance to the query, matching only the second word
of the title.
• ‘diam’ → ‘diamond dreams baseball performance podcast’: Not recent (2014-2015), not
popular.
• ‘espn’ → ‘espn australia/nz’: This is too specific/unpopular for US market. Better and more
popular content would be available for this input. Note: The rating would be different for
this in a more relevant market.
• ‘hotel’ → ‘the hotel marketing podcast’: Relevant to the query, not recent (2008-2015), not
popular.
• ‘lex’ → ‘lex fridman podcast | 5 minute podcast summaries’: The hint points to a very specific
podcast show and it would poorly satisfy a secondary intent.
• ‘lex’ → ‘lex frid’: The hint is incomplete, and the intent is for the podcast host, Lex Fridman.
• 'mur' → 'murder and magnolia': The intended content would be 'murder and magnolias', the
query is incomplete and should be rated Acceptable.
 • ‘p’→ ‘podcast’: The hint is too vague.
• ‘ser’ → ‘the serials on radio’: Not recent (2009-2013).
• ‘t’ → ‘theology’: The hint is too specific given the length of the input.
• ‘trev’ → ‘babysitting trevor with carl donnelly, chris’: Hint is incomplete and hence acceptable,
the full title would be ‘babysitting trevor with carl donnelly, chris martin and trevor crook.’

25 l
• ‘2’ → ‘2 girls 1 ghost’: Although the correct spelling is ’two girls one ghost’ (special
characters instead of written words,) the hint merits an Acceptable rating as we are

65 tia
returning the number version of the show’s title.
• ‘wine’ → ‘wine and crime’: Although the correct spelling is ‘wine & crime’ (special character
is missing,) the hint merits an Acceptable rating as the change from &→and, is an
acceptable behavior.

Unacceptable: Spelling

37 n
A suggestion that contains spelling errors, spacing errors, punctuation errors, etc. Please note that
capitalization does not matter in the context of this evaluation. Note, hints that are misspelled and
incomplete should be rated as Unacceptable: Spelling too.
Should you be unsure if a suggestion should fall into this category, it is better to be more critical and
use this rating than not.
e
Examples:

• ‘lex frid’→ ‘lex friedman’: The correct spelling is ‘lex fridman’ (misspelling).
14 id
• ‘osc’ → ‘oscar pellus’: The correct spelling is ‘òscar pellus’ (accent missing).
• 'con' → 'conan obrien': The correct spelling is 'conan o'brien' (special character is missing).
• ‘dr’ → ‘dr phil’: The correct spelling is ‘dr. phil’ (special character is missing).
• ‘obam’ → ‘the michele obama’ ( misspelled and incomplete)
nf

Unacceptable: Other
This rating should be used for any other issue you may encounter where the output is unsuitable for
the input. For example, when the output is not relevant to the query, points to non-existent content,
contains inappropriate adult content not explicitly indicated in the input, or the content is
not podcast-related.
Co

For example:

• Output is irrelevant to the input

o ‘stock market’ → ‘marketplace’: There is a partial keyword matching, however
the intent is clear and the hint would not satisfy the intent.
o 't' → 'what to expect' - The hint is not relevant to the query. User would not
search this content with the preposition starting with 't', but most likely with a
query starting with 'w'.
o ‘coun’ → ‘country doctor clinic’: The hint points to non existent podcast content,
not relevant to podcasts.

Adult Content
We assume that all adult or pornographic content is Off-Topic for non-adult and non-porn queries.
For example, any adult or porn content is should be rated as Unacceptable: Other for the query
[popular podcasts] because the user has not expressly indicated that they want content of this type.
On the other hand, if the query is [erotica] or [porn], any adult or porn content is relevant.
 Unacceptable: Extra Language [Removed]

25 l
NOTE: This rating option should not be used for this evaluation even though the option is visible in
the rating scale.

65 tia
Problem: Other
Use this rating only if there is a problem with the task itself in BaseLine. For example, there's no
output to judge.

37 ne
14 id
BaseLine unavailable
content

Problem: Other ratings should not be used if the content is unavailable in Apple Podcasts but it is
nf

still possible to determine relevance between input and output based on the information provided in
BaseLine, and by completing a side search on Google and/or on other websites.
Co