Suisun Guidelines

CONTEXT: Goal of this workflow is to label whether a video is language

neutral or not in order to train a model which can predict language
neutrality of a video, and also evaluate such model’s accuracy

Definitions of Labels

Language Neutral
“Language neutral” videos are those which any user can enjoy (i.e. watch the whole or a
significant chunk of the video and engage with it (e.g. like, share)) regardless of what
language(s) they understand. They may have near-zero speech/caption, but can also contain
some speech/caption as long as users don’t need to understand them in order to enjoy the
video.

Examples:
● Videos without any speech/caption
● Videos with some speech/captions, however, understanding them isn’t important for
enjoying the video
● Videos with only speech/captions being the lyrics of the background music (background
music isn’t a focus of a video)

Language Focused
“Language focused” videos are those which users are not likely to enjoy if they don’t understand
the speech/caption language of the video.

Examples:
● News videos
● Speech heavy drama/movie videos
● Variety show videos

Borderline
Videos which aren’t clearly language neutral or focused are defined as “borderline”. Significant
% of users can still enjoy borderline videos even if they don’t understand the speech/caption
language of the video, however, some users don’t enjoy them because they don’t understand
the language.
Examples:
● Song videos (users don’t necessarily need to understand the lyrics of the song in order
to enjoy it, however, it still affects how much users can enjoy it)
● Sports commentary videos (the main focus of these videos is to watch sports, but
listening to the commentary is an important aspect too)
● Prank videos with small amounts of speech/captions (these videos are oftentimes easy
to follow the plot without understanding the speech/captions and can still be pretty funny)

* If there are subtitles and reading them is an important part of the video watching experience, it
needs to be labelled as Language Focused.
* Note that users wouldn’t enjoy videos if they are of low quality or the contents don’t match
users’ preference, however, for the purpose of this labelling exercise you need to assume that
the videos’ quality is high and the contents match users’ preference.

Tooling:
● Left panel (top to bottom):
○ Video title
○ Video title language (model-based prediction)
○ Translation of video title (model-based translation)
○ Video description
○ Video description language (model-based prediction)
○ Translation of video description (model-based translation)
○ Whether the video contains an open (burned-in) caption or not (model-based
prediction)
○ Speech language (model-based prediction) of video
■ This includes the 26 languages mentioned here and also “other language”
and “no speech”
○ Video
● Right panel:
○ Question: Is the video language neutral?
■ Language Neutral
■ Language Focused
■ Borderline
■ Cannot Label

Watch Instructions:
● Skim the title and the description of the video to understand the topic (e.g. drama, news,
song, gaming) of the video.
● You do not need to watch the whole video, spend 60 seconds max per video.
 ● Spot check throughout the video is very helpful to get a good overview. You need to at
least spot check the video at 3 different locations: beginning, middle, and end.

TODO: put a screenshot of the review UI here

Rating Instructions:
You should follow the watch instructions above in order to judge whether users need to
understand a certain language in order to enjoy the video.

A. When to Select “Language Neutral”

● There’s no speech/captions in the video
● There are some speech/captions, however, understanding them isn’t important for
enjoying the video
● Tips: Imagine if you were the creator of the video and is trying to make the video popular
in another country… would you bother to add a subtitle in the local language? If not,
label Language Neutral.

B. When to Select “Language Focused”

● There are speech/captions in the video and understanding them is required in order to
enjoy the video
● Tips: Imagine if you were a user and you didn’t understand the language of the
speech/caption in the video, but you were interested in the topic of the video. If you still
wouldn’t enjoy the video, label Language Focused.

C. When to Select “Borderline”

● There are speech/captions in the video, but understanding them isn’t a hard requirement
in order to enjoy the video
● There would be a significant % of users who don’t understand the language of the
speech/captions in both cases; those who still enjoy the video and those who don’t (i.e.
it’s Language Neutral for some users, Language Focused for others)
● Choose “Borderline” whenever the video isn’t clearly Language Neutral nor Language
Focused

D. When to Select “Cannot Label”

● Video does not load
 ● Any other technical faults which prevents an accurate labelling

Rating Examples
Refer to this spreadsheet for the examples: Language Neutral Labelling Known Questions