Netflix - Hulu Usability Testing White Paper

Netflix and Hulu Mobile UX Testing
By Brandon Garfield
Netflix/Hulu Mobile UX Testing 2
Table of Contents
Introduction…………………………………….………………………………………….3
Methodology ……………………………………………………………………4
Test Environment………………….……………..………………..4
Task List……………………………..…...…………………………….4
Test Metrics………..…………………………………………………5
Results………………………………………………………………………………………6
Task #1………………………………………………………………………………6
Task #2………………………………………………………………………………7
Task #3………………………………………………………………………………8
Task #4……………………………………………………………………………10
Task #5……………………………………………………………………………11
Task #6……………………………………………………………………………12
Task #7……………………………………………………………………………13
Task #8……………………………………………………………………………14
Task #9……………………………………………………………………………15
Task #10.…………………………………………………………………………16
Recommendations……………………………………………………………………17
Raw Data………………………….………………………………………………………18
Introduction
This product usability test is being conducted on two popular
streaming service’s mobile apps. As handheld electronics are
advancing, more users of these services access them through their
handheld devices. With a significantly smaller screen size on mobile
devices in comparison to laptop computer or television screen, mobile
applications provide an experience that is different for users. I am
testing these apps having never used the mobile versions before
conducting these tests. The selected tests are intended to demonstrate
the regular functions commonly used for the services on a larger
screen.
Methodology
Test Environment
All tests were conducted using my personal Moto G5SPlus running
Android 8.1.0. The latest versions of both the Netflix and Hulu mobile
applications were installed on November 6, 2019 from the Google Play
Store.
Displayed versions of these applications may vary based on differing
phone operating systems ie: Android compared to Apple.
Tasks Used For Evaluation

Task 1: Find a known movie
Task 2: Find a known television show
Task 3: Navigate forward or backwards mid show
Task 4: Navigate to another Movie
Task 5: Select next television show episode
Task 6: Display subtitles for movie or television show
Task 7: Change audio and subtitle language of movie or television
show
Task 8: Add movie or television show to personalized list
Task 9: Locate movie or television show within personalized list
Task 10: Remove movie or television show from personalized list
Test Metrics
All tasks are evaluated using the categories listed below on a 1-5
Likert scale with 1 being Strongly Disagree, 2 being Disagree, 3 being
Neutral, 4 being Agree and 5 being Strongly Agree.
Effective – I was able to complete this task.

Efficient – I was able to complete this task in a reasonable amount
of time
Engaging – The function was easy to find and use
Error Tolerant- I did not need to close the app or return to the
opening page to correct an error
Easy to Learn – The layout felt familiar and consistent with other
mobile video formatting
Results
All tasks are given a score out of twenty-five possible
points and compared with the other apps performance on the
given task. Higher points awarded is considered good. Please
note for the pictures used below that the Hulu streaming
service would not allow me to take screenshots of certain
functions.
Task 1: Find a known movie
__24/25_
__23/25_
Both services (Hulu pictured
left, Netflix pictured right)
scored high marks for being
able to find a known movie.
Both services had readily
accessible search bars but
the first view of shows and
movies may keep consumers
from finding what they want
faster if they opt not to use
the search function and scroll
through the various options presented.
Task 2: Find a known television show
__25/25_
__25/25_
Both Hulu (left) and Netflix (right) have a wide

variety of easily recognizable shows. This
makes finding a known show easy. For this
test I used the search function as I knew the
shows that I would be searching for. This test
was influenced by my previous experience
with searching for a movie. After my initial
exposure, using the search functions was easy
to do leaving both apps
with a perfect score.
Task 3: Navigate forward or backwards mid video
__25/25_
__24/25_
Hulu and Netflix

provided the option
to move forward or
back by ten seconds.
The pictured screen
could be accessed by
tapping on the
screen and then
move forward or back with a tap on either button. For a
dragging function of moving forward or back Hulu has
something a little different but is effective. Although not
pictured it does function the same, but by changing the whole
screen and not just a small window like the Netflix app pictured
below.
Task 4: Navigate to another Movie
__24/25_
__25/25_
Both applications made it easy to

navigate to another movie by including
suggestions below the movie being
played. Netflix (left) was marked down
because as soon as you play a video it
immediately changes to full screen
adding an additional
step of exiting full
screen and then
scrolling in order to
find another movie to
play. Both services
offered auto-play.
Task 5: Select next television show episode
__25/25_
__25/25_
Both services listed the next episodes

lower on the page of the show being
streamed and the addition of a “next
episode” button available after tapping
the screen. Hulu not pictured.
Task 6: Display subtitles for movie or television show
__25/25_
__19/25_
Netflix (pictured above) scored significantly higher than Hulu

for being able to add subtitles while streaming. The button was
easily located by it’s label by tapping on the screen. Navigating
to the subtitles was not nearly as intuitive for the Hulu app (not
pictured). The unlabeled gear in the top right lead to a menu
with “accessibility settings”, I discovered that this was the
subtitles and this was misleading and made it harder to find
and use for me.
Task 7: Change audio and subtitle language of movie or

television show
__25/25_
__5/25_(lowest possible score)
Not pictured is the hulu app which prevented me from taking

screen shots at this particular point. Netflix was easy and
intuitive with the audio setting being clearly labeled. I was
unable to change the audio settings for Hulu and was unable to
find this function meaning it received the lowest possible score
on the Likert scale.
Task 8: Add movie or television show to personalized list
__22/25_
__22/25_
For both Netflix and Hulu the add to

my list or my stuff was on the landing
page for the television or movies. If
you were already watching something
it required you to return to the home
screen and re-
search the show in
order to arrive back
at the landing page
and add the show
to your
personalized list.
Task 9: Locate movie or television show within personalized

list
__22/25_
__25/25_
Hulu (left) made it very easy to access your

personalized list by having a button located
on the homescreen, whereas Netflix (right)
includes your list as a ribbon and required
scrolling almost to the bottom of the home
page to find my list and
not additional
recommendations.
Task 10: Remove movie or television show from personalized

list
__25/25_
__25/25_
Removing a television show or movie from your personalized
list was as easy as adding it. By clicking the same button that
was used to add the video/series you remove them as well. This
was very easy on both services.
Recommendations
Netflix scored more points overall in the UX testing than
Hulu did, but both services were easy to use and adapt to. If
you want to stream shows and movies on your mobile device in
more than one language then Netflix is the obvious choice as I
was unable to locate any change language feature in the Hulu
app. If you utilize personalized lists for shows and movies then
Hulu might be the better choice as it was far quicker in Hulu to
navigate to my personalized list.
Raw Collected Data
sks Netflix Hulu

sk 1 Find a known movie Find a known movie
ficient I was able to complete this task in a reasonable I was able to complete this task in a reasonable
amount of time. amount of time.
Strongly Somewhat Neutral Somewhat Strongly Strongly Somewhat Neutral Somewhat Strongly
Disagree Disagree Agree Agree Disagree Disagree Agree Agree
1 2 3 4 5 1 2 3 4 5
fective I was able to complete this task I was able to complete this task
1 2 3 4 5 1 2 3 4 5
ngaging The function was easy to find and use The function was easy to find and use
1 2 3 4 5 1 2 3 4 5
ror I did not need to close the app or return to the I did not need to close the app or return to the
lerant opening page to correct an error opening page to correct an error
1 2 3 4 5 1 2 3 4 5
asy to The layout felt familiar and consistent with other The layout felt familiar and consistent with other
arn mobile video formatting mobile video formatting
1 2 3 4 5 1 2 3 4 5
dditional There was a lot of information on the home page Hulu recommends a lot of movies and it was
otes and having the search function as an option on the somewhat hard to find a movie that I knew of
bottom took me a moment to find as I instinctively without using the search function. The search
looked for it at the top. function was on the home page though, which
does help if you know what you are looking for.
sks Netflix Hulu

sk 2 Find a known television show Find a known television show
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
dditional Now knowing where the search function was Using the search function was intuitive and quickly
otes located made it much quicker to search for a found the show that I was searching for.
television show.
*Finding the show through the recommendations

took much longer and was not helpful as it took a
significant amount of time to complete the task for
both apps.
sks Netflix Hulu

sk 3 Navigate forward or backwards mid show Navigate forward or backwards mid show
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
dditional Both Netflix and Hulu have back and forward This task was easily used and the only reason that
otes buttons in 10 second increments but Netflix provides Hulu was marked as not being as consistent with
an image of the time when jumping to another other mobile video formatting is that their fast
spot in the video. forward function does not display a video image.
*Hulu app does not allow for screenshots
sks Netflix Hulu

sk 4 Navigate to another movie Navigate to another movie
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
dditional Netflix automatically plays video full screen and as Related movies were displayed directly below the
otes such it took an additional step to locate the related playing one unless it was on full screen making it
movies. easy to find.
sks Netflix Hulu

sk 5 Select next episode Select next episode
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
dditional Next episode icon was displayed on the bottom of Next episodes and related shows were displayed
otes the screen at the start of the episode. immediately below the running television show.
sks Netflix Hulu

sk 6 Display subtitles Display subtitles
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
dditional Subtitles/audio settings were displayed at the Turing on the subtitles required using the gear at th
otes bottom of the screen when the episode started top left of the screen and which was not as intuitiv
making it very convenient to access and change. or easy to find, within the menu it was not labeled
as subtitles but as “accessibility settings” which wa
not very helpful.
sks Netflix Hulu

sk 7 Change audio and subtitle language Change audio and subtitle language
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
dditional The setting was connected with the subtitle settings I was unable to complete the task of changing the
otes and was easy to find and use. audio language.
sks Netflix Hulu

sk 8 Add movie or television show to personalized list Add movie or television show to personalized list
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
dditional *for both Hulu and Netflix the add to my list function
otes was on the landing page for the television show
and required that I returned to the home page to
research the show to add it to my list.
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
dditional Netflix lists everything on your list as it’s own ribbon Hulu has a “my list” button on the home page
otes and requires that you scroll almost to the bottom of making it exceptionally easy to locate your
the home page to access it. personalized list.
sks Netflix Hulu

sk 10 Remove movie or television show from personalized Remove movie or television show from personalize
list list
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5
dditional *This process was unclicking the button that was

otes used to add the show and was very straight
forward.

Netflix - Hulu Usability Testing White Paper

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Netflix - Hulu Usability Testing White Paper

Uploaded by

Copyright:

Available Formats

Netflix and Hulu Mobile UX Testing

Tasks Used For Evaluation

Effective – I was able to complete this task.

Task 2: Find a known television show

Both Hulu (left) and Netflix (right) have a wide

Task 3: Navigate forward or backwards mid video

Hulu and Netflix

Task 4: Navigate to another Movie

Both applications made it easy to

Task 5: Select next television show episode

Both services listed the next episodes

Task 6: Display subtitles for movie or television show

Netflix (pictured above) scored significantly higher than Hulu

Task 7: Change audio and subtitle language of movie or

__5/25_(lowest possible score)

Not pictured is the hulu app which prevented me from taking

Task 8: Add movie or television show to personalized list

For both Netflix and Hulu the add to

Task 9: Locate movie or television show within personalized

Hulu (left) made it very easy to access your

Task 10: Remove movie or television show from personalized

Raw Collected Data

sks Netflix Hulu

sks Netflix Hulu

*Finding the show through the recommendations

sks Netflix Hulu

sks Netflix Hulu

sks Netflix Hulu

sks Netflix Hulu

sks Netflix Hulu

sks Netflix Hulu

sks Netflix Hulu

dditional *This process was unclicking the button that was

You might also like