EME6208 Interactive Media - IMA2: Usability testing report

IMA 2 - Usability testing report

Duolingo

Dan Krupa

3/17/2019

1
 EME6208 Interactive Media - IMA2: Usability testing report

Introduction
Duolingo is a foreign language learning mobile device app that allows learners
to master a foreign language by participating in daily practice sessions. The
one feature that distinguishes Duolingo from most other mobile language
apps is that it uses it is designed to function like a game, with the purpose of
keeping learners motivated. At first glance, the app gives the appearance that
it is designed for children, due to its cartoon-like appearance. However, as
demonstrated in this usability study, it can be very effective in teaching the
people of all ages a new language. In addition, as noted by the study
participants, most would find it to be user-friendly.

Duolingo is based on the understanding that each individual, when trying to

learn a new language, come from different backgrounds. Some people will
have experience with that particular language, while others may not have
any. That’s why Duolingo has each person create an account, which begins
with a diagnostic test, and then places him or her into a certain level. As that
person completes more practices, he or she can move up to different levels.
Duolingo also allows a user to set daily reminders for practice sessions, an
can also set the length of time for those sessions. What if someone wants to
learn more than one language? No problem! A user’s account can easily be
switched between different languages.

A usability study was conducted by myself (Dan Krupa) for the purpose of
determining the effectiveness of Duolingo in learning new languages. Five
participants was chosen, who differed from each other in age, occupation, and
other walks of life. This allowed the researcher to determine if anoy of these
factors had an impact on the users rating of this learning app.

As is the case with any learning app, Duolingo has both advantages and
disadvantages. As had been noted by the participants in the study, it is
simple to set up an account, gives users a diagnostic test on a certain
language, and allows users to set up daily practice sessions for that language.
It also allows users to switch between different languages, is they are
interested in learning more than one. However, some disadvantages were
noted. For example, the audio icon was not easy to locate, and most of the
users had to navigate to try to find it. Also, it was not clear on how to click to
change the language. Finally, some users noted that it may take time to fully
understand the workings of the app.

Overall, all participants stated that Duolingo is a fun and easy language
learning app and, most importantly, it very users-friendly and keeps them
actively engaged.

2
 EME6208 Interactive Media - IMA2: Usability testing report

Methodology
Sessions
This usability study involved participants from different walks of life. Due to
their varied time schedules, the study was not done in one session, but with
each person individually, at different times (with the exception of participants
1&2, who were both done on the same date and time). Each individual
session lastly between 20-30 minutes, starting with each participant creating
an account. In order to create a relaxing atmosphere, the researcher
conducted each of the sessions in a home environment. In addition, this was
also important because, as users of the Duolingo app, most people will be
learning a new language in their home (and maybe work) environments. Each
participant in the study downloaded the app to his or her phone, and created
an account. This was the first step in the study. After that, they each
completed the required tasks, in order. These included tasks, such as
selecting a language, completing the included diagnostic test, changing to
another language, turning on and off the audio, and changing other settings.

The researcher in this study asked each participant some background

questions that could have an effect on his or her experience with Duolingo.
The two most important were:

1.) How proficient are you in your chosen language?

2.) Do you have any previous experience using a language learning app?

All of the participants had previous exposure to his or her selected language.
However, they could only recognize a few of the words, and none of them
could speak his or her language fluently. In addition, only two of the
participants have tried another language learning app, neither of which were
Duolingo.

Learnability
Duolingo is a language-learning app that is fairly self-explanatory. Users first
Download the app to a mobile device, and then create a personal account.
These initial steps are straightforward, and all of the participants in this study
Found these tasks to be fairly simple. After initial set-up, users then select a
language that they would like to learn, and are asked if they have any
experience with that language. The answer to this question will then
determine which diagnostic test will be given. It gets more complicated,
however, when users want to change settings, such as switching languages or
changing audio settings. Some of the participants in this study found these
tasks to be somewhat challenging. Two of the participants could not even
complete them on their own, and needed the guidance of the administrator.

3
 EME6208 Interactive Media - IMA2: Usability testing report

Evaluation Tasks/Scenarios
The participants in the Duolingo usability study were evaluated on their
completion of five different tasks while using the app. The first was the initial
task of setting up the account. Second, participants decided on which
language he or she would like to learn, and completed the placement test on
that language. Some participants were more proficient in their selected
languages than others. Some even used their family background to decide
which language to try, Third, the participants were to switch to a new
language. Most of them found this to be somewhat challenging, as Duolingo
does not clearly designate were to go on the app to change the language
setting. The fourth, and most challenging task, was for the participants to
change the auditory settings on the app. Just like the language, these
settings were shown to be somewhat difficult to locate. Finally, the fifth task
required them to change any setting of his or her choosing on the Duolingo
app. As a result of learning how to change other previous setting, most found
this task to be fairly simple.

Results
Task Completion Success Rate
Each participant in the study successfully completed tasks #1-3 without any
assistance from the administrator. Participants 2 and 5 required assistance on
the last 2 tasks, which included turning on and off the auditory sound and
changing another setting on the app.

The results of the participants completing each of the tasks successfully are
summarized I the data table below.

Task Completion Rates

Participant Task 1 Task 2 Task 3 Task 4 Task 5

1 √ √ √ √ √

2 √ √ √ - -

3 √ √ √ √ √

4 √ √ √ √ √

5 √ √ √ - -

Success 5 5 5 3 3
Completion
100% 100% 100% 60% 60%
Rates

Learnability: Time to complete a task successfully

Each team member recorded the time on task for each participant. Some
tasks were inherently more difficult to complete than others and is reflected
by the average time on task. As shown in the data table below, the first 3
tasks took the longest. These times decreased as the participants move on to
the next tasks.

4
 EME6208 Interactive Media - IMA2: Usability testing report

Time on Task
P1 P2 P3 P4 P5 TOT Avg.
Task 1 480 540 180 190 180 1570 314

Task 2 469 780 300 150 180 1879 376

Task 3 475 240 180 30 360 1285 257

Task 4 50 50 55 15 45 215 43

Task 5 30 45 30 10 180 295 59

Errors
The test administrator captured the number of errors participants made while
trying to complete the task scenarios.

A non-critical error is an error that does not prevent successful completion of

the scenario. As can be viewed in the test results, even though some
participants had errors, they were all non-critical, meaning that each
participant successfully completed each task. The highest number was in task
#3 and 4.

5
 EME6208 Interactive Media - IMA2: Usability testing report

Errors
P1 P2 P3 P4 P5 TOT Avg.
Task 1 1 0 1 1 1 4 0.80

Task 2 1 1 1 0 0 3 0.60

Task 3 2 2 0 0 3 7 0.71

Task 4 0 1 2 2 1 6 0.83

Task 5 0 0 1 1 2 4 0.80

Summary of Data
The table below displays a summary of the test data. Low completion rates
and satisfaction ratings and high errors and time on tasks are highlighted in
bold. The highest number of errors occurred in task #4, which involved
changing the audio setting on the app.

Summary of Completion, Learnability (Time on Task), Errors

Learnability
Task Task Completion Errors
(Time on Task)
1 5 314 0.80
2 5 376 0.60
3 5 257 0.71
4 3 43 0.83
5 3 59 0.80

Task and Satisfaction Questionnaire Results

When questioned on the ease of the 5 tasks, all participants agreed that tasks
1-3 were easy or super easy. The results began to vary when they got to
question 4, which began to rate as more challenging.

6
 EME6208 Interactive Media - IMA2: Usability testing report

Task Questionnaire table (example)

Super Not easy Super Mean Percent

Easy Hard
easy or hard hard Rating Agree

Task 1 4 1 1.2 100%

Task 2 3 2 1.8 100%

Task 3 4 1 1.2 100%

Task 4 3 2 1.8 60%

Task 5 3 2 1.8 100%

*Percent Agree (%) = Agree & Strongly Agree Responses combined

7
 EME6208 Interactive Media - IMA2: Usability testing report

Satisfaction Questionnaire table (example)

4
1 7
Do not Mean Percent
Strongly 2 3 5 6 Strongly
agree or Rating Agree
Disagree Agree
disagree
1. Overall, I am satisfied with
how easy it is to use this 1 2 2 6.2 80%
language learning app.
2. It was simple to use this
3 2 6.4 100%
language learning app.
3. I could effectively complete
the tasks using this language 1 2 2 5.8 80%
learning app.
4. I was able to complete the
tasks quickly using this language 3 2 6.4 100%
learning app.
5. I was able to efficiently
complete the tasks using this 1 3 1 7.0 80%
language learning app.
6. I felt comfortable using this
1 1 3 6.4 80%
language learning app.
7. It was easy to learn to use this
1 1 3 6.4 80%
language learning app.
8. I believe I could become
productive quickly using this 1 3 1 5.4 80%
language learning app.
9. This language learning app
gave error messages that clearly 1 1 1 2 5.2 60%
told me how to fix problems.
10. Whenever I made a mistake
using the language learning app,
I could recover easily and
1 2 2 6.2 80%
quickly.

8
 EME6208 Interactive Media - IMA2: Usability testing report

4
1 7
Do not Mean Percent
Strongly 2 3 5 6 Strongly
agree or Rating Agree
Disagree Agree
disagree
11. The information (such as on-
line help, on-screen messages
and other documentation) 1 3 1 6.0 80%
provided with this language
learning app was clear.
12. It was easy to find the
1 1 3 5.4 60%
information I needed.
13. The information provided for
this language learning app was 1 2 2 6.2 80%
easy to understand.
14. The information was
effective in helping me complete 1 3 1 6.0 80%
the tasks.
15. The organization of
information on the language 1 1 1 2 5.8 60%
learning app screens was clear.
16. The interface of this
language learning app (buttons, 1 2 2 6.0 80%
menus, graphics) was pleasant.
17. I liked using the interface of
this language learning app 1 1 3 6.2 80%
(buttons, menus, graphics).
18. This language learning app
has all the functions and 1 4 5.4 80%
capabilities I expect it to have.
19. Overall, I am satisfied with
1 2 2 6.0 80%
this language learning app.
*Percent Agree (%) = Agree & Strongly Agree Responses combined

9
 EME6208 Interactive Media - IMA2: Usability testing report

User experience
While there were a range of comments by each of the participants, most agreed that the Duolingo app was fairly simple to use.
Although some needed prompting for the later tasks, all of them were completed and everyone had mentioned that, given
enough time and use of the app, they could began proficient in learning the language of his or her choice.

10
 EME6208 Interactive Media - IMA2: Usability testing report

Recommendations
Based on the reviews from the surveys completed both during and after the Duolingo
usability study, recommendations have been suggested by the test coordinator (Dan
Krupa) While the ratings from all of the participants in the Duolingo study gave
mostly positive ratings, there are some areas of possible improvement. These
include:

1.) The icons on the screens should more clearly identify how to change certain
settings, such as language or audio
2.) Provide a short tutorial on usage of the app after users creates an account
3.) Include a Help icon on the screen, allowing a user to get assistance by typing in a
keyword or question
4.) Improve the overall clarity of instruction

Conclusion
In this usability study of Duolingo, all of the participants completed each task
successfully. Even participants that struggled with some the tasks initially, such as
changing the language or audio setting, found those tasks to be simple once they
were given guidance by the test administrator. This allowed all of them to
satisfactorily complete the fifth and final task.

The key issue for most participants was learning how to switch between the different
settings. However, after some had initial difficulty, they found this not to be an issue
when completing the final task. Overall, the participants rated Duolingo favorably
and most stated that, given enough time to work with the app, it can be a very
useful tool for learning a new language.

Attachments
Each participant in the study has an attached:
Completed observation sheet
Completed task questionnaire
Completed satisfaction questionnaire

11