Usability Testing (M33121)

Level 6

Coursework Specification
Module Coordinator: claudia.iacob@port.ac.uk

As part of this assignment, you will design, run, and report on a usability evaluation study of
a popular application/website. A list of popular apps/websites is available on Moodle. Please
select one app/website that has not been selected by more than two students. This will be
the app/website your study will focus on.

Your submission should include 3 parts:

1. The design of the study

2. A discussion of the results and their implications
3. An analysis of the threats to validity of the study.

Part 1. Study design

Part 1 should report on the design of your usability study, and it should be structured to
cover all the topics below:

1.1. Usability goals. Identify 3 possible usability testing goals for the application/website
you have chosen. Present these goals as succinct statements. Below, a few examples:

● Decide whether this app will be perceived similarly efficient by novice users and
advanced users.
● Determine whether this app will encourage elderly users to book flight tickets online.
● Determine the extent to which this app helps students manage their finances.

1.2. Methodology. Identify the usability concepts you need to measure to be able to meet
the goals defined for the study.

For each of the concepts identified, define metrics that can measure that concept.

Identify and discuss the methods and techniques that will allow you to capture the data
you need to calculate these metrics. For each method, identify the data you will be collecting
and the ways in which you will derive the metrics based on this data.
1.3. User profiles. Define your participants’ user profiles. Discuss the rationale behind every
user characteristic chosen with reference to the goals defined for the study.

1.4. Procedure. Provide a step-by-step workflow of the testing session. For each step,
identify who is involved, what is achieved, how much time is required, what possible risks
can you identify and what are the plans to mitigate them.

Define one or more scenarios (maximum five) to be used during the testing session.

Discuss any requirements for material or equipment used.

Part 2. Results and implications

Part 2 should report on the data analysis, the results of the study, and the implications of the
results. A minimum of 8 participants is required, but for statistically significant results, please
aim to include 10-12 participants. Part 2 should discuss all the topics below.

2.1. Data analysis. Possible data collected may include video recordings, audio recordings,
survey answers, interview data, etc. For each data type collected, define an analysis
protocol. At least two data types need to be considered in the data analysis. At least one
data type must be qualitative.

For each data type collected, apply the analysis protocol. Your submission should include a
sample of each of the types of data collected and the specific analysis done on that sample.
At least two data types need to be considered in the data analysis. At least one data type
must be qualitative.

2.2. Results interpretation and implications. Discuss the results obtained in relation to
each of the metrics defined for the study. Discuss the meaning of the results. Focus on any
patterns identified in the data, and the possible reasons for them.

Discuss the implications of the results in terms of the goals set for the study and the design
of future iterations of the app.
Part 3. Limitations and recommendations

Part 3 should report on the limitations of the study (i.e. threats to validity)) and their possible
implications on the results, as well as any recommendations for improving the usability of the
app/system evaluated. It should discuss the topic below.

3.1. Limitations and recommendations. Discuss any biases you think might have
influenced your study. Focus the discussion on each design decision, and analyse what
biases each decision had the potential to lead to.

Discuss alternative designs in answer to the usability issues identified as part of the study.