Slide 1

L2
High graded answer to the activity L1
Define
The Necker's cube is a bistable visual illusion. The same sensory data can support two
perceptual inferences (one in which corner 1 is closest to the observer, the other in which
corner 2 is closest).
Explain
This exemplifies that perception, as also suggested by Nour & Nour (2017) is the product of
two sources of information: the sensory data (bottom-up) and prior expectations (top-down).
Conclude
In this case, there is a conflict between these two modalities bottom-up and top-down that
could bring to perceive the cube heading towards different directions on the basis of which
modality dominates in a specific moment.

Why this answer is good

+ It answers the question (minimal mandatory requirement)
+ Terminology is correct (correctness)
+ Provide background and refer to the material (appropriateness)
+ It presents the answer in a short, logic and consistent way (logic and
consistency)

Slide 2
Agenda
• Usability and UX Methods and Definition(s)
• Tools and methods for ideation and testing

Slide 3
The decision making that you perform when you are going to design a process
or a product will be always affected by the context of use, the human and
psychological factors involved.
If you do not have enough information (about the context, the process of use, the
people involved etc.) well you could risk designing something (Like the wide
research for a missing woman at night in Iceland) that will be not useful by
wasting resources and energy.

This is the reason why user research, user experience and usability assessment
are important!

Slide 4
User experience: Definition, Issues and solutions

Slide 5
The ISO 9241-210 Defines the concept of User experience as a process in which
people perceptions and responses that result from the use and/or anticipated use
of a product, system or service.

This means that to measure UX we need to consider end-user:

• emotions,
• beliefs,
• preferences,
• perceptions,
• physical and psychological responses,
• behaviours and accomplishments

that occur before, during and after the use of a product or service>>
Anticipated use: Refer to people believes, attitude and Trust toward a product

Slide 6
User experience is different from Usability.

User Experience is focused on creating first value for the end-users, UX

plays on usefulness for people and on their emotion to create and promotes
positive emotions toward a service or a product.

Usability is more about safety, learnability etc.

Slide 7
However, A service/product with poor usability is more likely to provide a low
level of UX
Usability remains vital, and yet the interface-centred methods and perspectives
of human-computer interaction do not address all dimensions of web design. In
short, usability is necessary but not sufficient.

But UX also includes

Usefulness. As practitioners, we can’t be content to paint within the lines drawn
by managers. We must have the courage and creativity to ask whether our
products and systems are useful and to apply our knowledge of craft + medium
to define innovative solutions that are more useful.

Products and systems should be:

Desirable. Our quest for efficiency must be tempered by an appreciation for the
power and value of image, identity, brand, and other elements of emotional
design.
Findable. We must strive to design navigable web sites and locatable objects, so
users can find what they need.
Accessible. Just as our buildings have elevators and ramps, our web sites should
be accessible to people with disabilities (more than 10% of the population).
Today, it’s good business and the ethical thing to do. Eventually, it will become
the law.
Credible. Thanks to the Web Credibility Project, we’re beginning to understand
the design elements that influence whether users trust and believe what we tell
them.
Valuable. Our sites must deliver value to our stakeholders (not only the users).
For non-profits, the user experience must advance the mission. With for-profits,
it must contribute to the bottom line and improve customer satisfaction.

Slide 8
Key points about interaction assessment

Slide 9
User interface (UI) design and User Experience(UX) are two connected but
different aspects. UI is more about design and UX is more about “quantifying”
the experience of people with the product including the experience provided by
the UI
Slide 10
A well-prepared designer can do User Interface
But nobody can do UX alone!

Slide 11
Designers can not deal with UX alone! Even if they know how to perform UX
research. Only a team of experts can do that introducing in the design process
multiple perspectives and insights from in filed research that designers may use
to drive their creative process

Slide 12
Quite often we invest a significant amount of money to develop solutions for
problems that nobody has!

Slide 13
To provide a good experience means to know the needs of the end-users and
stakeholders and to provide value to them in their context.
We need to ask our self
Who really are our users?
What makes them tick?
What are they are hoping to accomplish?
Which are the constraints in the process of use and in the context?

Slide 14
Even stressful events, like performing an MRI scan, can be improved if we
combine usability to improve the experience of the end-users.
This does not mean that we do not have to pose the right question during the
development:
Is this product cost-effective?
it is safe to use?
it reduces the workload, the stress or facilitates people life, and well-being?
it increases the satisfaction of all the stakeholders involved.

These questions are not secondary ones. We need to know the answers to these
questions at the early stages of design!
Slide 15
End-Users
People who are using (actively or passively) a technology or service

Stakeholders
People who are affected or are affected by technology or a service

Example: An assistive technology that is right for the end-users, but bad for the
caregiver is more likely to be abandon

Slide 16
Affordances are perceived possibilities of action (Norman, 1988)
Affordances are the visual clue to explain without a verbal explanation the
function of the object (an opportunity for action)

Slide 17
Good affordances enable users to interact in a natural and comprehensible way

Slide 18
Affordances and Active users
If we do not design a clear pathway for the end-users with signifiers and
perceivable affordances and constraints, the end-users will find their way to use
a system.

Slide 19
How you designing and presenting information is important
To have a prototype of your product, or to design appropriate stimuli for your
experiment, is important because what you show to your participants and how
(methods and questions) is going to affect the reaction of the participants.
How you design the process of use, the space between elements, their
relationship, the colours, the labels and the words you are using to present the
contents are going to convey different information and affordances to
participants.
If you look at this example of the USA electoral map it seems that the RED
(Republican) received more votes than the BLUE (Democrats) but if you
visualize the actual vote of people in a different modality the visual information
is significantly different.
We are priming and biasing people by design, we need to be aware of that and to
control for that.

Slide 20
Questions to answer at early stages of development
THE WHO AND WHAT OF THE SOLUTION
• For who this tech/service is NOT designed for?
• Who are our users? And who are the stakeholders?
• What makes them tick?
• What are they hoping to accomplish? What are the tasks?

THE WHY OF THIS SOLUTION

• What are the social needs?
• There is value in the service/product?
• Is this product cost-effective?
• it reduces the workload, the stress or facilitates people life, and well-being?
• Does it increase the satisfaction of all the stakeholders involved?

THE HOW OF THE SOLUTION

• What happens when it goes wrong? it is safe to use?
• Does it improve the overall experience?
• Does it provide value and usefulness?

Slide 21
How to answer the questions: Testing, testing, testing…

Slide 22
User research at different points
UX is a process
Cannot be measured with one score, or with one method or in one specific
moment. You need a plan!

Usability is property
Quality of interaction

Usability testing occurs during and after the use of a product

Slide 23
User testing is mainly for products already in the market. But some techniques
could be used also in concept/prototyping phases
UX evaluation requires at least a fully functional prototype. However, aspects
such as the intention of use and perceived usefulness which are important to the
overall experience could be measured with raw concepts or mock-ups

Slide 24
Please note that:
USABILITY TESTING GOAL IS TO ASSESS THE INTERACTION
QUALITY: the fact that usability testing also provides evidence to redesign
is a consequence, or a secondary goal, not the main objective of the usability
testing.

Slide 25
The landscape of User Research methods
The graphic presentation shows here methods from the more Qualitative and
attitudinal to the more quantitative and behavioural

https://www.nngroup.com/articles/which-ux-research-methods/

Slide 26
In this link, you can find useful resources to drive you in the selection of
appropriate methods of assessment: https://servicedesigntools.org/tools

Slide 27
A meaningful UX evaluation: Requires multiple evaluation points with
objective and subjective measures, and well-defined procedures to collect data
over time to support design and redesign

Usability: requires a set of well-planned moments of review and test at different

stages (formative or summative)

Slide 28
Usability definition
Slide 29
Usability is usually described in terms of Effectiveness, Efficiency, Satisfaction
in line with the ISO9241-11, 2018:

The extent to which a product can be used by specified users to achieve

specified goals with effectiveness, efficiency and satisfaction in a specified
context of use.

Slide 30
What usually is not reported in usability studies is The context of use!
THIS IS A SERIOUS LIMITATION! No context= no possibility to
replicate

This Accounts for the specificity of the test and the type of participants
• Individual variables (attitudes, habits previous experiences, expectations)
are collected.
• which type of tasks and scenarios you asked participants to perform, how
these tasks were presented to the end-users.
• Which equipment did you use for the test e.g. the same PC or laptop, which
operative system, touch or mouse interaction?
• Which size of the screen/monitor?
• If it is a physical prototype, how this was presented? Which dimensions?
Did you record the audio and the video?
• With which tool for audio recordings?

Slide 31
Context
<< The information about the users (demographic information, their attitudes
preferences etc.), the tasks that they have to perform, information about the
physical and social environments in which a product is used >>

What to report about the context you simulated in the test e.g.:
Data about participants (how much they represent the stakeholders)
questionnaires about users attitudes, preferences, competences
the qualitative observation about people behaviour, attitudes
description of the tasks and their reasons,
information about equipment used in the test,
the observation about the usage of the product in the real environment of use.
Slide 32
Effectiveness
<<Accuracy and completeness with which users achieve certain goals.
Indicators of effectiveness include the quality of solution and error rates.>>
Measured by e.g.:
NUMBER OF ERRORS PERFORMED or VERBALISED,
ISSUES EXPERIENCED,
MISSING TASKS

Slide 33
Efficiency
Measured e.g.:
TIME TO PERFORM CORRECTLY
TASK COMPLETION
LEARNING TIME
This could also include the measure of stress and cognitive workload see for
instance the NASA-TLX or the Rating Scale Mental Effort (RSME)

Slide 34
Satisfaction
<<Extent to which the user's physical, cognitive and emotional responses that
result from the use of a system, product or service meet the user’s needs and
expectations. Satisfaction includes the extent to which the user experience that
results from actual use meets the user’s needs and expectations. Anticipated use
(expectations) can influence satisfaction with actual use>>

Measured e.g.:
STANDARDISED SATISFACTION QUESTIONNAIRE.
This could include also measure of likelihood to recommend

Slide 35
Satisfaction in the context of usability is a very specific concept:
TO MEASURE SATISFACTION YOU MUST USE A STANDARDISE AND
RELIABLE SCALE
To ask qualitative questions regarding preferences, pleasure etc. is also possible
at the end of the usability test but these questions are not a measure of
satisfaction. Just additional information to inform about design preferences

Slide 36
There are free scales you can use
• System Usability Scale (SUS)
https://www.usability.gov/how-to-and-tools/methods/system-usability-
scale.html

• Usability Metric for User Experience (UMUX)

Finstad, K. (2010). The usability metric for user experience. Interacting with
Computers, 22(5), 323-327.

• User experience questionnaire (UEQ)

https://www.ueq-online.org/

These scales are validated and reliable, and each one of this scale has a
procedure to calculate and report an overall score of satisfaction from 0 to 100

Slide 37
Three variations of the USABILITY definition (9241-11) to enable the
application with different products. Software, medical devices, Everyday
products

Slide 38
We identified 54 standards which refer to ISO 9241-11. All these standards refer
to the usability umbrella as the framework to assess the interaction with 19
different types of products

Slide 39
When it comes to software, Usability is a sub-set of the quality of interaction:
• Efficiency
• Effectiveness
 • Satisfaction
• Context of use
• Appropriateness recognisability,
• Learnability,
• Operability,
• User error protection,
• User interface aesthetics,
• Accessibility

Slide 40
UX and Usability: Test what? How? When?

Slide 41
As represented in the picture, different methods of assessment are associated
with the development phase of a product

Slide 42
BREAK

Slide 43
Ideation phase: Inform the design and the requirements (context analysis)

Slide 44
Some methods for the ideation phase: Observation, interview, focus group, card
sorting, survey

Slide 45
In focus groups or interview you want to get out information about all the
possibilities of stakeholders

A useful way to gather information about who are the stakeholders is to use
Matrix Clarification Tool1.

1
See pages from 1 to 8 of this document
https://iris.wpro.who.int/bitstream/handle/10665.1/5508/9290611863_mod2_eng
.pdf
This is a simple way to ask during interview or survey regarding all the
people that can benefit from, influence the adoption of technology and their
importance

- INFLUENCE: the power stakeholders have over the adoption and usage
of a tool
- IMPORTANCE: how important is that the product is attractive to solve
the problem of that stakeholder

Slide 46
Technology and services never work in isolation you need to map the process of
use from different perspectives by:
- Observation
- System mapping with stakeholders
- User Journey
- Card sorting to organise contents
- Survey

Moreover, experts analysis with Cognitive walkthrough, heuristic evaluation and

task analysis are associated with system mapping process with stakeholders
which aim to map from a different perspective the use of the product in
(complex) sociotechnical environments

Slide 47
Design and be aware of key tasks and scenarios
From concept to prototype

Slide 48
In the slide, you have a summary of methods for the concept/prototype phase

Slide 49
Task analysis
There are different types of task analysis e.g.,
• Hierarchical looks at the relationship among tasks in the process
• Cognitive looks at the decisions and cognitive effort to perform the tasks
We will see an example in the case study!
For more info:
http://www.idemployee.id.tue.nl/g.w.m.rauterberg/lecturenotes/UFTtask-
analysis.pdf

Slide 50
Tasks and goals
An interactive system requires a certain set of actions to achieve a goal. For
instance, in this example, a user wants to login into a system.
If you know the system you also know the sequence of correct actions that
people should perform, so that:
• If participants do something outside of the list of task = You can report a
usability issue
• If participants say that they do not know what to do = You can report a
usability issue
• If participants perform the steps but there is a technical problem = You can
report a usability issue

Slide 51
Usually, when you involve participants you present the tasks in the form of
scenarios in which you do not say what they have to do but what they want to
achieve (goal)

Slide 52
Iterative Cycle of Design
After that you have defined the product (ideation) You will start by defining
tasks and performing a review e.g., Heuristics principles.
This will provide you with a set of potential interactive issues that you will try to
solve by a redesign (prototype). Then you could perform a usability test that will
provide you with a list of issues experienced by participants. This will enable
you to comment about the usability of your prototype and to take further steps.

Slide 53
Inspection methods
You can explore the limitations of a system by using principles to rapidly inform
the redesign (e.g., heuristics) but this will be never as powerful as the user
testing. We will discuss this type of assessment in L3

Slide 54
Test with users (Prototype or product)

Slide 55
To involve people and let them interact with a prototype or a product IS NOT
ALWAYS equal to usability testing. For instance, in a participatory design, you
can invite end-users to redesign a product with you, this is a powerful way to
generate ideas and to improve the design, but it is not a form of assessment

Slide 56
There is always a distance between manufacturers and stakeholders point of
view. To reduce this distance it is necessary to show to manufacturers and to the
stakeholders the other perspective.

Slide 57
DIFFERENTLY FROM OTHER METHODS ONLY WITH USABILITY
TESTING:
you are exploring in a formal and reliable/replicable way the user performance
with the product /prototype to measure the usability e.g., efficiency,
effectiveness and satisfaction.

THERE ARE NO OTHER WAYS TO FULLY MEASURE USABILITY!

Slide 58
Why usability is important for every product
https://www.youtube.com/watch?v=Bzy5hVvbei8

Slide 59
There are different forms of Usability testing in-presence or in a remote. How
you set up your protocol is important, because you need to collect enough data
to advance your design in a reliable way
Slide 60
Effectiveness: missing steps, verbalization of problems (Verbalization could be
collected during (Concurrent) or after the interaction (retrospective)), issues in
understanding how to progress toward the goal, time to achieve the task
Efficiency: Time to perform correctly or to fail the task
Perceived satisfaction: Satisfaction scales, other important factors for the
specific product e.g., intention to promote, workload etc.
Qualitative perspective: User preference (verbal) and qualitative comments
(during the debriefs)

Slide 61
Level of prototype
When you do not have a functional product you can use a wizard of Oz
technique:
<<user interact with a product and responses are (all or partially)
generated by a human behind-the-scenes.>>

Slide 62
Wizard of Oz
Is a method of testing a system that does not exist (the listening typewriter, IBM
1984)

Human ‘wizard’ simulates system response:

• interprets user input according to an algorithm
• controls computer to simulate the appropriate output
• uses a real or mock interface
• wizard sometimes visible, sometimes hidden

good for:
adding simulated and complex vertical functionality
testing futuristic ideas

Slide 63
Advantages
Data are used to inform how to progress the design before implementing a fully
functional prototype
Good for A/B testing
Good as a driver for focus group and expert review

Disadvantages
Sometimes is less expensive to have a functional prototype (even not
aesthetically pleasing)
Data about efficiency, effectiveness and satisfaction could be affected by the
simulation and could be not representative of the real interaction with the
product.

Slide 64 Of usability test of fully functional prototype

Advantages
Data are used to inform how to progress the design by controlling the usability
You can model the likelihood of people finding issues

Disadvantages
Time and costs

Slide 65
Tips to organise users testing

Slide 66
The starting point of your evaluation should be a good (set of) question(s):

What you want out of your assessment?

Why you are doing the evaluation?

Slide 67
In a test:
You are empirically exploring a system with the end-users observing what
participants are doing, their reaction and the problems they are experiencing or
verbalise in order to achieve a task.

Slide 68
What you are testing?
Are people able to perform correctly the tasks presented as scenarios?

• If participants…
… do something outside of the list of correct steps= Issue
… say that they do not know what to do = Issue
… perform the correct steps but there is an technical problem = Issue
… can not achieve the goal or fail to achieve it= Issue
… verbalised issues due to e.g. information organisation = Issue
… spend an unreasonable amount of time to achieve the task = Issue

Everything that prevents a smooth interaction in terms of time and performance

is a usability issue

Slide 69
You test to obtain data, such as: time to perform, task achievement, task
failure, number and types of errors, verbalised issues, frustration/ satisfaction

These will help you to identify elements of the system/prototype which may be
improved.

Slide 70
Goal:
Identify if something does not work; Test new functions; Test uncertainty;
Decide how to redesign the system in future

Set up the procedure:

-Tasks that people have to perform (know when participants are right or wrong)
-Type of data you want to gather;
-Type of questions you are going to ask.

Minimal equipment
• Conference system
• Interactive Prototype link
• Participant: device PC/Mobile/Tablet
• Audio and video recording
Slide 71
Test don’t judge
You aim to enable a good interactive experience! You are doing a test not to
assess (how good is your product or how terrible are end-users) but to get
information to better support your decision making about the design of your
product.

Slide 72
You may choose different approaches to let people verbalise their
experience:

• Let people talk aloud during the interaction (Concurrent thinking aloud)
• Ask people to interact, video record their screen actions and audio. Ask them
to watch a video of their interaction, and to comment about issues
(Retrospective thinking aloud)

Slide 73
Each usability test is composed at least by the three following three
elements:
- Consent and questionnaire before the interaction (demographic and other
aspects you want to collect)
- Interaction with the product (observation, data recording) and a technique
to enable participants to verbalise issues. Here you should use your
scenarios!
- Debrief phase for final remarks of participants e.g., measures of
satisfaction, cognitive workload and likelihood to use scale etc.

Slide 74
Evaluation protocol
You should know since from the beginning:
• What you want out of the test
• How and why you are going to record and use the data you will collect
(each measure counts!)
• How you are going to report these data in your report.

Try to put down on paper your procedure:

- What participants will do from when they arrive, since when they will
finish the assessment
 - The order of the events: Questionnaires, the order of tasks, etc.
- What you will do: who will observe, who will support the participant,
who will record etc.
- How you will do that: e.g., ask participants to verbalise the issues? how
you are going to answer if they have problems?
- Which equipment you will use to collect data
- How you will store data
- How you are going to analyse the data

Slide 75
As an evaluator you will assist the participant remotely, you should drive and
keep people in the test, but you can not affect people decisions.

Slide 76
SOME EXAMPLE OF DEALING WITH PARTICIPANTS:
If participants struggle to find something or to progress. Do not help them!
You can ask:
Do you remember your goal?
What are you thinking?
What are you doing right now?

If you notice that participants get frustrated because they can not achieve the
goal?
You can say:
<<You believe that this task is achievable? If not, we can go to the next one?
Remember that if the system is not good enough to enable you to find something
is not your fault!>>
<<You are the end-user, your opinion is what matters if you believe that
something is not doable, for us this is already a valuable result.>>

Slide 77
Tips for the testers
• Define the critical and key tasks of the product
• Design the test procedure
• Present the tasks in terms of reliable scenarios to participants
 • Observe, record data, take notes
• Always add standardised evaluation scales –this does not mean that you
can not use qualitative scales tailored on your product, just also add
standardised scales to enable benchmarking and replicability
• Use the data to inform how to improve your redesign

Slide 78
Remember that a usability report should contain
+ data about effectiveness e.g. Errors, Issues, failed tasks
+ data about efficiency e.g. time to perform
+ data about satisfaction e.g. SUS
+ data about participants
+ how you tested (scenarios, tasks, information to replicate your test)

Slide 79
Your report should also explain how you selected your evaluation methods e.g.,
Because :
+ Suit your constraints (budget/time/capacity)
+ May inform the design phase
+ Reliable insights at low costs

Slide 80
Summary
+UX requires a systems perspective
+ UX is a process
+ UX & usability go together
+ Evaluation requires expertise
A designer with some expertise in evaluation and research is not equal to a
User Experience Researcher. Evaluation is not easy! Bad or partial
evaluation and analysis bring to disaster!
Slide 81
END