Human-Computer Interaction

Interaction
In Today's Lecture
Interaction Methods

Interfaces Types
Interaction Methods
Examples:

GUI

Text based UI

What else ?
Interaction Methods
Examples:

GUI Head-up display (HUD)

Text based UI Augmented reality (AR)

Multitouch Gesture-based

Speech ……. and more

Interaction Types

Instructing conversing Manipulating Exploring

Interaction Types
Instructing — where users issue
instructions to a system. (e.g., typing
commands, selecting menus,
speaking aloud commands, gesturing,
pressing buttons, function keys.)
Interaction Types
Conversing — where users have a dialog with a
system. (e.g, users speak or type in questions to which
the system replies via text or speech output)

● Assistants: Siri, Cortana, Google Assistant, Alexa

● GPS
Interaction Types
Manipulating — where users interact
with objects in a virtual or physical space
by manipulating them (e.g. opening,
holding, closing, placing).
Interaction Types
Exploring — where users move through a virtual environment or a physical
space. (e.g., virtual environments, 3D worlds, augmented and virtual reality,
smart home)
Interaction Types

Instructing conversing Manipulating Exploring

 20 Interfaces
Command-based Air-based gesture
WIMP and GUI Haptic
Multimedia Multimodal
Visualization and dashboards Shareable
Web Tangible
Consumer electronics and appliances Wearable
Mobile Brain—computer interaction (BCI)
Speech Virtual reality
pen Augmented reality
Touch Mixed reality
Command Based (CLI)
● Simpler
● Faster
● Expert Mode
● Disabilities
WIMP(GUI)

Windows (that could be scrolled, Menus (offering lists of options that

stretched, overlapped, opened, could be scrolled through and
closed, and moved around the selected in the way a menu is used in
screen using the mouse). a restaurant).

Icons (to represent applications, Pointing device (a mouse

objects, commands, and tools that controlling the cursor as a point of
were opened or activated when entry to the windows, menus, and
clicked on). icons on the screen).
WIMP(GUI)

Windows
Invented to overcome the physical
constraints of a computer display
WIMP(GUI)

Windows
Invented to overcome the physical
constraints of a computer display Macintosh OS
Design Challenges:

● Difficult to find specific one

● Switching between windows

Microsoft Windows
 1981
The Xerox
WIMP(GUI) 8010

Icons 1983
History Apple
Lisa

1984
Apple
Macintos
h System
 1985
Atari TOS
WIMP(GUI) 1985
Amiga Workbench
Icons
History 1985
Windows 1.0x

1991
Macintosh
System 7
 1992
Windows 3.1
WIMP(GUI)

Icons
History
2001
Mac OS X v10.0
WIMP(GUI)

Icons
History

2001 Windows XP
WIMP(GUI)

Icons
Designs

Flat Icon Design in iOS & Windows

WIMP(GUI)

Icons
History

Aqua Icons used in iOS & Mac

WIMP(GUI)

Menus
Challenges

● Overshooting
● Scrolling

Cascading Menu
WIMP(GUI)

Menus
Category Menu
WIMP(GUI)

Menus
Cascading Menu
WIMP(GUI)

Menus
Iconic Menu
WIMP(GUI)

Menus
Iconic Menu
WIMP(GUI)

Menus
Container Menu
WIMP(GUI)

Menus
Container Menu
WIMP(GUI)

Menus
Circular Menu
Multimedia
Media is just the plural of medium. And a medium is just a format of expression.

Multimedia it's a mix of different medias. Like audio/video visuals (shorts movies,
audiobooks, animation) or text through interactive platforms (web, apps, interactive
installations)

Extremely Useful for Educational Purposes

Multimedia
Example: multimedia-based encyclopedia
about Heart
- One or two video clips of a real live heart pumping
and possibly a heart transplant operation.

- Audio recordings of the heart beating and perhaps

an eminent physician talking about the cause of
heart disease.

- Static diagrams and animations of the circulatory

system, sometimes with narration.

- Several columns of hypertext, describing the

structure and function of the heart.
Visualization (infoviz) & Dashboards
Amplify human cognition, enabling users to see patterns, trends, and anomalies in
the visualization and from this to gain insights

Dashboards is an interactive way to slice and dice the data, so that users can
home in on what is important to them.
Visualization (infoviz) & Dashboards
Visualization (infoviz) & Dashboards
One of the best tools for
visualizations is D3 (Data
Driven Documents)
https://d3js.org
Web
• Text only
1980's • Black and White

• Button and Menus

• Navigation
1990's • More color, less Text

• Flash
• Readability
2000's • Functionality

• Responsive
• Minimalist
2010's • Animation

2020's • ??
Web
Consumer Electronics & Appliances
Washing machines, DVD players, vending machines, remotes, photocopiers,
printers, navigation systems, MP3 player, digital clock, digital camera, … etc

Get something specific done in a Short Period of Time

Mobile
New UI Experience

● Compact display design

● Animated menus
● Virtual Keyboard
● Monotask
● Camera as input device (QR "quick response" Code)
Speech
A person talks with a system that has a spoken language application, like a train
timetable, a travel planner, or a phone service

Example:
System: Which city do you want to fly to?
Caller: London

System: Which airport — Gatwick, Heathrow, Luton, Stansted, or City?

Caller: Gatwick

System: What day do you want to depart?

Caller: Monday next week

System: Is that Monday 5th May?

Caller: Yes
Speech
A person talks with a system that has a spoken language application, like a train
timetable, a travel planner, or a phone service

Example:
System: How can I help you?
Caller: I'd like to go to Paris next Monday for two weeks

System: You will travel to Paris from Monday 5th May till Sunday 19th May
Caller: Yes

Example:
Google Voice or Apple Siri
Pen
Pen-based devices (light-pens or styluses) enable people to write, draw, select,
and move objects at an interface using hand drawing and writing skills
Touch
● Touch screens used in ATMs, ticket
machines, museum guides, … cell
phones
● It works By detecting the presence
and location of a person's touch on
the display
● Multi-touch screens can use swiping,
flicking, pinching, pushing, and
tapping
Air-based Gesture
Camera capture, sensor, and computer vision techniques have advanced such that
it is now possible to fairly accurately recognize people's body, arm, and hand
gestures in a room

Examples:

● Sony's Move uses Eye camera to track users

● Nintendo's Wii uses Wii remote and nunchuk controllers
● Microsoft's Kinect uses camera, a depth sensor and a multi-array microphone
Air-based Gesture
Air-based Gesture
Haptic
Haptic interfaces provide tactile feedback, by
applying vibration and forces to the person, using
actuators that are embedded in their clothing or a
device they are carrying, such as a smartphone
or smartwatch

Examples:

● Hug Shirt
● Steering wheels in car simulation
● Vibration in games
● Music Jacket using vibrotactile feedback
Multimodal
Combines different modalities, i.e. touch, sight, sound, speech. Different
input/outputs may be used at the same time, e.g. using voice commands and
gestures simultaneously to move through a virtual environment, or alternately
using speech commands followed by gesturing

Examples:

● Speech and gesture (e.g., Ford's SUV concept cars)

● Eye-gaze and gesture
● Pen input and speech
● Speech and Vision (e.g., Microsoft's Kinect)
Shareable
Shareable interfaces are designed for
more than one person to use at the
same time.

Examples:

● Smart Boards
● Interactive tabletops (e.g.,
Microsoft's Surface, Smart's
SmartTable)
Tangible
Tangible interfaces use sensor-based
interaction, where physical objects, e.g.
bricks, balls, and cubes, are coupled with
digital representations

Examples:

● Learning, design activities and

playfulness
● Planning tools for landscape and urban
planning
Urp System for planning tools for landscape and urban
planning
TinkerSheets system for exploring & solving warehouse
logistics
Wearable
Embed technologies on people in the clothes they wear. Jewelry, head-mounted
caps, glasses, shoes, and jackets to provide the user with a means of interacting
with digital information
Brain-Computer Interaction (BCI)
Provide a communication pathway between a person's brain waves and an
external device, such as a cursor on a screen or a tangible puck that moves via
airflow.

How it works:

1. The person is trained to concentrate on the task

2. Detect changes in the neural functioning in the brain using headset
3. Small electric signals rapidly move from neuron to neuron
Virtual Reality
The illusion of participation in a synthetic environment rather than external
observation of such an environment

How this is done

● Images are displayed stereoscopically to the users — through shutter glasses

● Objects within the field of vision can be interacted with via an input device like
a joystick.

Useful for simulation purposes

Common 3D Software toolkits is Alice (www.alice.org).

Virtual Reality
First Person VR Third Person VR

Flying/driving simulations and games, where it is In games, learning environments, and

important to have direct and immediate control simulations, where it is important to see a
representation of self with respect to the
environment
Augmented Reality
Virtual representations are superimposed on physical devices and objects

Examples:

● X-rays and scans, were overlaid on part of a patient's body

● Games (e.g., Pockemon Go)
● Head-up displays (HUDs) are used in planes to aid pilots when landing during
poor weather conditions
● Overlaid maps to assess the effects of flooding and traffic
Augmented Reality
Mixed Reality
Views of the real world are combined with views of a virtual environment, and
virtual objects interacts with the real world objects

More advanced than Augmented Reality

Examples:
● Enhanced environment: A television app placed in a comfortable spot for viewing
● Blended environment: A bed is "covered" and replaced by a mixed reality fruit stand
● Immersive environment: Walk through a famous building, museum, popular city
Enhanced Environment
Blended Environment
Immersive Environment
 VR AR

Digital environments Digital content on Digital content

that shut out the real top of your real interacts with your
world. world. real world.
 20 Interfaces
Command-based Air-based gesture
WIMP and GUI Haptic
Multimedia Multimodal
Visualization and dashboards Shareable
Web Tangible
Consumer electronics and appliances Wearable
Mobile Brain—computer interaction (BCI)
Speech Virtual reality
pen Augmented reality
Touch Mixed reality