CHAPTER 3: SOCIAL AND EMOTIONAL INTERACTION of an implicit cue is when a participant looks at their watch, signaling indirectly to the
articipant looks at their watch, signaling indirectly to the other
PART 1: SOCIAL INTERACTION
1.1 Introduction
People are naturally social and have developed various technologies to continue being social even
when physically separated. These technologies include smartphones, video chat, social media,
gaming, messaging, and telepresence.
1.2 Being Social
The use of social media and other social-based technologies has dramatically increased and changed
the way people connect and interact with each other. People now communicate online through:
 Texting,
 Emailing,
 social media platforms, and
 workplace communication tools
The adoption of social media has raised questions about how it has affected people's ability to
connect and interact with each other.
People switch between different modes of communication when planning and coordinating social
activities and the challenges that can arise from using different apps and threads.
A growing concern is being raised within society about the negative impact of excessive smartphone
use on people's well-being, especially in terms of reducing opportunities for face-to-face
conversation and empathizing with others. The average person in the UK spends more than a day a
week online, and checking one's phone has become a habit for many.
However, some technologies, such as voice assistants on smart speakers, have been designed to
encourage social interaction and joint use.
1.3 Face-to-Face Conversations
Talking is something that is effortless and comes naturally to most people. And yet, holding a
conversation is a highly skilled collaborative achievement, having many of the qualities of a musical
ensemble.
Such mutual greetings are typical. A dialogue may then ensue in which the participants take turns
asking questions, giving replies, and making statements. Then, when one or more of the participants
wants to draw the conversation to a close, they do so by using implicit or explicit cues. An example
participants that they want the conversation to draw a close. Either way, the first participant may
then offer an explicit signal, by saying, "Well, I have to go now. following the acknowledgement by
the other participants of such explicit and implicit signals, the conversation draws to a close, with a
farewell ritual. The different participants take turns saying, "Goodbye," "Bye," "See you," repeating
themselves several times until they finally separate."
What is a Conversational Mechanism?
- allowing people to know how to start and stop. Throughout the conversation, further turn-
taking rules are followed that enable people to know when to listen, when to speak, when to
stop to allow another speaker to speak.
Three basic rules describe by Sacks et al. (1978) a conversation analysis.
Rule 1: The current speaker chooses the next speaker by asking a question or inviting opinion or
making a request.
Rule 2: Another person decides to start speaking.
Rule 3: The current speaker continues talking.
The rules are assumed to apply so there is an opportunity for a change of speaker. If rule 1 is applied
if the person chooses to be a next speaker is not accepting rule 2 is applied to someone else taking
part in the conversation. If this does not happen rule 3 is applied and the current speaker continues
talking.
1.4 Remote Conversations
Definition of Remote Conversation
- a way of communicating with others online.
- remote collaboration through technological means is known as virtual or remote
communication. Data transfers between two devices that are not in the same area take place
throughout this process.
Remote Conversation as a mean of communication
 some of the first remote communication that was used in the early ages are
telephone. The telephone was invented by Alexander Graham Bell in the 19th
century, allowing people to communicate over distances. Other technologies were
later developed, including videophones in the 1960s-1970s and "media spaces" in the
late 1980s-1990s, which combined audio, video, and computer systems to enable
remote communication and interaction as if people were physically present.
Video conferencing has become mainstream thanks to the availability of cheap webcams and
cameras that are now embedded as a default feature in devices such as tablets, laptops, and
phones.
o Zoom or other video calls.
 o Phone calls.
o Virtual meetings. 1.5.2 Awareness
o SMS text messages. Awareness involves knowing who is around, what is happening, and who is talking with whom.
o Slack or Microsoft Teams conversations.
Peripheral and situational awareness are essential in technology-rich work domains, such as air
This development can also be described in terms of the level of telepresence achieved. traffic control and operating theaters. People who work closely together develop strategies for
coordinating their work based on an up-to-date awareness of what the others are doing. This is
Robots are designed for telepresence to allow individuals to remotely attend events and especially important for interdependent tasks, such as putting on a show.
communicate with others by controlling the robot.
1.5.3 Shareable Interfaces
Telepresence robots are being studied to determine if they can assist individuals with A number of technologies have been designed to capitalize on existing forms of coordination and
developmental difficulties to visit places such as museums remotely. They have also become a awareness mechanisms such as:
common feature at conferences, such as the ACM CHI conference, enabling remote attendance for o whiteboards
those unable to travel. o large touch screens
o multi touch tables
1.5 Co-presence to enable groups of people to collaborate while interacting at the same time with digital content.
Several studies have investigated whether different arrangements of shared technologies can help
co-located people work better together, with shareable interfaces providing more opportunities for
Technology has been deployed to enable more than one person to use them at the same time,
flexible collaboration.
enabling co-located groups to collaborate more effectively when working, learning, and socializing.
Research has shown that it is possible to design shareable technologies so that people can
Examples of commercial products include Smartboards and Surfaces, multi-touch, and Kinect. To
participate more equally. The interface should invite users to select, add, manipulate, or remove
understand how effective these technologies are, it is important to consider the coordination and
digital content from displays and devices. A user study showed that a tabletop that allowed group
awareness mechanisms already in use by people in face-to-face interactions.
members to add digital content by using physical tokens resulted in more equitable participation
than if only digital input was allowed via touching icons and menus.
1.5.1 Physical Coordination
When working closely together, people often use gestures to coordinate their actions.
Nonverbal communication is also used such as: 1.6 Social Engagement
o Nods
o Shakes Definition of Social Engagement
o Winks - It refers to participation in activities of social groups (Anderson and Binstock, 2012).
o glances - It involves some form of social exchange where people give or receive something from others.
o hand-raising - It is voluntary and unpaid.
Hand signals have evolved, with their own set of standardized syntax and semantics. - Increasingly, different forms of social engagement are mediated by the Internet.
Universal gestures such as:
o beckoning KEY POINTS
o waving
o halting hand movement  Social interaction is central to our everyday lives.
are also used by people in their everyday settings.  Social mechanisms have evolved in face-to-face and remote contexts to facilitate
conversation, coordination, and awareness.
Physical objects such as:  Talk and the way it is managed are integral to coordinating social interaction.
o wands and batons  Many kinds of technologies have been developed to enable people to communicate
can also facilitate coordination, as they can be used as external thinking props to explain principles, remotely with one another.
an idea, or a plan to the others. They can help collaborators gain a better overview of the group  Keeping aware of what others are doing and letting others know what you are doing
activity and increase awareness of others' activities. are important aspects of collaboration and socializing.
  Social media has brought about significant changes in the way people keep in touch people can use their nervousness or negative emotions to their advantage, while others
and manage their social lives. struggle to cope with them.
Andreas Komninos (2017) suggests that it is the autonomous system "telling" people to
PART 2: EMOTIONAL INTERACTION avoid these kinds of potentially humiliating or embarrassing experiences.

2.2 Emotions and the User Experience How emotions work provides a way of considering how to design for user experiences
o Don Norman (2005) suggests that being in a positive state of mind can enable people to
be more creative as they are less focused.
What is Emotion?
o In contrast, when someone is anxious or angry, they are more likely to be less tolerant.
- Emotions can be simple and short-lived or complex and long-lasting. Researchers have
described them in terms of being either automatic or conscious.
Automatic emotions - (also knowns as affect) happen rapidly, typically within a fraction of
The Model of Emotion and Behavior
a second and,
likewise, may dissipate just as quickly.
Conscious emotions - tend to be slow to develop and equally slow to dissipate, and they
are often the result of a conscious cognitive behavior, such as weighing the odds,
reflection, contemplation.

The concept of emotional interaction in the context of user experience design

o Emotional interaction is concerned with understanding and leveraging human emotions

to inform the design of different aspects of the user experience.

o However, deciding how to design an interface to elicit a particular emotional response is

not straightforward.

o By understanding how emotions influence behavior, designers and advertisers can create
more effective and engaging experiences for users. o Anthony Ortony et.al’s (2005) model of emotional design showing three levels: visceral,
Ways to understand how emotions affect behavior and how behavior affects emotions
behavioral, and reflective
include:
o examine how people express themselves and read each other's expressions
Visceral levels - are parts of the brain that are prewired to respond automatically to
o understanding the relationship between facial expressions, body language, events happening in the physical world.
gestures, and tone of voice.
o Emotional Skills - ability to express and recognize emotions, and what emotions Behavioral level - control everyday behavior
to express in a given situation.
Reflective level - involved in contemplating
o There is an ongoing debate about whether and how emotion causes certain behaviors.
Roy Baumeister et al. (2007) argue that the role of emotion is more complicated than a o One way of using the model is to think about how to design products in terms of the three
simple cause-and-effect model. levels.

Many theorists, however, argue that emotions cause behavior. Visceral design refers to making products look, feel, and sound good.
.
However, these responses can also lead to unpleasant experiences, such as sweating, Behavioral design is about use and equates to the traditional values of usability.
butterflies in the stomach, quick breathing, heart pounding, and feelings of nausea. Some
 Reflective design is about considering the personal value of a product in a particular
culture.
o contextual videos.
A number of online stores and travel agencies also began including automated virtual agents in the
form of cartoon characters who acted like sales agents on their websites. The agents appeared
above or next to text boxes where the user could type in their query. To make them appear as if
they were listening to the user, they were animated in a semi-human-like way.

2.3 Expressive Interfaces and Emotional Design Interfaces, if designed poorly, can make people sometimes feel insulted, frustrated, or threatened.
The effect can be to annoy them to the point of losing their temper. There are many situations that
cause such negative emotional responses.
Designers use various features such as:
These include the following:
o Images,
o When an application doesn't work properly or crashes.
o Sounds,
o When a system doesn't do what the user wants it to do.
o Colors,
o When a user's expectations are not met.
o Shapes,
o When a system does not provide sufficient information to let the user know what to do.
o icons, and
o When error messages pop up that are vague or obscure.
o virtual agents
o When a system requires users to carry out too many steps to perform a task, only to discover a
to create expressive interfaces that can elicit emotional responses from users.
mistake was made somewhere along the line and they need to start all over again.
Emotional icons were used in the past to convey the current state of a computer or phone, but now
more aesthetically pleasing forms of feedback are used, such as spinning colorful beach balls and
moving check marks. These new forms of feedback are more impersonal but still visually appealing. 2.5 Affective Computing and Emotional AI
Other ways of conveying expressivity include the following:
o Animated icons (for example, a recycle bin expanding when a file is placed in it Affective computing is concerned with how to use computers to recognize and express emotions in
and paper disappearing in a puff of smoke when emptied!) the same way as humans do (Picard, 1998).
o Sound effects indicating actions and events (such as a "click" for a window being - It involves designing ways for people to communicate their emotional states, through
closed, "schlook" for a file being dragged, or a "ding" for a new email arriving) using novel, wearable sensors and creating new techniques to evaluate frustration,
o Vibrotactile feedback, such as distinct smartphone buzzes that represent stress, and moods by analyzing people's expressions and conversations.
specific messages from friends or family.
- It also explores how affect influences personal health (Jacques et al., 2017).
2.4 Annoying Interface

Emotional AI has emerged as a research area that seeks to automate the measurement of feelings
In many situations, interfaces may inadvertently elicit negative emotional responses, such as anger.
and behaviors by using AI technologies that can analyze facial expressions and voice in order to infer
This typically happens when something that should be simple and easy to use turns out to be
emotions.
complex.
The main techniques and technologies that have been used to do this are as follows:
The most common examples are:
o Cameras for measuring facial expressions
o remote controls,
o Biosensors placed on fingers or palms to measure galvanic skin response (which is used to
o printers,
infer how anxious or nervous someone is as indicated by an increase in their sweat)
o digital alarm clocks, and
o Affective expression in speech (voice quality, intonation, pitch, loudness, and rhythm)
o digital TV systems.
o Body movement and gestures, as detected by motion capture systems or accelerometer
sensors placed on various parts of the body.
That does not mean that developers are unaware of such user usability problems. Several methods
Six fundamental emotions are classified based on the facial expressions that Affdex collects:
have been devised to help the novice user adapt quickly and become familiar with technology.
o Anger
o Contempt
These methods include:
o pop-up help boxes, and o Disgust
 o Fear One is called All the Feels - which provides an overlay of biometric and webcam-
o Joy derived data of a streamer onto the screen interface
o Sadness (Robinson et al., 2017).

A dashboard provides a visualization of the streamer's heart rate, skin conductance, and emotions.
This additional layer of data has been found to enhance the spectator experience and improve the
They provide the data that determines the type of facial expression being shown, in terms of connection between the streamer and spectators.
detecting the presence or absence of the following:
o Smiling 2.6 Persuasive Technologies and Behavioral Change
o Eye widening
o Brow raising A diversity of techniques has been used at the interface level to draw people’s attention to certain
o Brow furrowing kinds of information in an attempt to change what they do or think.
o Raising a cheek Some of the methods that are being deployed on a computer or smartphone interfaces include:
o Mouth opening o Pop-op ads,
o Upper lip raising o warning messages,
o Wrinkling of the nose o reminders,
o prompts,
Note: A user screwing up their face when an ad pops up suggests they feel disgust. o personalized messages, and
If a user starts smiling, it suggests they are feeling happy. o recommendations

The various techniques that have been developed have been referred to as persuasive design (Fogg,
The emotional AI software perceives if a driver is angry and then suggests an intervention. 2009).
For example, a virtual agent in the car might suggest to the driver to take a deep breath and play They include:
soothing music to help relax them. In addition to identifying particular emotions through facial o Enticing
expressions (for example, joy, anger, and surprise), Affectiva uses particular markers to detect o cajoling, or nudging someone into doing something through the use of persuasive
drowsiness technology.
Other indirect methods that are used to reveal the emotional state of someone include:
o eye-tracking,
o finger pulse, HAPIfork - a device that was developed to help someone monitor and track their eating habits.
o speech, and If it detects they are eating too quickly, it will vibrate, and an ambient light will
o the words/phrases they use when tweeting, chatting online, or posting to Facebook (van appear at the end of the fork, providing the eater with real-time feedback intended
den Broek, 2013). to slow them down.

The level of affect expressed by users, the language they use, and the frequency with which they
express themselves when using social media can all indicate their mental state, well-being, and
aspects of their personality (for instance, whether they are an extrovert or introvert, neurotic or
calm, and so on).
The most popular site is Twitch; millions of viewers visit it each day to watch others compete in
games, such as Fortnite. The biggest streamers have become a new breed of celebrity, like
YouTubers. Some even have millions of dedicated fans.
Mobile apps and personal tracking devices are available to help people monitor their fork behavior
and change it based on the data collected and displayed back to them. Data is collected about how
long it takes them to finish their meal, the number of fork servings per minute, and the time
between them.
These devices include Fitness trackers, for example:
o Fitbit
o and weight trackers

Various tools have been developed to enhance the viewers' experience.

  Anthropomorphism is the attribution of human qualities to objects.
 Robots are being used in a variety of settings, including households and assisted-living
homes

2.7 Anthropomorphism

What is Anthropomorphism?
Anthropomorphism - is the act of attributing human-like qualities or behavior to non-
human entities, such as animals, objects, or even gods.

Anthropomorphism is the tendency of individuals to assign human-

like characteristics to objects or animals.

Examples:
o Books like "Alice in Wonderland," "Peter Rabbit," and "Winnie-the-Pooh" are well-known
anthropomorphism works.
o Individuals sometimes treat inanimate objects as if they were living beings, such as talking
to computers like they are humans, referring to robot cleaners as pets, and giving names
to devices like routers and mobile phones.

Additional Information
ActiMates, an early commercial product, aimed to promote learning through play for children. One
of the first products, Barney the dinosaur, utilized human-like speech and movement to motivate
play in children. The toys were designed to respond to children and provide feedback while
watching TV or engaging in computer-based activities together.

Furnishing technologies with human-like attributes can make them more fun to interact with and
can motivate people to engage in activities like learning.

KEY POINTS
 Emotional aspects of interaction design are concerned with how to facilitate certain states
(for example, pleasure) or avoid certain reactions (such as frustration) in user experiences.
 Well-designed interfaces can elicit good feelings in people.
 Aesthetically pleasing interfaces can be a pleasure to use.
 Expressive interfaces can provide reassuring feedback to users as well as be informative
and fun.
 Badly designed interfaces often make people frustrated, annoyed, or angry.
 Emotional AI and affective computing use AI and sensor technology for detecting people's
emotions by analyzing their facial expressions and conversations.
 Emotional technologies can be designed to persuade people to change their behaviors
or attitudes.