All Slides From THIS YEAR Merged

CSE 3210: Collaborative AI:
Course Introduction
Pradeep K. Murukannaiah
Catholijn M. Jonker
Myrthe L. Tielman
Luciano Siebert
Interactive Intelligence, INSY, TU Delft

1
Welcome!
Today we are going to talk about …
• Course organization
• What is collaborative AI?

– Challenges
– Opportunities
• Course contents
2
Course Organization
3
Teaching Team: Lecturers
Pradeep Catholijn Jonker

Murukannaiah C.M.Jonker@tudelft.nl
P.K.Murukannaiah@tudelft.nl
Myrthe L. Tielman Luciano Siebert

M.L.Tielman@tudelft.nl L.CavalcanteSiebert@tudelft.nl
4
Teaching Team: Course Assistants
• Enrico Liscio (Head TA)
• Florentin Arsene
• Nathan Ordonez Cardenas
• Radek Kargul
• Simran Karnani
• Imko Marijnissen
• Bram Renting
• Carolina Jorge
• Ruben Verhagen
5
Communication Channels
• Stackoverflow (TU Delft instance)
– All content-related questions and discussions
– We count on your contributions!
• Email collabai-cs-ewi@tudelft.nl
– For questions concerning individuals
– Feedback on the course
• Brightspace
– All important announcements
– Discussion forums, e.g., for team formation in the first week
6
Course Organization
• Two lectures per week

– Tuesday 10.45–12.45 Hybrid
– Thursday 08.45–10.45 Hybrid
• Two (shared) lab sessions per week

– Monday 13.45–17.45 Hybrid
– Wednesday 08.45–12.45 Hybrid
7
Hybrid Teaching
• Lectures
– Enroll in the queue for attending lectures on campus
• The queue for a lecture closes 4 days in advance
– Use the fixed Zoom link for attending online
• Labs
– Separate queues for on-campus and online support
– Detailed instructions on Brightspace.
8
Collaborative AI:
What, Why, and How?
LO: Compare centralized and

collaborative AI paradigms
9
Activity: Centralized or Collaborative?
• Provide examples for centralized AI systems

– Go to www.menti.com and use the code 6488 8584
– https://www.menti.com/cdvhhkpwh3
• Provide examples for collaborative AI systems

– Go to www.menti.com and use the code 8639 6371
– https://www.menti.com/u4axhuskd8
Include your own examples or pick the examples from

the next slide
10
Consider Some AI Systems…
Internet and e-commerce Healthcare Search and Rescue
CT Scan Analyzer Robot swarm
Netflix (movie recommender)
Fitbit
Yelp (POI recommender)
Google (search) Transport Manufacturing
Facebook (social networking) Smart traffic control Industrial robots
Alexa (virtual assistant) Autonomous vehicle
Education Gaming Law

Virtual tutor AlphaGo COMPAS
(predicts relapse likelihood)
IBM Debater
11
Artificial Intelligence (AI)
AI is the study and design of Intelligent agents
Intelligent agents:
• Have own thread of control, continually executing
• Are autonomous
• Exhibit proactive, reactive, and social behavior
• Can be physical or virtual, embodied or software-only
12
Centralized AI Systems
Centralized AI System  Single-Agent System
Centralized computation
and control 
Source: https://pixabay.com/photos/robot-softbank-pepper-tablet-white-1695653/
 Distributed computation
yet centralized control!
Source: https://www.flickr.com/photos/ibm_media/33838065805
13
Collaborative AI Systems
Collaborative AI System  Multi-Agent System
• Decentralized computation and control
http://groups.csail.mit.edu/mers/projects/human-robot-teamwork-in-manufacturing/ https://www.ctif.org/news/human-robot-teamwork-technology-robot-assisted-dis https://www.ledgerinsights.com/lg-cns-autonomous-cars-blockchain-5

aster-response-lecture-ivana-kruijff g-decentralized-identity/
14
Collaborative AI
• Collaborative AI is
the study and design of multi-agent systems
– Example: https://youtu.be/GdUkQObeKbY
15
Collaborative AI: Why? (1)
• Each agent has incomplete information or

capabilities for solving a problem
• There is no system for global control
• Data is decentralized
• Computation is asynchronous
16
Collaborative AI: Why? (2)
Facilitate hybrid (human–AI) intelligence
• Humans assisting machines

– Training, explaining, sustaining
• Machines assisting humans

– Amplifying, interacting, embodying
17
Collaborative AI:
A Sociotechnical Architecture
18
Collaborative AI: How?
You tell me …
Participate in an activity on
Brightspace  Quizzes and Activities
https://brightspace.tudelft.nl/d2l/le/content/399757/v
iewContent/2324105/View
19
What makes engineering a collaborative
AI system challenging?
Go to www.menti.com and use the code 9454 8532
https://www.menti.com/rjzqvw8ssa
20
Collaborative AI Creates New
Opportunities
Example Use Case: Activity Recommender
User: Alex
Goal: Lead a healthy life
• Alex’s activity tracking agent can talk to his physician’s agent to

receive data about Alex’s bloodwork (multi-agent)
• The two agents can ask the doctor to intervene if serious advice is to
be given (hybrid intelligence)
• The agents can talk to Alex’s family members to motivate Alex to

exercise more (hybrid intelligence)
21
Collaborative AI: Who Cares?
22
Collaborative AI: The Future of AI?
• DARPA Robotics Challenge (2015)
– Promoting innovation in human-supervised robotic technology for disaster-
response
• Microsoft Research: The Malmo Collaborative AI Challenge (2017)

– instill machines with the ability to work together with both other agents and humans
• The Hybrid Intelligence Center (2020—2029)

– combination of human and machine intelligence, expanding human intellect
instead of replacing it
• The Humane AI Net (2020—2023)

– European Network of Human-Centered Artificial Intelligence
23
Course Content
24
Learning Objectives
After completing this course, you should be able to:
• Compare centralized and collaborative AI paradigms.
• Describe the principles of automated negotiation, facilitating cooperation.
• Create an automated negotiating agent in the GeniusWeb platform
• Apply co-active design to solve a collaborative AI problem.
• Describe the conceptual underpinnings of agent coordination mechanisms.
• Describe the conceptual underpinnings of agent interaction protocols.
25
Course Contents and Schedule
Emerging topic! Suggestions are very welcome!

26
Course Assessment
• 50% Exam
– A combination of MCQs and open-ended questions
• 50% Practical: Two assignments

– Negotiating Agent: 0.5 weight (three weeks)
– Collaborative Agent: 0.5 weight (three weeks)
• Requirements to pass the course

– The exam >= 5.0
– Each practical assignment >= 5.0
– The total >= 5.75
27
Practical Assignment 1: Negotiating Agent
• Task: Develop an automated negotiating agent

• Programming language: Java or Python 3
• Platform: GeniusWeb
• Agents compete in a class tournament
28
Practical Assignment 2: Collaborative
Agent
• Task: Develop an agent to
solve the Block Worlds for
Teams (BW4T) task
• Programming language:
Python 3
• Platform: MATRX
• Efficient and
generalizable agents score
high!
29
Reading Material and Resources
• Multiagent Systems, 2nd Edition.
Edited by Gerhard Weiss
– Chapters 1, 2, 4, 6, 8, and 9
– E-book accessible via TUD login
• Research papers and handbooks

– Brightspace: Resources and tools
• Lecture slides
30
Summary
• Collaborative AI: What, why, and how?
– Compare centralized and collaborative AI paradigms
• Course organization
• TODOs for you

– Enroll in a team on Brightspace
• Deadline: Thursday, 10 Feb
– Work on the Negotiating Agent project

• First shared lab: Monday, 14 Feb, 13.45—17.45
• Next three lectures: Negotiation
31
Negotiation
Interactive Intelligence Group,TU Delft
Tuesday, March 8, 2022 1

Outline
• Why negotiate ?
• What Kinds of Negotiations Exist?
• Guidelines for Negotiation
• Negotiation Analysis
• Do it yourself !

Negotiate or not?
Why Negotiate?
• Positive situation:
• Win-win opportunities
• Short term relations
• Long term relations
• Conflict:
• Power play (or war) will not lead to a solution
• All parties need a solution
• All parties need to at least pretend to be willing
3
Definitions (1)
• Party or Stakeholder: someone that has a stake in the

outcome of the negotiation.
• Mediator: someone that has no stake in the outcome of the
negotiation, guards the negotiation process, aims for a fair
outcome to all parties.
• Issue: something that the parties need to agree on for a
successful negotiation.
• Open, closed or in between:
• Open: all parties provide full insight in their preferences
and interests
• Closed: only bids are exchanged

Kinds of Negotiations
What Kinds of Negotiations Exist?

Negotiations Come in Many Different Kinds…
Number of Parties Number of Issues

• 1 on 1 • 1
• 1 on m • N>1
• n on m
Mediated? Open?
5
Bilateral Negotiation
• Negotiation between two parties

• A buyer and a seller negotiate over cost and delivery time
• A wife and husband negotiate over the details of the party
that they will organize next week
Consumer Provider
Agreement

Multiple bilateral negotiation
• One party may negotiate with different parties
simultaneously.
• E.g. one buyer negotiates with multiple sellers
• The negotiation with one party may have an influence on
the negotiation with another party.
Service Service Service

Provider 1 Provider 2 Provider 3
Agreement 2
Agreement 1
Agreement 3
Service
Consumer
Multilateral Negotiation
• Negotiation among more than two parties
• All parties mutually agree on the final decision/offer
• A group of service providers would like to give a service together and
they need to mutually agree on some issues.
• A group of friends are planning to go on holiday together and they
need to find a joint agreement on some issues such as location,
duration etc.
Alic
Agreement
Bob e
Mary

Negotiator Guidelines
Principles That Help You Become a

Good Negotiator
From Positional Bargaining to …
… to Win-Win Negotiation
Position - the first demand of a party, often vocal and self-centered. Requests for cash ($) compensation is often a position.
Interest - the deeper motivations underlying a party’s position, and which offers greater room to explore a range of options
from which to compile a solution that satisfies the most number of parties.
9
Source: http://www.snsi.org/negotiation.php
Positional Bargaining
• Holding a fixed position of what you want regardless

of any underlying interest
How much do you want for
this painting?
500 Euro
Himm, I can pay
I cannot accept you 100 Euro.
less than 300 Euro

Negotiation: bidding
11
Negotiation: underlying interests
12

Good Negotiator
Principle 1
Build trust through mutual understanding and
meaningful communication
How?
• Ensure that all parties are identified and represented
• Develop understanding through active listening
• Remember that perceptions can be real, since they
may drive decision-making
13
14
15

Good Negotiator
Principle 2
Focus on revealing underlying interests rather
than positions
How?
• Adopt consensual negotiation style, not a positional
• Identify peoples underlying concerns, fears,
motivations, deadlines, aspirations & values
• Create room to explore a wide range of options
16
17

Good Negotiator
Principle 3
Widen the options for finding a solution
How?
• Use creativity and engage in joint problem solving
• Apply brainstorming techniques
• Assess the uncertainty and risks arising from the
proposed solution
18

Good Negotiator
Principle 4
Reach agreement that satisfies interests and adds
value for all parties
How?
• Ensure that parties implement commitments
• Incorporate a grievance mechanism to address
concerns over implementation
• Evaluate resulting relationships and outcomes
19
20
Phases of Negotiation
Private preparation: Joint exploration:

Domain modelling Domain modelling
User modelling User preferences
Opponent modelling Opponent modelling
Alternatives Relationships
Closing: Bidding:
Contract Strategy determination
Relationships Bid evaluation
Reflection Next bid determination
Bidding analysis
21
Definitions
Definitions (2)
• Position: the first demand of a party, often vocal and self-centered.

Requests for cash ($) compensation is often a position.
• Interest: the deeper motivations underlying a party’s position, and
which offers greater room to explore a range of options from which to
compile a solution that satisfies the most number of parties.
• Preference profile: the preferences an agent has regarding the
possible outcomes of the negotiation. These refer to the importance of
issues, but also to the values per issue. Preference profiles can be
qualitative or quantitative.
• Bid: for a set of issues of the negotiation attach a proposed value
• Partial bid: not all issues are assigned a value
• Complete bid: all issues are assigned a value. To find the best
possible outcomes, you are advised to only bid complete bids.
22
Definitions
Definitions (3)
• Utility function: a quantitative representation of a preference

profile. It maps bids to utility values. For easy of computation the
range of utility values is often [0, 1]. Typically utility functions are
only defined for complete bids.
• Reservation value: in terms of utility, it is a utility value below
which bids are unacceptable. Each party should determine its
reservation value.
• BATNA: Best Alternative To No Agreement – If the negotiation
fails, this is what you can fall back on.
• The better your BATNA, the higher you can set your reservation
value.
• The better your BATNA, the more relaxed you will be, the easier
it will be creative, to think “out of the box”
23
How To: Prepare!
Prepare!
• Read books on negotiations!

• Which interests are at stake? For me, for other parties?
• Short term and long term interests?
• How important is success to the participants?
• Create preference profiles
• What would I prefer?
• What would the others prefer?
• Convert preference profiles to utility functions for all agents (overall
function that assign a grade to a bid)
• What is normal in this domain (e.g., the going prices for real estate)?
• What is my reservation value?
• Make sure of a BATNA
24
How To: Prepare!
Prepare (2)
•Aspects •What is our relation?

•Interests mentioned •Business
•Hidden agenda •Friend / family/ enemy
•Experience •One time, long term
•Power differences
•Network
•(unsolved) past
•Emotions
•Which parties participate?

•What do I know
•about myself? •Which protocol?
•about other(s)?
March 8, 2022 25
How To: Joint Exploration
To Be Open or Not To Be Open…

The Negotiation Dilemma: Share Information !/?
Being Open in a Negotiation: Not Being Open in a Negotiation:

• Putting all your cards on the table, • Keep all your cards to yourself, not
sharing all information sharing any information.
• Outcome can be checked for • Outcome cannot be judged for
fairness. fairness.
• But... danger of exploitation by • But... how to reach a win-win
other party! outcome!?
Reciprocate:
Share Information if Other Reciprocates!
26
How To: Joint Exploration
Negotiation Atmosphere
27
Negotiation Styles
Mastenbroek’s
advice
Activity Scale
Interests
lenient hard
Power
bending, dominant
subdued
Climate
jovial hostile
personal formal
Flexibility
exploring repetitive
avoiding
What is your style?

Can you switch styles according to circumstances? 28
Which Style does Your Opponent Adopt?

How To: all phases
Conflict Handling Styles
Thomas-Kilmann Conflict Mode Instrument

What is your style?
Which is your Opponent’s style?
29
How To: all phases
Styles versus Tactics
30
How To: Evaluate
Analysis of Negotiation Results

• How do you rate a bid?
• Balance between what you give and take
• Utility: grade between 0 and 1
• Comparing results
give take
March 8, 2022 31
Definitions
Utility: Determine the Value of a Bid

• Utility is like a grade, determined as a weighted sum
of the evaluation values of each issue.
• The utility of <Sea, Summer, One-week>=

0.5 * U1(Sea) + 0.3 * U2 (Summer) +0.2 * U3(One-Week)
= 0.5*0.9 + 0.3 *0.7+0.2*0.5 = 0.76
where w1 (Location)= 0.5, w2 (Season)= 0.3, w3(Duration) =0.2

U1(Sea)= 0.9, U2 (Summer) =0.7, U3(One-Week)=0.5
32
How To: Evaluate
Utility
• n issues: • 2 issue example
b = (v1, …, vn) G: g1, g2
v1  D1, …, vn  Dn H: h1, h2
D = D1 x … x Dn • Agent A
• For every bid b  D
EA,1(g1) = 1
Utility of A: UA(b)
EA,1(g2) = 0.5
Utility of B: UB(b)
EA,2(h1) = 0.3
• UX : D  [0, 1]
EA,2(h2) = 1
• UX(b) = j wX,jEX,j(b)
wA,1 = 0.7
• wX,j : weight issue j wA,2 = 0.3
• EX,j(b) : evaluation of value of issue j UA((g1, h1)) = 0.7*1 + 0.3*0.3 = 0.7
in bid b + 0.09 = 0.79
How To: Evaluate
Utility (2)
• 2 issue example
• 2 issue example
G: g1, g2
G: g1, g2
H: h1, h2
H: h1, h2
• Agent B
• Agent A
EB,1(g1) = 1
EA,1(g1) = 1
EB,1(g2) = 0.6
EA,1(g2) = 0.5
EB,2(h1) = 1
EA,2(h1) = 0.3
EB,2(h2) = 0.4
EA,2(h2) = 1
wB,1 = 0.4
wA,1 = 0.7
wB,2 = 0.6
wA,2 = 0.3
UB((g1, h1)) = 1
UA((g1, h1)) = 0.79
UB((g2, h1)) = 0.84
UA((g2, h1)) = 0.44
UB((g1, h2)) = 0.64
UA((g1, h2)) = 1
UB((g2, h2)) = 0.48
UA((g2, h2)) = 0.65
How To: Evaluate
Result space
• Result of a bid:
(g1,h2) • R : D  ([0, 1], [0, 1])
• R(b) = (UA(b), UB(b))
1
• 2 issue example
(g1,h1) G: g1, g2
(g2,h2)
Utility A
H: h1, h2
• Bid UA UB
(g2,h1) (g1,h1) 0.79 1
(g2,h1) 0.44 0.84
(g1,h2) 1 0.64
(g2,h2) 0.65 0.48
0 Utility B 1
How To: Evaluate
The bidding from your own

perspective
March 8, 2022 36
How To: Evaluate
Comparing results
You think The other partner thinks
give take give take
March 8, 2022 37
How To: Evaluate
How Well Did You Do?

Kalai-Smorodinsky
(Ubuyer=Useller)
Utopia
1
“Non-Existent”
Buyer
Pareto Optimal Frontier
Nash product
max(Ubuyer  Useller)
0 Seller
38
Definitions
Definitions
• Let UA and UB be the utility functions of party A and
party B. Both functions are defined on the domain D of
all possible bids that the agents can make, mapping D to
the range [0, 1]. Thus,
UA : D  [0, 1] and UB : D  [0, 1].
• The Pareto Optimal Frontier is the set of bids, such
that there is no other bid that is better for at least one
party, without making things worse for the other parties.
Thus, if two agents A and B are involved,
POF = { b  D | b’  D : bb’  ( UA(b’) < UA(b) 
UB(b’) < UB(b) ) }
39
Definitions (2)
• Nash Product: the bid (or bids) that maximizes the

product of utilities of the parties, under the assumption
that all utility values are positive. If that is not the
case, you have to transpose the utility space to positive
values.
• The equal proportion of potential line (EPP) is the
line from (0,0) to (1,1).*
• Kalai-Smorodinsky: maximizes the minimum of the
utilities of the parties. To be found at the intersection
of POF and EPP, i.e., the bid(s) closest to that
intersection.
*It is a bit more technical than this if the domain is not continuous
How To: Evaluate
Result space
• Result of a bid:
(g1,h2) • R : D  ([0, 1], [0, 1])
• R(b) = (UA(b), UB(b))
1
• 2 issue example
(g1,h1) G: g1, g2
(g2,h2)
Utility A
H: h1, h2
• Bid UA UB
(g2,h1) (g1,h1) 0.79 1
(g2,h1) 0.44 0.84
(g1,h2) 1 0.64
(g2,h2) 0.65 0.48
0 Utility B 1
Relation to Artificial Intelligence
Negotiation Has Entered the Digital World:

• e-mail, e-commerce, …
Artificial Intelligence:
• Intelligent decision making
• Engineering Heuristic Approaches
for Machines
• Engineering Cognitive Skills used
in Negotiation
Russell & Norvig: 2 – 6, 10, 13 - 22
The Future:
• Enhanced negotiations by using the Negotiation Support Systems
42
Some classics on negotiation from a
human perspective
• Harvard Business Review Staff, (2003). Negotiation (Harvard
Business Essentials Series). Harvard Business School Press.
• Lewicki, R.J., Saunders, D.M., and Minton, J.W., (1999),
Negotiation: readings, exercises, and cases. Boston, McGraw-
Hill/Irwin.Raiffa, H. (1982). The art and science of
negotiation, how to resolve conflicts and get the best out of
bargaining. Cambridge, Mass., Belknap Press of Harvard
University Press.
• Shell, G.R., (2000), Bargaining for advantage: negotiation
strategies for reasonable people. Penguin Books.
March 8, 2022 43
Classics on the way to automated
negotiation
• Rosenschein, J.S., and Zlotkin, G., (1994). Rules of

Encounter: Designing Conventions for Automated
Negotiation Among Computers. MIT Press.
• Raiffa, H., Richardson, J., and Metcalfe, D., (2002).
Negotiation Analysis: The Science and Art of
Collaborative Decision Making, Cambridge, MA:
Belknap Press of Harvard University Press.
44
State of the art and classics for
automated negotiation
• Baarslag T, Fujita K, Gerding EH, Hindriks K, Ito T, Jennings NR, Jonker CM,
Kraus S, Lin R, Robu V, Williams CR, (2013). Evaluating Practical Negotiating
Agents: Results and Analysis of the 2011 International Competition, Artificial
Intelligence, 198, pages:73 - 103, issn: 0004-3702, doi:
10.1016/j.artint.2012.09.004.
http://www.sciencedirect.com/science/article/pii/S0004370212001105?v=s5
• Jonker CM, Hindriks K, Wiggers P, Broekens JD, (2012). Negotiating Agents,
AI Magazine,
33. http://www.aaai.org/ojs/index.php/aimagazine/article/view/2421
• Luo, Xudong, Miao, C, Jennings, Nick, He, Minghua, Shen, Z and Zhang, M
(2012) KEMNAD: A Knowledge Engineering Methodology for Negotiating
Agent Development. Computational Intelligence, 28, (1), 51-105.
45
Supporting humans
• Michele Joy Gelfand and Ya'akov Gal. The brave new negotiating world:
Challenges and opportunities. In B. Goldman & D. Shapiro (Eds.) The psychology
of negotiations in the21st Century Workplace. SIOP Frontiers Book, Routledge,
2013.http://www.routledge.com/books/details/9780415871150/
• Ya'akov Gal, Sarit Kraus, Michele Gelfand, Hilal Khashan, Elizabeth Salmon.
Negotiating with People across Cultures using an Adaptive Agent. ACM
Transactions on Intelligent Systems and Technology 3(1), Article 8, 2011.
http://www.eecs.harvard.edu/~gal/Papers/culture10.pdf
• http://ii.tudelft.nl/negotiation/index.php/Pocket_Negotiator
• Jonker C.M., Aydogan R, Baarslag T, Broekens J, Detweiler C, Hindriks K.V.,
Huldtgren A, Pasman W, (2017). An Introduction to the Pocket Negotiator: A
General Purpose Negotiation Support System. In: Multi-Agent Systems and
Agreement Technologies, Multi-Agent Systems and Agreement Technologies –
Proceedings of the 14th European Conference, (EUMAS 2016), LNAI 10207,
Springer International Publishing, pp:13—27.
• The InterNeg Research Centre: http://interneg.concordia.caa
46
Formalizing Negotiations
Interactive Intelligence Group,TU Delft
* 1
Outline
• Recap last week

• Domain model
• Preferences
• Utility functions
• Negotiation Analysis
* 2
Recap
• Why negotiate: joint decision needed, hidden

information, cannot be decided otherwise (power,
deliberation)
• Principles for human negotiators
• Phases of negotiation
• Definitions
• Styles: negotiation & conflict handling
* 3
How To: Evaluate
How Well Did You Do?

Kalai-
Smorodinsky
Utopi
(Ubuyer=Useller)
a
1
“Non-
Existent”
Buyer
Pareto Optimal
Frontier
Nash product
max(Ubuyer ⋅
Useller)
0 Seller
4
Social Welfare (Fairness)
Simple notion of social welfare

argmaxa∈F{Σp∈P up(a)}
Egalitarian Point (aka Rawls Point):

argmaxa∈F {argminp∈P up(a)}
Utilitarian Point (aka Kalai-Smorodinsky Point)

argmina∈F {ra − req}
5
When no equal utility solution exists
Only three possible outcomes, and no one

likes outcome B.
Enlarge the outcome space with lottery points.

[p1, A; 1-p1, C]
…
[pn, A; 1-pn, C]
New point D = [ 0.5, A; 0.5, C]
C
How to set the probabilities?
What will the negotiation be about if you
B
do this? What is a fair outcome then?
0
Compare to the literature on “The Battle of the Sexes”.

Domain
• Validity and Added value: The added value of using
artificial intelligence techniques highly depends on the
quality/validity of the model:
• Weak model → weak support by the technique
• Faulty model → faulty advice
• Transparency: Model should be transparent for the
user of technique
• Trust
• Understanding the advice
* 7
Domain model
• For the negotiations we do we need
• A set of issues X, and for each issue x ∈ X a set of
values for each of those issue: V(x)
• Another word for issues is attributes.
• For each stakeholder we need a preference profile for
this domain
• If repeated negotiations in a domain, then apply ML to
build and extend the domain and cluster preferences
to a set preferences that occur most.
• Learn from other negotiating agents by sharing
anonymized negotiation logs.
* 8
Domain Modelling
• Domain modelling is difficult for humans:
• Constructive process: humans become aware of them when
working with the material/problem. For negotiation this
implies that they might realise that something is an issue they
would like to negotiate about when the other party makes a
remark about it.
• Lack of experience in structuring problems
• Lack of formal techniques (math/logic, data structures)
• Can the stakeholder understand your model?
• An exam question might be: construct and explain a
domain model for the following domain. <followed by
an informal description of the domain>
* 9
Preferences
• Given a finite set of issues X, and i and j ∈ X and sets

of values for each of those issues: V(i), V(j)
• Stakeholders have preferences over the values per
issue and over combinations (bids) of values for a
range of issues, expressed by a preference relation <p
• Suppose a, b ∈ V(i), for some issue i, then a <p b
expresses that the stakeholder prefers b over a.
• Furthermore, suppose that, c, d ∈ V(j), for some j ≠ i,
• Then similarly (a, c) <p (b, d), expresses that the
combination (b,d) is preferred over (a,c).
* 10
Preferences
• Preference Independence: A set of attributes Y ⊂ X is
preferentially independent of its complement X-Y when the
preference order over outcomes with varying values of attributes
in Y does not change when the attributes of X-Y are fixed to any
value.
• Independence allows for linear additive utility functions.
Given a set of attributes X,

•an additive value function vX = Σx ∈ X (λx vx), where
Σ x ∈ X λ x =1, and for all x∈X: λx ∈[0,1]
Otherwise more complex functions are needed.
* 11
Preference elicitation
• Preferences are constructive in nature: humans

become aware of them when working with the
material/problem. For negotiation this implies that
they might realise they have a preference for an issue
when the other party makes a remark about it.
• You need a good elicitation technique for agents
supported humans in negotiation preparation (see
literature).
* 12
The Pocket Negotiator
http://ii.ewi.tudelft.nl:8080/PocketNegotiator/index.jsp
• Domain specification
• Preference elicitation
• Underlying interests
• Weights of issues
• Evaluation functions of issues
• Bidding support
• What to bid
• When to stop
* 13
IAGO platform – Group Jon Gratch
* 14
Additional literature
• Preference elicitation
Chen, L., & Pu, P. (2004). Survey of preference elicitation
methods (No. EPFL-REPORT-52659).
* 15
Automated Negotiation
Catholijn M. Jonker
Ongoing collaboration with: Tim Baarslag and Reyhan Aydogan
1
Contents of these slides
• Recap: Formalizing Negotiations
o Domain models
o Preferences and preference elicitation
o Analysing results
• Protocols
o bilateral
o Multi-lateral
• Negotiation strategies
o BOA framework
o An overview of important strategies
• Analysis of Negotiation Dynamics
• Using the BOA framework
• Prepare for Tutorial and Test
2
Why negotiation is difficult for
humans?
• Leaving money on the table

• Sub-optimal outcomes for both sides.
• Bounded rationality
• Outcome space might be too big.
• Emotions
• May have negative effect on acting rationally
• Some people are too shy to ask what they want.
Saturday, September 12, 2020 3
Automated Negotiation
• Intelligent software agents negotiate on behalf of their

users.
• Possible application domains: e-commerce, politics, cloud-
computing, crisis management system, task allocation, etc.
Consumer Producer
Agent Agent
Asking for a service
Offering an alternative service

Custom Provider
er ………
How can a software agent negotiate
its user’s behalf?
• Evaluating bids
• Reasoning on its user’s
preferences
• Employing a negotiation strategy
• Which action the agent will take
• How the agent will generate its offer
• When the agent will accept the
opponent’s counter offer
• Communicating with other agent (s)

based on predefined rules
(negotiation protocol)
For the purpose of Analysing
Negotiation strategies: Normalization
• You need first to normalize all evaluation functions
(also named valuation functions in last week’s slides).
I discuss it here for discrete domains:
• Pick the item of the issue’s range that has
maximum value, the normalized evaluation function
for that issue maps that item to 1.
• For all other items in the range normalize in
proportion to the maximum value to determine it’s
normalized value.
6
Normalization (2/3)
• Formally: Given issue j with range Dj, and valuation
function vj
• Let m=max({vj(x)| x ∈ Dj}),
• We define the normalized evaluation function
ej : Dj → [0,1] by: ej(x)=vj(x)/m
• Then to determine the normalized utility of a bid:

• Multiply the normalized evaluation of the issue wrt
the bid with the weight of that issue
• Do this for all issues
• Add it all up
7
Normalization (3/3)
• Formally:
• Let D = D1 x D2 x …x Dn be the cartesian product of the
ranges of all issues. D is also the bid space.
• Let wj be the weight of issue j, for all j: 1 ≤ j ≤ n.
• ∑1 ≤ j ≤ n (wj = 1)
• Let ej be the normalized evaluation function of issue j.
• The normalized utility function u: D → [0,1] is defined
by: u(b) = ∑1 ≤ j ≤ n (wj * ej(bj) ), where bj is the
projection of b on issue j.
8
NEGOTIATION PROTOCOLS
9
Negotiation Protocol
• A negotiation protocol governs the interaction between

negotiating parties by determining
• how the parties interact/exchange information
• “ who can say what and when they can say it ”
• when the negotiation ends
• Bilateral Negotiation
• Alternating Offers Protocol (Rubinstein 1982)
• Multiparty Negotiation
• Mediated Single Text Protocol
10
Alternating Offers Protocol (Rubinstein 1982)
• One of the agents initiates negotiation with an offer.

Agent
P Agent C
• The agent receiving an offer can
• accept that offer Offer [10days, 100$]
• make a counteroffer Counter Offer [15 days, 80 $]
• end negotiation Offer [11days, 99$]

………
• This process continues in a turn-taking fashion until

having a consensus or reaching a termination condition
such as a deadline.
11
Mediated Single Text Negotiation Protocol
(based on Raiffa 1982)
Mediator generates an offer and asks negotiating agents for their votes
either to accept or to reject this offer.
Mediator
(Paris, Summer, 1-week holiday)
Party 1 Party 2 ... Party n

• Negotiating agents send their votes for the current bid according to
their acceptance strategy.
• If all negotiating agents vote “accept”, the bid is labeled as the most
recently accepted.
Mediator
Accept
Accept ...
Accept
E.g. MRA Bid: (Paris, Summer, 1-week holiday)

• Mediator modifies the most recently accepted bid by exchanging one
value arbitrary and asks negotiating agents’ votes again.
• It updates the most recently accepted bid if all negotiating agents vote
as “accept”.
Mediator
(Rome, Summer, 1-week holiday)
This process continues iteratively until reaching a predefined number of

bids.
How does a mediator find a solution
acceptable to all?
• The role of the mediator is to propose new ideas that

are acceptable to all.
• How to find these?
• Trusted mediator might know all profiles (partially)
• What if mediator doesn’t know the profiles?
• Mediated Hill-Climber Agent
• Mediated Annealer Agent
15
Mediated Single Text Negotiation
Hill-Climber Agent (Klein et al., 2003)
• Accept a bid if its utility is higher than the utility of the
most recently accepted bid
• MRA Bid= (Antalya, Summer, 1-week),

• Bid6= (Antalya, Summer, 2-week),
• U(Bid6)=0.95 >U(MRA Bid )=0.87 🡪 ACCEPT
Note: If the utility of initial bid is quite high for one of the
agents, that agent may not accept other bids even though
those bids might be better for the majority.
http://ebusiness.mit.edu/research/Briefs/4Klein_Negotiation_Brief_Final.pdf
Mediated Single Text Negotiation:
Annealer Agent (Klein et al., 2003)
• Calculates the probability of acceptance for the current bid:
T: Virtual temperature gradually declines over time
• Higher probability for acceptance

• The utility difference is small & virtual temperature is high
• Tendency to accept individually worse bids earlier so the
agents find win-win bids later
Baarslag et al., for more information see phd thesis
NEGOTIATION STRATEGIES
18
Negotiation Strategy
• Determines
• which action the agent will take
• how the agent will generate its offers
• how the agent decide whether the opponent’s counter-offer is acceptable
O B A
19
Bidding strategies
B
Random Walker
• It generates an offer randomly as follows:
• Selects values of issues randomly
• Proposes only those bids whose own utility greater than

its reservation utility (RU=0.6).
21
Time-dependent Concession Strategy
[Faratin, Sierra & Jennings, 1998]
• Each agent has a deadline and the agent's behavior changes with
respect to the time.
• An offer which is not acceptable at the beginning, may become
acceptable over time (conceding while approaching the deadline).
• A function determines how much the agent will concede
• Remaining negotiation time
• Parameter related to concession speed (β)
• Conceder Tactic:
• β >1 concedes fast and goes to its reservation value quickly.
• Boulware Tactic:
• β <1 hardly concedes until the deadline
22
Trade-Off Strategy (1)
• Not only considers its own utility but also take its
opponent’s utility into account.
• The importance of the issues may be different for the

negotiating agents.
• E.g. delivery time might be more important for the consumer
• The agent may demand more on some issues while

concedes on other issues without changing its overall utility
as if possible.
• E.g. higher price in order to have an earlier delivery
23
Trade-Off Strategy (2)
[Faratin, Sierra, Jennings, AIJ 2002]
• Selects a subset of bids having the

same utility with its previous offer (iso-
curve)
• If not possible, it makes minimal
concession such as 0.05.
• Among those bids, choose the bids

which might be more preferred by its
opponent
• Heuristic: the most similar one to
opponent’s last bid
This figure is taken from Faratin’s PhD Thesis .
24
Behaviour Dependent Strategies:
[Faratin, Sierra & Jennings, 1998]
• The agent imitates its opponent’s behaviour.
• The degree of imitation may vary
• Absolute Tit-For-Tat:
• E.g. The opponent increases the price by 50 units then the
agent will decrease the price by 50 units.
• Relative (proportionally) Tit-For-Tat:
• Taking into account the changes of its opponent’s
behaviour in a number of previous steps.
• Averaged Tit-For-Tat
• Taking into account the average changes within a window
of size of its opponent history
25
Opponent modelling strategies
O
Opponent Modelling (1) Why?
• Exploit the opponent
• Maximize chance of reaching an agreement
• Requiring outcome with acceptable utility for opponent, i.e. resolving
the conflict of interest.
• Increase the efficiency of a negotiated agreement
• Searching through the outcome space for outcomes that are
mutually beneficial
• Reaching better/optimal agreements
• Avoid unfortunate moves

• which is worse for both agents
• Make trade-offs and maximize social welfare
• Reach agreements early
• Reducing communication cost
27
Opponent Modeling
Outcome Space
Contrac Pareto
tA frontier
My Utility
Contract
B
Utility of other side

Opponent Modelling (2) What?
• Learning which issues are important for the opponent

• Issue weights
• Learning opponent’s preferences

• Evaluation of issue values
• Preference ordering of issue values
• Learning about opponent’s strategy

• Predicting the utility of its next offer
• Learning what kind of offers are not acceptable

• Reservation value
• Constraints
29
Some Examples (1):
• Kernel density estimation for estimating the opponent’s

issue weights [Coehoorn and Jennings 2004]
• Intuition: The opponent has a tendency to concede slowly on
important issues.
• Assumption: Weighted scoring function & Concession based
strategy
• Bayesian Learning for predicting evaluation functions

and weights [Hindriks and Tykhonov, 2008]
• Hypothesis for evaluation functions: uphill, downhill, triangular
• Assumption: Linear additive functions & Concession based
strategy
30
Some Examples (2)
• A guessing heuristic for predicting the opponent’s

unknown weights [Jonker, Robu & Treur, 2007]
• Some of the weights are revealed by the opponent
• Requiring domain knowledge
• Concept-based Learning (RCEA) for classifying offers

regarding their acceptability [Aydogan & Yolum 2012]
• Assumption: Conjunctive & Disjunctive Constraints
• Intuition: Avoid offering unacceptable offers to opponent
31
A Simple Example:
Frequency Analysis Heuristic
• A heuristic adopted by some of the agents in ANAC
competition such as HardHeaded Agent
• Based on how often the value of an issue changed and
the frequency of appearance of values in offers
• Learning issue weights: importance of issues
• Heuristic: If the value is often changed, then the issue gets a
low weight.
• Learning issue value weights: evaluations of the issue values
• Heuristic: A preferred value will appeared more often in agent’s
offers than a less preferred value.
32
Frequency Analysis Heuristic (2)
Estimation of issue weights
• Assume that we have two issues (X, Y) and opponent ’s first

offer is [x1,y1].
• Take the predicted weights 0.5 and 0.5 for X, Y respectively
• Second offer [x1, y2]
• W1=0.5 +n since opponent didn’t change the value of X
• W2=0.5
• If n= 0.1 then new weights will be 0.6, 0.5 respectively
• W1new= 0.55 W2new= 0.45
33
Frequency Analysis Heuristic (3)
Estimation of evaluation values for issues
• Assume negotiation round=45 and our opponent’s

offer history
• “Brand” issue in Laptop domain
Issue Values: Dell Mac HP

# of times 20 15 10
appeared in offers
Estimated 1.0 (20/20) 0.75 (15/20) 0.5 (10/20)
Evaluation
34
Acceptance Conditions
A
Introduction
Why and when should we accept?
• In every negotiation with a deadline, one of the

negotiating parties has to accept an offer to avoid a
break off.
• A break off is usually an undesirable outcome;

therefore, it is important to consider under which
conditions to accept.
Introduction
The Acceptance Dilemma
• When designing such conditions one is faced with the

acceptance dilemma:
• Accepting too early may result in suboptimal
agreements
• On the other hand, accepting too late may result
in a break off
• We have to find a balance:

Selection of Existing Acceptance Conditions
• In literature and current agent implementations, we see the

following recurring acceptance conditions:
• Accept when the opponent’s bid is better than
• Accept when the opponent’s bid is better than our

upcoming bid
• Accept when time has passed

Example t = 0.7
0.1
0.2
0.3
0.4
0.5
0.6
Combining Acceptance Conditions
• We can also combine acceptance conditions, e.g.:
• splits the negotiation time into two phases: and
• We can also choose non-constant values for such as average utility

so far received (AVG), or maximum utility (MAX).
Conclusion
• is often used, but does not always give the

best results.
• performs worst of all AC’s, as a good

value for is highly domain-dependent.
• always reaches an agreement, but of

relatively low utility.
• We need combinations of different approaches.
Conclusion:
Challenges in Automated Negotiation
• Designing negotiation protocols & strategy
• Representing and Reasoning on Preferences in

Negotiation
• Predicting Other Agent’s Preferences during

Negotiation
• Acceptance Strategies
Hindriks, Jonker, Tykhonov, 2011
ANALYSIS OF NEGOTIATION
DYNAMICS
Analysis of negotiation strategies
• What kind of bids to make:

• Process analysis
• Step analysis
• Dynamic properties
• What kind of bids to accept:
• Outcome analysis
• Nash product
• Kalai-Smorodinsky
• Pareto-optimal
Outcome Analysis
Kalai-Smorodinsky
(Ubuyer=Useller)
Utopia
1
“Non-Existent”
Buyer
Pareto Optimal Frontier
Nash product
max(Ubuyer ⋅ Useller)
0 Seller
1
46
Negotiation traces
Negotiation step
by A
Utility of Agent A
Current Bid
of Agent A
Current Bid
of Agent B
Utility of Agent B
Negotiation traces tA: trace of A
tB: trace of B
t = tA + tB is the complete nego trace
t
A
Utility of Agent A
t
B
Utility of Agent B
Utility, negotiation steps, and traces
US(b): utility of “Self” for bid b

UO(b): utility of “Other” for b
∆a(b, b’) = Ua(b’)-Ua(b), a∈{S,O}
∆a(s): ∆a(b, b’) for a step s = b→b’.
A trace t is a series of negotiation steps, i.e., transitions
b→b’ with b, b’ offers.
Concession step denoted by (S-, O≥), s is a
concession step iff:
Utility of Agent A ∆S(s)<0, and ∆O(s)≥0.
concession
Utility of Agent B
Bosse, Jonker, RRS 2005

Unfortunate step denoted by (S≤, O-), s is an
unfortunate step iff:
Utility of Agent A ∆S(s) ≤ 0, and ∆O(s)<0.
unfortunate
Utility of Agent B
Fortunate step denoted by (S+, O+), s is an
unfortunate step iff:
Utility of Agent A
fortunate
∆S(s)>0, and ∆O(s)>0.
Utility of Agent B
Selfish step denoted by (S+, O≤), s is a selfish
step iff:
Utility of Agent A
selfish ∆S(s) > 0, and ∆O(s) ≤ 0.
Utility of Agent B
Silent step denoted by (S=, O=), s is a silent
step iff:
Utility of Agent A ∆S(s) = 0, and ∆O(s) = 0.
Utility of Agent B
Nice step denoted by (S=, O+), s is a nice
step iff:
Utility of Agent A ∆S(s) = 0, and ∆O(s) > 0.
Utility of Agent B
Classification of negotiation steps
Step
Classes
of Agent A
selfish fortunate
silent
nice
Utility of Agent A
unfortunate concession
Current Bid
of Agent A
Utility of Agent B
Classification of negotiation steps
Step
Classes
of Agent A
selfish fortunate
silent
Utility of Agent A
nice
Steps of
unfortunate concession Agent B
Current Bid
nice
concession fortunate
of Agent A
silent
Current Bid unfortunate selfish

of Agent B
Utility of Agent B
Sensitivity to Opponent Preferences
A rational negotiator would try to make
fortunate, nice, or concession steps.
• In case no selfish, unfortunate or silent steps are made we

stipulate that sensitivitya(t)=∞.
• If sensitivitya(t)<1, then an agent is more or less insensitive to
opponent preferences;
• If sensitivitya(t)>1, then an agent is more or less sensitive to the
opponent’s preferences, with complete sensitivity for
sensitivitya(t)=∞.
September 12, 2020 58

The Three Strategies
• ABMP [Jonker, Treur, 2001]
• does not use any knowledge about opponent;
• calculates concession step on every round of negotiation;
• always make concession on every issue;
• Trade-off [Faratin, Sierra, Jennings, AIJ 2002]
• uses domain knowledge;
• tries to find bids on the same iso-level of own utility function
that is closer to the current opponent’s bid, makes concession
of 0.05 if stuck;
• uses opponent’s bid to make trade-offs;
• Random-Walker [Hindriks, Jonker, Tykhonov, IAT 2007]
• Selects values of issues randomly
• Proposes only those bids that have own utility >0.6
The Three Domains
• Second hand car selling domain:
• 5 issues (4 discrete issues and price issue),
• only the buyer’s preferences and the price issue are
predictable
• Service-oriented negotiation (SON):
• 4 continues issues;
• all issues are predictable;
• AMPO vs City
• 10 issues;
• only 8 issues are predictable;
Trade-Off (City) vs
Trade-Off strategy (AMPO)
Trade-Off (City) vs
Random Walker (AMPO)
©Hindriks, Jonker, Tykhonov,

Trade-Off (City) vs ABMP (AMPO)

Random Walker (City) vs ABMP (AMPO)

Random Walker (City) vs
Trade-Off strategy (AMPO)
Outcome Utility
• Overall utility:
• ABMP 0.72,
• Trade-Off 0.74, and
• Random Walker 0.69.
• Trade-Off:
• Outperforms ABMP on the SON domain with complete
information and on the AMPOvsCity domain;
• Underperforms wrt ABMP on the second hand car domain due
to wrong weights and unpredictable issues;
• ABMP:
• Strong on the second hand car domain;
• Underperforms on the SON domain.
Conclusions
• Want to negotiate efficiently? Know your partner!
• It is impossible to avoid unfortunate steps without

sufficient domain knowledge or opponent
knowledge.
• In the analysis of negotiation strategies, not only

the outcome of a negotiation is relevant, but also
the bidding process itself is important.
• When developing a general negotiation strategy

test against many opponents and in many domains.
(Baarslag 2014): Tim Baarslag, Alexander Dirkzwager, Koen
Hindriks, and Catholijn Jonker. The significance of bidding, accepting
and opponent modeling in automated negotiation. In: 21st European
Conference on Artificial Intelligence, 2014.
THE BOA FRAMEWORK

Introduction
In search of an efficient automated negotiator
• Challenge: from the outside, agent architectures are
essentially a ‘black box’.
• A negotiation strategy is a result of complex

interaction between various components, of which
the individual performance may vary significantly.
• Overall performance measures make it hard to
pinpoint reasons for success.
Introduction
Component-based negotiation
• Many agent strategies are comprised of a fixed set of

modules; generally, a distinction is made between
three different modules:
• One that decides which set of bids could be proposed next;

B • One that tries to guess the opponent’s preferences and takes
O this into account when selecting an offer to send out.
A • One module that decides whether the opponent’s bid is
acceptable;
The BOA framework
Negotiation flow
O B A
Applying the BOA Framework
Combining components
Tit for Tat Conceding

B’s
strategy strategy
Accept as late as
Accept early possible
AC next discount
AC next discount Accept very good
Frequency model (1.0; 0.0; 1.0, 0.0)
(1.0;Accept above
0.0; 1.0, 0.0)a bids only
No opponent model Bayesian learning fixed threshold Accept at the last
Accept the best Never accept
moment
Bayesian learning
No opponent offer so far
Frequency model
model
O’s A’s
73
Required reading
• Klein et al., 2003

• Hindriks et al., 2011
74
References (1)
• Baarslag, T., Aydogan, R., Hindriks, K.V., Jonker, C.M., Fujita, K., & Ito, T.,
(2015). The Automated Negotiating Agents Competition, 2010-2015, AI Magazine,
36, pp:115-118.
• Tim Baarslag, Alexander Dirkzwager, Koen Hindriks, and Catholijn Jonker. The
significance of bidding, accepting and opponent modeling in automated
negotiation. In 21st European Conference on Artificial Intelligence, 2014.
• Baarslag T, Fujita K, Gerding EH, Hindriks K, Ito T, Jennings NR, Jonker CM, Kraus
S, Lin R, Robu V, Williams CR, (2013). Evaluating Practical Negotiating Agents:
Results and Analysis of the 2011 International Competition, Artificial Intelligence,
198, pages:73 - 103, issn: 0004-3702, doi: 10.1016/j.artint.2012.09.004.
http://www.sciencedirect.com/science/article/pii/S0004370212001105?v=s5
• Jonker CM, Hindriks K, Wiggers P, Broekens JD, (2012). Negotiating Agents, AI
Magazine, 33. http://www.aaai.org/ojs/index.php/aimagazine/article/view/2421
• Luo, Xudong, Miao, C, Jennings, Nick, He, Minghua, Shen, Z and Zhang, M (2012)
KEMNAD: A Knowledge Engineering Methodology for Negotiating Agent
Development. Computational Intelligence, 28, (1), 51-105.
75
References (2)
Klein, M., et al., Protocols for Negotiating Complex Contracts. IEEE
Intelligent Systems, 2003. 18(6): p. 32 - 38.
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=1249167&filter%3DAND%28p_IS_Num
ber%3A27968%29
but easily accessible:
http://ebusiness.mit.edu/research/Briefs/4Klein_Negotiation_Brief_Final.pdf
Rosenschein, J.S., and G. Zlotkin. Rules of Encounter. MIT Press,
Cambridge, MA, 1994. 16.
Rubinstein, A., Perfect equilibrium in a bargaining model. Econometrica,
50:97–109, 1982. http://arielrubinstein.tau.ac.il/papers/11.pdf
Raiffa, H., The art and science of negotiation. 1982, Cambridge, Mass.:
Belknap Press of Harvard University Press. x, 373.
Baarslag T. 2014. What to Bid and When to Stop. Delft University of
Technology. http://mmi.tudelft.nl/sites/default/files/thesis.pdf
Hindriks KV, Jonker CM, Tykhonov D, (2011). Let's dans! An analytic
framework of negotiation dynamics and strategies, Web Intelligence and
Agent Systems, 9, pages:319-335 (see reading material on blackboard for
the paper) 76
Computational Coalition Formation
CSE 3210: Collaborative AI
Slides adapted from (Elkind, Rahwan, and Jennings, 2013)
1
Collaborative AI: Learning Objectives
• Compare centralized and collaborative AI paradigms
• Describe the principles of negotiation for agent cooperation

– Describe the conceptual underpinnings of agent interaction protocols
– Create an automated negotiating agent in the Genius Web platform
• Describe the conceptual underpinnings of agent coordination

– Trust, computational social choice, and coalition formation
• Apply co-active design for a collaborative AI problem
2
Learning Objectives for Today
After this lecture, you should be able to:
• Describe the computational coalition formation

problem and its subproblems
• Apply a dynamic programming algorithm

(DP/IDP) for coalition structure generation
• Apply an integer programming algorithm (IP) for

anytime coalition structure generation
3
Reading Material and Resources
• Multiagent Systems, 2nd Edition.
Gerhard Weiss
– Chapter 8
• Sections 1, 2, 5.1, 5.2, and 5.3
• A research paper by
Rahwan et al., JAIR, 2009
– Sections 1--4
– Brightspace: Resources and tools
• Lecture slides
4
What is an Agent?
Hello…
I am an intelligent entity….
I am situated in an environment…
I can perceive the environment
I have goals…
I have a list of available actions…
I act autonomously to satisfy my goals
and…
most importantly…
I am social !!!
5
Example: the Prisoner’s Dilemma
• Two agents committed a crime.

• Court does not have enough evidence to convict them of the crime,
but can convict them of a minor offence
(1 year in prison each)
• If one suspect confesses (acts as an informer), he walks free, and
the other suspect gets 4 years
• If both confess, each gets 3 years
• Agents have no way of communicating or making binding
agreements
6
What should a “Rational” Agent do in the
Prisoner’s Dilemma?
https://www.menti.com/2ommwhq7ew
7
Prisoners’ Dilemma: Matrix
Representation
a2 quiet confess
a1
(-1,-1) (-4, 0)
quiet
(0, -4) (-3, -3)
confess
• Interpretation: the pair (x, y) at the intersection

of row i and column j means that the row player
gets x and the column player gets y
8
Prisoners’ Dilemma: the Rational
Outcome
• a1’s reasoning: a
a1 2 Quite Confess
– if a2 stays quiet, (-1,-1) (-4, 0)

I should confess Quite
– if a2 confesses, (0, -4) (-3, -3)

I should confess, too Confess
• a2 reasons in the same way
• Result: both confess and get 3 years in prison.

• But: they could have got only 1 year each!
• So why do they not cooperate?
9
Cooperative vs. Non-Cooperative Games
• Game theory studies interactions between agents in
situations known as games
• In Non-Cooperative games, players cannot make

binding agreements
• But what if binding agreements are possible?
• Cooperative games model scenarios, where

– binding agreements are possible
– agents can benefit by cooperating
10
Coalition Game Theory
• Coalition Formation is studied in a field of Game

theory, called Cooperative Game Theory
• In a cooperative game, agents, whether selfish

or working towards a common goal, benefit from
cooperation
11
Organizations in Multiagent Systems
Hierarchies
Compounds
Markets
MatrixControl
Organizations
Teams Coalitions
Holarchies
Congregations
Federations
Societies Data
12
Hierarchies
Compounds
Markets
Matrix
Organizations
Teams Coalitions
Holarchies
Congregations
Federations
Societies seller buyer
13
Hierarchies
Compounds
Markets
Matrix
Organizations
Teams Coalitions
Holarchies
Congregations
Federations
Societies
14
Agent Coalitions
Main characteristics
Coalitions in general are goal-directed and short-lived
No coordination among members of different coalitions
The organizational structure within each coalition is flat
15
Why Agents form Coalitions?
Distributed Vehicle Routing (Sandholm and Lesser, 1997)

16
Coalition Formation: Applications
Smart Energy Grids
Intelligent appliances and energy
storage devices coordinate for
optimal energy use
Electronic-commerce
Cooperation among buyers to
obtain quantity discounts, and
sellers to maintain pricing.
Disaster Management
UN report said: “Efforts by the
United Nations in Haiti have
lacked sufficient coordination”
17
Coalition Formation: Applications
Distributed sensor networks:
Coalitions of sensors can work together to track
targets of interest [Dang et al. 2006]
Distributed vehicle routing:

Coalitions of delivery companies can be formed to
reduce the transportation costs by sharing deliveries
[Sandholm and Lesser, 1997].
Information gathering:
Several information servers can form coalitions to
answer queries [Klusch and Shehory, 1996].
18
Coalition Formation Games
Partition Function
Does a coalition Yes Game (PFG)
influence other
co-existing
coalitions? No Characteristic
Function Game
Cooperative (CFG)
Game
Transferable
Yes Utility (TU)
Can a player
Game
transfer part of its
utility to
another? No Non-Transferable
Utility (NTU) Game
19
Example: Buying Ice-Cream
• n children, each has some money:
• Supermarkets sells many ice-cream tubs, in different sizes:
– Type 1 contains 500g, costs $7
– Type 2 contains 750g, costs $9
– Type 3 contains 1kg, costs $11
• Children have utility for ice-cream, and don‘t care about money
• The payoff of a group is the maximum amount of ice-cream the members
of the group can buy by pooling their money
• TU or NTU? TU (ice-cream is transferable)

• CFG or PFG? CFG (many available tubs)
20
Example: Writing Papers
• n researchers working at n different universities can form
groups to write papers
• the composition of a group determines the quality of the
paper they produce
• each author receives a payoff
from his own university:
– promotion
– teaching load reduction
21
Example: Growing Fruits
• n farmers can cooperate to grow fruit

• Each group grows apples or oranges
• a group of size k can grow f(k) tons of
apples, or g(k) tons of oranges, where
f() and g() are convex functions of k
• The market price of a fruit drops monotonically as the

number of tons available in the market increases
22
What types of coalition formation games
are the “writing papers” and “growing
fruits” examples?
https://www.menti.com/q2its8o42j
23
Coalition Formation Games
Partition Function
Does a coalition Yes Game (PFG)
influence other
co-existing
coalitions? No Characteristic
Function Game Focus
Cooperative (CFG) of this
Game
lecture
Transferable
Yes Utility (TU)
Can a player
Game
transfer part of its
utility to
another? No Non-Transferable
Utility (NTU) Game
24
Transferable Utility Games Formalized
• A transferable utility game is a pair (A, v), where:
– A ={a1, ..., an} is the set of players (or agents)
– v: 2A → R is the characteristic function
• for each C ⊆ A, v(C) is the value of C, i.e., the payoff that
members of C can attain by working together
– Usually, it is assumed that
• v(Ø) = 0
• v(C) ≥ 0 for any C ⊆ A
• v(C) ≤ v(D) for any C, D such that C ⊆ D
• The biggest possible coalition (the one containing
all agents) is called the grand coalition
25
Ice-Cream Game: Characteristic Function
C: $6, M: $4, P: $3
w = 500g w = 750g w = 1000g

p = $7 p = $9 p = $11
• v(Ø) = v({C}) = v({M}) = v({P}) = 0

• v({C, M}) = 750, v({C, P}) = 750, v({M, P}) = 500
• v({C, M, P}) = 1000
26
Transferable Utility Games: Outcome
An outcome of a TU game G =(A, v) is a pair (CS, x), where:
• CS =(C1, ..., Ck) is a coalition structure,

i.e., a partition of A into coalitions:
– C1  ...  Ck = A,
– Ci  Cj = Ø for i ≠ j
• x = (x1, ..., xn) is a payoff vector, which

specifies the payoff of each agent:
– xi ≥ 0 for all ai A
– xi = v(C) for all CCS
27
Coalition Formation Process
optimal?
Coalition structure
generation
Value
Value Value
Payoff
distribution
Value
28
Coalition Structure Generation in CFGs
Given 3 agents, the set of agents is:
{a1,a2,a3}
The possible coalitions are:

{a1} {a2} {a3} {a1,a2} {a1,a3} {a2,a3} {a1,a2,a3}
20 40 30 70 40 65 95
The possible coalition structures:

{{ a1},{a2},{90
20+40+30=
a3}} {{a70+30
1,a2},={100
a3}} {{a40+40
2},{a1,a 3}}
= 80
{{a120+65
},{a2,a=385
}} {{a1,a95
2,a3}}
Input: a value of every possible coalition

Output: a coalition structure in which the sum of values is maximized
29
value L4 value L3 value L2 value L1
140 {1,2,3,4} 90 {1, 2, 3} 50 {1, 2} 30 {1}
120 {1, 2, 4} 60 {1, 3} 40 {2}
100 {1, 3, 4} 80 {1, 4} 25 {3}
115 {2, 3, 4} 55 {2, 3} 45 {4}
70 {2, 4}
80 {3, 4}
Exercise
What is the optimal
coalition structure ?
Answer
{ {1}, {2}, {3,4} }
30
value L6 value L5 value L4 value L3 value L value L
2 1
6217 1,2, 3, 4, 5, 6 4804 1, 2, 3, 4, 5 4355 1, 2, 3, 4 3352 1, 2, 3 1750 1, 2 826 1

5657 1, 2, 3, 4, 6 4373 1, 2, 3, 5 3102 1, 2, 4 1670 1, 3 1108 2
4609 1, 2, 3, 5, 6 3770 1, 2, 3, 6 3301 1, 2, 5 1989 1, 4 1065 3
4829 1, 2, 4, 5, 6 3528 1, 2, 4, 5 3119 1, 2, 6 1664 1, 5 890 4
5650 1, 3, 4, 5, 6 3967 1, 2, 4, 6 3287 1, 3, 4 2023 1, 6 907 5
5852 2, 3, 4, 5, 6 3647 1, 2, 5, 6 1, 3, 5 2083 2, 3 1024 6
2696
4142 1, 3, 4, 5 1, 3, 6 2272 2, 4
2950
3875 1, 3, 4, 6 3324 1, 4, 5 2082 2, 5
3905 1, 3, 5, 6 2460 1, 4, 6 1995 2, 6
3645 1, 4, 5, 6 1, 5, 6 1807 3, 4
3134
3850 2, 3, 4, 5 2, 3, 4 2529 3, 5
2943 2045
4099 2, 3, 4, 6 2, 3, 5 3, 6
2956 1683
3967 2, 3, 5, 6 2, 3, 6 4, 5
3318 2, 4, 5, 6 2950 2, 4, 5 2115 4, 6
3576 3, 4, 5, 6 3661 2, 4, 6 1956 5, 6
2618 2, 5, 6
2906 3, 4, 5
2769 3, 4, 6
3070 3, 5, 6
3135 4, 5, 6
Exercise
What is the optimal coalition
structure ?
31
Complexity of Coalition Structure Generation
• Bell number
• ω(nn/2)
• O(nn)
32
Coalition Structure Generation Algorithms
Dynamic Programming techniques

[Rahwan & Jennings N. R., An Improved Dynamic Programming
IDP Algorithm for Coalition Structure Generation, AAMAS 2008].
Anytime with guarantees on solution quality

[Rahwan et al., An Anytime Algorithm for Optimal Coalition
IP Structure Generation, AAAI 2007, JAIR 2009].
33
Dynamic Programming Algorithm
Main observation: To find the optimal partition of a set of agents, it is sufficient to:
• Try the possible ways to split the set into two sets, and
• For every half, find the optimal partition of that half.
34
f Best split Evaluations performed before setting f coalitionsize
{1} V({1})=30 {1} 301
{2} V({2})=40 {2} 40
{3} V({3})=25 {3} 25
{4} V({4})=45 {4} 45
{1,2} V({1,2})=50 f({1})+f({2})=70 {1} {2} 702

{1,3} V({1,3})=60 f({1})+f({3})=55 {1,3} 60
{1,4} V({1,4})=80 f({1})+f({4})=75 {1,4} 80
{2,3} V({2,3})=55 f({2})+f({3})=65 {2} {3} 65
{2,4} V({2,4})=70 f({2})+f({4})=85 {2} {4} 85
{3,4} V({3,4})=80 f({3})+f({4})=70 {3,4} 80
{1,2,3} V({1,2,3})=90 f({1})+f({2,3})=95 f({2}) {2} {1,3} 3

100
+f({1,3})=100 f({3})+f({1,2})=95
{1,2,4} V({1,2,4})=120 f({1})+f({2,4})=115 f({2}) {1,2,4} 120

+f({1,4})=120 f({4})+f({1,2})=115
{1,3,4} V({1,3,4})=100 f({1})+f({3,4})=110 {1} {3,4} 110

f({3})+f({1,4})=105 f({4})+f({1,3})=105
{2,3,4} V({2,3,4})=115 f({2})+f({3,4})=120 {2} {3,4} 120

f({3})+f({2,4})=110 f({4})+f({2,3})=110
V({1,2,3,4})=140 f({1})+f({2,3,4})=150 4
{1,2,3,4} {1,2} {3,4} 150
f({2})+f({1,3,4})=150 f({3})+f({1,2,4})=145
f({4})+f({1,2,3})=145 f({1,2})+f({3,4})=150
f({1,3})+f({2,4})=145 f({1,4})+f({2,3})=145 35
Improved Dynamic Programming Algorithm (IDP)
We define a subset of edges E* in the coalition

structure graph
We prove that the edges in E* are sufficient to form

a path to every node in the graph
We modify the original algorithm such that it only

evaluates the movements through the edges in E*
36
The Coalition Structure Graph
{a1},{a2},{a3},{a4}
optima
l
{a1},{a2},{a3,a4} {a3},{a4},{a1,a2} {a1},{a3},{a2,a4} {a2},{a4},{a1,a3} {a1},{a4},{a2,a3} {a2},{a3},{a1,a4}
{a1},{a2,a3,a4} {a1,a2},{a3,a4} {a2},{a1,a3,a4} {a1,a3},{a2,a4} {a3},{a1,a2,a4} {a1,a4},{a2,a3} {a4},{a1,a2,a3}
{a1,a2,a3,a4}
37

IDP Algorithm for Coalition Structure Generation, AAMAS 2008].

[Rahwan et al., An Anytime Algorithm for Optimal Coalition
IP Structure Generation, AAAI 2007, JAIR 2009].
38
Basic Idea of IP
Upper bound = 550

Sub-space 500 Upper bound = 450
Lower bound = 200 Sub-space
Lower bound = 250
Search space
Upper bound = 300
Upper bound = 250
Sub-space
Lower bound = 100
Upper bound = 600
Sub-space Sub-space 400

Lower bound = 200 Lower bound = 100
Upper bound = 350

Sub-space
Lower bound = 150
39
Basic Idea of IP
S
Upper bound [4]
= 550 {{a1, a2, a3, a4}}
Sub-space 500 Upper bound = 450
{{abound
1,a2}, ={a200
3,a4}} Sub-space
S [2,2] Lower
{{a1,a3}, {a2,a4}} {{a=1},250{a2}, {a3,a4}}
Lower bound
{{a1,a4}, {a2,a3}} {{a1}, {a3}, {a2,a4}}

S
Upper bound = 250
[1,1,2]
Sub-space {{a1}, {a4}, {a2,a3}}
Upper bound = 300 {{a }
{a1}}, {a2,a3,a4}} Upper bound = 600
Lower bound = 100 {{a2}, {a3}, {a1,a4}}
SSub-space
1 3
[1,3] {{a }
{a2}}, {a1,a3,a4}} Sub-space 400
{{a2},bound
Lower
{a4}, ={a100
1,a3}}
Lower bound = 200 {a3}}, {a1,a2,a4}}
{{a }
{{a3}, {a4}, {a1,a2}}
} Upper bound = 350
{a4}}, {a1,a2,a3}}
{{a
S[1,1,1,1] Lower Sub-space
{{a1bound
}, {a2},= 150
{a3}, {a4}}
40
How the Bounds are Computed
value L4 value L3 value L2 value L1
425 {1, 2, 3, 4} 200 {1, 2, 3} 175 {1, 2} 125 1
150 {1, 2, 4} 150 {1, 3} 50 2
Max4 = 425
300 {1, 3, 4} 100 {1, 4} 75 3
Avg4 = 425
150 {2, 3, 4} 150 {2, 3} 150 4
200 {2, 4}
Max1 = 150 Max3 = 300
125 {3, 4}
Avg1 = 100 Avg3 = 200
Max2 = 200
Avg2 = 150 { {a1}, {a1, a2, a3} }
{ {a1}, {a1, a2, a4} }
S [4] {{a1, a2, a3, a4}}
{ {a1}, {a1, a3, a4} }
Avg=425 UB=425 { {a1}, {a2, a3, a4} }
{{a1,a2}, {a3,a4}} { {a2}, {a1, a2, a3} }

S [2,2] {{a }, {a
{ {a2},
1
{a1,,a
2
a2,,a
3
a4}}}}
4
Avg=300 UB=400 {{a1,a3}, {a2,a4}} S [1,3] { {a2}, {a1, a3, a4} }
{ {a2}, {a2,,a
a3,,a
a4}}}}
{{a }, {a
2 1 3 4
{{a1,a4}, {a2,a3}} Avg=300 UB=450
{ {a3}, {a1, a2, a3} }
{{a 3}, {a 1,a 2,aa4}
4}}}
{{a1}, {a2}, {a3,a4}} { {a3}, {a1, a2,
{ {a3}, {a1, a3, a4} }
{{a 4}, {a 1,a 2,aa4}
3}}}
{{a1}, {a3}, {a2,a4}} { {a3}, {a2, a3,
S[1,1,2] S [1,1,1,1]
Avg=350 UB=500 {{a1}, {a4}, {a2,a3}} Avg=400 UB=600 { {a4}, {a1, a2, a3} }
{ {a4}, {a1, a2, a4} }
{{a2}, {a3}, {a1,a4}} { {a4}, {a1, a3, a4} }
{{a1}, {a2}, {a3}, {a4}} { {a4}, {a2, a3, a4} }
{{a2}, {a4}, {a1,a3}} 41
Searched initially [8] Upper bound = 700
(contains one solution)
Lower bound = 550
Searched while [4,4] [3,5] [2,6] [1,7]
scanning the input
[2,3,3] [2,2,4] [1,3,4] [1,2,5] [1,1,6]

Avg=350 UB=400 Avg=450 UB=575 Avg=500 UB=700 Avg=390 UB=480 Avg=460 UB=510
[2,2,2,2] [1,2,2,3] [1,1,3,3] [1,1,2,4] [1,1,1,5]

Avg=520 UB=600 Avg=450 UB=520 Avg=440 UB=480 Avg=550 UB=620 Avg=520 UB=540
[1,1,2,2,2] [1,1,1,2,3] [1,1,1,1,4]

Avg=380 UB=490 Avg=470 UB=525 Avg=420 UB=475
[1,1,1,1,2,2] [1,1,1,1,1,3]
Avg=500 UB=650 Avg=320 UB=400
[1,1,1,1,1,1,2]
Avg=360 UB=390
Searched initially [1,1,1,1,1,1,1,1] 42

Searched initially [8] 700
Upper bound = 650
625
550
Lower bound = 600
625
Searched while [4,4] [3,5] [2,6] [1,7]
scanning the input
? ?
[2,3,3] c h
[2,2,4]
r
[1,3,4]
Avg=500a UB=700
[1,2,5] [1,1,6]
Se
Avg=350 UB=400 Avg=450 UB=575 Avg=390 UB=480 Avg=460 UB=510
?
[2,2,2,2]
Avg=520 UB=600
[1,2,2,3]
Avg=450
[1,1,2,2,2]
UB=520
[1,1,3,3]
Avg=440 UB=480
[1,1,1,2,3]
?
[1,1,2,4]
Avg=550 UB=620
[1,1,1,1,4]
[1,1,1,5]
Avg=520 UB=540
Avg=380 UB=490 Avg=470 UB=525 Avg=420 UB=475
? c h
[1,1,1,1,2,2] [1,1,1,1,1,3]
a r UB=650
Se
Avg=500 Avg=320 UB=400
[1,1,1,1,1,1,2]
Avg=360 UB=390
Searched initially [1,1,1,1,1,1,1,1] 43

IDP Algorithm
for Coalition Structure Generation, AAMAS 2008].
IP [Rahwan et al., An Anytime Algorithm for Optimal Coalition
Structure Generation, AAAI 2007, JAIR 2009].
Property
Algorithm
IDP IP
Worst case
n
performance O(3 ) O(nn)
Return solutions
anytime False True
Time to return
optimal solution Slow Fast
44
Summary
• Coalition formation as a cooperative game
• Formulate the coalition structure generation

(CSG) problem
• Describe IDP and IP algorithms for CSG
45
http://www.ihmc.us/users/mjohnson/publications.html
Seven “deadly” myths of autonomy
1. “Autonomous systems” are autonomous
2. Autonomy is unidimensional
3. The conceptualization of “levels of
autonomy” is a useful scientific
grounding
4. Autonomy is a widget
5. Once “achieved,” “full autonomy”
obviates the need for human-machine
collaboration
6. As machines acquire more “autonomy,”
they work as simple multipliers of
human capability
7. “Full autonomy” is not only possible, but
always desirable
-
Bradshaw, J.M, Robert R. Hoffman, Matthew Johnson, and David D. Woods. The Seven Deadly Myths of
"Autonomous Systems.” IEEEAutonomy
Intelligent Systems, May/June 2013 (vol. 28 iss. 3), pp. 54-61.
-
Lessons learned about autonomy
1. Autonomy != Replacement
2. It is not just about task allocation but also

about system integration
Dependent - Independent
Autonomy
-
2. It is not just about task allocation but also

about system integration
3. The goal should not be working independently,

but interdependently
- Inter
Autonomy Teamwork
- -
- Inter
Autonomy Teamwork
- -
- Inter
Autonomy Teamwork
- -
- Inter
Autonomy Teamwork
- -
• “improve the teaming of unmanned systems with the manned force” -
Unmanned Systems Integrated Roadmap FY2011-2036
• “The goal of the National Robotics Initiative is to accelerate the development

and use of robots in the United States that work beside, or cooperatively
with, people.” and “…symbiotic relationship...” – NRI 2013
• “human-system collaboration” – Defense Science Board 2012
• “combine the inherent strengths of manned platforms with the strengths

of UAS” – UAS Roadmap 2010-2035
• “shared-control concept … the operator works “in-the-loop” with the robot”

– Roadmap for US Robotics 2009
So, what does this mean to an

2. It is not just about allocation but also about

integration
3. The goal should not be working independently,

but interdependently
4. Interdependence has important

implications that permeate all design
elements (including algorithms and
interfaces)
- Inter
Autonomy Teamwork
- -
What is Coactive Design?
• Coactive Design is about designing
human-robot systems that support
interdependence.
• Coactive Design is about enabling

autonomy to reach its potential by
designing for interdependence.
The Theory of
Interdependence
The Theory of
Interdependence
“Interdependence” describes the set of

complementary relationships that two or more
parties rely on to manage required or
opportunistic dependencies in joint activity
The Theory of
Interdependence

involves both the control algorithms and the interface
The Theory of
Interdependence

opportunistic dependencies
Not limited to compensating in joint
for some missing ability activity
– or hard
constraints, but also involves normal supportive behavior associated
with teamwork – or soft constraints.
The Theory of
Interdependence

What kind of relationships are used to
manage dependencies in joint activity?
What is Coactive Design?
Interdependence
Observability
Predictability
Directability
Interface Design and Human Factors Automation and Control Theory

(Sheridan & Verplank, 1978)
These are where the requirements for

supporting interdependence come
Core Issues in Robotics Providing Support
Human Needs
What is the robot doing?
What is the robot going

to do next?
How can we get the robot

to do what we need?
Human Needs Issues
What is the robot doing? Observability
What is the robot going Predictability

to do next?
How can we get the robot Directability

to do what we need?
Human Needs Issues Robot Needs
What is the robot doing? Mutual Observability What is the intent of the
human?
What is the robot going Mutual Predictability What does the human
to do next? need from me?
How can we get the robot Mutual Directability Can the human provide
to do what we need? help?
How to design for
interdependence?
• Coactive Design is a design method
that helps design support for
interdependence and understand the
impact of change.
• Coactive Design highlights three key

capabilities that are needed for
effective human-robot teamwork:
observability, predictability and
directability (OPD).
Identification
process
Selection and
implementatio
n process
Evaluation of
change process
Identification
process
Selection and
implementatio
n process
Evaluation of
change process
IA Table Example for Collaborative Control
Interdependence Analysis (IA) Table
Team Member Role Alternatives
Performer Supporting Team Members
I can do it all My assistance could improve efficiency

I can do it all but my reliability is < 100% My assistance could improve reliability
I can contribute but need assistance My assistance is required
I cannot do it I cannot provide assistance
Team Member Role Alternatives

Performer Supporting Team Members
I can do it all My assistance could improve efficiency

I can do it all but my reliability is < 100% My assistance could improve reliability
I can contribute but need assistance My assistance is required
I cannot do it I cannot provide assistance
Feasible Interdependence combinations
notification predictability
Identification
process
Selection and
implementatio
n process
Evaluation of
change process
The mechanisms that
support an interdependent relationship
are the creative medium of the designer.
Selection and
implementatio
n process
notification predictability
Next we will get more
complex…
IA Table Example for DRC
Hose Task IA Table
Hose Task IA Table
Now for the OPD

requirements…
What would a specification look
http://www.ihmc.us/users/mjohnson/publications.html
COLLABORATIVE AI 7. - Trust
1
TRUST
2
TRUST
What is trust, and why is it relevant for Collaborative AI
Dimensions of trust
Evaluating Trust
Trustworthy AI
Explainability
This week: Trust concepts from the human-human & human-AI perspective
Next week: Trust modelling from the AI-AI & human-AI perspective
3
WHY TRUST?
What is Trust:
 “if A believes that B will act in A’s best interest, and accepts vulnerability to B’s actions, then A
trusts B 1”
We need trust to:

- Mitigate uncertainty & risk by:
- Enabling the anticipation that:
- Others will act in our best interest
1 Roger C Mayer, James H Davis, and F David Schoorman. 1995. An integrative model of organizational trust. Academy of management
review 20, 3 (1995), 709–734.
4
WHEN DO WE NEED TRUST?
All circumstances in which we are in any way
dependent on other’s actions
So: basically always when other people are

choosing actions and making decisions that
influence you
This happens quite a lot!
5
BACK TO AI?
6
RISK
Do we need to trust if there is no vulnerability?
Trust is most relevant if there is risk!
To accept vulnerability to X’s decision/action in

high-risk situations, we need to know the type of
decision/action and situation
8
TRUST AS A CONTRACT?
I trust X to do Y in situation Z
I trust my doctor to give me advice on medical matters

I do not trust my doctor to fix my plumbing
Contracts specify what behaviour we anticipate

Even when we talk about ‘general’ trust, there is typically an implicit contract!
I trust my doctor
9
TRUST VS. TRUSTWORTHINESS?
Trust
Trust is always something directional from one agent to another
 X trusts Y -> Attitude of X
Trustworthiness is inherent to an agent
 X is trustworthy -> Property of X
Not Trustworthy Trustworthy
Going back to contracts:

 Someone is trustworthy wrt some contract if they will maintain this contract
 My doctor is trustworthy if they will give me good advice on medical matters
Trust is perceived trustworthiness!
10
TRUST VS. TRUSTWORTHINESS
Warranted trust
 Trust is based on information about actual trustworthiness
Appropriate trust
 Trust is appropriate given trustworthiness
Trust calibration
 The process of calibrating trust to match trustworthiness
 If successful: leading to warranted appropriate trust.
11
WARRANTED/UNWARRANTED TRUST
Unwarranted trust/distrust :
Trust does not depend on trustworthiness
 Can lead to inappropriate trust:
 Distrust in a trustworthy system
 Trust in an untrustworthy system
Warranted trust/distrust :
Trust depends on trustworthiness
 Expected to lead to appropriate trust:
 Trust in a trustworthy system
 Distrust in an untrustworthy system
Jacovi et al.12 2010

INAPPROPRIATE TRUST IN TECHNOLOGY
Inappropriate trust
- Misuse (Overreliance)
 Monitoring failures
 Commission errors
 Reduced situation awareness
Inappropriate distrust
- Disuse
 Missed signals
 Missed opportunity
Parasuraman et13 al. 1997

THIS IS?
Mary has received a letter from her energy company ASUN that she can get solar panels.
She’d like to save energy, but she has heard from a friend who has solar panels from the
company B-WIND that the procedure is a hassle. As a result, she doesn’t believe that ASUN
can properly install solar panels. ASUN indeed has had problems with failed installations in
the past year.
1. Appropriate & unwarranted trust

2. Inappropriate & warranted trust
3. Inappropriate & unwarranted distrust
4. Appropriate & warranted distrust
5. Appropriate & unwarranted distrust
14
DIMENSIONS OF TRUST
Trustworthiness:
 Someone is trustworthy wrt some contract if they will maintain this contract
The article says:

 Someone is trustworthy wrt some contract if they are capable of maintaining this contract
However, when we say we don’t trust Facebook not to share our data with 3rd
parties:
 Do we mean we think they aren’t capable of not doing this?
 Or do we believe they are not willing to not do this?
Trustworthiness is about more than capability!

15
ABI MODEL
Mayer & Davis (1995)
The model developed here is

designed to focus on trust in an
organizational setting involving two
specific parties: a trusting party
(trustor) and a party to be trusted
(trustee)
16
ABI MODEL
Factors of perceived
trustworthiness:
 Ability
 Ability is that group of skills, competencies,
and characteristics that enable a party to
have influence within some specific domain.
 Benevolence
 Benevolence is the extent to which a trustee
is believed to want to do good to the trustor,
aside from an egocentric profit motive.
 Integrity
 The relationship between integrity and trust
involves the trustor's perception that the
trustee adheres to a set of principles that
the trustor finds acceptable
17
ABI Johan has a new smart watch. However, he values his
privacy and he doesn’t really trust the smart watch to
not share his personal data with third parties.
Therefore he doesn’t really wear the watch.
Ability Robin needs to give an important presentation

 Are you able for his work. He has a virtual assistant Val which
 Are you capable given the circumstances is trained to help him create slides for talks.
Johan knows that Val has been trained mostly
on different types of talks, so he’s a bit unsure
about using her for this particular talk, as it’s so
Benevolence important.
 Inter-personal relationship
 If you know someone well, they want to do well Lotte has a personal robot which helps her with her
by you in particular exercise regime. Her robot is called Exie, and learns
about Lotte’s habits and preferences over time. One
day, one of Exie’s motors breaks. While Lotte is
Integrity waiting for her new Exie, she can temporarily share
her sister’s personal robot Bexie, who has received all
 Shared personal values of Exie’s data about Lotte. However, Lotte feels like
 Honest, fair, principled Bexie sometimes prioritizes her sister over her, and
doesn’t trust Bexie to always be there for her for that
reason.
18
ABI MODEL
Trust =
 Perceived trustworthiness *
 Propensity to trust
Propensity to trust:
 Quality of the trustor, irrespective of the
trustee
 How likely are you, as a person, to trust?
19
ABI MODEL
Risk taking in relationship
 Behavior based on trust
 But also on perceived risk!
There is no direct link of only trust

to behaviour. Perceived risk always
also has an influence!
The higher the perceived risk, the

higher the necessary trust to take
that risk.
20
EVALUATING TRUST
Behavior
 Risk taking behaviour / Reliance
Perceived
Trust
Risk
Given ABI, to properly estimate trust from behaviour, you need to
 Know perceived risk
 Control for it!
Compare trust in setting x vs setting y on behavior

Behavior
 Measure perceived risk
 Control for this factor in your analysis
21
EVALUATING TRUST
Trust Many scales exist!
 Perceived Trustworthiness  Trust in organisations
 Perceived = Subjective  Trust in economics
 Trust in 1 other person
 Subjective measures
 Trust in automation
 Questionnaires
Choose the questionnaire that fits
22
QUESTIONNAIRES ON TRUST
Multidimensional?
 ABI is not the only model!
Ullman & Malle (2018) Trust in robots

 Capable
 Ethical
 Sincere
 Reliable
23
QUESTIONNAIRES ON TRUST
(1) I believe that there could be negative consequences when using
Willingness, Benevolence (—)
(2) I feel I must be cautious when using (—)
(3) It is risky to interact with (—)
(4) I believe that (—) will act in my best interest
Gulati et al. 2019 (HCI trust scale) (5) I believe that (—) will do its best to help me if I need help
 Perceived Risk (6) I believe that (—) is interested in understanding my needs and
 Benevolence preferences
 Competence (7) I think that (—) is competent and effective in (—)
(8) I think that (—) performs its role as (—) very well
 Reciprocity
(9) I believe that (—) has all the functionalities I would expect from
(—)
(10) If I use (—), I think i would be able to depend on it completely
(11) I can always rely on (—) for (—)
(12) I can trust the information presented to me by (—)
24
TRUSTWORTHY AI?
Trust
Perceived trustworthiness
Trustworthiness?
Trustworthy AI
 Trustworthiness as general concept
 What does it mean to be benevolent, competent and have integrity?
25
IEEE STANDARD – ETHICALLY ALIGNED DESIGN
“Ultimately, A/IS (autonomous and intelligent systems) should deliver services that
can be trusted.
This trust means that A/IS will

 reliably,
 safely,
 and actively
accomplish the objectives for which they were designed while

 advancing the human-driven values they were intended to reflect.”
26
GENERAL PRINCIPLES AS IMPERATIVES
We offer high-level General Principles in Ethically Aligned Design that we consider to be imperatives for creating and
operating A/IS that further human values and ensure trustworthiness. In summary, our General Principles are:
1. Human Rights – A/IS shall be created and operated to respect, promote, and protect internationally recognized human
rights.
2. Well-being – A/IS creators shall adopt increased human well-being as a primary success criterion for development.
3. Data Agency – A/IS creators shall empower individuals with the ability to access and securely share their data, to
maintain people’s capacity to have control over their identity.
4. Effectiveness – A/IS creators and operators shall provide evidence of the effectiveness and fitness for purpose of A/IS.
5. Transparency – The basis of a particular A/IS decision should always be discoverable.
6. Accountability – A/IS shall be created and operated to provide an unambiguous rationale for all decisions made.
7. Awareness of Misuse – A/IS creators shall guard against all potential misuses and risks of A/IS in operation.
8. Competence – A/IS creators shall specify and operators shall adhere to the knowledge and skill required for safe and
effective operation. 27
TRUSTWORTHY AGENTS?
1. Human Rights
2. Well-being I would trust my collaborative agent to…
3. Data
4. Effectiveness
5. Transparency
6. Accountability
Some are easier than others, but we have a
7. Awareness of Misuse way to go to achieve them all!
8. Competence
28
QUESTIONS
29
EXPLAINABLE AI
31
EXPLAINABILITY & TRUST
One of explainable AI (XAI)’s main objectives is to promote trust
Because:
- We need appropriate trust in human-AI systems
- For warraappropriate trust, we need to understand trustworthiness
- To understand trustworthiness, we need to understand if AI systems will
stick to the trust contract
- So we need to understand the AI
32
XAI & TRUST
Use case:
Search & Rescue drone to scout dangerous buildings
- We need to neither over rely on the drone, nor disuse it

 Over reliance can lead to dangerous situations, e.g. going in and hitting a dead end
 Disuse can lead to dangerous situations, e.g. going in without all available information
- For warranted trust, we need to understand trustworthiness

 We need to understand if the drone will fulfil it’s contract:
- To understand trustworthiness, we need to understand if AI systems will stick to the trust contract
 Will the drone provide truthful info about the lay-out of the building?
- So we need to understand the AI

 We need to understand when information is incomplete, or perhaps uncertain.
 This is easier if we understand the decision making process! (E.g. sometimes the drone confuses a window with a door)
33
XAI
If the goal is to promote warranted trust, an explanation should give insight into:
Whether a system will fulfil a contract
 E.g.
 I am 90% reliable on classifying pictures of cats and dogs
 I have been trained on pictures of high quality
 I am not capable of doing anything else
XAI attempts to increase understandability & predictability of systems to that end

 Provide information about it’s purpose/capabilities
 Provide information on how a decision was made
 Provide information on what information it based that decision on
34
XAI
Explanation: “the details or reasons that someone gives to make something clear or easy to understand”1
So the other needs to understand something!
So you need to understand the other
De Graaf and Malle:

“because people assign human-like traits to artificial agents, people will expect explanations using the
same conceptual framework used to explain human behaviours.” 2
1 https://dictionary.cambridge.org/es/diccionario/ingles/explanation
2 Miller 2019
35
XAI
•Social science: understanding people

•HCI: understanding interaction
•AI: understanding system
Further reading: Explanation in artificial intelligence: Insights from the social

sciences, Miller 2019
36
EXPLANATIONS
• Contrastive
• Selected
• Causality over probability
• Social
37
EXPLANATIONS
• Contrastive Why did the branch fall of the tree?
• Selected
• Causality over probability 1. Branches have a .05% chance of
breaking
• Social
2. Because there was a strong wind
38
EXPLANATIONS
• Contrastive Why did Elisabeth open the door?
Instead of leaving it closed
• Selected
Instead of turning on the airco
• Social
1. Because she was hot
2. Because the air-conditioner wasn’t
working
You need to understand what the

alternative was!
39
EXPLANATIONS
• Contrastive How did you boil that egg?
• Selected
1. I put it in boiling water for 5 minutes
• Social
2. I took the egg out of the fridge, put it on
the countertop, filled a pan with water,
put on the stove, waited until the water
was boiling, put the egg in carefully,
waited for 5 minutes, took the pan off
the heat, took out the egg with a spoon,
drained the water, and put the egg on a
plate.
40
EXPLANATIONS
• Contrastive Sorry I was late, I got stuck in traffic this
morning.
• Selected
Please put on your coat, it’s cold today.
• Social
You can probably imagine the type of

social situations and relationships
involved here. Explanations happen in a
social context!
41
EXPLANATORY QUESTIONS
Abnormal or unexpected events (from the questioner’s point of view)
Goal:
Create shared understanding
Creating trust:
- Persuasion that a decision was correct might sometimes be better than to achieve
full shared understanding.
 Do keep the long-run in mind though!
42
EXPLANATORY QUESTIONS
43
EXPLAINING YOURSELF
Explanations are causal
XAI attempts to explain itself / it’s own reasoning
So:
XAI agents should be able to reason about their own causal model
This is a lot easier for some agents then others!

 Goal based agents have explicit internal states & rules
 Data-driven AI typically has not
44
CAUSALITY & COUNTERFACTUALS
Regularity theory:
- There is a cause between event type A and event type B if :
 events of type A are always followed by events of type B
Classical counterfactual model of causality:

- Event type C causes an event type D if:
 under the hypothetical case that C did not occur, D also wouldn’t have occurred
Probabilistic theories:
- Event type E causes an event type F iff:
 The occurrence of an event type E increases the probability of F occurring
45
CONTRASTIVE
The agent giving the explanation should understand the counterfactual case
 Explanations are asked for when unexpected or abnormality is detected.
 The counterfactual case is the expected / normal
What happened: Fact

Counterfactual case: Foil
Why did Elisabeth open the door? (foil: leave it closed)

 Because she was hot
Why did Elisabeth open the door? (foil: turn on the air-conditioner)
 Because the air-conditioner doesn’t work
47
FACT & FOIL?
For yourself: write down the fact & foil for these questions:
 Try how specific you can be!
Why did Mary wake up at 12:00?
Why did she eat pizza for breakfast?
Why did he go for a run today?
48
FINDING THE FOIL?
To understand what is ‘normal & expected’, you need to understand either:
 The world
 The user
Understanding the world is challenging to AI

Understanding the user might be easier
 What did the person react to?
 Look at?
 Point towards?
49
SELECTING EXPLANATIONS
Causal chains
 To give an explanation, one should first know some causes for what needs to be explained
Why did he die?
Because his heart failed – Because he had poison in his body – Because he drank poison – Because
someone put poison in his drink – Because he had betrayed them
51
Causal chains
Fact and foil

 To select a cause, people look at the difference between fact and foil
“This theory rests on two ideas. The first is that the effect or the explanandum, i.e. the event to be explained,
should be construed, not as an object’s having a certain property, but as a difference between objects with
regard to a certain property. The second idea is that selection and weighting of causes is determined by
explanatory relevance.” [Emphasis from the original source] — Hesslow[69, p. 24].
52
Causal chains
 Because his heart failed – Because he had poison in his body – Because he drank poison – Because
Fact and foil

 Fact: he died. Foil: he would still be alive.
53
Causal chains
Fact and foil

Intentional vs. unintentional

 “Causes will be “traced through” a proximal (more recent) abnormal condition if there is a more distal
(less recent) event that is intentional. “
54
Causal chains
 Because his heart failed – Because he had poison in his body – Because he drank poison – Because
Fact and foil

 Fact: he died. Foil: he would still be alive.

 “Causes will be “traced through” a proximal (more recent) abnormal condition if there is a more distal
(less recent) event that is intentional. “
 Because he drank poison, because someone put poison in his drink, because he had betrayed them.
55
Causal chains
Fact and foil


 “Causes will be “traced through” a proximal (more recent) abnormal condition if there is a more distal (less
recent) event that is intentional. “
Necessity & sufficiency

 Necessary causes are preferred to sufficient causes
Responsibility
 An event considered more responsible for an outcome is likely to be judged as a better explanation than
other causes
56
EVALUATING EXPLANATIONS
1. Coherence
2. Simplicity
3. Generality
4. Truth
5. Probability
Roughly in order of importance.

So a coherent and simple explanation might be better than a very truthful one!
57
SOCIAL EXPLANATIONS
Explanations are a part of conversation.
So they typically follow the rules of conversation:
 Quality
 Say what you believe to be true
 Quantity
 Say only as much as is necessary
 Relation
 Say only what is relevant
 Manner
 Say it in a nice way
58
SOCIAL EXPLANATIONS
Theory of mind
 Understand what the other already knows
 Why did the man die?
 When asking the coroner: it was poison
 When, afterwards, asking the perpetrator: because he betrayed me
Social norms
 Norms govern what we ‘should’ or ‘should not’ do
 Prohibitions, Obligations, Permissions
 Fact and foil?
 Norms relate what is socially expected
 “I will not shake your hand today, as I have a cold”.
59
QUESTIONS
60
Trust
“if A believes that B will act in A’s best interest, and accepts
vulnerability to B’s actions, then A trusts B”
Trust as a contract
Warranted & Appropriate trust
Dimensions of trust
ABI Model
RE-CAP OF TODAY Evaluating trust

Behavior
Questionnaires
Trustworthy AI
XAI
Explanations
Causality & Contrastive
Evaluating explanation
61
COLLABORATIVE AI 8. - Trust
1
TRUST
2
TRUST IN MAS
Goal: Social control mechanism
To help multi-agent systems in becoming robust

Decentralized control
Soft security
 Security while maintaining individual autonomy
3
REPRESENTING TRUST
To keep in mind when judging a representation:
 Inspired by social concepts
 But with a goal (e.g. security)
Expressiveness
Simplicity
4
REPRESENTING TRUST - BOOLEAN
No trust Trust
5
REPRESENTING TRUST - BOOLEAN
No trust Trust
Simple
Easy to base decisions on
Not very expressive
Not very fine-grained
6
REPRESENTING TRUST - NUMBERS
Integer scale
Interval
But also think about
-1 to 1
 What does 0 mean?
-5 to 5
 Is 0,5 good?
0 to 10  Is 0,6 better in an important way?
 When do you act in what way?
7
REPRESENTING TRUST - NUMBERS
Integer scale Allows for direct comparison of higher-
lower trust
Interval
More fine-grained than Boolean
-1 to 1
What does trust of 0 mean? – hard to
-5 to 5 interpret for people
0 to 10 How do you get these numbers? – hard
to establish for people
8
REPRESENTING TRUST - LABELS
Trust is by nature a vague concept.
What really is the difference between 0,6 and 0,7?
Labels might represent this better?

{bad, neutral, good}
{very_bad, bad, neutral, good, very_good}
9
REPRESENTING TRUST - LABELS
Easier for people to interpret than
Trust is by nature a vague concept. numbers
What really is the difference between 0,6 and 0,7? More fine-grained comparison than
Boolean
Labels might represent this better? Are distances between labels equal? –
more difficult to compare than numbers
{bad, neutral, good}
{very_bad, bad, neutral, good, very_good}
10
REPRESENTING – UNCERTAINTY & VAGUENESS?
When we talk about trust, we’re often not precise
 How much do you trust your doctor on a scale from 0 to 10?
Trust is difficult to define precisely!
So how do we add uncertainty & vagueness in representations?

2 possible options:
 Probability distributions
 Fuzzy sets
11
PROBABILITY DISTRIBUTIONS
Probability distribution over
sorted discrete sets
Represent uncertainty
[very bad, bad, normal, good,

very good]
A: 0.75 p of very bad, 0.25 p

of bad
B: 0.5 p of very bad, 0.5p of
very good
C: Completely unpredictable
12
FUZZY SETS
Set where the membership is ‘fuzzy’, e.g. as represented with a truth level/membership level
 Your reputation is 30 out of 100. R=30
 Set: not very trustworthy, membership range 0-1
 Truth value of you not being very trustworthy: 0,7
Fuzzy sets don’t reason about uncertainty, they reason about vagueness!
Probability:
 0,75 p of you being trustworthy
Fuzzy set
 3/10 reputation: {not very trustworthy, membership value = 0.7}
13
FUZZY SETS
Often help to describe meaning of
language
 Very much
 Somewhat
 A little
Example For trust:

Linguistic modifiers (extremely, somewhat)
modify how wide the ‘fuzzy band’ is.
 Fuzzy band: level of truth
14
PROBABILITY VS FUZZY
John is very tall to huge, which means he is either 1,9m or 2m tall
{very tall, 0.5},{huge, 0.5}
 Very tall = 1,9m & Huge = 2m
John is 1,95 tall. This is referred to as either very tall, or huge depending on who you ask
{very tall, 0.5},{huge, 0.5}
 John is 1,95m, half of the people would describe this as very tall, the other half as huge
Probability: uncertainty
Fuzziness: vagueness in language
15
PROBABILITY VS FUZZY
Good at representing uncertainty Good at representing vagueness
Usable with both integers or labels Can capture language use
Easy to interpret
Difficult to understand for many
More complex than other options How do you establish truth values?
How do you establish uncertainty?
16
HOW WOULD YOU REPRESENT?
You are building a team manager robot which will work in a storage center. This manager robot is in charge of
delegating tasks to a diverse team of different robots (there are no humans involved in the team). It uses the concept
of trust to do this.
For every trust decision, the robot wants to use the simplests representation of trust which still expresses what it needs.
1. In one situation, the manager robot is meeting another robot, and it needs to decide whether or not to trust this other robot for this
task. Given the story above, what representation of trust should the manager robot use and why?
2. In another situation, the manager robot needs to decide between two different robots. It needs to choose which robot it trusts to
do the task more. Given the story above, what representation of trust should the manager robot use and why?
3. In a third situation, the robot needs to communicate to an external human regulatory operator about its team. It needs to
communicate about how much it trusts its different robot team mates. The robot isn’t always sure of how trustworthy the other
robots are, however. Given the story above, what representation of trust should the manager robot use and why?
4. In a fourth situation, the manager robot has a new job in a new storage center, which does include human teammates. The
previous manager (human), debriefs the robot manager about the robot team. They tell the robot manager that robotA is
somewhat untrustworthy, that robotB is very trustworthy, and that robotC is extremely untrustworthy. Give a representation of the
trustworthiness of these three robots using fuzzy sets. Describe your truth value and trust value, and why you chose those values
for that specific robot. You can determine the ranges yourself, but mention them and give a justification for your choice.
17
BDI - TRUST
Belief
Desire
Intention
Trust is represented in terms of beliefs:
I belief you are x trustworthy

I believe there is y chance of you being untrustworthy
I believe you have reputation score of z
18
TRUST BELIEFS
Depending on how you define trust
You also need to specify your belief
Trust: “an agent i trusts another agent j in order to do an action α with respect to a
goal ϕ” 1
Then we need an i, j, α and ϕ!
1 Castelfranchi & Falcone
19
REPAGE
We can infer that:
BDI RepAge
 Representing reputation as beliefs
 Probability distribution S(buy(j), V BadProduct, 0.6, seller)
S(buy(j),BadProduct, 0.1, seller)
Reputation value: S(buy(j),OKProduct, 0.1, seller)
S(buy(j),GoodP roduct, 0.1, seller)
Rep( j, seller, [0.6,0.1,0.1,0.1,0.1])
 j = the agent S(buy(j), V GoodProduct, 0.1, seller)
 seller = role of the agent
 Probability defined over the discrete set: S = what people/agents say
 {VBadProduct,BadProduct,OKProduct,GoodProduct,VGoodProduct}  So subjective!
Format: S(action(agent), outcome, probability, role j)

20
WILLINGNESS & COMPETENCE
Castelfranchi & Falcone (2011):
First, x has a goal g x tries to achieve by using y: this is what x would like to
"delegate to" y, its "task"
Then x has some specific beliefs:
1. Competence Belief: a positive evaluation of y is necessary, x should believe that y
is useful for this goal of its, that y can produce/provide the expected result, that y
can play such a role in x’s plan/action, that y has some function.
2. Disposition Belief: Moreover, x should think that y not only is able and can do that
action/task, but y actually will do what x needs. With cognitive agents this will be a
belief relative to their willingness: this make them predictable.
21
TRUST PROCESSES IN MAS
MAS = Multi-Agent Systems
Trust processes in MAS often consist of two main parts:

Trust evaluation
 Trust as an expectation
 I believe you will do X
Trust decision
 Trust as an action
 I will rely on you to do X
22
TRUST PROCESS IN MAS
Image (Conte & Paolucci), “an evaluative
belief; it tells whether the target is good or
bad with respect to a given behaviour”
 Direct experiences
 Information about interactions between trustor and trustee
 Communicated experiences
 Information about interaction between trustee and other agent
 Social information
 What is the social position of the trustee?
Reputation:
 “what a social entity says about a target regarding
his or her behaviour”
 (often): Communicated image rather than experience!
23
TRUST PROCESS IN MAS
Self motivations
 Stakes
 Risk tolerance
Context
 The current context is different from ‘normal’ (from
past & communicated experiences)
 Good car mechanic, but right now I see a long cue
24
ABI VS MAS?
25
ABI VS MAS?
Willingness & Competence?
26
TRUST EVALUATION
How do we incorporate what input?
Filtering
 Context
 Are the experiences relevant for the current context?
 Information source
 Is this source reliable
 Own experiences similar to source? (use distance as reliability measure)
 Correlated evidence?
 Many communicate about the same experience
 Experiences are typically not identified!
 Consider social circles?
27
AGGREGATION
Discrete labels:
 9 & 10 = very good
 8 & 7 = good
 6 & 5 = okay
 4 & 3= bad
 1 & 2 = very bad
t(trustee, context, {bad, okay, good, very good}

What label is given most?
 By trustor / by others?
 Good trustworthiness
28
AGGREGATION
With numbers, you can do more!
Average?
 Input from others: (4+7+8+6+7+7+7)/7
Weighted average
 Recency of input
 Reliability of the source
 One per social cluster?
29
TRUST BELIEFS
DispTrust(Alice,Tom, inform(weather),
Don’t necessarily arise from calculations know(Alice,weather), asked(Alice,
Tom,weather))
Alice trusts Tom to inform her about the
Rather, they are defined! weather (goal: Alice knows about the weather) if
 Trust evaluation belief (dispositional trust) Alice asks Tom about this.
 Trust decision belief
You are trustworthy when some conditions are

satisfied
 Typically about action, goal & contextual conditions
30
TRUST BELIEFS - EXAMPLE
ForTrust Model:
Dispositional trust:
i = trustor PotGoali(𝜑 ,K) = trustor i has the potential goal 𝜑 given circumstances K
j = trustee G 𝜑 = globally 𝜑 (always, temporally)
a = action F 𝜑 = eventually 𝜑 (¬G¬ 𝜑)
𝜑 = goal Choicei𝜑 = i has chosen goal 𝜑
K = conditions
Afterj:a;𝜑 = immediately after j does a 𝜑 holds
31
TRUST BELIEFS - EXAMPLE
ForTrust Model:
Dispositional trust:
i = trustor
j = trustee
a = action
𝜑 = goal
K = conditions
32
TRUST DECISION
Trust evaluations lead to trust decisions
 Formalization of evaluation directs formalization of
decision
If trust evaluation > x, decide a, otherwise b

 So links a numerical value from the trust evaluation to
whether to trust or distrust
 If trust evaluation > 6: trust
 Stakes can help set the threshold!
33
QUESTIONS
35
REPUTATION
Related to trust, though slightly different
But many definitions exist!
One that we will use:

 What a social entity says about a target regarding his or her behaviour.
 Social entity: group of individuals (but irreducible to those individuals)
 Says: what is communicated (so doesn’t need to be equal to agent’s belief!)
 Here: behaviour
 Other uses: Image, trust evaluation
 Target: trustee
36
REPUTATION
Reputation is typically used to calculate trust if direct experience isn’t available
Reputation is subjective
 It depends on the opinions of others!
So how much should we rely on it?
Option:
 Calculate reliability value
 Number of opinions it is based on
 Variance of opinions it is based on
 Recency of the opinions
 Credibility of the agents behind the opinions
37
REPUTATION
Option: agent(time, agent(time,
Reputation (Mean) = 6,1
 Calculate reliability value Nr. Agents : 9
opinion, opinion,
 Number of opinions it is based on Variance: 5,88
credibility) credibility)
 Variance of opinions it is based on Recency (now t=5): 2,6
 Recency of the opinions Credibility agents: 6,7
a1(1, 2, 7) a1(2, 6, 5)
 Credibility of the agents behind the
opinions a2(1, 4, 8) a2(3, 7, 8)
a3(1, 4, 6) a3(3, 8, 4)
Reputation (Mean) = 6,8
a4(2, 5, 7) a4(4, 5, 5)
Nr. Agents: 5
a5(2, 9, 8) a5(4, 8, 7)
Which reputation value is Variance: 1,7
a6(3, 5, 5)
Recency (now t=5): 1,8
more reliable? a7(3, 8, 7)
Credibility agents: 5,8
a8(3, 9, 6)
a9(4, 7, 8)
a2(4, 8, 8)
39
REPUTATION PROCESS
Basic components:
Communicated image
Communicated reputation
Social Information
40
REPUTATION PROCESS
Communicated image
 Image: “an evaluative belief; it tells whether the
target is good or bad with respect to a given
behaviour”
 Communicated image aggregates!
 Important that the individuals who communicate
the image are representative of the whole
 So small samples can be a problem!
Solution: reputation of communication?

 Social information
 Most Representative
 Most credible
41
REPUTATION PROCESS
Communicated Reputation
 Aggregation of reputation information of others
 Image: evaluative belief, Reputation: what others say about
behavior
42
REPUTATION PROCESS
Communicated Reputation
 Aggregation of reputation information of others
Inherited reputation
 Reputation which is directly taken from third-
party agents with who the agent has a social
relation
 Reputation doesn’t depend on behaviour of
subject!
 Reputation associated with the role the subject
has in society
43
(DE)-CENTRALIZED REPUTATION?
Centralized reputation:
 A central service collects the raw information and aggregates into a reputation
 All individuals in the community contribute
 Less impact of a few outliers
 Public reputation, so newcomers benefit
 But:
 Trust in the central service is crucial
 Personal biases and preferences are ignored
 Vulnerability, as the central service is a bottleneck
 Examples?
 eBay reputation
 Internet review sites
44
(DE)-CENTRALIZED REPUTATION?
De-centralized reputation:
 Relies on individual information that each agent can obtain
 No external entity to trust
 Scalable
 Different agents can follow different reputation functions
 But:
 Takes time to collect enough information
 Requires more complex agents, as the calculation process is more complex
 Examples:
 Most multi-agent reputation models
45
USING REPUTATION
Reputation as source of trust
 Useful if direct information is lacking
Reputation for social order

 Social order: social structures, institutions and practises that conserve, maintain and
enforce the ‘normal way’ of relating and behaving
 Social norms
 Shake someone’s hand when meeting them
 Reputation to specify what is socially acceptable
46
USING REPUTATION - PITFALLS
Unfair ratings
 Deliberate sending wrong information
Ballot stuffing
 Send more information than is warranted (e.g. 4 messages about 1 interaction)
Dynamic personality
 Agents taking advantage of high reputations
Whitewashing
 Change your name/id to escape a bad reputation
Collusion
 Unfair cooperation (for instance as a group give inflated scores)
Sybil attacks
 Create ‘fake’ identities (for instance to all give you high ratings)
Reputation lag
 Take advantage of delay between bad action and bad reputation
47
TRUST IN NEGOTIATION
Negotiate about trust
 When a group of agents together need to make
trust decisions
 What characteristics are important for us to trust
another agent/agency?
Trust in negotiation
 Evaluate your negotiating partner
 Do you trust them to keep their word?
48
SOCIAL NORMS
Norm: social rule for behaviour
 You do not leave a party without saying goodbye to the host
 You bring a present to a birthday
 You do not walk away during a conversation
Social norms are shared among a social group

Social can be broken, but that often has a social consequence
How do you assess whether someone will stick to a norm?

 Trust to assess norm obedience
49
QUESTIONS
50
RESEARCH PROJECTS Q4
User-aware eXplainable AI for improving human-AI teamwork
Agent tailoring
explanations
• How can an agent model and use human trust in the agent to tailor explanations? to human
• How can an agent model and use human workload to tailor explanations?
• How can an agent model and use (epistemic) relevance to tailor explanations?
• How can an agent model and use human understanding of the agent to tailor explanations?
• How can an agent model and use human situation awareness to tailor explanations?
• How can an agent model and use human reliance on the agent to tailor explanations? Agent actions
• How can an agent model and use human performance to tailor explanations? affecting human
trustworthiness
Towards trust in human-AI teams

• How does (an artificial agent) offering help affect human trustworthiness?
• How does (an artificial agent) conflicting with human actions affect human trustworthiness?
• How does (an artificial agent) asking human for advice/help affect human trustworthiness?
• How does (an artificial agent) directing the human affect human trustworthiness?
• How does (an artificial agent) being directable by the human affect human trustworthiness?
• How does (an artificial agent) being friendly affect human trustworthiness?
51
Computational Social Choice
CSE3210 Collaborative Artificial Intelligence
Luciano Cavalcante Siebert

Assistant Professor
Interactive Intelligence group, Intelligent Systems Department, EEMCS
l.cavalcantesiebert@tudelft.nl
1
You have learned about
• Centralized x collaborative AI
• (Automated) Negotiation
• Multiagent teamwork: Coalition formation
• Human-agent teamwork: Coactive design
• Trust
Next
• Computational social choice
2
Outline
• What is (computational) social choice theory?
• Preference aggregation
• Social Welfare Functions
• Axioms
• Arrow’s theorem
3
Study material
• Slides
• Chapter 6 of the book

– First lecture: 1; 1.1; 1.2; 1.3; 2; 2.1
– Second lecture: 3; 3.1; 3.2 ;3.2.1; 3.2.2;3.2.3
• Additional material:
– Handbook of Computational Social Choice
http://www.cse.unsw.edu.au/~haziz/comsoc.pdf
– Articles mentioned in the slides
4
Part 1: Introduction and Preference Aggregation

Assistant Professor
5
Introductory example Group A
4 people
Group B
3 people
Group C
2 people
1st Gin tonic Beer Wine
• House party: 2nd Wine Wine Beer
– You are planning a party for 9 3rd Beer Gin tonic Gin tonic
people
– You only want one type of
drink in the party
– You asked the guests to
provide their preferences
beforehand
– Which drink should you serve?
?
?
6
Introductory example Group A
4 people
Group B
3 people
Group C
2 people
1st Gin tonic Beer Wine
• House party: 2nd Wine Wine Beer
3rd Beer Gin tonic Gin tonic
Plurality rule: Only considers how often each alternative is ranked in the first place Gin tonic
• However, the majority (groups B and C combined) will be dissatisfied with this choice
Condorcet method: Determine which alternative defeats every other alternative in pairwise
majority comparisons:
Wine
- Gin preferred to wine? 4 people - Wine preferred to gin? 5 people
- Beer preferred to wine? 3 people - Wine preferred to beer? 6 people
7
Group A Group B Group C
Introductory example 1st
4 people
Gin tonic
3 people
Beer
2 people
Wine
2nd Wine Wine Beer

• House party: 3rd Beer Gin tonic Gin tonic
• Another method:
Single transferable vote (STV): eliminate the options that are ranked first by the lowest
number of agents
- Remove wine from the choice set (first option of only 2 people)
Beer
A (4) B (3) C (2)
1st Gin tonic Beer Beer
2nd Beer Gin Gin
8
Introductory example
• This simple example shows that collective choice is not a trivial matter
What method
should I use?
How to deal
with strategic
manipulation?
How to deal
with ties?
9
Social choice theory
• Social choice theory: methods for aggregating the preferences of
multiple agents
– How to fairly allocate resources to the members of a society?
– How to elect a president given people’s preferences?
– How to aggregate the views of different judges in a court case?
• Origins:
Political
Economics Mathematics
science
10
Social choice theory
• Early days:
– Jean-Charles de Borda (1733-1799)
– Marquis de Condorcet (1743-1797)
– Charles L. Dodgson a.k.a. Lewis Carroll (1832-1898)
• Social choice theory as a scientific discipline:

– Kenneth J. Arrow (1921-2017)
• Arrow’s impossibility theorem
– Amartya Sen (1933)
11
Computational social choice
• Computational social choice adds an algorithmic perspective to the
formal approach of social choice theory
12
• Computational social choice:
– Bartholdi, Orlin, Tovey, and Trick (around 1990):
Complexity theory as a barrier for strategic manipulation
– U. Endriss and J. Lang: 1st International Workshop on
Computational Social Choice (COMSOC, 2006)
13
A
• Multiagent system as a “society” of agents A2
A
– Recommender systems on the basis of choices
made by other users in the past B
– Metasearch engine that combines results of A3
several internet search engines
A1
B
A
A4
A4
Valcarce, D., Parapar, J., & Barreiro, Á. (2017). Combining top-n recommenders with metasearch algorithms. In ACM SIGIR
14
A
• Multiagent system as a “society” of agents A2
A
– Collaborative large-scale peer grading (e.g.
MOOC) B
– Ethical decision making: Bottom-up approaches to A3
aggregate people’s (moral) preferences
A1
B
A
A4
A4
Caragiannis, I. (2017). Recent advances in large-scale peer grading. Trends in Computational Social Choice, 327.
Noothigattu, R., et al. (2018). A voting-based system for ethical decision making. In AAAI Conference on Artificial Intelligence. 15
Fundamentals
• How to compare?
– Cardinal utility*
• Is today warmer than or colder than yesterday?
• 𝑇𝑡𝑜𝑑𝑎𝑦 = 10°𝐶
• 𝑇𝑦𝑒𝑠𝑡𝑒𝑟𝑑𝑎𝑦 = 8°𝐶
• Today is warmer than yesterday, by 2°𝐶
– Ordinal utility (or preference)
• Do you prefer pizza or pasta? Do you prefer candidate A or B?
• In the absence of a common basis of comparison (e.g. money,
temperature), the meaning of individual (cardinal) utility values
is quite controversial
• Specially, how to compare between different people
(interpersonal comparisons)?
16
Fundamental
• In our lectures we will follow the research in computational social choice with
the classical perspective in which agents (or voters) are assumed to have
ordinal preferences
• There is, however, a different, utilitarian approach to the aggregation of group

preferences that assumes individuals have cardinal utility functions that
quantify the desirability of various outcomes*
– Usually this means to maximizes the sum of the individual utilities, that is, social
welfare
*For a review on Social Choice from a Utilitarian Perspective (i.e. using cardinal preferences), see section 10.8 of the
Handbook of Computational Social Choice http://www.cse.unsw.edu.au/~haziz/comsoc.pdf 17
Fundamentals
• Let’s consider a finite set 𝑁 = {1, … , 𝑛} of at least 2 agents (a.k.a
individuals, voters),
• and a finite universe 𝑈 = {𝑎, … , 𝑧} of at least 2 alternatives (a.k.a
candidates)
• Each agent 𝑖 has preferences over the alternatives in 𝑈, which are
represented by a transitive and complete preference relation ≿𝑖
– Transitivity: If 𝑎 ≿𝑖 𝑏 and 𝑏 ≿𝑖 𝑐 implies that 𝑎 ≿𝑖 𝑐
– Completeness: requires any pair of alternatives 𝑎, 𝑏 ∈ 𝑈 to be comparable
– Usually, we also assume antisymmetry: 𝑎 ≿𝑖 𝑏 and 𝑏 ≿𝑖 𝑎 implies that 𝑎 = 𝑏
• We denote 𝑎 ≻𝑖 𝑏 (strict preference), if 𝑎 ≿𝑖 𝑏 but not 𝑏 ≿𝑖 𝑎
• A preference profile is denoted as 𝑅 = ≿1 , … , ≿𝑛
18
Preference Aggregation
• Preference aggregation is the aggregation of several agents' preference
rankings of two or more alternatives into a single, collective preference ranking
(or choice) over these alternatives
• It can be seem as a Social Welfare Function (SWF) or a Social Choice

Function (SCF)
19
ℛ(𝑈)𝑛 ℛ(𝑈)
• Aggregates ≿1 𝑎 ≿1 𝑏 …
Social preferences of
Welfare individual agents ≿2 𝑎 ≿2 𝑏 ≿
Function into collective
…
(SWF) preferences
≿𝑛 𝑏 ≿𝑛 𝑎
• 𝑓: ℛ(𝑈)𝑛 → ℛ(𝑈)
20
ℛ(𝑈)𝑛 ℱ(𝑈) ℱ(𝑈)

For
• Maps the individual preferences ≿1 example:
Social of the agents and a feasible ?
subset of the alternatives (e.g. ≿2 𝑎
Choice the set of all non-empty subsets
…
Function of U) to a set of preferred 𝑏
…
(SCF) alternatives, a.k.a. the choice set
• 𝑓: ℛ(𝑈)𝑛 × ℱ(𝑈) → ℱ(𝑈) ≿𝑛
…
21
• A variety of interesting SWFs and SCFs have been proposed, often based on
mechanisms that seem to calculate the winner in an intuitively “fair” way
• After further analysis these mechanisms may show unintended consequences
that undercut their initial appeal
• Instead of based solely on intuitive appeal, social choice theorists argue they
should rely on axioms, i.e. precisely defined desirable properties
Axiom Property
– An axiom is a statement that is assumed to be true in order to help provide a

foundation from which other statements can be proved
– A property is a statement that is true in some particular context
22
• Not every SWF or SCF will satisfy every axiom
• In this lecture we will consider axioms for SWFs, but very similar
axioms can be applied to SCFs
• What properties would you consider desirable for a SWF?
23
Some basic axioms
• Anonymity: symmetry w.r.t. agents
– If alternative 𝑎 wins with 𝑛1 and 𝑛2 for and 𝑛3 against, then 𝑎 should also
win with 𝑛2 , 𝑛3 for and 𝑛1 against
• Neutrality: symmetry w.r.t. alternatives
– If we swap alternatives 𝑎 and 𝑏 in a ballot, the outcome should be the
same
• Positive responsiveness:
– If the group decision is indifferent or favourable to 𝑎, and if individual
preferences remain the same except a single agent changes their ranking
in favour of 𝑎, then the group decision should favour 𝑎
24
May’s Theorem
• May’s Theorem: For two alternatives and an odd number of agents, majority
rule is the unique SWF that satisfies anonymity, neutrality, and positive
responsiveness
– Majority rule: Given two alternatives, 𝑎 should be preferred over 𝑏, iff (if and only if) there are
more agents who prefer 𝑎 to 𝑏, than 𝑏 to 𝑎
• 𝑈 = 𝑎, 𝑏
• 𝑁 = {𝑛1 , 𝑛2 , 𝑛3 }
𝑛1 𝑛2 𝑛3 Kenneth O. May
a a b 𝑎≻𝑏
b b a
25
May’s Theorem
• Anonymity
𝑛1 𝑛2 𝑛3 𝑛1 ′ 𝑛2 ′ 𝑛3 ′
a a b a b a 𝑎≻𝑏
b b a b a b
• Neutrality
– E.g. if votes are reversed, results follow
𝑛1 𝑛2 𝑛3
b a b 𝑏≻𝑎
a b a
26
May’s Theorem
• Positive responsiveness
𝑛1 𝑛2 𝑛3 𝑛1 𝑛2 𝑛3
a a b 𝑎≻𝑏 a a a 𝑎≻𝑏
b b a b b b
27
• What happens if we add a third alternative?
– Majority rule can result in cycles when there are more than two alternatives
1 1 1 a
Majority (2 out of 3): 𝑎 ≻ 𝑏
a b c
Majority (2 out of 3): 𝑏 ≻ 𝑐
b c a
Majority (2 out of 3): 𝑐 ≻ 𝑎
c a b c b
– The pairwise majority relation in this example is cyclic, therefore not a well-formed
preference relation
– Hence, the majority rule does not satisfy anonymity, neutrality, and positive
responsiveness for 𝑈 ≥ 3
• Marquis de Condorcet was the one who first noted that social preference relations can
be problematic. We call these cyclic relations a Condorcet paradox
28
Axioms
Formalizing the Axioms:
• A SWF 𝐹 is anonymous if agents are treated symmetrically:
𝐹 𝑅1 , … , 𝑅𝑛 = 𝐹 𝑅𝜋(1) , … , 𝑅𝜋(𝑛)
for any profile 𝑅 and any permutation 𝜋: 𝑁 → 𝑁
• A SWF 𝐹 is neutral if alternatives are treated symmetrically
𝐹 𝜋(𝑅) = 𝜋 𝐹(𝑅)
for any profile 𝑅 and any permutation 𝜋: 𝐴 → 𝐴
29
Axioms
• Positive responsiveness:
– For all 𝑎, 𝑏 ∈ 𝑈 and any two profiles 𝑅 and 𝑅′ ∈ ℛ(𝑈)𝑛 ,
– if 𝑎 ≿ 𝑏 for 𝑅 and
– for 𝑅′ a single agent changes their ranking in favour of 𝑎
– then 𝑎 ≻ 𝑏
• Are there other axioms that might be interesting?

– In this lecture we will see Pareto Optimality (a.k.a. unanimity),
independence of irrelevant alternatives (IIA), and non-dictatorship
30
Axioms
• An SWF satisfies the Pareto condition if, whenever all individuals rank 𝑎
above 𝑏, then so does society
𝑎 ≻𝑖 𝑏 for all 𝑖 𝜖 𝑁 implies that 𝑎 ≻ 𝑏
for all 𝑖 𝜖 𝑁
𝑎 ≻𝑖 𝑏
≿1 𝑎 ≻1 𝑏 ≻1 𝑐 ≻1 𝑑 𝑎 ≻𝑖 𝑐 𝑎≻𝑏≻𝑐≻𝑑 𝑏≻𝑎≻𝑐≻𝑑
≿2 𝑎 ≻𝑖 𝑑 𝑑≻𝑎≻𝑏≻𝑐 𝑑≻𝑏≻𝑐≻𝑎
𝑐 ≻2 𝑑 ≻ 2 𝑎 ≻2 𝑏
…
𝑐≻𝑎≻𝑑≻𝑏 𝑐≻𝑏≻𝑑≻𝑎
≿3 𝑑 ≻ 3 𝑎 ≻3 𝑐 ≻3 𝑏 𝑑 ≻𝑖 𝑐
…
31
Axioms
• An SWF satisfies independence of irrelevant alternative (IIA) if the social
preferences between any pair of alternatives only depends on the individual
preference profiles restricted to these two alternatives
– If 𝑎 is socially preferred to 𝑏, then this should not change when 𝑖 changes
its ranking of 𝑐
– Let 𝑅 and 𝑅’ be two preferences profiles and 𝑎 and 𝑏 be two alternatives
such that 𝑅|{𝑎,𝑏} = 𝑅′ |{𝑎,𝑏} , i.e. the pairwise comparisons between 𝑎 and 𝑏
are identical in both profiles
– Then, IIA requires that 𝑎 and 𝑏 are also ranked identically:
≿ | 𝑎, 𝑏 =≿ ′| 𝑎, 𝑏
32
• Independence of irrelevant alternative (IIA). Example:
≿1 𝑏 ≻1 𝑎 ≻1 𝑐 ≻1 𝑑 Satisfy IIA Do not satisfy IIA
≿2 𝑎 ≻2 𝑏 ≻2 𝑐 ≻2 𝑑 What changed? 𝑎≻𝑏≻𝒄≻𝒅 𝑎≻𝑏≻𝑐≻𝑑

𝑎, 𝑏 Same
…
…
≿3 𝑐 ≻3 𝑑 ≻ 3 𝑎 ≻3 𝑏
𝑎, 𝑐 Same
𝑎, 𝑑 Same
ℛ′(𝑈)𝑛 𝑏, 𝑐 Same ℛ′(𝑈)
𝑏, 𝑑 Same 𝑏≻𝑎≻𝑐≻𝑑
≿1 𝑏 ≻1 𝑎 ≻1 𝒅 ≻𝟏 𝒄
𝑐, 𝑑 Changed 𝑑≻𝑏≻𝑐≻𝑎
≿2 𝑎 ≻2 𝑏 ≻2 𝑐 ≻2 𝑑 𝑎≻𝑏≻𝒅≻𝒄
𝑐≻𝑏≻𝑑≻𝑎
…
≿3 𝒅 ≻𝟑 𝒄 ≻ 3 𝑎 ≻3 𝑏
…
33
Axioms
• An SWF is non-dictatorial if there is no agent who
can dictate a strict ranking no matter which
preferences the other agents have
– There is no agent 𝑖 such that for all preference
profiles 𝑅 and all alternatives 𝑎, 𝑏, 𝑎 ≻𝑖 𝑏 implies
that 𝑎 ≻ 𝑏
34
• Why do we need axioms?
• With a formal definition of these desirable properties one can establish:
– Characterisation theorems: Show that a particular (class of)
mechanism(s) is the only one satisfying a given set of axioms
– Impossibility theorems: Show that there exists no aggregation
mechanism satisfying a given set of axioms
35
Arrow’s theorem
• Arrow’s Theorem: There exists no SWF that
simultaneously satisfies IIA, Pareto optimality,
and non-dictatorship whenever U ≥ 3
• Paul Samuelson (one of the most influential economist of the

later 20th century) considered Arrow’s theorem as one of the
significant intellectual achievements of our time
Arrow, K. J. (2012). Social choice and individual values (Vol. 12). Yale university press. First edition published in 1951.
36
Arrow’s theorem
• Let us assume a SWF that satisfies IIA and Pareto optimality, and U ≥ 3
• Step 1: If every agents puts an outcome 𝑏 at either the very top or the very
bottom of its preference list, 𝑏 must be at either the very top or very bottom of
≿ as well
1 1 1
𝑏 𝑏 𝑎 𝑏 ≻? ≻?
𝑎 𝑐 𝑐
? ≻? ≻ 𝑏
𝑐 𝑎 𝑏
– Let us consider for contradiction that the above claim is not true
– This means that there is an option above 𝑏 and an option below 𝑏, for example we
could have:
𝑎≻𝑏≻𝑐
37
Arrow’s theorem
• Let us modify the preference profile so that every voter moves 𝑐 above 𝑎, and
otherwise leave the ranking unchanged
1 1 1 1 1 1
𝑏 𝑏 𝑎 𝑏 𝑏 𝑐
𝑎 𝑐 𝑐 𝑐 𝑐 𝑎
𝑐 𝑎 𝑏 𝑎 𝑎 𝑏
• IIA says that for 𝑎 ≻ 𝑏 to change we need to change the pairwise relationship
between 𝑎 and 𝑏; the same applies for 𝑏 ≻ 𝑐
• However, as 𝑏 has an extremal position for all agents (top or bottom), 𝑐 can be
moved above 𝑎 without changing either of these pairwise relations
38
Arrow’s theorem
• From transitivity we have that if 𝑎 ≻ 𝑏 and b ≻ 𝑐 , then 𝑎 ≻ 𝑐
• However, the Pareto condition requires that 𝑐 ≻ 𝑎, since every agent ranked
𝑐 above 𝑎.
1 1 1
𝑏 𝑏 𝑐
𝑐 𝑐 𝑎
𝑎 𝑎 𝑏
• We have a contradiction!
• Hence, to respect both IIA and the Pareto condition 𝑏 must be either at the
bottom or at the top of ≿
39
Arrow’s theorem
• Step 2: There is an agent 𝑛∗ who is extremely pivotal in the sense that by
changing its preferences, it can move a given outcome 𝑏 from the bottom to
the top of ≿
• Let us assume a preference profile in which every voter ranks 𝑏 last
1 1 1
𝑎 𝑎 𝑐
? ≻? ≻ 𝑏
𝑐 𝑐 𝑎
𝑏 𝑏 𝑏
• The Pareto condition requires that 𝑏 is ranked last
40
Arrow’s theorem
• Now, let agents successively modify their preferences by moving 𝑏 from the
bottom to the top, preserving all other relative rankings
• Let us denote 𝑛∗ as the first agent who change causes the social ranking of 𝑏
to change 1 1 1
𝑏 𝑎 𝑐 ? ≻? ≻ 𝑏
≿1
𝑎 𝑐 𝑎
𝑐 𝑏 𝑏
𝑏 must be at
𝑛∗
1 1 1 the top or
bottom
𝑏 𝑏 𝑐
≿2 𝑏 ≻? ≻?
𝑎 𝑎 𝑎
𝑐 𝑐 𝑏 41
Arrow’s theorem
• There must always be some such agent as 𝑛∗ , because when all agents move
𝑏 to the top, the Pareto condition will require that 𝑏 is ranked at the top
• The only difference between ≿1 and ≿2 , are the preferences of 𝑛∗
• Step 3: 𝑛∗ is a dictator over any pair not involving 𝑏 (in our case 𝑎, 𝑐)
• Let us consider first 𝑎, and construct a new profile ≿3 from ≿2 by: 1) moving 𝑎
to the top of 𝑛∗ ’s preference ordering, and 2) rearranging the relative ranking of
𝑎 and 𝑐 for all agents other than 𝑛∗
1 1(𝑛∗ ) 1 1 1(𝑛∗ ) 1
𝑏 𝑏 𝑐 𝑏 𝑎 𝑎
≿2 ≿3
𝑎 𝑎 𝑎 𝑐 𝑏 𝑐
𝑐 𝑐 𝑏 𝑎 𝑐 𝑏
42
Arrow’s theorem
• In ≿1 , we had that 𝑎 ≻ 𝑏, as 𝑏 was at the bottom of ≿
• When we compare ≿1 to ≿3 , relative rankings between 𝑎 and 𝑏 are the same
for all agents
1 1 1 1 1(𝑛∗ ) 1
𝑏 𝑎 𝑐 𝑏 𝑎 𝑎
≿1 ≿3
𝑎 𝑐 𝑎 𝑐 𝑏 𝑐
𝑐 𝑏 𝑏 𝑎 𝑐 𝑏
• From IIA, we must have 𝑎 ≻ 𝑏 in ≿3 as well
43
Arrow’s theorem
• In ≿2 , we had that 𝑏 ≻ 𝑐, as 𝑏 was at the top of ≿ (being 𝑛∗ pivotal)
• Comparing ≿2 to ≿3 , relative rankings between 𝑏 and 𝑐 are the same
1 1(𝑛∗ ) 1 1 1(𝑛∗ ) 1
𝑏 𝑏 𝑐 𝑏 𝑎 𝑎
≿2 ≿3
𝑎 𝑎 𝑎 𝑐 𝑏 𝑐
𝑐 𝑐 𝑏 𝑎 𝑐 𝑏
𝑎≻𝑏≻𝑐
• Thus, in ≿3 , we have 𝑏 ≻ 𝑐
• As we have that 𝑎 ≻ 𝑏 and 𝑏 ≻ 𝑐, from transitivity we have that 𝑎 ≻ 𝑐 in ≿3
44
Arrow’s theorem
• Let us construct one more (last one! ) preference profile by changing ≿3 in
two ways: 1) Arbitrarily change the position of 𝑏 for each agent while keeping
all other relative preferences the same; 2) Move 𝑎 to an arbitrary position in
𝑛∗ ’s preference ordering, with the constraint that 𝑎 ≿𝑛∗ 𝑐
1 1(𝑛∗ ) 1 1 1(𝑛∗ ) 1
𝑏 𝑎 𝑎 𝑐 𝑏 𝑎
≿3 ≿4
𝑐 𝑏 𝑐 𝑏 𝑎 𝑏
𝑎 𝑐 𝑏 𝑎 𝑐 𝑐
45
Arrow’s theorem
• Note that the only assumption we have now for ≿4 is that 𝑎 ≿𝑛∗ 𝑐
• For ≿3 and ≿4 all agents have the same relative preference between 𝑎 and 𝑐
1 1(𝑛∗ ) 1 1 1(𝑛∗ ) 1
𝑏 𝑎 𝑎 𝑐 𝑏 𝑎
≿3 ≿4
𝑐 𝑏 𝑐 𝑏 𝑎 𝑏
𝑎 𝑐 𝑏 𝑎 𝑐 𝑐
• From IIA, we have that a ≻ 𝑐 only with the assumption that 𝑎 ≿𝑛∗ 𝑐
• If 𝑛∗ would change its preference to 𝑐 ≿𝑛∗ 𝑎, according to Pareto optimality
would follow that c ≻ 𝑎
• Hence, we cay that 𝑛∗ is a dictator, at least over (𝑎, 𝑐)
46
Arrow’s theorem
• Step 4: 𝑛∗ is a dictator over any pair of alternatives (including
pairs that include 𝑏).
• By the argument of Step 2, we can say that there is an agent
𝑛∗∗ who is extremely pivotal for 𝑐
• By the argument in Step 3, 𝑛∗∗ is a dictator over any pairs not
involving 𝑐
• We have previously observed that 𝑛∗ can affect the relative
preferences between 𝑎 and 𝑏 (remember ≿1 and ≿2 , when
we identified 𝑛∗ )
• Hence, as 𝑛∗∗ is a dictator over any pairs not involving 𝑐, and
𝑛∗ can influence 𝑏, we can conclude that 𝒏∗∗ and 𝒏∗ must be
the same agent!
47
Arrow’s theorem
• Take into consideration that:
– “Most systems are not going to work badly all of the time. All I proved is that all can
work badly at times”, K. Arrow
– The conditions of Arrow’s theorem can be relaxed to find more realistic solutions
– We can call the assumptions into questions e.g. in many applications, a full social
preference relation or IIA might not be needed
• Further reading and alternative proofs:

– https://www.rochester.edu/college/faculty/markfey/papers/ArrowProof.pdf
– http://dido.econ.yale.edu/~gean/art/p1116.pdf
– https://www.youtube.com/watch?v=QLi_5LCwJ20
48
Conclusion
• Today we learned the fundamentals of social choice, and its most
discussed axioms
• The purpose of presenting the limitations is not convince you that
social choice is hopeless!
• Rather, it is intended to make one think about social choice in a
precise manner and to have more realistic expectations
• In the next lecture, we will move to more concrete procedures for
making decisions based on the preference of multiple agents
– Voting rules, scoring rules, Condorcet extensions
– Strategic manipulation
49

Assistant Professor
1
Recap
• Preference aggregation
–SWF: 𝑓: ℛ(𝑈)𝑛 → ℛ(𝑈)
–SCF: 𝑓: ℛ(𝑈)𝑛 × ℱ(𝑈) → ℱ(𝑈)
• Axioms:
–Anonymity, Neutrality, Positive responsiveness
–Pareto condition, Independence of Irrelevant Alternatives (IIA), Non-dictatorial
•Arrow’s theorem:
–There exists no SWF that simultaneously satisfies IIA, Pareto optimality, and non-
dictatorship whenever U ≥ 3
2
Part 2: Voting methods

Assistant Professor
3
Outline
PART II: Voting methods
• Voting rules
• Scoring rules
• Condorcet extensions
• Strategic manipulation
4
Study material
• Slides
• Chapter 6 of the book

– First lecture: 1; 1.1; 1.2; 1.3; 2; 2.1
– Second lecture: 3; 3.1; 3.2 ;3.2.1; 3.2.2;3.2.3
• Additional material:
– Handbook of Computational Social Choice
http://www.cse.unsw.edu.au/~haziz/comsoc.pdf
– Articles mentioned in the slides
5
Voting rules
• A voting rule is a function 𝑓: ℛ(𝑈)𝑛 → ℱ(𝑈)
• Every voting rule is usually seen as a Social

Choice Function (SCF), but not always the
other way around
– A SCF, defined by 𝑓: ℛ(𝑈)𝑛 × ℱ(𝑈) → ℱ(𝑈),
require the definition of a feasible subset of the
alternatives
– Some of the SCFs can hardly be considered
voting rules because they are not
discriminatory enough
6
Voting rules
Properties often required of voting rules:
• Resoluteness: they should always yield a unique winner
– 𝑓 is resolute if 𝑓 𝑹 = 1 for all preferences profiles R
• Anonymity: symmetry w.r.t. agents
• Neutrality: symmetry w.r.t. alternatives
However, even these simple properties can often not be satisfied

• Example: Two agents and two alternatives.
1 1
a b
7
Voting rules
• Plurality rule:
– Only considers how often each alternative is ranked in the first place
3 2
𝑎 𝑏
𝑏 𝑐
𝑐 𝑎
– Winner: 𝒂
8
Voting rules
• A common objection to rules such as the plurality rule is that an alternative
ought to get some credit for being ranked, say, in second place by a voter
9
Scoring rules
• Under a scoring rule, each time an alternative is ranked 𝑖 − 𝑡ℎ by some voter,
it gets a particular score 𝑠𝑖
• For a fixed number of alternatives 𝑚, we define a score vector
𝑠 = {𝑠1 , … . , 𝑠𝑚 } such that 𝑠1 ≥ ⋯ ≥ 𝑠𝑚 and 𝑠1 > 𝑠𝑚
10
Plurality and Anti-plurality rules
• Plurality rule: We can also define the plurality rule as a scoring rule. The
score vector for the plurality rule is (1, 0, … , 0).
– The cumulative score of an alternative equals the number of voters by which it is
ranked first
• Anti-plurality rule: The score vector for the anti-plurality rule (a.k.a veto rule)
is (1, … , 1, 0).
– It choose those alternative that are least-preferred by the lowest number of votes.
– Sometime the anti-plurality rule is also defined as (0, … , 0 , -1)
11
Plurality and Anti-plurality rules
3 2
• Plurality rule: (1, 0, … , 0)
𝒂: 𝟑 𝑎 𝑏
𝑏: 2 𝑏 𝑐
𝑐: 0
𝑐 𝑎
• Anti-plurality rule: (1, … , 1, 0)
𝑎: 3
𝒃: 𝟑 + 𝟐 = 𝟓
𝑐: 2
12
Borda’s rule
• Borda’s rule: Alternative 𝑎 gets 𝑘 points from voter 𝑖, if 𝑖 prefers 𝑎 to 𝑘 other
alternatives, i.e. the score vectors is ( 𝑈 − 1, 𝑈 − 2, … , 0)
– Borda rule chooses those alternatives with the highest average rank in individual rankings
3 2
𝑎 𝑏 2 points 𝑎: 3 × 2 = 6
𝑏 𝑐 1 point 𝒃: 𝟐 × 𝟐 + 𝟑 × 𝟏 = 𝟕
𝑐: 2 × 1 = 2
𝑐 𝑎 0 point
13
Research example
• Ethical decision making for autonomous systems
– We want autonomous systems to respect given ethical principles… but which principles should
they favor in case of dilemmas?
– We modeled 6 ethical principles, and analyzed people’s preference w.r.t. these principles using
data from the Moral Machine experiment
Participants Plurality rule Borda’s rule
Principles
2.92 1.05 0.77
violated (/6)
Principles
1.84 2.03 2.31
respected (/6)
Siebert, L.C., Andriamahery, K., Jonker, C., van den Hoven, J. Ethical decision-making for autonomous systems under 14
moral uncertainty. Work in progress.
Borda’s rulExe
• Exercise: Apply the Borda rule
1 3 2
𝑎 𝑐 𝑏 𝒂: 𝟏 × 𝟐 + 𝟑 × 𝟏 + 𝟐 × 𝟏 = 𝟕
𝑏 𝑎 𝑎 𝑏: 2 × 2 + 1 × 1 = 5
𝑐: 3 × 2 = 6 Winner: a
𝑐 𝑏 𝑐
15
Condorcet extensions
• Scoring rules have been criticized for failing to select the Condorcet
winner for some preference profiles
– Condorcet winner is an alternative that beats every other alternatives in pairwise majority
comparisons
– In the previous lecture we saw that there are some preference profiles that do not admit a
Condorcert winner
1 1 1 Majority (2 out of 3): 𝑎 ≻ 𝑏

a b c
Majority (2 out of 3): 𝑏 ≻ 𝑐
b c a
Majority (2 out of 3): 𝑐 ≻ 𝑎
c a b 16
• Let us consider the following example and the score vector (1,𝑠2 ,0), with
0 ≤ 𝑠2 ≤ 1 Pairwise comparisons
(𝑎, 𝑏) → 9 out of 17 prefers 𝑎 to 𝑏
6 3 4 4
(𝑎, 𝑐) → 10 out of 17 prefers 𝑎 to 𝑐
𝑎 𝑐 𝑏 𝑏 𝒂 is the Condorcet winner
𝑏 𝑎 𝑎 𝑐 if 𝑠2 =1 if 𝑠2 =0.5 if 𝑠2 =0
Scoring
𝑐 𝑏 𝑐 𝑎 𝑎’s score: 6 + 7s2 𝑎 = 13 𝑎 = 9.5 𝑎 =6
𝑏’s score: 8 + 6s2 𝒃 = 𝟏𝟒 𝒃 = 𝟏1 𝒃 =8
𝑐’s score: 3 + 4s2 𝑐=7 𝑐=5 𝑐 =3
𝒃 is the scoring rule winner, regardless of the value of 𝑠2

• For this example, 𝑏 is the unique winner for any scoring rule (1,𝑠2 ,0), even
though 𝑎 is the Condorcet winner
17
• There are some rules that try to satisfy Condercet’s
criterion: they are called Condorcet extensions
• These rules will elect the Condorcet winner if there is one
• They might also elect a winner when there is no
Condorcet winner (different Condorcet extensions may
elect different winners in the case of a cycle)
• Condorcet extensions differ on which other properties
they satisfy
18
• Copeland’s rule:
– An alternative gets a point for every pairwise majority win, and some fixed number of points
between 0 and 1 (say, ½) for every pairwise tie
– The winners are alternatives with the greatest number of points
6 3 4 4 𝑎, 𝑏 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 9−0−8 (1 point for a) 𝒂: 𝟐 (Condorcet winner)

𝑎, 𝑐 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 10 − 0 − 7 (1 point for a) 𝑏: 1
𝑎 𝑐 𝑏 𝑏 𝑏, 𝑐 𝑐: 0
(𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 14 − 0 − 3 (1 point for b)
𝑏 𝑎 𝑎 𝑐 𝑏, 𝑎 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 8 − 0 − 9
𝑐, 𝑎 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 7 − 0 − 10
𝑐 𝑏 𝑐 𝑎 𝑐, 𝑏 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 3 − 0 − 14
19
• Exercise: Apply Copeland’s rule (consider ½ point for ties)
𝑎, 𝑏 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 4−0−2 (1 point for a)

1 3 2
𝑎, 𝑐 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 3−0−3 (1/2 point for a and c)
𝑎 𝑐 𝑏 𝑏, 𝑐 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 3 − 0 − 3 (1/2 point for b and c)
𝑏 𝑎 𝑎 𝑏, 𝑎 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 2 − 0 − 4
𝑐, 𝑎 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 3 − 0 − 3
𝑐 𝑏 𝑐 𝑐, 𝑏 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 3 − 0 − 3
𝒂: 𝟏. 𝟓
𝑏: 0.5
𝑐: 1
Winner: a
20
• Maximin (a.k.a. Simpson–Kramer method):
– Under this rule, we consider the magnitude of pairwise election results (by how many voters one
alternative was preferred to the other)
– We evaluate every alternative by its worst pairwise defat by another alternative
– The winners are those who lose by the lowest margin in their worst pairwise defeats (if there are
alternatives with no pairwise defeats, they win!)
– Let 𝑋, 𝑌 denote the pairwise score for 𝑋 against 𝑌. The candidate selected by maximin is given by:
𝑊 = arg min(max( 𝑑 𝑋, 𝑌 − 𝑑 𝑌, 𝑋 ))
𝑥 𝑦
6 3 4 4
𝑎, 𝑏 =9−0−8 • Pairwise defeats:
(𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡)
𝑎 𝑐 𝑏 𝑏 𝒂: −(𝟎)
𝑎, 𝑐 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 10 − 0 − 7 𝑏: 1
𝑏 𝑎 𝑎 𝑐 𝑏, 𝑐 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 14 − 0 − 3 𝑐: {3,11}
𝑐 𝑏 𝑐 𝑎 𝑏, 𝑎 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 8 − 0 − 9 b: margin of 1
𝑐, 𝑎 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 7 − 0 − 10 c: margin of 3 arg min(max{𝟎, 1,11})) : 𝑎
𝑥 𝑦
c: margin of 11
𝑐, 𝑏 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 3 − 0 − 14 Winner: 𝒂 (Condorcet winner)
21
• Exercise: Apply the Maximin rule
𝑎, 𝑏 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 4−0−2
1 3 2
𝑎, 𝑐 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 3−0−3
𝑎 𝑐 𝑏 𝑏, 𝑐 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 3 − 0 − 3
𝑏 𝑎 𝑎 𝑏, 𝑎 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 2 − 0 − 4 b: margin of 2
𝑐, 𝑎 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 3 − 0 − 3
𝑐 𝑏 𝑐 𝑐, 𝑏 (𝑤𝑜𝑛−𝑡𝑖𝑒𝑑−𝑙𝑜𝑠𝑡) = 3 − 0 − 3
Winners: Tie between a and c (no losts)
22
• Nanson’s rule:
– Runoff method (i.e. multiple rounds)
– Series of Borda elections. Two variants:
• Fishburn variant (a.k.a Baldwin’s rule): For each Borda election, exclude the alternative with the lower
Board score unless all alternatives have identical Borda score, in which case these candidates are
declared the winner(s).
• Schwartz variant (the one we will use!): For each Borda election, exclude the alternative(s) which have
less than the average Borda score unless all alternatives have identical Borda score, in which case these
candidates are declared the winners
6 3 4 4 𝑎: 6 × 2 + 7 × 1 = 19 6 3 4 4
𝑎 𝑐 𝑏 𝑏 2 points 𝒃: 𝟖 × 𝟐 + 𝟔 × 𝟏 = 𝟐𝟐 𝑎 𝑎 𝑏 𝑏 1 point
𝑐: 3 × 2 + 4 × 1 = 10
𝑏 𝑎 𝑎 𝑐 1 point 𝑏 𝑏 𝑎 𝑎 0 point
𝑐 𝑏 𝑐 𝑎 0 point Average Borda score = 17
Remove alternative c 𝒂: 𝟗 (Condorcet winner)

𝑏: 8 23
• Exercise: Apply the Nanson’s rule (Schwartz variant )
1 3 2 𝑎: 1 × 2 + 5 × 1 = 7 1 3 2 𝑎: 3 × 2 + 3 × 1 = 9
𝑏: 2 × 2 + 1 × 1 = 5 𝑐: 3 × 2 + 3 × 1 = 9
𝑎 𝑐 𝑏 𝑐: 3 × 2 = 6 𝑎 𝑐 𝑎
𝑏 𝑎 𝑎 𝑐 𝑎 𝑐 Same Borda score
Average Borda score = 6
𝑐 𝑏 𝑐
Remove alternative b
Winners: Tie between a and c
24
Other rules
Finally, there are many others rules that are neither scoring rules, nor Condorcet
extensions, for example:
• Single Transferable Vote (STV):
– Looks for alternatives that are ranked in first place the least often, removes them from all voter’s
ballot, and repeats
– The alternatives removed in the last round win
6 3 4 4
6 3 4 4
𝑎 𝒄 𝑏 𝑏
𝑎 𝑎 𝒃 𝒃 Winner: 𝒂
𝑏 𝑎 𝑎 𝑐
𝑏 𝑏 𝑎 𝑎
𝑐 𝑏 𝑐 𝑎
25
Other rules
• Exercise: Apply Single Transferable Vote (STV)
1 3 2
1 3 2
𝑎 𝑐 𝑏
𝑏 𝑐 𝑏 Winner: b and c (tie)
𝑏 𝑎 𝑎
𝑐 𝑏 𝑐
𝑐 𝑏 𝑐
26
Other rules
• Bucklin’s rule:
– Check whether there is any alternative that is ranked first by more than half the voters; if so, this
alternative wins
– If not, check whether there are any alternatives that are ranked in either first or second place by
more than half the voters; if so, they win
– If not, consider the first three positions, etc.
– When multiple alternatives cross the 𝑛/2 threshold simultaneously, it is common to break ties by
the margin by which they crossed the threshold
6 3 4 4 Ranked first place: Ranked first or second place:

𝑎 𝑐 𝑏 𝑏 𝑎: (6/17) 35.3% < 50% 𝑎: (13/34) 38.2% < 50%
𝑏: (8/17) 47.1% < 50% 𝑏: (14/34) 41.2% < 50%
𝑏 𝑎 𝑎 𝑐
𝑐: (3/17) 17.6% < 50% 𝑐: (7/34) 20.6% < 50%
𝑐 𝑏 𝑐 𝑎
Winner: No winner (or tie between a, b, and c)
27
Other rules
• Exercise: Apply Single Transferable Vote (STV)
1 3 2
Ranked first place: Ranked first or second place:
𝒂 𝑐 𝑏 𝑎: (1/6) < 50% 𝑎: (6/12) = 50%
𝑏 𝑎 𝑎 𝑏: (2/6) < 50% 𝑏: (3/12) < 50%
𝑐: (3/6) = 50% 𝑐: (3/12) < 50%
𝑐 𝑏 𝑐
No winner
28
Interactive activity
• Public consultation on policies to
combat climate change. Consider
the following options:
– Introduction of meat tax
– More windmills and solar parks
– Improvement of household’s insulation
– Increase tax on natural gas
– Extension of subsidy for electric
vehicles
– Subsidy to reduce CO2 emissions from
industry
• What options do you prefer? (~3-5
minutes)
– Go to https://bit.ly/3pPtXp6 and state
your preferences
– The order of the options are
randomized
– Type some random characters as your
“Nickname” (let’s keep it anonymous)
29
Interactive activity
• Which method do you find more suitable for this participatory process? Why?
• Interested in the topic? This was a study in which 10.000 participated. See more info and
the insights here (Dutch-language only)
– https://www.tudelft.nl/tbm/pwe/case-studies/klimaatraadpleging
30
Strategic Manipulation
• So far, we assumed that the true preferences of all agents are known
• However, we only have access to their reported preferences
• This is an unrealistic assumption because agents might misrepresent their preferences
to exploit the voting rules
• Plurality rule:
1 2 2 2 1 2 2 2
𝑎 𝑎 𝑏 𝑐 𝑎 𝑎 𝑏 𝑏
𝑏 𝑐 𝑑 𝑏 Winner: 𝑎 𝑏 𝑐 𝑑 𝑐 Winner: 𝑏
𝑐 𝑏 𝑐 𝑑 𝑐 𝑏 𝑐 𝑑
𝑑 𝑑 𝑎 𝑎 𝑑 𝑑 𝑎 𝑎
They
prefer 𝑏
to 𝑎
31
• Borda rule:
1 2 2 2
𝑎 𝑎 𝑏 𝑐 𝑎: 3 × 3 = 9
𝑏: 𝟐 × 𝟑 + 𝟑 × 𝟐 + 𝟐 = 𝟏𝟒
𝑏 𝑐 𝑑 𝑏 𝑐: 2 × 3 + 2 × 2 + 3 × 1 = 13
𝑐 𝑏 𝑐 𝑑 𝑑: 2 × 2 + 2 × 1 = 6
𝑑 𝑑 𝑎 𝑎
1 2 2 2
They
𝑎 𝑐 𝑏 𝑐 𝑎: 1 × 3 + 2 × 2 = 7
prefer 𝑐 𝑏: 2 × 3 + 3 × 2 + 2 = 14
𝑏 𝑎 𝑑 𝑏 𝒄: 𝟒 × 𝟑 + 𝟑 × 𝟏 = 𝟏𝟓
to 𝑏
𝑐 𝑏 𝑐 𝑑 𝑑: 2 × 2 + 2 × 1 = 6
𝑑 𝑑 𝑎 𝑎
32
• Why avoid manipulation?
– It can decrease fairness since manipulative skills are not spread evenly
– Hard to evaluate whether the outcome respects agent’s “true preferences” or not
• Unfortunately, manipulation cannot be avoided in general: every voting rule is

susceptible to manipulation
• The Gibbard-Satterthwaite Impossibility theorem (1973/1975): Every non-imposing,

strategyproof, resolute voting rule is dictatorial when 𝑈 ≥ 3
– Non-imposing: for every alternative, there exist a preference profile that would make that alternative win
– Strategyproof: It is not manipulable
33
Manipulation
• Research in computational social choice has investigated the question of whether
manipulation can be made computationally difficult
• The Gibbard-Satterthwaite theorem requires that the voting rule is defined for all
possible preference profiles
• However, if we restrict the domain of possible preference profiles, we might create
strategyproof solutions:
– One well-know approach is the domain of single-peaked preferences e.g. when it is possible to
establish a linear ordering of the alternatives
• Example: tax rate {20%, 25%, 30%}
• Preferences are single-peaked if for every voter, as we move away from the voter’s most
preferred alternative, the alternative will become less preferred for that voter
• if 𝑥 < 𝑦 < 𝑧 or 𝑧 < 𝑦 < 𝑥 , then 𝑥 ≻𝑖 𝑦 implies 𝑦 ≻𝑖 𝑧 for every 𝑖 ∈ 𝑁
34
Manipulation
• In many real-life setting we can not expect restriction of preferences to hold
– There might be other reasons for voters to rank a given alternative higher or lower
• Even though manipulation is theoretically possible, opportunities for beneficial

manipulations are usually computationally hard to find (NP-hard) e.g. Nanson’s
rule
• NP-hardness is a worst-case measure i.e. it is unlikely that there is an efficient

algorithm that solves all instances of the manipulation problem
• However, there may still be an efficient algorithm that solves many of these
instances fast
35
Conclusion
• Computational social choice can foster collaboration among agents
• Even though many collective decision problems can be modelled as problems
of social choice, it is also true that many practical problems will not necessarily
fit the template provided by classical frameworks
• There is much more to computational social choice: Tournaments,
combinatorial domains, fair allocation…
“No one pretends that democracy is perfect or all-wise. Indeed it has been said
that democracy is the worst form of Government except for all those other forms
that have been tried from time to time.…”
Winston S Churchill
36
Ethics of Collaborative AI: Values
Enrico Liscio and Pradeep K.
Murukannaiah
What is Ethics?
The field of ethics involves systematizing, defending, and
recommending concepts of right and wrong behavior
[Fieser, The Internet Encyclopedia of Philosophy]
Ethical Dilemmas are scenarios where

there is no obviously good choice
Why should you Study Ethics …
▪ … as a computer scientist or a software engineer?
▪ To communicate with specialists in ethics (e.g., philosophers)
▪ To approach ethics from a technical standpoint
▪ … as an AI engineer?
▪ Because AI applications take increasingly sophisticated actions that affect humans
▪ … as a Collaborative AI engineer?
▪ Ethics is inherently a multi-agent concern
Values: A Building Block of Ethical AI
▪ What are values?
▪ When we think of values, we think

of what is important to us in life
▪ What are the features of values?
▪ Schwartz (2012) identifies six

features of all values
Preferences: Values vs. Interests
▪ A preference means a more positive attitude (leaning) toward one alternative
over another (or other) alternative(s)
▪ Preferences can be over values, interests, or other arbitrary choices
▪ Preferences over values vs. preferences over interests

▪ An interest is manifested as sustained attention involving cognition of the interest object,
accompanying positive affect
▪ A value is manifested as affective valuation
▪ Both values and interests influence behavior
▪ When judgment in preference is based on liking (i.e., attraction), it is an interest; when the
basis is importance (i.e., significance or meaning), the preference is a value
Value-Sensitive Design (VSD)
▪ VSD methods are not
standalone but are Intended
to be integrated with other
methods and processes for
technical design and
development
▪ Our focus for today

▪ Value identification
▪ Value preference elicitation
Value Identification
▪ Use an existing model of values
▪ For example, Schwartz Value

Model is a popular value model
▪ The Schwartz model consists

of ten universal values
Value Preference Elicitation
▪ How can we elicit the value preferences of an individual?
▪ There are questionnaires for eliciting value preferences
▪ Schwartz Value Survey (SVS)

▪ 57 items about potentially desirable end states or ways of acting
▪ Rate the importance of each item “as a guiding principle in MY life”
▪ Nine-point asymmetric rating scale
▪ Portrait Value Questionnaire (PVQ)

▪ Short (gender-matched) verbal portraits of 40 different people
▪ Question: How much like you is this person?
▪ Six-point rating scale (very much like me to not at all like me)
Context-Specific Values
Concrete Applications involving Values
Concrete examples of usage of values with artificial (collaborative)
agents:
▪ Health support
▪ Autonomous driving
▪ Public policy making
15-03-2022
Practical Example
Health support agent for grandpa in different
contexts:
▪ COVID-19 regulations
▪ Driving
We use different words in different

contexts.
15-03-2022
General Values
What values are relevant in the
following situations?
▪ Commuting
▪ Green energy transition
Not all values are relevant to all

contexts.
15-03-2022
Value Identification
We need to identify and define values in a context: context-specific values.
We use Axies, a hybrid methodology for identifying context-specific

values.
To be effective for collaborative agents, the methodology needs to be:
▪ Based on population input
▪ Democratic
▪ AI + human intuition
▪ Hybrid
▪ Performed by a group of
▪ Unbiased
annotators
▪ Simple ▪ On a web platform
E. Liscio, M. van der Meer, L. C. Siebert, C. M. Jonker, N. Mouter, and P. K.
15-03-2022
Murukannaiah (2021). “Axies: Identifying and Evaluating Context-Specific Values”.
In: Proc. AAMAS 2021, pp. 799-808.
Axies Input - Word Embeddings
The input of the methodology is population feedback on a
context (e.g., tweets). Words and sentences are represented as
points in a n-dimensional space (typically ) where semantic
similarities are preserved.
man
woman
king
queen
N. Reimers and I. Gurevych (2019). “Sentence-BERT: Sentence

15-03-2022
Embeddings using Siamese BERT-Networks”. In: Proc. EMNLP 2019,
pp. 3982–3992.
Axies Methodology - Exploration
In the exploration phase, each annotator independently develops a value list
by analysing the population input.
All input is embedded through sentence embeddings, the next opinion shown
to the annotator is selected with the Farthest First Traversal algorithm.
“Everybody should
“The economy must restart”
wear mouth
masks”
“Loneliness kills”
S. Basu, A. Banerjee, and R. J. Mooney (2004). “Active semi-supervision
15-03-2022
for pairwise
constrained clustering”. In: Proc. SDM 2004, pp. 333–344.
Axies Methodology - Exploration
Given as the set of visited points and the set of novel points, FFT
selects the next point as follows:
5 11
5
4
12
S. Basu, A. Banerjee, and R. J. Mooney (2004). “Active semi-supervision

15-03-2022
for pairwise
constrained clustering”. In: Proc. SDM 2004, pp. 333–344.
Axies Methodology - Consolidation
In the consolidation phase, the annotators in a group
collaborate to merge their individual value lists.
Annotator 1
Freedom
Safety
Group
Fairness
Freedom
Mental Health
Safety
Fairness
Annotator 2 Mental Health
Freedom Prosperity
Safety Pleasure
Prosperity
Pleasure
15-03-2022
Value Elicitation
The result of Axies is a list of values relevant to
a context.
A concrete application is the elicitation of value
preferences through natural language.
Estimating values from natural language allows:

▪ Humans to express values naturally;
▪ Agents to have meaningful conversations with
us;
▪ Agents to learn a user’s value profile by
aggregating values estimated from sentences.
15-03-2022
Value Classification
We can treat value classification as a supervised classification.
We need:
▪ Output of Axies
▪ Labelling schema
▪ Tweets, surveys, etc.
▪ Data
▪ Crowd annotations
▪ Labels
Then we can just perform a regular supervised classification.
Wrong!
15-03-2022
Label Subjectivity
The subjectivity of the interpretation of values must be considered,
especially in collaborative settings.
Textual inputs must be annotated by multiple annotators. Then, we
select the majority annotation to perform supervised classification.
Wrong!
That would result in the tyranny of the majority. We must consider

a plurality of opinions. We must build uncertainty in the model.
15-03-2022
Evaluation of Supervised Classifiers
Supervised classifiers are typically evaluated with the
score, assuming that a prediction is either correct or wrong.
Real label
Positive Negative
True Positive False
Predicted Positive (TP) Positive (FP)
label False True
Negative
Negative Negative
(FN) (TN)
15-03-2022
Data Representation
A1: [0, 1, 0,
Example: three annotators have annotated one datapoint 0]
A2: [0, 1, 1,
Examples of data representation are: 0]
A3: [0, 0, 0,
▪ Majority agreement aggregation: [0, 1, 0, 0]
1]
▪ Repeated labels: [0, 1, 0, 0] [0, 1, 1, 0] [0, 0, 0, 1]
▪ Distribution of labels: [0, 0.66, 0.33, 0.33]
▪ Filter hard items, factor in annotator quality, item difficulty,

etc.
A. N. Uma, T. Fornaciari, D. Hovy, S. Paun, B. Plank, M. Poesio (2021). Learning
15-03-2022
from Disagreement: A Survey. Journal of Artificial Intelligence Research, 72, 1385-
1470.
Evaluation Metrics
Examples of evaluation metrics are:
▪ Accuracy and (class-weighted) score;
▪ Disagreement-weighted score (items with a lot of disagreement are
weighted less than “easy” items);
▪ Cross-entropy (to compare annotated and predicted distributions of

labels):
15-03-2022
Takeaways
▪ Artificial agents must understand values operate among humans
▪ Values represent what humans (and organizations) find as

important in life
▪ Values are subjective and context dependent
▪ Values can identified and elicited via systematic AI (NLP)

techniques
15-03-2022
Readings
▪ Required: Lecture slides
▪ Optional
▪ Values: Schwartz, S. H. (2012). An overview of the Schwartz theory of basic values

. Online readings in Psychology and Culture, 2(1), 2307-0919.
▪ Context-specific values: Liscio, E., van der Meer, M. T., Siebert, L. C., Jonker, C. M.,
Mouter, N., & Murukannaiah, P. K. (2021).
Axies: Identifying and Evaluating Context-Specific Values. In Proceedings of the 20th
international conference on autonomous agents and MultiAgent systems (pp. 799-808).
Thank you for your attention!
Enrico Liscio and Pradeep K. Murukannaiah
CREDITS: This presentation includes icons by Flaticon and infographics & images by Freepik
Ethics of Collaborative AI: Social Norms
The Essence of Collaborative AI
▪What is one thing that is essential for collaboration?
▪Knowing mutual expectations among the collaborating parties

What are the Essential Elements of an Expectation?
▪ Consider an expectation in an e-commerce socio-technical system (STS) :
A seller must send the goods to a buyer upon payment
▪ What is the expectation? Send goods
▪ Who should meet the expectation? Seller
▪ Who is the expectation from? Buyer
▪ When is the expectation in force? Upon payment

A Social Norm: A Formal Representation of a Social Expectation
▪ A social norm is a directed social expectation between principals

(stakeholders)
Formal representation:
▪ Subject and Object: The two principals involved in the norm
▪ Antecedent: The condition that brings a norm into force
▪ Consequent: The outcome determining the satisfaction or violation of a norm

A Social Norm: A Computational Entity
Life cycle of a norm:

Why Social Norms?
▪ Accountability
▪ A principal can call another to account for its actions
▪ Flexibility
▪ A principal can violate norms (but bears the consequences)
▪ Explainability
▪ Provides an opportunity for principals to explain their actions
Types of Social Norms
▪ There are a few norm types, which can many instances in an STS
▪ The norm type defines the semantics of expectations underlying a norm
Types of Norms: Commitment
▪ Within an organizational context,
the subject (i.e., debtor) commits to the object (i.e., creditor) that if the
antecedent holds, the debtor will bring about the consequent
▪ Example in an e-commerce STS:
A seller must send the goods to a buyer upon payment
Norm:
Types of Norms: Prohibition
The object prohibits (i.e., forbids) the subject from bringing about the consequent
provided the antecedent holds
▪ Example in a healthcare STS:
Patient’s personal health information (PHI) should not be published online under
any circumstances
Norm:
Types of Norms: Authorization
the object authorizes (i.e., permits) the subject to bring about the consequent
when the antecedent holds
▪ Example in a healthcare STS:
In emergencies, hospital physicians may share a patient’s PHI with outside

physicians
Norm:
Types of Norms: Sanction
the object would sanction (i.e., reward or punish) the subject by bringing
about the consequent provided the antecedent holds
▪ Example in an e-commerce STS:
A buyer can sanction a seller by providing a poor rating if the product received
is not as advertised
Norm:
Types of Norms: Power
when the antecedent holds, the object empowers the subject to bring about
the consequent at will
▪ Example in a university STS:
The university empowers the system administer to create accounts
Norm:
▪ Power does not mean authorization!

Takeaways
▪ Social norms represent expectations among principals
▪ Social norms are computational entities
▪ Modeling social norms promotes accountability
▪ An STS can have many inorm instances, but they belong to a few
norm types
15-03-2022
Readings
▪ Required: Lecture slides
▪ Optional
▪ Social norms: Singh, M. P. (2014).

Norms as a basis for governing sociotechnical systems. ACM Transactions on Intelligent
Systems and Technology (TIST), 5(1), 1-23.
Collaborative AI: Synthesis
The Centralized AI Paradigm
▪ Today’s view of AI involves how an agent,

e.g., a prediction algorithm, deals with people
▪ The focus is on automation, not autonomy

The Collaborative AI Paradigm
▪ The collaborative AI paradigm

includes social entities—
principals (humans and
organizations)—and technical
entities—agents
▪ The focus is on enhancing the

autonomy of principals and
facilitating collaboration
among principals and agents
Collaborative AI: Learning Objectives
After completing this course, you should be able to:
▪ Compare centralized and collaborative AI paradigms.

▪ Describe the principles of automated negotiation, facilitating cooperation.
▪ Create an automated negotiating agent in the GeniusWeb platform
▪ Apply co-active design to solve a collaborative AI problem.
▪ Describe the conceptual underpinnings of agent coordination mechanisms.
▪ Describe the conceptual underpinnings of agent interaction protocols.
Collaborative AI: Course Assessment
▪ 50% Exam
▪ A combination of MCQs and open-ended questions
▪ 50% Practical: Two assignments

▪ Negotiating Agent: 0.5 weight (three weeks)
▪ Collaborative Agent: 0.5 weight (three weeks)
▪ Requirements to pass the course

▪ The exam >= 5.0
▪ Each practical assignment >= 5.0
▪ The total >= 5.75
Responsible and Ethical AI

Assistant Professor
1
ROBO
Robot for Burning Offices
Example adapted from the edX TU Delft MOOC “Mind of the Universe - Robots in Society: Blessing or
Curse?”, as described by Rijk Mercuur. https://www.youtube.com/watch?v=9fb_MENUfS4 2
ROBO’s main goal is to
support fire emergency
evacuation, including:
• Lead people to the best exit
route
• Remove obstacles
• Extinguish fire
• …
Example adapted from the edX TU Delft MOOC “Mind of the Universe - Robots in Society: Blessing or
Curse?”, as described by Rijk Mercuur. https://www.youtube.com/watch?v=9fb_MENUfS4 3
What should ROBO do?
4
One possible way: Act according to what people want
• Instructions Wow, fire alarm! Get

me out of here now.
• Revealed preferences: what my
behavior reveals I prefer
• Informed preferences: Do what I
would want if I were fully rational
and informed
• Act according to my value
preferences and social norms
Gabriel, I. (2020). Artificial intelligence, values, and alignment. Minds and machines, 30(3), 411-437.
Russell, S. (2019). Human compatible: Artificial intelligence and the problem of control. Penguin. 5
One possible way: Act according to what people want
Don’t worry about me. …
• What if there is no Go to the fifth floor,
agreement? there is someone there
that needs assistance.
• What if people are not
consistent?
• What if this behavior cannot
be modelled (or learned)?
…
• Could we find an overall

solution that would make
everyone satisfied?
6
Another possible way: Act according to what “is right”
• Goal
– Maximize happiness and well-being for the majority of a
population (Utilitarianism)
• What if:
– ROBO blocks one room to extinguish the fire but there are people
inside?
• Goal
– Morality should be based on whether a action itself is right or
wrong (Kantianism)
• What if:
– ROBO saves someone in a wheelchair but dozens of people get
severely injured?
7
Artificial moral agents
• Any approach may have severe ethical implications
– People will disagree on what is right or wrong ?
– Unexpected and emergent situations might arise
– Specially for wicked problems: class of problems for which
science provides insufficient or inappropriate resolution
• There is no one size solution fits all
• More (and better) technology isn’t always the answer
8
Hybrid (moral) Intelligence
• Hybrid intelligence (HI) as the
combination of human and machine
intelligence
– Augments human intellect and capabilities
instead of replacing them and achieving
goals
• Machine should engage with people,

explain their reasoning, behave
responsible, and learn from their mistakes
Akata, Z. et al (2020). A research agenda for hybrid intelligence: augmenting human intellect with collaborative, adaptive,
responsible, and explainable artificial intelligence. Computer, 53(08), 18-28. 9
TDP1: Human moral decision maker
• Agents do not require

moral competencies
• Humans have the “final
decision”
• Humans may suffer from
cognitive overload
• Human may become an
ethical scapegoat
van der Waa, J., van Diggelen, J., Siebert, L. C., Neerincx, M., & Jonker, C. (2020, July). Allocation of Moral Decision-Making in Human-Agent
Teams: A Pattern Approach. In International Conference on Human-Computer Interaction (pp. 203-220). Springer, Cham. 10
TDP2: Supported moral decision-making
• Humans usually get

distracted when not
supervising the direct
task
• Explanations may bias
the human moral
decision-making
TDP3: Coactive moral decision making
• Human agent cannot

intervene
• Handover may introduce
too much overhead and
delay important decisions
TDP4: Autonomous moral decision making
• Impossible with the

current state of the art
• Human values may be
extremely hard or even
impossible to elicit for all
situations
Collaborative AI = Ethical AI?
• Does having an “intelligent” agent guarantees ethical

behavior?
• Does a collaborative agent guarantees ethical behavior?
• Does having a human “in the loop” guarantees ethical
behavior?
• No; No; No.
14
Keeping control
• Humans must be in a position to be capable of being in control
of the system
• Machines should be able to understand and be responsive to
our moral standards
Synergistic &
transparent
collaboration
15
Meaningful Human Control
Humans not computers and their

algorithms should ultimately remain
in control of, and thus be morally
responsible for relevant decisions
Santoni de Sio, F., & Van den Hoven, J. (2018). Meaningful human control over 16
autonomous systems: a philosophical account. Frontiers in Robotics and AI, 5, 15.
• Some necessary conditions
– Define an moral operational
design domain (moral-ODD)
– Achieve appropriate and

mutually compatible
representations between
humans and AI
Siebert, L. C., Lupetti, M. L., Aizenberg, E., Beckers, N., Zgonnikov, A., Veluwenkamp,
H., ... & Lagendijk, R. L. (2021). Meaningful human control over AI systems: beyond
talking the talk. arXiv preprint arXiv:2112.01298. https://arxiv.org/pdf/2112.01298.pdf 17
• Some necessary conditions
– Make sure humans have ability
and authority to intervene and
steer the system
– Keep traceability
Siebert, L. C., Lupetti, M. L., Aizenberg, E., Beckers, N., Zgonnikov, A., Veluwenkamp,
H., ... & Lagendijk, R. L. (2021). Meaningful human control over AI systems: beyond
talking the talk. arXiv preprint arXiv:2112.01298. https://arxiv.org/pdf/2112.01298.pdf 18
Conclusion
• Only AI
– Unexpected situations
– Might misalign with our moral values (including emergent patterns)
– Who gets to decide which values it should follow?
• Only human
– Might be difficulty to cope with speed and data volume of AI
• Hybrid intelligence (Human + AI)
– Working together to avoid pitfalls
– AI under meaningful human control, aligned with our moral values
• To design and develop AI systems is not a value-neutral activity
• Multidisciplinary challenge!
19

All Slides From THIS YEAR Merged

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

All Slides From THIS YEAR Merged

Uploaded by

Copyright:

Available Formats

CSE 3210: Collaborative AI:

Interactive Intelligence, INSY, TU Delft

• What is collaborative AI?

Pradeep Catholijn Jonker

Myrthe L. Tielman Luciano Siebert

• Two lectures per week

• Two (shared) lab sessions per week

LO: Compare centralized and

• Provide examples for centralized AI systems

• Provide examples for collaborative AI systems

Include your own examples or pick the examples from

Education Gaming Law

• Have own thread of control, continually executing

• Exhibit proactive, reactive, and social behavior

• Can be physical or virtual, embodied or software-only

Collaborative AI System  Multi-Agent System

• Decentralized computation and control

http://groups.csail.mit.edu/mers/projects/human-robot-teamwork-in-manufacturing/ https://www.ctif.org/news/human-robot-teamwork-technology-robot-assisted-dis https://www.ledgerinsights.com/lg-cns-autonomous-cars-blockchain-5

• Each agent has incomplete information or

• There is no system for global control

Facilitate hybrid (human–AI) intelligence

• Humans assisting machines

• Machines assisting humans

Go to www.menti.com and use the code 9454 8532

• Alex’s activity tracking agent can talk to his physician’s agent to

• The agents can talk to Alex’s family members to motivate Alex to

• Microsoft Research: The Malmo Collaborative AI Challenge (2017)

• The Hybrid Intelligence Center (2020—2029)

• The Humane AI Net (2020—2023)

• Compare centralized and collaborative AI paradigms.

• Describe the principles of automated negotiation, facilitating cooperation.

• Create an automated negotiating agent in the GeniusWeb platform

• Apply co-active design to solve a collaborative AI problem.

• Describe the conceptual underpinnings of agent coordination mechanisms.

• Describe the conceptual underpinnings of agent interaction protocols.

Emerging topic! Suggestions are very welcome!

• 50% Practical: Two assignments

• Requirements to pass the course

• Task: Develop an automated negotiating agent

• Research papers and handbooks

• TODOs for you

– Work on the Negotiating Agent project

• Next three lectures: Negotiation

Interactive Intelligence Group,TU Delft

Tuesday, March 8, 2022 1

• What Kinds of Negotiations Exist?

• Guidelines for Negotiation

Tuesday, March 8, 2022 2

• Party or Stakeholder: someone that has a stake in the

Tuesday, March 8, 2022 4

What Kinds of Negotiations Exist?

Number of Parties Number of Issues

• Negotiation between two parties

Tuesday, March 8, 2022 6

Service Service Service

Tuesday, March 8, 2022 8

Principles That Help You Become a

• Holding a fixed position of what you want regardless

Tuesday, March 8, 2022 10

Negotiation: underlying interests

Principles That Help You Become a

Principles That Help You Become a

Principles That Help You Become a

= 0.50.9 + 0.3 0.7+0.2*0.5 = 0.76