# Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Bayesian Multi Agent Systems
Ahmad Ashar Group 256 Modelling and Simulation

December 17, 2012

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

1

2 3

4

Motivation and Aim What is Bayesian Bayesian vs Frequentist Avoiding Over ﬁtting Personal Motivation Applications MAS:A Brief Overview Decision Theory Game Theory A Bayesian Framework Markov Decision Process for Single Agent
Value Functions and Bellman Equations

5

MDP for MultiAgent System in Reinforcement learning Conclusions
Ahmad Ashar Group 256 Modelling and Simulation Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Avoiding Over ﬁtting Personal Motivation

The questions

Why Bayesian ? What is Bayesian ? Why is it an important paradigm ? Why am I studying this?

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Avoiding Over ﬁtting Personal Motivation

What is Bayesian ? p(θ|D) = p(D|θ)p(θ0 ) θ=∞ θ=−∞ p(D|θ)p(θ)dθ

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Avoiding Over ﬁtting Personal Motivation

Bayesian vs Frequentist

Why Bayesian ?

Figure: Human Evolution

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Avoiding Over ﬁtting Personal Motivation

Bayesian Marginalization avoids the problem of overﬁtting ”.. the phenomenon of over-ﬁtting is really an unfortunate property of maximum likelihood and does not arise when we marginalize over parameters in a Bayesian setting.” ”..It is a property of the marginal likelihood that it automatically incorporates a trade-oﬀ between model ﬁt and model complexity ” Above quotes courtesy Bishop’s book on Pattern Recognition

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Avoiding Over ﬁtting Personal Motivation

The details..
”Overﬁtting occurs when a statistical model describes random error or noise instead of the underlying relationship” [Wikipedia]

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Avoiding Over ﬁtting Personal Motivation

Why am I studying this?

Interest in Statistical Machine Learning Bayesian Non-parametrics: possibly the hottest thing in Machine Learning today My interests in Gaussian Process Regression,Classiﬁcation Research includes tools from
1 2 3

Statistical Physics : Mean Field Approximations Data Analysis: Kernel Methods Neural Computation : ANN

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Applications

Data Analysis - Google,Microsoft,Yahoo! Fraud Detection Geo-statistical applications Intelligent systems (Robots, Recommender sytems :Netﬂix prize) Bioinformatics, Cheminformatics, NLP, Phylogenetic trees Information Theory, Decision Theory Practically anything which has a large data set!

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Decision Theory Game Theory

MAS :Decision Theory

Decision Theory Maximizing Utility over various policies (series of steps) State → Percept → Action → NewState π∗ Found by searching various possible states s Search problem → Decision Theory problem Reward R : S × S × A → R Distribution over actions p(a|s) for each s ∈ S where a ∈ A(s)

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Decision Theory Game Theory

MAS :Game Theory

Game Theory Framework for Co-operation and Communication Treat MAS as a game with agents as players Use mathematical results in co-operative games with incomplete information as strategies for agents Highly complex and advanced ﬁeld of research

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Markov Decision Process for Single Agent MDP for MultiAgent System in Reinforcement learning

Bayesian Decision Making: Markov Decision Process

State transition properties depend on the current state and action State transition probabilities are multinomial distributions When the rewards or transition probabilities are unknown the problem reduces to that of reinforcement learning

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Markov Decision Process for Single Agent MDP for MultiAgent System in Reinforcement learning

Value Functions and Bellman Equations
Reward function:
N

γ t rt+1
t=0

where γ > 0 State value function

V (s) = E[
t=0

p

γ t rt+1 |so = s]

Bellman Equation p(a|s)
a∈A(s) s ∈S

p(s |s, a)[r (s, a, s ) + γV p (s )]

State Action Value Function
Ahmad Ashar Group 256 Modelling and Simulation

=

p(s |s, a)[r (s, a, s ) + γV p (s )]
Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Markov Decision Process for Single Agent MDP for MultiAgent System in Reinforcement learning

A Bayesian Framework : MDP for MAS in RL

Acting, Coordinating should be integrated One agent’s optimal policy aﬀects the other’s Solution: Put Bayesian Priors over MAS optimal policy Solution: Likelihood could be factorized Bayesian inference could be carried out, Often approximations Have to model some agents joint behaviour for Machine Learning of the parameters Problem with scaling :With more agents Parameters could tend to ∞

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems

Motivation and Aim Applications MAS:A Brief Overview A Bayesian Framework Conclusions

Conclusions

MAS : Mixing of Decision Theory and Game theory Making MAS an extension of probabilistic single agents with incomplete information Using Bayesian Inference for optimal policy determination Challenge: To fully develop a detailed Bayesian MAS mathematically

Ahmad Ashar Group 256 Modelling and Simulation

Bayesian Multi Agent Systems