ECE 6504: Advanced Topics in

Machine Learning
Probabilistic Graphical Models and Large-Scale Learning

Topics:
–  Bayes Nets: Representation/Semantics
–  v-structures
–  Probabilistic influence, Active Trails

Readings: Barber 3.3; KF 3.3.1-3.3.2

Dhruv Batra
Virginia Tech
 Plan for today
•  Notation Clarification

•  Errata #1: Number of parameters in disease network

•  Errata #2: Car start v-structure example

•  Bayesian Networks
–  Probabilistic influence & active trails
–  d-separation
–  General (conditional) independence assumptions in a BN

(C) Dhruv Batra 2

 A general Bayes net
•  Set of random variables

•  Directed acyclic graph

–  Encodes independence assumptions

•  CPTs
–  Conditional Probability Tables

•  Joint distribution:

(C) Dhruv Batra 3

 Factorized distributions
•  Given Flu Allergy
–  Random vars X1,…,Xn
–  P distribution over vars Sinus

–  BN structure G over same vars

Headache Nose

•  P factorizes according to G if

(C) Dhruv Batra Slide Credit: Carlos Guestrin 4

 How many parameters in a BN?
•  Discrete variables X1, …, Xn

•  Graph
–  Defines parents of Xi, PaXi

•  CPTs – P(Xi| PaXi)

(C) Dhruv Batra 5

 Independencies in Problem
World, Data, reality: BN:

True distribution P
contains independence
assertions

Graph G
encodes local
independence
assumptions

(C) Dhruv Batra Slide Credit: Carlos Guestrin 6

 Bayes Nets
•  BN encode (conditional) independence assumptions.
–  I(G) = {X indep of Y given Z}

•  Which ones?
•  And how can we easily read them?

(C) Dhruv Batra 7

 Local Structures
•  What’s the smallest Bayes Net?

(C) Dhruv Batra 8

 Local Structures
Indirect causal effect:

X Z Y

Indirect evidential effect: Common effect:

X Z Y
X Y

Common cause: Z
Z
X Y

(C) Dhruv Batra 9

 Car starts BN

•  18 binary attributes

•  Inference
–  P(BatteryAge|Starts=f)

•  218 terms, why so fast?

(C) Dhruv Batra Slide Credit: Carlos Guestrin 10

 Bayes Ball Rules
•  Flow of information
–  on board

(C) Dhruv Batra 11

 Active trails formalized
•  Let variables O ⊆ {X1,…,Xn} be observed

•  A path X1 – X2 – · · · –Xk is an active trail if for each

consecutive triplet:

–  Xi-1→Xi→Xi+1, and Xi is not observed (Xi∉O)

–  Xi-1←Xi←Xi+1, and Xi is not observed (Xi∉O)

–  Xi-1←Xi→Xi+1, and Xi is not observed (Xi∉O)

–  Xi-1→Xi←Xi+1, and Xi is observed (Xi∈O), or one of its

descendents is observed

(C) Dhruv Batra Slide Credit: Carlos Guestrin 12

 Active trails and Independence
A B
•  Theorem: Variables Xi and Xj
are independent given Z if
–  no active trail between Xi and Xj C
when variables Z⊆{X1,…,Xn} are E
observed D

G
F

H J

I
K

(C) Dhruv Batra Slide Credit: Carlos Guestrin 13

 Name That Model

Naïve Bayes:

(C) Dhruv Batra Slide Credit: Erik Sudderth 14

 Name That Model

Tree-Augmented Naïve Bayes (TAN)

(C) Dhruv Batra Slide Credit: Erik Sudderth 15

 Name That Model

Y1 = {a,…z} Y2 = {a,…z} Y3 = {a,…z} Y4 = {a,…z} Y5 = {a,…z}

X1 = X2 = X3 = X4 = X5 =

Hidden Markov Model (HMM)

(C) Dhruv Batra Figure Credit: Carlos Guestrin 16