EM Algorithm

Week-10
CSEg4301

Expectation Maximization Algorithm

When to use EM Algorithm
 Missing or Hidden data present in dataset
 When something is unknown
 When there is no class label in the dataset
 Unsupervised learning
 It is an iterative procedure to compute MLE
(Maximum Likelihood Estimation) in the process of
missing or hidden data.
EM Algorithm
 Expectation-Maximization (EM) Algorithm

The Expectation-Maximization (EM) algorithm is an iterative approach to

find the maximum likelihood estimate (MLE) of a statistical model when
some of the data is incomplete or missing. It works by alternating between
two steps:
1. Expectation (E) Step:
In this step, we estimate the missing data based on the current estimates of
the model parameters and the observed data. This is done by calculating the
expected value of the complete-data likelihood function, where the
expectation is taken over the missing data distribution.
2. Maximization (M) Step:
In this step, we maximize the expected complete-data likelihood function
with respect to the model parameters. This provides updated estimates of
the model parameters that are closer to the true MLE.
These two steps are repeated until the difference between the estimates in
successive iterations falls below a certain threshold.
EM ALGORITHM

MCAR: Missing completely at random or MAR- Missing at Random

How it works?
 EM-Algorithm &Applications
EM Algorithm:

Initialization: We start with initial estimates of the bias for each coin (e.g., p1 = p2 = 0.5).
E-Step: For each observed flip, we calculate the probability that it was generated by each
coin.
M-Step: Based on the probabilities calculated in the E-Step, we estimate the new bias for
each coin.
Repeat: We repeat steps 2 and 3 until the estimates converge.
Applications:
General Idea of EM
Idea of EM Alg.
(MLE) Maximum Likelihood Estimation
EM Usecase
Assume that we have two coins C1 and C2
EM - Coin Toss Usecase
EM- coin Toss Usecase
First Experiment
EM- coin Toss Usecase
E:step
E-Step
M:step
For row-2
EM- coin Toss Use case
EM- coin Toss Use case
EM- coin Toss Use case
 EM- coin Toss Usecase End

Convergence:
After a few iterations, the estimates of p1 and p2 will converge to their true values. In this
example, the true values of p1 and p2 are 0.8 and 0.54, respectively.
Problems with EM
End of Week-10: EM Algorithm
Thank u!