Assignment 11

Introduction to Machine Learning

Prof. B. Ravindran
1. Given n samples x1 , x2 , . . . , xN drawn independently from an Exponential distribution un-
known parameter λ, find the MLE of λ.
Pn
(a) λM LE = i=1 xi
Pn
(b) λM LE = n i=1 xi
(c) λM LE = Pnn
i=1 xi
Pn
i=1 xi
(d) λM LE = n
(e) λM LE = Pn−1
n
i=1 xi
Pn
i=1 xi
(f) λM LE = n−1

Sol. (c)

n
Y n
Y Pn
L(λ, x1 , . . . , xn ) = f (xi , λ) = λe−λx = λn e−λ i=1 xi

i=1 i=1

Pn
d ln λn e−λ i=1 xi


d ln (L(λ, x1 , . . . , xn ))
=
dλ dλ Pn
d ln (n ln(λ) − λ i=1 xi )
=
dλ
n
n X
= − xi
λ i=1

Set the above term to zero to obtain MLE of λ

n
λ= P
n
xi
i=1

2. Given n samples x1 , x2 , . . . , xn drawn independently from an Geometric distribution unknown

parameter p given by pdf Pr(X = k) = (1 − p)k−1 p for k = 1, 2, 3, · · · , find the MLE of p.
Pn
(a) pM LE = i=1 xi
Pn
(b) pM LE = n i=1 xi
(c) pM LE = Pnn
i=1 xi
Pn
i=1 xi
(d) pM LE = n
(e) pM LE = Pn−1
n
i=1 xi
Pn
i=1 xi
(f) pM LE = n−1

1
 Sol. (c)

3. (2 marks) Suppose we are trying to model a p dimensional Gaussian distribution. What is the
actual number of independent parameters that need to be estimated in mean and covariance
matrix respectively?
(a) 1, 1
(b) p − 1, 1
(c) p, p
(d) p, p(p + 1)
(e) p, p(p + 1)/2
(f) p, (p + 3)/2
(g) p − 1, p(p + 1)
(h) p − 1, p(p + 1)/2 + 1
(i) p − 1, (p + 3)/2
(j) p, p(p + 1) − 1
(k) p, p(p + 1)/2 − 1
(l) p, (p + 3)/2 − 1
(m) p, p2
(n) p, p2 /2
(o) None of these
Sol. (e)
Explanation Mean vector has p parameters. The covariance matrix is symmetric (p × p) and
hence has p p+1
2 independent parameters.

4. (2 marks) Given n samples x1 , x2 , . . . , xN drawn independently from a Poisson distribution

unknown parameter λ, find the MLE of λ.

2
 Pn
(a) λM LE = i=1 xi
Pn
(b) λM LE = n i=1 xi
(c) λM LE = Pnn
i=1 xi
Pn
i=1 xi
(d) λM LE = n
(e) λM LE = Pn−1
n
i=1 xi
Pn
i=1 xi
(f) λM LE = n−1

Sol. (d)
Write the likelihood:
Y λxi e−nλ
l(λ; x) = e−λ = λx1 +x2 +···+xn
i
xi ! x1 !x2 ! · · · xn !

Take the log and differentiate the log-likelihood with respect to λ and set it to 0.
5. (2 marks) In Gaussian Mixture Models, πi are the mixing coefficients. Select the correct
conditions that the mixing coefficients need to satisfy for a valid GMM model.
(a) −1 ≤ πi ≤ 1, ∀i
(b) 0 ≤ πi ≤ 1, ∀i
P
(c) i πi = 1
P
(d) i πi need not be bounded

Sol. (b), (c)

6. (2 marks) Expectation-Maximization, or the EM algorithm, consists of two steps - E step and

the M-step. Using the following notation, select the correct set of equations used at each step
of the algorithm.
Notation.
X: Known/Given variables/data
Z: Hidden/Unknown variables
θ: Total set of parameters to be learned
θk : Values of all the parameters after stage k
Q(, ): The Q-function as described in the lectures
(a) E-step: EZ|X,θ [log(P r(X, Z|θm ))]
(b) E-step: EZ|X,θm−1 [log(P r(X, Z|θ))]
P
(c) M-step: argmaxθ Z P r(Z|X, θm−2 ) · log(P r(X, Z|θ))
(d) M-step: argmaxθ Q(θ, θm−1 )
(e) M-step: argmaxθ Q(θ, θm−2 )
Sol. (b), (d)