Scribd
Upload
18 views1 page

Homework 03: Probability & Bayes' Theorem

"But I must explain to you how all this mistaken idea of denouncing pleasure and praising pain was born and I will give you a complete account of the system, and expound the actual teachings of the great explorer of the truth, the master-builder of human happiness. No one rejects, dislikes, or avoids pleasure itself, because it is pleasure, but because those who do not know how to pursue pleasure rationally encounter consequences that are extremely painful. Nor again is there anyone who loves or

Uploaded by

Dũng Minh
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
18 views1 page

Homework 03: Probability & Bayes' Theorem

"But I must explain to you how all this mistaken idea of denouncing pleasure and praising pain was born and I will give you a complete account of the system, and expound the actual teachings of the great explorer of the truth, the master-builder of human happiness. No one rejects, dislikes, or avoids pleasure itself, because it is pleasure, but because those who do not know how to pursue pleasure rationally encounter consequences that are extremely painful. Nor again is there anyone who loves or

Uploaded by

Dũng Minh
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

COMP3020 – Homework 03

Thai Minh Dung


2024-11-22

Problem 1: Writing Problems


1.1 Join Distribution:
We have:
ℙ[𝑌 > 0 | 𝑋 = +1] = ℙ[1 + 𝑁 > 0] = ℙ[𝑁 > −1] = 1 − ℙ[𝑁 ≤ −1] = 1 − Φ(−1)

As 𝑁 ∈ Gaussian(0,1) ⇒ ℙ[𝑁 ≤ 1] = Φ(−1).


Similarly, we have:
ℙ[𝑌 > 0|𝑋 = −1] = 1 − Φ(+1)
⇒ ℙ[𝑌 > 0] = ℙ[𝑌 > 0 | 𝑋 = −1] ⋅ ℙ[𝑋 = −1] + ℙ[𝑌 > 0 | 𝑋 = +1] ⋅ ℙ[𝑋 = +1]
1
⇒ ℙ[𝑌 > 0] = 1 + (Φ(+1) + Φ(−1))
2
1
ℙ[𝑌 > 0] = 1 + (1 − Φ(−1) + Φ(−1))
2
1
ℙ[𝑌 > 0] = .
2
Then, we have:
ℙ[𝑌 > 0|𝑋 = +1] ⋅ ℙ[𝑋 = +1]
ℙ[𝑋 = +1|𝑌 > 0] =
ℙ[𝑌 > 0]
= ℙ[𝑌 > 0|𝑋 = +1] = 1 − Φ(−1) = 0.8413
⇒ ℙ[𝑋 = −1|𝑌 > 0] = 1 − 0.8413 = 0.1587

1.2 Bayes’s Theorem:


Let:
• A : the event that people who studied Mechanics actually switch to IT.
• D : the event that a person has described characteristics
• F : the event that a person is a friend.
Then we have:
1
ℙ(𝐴) = = 0.05
20
ℙ(𝐷|𝐹 , 𝐴) = 0.7
ℙ(𝐷|𝐹 , ¬𝐴) = 0, 1

And:
ℙ(𝐷|𝐹 ) = ℙ(𝐷|𝐹 , 𝐴) ∗ ℙ(𝐴) + ℙ(𝐷|𝐹 , ¬𝐴) ∗ (1 − ℙ(𝐴))
= 0.7 ∗ 0.05 + 0.1 ∗ (1 − 0.05)
= 0.035 + 0.095 = 0.13
Then:
ℙ(𝐴)
ℙ(𝐴|𝐹 , 𝐷) = ℙ(𝐷|𝐹 , 𝐴) ⋅
ℙ(𝐷|𝐹 )
= 0.269

We need to find, ℙ(𝐴|𝐹 , 𝐷):


Ans = 0.269 = 26.9%

1.3 Maximum Likelihood Estimation:


1.3.a
We have:

𝜆𝑥𝑛 ⋅ 𝑒−𝜆
𝑃 (𝑋𝑛 = 𝑥𝑛 ) = ∀𝑥𝑛 ∈ ℕ
𝑥𝑛 !

Hence,

𝑃 (𝑌𝑛 = 0) = 𝑃 (𝑋𝑛 = 0) = 𝑒−𝜆

which indicates that:

𝑃 (𝑌𝑛 = 1) = 𝑃 (𝑋𝑛 ≥ 1) = 1 − 𝑃 (𝑋𝑛 = 0) = 1 − 𝑒−𝜆

We have the PMF of 𝑌𝑛 :

𝑒−𝜆 if 𝑦𝑛 = 0
𝑃 (𝑌𝑛 = 𝑦𝑛 ) = {
1 − 𝑒−𝜆 if 𝑦𝑛 = 1

1.3.b
Let the proportion of time 𝑌𝑛 = 1 in T experiments be 𝑝, we have:

∑𝑇𝑛=1 𝑦𝑛
𝑝=
𝑇
As 𝑌𝑛 = 1 has an underlying PMF of 𝑃 (𝑌𝑛 = 1) = 1 − 𝑒−𝜆 , we have:

𝑝 = 1 − 𝑒−𝜆
∑𝑇𝑛=1 𝑦𝑛
= 1 − 𝑒−𝜆
𝑇

−𝜆
∑𝑇𝑛=1 𝑦𝑛
𝑒 =1−
𝑇
∑𝑇𝑛=1 𝑦𝑛
−𝜆 = ln(
(1 − )
)
𝑇
( )
∑𝑇𝑛=1 𝑦𝑛
𝜆 = − ln(
(1 − )
)(𝑄.𝐸.𝐷)
𝑇
( )

1.4 Bayesian Linear Regression:


We have: Likelihood 𝑝(𝑦|𝜃):

𝑝(𝑦|𝜃) = 𝑁 (𝑦|𝑋𝜃, 𝜎2 𝐼),

with PDF:
1
𝑝(𝑦|𝜃) ∝ exp(− ‖𝑦 − 𝑋𝜃‖2 ).
2𝜎2
Prior 𝑝(𝜃):
𝑝(𝜃) = 𝑁 (𝜃|𝜇𝜃 , Σ𝜃 ),

with PDF:
1
𝑝(𝜃) ∝ exp(− (𝜃 − 𝜇𝜃 )𝑇 Σ−1
𝜃 (𝜃 − 𝜇𝜃 )).
2
Apply Bayes theorem, 𝑝(𝜃|𝑦) ∝ 𝑝(𝑦|𝜃)𝑝(𝜃),
1 1
𝑝(𝜃|𝑦) ∝ exp(− 2
‖𝑦 − 𝑋𝜃‖2 − (𝜃 − 𝜇𝜃 )𝑇 Σ−1
𝜃 (𝜃 − 𝜇𝜃 )).
2𝜎 2
Expand ‖𝑦 − 𝑋𝜃‖2 :

‖𝑦 − 𝑋𝜃‖2 = 𝑦𝑇 𝑦 − 2𝑦𝑇 𝑋𝜃 + 𝜃𝑇 𝑋 𝑇 𝑋𝜃.

Combine terms:
1 1 1 𝑇
𝑝(𝜃|𝑦) ∝ exp(− [𝜃𝑇 ( 2 𝑋 𝑇 𝑋 + Σ−1 𝑇 −1
0 )𝜃 − 2𝜃 ( 2 𝑋 𝑦 + Σ0 𝜇0 )])(1)
2 𝜎 𝜎
Set
1 𝑇
Σ−1 −1
1 = Σ0 + 𝑋 𝑋,
𝜎2
1 𝑇
𝜇1 = Σ1 (Σ−1
0 𝜇0 + 2 𝑋 𝑦).
𝜎
Substitute into (1):
1 1 𝑇 −1
𝑝(𝜃|𝑦) ∝ exp(− [𝜃𝑇 Σ−1 𝑇 −1 𝑇 −1 𝑇 −1
1 𝜃 − 2𝜃 Σ1 𝜇1 ]) ∝ exp(− [𝜃 Σ1 𝜃 − 2𝜃 Σ1 𝜇1 + 𝜇1 Σ1 𝜇1 ])
2 2
Hence, the posterior 𝑝(𝜃|𝑦) is Gaussian:
𝑝(𝜃|𝑦) = 𝑁 (𝜃|𝜇1 , Σ1 ),

where:

Σ−1 −1 −2 𝑇
1 = Σ0 + 𝜎 𝑋 𝑋

𝜇1 = Σ1 (Σ−1 −2 𝑇
0 𝜇0 + 𝜎 𝑋 𝑦).

You might also like