STSM 2626

BAYESIAN STATISTICAL INFERENCE

2023
Notes prepared by Dr I. Garisch. Notes edited by Dr D. Chikobvu and Dr M. Sjölander

Slides (from notes) by Dr M. Sjölander

Presented by Dr M. Sjölander
When finding the joint pdf of the data (Xi’s) i.e. the
Likelihood function:
• Take note:
• Πc = cn
• Πxib = (Πxi)b
• Πbxi = bΣxi
• Π(cxibbxi) = cnbΣxi(Πxi)b
• lnΠxi = Σln(xi)
• This is because:
• Πc = c…c = cn
• Πxib = x1b…xnb = (x1…xn)b = (Πxi)b
• Πbxi = bx1…bxn = bx1+…+xn = bΣxi
• Similarly Π(cxibbxi) = cnbΣxi(Πxi)b
• lnΠxi = ln(x1…xn) = lnx1+…+lnxn = Σln(xi)
 See P9 – Law of total
probability for the
continuous case
– y replaced by θ

See P8 – Law of total

probability for the
discrete case
– y replaced by θ

In Bayesian Statistics: (θ’s are R.V.’s)

Prior: p(θ) p(x|θ) is the pdf of X, and p(x) is just a
Data’s pdf: p(x|θ) constant k and not the pdf of X.
Posterior: p(θ|x) p(x|θ) and p(x) are NOT the same thing.

Method 1 to find posterior In Frequentist Statistics: (θ’s are const.)

The pdf of X is p(x|θ) or p(x)
p(x|θ) and p(x) are the same thing.
Random Variable here is θ: where k = 1/p(x)

X ∝ Y means X = kY
X ∝ Y implies Y ∝ X as then Y = cX where c = 1/k
Random Variable here is θ: where k = 1/p(x)

X ∝ Y means X = kY
X ∝ Y implies Y ∝ X as then Y = cX where c = 1/k

Method 2 to find
posterior with
k found by:
This is obvious because on page 12 we have:
3 methods to find the posterior
We have thus seen method 1 and 2 as follows:
Method 1:
Continuous case: Discrete case:
(Tutorial 2’s scope ↑)
Method 2:
i.e. i.e.
We find k by noting that:
Continuous case: Discrete case:
(Example 6.4 ↑)
a third method is:
Method 3:
i.e. i.e.
Look at the formula sheet and find k by recognising which distribution
the posterior has. The pdf on the formula sheet will have the k.
(Example 6.3 ↑)
Solution:
Prior: θ ~ Uniform(0,1) so p(θ) = 1 for 0 ≤ θ ≤ 1

This is from page 6:

Random variable is now θ,

so x becomes θ.
Also, a = 0 and b = 1.
Likelihood: note that Xi~Bernoulli(θ)

L(θ|x) =

= θx1 (1 – θ)1 – x1 · θx2 (1 – θ)1 – x2 · … · θxn (1 – θ)1 – xn

= θx1 · θx2 · … · θxn · (1 – θ)1 – x1 · (1 – θ)1 – x2 · … · (1 – θ)1 – xn
= θx1+x2+…+xn · (1 – θ)1 – x1 + 1 – x2 +…+1 – xn You must be able
to skip the blue
= θx1+x2+…+xn · (1 – θ)1 + 1 + … + 1 – x1 – x2 – … – xn
= θΣxi (1 – θ)n – Σxi steps eventually
where y = Σxi
 where y = Σxi

Posterior:
.1

as we saw p(θ) = 1 for 0 ≤ θ ≤ 1 and p(x|θ) = θΣxi (1 – θ)n – Σxi

We thus know that p(θ|x) = k θy(1 – θ)n – y.1 = k θy(1 – θ)n – y
What type of distribution has this form?
Let’s look at pages 4 – 6.
 where y = Σxi

Posterior:
.1

as we saw p(θ) = 1 for 0 ≤ θ ≤ 1 and p(x|θ) = θΣxi (1 – θ)n – Σxi

We thus know that p(θ|x) = k θy(1 – θ)n – y
What type of distribution has this form?
Let’s look at pages 4 – 6.
Now the Binomial Distribution has the form p(x|p) = k px(1-p)n-x
and the Beta Distribution has the form p(x|p) = k xa-1(1-x)b-1
Which distribution do we have?
Well our variable is θ and we want a pdf of the form k θ?(1 – θ)?
For the Beta distribution on the formula sheet, the variable is x
and we have a pdf of the form k xa-1(1 – x)b-1 = k x?(1 – x)?
For the Binomial distribution on the formula sheet, the variable is x
and we have a pdf of the form k px(1 – p)n-x = k ?x ?n-x. Remember, our variable is θ.
Thus the Beta distribution is the distribution that we have (a – 1 = y, b – 1 = n – y)
 This is from page 6:

Random variable is now θ, so that’s

why θ replaced x. We also saw that:
a = y+1 and b = n-y+1 and apply
this to the constant term as well.
Our posterior distribution is:
θ|x ~ Beta( y + 1 , n – y + 1)

Note: If you can recognize the type of distribution (e.g. Beta) that the posterior
distribution has, then you must also give the type (e.g. Beta), not just the formula
of the posterior pdf.

If you cannot recognize the type of distribution that the posterior distribution has,
then you must just give the formula of the posterior pdf.
 Find the posterior distribution of θ.
Solution:

= e-θ k e-kx ekθ = k e-kx ekθ e-θ = k e-kx e(k-1)θ

Random variable is θ, so everything else is treated like a constant, thus k and e -kx is constant.
Note that X ∝ Y means X = kY, so ke-kxY ∝ Y as ke-kxY = k1Y.
Now we have p(θ|x) ∝ e(k-1)θ which means p(θ|x) = c e(k-1)θ , 0 < θ < x