Sulaimani polytechnic university

Technical college of engineering

Communication engineering department

Information theory & coding

ASSIGNMENT

Prepared by : Sida Sana kiumars

Fourth stage
Bayes' Theorem:
Bayes' Theorem, named after 18th-century British mathematician Thomas
Bayes, is a mathematical formula for determining conditional probability.
Conditional probability is the likelihood of an outcome occurring, based on
a previous outcome having occurred in similar circumstances. Bayes'
theorem provides a way to revise existing predictions or theories (update
probabilities) given new or additional evidence.

In finance, Bayes' Theorem can be used to rate the risk of lending money
to potential borrowers. The theorem is also called Bayes' Rule or Bayes'
Law and is the foundation of the field of Bayesian statistics.

Conditional property:
 Conditional probability refers to the chances that some outcome
occurs given that another event has also occurred.
 It is often stated as the probability of B given A and is written as
P(B|A), where the probability of B depends on that of A happening.
 Conditional probability can be contrasted with unconditional
probability.
 Probabilities are classified as either conditional, marginal, or joint.
 Bayes' theorem is a mathematical formula used in calculating
conditional probability.
 If A and B are two events, then the conditional probability me be
designated as P(A given B) or P(A|B).
  Conditional probability can be calculated from the joint
probability(A | B) = P(A, B) / P(B)
  The conditional probability is not symmetrical; For example P(A |
B) != P(B | A).

We can also calculating conditional probability includes using the

other conditional probability :

P(A|B) = P(B|A) * P(A) / P(B)

Reverse is also used

P(B|A) = P(A|B) * P(B) / P(A)

Formula for Bayes' Theorem:

where: A and B are events and P(B) is not equal to zero.

P(A)= The probability of A occurring

P(B)= The probability of B occurring

P(A∣B)=The probability of A given B

P(B∣A)= The probability of B given A

P(A⋂B))= The probability of both A and B occurring

Bayes' theorem may be derived from the definition of conditional

probability:
P(A|B) = P(A ⋂ B)/ P(B), where P(B) ≠ 0
P(B|A) = P(B ⋂ A)/ P(A), where P(A) ≠ 0
Here, the joint probability P(A ⋂ B) of both events A and B being
true such that,

P(B ⋂ A) = P(A ⋂ B)
P(A ⋂ B) = P(A | B) P(B) = P(B | A) P(A)

P(A|B) = [P(B|A) P(A)]/ P(B), where P(B) ≠ 0

Equivalently, Bayes Theorem can be written as:

posterior = likelihood * prior / evidence

This words might sound fancy but the underlying idea behind them is
really simple. You can always refer back to this section whenever you have
any doubts or reach out to me directly in the comments section below.

Applications of Bayes Theorem:

 
The Bayes' Theorem has several practical uses. If you are unable to
immediately grasp all of the mathematics required, don't be concerned.
Starting with just obtaining an understanding of how it operates is
sufficient.
 
A statistical method for solving the categorization of pattern issues is
called Bayesian Decision Theory. This concept assumes that the categories'
fundamental probability density function is known. As a result, we can
benchmark the effectiveness of all other classifiers against the ideal Bayes
Classifier.
 
We'll go through Bayes' Theorem's five primary applications:
 
 Naive Bayes’ Classifiers
 Discriminant Functions and Decision Surfaces
 Bayesian Parameter Estimation
 Bayesian Optimization
 Bayesian Belief Networks
 Example of Bayes’ Theorem:

-Imagine you are a financial analyst at an investment bank.

According to your research of publicly-traded companies, 60% of
the companies that increased their share price by more than 5% in
the last three years replaced their CEOs during the period.

At the same time, only 35% of the companies that did not increase
their share price by more than 5% in the same period replaced
their CEOs. Knowing that the probability that the stock prices grow
by more than 5% is 4%, find the probability that the shares of a
company that fires its CEO will increase by more than 5%.

Before finding the probabilities, you must first define the notation
of the probabilities.

 P(A) – the probability that the stock price increases by 5%

 P(B) – the probability that the CEO is replaced
 P(A|B) – the probability of the stock price increases by 5%
given that the CEO has been replaced
 P(B|A) – the probability of the CEO replacement given the
stock price has increased by 5%.

Using the Bayes’ theorem, we can find the required probability:

Thus, the probability that the shares of a company that replaces its
CEO will grow by more than 5% is 6.67%.
Plotting entropy(H) graph using Matlab codes:
The code is:
%Define a vector of probabilities
p = linspace (0.001,0.999, 1000) :
%Calculate the entropy for each probability
H = -p. *log2 (p) - (1-p).*log2 (1-p) ;
10
11
12
% Plot the entropy as a function of probability plot (p, H) ;
xlabel ('Probability');
ylabel (' Entropy (bits) ');
title ('Entropy vs. Probability');

And this is the result of plot: