Machine Learning for Signal Processing

EHB 328E

Ibraheem Shayea
Electronics and Communication Engineering Department,
Faculty of Electrical and Electronics Engineering
İstanbul Teknik Üniversitesi - İTÜ
 Syllabus 1

Fundamentals of Probability and

Statistics
Part A
Outline

❑ Introduction

❑ Sample Space and Events

❑ Algebra of Sets

❑ Probability Space
Introduction
Introduction
What is Probability?

❑ Probability is an important part of any mathematical education.

❑ Probability is an essential tool in applied mathematics and

mathematical modeling.

❑ Probability is also an essential tool in statistics.”

Introduction
What is Probability?
❑ Probability is the measure of chance that an event will occur, and as
such finds applications in disciplines that involve uncertainty.

❑ The study of probability stems from the analysis of certain games of

chance
Introduction
What is Probability?
❑ The outcome of a random event cannot be determined before
it occurs, but it can be any one of a number of possible outcomes. The
actual outcome is considered to be determined by chance.
Introduction
What is Probability Theory ?

❑ Probability Theory, a branch of mathematics concerned with the

analysis of random phenomena.

❑ Probability Theory deals with the study of random phenomena,

which under repeated experiments yield different outcomes that have
certain underlying patterns about them.
Introduction
What is Probability / Probability Theory ?

Probability Theory is a fundamental pillar of modern mathematics with

relations to other mathematical areas like algebra, topology, analysis,
geometry or dynamical systems.
Introduction
What is Probability ?
Example: Tossing a Coin
When a coin is tossed, there are two possible outcomes:
o Heads (H) Thus, we can say that :
• the probability of the coin landing H is ½
o Tails (T)
• the probability of the coin landing T is ½

Tails (T)

Heads (H)
Introduction
What is Probability ?
Example: Throwing Dice

When a single Dice is thrown, there are six possible outcomes:

o 1
o 2
o 3
o 4
o 5
o 6

Thus, we can say that :

• The probability of any one of them is 1/6

Introduction
How Probability can be calculated ?

The probability of an 𝐸𝑣𝑒𝑛𝑡 𝐻𝑎𝑝𝑝𝑒𝑛𝑖𝑛𝑔 can be represented

mathematically by:

𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐸𝑣𝑒𝑛𝑡𝑠 𝑖𝑡 𝐶𝑎𝑛 𝐻𝑎𝑝𝑝𝑒𝑛𝑒𝑑

𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑜𝑓 𝑎𝑛 𝐸𝑣𝑒𝑛𝑡 𝐻𝑎𝑝𝑝𝑒𝑛𝑖𝑛𝑔 =
𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑠

𝑁𝐸
𝑃(𝐴) =
𝑁𝑇
Introduction
How Probability can be calculated ?
Example: What is the probability of Throwing Dice one time ?
❑ Let us assume the outcome in this time is 1

1
𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑜𝑓 𝑡ℎ𝑒 𝐸𝑣𝑒𝑛𝑡 𝐻𝑎𝑝𝑝𝑒𝑛𝑖𝑛𝑔 =
6
Introduction
How Probability can be calculated ?
Example:
❑ Let us assume we throwing a dice two times
❑ What is the probability of the sum of outcomes is 5?

Answer:

?
Introduction
How Probability can be calculated ?
Example:

S: {(1,1), (1,2), (1,3), ……etc.}

o There are 36 possible outcomes in S, all equally likely.
o Thus, the probability of any one of them is 1/36.
o Events having a sum of 5 has 4 possible outcomes (1&4, 2&3, 4&1, 3&2)
o 36 outcomes in sample space
o probability of the sum of outcomes is 5 = 4/36
Introduction
How Probability can be calculated ?
Example:
❑ Let us assume we throwing a dice two times
❑ What is the probability of getting (3, 1)?

Answer:

?
Introduction
How Probability can be calculated ?
Example:

S: {(1,1), (1,2), (1,3), ……etc.}

o There are 36 possible outcomes in S, all equally likely.
o Thus, the probability of any one of them is 1/36.
o Events having 3 and 1has 1 possible outcomes (3&1)
o 36 outcomes in sample space
o = 1/36
Introduction
How Probability can be calculated ?
Exercise:
❑ Let us assume we throwing a dice three times
❑ What is the probability the outcome in these three times are 4, 3, and 2 ?

Answer:

?
Introduction
How Probability can be calculated ?

Exercise:

What is the probability of getting 1 if we Throwing Dice 700 times ?

?
Introduction
Applications of Probability Theory ?
Probability theory is extensively used in a host of areas :

Science Engineering Medicine

Business
Sample Space and Events
Sample Space and Events
Random Experiments

The concept of an experiment assumes a set of repeatable

conditions that allow any number of identical repetitions. When an
experiment is performed under these conditions, certain elementary
events occur in different but completely uncertain ways.
Sample Space and Events
Random Experiments

❑ Experiment: In the study of probability, any process of observation

is referred to as an experiment.

❑ Random Experiment: An experiment is called a random

experiment if its outcome cannot be predicted.
Sample Space and Events
Random Experiments
Examples
Typical examples of a random experiment are :
❖ Roll of a Die

❖ Toss of a Coin

❖ Drawing a Card from a Deck

Sample Space and Events
Outcomes of Random Experiments

The results of an observation are called the outcomes of the experiment.

Example: What is the probability of Throwing Dice one time ?

❑ Let us assume the outcome in this time is 1

Outcomes

1
𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑜𝑓 𝑡ℎ𝑒 𝐸𝑣𝑒𝑛𝑡 𝐻𝑎𝑝𝑝𝑒𝑛𝑖𝑛𝑔 =
6
Sample Space and Events
Sample Space

❑ Sample Space: The set of all possible outcomes of a random

experiment is called the sample space (or universal set), and it is
denoted by S.

The Sample Space is all possible outcomes of a random

experiment.
Sample Space and Events
Sample Space

Sample space for the experiment of Tossing three coins.

Sample Space and Events
Sample Space
Example :
Find the sample space for the experiment of tossing a coin :
(a) Once
(b) Twice.

Heads (H) Tails (T)

Solution:

?
Sample Space and Events
Sample Space
Example :
Find the sample space for the experiment of tossing a coin :
(a) Once
(b) Twice.

Heads (H) Tails (T)

Solution:
Sample Space and Events
Sample Space
Exercise :
Find the sample space for the experiment of of Throwing Dice:

(a) One times ?

(b)Two times ?
Solution:

???
Sample Space and Events
Sample Space
Example :
Find the sample space for the experiment of tossing a coin repeatedly
and counting the number of tosses required until the first head appears.

Solution:

???
Sample Space and Events
2Sample Space
Example :
Find the sample space for the experiment of tossing a coin repeatedly and
counting the number of tosses required until the first head appears.

Solution:

Clearly all possible outcomes for this experiment are the terms of the
sequence 1, 2,3,...Thus:

𝑺 = { 𝟏, 𝟐, 𝟑, … , ∞}

Note that there are an infinite number of outcomes

Sample Space and Events
Sample Space

Exercise :
Find the sample space for the experiment of Throwing Dice repeatedly and
counting the number of Throwing are required until the side of one dot is
appeared one time?

Solution:

???
Sample Space and Events
Sample Space
Exercise :
Find the sample space for the experiment of of Throwing Dice repeatedly
and counting the number of Throwing are required until the side of one dot
is appeared Ten times?

Solution:

???
Sample Space and Events
Sample Space
Example :
Find the sample space for the experiment of measuring (in hours) the
lifetime of a transistor.

Solution:

???
Sample Space and Events
Sample Space
Example :
Find the sample space for the experiment of measuring (in hours) the
lifetime of a transistor.

Solution:
Clearly all possible outcomes are all non-negative real numbers.
That is,

where 𝜏: represents the life of a transistor in hours.

Sample Space and Events
Sample Space
Types of Sample Space
Note that any particular experiment can often have many different
sample spaces depending on the observation of interest (Probs. 1 and 2).

❑ Continuous : A sample space S is said to be continuous if the sample

points constitute a continuum.

❑ Discrete : A sample space S is said to be discrete if it consists of a finite

number of sample points (as in the previous Example) or Countable
infinite sample points.

Countable : A set is called countable if its elements can be placed in a one-

to-one correspondence with the positive integers.
Sample Space and Events
Sample Point

❑ Sample Point: An element in S is called a sample point. Each

outcome of a random experiment corresponds to a sample point.
Sample Space and Events
Events
❖ Event
Any subset of the sample space S is called an event.
o Events are subsets of the sample space S
Sample Space and Events
Events

❖ Elementary Event
A Sample Point of S is often referred to as an elementary event.

❖ Certain Event
The sample space S is the subset of it self: that is, 𝑆 ⊂ 𝑆. Since S is the
set of all possible outcomes, it is often called the certain event.
Sample Space and Events
Events
Example 4:
Consider the experiment of Example 2.
o Let A be the event that the number of tosses required until the first head
appears is even.
o Let B be the event that the number of tosses required until the first head
appears is odd.
o Let C be the event that the number of tosses required until the first head
appears is less than 5.
Express events A, B, and C ?
Solutions:
A = {2, 4, 6,...}
B = {l, 3, 5,...}
C = {l, 2, 3, 4}
Sample Space and Events
Sample Space, Sample Point, Event

❑ Sample Space : Sample Space is all possible outcomes of a

random experiment.
❑ Sample Point: A Sample Point is just one possible outcome
❑ Event: Event can be one or more of the possible outcomes.
Sample Space and Events
Random Experiments, Outcomes, Sample Space
Algebra of Sets
Algebra of Sets
Set Operations
Algebra of Sets
Set Operations
Equality

Definition: Two sets are equal if and only if they have the same
elements.

❑ Two sets A and B are equal, denoted : 𝐴 = 𝐵

if and only if : 𝐴 ⊂ 𝐵 and 𝐵⊂𝐴

Example:
A = {1,2,3}
B = {3,1,2}
C = {1,2,1,3,2}
{1,2,3} = {3,1,2} = {1,2,1,3,2}
Algebra of Sets
Set Operations
Complementation

Suppose 𝐴⊂𝑆

The complement of set A is the set containing all elements in S but not in A

The complement of set A, denoted 𝐴ҧ

𝐴ҧ

𝐴ҧ
Algebra of Sets
Set Operations
Union

The union of sets A and B is the set containing all elements in either A or B
or both.

The union of sets A and B, denoted :

𝐴∪𝐵
Algebra of Sets
Set Operations
Intersection
❑ The intersection of sets A and B is the set containing all elements in
both A and B.
❑ The intersection of sets A and B, denoted :

𝐴∩𝐵
Algebra of Sets
Set Operations
Difference

❑ The difference of sets A and B is the set containing all elements in A but
not in B.
❑ The difference of sets A and B, denoted :
𝐴∖𝐵
𝐴∖𝐵

Home Exercise
Proof this ?
Algebra of Sets
Set Operations
Symmetrical Difference

❑ The symmetrical difference of sets A and B is the set of all elements that
are in A or B but not in both
❑ The symmetrical difference of sets A and B, denoted :

𝐴△𝐵

𝐴△𝐵

Home Exercise
Proof this ?
Algebra of Sets
Set Operations
Null Set

❑ The set containing no element is called the null set. Note that

In mathematical sets, the null set, also called the empty set, is the
set that does not contain anything.

❑ The Null set denoted :

= ⊘= 𝑆
Algebra of Sets
Set Operations
Disjoint Sets

❑ Two sets A and B are called disjoint or mutually exclusive if they contain
no common element.
𝑻𝒉𝒂𝒕 𝒊𝒔 𝒊𝒇 𝑨 ∩ 𝑩 = 𝟎

𝐴 ∩𝐵 = ⊘
Algebra of Sets
Set Operations
Disjoint Sets

A B
Technologies Cities

Microsoft Istanbul Kuala

Google
Lumpur

Kuala
Cisco Facebook Paris
Lumpur

Twitter London

𝐴 ∩𝐵 = ⊘
Algebra of Sets
Venn Diagram
Algebra of Sets
Venn Diagram
Algebra of Sets
Identities
❑ By the above set definitions or reference we obtain the following
identities

Home Exercise
Proof these properties ??
Algebra of Sets
Identities

Commutative Laws

Home Exercise
Proof these properties ??
Algebra of Sets
Identities
Associative Laws
Home Exercise
Proof these properties ??

Distributive Laws:

Home Exercise
Proof these properties ??
Algebra of Sets
Identities
De Morgan's Laws

Home Exercise
Proof these properties ??

any element that is contained in the set on the left side of the equality
sign is also contained in the set on the right side, and vice versa.
Probability Space
Probability Space
Event Space
❑ Events are subsets of the sample space S

❑ A subset A of S can be an event if it belongs to a collection F of

subsets of S, satisfying the following conditions:

The collection F is called an event space. In mathematical

literature, event space is known as sigma field (𝝈 − 𝒇𝒊𝒆𝒍𝒅) or 𝝈 −
𝒂𝒍𝒈𝒆𝒃𝒓𝒂
Probability Space
Probability Space
❑ A Probability Space is a mathematical model that provides a formal model of a
random process or "experiment".

❑ It is made up of three parts:

1. Sample Space: is the set of all possible outcomes.

2. Event Space: is the set of all events.

3. Probability Function: is the assignment of probability to the event.

Assigns each event in the event space a probability, which is a number between 0
and 1.

For example:
1
The odds of tossing a coin and getting a heads is 50%, .
2
Probability Space
Probability Measure
Classical Definition

Classical Probability is the statistical concept that measures the

possibility (probability) of something happening.

In a classic sense, it means that every statistical experiment will contain

elements that are equally likely to happen (equal chances of occurrence of
something).

Therefore, the concept of classical probability is the simplest form of

probability that has equal chances of something happening.
Probability Space
Probability Measure
Classical Definition
Probability Examples

Example 1
Tossing an unbiased coin. There is an equal probability that your toss will yield either head or
tail.

Example 2
Rolling of a fair dice because it is equally probable that top face of die will be any of the 6
numbers on the die: 1, 2, 3, 4, 5, or 6.

Example 3
Guess a multiple choice quiz test with (about) four possible answers A, B, C or D. Each option
(choice) has the same chances (equal probabilities) of being chosen (assuming you select randomly
and do not follow any pattern).
Probability Space
Probability Measure
Classical Definition
Formula for Classical Probability

The probability of a simple event happening is the number of times the event can happen, divided
by the number of possible events (outcomes).

Mathematically
𝑓
𝑃(𝐴) = 𝑁
where,
𝑷(𝑨) : “probability of event A” (event A is whatever event you are looking for),

𝒇 : frequency, or number of possible times the event could happen and

𝑵 : Total number of times the event could happen.

Probability Space
Probability Measure
Classical Definition
Consider an experiment with equally likely finite outcomes. Then the
classical definition of probability of event A, denoted P(A),is defined
by:

𝑨
𝑷 𝑨 =
𝑺
Probability Space
Probability Measure
Classical Definition

Properties

Let S be a sample space with probability measure P. Also, let A and B be any events in S.
Then the following hold:
• ഥ =𝟏−𝑷 𝑨
𝑷 𝑨 Proof :

• 𝑷 ∅ =𝟎
𝑺
• 𝑷 𝑺 = = 𝟏
𝑺

• 𝑰𝒇 𝑨 ⊆ 𝑩, 𝒕𝒉𝒆𝒏 𝑷 𝑨 ≤ 𝑷 𝑩 . Home Exercise: Proof : ?

• 𝑷 𝑨 ≤𝟏
Probability Space
Probability Measure
Classical Definition
Properties

• If A and B are joint

𝑷(𝑨 ∪ 𝑩) = 𝑷(𝑨) + 𝑷(𝑩) − 𝑷(𝑨 ∩ 𝑩)

Home Exercise: Proof : ?

Probability Space
Probability Measure
Classical Definition
Example

𝐴 ∩𝐵 = 0 𝐶 ∩𝐵 = 7

𝐴 ∩𝐵 = ⊘
3
𝑃(𝐶) =
3 6
𝑃(𝐴) =
6 3
𝑃 𝐷 =
6
3
𝑃 𝐵 =
6 1
P(C∩D) =
6
P(A∩B) = 0
𝑃 𝐶 ∪ 𝐷 = 𝑃(𝐶) + 𝑃(𝐷) − 𝑃(𝐶 ∩ 𝐷)
3 3 3 3 1 5
𝑃 𝐴 ∪ 𝐵 = 𝑃(𝐴) + 𝑃(𝐵) = + =1 = + − =
6 6 6 6 6 6
Probability Space
Probability Measure
Classical Definition

Example: Consider an experiment of rolling a die.The outcome is

Probability Space
Probability Measure
Classical Definition

Note: the classical definition, P(A) is determined a priori without actual

experimentation and the definition can be applied only to a limited class of
problems such as only if the outcomes are finite and equally likely or equally probable.
Probability Space
Probability Measure
Relative Frequency Definition:

Relative Frequency or Empirical Probability or A Posterior Approach

So if an event A occurs m times, then the probability of the occurrence of the event A is
defined by

𝑚
𝑃(𝐴) = lim
𝑥→∞ 𝑛

Fore example, if we assume that the probability of a number n child will be a boy is 1/2,
then it means that over a large number of children born 50% of all will be boys.
Probability Space
Probability Measure
Relative Frequency Definition:

Suppose that the random experiment is repeated n times. If event A occurs 𝑛(𝐴)
times, then the probability of event A, denoted P(A), is defined as:

where 𝒏(𝑨)Τ𝒏 is called the relative frequency of event A.

Probability Space
Probability Measure
Classical Definition

Properties

• If A and B are joint

𝑷(𝑨 ∪ 𝑩) = 𝑷(𝑨) + 𝑷(𝑩) − 𝑷(𝑨 ∩ 𝑩)

Home Exercise: Proof : ?

The End

Thank You