Probability and Statistics

Prof. Zheng Zheng

z. zheng 1
 Content
1. Introduction (1 lectures)
– Concept of probability and
probability spaces
– Elementary probability theory
– Conditional probability and
Bayes' theorem
2. Repeated Trials (1 lectures)
– Combined experiments
– Bernoulli trials
– Poisson theorem
3. Random Variables and
Distribution (4 lectures)
– Discrete and continuous
random variables
– Functions of random variables
– Joint and marginal
distributions
– Independence
4. Expectation
– Mean, variance and covariance
– Conditional distribution and
conditional expectation
– Least squares estimation for
Gaussian random vectors
5. Limit theorems
– Laws of large numbers
– Central limit theorem
6. Statistics
– Parameter estimation
– Hypothesis testing
z. zheng 2
 What is ‘probability’? (1)
Classical definition:
– P(A)=NA/N
– N: # of all possible outcome; NA: # of outcomes
favor A
– a priori
– ‘Equal Probability’ assumption
Example:
Balls in a box

z. zheng 3
 What is ‘probability’? (2)
Relative frequency definition:
n A
– P ( A )  lim
n   n
– N: # of experiments; NA: # times of A’s
occurrence

Example:
Toss a coin, many, many times…

z. zheng 4
 What is ‘probability’? (3)
Axiomatic definition:
• Experiment under repeatable conditions
• Elementary events i
•  consists of i
   1 ,  2 ,  ,  k ,  
• A, B, C, … are subsets of denoted as A  
– e.g.   A implies   .

z. zheng 5
 Set Theory--Definition
• Set: a collection of elements
A    1 ,  2 ,...,  n 
• Subset: consists of elements that are also
element of the set A B  A
• Belong or not belong to a set:  i  A  i  A
• Empty (null) set:  0 

z. zheng 6
 Operations on Subsets
• Union A  B    |   A or   B
• Intersection A  B    |   A and   B
• Complement A    |   A
• Transitivity Property: If   A and A  B ,   B

A
A B A B A

A B A B A

z. zheng 7
 More definitions
• Mutually exclusive (M.E) or Disjoint
A B

A B 

• A partition of : a collection of A1
A2
mutually exclusive subsets of . Ai
Aj An
Ai  A j   , and A i  .
i 1

z. zheng 8
 De-Morgan’s Laws

A B  A B ; A B  A B

A B A B A B A B

A B A B A B

z. zheng 9
 Field
• A collection of subsets of a nonempty set 　
forms a field F if
(i)  F
(ii) If A  F , then A  F
(iii) If A  F and B  F , then A  B  F.
• it is easy to show that A  B , A  B , etc.,
also belong to F. Thus
F   0 , A , B , A , B , A  B , A  B , A  B ,  .
z. zheng 10
 Axioms of Probability
• For any event A, the probability of A
assigned as P(A) satisfies:
(i) P ( A)  0 (Probabili ty is a nonnegativ e number)
(ii) P ( )  1 (Probabili ty of the whole set is unity)
(iii) If A  B   , then P ( A  B )  P ( A)  P ( B ).

z. zheng 11
 Some Basic Rules
• P    0 .
• P( A  A)  P ( A)  P( A)  1 or P( A)  1  P ( A).
• P ( A  B )  ? for not M.E. A and B
Proof: A  B  A  AB ,
P ( A  B )  P ( A  A B )  P ( A )  P ( A B ).
P ( B )  P ( BA )  P ( B A),
A AB
P ( A  B )  P ( A)  P ( B )  P ( AB ).
A B
z. zheng 12
 Conditional Probability
• P(A|B) = Probability of “the event A given
that B has occurred”.
P ( AB )
• We define P ( A | B )  , provided P( B)  0.
P( B)

• This definition satisfies:

– P( A | B)  0
– P ( | B )  1
– P ( A  C | B )  P ( A | B )  P (C | B ) , if A  C  0 .
z. zheng 13
 Properties of Conditional Probability
• If B  A , P ( A | B )  1
• If A  B , P ( A | B )  P ( A )
n
• P ( B )   P ( B | Ai ) P ( A i )
i 1

n
if Ai A j   and  Ai  
i1

Total Probability Theorem

z. zheng 14