Course description

• General topic: theory of statistical inference, and some probability theory

– First half: Probability Theory ( B&E, Chapters 1-7)
univariate, joint, marginal, and conditional probabilities; random variables and their
functions, distribution characteristics of random variables, basic properties and
asymptotic theory
– Second half: Theory of Statistical Inference (B&E, Chapters 8-12)
principles and methods of constructing point estimators, confidence intervals,
and hypothesis testing…

• Course is required for

– MS comprehensive examination, and for
– more advanced courses, including most core course taken in later years.

• Structure of the course will closely follow the B&E textbook

– Slides’ legend: Very important facts, Extra (non-required) facts
– All examples will be from B&E unless otherwise noted

• Assignments will be based on the covered material

– hopefully will help remember the taught material
 Topic 1
Probability

Lecture 1
 In Topic 1
1. Elementary set operations and their properties
2. Notation and terminology
3. Probability
4. Some properties of probability
5. Counting techniques (next lecture)
6. Conditional probability
 Basic constructs
• “Event” is a well-defined collection of possibilities
health-related: discovering disease in an individual, observing 3 deaths within a given period of
time, observing a low rate of infection in the population
more abstract and structured (often game-related): observing a tail on a coin flip, certain
combination on dice throw, drawing a certain combination of cards from a deck

• “Experiment” is a well-defined reproducible set of conditions for

observing events
health-related: obtaining radiologist’s interpretation of a chest X-ray for a randomly selected
person from a lung-cancer screening cohort
more abstract: flipping a coin, throwing a balanced die, drawing a card from a shuffled deck

• Basic models try to define the “experiment” and events that better fit a
natural phenomenon (to better quantify)
health-related setting: Observing a positive result in a randomly selected individual among people
undergoing testing
abstract model: Flipping an unbalanced (e.g., heavy-tailed) coin

• The formalism of statistical inference stems from the probability theory,

which stems from set and measure theory
events are visualized and operated with as sets; probability is a measure of a set
 Probability: a concept
• The frequency interpretation:
The probability that some specific event is obtained = the relative
frequency with which the event would occur if the process (experiment) were
repeated a large (infinite) number of times under similar conditions
– Example: coin is tossed, the probability of a head is ½; we expect a head
on approximately one-half of many (e.g., 1000) tosses
– Difficulties: “similar conditions”, “large number of times”, “approximately”,
“repetitions” are not easily interpretable for all problems

• In practice, calculation of probabilities is sometimes based on the

concept of equally likely outcomes
– Example: coin is tossed, there are two possible outcomes (a head or a tail);
if they are equally likely to occur, then the probability of each must be ½
– Shortcomings: concept of equally likely is based on the concept of
probability; sometimes it is difficult to correctly determine/find equally
likely outcomes
(often equal probability events are more evident from the frequency
considerations)
 Sample Space
• Experiment – obtain (observe) result (outcome) of a phenomenon
– e.g., toss of a coin (H-head, T-tail); toss of two coins

• Elementary event, ‘e’ –exactly one outcome of an experiment

– e.g., for tossing two coins: {HH}, {HT},...
– the impossible event: ∅ (never occurs)
– different elementary events do not overlap, i.e., are mutually exclusive e1∩e2= ∅

• Event, ‘A’ – a collection of possible outcomes of an experiment

– e.g., “observing at least one head” = {HH,HT,TH}={HH}∪ {HT} ∪ {TH}

• Sample space ‘S’ – the set of all possible outcomes of an experiment

– e.g., S={HH,HT,TH,TT}
– sample space S is a “sure” event (always occurs)

• Sample space S can be

– finite if it consists of a finite number of outcomes (e.g., for toss of two coins)
– countably infinite if its outcomes can be put into a one-to-one correspondence with
the positive integers (i.e., counted) (e.g., indefinitely repeated toss of a coin)
– discrete sample space if it is finite or countably infinite
– “continuous” sample space (e.g., location of a coin tossed on a table)
 Basic Event (Set) Operations
• Event = Set – is collection of distinct objects (elements)
– Trivial sets(events): empty (impossible), ∅; universal (sure), S (contains all)
• Relationships: (implies)
– Containment (“implication”) 𝐴𝐴 ⊂ 𝐵𝐵 ⟺ (∀𝑥𝑥 ∈ 𝐴𝐴 ⇒ 𝑥𝑥 ∈ 𝐵𝐵) (A is a subset of B)
– Equality 𝐴𝐴 = 𝐵𝐵 ⟺ (𝐴𝐴 ⊂ 𝐵𝐵 and 𝐵𝐵 ⊂ 𝐴𝐴)
• Operations (note analogy with Arithmetic and Logic):
– Union (“either”) 𝐴𝐴 ∪ 𝐵𝐵 = {𝑥𝑥: 𝑥𝑥 ∈ 𝐴𝐴 𝑜𝑜𝑜𝑜 𝑥𝑥 ∈ 𝐵𝐵} (note: mutually
– Intersection (“both”) 𝐴𝐴 ∩ 𝐵𝐵 = {𝑥𝑥: 𝑥𝑥 ∈ 𝐴𝐴 𝑎𝑎𝑎𝑎𝑎𝑎 𝑥𝑥 ∈ 𝐵𝐵} exclusive: 𝐴𝐴 ∩ 𝐵𝐵 = ∅)

– Complement (“not”) 𝐴𝐴̅ (𝐴𝐴′ 𝑜𝑜𝑜𝑜 𝐴𝐴𝑐𝑐 ) = 𝑥𝑥: 𝑥𝑥 ∈ 𝑆𝑆 𝑎𝑎𝑎𝑎𝑎𝑎 𝑥𝑥 ∉ 𝐴𝐴 (note: 𝐴𝐴̅ = 𝐴𝐴 )
• Properties
– Commutative 𝐴𝐴 ∪ 𝐵𝐵 = 𝐵𝐵 ∪ 𝐴𝐴; 𝐴𝐴 ∩ 𝐵𝐵 = 𝐵𝐵 ∩ 𝐴𝐴
B&E Appendix A
– Associative 𝐴𝐴 ∪ 𝐵𝐵 ∪ 𝐶𝐶 = 𝐴𝐴 ∪ 𝐵𝐵 ∪ 𝐶𝐶
𝐴𝐴 ∩ 𝐵𝐵 ∩ 𝐶𝐶 = (𝐴𝐴 ∩ 𝐵𝐵) ∩ 𝐶𝐶
– Distributive 𝐴𝐴 ∩ 𝐵𝐵 ∪ 𝐶𝐶 = (𝐴𝐴 ∩ 𝐵𝐵) ∪ (𝐴𝐴 ∩ 𝐶𝐶)
𝐴𝐴 ∪ 𝐵𝐵 ∩ 𝐶𝐶 = (𝐴𝐴 ∪ 𝐵𝐵) ∩ (𝐴𝐴 ∪ 𝐶𝐶)
– DeMorgan’s Laws 𝐴𝐴 ∪ 𝐵𝐵 = 𝐴𝐴̅ ∩ 𝐵𝐵�
𝐴𝐴 ∩ 𝐵𝐵 = 𝐴𝐴̅ ∪ 𝐵𝐵�
 Mathematical theory of probability
• Probability is a real-valued function defined on a certain collection of
sets (events). This collection of sets has to obey certain properties.
• Definition
A collection of subsets of 𝑆𝑆 is called a sigma algebra ℬ (or Borel field) if it satisfies the
following three properties:
1) ∅∈ℬ (the empty set is an element of ℬ)
2) If 𝐴𝐴 ∈ ℬ ⇒ 𝐴𝐴̅ ∈ ℬ (ℬ is closed under complementation)
∞
3) If 𝐴𝐴1 , 𝐴𝐴2 , … ∈ ℬ ⇒ ∪ 𝐴𝐴𝑖𝑖 ∈ ℬ (ℬ is closed under countable unions)
𝑖𝑖=1

• Corollary:
o 𝑆𝑆 ∈ ℬ
∞
o If 𝐴𝐴1 , 𝐴𝐴2 , … ∈ ℬ ⇒ ∩ 𝐴𝐴𝑖𝑖 ∈ ℬ (ℬ is closed under countable intersections)
𝑖𝑖=1

• Note:
– Many different sigma algebras can be associated with the same 𝑆𝑆
– We are usually interested in the smallest one that contains all of the open sets in a
given sample space 𝑆𝑆
• For countable S, the sigma algebra ℬ is a collection of all subsets including trivial
 Probability: formal definition
• Definition
Let S be a sample space associated with a given experiment, and ℬ is a
sigma algebra on 𝑆𝑆. Probability (or a probability function) is a
real-valued function 𝑃𝑃 with domain ℬ (𝑃𝑃: ℬ →ℝ) that satisfies:
1) 0 ≤ 𝑃𝑃 𝐴𝐴 ∀ 𝐴𝐴 ∈ ℬ
2) 𝑃𝑃 𝑆𝑆 = 1
3) If 𝐴𝐴1 , 𝐴𝐴2 , … ∈ ℬ are mutually exclusive (i.e. 𝐴𝐴𝑖𝑖 ∩ 𝐴𝐴𝑗𝑗 = ∅ if 𝑖𝑖 ≠ 𝑗𝑗), then
∞ ∞
𝑃𝑃 ∪ 𝐴𝐴𝑖𝑖 = �𝑖𝑖=1𝑃𝑃(𝐴𝐴𝑖𝑖 )
𝑖𝑖=1
• Corollary:
– (finite union) If 𝐴𝐴1 , 𝐴𝐴2 ,…, 𝐴𝐴𝑘𝑘 ∈ ℬ are mutually exclusive, then
𝑃𝑃 𝐴𝐴1 ∪ ⋯ ∪ 𝐴𝐴𝑘𝑘 = 𝑃𝑃 𝐴𝐴1 + ⋯ + P(𝐴𝐴𝑘𝑘 )
– 𝑃𝑃 ∅ = 0
– 𝑃𝑃 𝐴𝐴 ≤ 1 ∀ 𝐴𝐴 ∈ ℬ
 Properties of probability
• Useful for computing probabilities of various events
– All can be proved directly from the definition of probability

1. P( A) = 1 − P( A) ∀ A ∈ B
2. P( A) ≤ 1 ∀ A ∈ B
3. P( A ∪ B)= P( A) + P( B) − P( A ∩ B) ∀ A, B ∈ B
4. P ( A ∪ B ∪ C= ) P ( A) + P ( B ) + P (C ) + P ( A ∩ B ∩ C )
− P ( A ∩ B ) − P ( A ∩ C ) − P( B ∩ C ) ∀ A, B, C ∈ B

5. A ⊂ B ⇒ P( A) ≤ P( B) ∀ A, B ∈ B
 n  n
6. Bonferroni's − Boole's Inequality: P   Ai  ≤ ∑ P( Ai ) ∀ A1 , A2 ,... ∈ B
 i =1  i =1
 Probability in Discrete Sample Spaces
• In practice often one need to define a proper probability model

• Theorem (C&B 1.2.6)

Let 𝑆𝑆 = {𝑒𝑒1 , … , 𝑒𝑒𝑛𝑛 } be a finite set of distinct elements ;
Let 𝑝𝑝1 , … , 𝑝𝑝𝑛𝑛 ≥ 0 and such that ∑ 𝑝𝑝𝑖𝑖 = 1 . For any 𝐴𝐴 ∈ ℬ define 𝑃𝑃(𝐴𝐴) = ∑𝑒𝑒𝑖𝑖∈𝐴𝐴 𝑝𝑝𝑖𝑖
Then 𝑃𝑃 is a probability function on ℬ (ℬ be any sigma algebra defined on 𝑆𝑆) .

• Classical probability model (used with counting techniques)

If in the finite sample space 𝑆𝑆 = {𝑒𝑒1 , … , 𝑒𝑒𝑁𝑁 } elementary events are equally likely
1 # outcomes in 𝐴𝐴
to occur, then 𝑝𝑝𝑖𝑖 = ⇒ 𝑃𝑃 𝐴𝐴 =
𝑁𝑁 𝑁𝑁

• Example 1.3.2
health-related setting: testing two individuals for a condition with expected prevalence of 50%
abstract model for the experiment: tossing two balanced coins

Tossing two coins, 𝑆𝑆 = 𝑒𝑒𝑖𝑖 , 𝑖𝑖 = 1, … , 4 = {HH,HT,TH,TT}

1
Elementary events (4) are equally likely ⇒ 𝑝𝑝𝑖𝑖 = , 𝑖𝑖 = 1, … , 4
4
1 1 1
A = {obtain exactly one head}={HT,TH} ⇒ 𝑃𝑃 𝐴𝐴 = + =
4 4 2
1 1 1 3
B = {obtain at least one head}={HH,HT,TH} ⇒ 𝑃𝑃 𝐵𝐵 = + + =
4 4 4 4
1 1 1 3
C = {obtain at least one tail}={HT,TH,TT} ⇒ 𝑃𝑃 𝐶𝐶 = + + =
4 4 4 4
 Probability and Random Selection
• Definition (random selection)
Object is selected at random if it is selected from a collection of distinct
objects in such a manner that each object has the same probability of being
selected

• Example 1.3.3
Health-related setting: random selection from a cohort of patients obtaining chest X-ray, with 52
possible types of findings
Abstract model: Drawing a card at random from an ordinary deck of (52) cards.

1
Each card has the same probability of being selected 𝑝𝑝 =
52

• Example 1.4.2 (Ex. 1.3.3 continued)

A={draw a “red Ace”} B={draw a Heart card}
2 13 1
𝑃𝑃 𝐴𝐴 = , 𝑃𝑃 𝐵𝐵 = , 𝑃𝑃 𝐴𝐴 ∩ 𝐵𝐵 = ⟹
52 52 52
2 13 1 14
𝑃𝑃 𝐴𝐴 ∪ 𝐵𝐵 = + − =
52 52 52 52