Chapter 2 Summary Absolutely Continuous Random Variables

P
• If x∈R P (X = x) = 0, then X is a continuous random
Discrete Random Variables variable.
P
• If x∈R P (X = x) = 1, then X is a discrete random • An absolutely continuous random variable X has a den-
variable. sity function fX (x) such that
• The probability function is pX (x) = P (X = x). Z b
P (a < X < b) = fX (x) dx
• The law of total probability, discrete random variable ver- a
sion, is
for all a ≤ b.
X
P (A) = P (X = x) P (A | X = x) . • For
R ∞ density functions, f (x) ≥ 0 for all x ∈ R and
x∈R f (x) dx = 1.
−∞ X

• Degenerate distributions: P (X = c) = 1 for some c. • Uniform distributions: X ∼ Uniform(L, R).

• Bernoulli distributions: X ∼ Bernoulli(θ). Single f (x) = 1/(R − L) for L < x < R .

coin toss. P (X = 1) = θ, P (X = 0) = 1 − θ where
X ∼ Uniform(0, 1) is a special case.
0 < θ < 1 is a parameter.
• Exponential distributions: X ∼ Exponential(λ). For
• Binomial distributions: X ∼ Binomial(n, θ). Number
parameter λ > 0, the density is
of heads in n coin tosses, probability of a head is θ for
each toss where 0 < θ < 1 and n is a positive integer. f (x) = λe−λx for x ≥ 0 .
 
P (X = k) =
n k
θ (1 − θ)n−k for k = 0, 1, . . . , n . The right tail probability is P (X > x) = e−λx .
k
• Gamma distributions: X ∼ Gamma(α, λ), for shape
X = X1 + · · · + Xn where Xi ∼ i.i.d. Bernoulli(θ). parameter α > 0 and scale parameter λ > 0. The density
is
• Geometric distributions: X ∼ Geometric(θ). Num- λα α−1 −λx
f (x) = x e for x ≥ 0,
ber of tails before the first head in repeated coin tosses Γ(α)
with head probability θ for each toss where 0 < θ < 1. where Z ∞
P (X = k) = θ(1 − θ) k
for k = 0, 1, 2, . . . . Γ(α) = xα−1 e−x dx
0

• Negative Binomial distributions: is the gamma function, defined when α > 0.

X ∼ Negative Binomial(r, θ). Number of tails before the Note that√Γ(n + 1) = n! for nonnegative integer n,
rth head in repeated coin tosses with head probability θ Γ(1/2) = π, and Γ(α + 1) = αΓ(α).
for each toss where 0 < θ < 1. Gamma(1, λ) is Exponential(λ).
 
k+r−1 r X = X1 + · · · + Xn where Xi ∼ i.i.d. Exponential(λ).
P (X = k) = θ (1 − θ)k for k = 0, 1, 2, . . . .
r−1
• Normal distributions: X ∼ N(µ, σ 2 ). Bell-shaped
X = X1 + · · · + Xn where Xi ∼ i.i.d. Geometric(θ). curves. The density is
1 2
• Poisson distributions: X ∼ Poisson(λ). For parame- f (x) = √ e−((x−µ)/σ) /2 (−∞ < x < ∞) .
ter λ > 0, σ 2π

e−λ λk In the standard normal distribution, µ = 0, σ 2 = 1 and

P (X = k) = for k = 0, 1, 2, . . . . the density is
k! 1 2
φ(z) = √ e−z /2 .
If Xn ∼ Binomial(n, θn ) for λ = nθn , n large and θn very 2π
small, then P (Xn = k) ≈ P (X = k). If Z ∼ N(0, 1), then X = µ + σZ ∼ N(µ, σ 2 ).
• Hypergeometric distributions: The cumulative distribution function of the standard nor-
X ∼ Hypergeometric(N, M, n). Sample n balls without mal distribution is
replacement from M white and N −M black balls, count- Z z
ing the number of white balls in the sample. Φ(z) = φ(z)dz .
−∞
M N −M



2
P (X = k) = k n−k
, If X ∼ N(µ, σ ), then
N


n
 
X −µ
P (X ≤ x) = Φ .
where k = max{0, n − N + M }, . . . , min{M, n}. σ

1
 • Beta distributions: X ∼ Beta(a, b). With parameters • For absolutely continuous random variables X and Y ,
a > 0, b > 0, the density is the joint density function fX,Y (x, y) defined so that
RdRb
P (a ≤ X ≤ b, c ≤ Y ≤ d) = c a fX,Y (x, y) dx dy deter-
1
f (x) = xa−1 (1 − x)b−1 for 0 < x < 1 . mines the joint distribution.
B(a, b) R∞ R∞
• −∞ −∞ fX,Y (x, y) dx dy = 1
where the beta function B(a, b) is defined for a > 0, b > 0
as • For discrete random variables X and Y ,
Z 1 X
Γ(a)Γ(b) P (X = x) = P (X = x, Y = y)
B(a, b) = xa−1 (1 − x)b−1 dx = .
0 Γ(a + b) y
X
Beta(1,1) is Uniform(0,1). P (Y = y) = P (X = x, Y = y)
x

Cumulative Distribution Functions • For absolutely continuous random variables X and Y

• The cumulative distribution function (cdf ) is defined as with joint density fX,Y ,
F (x) = P (X ≤ x). Z ∞
fX (x) = fX,Y (x, y) dy
• F determines the distribution. −∞
Z ∞
• For an absolutely continuous random variable X with fY (y) = fX,Y (x, y) dx
d −∞
density f , dx F (x) = f (x).
• 0 ≤ F (x) ≤ 1 for all x, F is nondecreasing. Conditional Distributions

• limx→−∞ F (x) = 0, limx→∞ F (x) = 1. • For discrete random variables X and Y , the conditional
distribution of Y given X = x is the collection of proba-
One-Dimensional Change of Variable bilities
P (Y ∈ B ∩ X = x)
• If X is a discrete random variable and Y = h(X), then P (Y ∈ B | X = x) =
P (X = x)
X
P (Y = y) = P (X = x) . defined whenever P (X = x) > 0.
x:h(x)=y
• The conditional probability function of Y given X is
• If X is an absolutely continuous random variable with
P (X = x, Y = x)
density fX ; if Y = h(X) where h is strictly increasing or pY |X (y | x) =
strictly decreasing when fX (x) > 0 so that y = h(x) im- P (X = x)
plies that x = h−1 (y) is unique; and if h is differentiable defined whenever P (X = x) > 0.
with derivative h0 ; then
• For absolutely continuous random variables X and Y ,
fY (y) = fX (h−1 (y))/|h0 (h−1 (y))| . the conditional density of Y given X = x is defined to be

Joint Distributions fX,Y (x, y)

fY |X (y | x) =
fX (x)
• The joint distribution of random variables X and Y is
the collection of probabilities P ((X, Y ) ∈ B) for all mea- defined whenever fX (x) > 0.
surable subsets B ∈ R2 . • The conditional distribution of Y given X is the collection
• The joint cumulative distribution function FX,Y (x, y) = of probabilities
P (X ≤ x, Y ≤ y) determines the joint distribution of X Z b
and Y . P (Y ∈ B | X = x) = fY |X (y | x) dy
a
• The joint distribution of X and Y determines the
marginal distributions of X and of Y . for a ≤ b defined whenever P (X = x) > 0.

• For discrete random variables X and Y , the joint proba- Independence

bility function pX,Y (x, y) = P (X = x, Y = y) determines
the joint distribution. • Two random variables X and Y are independent if
P P P (X ∈ B1 , Y ∈ B2 ) = P (X ∈ B1 ) P (Y ∈ B2 ) for all
• x y p X,Y (x, y) = 1. measurable sets B1 , B2 ⊂ R. In other words, random
variables X and Y are independent if any event involv-
ing X and any event involving Y are independent.

2
 • Discrete random variables X and Y are independent if • If X1 , . . . , Xn are absolutely continuous random variables
and only if P (X = x, Y = y) = P (X = x) P (Y = y) for with joint density fX1 ,...,Xn and if random variables Yi =
all x, y ∈ R. In addition, discrete random variables X hi (X1 , . . . , Xn ) are defined so that hi is differentiable and
and Y are independent if and only if P (Y = y | X = x) = so that h = (h1 , . . . , hn ) is one-to-one on the region where
P (Y = y) for all y ∈ R and all x ∈ R such that fX1 ,...,Xn (x1 , . . . , xn ) > 0, then
P (X = x) > 0.
fX1 ,...,Xn (h−1 (y1 , . . . , yn ))
• Absolutely continuous random variables X and Y are fY1 ,...,Yn (y1 , . . . , yn ) =
|J(h−1 (y1 , . . . , yn ))|
independent if and only if the densities can be chosen
so that fX,Y (x, y) = fX (x)fY (y) for all x, y ∈ R. In Convolution
addition, absolutely continuous random variables X and
Y are independent if and only if fY |X (y | x) = fY (y) for • If X and Y are independent discrete random variables
all y ∈ R and all x ∈ R such that fX (x) > 0. and Z = X + Y , then
• If random variables X and Y are independent, then ran-
X
P (Z = z) = P (X = x) P (Y = z − x) for all z ∈ R .
dom variables g(X) and h(Y ) are also independent. x

• Collections of discrete random variables X1 , X2 , . . . , Xn

• If X and Y are independent absolutely continuous ran-
are independent if
dom variables and Z = X + Y , then
n
Y Z ∞
P (X1 = x1 , . . . , Xn = xn ) = P (Xi = xi ) fZ (z) = fX (x)fY (z − x) dx for all z ∈ R .
i=1 −∞

for all numbers x1 , . . . , xn ∈ R. Simulation

• Collections of absolutely continuous random variables • Some special distributions can be simulated from directly
X1 , X2 , . . . , Xn are independent if density functions can as functions of uniform random variables.
be chosen so that
n • If U ∼ Uniform(0, 1), then X = (R − L)U + L ∼
Y
fX1 ,...,Xn (x1 , . . . , xn ) = fXi (xi ) Uniform(L, R).
i=1
• If U ∼ Uniform(0, 1), then X = − ln(U )/λ ∼
for all numbers x1 , . . . , xn ∈ R. Exponential(λ).

• A collection of random variables is independent and iden- • If U1 , U2 ∼ i.i.d. Uniform(0, 1), and we define
tically distributed, or i.i.d. if they are independent and p
have the same marginal distributions. X = −2 ln(U1 ) cos(2πU2 )

and
Order Statistics p
Y = −2 ln(U1 ) sin(2πU2 ),
• Random variables X1 , . . . , Xn sorted from smallest to then X ∼ N(0, 1), Y ∼ N(0, 1), and X and Y are inde-
largest are labeled pendent.
X(1) , . . . , X(n) • If Z ∼ N(0, 1), then X = µ + σZ ∼ N(µ, σ 2 ).

and are called the order statistics of the sample. • If X is a random variable with cumulative distribution
function F , then F −1 (t) = min{x : F (x) ≥ t} for 0 <
• X(1) is the minimum, X(n) is the maximum. t < 1 is the inverse cdf or quantile function of X.

Multidimensional Change of Variable • If U ∼ Uniform(0, 1), then Y = F −1 (U ) is a random

variable with cdf F .
• If h = (h1 , . . . , hn ) : Rn → Rn is a differentiable function,
its Jacobian derivative is defined as
∂(h1 , . . . , hn )
= J(x1 , . . . , xn ) = det M
∂(x1 , . . . , xn )
∂hj
where M is an n × n matrix whose i, j element is ∂xi . In
particular for n = 2,
∂h1 ∂h2 ∂h1 ∂h2
J= −
∂x1 ∂x2 ∂x2 ∂x1