Probability Theory

Measure Spaces
Definition 1. Sigma Algebra
Let E be a set. A non-empty collection E of subsets of E is called a sigma-algebra on E if
(1) A E E A S
E
(2) {Ai }iN E iN Ai E
Corollary 1. Let E be a set and E be a sigma-algebra on E. Then
(1) E, E
T
(2) {Ai }iN E iN Ai E
Definition 2. Measurable Space
A pair (E, E) where E is a set and E is a -algebra on E is called a measurable space.
Throughout, fix a measurable space (E, E).
Definition 3. Measurable Function
Let (E, E) and (F, F) be measurable spaces. A function f : E F is called measurable (with respect to
F and E) if
f 1 (A) = {x E|f (x) A} E f or each A F
Corollary 2. Let (E, E), (F, F) and (G, G) be measurable spaces and f : E F, g : F G be measurable.
Then g f : E G is measurable.
Definition 4. Measure
A function : E R+ is called a measure on (E, E) if
(1) ()
F =0
P
(2) ( iN Ai ) = iN (Ai ) for each disjoint {Ai }iN E
Definition 5. Measure Space
A triple (E, E, ) is called a measure space if (E, E) is a measurable space and is a measure on it.
Integration
Throughout, fix a measure space (E, E, ) and let R be equipped with its Borel sigma algebra B(R), so
that the pair form a measurable space.
Definition 6. Simple
A function : E R is called simple if for some disjoint {Ai }ni=1 E and {ai R}ni=1
=

n
X

ai 1Ai

i=1

Corollary 3. Simple functions are measurable.

Theorem 1. A function f : E R+ is measurable if and only if it can be expressed as the monotone limit
of a sequence of simple functions.
Indeed, if f is measurable then the sequence {i }iN defined by let n : E R+ be defined by

n if f (x) n
n (x) =
i1
i
n
if i1
2n
2n f (x) < 2n for i = 1, ..., n2
is a monotone sequence of simple functions converging to f.
Theorem 2. Let f : E R be measurable. Then the functions f + , f : E R+ defined by
f + (x) = max{f (x), 0} =

1
1
(|f (x)| + f (x)) and f (x) = max{f (x), 0} = (|f (x)| f (x))
2
2

are measurable, and clearly satisfy

f = f+ f

and
1

f +, f 0

Definition 7. Integral
Pn
Let : E R+ be simple so that = i=1 ai 1Ai . Then the integral of f with respect to is
Z
(x) (dx) =
E

n
X

ai (Ai )

i=1

R
and in particular note that E 1A (dx) = (A).
More generally, let f : E R+ be measurable and let {i }iN be a monotone sequence of simple functions
converging to f. Then the integral of f with respect to is
Z
Z
f (x) (dx) = lim
n (x) (dx)
n

R
R
Finally, let f : E R be measurable and such that at least one of E f + (x) (dx), E f (x) (dx) is
finite. Then the integral of f with respect to is
Z
Z
Z
f (x) (dx) =
f + (x) (dx)
f (x) (dx)
E

Definition 8. Integral over a set

Let A E and f : E R be measurable. Then the integral over A of f is defined as
Z
Z
f (x) (dx) =
f (x) 1A (x) (dx)
A

Definition 9. Monotone Class

A monotone class of functions is a collection M of measurable functions f : E R such that
(1) 1 M
(2) Bounded f, g M; a, b R af + bg M
(3) {fn }nN M; fn f f M
Theorem 3. Monotone Class Theorem for functions
Let M be a monotone class of functions on E such that 1A M for each A E. Then M contains all
positive (or negative) measurable functions and all bounded measurable functions f : E R.
Probability Spaces and Random Variables
Definition 10. Probability Space
A probability space is a measure space (, H, P) such that P() = 1.
Throughout, fix a probability space (, H, P).
Definition 11. Random Variable
A random variable is a measurable function X : E.
Definition 12. Image Measure
Let (E, E, ) be a measure space, (F, F) be a measurable space and f : E F be a measurable function.
The image measure of under f is the measure f : F R+ on F defined by
f = h1

so that

f (A) = f 1 (A) = ({x E|f (x) A})

Corollary 4. The image measure can only be defined for a measurable function.
Definition 13. Distribution Measure of a Random Variable
Let X : E be a random variable. The distribution measure of X is the image measure of P under X.
Corollary 5. Let X : E be a random variable and PX denote the distribution measure of X. Then
P(X A) = P({|X() A}) = PX (A)
2

f or each A E

Definition 14. Expected Value

Let X : R be a real-valued random variable. Then the expected value of X is defined as
Z
E(X) =
X P(d)
E

Corollary 6. E(1A ) = P(A) for each A H

Example 1. Discrete Random Variables
Let X be a real-valued random variable that takes on at most countably many values, say {ai }iN , and
let Ai = X 1 (ai ) = { |X() = ai } for each i N. Then X can be expressed
Z
X
X
X
X=
ai 1Ai
and thus
E(X) =
X() P(d) =
ai P(Ai ) =
ai P(X = ai )

iN

iN

iN

More generally, let g : R R be a measurable function. Then g X : R is again a random variable

which takes only the countably many values {g(ai )}iN . Thus, arguing as before, we have
Z
X
X
E(g(X)) =
g(X()) P(d) =
g(ai ) P(g(X) = g(ai )) =
g(ai ) P(X = ai )

iN

iN

Definition 15. Absolutely Continuous

Let , : E R+ be measures. is called absolutely continuous with respect to , denoted  if
(A) = 0 (A) = 0

f or each A E

Theorem 4. Radon-Nikodym Theorem and Derivative

Let , : E R+ be measures. Then  if and only if there exists a measurable f : E R+ such
that for each B E we have
Z
Z
Z
(B) =
f (x) (dx)
or equivalently
g(y) (dy) =
g(x) f (x) (dx)
B

for each measurable g : E R. In this case, f is called the Radon-Nikodym derivative, or density, of with
respect to .
Example 2. Continuous Random Variables
Let (R, B(R), Leb) denote the measure space of the real numbers equipped with their Borel sigma algebra
and Lebesgue measure. Let X be a real-valued random variable such that P (X Z) = 0 for each measure
zero set Z B(R), or equivalently, such that PX  Leb. Then by the Radon-Nikodym theorem there exists
f such that
Z
P(X A) = PX (A) =

f (x) dx
A

Theorem 5. The Change of Variables Theorem

Let (E, E, ) be a measure space, (F, F) be a measurable space and f : E F, g : F R be measurable
functions. Then
Z
Z
g f (x) (dx) =
E

g(y) f (dy)
F

Example 3. Continuous Random Variables Continued

As before, let X be a real valued random variable such that PX  Leb and further let g : R R be
measurable. Then
Z
Z
Z
E(g(X)) =
g(X()) P(d) =
g(x) PX (dx) =
g(x) f (x) dx

and in particular, letting g be the identity function g(x) = x we obtain

Z
E(X) =
x f (x) dt
R

Example 4. Cummulative Distribution Functions

Again, let X be a real valued random variable. Then the cummulative distribution function is defined as
F (x) = PX ((, x)) = P(X x)
Now if as above PX  Leb, then we may express this as
Z

F (x) = PX ((, x)) =

f (y) dy

and when this is the case, Lebesgues fundamental theorem of calculus yeilds that F is differentiable almost
everywhere and F 0 (x) = f (x).
Sigma Algebras and Determinability
Recall we have a fixed probability space (, H, P) and measure spaces (E, E, ), (F, F, ), (R, B(R), Leb).
Definition 16. Sigma Algebra Generated by a Random Variable
Let X : E be a random variable (that is, measurable). The sigma algebra X generated by X is the
smallest sigma algebra on such that X is measurable, and equivalently,
f = {X 1 A|A E}
Theorem 6. Let X : E be H-measurable. Another random variable Y : F is X-measurable if
and only if there exists a measurable f : E F such that
Y =f X
Corollary 7. Let {Xi : E} be H-measurable. Another random variable Y : F is {Xi }-measurable
if and only if there exists a measurable f : E F such that
Y = f (Xt1 , Xt2 , ...)
for some sequence {Xti }ti N {Xi }
Example 5. Let F H be a sub-sigma algebra on . Then
F = {1A |A F}
so that measurability with respect to an arbitrary sigma algebra may be understood in terms of the above
theorem.
Conditional Expectations and Probabilities
Definition 17. Conditional (on a sigma algebra) Expectation
Let X : R+ be a random variable and F H be a sub-sigma algebra on . The conditional
expectation of X given F is a random variable EF X such that
(1) EF X is F measurable.
(2) EEF X 1A = EX 1A for each A F
or equivalently (to 2)
(2) EEF X Y = EX Y for each F-measurable random variable Y .
More generally, for (not necesarilly positive) random variables X : R such that E(X) exists, we
define
EF X = EF X + EF X
Definition 18. Conditional (on a random variable) Expectation
Let X, Y : R+ be random variables. The conditional expectation of X given Y is
EY X = EY X
Definition 19. Conditional Probability
Let A H and F H be a sub-sigma algebra on . The conditional probability of A given F is
PF (A) = EF (1A )