Professional Documents
Culture Documents
• Probability Spaces
• Random Variables
• Scalar detection
• F does not have to be the entire power set (more on this later)
• Law of Total Probability: Let A1, A2,SA3, . . . be events that partition Ω, i.e.,
disjoint (Ai ∩ Aj = ∅ for i 6= j) and i Ai = Ω. Then for any event B
P
P(B) = iP(Ai ∩ B)
The Law of Total Probability is very useful for finding probabilities of sets
• Let A1, A2, . . . , An be nonzero probability events that partition Ω, and let B be
a nonzero probability event
• We know P(Ai ) and P(B | Ai), i = 1, 2, . . . , n, and want to find the a posteriori
probabilities P(Aj | B), j = 1, 2, . . . , n
• We know that
P(B | Aj )
P(Aj | B) = P(Aj )
P(B)
• By the law of total probability
X n n
X
P(B) = P(Ai, B) = P(Ai )P(B | Ai)
i=1 i=1
ω X(ω)
• Notations:
◦ We use upper case letters for random variables: X, Y, Z, Φ, Θ, . . .
◦ We use lower case letters for values of random variables: X = x means that
random variable X takes on the value x, i.e., X(ω) = x where ω is the
outcome
• Specifying a random variable means being able to determine the probability that
X ∈ A for any Borel set A ⊂ R, in particular, for any interval (a, b ]
• To do so, consider the inverse image of A under X , i.e., {ω : X(ω) ∈ A}
set A
inverse image of A under X(ω), i.e., {ω : X(ω) ∈ A}
Q(x)
x
The Q(·) function is used to compute P{X > a} for any Gaussian r.v. X :
Given Y ∼ N (µ, σ 2), we represent it using the standard X ∼ N (0, 1) as
Y = σX + µ
Then
y−µ y−µ
P{Y > y} = P X > =Q
σ σ
√
◦ The complementary error function is erfc(x) = 2Q( 2 x)
• Suppose we are given a r.v. X with known cdf FX (x) and a function y = g(x).
What is the cdf of the random variable Y = g(X)?
• We use
FY (y) = P{Y ≤ y} = P{x : g(x) ≤ y}
x y
y
{x : g(x) ≤ y}
√ x
− y √
y
where x1, x2, . . . are the solutions of the equation y = g(x) and g ′(xi ) is the
derivative of g evaluated at xi
+a
−1
x
+1
−a
◦ y ≥ a: Here FY (y) = 1
Combining the results, the following is a sketch of the cdf of Y
FY (y)
y
−a a
y = F −1(x)
• Note: F does not need to be continuous for the above to work. For example, to
generate Y ∼ Bern(p), we set
(
0 X ≤1−p
Y =
1 otherwise
x = F (y)
1−p
y
0 1
• Conclusion: We can generate a r.v. with any desired distribution from a U[0, 1] r.v.
• A pair of random variables X and Y defined over the same probability space are
specified by their joint cdf
FX,Y (x, y) = P{X ≤ x, Y ≤ y} , x, y ∈ R
FX,Y (x, y) is the probability of the shaded region of R2
y
(x, y)
• To find pX (x), the marginal pmf of X , we use the law of total probability
X
pX (x) = p(x, y) , x ∈ X
y∈Y
X ∼ Bern(p) Y =X ⊕Z
• X and Y are jointly continuous random variables if their joint cdf is continuous
in both x and y
In this case, we can define their joint pdf, provided that it exists, as the function
fX,Y (x, y) such that
Z x Z y
FX,Y (x, y) = fX,Y (u, v) du dv , x, y ∈ R
−∞ −∞
◦ fX,Y (x, y) ≥ 0
Z ∞Z ∞
◦ fX,Y (x, y) dx dy = 1
−∞ −∞
• X and Y are independent iff fX,Y (x, y) = fX (x)fY (y) for every x, y
• Conditional cdf and pdf: Let X and Y be continuous random variables with
joint pdf fX,Y (x, y). We wish to define FY |X (y | X = x) = P{Y ≤ y | X = x}
We cannot define the above conditional probability as
P{Y ≤ y, X = x}
P{X = x}
because both numerator and denominator are equal to zero. Instead, we define
conditional probability for continuous random variables as a limit
FY |X (y|x) = lim P{Y ≤ y | x < X ≤ x + ∆x}
∆x→0
P{Y ≤ y, x < X ≤ x + ∆x}
= lim
∆x→0 P{x < X ≤ x + ∆x}
Ry Z y
fX,Y (x, u) du ∆x fX,Y (x, u)
= lim −∞ = du
∆x→0 fX (x)∆x −∞ fX (x)
X ∼ N (0, P ) Y
Θ Y
• We wish to find the estimate Θ̂(Y ) (i.e., design the decoder) that minimizes the
probability of error:
△
Pe = P{Θ̂ 6= Θ} = P{Θ = θ0, Θ̂ = θ1} + P{Θ = θ1, Θ̂ = θ0}
= P{Θ = θ0}P{Θ̂ = θ1 | Θ = θ0} + P{Θ = θ1}P{Θ̂ = θ0 | Θ = θ1}
and the probability of error is minimized if we pick the largest pΘ|Y (Θ̂(y)|y) for
every y, which is precisely the MAP decoder
• If p = 12 , i.e., equally likely signals, using Bayes rule, the MAP decoder reduces
to the maximum likelihood (ML) decoder
(
θ0 if fY |Θ(y|θ0) > fY |Θ(y|θ1)
Θ̂(y) =
θ1 otherwise
√ √
f (y |− P ) f (y |+ P )
√ √ y
− P + P