Math 162: Probability generating functions

Ümit Işlak

Boğaziçi Üniversitesi, Matematik Bölümü

May 6, 2022

Ümit Işlak Math 162: Probability generating functions

Definition
Let X be a random variable taking the values 0, 1, 2, . . . with
probabilities p0 , p1 , p2 , . . .. The probability generating function of
the random variable X (or, of the corresponding distribution
{pk : k ≥ 0}) is the function F defined by
X
F (x) = pk xk .
k≥0

Ümit Işlak Math 162: Probability generating functions

Example
(Uniform distribution) For a ∈ N, let
1
pk = , k = 1, . . . , a.
a
. Then the corresponding probability generating function is
a
x − xa+1
 
X
k 1
1
F (x) = pk x = x + x2 + · · · + xa = .
a a 1−x
k=1

Ümit Işlak Math 162: Probability generating functions

Example
(Binomial) Let X be a binomial random variable whose pmf is
given by
(

n k n−k , if k = 0, 1, . . . , n,
f (k) = k p (1 − p)
0, otherwise.

Find the pgf of X.

Solution. We have
n n  
X
k
X n k
F (x) = pk x = p (1 − p)n−k xk
k
k=0 k=0
n  
X n
= (px)k (1 − p)n−k
k
k=0
= (1 − p + px)n ,
where the last step uses the binomial theorem. 
Ümit Işlak Math 162: Probability generating functions
Example
(Binomial) Let X be a binomial random variable whose pmf is
given by
(

n k n−k , if k = 0, 1, . . . , n,
f (k) = k p (1 − p)
0, otherwise.

Find the pgf of X.

Solution. We have
n n  
X
k
X n k
F (x) = pk x = p (1 − p)n−k xk
k
k=0 k=0
n  
X n
= (px)k (1 − p)n−k
k
k=0
= (1 − p + px)n ,
where the last step uses the binomial theorem. 
Ümit Işlak Math 162: Probability generating functions
Theorem
Let X be a random variable taking the values 0, 1, 2, . . . with
probabilities p0 , p1 , p2 , . . .. Let F be the corresponding probability
generating function. Then

µ = E[X] = F 0 (1)

and

σ 2 = V ar(X) = E[X 2 ] − (E[X])2 = F 0 (1) + F 00 (1) − (F 0 (1))2 .

P∞
Proof. For the first claim, observing F 0 (x) = n=0 npn x
n−1 , we
see that
X∞
0
F (1) = npn = E[X].
n=0

Proof of the second claim is similar, and is left for you.

Variance formula is an exercise for you. 
Ümit Işlak Math 162: Probability generating functions
Theorem
Let X be a random variable taking the values 0, 1, 2, . . . with
probabilities p0 , p1 , p2 , . . .. Let F be the corresponding probability
generating function. Then

µ = E[X] = F 0 (1)

and

σ 2 = V ar(X) = E[X 2 ] − (E[X])2 = F 0 (1) + F 00 (1) − (F 0 (1))2 .

P∞
Proof. For the first claim, observing F 0 (x) = n=0 npn x
n−1 , we
see that
X∞
0
F (1) = npn = E[X].
n=0

Proof of the second claim is similar, and is left for you.

Variance formula is an exercise for you. 
Ümit Işlak Math 162: Probability generating functions
Example
Recall that a binomial
(
random variable X
n k n−k , if k = 0, 1, . . . , n,
has pmf f (k) = k p (1 − p)
0, otherwise.
and pgf
F (x) = (1 − p + px)n , x ∈ R.
Then the previous theorem says that expectation of X is

F 0 (1) = np(1 − p + px)n−1 x=1 = np.



Ümit Işlak Math 162: Probability generating functions

Example
(Poisson) Let λ > 0 and consider the distribution determined by
−λ n
the sequence pn = e n!λ , n = 0, 1, 2, . . .. Denoting the
corresponding pgf by F ,
(i) Find F (x).
(ii) Using (i), find the expectation of a random variable Y with the
given probability sequence pn .

Solution. (i) We have

∞  −λ n  ∞
X e λ X (λx)n
F (x) = xn = e−λ = e−λ eλx = eλ(x−1) .
n! n!
n=0 n=0

(ii) Using the theorem, we obtain

E[Y ] = F 0 (1) = λeλ(x−1) x=1 = λ.




Ümit Işlak Math 162: Probability generating functions
Example
(Poisson) Let λ > 0 and consider the distribution determined by
−λ n
the sequence pn = e n!λ , n = 0, 1, 2, . . .. Denoting the
corresponding pgf by F ,
(i) Find F (x).
(ii) Using (i), find the expectation of a random variable Y with the
given probability sequence pn .

Solution. (i) We have

∞  −λ n  ∞
X e λ X (λx)n
F (x) = xn = e−λ = e−λ eλx = eλ(x−1) .
n! n!
n=0 n=0

(ii) Using the theorem, we obtain

E[Y ] = F 0 (1) = λeλ(x−1) x=1 = λ.




Ümit Işlak Math 162: Probability generating functions
NEXT: Coupon collecting problem

Following Wilf we will next see a neat application of probability

generating functions on the coupon collecting problem. The
classical model is as follows:

Ümit Işlak Math 162: Probability generating functions

 A set contains m distinct objects (which will be called
”coupons”): balls in an urn, letters in an alphabet, pictures
of soccer players sold with gums, etc.. The collector samples
from the set with replacement. On each trial she has a fixed
probability qi = 1/m of drawing a type i object, indepen-
dently of all past events.

Let pn be the probability that exactly n trials are needed in order

to have a complete collection for the first time. We have the
following claims:

Ümit Işlak Math 162: Probability generating functions

1 We have
m!S2 (n − 1, m − 1)
pn = ,
mn
where S2 (n, k) is the Stirling number of the second kind.
2 Letting F (x) be the generating function of the sequence pn ,
we have
(m − 1)!xm
F (x) = .
(m − x)(m − 2x) · · · (m − (m − 1)x)
3 The average number of trials that are needed to get a
complete collection of all m coupons is given by
 
1 1
µ = m 1 + + ··· + ∼ m ln m.
2 m
4 The standard deviation of number of trials that are needed to
get a complete collection of all d coupon is given by
v
u m  
u X
2
1 1 1
σ= mt − m 1 + + ··· + .
i2 2 m
i=1

Ümit Işlak Math 162: Probability generating functions

1 We have
m!S2 (n − 1, m − 1)
pn = ,
mn
where S2 (n, k) is the Stirling number of the second kind.
2 Letting F (x) be the generating function of the sequence pn ,
we have
(m − 1)!xm
F (x) = .
(m − x)(m − 2x) · · · (m − (m − 1)x)
3 The average number of trials that are needed to get a
complete collection of all m coupons is given by
 
1 1
µ = m 1 + + ··· + ∼ m ln m.
2 m
4 The standard deviation of number of trials that are needed to
get a complete collection of all d coupon is given by
v
u m  
u X
2
1 1 1
σ= mt − m 1 + + ··· + .
i2 2 m
i=1

Ümit Işlak Math 162: Probability generating functions

The table below show how much we should wait on the average for
varying m:

m 3 4 5 10 20 50 100 500 1000

mean waiting time 3.3 5.5 8 23 60 196 461 3107 6908

So if you are trying to collect the photos of soccer players of 32

countries in a World Cup, you will need to around 3000 cards on
the average! And of course the probabilities are not uniform in real
life! That is, some players will have real small probability to appear.

Ümit Işlak Math 162: Probability generating functions

The table below show how much we should wait on the average for
varying m:

m 3 4 5 10 20 50 100 500 1000

mean waiting time 3.3 5.5 8 23 60 196 461 3107 6908

So if you are trying to collect the photos of soccer players of 32

countries in a World Cup, you will need to around 3000 cards on
the average! And of course the probabilities are not uniform in real
life! That is, some players will have real small probability to appear.

Ümit Işlak Math 162: Probability generating functions

Proof of claims.
1. Let’s consider sequences of length n of pictures for which we
complete the collection at exactly nth picture/trial. For now, let’s
say that we obtain mth type of picture in nth trial.
Since we complete it at exactly nth one, the picture at that time
must occur for the first time at that trial. Now we will partition
the set {1, 2, . . . , n − 1} (first n − 1 pictures) into m − 1 groups so
that if the ith type of the pictures occurs in jth selection
(1 ≤ i ≤ m, 1 ≤ j ≤ n − 1), then we put j into ith group. Note
that none of these m − 1 groups can be empty.
Since the type of the picture at nth is the mth type (for now),
there are (m − 1)!S2 (n − 1, m − 1) distinct ways to partition
{1, 2, . . . , n − 1} into m − 1 groups as we described. Here,
(m − 1)! is due to the fact that the types of the pictures may
occur in this many different orders.

Ümit Işlak Math 162: Probability generating functions

Proof of claims.
1. Let’s consider sequences of length n of pictures for which we
complete the collection at exactly nth picture/trial. For now, let’s
say that we obtain mth type of picture in nth trial.
Since we complete it at exactly nth one, the picture at that time
must occur for the first time at that trial. Now we will partition
the set {1, 2, . . . , n − 1} (first n − 1 pictures) into m − 1 groups so
that if the ith type of the pictures occurs in jth selection
(1 ≤ i ≤ m, 1 ≤ j ≤ n − 1), then we put j into ith group. Note
that none of these m − 1 groups can be empty.
Since the type of the picture at nth is the mth type (for now),
there are (m − 1)!S2 (n − 1, m − 1) distinct ways to partition
{1, 2, . . . , n − 1} into m − 1 groups as we described. Here,
(m − 1)! is due to the fact that the types of the pictures may
occur in this many different orders.

Ümit Işlak Math 162: Probability generating functions

Now, noting that we could have m distinct cases for the nth
selection, we see that there are
m(m − 1)!S2 (n − 1, m − 1) = m!S2 (n − 1, m − 1) ways in which
we complete the collection at exactly nth selection. Since there are
mn many sequences when there are no restrictions, the probability
turns out to be as claimed.

Ümit Işlak Math 162: Probability generating functions

2. Recall that the generating function of S2 (n, k) is
∞
X xk
S2 (n, k)xn = .
(1 − x)(1 − 2x) · · · (1 − kx)
n=0

Then we have
X m!S2 (n − 1, m − 1)
F (x) = xn
mn
n≥0
X  x n
= m! S2 (n − 1, m − 1)
m
n≥1
x X  x n
= m! S2 (n, m − 1)
m m
n≥0
x  x m−1 1
= m! (m−1)x
m m (1 − mx
− 2x
)(1 m ) · · · (1 − m )
(m − 1)!x m
= .
(m − x)(m − 2x) · · · (m − (m − 1)x)

Ümit Işlak Math 162: Probability generating functions

  
m
3. Noting that F (x) = (m − 1)! Qm−1x , we have
j=1 (m−jx)

Qm−1
mxm−1 (m − jx) − xm m−1
P Q
0 j=1 i=1 (−i) j6=i (m − jx)
F (x) = (m−1)! Qm−1 2
j=1 (m − jx)

Substituting x = 1 gives
m(m − 1)! + m−1
!
P i
i=1 m−i (m − 1)!
F 0 (1) = (m − 1)!
((m − 1)!)2
m−1
X i
= m+
m−i
i=1
= ···  
1 1
= 1 + m 1 + + ··· +
2 m−1
 
1 1
= m 1 + + ··· +
2 m
∼ m ln m.
Ümit Işlak Math 162: Probability generating functions
4. Follows from the corresponding formula for the variance in
terms of the generating function. Details are left for you. 

Ümit Işlak Math 162: Probability generating functions

Arrivederci

Next lecture:

Selamlar, iyi çalışmalar,

Ümit

Ümit Işlak Math 162: Probability generating functions