2.

1 Limit Theorems 47

Applying Chebyshev’s inequality yields that the probability of a deviation

σ2
of  from the mean µ is bounded by m 2 . For fixed failure probability δ =

Pr(|X̄m − µ| > ) we have

√
δ ≤ σ 2 m−1 −2 and equivalently  ≤ σ/ mδ.
This bound is quite reasonable for large δ but it means that for high levels
of confidence we need a huge number of observations.

Much stronger results can be obtained if we are able to bound the range
of the random variables. Using the latter, we reap an exponential improve-
ment in the quality of the bounds in the form of the McDiarmid [McD89]
inequality. We state the latter without proof:

Theorem 2.11 (McDiarmid) Denote by f : Xm → R a function on X

and let Xi be independent random variables. In this case the following holds:
Pr (|f (x1 , . . . , xm ) − EX1 ,...,Xm [f (x1 , . . . , xm )]| > ) ≤ 2 exp −22 C −2 .

Here the constant C 2 is given by C 2 = m 2

P
i=1 ci where
f (x1 , . . . , xi , . . . , xm ) − f (x1 , . . . , x0i , . . . , xm ) ≤ ci
 

for all x1 , . . . , xm , x0i and for all i.

This bound can be used for averages of a number of observations when

they are computed according to some algorithm as long as the latter can be
encoded in f . In particular, we have the following bound [Hoe63]:

Theorem 2.12 (Hoeffding) Denote by Xi iid random variables with bounded

range Xi ∈ [a, b] and mean µ. Let X̄m := m−1 m
P
i=1 Xi be their average.
Then the following bound holds:
2m2
 
 

Pr X̄m − µ >  ≤ 2 exp −
  . (2.15)
(b − a)2
Proof This is a corollary of Theorem 2.11. In X̄m each individual random
variable has range [a/m, b/m] and we set f (X1 , . . . , Xm ) := X̄m . Straight-
forward algebra shows that C 2 = m−2 (b − a)2 . Plugging this back into
McDiarmid’s theorem proves the claim.
Note that (2.15) is exponentially better than the previous bounds. With
increasing sample size the confidence level also increases exponentially.

Example 2.3 (Hoeffding bound) As in example 2.2 assume that Xi are

iid random variables and let X̄m be their average. Moreover, assume that