STA 741 / ECE 741: Homework 0

Compressed Sensing and Related Topics

Duke University, Spring 2016
Due: Never
Remark: You do not need to turn in this homework. It is meant to be a self-diagnosis tool for
the material based on the prerequisites for this class. Unmarked problems should be solvable immediately. Problems marked with a single star (*) are problems which may require more thoughts.
Problems marked with a double star (**) use some more advanced concepts which may not have
been covered before.
If you find the problems on this homework very challenging, you may find the class to be difficult
without significant extra effort. If you do not understand some of the problems on the homework,
you probably do not have the sufficient mathematical background to follow the material in the
course. Students are welcome to audit the course without receiving credit.
Problem 0.1 There is a 1% chance I have a certain disease. I take a test for this disease which is
90% accurate (i.e. P[test is pos|I have disease] = P[test is neg|I dont have disease] = 0.9). Given
the test is positive, what is the probability I have the disease.
Problem 0.2 Let Y be a non-negative continuous random variable with mean , variance 2 , and
cdf F (y). Compute the following:
(a) P[F (Y ) > 3/4]
R
(b) (*) 0 (1 F (y))dy
(c) the mean and variance of distribution Z = XY where X is Bernoulli-p (i.e. P[X = 1] =
1 P[X = 0] = p) and independent of Y .
Problem 0.3 Two players A and B alternate flipping a coin until one of them flips a head.
Assuming that A starts, what the probability that B wins.
Problem 0.4 Let X and Y be discrete random variables defined on some probability space with a
joint probability mass function (pmf) given by p(x, y). Note that X and Y need not be independent.
(a) Prove that E[X + Y ] = E[X] + E[Y ]. Do not assume independence.
(b) Prove that if X and Y are independent, then E[XY ] = E[X] E[Y ].
(c) Assume that X and Y are not independent. Find an example for which E[XY ] = E[X]E[Y ]
and another example for which E[XY ] 6= E[X]E[Y ].
Problem 0.5 (*) Let X, Y, Z be discrete random variables whose joint distribution is such that
their values are distinct with probability 1. Let a = P (X > Y ), b = P (Y > Z) and c = P (Z > X).
Show that min{a, b, c} 2/3. Find an example where there is an equality.
Problem 0.6 Give an example of a random variable X taking values on positive numbers for
which E[1/X] 6= 1/E[X]. What can you say in general about E[1/X] and 1/E[X] (e.g. find an
inequality). Is it ever possible for them to be equal?
Problem 0.7 Let X1 , X2 , . . . , Xn be a sequence of n i.i.d. random variables taking values in {0, 1}
and P (Xi = 1) = r < 1/2.

STA 741 / ECE 741 Homework 0

(a) What is the probability mass function (pmf) for X1 , X2 , . . . , Xn ? Let k denote the number of
1s. Find a function (r, k, n) such that your answer can be written as exp(n (r, k, n)).
(b) What is the probability that exactly k of the Xi are equal to 1?
(c) (*) Minimize the function (r, k, n) over r for a fixed k and n. What is the maximizing value?
Explain why this makes sense in terms of the probability of the sequence. If you fix r and n,
what is the k that maximizes (r, k, n)?
Problem 0.8 Which of the following are valid density functions?
(a) f (x) =

1
1+x2

(b) f (x, y) =

for x R

1
2 2

exp 12

(c) f (x, y) = 3 ex +
(d) f (x) =

1
2

1 2
2x

2 x
3 e

+ y2

for (x, y) R2

for x, y 0

for x {1, 1}

Problem 0.9 Suppose X = (X1 , X2 , . . . , Xd ) is a d-dimensional jointly Gaussian vector with

distribution N (0, ) for some positive definite matrix .
P
(a) Write down the distribution of Y = di=1 Xd as a function of .
(b) Suppose d = 3 and

3 2 1
= 2 3 2 .
1 2 3
Find the conditional distribution of (X1 , X2 ) given X3 .
(c) (**) For the same in the previous part, can you find a matrix A such that X = AU where
U N (0, I)?
Problem 0.10 Which of the following functions are convex?
(a) f (z) = max{0, 1 z}
(b) f (z) = log(1 + ez )
(c) f (x, y) = 4x 3y
Problem 0.11 (*) Suppose that a probability distribution over a finite set X = {1, 2, . . . , M } has
the following form. Given a vector of parameters Rd , the probability of x X is given by


P (x; ) = exp > (x) A() ,
where : X Rd is a function that assigns a d-dimensional vector to each x X .
(a) Give a formula for A() in terms of {(x) : x X }.
(b) Calculate A() (the gradient as a function of ).
(c) (**) Calculate 2 A() (the Hessian as a function of ).
(d) (**) Is A() convex? Why or why not?