Chair of Optimization and Data Analysis

Department of Mathematics
Technical University of Munich
"

Note:
• During the attendance check a sticker containing a unique code will be put on this exam.
E0177 • This code contains a unique number that associates this exam with your registration
Place student sticker here number.
• This number is printed both next to the code and to the signature field in the attendance
check list.
"

Foundations in Data Analysis

cit-fda-1-20230807-E0177-01

Exam: MA4800 / Summer Term Date: Monday 7th August, 2023

Examiner: Prof. Dr. Massimo Fornasier/ Dr. Anna Veselovska Time: 15:30 – 17:00
"

P1 P2 P3 P4

I
"

Working instructions
• This exam consists of 16 pages with a total of 4 problems.
cit-fda-1-20230807-E0177-01

Please make sure now that you received a complete copy of the exam.
"

• The total amount of achievable credits in this exam is 100 credits.

• Detaching pages from the exam is prohibited.
• Allowed resources:
– a A4 sheet of paper written on both sides by hand

• Answers are only accepted if the solution approach is documented. Give a reason for each
answer unless explicitly stated otherwise in the respective subproblem.
"

• Do not write with red or green colors nor use pencils.

• Physically turn off all electronic devices, put them into your bag and close the bag.
"

Left room from to / Early submission at

Exam empty – Page 1 / 16 – cit-fda-1-20230807-E0177-01

 Problem 1 Projections and Singular Value Decomposition (25 credits)

"
0 a) Consider the `2 -ball of radius r in Rn defined as B2r := {x ∈ Rn : kx k2 ≤ r }. Show that for any vector x ∈ Rn ,

cit-fda-1-20230807-E0177-02
1 its `2 projection onto B2r can be computed as
2
3 x
PB2r (x) = r · .
4 max{r, kx k2 }
5
Is the same formular valid the sphere Srn−1 := {x ∈ Rn : kx k2 = r }? Justify your answer.

"
"
cit-fda-1-20230807-E0177-02
"
"
"
cit-fda-1-20230807-E0177-02
"

cit-fda-1-20230807-E0177-02 – Page 2 / 16 – Page empty

 b) Consider some number ε ∈ (0, 1) and the matrix 0
  1
"

−1 0 2
A= ,
0 −1 − ε 3
4
compute the singular values and the right and left singular vectors of A . Write down the SVD of A . 5
6
7
8
9
10
"
cit-fda-1-20230807-E0177-03

"
"
cit-fda-1-20230807-E0177-03

"
"
"

Page empty – Page 3 / 16 – cit-fda-1-20230807-E0177-03

0 c) Now consider the matrix A
e defined as A
e := A + E with
1  

"
2 0 ε
3
E= ,
ε ε

cit-fda-1-20230807-E0177-04
4
for the same matrix A ∈ R2×2 and ε ∈ (0, 1) as in Problem 1.b). The leading right singular vector of A e is given
by ev1 = √12 (−1, 1)T Write down A
e explicitly, check that e
v1 is a singular vector of A
e and explain why the factor
√1 is necessary here.
2

"
"
cit-fda-1-20230807-E0177-04
"
"
"
cit-fda-1-20230807-E0177-04
"

cit-fda-1-20230807-E0177-04 – Page 4 / 16 – Page empty

 d) Let now V and V e be the one-dimensional spaces spanned by the leading right singular vector of A and 0
the leading right singular vector of A
e , respectively. 1
"

Write down the orthogonal projection operators PV and Pe in matrix form and compute the value of kPV − Pe k . 2
V V F
What can this value tell us about the stability of SVD computation? 3
4
5
6
"
cit-fda-1-20230807-E0177-05

"
"
cit-fda-1-20230807-E0177-05

"
"
"

Page empty – Page 5 / 16 – cit-fda-1-20230807-E0177-05

 Problem 2 Probability and Dimensionality Reduction (25 credits)

"
0 a) Let g be a standard Gaussian random variable. Show that g 2 is subexponential for suitable constant

cit-fda-1-20230807-E0177-06
1 parameters. Namely, show that there exist constants κ, β > 0 such that P(g 2 ≥ t) ≤ β exp(−κt). Determine
2 the constants β and κ explicitly.
3
4
5
6

"
"
cit-fda-1-20230807-E0177-06
"
"
"
cit-fda-1-20230807-E0177-06
"

cit-fda-1-20230807-E0177-06 – Page 6 / 16 – Page empty

 b) Now consider the matrix A ∈ Rk ×d , whose entries are sampled independently and identically according to 0
the standard Gaussian distribution. Show that for any deterministic vector x ∈ Rd , we have 1
"

2
1 2 3
Ek √ Ax k2 = kx k22 4
k
5
2 6
Hint: Analyzing E(Ax)2j = Ehaj , x i , where aj denotes the j -th row of the matrix A , for each j ∈ {1, .., k }, might 7
be useful. 8
"
cit-fda-1-20230807-E0177-07

"
"
cit-fda-1-20230807-E0177-07

"
"
"

Page empty – Page 7 / 16 – cit-fda-1-20230807-E0177-07

0 c) For the same matrix A ∈ Rk ×d as in Problem 2.b) and a random variable Zj defined as
1

"
2 haj , x i
3
Zj := , (??)
kx k2

cit-fda-1-20230807-E0177-08
4
where aj again denotes the j -th row of the matrix A , compute E[Zj ] and Var[Zj ].

"
"
cit-fda-1-20230807-E0177-08
"
"
"
cit-fda-1-20230807-E0177-08
"

cit-fda-1-20230807-E0177-08 – Page 8 / 16 – Page empty

 d) Recall that Z1 , ... , Zk defined in (??) satisfy 0
1
"

k
X  2
P Zj2 > (1 + ε)k ≤ exp(−k /4(ε2 − ε3 )) 3
j=1 4
5
and use this to show that for any deterministic vector x ∈ Rd it holds that 6
7
 1 
2
P k √ Ax k2 > (1 + ε)kx k22 ≤ exp(−k /4(ε2 − ε3 )).
k

How is this result related to the Johnson-Lindenstrauss Lemma?

"
cit-fda-1-20230807-E0177-09

"
"
cit-fda-1-20230807-E0177-09

"
"
"

Page empty – Page 9 / 16 – cit-fda-1-20230807-E0177-09

 Problem 3 Duality and Optimization (25 credits)
For some non-zero matrix A ∈ Rm×n , consider the following optimization problem

"
1

cit-fda-1-20230807-E0177-10
minx ∈Rn kx k22 subject to Ax = 0. (?)
2

0 a) Find the convex conjugate of the function f (x) = 21 kx k22 .

1
2
3
4

"
5
6
7
8

"
cit-fda-1-20230807-E0177-10
"
"
"
cit-fda-1-20230807-E0177-10
"

cit-fda-1-20230807-E0177-10 – Page 10 / 16 – Page empty

 b) State the Lagrange function L(x, ξ ) of the problem (?), and compute the corresponding Lagrange dual 0
function H(ξ ). All suprema and infima need to be explicitly resolved. 1
"

2
3
4
5
6
7
8
"
cit-fda-1-20230807-E0177-11

"
"
cit-fda-1-20230807-E0177-11

"
"
"

Page empty – Page 11 / 16 – cit-fda-1-20230807-E0177-11

0 c) State and solve the dual problem to the primal problem (?). Does the strong duality for the primal problem
1 (?) and its dual problem hold? Justify your answer, and based on it provide either the optimal value of the

"
2 primal problem (?) or a lower bound for this optimal value.
3

cit-fda-1-20230807-E0177-12
4
5
6
7
8
9

"
"
cit-fda-1-20230807-E0177-12
"
"
"
cit-fda-1-20230807-E0177-12
"

cit-fda-1-20230807-E0177-12 – Page 12 / 16 – Page empty

 Problem 4 Gradient-based Algorithms (25 credits)
Consider the function f : R2 → R defined for each x = (x1 , x2 )T ∈ R2 as
"

1 T
f (x) = · x Ax + b T x
2
 
2 1
for the positive semi-definite matrix A = and the vector b = (−1, 1)T .
1 2
In this task, we will set up a gradient descent algorithm in order to find the minimizer of f in R2 .

a) Compute ∇f , and prove that the function f is convex using its Hessian ∇2 f . 0
1
"

2
3
4
5
cit-fda-1-20230807-E0177-13

6
7
"
"

b) Show that ∇f is 3-Lipschitz continuous function. You can use without proof that the spectral norm kA k = 3. 0
1
2
cit-fda-1-20230807-E0177-13

3
"

4
5
"
"

Page empty – Page 13 / 16 – cit-fda-1-20230807-E0177-13

0 c) Set up the gradient descent algorithm in order to find a minimizer of the above-stated function f and
1 calculate the first step of this algorithm, i.e. x (1) , with the step size (learning rate) α = 1/4 and the initial value

"
2 x (0) = (1, 1)T . Indicate a range for α such that the algorithm is guaranteed to converge to the global minimizer.
3

cit-fda-1-20230807-E0177-14
4
5
6
7

"
"
cit-fda-1-20230807-E0177-14
"
"
"
cit-fda-1-20230807-E0177-14
"

cit-fda-1-20230807-E0177-14 – Page 14 / 16 – Page empty

 d) Are there different initial values for which the iterates of the above-set gradient descent algorithm with 0
optimal step size will be stuck at a local minimum? Justify your answer. 1
"

2
3
4
5
6
"
cit-fda-1-20230807-E0177-15

"
"
cit-fda-1-20230807-E0177-15

"
"
"

Page empty – Page 15 / 16 – cit-fda-1-20230807-E0177-15

Additional space for solutions–clearly mark the (sub)problem your answers are related to and strike
out invalid solutions.

"
cit-fda-1-20230807-E0177-16
"
"
cit-fda-1-20230807-E0177-16
"
"
"
cit-fda-1-20230807-E0177-16
"

cit-fda-1-20230807-E0177-16 – Page 16 / 16 – Page empty