Professional Documents
Culture Documents
to Data Mining
Master 1
Computational Colour and Spectral Imaging (COSI)
Cyber‐Physical Social System (CPS2)
Saint-Étienne, France
Fabrice MUHLENBACH
https://perso.univ-st-etienne.fr/muhlfabr/
e-mail: fabrice.muhlenbach@univ-st-etienne.fr 1
Tutorial
• Convex Optimization
➢ Exercise 1: we want to solve:
max 𝑥. 𝑦
ቊs. t. 𝑥 + 3𝑦 = 24
➢ Questions:
o what are the objective and constraint functions?
o solve this system by using the method of Lagrange multipliers
• Convex Optimization
➢ Solution:
max 𝑥. 𝑦
ቊs. t. 𝑥 + 3𝑦 = 24
• Convex Optimization
o ℒ 𝑥, 𝑦, 𝜆 = 𝑥. 𝑦 − 𝜆 𝑥 + 3𝑦 − 24 = 𝑥. 𝑦 − 𝜆𝑥 − 3𝜆𝑦 + 24𝜆
o optimizing a function → finding the critical points
→ finding when the derivative of the function is equal to zero
o we have 3 variables: 𝑥, 𝑦, 𝜆, therefore the function need to be derived
3 times
o there are 3 first order conditions:
𝜕ℒ
o =0⇔𝑦−𝜆 =0⇔𝜆 =𝑦 1
𝜕𝑥
𝜕ℒ 𝑥
o = 0 ⇔ 𝑥 − 3𝜆 = 0 ⇔ 𝜆 = 2
𝜕𝑦 3
𝜕ℒ
o = 0 ⇔ 𝑥 + 3𝑦 − 24 = 0 3
𝜕𝜆
From Statistics 2 DM –Tutorial F. Muhlenbach 14
Tutorial
• Convex Optimization
o ℒ 𝑥, 𝑦, 𝜆 = 𝑥. 𝑦 − 𝜆 𝑥 + 3𝑦 − 24 = 𝑥. 𝑦 − 𝜆𝑥 − 3𝜆𝑦 + 24𝜆
𝑥
o 1 and 2 : 𝑦 = 4
3
𝑥
o 3 and 4 : 𝑥 + 3 × − 24 = 0
3
o ⇔ 𝑥 + 𝑥 − 24 = 0
o ⇔ 2𝑥 = 24
o ⇔ 𝑥 = 12
𝑥 12
o with 4 : 𝑦 = = =4
3 3 𝑥 = 12
o with 1 : 𝜆 = 𝑦 = 4 therefore ቐ 𝑦 = 4
𝜆=4
From Statistics 2 DM –Tutorial F. Muhlenbach 15
Tutorial
• Convex Optimization
➢ Exercise 2: we want to solve:
min 2𝑥 + 2𝑦
ቊ
s. t. 𝑥. 𝑦 = 4
➢ Solution: → constraint: 𝑥. 𝑦 = 4 ⟺ 𝑥. 𝑦 − 4 = 0
o ℒ 𝑥, 𝑦, 𝜆 = 2𝑥 + 2𝑦 − 𝜆 𝑥. 𝑦 − 4 . FOC:
𝜕ℒ 2
o = 0 ⇔ 2 − 𝜆𝑦 = 0 ⇔ 2 = 𝜆𝑦 ⇔ 𝜆 = 1
𝜕𝑥 𝑦
𝜕ℒ 2
o = 0 ⇔ 2 − 𝜆𝑥 = 0 ⇔ 2 = 𝜆𝑥 ⇔ 𝜆 = 2
𝜕𝑦 𝑥
𝜕ℒ
o = 0 ⇔ 𝑥. 𝑦 − 4 = 0 3
𝜕𝜆
• Convex Optimization
o ℒ 𝑥, 𝑦, 𝜆 = 2𝑥 + 2𝑦 − 𝜆 𝑥. 𝑦 − 4
2 2
o 1 and 2 : = ⇔𝑥=𝑦 4
𝑦 𝑥
o 3 and 4 : 𝑥 2 − 4 = 0 ⇔ 𝑥 2 = 4 ⟺ 𝑥 = 2 5
o 4 and 5 : 𝑦 = 2 and 𝜆 = 1
𝑥=2
o therefore ቐ𝑦 = 2
𝜆=1
• Convex Optimization
➢ Exercise 3: we want to solve:
𝑓 𝑥, 𝑦 = max 𝑥 2 𝑦
൝
s. t. 𝑥 2 + 𝑦 2 = 1
• Convex Optimization
➢ Solution: we want to solve:
𝑓 𝑥, 𝑦 = max 𝑥 2 𝑦
൝
s. t. 𝑥 2 + 𝑦 2 = 1
o graphical interpretation of 𝑥 2 + 𝑦 2 = 1:
𝑦 ❑ 𝑥2 + 𝑦2 = 1
❑ value for 𝑥: 𝑥𝐴
❑ ∀𝐴, 𝑥𝐴2 + 𝑦𝐴2 = 𝑥𝐴 , 𝑦𝐴 2 = 1
❑ value for 𝑦: 𝑦𝐴 ❑ 𝑥𝐴 , 𝑦𝐴 = 1
2 2 2
❑ 𝑥𝐴 + 𝑦𝐴 = 𝑥𝐴 , 𝑦𝐴 𝑦𝐴 𝐴 ⇔ 𝑂, 𝐴 = 1
❑ 𝑥𝐴 , 𝑦𝐴 = [𝑂, 𝐴] → circle with
𝑥 radius with the
𝑂 𝑥𝐴 size of 1
From Statistics 2 DM –Tutorial F. Muhlenbach 19
Tutorial
• Convex Optimization
o graphical interpretation of 𝑥 2 + 𝑦 2 = 1: circle with radius =1
o 𝑥2 + 𝑦2 = 1 ⟺ 𝑦2 = 1 − 𝑥2
o ⟺ 𝑦 = 1 − 𝑥2 or 𝑦 = − 1 − 𝑥 2
• Convex Optimization
1
o graphical interpretation of 𝑥 2 𝑦: 𝑦 = 𝑧 ×
𝑥2
𝑧=3
𝑧=2
𝑧=1
𝑧=0
𝑧 = −1
𝑧 = −2
• Convex Optimization
1
𝑓 𝑥, 𝑦 = max 𝑥 2 𝑦 ⇒ 𝑦 = 𝑧 ×
o graphical interpretation of ቐ 𝑥2
s. t. 𝑥 2 + 𝑦 2 = 1 ⇒ 𝑦 = ± 1 −𝑥2
o when 𝑓 𝑥, 𝑦 = 𝑧 ≤ 0.3, the
𝑧 = 1.0
constraint intersects with the
𝑧 = 0.4
𝑧 = 0.3 circle, some solutions for the 2
𝑧 = 0.2
𝑧 = 0.1 conditions exists
𝑧=0
o when 𝑓 𝑥, 𝑦 = 𝑧 ≥ 0.4, no
intersection → no solution
o maxima → tangency points
o 𝑧 ? graphically, 𝑧 ∈ 0.3; 0.4
From Statistics 2 DM –Tutorial F. Muhlenbach 22
Tutorial
• Convex Optimization
➢ Solution: we want to solve:
𝑓 𝑥, 𝑦 = max 𝑥 2 𝑦
൝
s. t. 𝑥 2 + 𝑦 2 = 1
➢ constraint: 𝑥 2 + 𝑦 2 = 1 ⟺ 𝑥 2 + 𝑦 2 − 1 = 0
o ℒ 𝑥, 𝑦, 𝜆 = 𝑥 2 𝑦 − 𝜆 𝑥 2 + 𝑦 2 − 1 . FOC:
𝜕ℒ
o = 0 ⇔ 2𝑥. 𝑦 − 𝜆2𝑥 = 0 ⇔ 2𝑥. 𝑦 = 𝜆2𝑥 ⇔ 𝑥 ≠ 0, 𝜆 = 𝑦 1
𝜕𝑥
𝜕ℒ
o = 0 ⇔ 𝑥 2 − 𝜆2𝑦 = 0 ⇔ 𝑥 2 = 𝜆2𝑦 ⇔ 1 ⇒ 𝑥 2 = 2𝑦 2 2
𝜕𝑦
𝜕ℒ
o = 0 ⇔ 𝑥2 + 𝑦2 − 1 = 0 3
𝜕𝜆
• Convex Optimization
o ℒ 𝑥, 𝑦, 𝜆 = 𝑥 2 𝑦 − 𝜆 𝑥 2 + 𝑦 2 − 1
o 3 and 2 : 2𝑦 2 + 𝑦 2 − 1 = 0 ⇔ 3𝑦 2 = 1
1 1
o ⇔ 𝑦2 = ⇒𝑦=± 4
3 3
1 2 2
o 2 and 4 : 𝑥 2 = 2𝑦 2 = 2 × ⇔ 𝑥 2 = ⇒ 𝑥 = ± 5
3 3 3
1
o 1 and 4 : 𝑦 = 𝜆 ⇒ 𝜆 = ±
3
2 1 2 1 2 1 2 1
o solutions: , , ,− , − , and − ,−
3 3 3 3 3 3 3 3
• Convex Optimization
o ℒ 𝑥, 𝑦, 𝜆 = 𝑥 2 𝑦 − 𝜆 𝑥 2 + 𝑦 2 − 1
2 1 2 1 2 1 2 1
o solutions: , , ,− , − , and − ,−
3 3 3 3 3 3 3 3
o → the possible solutions are the four points where the contour lines
are tangent
o which one maximize the function 𝑓 𝑥, 𝑦 = max 𝑥 2 𝑦
o but 𝑦 cannot be negative because 𝑥 2 𝑦 will be negative
2 1 2 1 1 2
o 𝑓 , =𝑓 − , = → 2 solutions
3 3 3 3 3 3
• Convex Optimization
o function to
maximize:
1
𝑧 = 2Τ3 1Τ3
1
→𝑦=𝑧× 2
𝑥
-1
o constraint:
0 1
𝑦 = ± 1 − 𝑥2
o solutions:
2 1 2 1
, , − ,
3 3 3 3