Organization and

Introduction
Michael Moeller

Chapter 0
Organization and Introduction
About this lecture

Ill-Posed Problems in Image and Signal Processing Examples of Ill-Posed

Inverse Problems
WS 2014/2015 Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Michael Moeller
Optimization and Data Analysis
Department of Mathematics
TU München
updated 22.09.2014
 Organization and
Objectives of the lecture Introduction
Michael Moeller

What is this lecture about?

• Ill-Posed Problems
About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

updated 22.09.2014
 Organization and
Requirements, or “is this something for me?” Introduction
Michael Moeller

Necessary

• Interest in combining mathematical theory with practical

About this lecture
applications Examples of Ill-Posed
Inverse Problems
• Working knowledge about linear algebra, and calculus
Linear Ill-Posed
• Basic knowledge about numerics (Matlab) Problems

Regularization and
Singular Value
Decomposition

Nice to know Tikhonov Type

Regularizations

Error Estimates
• Image processing
Bregman Iteration
• Calculus of variation
• Partial differential equations
• Functional analysis
• Optimization

updated 22.09.2014
 Organization and
Formalities Introduction
Michael Moeller

Exercises
About this lecture
• Homework every week.
Examples of Ill-Posed
Inverse Problems
• Typically, 3 theory exercises, 1 programming problem.
Linear Ill-Posed
• Homework will be turned in for correction in groups of 2. Problems

Regularization and
• Students having more than 80% of the possible homework Singular Value
Decomposition
points will receive a one step bonus on a passing grade. Tikhonov Type
Regularizations
• Due date is Mondays.
Error Estimates
• I’ll grade the homework until Tuesday. Bregman Iteration

• Exercises will be discussed in an exercise group every

weeks, to be held every Tuesday from 2:15pm to 3:45pm
in room 02.10.011.

updated 22.09.2014
 Organization and
Formalities Introduction
Michael Moeller

Examination

• Depending on the number of attendees, the final exam will

be either oral or written. About this lecture

• Requirements: attendance and active participation in the Examples of Ill-Posed

Inverse Problems
exercise group. Linear Ill-Posed
Problems
• ECTS credits: 6
Regularization and
Singular Value
Decomposition

Miscellaneous Tikhonov Type

Regularizations

Error Estimates
• My office: 02.09.061
Bregman Iteration
• Office hours: Tuesday 4:00 - 5:00pm.
• Lecture: Starts at 10:15am. Short break in between.
• http://www-m15.ma.tum.de/Allgemeines/IllposedProblems/
• Tomorrow: Short exercise (hints for the solutions,
answering questions).

updated 22.09.2014
 Organization and
Overview Introduction
Michael Moeller

1 Examples of Ill-Posed Inverse Problems

About this lecture

Examples of Ill-Posed
2 Linear Ill-Posed Problems Inverse Problems

Linear Ill-Posed
Problems

3 Regularization and Singular Value Decomposition Regularization and

Singular Value
Decomposition

Tikhonov Type
Regularizations
4 Tikhonov Type Regularizations
Error Estimates

Bregman Iteration

5 Error Estimates

6 Bregman Iteration

updated 22.09.2014
 Organization and
What are ill-posed problems and why study them? Introduction
Michael Moeller

Inverse Problem
About this lecture

Examples of Ill-Posed
f = Au Inverse Problems

Linear Ill-Posed
Measure data f , linear operator A, desired solution u Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

updated 22.09.2014
 Organization and
What are ill-posed problems and why study them? Introduction
Michael Moeller

Inverse Problem
About this lecture

Examples of Ill-Posed
f = Au Inverse Problems

Linear Ill-Posed
Measure data f , linear operator A, desired solution u Problems

Regularization and
Singular Value
Decomposition
Well-posedness
Tikhonov Type
Regularizations
• A solution exists Error Estimates

• The solution is unique Bregman Iteration

• The solution depends continuously on the data

The majority of practically relevant problems is ill-posed!

updated 22.09.2014
 Organization and
Differentiation Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Data from: Microsoft Research GeoLife GPS Trajectories

Time ’12:44:12’ ’12:44:13’ ’12:44:15’

Latitude 39.974408918 39.974397078 39.973982524
Longitude 116.30352210 116.30352693 116.30362184

How fast did this person go?

updated 22.09.2014
 Organization and
Differentiation Introduction
Michael Moeller

x(ti ) − x(ti−1 )
v (ti ) = ≈ ∂t x(ti )
ti − ti−1

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

updated 22.09.2014
 Organization and
Differentiation Introduction
Michael Moeller

x(ti ) − x(ti−1 )
v (ti ) = ≈ ∂t x(ti )
ti − ti−1

15 About this lecture

Examples of Ill-Posed
Usain Bolt World Record Inverse Problems

Linear Ill-Posed
Problems

Regularization and
10
meters per second

Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates
5
Bregman Iteration

0
0 100 200 300 400 500 600 700 800 900 1000
Measurements

New world record? Top speed of 161.78 km/h?

updated 22.09.2014
 Organization and
Deblurring Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Original image

updated 22.09.2014
 Organization and
Deblurring Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Blurry image f = A ∗ u

updated 22.09.2014
 Organization and
Heat Equation Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Simple Solution: Enhance! Regularization and

Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

updated 22.09.2014
 Organization and
Deblurring Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Blurry image f = A ∗ u ⇒ F(f ) = F(A) · F(u)

updated 22.09.2014
 Organization and
Deblurring Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Reconstructed image u = F −1 (F(f )/F(A))

updated 22.09.2014
 Organization and
Deblurring Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Blurry image f = A ∗ u ⇒ F(f ) = F(A) · F(u)

updated 22.09.2014
 Organization and
Deblurring Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Blurry noisy image f = A ∗ u + n, ⇒ F(f ) ≈ F(A) · F(u)

updated 22.09.2014
 Organization and
Deblurring Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Reconstruction by F −1 (F(f )/F(A))

updated 22.09.2014
 Organization and
Heat Equation Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Example movie Regularization and

Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

updated 22.09.2014
 Organization and
Medical Imaging Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Idea of a CT Scanner

updated 22.09.2014
 Organization and
Medical Imaging Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

updated 22.09.2014
 Organization and
Medical Imaging Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates
CT Reconstruction: Bregman Iteration

• Given: Sinogram (=different line integrals)

• Desired: Underlying image

updated 22.09.2014
 Organization and
Medical Imaging Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Original image
updated 22.09.2014
 Organization and
Medical Imaging Introduction
Michael Moeller

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Naive reconstruction
updated 22.09.2014
 Organization and
Summary of the Examples Introduction
Michael Moeller

• Differentiation: Finding u(x) for given

Z x
u(y )dy About this lecture
0
Examples of Ill-Posed
Inverse Problems
is ill-posed. Linear Ill-Posed
Problems
• Deconvolution: Finding u(x) for given
Regularization and
Z Singular Value
Decomposition
K (x, y )u(y)dy Tikhonov Type
Ω Regularizations

Error Estimates
with smoothing kernel K is ill-posed. Bregman Iteration

• CT reconstruction: Finding u(x) for given

Z
u(x)dx ⊥
~
x·Θ=s

is ill-posed.

updated 22.09.2014
 Organization and
Overview Introduction
Michael Moeller

1 Examples of Ill-Posed Inverse Problems

About this lecture

Examples of Ill-Posed
2 Linear Ill-Posed Problems Inverse Problems

Linear Ill-Posed
Problems

3 Regularization and Singular Value Decomposition Regularization and

Singular Value
Decomposition

Tikhonov Type
Regularizations
4 Tikhonov Type Regularizations
Error Estimates

Bregman Iteration

5 Error Estimates

6 Bregman Iteration

updated 22.09.2014
 Organization and
Ill-posedness, infinite dimensional spaces, inverse crimes Introduction
Michael Moeller

Question
About this lecture
Is the inversion of integral operators ill-posed in general? Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems
Plan:
Regularization and
Singular Value
• Study linear operators via singular value decompositions Decomposition

• Small but nonzero singular values cause problems Tikhonov Type

Regularizations

• Finite dimensions: Ill-conditioned problems Error Estimates

Bregman Iteration
• Infinite dimensions: Ill-posed problems

Find rather general answer to the above question!

updated 22.09.2014
 Organization and
Overview Introduction
Michael Moeller

1 Examples of Ill-Posed Inverse Problems

About this lecture

Examples of Ill-Posed
2 Linear Ill-Posed Problems Inverse Problems

Linear Ill-Posed
Problems

3 Regularization and Singular Value Decomposition Regularization and

Singular Value
Decomposition

Tikhonov Type
Regularizations
4 Tikhonov Type Regularizations
Error Estimates

Bregman Iteration

5 Error Estimates

6 Bregman Iteration

updated 22.09.2014
 Organization and
From the truncated SVD to Tikhonov Regularization Introduction
Michael Moeller

Observation
Small but nonzero singular values cause problems in Au = f .
About this lecture

Examples of Ill-Posed
Question Inverse Problems

Linear Ill-Posed
How can we try to avoid these problems? Problems

Regularization and
Singular Value
Decomposition
Plan:
Tikhonov Type
• Approximate A by Aα such that the singular values are Regularizations

Error Estimates
greater than α.
Bregman Iteration
• Two reasons for errors
• Approximation error k(A−1 − A−1 α )f k
• Data error k(Aα )−1 (f − f δ )k
• Regularization strategy Rα = (Aα )−1
• Regularization parameter α

updated 22.09.2014
 Organization and
Overview Introduction
Michael Moeller

1 Examples of Ill-Posed Inverse Problems

About this lecture

Examples of Ill-Posed
2 Linear Ill-Posed Problems Inverse Problems

Linear Ill-Posed
Problems

3 Regularization and Singular Value Decomposition Regularization and

Singular Value
Decomposition

Tikhonov Type
Regularizations
4 Tikhonov Type Regularizations
Error Estimates

Bregman Iteration

5 Error Estimates

6 Bregman Iteration

updated 22.09.2014
 Organization and
Imposing Prior Knowledge through Regularization Introduction
Michael Moeller
Tikhonov-type regularization
A particular type of regularization strategy is

Rα (f ) = arg min kAu − f k2 + αkuk2 ,

u About this lecture

Examples of Ill-Posed
Inverse Problems
and motivates more general regularizations
Linear Ill-Posed
Problems
2
Rα (f ) = arg min kAu − f k + αJ(u). Regularization and
u Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Blurry noisy image αkuk2

updated 22.09.2014
 Organization and
Imposing Prior Knowledge through Regularization Introduction
Michael Moeller
Tikhonov-type regularization
A particular type of regularization strategy is

Rα (f ) = arg min kAu − f k2 + αkuk2 ,

u About this lecture

Examples of Ill-Posed
and motivates more general regularizations Inverse Problems

Linear Ill-Posed
Problems
2
Rα (f ) = arg min kAu − f k + αJ(u). Regularization and
u Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

R

Blurry noisy image α supp∈C ∞ (Ω) Ω
u ∇ · p dx
0 updated 22.09.2014
 Organization and
Overview Introduction
Michael Moeller

1 Examples of Ill-Posed Inverse Problems

About this lecture

Examples of Ill-Posed
2 Linear Ill-Posed Problems Inverse Problems

Linear Ill-Posed
Problems

3 Regularization and Singular Value Decomposition Regularization and

Singular Value
Decomposition

Tikhonov Type
Regularizations
4 Tikhonov Type Regularizations
Error Estimates

Bregman Iteration

5 Error Estimates

6 Bregman Iteration

updated 22.09.2014
 Organization and
Bregman Distances for Error Estimates Introduction
Michael Moeller

Error estimation About this lecture

Let A be an injective linear operator, and f = Aû. Let f δ be Examples of Ill-Posed

Inverse Problems
such that kf − f δ k = δ. How close is Rα (f δ ) to û? Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Questions: Tikhonov Type

Regularizations
• How do we measure closeness? Error Estimates

• Additional assumptions for error estimates? Bregman Iteration

• Can we adapt our understanding of closeness to the

regularization used?

updated 22.09.2014
 Organization and
Overview Introduction
Michael Moeller

1 Examples of Ill-Posed Inverse Problems

About this lecture

Examples of Ill-Posed
2 Linear Ill-Posed Problems Inverse Problems

Linear Ill-Posed
Problems

3 Regularization and Singular Value Decomposition Regularization and

Singular Value
Decomposition

Tikhonov Type
Regularizations
4 Tikhonov Type Regularizations
Error Estimates

Bregman Iteration

5 Error Estimates

6 Bregman Iteration

updated 22.09.2014
 Organization and
Correcting the Bias of Convex Variational Regularizations Introduction
Michael Moeller

For a given regularization functional J our solution Rα (f δ )

seems somewhat suboptimal:

About this lecture

Examples of Ill-Posed
Inverse Problems

Linear Ill-Posed
Problems

Regularization and
Singular Value
Decomposition

Tikhonov Type
Regularizations

Error Estimates

Bregman Iteration

Total Variation Bregman Iteration

We will learn how to use Bregman iteration to fix this loss of

contrast!

updated 22.09.2014