LESSON – CAP 5 BOOK

Recall again the general model for image degradation that we saw last lesson.

~
f (x , y )

Figure 1. A general degradation model.

Now, we assume to have learnt how to remove noise and hence to be able to have an estimation of the
~
image f (x , y ).

Mathematically, we have now the equation

~
f ( x , y )=f ( x , y )∗h ( x , y )
Where we recall that symbol * denotes convolution.

The problem of estimating f(x,y) now is called inverse filtering or deconvolution due to the fact that the
convolution operation is a filtering operation.

Inverse filtering is a procedure difficult to be implemented and it is possible only if we know exactly the
mask of the filter, h(x,y).

If we assume to know it, then by passing to the Fourier domain, we have

G ( w 1 , w 2 )=F ( w1 , w2 ) ∙ H ( w1 , w2 )

And hence we may extract F ( w1 , w2 ) simply by dividing G ( w 1 , w 2 ) by H ( w 1 , w 2 ) as follows:

G (w 1, w 2)
^
F ( w1 , w2 ) =
^ (w 1, w 2)
H

We indicate the Fourier transform for f(x,y) and h(x,y) in red to recall that this is an estimation since it is
impossible to know exactly the Fourier transform of the filter that introduced blurring.

There are some critical issues to discuss before going on:

1. The estimation of ^
H ( w 1 , w 2 ). It depends on the application and there can be several approaches to
estimate it.

2. The division by a quantity in general has problems due to the zeros of the quantities.

Let’s consider a problem at a time.

Estimation of ^
H ( w 1 , w 2 ).

Image blurring may be caused by several phenomena.

- Movement during image acquisition of the camera or of the object.

- Out of focus of some parts of the scene
- Atmospheric turbulence in a transmission channel
- Low exposure time (low number of photons incident)
- Scattering phenomenon

In this section, we provide some approaches for the estimation of ^

H ( w 1 , w 2 ).

1. The first category of approach is called by observation.

In some situations, if we observe the image in regions where there were a clear thin object (i.e., a line or a
white point) whose smoothing effect is evident, then we can estimate the amount of movement, if any, the
direction etc.

Figure 1. An example of estimation by observation

If we look at the extreme of the letter l in word “Digital”, we observe that the point spreads into a segment
oriented at about 45°. The orientation of the green segment along with the length of the segment can be
used to construct a filter called a “motion filter” according to specific mathematical model (Matlab coding).
Hence you can estimate the filter shape in a local region Sxy, and under the assumption that the filter
remains the same (LSI meaning), then you can use that estimation for the whole image.
So, we have

G (w 1, w 2)
^
F ( w1 , w2 ) =
^
H Sxy ( w1 , w2 )

2. Estimation by experimentation
If the equipment responsible for the blurring is available, then you can resimulate the blurring with an
impulse image. See Figure 2 as example.

Figure 2. An example of estimation by experimentation.

Preparing an impulse image (left) and passing it to the acquisition system, we know that the resulting image
is the filter mask translated to the position of the spike (right). By translating again across the origin of the
axis, we obtain exactly the estimation of the filter mask, in space. By passing to the Fourier domain, we
then have the ^ H ( w 1 , w 2 ).

3. Estimation by direct modelling

According to the situation we have two kinds of possible models for ^

H ( w 1 , w 2 ).

a) Atmospheric turbulence (1964)

In this model, we assume

2 2 5/6
^
H ( w 1 , w 2 )=e− K (w 1 +w 2 )

It seems a Gaussian shape but the exponent 5/6 makes it different. The advantage of such model is that it is
never null. Hence, there is no any problems related to the division by ^
H ( w 1 , w 2 ) in the formula. The
parameter K depends on the turbulence nature.

b) Linear uniform motion in a given direction

When the blurring is assumed to be related to movement, and the motion is uniform and on a given
direction, then we may assume that the coordinates of the system moves as follows:

{ x ( t )=a ∙ t/T
y ( t )=b ∙t /T

Where T is the duration of movement, a and b are the speed components of the motion along the two
directions.

Using this formulation, the final estimation of the Fourier transform of the filter is as follows:
 ^ T
sin [ π ( a∙ w 1+ b ∙ w2) ] e
− jπ(a ∙ w +b ∙w )
H ( w 1 , w 2 )= 1 2

π (a ∙ w1 +b ∙ w 2)
It can be noted that the shape of the estimation in the frequency domain is a modulated sinc function. The
formulation provided presents some problems related to the fact that the sin function has periodic zeros in
the spectrum. In particular, according to the values of a and b, zeros are when ( a ∙ w 1+ b ∙ w2 )=k with k any
integer.

Once the filter has been estimated, the procedure needed to extract a possible estimation of f(x,y) is the
so-called inverse filtering.

Inverse filtering

The standard way to proceed is to calculate

G (w 1, w 2)
^
F ( w1 , w2 ) =
^ (w 1, w 2)
H

However, when the filter has zeros in Fourier domain, one possible solution is the so called pseudo-inverse
filtering. In this approach, the algorithm is the following:

{
G (w 1, w 2)
^ , when ¿^H ( w1 , w2 ) ∨¿ 0
F ( w1 , w2 ) ^
H (w 1, w 2)
0 , when H ^ ( w , w )=0
1 2

In this way, the problem of dividing by zeros is solved in place of some artifacts introduced by the non
continuous assignment. Remembering that at the end of the procedure the finally estimated image f(x,y) is
derived by the inverse Fourier transform,
π π
1
f ( x , y )= ∫ ∫ ^F ( w1 , w2 ) e j w x e j w y d w1 d w2
2 π −π −π
1 2

Hence, the artifacts introduced by the non continuous settings propagate in unpredictable manner through
the inverse transform and produce unwanted patterns in the reconstructed image.

Final considerations

We previously saw several methods to reduce the presence of noise before moving to deblurring step.
Anyway, we also illustrate drawbacks and difficulties in obtaining a perfect denoising result. Smoothing
and/or residual noise are always present and should be faced off.

A more general approach able to simultaneously deal with noise and blurring is the so called Wiener filters.

Wiener filters or Minimum mean square error filter

We previously assume a general model that we recall here,

g ( x , y )=f ( x , y )∗h ( x , y ) +η(x , y)

We observed that since η ( x , y ) is a random process, also g ( x , y ) can be consider a random process.
Actually, the same holds for any estimation of the processes involved also of f ( x , y ) since residual noise is
always present in the final estimation.

Let us indicated with ^f ( x , y ) the estimation we are looking for (without noise and without blurring). Then,
we may construct a cost function to be minimized as follows

E ([ F
^ ( w 1 , w 2 )−F ( w1 , w2 ) ] 2)

Where
G ( w 1 , w 2 )−N ( w 1 , w 2 )
F ( w1 , w2 ) =
^
H ( w1 , w2 )

The E ( ∙ ) is called the expected operator in probability theory, but think like it is a mean operator.

By minimizing the cost function of above, we are able to produce an expression for ^
F ( w1 , w2 ) as follows:

^ ⊥
H (w 1, w 2)
^
F ( w1 , w2 ) =
Sη ( w 1 , w 2 )
¿^
2
H ( w 1 , w 2 )∨¿ + G ( w1 , w2 ) ¿
S f ( w1 , w2 )

where the symbol ⊥ denotes the conjugate that is the function with the same real part and imaginary part
with opposite sign,

Sη ( w 1 , w 2 ) indicates the power spectrum of noise (i.e., the Fourier transform of the Autocorrelation
function (for deepening see Papoulis)

S f ( w1 , w2 ) is the square spectrum of the signal, S f ( w1 , w2 ) =¿ F ( w 1 , w2 )∨¿ 2 ¿.

Unfortunately, as seen, the formula seems self-referenced, because of the presence of the term S f ( w1 , w2 )
that is unknown by definition.

S η ( w1 , w2 )
To solve the problem, it is usually considered a term in place of the ratio under certain
S f ( w1 , w2 )
assumptions. If the noise is the so called “white noise” meaning that the spectrum is constant in frequency,
we may substitute Sη ( w 1 , w 2 ) with a constant term K1. In addition, to solve the problem of unknown
S f ( w1 , w2 ) , we may substitute to the entire ratio a constant K2 such that the new formulation for
deconvolution becomes:
^ ⊥ ( w1 , w2 )
H
^
F ( w1 , w2 ) =
¿^
H ( w 1 , w 2 )∨¿ 2+ K 2 G ( w1 , w2 ) ¿

By cycling over various values of K2, we may obtain an optimal result for visual point of view at least
(Matlab coding).
Of course, this is only an approximation since even if noise is white, the same cannot be verified for the
spectrum of the image f(x,y) that cannot be constant too.

Another possibility is to use instead of the constant, the SNR of the image, if available.

A further approach that can be used is the so called regularized filtering

The final formulation for this approach is

^
H ⊥ ( w1 , w2 )
^
F ( w1 , w2 ) = 2
¿^
H ( w 1 , w 2 )∨¿ 2+ γ |P(w1 , w2 )| G ( w 1 , w 2 ) ¿

where P(w1 , w2 ) is the Fourier transform of the Laplacian filter mask (we’ll see later in the course), given
by

| |
0 −1 0
p(x,y)= −1 4 −1
0 −1 0

By using this approach, the method applies a sort of smoothing to the image that reduces the presence of
noise. The parameter γ has to be chosen again by visual evaluation of the result.

Recall that when γ =0 or when K2=0 in the formula above, then the formulation reduces to the inverse
filtering (or pseudo-inverse filtering implementation).

Image quality evaluation metrics

When an image has been deblurred or denoised, we expect to quantify the result by quantitative indicators
rather than visual evaluation.

It is not so easy to extract objective quantitative indicators of image quality that resemble the human visual
assessment result.

According to what we have about the original expected results, we may design different metrics and
procedures for assessment of images.

1) Full reference case. In this situation, we have the original ideal image since we are testing an algorithm
and we artificially add degradation to a given ideal image in order to validate the effectiveness in restoring
it. In this case we have three different metrics:

1
∑ ¿ f ( x , y )− f^ (x , y)∨¿ ¿
2
- Mean Square Error (MSE) =
¿ pixels (x , y)

max ⁡( f ( x , y ) )
- Peak Signal to Noise Ratio (PSNR) = 20 log10
MSE
- Structure SIMilarity (SSIM) it is a sort of extension of MSE but it should be higher for high quality

It is a complex indicator made by three contributions:

α β γ
= (I 1) ∙(I 2 ) ∙( I 3 )
 Where
mf ∙ mf^ + k 1
I 1= is the indicator for average luminance comparison
mf 2 +m^f 2 +k 1

σ f ∙ σ f^ + k 2
I 2= is the indicator for contrast comparison
σ f 2+ σ f^ 2+ k 2

2 σ f ^f +k 3
I 3= is the indicator for the structure comparison thanks to the covariance of the two images and
σ f ∙ σ ^f +k 3
σ f f^ the covariance of the two images is defined as

1
σ f f^ = ∑ (f ( x , y )−¿ mf )( f^ ( x , y )−m^f )¿
¿ pixels (x , y )

α , β , and γ are constants needed for weighting in different way the three contributions, k 1 , k 2 , and k 3 are
constants related to the calculation of the indicators. Usually they are set to 0.1.

2) Semi-reference evaluation. In this case, we do not have the entire reference image f(x,y), but only some
indicators from it, such as the variance (i.e. the expected contrast), the average value (the expected
average luminance), hence we may only compare the average appearance of the two images.

3) Blind reference. We do not know anything about the reference image. In this case, we may conduct a
peer evaluation in which some human candidates are asked to evaluate the quality of some resulting
images. Then, some cooperative technique such as majority voting can be applied to finally decide the best
result. Or as an alternative, the experts are asked to assign a score to each result and at the end we may
collect the so called Mean Opinion Score (MOS), by summing up the scores provided by the experts. At the
end the result with the highest score is the best.

To understand the difficulty in using numerical indicators, see the following images:

(1) (2) (3)

(4) (5)
((b) (6)

Each of us can assign a score from 1 to 5 to each image, according to what kind of alterations is less tedious.
Anyway, the MSE value is often not in agreement with visual result.
Homework. Please assign a score from 1 to 5 to images (1)-(6) (independent each other among you) and
send me in private chat or on skype in this order (they should be 6 values). We’ll collect them and prepare a
MOS for the images.

1 is not good, and 5 is very good,