This article has been accepted for publication in a future issue of this journal, but has not been
fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012
Blind Deblurring and Denoising of Images
Corrupted by Unidirectional Object Motion Blur
and Sensor Noise
Yi Zhang, Member, IEEE
Ford Motor Company
AbstractLow light photography suffers from blur and noise.
In this article, we propose a novel method to recover a dense
estimate of spatially varying blur kernel as well as a denoised
and deblurred image from a single noisy and object motion
blurred image. Proposed method takes advantage of the sparse
representation of double discrete wavelet transform (DDWT)a
generative model of image blur that simplifies the wavelet analysis
of a blurred imageand the Bayesian perspective of modeling
the prior distribution of the latent sharp wavelet coefficient and
the likelihood function that makes the noise handling explicit. We
demonstrate the effectiveness of proposed method on moderate
noise and severely blurred images using simulated and real
camera data.
Index TermsNoisy image blind deblurring, spatially varying
blur, Gaussian scale mixture, double discrete wavelet transform,
sparse representation, statistical model.
I. I NTRODUCTION
Image blur is a phenomenon caused by pixel sensors record-
ing light from multiple sources. If the object represented in the
image is Lambertian, such degradation can be characterized
by a point spread function or the blur kernel. Camera shake,
defocus, and object motion are three common types of blur in
consumer imaging. Blur casused by camera shake is spatially
slowly varying (often approximated by a global blur [1], [2],
[3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15],
[16], [17]), whereas the spatial variation of defocus and object
motion causes sharp blur transitions because they depend on
the depth or the speed of the object, respectively. As such,
the homographic transformations typically used to model the
slow spatial variation of the camera shake (e.g. [18], [19],
[20]) are clearly mismatched for the defocus and the object
motion blurs. This is evidenced by Figure 2(b) where the blur
kernel boundary between the foreground and the background
was missed entirely by the blind image deblurring method.
Hence detection and removal of blur stemming from object
motionthe subject of this paperrequires a fundamentally
different technique, one that have yet to reach its maturity.
With the exception of [21], [22], [23], methods designed to
handle sharp blur kernel boundaries have restricted the blur
shape to a line (for an object motion) [24], [25], [26], [27];
or a line or a circle (for a defocus blur) [28], [29], [30]. The
parameterized blur model constrains the motion blur problem
to the point that the blur kernel boundaries can be recovered
with the help of local image statistics [26], [27], boundary
aware regularization [24], matting [29], dictionary learning
[30] and double discrete wavelet transform [28], [25].
1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
1
Keigo Hirakawa, Senior Member, IEEE
University of Dayton
Another prevalent problem in object motion blur is the
sensor noise. In low light photography, the exposure time is
increased to accumulate more light. Since the degree of blur
scales with the object speed and the exposure time, a real-
world object motion blur is also accompanied by (moderate
to severe) noise. Though existing deblurring methods tolerate
noise to a degree [4], [5], [6], [7], [8], [9], [10], [11], [12],
[13], [14], [15], [16], the prior studies exploring the limits
of deblurring in noise have been limited to the global blur
case [3], [17]. Even a moderate level of noise is enough to
severely deteriorate the image deblurring methods handling
the spatially varying blur cases, as evidenced by a motion
object blur estimation method in [26] failing under noise in
Figure 2(c).
The goal of this article is to remove moderate to severe noise
and to reverse the effects of the (unknown) spatial varying blur
caused by object motion. We accomplish this task by appealing
to the sparse representation of blurry images via the double
discrete wavelet transform (DDWT) we proposed previously
[28], [25]. DDWT is an analytical image representation that
significantly simplifies the wavelet-based generative modeling
of blurred images by decoupling a blur from the signal. Adopt-
ing the Bayesian perspectivethat is, the prior distribution
of the latent image wavelet coefficients are modeledand a
likelihood function that makes the noise handling explicit, we
propose (i) a maximal likelihood estimation of the blur kernel
and (ii) a Bayesian least squares estimate of the latent sharp
denoised image. Specifically, we draw on the work of [31],
[32], [21] to model the distribution of the latent sharp image
wavelet coefficient by the Gaussian scale mixture. The heavy-
tailed prior density of the wavelet coefficients helps resolve the
ambiguities caused by the noise and the blur. At the higher
level, the present problem is simpler than the camera shake
problem in the sense that the blur kernel is parameterized
following the examples of [24], [25], [26], [27], our method
approximates the blur shape by one dimensional linear motion.
But it is more complex than the camera shake deblurring
techniques (e.g. [18], [19], [20]) in the sense that dense esti-
mation of blur kernel tolerates sharp blur kernel boundaries;
and far more robust to noise than the existing object motion
blur detection/removal techniques.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 2

(a) DDWT analysis in (3) (b) DDWT analysis in (4)

Fig. 1: The two processing pipelines above are equivalent. Though (a) is the direct result of applying DDWT (di ? dj ) to the
observed blurry image y, (b) is the interpretation we give to the DDWT coefficients.
Figure 1(b) is the interpretation we give to (3) (remark: a literal
implementation of system in Figure 1(b) is not realizable). The
key is to realize that v ij is the result of convolving a sparse
signal uj with a sparse filter q i , plus the noise ij .
Assume for the moment that we have a horizontal motion
(with a constant velocity during exposure) with the pixel
velocity k. The blur kernel under this circumstance is:
(a) Input (b) Whyte [19] (c) Chakrabarti [26] n 
 
 o
1 0 0
h(n) = k(n) step n + 21 k(n) step n 12 k(n) ,
Fig. 2: Examples illustrating how moderate noise is enough to
severely deteriorate existing image deblurring methods. (5)
where k(n) is the pixel velocity at the location n. When di
denotes a Haar wavelet transform, the DWT response q i of
II. BACKGROUND A ND R ELATED W ORK blur kernel h in (5) and the corresponding DDWT coefficients
A. Double Discrete Wavelet Transform v ij of blurred image in (1) respectively are
Below, we provide a requisite review for over-complete n 
 
 o
0 0
DDWT, a signal decompositional technique we proposed in q i (n) = k(n)
1
n + 12 k(n) n 12 k(n) ,
[28] to sparsify the sharp image and the blur kernel simulta- n 
0
 
0
o
v ij (n) = k(n)
1
uj n + 12 k(n) uj n 12 k(n)



neously. Let x : Z2 R be the latent sharp image. Assuming
Lambertian reflectance surface, the observation y : Z2 R + ij (n). (6)
corrupted by noise and blur is assumed to be given by:
We conclude that v ij is a noisy observation of a difference
y(n) = {x ? hn }(n) + (n), (1) of two DWT coefficients uj displaced by k pixels. Figure 3
where  : Z2 R is the measurement noise, n Z2 is the shows a comparison between DWT (Figure 3(b)) and DDWT
pixel location index and ? denotes a convolution operation. (Figure 3(c)) coefficients of a noisy and blurry image. Based
The point spread function hn : Z2 R denotes a (possibly on the intuition that DWT coefficients capture edge and texture
local) blur kernel acting at the pixel location n. Upper and information and by (6), we expect double edges to appear in
lower case variables in bold font hereafter indicate matrices DDWT on the moving objects of where the distance between
and vectors, respectively. Where understood, the subscript n double edges correspond to the velocity in pixels. Indeed,
is omitted from hn (n). the blur detection is simplified to detecting the displacement
DDWT y 7 v ij is defined as application of an over- of these double edges and textures. In our earlier work, we
complete wavelet filter twice to the observed image: estimated the repeated DWT features using an autocorrelation
function [28].
v ij (n) := {di ? dj ? y}(n), (2)
where di (n) and dj (n) are the ordinary discrete wavelet
transform (DWT) analysis filters in i and jth subbands, respec- B. Arbitrary Motion Direction
tively. By the associativity and the commutativity properties For an arbitrary direction motion, we use image shearing to
of convolution operators, we have skew the image to obtain a horizontal direction DDWT vector.
Let k(n) = r[sin , cos ]T R2 denote a vectored arbitrary
v ij (n) ={di ? dj ? y}(n) (3)
i j i j
motion blur parameterized by a direction [/2, /2) and
={d ? h} ? {d ? x}(n) + {d ? d ? }(n) a length r R+ . Let M R22 denote a shearing matrix
={q i ? uj }(n) + ij (n), (4) of the form:
where uj := dj ? x and q i := di ? h are the overcomplete
" #
1 tan
wavelet transforms of x and h, respectively; and ij :=

, if || /4
0 1


di ? dj ?  is noise in the DDWT domain. The relation between

(3) and (4) is illustrated in Figure 1. The pipeline in Figures M = . (7)
" #
0 1

1(a) and 1(b) are equivalent1(a) is the direct result of


, otherwise
applying the DDWT on the observed blurred noisy image, but
1 cot

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 3

(a) Blurred input image y (b) DWT coefficients wj (c) DDWT coefficients v ij (d) Intermediate blur kernel estimate (e) Final blur kernel estimate

Fig. 3: Example of DWT and DDWT coefficients using data corrupted by noise. The motion blur manifests itself as a double
edge in DDWT, where the distance between the double edges (yellow arrows in (c)) correspond to the speed of the object;
(d) is the coarse dense blur kernel estimation in (20); (e) final blur kernel estimation including graph cut described in Section
III.D.
Define: C. Gaussian Scale Mixture
The distribution of natural images in wavelet domain typ-
vij (n) :={di () ? dj (M ) ? y(M )}(n) ically exhibits non-Gaussian behavior (e.g. tails are heavier
={di () ? dj (M ) ? (h(M ) ? x(M ) + (M ))}(n)than Gaussian). Wainwright et al. proposed Gaussian scale
={di () ? h(M )} ? {dj (M ) ? x(M )}(n) mixture (GSM) [32] to model such behavior. Let u(n)
corresponds
to the latent sharp DWT coefficients within an
+ {di () ? dj (M ) ? (M )}(n) N N spatial neighborhood of n (overlapped, and vec-
={qi ? uj }(n) + ij (n), (8) torized to RN ), a random vector u RN is a Gaussian scale
mixture if it can be written as
where
u|z N (0, z), z p(z), (13)
uj (n) j
:= {d (M ) ? x(M )}(n) = u (M n) j
where z R is a positive scalar random variable distributed
qi (n) := {di () ? h(M )}(n) according to the density function p(z) and denotes the
covariance of u. The prior density p(u) is normal density
ij (n) := {di () ? dj (M ) ? (M )}(n). (9)
p(u|z) marginalized over the hidden parameter z, as follows:
Z
Owing to the fact that h(M n) is a horizontal blur, we may
p(u) = p(u|z)p(z)dz
simplify qi (n) and vij (n) as:
Z exp uT (z)1 u
 
n
o 2
qi (n) = kk(M1 n)k n + 12 k (n) n 12 k (n)


= p(z)dz, (14)
(2)N/2 |z|0.5
uj (n+ 21 k (n))uj (n 21 k (n))
vij (n) = kk(M n)k + ij (n), (10) where the multiplier z is known as hidden variable and control
the contribution of each Gaussian in the mixture. The benefit of
where GSM is that it simplifies non-Gaussianity by marginalization
( of Gaussians over all possible values of z. Portilla et al. [31]
[0 r cos ]T
if || /4 investigates several solutions including maximum likelihood
k (n) := M1 k(M n) = .
[0 r sin ]T
otherwise estimation of z and Jeffreys prior [33] in their denoising work.
(11) In this paper, we follow the examples of Portilla et al. in [31]
Note that when M is a non-integer matrix, it requires to adopt Jefferys prior for p(z).
interpolation. This is problematic for low resolution images,
but the sacrifice is less severe for high resolution images. III. P ROPOSED W ORK
Unshearing vij (n) results in the following
The main drawback of an autocorrelation approach to the
DDWT-based blur kernel estimation in [28] is that it is
v ij (n, ) : = v (M1 n)
sensitive to noise (like most blind deblurring methods). We
uj (n+ 12 k(n))uj (n 12 k(n))
= kk(n)k + ij (n, ), investigate a Bayesian-based approach to the DDWT-based
blur kernel estimation that is more robust to noise.
(12)

where ij (n, ) is the unsheared version of ij (n). Therefore
in the remainder of the paper, we are concerned with recover-
ing uj from v ij , which is the noisy sum of the shifted versions
of uj . For simplicity, the superscripts i and j are omitted from
{uj , q i , v ij , ij , uj , qi , vij , ij }.
A. Noisy Blurred Image Probability Model
In contrast to the discriminative modelingincluding reg-
ularization approaches that are popular in blind and non-
blind deblurring ([7], [8], [9], [10], [11], [12], [13], [14],
[15], [16])DDWT admits a generative modeling of blur that

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 4

(a) log-likelihood when = 10 , kk(n)k = 14 (b) log-likelihood when = 0 , kk(n)k = 0

Fig. 4: Examples of log-likelihood function with and without object motion. (a) is the 2D likelihood function over blur direction
and length when there is motion blur; (b) is the 2D likelihood function when image is sharp.

lends itself naturally to working directly with the posterior Here, we effectively choose the most likely blur length and
densities of the latent sharp images. By DDWT relation direction by searching over the vectored candidate blur ker-
in (12) and the GSM model of (13), the motion blurred nel ` parameterized by its length > 0 and its direction
image yields a pairwise jointly Gaussian distribution between [, ), aimed at estimating the latent ground truth blur
the DDWT coefficients (sheared and unsheared; recall (12)) kernel k(n) = r[sin , cos ]T parameterized by the actual
v n + 12 k(n), and v n 12 k(n), as follows:



  r > 10 and
length
direction  [, ). The likelihood
 v n + 2 `,

u(n) p k(n) = ` is computed by marginaliz-
v n + 21 k(n),



v n 12 `, 

k(n), u (n + k(n))
v n 21 k(n),  ing over all possible hidden variables of (16):
u (n k(n))
v n + 12 `,

     
 
1 u (n + k(n)) u(n) 0 p k(n) = `
N , , v n 12 `, 
kk(n)k u (n k(n)) + u(n) 0
v n + 12 `,
 k(n) = `
Z 
 
(15)
= p
v n 12 `,  z(n, `)
where u(n), v(n, ) ZN denote the DWT and the DDWT R3

coefficients within the N N spatial neighborhood of n, p(z(n, `))dz(n, `). (19)

respectively. The conditional likelihood is given by:
Figure 4 shows
 a typical behavior of the
 log-likelihood
v n + 12 k(n),
 k(n)


v n + 12 `,



function log p k(n) = ` for moving and
v n 12 k(n),  z(n, k) v n 12 `, 
 
0 1
  non-moving locations. The maximum at = 10 and = 14
(z (n + k(n)) + z(n)) z(n)
N , z(n) (z (n k(n)) + z(n)) in Figure 4(a) coincides with the actual object motion speed
0 kk(n)k2
  k(n), verifying that the MLE of blur kernel is reliable. In the
0
+ , (16) absence of motion, another maximum is found at = 0 and
0
some small number = L where L corresponds to the Haar
T
where z (n, k) = [z(n), z (n + k(n)) , z (n k(n))] and window size (L = 2 in Figure 4(b)). This agrees with the
intuition that the DWT coefficients of the spatial support size
E[uuT ] = E[z] . (17)
L cannot be used to detect the blur that is smaller than L. In
In our work, the model parameter was trained from a
practice, we set k(n) to zero when the maximum occurs at
large number of sharp images by dividing out E[z] from the L.
covariance matrix of u. The noise covariance matrix was It is reasonable to expect that the neighboring pixels have
derived from the noise variance estimated via [34]. similar pixel velocity (except at the object boundaries). As
such, we borrow strengths from the neighboring pixels by
B. Blind Motion Blur Deblurring with Marginalized Hidden spatially averaging the log-likelihood function as follows:
Variables
1) Maximum Likelihood Blur Estimation: We propose to

k(n)
estimate the blur kernel stemming from the object motion at
v n0 + 21 `,

 
 
every pixel by leveraging the probabilistic model of (15) and 0
= arg max Bn0 (n) log p k(n ) = ` ,
(16). Specifically, the maximal likelihood estimate (MLE) of sin v n0 12 `, 
`=
the blur kernel k(n) takes the form: cos
(20)
v n + 12 `,


 

k(n) = arg max p k(n) = ` . (18)
sin v n 12 `,  where (n) denotes a neighborhood centered at n and B{} is
`=
cos
a bilateral filter (see appendix A). The proposed algorithm for

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 5

MLE estimate of the blur kernel is summarized in Algorithm

1.

Algorithm 1 Blur kernel estimation with numerical integration

1: Compute single scale vertical subband DDWT coefficients
v by (12).
2: by MLE in (20).
Estimate the blur kernel k
3: Refine blur kernel by (31).

2) Bayesian Least Squares Image Deblurring: Recall

(15),

where the DWT vector u(n) is the vectorization of N N
neighborhood. Our goal of deblurring is to recover u(n) given
the DDWT coefficients vector v(n, ) and the blur kernel
length k(n). The Bayesian least squares estimate (BLS) is
the posterior mean:
 
 v n + 1 k(n),

  Fig. 5: Examples of posterior distribution
E u(n) 2 , k(n)
v n 12 k(n),



 v n + 1 k(n),
Z 
 
2 k(n)
= E u(n) , Bayes theorem in the usual way
v n 21 k(n),


R3 z(n, k)

 v n + 1 k(n),

 

 v n + 1 k(n),

 
p z(n, k)  2 , k(n) dz(n, k). (21)
v n 21 k(n),

2
p z(n, k) , k(n)
v n 12 k(n),

Equation (21) suggeststhat the posterior mean
is the condi- v n + 12 k(n),
 k(n)

 
 v n + 1 k(n),
 p 1 p(z(n, k))
k(n) v n 2 k(n), z(n, k)

tional BLS estimate E u(n) 2 , = R  . (25)
v n 21 k(n),


v n + 12 k(n),
 k(n)

 
z(n, k) p p(z(n, k))dz(n, k)
R3 1
marginalized
 over the posterior mixture parameter v n 2 k(n), z(n, k)

 v n + 1 k(n),


z(n, k) 2
, k(n) .
v n 21 k(n), We numerically integrate over z(n, k) in (21) and (25). The
Thanks to GSM, the latent local DWT vector u(n)|z(n) is final deblurred and denoised image is recovered by the inverse
conditionally normal. Assuming the noise is additive Gaussian wavelet transform of the wavelet coefficients estimated by
in (15), conditional BLS estimation is a linear minimum mean (21). The proposed algorithm for Bayesian least squares image
square error (LMMSE) estimate: deblurring is summarized in Algorithm 2.

 v n + 1 k(n),

 
2 k(n)
E u(n) , Algorithm 2 Non-blind deblurring with numerical integration
v n 21 k(n),


z(n, k)
1
Compute multi-scale DDWT coefficients v ij via (12).


1 v n + 2 k(n),
1:
=uv|k,z (v|k,z ) , (22)
v n 21 k(n), 2: for each subband j do
3: j from DDWT vector
Reconstruct sharp DWT vector u
where ij
v using (21).
"
T  #
v n + 12 k(n),
 k(n) end for

4:
uv|k,z = E u(n) Inverse DWT to recover sharp image x .
v n 21 k(n), z(n, k) 5:


z(n)  
= , (23)
kk(n)k
and
C. Speed Optimization
v|k,z
"
T  #
The marginalizations over the hidden variables used in (19),
v n + 12 k(n),
v n + 12 k(n),
 k(n)

 
=E (21) and (25) involve a triple integration that is computation-
v n 12 k(n), v n 21 k(n), z(n, k)

ally expensive. It stems from the fact that the (conditional)
1
 
(z(n + k(n)) + z(n)) z(n) probability density function on z cannot be circumvented in
=
kk(n)k2 z(n) (z(n k(n)) + z(n)) (19), (21) and (25). In order to speed up the inference, we
 
0 propose a simplification aimed at replacing integral with fast
+ . (24)
0 searches.
The  posterior
 mixture parameter 1) Blur Estimation by Joint Maximum Likelihood (ML) and
 v n + 1 k(n), Maximum a-Posteriori (MAP) Estimation: As an alternative to
z(n, k) 2
, k(n) can be handled by
v n 21 k(n), integration in (18), we propose a joint ML estimation of k(n)

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 6

and MAP estimation of z(n, k) of the form:

(z(n, k) z(n, k)). This is justifiable when the posterior
distribution of z is sparse. Indeed, Figure 5 shows an example
v n + 21 `,


  

arg max p , z  k(n) = `
 of the posterior distribution of real data. Unimodal and narrow

sin
 v n 12 `,
z,`=
cos
peak distribution guarantees that the approximation in (29) is
 1

  valid. When the posterior of z is concentrated near zero, the
v n + 2 `,
 k(n) = `
= arg max p 1 p(z|k(n) = `). posterior mode is especially peaky.

sin
 v n 2 `,  z
z,`=
cos
In terms of complexity, (29) is computed by plugging in
(26)
z(n, k) (already computed in (27)) into (22). By comparison,
implementation of (21) requires a numerical integration where
The above equation is equivalent to: (22) is computed repeatedly for different values of z(n, k).

k(n) In fact, the numerical integration over z(n, k) must be over
  1

   a dense grid, especially when the posterior density of z(n, k)
v n + 2 `,
 k(n) = `
= arg max arg max p p(z) has a narrow peak. The proposed algorithm for image deblur-
z v n 12 `,  z
`=
sin
cos
ring with plugin estimate of hidden variable is summarized in
Algorithm 4.
v n + 12 `,
 k(n) = `

 
= arg max p , (27)
sin v n 12 `,  z = z(n, `) Algorithm 4 Non-blind deblurring with Laplacian approxima-
`=
cos
tion
where we used the fact that p(z|k(n) = `) = p(z) is expected 1: Compute multi-scale DDWT coefficients v ij via (12).
to hold, and 2: for each subband j do
1 Estimate hidden variable z by MAP estimation (28)

 
v n + 2 `,
 k(n) = `
 3:

z(n, `) = arg max p 1 p(z), via EM algorithm (30).
z v n 2 `,  z
(28) 4: Reconstruct sharp DWT vector u j from DDWT vector
ij
is a MAP estimation of z with k(n) = ` given. The compu- v using (29) with estimated z.
tation of z(n, `) can be implemented efficiently by Newtons 5: end for
method [35] or by Expectation-Maximization (EM) algorithm 6: Inverse DWT to recover sharp image x .
[36] which converges after only a few iterationssee III.C.3.
The accumulation of local blur evidence by using bilateral 3) Maximum a-Posterior Estimation of Hidden Variables:
filter remain the same as described in section III.B.1. The Revised blind deblurring of (27) and (29) avoids the triple
proposed algorithm for joint MLE-MAP blur kernel estimation integration. To accomplish this, the MAP estimation of hid-
is summarized in Algorithm 3. den variables z(n, k) must be computationally efficient. To
this end, we propose to use Expectation-Maximization (EM)
Algorithm 3 Blur kernel estimation with joint MLE-MAP algorithm [36] to iteratively solve for MAP estimation on
1: Compute single scale vertical subband DDWT coefficients hidden variables. Such iteration alternates between E-step
v by (12). the expectation of posterior density; and M-stepfind hidden
2: Estimate hidden variable z by MAP estimation (28) via variables z that maximize the posterior. Combined, the EM
EM algorithm (30). iteration takes the form:
3: Estimate blur kernel k by joint MLE-MAP in (27) using
p+1
estimated z. zm

4: Refine blur kernel by (31). log |z|
= arg max E log p(z)
z 2

 v n + 1 `,


2) Image Deblurring by Posterior Sparsity of Hidden Vari- 2
uTm 1 um  v n 12 `,


able: Suppose we approximate (21) by replacing the triple
, m = 0, 1, 2, (30)
integration with a plugin estimate of z, as follows: 2z 
 k(n) = `
p
  1
 v n + k(n),

   z = z (n, `)
E u(n) 2
, k(n)
v n 12 k(n), where z0p , z1p , z2p denote the MAP estimation of z after p

 v n + 1 k(n),

 
iterations at locations (n), (n + `), (n `), respectively; and
Z
= E u(n) 2
1

, k(n)
R 3 v n 2 k(n), z(n, k) u 0 , u1 , u2 denote u(n), u(n+`), u(n`), respectively. (See
 
 v n + 1 k(n),

  Appendix B for derivation.) Notice that three hidden variables
p z(n, k) 2
, k(n) dz(n, k)
v n 21 k(n), z p (n) and z p (n`) can be updated separately in each iteration
 
 v n + 1 k(n),

  which further improves the speed.
k(n)
E u(n) 2 , , (29)
v n 12 k(n),



z(n, k)
D. Regularizing Blur Estimation Map
where
z(n, k) is the MAP estimate of z
as in (27). This approximation is equivalent The blur kernel estimation in III.B.1 and III.C.1 can be
 v n + 1 k(n), further improved by post-processing. Following examples of


to p z(n, k) 2
, k(n) Chakrabarti [37] and Levin [38], we smooth the blur kernel
v n 21 k(n),

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012
estimation map by graph cut [39] based on Markov random
field (MRF) over the entire image. Such optimization process
searches the estimation map that minimizes following energy:
X X
E(k) = E1 (k(n)) + E2 (k(n), k(n0 )). (31)
n n,n0
Here E1 enforces output blur map similar to (18) or (27). We
choose the binary cost:
( various types of details (e.g. line, curve, different shapes).
0 if k(n) = k(n)
E1 = . (32)
1 otherwise
On the other hand, E2 encourages discontinuities to align with
image edges. Such cost is defined by:
(
0 , k(n) = k(n0 )
E2 = 2 2 , (33)
ekn k / , otherwise
where is set to be four times of the median value of kn k2 ,
and kn k2 denotes the image gradient at n [37].
IV. E XPERIMENTS
We evaluate our proposed method using simulated and real
camera data degraded by blur and noise. Experiments are
implemented in MATLAB and executed on PC using Intel(R)
Core(TM) i7-2600 CPU @ 3.4GHz with 16GB memory.
Below we describe the simulated and real-image experiments.
A few representative output images are shown in Figures 8
to 10; many more examples can be found in the supplementary
material.
We compared our proposed method to state-of-art blind
image deblurring algorithms of [6], [15], [19], [3], [4], [5],
[13], [16], [17], [28]. We also compared the motion blur kernel
estimation/detection to the methods in [26], [28], [25] and
[22]. We used parameter values recommended by each authors.
Since there are no alternatives for handling object motion blur
in presence of noise, we acknowledge the limitation to our
studyexisting methods use parametric blur kernels to handle
blur kernel boundaries (typical for object motion and defocus)
but without considerations for severe noise [28], [25], [26];
use nonparametric blur kernels but do not handle sharp blur
kernel boundaries (typical for camera shakes) [19]; and/or
are designed for heavy noise, but without consideration for
spatial variations of blur [17], [3]. In the real-image results
below, the global deblurring methods of [40], [41], [17], [4],
[3] were applied only to the blurred regions of the images,
as determined by our MLE blur kernel estimation method in
Algorithm 3 (See Section IV.B for details).
We set GSM neighborhood N = 3 and N = 1 in
Algorithms 1, 2 and Algorithms 3, 4, respectively. The range of
numerical integration in Algorithms 1 and 2 is truncated within
z [exp(20.5), exp(3.5)]3 . We use the single scale and ver-
tical highpass subband (recall blur estimation is evaluated by
image shearing) of the wavelet coefficients for blur estimation.
Specifically, we use the third level vertical subband of DWT
coefficients in the cases of noise variances equal to 625 in
synthetic experiment (Figure 6), as well as real image example
(Figures 8(b,c)). We use the second level vertical subband for
the rest of the experiment. For image deblurring, we recover
1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
7
sharp DWT coefficients at each subband, and final sharp image
is reconstructed by taking inverse DWT of recovered DWT
coefficients.
A. Simulation Study
We simulated a blurred noisy camera data using 50 8-
bit images from Berkeley segmentation dataset [42] with
The synthetic blurry and noisy images are created by applying
global translational blur of various lengths in the horizontal
direction as well as additive white Gaussian noise. We quan-
tify the overall performance (blur estimation, deblurring, and
denoising) by comparing the average peak signal-to-noise ratio
(PSNR) of recovered images. Figure 6 summarizes the PSNR
scores of blind deblurring performance. Overall, the proposed
work clearly outperformed the alternatives when the noise
variances are 25, 100 and 625. For this study, our methods
were implemented to estimate the global blurthat is, we used
(20) and (27) to find the global maximizer of the likelihood
function. Among our methods, the full integration versions
as described by Algorithms 1 and 2 performed better than
the accelerated version of Algorithms 3 and 4, though the
differences were small. The performance of [4] for a low noise
case (variance = 9) were comparable. PSNRs of [13] when
noise variance = 625 are intentionally left out due to the fact
that the code provided by the authors caused error when the
noise variance was high.
We also compare the deblurring techniques of Algorithms
2 and 4 to the non-blind deblurring techniques of [6], [3],
[4], [41], [13], [17], [28]. In Figure 7, the ground truth blur
kernels were provided. Proposed non-blind deblurring methods
are comparable to state-of-art methods ([41], [17]) when noise
variance = 100 and 625.
B. Real Data
The real data experiment is conducted on raw sensor
data captured by Nikon D90 indoor (low light) or outdoor
(sunset/sunrise in cloudy weather) with high ISO and small
aperture size. All input images suffer from noise and uni-
form or non-uniform translational blur (in various directions)
stemming from object motion. The raw data was first treated
with demosaicking method of [43]; then pixel neighborhood
of 22 was averaged and the image was downsampled by 2 to
ensure that the demosaicking artifacts has minimal impact on
deblurring process. We trained the covariance matrix E[uuT ]
of the sharp DWT coefficients in (17) for each scale and
subband based on 112 sharp images captured by the same
camera and lens, and downsampled in the same manner. We
report the noise standard deviation estimated by [34] in the
caption of each figure. We are aware that such estimation is not
accurate for the true image noise model since the distribution
of the image sensor noise is better characterized as Poisson
[44], though it is still a good proxy for the level of noise
we see in each image. We begin the real data experiments
by considering an image with a global object motion and
noise. We captured an image of a color picture mounted on a
Yaskawa Motoman HP3 robot. Together with the motion blur
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 8

(a) Noise variance = 9 (b) Noise variance = 25

(c) Noise variance = 100 (d) Noise variance = 625

Fig. 6: Blind deblurring performance on images synthetically corrupted by noise and global blur.

speed and direction that can be controlled with high precision
and the relative camera coordinates, we have the ground truth
motion data for these images. This experiment ensures that
blind deblurring methods in [3], [17], [4], [5], [13] and a
combination of [40] and [41] designed for global blur can
be tested without additional measures.
Figure 8 show the results of processing uniformly blurred
image (more global blur images in the supplementary mate-
rial). In this example, noise is clearly visible. In Figure 8(e,f,h),
the alternative blind deblurring methods fail at the blur kernel
estimation stage. We found that our previous method (based on
autocorrelation function of DDWT) yielded satisfactory blur
estimation, but deblurring was low quality (Figure 8(d)). Meth-
ods in [17], [5], [13] yield acceptable blur kernel estimations
but deblurring results are not satisfactory (Figure 8(g,i,j)).
The proposed methods (implemented with (20) and (27)
to find the global maximizer of the likelihood function) also
yielded satisfactory blur kernel estimation and image deblur-
ring resultsconsistent with our findings in Figures 6 and
7. As was also suggested by Figures 6 and 7, the integral
method as described by Algorithms 1 and 2 yielded better
results than the approximated method by Algorithms 3 and
4, though the latter was far faster in execution time. The
blur kernel estimation in Algorithm 1 takes 765 seconds to
process color image of size 500x500, while the accelerated
method in Algorithm 3 takes 393 seconds. Note however that
the bilateral filtering both methods takes 385 secondsone
can expect significant speed up from bilateral filtering itself
by considering fast implementations, such as that of [45].
Deblurring methods in Algorithms 2 and 4 take 228 and 9.5
seconds, respectively.
In a more likely scenario, Figure 9(a) shows an example
of moving foreground against a stationary background; and
Figure 10(a) shows a stationary foreground against moving
background (more examples in the supplementary material).
Subfigures (m,n) in Figures 9 and 10 show the recovered
object speed, direction, and sharp blur kernel boundaries using
the proposed Algorithms 1 and 3, respectively. Contrast this
to the blur kernel estimation techniques in [28], [25], [26]
and the blur detection method in [22] that reported promising
results in low noise case. These algorithms prove less reliable
for identifying object motion blur in presence of more noise,
however, as Figure 9(o-r) and Figure 10(o-r) have shown. As
evidenced by the output images in Figures 9(b,c) and Figures
10(b,c), the proposed deblurring methods in Algorithms 2
and 4 yield satisfactory results, largely free of instabilities
caused by noise and ringing artifacts. For the most part, the
edge details were recovered correctly, albeit somewhat softer
in appearance. The output from our previous DDWT-based
method in [28] yielded considerably softer edges, as well as
the noise artifacts (Figure 9(d) and Figure 10(d)).
We also compared our results to the deblurring methods

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 9

(a) Noise variance = 9 (b) Noise variance = 25

(c) Noise variance = 100 (d) Noise variance = 625

Fig. 7: Non-blind deblurring performance with ground truth blur kernel on images synthetically corrupted by noise and global
blur.

in [3], [40], [41], [17], [4]. Recall that these methods cannot
handle sharp blur boundaries (and indeed, they will fail for
object motion blur; recall Figure 2). Thus we first isolated
the blurred regions of the image by motion matte, computed
by convolving a binary mask of moving pixels in Figures
9(n) and 10(n) by the estimated blur kernel (one may also
consider techniques such as [46] and [47]). In Figures 9(e-
h) and 10(e-h), blur kernel and the latent sharp image were
estimated by the blind image deblurring methods in [3],
[40]+[41], [17] and [4]. In Figures 9(i-l) and 10(i-l), the latent
sharp image was estimated by non-blind image deblurring
techniques of [3], [41], [17], [4] using the blur kernel estimated
in Figures 9(n) and 10(n), respectively. Despite the relatively
good performance in simulated results we reported in Figure
7, the results showed sensitivities to noise (Figures 9(i, e, h),
9(e, f)), susceptibility to ringing artifacts (Figures 9(j), 10(f, i,
j, k, l)), and failure to recover edges (Figures 9(e, g, h), 10(e,
g, h, i, k ,l)). Overall, deblurring results of [3], [4] worked
very well on some images and failed badly on others. The
combination of matting based Algorithm 3 and [17] performed
more consistently.
C. Discussion
There are four main criteria for evaluating the quality of re-
covering sharp image from images corrupted by object motion
blur: (i) recovery of sharp blur kernel transition, (ii) dense blur
kernel estimation, (iii) deblurring with blur kernel boundaries,
and (iv) robustness to noise. Most existing methods are not
designed for sharp blur kernel boundaries and noise. Since
we tend to use long exposure and high gain to overcome
noise in low light photography, deblurring techniques for blur
caused by object motion must be robust to noise in order to
be pragmatic and useful for real scenarios.
We conclude from Figures 9-10(m,n) (as well as the sup-
plemental material) that our previous methods of [28] and
[25] as well as the techniques of [26] and [22] cannot extract
blur boundaries accurately, even though these methods have
been reported to do better when the noise is less severe. By
comparison, the proposed Algorithms 1 and 3 successfully
recover the object motion with sharp blur kernel transitions in
presence of noise. The proposed methods have benefited from
the likelihood function that makes noise handling explicit.
DDWT also plays a critical rolethe bivariate likelihood func-
tion in (18) is computationally/analytically tractable thanks to
DDWT that transforms the blurred DWT coefficients into only
two significant DDWT coefficients.
Blur kernel estimation and deblurring of images corrupted
by noise and blur caused by object motion pose unique
challenges for techniques not specifically designed for such
purpose. Majority of prior work in image deblurring are

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 10

designed to handle spatially uniform blur with low noise. Our V. C ONCLUSION
experiments confirm that there is no straightforward way to
repurpose existing deblurring method for the object motion
We extended the the DDWT deblurring of [28] to tolerate
problem. Specifically, by isolating the blurred object from
sensor noise and sharp blur boundaries common in object
otherwise stationary scene using motion matting, we can
motion blurs. In order to consider the effect of noise, we
effectively treat the scene as a piecewise-uniform blur with
proposed maximum likelihood blur kernel estimation and
noise. However, experiments with real image data confirm
Bayesian least squares deconvolution using Gaussian scale
that deblurring results in instabilities such as ringing artifacts
mixture modeling. We accelerate algorithm by incorporating
that are often severe, often seen near the sharp blur kernel
expectation maximization algorithm. We demonstrated that
boundaries. Noise amplification was evident in the presence
GSM modeling combined with explicit noise modeling pro-
of moderate to heavy noise, also.
vides a significant advantage in realistic image deblurring
Compared to these alternatives, the proposed deblurring
scenarios for object motion. Our future research plan includes
technique handles the blur kernel transitions better. Specif-
blur estimation by maximum a posteriori and minimum mean
ically, the proposed DDWT-GSM algorithm was the only
square estimator using prior model on blur kernel size and
method consistently and reliably outputting deblurred images
direction.
without noticeable ringing artifacts. The absence of ringing
can be explained by the fact that each DDWT coefficients
influences the recovery of only two DWT coefficients (contrast
this to the conventional deblurring approach where the recon- A PPENDIX A
struction of each pixel depends on a very large group of pixels M ODIFIED B ILATERAL FILTER
in its neighborhood). While there are instances where certain
image features were recovered by the existing methods more
convincingly than ours, our algorithm significantly improved Bilateral filter is exploited to carryout the local smoothness
stability and overall output image quality. of log-likelihood. For notational simplicity, we define
On the other hand, deblurring results with modest/severe
noise are obviously not as high quality compared to the v n + 12 `,


 
low noise deblurring case (obviously). GSM-based image p (V (n, `, )) := p k(n) = ` . (34)
v n 12 `, 
reconstruction (such as ours) are known to yield somewhat
softer appearance, owing to the shrinkage stemming from
The bilateral filtering can be discribed by:
coring function. As suggested by Figure 7, for example, we
may consider in the future a hybridization between the blur
kernel estimation of Algorithm 1 (or 3) and existing state-of- Bn0 (n) {log p (V (n0 , `, ))}
art image deblurring techniques in the case that noise is less 0 0
P
n0 (n) W (n ) log p (V (n , `, ))
significant. = P 0
, (35)
There are further design considerations for Algorithm 1 and n0 (n) W (n )

3. DDWT in (6) and the corresponding likelihood function

in (15) remain valid for all subbands of DWT, regardless of where (n) defines the local neighborhood, n is the center
scale, orientation, or the type of wavelet transform. In practice, pixel location, and W (n0 ) denotes the bilateral weight:
coarse scale wavelet coefficients are more robust to noise,
making it more desirable for our intended application. On
W (n0 )
the other hand, the fine scale wavelet coefficients are sparser, !
2
kn n0 k2 kLab(n0 ) Lab(n)k
 
which in turn helps the blur kernel estimation. The trade off
= exp exp ,
has not been fully explored yet. d2 2r2
Finally, the proposed method is not without limitations. (36)
The blur kernel model is highly restrictivethe unidirec-
tional/linear shift blur kernel may be parameterized by the
blur direction and length. Although this simplification is not where d2 and r2 are supported spatial and intensity variance.
applicable to other common causes of blur (defocus and Here, Lab transforms input color image into CIE-Lab space.
camera shake), the parameterization was appropriate and nec-
essary for enabling a dense estimation of blur kernel that
tolerate sharp blur kernel boundaries. Moreover, it is by this
A PPENDIX B
parameterization that DDWT achieves sparsification of blurred
M AXIMUM A -P OSTERIOR E STIMATION OF H IDDEN
DWT coefficients, which subsequently prevent the ringing
VARIABLE USING EM-A LGORITHM
artifacts from emerging. In short, the proposed deblurring
techniques in Algorithms 2 and 4 are simpler than [41] and
[17] in the sense that the former only handle linear motion; Instead of finding closed form solution that can be in-
more complex in the sense that the latter only handle global tractable, we use EM algorithm to iteratively seek for MAP
blur with no sharp blur kernel boundaries. estimation of hidden variables.

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 11

(a) (b) (c) (d) (e)

(f) (g) (h) (i) (j)

Fig. 8: Example of uniform blur image with noise standard deviation equals to 0.076 (the maximum pixel value is scaled to
one) estimated by [34]. (a) Blurry and noisy image with ground truth blur kernel from robot motion; (b) Latent clean image;
(c) Algorithm 1 + 2; (d) Algorithm 3 + 4; (e) Molina [3]; (f) Sun [40] + Zoran [41]; (g) Zhong [17]; (h) Perrone [4]; (i)
Michaeli [5]; (j) Shan [13].

Following (28), we have: under current estimation of z (i.e. z p ):

z(n, `) Q(z, z p (n, `))

v n + 21 `,
 k(n) = `

 
u(n) 
= arg max p p(z) 
z v n 12 `,  z =E log p(z) + log p
u (n + `) z


u(n)
u (n `) 
1
Z 

v n + 2 `, 
 v n + 1 `,

 
= arg max p(z) p
, u (n + `) k(n) = `
z v n 12 `,  2
 v n 1 `, , .
z = z p (n, `)


R3N u (n `) 2


k(n) = `
 u(n) (39)
 d u (n + `)
 z
u (n `) M-step: Choose z that maximizes Q function:

Z  1

 u(n)
v n + 2 `,

z p+1 (n, `) = arg max Q(z, z p (n, `)).

= arg max p(z) p u (n + `) (40)
z R3N v n 12 `,  z
u (n `)

 u(n)  u(n)
u(n)

k(n) = `  
, p u (n + `)z d u (n + `) , (37)

z Notice that p u (n + `) z is normal and each z can be

u (n `)  u (n `) u (n `) 
optimized separately. Hence, for m = 0, 1, 2:
where

p+1 log |z|
= arg max E log p(z)

zm

 1

 u(n)
v n+ 2 `,
u (n + `) , k(n) = `
 z 2
p 1 
 v n + 1 `,
n


v 2 `,  u (n `)
 z 2
uTm 1 um  v n 12 `,



 . (41)

 1

 u(n) k(n) = `
v n+ 2 `, 

u (n + `) , k(n) = ` 2z 
=p 1 (38) 
 z = z p (n, `)
v n 2 `,  u (n `)


 v n + 1 `,


is not a function of z and is zero when taking derivative with  2
 v n 1 `,


respect to z. Also notice that um  2 has the conditional
k(n) = `
E-step: Compute the expected value of log-posterior

 z = z p (n, `)

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 12

(a) (b) (c) (d)

(e) (f) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p) (q) (r)

Fig. 9: Example of spatially varying blur image with noise standard deviation equals to 0.022 (maximum pixel value is scaled
to one) estimated by [34]. (a) Input; (b) Algorithm 1 + 2; (c) Algorithm 3 + 4; (d) Zhang [28]; (e) Molina [3] + matting based
Algorithm 3; (f) Sun [40] + Zoran [41] + matting based Algorithm 3; (g) Zhong [17] + matting based Algorithm 3; (h) Perrone
[4] + matting based Algorithm 3; (i) Molina [3] + Algorithm 3; (j) Zoran [41] + Algorithm 3; (k) Zhong [17] + Algorithm 3;
(l) Perrone [4] + Algorithm 3; (m) Algorithm 1; (n) Algorithm 3; (o) Zhang [28]; (p) Zhang [25]; (q) Chakrabarti [26]; (r) Shi
[22], a blur confidence map where bright pixels represent small chance to miss true positive and dark pixels represent small
false alarm. Estimations m to q are HSV color coded where hue represents the direction, saturation represents blur kernel
length and brightness is set to the constant.

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 13

(a) (b) (c) (d)

(e) (f) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p) (q) (r)

Fig. 10: Example of spatially varying blur image with noise standard deviation equals to 0.010 (maximum pixel value is scaled
to one) estimated by [34]. (a) Input; (b) Algorithm 1 + 2; (c) Algorithm 3 + 4; (d) Zhang [28]; (e) Molina [3] + matting based
Algorithm 3; (f) Sun [40] + Zoran [41] + matting based Algorithm 3; (g) Zhong [17] + matting based Algorithm 3; (h) Perrone
[4] + matting based Algorithm 3; (i) Molina [3] + Algorithm 3; (j) Zoran [41] + Algorithm 3; (k) Zhong [17] + Algorithm 3;
(l) Perrone [4] + Algorithm 3; (m) Algorithm 1; (n) Algorithm 3; (o) Zhang [28]; (p) Zhang [25]; (q) Chakrabarti [26]; (r) Shi
[22], a blur confidence map where bright pixels represent small chance to miss true positive and dark pixels represent small
false alarm. Estimations m to q are HSV color coded where hue represents the direction, saturation represents blur kernel
length and brightness is set to the constant.

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 14

mean: Hence

 v n + 1 `,



 v n + 1 `,

 2
uTm (z)1 um  v n 1 `,


 2
 v n 1 `,

2
E 
m
=E u m
 2 2  k(n) = `

 k(n) = ` 
 z = z p (n, `)
 z = z p (n, `) XX
m ]ij [1 ]ij +
[ Tm 1
m
v n + 12 `,



1
=um v|`,zp v|`,zp (42) i j
v n 12 `, = (49)
2z
and conditional covariance To solve (41), we take derivatives of each term with respect
to z:

 v n + 1 `,


m
2
uTm (z)1 um  v n 12 `,



 v n + 1 `,


2 E
 1

z 2 k(n) = `
T  v n 2 `,
 
=E (um m )(um m ) 

 z = z p (n, `)
k(n) = `
P P 1 Tm 1

 z = z p (n, `) i j [m ]ij [ ]ij + m
= (50)
= um v|`,zp 1
v|`,z p vum |`,z .
p (43) 2z

2
1


 v n + `,
 2
 v n 1 `,


log|z| = N
In (42) and (43), conditional cross correlation matrix take the E
 2 (51)
form of following: z 2   k(n) = ` 2z
p
 z = z (n, `)

 v n + 1 `,


z0p (n) 2
u0 v|`,zp = [, ] (44)

1  v n 12 `,


k`k = 1
E log 
(52)
z z k(n) = ` z
z p (n)

u1 v|`,zp = 1 [, 0] (45)  z = z p (n, `)
k`k

z p (n) where the local spatial neighborhood has a size of N N .
u2 v|`,zp = 2 [0, ]. (46)
k`k The final solution is obtained by taking summation of (50),
(51) and (52) and set it to zero.
Notice that vum |`,zp = Tum v|`,zp , where m = 0, 1, 2, and P P [ ] [1 ] + T 1
i j 0 ij ij 0 0

N +2
P P
v|`,zp
j [1 ]ij [
1
T
]ij +1
1
1

z p+1 (n, `) =

"
T  # i (53)
N +2
v n + 21 `,
v n + 12 `,
 k(n) = `

 

=E
v n 12 `, v n 12 `, z = z p (n, `)
 P P
1 1
i j [2 ]ij [ T
]ij +2 2

N +2
z p (n + `) + z p (n) z p (n)
 
=
k`k2 z p (n) z p (n `) + z p (n) ACKNOWLEDGMENT
 
0
+ . (47) We would like to thank the authors of [6], [15], [19], [3],
0 [4], [5], [13], [16], [17], [26], [22] for providing the code that
were used in our experiments. We also thank authors of [42]
In (41), for making the image dataset available.

 v n + 1 `,


 2 R EFERENCES
T 1  v n 1 `,


E um um  2
[1] F. Xue and T. Blu, A novel sure-based criterion for parametric psf

k(n) = `
 estimation, Image Processing, IEEE Transactions on, vol. 24, no. 2,
 z = z p (n, `) pp. 595607, 2015.
[2] J. P. Oliveira, M. Figueiredo, J. M. Bioucas-Dias et al., Parametric

 v n + 1 `,


 2 blur estimation for blind restoration of natural images: Linear motion
 v n 1 `,


T 1 and out-of-focus, Image Processing, IEEE Transactions on, vol. 23,

= E (um
m ) (um m )
 2
k(n) = ` no. 1, pp. 466477, 2014.
[3] R. Molina, J. Mateos, and A. K. Katsaggelos, Blind deconvolution

 z = z p (n, `)
using a variational approach to parameter, image, and blur estimation,
+ Tm 1
m Image Processing, IEEE Transactions on, vol. 15, no. 12, pp. 3715
XX 3727, 2006.
= m ]ij [1 ]ij +
[ Tm 1
m (48) [4] D. Perrone and P. Favaro, Total variation blind deconvolution: The devil
i j is in the details, in Computer Vision and Pattern Recognition (CVPR),
2014 IEEE Conference on. IEEE, 2014, pp. 29092916.
[5] T. Michaeli and M. Irani, Blind deblurring using internal patch recur-
where i, j corresponds to row and column indices, respectively. rence, in Computer VisionECCV 2014. Springer, 2014, pp. 783798.

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing
JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012
[6] A. Levin, Y. Weiss, F. Durand, and W. T. Freeman, Efficient marginal
likelihood optimization in blind deconvolution, in Computer Vision and
Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2011,
pp. 26572664.
[7] L. Lucy, Bayesian-based iterative method of image restoration, Journal
of Astronomy, vol. 79, no. 745-754, 1974.
[8] A. Levin, Y. Weiss, F. Durand, and W. Freeman, Understanding and
evaluating blind deconvolution algorithms, in Computer Vision and
Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE,
2009, pp. 19641971.
[9] R. Fergus, B. Singh, A. Hertzmann, S. Roweis, and W. Freeman,
Removing camera shake from a single photograph, ACM SIGGRAPH,
2006.
[10] J. Cai, H. Ji, and Z. Shen, Blind motion deblurring from a single image
using sparse approximation, CVPR, vol. 62, pp. 291294, January 2009.
[11] M. Ben-Ezra and S. Nayar, Motion deblurring using hybrid imaging,
in Computer Vision and Pattern Recognition, 2003. Proceedings. 2003
IEEE Computer Society Conference on, vol. 1. IEEE, 2003, pp. I657.
[12] S. Nayar and M. Ben-Ezra, Motion-based motion deblurring, Pattern
Analysis and Machine Intelligence, IEEE Transactions on, vol. 26, no. 6,
pp. 689698, 2004.
[13] Q. Shan, J. Jia, and A. Agarwala, High-quality motion deblurring from
a single image, ACM Transactions on Graphics (SIGGRAPH), 2008.
[14] T. Cho, S. Paris, B. Horn, and W. Freeman, Blur kernel estimation
using the radon transform, in Computer Vision and Pattern Recognition
(CVPR), 2011 IEEE Conference on. IEEE, 2011, pp. 241248.
[15] S. Cho and S. Lee, Fast motion deblurring, in ACM Transactions on
Graphics (TOG), vol. 28, no. 5. ACM, 2009, p. 145.
[16] L. Xu, S. Zheng, and J. Jia, Unnatural l0 sparse representation for
natural image deblurring.
[17] L. Z. S. Cho, D. Metaxas, S. Paris, and J. Wang, Handling noise in
single image deblurring using directional filters.
[18] M. Hirsch, C. J. Schuler, S. Harmeling, and B. Scholkopf, Fast removal
of non-uniform camera shake, in Computer Vision (ICCV), 2011 IEEE
International Conference on. IEEE, 2011, pp. 463470.
[19] O. Whyte, J. Sivic, A. Zisserman, and J. Ponce, Non-uniform deblur-
ring for shaken images, in Computer Vision and Pattern Recognition
(CVPR), 2010 IEEE Conference on. IEEE, 2010, pp. 491498.
[20] C. Paramanand and A. N. Rajagopalan, Non-uniform motion deblurring
for bilayer scenes, in Computer Vision and Pattern Recognition (CVPR),
2013 IEEE Conference on. IEEE, 2013, pp. 11151122.
[21] K. Schelten and S. Roth, Localized image blur removal through non-
parametric kernel estimation, in Pattern Recognition (ICPR), 2014 22nd
International Conference on. IEEE, 2014, pp. 702707.
[22] J. Shi, L. Xu, and J. Jia, Discriminative blur detection features, in
Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Confer-
ence on. IEEE, 2014, pp. 29652972.
[23] Q. Qian and B. K. Gunturk, Space-varying blur kernel estimation and
image deblurring, in IS&T/SPIE Electronic Imaging. International
Society for Optics and Photonics, 2014, pp. 90 230E90 230E.
[24] T. H. Kim and K. M. Lee, Segmentation-free dynamic scene deblur-
ring, in Computer Vision and Pattern Recognition (CVPR), 2014 IEEE
Conference on. IEEE, 2014, pp. 27662773.
[25] Y. Zhang and K. Hirakawa, Fast spatially varying object motion
blur estimation, in Image Processing (ICIP), 2015 IEEE International
Conference on. IEEE, 2015.
[26] A. Chakrabarti, T. Zickler, and W. T. Freeman, Analyzing spatially-
varying blur, CVPR, 2010.
[27] A. Levin, Blind motion deblurring using image statistics, Advances in
Neural Information Processing Systems, vol. 19, p. 841, 2007.
[28] Y. Zhang and K. Hirakawa, Blur processing using double dis-
crete wavelet transform, in Computer Vision and Pattern Recognition
(CVPR), 2013 IEEE Conference on. IEEE, 2013, pp. 10911098.
[29] S. Dai and Y. Wu, Removing partial blur in a single image, in
Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE
Conference on. IEEE, 2009, pp. 25442551.
[30] F. Couzinie-Devy, J. Sun, K. Alahari, and J. Ponce, Learning to estimate
and remove non-uniform image blur, in Computer Vision and Pattern
Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013, pp. 1075
1082.
[31] J. Portilla, V. Strela, M. Wainwright, and E. Simoncelli, Image denois-
ing using scale mixtures of gaussians in the wavelet domain, Image
Processing, IEEE Transactions on, vol. 12, no. 11, pp. 13381351, 2003.
[32] M. J. Wainwright and E. P. Simoncelli, Scale mixtures of gaussians and
the statistics of natural images. in NIPS. Citeseer, 1999, pp. 855861.
1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
15
[33] H. Jeffreys, An invariant form for the prior probability in estimation
problems, Proceedings of the Royal Society of London. Series A.
Mathematical and Physical Sciences, vol. 186, no. 1007, pp. 453461,
1946.
[34] D. Donoho and I. Johnstone, Adapting to unknown smoothness via
wavelet shrinkage, Journal of the american statistical association,
vol. 90, no. 432, pp. 12001224, 1995.
[35] E. Polak, Optimization: algorithms and consistent approximations.
Springer-Verlag New York, Inc., 1997.
[36] A. P. Dempster, N. M. Laird, and D. B. Rubin, Maximum likelihood
from incomplete data via the em algorithm, Journal of the Royal
Statistical Society. Series B (Methodological), pp. 138, 1977.
[37] A. Chakrabarti and T. Zickler, Depth and deblurring from a spectrally-
varying depth-of-field, in Computer VisionECCV 2012. Springer,
2012, pp. 648661.
[38] A. Levin, R. Fergus, F. Durand, and W. T. Freeman, Image and depth
from a conventional camera with a coded aperture, in ACM Transactions
on Graphics (TOG), vol. 26, no. 3. ACM, 2007, p. 70.
[39] Y. Boykov and V. Kolmogorov, An experimental comparison of min-
cut/max-flow algorithms for energy minimization in vision, Pattern
Analysis and Machine Intelligence, IEEE Transactions on, vol. 26, no. 9,
pp. 11241137, 2004.
[40] L. Sun, S. Cho, J. Wang, and J. Hays, Edge-based blur kernel estimation
using patch priors, in Computational Photography (ICCP), 2013 IEEE
International Conference on. IEEE, 2013, pp. 18.
[41] D. Zoran and Y. Weiss, From learning models of natural image patches
to whole image restoration, in Computer Vision (ICCV), 2011 IEEE
International Conference on. IEEE, 2011, pp. 479486.
[42] D. Martin, C. Fowlkes, D. Tal, and J. Malik, A database of human
segmented natural images and its application to evaluating segmentation
algorithms and measuring ecological statistics, in Proc. 8th Intl Conf.
Computer Vision, vol. 2, July 2001, pp. 416423.
[43] E. Dubois, Frequency-domain methods for demosaicking of bayer-
sampled color images, IEEE Signal Processing Letters, vol. 12, no. 12,
p. 847, 2005.
[44] R. C. Gonzalez, R. E. Woods, and S. L. Eddins, Digital image processing
using MATLAB. Pearson Education India, 2004.
[45] C. Karam, C. Chen, and K. Hirakawa, Stochastic bilateral filter for
high-dimensional images, in Image Processing (ICIP), 2015 IEEE
International Conference on. IEEE, 2015, pp. 192196.
[46] A. Zomet, A. Levin, S. Peleg, and Y. Weiss, Seamless image stitching
by minimizing false edges, Image Processing, IEEE Transactions on,
vol. 15, no. 4, pp. 969977, 2006.
[47] M. W. Tao, M. K. Johnson, and S. Paris, Error-tolerant image com-
positing, International journal of computer vision, vol. 103, no. 2, pp.
178189, 2013.
Yi Zhang Yi Zhang (S12-M16) received the B.E.
degree in Electronic and Information Engineering
from Nanjing University of Information Science and
Technology, Nanjing, Jiangsu, China, in 2008, the
M.S. and Ph.D. degrees in Electrical and Computer
Engineering from University of Dayton, Dayton,
OH, in 2011 and 2015, respectively. Dr. Zhang
is currently a research scientist at Ford Research
and Innovation Center, Palo Alto, CA. His current
research interests include image processing, statis-
tical signal processing, computer vision and deep
learning.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIP.2016.2583069, IEEE
Transactions on Image Processing

JOURNAL OF LATEX CLASS FILES, VOL. 11, NO. 4, DECEMBER 2012 16

Keigo Hirakawa Keigo Hirakawa (S00-M05-

SM11) received the B.S. degree in Electrical En-
gineering from Princeton University, Princeton, NJ,
in 2000 with high honors, the M.S. and Ph.D. de-
grees in Electrical and Computer Engineering from
Cornell University, Ithaca, NY, in 2003 and 2005,
respectively, and the M.M. degree in Jazz Perfor-
mance Studies from the New England Conservatory
of Music, Boston, MA, in 2006 with high honors. He
was a research associate at Harvard University, Cam-
bridge, MA, from 2006 until 2009. He is currently an
associate professor at the University of Dayton, Dayton, OH. Prof. Hirakawa
heads Intelligent Signal Systems Laboratory at University of Dayton, where
the group focuses on statistical signal processing, color image processing, and
computer vision.

1057-7149 (c) 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.