Robust Nonlinear Diusion

of
Vector Fields
Ashish Doshi, BEng.(Hons), MIET
Submitted for the degree of Doctor of Philosophy
Department of Computer Science
August, 2008
Copyright c _ by Ashish Doshi, 2008.
All rights reserved. No part of this work may be reproduced or transmitted
in any form or by any means, electronic or mechanical, including photocopy,
recording or any information storage or retrieval system, without permission
in writing from the author.
To,
Savita
Abstract
Diusion methods are fundamentally important to reduce the eects of
noise whilst enhancing edges and the general perception of images. Although
diusion is primarily applied to images, applying such methods on vectorial
elds is an important and challenging problem under the conditions of uncer-
tainty and corruption which exist in most vectorial eld estimation problems.
In this thesis, a novel anisotropic diusion method is developed which
is based on the dynamics of the local Hessian. The proposed algorithm is
kernel based derived from the heat equation. Statistically robust methods
such as the median and the alpha-trimmed mean are integrated within the
algorithm. The result is a smoothed optical ow with reduced outliers, given
an initial noisy ow. The technique is evaluated on synthetic data as well as
vector elds extracted from real-world image sequences.
Further study on the eects of diusion on structural ows is conducted.
Structural ows are obtained from real-world medical data for the purpose
of 3D volumetric interpolation from 2D sparse data. As a result, the eects
of smoothing are visualised in reconstructed 3D volumes.
In addition to the proposed robust diusion methods, a computational
uid dynamics based nonlinear model for optical ow smoothing is intro-
duced. An experimental analysis of the model which uses the Navier-Stokes
equation, is carried out to verify its suitability for motion smoothing of real-
world complex and turbulent image sequences and deformation of moving
objects over time. The uniqueness of the proposed algorithm is that it com-
bines the eectiveness of the explicit and the speed of the implicit nite
dierencing schemes. The performance of the model is compared to that of
i
other alternatives reported in the literature.
Finally, the robust Hessian based diusion kernels developed earlier are
extended to the nonlinear model for fast, smooth and ecient optical ow
smoothing. Our techniques are used to reliably estimate the presence of vor-
tices from articially created von Karman vortex sheet and real-time satellite
imagery. Experiments show that it is possible to extract features and model
the dynamics of vortices blindly, lacking the detailed knowledge of physical
properties of the event captured in each frame. The work provides the foun-
dation for the development of a robust real-time identication and tracking
of storms.
ii
Contents
Declaration vii
Acknowledgements viii
List of Publications ix
List of Figures x
List of Tables xv
List of Abbreviations xvii
List of Symbols xix
1 Introduction 1
1.1 Optical Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Literature Review 7
2.1 Optical Flow Estimation . . . . . . . . . . . . . . . . . . . . . 7
2.2 Diusion Algorithms . . . . . . . . . . . . . . . . . . . . . . . 11
iii
2.3 3D Volumetric Interpolations . . . . . . . . . . . . . . . . . . 15
2.4 Navier-Stokes Equations and Vortex
Identication . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3 Robust Hessian-based Anisotropic Diusion 24
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2 Optical Flow Estimation . . . . . . . . . . . . . . . . . . . . . 26
3.3 Anisotropic Diusion of Optical Flow . . . . . . . . . . . . . . 29
3.3.1 Diusion kernel . . . . . . . . . . . . . . . . . . . . . . 29
3.3.2 Embedded Hessian diusion kernel . . . . . . . . . . . 31
3.3.3 Multiple 2D Hessian kernels . . . . . . . . . . . . . . . 34
3.3.4 3D Hessian kernel . . . . . . . . . . . . . . . . . . . . . 35
3.4 Robust Hessian Diusion Kernels . . . . . . . . . . . . . . . . 36
3.4.1 Outlier robustness study . . . . . . . . . . . . . . . . . 36
3.4.2 Median of directional Hessians kernel . . . . . . . . . . 38
3.4.3 Alpha-trimmed mean kernel . . . . . . . . . . . . . . . 40
3.5 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 42
3.5.1 Smoothing noisy articial vector elds . . . . . . . . . 43
3.5.2 Smoothing motion elds . . . . . . . . . . . . . . . . . 49
3.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4 3D Volumetric Interpolation from Structural Flows 67
4.1 Research Objective . . . . . . . . . . . . . . . . . . . . . . . . 67
4.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.3 Volumetric Image Interpolation . . . . . . . . . . . . . . . . . 69
iv
4.4 Structural Flow Initialisation . . . . . . . . . . . . . . . . . . 71
4.5 Smoothing Structural Flows . . . . . . . . . . . . . . . . . . . 77
4.6 Slice Interpolation . . . . . . . . . . . . . . . . . . . . . . . . 78
4.7 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 81
4.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5 Robust Physics based Diusion Solver 109
5.1 Research Objective . . . . . . . . . . . . . . . . . . . . . . . . 109
5.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.3 The Stable Fluid Model . . . . . . . . . . . . . . . . . . . . . 112
5.4 Robust Hybrid Fluid Model . . . . . . . . . . . . . . . . . . . 116
5.5 Vortex Core Detection . . . . . . . . . . . . . . . . . . . . . . 121
5.6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 126
5.6.1 Synthetic data simulation set-up . . . . . . . . . . . . . 127
5.6.2 Results on synthetic data . . . . . . . . . . . . . . . . 131
5.6.3 Real image sequences and remarks . . . . . . . . . . . 142
5.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
6 Conclusion and Outlook 152
6.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . . 152
6.1.1 Robust nonlinear diusion . . . . . . . . . . . . . . . . 152
6.1.2 3D volumetric slice interpolation . . . . . . . . . . . . 153
6.1.3 Diusion based on uid dynamics . . . . . . . . . . . . 154
6.2 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
A Dataset 157
v
List of References 159
vi
Declaration
This thesis has not previously been accepted in substance for any degree
and is not being concurrently submitted in candidature for any degree other
than Doctor of Philosophy at the University of York. This thesis is the result
of my own investigations, except where otherwise stated. Other sources are
acknowledged by explicit references.
Some of the material in the following chapters has been previously pub-
lished by the author. A complete list of refereed publications can be found
on page ix.
I hereby give consent for my thesis, if accepted, to be made available for
photocopying and for inter-library loan, and for the title and summary to be
made available to outside organisations.
vii
Acknowledgements
First and foremost, I would like to thank my supervisor, Dr. Adrian G.
Bors, for his continued support, advice and suggestions during my research
and writing up. His breadth of knowledge and experience, coupled with his
dedication towards his students is invaluable both as a supervisor and as a
friend.
Secondly, appreciation goes to my assessors, Prof. Edwin Hancock and
Prof. Nishan Canagarajah for their constructive feedback on this thesis and
various aspects of my research, including reports and presentations.
My sincere thanks go to various members, past and present, of the com-
puter vision group for their invaluable technical discussions and brainstorm-
ing sessions.
Finally, to my family, for their encouragement, wholesome support and
love.
viii
List of Publications
Conference Papers:
A. Doshi, A. G. Bors. Navier-Stokes formulation for mod-
elling turbulent optical ow. In Proceedings of 18th British
Machine Vision Conference, Warwick, United Kingdom. 10-13
September 2007.
A. Doshi, A. G. Bors. Robustied heat kernel smoothing
of structural ows for volumetric image interpolation. In
Proceedings of IEEE International Conference on Image Process-
ing, Atlanta, USA. 8-11 October 2006.
A. Doshi, A. G. Bors. Robust diusion kernels for optical
ow smoothing. In Proceedings of IEEE International Work-
shop on Machine Learning for Signal Processing, Maynooth, Ire-
land. 6-8 September 2006.
A. Doshi, A. G. Bors. Structural ow smoothing for shape
interpolation. In Proceedings of 18th International Conference
on Pattern Recognition, Hong Kong, China. 20-24 August 2006.
A. Doshi, A. G. Bors. Optical ow diusion with robusti-
ed kernels. In Proceedings of 11th International Conference
on Computer Analysis of Images and Patterns, Lecture Notes
in Computer Science 3691, pp. 222-230, Versailles, France. 5-8
September 2005.
ix
List of Figures
1.1 Top-level research model where I is the image frame and V
represents the motion eld. . . . . . . . . . . . . . . . . . . . . 4
3.1 Block matching algorithm. . . . . . . . . . . . . . . . . . . . . 27
3.2 Articial vector eld. . . . . . . . . . . . . . . . . . . . . . . . 31
3.3 Smoothed articial vector elds. . . . . . . . . . . . . . . . . . 33
3.4 The eect of diusion on outliers. . . . . . . . . . . . . . . . . 38
3.5 Calculation of the median of directional Hessian kernels. . . . 41
3.6 Synthetic vector elds, original and after being corrupted with
noise. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.7 Examples of Synthetic-1 vector elds after corruption with
Gaussian noise with
2
= 0.1 after ve iterations of smoothing. 46
3.8 Articial vector eld after corruption with Poisson noise with

2
= 0.25 after ve iterations of smoothing. . . . . . . . . . . . 49
3.9 Frames from three image sequences and their corresponding
optical ows. . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.10 Frames from additional three image sequences and their cor-
responding optical ows. . . . . . . . . . . . . . . . . . . . . . 52
x
3.11 PSNR convergence for the reconstructed frame 8 from Con-
corde sequence. . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.12 Predicted frame PSNR evaluation, using ATM-2DH smooth-
ed vector elds when varying the alpha parameter for various
window sizes. . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.13 Smoothed optical ows, initialised using BMA and the result-
ing predicted 5th frame of the Taxi sequence. . . . . . . . . . 58
3.14 Smoothed optical ows, initialised using LK and the resulting
predicted 6th frame of the Concorde sequence. . . . . . . . . . 59
3.15 Results when applying optical ow smoothing for the image
sequences considered in this study when the optical ow has
been initialised using BMA. . . . . . . . . . . . . . . . . . . . 60
3.16 Results when applying optical ow smoothing for the image
sequences considered in this study when the optical ow has
been initialised using LK. . . . . . . . . . . . . . . . . . . . . 61
3.17 PSNR of the predicted frame when tracking scene change in
two image sequences for the best ve diusion methods when
the convergence criterion is set to < 10
1
. . . . . . . . . . . 62
4.1 An illustration of intermediate slice reconstruction. . . . . . . 70
4.2 Block matching process in DBMA. . . . . . . . . . . . . . . . 72
4.3 DBMA structural ows. . . . . . . . . . . . . . . . . . . . . . 73
4.4 Bad edge matching. . . . . . . . . . . . . . . . . . . . . . . . . 74
4.5 GrowingRegion(BlkOrig, BlkRef): Pseudocode of growing
algorithm used for regions of object boundaries, where nnz()
is a function that nds the number of nonzero pixels in a block. 75
xi
4.6 Visualisation of growing algorithm. . . . . . . . . . . . . . . . 76
4.7 The slice interpolation owchart. . . . . . . . . . . . . . . . . 79
4.8 Sample slices of the data sets. . . . . . . . . . . . . . . . . . . 82
4.9 Sample slices from the Incisor data set. . . . . . . . . . . . . . 84
4.10 Structural ows and reconstructed slice of an Incisor. . . . . . 85
4.11 Further results for the Incisor. . . . . . . . . . . . . . . . . . . 86
4.12 3D Incisor reconstructions. . . . . . . . . . . . . . . . . . . . . 88
4.13 Accuracy of the middle slice reconstruction considering both
shape structure and grey-level. . . . . . . . . . . . . . . . . . . 90
4.14 Example of Knee slices for experimentations. . . . . . . . . . . 93
4.15 Structural ows and reconstructed slice of a Knee MRI. . . . . 94
4.16 Further results on structural ows and reconstructed slices of
a Knee MRI. . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.17 Original Humerus bone. . . . . . . . . . . . . . . . . . . . . . 97
4.18 3D Humerus bone reconstructed volumes. . . . . . . . . . . . 98
4.19 Original Iliac bone. . . . . . . . . . . . . . . . . . . . . . . . . 99
4.20 3D Iliac bone reconstructed volumes when skipping 5 consec-
utive slices between the remaining 2 slices. . . . . . . . . . . . 100
4.21 Results on structural ows and reconstructed slices of a Female
Chest MRI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
4.22 Results on structural ows and reconstructed Sheeps Heart
MRI slices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
xii
4.23 Reconstruction of bone contours for Humerus and Iliac bones.
From top to bottom, the contours are from smoothed recon-
structions using original slices (blue), DBMA (red), Perona-
Malik (green), Black (magenta), 2DH (yellow), ATM-2DH
(cyan) and MED-2DH (black). . . . . . . . . . . . . . . . . . . 108
5.1 The stable uid solver algorithm. . . . . . . . . . . . . . . . . 114
5.2 Robust hybrid solver. . . . . . . . . . . . . . . . . . . . . . . . 116
5.3 Helmholtz-Hodge decomposition of a closed lid driven cavity
laminar ow. . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5.4 Evaluation of the Q
w
-criterion from a 5by5 window, where
the location of each vector in the window is shown with a
dierent marker according to its corresponding Q
i
set. . . . . . 126
5.5 Driven-lid cavity ow diagram. . . . . . . . . . . . . . . . . . 127
5.6 Synthetic closed driven-lid cavity ows with noise. . . . . . . . 129
5.7 Half-cylinder model for von Karman ow. . . . . . . . . . . . 130
5.8 Closed cavity vector eld smoothing comparisons. . . . . . . . 132
5.9 Representing von Karman ows. . . . . . . . . . . . . . . . . . 134
5.10 Smoothing noisy von Karman ows. . . . . . . . . . . . . . . 135
5.11 Evaluating Q and Q
w
on the von Karman ow using equations
(5.14) and (5.18), respectively. . . . . . . . . . . . . . . . . . . 137
5.12 MSE after smoothing the noisy von Karman ow. . . . . . . . 138
5.13 Performance of the vorticity segmentation using
w
. . . . . . 139
5.14 Evaluating and
w
on a smoothed von Karman ow from
equations (5.17) and (5.19), respectively. . . . . . . . . . . . . 141
xiii
5.15 Smoothing optical ows in image sequences displaying turbu-
lent motion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
5.16 Finding vortices in the Superstorm Andrea sequence. . . . . . 145
5.17 Additional results of vortex segmentation of Superstorm An-
drea sequence. . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
5.18 Smoothing optical ow corresponding to cloud movement for
Superstorm Andrea. . . . . . . . . . . . . . . . . . . . . . . . 147
5.19 Finding vortices in the Solar Flare sequence. . . . . . . . . . . 149
A.1 Incisor dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . 158
xiv
List of Tables
3.1 Numerical results for Synthetic-1 data after one iteration of
diusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.2 Numerical results for Synthetic-2 data after one iteration of
diusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.3 PSNR (dB) of the reconstructed frame using the smoothed
optical ow with the number of iterations (No) necessary to
reach convergence for each method and in the case of each
image sequence when the motion eld is initialised using BMA. 63
3.4 PSNR (dB) of the reconstructed frame using the smoothed
optical ow with the number of iterations (No) necessary to
reach convergence for each method and in the case of each
image sequence when the motion eld is initialised using LK. . 64
4.1 Summary of the slice dimensions and voxel sizes for the dif-
ferent data sets. . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.2 Average percentage of reconstruction errors with DBMA as
the initialisation. . . . . . . . . . . . . . . . . . . . . . . . . . 104
4.3 Average peak signal-to-noise ratio of slice reconstructions with
DBMA as the initialisation. . . . . . . . . . . . . . . . . . . . 104
xv
4.4 Average PSNR of original middle slice reconstructions after
removing 5 intermediate slices with LK as the initialisation. . 105
4.5 Hausdor distance for Humerus bone. . . . . . . . . . . . . . . 105
4.6 Hausdor distance for Iliac bone. . . . . . . . . . . . . . . . . 106
5.1 Mean cosine error of smoothed vector elds. . . . . . . . . . . 132
5.2 Evaluation of smoothed noisy von Karman ows when the
noise variance is 0.10. . . . . . . . . . . . . . . . . . . . . . . . 138
xvi
List of Abbreviations
1D One Dimensional
2D Two Dimensional
3D Three Dimensional
ATM-2DH Alpha-Trimmed Mean 2D Hessian kernel
ATM-3DH Alpha-Trimmed Mean 3D Hessian kernel
ATM-M2DH Alpha-Trimmed Mean Multiple eld
2D Hessian kernel
Black Black et. al.s method
BMA Block Matching Algorithm
CFD Computational Fluid Dynamics
CT Computer Tomography
dB Decibel
DBMA Dual directional Block Matching Algorithm
L.H.S. Left Hand Side
LK Lucas-Kanades algorithm
M2DH Multiple eld 2D Hessian kernel
MAD Median of Absolute Deviation
MCE Mean Cosine Error
MED-2DH Median of 2D Hessian kernel
MED-3DH Median of 3D Hessian kernel
MED-M2DH Median of Multiple eld 2D Hessian
kernel
MedH-SFS Robust Hybrid Fluid Solver
xvii
MSE Mean Square Error
MRI Magnetic Resonance Image
OFE Optical Flow Equation
OFCE Optical Flow Constrained Equation
PDE Partial Dierential Equation
PIV Particle Image Velocimetry
PM Perona-Malik method
PSNR Peak Signal-to-Noise Ratio
R.H.S Right Hand Side
SFS Stable Fluid Solver
SVD Singular Value Decomposition
xviii
List of Symbols
I Still video frame
V Vector eld
x, y Eulerian coordinates
t Time
V
x
, u x component of vector eld
V
y
, v y component of vector eld
H Hessian
i, j, k, l Array indices
M, N Length of array
Dierence
S
x
Search region in the x direction
S
y
Search region in the y direction
c Energy
Lagrangian multiplier
w Weights
() Windowed neighbourhood centred around
Scalar eld
f(),

f() Functions
Pi
Alpha trimmed mean trimming ratio
Angle

2
Variance
( Gaussian function
xix
(
P
Poisson function
e, exp Exponential function
L Total number of vectors
()
T
Vector or matrix transposed
Convergence threshold
/ Manifold space
(
K
Heat kernel
IR Real space
d Normalisation coecient
Covariance matrix
L Laplacian matrix
Vorticity function
Gradient operator
H Hausdor distance
Divergence operator
P Pressure
Density
Viscosity

2
Laplace operator
f Force
n Surface normal
Bound domain
T Projection operator
S Rate of strain tensor
Vorticity tensor
xx
Discriminant
1() Indexing function
Stream function
Curl operator
xxi
Chapter 1
Introduction
In modern times, there has been a tremendous technological breakthrough in
the areas of digital computation and telecommunications. Of particular in-
terest has been the participation of the general public in these developments
as aordable computers and the incredible explosion of the World Wide Web
has brought a ood of instant information to an increasing percentage of
homes and businesses. Most of this information is designed for visual con-
sumption in the form of text, graphics and pictures or integrated multimedia
presentations. The rate at which information is transmitted, stored, pro-
cessed and displayed in a digital visual format is increasing rapidly and thus,
the design of engineering methods for eciently transmitting, maintaining
and even improving the visual integrity of this information is of great interest.
1.1 Optical Flow
Analysing motion patterns is essential for understanding visual surround-
ings. Representative applications of motion analysis include video coding [1],
1
robotic vision, super-resolution reconstruction, enhancement, etc. Typically,
when estimating motion, it is assumed that all pixel intensities are locally
translated from one frame to the next and that the shifted values are pre-
served. This constraint implies that the intensity of a moving pixel in the
image plane remains constant along the trajectory of that pixel in time. This
assumption is the basis of the optical ow constraint equation [2]. Conse-
quently, optical ow equations are widely used to estimate motion between
consecutive frames.
From the point of view of image analysis, the optical ow estimation al-
gorithms can be classied as gradient-based and feature-based methods. The
method employed can be either deterministic or stochastic, local or global.
The most widely used method for motion estimation is the block matching
algorithm (BMA) due to its simple implementation scheme. The BMA esti-
mates the vector eld based on the correlation between each pixel microblock
in one frame and the corresponding microblock within a macroblock (search
region) in the subsequent frame. However, unconstrained pixel intensity and
lack of local contrast variation can lead to erroneous optical ow estimations.
In order to overcome such problems, regularisation terms are introduced,
hence the wide variations of BMA available within the research community.
The focus of this research is the use of anisotropic diusion for accurate
estimation of the velocity elds initialised by BMA and Lucas-Kanades (LK)
gradient based algorithm [3]. Typically, most BMAs use either windowed
median operators or other statistical cost measures to obtain smooth vectors.
This is also true for LK algorithms which use weighted windowed Gaussian
kernels. However, the performance of such methods when processing 2D
2
motion vectors depicting spatial-temporal object deformation is questionable,
hence the interest in nonlinear schemes such as partial dierential equations
(PDE). Although a considerable amount of research has been conducted in
this area, most of it has been oriented towards understanding mathematical
properties of anisotropic diusion [4] and modifying the diusion equations
for specic applications [5, 6]. A lot of this research is primarily applied for
diusing and segmenting colour images and image inpainting [6, 7, 8].
Robust statistical methods are adopted in computer vision to improve
the performance of feature extraction algorithms at the bottom level of the
vision hierarchy. These methods, to varying degrees, tolerate the presence of
data points that do not obey the assumed model. These points are typically
known as outliers. The robustness, in this context, can be attributed to the
breakdown point. This can be dened as a point at which the smallest fraction
of outliers can cause the estimator to produce arbitrarily bad results. This
is usually regarded as the worst-case scenario in statistics. Some examples
of such methods are presented in [9, 10]. These type of kernels use rank
ordering statistics for outlier rejection. Notably, results can be skewed if
data is uneven.
Further details of the anisotropic diusion algorithms developed and their
novelty is provided in Chapter 3. These algorithms, embedded with stochas-
tic processes form the core basis for the remainder of this thesis.
1.2 Motivation
Robust methods for motion estimation, motion compensation and regulari-
sation are very important in image processing. These are the pre-processing
3
techniques that make standard image processing methods work. To sum it
up, when dealing with image sequences, motion is estimated through the
computation of the image velocity vectors. Diusion schemes (lter) are ap-
plied during vector eld computation to enhance the structure of moving
objects whilst smoothing out unwanted information (mostly due to noisy ar-
tifacts). When a clean optical ow is obtained, this information is used in
image processing lters to compensate for motion for image enhancement,
segmentation, tracking and classication. This process is highlighted in Fig-
ure 1.1.
Figure 1.1: Top-level research model where I is the image frame and V
represents the motion eld.
More recently though, signicant research has been done in developing
image pre-processing techniques to obtain better feature extraction results.
Hence, the main aim of this research is to develop image pre-processing tech-
4
niques, mainly anisotropic diusion using nonlinear measures such as kernel
based Gaussian function and nite dierencing methods from uid mechanics
for feature extraction of non-rigid objects which are deformable over time.
The other purpose of this research is to develop a generalised nonlinear
smoothing model which can either be used on its own or as a secondary lter
for optical ow processing.
1.3 Thesis Overview
The structure of the thesis is organised as follows:
Chapter 2 surveys signicant contributions to the eld of optical ow es-
timation and processing, isotropic and anisotropic diusion algorithms, 3D
volumetric reconstructions (mainly from medical imaging) and uid mechan-
ics methods that have been implemented in computer vision and pattern
matching.
Chapter 3 of the thesis presents the novel contribution of nonlinear sm-
oothing kernel which is based on the local Hessian information of the vector
eld. Included in the chapter is the formulation of the dierential smoother
with statistical robustness. Also this chapter provides are experimental re-
sults and analysis of techniques based on the local heat kernel and Hessian
matrix of the optical ow. Further discussions on the comparative perfor-
mances of the proposed algorithms against known methods in the eld is also
covered.
In Chapter 4, the focus of research is on the utilisation of structural
ows to reconstruct 3D volumetric models of human tissue structures. This
chapter will show how structural ows are obtained and further smooth-
5
ed using anisotropic diusion methods developed in Chapter 2 to obtain
intermediate slices and subsequently reconstruct 3D volumes. The proposed
methodology is shown to obtain positive results which are comparable with
other known implementations.
Chapter 5 presents another novel diusion algorithm which is based on the
Navier-Stokes equation used in computational uid dynamics (CFD). The
proposed algorithm is a hybrid model which is dependent on both the explicit
and implicit dierencing schemes. As in prior chapters, this chapter will also
describe models used to model ow elds and provide experimental results
from both articial and real-world sequence based vector elds. Additionally,
there is also a section on vortex dynamics which covers the identication and
segmentation of vortex cores. The methods described here can be used for
vortex extraction from satellite imagery.
Finally, Chapter 6 summarises the contributions to the thesis, highlight-
ing strengths and weaknesses as well as suggestions on future research direc-
tions.
6
Chapter 2
Literature Review
The literature review survey conducted for the preparation of this thesis
covers numerous disciplines such as mathematics, physics, engineering, bio-
science, neuroimaging, medical imaging and computational uid dynamics.
This chapter is divided into four sections, each describing major contribu-
tions to the specic area of interest for the purpose of this work. Other
contributions, though not signicant for the current research, are neverthe-
less mentioned in the review.
2.1 Optical Flow Estimation
Optical ow estimation from image sequences has been identied as an ill-
posed problem [1] which requires a regularisation methodology as shown since
early eighties by Horn and Schunck [11], Lucas and Kanade [3] as well as in
the comparative study from [12, 13]. In their paper, Horn and Schunck pro-
posed a method to calculate the moving eld (u and v components) at each
point using image intensity dierences. Their work involved some assump-
7
tions such as the surface object is at to avoid shading variations, assume
initial pixel intensity to be uniform across the surface with no spatial dis-
continuities. Horn and Schuncks algorithm was among the rst that gave
better smoothing of the velocity eld resulting in better segmentation of the
moving objects. They also introduced a way of implementing the derivatives
digitally. They proposed averaging four nite dierences (based on 2-by-2
cube representation) [2].
Quite often the estimated optical ow is noisy and contains outliers. This
is even more evident in the case of the block matching algorithm used in video
compression algorithms [1, 14]. However, a paper presented by Verri and Pog-
gio in 1986 [15] noted some disadvantages of Horn and Schuncks proposal.
They argued that the latters algorithm does not take into account any rota-
tional movement and that the optical ow and motion eld are generally not
identical concluding that two dimensional motion eld generally cannot rep-
resent three dimensional velocity eld unless special conditions are satised.
They suggest that feature-based matching algorithms are reliable enough to
accurately recover strong structures (segmentation) from motion, however in
order to determine the velocity eld, qualitative measurements are needed.
Firstly, they propose the use of scene radiance and image irradiance with the
aid of the Lambertian model to compute the dierence between the velocity
eld and motion eld. Secondly, they introduce minimal optical ow which
relates to perceived motion in the image.
There is no denite solution for occluded/unoccluded problem. Also,
there exist two unknowns (u and v velocity components) for each observa-
tion. And nally, he suggests that we can only determine the motion ow
8
orthogonal to the spatial image gradient (normal ow) at each pixel. Due
to this problem, there are certain assumptions that need to be made with
regard to the structure of the motion eld. These are:
Some sort of smoothing constraints or uniformity has to be imple-
mented on the motion eld (non-parametric model).
Six motion parameters to constrain local ow vectors to lie on a specic
line (quasi-parametric model).
More recently though, Tekalp [1] identies numerous motion eld estima-
tors. Among them are methods based on the optical ow equation (OFE),
pixel-recursive based methods (extended to Wiener lter type motion esti-
mation), Bayesian based methods, phase correlation based method and block
matching methods.
Bors [14, 16] used radial basis function (RBF) networks for motion esti-
mation and moving object segmentation [14]. His method implied the use of
median radial basis function (MBRF) network for segmentation. Simultane-
ously, subsequent image sequence frame is predicted using prior information
for tracking [14]. In [16], tracking was employed for moving object prediction
in video frame prediction. The process is repeated to obtain better motion
estimation and frame prediction, but the MBRF network has to be retrained
if an object has either entered or left a scene in the frame.
There are other methods that are being researched in the eld of motion
estimation. For example, P erez et. al. [17] proposed using the RANSAC
1
algorithm to estimate rigid motions through the image sequence. The RA-
NSAC algorithm is a robust tting model in the presence of data outliers.
1
(Random Sample Consensus) Algorithm
9
They then use singular value decomposition technique to estimate the scale
space representation of the local motion from the data. Other examples of
motion estimation techniques are fuzzy based as outlined by Erdem et. al. [18]
and Peacock et. al. [19], blind estimation using generalized cross validation
technique as proposed by Foroosh [20] and adaptive based methods using
recursive optical ow estimation as proposed by Elad [21] or using tensor
based models as proposed by Liu et. al. [22], all of which are robust and
ecient on simulated sequences yet untested on arbitrary video sequences.
Energy based minimisation frameworks have been dened for estimating
the optical ow by taking into account various determining factors [14, 23,
24, 25].
Recently, the research in optical ow computation has focused on stochas-
tic approaches. Roth and Black [25] proposed using spatial statistics of vector
elds to obtain better accuracy in the ow estimation. Their method relies
on learning statistical information from a synthetic training set of optical
ows to obtain an appropriate statistical energy function. This function is
used within the Combined-Local-Global (CLG) [26] framework. Altough their
method performs reasonably well, it is limited (as suggested in [25]) due to
the fact the training set used is made up of static images, occlusion and
movement into/out-of scene is not considered.
Dierent applications such as moving object segmentation [14, 23], track-
ing in image sequences and video coding quality [1] depend on smooth optical
ows.
10
2.2 Diusion Algorithms
Regularisation techniques using partial dierential equations (PDE) have
found several applications in image processing and computer vision. The
theoretical background of these approaches arises from the methodology used
for describing physical phenomena such as the dissipation of heat or the
movement of uids [27, 28, 29]. The solution of the heat equation provides
a kernel function which is adaptive to the local manifold [30, 31]. If the
manifold is constant in all directions, then an isotropic kernel can be used
[27]. However, this is a particular case of data and most often the variation
in the local manifold should be taken into account by using an anisotropic
kernel. PDEs used in the diusion context have been applied for image
smoothing while preserving main features in images such as object contours
and corners [7, 32, 33, 34].
Anisotropic diusion was employed for smoothing images while preserving
edges by Perona and Malik [4]. Their work was extended by Black et. al.
[7] by using a dierent edge detection function in order to preserve sharper
boundaries in images. Hummel [35] oers some very important insight on the
relationship between Gaussian blurring and the heat equation. Hummel is
able to demonstrate that there is a precise relationship between the standard
deviation of the Gaussian and time t of the heat diusion equation. However,
a caveat of the heat equation is that true locations of the boundaries in very
noisy images makes the boundary preservation questionable since it may be
impossible to determine the existence of the boundary.
Weickert et. al. used the additive operator splitting (AOS) [34] for smoo-
thing volumetric images by extending the diusion approach from [33], which
11
is similar to Perona and Malik [4], but using a Gaussian kernel instead of
gradient of image. The AOS scheme guarantees equal treatment of all co-
ordinate axes. It can be implemented easily in arbitrary dimensions, has good
rotational invariance and reveal a computational complexity and memory
requirement which is linear in the number of pixels. Weickert et. al. have
proved that AOS schemes are at least 10 times more ecient than the widely
used explicit schemes.
The above work has been inuenced by a number of related approaches.
Implicit splitting based approaches for linear diusion ltering have been
proposed in [36] and [37] and also in [38] and [39] where their realisation
for recursive lters is suggested. Impressive results on improved eciency
by recursive ltering can be found in [40, 41] and the close relation between
recursive lters and linear scale-space approaches has been claried in [42].
In the nonlinear diusion eld, one can nd several approaches which
aim to be ecient alternatives to the conventional two-level explicit nite-
dierence scheme, for instance three-level methods, semi-implicit approaches
[33], multiplicative splittings, multigrid methods [43], nite element tech-
niques with adaptive mesh coarsening, numerical schemes with wavelets as
trial functions and pseudo spectral methods [44]. Hardware proposals for
nonlinear diusion ltering can be found in the literature [4].
Luck et. al. [45] combined the median lter and the diusion operator of
Black et. al. [7] for smoothing medical confocal images. The adaptivity of
the Gaussian kernel to the local manifold is given by its covariance matrix
[46].
A suitable model of the local manifold geometry is indicated by the Hes-
12
sian matrix calculated from the local data [5, 6, 47]. The Hessian matrix,
represents the multidimensional second order derivates and can be used to
detect local maxima and minima as well as the transition areas from images.
When the Hessian is embedded in certain functions, it can be used to improve
the object representation as estimated from local surface normals [10].
On a more recent note, Tschumperl e and Deriche [6] proposed PDE based
vector diusion method. In a nutshell, their method uses tensor and Hessian
matrices which are constructed from a still image or image sequence and
is then integrated within an anisotropic diusion kernel of Gaussian nature
for smoothing optical ow eld or image intensities. The advantage of this
method is that the Gaussian kernel used is multivariate and tends to work
better when there exists multiple sources of noise or presence of complex mo-
tion. On the other hand, Burgi [48] uses diusion kernel proposed in [12] for
the smoothing of ow obtained through intensity gradient direction method.
Petrovic et. al. [49] have introduced an unsupervised segmentation tech-
nique to study the multi-scale structure of images. They accomplish this by
generating a scale-space stack and building a hierarchical tree of the scales.
Linking is then performed between the scales and diusion is applied to
smooth and enhance the edges. Their method has been applied mainly on
still greyscale and colour images. An interesting approach is implemented by
Preusser and Rumpf in [50]. Their method utilises two dimensional optical
ow computation to represent three dimensional velocity vectors. The core
of their method is the anisotropic geometric diusion scheme which is driven
by the structure of the image and its temporal acceleration.
In stark contrast to applying diusion based smoothing, marginal me-
13
dian is a simple statistical ordering method [9, 10]. It relies heavily on the
distribution and amount of data to be considered in the kernel. The data is
ranked and median of the data is selected. This type of method can be useful
to obtain good optical ows provided the structure of the moving objects are
strong and the neighborhood distribution has been selected appropriately.
However, the results can be skewed if the data is uneven.
There are numerous diusion kernels being developed with various degree
of complexity. Most of which is beyond the scope of this thesis. To sum-
marise, many of the methods reviewed have direct relevance to this research.
Although some methods have been applied to image stills, it is possible to
extend them to image sequences albeit some degree of complexity may be
involved. The methods described above have been proposed to work indi-
vidually, but our opinion is that the demonstration of succesful working of a
combination of methods would be an interesting concept to consider. Most of
the diusion kernel methods reviewed, integrate Gaussian (or its variation)
distribution within the kernel. The reasoning behind the distribution choice
is that on most occasion of motion blur, it is assumed that a Gaussian point
spread function is the cause of blurring and hence, the Gaussian diusion
kernel is best to de-blur. By assuming the distribution is Gaussian, it also
simplies the complexity of the algorithms involved thus reducing the com-
putational cost. However, there are other types of diusion kernels being
researched in the eld. Interesting ones include Weibull distribution based
kernel [51], principal component analysis (PCA) based kernel [52, 53] and
the Fisher criterion based kernel [54].
14
2.3 3D Volumetric Interpolations
Recent advances in medical imaging highlights an increasing need for 3D
models of human anatomy for medical diagnostics. Due to the nature of the
medical applications, it is extremely important that an accurate 3D model
of the anatomy is used. However, the task to reconstruct an accurate 3D
model is dicult from a group of sparse image slices. Traditionally, medical
image slices are obtained through mechanical slicing and digitisation, thus
producing image slices which are not equidistant between slices. Modern
technology provides images of slices through magnetic resonance imaging
(MRI) or computer tomography (CT). Even with such tools at a researchers
disposal, it is still improbable for the machines to take sucient digitised
images for full 3D reconstructions, hence the problem of data extraction
from sparse information.
There has been considerable amount of research devoted to reconstruct-
ing accurately the 3D models of a human body part. Most methods are
either a variation of morphological operations [55, 56] or interpolation meth-
ods [57, 58, 59, 60, 61]. For example, Bors [55] uses reciprocal binary
morphology to obtain intermediary slices to reconstruct a 3D tooth model.
Similarly, Lee [56] aligns the slices rst using object centralisation before
applying morphological operators. However, Lees [56] process only inter-
polate non-matching regions between slices. Schaller et. al. [60] proposed
using spiral interpolation based on azimuthal rebinning and Lee [61] pro-
poses shape based interpolation using warping. Comparisons between vari-
ous interpolation methods are presented in [58] and [59]. Other examples
of reconstruction methods in use include regularised maximum-likelihood
15
(ML) method [62], statistically based principal component analysis (PCA)
method [63], Bayesian framework based on alignment [64, 65], matching
based methods [57, 66, 67, 68, 69] and shape based methods [70, 59]. Other
than standard methods being researched, Turk and OBrien [71] considered
using a variational approach for shape transformation and interpolation.
Their method is based on an implicit implementation and uses contours
of objects as boundaries, which enables surface warping from one object to
another.
Weng et. al. [72] used optical ows from [2] to model the surface pro-
jection from one ultrasound image slice to another. Dougherty et. al. [68]
used optical ows to model myocardiac displacements within the heart. This
would then allow them to tag and track tissue deformation over time. Hata
et. al. [73] used the gradient based optical ow formulation to measure vol-
umetric brain deformations from MR images. They regularised the ows
using a Gaussian kernel for better estimation. However, they state that it is
important to segment and then remove skin from the processing stage as it
can inuence the accuracy of the ow estimations. Abr` amo and Viergever
[74] proposed a variation of Lucas and Kanades [3] method to visualise the
movement of soft tissue in 3D volumetric Orbit images. These images tend
to resemble soft tissues like optic nerves within the human or animal brain.
In contrast to gradient based methods, Weruaga et. al. [75] proposed a para-
metric approach to estimate the volumetric motion of the human thorax.
Furthermore, Ray and Acton [76] proposed using the energy minimisation
method of motion vector gradient vector ow to capture and track cells in
microscopic imaging. Also of interest is the myocardial motion eld captured
16
by applying a non-rigid registration technique [77] as presented by Rao et. al.
[78]. They also test their method on modelling brain deformations.
Variations of the block matching algorithm have been used for registra-
tion in medical images as described in [69, 79]. In [69], Penny et. al. realises
that for a registration based technique, it is important that initial neigh-
bouring slices contain similar anatomical features, otherwise the technique is
unreliable. Once the above criteria of slices having similar features are met,
they proceed to use B-spline method as in [77] to perform matching between
slices. The interpolated image is obtained by using the matching informa-
tion and linking it with the interpolation plane between the two slices. The
linking is done by linear interpolation between matching correspondes in the
slices and alignment is made appropriately.
Aside from the main focus of 3D volumetric interpolation schemes in
medical imaging, there are other kinds of research in this area that has some
relation to the work carried out in the thesis. One such method is the work
of Sharp and Hancock [80] which uses probabilistic relaxation model to cor-
relate the contour displacements of MRI surfaces with the corresponding
intensity features from successive frames to track its deformation. Rueckert
et. al. [77] used a nonrigid registration approach to model deformation in
breast magnetic resonance images. They apply ane transformations with
spline based free form deformation methodology to achieve some degree of
success. Garza-Jinich et. al. [81] used an automatic segmentation algorithm
to segment regions of interest from MR image slices. Their proposal is robust
and the segmentation regions are then stacked to produce a 3D volume for
visualisation purposes.
17
Also of interest is the implementation of PDE based models in medical
imaging. Sarti et. al. [82] implemented an improved Perona-Malik [4] diu-
sion algorithm with a multiscale method to give a better representation of
a 3D volumetric echocardiographic sequence. Chung and Sapiro [83] used
a PDE based morphological algorithm to successfully segment skin legions
from images. Krissian [84] used uid based ux algorithm to anisotropically
diuse CT images of the liver. By applying such an algorithm, he is able to
obtain a smooth image whereby the vascular structures of the liver could be
segmented for future analysis. Abd-Elmoniem et. al. [85] suggested using
a Hessian based diusion kernel to smooth speckle images. Their algorithm
is anisotropic in design and is applied on ultrasound images of the heart.
Also anisotropic is the diusion algorithm proposed by Ling and Bovik [86].
Their algorithm utilises the median of Peronal-Maliks [4] implementation to
achieve very good smoothing of structures within molecular images, which
have low signal-to-noise ratio. Chung [87] suggested using the heat kernel
for smoothing in neuroimaging. He acknowledges that it is not appropriate
to use isotropic kernels to smooth soft tissue surfaces as it is in a curvlin-
ear space. Hence, the favoured approach of anisotropic smoothing using the
heat kernel, where smoothing only occurs uniformly within a common neigh-
bourhood. He uses this model to investigate cortical thickness of the brain
between autistic and normal children.
18
2.4 Navier-Stokes Equations and Vortex
Identication
Very often, the modelling of natural phenomena involves the motion of dy-
namic uids, which diers radically from that of rigid bodies. Classical opti-
cal ow estimation algorithms would fail in such cases. The use of uid ow
modelling for motion estimation can be traced back to the work of Fitzpatrick
[88], who compared optical and uid ow methods. The computation of ows
depends largely on the specic nature of the application. Using Fitzpatricks
analysis as a basis, Song and Leahy [89], employed the equation of conti-
nuity as an additional constraint to Horn and Schuncks algorithm [11] in
order to obtain better motion estimation of the beating heart. Navier-Stokes
equations have been extensively studied in uid mechanics for modelling the
behaviour of uids under various conditions and constraints [90, 91].
The Navier-Stokes and optical ow constraint equations have been em-
ployed for modelling von Karman ows in [92]. Nakajima et. al. [92] min-
imised the overall cost function of the constraints to obtain a better estima-
tion of the ow for dye shading movements. Although the method seems
to yield good results, it was only tested on simulated sequences and their
algorithms is similar to that of [89]. Bertalmio et. al. applied the Navier-
Stokes equations to image and video inpainting [93]. Their approach uses
the vorticity-stream formulation of the uid ow equation, which can be at-
tributed to the image intensity-Laplacian relationship. Corpetti et. al. used
the vorticity-stream formulation to recover dense motion of water vapours
[94].
19
In recent years, extensive research has been conducted in optical ow
construction using uid based algorithms. One particular work was done by
Kohlberger et. al. [95] which used variational domain decomposition method
to obtain a fast converging optical ow construction method, especially for
large scale computations. As such, they also introduced a parallelisation
approach to achieve fast computations.
Navier-Stokes equations have been used in computer graphics for vi-
sualising ames and building animation tools based on uid-like motion
[96, 97, 98, 99]. The stable uid solver (SFS) algorithm implements Navier-
Stokes equations and consists of a set of consecutive processing steps [97],
such as: advection, diusion and mass conservation. The boundary condi-
tions are important in constraining the uid motion [91]. The boundaries
have been processed as a set of constraints on a grid [98], by enforcing repe-
tition and employing the Fast Fourier Transform (FFT) [97] or by using level
sets [96].
It is very well to use Navier-Stokes equations to model and smooth veloc-
ity ows, how is it possible to verify the results obtained? One possible way
is to identify the presence of vortices or coherent structures [100] in a ow
eld. Unless there is large perturbations in the eld, whereby a vortex could
easily be detected, the identication of vortices in noisy elds with small per-
turbations is dicult. Jeong and Hussain [100] suggest using regularisation
functions to obtain a uniform and smooth eld, with high frequency com-
ponents removed. This would full the properties of Galilean-invariant eld
for better detection [100]. They have identied that the vorticity magnitude
measurement, [[ is not a reliable way to detect vortices. This is because
20
there are other physical states where the vorticity magnitude could not de-
tect the presence of a vortex. Please refer to [100] for a detailed discussion on
this. They also argue the reliability of Hunt et. al.s Q-criterion measurement
[101]. Instead, they propose the
2
-denition as an additional constraint for
better vortex identication.
Recently, Haller [102] proposed M
Z
and Q
S
criterions for better repre-
sentation of the vortex detection from multiple frames, in three and two
dimensions respectively, compared to [101] and [100] algorithms. Addition-
ally, Haller also presented a secondary criteria for hyperbolic (of saddle-type)
structures around the vortex centre.
In computer vision, Zhong et. al. [103] were among the rst to implement
uid mechanics constraints [100, 101] for vortex detection from 3D turbulent
motion elds. However, they only managed to test the algorithms on arti-
cially generated 2D and 3D PIV (particle image velocimetry) turbulent elds.
Another example is the work conducted by Ford and Strickland [104]. They
proposed using partition based method to segment regions of interest (con-
verging and diverging focal points) using polynomial based nonlinear least
squares estimation. This is then used to recover critical uid structures from
streamline imageries.
However, in computer graphics, a lot of work has been conducted for
vortex extraction, namely by Laramee et. al. [105] and Peikert et. al. [106,
107, 108, 109]. Laramee et. al. [105] uses topology based visualisation to
extract and track vortices. In contrast, Peikert et. al. [106, 107] suggest using
vector parallelism and scale space technique, with implicit nite dierencing
scheme to identify and track vortices.
21
Other notable research in computer graphics and vision using uid meth-
ods for vortex extraction was conducted by Sahner et. al. [110] using the
M
Z
-criterion [102], Cuzol and Memin [111] using probabilistic energy min-
imisation method to extract vortex from phantom satellite imagery and Zhou
et. al. [112] used uid methods with local and global smoothness constraints
to obtain scene structures for segmentation and tracking of clouds for cloud
modelling and weather prediction. A good review of current methods being
used in the eld for vortex extraction is presented in [113].
2.5 Conclusions
In computer vision, it is increasingly dicult to nd a niche in the eld
to specialise. This is especially true in the area of motion estimation, as
it is a huge eld in itself. The literature survey reported in this chapter,
namely in optical ow estimations, diusion, volumetric reconstructions and
uid methods in vision, only represents a small percentage of research in this
area.
However, there is little work in the area of non-rigid optical ow esti-
mations and regularisation techniques. Hence, the motivation to focus the
thesis in nonlinear models for smoothing of optical ows from complex and
turbulent image sequences, with scenes of non-rigid objects. Though there
are notable review publications on diusion, they are mostly specic on still
images and not on velocity elds.
Although extending optical ows to structural ows pertaining to anato-
mical organ slices is not new, there is no review on the eects of diusion on
such ows and how this would eect 3D volumetric reconstructions.
22
Furthermore, we have identied an area of optical ow estimation which
is still in its infancy, but slowly gathering pace in its level of interest by vi-
sion researchers. Using uid methods in computer graphics is quite common
nowadays. However, there is signicant lack of work in optical ows using
Navier-Stokes equations. Though in the thesis we use Navier-Stokes equa-
tions for regularisation purposes, this is viewed as a stepping stone for an
optimised optical ow estimation using Navier-Stokes equations.
It is the aim of the work described in this thesis to address some gaps in
the eld.
23
Chapter 3
Robust Hessian-based
Anisotropic Diusion
3.1 Introduction
Optical ow estimation from image sequences has been identied as an ill-
posed problem which requires a regularisation methodology as shown through
the works of Horn and Schunck [11] as well as Lucas and Kanade [3]. Quite
often the estimated optical ow is noisy and contains outliers. This is more
evident in the case of the block matching algorithm used in video compres-
sion algorithms [1, 14]. Energy based minimisation frameworks have been
dened for estimating the optical ow by taking into account various deter-
mining factors [14, 23, 24, 25]. Dierent applications such as moving object
segmentation [14, 23], tracking in image sequences and video coding quality
[1] depend on smooth optical ows.
The aim of this chapter is to analyse the performance of a set of al-
gorithms that perform robust diusion on vectorial elds. The proposed
24
algorithms combine the smoothing ability of the heat kernel with the outlier
rejection mechanisms of robust statistics algorithms. The diusion kernel is
Gaussian with the covariance matrix considered as the Hessian calculated
from the data located in a certain neighbourhood. The kernel implicitly em-
beds the local changes in the optical ow. The novelty in this section of
the research is the introduction of Hessian based diusion kernels which dif-
fuses data anisotropically in the presence of outliers. Under these conditions,
classical diusion introduces a bias in the whole data set. Robust statistics
operators are shown to improve the results provided by Hessian-based diu-
sion by rejecting outliers and by enhancing the smoothing ability of diusion
algorithms. Alpha-trimmed mean and median statistics have been consid-
ered for robustifying the diusion kernels. The robust diusion process is
extended to 3D lattices by using 3D Hessians. The proposed algorithms are
used for smoothing articial vector elds as well as optical ows estimated
from various image sequences. They combine anisotropic diusion and ro-
bust statistics for smoothing the optical ow obtained when using simple
motion estimation algorithms.
This chapter is organised as follows: Section 3.2 describes the estimation
algorithms of optical ows that have been used in the thesis. Section 3.3
outlines the application of anisotropic diusion on vector elds. Section 3.4
contains a study analysing the bias introduced when diusing outliers and
describes the proposed set of algorithms that combine robust statistics with
Hessian based diusion. Section 3.5 presents the comparative results for the
proposed methodology, while Section 3.6 outlines the conclusion of the study.
25
3.2 Optical Flow Estimation
The optical ow extracted from a video sequence is represented as a vector
eld which warps one image into another providing its changes in time [3, 11,
12]. The methodology underlying the optical ow estimation and smoothing
is described in the following subsection.
The optical ow can be represented using a Taylor expansion of a frame
from an image sequence with respect to the other frames, considering the
rst derivative and neglecting higher order derivatives [3, 11]:
I(x + x, t + t) I(x, t) +I x + I
t
t (3.1)
where I(x, t) represents an area located at x = (x, y) in frame t which is
part of the image sequence I, x represents a translation in space and t
represents the variation in time, while I =
_
I
x
,
I
y
_
and I
t
represent rst
order partial spatial and temporal derivatives, respectively. These represen-
tation are chosen to be consistent with literature within the eld of motion
estimation. If the displacements are small enough, it can be assumed that
I(x + x, t + t) I(x, t) and after dividing equation (3.1) with t yields
the constrained optical ow equation:
I V
t
+ I
t
= 0 (3.2)
where V
t
= (V
x
, V
y
) denotes the motion vector at time t. Extending the cal-
culation for multiple frames, a vector eld on a 3D lattice is obtained, where
each plane of the lattice corresponds to the motion between two consecutive
frames. Motion vectors can be initially calculated by either using gradient
methods (3.2) [3, 11] or feature matching, for example the block matching
26
algorithm [1, 14] used in video coding. The block matching algorithm relies
on the correlation between blocks from one frame and blocks from a search
region from another frame of the same image sequence as in
V
t
= arg
k,l
min
_
M

i=1
N

j=1
[I(i, j, t) I(i + k, j + l, t + 1)[
_
(3.3)
where MN is the block size and (k, l) are assumed to be inside a predened
search region (S
x
, S
y
). Consequently, for each block of pixels a motion vector
is determined representing the displacement between the coordinates of the
two blocks from the two frames as given by V
t
. The basic block matching
algorithm is shown in Figure 3.1.
Figure 3.1: Block matching algorithm.
However, motion estimation algorithms often lead to wrong decisions,
particularly in areas with constant texture and colour, where the gradient is
constant in all directions and the correlation is identical for several pixel block
27
combinations. Rotational and complex motion is only roughly approximated
by motion detection algorithms. Moreover, the image noise and changes
in illumination conditions produce motion vectors that are not consistent
with the optical ow. Due to all these causes, the resulting optical ow is
noisy in nature, displaying both slow varying noise as well as sudden large
changes. These errors in the optical ow should be smoothed out or removed
altogether.
Other optical ow estimators used for initialisation are known as gra-
dient based algorithms (3.2). Although equation (3.2) is constrained and
is now a well-posed problem, additional constraints are needed to improve
the eciency of the solution. Horn and Schunck [2] used energy minimi-
sation technique, which minimises the combined global smoothness and the
constrained intensity gradient as shown below,
c =
__

2
_
u
2
x
+ u
2
y
+ v
2
x
+ v
2
y
_
+ (I
x
u + I
y
v + I
t
) dxdy (3.4)
where u
x
, u
y
, v
x
, v
y
are rst order spatial velocity derivatives and I
x
, I
y
are
rst order spatial intensity derivatives with the Lagrangian multiplier, .
Another popular method of computation that is often used is Lucas
and Kanades algorithm [3], which implements regularisation in the pre-
computation stage of the optical ow. Their algorithm assigns weights, w to
corresponding centered pixels which are locally inuenced by the surround-
ing neighbourhood before using the gradient constraint equation (3.2). Their
implementation is shown to be
min

x
w
2
(x) [I(x, t) u + I
t
(x, t)]
2
(3.5)
28
where x = (x, y) spatial locations and is the local windowed neighbourhood.
However, as in most optical ow estimators, indiscriminate smoothing
could produce undesired eects in the resulting motion eld. The next section
explains the optical ow smoothing methodology using anisotropic diusion.
3.3 Anisotropic Diusion of Optical Flow
3.3.1 Diusion kernel
Anisotropic diusion underpins the modelling of complex processes in physics
and chemistry. The heat equation of a geometric manifold can be described
as [28, 29, 31, 114]:
V
t
(x)
t

2
V
t
(x) = 0 (3.6)
where V
t
(x) is the heat vector located at x and at time t and
2
V
t
(x)
represents the Laplacian of the vector eld which produces a tensor. This
equation assumes a given initial condition.
The general solution to the heat equation from (3.6) yields [31]:

V
t
(x) =
_
M
(
K
(x )V()d (3.7)
where is an arbitrary variable, (
K
(x ) is the heat kernel, representing a
Green function, which applies the diusion onto the manifold /and x /.
The heat kernel is the natural candidate for measuring the similarity between
two points on the same manifold, while respecting the manifold geometry
[114]. In image processing, this operation was seen as being equivalent with
29
the local convolution of an image with a kernel function [30, 31]. In the case of
real data, the heat kernel from equation (3.7) is the Gaussian kernel [30, 114].
Isotropic smoothing using PDEs in the context of images was rst for-
mulated by Koenderink [27]. In the case of images, the presence of edges
and details require an anisotropic smoothing approach. Perona and Malik
[4] proposed a smoothing function based on the Lorentzian error norm. Sim-
ilar to the statistical approach in [4], Black et. al. [7] used Tukeys biweight
function to achieve similar results as obtained by Perona and Malik [4]. More
recently though, Tschumperl e and Deriche [6] used oriented Laplacians for
the same purpose. Dierent to the mentioned methods, Weickert et. al. [34]
proposed an additive operator splitting (AOS) method which is based on the
CLMC method [33].
In contrast, a multivariate approach for nonlinear diusion was considered
in this work. Assuming that the local manifold is warped to real space, then
it can be deduced that points on the manifold are appropriately correlated
to points in real space, i.e. / IR, then the heat kernel is the Gaussian
kernel [30, 114]. In this case, the solution to the heat equation (3.6) yields:

V
t
(z
c
) =
1

4d
_
x
exp
_

1
4d
(x z
c
)
T

1
(x z
c
)
_
V
t
(x)dx (3.8)
where represents the covariance matrix, x is a location in a neighbourhood
IR, z
c
is the central location of the windowed neighbourhood and d is a
normalisation coecient. The proposed method uses a multivariate Gaussian
kernel which is based on the local Hessian information. The local Hessian
represents the curvatures of the local manifold [6, 10] and it can be used as
a feature detector [47]. The embedding of the Hessian in the heat kernel is
30
discussed in the following subsection.
3.3.2 Embedded Hessian diusion kernel
(a) Original vector eld (b) Degraded vector eld
Figure 3.2: Articial vector eld.
Figure 3.2 shows the articial vector eld that is used to evaluate which
of the covariance (), Laplacian (L) and Hessian (H) matrices to be used
in the diusion kernel function in (3.8). Figure 3.2(b) shows the degraded
eld with additive Gaussian noise of mean zero and variance
2
= 0.3. This
results in a more dicult smoothing task for the diusion kernels to recover
the original vector eld in Figure 3.2(a).
Let us consider the calculation of the following measures on a neighbour-
hood of the vector eld, with V
x
and V
y
being mean vectors of V
x
and V
y
,
respectively:
Covariance () =
N

i=1
(V
x,i
V
x
)(V
y,i
V
y
)
T
N
(3.9)
31
Laplacian (L
2D
) =
_

2
V
x
x
2

2
V
x
y
2

2
V
y
x
2

2
V
y
y
2
_

_
(3.10)
Hessian (H
2D
) =
_

2
V
x
x
2

2
V
x
xy

2
V
y
yx

2
V
y
y
2
_

_
(3.11)
The results in Figure 3.3 show the eects of Hessian (a), Laplacian (b)
and covariance (c) kernel based smoothing on the degraded articial vector
eld shown in Figure 3.2(b). Mean cosine error (MCE) of the smoothed
elds is calculated as in equation (3.26). This evaluates as to how best the
smoothed elds match the original vector eld.
The Hessian smoothed eld in Figure 3.3(a) yielded a mean cosine error
of 0.9620 after 2 iterations. The Laplacian smoothed eld in Figure 3.3(b)
yielded a mean cosine error of 0.9590 after 3 iterations and the covariance
smoothed eld in Figure 3.3(c) yielded a mean cosine error of 0.9566 after
5 iterations. The covariance matrix is a global measure of how dierent the
V
x
and V
y
components of the vector eld dier from each other. Hence,
its relative poor performance. There is only 0.3% dierence between the
Hessian and Laplacian smoothed vector elds, with the Hessian being better.
As can be observed, the Laplacian tend to oversmooth the eld between
the horizontal and vertical vectors, which the Hessian does not. Hence the
Hessian matrix is chosen to be the most suitable to be integrated into the
diusion kernel.
By embedding the local data in the Hessian difusion kernel, we can au-
tomatically direct the diusion along the main data features. On the other
hand, in homogeneous regions there is no preferred direction of smoothing
32
(a) Hessian smoothed (b) Laplacian smoothed
(c) Covariance smoothed
Figure 3.3: Smoothed articial vector elds.
and isotropic diusion can be used.
The proposed method is based on the use of a diusion kernel which
adapts itself to the local manifold variations. The local Hessian consists of an
appropriate measure of variation in the geometry of the local statistics [114]
and can be viewed as the application of a moving mask that models the
second derivative of the local manifold.
The most common way of calculating the local Hessian is by using the
local second order central dierence approximation of the second derivatives.
The eigenvalues of the Hessian matrix as given in (3.11) can be used as
33
a detector of change in the direction of the optical ow by indicating the
localisation and orientation of the moving object boundaries.
In the following analysis, the heat kernel is formulated as an adaptive
anisotropic lter that implicitly considers the local manifold variation by
means of its Hessian. The local Hessian (3.11) is embedded as the covariance
matrix in the heat kernel from (3.7). Considering that the Gaussian kernel
is a solution to the heat equation, the updated discretised and normalised
equation is:

V
t+1
kc
=

x
i
(zc)
V
t
ki
exp[(x
i
z
c
)
T
H
1
2D,c
(x
i
z
c
)]

x
i
(zc)
exp[(x
i
z
c
)
T
H
1
2D,c
(x
i
z
c
)]
(3.12)
where V
t
ki
is the vector at location i within a neighbourhood (z
c
), which
denes a symmetric region centered at the location z
c
, t denotes iteration
number and k is the frame index. The non-singularity of the local Hessian
can be enforced by using various procedures, for example by calculating its
pseudo-inverse.
3.3.3 Multiple 2D Hessian kernels
The 2D Hessian kernel from (3.12) is used to perform smoothing on motion
vector elds calculated between pairs of consecutive frames. This approach
is extended to consider multiple frames. In this instance, a 3D lattice of
parallel and equidistant vector elds modelling the optical ow from the
entire image sequence is obtained. Firstly, local smoothing is applied in the
spatial neighbourhood as in (3.12). Thereafter, the smoothing is extended by
considering consecutive vector elds on both sides of the frame k containing
34
the central location, z
c
:

V
t+1
kc
=

x
i
(zc)
V
t
ji
exp [(x
ji
z
jc
)
T
H
1
2D,jc
(x
ji
z
jc
)]

x
i
(zc)
exp [(x
ji
z
jc
)
T
H
1
2D,jc
(x
ji
z
jc
)]
(3.13)
where j = k K, . . . , k +K and 2K represents the number of frames under
consideration for smoothing.
3.3.4 3D Hessian kernel
The 2D Hessian kernel is extended to 3D in order to accommodate the spatio-
temporal variation in the optical ow. Here, the diusion process is modelled
as:

V
t+1
kc
=

x
i

3D
(zc)
V
t
ki
exp[(x
i
z
c
)
T
H
1
3D,c
(x
i
z
c
)]

x
i

3D
(zc)
exp[(x
i
z
c
)
T
H
1
3D,c
(x
i
z
c
)]
(3.14)
where the neighbourhood is dened symmetrically in 3D as
3D
(z
c
) and z
c
is the central location in the middle frame. By processing a larger amount
of data, the optical ow transitions and moving object boundaries would be
better modelled by the 3D Hessian whilst diusing the vector eld. The 3D
Hessian matrix is given by:
H
3D
=
_

xx

xy

xk

yx

yy

yk

kx

ky

kk
_

_
(3.15)
where the entries of the matrix are
xx
=

2
V
x
x
2
,
yy
=

2
V
y
y
2
,
xy
=

2
V
x
xy
,

yx
=

2
V
y
yx

xk
=

2
V
x
xk
,
yk
=

2
V
y
yk
,
kx
=

2
V
x
kx
,
ky
=

2
V
y
ky
and
kk
is
35
the frame index dierence, (V
x
, V
y
) is the given vector eld dened on a 3D
lattice and k denotes the frame index.
3.4 Robust Hessian Diusion Kernels
The directional anisotropic diusion approaches introduced by Tschumperl e
and Deriche [5, 6] embed the local data geometry in the diusion kernel
in order to avoid over smoothing of important data features. Section 3.3.2
discussed the expansion of this approach for use on vector elds. While
directional anisotropic diusion is able to smooth according to the local data
ow geometry, it is not able to identify the noise, particularly outliers. Hence,
another novel contribution of this work is the development of statistically
robust diusion framework of noisy vector elds.
3.4.1 Outlier robustness study
Consider the following one-dimensional (1D) signal:
f(x) =
_
_
_
y , x ,= K
y + M , x = K
(3.16)
This signal has an outlier of height M at location x = K, and its dis-
cretised version is displayed in Figure 3.4(a) when M = 100 and K = 10.
This signal is used as an example to mimic the presence of noise with an
overwhelming value compared to its neighbourhood. It is subsequently used
to test the eects of diusion in the neighbourhood.
Consider the implementation of expression (3.12) in 1D, the Hessian be-
comes the second derivative and the resulting diused value at location z
c
36
is:

f(z
c
) =
N/2

j=N/2
f(z
c
+ j) exp
_
4j
2
f(z
c
+ j + 1) 2f(z
c
+ j) + f(z
c
+ j 1)
_
N/2

j=N/2
exp
_
4j
2
f(z
c
+ j + 1) 2f(z
c
+ j) + f(z
c
+ j 1)
_
(3.17)
where N is the size of the diusion window and where the dierentiating
operator is approximated by central dierences. The diusion is calculated
at the center of the window which is located at z
c
= Ki. It can be observed
that all the second derivatives in the signal from (3.16) are zero except those
at the locations j = i 1, i, i + 1. After replacing (3.16) into (3.17), the
following result is obtained:

f(K i) = y +
M exp
_
i
2
2
M
_
exp
_
(i 1)
2
4
M
_
+ exp
_
i
2
2
M
_
+ exp
_
(i + 1)
2
4
M
_
(3.18)
The bias at a site located at a distance of i positions from that of the
outlier in the signal from (3.16) is evaluated as [

f(Ki) y[. The fractional

part on the r.h.s. of equation (3.18) represents the bias resulted from diusing
the outlier present in the signal from (3.16). The diusion of the outlier will
inuence several signal values in its neighbourhood. This situation could
happen when diusion is performed on images where outliers are present near
edges. The Hessian detects the edge, but not the outliers. Consequently, the
outliers will be diused in the 2D neighbourhood resulting in a bias as that
indicated in (3.18). Assuming the window size is N = 3, diusion is applied
37
repeatedly on the signal displayed in Figure 3.4(a). The results produced by
diusing the signal from (3.16) during successive iterations t = 1, 3, 6, 12
are shown in Figure 3.4(b). It can be observed that the outlier is diused in
the surrounding signal. After a certain number of iterations, the values of
the diused signal are stationary and they are always biased with respect to
the original signal which in this case corresponds to y = 10.
(a) Initial signal (b) Diused signal
Figure 3.4: The eect of diusion on outliers.
Other diusion functions such as those from [7, 34] are not able to distin-
guish between outliers and data features. However, the following subsections
introduces a new methodology that enhances the ability of anisotropic diu-
sion with an outlier rejection mechanism.
3.4.2 Median of directional Hessians kernel
Robust statistics is known for its ability to preserve edges while eliminating
outliers and have been used for image ltering [115]. Two robust statistics-
based lters, the median and the alpha-trimmed mean have been applied
together with radial basis functions for eliminating outliers from the opti-
38
cal ow [14] and for segmenting volumetric images [116], respectively. The
proposal here is to combine the performance of anisotropic diusion in de-
tecting the main data features with the ability of robust statistics algorithms
to eliminate outliers.
In the rst instance, median estimation is applied on the results provided
by directional diusion. This robust diusion approach consists of two stages.
We consider a N by N window, where N is an odd number. For each
location in this window, we dene an additional window in order to evaluate
the Hessian based diusion outputs as in (3.11) along a specic direction. All
the directions envisaged start from the central window location z
c
and are
spaced at intervals of /4 from each other. Consequently, the calculation of
the diused vectors is not done centrally as in (3.11), but in the rst instance
is evaluated at locations situated at the margins of their corresponding central
window. All such extended windows are used to calculate Hessians along
directions towards the center of the initial N by N window. The diused
vector from the center of the initial N by N window is calculated as
well. The diused vectors correspond to a specic directionally oriented
geometry of their neighbourhoods and this process considers a certain degree
of overlapping for the diusion windows. The locations where the diusion
is calculated are indicated by crosses in Figure 3.5 for N = 3. In this case,
the resulting neighbourhood is increased from 3by 3 to 5by 5 vectors.
This results a total of N
2
diused vectors including the one in the center.
The resulting diused vectors are ranked. Ranking in vectorial data can
be performed marginally, i.e. along each entry separately, or with respect to
the distance to a central data sample. In our approach, we use the marginal
39
median where all the corresponding entries are ranked separately and the
middle value is chosen for each entry [14]. The median operator is applied
onto the results produced by the directional diusions resulted as described
above:

V
kc
= Med (

V
i
, (z
c
)) (3.19)
where (z
c
) represents the neighbourhood dened by the window centered at
z
c
, containing vectors resulting from directional diusions. This algorithm
takes into account extended neighbourhoods aiming to reduce overlaps among
local estimates. It can be applied to the 2D Hessian, multiple frame 2D
Hessian calculated in Section 3.3.3 or to the 3D Hessian, calculated according
to (3.14) and as described in Section 3.3.4. The inuence of outliers will be
diused during the rst operation and eliminated during the second step of
median ltering. When applied to the 3D Hessian, the orientations for the
diusion calculation are dened in the 3D neighbourhood leading to greater
reliability in data statistics [117].
3.4.3 Alpha-trimmed mean kernel
Another robust statistics based approach is the inter-quartile averaging, also
known as the alpha-trimmed mean algorithm, which is suitable for smooth-
ing medium and long tailed data distributions [115, 116]. This method ranks
the given data and eliminates a certain percentage of data samples at the ex-
tremes of the ranked array. The aim of this method is to remove outliers and
to apply the diusion algorithm only on data that are statistically consistent
with each other. After ranking the data, the extreme vectors are eliminated.
40
Diff Diff
Diff Diff
Diff
MED
Diff
Diff
Diff
Diff
Figure 3.5: Calculation of the median of directional Hessian kernels.
When ranking is performed according to the distance from a central vector,
the alpha-trimmed mean algorithm eliminates the data which are located far
away from that central data sample [116]. The updating equation is:

V
t+1
kc
=
NN

i=N
V
t
i
exp[(x
i
z
c
)
T
H
1
c
(x
i
z
c
)]
NN

i=N
exp[(x
i
z
c
)
T
H
1
c
(x
i
z
c
)]
(3.20)
where [0, . . . , 0.5] is the trimming ratio of a ranked array of N vectors
from the neighbourhood
(
z
c
) and V
t
i
represents the i-th ordered vector at
location x
i
for iteration t. This algorithm reduces the computational cost of
the diusion per iteration by considering diusion on a reduced set of vectors.
When = 0, the diusion algorithm corresponds to the Hessian based diu-
sion from (3.12), while when = 0.5, only one vector is taken into account
corresponding to the median of the ranked array, although the calculation in
this case is dierent from the one described in Section 3.4.2. The Hessian can
41
be H
2D
from (3.11), H
3D
from (3.15) or calculated by using H
2D
on multiple
frames and averaging the outputs as described in Section 3.3.3.
3.5 Experimental Results
A comparative experimental study has been performed using the proposed
robust diusion methodology and various other diusion algorithms for sm-
oothing articial vector elds corrupted by noise as well as motion elds
extracted from real image sequences. The algorithms are denoted according
to the type of kernel that has been used for smoothing: 2DH - diusion
algorithm using 2D Hessian (3.11), M2DH - multiple frame 2D Hessian (3.12),
3DH - 3D Hessian (3.14), MED-2DH - median of 2D Hessian (3.19), MED-
M2DH - median of multiple frame 2D Hessian, MED-3DH - median of 3D
Hessian, ATM-2DH - alpha trimmed mean using 2D Hessian (3.20), ATM-
M2DH - alpha trimmed mean of multiple 2D Hessian, ATM-3DH - alpha
trimmed mean of 3D Hessian. All the vector neighbourhoods are assumed as

2D
= 3 by 3, while for multiple frames they become 3 by 3 by 4
and for 3D kernels
3D
= 3 by 3 by 3.
The other algorithms considered for comparisons are Black - Black et. al.
[7] with parameter = 1/(4 t), PM - Perona-Malik [4] with parameters
= 30 and = 1/(4 t), TD - Tschumperl e and Deriche [6] with parameter
dt = 20 and AOS - Additive Operator Splitting scheme [34] with parameters
T = 10s, maximum iterations = 1, = 0.1, = 1 and diusion timesteps =
1.
42
3.5.1 Smoothing noisy articial vector elds
Two articial vector elds have been considered. The rst vector eld,
Synthetic-1 is modelled by:
_
_
V
x
V
y
_
_
=
_
_
c s
s c
_
_
_
_
D + S R
R D S
_
_
_
_
c s
s c
_
_
_
_
x
y
_
_
(3.21)
where V
x
and V
y
are the velocity components in the x and y directions,
c = cos(), s = sin(), = 0, D = 0.8 is the dilation coecient, S = 0.05
is the shear coecient, R = 0.1 is the rotation coecient, and = 31 is the
center of the resultant ow. This vector eld models a complex variation of
zooming in and out, and is displayed in Figure 3.6(a). The second vector
eld, Synthetic-2 is created by dierentiating the following expression:
Z(x, y) = 3(1 x)
2
e
x
2
(y+1)
2
10
_
x
5
x
3
y
5
_
e
x
2
y
2

1
3
e
(x+1)
2
y
2
(3.22)
The velocity components are obtained as V =
_
Z
x
,
Z
y
_
. This vec-
tor eld has two attractors and two divergent centers and is displayed in
Figure 3.6(b).
These vector elds are corrupted by noise and independently generated
for each entry. In order to model a variety of possible corruption by noise,
both additive Gaussian and Poisson noise distributions were considered. The
Gaussian noise is commonly used as a noise model and is represented as:
((
2
) =
1

2
exp
_
v
2
2
2
_
(3.23)
43
(a) Synthetic-1 (b) Synthetic-2
0 10 20 30 40 50 60
0
10
20
30
40
50
60
(c) Gaussian
2
= 0.10 (d) Gaussian
2
= 0.10
(e) Poisson
2
= 0.25 (f) Poisson
2
= 0.25
Figure 3.6: Synthetic vector elds, original and after being corrupted with
noise.
44
where v is the random variable associated with the additive noise and
2
is the variance of the Gaussian distribution. The Poisson distribution is
characterised by a long tail distribution and can be used to contaminate
data with outliers. It is modelled by:
(
P
(
2
) =
e
(
2
)
(
2
)
v
v!
(3.24)
where v! is the factorial of v, the number of occurrences, assumed to be
an integer, and
2
> 0 is the variance of the Poisson distribution [118].
The width of the Poisson distribution is controlled by its variance
2
, which
is directly related to the percentage of outliers in data. The vector elds
modelled by equations (3.21) and (3.22) corrupted by Gaussian noise with

2
= 0.10 are displayed in Figures 3.6(c) and 3.6(d), respectively. The
same vector elds corrupted by Poisson noise with
2
= 0.25 are displayed in
Figures 3.6(e) and 3.6(f), respectively. Five dierent values for the variance
have been considered in each combination of noise distribution and data set.
The proposed algorithms as well as the diusion algorithm of Black et. al.
[7] which has been adapted for the use on vectorial data, are applied for
smoothing the noisy vector elds with the aim of trying to reconstruct the
original vector elds. These experiments has only been tested on the follow-
ing algorithms: 2DH, ATM-2DH, MED-2DH and Black [7]. For the alpha-
trimmed mean smoothing algorithm in the case of the
2D
, = 0.33 and
hence 6 vectors are eliminated from the diusion process.
The numerical results are assessed with respect to two error measures
representing the dierence between the original and the smoothed vector
elds. The error measures consist of the mean square error (MSE) and the
45
(a) 2DH (b) Perona-Malik
(c) ATM-2DH (d) MED-2DH
Figure 3.7: Examples of Synthetic-1 vector elds after corruption with Gaus-
sian noise with
2
= 0.1 after ve iterations of smoothing.
mean cosine error (MCE). The MSE is given by:
MSE =
L

i=1
(V
i


V
i
)
T
(V
i


V
i
)
L
(3.25)
where L is the total number of vectors in the given vector eld, V
i
is the
ground truth before considering the noise and smoothing, and

V
i
is the
result obtained after smoothing the noisy vector eld at location i. The
MCE measures the angular error in the orientation of the vector as in [10]
46
Method
Noise
Black 2DH ATM-2DH MED-2DH
(
2
)
MSE MCE MSE MCE MSE MCE MSE MCE
G
a
u
s
s
i
a
n
0.01 0.007 0.991 0.016 0.982 0.017 0.973 0.007 0.995
0.10 0.063 0.933 0.097 0.931 0.154 0.860 0.059 0.973
0.25 0.180 0.871 0.237 0.887 0.444 0.740 0.126 0.952
0.30 0.229 0.854 0.276 0.873 0.528 0.721 0.158 0.947
0.40 0.286 0.818 0.327 0.852 0.680 0.690 0.186 0.933
P
o
i
s
s
o
n
0.01 0.015 0.993 0.009 0.998 0.001 0.998 0.002 0.999
0.05 0.309 0.963 0.182 0.978 0.005 0.995 0.094 0.987
0.10 1.052 0.934 0.728 0.959 0.036 0.985 0.501 0.975
0.25 7.594 0.797 7.091 0.830 1.627 0.867 6.454 0.851
0.40 20.873 0.649 20.767 0.668 10.629 0.712 19.470 0.692
Table 3.1: Numerical results for Synthetic-1 data after one iteration of dif-
fusion.
and is given by:
MCE =
L

i=1
V
i


V
i
|V
i
| |

V
i
| L
=
cos(
i
)
L
(3.26)
The normalised dot product between the two vectors provides the cosine
of the angle between them, denoted as
i
. The minimum error, as indicated
by MCE between the smoothed vector eld and the original one, should
be ideally close to 1. The numerical results obtained after smoothing the
synthetic vector elds corrupted by noise, after one iteration by the given
algorithms, are provided in Table 3.1 for Synthetic-1 data and in Table 3.2
47
Method
Noise
Black 2DH ATM-2DH MED-2DH
(
2
)
MSE MCE MSE MCE MSE MCE MSE MCE
G
a
u
s
s
i
a
n
0.01 0.011 0.722 0.019 0.682 0.023 0.626 0.012 0.776
0.10 0.061 0.532 0.088 0.531 0.239 0.428 0.043 0.612
0.25 0.156 0.435 0.195 0.473 0.473 0.350 0.108 0.554
0.30 0.213 0.469 0.253 0.493 0.482 0.362 0.146 0.572
0.40 0.425 0.414 0.522 0.443 0.658 0.320 0.342 0.546
P
o
i
s
s
o
n
0.01 0.021 0.961 0.027 0.956 0.007 0.974 0.008 0.985
0.05 0.291 0.805 0.338 0.800 0.013 0.969 0.113 0.889
0.10 1.364 0.633 1.482 0.633 0.070 0.923 0.832 0.703
0.25 8.008 0.376 8.106 0.372 2.427 0.623 6.883 0.359
0.40 19.591 0.240 20.073 0.237 10.871 0.359 17.994 0.219
Table 3.2: Numerical results for Synthetic-2 data after one iteration of dif-
fusion.
for Synthetic-2 data. The best results are highlighted in bold for each data
set, noise distribution and noise variance in these tables. MED-2DH provides
the best performances when smoothing out Gaussian noise, whilst ATM-2DH
is the best kernel for smoothing out Poisson noise. Figures 3.7(a)-3.7(d)
show the results after applying diusion onto the vector eld Synthetic-1,
while Figures 3.8(a)-3.8(d) show the results after applying diusion kernels
onto the vector eld Synthetic-2. From these results, it can be observed that
despite the high level of additive noise, MED-2DH provides smooth vector
elds that are similar with the original vector elds provided in Figures 3.6(a)
and 3.6(b), respectively. From Figure 3.8(c) it can be observed that after
48
(a) 2DH (b) Perona-Malik
(c) ATM-2DH (d) MED-2DH
Figure 3.8: Articial vector eld after corruption with Poisson noise with

2
= 0.25 after ve iterations of smoothing.
smoothing using the ATM-2DH algorithm, there is clear identication of the
vector eld structure from Synthetic-2 data whilst most of the noise was
correctly cleared out.
3.5.2 Smoothing motion elds
The proposed diusion algorithms have also been evaluated on motion elds
extracted from real image sequences. The image sequences that have been
considered show a large variety of motion and a diversity of factors aecting
49
motion detection. One frame from each of the sequences Taxi, Concorde and
Fighter are shown in Figures 3.9(a), 3.9(c) and 3.9(e), while frames from each
of the sequences Clouds, Tornado and Trac are shown in Figures 3.10(a),
3.10(c) and 3.10(e). The optical ow is generated using two dierent ap-
proaches. The rst method uses the block matching algorithm (BMA) ac-
cording to (3.3), which is widely used in video compression algorithms [1].
The optical ow estimated using BMA is shown in Figure 3.9(b) for the Taxi
frame, in Figure 3.10(b) for the Clouds frame and in Figure 3.10(f) for the
Trac frame. The second method employs the Lucas-Kanade algorithm [3]
which relies on a gradient based approach for estimating the motion (3.7)
while embedding certain regularisation constraints. The results provided by
this algorithm are shown in Figure 3.9(d) for Concorde, in Figure 3.9(f) for
the Fighter and in Figure 3.10(d) for the Tornado sequence. In both motion
estimation methods and for all the image sequences, 4 by 4 pixel blocks
have been considered. It can be observed that the optical ow estimation
using block matching usually results in additional outliers when compared
to Lucas-Kanade algorithm.
A measure for assessing the ability of the smoothing algorithms is pro-
vided by the frame reconstruction accuracy when considering the smoothed
optical ow. Such a measure is widely used for assessing the eciency of
motion compression algorithms [1]. As it can be observed from equation
(3.1), we can predict an image frame by using the values of the current frame
I(x, t):

I(x, t) = I(x + t

V, t + t) (3.27)
50
(a) 5th frame from the Taxi sequence (b) Optical ow produced by
BMA
(c) 6th frame from the Concorde sequence (d) Optical ow produced by
Lucas-Kanade algorithm
(e) 173rd frame from the Fighter sequence (f) Optical ow produced by
Lucas-Kanade algorithm
Figure 3.9: Frames from three image sequences and their corresponding op-
tical ows.
51
(a) 474th frame from the Clouds sequence (b) Optical ow produced by
BMA
(c) 341th frame from the Tornado sequence (d) Optical ow produced by
Lucas-Kanade algorithm
(e) 59th frame from the Trac sequence (f) Optical ow produced by
BMA
Figure 3.10: Frames from additional three image sequences and their corre-
sponding optical ows.
52
where

V is the smoothed vector eld at location x and time t. Based on
equation (3.27) and by considering the image values in the rst frame as
known, we can predict all the other frames by appropriately varying t.
However, the prediction of the scene based on the previous frame and the
motion eld, according to (3.27), does not take into account atmospheric
conditions, changes in illumination or the perspective distortions caused by
the 3D characteristics of the scene. The frame reconstruction accuracy us-
ing motion eld based prediction is given by the peak signal-to-noise ratio
(PSNR) which is calculated in decibels as:
PSNR = 20 log
10
_
_
_
_
_
_
255MN

i
N

j
(

I(x
ij
, t) I(x
ij
, t))
2
_
_
_
_
_
_
(3.28)
where 255 is the maximum value in a greyscale image of size M by N
and we consider the prediction for a single frame (t = 1) without taking
into account the predicted frame dierence that usually carries the additional
information needed for a perfect image reconstruction. A higher PSNR corre-
sponds to a better frame prediction, consequently indicating a better motion
eld smoothing algorithm. The frame prediction method ignores occlusion
and does not consider the eect of multiple pixels converging to the same
location.
The experimental results are a snapshot of the conditions at convergence.
The condition of convergence for all the smoothing algorithms is given by:
1
L
L

i=1
(

V
n+1
i


V
n
)
T
(

V
n+1
i


V
n
) < (3.29)
53
Figure 3.11: PSNR convergence for the reconstructed frame 8 from Concorde
sequence.
where = 10
2
is the convergence threshold, L is the total number of vectors
that are smoothed and n is the iteration number. The convergence results
provided by (3.29) and represented using the PSNR, as dened in equation
(3.28), is shown for Concorde sequence in Figure 3.11, when the optical ow
is initially estimated by LK algorithm. It can be observed that most of
the proposed robust diusion algorithms smoothly converge in just a few
iterations.
In order to assess the eciency of the -trimmed mean anisotropic dif-
fusion method, we calculate the PSNR of the predicted frame using (3.28)
when smoothing the vector elds produced by either the block matching or
the Lucas-Kanade algorithms for the ATM-2DH algorithm for various win-
dow sizes and for a variety of trimming values. The results produced by
54
these tests are shown in Figure 3.12. It can be observed that the best re-
sults are obtained when using a 3 by 3 window and = 0.2, hence 4
vectors are eliminated from the diusion calculation. When increasing the
neighbourhood size, the vector elds tend to be over smoothed, eventually
losing important information from the resulting optical ow. Denitely, by
eliminating outliers, we improve the results produced by the diusion pro-
cess, although when getting closer to the median (i.e. when = 0.4), best
performance is not necessarily achieved. The results from Figure 3.12 are
consistent for all the image sequences under consideration.
Figure 3.12: Predicted frame PSNR evaluation, using ATM-2DH smoothed
vector elds when varying the alpha parameter for various window sizes.
The results provided by some of the proposed methods when applied on
55
the optical ow extracted from the given set of image sequences are shown
in Figures 3.13, 3.14, 3.15 and 3.16. Figure 3.13 displays the results for
the Taxi sequence. The Taxi sequence displays the motion of several rigid
moving objects which is mainly translational. Figures 3.13(a), 3.13(c) and
3.13(e) show the optical ow estimated between frames 3 and 5 and displayed
in Figure 3.9(b), after being smoothed using ATM-M2DH, MED-M2DH and
3DH kernels, respectively. Figures 3.13(b), 3.13(d) and 3.13(f) show the
resulting predicted frames corresponding to the smoothed motion elds. The
frame reconstruction uses the smoothed motion elds according to equation
(3.27).
Figure 3.14 shows the results obtained for the Concorde sequence. This
sequence was chosen for its complex motion characteristics such as the ro-
tational movement, turbulent air from jet thrusters, blocky artifacts from
compression and camera movement combined with a rigid moving object.
The initial optical ow was extracted between frames 4 and 6 using the
Lucas-Kanade motion estimation algorithm which is shown in Figure 3.9(d).
Figures 3.14(a), 3.14(c) and 3.14(e) show the Concorde sequence optical ow
from Figure 3.9(d) after being smoothed using 2DH, MED-2DH and MED-
3DH kernels, respectively. Figures 3.14(b), 3.14(d) and 3.14(f) show the
resulting predicted frames corresponding to the smoothed optical ow.
The best results for the other image sequences in terms of the optical ow
smoothness as well as the PSNR for the reconstructed frame are shown in
Figures 3.15 and 3.16. The optical ow from Figure 3.9(f), initially estimated
using the Lucas-Kanade algorithm from the Fighter image sequence, when
smoothed by MED-M2DH, is shown in Figure 3.16(a). The frame recon-
56
structed based on this smoothed optical ow is provided in Figure 3.16(b).
The optical ow from Figure 3.10(b), initially estimated using the block
matching algorithm from the Clouds image sequence, when smoothed by
MED-3DH is shown in Figure 3.15(a). The frame reconstructed based on
this smoothed optical ow is provided in Figure 3.15(b). The optical ow
from Figure 3.10(d), initially estimated using the Lucas-Kanade algorithm
from the Tornado image sequence, when smoothed by MED-2DH is shown in
Figure 3.16(c). The frame reconstructed based on this smoothed optical ow
is provided in Figure 3.16(d). The optical ow from Figure 3.10(f), initially
estimated using the block matching algorithm from Trac image sequence,
when smoothed by MED-2DH is shown in Figure 3.15(c). The frame recon-
structed based on this smoothed optical ow is provided in Figure 3.15(d).
As it can be observed from these gures, the main motion structure is better
recovered after smoothing the initially noisy vector elds.
The PSNR results when tracking several frames from Taxi and Concorde
sequences are shown in Figures 3.17(a) and 3.17(b), respectively. These plots
prove that the observed results are consistent for all the frames from the
image sequences under analysis despite the various motion characteristics
in these image sequences. It should be noted here that for both graphs
in Figure 3.17, the diusion lters that performed the best compared to
the other four kernels plotted are robust Hessian based kernels that uses
information from multiple elds. This shows that an object movement path
in predicted frames is traced more eectively using prior information from
previous image frames/vector elds.
Tables 3.3 and 3.4 show the comparison between the dierent diusion
57
(a) ATM-M2DH smoothed ow (b) Predicted frame 5 from (a)
(c) MED-M2DH smoothed ow (d) Predicted frame 5 from (c)
(e) 3DH smoothed ow (f) Predicted frame 5 from (e)
Figure 3.13: Smoothed optical ows, initialised using BMA and the resulting
predicted 5th frame of the Taxi sequence.
58
(a) 2DH smoothed ow (b) Predicted frame 6 from (a)
(c) MED-2DH smoothed ow (d) Predicted frame 6 from (c)
(e) MED-3DH smoothed ow (f) Predicted frame 6 from (e)
Figure 3.14: Smoothed optical ows, initialised using LK and the resulting
predicted 6th frame of the Concorde sequence.
59
(a) Smoothed ow from Clouds (b) Predicted 474th frame from
using MED-3DH kernel ow smoothed by MED-3DH
(c) Smoothed ow from Trac (d) Predicted 59th frame from
using MED-2DH kernel ow smoothed by MED-2DH
Figure 3.15: Results when applying optical ow smoothing for the image
sequences considered in this study when the optical ow has been initialised
using BMA.
algorithms, i.e. proposed set of algorithms is considered as well as other dif-
fusion algorithms such as Perona-Malik (PM) [4], Black [7], additive operator
splitting (AOS) [34], as well as Tschumperl e and Deriche (TD) [6]. The basis
for comparison is the PSNR of the reconstructed frames at convergence. The
results from Table 3.3 are obtained when smoothing the optical ow gener-
ated by the block matching algorithm (BMA), while those from Table 3.4
60
(a) Smoothed ow from Fighter (b) Predicted 173th frame from
using MED-M2DH kernel ow smoothed by MED-M2DH
(c) Smoothed ow from Tornado (d) Predicted 341th frame from
using MED-2DH kernel ow smoothed by MED-2DH
Figure 3.16: Results when applying optical ow smoothing for the image
sequences considered in this study when the optical ow has been initialised
using LK.
are achieved when smoothing the motion eld which is initialised by Lucas
and Kanade (LK) approach [3]. In each of these tables, we also provide the
PSNR of the reconstruction error when no smoothing is performed, i.e. when
using the optical ow calculated using BMA and LK respectively.
For each method and each image sequence, the number of iterations nec-
essary to achieve the convergence according to (3.29) is provided. As it can
61
(a) Taxi sequence
(b) Concorde sequence
Figure 3.17: PSNR of the predicted frame when tracking scene change in two
image sequences for the best ve diusion methods when the convergence
criterion is set to < 10
1
.
62
Method Taxi Concorde Fighter Clouds Tornado Trac
dB No dB No dB No dB No dB No dB No
BMA 17.72 15.78 15.52 15.40 21.39 13.21
PM 19.99 4 18.54 2 18.35 2 19.52 2 22.46 2 15.12 2
Black 20.05 4 18.54 2 18.35 2 19.52 2 22.46 2 15.13 2
2DH 21.55 8 18.98 4 18.58 3 19.33 3 21.70 2 14.53 2
M2DH 21.20 5 19.48 2 19.57 3 17.85 2 22.14 2 14.13 2
ATM-2DH 21.38 9 18.90 10 19.12 8 20.24 6 23.02 4 14.81 4
MED-2DH 22.17 6 19.59 3 20.18 3 21.21 2 23.45 2 16.93 2
ATM-M2DH24.42 1 17.52 1 16.10 1 16.49 1 23.15 1 13.64 1
MED-M2DH22.16 5 20.80 2 20.94 2 21.08 2 24.16 2 17.21 2
3DH 21.46 7 18.94 4 18.98 4 19.35 3 21.83 2 14.44 2
ATM-3DH 21.79 1 16.84 1 15.61 1 15.80 1 21.83 1 12.29 1
MED-3DH 22.25 6 19.59 3 20.18 3 21.66 2 23.34 2 16.83 2
AOS 20.11 5 18.34 1 22.26 4 18.97 1 22.93 2 15.96 2
TD 22.91 83 12.55 3 14.99 6 12.40 4 18.88 6 12.70 7
Table 3.3: PSNR (dB) of the reconstructed frame using the smoothed optical
ow with the number of iterations (No) necessary to reach convergence for
each method and in the case of each image sequence when the motion eld
is initialised using BMA.
be observed, the number of iterations necessary to achieve the convergence
varies according to the method adopted and depends on the image sequence
as well. It is observed that the robust diusion algorithms usually require
fewer iterations in order to achieve convergence when compared to the classi-
63
Method Taxi Concorde Fighter Clouds Tornado Trac
dB No dB No dB No dB No dB No dB No
LK 20.46 14.67 16.78 17.11 21.59 15.41
PM 21.74 2 17.11 4 19.10 2 19.65 2 23.93 2 17.40 2
Black 21.74 2 17.21 4 19.10 2 19.65 2 23.93 2 17.40 2
2DH 20.50 1 19.27 10 19.34 4 19.25 2 23.11 2 15.82 1
M2DH 20.42 2 18.22 4 19.42 6 18.70 2 21.51 2 15.74 2
ATM-2DH 20.28 3 19.51 14 19.25 11 20.48 8 23.42 5 15.13 3
MED-2DH 22.09 2 20.30 8 20.78 4 22.12 3 25.13 2 18.67 2
ATM-M2DH 19.00 1 16.25 1 16.38 1 16.58 1 20.14 1 14.20 1
MED-M2DH22.19 2 19.22 4 21.36 6 20.43 2 22.48 2 17.80 2
3DH 20.50 1 19.14 10 19.31 4 19.24 2 23.25 2 15.84 1
ATM-3DH 19.08 1 16.07 1 16.18 1 16.08 1 20.06 1 13.27 1
MED-3DH 22.15 2 20.34 8 20.82 4 21.99 3 25.00 2 18.69 2
AOS 22.18 1 16.86 2 19.08 1 20.81 1 24.83 2 17.76 1
TD 19.88 8 17.92 30 17.85 12 16.02 6 20.64 10 14.77 100
Table 3.4: PSNR (dB) of the reconstructed frame using the smoothed optical
ow with the number of iterations (No) necessary to reach convergence for
each method and in the case of each image sequence when the motion eld
is initialised using LK.
cal diusion algorithms. The best PSNR of the frame reconstruction is high-
lighted for each image sequence and initialisation. The initial optical ow
provided by the Lucas and Kanade (LK) algorithm provide better PSNR for
the frame reconstruction than the ones given by the block matching algo-
64
rithm (BMA). It is observed that MED-2DH, MED-M2DH and MED-3DH
provide the best results in terms of smoothing and reconstruction ability.
These results show clear improvement when combining diusion and robust
statistics as well as when considering the motion from several consecutive
frames, according to the results from Tables 3.3 and 3.4. The smoothing re-
sults provided by the robust diusion algorithms are better when the optical
ow is complex, for example when the optical ow represents the motion of
several objects which move in a variety of ways (including swirling motions)
or when representing uids in motion. Frame prediction can be further im-
proved by inpainting the uncovered areas in the predicted frames using the
neighbourhood information.
3.6 Conclusions
A set of robust diusion algorithms have been applied for vector eld smoo-
thing. The proposed methodology is tested on optical ow, when the initial
motion estimation is noisy due to various factors. The diusion kernel is
multivariate Gaussian and embeds the local Hessian as its covariance ma-
trix. This type of kernel ensures that smoothing occurs along the motion
eld structure, thus preserving the moving objects borders and the main
ow structure. The extension from 2D to 3D Hessian based Gaussian ker-
nels considers the temporal information from multiple frames. The analysis
has shown that following repetitive diusion, outliers from data are spread
around introducing a bias in the resulting diused signal. Robust statistics
algorithms such as the inter-quartile averaging and the marginal median are
employed together with the diusion kernels for removing the outliers and for
65
enhancing vector smoothing. The algorithms are applied onto articial vec-
tor elds and onto the motion elds extracted from various image sequences.
The initial motion elds are generated using the block matching algorithm
and the Lucas-Kanade algorithm. The proposed robust statistics based dif-
fusion provides clear improvements over classical diusion algorithms. The
improvements are evident when dealing with complex motion elds which
are noisy. Motion eld smoothing can be used as a processing module for
various systems such as motion estimation and segmentation, tracking and
classication of moving objects as well as in video compression systems that
rely on motion based frame prediction.
66
Chapter 4
3D Volumetric Interpolation
from Structural Flows
4.1 Research Objective
Chapter 3 described the use of robust Hessian based kernels to perform diu-
sion on optical ows from synthetic and real image sequences. It was thought
that the research needed to be more diverse and not restricted to regularising
motion from video sequences. Hence, the diusion kernels have been tested
on structural ows obtained from 3D volumetric images representing series
of sparse medical images of soft and hard tissue slices. As a result of the
work in improving the recovery of structural ows, the research is expanded
to include synthesising intermediary slices and 3D volumetric visualisation
of the object in the slices.
67
4.2 Introduction
This chapter focusses on robust structural ows as a methodology to model
the morphing between consecutive slices in sequences of cross-sectional im-
ages. The resulting smoothed structural ows are applied to volumetric
image interpolation with the aim to reconstruct 3D shapes. The methods
considered for volumetric image interpolation can be categorised [58] into
shape-based [55, 59, 70] and grey-level based [69]. A method that considers
the grey-level information as a surface followed by interpolation and there-
after collapse it back to a slice was proposed by Grevera and Udupa [119].
In [55] the reciprocal morphing of one slice into the next one was performed
using two mathematical morphology operators.
Vector elds have been used to represent the optical ow in image se-
quences [1, 14, 120]. In this case, the vector elds show the temporal warping
of one video frame into another. In the medical imaging context, optical ow
has been used to model the movement of the heart [68, 121], registration of
cross-sections from the brain [122], deformations of skin tissue [123] as well
as changes in heart shape using diuse tensor magnetic resonance images
(MRI) [79].
The proposed method models the variation in the internal structure rep-
resented in volumetric images. Inline with the block matching algorithm [1],
each image slice is split into blocks of pixels. The method uses correlation
to decide the correspondence between each block of pixels from one slice to
the next [14]. Variations of the block matching algorithm have been used
for registration in medical images as described in [69, 79]. In this thesis, the
proposed dual block matching algorithm (DBMA) models the structural ow
68
through the volumetric image by considering matching in both directions of
the ordered slices. This algorithm provides two structural ows located on a
regular 3D lattice. Each vector from the rst structural ow represents the
correspondence of a set of voxels from one slice to the next one, while the
second vector eld models the reverse structural ow. However, the lack of
contrast or missing data can lead to erroneous matches and unsmoothness in
the resulting dual structural ows. In [14], a reliability coecient was used
to model the condence of the block matching estimates in video sequences.
In order to overcome the above mentioned problems, this study has de-
veloped a methodology that combines the advantages of two dierent ap-
proaches: anisotropic diusion and robust statistics. The Hessian represents
second derivatives and is known for its capability to model shape geome-
try in images [124]. In medical imaging, the local Hessian has been used
for feature localisation in [47] and for segmenting blood vessels [125]. The
anisotropic kernel functions developed in Section 3.3 and Section 3.4 are used
for automatically detecting signicant changes and smoothing accordingly
the structural ow.
4.3 Volumetric Image Interpolation
The proposed methodology aims to obtain structural ows from sparse med-
ical datasets. These ows are then smoothed by using robust diusion lters
embedding the Hessian of the local data. Once smoothed ows are obtained,
they are used to reconstruct intermediary slices between the images of the
original sets. This is illustrated in Figure 4.1 on a set of images showing
cross-sections of a tooth. In the Figure 4.1, the initial slice is Slice 17 and
69
the reference slice is Slice 19. The aim is to reconstruct intermediate slices
between the two original images. The arrows depict the direction of the vol-
umetric image stacking. In between Slice 17 and Slice 19, the intermediate
structural ow and its corresponding reconstructed image slice is shown. As
in [72], the slices (original and reconstructed) are stacked to reproduce a 3D
volume that appropriately represents the original 3D object. The following
sections describe the proposed framework.
Figure 4.1: An illustration of intermediate slice reconstruction.
70
4.4 Structural Flow Initialisation
In volumetric images, such as those provided by MRI, CT or by other means
[55], a set of image slices are provided, each representing a cross-section
through a volume. Vectorial elds have been used to represent the opti-
cal ow in image sequences as well as to model deformations, particularly
in medical imaging [68, 79, 121, 122]. One well known method to initialise
the optical ow is the block matching algorithm [14]. This algorithm nds
the best correlation between two blocks of pixels from a reference image
and an initial image, respectively, such that they have maximum correla-
tion. The equation implementing BMA is provided in equation (3.3). The
algorithm searches for the best match in a predened search region from the
reference image. Other registration methods use the gradient for nding cor-
respondences between regions of interest in medical images [69, 79]. In areas
that have similar texture or contain widespread repetition of similar features,
BMA produces erroneous decisions, thus resulting in noisy ows [14, 120].
The proposed algorithm for producing structural ows is called the dual
directional block matching algorithm (DBMA) and produces two vector ows
that are oriented opposite to each other with respect to the ordering of slices
in the volume. It is assumed that any specular, additive or distributive noise
has been removed from the background for the ease of estimation. Each slice
is split into rectangular blocks and the best matching (highest correlation)
is sought for each block by comparing its grey-level values with those of
blocks within a search area from the reference slice, as shown in Figure 4.2.
In the matching process, only segmented foreground of the object (tooth)
is considered. Unlike BMA, DBMA estimates two structural ows, forward
71
and backward respectively, each oriented in one or the other direction along
the main axis of the object structure. An example of dual structural ows
is shown in Figure 4.3 for a set of cross-sectional image slices representing a
tooth.
Figure 4.2: Block matching process in DBMA.
In the example shown in Figure 4.3, structural ows are computed be-
tween Slice 17 and Slice 19 of the Incisor tooth dataset [55]. For the rst
structural ow (forward direction), matching from slice t to t + 1 results in
the vector eld V
1
, while for the second, the reverse matching is computed
from t +1 to t resulting in the vector eld V
2
[117]. For displaying purposes,
the total ow, as shown in Figure 4.3(c) is obtained by adding forward and
reverse ows, i.e. V
1
+ V
2
. This ow represents the estimated deforma-
72
(a) Forward Flow, V
1
(b) Reverse Flow, V
2
(c) Combined forward and reversed ows
Figure 4.3: DBMA structural ows.
tion that took place between Slice 17 and Slice 19. Although edges of the
structure have been captured reasonably well, the internal structure remains
noisy.
Similar to the block matching algorithm used for estimating motion in
image sequences, the search area from the reference slice has the same centre
as the block region from the initial slice. However, due to the shape variation,
a large part of the search region may consist of the background, i.e. without
useful information for the structural ows. Careful consideration of the block
size is needed, especially for locating edge correspondences. An example of
73
matching errors which occurs at the boundary of the object is shown in
Figure 4.4.
Figure 4.4: Bad edge matching.
Perfect matching against the background is lost or diminished by reduc-
ing the proportion of zeros in the pixel block containing the object boundary
considered. It is important, however, to retain the object shape information.
Therefore, it can be said that a trade-o needs to be achieved between back-
ground matching and retaining the object shape as illustrated in Figure 4.3.
Figure 4.5 presents the pseudocode of the growing algorithm. The pseu-
docode shown is a subset of the main structural ow estimation program.
The input to the function GrowingRegion consists of the intensity blocks
from the original and reference slices. Line 1 of the pseudocode checks
whether the blocks are dierent, i.e. there are no matching correspondences,
by checking the number of non-zeros present in the blocks. If this state-
ment is TRUE, then the variables are set in lines 3 to 6. HopX and HopY
represent the hopping distance between blocks. NewBlk is taken to be the
array of imin NewHopX : imax + NewHopX in the x-direction and
74
1 if (nnz(BlkOrig) >= RegionSize) &&
(nnz(BlkRef) < RegionSize)
2 then
3 get imin,imax,jmin,jmax from main program
4 NewHopX HopX
5 NewHopY HopY
6 BoundaryBlk NewBlk
7 while (NoMatchingBlocks)
8 do
9 NewHopX NewHopX + HopX
10 NewHopY NewHopY + HopY
11 Length |NewHopX, NewHopY |
12 RegionSize (1.75 Length)/2
13 perform matching in the new search region
Figure 4.5: GrowingRegion(BlkOrig, BlkRef): Pseudocode of growing al-
gorithm used for regions of object boundaries, where nnz() is a function
that nds the number of nonzero pixels in a block.
jmin NewHopY : jmax + NewHopX in the y-direction. The condition
NoMatchingBlock in the while loop checks the presence of shape or edge in-
formation of the object within BoundaryBlk in the corresponding reference
slice. If no such information is found, i.e. the condition is TRUE, then the
size of the block is expanded as shown in lines 9 and 10. Once the edge of
the shape is found within the reference slice, the shortest path between the
initial point and the object boundary is traced. This distance (Euclidean) is
75
(a) Initial search block (b) New search block
Figure 4.6: Visualisation of growing algorithm.
then used in line 12 to obtain a new search region. A value of 1.75, found
by experimentation, is used as a scaling constant. The new search region is
then fed back into the main program to re-estimate the structural ow.
Another problem that may arise, as it can be observed in Figure 4.3, is
attributed to the boundary structure of the object. This is an interesting
problem since matching between boundary regions in the initial slice and
background at the same pixel location in the reference slice will lead to
wrong ow estimation. Therefore, a growing algorithm is implemented within
DBMA, which is similar to mathematical morphology method employed in
[55]. The functioning of this algorithm is illustrated in Figure 4.6 on two slices
of the Incisor data set. With reference to Figure 4.6, a growing algorithm
is applied by progressively increasing the size of the search region as shown
in Figure 4.6(a), until a part of the object contained in the reference slice is
detected as shown in Figure 4.6(b). Then, the matching proceeds until the
best correlated block is found. For the second structural ow, the algorithm
76
is applied in a reverse manner for the two slices, seeking the best correlations
for blocks from the reference slice from inside a search region dened in the
initial slice. Due to the variations in the circumstances of choosing the search
region each time from a dierent reference image, the two structural ows are
not identical in the absolute value. While DBMA improves the modelling of
feature correspondences by creating a better representation of the variation
in the volumetric image, it also results in noisy vector elds in the same
way as BMA does. The following Section 4.5, addresses the smoothing of
structural ows.
4.5 Smoothing Structural Flows
Why is smoothing necessary for structural ows? As it can be observed in
Figure 4.3, the resultant ows are quite noisy. If these ows were to be
used for interpolating intermediate slices and hence reconstruct the 3D vol-
ume, it is more than likely that the results would not be very good. This
is because unsmoothed estimations can have detrimental inuence on recon-
structed data. Hence, it is important to lter the estimations for obtaining
more accurate reconstructions. Basically, noisy (corrupted) ows lead to bad
interpolations and smoothed ows lead to more reliable outcomes.
The process of smoothing structural ows takes place in a similar manner
to that of the optical ows from image sequences. The smoothing lters used
are those developed in Section 3.3 and Section 3.4. The proposed Hessian-
based diusion kernels are able to either reduce or remove noise in order to
obtain a better representation of the inner and outer volumetric structure.
The normalised updating kernel for smoothing structural ows is given as
77

V
t+1
kc
=

x
i
(zc)
V
t
ki
exp[(x
i
z
c
)
T
H
1
2D,c
(x
i
z
c
)]

x
i
(zc)
exp[(x
i
z
c
)
T
H
1
2D,c
(x
i
z
c
)]
(4.1)
where V
t
ki
is the vector at the location i within a neighbourhood = 3by3,
t denotes the iteration number, k is the slice number, and z
c
represents the
central location of the neighbourhood. For comparative analysis, experi-
ments were carried out for smoothing using 2DH (3.12), ATM-2DH (3.20)
and MED-2DH (3.19) Hessian based kernels, which are provided in Chapter
3.
It is worth pointing out that smoothing is not conducted on the full
structural ow for an intermediate slice as it would be expected. As shown
in Figure 4.7 and described in the next section, smoothing is performed on
the forward and reverse ows separately. This ensures that the structural
ow estimations of the object deformations are modelled reliably throughout
the whole reconstruction process.
4.6 Slice Interpolation
In many applications, particularly in the medical eld, it is essential to inter-
polate image slices in order to have an appropriate volumetric representation
[55, 58, 59, 69]. The proposed model uses smoothed structural ows to in-
terpolate additional slices in between the cross-sectional slices of the existing
set. Volumetric image information is interpolated in both shape structure and
grey-level information along the structural ows. The proposed intermediary
slice interpolation algorithm is presented in the chart shown in Figure 4.7.
With reference to Figure 4.7, structural ows (Flow 1 and Flow 2) are
78
Figure 4.7: The slice interpolation owchart.
computed in the forward and reverse directions, respectively, as described
in Section 4.4. The main purpose of computing the structural ows in this
way is to estimate the deformations from one slice to the next smoothly.
This is essential if the information of object structure from both slices are
to be used for intermediate slice reconstruction. The ows model the shape
79
morphing as well as the intensity variation from one slice to another (Slice
1 and Slice 2) and consist of a 3D representation of the variation in the
volumetric shape [117]. Using this framework, up to several slices can be
generated along both, forward and backward ows, between the two given
slices. Consequently, the interslice dierence is reduced by
1
n 1
. As it
can be seen from Figure 4.7, the structural ows interpolate the deformation
between the given pair of slices. In order to reconstruct intermediary slices,
a proportion of these ows is considered for slice reconstruction along each
ow. For example, in order to reconstruct the rst slice from a total number
of n = 20 slices between pairs, the corresponding forward ow will be 1/20th
of Flow 1 and the reverse ow will be 19/20th of Flow 2. This is appropriate
since the intermediary slice to be reconstructed is immediately adjacent to
the original Slice 1, representing the volumetric image information located
at quite a distance away from Slice 2. It should be noted that during the
intermediary slice reconstruction stage, image slices obtained by B-splines
[122] or other methods [55, 56, 60, 61, 64, 77, 119] can also be used.
During the smoothing stage, the ows are independently smoothed us-
ing the diusion kernels proposed and described in Section 3.3 and Section
3.4. The reconstruction of each part of the intermediary slice consists of the
contour, the texture and other features that follow the direction of the struc-
tural ows in the volumetric image. Intermediary slice parts are produced
by displacing blocks of pixels along both structural ows from initial slice
pairs. The resulting values from the two structural ows are combined in the
last step in order to generate the interpolated slice. For areas that have ob-
ject grey-level assignments for both structural ows, the resulting grey-level
80
is decided by proportionally averaging the two values. Therefore, the kth
intermediary slice between the image slicess I
1
and I
2
is generated by the
following equation :
I
1,k
(x) =
kI
1
(x +

V
1
) + (n k)I
2
(x +

V
2
)
n
(4.2)
where x is the location of the interpolated result,

V
1
and

V
2
are the smoothed
vector elds modelling the warping along the forward and reverse ows, and
I
2
corresponds conventionally to I
1,n
from a total of n slices. In the case
where there is only one data assignment from one of the structural ows
while according to the reverse structural ow there is no assignment, the
positive decision has priority. Results on the eects of diusion on structural
ows and analysis are presented in the following section.
4.7 Experimental Results
The proposed methodology has been applied on real volumetric medical im-
ages. The given data sets are composed of sequences of digitised cross-
sections of medical data volumes representing both hard and soft tissue.
Three datasets are used representing an incisor
1
, a humerus bone
2
(part of
the upper arm) and an iliac bone
2
(part of the hip joint), respectively. An-
other three data sets, comprising of a sheeps heart
3
, CT scan of a female
chest
3
and MRI of a knee
3
have been used. An example of original slices, one
1
The Incisor was used in [55]
2
The image datasets used in this experiment were from the Laboratory of Human Anatomy and Embryology,
University of Brussels (ULB), Belgium. http://isbweb.org/data/vsj/index.html
3
Obtained from The Volume Library at http://www9.informatik.uni-erlangen.de/External/vollib/
81
for each data set is shown in Figure 4.8. In Table 4.1, the technical details
of these image slices are provided.
(a) Incisor (b) Humerus bone (c) Iliac bone
(d) Sheeps heart (e) Female chest (f) Knee
Figure 4.8: Sample slices of the data sets.
We have initialised the structural ows using the DBMA (producing two
ows) as explained in Section 4.4 and Lucas-Kanade algorithm [3] producing
a single ow. The proposed vector smoothing methodology is applied on all
these structural ows.
The rst dataset, which has also been used in [55], contains 22 slices
and represents an incisor. The slices have been obtained by mechanical
slicing followed by digitisation. After segmenting the tooth body from the
background as well as its root canal, the slices are aligned using a semi-
automatic procedure.
Three slices of the incisor sequence are shown in Figures 4.9(a), 4.9(b)
82
Data Set In-plane In-plane No. Slice
grid sizes dimensions slices spacing
(mm) (mm)
Incisor 500x500 12.40x12.40 22 2.00-2.20
Humerus 512x512 180.00x180.00 401 0.50-1.00
Iliac 512x512 250.00x250.00 260 0.50-1.00
Sheeps Heart 352x352 1.00x1.00 256 1.00
Female Chest 384x384 1.00x1.00 240 1.00
Knee 512x512 0.25x0.25 87 1.50
Table 4.1: Summary of the slice dimensions and voxel sizes for the dierent
data sets.
and 4.9(c). As it can be observed, these slices correspond to cross-sections
through the tooth in the region where the root canal emerges. The for-
ward structural ow using DBMA, calculated between Slice 17 and Slice 19
is shown in Figure 4.10(a), while Slice 18 reconstructed using this struc-
tural ow is displayed in Figure 4.10(b). The structural ows resulted from
smoothing by 2DH kernel, Perona-Malik [4], ATM-2DH kernel, Black et. al.s
algorithm [7] and MED-2DH are displayed in Figures 4.10(c), 4.10(e), 4.11(a),
4.11(c) and 4.11(e), respectively. Reconstructed slices corresponding to Slice
18 after using DBMA ows smoothed by various algorithms are shown in
Figures 4.10(d), 4.10(f), 4.11(b), 4.11(d) and 4.11(f).
As it can be observed from these gures, all the diusion based algorithms
improve the initial results provided by DBMA structural ows. The direc-
tions of morphing one slice into another are captured well by the structural
ows. The structural ows smoothed by PM [4] and Black [7] algorithms
83
(a) Slice 17 (b) Slice 18
(c) Slice 19
Figure 4.9: Sample slices from the Incisor data set.
are noisier than the structural ows smoothed by the proposed diusion al-
gorithms. The robust diusion algorithms provide smooth structural ows
while eliminating the inuence of outlying vectors. The root canal is al-
most completely closed in the reconstructions provided by 2DH, ATM-2DH
and MED-2DH smoothed structural ows (as it should be according to the
ground truth slice from Figure 4.9(b)). The best visually assessed recon-
84
(a) DBMA ow (b) DBMA reconstructed
(c) 2DH smoothed (d) 2DH reconstructed
(e) PM smoothed (f) PM reconstructed
Figure 4.10: Structural ows and reconstructed slice of an Incisor.
85
(a) ATM-2DH smoothed (b) ATM-2DH reconstructed
(c) Black smoothed (d) Black reconstructed
(e) MED-2DH smoothed (f) MED-2DH reconstructed
Figure 4.11: Further results for the Incisor.
86
struction that also provides the most compact reconstruction of this slice is
that provided by the MED-2DH smoothed structural ows.
As shown in Figures 4.10 and 4.11, the structural ow is computed be-
tween Slice 17 and Slice 19 with the aim of reconstructing Slice 18. Though
it is assumed that the slices have equal distance between each other, in re-
ality this is not always the case since the results prove that the middle slice
reconstructions are not always identical to the original slices. It should be
noted that the main cause of this dierence in the reconstructions is that
there is signicant shape variation from one slice to another from the data
set. If the data set is not sparse, then bad shape reconstructions would be
nullied. The focus however is to observe the eects of smoothed structural
ows on intermediary slice reconstructions.
Figure 4.12 shows the 3D volume visualisation of the entire set of 420
slices obtained through interpolation for structural ows smoothed by all six
algorithms. This means that a total of 20 slices are interpolated between pairs
of original slices, including the originals. A low value indicates the volume is
more likely to appear squashed along the central axis, while a large value will
indicate the volume to appear elongated. It is necessary to obtain a balance
in order to obtain a suitable volume. The number of interpolated slices will
depend on the volume to be reconstructed. From the results obtained, MED-
2DH provides the smoothest surface for the 3D incisor reconstruction closely
followed by the 2DH kernel and DBMA (reconstruction based on unsmoothed
ows), while PM and Black does not provide very smooth shapes.
Figure 4.13 provides numerical results for reconstructing the middle slice
for the whole Incisor data set when skipping one slice at a time and aiming
87
(a) DBMA (b) Perona-Malik (c) Black
(d) 2DH (e) ATM-2DH (f) MED-2DH
Figure 4.12: 3D Incisor reconstructions.
to reconstruct it from its two neighbouring slices. Figure 4.13(a) provides
the shape representation accuracy by evaluating the percentage of estimated
pixels that are not correctly placed in the estimated slice when compared to
the original from the initial data set. This percentage of error, is calculated
88
as
=
N

i
i
o

N

j
j
p
N

i
i
o
100 I
o,i
, I
p,j
,= 0 (4.3)
where N is the size of the data, i
o
represent the spatial position in I
o,i
where
the object structure is represented and j
p
represent the spatial position in
I
p,j
where the estimated object structure is represented. In equation (4.3),
the percentage of pixel error is obtained by deducting the sum of correctly
placed nonzero pixels in the estimated slice from the total number of nonzero
pixels in the original slice over the sum of the nonzero pixels in the original
slice multiplied by 100. Lower percentage of error shows a higher degree of
accurate reconstruction. From the plot in Figure 4.13(a), it can be observed
that the robust Hessian based kernels perform better at correctly identifying
positions for new intensity pixels when compared to the other kernels. It can
be also observed that sudden changes in the graph generally represent major
dierences between slices, i.e. large shape deformation or sharp changes in
grey-level intensity. The estimation of the structural ows is based on the
assumption that the slices are equidistant, which is not always the case, par-
ticularly in this experiment where the incisor had been mechanically sliced.
BMA [1] is the classical block matching algorithm, using a single struc-
tural ow for evaluating the displacement between two consecutive slices
instead of the dual structural ow as in DBMA. Figure 4.13(b) evaluates the
grey-level reconstruction using various diusion kernels by calculating the
peak signal-to-noise ratio (PSNR) between the estimated object/slice and
the original slice. It can be observed from Figure 4.13(b) that the recon-
89
(a) Shape reconstruction error rate
(b) Peak signal-to-noise ratio (PSNR)
Figure 4.13: Accuracy of the middle slice reconstruction considering both
shape structure and grey-level.
90
struction results for the Hessian based kernels are not very good when the
reconstructed slice has to recover signicant morphological change between
two consecutive slices.
However, the results are misleading. With BMA and DBMA, although
the shape deformation is not estimated accurately, the intensity value at cor-
rectly placed pixel matches the original, hence the high PSNR value. On
the other hand, the Hessian based kernels do the opposite. The shape defor-
mation is modelled accurately, but the grey-level reconstruction is somewhat
poor. The Perona-Malik and Black kernels are somewhere in the middle.
Furthermore, the estimation of the structural ows are based on the assump-
tion that the slices are equidistant, which in reality is not, as can be observed
in Figure A.1. The higher the PSNR, the closer the slice is to the absolute
middle, while low PSNR would reect the approximate distances of the real
slices. The PSNR between the original (ground truth) and the reconstructed
slice is calculated in decibels as:
PSNR(k+1) = 20 log
10
_
_
_
_
_
_
255M

i=1
(

I
k,k+2
(x
i
) I
k+1
(x
i
))
2
_
_
_
_
_
_
(4.4)
where M is the number of pixels located in the foreground at x co-ordinates
in both the original k +1th frame, I
k+1
, and the predicted

I
k,k+2
frame. The
predicted frame of

I
k,k+2
is reconstructed from the structural ows between
the k and k + 2 frames. BMA and DBMA provide quite good PSNR recon-
struction values but on the other hand they are not able to appropriately
model the variation in the shape according to the plot in Figure 4.13(a).
This is due to the fact that both BMA and DBMA rely on maximising the
91
correlation between blocks of pixels which results in a procedure ideal for
maximising the reconstruction PSNR. This is actually the main reason why
BMA is used in predictive based coding currently embedded in video coding
algorithms.
Diusion based kernels, on the other hand, provide good shape recon-
structions while their grey-level restoration is not much worse than that pro-
vided by BMA and DBMA. The improvement provided by DBMA over BMA
in terms of reconstruction error is clear from the plot in Figure 4.13(a). ATM-
2DH gives the best shape reconstruction results closely followed by 2DH, PM
and MED-2DH algorithms for this 3D volume data.
Another data set that has been used for comparison purposes is the knee
MRI data set. Three slices of the knee sequence are shown in Figures 4.14(a),
4.14(b) and 4.14(c). In this case, these slices correspond to transversal-
sections through the knee and display both soft and hard human tissue.
The initialisation for this data set computed between slices 30 and 32
using the Lucas-Kanade [3] algorithm is shown in Figure 4.15(a), while slice
31 reconstructed using this structural ow is displayed in Figure 4.15(b).
The structural ow resulted from smoothing by 2DH kernel, Perona-Malik
[4], ATM-2DH kernel, Blacks algorithm [7] and MED-2DH are displayed in
Figures 4.15(c), 4.15(e), 4.16(a), 4.16(c) and 4.16(e), respectively. Recon-
structed slices corresponding to slice 31 after using LK ows smoothed by
various algorithms are shown in Figures 4.15(d), 4.15(f), 4.16(b), 4.16(d) and
4.16(f).
92
(a) Slice 30 (b) Slice 31
(c) Slice 32
Figure 4.14: Example of Knee slices for experimentations.
As it can be observed from Figures 4.15 and 4.16, all the diusion based
algorithms improve the slice reconstruction results provided by LK structural
ows. The directions of morphing one slice into another are captured well
by the structural ows. The structural ows smoothed by PM [4] and Black
[7] algorithms are noisier than the structural ows smoothed by the pro-
posed diusion algorithms. The robust diusion algorithms provide smooth
structural ows while eliminating the inuence of outlying vectors. The best
visually assessed reconstruction that provides the most compact reconstruc-
93
(a) LK ow (b) LK reconstructed
(c) 2DH smoothed (d) 2DH reconstructed
(e) PM smoothed (f) PM reconstructed
Figure 4.15: Structural ows and reconstructed slice of a Knee MRI.
94
(a) ATM-2DH smoothed (b) ATM-2DH reconstructed
(c) Black smoothed (d) Black reconstructed
(e) MED-2DH smoothed (f) MED-2DH reconstructed
Figure 4.16: Further results on structural ows and reconstructed slices of a
Knee MRI.
95
tion of this slice is that provided by the MED-2DH smoothed structural
ows. The observation shows that MED-2DH have signicantly reduced the
inuence of outliers and modelled the deformation of the hard tissue of the
knee more accurately when compared to the other algorithms used in the
experiments. MED-2DH have also accurately modelled the deformation of
soft tissue in the muscular area of the knee whilst smoothing out unwanted
specular eects of the image.
The small black gaps in the reconstructed slices observed in Figures 4.10,
4.11, 4.15 and 4.16 caused due to pixel block displacement can be covered
using interpolation [119], mathematical morphology operator [55], diusion
algorithms for inpainting [8] or the diusion methodology proposed in this
thesis, when applied on grey-level images.
Four other data sets representing a mixture of soft and hard human tis-
sues have also been used for experimental purposes. The original Humerus
and Iliac data sets are used as ground truth and are subsampled by 1:6 aiming
to replace the slices that are initially skipped for numerically assessing the
reconstruction methodology. The structural ows for these slices of human
bones have been initialised using the DBMA algorithm. Figure 4.17 shows
the original 3D Humerus bone, while Figure 4.18 represent 3D volume recon-
structions using interpolated structural ows smoothed by diusion kernels.
Figure 4.19 shows the original Iliac bone and Figure 4.20 shows the pro-
cessed 3D volumes for the Iliac bone from the structural ows smoothed by
various diusion kernels.
The surfaces of the reconstructed objects are smoother for the 3D Incisor
and Humerus than those of the Iliac bone which has a more complex mor-
96
Figure 4.17: Original Humerus bone.
phology. The 3D Humerus volume reconstructed from the structural ows
smoothed by MED-2DH is smoother everywhere as compared to the other
algorithms apart from the regions at the extreme end slices of the bone. At
the extreme end of the bone slices, PM [4] and Black [7] algorithms perform
better. The external surfaces of the 3D shapes can be further smoothed using
additional post-processing 3D shape smoothing algorithms.
The soft tissue data sets used in this study, female chest and sheeps
heart MRI scans have been initialised using the LK algorithm [3]. Prior
segmentation have not been conducted on these data sets. As a consequence,
it is not possible to show a 3D rendered volume of the chest and heart. Figures
4.21 and 4.22 show some intermediate slice ow and reconstruction results
for the chest and heart data sets, respectively. These results attempt to
reconstruct the middle slices, whereby ve original intermediate slices have
97
(a) DBMA (b) PM (c) Black
(d) 2DH (e) ATM-2DH (f) MED-2DH
Figure 4.18: 3D Humerus bone reconstructed volumes.
98
Figure 4.19: Original Iliac bone.
been removed.
The results displayed in Figures 4.21 and 4.22 are intermediate slice recon-
structions from Black et. al. [7] and MED-2DH smoothed structural ows.
As it can be observed, the MED-2DH smoothed structural ows seemed to
be cleaner and less noisy when compared to the result produced by Blacks
algorithm. All the structural ows computed using the LK algorithm have
been subsampled by a factor of 2 in x and y directions, for visual clarity
purposes. This is because the LK algorithm is a gradient based method and
produces dense ow elds, which are dicult to visualise, especially for large
slices.
The reconstruction errors are calculated by skipping one intermediary
frame for each pair of slices from the original Incisor data set and 5 slices
for the Humerus and Iliac data sets. These slices are reconstructed back as
described in this chapter using the DBMA method as the initialisation. The
dierence, calculated as the percentage of wrong shape assignment between
99
(a) DBMA (b) Perona-Malik (c) Black
(d) 2DH (e) ATM-2DH (f) MED-2DH
Figure 4.20: 3D Iliac bone reconstructed volumes when skipping 5 consecu-
tive slices between the remaining 2 slices.
100
(a) Slice 54 to be reconstructed
(b) Black smoothed (c) Black reconstructed
(d) MED-2DH smoothed (e) MED-2DH reconstructed
Figure 4.21: Results on structural ows and reconstructed slices of a Female
Chest MRI.
101
(a) Slice 177 to be reconstructed
(b) Black smoothed (c) Black reconstructed
(d) MED-2DH smoothed (e) MED-2DH reconstructed
Figure 4.22: Results on structural ows and reconstructed Sheeps Heart
MRI slices.
102
the original and the reconstructed data set are provided in Table 4.2. As
expected, MED-2DH is the best algorithm for volumetric reconstruction.
Also presented is the average PSNR of the intermediate slice reconstruc-
tions for the whole 3D volume of Incisor, Humerus and Iliac data sets in
Table 4.3. The numerical results in Table 4.3 have been obtained by tak-
ing the average PSNR of all slices needed to reconstruct a 3D volume of a
particular object. The average PSNR has been compared against dierent
diusion lters. For the Incisor, the Hessian based kernels performed bet-
ter because the dataset is sparse with large shape variations between slices.
Hence, the Hessian based kernels were able to model the shape variations
more accurately than the other methods, thereby interpolating the pixels
more accurately. However, the Humerus and Iliac datasets not being sparse,
suggested minimal variations between slices. Therefore, with shape interpo-
lation no longer an issue, DBMA (no smoothing) achieves a higher accuracy
of pixel interpolation compared to diusion methods.
Table 4.4 presents the average PSNR after removing 5 slices from the
original set (for all 6 data sets), aiming to reconstruct the middle slice with
LK algorithm as the initialisation. From the result observed, it is thought
that gradient based ow methods tend to be more ecient than the block
based methods.
A third measure considered to validate the shape reconstructions is the
Hausdor distance measure. The Hausdor distance between two contours
f
i
and

f
i
is dened as in [126]:
H(f
i
,

f
i
) = maxh(f
i
,

f
i
), h(

f
i
, f
i
) (4.5)
103
Object 3D shape reconstruction error (%)
DBMA PM Black 2DH ATM-2DH MED-2DH
Incisor 17.81 14.24 14.24 11.09 12.15 10.43
Humerus 12.12 6.59 6.60 6.97 7.52 5.85
Iliac 12.35 10.24 9.60 8.21 8.12 7.08
Table 4.2: Average percentage of reconstruction errors with DBMA as the
initialisation.
Object PSNR (dB)
DBMA PM Black 2DH ATM-2DH MED-2DH
Incisor 13.83 13.92 13.92 14.05 14.14 14.13
Humerus 13.87 13.21 13.21 13.25 13.34 13.24
Iliac 14.67 14.14 14.04 14.08 14.30 14.13
Table 4.3: Average peak signal-to-noise ratio of slice reconstructions with
DBMA as the initialisation.
where
h(f
i
,

f
i
) = max
af
i
min
b

f
i
|a b| (4.6)
where a and b are elements of the original contour f
i
and of reconstructed
contour

f
i
, respectively, and | | denotes the Euclidean distance. The Haus-
dor distance calculated between the contours of the horizontal projections
of the 3D shapes, when skipping 1, 3, 5 and 9 slices is provided in Table 4.5
for the Humerus bone and Table 4.6 for the Iliac bone.
Based on the results from Table 4.2, MED-2DH smoothed structural ows
provide the best reconstruction followed by ATM-2DH. However, from the
104
Object Peak Signal-to-Noise Ratio (dB)
LK PM Black 2DH ATM-2DH MED-2DH
Incisor 17.63 17.36 17.30 16.79 16.97 16.72
Humerus 34.92 34.52 34.52 34.79 34.50 34.62
Iliac 29.59 28.94 28.94 28.89 29.13 28.72
Knee 24.22 23.98 23.98 23.92 23.84 23.81
Chest 37.97 37.82 37.79 37.81 37.74 37.66
Heart 21.02 20.74 20.74 20.69 20.66 20.57
Table 4.4: Average PSNR of original middle slice reconstructions after re-
moving 5 intermediate slices with LK as the initialisation.
Slices Hausdor distance
skipped DBMA PM Black 2DH ATM-2DH MED-2DH
1 8.94 7 7 8.06 8 7.07
3 13.89 11.66 11.66 12.53 12.21 12.53
5 13.04 10.82 10.82 13.89 12.81 13.60
9 14.77 11.31 11.31 13.42 14.20 13.60
Table 4.5: Hausdor distance for Humerus bone.
Hausdor distance results in Table 4.5, PM and Black give better recon-
struction results compared to the Hessian based kernels. The main reason
for these results is the consistent error in the reconstruction of the end slices,
which is independent of the number of skipped slices. Relating to the Iliac
bone, it is dicult to deduce a meaningful conclusion from the results in
Table 4.6. From Figure 4.19, the Iliac bone is an odd shape with lots of
indentations, a hole at the bottom and a large groove at the back which is
105
Slices Hausdor distance
skipped DBMA PM Black 2DH ATM-2DH MED-2DH
1 12.81 12.81 13.89 15.23 17.80 16.97
3 11.40 14.76 18.38 22.13 15.65 16.55
5 26.40 22.67 21.10 45.62 14.14 13.89
9 48.08 22.80 18.03 66.10 62.80 71.45
Table 4.6: Hausdor distance for Iliac bone.
expected due to its biological function. With every alternate and every third
slice being skipped, DBMA and PM (only for skip 1) are the nearest shape
match to the original contour, with the Hessian based kernels showing errors
in reconstruction around the hole. The Iliac bone reconstructed from MED-
2DH smoothed structural ows is the closest match to the original contour
compared to other methods. This shows the strength of the algorithm to
reconstruct an accurate shape. For every 10th slice that is used, the results
did not favour the Hessian based kernels with Blacks method coming out
on top. The signicant dierence in performance here is due to the Hessian
based kernels over compensating to reconstruct the concavity of the bone,
causing the shape error to be higher than it should really be. A probable
solution to this is to perform shape segmentation in the 3D space, which
should yield more realistic error measurements.
The 3D shape reconstruction rate in equation (4.3) is only applicable
for slices that have been segmented, i.e. noise in the background (beyond
the structure of the object in the slice) have been removed or signicantly
reduced. Since the three soft tissue data sets have not been pre-processed,
the only measurement that bears any signicance is the PSNR (4.4). The
106
comparative results for all six data sets with the structural ows initialised
using the LK algorithm [3] is presented in Table 4.4. From the table, highest
PSNR is achieved by reconstructions from the initial ow. This result is
expected since PSNR is intensity based measurement and the LK initial ow
does not infer any intensity changes on the reconstructed slice. The results
reported here are similar to the results obtained for structural ows initialised
by DBMA.
Figure 4.23 shows the stack of aligned contours for the original 3D bones
projected onto the horizontal plane together with those reconstructed using
the structural ows smoothed by various diusion algorithms. The contours
correspond to 3D volumes that have been reconstructed after eliminating 5 in-
termediary slices and thereafter interpolating them by using smoothed struc-
tural ows. The positive eect of robust diusion structural ow smoothing
is evident in some of these contours when compared to those obtained when
using the initial structural ows as provided by DBMA. All these results
highlight the advantages of using the proposed 3D interpolation methodol-
ogy particularly when applying robust diusion kernels to structural ows.
4.8 Conclusion
In this chapter, a new methodology for 3D volumetric reconstruction from
sets of sparse cross-sections has been proposed. A bi-directional correlation
algorithm between pairs of image slices to construct structural ows has been
used. These structural ows show the correspondence in the internal struc-
ture and among features for a given 3D object. Besides the proposed DBMA
algorithm, structural ows were also initialised using the LK algorithm. The
107
(a) Humerus Bones (b) Iliac Bones
Figure 4.23: Reconstruction of bone contours for Humerus and Iliac bones.
From top to bottom, the contours are from smoothed reconstructions using
original slices (blue), DBMA (red), Perona-Malik (green), Black (magenta),
2DH (yellow), ATM-2DH (cyan) and MED-2DH (black).
ows are smoothed by various diusion algorithms from Chapter 3. Interme-
diary slices are reconstructed using the smoothed structural ows resulting
in a 3D volumetric object. Both shape structure and grey-level texture is
reconstructed according to the structural ows and existing slices. The ex-
perimental results prove that the best outcome is achieved when using the
dual directional structural ows smoothed by robust local Hessian based ker-
nels, provided the structural ows have been initialised using feature based
method. A similar PSNR has been obtained for the interpolated slices for
all the diusion methods in Chapter 3 when used on the same data set.
However, the surface of the resulting 3D shapes is not always smooth. This
problem can be resolved by using additional post-processing algorithms to
smooth the resulting 3D shape surfaces.
108
Chapter 5
Robust Physics based Diusion
Solver
5.1 Research Objective
In uid dynamics, turbulence is characterised by chaotic and stochastic prop-
erty changes. This includes low momentum diusion, high momentum con-
vection, and rapid variation of pressure and velocity in spatial and temporal
space. Typically, turbulence is represented by multitude of swirling motions
around either a single or multiple vortex cores. In vortex dynamics, it is
numerically assumed that at the centre of the vortex core, the pressure is
minimum in all directions [100]. Hence forth, subsequent references to com-
plex and turbulent motion in this thesis will refer to a combination of swirling
motions of the object in the scene and noise/camera movement introduced
by the video capture.
A robust Hessian based diusion algorithm has been presented in Chap-
ter 3 for smoothing optical ows from video sequences. Additionally, it has
109
been shown in Chapter 4 that robust and ecient smoothing of structural
ows can inuence slice synthesisation and subsequent 3D volumetric recon-
structions. In this chapter, an improved diusion methodology that is based
on the Navier-Stokes equation is developed. The proposed method is an
improvement on the Stable Fluid Solver proposed by Stam [97, 98, 127] by
robustifying the diusion stage. The robust Hessian based diusion devel-
oped in Chapter 3 is embedded in the proposed method. A methodology for
detecting specic movement patterns such as vortices from smoothed vector
elds is developed in this chapter as well.
5.2 Introduction
It can be safely said that modelling the optical ow from motion in uid im-
age sequences can be an onerous task. The techniques that have been around
for a while, e.g. standard optical ow estimation methods work well when
modelling the movement of rigid objects. However, in the case of deformable
objects and uids, existing methods perform ineciently. The Navier-Stokes
equation has been frequently used to model the dynamics of uid ow in
computational uid dynamics. It was used to model heart movement in
echocardiography [89], for simulating the behaviour of natural phenomena
such as air, uids, clouds, etc, in computer graphics [97, 110, 112, 128], mod-
elling the dynamics of turbulence [129], blood ow analysis in angiography
[130], modelling behaviour of laminar ow around a cylinder [131], modelling
the radiation of stars in dust clouds in astrophysics [132], etc.
The Navier-Stokes equations in itself are complex and are traditionally
used to model the movement of heat and uid in complex structures. Such
110
natural phenomena are more likely to exhibit nonlinear, sometimes chaotic
like behaviour. Due to the presence of nonlinearities, we are unable to model
such behaviour easily. In order to make accurate assessments on the be-
haviour of such phenomena, for example heat and uid particles, ne grid
structures are a necessity to model the connement structure. Additional
constraints such as the initial and boundary conditions, explicit or implicit
nite dierencing schemes are important since they increase the stability and
reliability of the computations. The reason why Navier-Stokes equations is
so important is that we are increasingly aected by it in our daily life. It was
used to model injection and combustion engines, its been used to forecast cli-
mate change, model the trac and crowd ow, it has been used to model the
aerodynamics of wings and rotor blades, model the temperature distribution
of silicon chips and so on. However, the implementation of the Navier-Stokes
equations is application dependent. Despite the complexity involved, new
and innovative methods are continuosly being developed in order to make it
computationally more accessible and generalised.
In this chapter, an innovative smoothing method is proposed using the
Navier-Stokes equations. Together with supplemental equations, such as the
conservation of mass and well formulated boundary conditions, the proposed
methodology is used to smooth estimated optical ows of complex and turbu-
lent uid motion from image sequences. The proposed method is embedded
with a robust diusion algorithm, as described in Chapter 3 for better e-
ciency. Distinct features such as vortices (focus of rotation) are estimated
from turbulent optical ows. The proposed technique is applied on articial
data and image sequences representing atmospheric and solar phenomena.
111
The remainder of the chapter is presented as follows. Section 5.3 de-
scribes the stable uid model [97] and how it is being implemented in com-
puter graphics and visualisation. In Section 5.4, the methodology used to
implement the proposed robust hybrid uid solver is presented. In Section
5.5, we present methods to detect the presence of vortices in turbulent elds.
Experimental results and the analysis on simulated and real-world data is
presented in Section 5.6 while Section 5.7 concludes the chapter.
5.3 The Stable Fluid Model
Navier-Stokes methodology represents the basis for modelling a large variety
of phenomena such as those characterising weather, ocean currents, water
ow in a pipe, the air ow around an aircraft wing, the motion of stars
inside a galaxy, blood ow, nancial derivatives, etc [91]. In engineering,
they are also used in the analysis of the eects of pollution, the design of
aircraft and of power stations. Navier-Stokes methodology has been applied
in computer graphics in order to visualise and create the eects given by
the complex movement of uids such as coloured gases, air, clouds, liquids,
smoke, re, etc [96, 97]. The explicit model is generally used for the precise
computation of uid dynamics and involves heavy computational complexity
[91]. The von Neumanns stability analysis, as shown in [91], highlights
that the implicit model based on discretisation when calculating Navier-
Stokes equations is unconditionally stable, although it requires a complex
numerical implementation scheme [97, 98]. The stable uid solver (SFS)
algorithm proposed by Stam represents an implementation of the Navier-
Stokes methodology in an implicit scheme [97, 98] which has been applied
112
for complex graphics modelling.
In order to achieve visual eects, the Navier-Stokes equations are used
for both density and velocity modelling in the SFS algorithm [97, 98]. Unlike
in the original SFS approach, only the modelling of motion based on the
Navier-Stokes equations are considered in this work. In SFS, the area of
investigation (in this case an image or a segmented region from an image)
is split into cells located on a grid and is assigned a particle to each grid
location. We assume that the SFS system moves the particles around in the
vector eld, where each vector corresponds to a grid location. The Navier-
Stokes equation for a given system is derived using the conservation of mass,
momentum and energy for an arbitrary control volume [91] and is given by:
V
t
= (V ) V
P

+
2
V+f (5.1)
where the change of velocity V over time is represented with respect to the
advection, the gradient of the pressure P, diusion and the external forcing
function f, while is a viscosity constant that characterises the uid and is
a parameter. The pressure is assumed to be constant in the given eld and
its gradient is zero, i.e. the change in pressure from one spatial position to
another in the vector eld is negligible. Consequently, the equation employed
by the SFS method is:
V
t
= (V ) V +
2
V+f (5.2)
The diusion term
2
V characterises uids which are assumed incom-
pressible and Newtonian. Moreover, for incompressible uids it is important
to enforce the conservation of mass [91] condition:
113
V = 0 (5.3)
which states that the divergence of velocity components is zero for innites-
imal time steps. The density of a particle is constant between iterations,
thereby the total mass of the eld is conserved within the given region.
for k 1 to convergence or number of iterations
do
1 add force: V
1
= V
0
+f t
2 advect: V
2
(x) = adv(V
1
(x, t))
3 transform:

V
2
= FFT(V
2
)
4 diuse:

V
3
(z) =

V
2
(z)/(1 +tk
2
)
5 conserve:

V
4
= conserve(

V
3
)
6 transform: V
4
= FFT
1
(

V
4
)
Figure 5.1: The stable uid solver algorithm.
The SFS algorithm proceeds to calculate the velocity components V as
described in Figure 5.1 [97]. For each iteration, the rst step consists of
adding the external forcing function f which determines the initial conditions
in the processing cycle. The second step represents the advection term in
equation (5.2), which corresponds to the following:
(V ) V =
_
V
x
V
x
x
+ V
y
V
x
y
, V
x
V
y
x
+ V
y
V
y
y
_
(5.4)
where V = (V
x
, V
y
). The analysis of the advection process in real physical
phenomena is provided in [91]. The process described by equation (5.4) is
114
known as the self-advection of velocity. The advection step from the SFS
algorithm is implemented using an implicit nite dierence scheme. It is
quite impractical to model the motion from one grid location to another
using dierence schemes due to constraints such as computational stability
and ill-posed system of equations. However, by assuming that the motion
vector of each grid cell is a particle, these can be traced back in time with
t by backtracking the velocity eld. By doing this, the motion vector at
the backtraced location is interpolated from the neighbouring four locations.
This interpolated vector is then used as a basis for further computations.
The third step transforms the velocity eld to the frequency domain using
the Fast Fourier Transform (FFT). The requirement to set specic boundary
conditions is eliminated by extending the spatial repeatability of the area
under consideration and by applying FFT. The diusion term (fourth step)
represents the decay of high spatial frequencies in the velocity eld and is
computed in the Fourier domain with a Gaussian lter processing the velocity
component V by using the time step t and the uid kinematic viscosity .
The nite dierence implicit scheme is used here to discretise the diusion
term in order to obtain an unconditionally stable system [97]. The fth
step enforces the local incompressibility of the optical ow which requires
that the amount of ow entering a specic area should be equal to the ow
exiting that area. The nal step projects the ow back from the frequency
domain to the spatial-time domain using the inverse FFT transform. This
algorithm was modied in [98] by replacing the FFT transformations and the
processing in the frequency domain by dening a set of boundary constraints
on a grid-based representation of the ow.
115
5.4 Robust Hybrid Fluid Model
Figure 5.2: Robust hybrid solver.
The implementation of the stable uid solver [97] provided rather poor
performance in modelling turbulent optical ow estimated from image se-
quences. This is mainly caused due to the uncertainty in the initial estima-
tion of the optical ow which produces noise, particularly in image sequences
displaying complex motion. In order to improve the performance on optical
ows, a robust anisotropic kernel [133] is embedded in the diusion step of
the SFS. Figure 5.2 shows a ow diagram of the proposed robust hybrid uid
solver. The initial ow can be estimated using the block matching algorithm
116
as in [14] or other motion estimation algorithms [12]. Optical ows provided
by block matching or by using temporal gradient estimation are invariably
noisy [14], particularly in the case of image sequences representing moving
uids or other complex phenomena.
The rst processing block corresponds to a reinforcement step and the
proposed method is implemented by adding a proportion of the velocity from
the previous iteration to the current velocity:
V
1
(t + t) = (1 w)V
0
(t) + wtV
5
(t) (5.5)
where V
5
(t) is the motion vector from the previous iteration t, w (0, 1)
is a weighting factor modelling the degree of the reinforcement, V
0
(t) and
V
1
(t + t) represent the motion vector reinforced by force at times t and
t+t, respectively. At the rst iteration there is no reinforcement, i.e. w = 0.
The SFS algorithm described in Section 5.3 proposes to advect the initial ow
at Step 2 from Figure 5.1.
However, the algorithm described above produces unreliable estimation
when applied to noisy vector elds. The optical ow should have a degree of
smoothing before advection can be applied. In the proposed approach, the
noisy ow is diused before proceeding to the advection stage. The transfer
function of the original smoothing algorithm is a Gaussian function appropri-
ately dened within the frequency domain [97]. The proposed method is to
implement a Hessian based diusion that jointly processes the local geometry
and the statistics of the local vector eld as in [133]:
117

V
2
(t + t) =

x
i
(zc)
V
1,i
(t) exp[(x
i
z
c
)
T
H
1
(x
i
z
c
)]

x
i
(zc)
exp[(x
i
z
c
)
T
H
1
(x
i
z
c
)]
(5.6)
where

V
2
(t + t) is the intermediate diused value, H represents the local
Hessian, V
1,i
(t) is the vector at location i within a neighbourhood (z
c
),
centred at the location z
c
.
The eigenvector corresponding to the largest eigenvalue of the Hessian sh-
ows the local direction of the optical ow. This diusion kernel is anisotropic
and adapts to the local structure of the optical ow. Signicant optical
ow transitions are automatically detected and consequently not smoothed
over by the Hessian-based kernel. However, anisotropic diusion does not
deal properly with outliers as shown in a study provided in [133] and de-
scribed in Section 3.4.1. In order to properly process the local statistics
and eliminate outliers, the median algorithm is considered for robustifying
the Hessian based diusion in the neighbourhood (z
c
), as implemented in
equation (3.19).
At the advection stage, this model is only concerned with the nonlinearity
of the advection term from equation (5.4). As mentioned in Section 5.3, the
self-advection term represents the ability of the velocity components to move
their own values from one position to another on a grid in a time step interval,
t in order to model complex movement usually combining rotation and
translation. This procedure involves interpolating the velocity at the grid
points, using a neighbourhood approximation, from the previous time step
back to the position in the current time step [98].
The model is dependent on the initialisation and on boundary conditions
118
of the system under study. In the proposed approach, boundary conditions
are specically provided onto the grid in order to represent the physical limits
of the optical ow. Such boundary conditions can be the result of image or
motion segmentation algorithms or of the existence of a priori information
about the image sequence. There are two boundary conditions to consider.
The rst condition is determined by the physical boundary and is represented
by the von Neumann condition which species the normal component of the
ow to the boundary surface as:
V
n

= 0 (5.7)
where represents the boundary and n is its surface normal. This equa-
tion implies that the wall absorbs any ow particles coming towards it. This
condition is also commonly known as the no-slip boundary condition. For
the sake of reducing the required computation complexity, the walls of the
domain, are represented by zero values on a geometric grid, which are en-
forced at every stage of the computation in order to preserve the stability and
integrity of the numerical calculation. Since our proposal incorporates both
explicit and implicit nite dierencing schemes, it is absolutely imperative
that the model adheres to the stability criteria, given by t/(x)
2
1/2
[91], where x represents the location change during the time interval t.
The second condition relates to the conservation of mass of the velocity
eld. The conservation of mass, given by equation (5.3), should be main-
tained in order to ensure the incompressibility of the ow.
In order to maintain a divergence free velocity eld for every stage of
computation, the conservation of mass is enforced after both diusion and
119
advection stages. The conservation of mass stage corresponds to a data
normalisation process. This is enforced by using the Helmholtz-Hodge de-
composition [97] of the velocity eld. Using the projection method (based
on the Helmholtz-Hodge decomposition) as described in [127], the projection
of the pressure gradient onto a divergence free vector eld is thus given as
Tv = w and w = 0, Tv = v if and only if v = 0, TP = 0 [134].
This decomposition provides an exact solution so that the mass conserved
incompressible ow can be obtained by extracting the gradient of the ow
from the current vector eld. The decomposition can be shown to be
w = V+q (5.8)
where w is any vector eld, V is mass conserved and divergence free (5.3)
and q is the gradient of a smooth scalar eld. By taking the divergence of
equation (5.8), the Poisson equation as shown in the following equation is
obtained:
w = V+ q
w =
2
q (5.9)
where condition (5.3) is applied. The Poisson equation obtained in equa-
tion (5.9) is sparse and is solved using the Gauss-Seidel relaxation method,
similarly described in [128]. This is because the Gauss-Seidel relaxation
method is simple to implement, achieves fast convergence and new approxi-
mations are used instantly for subsequent computations. Although there are
better relaxation schemes available, this method is sucient for the current
implementation. Subsequently, the incompressible eld is obtained by
120
V = wq (5.10)
as suggested by the Helmholtz-Hodge decomposition. This decomposition
maintains the incompressibility and smoothness of the estimated velocity
eld. Mass conservation is important for realistically estimating optical ow
of uids. For example, the exact Helmholtz-Hodge decomposition of the
closed cavity laminar ow (articial data provided in Section 5.6) at the
thousandth iteration is shown in Figure 5.3.
Current Flow = Incompressible Flow + Gradient Flow
Figure 5.3: Helmholtz-Hodge decomposition of a closed lid driven cavity
laminar ow.
5.5 Vortex Core Detection
After smoothing and modelling the vector eld as described in Chapter 2
and Section 5.4, it is now possible to extract salient patterns that are char-
acteristic to turbulent ows such as vortices. Although the vector eld have
been smoothed, there is no clear process to verify whether the smoothed eld
is either good or bad ow. This is especially important for fuel mixing in
jet thrusters [135]. The presence of vortices in such ows could cause mas-
121
sive energy loss within the combustion chamber and subsequently cause an
unstable fuel chain reaction with potential damaging consequences. There-
fore the detection and tracking of vortices in turbulent ows is of signicant
importance. Furthermore, it is possible to identify types of ow elds from
the dynamics of vortices. Vortices in uid mechanics represent rotational
structures of concentrated energy around a centre [100, 103, 112, 136]. Usu-
ally, the centre is characterised by a minimum in the pressure and by zero
velocity. However, this is not the case in most turbulent elds.
A Galilean-invariant velocity eld is assumed to be of a uniformly smooth
ow eld [135]. The Galilean-invariant vortex detection criteria uses the
scalar velocity gradient decomposition [100, 103]:
V = S + (5.11)
where S is the rate of strain tensor:
S =
1
2
[V+ (V)
T
] (5.12)
and is the vorticity tensor:
=
1
2
[V(V)
T
] (5.13)
The Q-criterion denes a spatial region where the Euclidean norm of the
vorticity tensor dominates that of the rate of strain:
Q =
1
2
[||
2
|S|
2
] > 0 (5.14)
where | | represents the Euclidean norm of the tensor. This expression is
further simplied using equation (5.3) to obtain
122
Q =
V
x
y
V
y
x
> 0 (5.15)
The Q-criterion is further decomposed into Eulerian form [100] as in equa-
tion (5.15) for ease of vortex region identication. The equation (5.15) is
also obtained by applying the constraints for incompressibility, as described
in equation (5.3), into equation (5.14). Therefore, the criterion is dependent
on the mixed derivatives of the gradient velocity tensor. Hence, for the ex-
periments, the implementation is based on equation (5.15) since we assume
that the vector elds under study are incompressible. In addition to the Q-
criterion, the characteristic function of the deformation tensor for 2D vector
eld is given as [103]:

2
+ p + q = 0 (5.16)
where is the eigenvalue of the tensor V, while the co-ecients depend on
its determinant and trace as: q = Det(V) and p = Tr(V). Hence, the
discriminant of equation (5.16) is:
= p
2
4q (5.17)
where the discriminant is the root of the characteristic equation (5.16). Both
the Q-criterion and discriminant, , is used to characterise critical regions
of focus, node and saddle in a topology map of a turbulent ow [103]. If
is positive, then the velocity particle is likely to be located within the vortex
core region. If is negative, then it is likely to be within a saddle region
outside of the vortex core boundaries.
123
These formulae are applicable for vortex detection, on the assumption
that the vortex structures are smooth and can be integrated from a su-
ciently large energy basin. The smoothness condition is ensured by using the
methodology described in the previous sections by employing the Navier-
Stokes equations with a robust diusion step (Sections 5.3 and 5.4) or by
using diusion algorithms proposed in Sections 3.3 and 3.4. In [100, 103], a
single point evaluation for Q was used.
The Q-criterion is notably susceptible to the presence of noise and in-
variably fails when the velocity eld is not Galilean-invariant. The current
work attempts to improve this by using a running window in which several
Q
i
using (5.15) are evaluated. A set of four vectors, symmetrically located
at the extremes of a cross, window centered, is used for calculating each Q
i
.
The resulting Q
w
is estimated as a weighted average of all Q
i
, each calculated
according to equation (5.15), from a set of four vectors located as shown in
Figure 5.4:
Q
w
=
25

i=1
w
i
Q
i
25

i=1
w
i
(5.18)
where w
i
represents the inverse of the Euclidean distance from the windows
center to the location of each of the four vectors. For a 5by5 window, the
weights correspond to w
i
= 1,
1

2
,
1
2
,
1

5
,
1

5
,
1
2

2
, depending on the inverse
shortest path between pixel in a neighbourhood and its center.
The vortex structure is identied for > , where is an arbitrary
value. Our studies show that an appropriate should be about 0.2 from the
maximum value of topology map.
124
The weighted discriminant,
w
is also computed in a similar way as to
the Q
w
-criterion as shown in equation (5.18). The computation of
w
is
shown to be

w
=
25

i=1
w
i

i
25

i=1
w
i
(5.19)
As mentioned in [100], the discriminant, is used as another set of
constraints for identifying vortex core regions. This computation is useful
for segmenting specic regions especially when the eld is smooth. As with
the other constraints, this criterion fails in the presence of noise. Hence, the
introduction of
w
, which does provide a more reliable estimation of the
vortex structure.
Vortex structures are segmented by correlating the evidence in both the
velocity elds and the vorticity maps using > as a constraint. This is
done by rst thresholding the topology map with the function
1() =
_
_
_
1 , ( > 0.2)
0 , ( 0.2)
(5.20)
where 1() represents an indexing function. By now, it is expected that the
regions of interest (ROI), i.e. vortex regions are segmented in the index array.
Subsequently, the function 1 is convoluted with either V or [[ to obtain
segmented vortex regions in the vector eld or vorticity map. Additional
functions can be added to this process to allow tracking of vortices from
frame to frame. This is especially useful in weather forecasting and geo
remote sensing applications.
125
Figure 5.4: Evaluation of the Q
w
-criterion from a 5 by 5 window, where
the location of each vector in the window is shown with a dierent marker
according to its corresponding Q
i
set.
5.6 Experimental Results
The experimental results presented below show the evaluation of the pro-
posed algorithm on synthetic uid sequences and on real-world image se-
quences.
126
5.6.1 Synthetic data simulation set-up
This section describes the experimental set-up required to simulate arti-
cial sequences based on the Navier-Stokes equations. The rst part of the
research results focusses on two dierent types of articial sequences. The
rst is a closed driven-lid cavity ow and the second is the ow of the von
Karmans vortex sheet.
Closed driven-lid cavity ow
The rst synthetic sequence is created using the original Navier-Stokes equa-
tions [91] for a closed driven-lid cavity ow. The air ow within the cavity
is modelled and observed when the top wall is moved with constant velocity
as shown in Figure 5.5. The synthetic ow is created using the vorticity-
stream formulation of the Navier-Stokes equations instead of the standard
velocity-pressure formulation.
Figure 5.5: Driven-lid cavity ow diagram.
With reference to Figure 5.5, the walls of the cavity are marked with
A (top), B (left), C (right) and D (bottom). The initial and boundary
127
conditions for the four walls are given as follows:
A: V
x
= V e = 5m/s, = 0,
P
n
= 0, = 2

0,j

1,j
(y)
2
B: V = 0, = 0,
P
n
= 0, = 2

i,0

i,1
(y)
2
C: V = 0, = 0,
P
n
= 0, = 2

i,ny1

i,ny2
(y)
2
D: V = 0, = 0,
P
n
= 0, = 2

nx1,j

nx2,j
(y)
2
with being vorticity and being the stream functions of the velocity. Both
vorticity and stream functions can be represented in terms of velocity by
= [[ = [V[ =
V
y
x

V
x
y
(5.21)
and

y
= V
x
,

x
= V
y
(5.22)
In addition to the given boundary conditions, there is also a constant
pressure applied at grid position (0,0) of 2bars to move the lid (top wall).
The sequence is corrupted with additive Gaussian noise of zero mean and
test sequences when varying the noise variances are obtained as follows.
Figure 5.6(a) represents the ground truth synthetic eld that visualises
the movement of air ow moving inside the area of a closed driven-lid cavity
with a xed velocity. The gure shows the snapshot of the ow after a thou-
sand iterations with time intervals of 1ms between iterations. Figure 5.6(b)
shows ow degradation after adding Gaussian noise with variance
2
= 0.01,
Figure 5.6(c) is ow with Gaussian noise with variance
2
= 0.10 and Fig-
ure 5.6(d) is ow with Gaussian noise with variance
2
= 0.25. These are
128
(a) Ground truth ow (b)
2
= 0.01
(c)
2
= 0.10 (d)
2
= 0.25
Figure 5.6: Synthetic closed driven-lid cavity ows with noise.
examples of degraded vector elds in order to be used for testing the nonlin-
ear smoothing methodology.
von Karmans vortex sheet
The second synthetic sequence is the von Karman ow which is created
using models suggested in [92]. This model is used to simulate the creation
of articially complex and turbulent ows which is used as ground truth
for comparative analysis. The model used to create the ows is shown in
Figure 5.7 and is based on the velocity-pressure formulation of the Navier-
129
Stokes equations.
Figure 5.7: Half-cylinder model for von Karman ow.
With reference to Figure 5.7, the walls of the cavity are marked by A
(top), B (bottom), C (left) and D (right). The initial and boundary condi-
tions for the four walls are given as follows:
A:
V
n
= 0, P = 0
B:
V
n
= 0, P = 0
C: V
x
= V e = 1m/s
D:
V
x
= 0, P = 0
In addition to the numerical conditions at the walls, the default boundary
constraint on the half-cylinder, G is
V
n
= 0. This simple model, without
additional Dirichlet or Neumann conditions is simulated using Gerris Flow
Solver [137] with T = 9s and t = 0.1.
130
5.6.2 Results on synthetic data
Closed driven-lid cavity ow
Modelling results using the modied SFS (SFSM) algorithm [98] adapted for
usage on vector elds is shown in Figure 5.8(a), while vector eld smoothing
using Blacks anisotropic diusion algorithm [7] is shown in Figure 5.8(b).
Figure 5.8(c) shows the eects of using MED-2DH which is a robust Hessian
based diusion algorithm described in [133], while the robust hybrid uid
solver embedding the median of 2D Hessian diusion kernel (MedH-SFS)
algorithm, as described in Section 5.4, is shown in Figure 5.8(d). For better
visualisation, the vector from the upper-right corner of the SFSM vector eld
in Figure 5.8(a) has been rescaled.
The results in Figure 5.8 are obtained at convergence when the mean
square error dierence between vector elds at two successive iterations is
less than 0.01. The number of iterations necessary to achieve convergence
is provided in the parentheses from the caption of each result of Figure 5.8.
From these results, it can be observed that the vector eld modelled by SFSM
is still noisy at convergence, while the noise has been signicantly reduced
in the other smoothed vector elds. The results also show that MedH-SFS
provides the best outcome and the vortex recovered is better located when
compared to the vortices recovered using Black and MED-2DH.
For numerical comparisons, the mean cosine error (MCE) between the
recovered smoothed ow and the ground truth ow is considered. The MCE
is calculated as in equation (3.26). The MCE results provided in Table 5.1 is
after one iteration of smoothing. SFS algorithm as described in Section 5.3
was adapted from [97], while SFSM has been described in [98]. Both these
131
(a) SFSM (6) (b) Black (3)
(c) MED-2DH (4) (d) MedH-SFS (5)
Figure 5.8: Closed cavity vector eld smoothing comparisons.
Gaussian Noise (
2
) SFSM SFS MedH-SFS Black MED-2DH
0.01 0.7525 0.6211 0.7634 0.7226 0.7383
0.10 0.6020 0.5616 0.7327 0.6554 0.6997
0.25 0.4538 0.4523 0.6849 0.5584 0.6424
0.30 0.4373 0.4624 0.6704 0.5567 0.6058
0.40 0.4005 0.4184 0.5799 0.4958 0.5556
Table 5.1: Mean cosine error of smoothed vector elds.
132
algorithms have been adapted to process vector elds. It can be observed
that SFS performs reasonably on a vector eld corrupted with low noise
variance. However, its performance deteriorates signicantly when the noise
increases, because the corrupted vector eld departs signicantly from the
Navier-Stokes underlying model. The robust diusion hybrid uid algorithm
MedH-SFS provides better results than either SFS or SFSM methods in terms
of MCE when considering additive Gaussian noise, as it can be observed from
Table 5.1. MedH-SFS is also consistently better than Black [7] and MED-
2DH [133] anisotropic smoothers.
von Karmans sheet
The velocity eld estimation from the area cropped in Figure 5.9(a) is shown
in Figure 5.9(b) and is used as ground truth information for comparative
purposes. Figure 5.9(c) shows the vector eld corrupted by adding Gaus-
sian noise distribution with variance 0.10 on each of the x and y axes. The
noisy turbulent vector eld produced by the von Karman ow is smoothed
using the median of the Hessian Hybrid Fluid Solver algorithm (MedH-SFS)
described in Section 5.4, SFS (Section 5.3) provided in [97, 127], SFSM
algorithm described in [98] and the Hessian-based diusion algorithm of
Tschumperl e and Deriche (TD) [5, 6] adapted for use on optical ows. Fig-
ures 5.10(a)-(c) represent the smoothed ow obtained at convergence using
the MedH-SFS, SFSM and TD, respectively (all three results have been sub-
sampled by a factor of 2 along both x and y directions).
As it can be observed, the SFSM smoothed ow is visually the best
smoothed eld. However, SFSM took 4 iterations to converge and has over-
133
(a) Initial von Karman ow
(b) Motion ow extracted from the von Karman ow
(c) Noisy von Karman ow with
2
= 0.10
Figure 5.9: Representing von Karman ows.
smoothed the ow. The result shown could only be made visible after rescal-
ing by a factor of 8. In contrast, the TD smoothed ow (only 1 iteration,
134
(a) Smoothed MedH-SFS ow
(b) Smoothed SFSM ow
(c) Smoothed TD ow
Figure 5.10: Smoothing noisy von Karman ows.
135
as it fails thereafter) is hardly diused and remains quite noisy. In compar-
ison to SFSM, MedH-SFS does diuse the velocity eld quite well within
3 iterations without rescaling. Though the vortices have been successfully
recovered (albeit not perfectly), it is possible that some regions of the eld
may have been oversmoothed.
The method described in Section 5.5 is applied for nding vortex regions
using the Q-criterion and the discriminant . Figure 5.11(a) represents
the measure of the Q-criterion evaluated according to equation (5.14) for
measuring the vorticity and Figure 5.11(b) represents the locally weighted
Q
w
-criterion calculated as in equation (5.18). These results highlight the
saddle-type behaviour characteristic of the ow enveloping the vortex core
[100]. The high values in the map correspond to the 2D stable manifold
characterising the regions where the Euclidean norm of the vorticity ten-
sor dominates the rate of strain (5.14), while lighter colours in the gure
correspond to smaller dispersions of the vortex energy.
From Figure 5.11, it can be observed that the measure Q
w
from (5.18)
provides a clearer identication of the four intersections of the upper and
lower sides of the vortex rings (corresponding to hyperbolic regions around
the center of the vortex core) when compared to the locally estimated Q,
according to (5.14).
Table 5.2 presents numerical analysis results for the eects of smoothing
von Karman ows corrupted with a Gaussian noise of mean zero and variance
0.10 as shown in Figure 5.9(c), after one iteration for consistent comparison.
The table also presents the number of iterations required to achieve conver-
gence. The mean squared error (MSE) records the dierence between the
136
(a) Q-criterion
(b) Weighted average Q
w
-criterion
Figure 5.11: Evaluating Q and Q
w
on the von Karman ow using equations
(5.14) and (5.18), respectively.
ground truth, as shown in Figure 5.9(a) and the smoothed ows. The MSE
plot after smoothing various von Karman ows with noise of increasing vari-
137
ance,
2
[0.01, 0.80] is displayed in Figure 5.12. From this plot it can be
clearly seen that SFS performs better than the other algorithms with respect
to the reduction of MSE.
SFSM SFS MedH-SFS TD
No. iterations 6 1 5 1
MSE 3.917 0.456 2.677 13.282
MCE 0.828 0.842 0.894 0.197

E
(%) 1.37 1.77 1.29 1.37

w,E
(%) 1.73 6.82 1.13 3.53
Table 5.2: Evaluation of smoothed noisy von Karman ows when the noise
variance is 0.10.
0 0.01 0.05 0.10 0.15 0.20 0.25 0.30 0.50 0.60 0.80 1.00
0
2
4
6
8
10
12
14
Noise variance
M
e
a
n

S
q
u
a
r
e

E
r
r
o
r


SFSM
SFS
MedHSFS
TD
Figure 5.12: MSE after smoothing the noisy von Karman ow.
The constraints, Q, Q
w
, and
w
are used to detect the regions of
vorticity, according to > , where is approximately twenty percent of
138
0 0.01 0.05 0.10 0.15 0.20 0.25 0.30 0.50 0.60 0.80 1.00
0
1
2
3
4
5
6
7
8
9
Noise variance
S
e
g
m
e
n
t
a
t
i
o
n

E
r
r
o
r
,

w
,
E

(
%
)


SFSM
SFS
MedHSFS
TD
Figure 5.13: Performance of the vorticity segmentation using
w
.
the maximum discriminant () value. Such vorticity regions are detected
in both the ground truth ows as shown in Figure 5.14(b) as well as in the
smoothed ows, including those from Figures 5.10(a)-(c). The regions of
vorticity characterised by high are segmented and compared between the
segmented regions in both the smoothed and the ground truth eld.
Consequently, the percentage of mis-classication and of false alarms is
measured for the vorticity regions for all the tested algorithms SFS, MedH-
SFS, TD and SFSM, calculated as a percentage of the total number of vectors
in the vector eld. The vorticity error measure denoted as
E
and
w,E
is
provided in Table 5.2 for noise of variance 0.10. Similar to the Q and Q
w
criterions, the dierence between and
w
is their sensitivity to noise.
For example, Figure 5.14 shows the topology map for and
w
, computed
from a MedH-SFS smoothed ow after 1 iteration. In this example, the
initial ow was corrupted with Gaussian noise (variance = 0.01). As it can
139
be observed, the vortex regions in Figure 5.14(a) is visibly faint. However,
the vortex regions in Figure 5.14(b) can easily be identied with the weighted
discriminant,
w
.
The plot in Figure 5.13 shows the performance of vortex segmentation
for various algorithms when increasing the noise variance. Though SFS has
the least error initially, as the amount of noise is increased, its performance
gradually gets worse than MedH-SFS. The mean vortex segmentation error
measurement is calculated as

E
=
1
N
2
N
2

i=1
[1(
O
) 1()[
N
2

i=1
1(
O
)
100 (5.23)
where N
2
is the size of the vector eld, 1(
O
) is the index array of the dis-
criminant, calculated for > 20% of its maximum value for the ground truth
eld and 1() represents the index array of the discriminant, calculated for
> 20% of its maximum value for the smoothed ows. The same equa-
tion can be used for the weighted discriminant,
w,E
. However, it should
be noted that
w,E
gives better results than
E
when the variance is be-
tween 0.01 and 0.20. Thereafter,
E
is signicantly better. This behaviour
is expected for higher noise since there is a larger likelihood of segmentation
mis-classication due to larger
w
values.
The overall observed results show that among the smoothing methods
tested, generally MedH-SFS outperforms the other methods. This method
provides smooth ows and succeeds in recovering all 5 vortex core regions
(from the cropped eld) after 1 iteration. This is veried by the MCE results
140
(a)
(b)
w
Figure 5.14: Evaluating and
w
on a smoothed von Karman ow from
equations (5.17) and (5.19), respectively.
141
presented in Table 5.2, which shows that the MedH-SFS smoothed eld is
the best match to the ground truth ow. Unlike the uid based methods,
TD was unable to smooth the ows reliably and hence could not recover
the vortex regions properly. It is important to note that reconstructions
from TD smoothed ows were very poor, hence reecting the high errors
in Table 5.2. These error measurements are based on the magnitude and
direction of the vector. However, TD smoothed ow does give a reasonable
value for segmentation error. This is possible due to vectors being in the
correct location for segmentation purposes, despite bad ow estimations.
5.6.3 Real image sequences and remarks
The proposed methodology of hybrid uid smoothing is tested on optical
ows estimated from image sequences. Figure 5.15(a) represents a frame
from the Tornado image sequence, while Figure 5.15(b) shows a frame from
the Solar Flare sequence obtained from Kanzelh ohe Obervatorys solar and
environmental research website
1
. The rst sequence represents a complex
atmospheric phenomenon while the second image sequence is used to observe
and analyse solar surface activity.
The initial optical ows have been estimated using block matching algo-
rithm (BMA) and are shown in Figures 5.15(c) and (d), respectively. The
complexity of the motion in the scenes as well as the compression artefacts
inuence negatively the performance of the BMA algorithm. Figure 5.15(e)
and (f) show the smoothing result when using MedH-SFS algorithm on the
optical ow estimated from the Tornado sequence and from the Solar Flare
1
Sequence obtained from http://www.solobskh.ac.at/index en.php
142
(a) Original frame 341 (b) Original frame 220
(c) Initial BMA ow (d) Initial BMA ow
(e) MedH-SFS smoothed ow (f) MedH-SFS smoothed ow
Figure 5.15: Smoothing optical ows in image sequences displaying turbulent
motion.
143
optical ow, respectively, both after one iteration. The recovered optical ow
smoothed with MedH-SFS is a signicant improvement over the initial op-
tical ows. We can clearly identify moving twister and its boundaries after
using the proposed methodology as it is shown in Figure 5.15(e). Turbulent
movements of the solar surface can be properly identied in Figure 5.15(f).
The proposed methodology was also applied on real image sequences
showing atmospheric phenomena such as those that occurred during the Su-
perstorm Andrea. The satellite image sequence of the Superstorm Andrea
contains 24 images and captures the formation of the Superstorm Andrea.
2
The data set consists of 24 images acquired at 30 minute intervals per frame
covering a total period of 12 hours. Frame 22 from the Superstorm Andrea
image sequence is shown in Figure 5.16(a), while the optical ow is calcu-
lated, using the block matching algorithm, between frames 20 and 22 is shown
in Figure 5.18(a). As it can be observed, the ow is very noisy and is inu-
enced by the map information as well as by the ickering of lights (as this
is a night satellite image capture) and by additional geographic information.
The movement of clouds from frame to frame is not very clear when using the
block matching algorithm. Another possible cause of the noise is the random
bursts of clouds in the region. Figure 5.18(b) and Figure 5.18(c) show the
eect of smoothing using MedH-SFS and SFSM algorithms, respectively.
Figure 5.16(b) shows the topology map of the 2D discriminant ex-
tracted from the image sequence showing the turbulent storm. Nine main
concentrations of energy (vortices) are identied on this map and their loca-
tion is marked with distinctive numbers. On the right side of the map (over
the Atlantic Ocean), a concentration of detected vortices can be observed.
2
Sequence obtained from http://www.nrlmry.navy.mil/sat products.html
144
(a) Original reference frame 22 of Superstorm Andrea
(b) Vortex detection from the smoothed Andrea ow using MedH-SFS
based on topology
Figure 5.16: Finding vortices in the Superstorm Andrea sequence.
145
(a) Vortex detection from the smoothed Andrea ow using Blacks
algorithm based on topology
(b) Vortex detection from the smoothed Andrea ow using MED-2DH
based on
w
topology
Figure 5.17: Additional results of vortex segmentation of Superstorm Andrea
sequence.
146
(a) Initial ow from frame 20 to frame 22
(b) Smoothed ow using MedH-SFS
(c) Smoothed ow using SFSM
Figure 5.18: Smoothing optical ow corresponding to cloud movement for
Superstorm Andrea.
147
This area corresponds to the region of main activity for the Superstorm
Andrea. These results show clearly that the region of principal turbulent ac-
tivity, as recorded by the movement of clouds observed in this satellite image
sequence takes place mainly above the oceans. Further results showing topol-
ogy maps of and
w
are shown in Figures 5.17(a) and (b) respectively.
The segmentation shown in (a) is obtained from Black [7] smoothed ow.
In contrast, the segmentation in (b) is obtained from MED-2DH smoothed
ow. The main dierences between the results shown is that when using
w
,
the segmentation is clearer and vortex core regions can be clearly identied.
This shows that the
w
produces a clear improvement over .
In addition to the vortex region identication for the Superstorm Andrea
sequence, the algorithm is also tested on the Solar Flare sequence, as shown
in Figure 5.19. The regions of high energy in Figure 5.19(b) correlates to the
turbulent motion observed in Figure 5.19(a). An interesting point to note
is that some of these high energy regions do not mimic vortex behaviour.
Instead, they are either converging or diverging from points which are the
focus of particle movements.
From these results, the benets of using a robust diusion stage integrated
in the computational uid dynamics methodology such as that implemented
by Navier-Stokes equation is that it not only smoothes the vector eld, but
also contributes to detecting vortices and other regions of concentrated tur-
bulence. The proposed measure of vorticity detection using
w
proved to
give reliable and accurate estimates of the areas with signicant turbulence.
148
(a) Smoothed MedH-SFS ow after 6 iterations
(b) Vortex detection from the smoothed ow in (a) based on topology
Figure 5.19: Finding vortices in the Solar Flare sequence.
5.7 Conclusion
A uid dynamics based model for smoothing optical ow from image se-
quences representing complex and turbulent ows has been presented. The
Stable Fluid Solver (SFS) method is based on the Navier-Stokes equations.
The SFS method was originally used for computer graphics and visualisa-
tion applications. In this thesis, the SFS method has been modied for the
149
specic use on optical ows displaying turbulent motion.
An important point to note is that the optical ows used for the ex-
periments do obey incompressible and Newtonian assumptions. Although
optical ows are obtained from 3D real time scenes with changeable condi-
tions, the ows themselves are constrained by the boundaries of the eld at
every snapshot. No additional physical conditions could therefore aect the
ows. Furthermore, the volume given by the number of vectors of the ows
is conserved, since it is dependent on the motion estimation algorithm and
the boundaries are consequently dened.
The motion is estimated following a set of processing steps implementing
reinforcement, diusion, advection and mass conservation. An explicit dif-
fusion model is used after the reinforcement stage and before the advection
stage. The median of the local Hessian-based kernel is considered for the
diusion stage. This type of kernel ensures that smoothing occurs along the
structure of the motion eld maintaining the moving objects boundaries and
the main optical ow features. Moreover, the kernel embeds robust statis-
tics capability by reducing the impact of outliers and thus enhancing the
smoothness of the resulting optical ow.
After modelling the optical ow using the proposed method, an approach
for identifying salient patterns such as vortices (coherent structures) is used.
This method uses the Q-criterion and the discriminant , as described in
Section 5.5 to identify, segment and track vortices. Vortices are spatial re-
gions characterised by rotational movement where the Euclidean norm of the
vorticity tensor dominates that of the rate of the tensor. In this thesis, we
propose to use weighted estimates of these measures, calculated from all the
150
possible combinations of four vectors distributed at the extremes of a cross
from a given window. The proposed methodology is applied on articial
vector elds as well as on the turbulent ow representing natural phenom-
ena. The proposed methodology can be applied for identifying and detecting
turbulent phenomena from satellite image sequence and it can be used for
tracking storms.
151
Chapter 6
Conclusion and Outlook
This conclusion chapter commences with a summary of the contributions of
Chapters 3 to 5 which includes a discussion of the strengths and weaknesses
of the nonlinear diusion techniques. Finally, directions for future work will
be discussed.
6.1 Summary of Contributions
This thesis has three main contributions in the area of vector eld diusion.
6.1.1 Robust nonlinear diusion
The rst contribution to the computer vision eld is presented in Chapter
3. Here, we briey outline the our methodology for a robust anisotropic
diusion algorithm to smooth optical ows.
The proposed nonlinear diusion algorithms in Chapter 3 are kernel based
lters and are dependent on the information of the local Hessian. The kernel
is anisotropic due to the Hessian matrix, which provides orientation infor-
152
mation of the data structure and hence copes better with identifying edges
and specic data transition patterns. The Hessian based kernels are made
statistically robust using the median or the alpha-trimmed mean algorithms
for better performance. In contrast to other known methods, the proposed
diusion kernels are primarily designed for use on optical ows from image
sequences displaying object deformation over time, although the kernels can
be adapted for use on images as well. The diusion algorithms were tested on
synthetic vector elds and on optical ows extracted from real-world image
sequences displaying various types of motion. The algorithms were compared
against various other methods [4, 6, 7, 34].
6.1.2 3D volumetric slice interpolation
3D slice interpolation technique has been around for a while and has pre-
viously been used for 3D volumetric reconstructions [14, 72]. Furthermore,
there are many dierent types of techniques being used to obtain interme-
diate slices for the reconstruction. One example being the work of Weng
et. al. [72] which uses optical ows.
We redene optical ows as structural ows, since these describe bet-
ter the deformations between image slices of soft and hard tissue of human
anatomy and can be used on medical images in general. The ows are also
able to capture the direction of textural change within the inner structure
of the objects. Most ow algorithms are uni-directional. In contrast, the
proposed contribution in Chapter 4 is bi-directional and thus employs addi-
tional information to appropriately model the morphing between two original
slices. Robust diusion kernels, as proposed in Chapter 3, are then used to
153
remove outliers, especially around the boundaries in order to give better
and more accurate morphing. Intermediate slices are then obtained using
the information from the structural ows and original slices. The collated
slices, including the originals are then stacked to reconstruct a 3D volume,
as suggested in [72]. The Hausdor distance measure is used to give a better
representation on the 3D shape reconstruction of the volume. The proposed
framework has been tested on three hard tissue data sets and three soft tissue
data sets. The data sets were a mixture of pre-segmented and unsegmented
slices. Currently, only volumes of segmented sets can be rendered, i.e. In-
cisor, Humerus and Iliac bones. Three measurements were taken to validate
the volumetric reconstructions. They are average 3D shape reconstruction
error of intermediate slices, average peak signal-to-noise ratio of intermediate
slices and the Hausdor distance of a 3D synthesised volume compared to
the ground truth information.
6.1.3 Diusion based on uid dynamics
There is a huge amount of literature dedicated to using Navier-Stokes equa-
tions in the eld of uid mechanics. Navier-Stokes based methodology has
been used in computer graphics for simulating various natural eects. Main
reasons for the slow adoption of this method in computer vision is due to its
nonlinear characteristics, diculty in understanding, complex implementa-
tions and its heavy computational complexity.
A new approach based on Navier-Stokes equations is used for modelling
on optical ows. The main purpose of using this method is for better esti-
mation of ows from image sequences when complex and turbulent motion
154
is present, for example, for the modelling of vortex ow, explosive sequences,
sequences showing storm motion in satellite imagery, etc. Since optical ows
are constrained by construction, it is assumed that they exhibit incompress-
ible and Newtonian properties. The proposed method incorporates both
explicit and implicit dierencing schemes, and a robust outlier mechanism.
The diusion lter is anisotropic in design and the results obtained are com-
parable to standard diusion algorithms.
An improved vortex detection method is also detailed in Chapter 5. Vor-
tex detection is an important concept, since this enables engineers and re-
searchers to identify weak structures or potentially damaging natural ele-
ments. For structures where a smooth stream ow is needed, perturbations
of the eld is kept to a minimum to avoid breakdown of the overall ow.
Hence, the dynamics of a vortex is of immense interest. Though there are
numerous measurements [100, 102, 136] available to identify the vortex struc-
tures, we propose a computationally eective and robust method to identify
vortices in image sequences showing movement of turbulent uids.
Although the proposed algorithm is fast and ecient, the resulting smoo-
thed optical ows from the experiments are still fairly coarse when compared
to specialised anisotropic diusion lters.
6.2 Outlook
This thesis presents three dierent contributions to the eld of computer
vision. As such, continuation of the research presented is vital for continued
improvements of methods and possible contributions to future applications,
and to the advancement of the eld. It is suggested that the Hessian based
155
algorithms can be improved and optimised further for better performance.
The methodology pertaining to 3D volume reconstruction can be im-
proved for more accurate representation. Currently, this method is for seg-
mented hard tissue based structures, although there has been some degree
of success reconstructing soft tissue structures. There is scope to conduct
extensive research to soft tissue based organs.
Using Navier-Stokes equations in computer vision is still a new concept.
This is especially true in the area of motion estimation. The results presented
in the thesis are preliminary results and further detailed investigations are
needed for mainstream research. Though the proposal in the thesis is a
diusion lter methodology, it is much better to develop a new optical ow
technique which is based on the Navier-Stokes equations with an integrated
regularisation mechanism. There is ongoing work on this technique [24] and
is based on the energy minimisation scheme of Horn and Schunk [2]. It
is anticipated that this technique is better suited to modelling the motion
of complex and turbulent scenes such as waterfalls, hurricanes and twisters
(from satellite imagery), solar phenomena, human motion analysis and cloth
dynamics.
It is thought that uid based optical ows has many potential applications
in other elds, such as climatology, weather forecasting, gait analysis, facial
feature detection and computer graphics.
156
Appendix A
Dataset
The original Incisor dataset of the 22 slices is shown in Figure A.1. Interpo-
lated structural ows are obtained from the sparse dataset with segmented
foreground. 2D intermediary slices between original slices are reconstructed
from the structural ows. The 2D slices are then stacked and aligned to
reconstruct 3D volumes of the Incisor, as presented in Chapter 4.
157
Figure A.1: Incisor dataset.
158
List of References
[1] A. M. Tekalp. Digital video processing. Prentice Hall, Upper Saddle
River, NJ, 1995.
[2] B. K. P. Horn and B. G. Schunck. Determining optical ow. Tech-
nical Report A.I. Memo No. 572, Articial Intelligence Laboratory,
Massachusetts Institute of Technology, 1980.
[3] B. Lucas and T. Kanade. An iterative image registration technique
with an application to stereo vision. In International Joint Conference
on Articial Intelligence, pages 674679, Vancouver, Canada, 1981.
[4] P. Perona and J. Malik. Scale-space and edge detection using
anisotropic diusion. IEEE Transactions on Pattern Analysis and Ma-
chine Intelligence, 12(7):629639, July 1990.
[5] D. Tschumperl e and R. Deriche. Orthonormal vector sets regulariza-
tion with PDEs and applications. International Journal of Computer
Vision, 50(3):237252, 2002.
[6] D. Tschumperl e and R. Deriche. Vector-valued image regularization
with PDEs: a common framework for dierent applications. In IEEE
159
Computer Vision and Pattern Recognition, Madison, United States,
June 2003.
[7] M. J. Black, G. Sapiro, D. H. Marimont, and D. Heeger. Ro-
bust anisotropic diusion. IEEE Transactions on Image Processing,
7(3):421432, 1998.
[8] M. Bertalmio, L. Vese, G. Sapiro, and S. Osher. Simultaneous structure
and texture image inpainting. IEEE Transactions on Image Processing,
12(8):882889, 2003.
[9] I. Giakoumis and I. Pitas. Digital restoration of painting cracks. In
Proceedings of IEEE International Symposium on Circuits and Sys-
tems, volume 4, pages 269272, California, United States, 31 May3
June 1998.
[10] A. G. Bors, E. R. Hancock, and R. C. Wilson. Terrain analysis using
radar shape-from-shading. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 25(9):974992, 2003.
[11] B. Horn and B. Schunck. Determining optical ow. Articial Intelli-
gence, 17:185203, 1981.
[12] J. L. Barron, D. J. Fleet, and S. Beauchemin. Performance of optical
ow techniques. International Journal of Computer Vision, 12(1):43
77, 1994.
[13] A. Bab-Hadiashar and D. Suter. Robust optic ow computation. In-
ternational Journal of Computer Vision, 29(1):5977, 1998.
160
[14] A. G. Bors and I. Pitas. Optical ow estimation and moving object
segmentation based on median radial basis function network. IEEE
Transactions on Image Processing, 7(5):693702, May 1998.
[15] A. Verri and T. Poggio. Motion eld and optical ow: qualitative
properties. Technical Report A.I. Memo No. 917, Articial Intelli-
gence Laboratory and Center for Biological Information Processing,
Whitaker College, Massachusetts Institute of Technology, December
1986. C.B.I.P. Memo No. 22.
[16] A. G. Bors and I. Pitas. Prediction and tracking of moving objects in
image sequences. IEEE Transactions on Image Processing, 9(8):1441
1445, August 2000.
[17] N. P. de la Blanca, J. M. Fuertes, M. Lucena, and A. Garrido. Local
motion estimation from stereo image sequences. In F. J. Perales et. al.,
editor, IbPRIA, LNCS 2652, pages 740747. Springer-Verlag, 2003.
[18] C. E. Erdem, G. Z. Karabulut, E. Yanmaz, and E. Anarim. Motion
estimation in the frequency domain using fuzzy c-planes clustering.
IEEE Transactions on Image Processing, 10(12):18731879, December
2001.
[19] A. M. Peacock, D. Renshaw, and J. Hannah. A fuzzy data fusion
method for improved motion estimation. In Advanced Concepts for
Intelligent Vision Systems, Baden-Baden, Germany, August 1999.
161
[20] H. Foroosh. Adaptive estimation of motion using generalized cross
validation. In 3rd International (IEEE) Workshop on Statistical and
Computational Theories of Vision, 2003.
[21] M. Elad and A. Feuer. Recursive optical ow estimation - adaptive
ltering approach. In The 19th IEEE Conference in Israel, Jerusalem,
November 1996.
[22] H. Liu, R. Chellappa, and A. Rosenfeld. Accurate dense optical ow es-
timation using adaptive structure tensors and parametric model. IEEE
Transactions on Image Processing, 12(10):11701180, October 2003.
[23]

E. Memin and P. Perez. Hierarchical estimation and segmentation
of dense motion elds. International Journal of Computer Vision,
46(2):129155, 2002.
[24] N. Papenberg, A. Bruhn, T. Brox, S. Didas, and A. Weickert. Highly
accurate optic ow computation with theoretical justied warping. In-
ternational Journal of Computer Vision, 67(2):141158, 2006.
[25] S. Roth and M. J. Black. On the spatial statistics of optical ow. In
IEEE International Conference on Computer Vision, volume 1, pages
4249, Beijing, China, 2005.
[26] A. Bruhn, J. Weickert, and C. Schnorr. Lucas/Kanade meets Horn/Sc-
hunck: combining local and global optic ow methods. International
Journal of Computer Vision, 61(3):211231, 2005.
[27] J. J. Koenderink. The structure of images. Biological Cybernetics,
50:363370, 1984.
162
[28] D. V. Widder. The heat equation. Academic Press, 1975.
[29] S.-T. Yau. Surveys in dierential geometry: dierential geometry in-
spired by string theory, volume 5. International Press, 1999.
[30] R. A. Hummel, B. Kimia, and S. W. Zucker. Gaussian blur and the
heat equation: forward and inverse solutions. In IEEE International
Conference on Computer Vision and Pattern Recognition, pages 668
671, 1985.
[31] R. A. Hummel, B. Kimia, and S. W. Zucker. Deblurring gaussian blur.
Computer Vision, Graphics, Image Processing, 38(1):6680, 1987.
[32] G. Unal, H. Krim, and A. Yezzi. Stochastic dierential equations and
geometric ows. IEEE Transactions on Image Processing, 11(12):1405
1416, 2002.
[33] F. Catte, P.-L. Lions, J.-M. Morel, and T. Coll. Image selective smooth-
ing and edge detection by nonlinear diusion. SIAM Journal on Nu-
merical Analysis, 29(1):182193, 1992.
[34] J. Weickert, B. M. T. Romeny, and M. A. Viergever. Ecient and
reliable schemes for nonlinear diusion ltering. IEEE Transactions
on Image Processing, 7(3):398410, 1998.
[35] R. A. Hummel. Representation based on zero-crossings in scale space.
In IEEE International Conference on Computer Vision and Pattern
Recognition, June 1986.
[36] A. R. Gourlay. Implicit convolution. Image Vision Computing, 3:1523,
1985.
163
[37] L.-D. Cai. Some notes on repeated averaging smoothing. In Proceedings
of the 4th International Conference on Pattern Recognition, pages 697
605, London, UK, 1988. Springer-Verlag.
[38] L. Alvarez and L. Mazorra. Signal and image restoration using shock
lters and anisotropic diusion. SIAM Journal on Numerical Analysis,
31(2):590605, 1994.
[39] D. Zhao and B. Li. A new implementation of discrete multiscale lter-
ing. In IEEE International Conference on Image Processing, volume 1,
pages 383385, Lausanne, Switzerland, September 1996.
[40] R. Deriche. Fast algorithms for low-level vision. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 12(1):7887, January 1990.
[41] R. Deriche. Recursively implementing the gaussian and its derivatives.
Technical Report 1893, INRIA, May 1993.
[42] M. Nielsen, L. Florack, and R. Deriche. Regularization, scale-space and
edge detection lters. Journal of Mathematical Imaging and Vision,
7(4), 1997.
[43] S. T. Acton, A. C. Bovik, and M. M. Crawford. Anisotropic diusion
pyramids for image segmentation. In IEEE International Conference
on Image Processing, pages 478482, 1994.
[44] J. Fr ohlich and J. Weickert. Image processing using a wavelet algorithm
for nonlinear diusion. Technical Report 104, Laboratory of Techno-
mathematics, University of Kaiserslautern, Kaiserslautern, Germany,
March 1994.
164
[45] B. L. Luck, K. D. Carlson, A. C. Bovik, and R. R. Richards-Kortum.
A image model and segmentation algorithm for reectance confocal
images of In Vivo cervical tissue. IEEE Transactions on Image Pro-
cessing, 14(9):12651276, 2005.
[46] C. Wallraven, B. Caputo, and A. B. A. Graf. Recognition with lo-
cal features: the kernel recipe. In IEEE International Conference on
Computer Vision, volume 2, pages 257264. IEEE Press, 2003.
[47] Y. Kanazawa and K. Kanatani. Do we really have to consider covari-
ance matrices for image feature points? Electronics and Communica-
tions in Japan part III - Fundamental Electronic Science, 86(1):110,
2003.
[48] P.-Y. Burgi. Motion estimation based on the direction of intensity
gradient. Image and Vision Computing, 22:637653, 2004.
[49] A. Petrovic, E. O. Divorra, and P. Vandergheynst. Multiresolution
segmentation of natural images: from linear to non-linear scale-space
representations. IEEE Transactions on Image Processing, 2004.
[50] T. Preusser and M. Rumpf. Extracting motion velocities from 3D image
sequences. In SPIE Conference on Visualization and Data Analysis,
2003.
[51] R. Archibald, J. Hu, A. Gelb, and G. Farin. Improving the accuracy
of volumetric segmentation using pre-processing boundary detection
and image reconstruction. IEEE Transactions on Image Processing,
13(4):459466, April 2004.
165
[52] K. I. Kim, M. O. Franz, and B. Sch olkopf. Kernel hebbian algorithm for
single-frame super-resolution. In A. Leonardis and H. Bischof, editors,
Statistical Learning in Computer Vision, pages 135149, 2004.
[53] S. Mika, B. Sch olkopf, A. J. Smola, K.-R. M uller, M. Scholz, and
G. R atsch. Kernel PCA and de-noising in feature spaces. In M. S.
Kearns, S. A. Solla, and D. A. Cohn, editors, Advances in Neural In-
formation Processing Systems 11, pages 536542. MIT Press, 1999.
[54] S. Mika, G. R atsch, and K.-R. M uller. A mathematical programming
approach to the kernel sher algorithm. In T. K. Leen, T. G. Dietterich,
and V. Tresp, editors, Advances in Neural Information Processing Sys-
tems 13, pages 591597. MIT Press, 2001.
[55] A. G. Bors, L. Kechagias, and I. Pitas. Binary morphological shape-
based interpolation applied to 3D tooth reconstruction. IEEE Trans-
actions on Medical Imaging, 21(2):100109, 2002.
[56] T.-Y. Lee and W.-H. Wang. Morphology based three dimensional in-
terpolation. IEEE Transactions on Medical Imaging, 19(7):711721,
July 2000.
[57] D. H. Frakes et. al. Application of an adaptive control grid interpolation
technique to morphological vascular reconstruction. IEEE Transactions
on Biomedical Engineering, 50(2):197206, February 2003.
[58] T. M. Lehmann et. al. Survey: interpolation methods in medical image
processing. IEEE Transactions on Medical Imaging, 18(11):10491075,
November 1999.
166
[59] G. J. Grevera and J. K. Udupa. An objective comparison of 3D im-
age interpolation methods. IEEE Transactions on Medical Imaging,
17(4):642652, 1998.
[60] S. Schaller et. al. Spiral interpolation algorithm for multislice spiral CT
- part I: theory. IEEE Transactions on Medical Imaging, 19(9):822834,
September 2000.
[61] T.-Y. Lee and C.-H. Lin. Feature-guided shape-based image inter-
polation. IEEE Transactions on Medical Imaging, 21(12):14791489,
December 2002.
[62] C. A. Johnson, J. Seidel, and A. Sofer. Interior-point methodology
for 3D PET reconstruction. IEEE Transactions on Medical Imaging,
19(4):271285, April 2000.
[63] P. T. Fletcher, C. Lu, S. M. Pizer, and S. Joshi. Principal geodesic anal-
ysis for the study of nonlinear statistics of shape. IEEE Transactions
on Medical Imaging, 23(8):9951005, August 2004.
[64] S. C. Joshi, S. M. Pizer, P. T. Fletcher, P. A. Yushkevich, A. Thall,
and J. S. Marron. Multi-scale deformable model segmentation and
statistical shape analysis using medial descriptions. IEEE Transactions
on Medical Imaging, 21(5):538550, 2002.
[65] J. M. Sanches and J. S. Marques. Joint image registration and volume
reconstruction for 3D ultrasound. Pattern Recognition Letters, 24:791
800, 2003.
167
[66] A. Pitiot, G. Malandain, E. Bardinet, and P. M. Thompson. Piece-
wise ane registration of biological images. In J. C. Gee, J. B. A.
Maintz, and M. W. Vannier, editors, Second International Workshop
on Biomedical Image Registration, LNCS 2717, pages 91101. Springer-
Verlag, 2003.
[67] Y.-P. Wang, Y. Chen, and A. A. Amini. Fast LV motion estimation us-
ing subspace approximation techniques. IEEE Transactions on Medical
Imaging, 20(6):499513, June 2001.
[68] L. Dougherty et. al. Validation of an optical ow method for tag
displacement estimation. IEEE Transactions on Medical Imaging,
18(4):359363, April 1999.
[69] G. P. Penney et. al. Registration-based interpolation. IEEE Transac-
tions on Medical Imaging, 23(7):922926, July 2004.
[70] W. Higgins, C. Morice, and E. Ritman. Shape-based interpolation of
tree-like structures in three-dimensional images. IEEE Transactions
on Medical Imaging, 12(3):439450, 1993.
[71] G. Turk and J. OBrien. Shape transformation using variational im-
plicit functions. In SIGGRAPH, pages 335342, August 1999.
[72] N. Weng, Y.-H. Yang, and R. Pierson. Three-dimensional surface re-
construction using optical ow for medical imaging. IEEE Transactions
on Medical Imaging, 16(5):630641, 1997.
168
[73] N. Hata et. al. Three-dimensional optical ow method for measure-
ment of volumetric brain deformation from intraoperative MR images.
Journal of Computer Assisted Tomography, 24:531538, July 2000.
[74] M. D. Abramo and M. A. Viergever. Computation and visualization
of three-dimensional soft tissue motion in the orbit. IEEE Transactions
on Medical Imaging, 21(4):296304, April 2002.
[75] L. Weruaga, J. Morales-Sanchez, L. Nunez, and R. Verd u. Volumetric
motion estimation in human thorax for radiotherapy planning. IEEE
Transactions on Medical Imaging, 22(6):766772, 2003.
[76] N. Ray and S. T. Acton. Motion gradient vector ow: an external force
for tracking rolling leukocytes with shape and size constrained active
contours. IEEE Transactions on Medical Imaging, 23(12):14661478,
2004.
[77] D. Rueckert, L. I. Sonoda, C. Hayes, D. L. G. Hill, M. O. Leach, and
D. J. Hawkes. Non-rigid registration using free-form deformations: ap-
plication to breast MR images. IEEE Transactions on Medical Imaging,
18(8):712721, 1999.
[78] A. Rao, R. Chandrashekara, G. I. Sanchez-Ortiz, R. Mohiaddin, P. Al-
jabar, J. V. Hajnal, B. K. Puri, and D. Rueckert. Spatial transforma-
tion of motion and deformation elds using nonrigid registration. IEEE
Transactions on Medical Imaging, 23(9):10651076, 2004.
169
[79] Y. Cao, M. I. Miller, R. L. Winslow, and L. Younes. Large deformation
dieomorphic metric mapping of vector elds. IEEE Transactions on
Medical Imaging, 24(9):12161230, 2005.
[80] N. G. Sharp and E. R. Hancock. Density propagation for surface track-
ing. Pattern Recognition Letters, 19(2):177188, 1998.
[81] M. Garza-Jinich, P. Meer, and V. Medina. Robust retrieval of 3D struc-
tures from image stacks. Medical Image Analysis, 3(1):2135, 1999.
[82] A. Sarti, K. Mikula, and F. Sgallari. Nonlinear multiscale analysis of 3D
echocardiographic sequences. IEEE Transactions on Medical Imaging,
18:453466, 1999.
[83] D. H. Chung and G. Sapiro. Segmenting skin lesions with partial dif-
ferential equations based image processing algorithms. In IEEE Inter-
national Conference on Image Processing, 2000.
[84] K. Krissian. Flux-based anisotropic diusion applied to enhancement of
3D angiogram. IEEE Transactions on Medical Imaging, 21(11):1440
1442, 2002.
[85] K. Z. Abd-Elmoniem, A.-B. M. Youssef, and Y. M. Kadah. Real-time
speckle reduction and coherence enhancement in ultrasound imaging
via nonlinear anisotropic diusion. IEEE Transactions on Biomedical
Engineering, 49(9):9971014, 2002.
[86] J. Ling and A. C. Bovik. Smoothing low SNR molecular images via
anisotropic median-diusion. IEEE Transactions on Medical Imaging,
21(4):377384, 2002.
170
[87] M. K. Chung. Heat kernel smoothing and its application to cortical
manifolds. Technical Report TR 1090, Department of Statististics,
Universisty of Wisconsin-Madison, USA, 2004.
[88] J. M. Fitzpatrick. A method for calculating velocity in time depen-
dent images based on the continuity equation. In IEEE International
Conference on Computer Vision and Pattern Recognition, pages 7881,
1985.
[89] S. M. Song and R. M. Leahy. Computation of 3D velocity elds from
3D cine CT images of a human heart. IEEE Transactions on Medical
Imaging, 10(1):295306, 1991.
[90] A. J. Chorin. Numerical solution of the navier-stokes equations. Math-
ematics of Computation, 22(104):745762, October 1968.
[91] J. D. Anderson Jr. Computational uid dynamics: the basics with
applications. McGraw Hill, 1995.
[92] Y. Nakajima, H. Inomata, H. Nogawa, Y. Sato, S. Tamura, K. Okazaki,
and S. Torii. Physics-based ow estimation of uids. Pattern Recogni-
tion, 36:12031212, 2003.
[93] M. Bertalmio, A. L. Bertozzi, and G. Sapiro. Navier-stokes, uid dy-
namics, and image and video inpainting. In IEEE Conference on Com-
puter Vision and Pattern Recognition, volume 1, pages 355362, 2001.
[94] T. Corpetti,

E. M emin, and P. P erez. Dense estimation of uid
ows. IEEE Transactions on Pattern Analysis and Machine Intelli-
gence, 24(3):365380, March 2002.
171
[95] T. Kohlberger, C. Schnorr, A. Bruhn, and J. Weickert. Domain decom-
position for variational optical-ow computation. IEEE Transactions
on Image Processing, 14(8):11251137, August 2005.
[96] S. Premoze, T. Tasdizen, J. Bigler, A. Lefohn, and R. T. Whitaker.
Particle-based simulation of uids. In EUROGRAPHICS, Computer
Graphics Forum 22(3), pages 401410, 2003.
[97] J. Stam. A simple uid solver based on the FFT. Journal of Graphics
Tools, 6(2):4352, 2001.
[98] J. Stam. Real-time uid dynamics for games. In Game Developer
Conference, volume 4, pages 7692, March 2003.
[99] A. Treuille, A. McNamara, Z. Popovic, and J. Stam. Keyframe control
of smoke simulations. ACM Transactions On Graphics, 22(3):716723,
July 2003.
[100] J. Jeong and F. Hussain. On the identication of a vortex. Journal of
Fluid Mechanics, 285:6994, 1995.
[101] J. C. Hunt, A. A. Wray, and P. Moin. Eddies, stream, and convergence
zones in turbulent ows. Technical Report CTR-S88, NASA Ames,
Stanford University, USA, 1988.
[102] G. Haller. An objective denition of a vortex. Journal of Fluid Me-
chanics, 525:126, 2005.
[103] J. Zhong, T. S. Huang, and R. J. Adrian. Extracting 3D vortices
in turbulent uid ow. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 20(2):193199, 1998.
172
[104] R. M. Ford and R. N. Strickland. Representing and visualizing uid ow
images and velocimetry data by nonlinear dynamical systems. Graph-
ical Models and Image Processing, 57(6):462482, 1995.
[105] R. S. Laramee, H. Hauser, L. Zhao, and F. H. Post. Topology-based
ow visualization, the state of the art. In H. Hauser, H. Hagen, and
H. Theisel, editors, Topology-Based Methods in Visualization, Visual-
ization and Mathematics, pages 119. Springer-Verlag, 2007.
[106] R. Peikert and M. Roth. The parallel vectors operator - a vector eld
visualization primitive. IEEE Visualization, pages 263270, October
1999.
[107] D. Bauer and R. Peikert. Vortex tracking in scale-space. In Joint
Eurographics IEEE TCVG Symposium on Visualization, pages 140
147, May 2002.
[108] F. Sadlo, R. Peikert, and E. Parkinson. Vorticity based ow analysis
and visualization for pelton turbine design optimization, October 2004.
[109] F. Sadlo, R. Peikert, and M. Sick. Visualization tools for vorticity
transport analysis in incompressible ow. IEEE Transactions on Visu-
alization and Computer Graphics, 12(5):949956, 2006.
[110] J. Sahner et. al. Vortex and strain skeletons in eulerian and lagrangian
frames. IEEE Transactions on Visualization and Computer Graphics,
13(5):980990, 2007.
[111] A. Cuzol and

E. Memin. Vortex and source particles for uid motion
estimation. In Scale-Space, volume 3459, pages 254266, 2005.
173
[112] L. Zhou, C. Kambhamettu, D. B. Goldgof, K. Palaniappan, and A. F.
Hasler. Tracking nonrigid motion and structure from 2D satellite cloud
images without correspondences. IEEE Transactions on Pattern Anal-
ysis and Machine Intelligence, 23(11):13301336, 2001.
[113] M. Jiang, R. Machiraju, and D. Thompson. Detection and visualiza-
tion of vortices. In C. Johnson and C. Hansen, editors, Visualization
Handbook, pages 295309. Academic Press, 2005.
[114] J. Laerty and G. Lebanon. Diusion kernels on statistical mani-
folds. Technical Report CMU-CS-04-101, School of Computer Science,
Carnegie Mellon University, Pittsburgh, PA, United States, January
2004.
[115] I. Pitas and A. N. Venetsaonopoulos. Nonlinear digital lters: princi-
ples and applications. Kluwer Academic, Norwell, MA, 1990.
[116] A. G. Bors and I. Pitas. Object classication in 3D images using alpha-
trimmed mean radial basis function network. IEEE Transactions on
Image Processing, 8(12):17301744, December 1999.
[117] A. Doshi and A. G. Bors. Robustied heat kernel smoothing of struc-
tural ows for volumetric image interpolation. In IEEE International
Conference on Image Processing, pages 12251228, Atlanta, GA, USA,
2006.
[118] G. James. Modern engineering mathematics. Prentince Hall, third
edition, 2001.
174
[119] G. J. Grevera and J. K. Udupa. Shape-based interpolation of multidi-
mensional grey-level images. IEEE Transactions on Medical Imaging,
15:881892, December 1996.
[120] A. Doshi and A. G. Bors. Optical ow diusion with robustied ker-
nels. In LNCS 3691, International Conference on Computer Analysis
of Images and Patterns, pages 213221, Versailles, France, 2005.
[121] I. Mikic, S. Krucinski, and J. D. Thomas. Segmentation and track-
ing in echocardiographic sequences: active contours guided by optical
ow estimates. IEEE Transactions on Medical Imaging, 17(2):274284,
1998.
[122] Z. Xie and G. E. Farin. Image registration using hierarchical b-splines.
IEEE Transactions on Visualisation and Computer Graphics, 10(1):85
94, 2004.
[123] D. Zoccolan and V. Torre. Using optical ow to characterize sensory-
motor interactions in a segment of the medicinal leech. The Journal of
Neuroscience, 22(6):22832298, March 2002.
[124] P. L. Worthington and E. R. Hancock. New constraints on data close-
ness and needle map consistency for shape-from-shading. IEEE Trans-
actions on Pattern Analysis and Machine Intelligence, 21(12):1250
1267, 1999.
[125] Y. Sato et. al. Three-dimensional multi-scale line lter for segmentation
and visualization of curvilinear structures in medical images. Medical
Image Analysis, 2(2):143168, 1998.
175
[126] A. P. Huttenlocher and W. J. Rucklidge. A multi-resolution technique
for comparing images using hausdor distance. Technical Report TR
92-1321, Department of Computer Science, Cornell University, 1992.
[127] J. Stam. Stable uids. In SIGGRAPH, pages 121128, August 1999.
[128] N. Foster and D. Metaxas. Modeling the motion of a hot, turbulent
gas. Computer Graphics, 31(Annual Conference Series):181188, 1997.
[129] B. Galanti, J. D. Gibbon, and M. Heritage. Vorticity alignment results
for the three-dimensional euler and navierstokes equations. Nonlin-
earity, 10:16751694, 1997.
[130] J. R. Cebral, M. Hernandez, and A. F. Frangi. Computational analysis
of blood ow dynamics in cerebral aneurysms from CT and 3D rota-
tional angiography image data. In International Congress on Compu-
tational Bioengineering, Zaragoza, Spain, 2003.
[131] S. Turek, M. Schafer, and R. Rannacher. Evaluation of a CFD bench-
mark for laminar ows. In H. G. Bock, Y. A. Kuznetsov, R. Glowinski,
J. Periaux, and R. Rannacher, editors, Proceedings of the 2nd Euro-
pean Conference on Numerical Mathematics and Advanced Applica-
tions, pages 549563. World Science Publishing, 1998.
[132] S. Turek. A generalized mean intensity approach for the numerical
solution of the radiative transfer equation. Computing, 54(1):2738,
1995.
176
[133] A. Doshi and A. G. Bors. Robust diusion kernels for optical ow
smoothing. In IEEE Workshop on Machine Learning for Signal Pro-
cessing, pages 415420, Maynooth, Ireland, 2006.
[134] A. J. Majda and A. L. Bertozzi. Vorticity and Incompressible Flow.
Cambridge University Press, December 2001.
[135] A. Agrawal and A. K. Prasad. Organizational modes of large-scale
vortices in an axisymmetric turbulent jet. Flow, Turbulence and Com-
bustion, 68(4):359377, 2002.
[136] R. M. Ford. Critical point detection in uid ow images using dynam-
ical system properties. Pattern Recognition, 30(12):19912000, 1997.
[137] S. Popinet. Gerris: a tree-based adaptive solver for the incompress-
ible euler equations in complex geometries. Journal for Computational
Physics, 190(2):572600, 2003.
177