This action might not be possible to undo. Are you sure you want to continue?

of

Vector Fields

Ashish Doshi, BEng.(Hons), MIET

Submitted for the degree of Doctor of Philosophy

Department of Computer Science

August, 2008

Copyright c _ by Ashish Doshi, 2008.

All rights reserved. No part of this work may be reproduced or transmitted

in any form or by any means, electronic or mechanical, including photocopy,

recording or any information storage or retrieval system, without permission

in writing from the author.

To,

Savita

Abstract

Diﬀusion methods are fundamentally important to reduce the eﬀects of

noise whilst enhancing edges and the general perception of images. Although

diﬀusion is primarily applied to images, applying such methods on vectorial

ﬁelds is an important and challenging problem under the conditions of uncer-

tainty and corruption which exist in most vectorial ﬁeld estimation problems.

In this thesis, a novel anisotropic diﬀusion method is developed which

is based on the dynamics of the local Hessian. The proposed algorithm is

kernel based derived from the heat equation. Statistically robust methods

such as the median and the alpha-trimmed mean are integrated within the

algorithm. The result is a smoothed optical ﬂow with reduced outliers, given

an initial noisy ﬂow. The technique is evaluated on synthetic data as well as

vector ﬁelds extracted from real-world image sequences.

Further study on the eﬀects of diﬀusion on structural ﬂows is conducted.

Structural ﬂows are obtained from real-world medical data for the purpose

of 3D volumetric interpolation from 2D sparse data. As a result, the eﬀects

of smoothing are visualised in reconstructed 3D volumes.

In addition to the proposed robust diﬀusion methods, a computational

ﬂuid dynamics based nonlinear model for optical ﬂow smoothing is intro-

duced. An experimental analysis of the model which uses the Navier-Stokes

equation, is carried out to verify its suitability for motion smoothing of real-

world complex and turbulent image sequences and deformation of moving

objects over time. The uniqueness of the proposed algorithm is that it com-

bines the eﬀectiveness of the explicit and the speed of the implicit ﬁnite

diﬀerencing schemes. The performance of the model is compared to that of

i

other alternatives reported in the literature.

Finally, the robust Hessian based diﬀusion kernels developed earlier are

extended to the nonlinear model for fast, smooth and eﬃcient optical ﬂow

smoothing. Our techniques are used to reliably estimate the presence of vor-

tices from artiﬁcially created von Karman vortex sheet and real-time satellite

imagery. Experiments show that it is possible to extract features and model

the dynamics of vortices blindly, lacking the detailed knowledge of physical

properties of the event captured in each frame. The work provides the foun-

dation for the development of a robust real-time identiﬁcation and tracking

of storms.

ii

Contents

Declaration vii

Acknowledgements viii

List of Publications ix

List of Figures x

List of Tables xv

List of Abbreviations xvii

List of Symbols xix

1 Introduction 1

1.1 Optical Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.3 Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Literature Review 7

2.1 Optical Flow Estimation . . . . . . . . . . . . . . . . . . . . . 7

2.2 Diﬀusion Algorithms . . . . . . . . . . . . . . . . . . . . . . . 11

iii

2.3 3D Volumetric Interpolations . . . . . . . . . . . . . . . . . . 15

2.4 Navier-Stokes Equations and Vortex

Identiﬁcation . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3 Robust Hessian-based Anisotropic Diﬀusion 24

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.2 Optical Flow Estimation . . . . . . . . . . . . . . . . . . . . . 26

3.3 Anisotropic Diﬀusion of Optical Flow . . . . . . . . . . . . . . 29

3.3.1 Diﬀusion kernel . . . . . . . . . . . . . . . . . . . . . . 29

3.3.2 Embedded Hessian diﬀusion kernel . . . . . . . . . . . 31

3.3.3 Multiple 2D Hessian kernels . . . . . . . . . . . . . . . 34

3.3.4 3D Hessian kernel . . . . . . . . . . . . . . . . . . . . . 35

3.4 Robust Hessian Diﬀusion Kernels . . . . . . . . . . . . . . . . 36

3.4.1 Outlier robustness study . . . . . . . . . . . . . . . . . 36

3.4.2 Median of directional Hessians kernel . . . . . . . . . . 38

3.4.3 Alpha-trimmed mean kernel . . . . . . . . . . . . . . . 40

3.5 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 42

3.5.1 Smoothing noisy artiﬁcial vector ﬁelds . . . . . . . . . 43

3.5.2 Smoothing motion ﬁelds . . . . . . . . . . . . . . . . . 49

3.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

4 3D Volumetric Interpolation from Structural Flows 67

4.1 Research Objective . . . . . . . . . . . . . . . . . . . . . . . . 67

4.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4.3 Volumetric Image Interpolation . . . . . . . . . . . . . . . . . 69

iv

4.4 Structural Flow Initialisation . . . . . . . . . . . . . . . . . . 71

4.5 Smoothing Structural Flows . . . . . . . . . . . . . . . . . . . 77

4.6 Slice Interpolation . . . . . . . . . . . . . . . . . . . . . . . . 78

4.7 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 81

4.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

5 Robust Physics based Diﬀusion Solver 109

5.1 Research Objective . . . . . . . . . . . . . . . . . . . . . . . . 109

5.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

5.3 The Stable Fluid Model . . . . . . . . . . . . . . . . . . . . . 112

5.4 Robust Hybrid Fluid Model . . . . . . . . . . . . . . . . . . . 116

5.5 Vortex Core Detection . . . . . . . . . . . . . . . . . . . . . . 121

5.6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . 126

5.6.1 Synthetic data simulation set-up . . . . . . . . . . . . . 127

5.6.2 Results on synthetic data . . . . . . . . . . . . . . . . 131

5.6.3 Real image sequences and remarks . . . . . . . . . . . 142

5.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

6 Conclusion and Outlook 152

6.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . . 152

6.1.1 Robust nonlinear diﬀusion . . . . . . . . . . . . . . . . 152

6.1.2 3D volumetric slice interpolation . . . . . . . . . . . . 153

6.1.3 Diﬀusion based on ﬂuid dynamics . . . . . . . . . . . . 154

6.2 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

A Dataset 157

v

List of References 159

vi

Declaration

This thesis has not previously been accepted in substance for any degree

and is not being concurrently submitted in candidature for any degree other

than Doctor of Philosophy at the University of York. This thesis is the result

of my own investigations, except where otherwise stated. Other sources are

acknowledged by explicit references.

Some of the material in the following chapters has been previously pub-

lished by the author. A complete list of refereed publications can be found

on page ix.

I hereby give consent for my thesis, if accepted, to be made available for

photocopying and for inter-library loan, and for the title and summary to be

made available to outside organisations.

vii

Acknowledgements

First and foremost, I would like to thank my supervisor, Dr. Adrian G.

Bors, for his continued support, advice and suggestions during my research

and writing up. His breadth of knowledge and experience, coupled with his

dedication towards his students is invaluable both as a supervisor and as a

friend.

Secondly, appreciation goes to my assessors, Prof. Edwin Hancock and

Prof. Nishan Canagarajah for their constructive feedback on this thesis and

various aspects of my research, including reports and presentations.

My sincere thanks go to various members, past and present, of the com-

puter vision group for their invaluable technical discussions and brainstorm-

ing sessions.

Finally, to my family, for their encouragement, wholesome support and

love.

viii

List of Publications

Conference Papers:

• A. Doshi, A. G. Bors. Navier-Stokes formulation for mod-

elling turbulent optical ﬂow. In Proceedings of 18th British

Machine Vision Conference, Warwick, United Kingdom. 10-13

September 2007.

• A. Doshi, A. G. Bors. Robustiﬁed heat kernel smoothing

of structural ﬂows for volumetric image interpolation. In

Proceedings of IEEE International Conference on Image Process-

ing, Atlanta, USA. 8-11 October 2006.

• A. Doshi, A. G. Bors. Robust diﬀusion kernels for optical

ﬂow smoothing. In Proceedings of IEEE International Work-

shop on Machine Learning for Signal Processing, Maynooth, Ire-

land. 6-8 September 2006.

• A. Doshi, A. G. Bors. Structural ﬂow smoothing for shape

interpolation. In Proceedings of 18th International Conference

on Pattern Recognition, Hong Kong, China. 20-24 August 2006.

• A. Doshi, A. G. Bors. Optical ﬂow diﬀusion with robusti-

ﬁed kernels. In Proceedings of 11th International Conference

on Computer Analysis of Images and Patterns, Lecture Notes

in Computer Science 3691, pp. 222-230, Versailles, France. 5-8

September 2005.

ix

List of Figures

1.1 Top-level research model where I is the image frame and V

represents the motion ﬁeld. . . . . . . . . . . . . . . . . . . . . 4

3.1 Block matching algorithm. . . . . . . . . . . . . . . . . . . . . 27

3.2 Artiﬁcial vector ﬁeld. . . . . . . . . . . . . . . . . . . . . . . . 31

3.3 Smoothed artiﬁcial vector ﬁelds. . . . . . . . . . . . . . . . . . 33

3.4 The eﬀect of diﬀusion on outliers. . . . . . . . . . . . . . . . . 38

3.5 Calculation of the median of directional Hessian kernels. . . . 41

3.6 Synthetic vector ﬁelds, original and after being corrupted with

noise. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

3.7 Examples of Synthetic-1 vector ﬁelds after corruption with

Gaussian noise with σ

2

= 0.1 after ﬁve iterations of smoothing. 46

3.8 Artiﬁcial vector ﬁeld after corruption with Poisson noise with

σ

2

= 0.25 after ﬁve iterations of smoothing. . . . . . . . . . . . 49

3.9 Frames from three image sequences and their corresponding

optical ﬂows. . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3.10 Frames from additional three image sequences and their cor-

responding optical ﬂows. . . . . . . . . . . . . . . . . . . . . . 52

x

3.11 PSNR convergence for the reconstructed frame 8 from Con-

corde sequence. . . . . . . . . . . . . . . . . . . . . . . . . . . 54

3.12 Predicted frame PSNR evaluation, using ATM-2DH smooth-

ed vector ﬁelds when varying the alpha parameter for various

window sizes. . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

3.13 Smoothed optical ﬂows, initialised using BMA and the result-

ing predicted 5th frame of the Taxi sequence. . . . . . . . . . 58

3.14 Smoothed optical ﬂows, initialised using LK and the resulting

predicted 6th frame of the Concorde sequence. . . . . . . . . . 59

3.15 Results when applying optical ﬂow smoothing for the image

sequences considered in this study when the optical ﬂow has

been initialised using BMA. . . . . . . . . . . . . . . . . . . . 60

3.16 Results when applying optical ﬂow smoothing for the image

sequences considered in this study when the optical ﬂow has

been initialised using LK. . . . . . . . . . . . . . . . . . . . . 61

3.17 PSNR of the predicted frame when tracking scene change in

two image sequences for the best ﬁve diﬀusion methods when

the convergence criterion is set to τ < 10

−1

. . . . . . . . . . . 62

4.1 An illustration of intermediate slice reconstruction. . . . . . . 70

4.2 Block matching process in DBMA. . . . . . . . . . . . . . . . 72

4.3 DBMA structural ﬂows. . . . . . . . . . . . . . . . . . . . . . 73

4.4 Bad edge matching. . . . . . . . . . . . . . . . . . . . . . . . . 74

4.5 GrowingRegion(BlkOrig, BlkRef): Pseudocode of growing

algorithm used for regions of object boundaries, where nnz()

is a function that ﬁnds the number of nonzero pixels in a block. 75

xi

4.6 Visualisation of growing algorithm. . . . . . . . . . . . . . . . 76

4.7 The slice interpolation ﬂowchart. . . . . . . . . . . . . . . . . 79

4.8 Sample slices of the data sets. . . . . . . . . . . . . . . . . . . 82

4.9 Sample slices from the Incisor data set. . . . . . . . . . . . . . 84

4.10 Structural ﬂows and reconstructed slice of an Incisor. . . . . . 85

4.11 Further results for the Incisor. . . . . . . . . . . . . . . . . . . 86

4.12 3D Incisor reconstructions. . . . . . . . . . . . . . . . . . . . . 88

4.13 Accuracy of the middle slice reconstruction considering both

shape structure and grey-level. . . . . . . . . . . . . . . . . . . 90

4.14 Example of Knee slices for experimentations. . . . . . . . . . . 93

4.15 Structural ﬂows and reconstructed slice of a Knee MRI. . . . . 94

4.16 Further results on structural ﬂows and reconstructed slices of

a Knee MRI. . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

4.17 Original Humerus bone. . . . . . . . . . . . . . . . . . . . . . 97

4.18 3D Humerus bone reconstructed volumes. . . . . . . . . . . . 98

4.19 Original Iliac bone. . . . . . . . . . . . . . . . . . . . . . . . . 99

4.20 3D Iliac bone reconstructed volumes when skipping 5 consec-

utive slices between the remaining 2 slices. . . . . . . . . . . . 100

4.21 Results on structural ﬂows and reconstructed slices of a Female

Chest MRI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

4.22 Results on structural ﬂows and reconstructed Sheep’s Heart

MRI slices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

xii

4.23 Reconstruction of bone contours for Humerus and Iliac bones.

From top to bottom, the contours are from smoothed recon-

structions using original slices (blue), DBMA (red), Perona-

Malik (green), Black (magenta), 2DH (yellow), ATM-2DH

(cyan) and MED-2DH (black). . . . . . . . . . . . . . . . . . . 108

5.1 The stable ﬂuid solver algorithm. . . . . . . . . . . . . . . . . 114

5.2 Robust hybrid solver. . . . . . . . . . . . . . . . . . . . . . . . 116

5.3 Helmholtz-Hodge decomposition of a closed lid driven cavity

laminar ﬂow. . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

5.4 Evaluation of the Q

w

-criterion from a 5−by−5 window, where

the location of each vector in the window is shown with a

diﬀerent marker according to its corresponding Q

i

set. . . . . . 126

5.5 Driven-lid cavity ﬂow diagram. . . . . . . . . . . . . . . . . . 127

5.6 Synthetic closed driven-lid cavity ﬂows with noise. . . . . . . . 129

5.7 Half-cylinder model for von Karman ﬂow. . . . . . . . . . . . 130

5.8 Closed cavity vector ﬁeld smoothing comparisons. . . . . . . . 132

5.9 Representing von Karman ﬂows. . . . . . . . . . . . . . . . . . 134

5.10 Smoothing noisy von Karman ﬂows. . . . . . . . . . . . . . . 135

5.11 Evaluating Q and Q

w

on the von Karman ﬂow using equations

(5.14) and (5.18), respectively. . . . . . . . . . . . . . . . . . . 137

5.12 MSE after smoothing the noisy von Karman ﬂow. . . . . . . . 138

5.13 Performance of the vorticity segmentation using ∆

w

. . . . . . 139

5.14 Evaluating ∆ and ∆

w

on a smoothed von Karman ﬂow from

equations (5.17) and (5.19), respectively. . . . . . . . . . . . . 141

xiii

5.15 Smoothing optical ﬂows in image sequences displaying turbu-

lent motion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

5.16 Finding vortices in the Superstorm Andrea sequence. . . . . . 145

5.17 Additional results of vortex segmentation of Superstorm An-

drea sequence. . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

5.18 Smoothing optical ﬂow corresponding to cloud movement for

Superstorm Andrea. . . . . . . . . . . . . . . . . . . . . . . . 147

5.19 Finding vortices in the Solar Flare sequence. . . . . . . . . . . 149

A.1 Incisor dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . 158

xiv

List of Tables

3.1 Numerical results for Synthetic-1 data after one iteration of

diﬀusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

3.2 Numerical results for Synthetic-2 data after one iteration of

diﬀusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

3.3 PSNR (dB) of the reconstructed frame using the smoothed

optical ﬂow with the number of iterations (No) necessary to

reach convergence for each method and in the case of each

image sequence when the motion ﬁeld is initialised using BMA. 63

3.4 PSNR (dB) of the reconstructed frame using the smoothed

optical ﬂow with the number of iterations (No) necessary to

reach convergence for each method and in the case of each

image sequence when the motion ﬁeld is initialised using LK. . 64

4.1 Summary of the slice dimensions and voxel sizes for the dif-

ferent data sets. . . . . . . . . . . . . . . . . . . . . . . . . . . 83

4.2 Average percentage of reconstruction errors with DBMA as

the initialisation. . . . . . . . . . . . . . . . . . . . . . . . . . 104

4.3 Average peak signal-to-noise ratio of slice reconstructions with

DBMA as the initialisation. . . . . . . . . . . . . . . . . . . . 104

xv

4.4 Average PSNR of original middle slice reconstructions after

removing 5 intermediate slices with LK as the initialisation. . 105

4.5 Hausdorﬀ distance for Humerus bone. . . . . . . . . . . . . . . 105

4.6 Hausdorﬀ distance for Iliac bone. . . . . . . . . . . . . . . . . 106

5.1 Mean cosine error of smoothed vector ﬁelds. . . . . . . . . . . 132

5.2 Evaluation of smoothed noisy von Karman ﬂows when the

noise variance is 0.10. . . . . . . . . . . . . . . . . . . . . . . . 138

xvi

List of Abbreviations

1D One Dimensional

2D Two Dimensional

3D Three Dimensional

ATM-2DH Alpha-Trimmed Mean 2D Hessian kernel

ATM-3DH Alpha-Trimmed Mean 3D Hessian kernel

ATM-M2DH Alpha-Trimmed Mean Multiple ﬁeld

2D Hessian kernel

Black Black et. al.’s method

BMA Block Matching Algorithm

CFD Computational Fluid Dynamics

CT Computer Tomography

dB Decibel

DBMA Dual directional Block Matching Algorithm

L.H.S. Left Hand Side

LK Lucas-Kanade’s algorithm

M2DH Multiple ﬁeld 2D Hessian kernel

MAD Median of Absolute Deviation

MCE Mean Cosine Error

MED-2DH Median of 2D Hessian kernel

MED-3DH Median of 3D Hessian kernel

MED-M2DH Median of Multiple ﬁeld 2D Hessian

kernel

MedH-SFS Robust Hybrid Fluid Solver

xvii

MSE Mean Square Error

MRI Magnetic Resonance Image

OFE Optical Flow Equation

OFCE Optical Flow Constrained Equation

PDE Partial Diﬀerential Equation

PIV Particle Image Velocimetry

PM Perona-Malik method

PSNR Peak Signal-to-Noise Ratio

R.H.S Right Hand Side

SFS Stable Fluid Solver

SVD Singular Value Decomposition

xviii

List of Symbols

I Still video frame

V Vector ﬁeld

x, y Eulerian coordinates

t Time

V

x

, u x component of vector ﬁeld

V

y

, v y component of vector ﬁeld

H Hessian

i, j, k, l Array indices

M, N Length of array

∆ Diﬀerence

S

x

Search region in the x direction

S

y

Search region in the y direction

c Energy

λ Lagrangian multiplier

w Weights

η() Windowed neighbourhood centred around

ψ Scalar ﬁeld

f(),

ˆ

f() Functions

π Pi

α Alpha trimmed mean trimming ratio

θ Angle

σ

2

Variance

( Gaussian function

xix

(

P

Poisson function

e, exp Exponential function

L Total number of vectors

()

T

Vector or matrix transposed

τ Convergence threshold

/ Manifold space

(

K

Heat kernel

IR Real space

d Normalisation coeﬃcient

Σ Covariance matrix

L Laplacian matrix

ω Vorticity function

∇ Gradient operator

H Hausdorﬀ distance

∇ Divergence operator

P Pressure

ρ Density

ν Viscosity

∇

2

Laplace operator

f Force

n Surface normal

Ω Bound domain

T Projection operator

S Rate of strain tensor

Ω Vorticity tensor

xx

∆ Discriminant

1() Indexing function

Ψ Stream function

∇ Curl operator

xxi

Chapter 1

Introduction

In modern times, there has been a tremendous technological breakthrough in

the areas of digital computation and telecommunications. Of particular in-

terest has been the participation of the general public in these developments

as aﬀordable computers and the incredible explosion of the World Wide Web

has brought a ﬂood of instant information to an increasing percentage of

homes and businesses. Most of this information is designed for visual con-

sumption in the form of text, graphics and pictures or integrated multimedia

presentations. The rate at which information is transmitted, stored, pro-

cessed and displayed in a digital visual format is increasing rapidly and thus,

the design of engineering methods for eﬃciently transmitting, maintaining

and even improving the visual integrity of this information is of great interest.

1.1 Optical Flow

Analysing motion patterns is essential for understanding visual surround-

ings. Representative applications of motion analysis include video coding [1],

1

robotic vision, super-resolution reconstruction, enhancement, etc. Typically,

when estimating motion, it is assumed that all pixel intensities are locally

translated from one frame to the next and that the shifted values are pre-

served. This constraint implies that the intensity of a moving pixel in the

image plane remains constant along the trajectory of that pixel in time. This

assumption is the basis of the optical ﬂow constraint equation [2]. Conse-

quently, optical ﬂow equations are widely used to estimate motion between

consecutive frames.

From the point of view of image analysis, the optical ﬂow estimation al-

gorithms can be classiﬁed as gradient-based and feature-based methods. The

method employed can be either deterministic or stochastic, local or global.

The most widely used method for motion estimation is the block matching

algorithm (BMA) due to its simple implementation scheme. The BMA esti-

mates the vector ﬁeld based on the correlation between each pixel microblock

in one frame and the corresponding microblock within a macroblock (search

region) in the subsequent frame. However, unconstrained pixel intensity and

lack of local contrast variation can lead to erroneous optical ﬂow estimations.

In order to overcome such problems, regularisation terms are introduced,

hence the wide variations of BMA available within the research community.

The focus of this research is the use of anisotropic diﬀusion for accurate

estimation of the velocity ﬁelds initialised by BMA and Lucas-Kanade’s (LK)

gradient based algorithm [3]. Typically, most BMA’s use either windowed

median operators or other statistical cost measures to obtain smooth vectors.

This is also true for LK algorithms which use weighted windowed Gaussian

kernels. However, the performance of such methods when processing 2D

2

motion vectors depicting spatial-temporal object deformation is questionable,

hence the interest in nonlinear schemes such as partial diﬀerential equations

(PDE). Although a considerable amount of research has been conducted in

this area, most of it has been oriented towards understanding mathematical

properties of anisotropic diﬀusion [4] and modifying the diﬀusion equations

for speciﬁc applications [5, 6]. A lot of this research is primarily applied for

diﬀusing and segmenting colour images and image inpainting [6, 7, 8].

Robust statistical methods are adopted in computer vision to improve

the performance of feature extraction algorithms at the bottom level of the

vision hierarchy. These methods, to varying degrees, tolerate the presence of

data points that do not obey the assumed model. These points are typically

known as outliers. The robustness, in this context, can be attributed to the

breakdown point. This can be deﬁned as a point at which the smallest fraction

of outliers can cause the estimator to produce arbitrarily bad results. This

is usually regarded as the worst-case scenario in statistics. Some examples

of such methods are presented in [9, 10]. These type of kernels use rank

ordering statistics for outlier rejection. Notably, results can be skewed if

data is uneven.

Further details of the anisotropic diﬀusion algorithms developed and their

novelty is provided in Chapter 3. These algorithms, embedded with stochas-

tic processes form the core basis for the remainder of this thesis.

1.2 Motivation

Robust methods for motion estimation, motion compensation and regulari-

sation are very important in image processing. These are the pre-processing

3

techniques that make standard image processing methods work. To sum it

up, when dealing with image sequences, motion is estimated through the

computation of the image velocity vectors. Diﬀusion schemes (ﬁlter) are ap-

plied during vector ﬁeld computation to enhance the structure of moving

objects whilst smoothing out unwanted information (mostly due to noisy ar-

tifacts). When a clean optical ﬂow is obtained, this information is used in

image processing ﬁlters to compensate for motion for image enhancement,

segmentation, tracking and classiﬁcation. This process is highlighted in Fig-

ure 1.1.

Figure 1.1: Top-level research model where I is the image frame and V

represents the motion ﬁeld.

More recently though, signiﬁcant research has been done in developing

image pre-processing techniques to obtain better feature extraction results.

Hence, the main aim of this research is to develop image pre-processing tech-

4

niques, mainly anisotropic diﬀusion using nonlinear measures such as kernel

based Gaussian function and ﬁnite diﬀerencing methods from ﬂuid mechanics

for feature extraction of non-rigid objects which are deformable over time.

The other purpose of this research is to develop a generalised nonlinear

smoothing model which can either be used on its own or as a secondary ﬁlter

for optical ﬂow processing.

1.3 Thesis Overview

The structure of the thesis is organised as follows:

Chapter 2 surveys signiﬁcant contributions to the ﬁeld of optical ﬂow es-

timation and processing, isotropic and anisotropic diﬀusion algorithms, 3D

volumetric reconstructions (mainly from medical imaging) and ﬂuid mechan-

ics methods that have been implemented in computer vision and pattern

matching.

Chapter 3 of the thesis presents the novel contribution of nonlinear sm-

oothing kernel which is based on the local Hessian information of the vector

ﬁeld. Included in the chapter is the formulation of the diﬀerential smoother

with statistical robustness. Also this chapter provides are experimental re-

sults and analysis of techniques based on the local heat kernel and Hessian

matrix of the optical ﬂow. Further discussions on the comparative perfor-

mances of the proposed algorithms against known methods in the ﬁeld is also

covered.

In Chapter 4, the focus of research is on the utilisation of structural

ﬂows to reconstruct 3D volumetric models of human tissue structures. This

chapter will show how structural ﬂows are obtained and further smooth-

5

ed using anisotropic diﬀusion methods developed in Chapter 2 to obtain

intermediate slices and subsequently reconstruct 3D volumes. The proposed

methodology is shown to obtain positive results which are comparable with

other known implementations.

Chapter 5 presents another novel diﬀusion algorithm which is based on the

Navier-Stokes equation used in computational ﬂuid dynamics (CFD). The

proposed algorithm is a hybrid model which is dependent on both the explicit

and implicit diﬀerencing schemes. As in prior chapters, this chapter will also

describe models used to model ﬂow ﬁelds and provide experimental results

from both artiﬁcial and real-world sequence based vector ﬁelds. Additionally,

there is also a section on vortex dynamics which covers the identiﬁcation and

segmentation of vortex cores. The methods described here can be used for

vortex extraction from satellite imagery.

Finally, Chapter 6 summarises the contributions to the thesis, highlight-

ing strengths and weaknesses as well as suggestions on future research direc-

tions.

6

Chapter 2

Literature Review

The literature review survey conducted for the preparation of this thesis

covers numerous disciplines such as mathematics, physics, engineering, bio-

science, neuroimaging, medical imaging and computational ﬂuid dynamics.

This chapter is divided into four sections, each describing major contribu-

tions to the speciﬁc area of interest for the purpose of this work. Other

contributions, though not signiﬁcant for the current research, are neverthe-

less mentioned in the review.

2.1 Optical Flow Estimation

Optical ﬂow estimation from image sequences has been identiﬁed as an ill-

posed problem [1] which requires a regularisation methodology as shown since

early eighties by Horn and Schunck [11], Lucas and Kanade [3] as well as in

the comparative study from [12, 13]. In their paper, Horn and Schunck pro-

posed a method to calculate the moving ﬁeld (u and v components) at each

point using image intensity diﬀerences. Their work involved some assump-

7

tions such as the surface object is ﬂat to avoid shading variations, assume

initial pixel intensity to be uniform across the surface with no spatial dis-

continuities. Horn and Schunck’s algorithm was among the ﬁrst that gave

better smoothing of the velocity ﬁeld resulting in better segmentation of the

moving objects. They also introduced a way of implementing the derivatives

digitally. They proposed averaging four ﬁnite diﬀerences (based on 2-by-2

cube representation) [2].

Quite often the estimated optical ﬂow is noisy and contains outliers. This

is even more evident in the case of the block matching algorithm used in video

compression algorithms [1, 14]. However, a paper presented by Verri and Pog-

gio in 1986 [15] noted some disadvantages of Horn and Schunck’s proposal.

They argued that the latter’s algorithm does not take into account any rota-

tional movement and that the optical ﬂow and motion ﬁeld are generally not

identical concluding that two dimensional motion ﬁeld generally cannot rep-

resent three dimensional velocity ﬁeld unless special conditions are satisﬁed.

They suggest that feature-based matching algorithms are reliable enough to

accurately recover strong structures (segmentation) from motion, however in

order to determine the velocity ﬁeld, qualitative measurements are needed.

Firstly, they propose the use of scene radiance and image irradiance with the

aid of the Lambertian model to compute the diﬀerence between the velocity

ﬁeld and motion ﬁeld. Secondly, they introduce minimal optical ﬂow which

relates to perceived motion in the image.

There is no deﬁnite solution for occluded/unoccluded problem. Also,

there exist two unknowns (u and v velocity components) for each observa-

tion. And ﬁnally, he suggests that we can only determine the motion ﬂow

8

orthogonal to the spatial image gradient (normal ﬂow) at each pixel. Due

to this problem, there are certain assumptions that need to be made with

regard to the structure of the motion ﬁeld. These are:

• Some sort of smoothing constraints or uniformity has to be imple-

mented on the motion ﬁeld (non-parametric model).

• Six motion parameters to constrain local ﬂow vectors to lie on a speciﬁc

line (quasi-parametric model).

More recently though, Tekalp [1] identiﬁes numerous motion ﬁeld estima-

tors. Among them are methods based on the optical ﬂow equation (OFE),

pixel-recursive based methods (extended to Wiener ﬁlter type motion esti-

mation), Bayesian based methods, phase correlation based method and block

matching methods.

Bors [14, 16] used radial basis function (RBF) networks for motion esti-

mation and moving object segmentation [14]. His method implied the use of

median radial basis function (MBRF) network for segmentation. Simultane-

ously, subsequent image sequence frame is predicted using prior information

for tracking [14]. In [16], tracking was employed for moving object prediction

in video frame prediction. The process is repeated to obtain better motion

estimation and frame prediction, but the MBRF network has to be retrained

if an object has either entered or left a scene in the frame.

There are other methods that are being researched in the ﬁeld of motion

estimation. For example, P´ erez et. al. [17] proposed using the RANSAC

1

algorithm to estimate rigid motions through the image sequence. The RA-

NSAC algorithm is a robust ﬁtting model in the presence of data outliers.

1

(Random Sample Consensus) Algorithm

9

They then use singular value decomposition technique to estimate the scale

space representation of the local motion from the data. Other examples of

motion estimation techniques are fuzzy based as outlined by Erdem et. al. [18]

and Peacock et. al. [19], blind estimation using generalized cross validation

technique as proposed by Foroosh [20] and adaptive based methods using

recursive optical ﬂow estimation as proposed by Elad [21] or using tensor

based models as proposed by Liu et. al. [22], all of which are robust and

eﬃcient on simulated sequences yet untested on arbitrary video sequences.

Energy based minimisation frameworks have been deﬁned for estimating

the optical ﬂow by taking into account various determining factors [14, 23,

24, 25].

Recently, the research in optical ﬂow computation has focused on stochas-

tic approaches. Roth and Black [25] proposed using spatial statistics of vector

ﬁelds to obtain better accuracy in the ﬂow estimation. Their method relies

on learning statistical information from a synthetic training set of optical

ﬂows to obtain an appropriate statistical energy function. This function is

used within the Combined-Local-Global (CLG) [26] framework. Altough their

method performs reasonably well, it is limited (as suggested in [25]) due to

the fact the training set used is made up of static images, occlusion and

movement into/out-of scene is not considered.

Diﬀerent applications such as moving object segmentation [14, 23], track-

ing in image sequences and video coding quality [1] depend on smooth optical

ﬂows.

10

2.2 Diﬀusion Algorithms

Regularisation techniques using partial diﬀerential equations (PDE) have

found several applications in image processing and computer vision. The

theoretical background of these approaches arises from the methodology used

for describing physical phenomena such as the dissipation of heat or the

movement of ﬂuids [27, 28, 29]. The solution of the heat equation provides

a kernel function which is adaptive to the local manifold [30, 31]. If the

manifold is constant in all directions, then an isotropic kernel can be used

[27]. However, this is a particular case of data and most often the variation

in the local manifold should be taken into account by using an anisotropic

kernel. PDE’s used in the diﬀusion context have been applied for image

smoothing while preserving main features in images such as object contours

and corners [7, 32, 33, 34].

Anisotropic diﬀusion was employed for smoothing images while preserving

edges by Perona and Malik [4]. Their work was extended by Black et. al.

[7] by using a diﬀerent edge detection function in order to preserve sharper

boundaries in images. Hummel [35] oﬀers some very important insight on the

relationship between Gaussian blurring and the heat equation. Hummel is

able to demonstrate that there is a precise relationship between the standard

deviation of the Gaussian and time t of the heat diﬀusion equation. However,

a caveat of the heat equation is that true locations of the boundaries in very

noisy images makes the boundary preservation questionable since it may be

impossible to determine the existence of the boundary.

Weickert et. al. used the additive operator splitting (AOS) [34] for smoo-

thing volumetric images by extending the diﬀusion approach from [33], which

11

is similar to Perona and Malik [4], but using a Gaussian kernel instead of

gradient of image. The AOS scheme guarantees equal treatment of all co-

ordinate axes. It can be implemented easily in arbitrary dimensions, has good

rotational invariance and reveal a computational complexity and memory

requirement which is linear in the number of pixels. Weickert et. al. have

proved that AOS schemes are at least 10 times more eﬃcient than the widely

used explicit schemes.

The above work has been inﬂuenced by a number of related approaches.

Implicit splitting based approaches for linear diﬀusion ﬁltering have been

proposed in [36] and [37] and also in [38] and [39] where their realisation

for recursive ﬁlters is suggested. Impressive results on improved eﬃciency

by recursive ﬁltering can be found in [40, 41] and the close relation between

recursive ﬁlters and linear scale-space approaches has been clariﬁed in [42].

In the nonlinear diﬀusion ﬁeld, one can ﬁnd several approaches which

aim to be eﬃcient alternatives to the conventional two-level explicit ﬁnite-

diﬀerence scheme, for instance three-level methods, semi-implicit approaches

[33], multiplicative splittings, multigrid methods [43], ﬁnite element tech-

niques with adaptive mesh coarsening, numerical schemes with wavelets as

trial functions and pseudo spectral methods [44]. Hardware proposals for

nonlinear diﬀusion ﬁltering can be found in the literature [4].

Luck et. al. [45] combined the median ﬁlter and the diﬀusion operator of

Black et. al. [7] for smoothing medical confocal images. The adaptivity of

the Gaussian kernel to the local manifold is given by its covariance matrix

[46].

A suitable model of the local manifold geometry is indicated by the Hes-

12

sian matrix calculated from the local data [5, 6, 47]. The Hessian matrix,

represents the multidimensional second order derivates and can be used to

detect local maxima and minima as well as the transition areas from images.

When the Hessian is embedded in certain functions, it can be used to improve

the object representation as estimated from local surface normals [10].

On a more recent note, Tschumperl´ e and Deriche [6] proposed PDE based

vector diﬀusion method. In a nutshell, their method uses tensor and Hessian

matrices which are constructed from a still image or image sequence and

is then integrated within an anisotropic diﬀusion kernel of Gaussian nature

for smoothing optical ﬂow ﬁeld or image intensities. The advantage of this

method is that the Gaussian kernel used is multivariate and tends to work

better when there exists multiple sources of noise or presence of complex mo-

tion. On the other hand, Burgi [48] uses diﬀusion kernel proposed in [12] for

the smoothing of ﬂow obtained through intensity gradient direction method.

Petrovic et. al. [49] have introduced an unsupervised segmentation tech-

nique to study the multi-scale structure of images. They accomplish this by

generating a scale-space stack and building a hierarchical tree of the scales.

Linking is then performed between the scales and diﬀusion is applied to

smooth and enhance the edges. Their method has been applied mainly on

still greyscale and colour images. An interesting approach is implemented by

Preusser and Rumpf in [50]. Their method utilises two dimensional optical

ﬂow computation to represent three dimensional velocity vectors. The core

of their method is the anisotropic geometric diﬀusion scheme which is driven

by the structure of the image and its temporal acceleration.

In stark contrast to applying diﬀusion based smoothing, marginal me-

13

dian is a simple statistical ordering method [9, 10]. It relies heavily on the

distribution and amount of data to be considered in the kernel. The data is

ranked and median of the data is selected. This type of method can be useful

to obtain good optical ﬂows provided the structure of the moving objects are

strong and the neighborhood distribution has been selected appropriately.

However, the results can be skewed if the data is uneven.

There are numerous diﬀusion kernels being developed with various degree

of complexity. Most of which is beyond the scope of this thesis. To sum-

marise, many of the methods reviewed have direct relevance to this research.

Although some methods have been applied to image stills, it is possible to

extend them to image sequences albeit some degree of complexity may be

involved. The methods described above have been proposed to work indi-

vidually, but our opinion is that the demonstration of succesful working of a

combination of methods would be an interesting concept to consider. Most of

the diﬀusion kernel methods reviewed, integrate Gaussian (or its variation)

distribution within the kernel. The reasoning behind the distribution choice

is that on most occasion of motion blur, it is assumed that a Gaussian point

spread function is the cause of blurring and hence, the Gaussian diﬀusion

kernel is best to de-blur. By assuming the distribution is Gaussian, it also

simpliﬁes the complexity of the algorithms involved thus reducing the com-

putational cost. However, there are other types of diﬀusion kernels being

researched in the ﬁeld. Interesting ones include Weibull distribution based

kernel [51], principal component analysis (PCA) based kernel [52, 53] and

the Fisher criterion based kernel [54].

14

2.3 3D Volumetric Interpolations

Recent advances in medical imaging highlights an increasing need for 3D

models of human anatomy for medical diagnostics. Due to the nature of the

medical applications, it is extremely important that an accurate 3D model

of the anatomy is used. However, the task to reconstruct an accurate 3D

model is diﬃcult from a group of sparse image slices. Traditionally, medical

image slices are obtained through mechanical slicing and digitisation, thus

producing image slices which are not equidistant between slices. Modern

technology provides images of slices through magnetic resonance imaging

(MRI) or computer tomography (CT). Even with such tools at a researcher’s

disposal, it is still improbable for the machines to take suﬃcient digitised

images for full 3D reconstructions, hence the problem of data extraction

from sparse information.

There has been considerable amount of research devoted to reconstruct-

ing accurately the 3D models of a human body part. Most methods are

either a variation of morphological operations [55, 56] or interpolation meth-

ods [57, 58, 59, 60, 61]. For example, Bors [55] uses reciprocal binary

morphology to obtain intermediary slices to reconstruct a 3D tooth model.

Similarly, Lee [56] aligns the slices ﬁrst using object centralisation before

applying morphological operators. However, Lee’s [56] process only inter-

polate non-matching regions between slices. Schaller et. al. [60] proposed

using spiral interpolation based on azimuthal rebinning and Lee [61] pro-

poses shape based interpolation using warping. Comparisons between vari-

ous interpolation methods are presented in [58] and [59]. Other examples

of reconstruction methods in use include regularised maximum-likelihood

15

(ML) method [62], statistically based principal component analysis (PCA)

method [63], Bayesian framework based on alignment [64, 65], matching

based methods [57, 66, 67, 68, 69] and shape based methods [70, 59]. Other

than standard methods being researched, Turk and O’Brien [71] considered

using a variational approach for shape transformation and interpolation.

Their method is based on an implicit implementation and uses contours

of objects as boundaries, which enables surface warping from one object to

another.

Weng et. al. [72] used optical ﬂows from [2] to model the surface pro-

jection from one ultrasound image slice to another. Dougherty et. al. [68]

used optical ﬂows to model myocardiac displacements within the heart. This

would then allow them to tag and track tissue deformation over time. Hata

et. al. [73] used the gradient based optical ﬂow formulation to measure vol-

umetric brain deformations from MR images. They regularised the ﬂows

using a Gaussian kernel for better estimation. However, they state that it is

important to segment and then remove skin from the processing stage as it

can inﬂuence the accuracy of the ﬂow estimations. Abr` amoﬀ and Viergever

[74] proposed a variation of Lucas and Kanade’s [3] method to visualise the

movement of soft tissue in 3D volumetric Orbit images. These images tend

to resemble soft tissues like optic nerves within the human or animal brain.

In contrast to gradient based methods, Weruaga et. al. [75] proposed a para-

metric approach to estimate the volumetric motion of the human thorax.

Furthermore, Ray and Acton [76] proposed using the energy minimisation

method of motion vector gradient vector ﬂow to capture and track cells in

microscopic imaging. Also of interest is the myocardial motion ﬁeld captured

16

by applying a non-rigid registration technique [77] as presented by Rao et. al.

[78]. They also test their method on modelling brain deformations.

Variations of the block matching algorithm have been used for registra-

tion in medical images as described in [69, 79]. In [69], Penny et. al. realises

that for a registration based technique, it is important that initial neigh-

bouring slices contain similar anatomical features, otherwise the technique is

unreliable. Once the above criteria of slices having similar features are met,

they proceed to use B-spline method as in [77] to perform matching between

slices. The interpolated image is obtained by using the matching informa-

tion and linking it with the interpolation plane between the two slices. The

linking is done by linear interpolation between matching correspondes in the

slices and alignment is made appropriately.

Aside from the main focus of 3D volumetric interpolation schemes in

medical imaging, there are other kinds of research in this area that has some

relation to the work carried out in the thesis. One such method is the work

of Sharp and Hancock [80] which uses probabilistic relaxation model to cor-

relate the contour displacements of MRI surfaces with the corresponding

intensity features from successive frames to track its deformation. Rueckert

et. al. [77] used a nonrigid registration approach to model deformation in

breast magnetic resonance images. They apply aﬃne transformations with

spline based free form deformation methodology to achieve some degree of

success. Garza-Jinich et. al. [81] used an automatic segmentation algorithm

to segment regions of interest from MR image slices. Their proposal is robust

and the segmentation regions are then stacked to produce a 3D volume for

visualisation purposes.

17

Also of interest is the implementation of PDE based models in medical

imaging. Sarti et. al. [82] implemented an improved Perona-Malik [4] diﬀu-

sion algorithm with a multiscale method to give a better representation of

a 3D volumetric echocardiographic sequence. Chung and Sapiro [83] used

a PDE based morphological algorithm to successfully segment skin legions

from images. Krissian [84] used ﬂuid based ﬂux algorithm to anisotropically

diﬀuse CT images of the liver. By applying such an algorithm, he is able to

obtain a smooth image whereby the vascular structures of the liver could be

segmented for future analysis. Abd-Elmoniem et. al. [85] suggested using

a Hessian based diﬀusion kernel to smooth speckle images. Their algorithm

is anisotropic in design and is applied on ultrasound images of the heart.

Also anisotropic is the diﬀusion algorithm proposed by Ling and Bovik [86].

Their algorithm utilises the median of Peronal-Malik’s [4] implementation to

achieve very good smoothing of structures within molecular images, which

have low signal-to-noise ratio. Chung [87] suggested using the heat kernel

for smoothing in neuroimaging. He acknowledges that it is not appropriate

to use isotropic kernels to smooth soft tissue surfaces as it is in a curvlin-

ear space. Hence, the favoured approach of anisotropic smoothing using the

heat kernel, where smoothing only occurs uniformly within a common neigh-

bourhood. He uses this model to investigate cortical thickness of the brain

between autistic and normal children.

18

2.4 Navier-Stokes Equations and Vortex

Identiﬁcation

Very often, the modelling of natural phenomena involves the motion of dy-

namic ﬂuids, which diﬀers radically from that of rigid bodies. Classical opti-

cal ﬂow estimation algorithms would fail in such cases. The use of ﬂuid ﬂow

modelling for motion estimation can be traced back to the work of Fitzpatrick

[88], who compared optical and ﬂuid ﬂow methods. The computation of ﬂows

depends largely on the speciﬁc nature of the application. Using Fitzpatrick’s

analysis as a basis, Song and Leahy [89], employed the equation of conti-

nuity as an additional constraint to Horn and Schunck’s algorithm [11] in

order to obtain better motion estimation of the beating heart. Navier-Stokes

equations have been extensively studied in ﬂuid mechanics for modelling the

behaviour of ﬂuids under various conditions and constraints [90, 91].

The Navier-Stokes and optical ﬂow constraint equations have been em-

ployed for modelling von Karman ﬂows in [92]. Nakajima et. al. [92] min-

imised the overall cost function of the constraints to obtain a better estima-

tion of the ﬂow for dye shading movements. Although the method seems

to yield good results, it was only tested on simulated sequences and their

algorithms is similar to that of [89]. Bertalmio et. al. applied the Navier-

Stokes equations to image and video inpainting [93]. Their approach uses

the vorticity-stream formulation of the ﬂuid ﬂow equation, which can be at-

tributed to the image intensity-Laplacian relationship. Corpetti et. al. used

the vorticity-stream formulation to recover dense motion of water vapours

[94].

19

In recent years, extensive research has been conducted in optical ﬂow

construction using ﬂuid based algorithms. One particular work was done by

Kohlberger et. al. [95] which used variational domain decomposition method

to obtain a fast converging optical ﬂow construction method, especially for

large scale computations. As such, they also introduced a parallelisation

approach to achieve fast computations.

Navier-Stokes equations have been used in computer graphics for vi-

sualising ﬂames and building animation tools based on ﬂuid-like motion

[96, 97, 98, 99]. The stable ﬂuid solver (SFS) algorithm implements Navier-

Stokes equations and consists of a set of consecutive processing steps [97],

such as: advection, diﬀusion and mass conservation. The boundary condi-

tions are important in constraining the ﬂuid motion [91]. The boundaries

have been processed as a set of constraints on a grid [98], by enforcing repe-

tition and employing the Fast Fourier Transform (FFT) [97] or by using level

sets [96].

It is very well to use Navier-Stokes equations to model and smooth veloc-

ity ﬂows, how is it possible to verify the results obtained? One possible way

is to identify the presence of vortices or coherent structures [100] in a ﬂow

ﬁeld. Unless there is large perturbations in the ﬁeld, whereby a vortex could

easily be detected, the identiﬁcation of vortices in noisy ﬁelds with small per-

turbations is diﬃcult. Jeong and Hussain [100] suggest using regularisation

functions to obtain a uniform and smooth ﬁeld, with high frequency com-

ponents removed. This would fulﬁl the properties of Galilean-invariant ﬁeld

for better detection [100]. They have identiﬁed that the vorticity magnitude

measurement, [ω[ is not a reliable way to detect vortices. This is because

20

there are other physical states where the vorticity magnitude could not de-

tect the presence of a vortex. Please refer to [100] for a detailed discussion on

this. They also argue the reliability of Hunt et. al.’s Q-criterion measurement

[101]. Instead, they propose the λ

2

-deﬁnition as an additional constraint for

better vortex identiﬁcation.

Recently, Haller [102] proposed M

Z

and Q

S

criterions for better repre-

sentation of the vortex detection from multiple frames, in three and two

dimensions respectively, compared to [101] and [100] algorithms. Addition-

ally, Haller also presented a secondary criteria for hyperbolic (of saddle-type)

structures around the vortex centre.

In computer vision, Zhong et. al. [103] were among the ﬁrst to implement

ﬂuid mechanics constraints [100, 101] for vortex detection from 3D turbulent

motion ﬁelds. However, they only managed to test the algorithms on artiﬁ-

cially generated 2D and 3D PIV (particle image velocimetry) turbulent ﬁelds.

Another example is the work conducted by Ford and Strickland [104]. They

proposed using partition based method to segment regions of interest (con-

verging and diverging focal points) using polynomial based nonlinear least

squares estimation. This is then used to recover critical ﬂuid structures from

streamline imageries.

However, in computer graphics, a lot of work has been conducted for

vortex extraction, namely by Laramee et. al. [105] and Peikert et. al. [106,

107, 108, 109]. Laramee et. al. [105] uses topology based visualisation to

extract and track vortices. In contrast, Peikert et. al. [106, 107] suggest using

vector parallelism and scale space technique, with implicit ﬁnite diﬀerencing

scheme to identify and track vortices.

21

Other notable research in computer graphics and vision using ﬂuid meth-

ods for vortex extraction was conducted by Sahner et. al. [110] using the

M

Z

-criterion [102], Cuzol and Memin [111] using probabilistic energy min-

imisation method to extract vortex from phantom satellite imagery and Zhou

et. al. [112] used ﬂuid methods with local and global smoothness constraints

to obtain scene structures for segmentation and tracking of clouds for cloud

modelling and weather prediction. A good review of current methods being

used in the ﬁeld for vortex extraction is presented in [113].

2.5 Conclusions

In computer vision, it is increasingly diﬃcult to ﬁnd a niche in the ﬁeld

to specialise. This is especially true in the area of motion estimation, as

it is a huge ﬁeld in itself. The literature survey reported in this chapter,

namely in optical ﬂow estimations, diﬀusion, volumetric reconstructions and

ﬂuid methods in vision, only represents a small percentage of research in this

area.

However, there is little work in the area of non-rigid optical ﬂow esti-

mations and regularisation techniques. Hence, the motivation to focus the

thesis in nonlinear models for smoothing of optical ﬂows from complex and

turbulent image sequences, with scenes of non-rigid objects. Though there

are notable review publications on diﬀusion, they are mostly speciﬁc on still

images and not on velocity ﬁelds.

Although extending optical ﬂows to structural ﬂows pertaining to anato-

mical organ slices is not new, there is no review on the eﬀects of diﬀusion on

such ﬂows and how this would eﬀect 3D volumetric reconstructions.

22

Furthermore, we have identiﬁed an area of optical ﬂow estimation which

is still in its infancy, but slowly gathering pace in its level of interest by vi-

sion researchers. Using ﬂuid methods in computer graphics is quite common

nowadays. However, there is signiﬁcant lack of work in optical ﬂows using

Navier-Stokes equations. Though in the thesis we use Navier-Stokes equa-

tions for regularisation purposes, this is viewed as a stepping stone for an

optimised optical ﬂow estimation using Navier-Stokes equations.

It is the aim of the work described in this thesis to address some gaps in

the ﬁeld.

23

Chapter 3

Robust Hessian-based

Anisotropic Diﬀusion

3.1 Introduction

Optical ﬂow estimation from image sequences has been identiﬁed as an ill-

posed problem which requires a regularisation methodology as shown through

the works of Horn and Schunck [11] as well as Lucas and Kanade [3]. Quite

often the estimated optical ﬂow is noisy and contains outliers. This is more

evident in the case of the block matching algorithm used in video compres-

sion algorithms [1, 14]. Energy based minimisation frameworks have been

deﬁned for estimating the optical ﬂow by taking into account various deter-

mining factors [14, 23, 24, 25]. Diﬀerent applications such as moving object

segmentation [14, 23], tracking in image sequences and video coding quality

[1] depend on smooth optical ﬂows.

The aim of this chapter is to analyse the performance of a set of al-

gorithms that perform robust diﬀusion on vectorial ﬁelds. The proposed

24

algorithms combine the smoothing ability of the heat kernel with the outlier

rejection mechanisms of robust statistics algorithms. The diﬀusion kernel is

Gaussian with the covariance matrix considered as the Hessian calculated

from the data located in a certain neighbourhood. The kernel implicitly em-

beds the local changes in the optical ﬂow. The novelty in this section of

the research is the introduction of Hessian based diﬀusion kernels which dif-

fuses data anisotropically in the presence of outliers. Under these conditions,

classical diﬀusion introduces a bias in the whole data set. Robust statistics

operators are shown to improve the results provided by Hessian-based diﬀu-

sion by rejecting outliers and by enhancing the smoothing ability of diﬀusion

algorithms. Alpha-trimmed mean and median statistics have been consid-

ered for robustifying the diﬀusion kernels. The robust diﬀusion process is

extended to 3D lattices by using 3D Hessians. The proposed algorithms are

used for smoothing artiﬁcial vector ﬁelds as well as optical ﬂows estimated

from various image sequences. They combine anisotropic diﬀusion and ro-

bust statistics for smoothing the optical ﬂow obtained when using simple

motion estimation algorithms.

This chapter is organised as follows: Section 3.2 describes the estimation

algorithms of optical ﬂows that have been used in the thesis. Section 3.3

outlines the application of anisotropic diﬀusion on vector ﬁelds. Section 3.4

contains a study analysing the bias introduced when diﬀusing outliers and

describes the proposed set of algorithms that combine robust statistics with

Hessian based diﬀusion. Section 3.5 presents the comparative results for the

proposed methodology, while Section 3.6 outlines the conclusion of the study.

25

3.2 Optical Flow Estimation

The optical ﬂow extracted from a video sequence is represented as a vector

ﬁeld which warps one image into another providing its changes in time [3, 11,

12]. The methodology underlying the optical ﬂow estimation and smoothing

is described in the following subsection.

The optical ﬂow can be represented using a Taylor expansion of a frame

from an image sequence with respect to the other frames, considering the

ﬁrst derivative and neglecting higher order derivatives [3, 11]:

I(x + ∆x, t + ∆t) ≈ I(x, t) +∇I ∆x + I

t

∆t (3.1)

where I(x, t) represents an area located at x = (x, y) in frame t which is

part of the image sequence I, ∆x represents a translation in space and ∆t

represents the variation in time, while ∇I =

_

∂I

∂x

,

∂I

∂y

_

and I

t

represent ﬁrst

order partial spatial and temporal derivatives, respectively. These represen-

tation are chosen to be consistent with literature within the ﬁeld of motion

estimation. If the displacements are small enough, it can be assumed that

I(x + ∆x, t + ∆t) ≈ I(x, t) and after dividing equation (3.1) with ∆t yields

the constrained optical ﬂow equation:

∇I V

t

+ I

t

= 0 (3.2)

where V

t

= (V

x

, V

y

) denotes the motion vector at time t. Extending the cal-

culation for multiple frames, a vector ﬁeld on a 3D lattice is obtained, where

each plane of the lattice corresponds to the motion between two consecutive

frames. Motion vectors can be initially calculated by either using gradient

methods (3.2) [3, 11] or feature matching, for example the block matching

26

algorithm [1, 14] used in video coding. The block matching algorithm relies

on the correlation between blocks from one frame and blocks from a search

region from another frame of the same image sequence as in

V

t

= arg

k,l

min

_

M

i=1

N

j=1

[I(i, j, t) −I(i + k, j + l, t + 1)[

_

(3.3)

where MN is the block size and (k, l) are assumed to be inside a predeﬁned

search region (S

x

, S

y

). Consequently, for each block of pixels a motion vector

is determined representing the displacement between the coordinates of the

two blocks from the two frames as given by V

t

. The basic block matching

algorithm is shown in Figure 3.1.

Figure 3.1: Block matching algorithm.

However, motion estimation algorithms often lead to wrong decisions,

particularly in areas with constant texture and colour, where the gradient is

constant in all directions and the correlation is identical for several pixel block

27

combinations. Rotational and complex motion is only roughly approximated

by motion detection algorithms. Moreover, the image noise and changes

in illumination conditions produce motion vectors that are not consistent

with the optical ﬂow. Due to all these causes, the resulting optical ﬂow is

noisy in nature, displaying both slow varying noise as well as sudden large

changes. These errors in the optical ﬂow should be smoothed out or removed

altogether.

Other optical ﬂow estimators used for initialisation are known as gra-

dient based algorithms (3.2). Although equation (3.2) is constrained and

is now a well-posed problem, additional constraints are needed to improve

the eﬃciency of the solution. Horn and Schunck [2] used energy minimi-

sation technique, which minimises the combined global smoothness and the

constrained intensity gradient as shown below,

c =

__

λ

2

_

u

2

x

+ u

2

y

+ v

2

x

+ v

2

y

_

+ (I

x

u + I

y

v + I

t

) dxdy (3.4)

where u

x

, u

y

, v

x

, v

y

are ﬁrst order spatial velocity derivatives and I

x

, I

y

are

ﬁrst order spatial intensity derivatives with the Lagrangian multiplier, λ.

Another popular method of computation that is often used is Lucas

and Kanade’s algorithm [3], which implements regularisation in the pre-

computation stage of the optical ﬂow. Their algorithm assigns weights, w to

corresponding centered pixels which are locally inﬂuenced by the surround-

ing neighbourhood before using the gradient constraint equation (3.2). Their

implementation is shown to be

min

x∈η

w

2

(x) [∇I(x, t) u + I

t

(x, t)]

2

(3.5)

28

where x = (x, y) spatial locations and η is the local windowed neighbourhood.

However, as in most optical ﬂow estimators, indiscriminate smoothing

could produce undesired eﬀects in the resulting motion ﬁeld. The next section

explains the optical ﬂow smoothing methodology using anisotropic diﬀusion.

3.3 Anisotropic Diﬀusion of Optical Flow

3.3.1 Diﬀusion kernel

Anisotropic diﬀusion underpins the modelling of complex processes in physics

and chemistry. The heat equation of a geometric manifold can be described

as [28, 29, 31, 114]:

∂V

t

(x)

∂t

−∇

2

V

t

(x) = 0 (3.6)

where V

t

(x) is the heat vector located at x and at time t and ∇

2

V

t

(x)

represents the Laplacian of the vector ﬁeld which produces a tensor. This

equation assumes a given initial condition.

The general solution to the heat equation from (3.6) yields [31]:

ˆ

V

t

(x) =

_

M

(

K

(x −ς)V(ς)dς (3.7)

where ς is an arbitrary variable, (

K

(x −ς) is the heat kernel, representing a

Green function, which applies the diﬀusion onto the manifold /and x ∈ /.

The heat kernel is the natural candidate for measuring the similarity between

two points on the same manifold, while respecting the manifold geometry

[114]. In image processing, this operation was seen as being equivalent with

29

the local convolution of an image with a kernel function [30, 31]. In the case of

real data, the heat kernel from equation (3.7) is the Gaussian kernel [30, 114].

Isotropic smoothing using PDE’s in the context of images was ﬁrst for-

mulated by Koenderink [27]. In the case of images, the presence of edges

and details require an anisotropic smoothing approach. Perona and Malik

[4] proposed a smoothing function based on the Lorentzian error norm. Sim-

ilar to the statistical approach in [4], Black et. al. [7] used Tukey’s biweight

function to achieve similar results as obtained by Perona and Malik [4]. More

recently though, Tschumperl´ e and Deriche [6] used oriented Laplacians for

the same purpose. Diﬀerent to the mentioned methods, Weickert et. al. [34]

proposed an additive operator splitting (AOS) method which is based on the

CLMC method [33].

In contrast, a multivariate approach for nonlinear diﬀusion was considered

in this work. Assuming that the local manifold is warped to real space, then

it can be deduced that points on the manifold are appropriately correlated

to points in real space, i.e. / ≡ IR, then the heat kernel is the Gaussian

kernel [30, 114]. In this case, the solution to the heat equation (3.6) yields:

ˆ

V

t

(z

c

) =

1

√

4πd

_

x∈η

exp

_

−

1

4d

(x −z

c

)

T

Σ

−1

(x −z

c

)

_

V

t

(x)dx (3.8)

where Σ represents the covariance matrix, x is a location in a neighbourhood

η ∈ IR, z

c

is the central location of the windowed neighbourhood and d is a

normalisation coeﬃcient. The proposed method uses a multivariate Gaussian

kernel which is based on the local Hessian information. The local Hessian

represents the curvatures of the local manifold [6, 10] and it can be used as

a feature detector [47]. The embedding of the Hessian in the heat kernel is

30

discussed in the following subsection.

3.3.2 Embedded Hessian diﬀusion kernel

(a) Original vector ﬁeld (b) Degraded vector ﬁeld

Figure 3.2: Artiﬁcial vector ﬁeld.

Figure 3.2 shows the artiﬁcial vector ﬁeld that is used to evaluate which

of the covariance (Σ), Laplacian (L) and Hessian (H) matrices to be used

in the diﬀusion kernel function in (3.8). Figure 3.2(b) shows the degraded

ﬁeld with additive Gaussian noise of mean zero and variance σ

2

= 0.3. This

results in a more diﬃcult smoothing task for the diﬀusion kernels to recover

the original vector ﬁeld in Figure 3.2(a).

Let us consider the calculation of the following measures on a neighbour-

hood of the vector ﬁeld, with V

x

and V

y

being mean vectors of V

x

and V

y

,

respectively:

Covariance (Σ) =

N

i=1

(V

x,i

−V

x

)(V

y,i

−V

y

)

T

N

(3.9)

31

Laplacian (L

2D

) =

_

¸

¸

_

∂

2

V

x

∂x

2

∂

2

V

x

∂y

2

∂

2

V

y

∂x

2

∂

2

V

y

∂y

2

_

¸

¸

_

(3.10)

Hessian (H

2D

) =

_

¸

¸

_

∂

2

V

x

∂x

2

∂

2

V

x

∂x∂y

∂

2

V

y

∂y∂x

∂

2

V

y

∂y

2

_

¸

¸

_

(3.11)

The results in Figure 3.3 show the eﬀects of Hessian (a), Laplacian (b)

and covariance (c) kernel based smoothing on the degraded artiﬁcial vector

ﬁeld shown in Figure 3.2(b). Mean cosine error (MCE) of the smoothed

ﬁelds is calculated as in equation (3.26). This evaluates as to how best the

smoothed ﬁelds match the original vector ﬁeld.

The Hessian smoothed ﬁeld in Figure 3.3(a) yielded a mean cosine error

of 0.9620 after 2 iterations. The Laplacian smoothed ﬁeld in Figure 3.3(b)

yielded a mean cosine error of 0.9590 after 3 iterations and the covariance

smoothed ﬁeld in Figure 3.3(c) yielded a mean cosine error of 0.9566 after

5 iterations. The covariance matrix is a global measure of how diﬀerent the

V

x

and V

y

components of the vector ﬁeld diﬀer from each other. Hence,

its relative poor performance. There is only 0.3% diﬀerence between the

Hessian and Laplacian smoothed vector ﬁelds, with the Hessian being better.

As can be observed, the Laplacian tend to oversmooth the ﬁeld between

the horizontal and vertical vectors, which the Hessian does not. Hence the

Hessian matrix is chosen to be the most suitable to be integrated into the

diﬀusion kernel.

By embedding the local data in the Hessian difusion kernel, we can au-

tomatically direct the diﬀusion along the main data features. On the other

hand, in homogeneous regions there is no preferred direction of smoothing

32

(a) Hessian smoothed (b) Laplacian smoothed

(c) Covariance smoothed

Figure 3.3: Smoothed artiﬁcial vector ﬁelds.

and isotropic diﬀusion can be used.

The proposed method is based on the use of a diﬀusion kernel which

adapts itself to the local manifold variations. The local Hessian consists of an

appropriate measure of variation in the geometry of the local statistics [114]

and can be viewed as the application of a moving mask that models the

second derivative of the local manifold.

The most common way of calculating the local Hessian is by using the

local second order central diﬀerence approximation of the second derivatives.

The eigenvalues of the Hessian matrix as given in (3.11) can be used as

33

a detector of change in the direction of the optical ﬂow by indicating the

localisation and orientation of the moving object boundaries.

In the following analysis, the heat kernel is formulated as an adaptive

anisotropic ﬁlter that implicitly considers the local manifold variation by

means of its Hessian. The local Hessian (3.11) is embedded as the covariance

matrix in the heat kernel from (3.7). Considering that the Gaussian kernel

is a solution to the heat equation, the updated discretised and normalised

equation is:

ˆ

V

t+1

kc

=

x

i

∈η(zc)

V

t

ki

exp[−(x

i

−z

c

)

T

H

−1

2D,c

(x

i

−z

c

)]

x

i

∈η(zc)

exp[−(x

i

−z

c

)

T

H

−1

2D,c

(x

i

−z

c

)]

(3.12)

where V

t

ki

is the vector at location i within a neighbourhood η(z

c

), which

deﬁnes a symmetric region centered at the location z

c

, t denotes iteration

number and k is the frame index. The non-singularity of the local Hessian

can be enforced by using various procedures, for example by calculating its

pseudo-inverse.

3.3.3 Multiple 2D Hessian kernels

The 2D Hessian kernel from (3.12) is used to perform smoothing on motion

vector ﬁelds calculated between pairs of consecutive frames. This approach

is extended to consider multiple frames. In this instance, a 3D lattice of

parallel and equidistant vector ﬁelds modelling the optical ﬂow from the

entire image sequence is obtained. Firstly, local smoothing is applied in the

spatial neighbourhood as in (3.12). Thereafter, the smoothing is extended by

considering consecutive vector ﬁelds on both sides of the frame k containing

34

the central location, z

c

:

ˆ

V

t+1

kc

=

j

x

i

∈η(zc)

V

t

ji

exp [−(x

ji

−z

jc

)

T

H

−1

2D,jc

(x

ji

−z

jc

)]

j

x

i

∈η(zc)

exp [−(x

ji

−z

jc

)

T

H

−1

2D,jc

(x

ji

−z

jc

)]

(3.13)

where j = k −K, . . . , k +K and 2K represents the number of frames under

consideration for smoothing.

3.3.4 3D Hessian kernel

The 2D Hessian kernel is extended to 3D in order to accommodate the spatio-

temporal variation in the optical ﬂow. Here, the diﬀusion process is modelled

as:

ˆ

V

t+1

kc

=

x

i

∈η

3D

(zc)

V

t

ki

exp[−(x

i

−z

c

)

T

H

−1

3D,c

(x

i

−z

c

)]

x

i

∈η

3D

(zc)

exp[−(x

i

−z

c

)

T

H

−1

3D,c

(x

i

−z

c

)]

(3.14)

where the neighbourhood is deﬁned symmetrically in 3D as η

3D

(z

c

) and z

c

is the central location in the middle frame. By processing a larger amount

of data, the optical ﬂow transitions and moving object boundaries would be

better modelled by the 3D Hessian whilst diﬀusing the vector ﬁeld. The 3D

Hessian matrix is given by:

H

3D

=

_

¸

¸

¸

_

ψ

xx

ψ

xy

ψ

xk

ψ

yx

ψ

yy

ψ

yk

ψ

kx

ψ

ky

ψ

kk

_

¸

¸

¸

_

(3.15)

where the entries of the matrix are ψ

xx

=

∂

2

V

x

∂x

2

, ψ

yy

=

∂

2

V

y

∂y

2

, ψ

xy

=

∂

2

V

x

∂x∂y

,

ψ

yx

=

∂

2

V

y

∂y∂x

ψ

xk

=

∂

2

V

x

∂x∂k

, ψ

yk

=

∂

2

V

y

∂y∂k

, ψ

kx

=

∂

2

V

x

∂k∂x

, ψ

ky

=

∂

2

V

y

∂k∂y

and ψ

kk

is

35

the frame index diﬀerence, (V

x

, V

y

) is the given vector ﬁeld deﬁned on a 3D

lattice and k denotes the frame index.

3.4 Robust Hessian Diﬀusion Kernels

The directional anisotropic diﬀusion approaches introduced by Tschumperl´ e

and Deriche [5, 6] embed the local data geometry in the diﬀusion kernel

in order to avoid over smoothing of important data features. Section 3.3.2

discussed the expansion of this approach for use on vector ﬁelds. While

directional anisotropic diﬀusion is able to smooth according to the local data

ﬂow geometry, it is not able to identify the noise, particularly outliers. Hence,

another novel contribution of this work is the development of statistically

robust diﬀusion framework of noisy vector ﬁelds.

3.4.1 Outlier robustness study

Consider the following one-dimensional (1D) signal:

f(x) =

_

_

_

y , x ,= K

y + M , x = K

(3.16)

This signal has an outlier of height M at location x = K, and its dis-

cretised version is displayed in Figure 3.4(a) when M = 100 and K = 10.

This signal is used as an example to mimic the presence of noise with an

overwhelming value compared to its neighbourhood. It is subsequently used

to test the eﬀects of diﬀusion in the neighbourhood.

Consider the implementation of expression (3.12) in 1D, the Hessian be-

comes the second derivative and the resulting diﬀused value at location z

c

36

is:

ˆ

f(z

c

) =

N/2

j=−N/2

f(z

c

+ j) exp

_

−4j

2

f(z

c

+ j + 1) −2f(z

c

+ j) + f(z

c

+ j −1)

_

N/2

j=−N/2

exp

_

−4j

2

f(z

c

+ j + 1) −2f(z

c

+ j) + f(z

c

+ j −1)

_

(3.17)

where N is the size of the diﬀusion window and where the diﬀerentiating

operator is approximated by central diﬀerences. The diﬀusion is calculated

at the center of the window which is located at z

c

= K−i. It can be observed

that all the second derivatives in the signal from (3.16) are zero except those

at the locations j = ¦i − 1, i, i + 1¦. After replacing (3.16) into (3.17), the

following result is obtained:

ˆ

f(K −i) = y +

M exp

_

−i

2

2

M

_

exp

_

−(i −1)

2

4

M

_

+ exp

_

−i

2

2

M

_

+ exp

_

−(i + 1)

2

4

M

_

(3.18)

The bias at a site located at a distance of i positions from that of the

outlier in the signal from (3.16) is evaluated as [

ˆ

f(K−i) −y[. The fractional

part on the r.h.s. of equation (3.18) represents the bias resulted from diﬀusing

the outlier present in the signal from (3.16). The diﬀusion of the outlier will

inﬂuence several signal values in its neighbourhood. This situation could

happen when diﬀusion is performed on images where outliers are present near

edges. The Hessian detects the edge, but not the outliers. Consequently, the

outliers will be diﬀused in the 2D neighbourhood resulting in a bias as that

indicated in (3.18). Assuming the window size is N = 3, diﬀusion is applied

37

repeatedly on the signal displayed in Figure 3.4(a). The results produced by

diﬀusing the signal from (3.16) during successive iterations t = ¦1, 3, 6, 12¦

are shown in Figure 3.4(b). It can be observed that the outlier is diﬀused in

the surrounding signal. After a certain number of iterations, the values of

the diﬀused signal are stationary and they are always biased with respect to

the original signal which in this case corresponds to y = 10.

(a) Initial signal (b) Diﬀused signal

Figure 3.4: The eﬀect of diﬀusion on outliers.

Other diﬀusion functions such as those from [7, 34] are not able to distin-

guish between outliers and data features. However, the following subsections

introduces a new methodology that enhances the ability of anisotropic diﬀu-

sion with an outlier rejection mechanism.

3.4.2 Median of directional Hessians kernel

Robust statistics is known for its ability to preserve edges while eliminating

outliers and have been used for image ﬁltering [115]. Two robust statistics-

based ﬁlters, the median and the alpha-trimmed mean have been applied

together with radial basis functions for eliminating outliers from the opti-

38

cal ﬂow [14] and for segmenting volumetric images [116], respectively. The

proposal here is to combine the performance of anisotropic diﬀusion in de-

tecting the main data features with the ability of robust statistics algorithms

to eliminate outliers.

In the ﬁrst instance, median estimation is applied on the results provided

by directional diﬀusion. This robust diﬀusion approach consists of two stages.

We consider a N − by − N window, where N is an odd number. For each

location in this window, we deﬁne an additional window in order to evaluate

the Hessian based diﬀusion outputs as in (3.11) along a speciﬁc direction. All

the directions envisaged start from the central window location z

c

and are

spaced at intervals of π/4 from each other. Consequently, the calculation of

the diﬀused vectors is not done centrally as in (3.11), but in the ﬁrst instance

is evaluated at locations situated at the margins of their corresponding central

window. All such extended windows are used to calculate Hessians along

directions towards the center of the initial N −by −N window. The diﬀused

vector from the center of the initial N − by − N window is calculated as

well. The diﬀused vectors correspond to a speciﬁc directionally oriented

geometry of their neighbourhoods and this process considers a certain degree

of overlapping for the diﬀusion windows. The locations where the diﬀusion

is calculated are indicated by crosses in Figure 3.5 for N = 3. In this case,

the resulting neighbourhood is increased from 3−by −3 to 5−by −5 vectors.

This results a total of N

2

diﬀused vectors including the one in the center.

The resulting diﬀused vectors are ranked. Ranking in vectorial data can

be performed marginally, i.e. along each entry separately, or with respect to

the distance to a central data sample. In our approach, we use the marginal

39

median where all the corresponding entries are ranked separately and the

middle value is chosen for each entry [14]. The median operator is applied

onto the results produced by the directional diﬀusions resulted as described

above:

ˆ

V

kc

= Med (

ˆ

V

i

, η(z

c

)) (3.19)

where η(z

c

) represents the neighbourhood deﬁned by the window centered at

z

c

, containing vectors resulting from directional diﬀusions. This algorithm

takes into account extended neighbourhoods aiming to reduce overlaps among

local estimates. It can be applied to the 2D Hessian, multiple frame 2D

Hessian calculated in Section 3.3.3 or to the 3D Hessian, calculated according

to (3.14) and as described in Section 3.3.4. The inﬂuence of outliers will be

diﬀused during the ﬁrst operation and eliminated during the second step of

median ﬁltering. When applied to the 3D Hessian, the orientations for the

diﬀusion calculation are deﬁned in the 3D neighbourhood leading to greater

reliability in data statistics [117].

3.4.3 Alpha-trimmed mean kernel

Another robust statistics based approach is the inter-quartile averaging, also

known as the alpha-trimmed mean algorithm, which is suitable for smooth-

ing medium and long tailed data distributions [115, 116]. This method ranks

the given data and eliminates a certain percentage of data samples at the ex-

tremes of the ranked array. The aim of this method is to remove outliers and

to apply the diﬀusion algorithm only on data that are statistically consistent

with each other. After ranking the data, the extreme vectors are eliminated.

40

Diff Diff

Diff Diff

Diff

MED

Diff

Diff

Diff

Diff

Figure 3.5: Calculation of the median of directional Hessian kernels.

When ranking is performed according to the distance from a central vector,

the alpha-trimmed mean algorithm eliminates the data which are located far

away from that central data sample [116]. The updating equation is:

ˆ

V

t+1

kc

=

N−αN

i=αN

V

t

i

exp[−(x

i

−z

c

)

T

H

−1

c

(x

i

−z

c

)]

N−αN

i=αN

exp[−(x

i

−z

c

)

T

H

−1

c

(x

i

−z

c

)]

(3.20)

where α ∈ [0, . . . , 0.5] is the trimming ratio of a ranked array of N vectors

from the neighbourhood η

(

z

c

) and V

t

i

represents the i-th ordered vector at

location x

i

for iteration t. This algorithm reduces the computational cost of

the diﬀusion per iteration by considering diﬀusion on a reduced set of vectors.

When α = 0, the diﬀusion algorithm corresponds to the Hessian based diﬀu-

sion from (3.12), while when α = 0.5, only one vector is taken into account

corresponding to the median of the ranked array, although the calculation in

this case is diﬀerent from the one described in Section 3.4.2. The Hessian can

41

be H

2D

from (3.11), H

3D

from (3.15) or calculated by using H

2D

on multiple

frames and averaging the outputs as described in Section 3.3.3.

3.5 Experimental Results

A comparative experimental study has been performed using the proposed

robust diﬀusion methodology and various other diﬀusion algorithms for sm-

oothing artiﬁcial vector ﬁelds corrupted by noise as well as motion ﬁelds

extracted from real image sequences. The algorithms are denoted according

to the type of kernel that has been used for smoothing: 2DH - diﬀusion

algorithm using 2D Hessian (3.11), M2DH - multiple frame 2D Hessian (3.12),

3DH - 3D Hessian (3.14), MED-2DH - median of 2D Hessian (3.19), MED-

M2DH - median of multiple frame 2D Hessian, MED-3DH - median of 3D

Hessian, ATM-2DH - alpha trimmed mean using 2D Hessian (3.20), ATM-

M2DH - alpha trimmed mean of multiple 2D Hessian, ATM-3DH - alpha

trimmed mean of 3D Hessian. All the vector neighbourhoods are assumed as

η

2D

= 3 −by −3, while for multiple frames they become 3 −by −3 −by −4

and for 3D kernels η

3D

= 3 −by −3 −by −3.

The other algorithms considered for comparisons are Black - Black et. al.

[7] with parameter λ = 1/(4 ∗ t), PM - Perona-Malik [4] with parameters

κ = 30 and λ = 1/(4 ∗ t), TD - Tschumperl´ e and Deriche [6] with parameter

dt = 20 and AOS - Additive Operator Splitting scheme [34] with parameters

T = 10s, maximum iterations = 1, λ = 0.1, σ = 1 and diﬀusion timesteps =

1.

42

3.5.1 Smoothing noisy artiﬁcial vector ﬁelds

Two artiﬁcial vector ﬁelds have been considered. The ﬁrst vector ﬁeld,

Synthetic-1 is modelled by:

_

_

V

x

V

y

_

_

=

_

_

c −s

s c

_

_

_

_

D + S −R

R D −S

_

_

_

_

c s

−s c

_

_

_

_

x −µ

y −µ

_

_

(3.21)

where V

x

and V

y

are the velocity components in the x and y directions,

c = cos(θ), s = sin(θ), θ = 0, D = 0.8 is the dilation coeﬃcient, S = 0.05

is the shear coeﬃcient, R = 0.1 is the rotation coeﬃcient, and µ = 31 is the

center of the resultant ﬂow. This vector ﬁeld models a complex variation of

zooming in and out, and is displayed in Figure 3.6(a). The second vector

ﬁeld, Synthetic-2 is created by diﬀerentiating the following expression:

Z(x, y) = 3(1 −x)

2

e

−x

2

−(y+1)

2

−10

_

x

5

−x

3

−y

5

_

e

−x

2

−y

2

−

1

3

e

−(x+1)

2

−y

2

(3.22)

The velocity components are obtained as V =

_

∂Z

∂x

,

∂Z

∂y

_

. This vec-

tor ﬁeld has two attractors and two divergent centers and is displayed in

Figure 3.6(b).

These vector ﬁelds are corrupted by noise and independently generated

for each entry. In order to model a variety of possible corruption by noise,

both additive Gaussian and Poisson noise distributions were considered. The

Gaussian noise is commonly used as a noise model and is represented as:

((σ

2

) =

1

σ

√

2π

exp

_

−v

2

2σ

2

_

(3.23)

43

(a) Synthetic-1 (b) Synthetic-2

0 10 20 30 40 50 60

0

10

20

30

40

50

60

(c) Gaussian σ

2

= 0.10 (d) Gaussian σ

2

= 0.10

(e) Poisson σ

2

= 0.25 (f) Poisson σ

2

= 0.25

Figure 3.6: Synthetic vector ﬁelds, original and after being corrupted with

noise.

44

where v is the random variable associated with the additive noise and σ

2

is the variance of the Gaussian distribution. The Poisson distribution is

characterised by a long tail distribution and can be used to contaminate

data with outliers. It is modelled by:

(

P

(σ

2

) =

e

−(σ

2

)

(σ

2

)

v

v!

(3.24)

where v! is the factorial of v, the number of occurrences, assumed to be

an integer, and σ

2

> 0 is the variance of the Poisson distribution [118].

The width of the Poisson distribution is controlled by its variance σ

2

, which

is directly related to the percentage of outliers in data. The vector ﬁelds

modelled by equations (3.21) and (3.22) corrupted by Gaussian noise with

σ

2

= 0.10 are displayed in Figures 3.6(c) and 3.6(d), respectively. The

same vector ﬁelds corrupted by Poisson noise with σ

2

= 0.25 are displayed in

Figures 3.6(e) and 3.6(f), respectively. Five diﬀerent values for the variance

have been considered in each combination of noise distribution and data set.

The proposed algorithms as well as the diﬀusion algorithm of Black et. al.

[7] which has been adapted for the use on vectorial data, are applied for

smoothing the noisy vector ﬁelds with the aim of trying to reconstruct the

original vector ﬁelds. These experiments has only been tested on the follow-

ing algorithms: 2DH, ATM-2DH, MED-2DH and Black [7]. For the alpha-

trimmed mean smoothing algorithm in the case of the η

2D

, α = 0.33 and

hence 6 vectors are eliminated from the diﬀusion process.

The numerical results are assessed with respect to two error measures

representing the diﬀerence between the original and the smoothed vector

ﬁelds. The error measures consist of the mean square error (MSE) and the

45

(a) 2DH (b) Perona-Malik

(c) ATM-2DH (d) MED-2DH

Figure 3.7: Examples of Synthetic-1 vector ﬁelds after corruption with Gaus-

sian noise with σ

2

= 0.1 after ﬁve iterations of smoothing.

mean cosine error (MCE). The MSE is given by:

MSE =

L

i=1

(V

i

−

ˆ

V

i

)

T

(V

i

−

ˆ

V

i

)

L

(3.25)

where L is the total number of vectors in the given vector ﬁeld, V

i

is the

ground truth before considering the noise and smoothing, and

ˆ

V

i

is the

result obtained after smoothing the noisy vector ﬁeld at location i. The

MCE measures the angular error in the orientation of the vector as in [10]

46

Method

Noise

Black 2DH ATM-2DH MED-2DH

(σ

2

)

MSE MCE MSE MCE MSE MCE MSE MCE

G

a

u

s

s

i

a

n

0.01 0.007 0.991 0.016 0.982 0.017 0.973 0.007 0.995

0.10 0.063 0.933 0.097 0.931 0.154 0.860 0.059 0.973

0.25 0.180 0.871 0.237 0.887 0.444 0.740 0.126 0.952

0.30 0.229 0.854 0.276 0.873 0.528 0.721 0.158 0.947

0.40 0.286 0.818 0.327 0.852 0.680 0.690 0.186 0.933

P

o

i

s

s

o

n

0.01 0.015 0.993 0.009 0.998 0.001 0.998 0.002 0.999

0.05 0.309 0.963 0.182 0.978 0.005 0.995 0.094 0.987

0.10 1.052 0.934 0.728 0.959 0.036 0.985 0.501 0.975

0.25 7.594 0.797 7.091 0.830 1.627 0.867 6.454 0.851

0.40 20.873 0.649 20.767 0.668 10.629 0.712 19.470 0.692

Table 3.1: Numerical results for Synthetic-1 data after one iteration of dif-

fusion.

and is given by:

MCE =

L

i=1

V

i

ˆ

V

i

|V

i

| |

ˆ

V

i

| L

=

cos(θ

i

)

L

(3.26)

The normalised dot product between the two vectors provides the cosine

of the angle between them, denoted as θ

i

. The minimum error, as indicated

by MCE between the smoothed vector ﬁeld and the original one, should

be ideally close to 1. The numerical results obtained after smoothing the

synthetic vector ﬁelds corrupted by noise, after one iteration by the given

algorithms, are provided in Table 3.1 for Synthetic-1 data and in Table 3.2

47

Method

Noise

Black 2DH ATM-2DH MED-2DH

(σ

2

)

MSE MCE MSE MCE MSE MCE MSE MCE

G

a

u

s

s

i

a

n

0.01 0.011 0.722 0.019 0.682 0.023 0.626 0.012 0.776

0.10 0.061 0.532 0.088 0.531 0.239 0.428 0.043 0.612

0.25 0.156 0.435 0.195 0.473 0.473 0.350 0.108 0.554

0.30 0.213 0.469 0.253 0.493 0.482 0.362 0.146 0.572

0.40 0.425 0.414 0.522 0.443 0.658 0.320 0.342 0.546

P

o

i

s

s

o

n

0.01 0.021 0.961 0.027 0.956 0.007 0.974 0.008 0.985

0.05 0.291 0.805 0.338 0.800 0.013 0.969 0.113 0.889

0.10 1.364 0.633 1.482 0.633 0.070 0.923 0.832 0.703

0.25 8.008 0.376 8.106 0.372 2.427 0.623 6.883 0.359

0.40 19.591 0.240 20.073 0.237 10.871 0.359 17.994 0.219

Table 3.2: Numerical results for Synthetic-2 data after one iteration of dif-

fusion.

for Synthetic-2 data. The best results are highlighted in bold for each data

set, noise distribution and noise variance in these tables. MED-2DH provides

the best performances when smoothing out Gaussian noise, whilst ATM-2DH

is the best kernel for smoothing out Poisson noise. Figures 3.7(a)-3.7(d)

show the results after applying diﬀusion onto the vector ﬁeld Synthetic-1,

while Figures 3.8(a)-3.8(d) show the results after applying diﬀusion kernels

onto the vector ﬁeld Synthetic-2. From these results, it can be observed that

despite the high level of additive noise, MED-2DH provides smooth vector

ﬁelds that are similar with the original vector ﬁelds provided in Figures 3.6(a)

and 3.6(b), respectively. From Figure 3.8(c) it can be observed that after

48

(a) 2DH (b) Perona-Malik

(c) ATM-2DH (d) MED-2DH

Figure 3.8: Artiﬁcial vector ﬁeld after corruption with Poisson noise with

σ

2

= 0.25 after ﬁve iterations of smoothing.

smoothing using the ATM-2DH algorithm, there is clear identiﬁcation of the

vector ﬁeld structure from Synthetic-2 data whilst most of the noise was

correctly cleared out.

3.5.2 Smoothing motion ﬁelds

The proposed diﬀusion algorithms have also been evaluated on motion ﬁelds

extracted from real image sequences. The image sequences that have been

considered show a large variety of motion and a diversity of factors aﬀecting

49

motion detection. One frame from each of the sequences Taxi, Concorde and

Fighter are shown in Figures 3.9(a), 3.9(c) and 3.9(e), while frames from each

of the sequences Clouds, Tornado and Traﬃc are shown in Figures 3.10(a),

3.10(c) and 3.10(e). The optical ﬂow is generated using two diﬀerent ap-

proaches. The ﬁrst method uses the block matching algorithm (BMA) ac-

cording to (3.3), which is widely used in video compression algorithms [1].

The optical ﬂow estimated using BMA is shown in Figure 3.9(b) for the Taxi

frame, in Figure 3.10(b) for the Clouds frame and in Figure 3.10(f) for the

Traﬃc frame. The second method employs the Lucas-Kanade algorithm [3]

which relies on a gradient based approach for estimating the motion (3.7)

while embedding certain regularisation constraints. The results provided by

this algorithm are shown in Figure 3.9(d) for Concorde, in Figure 3.9(f) for

the Fighter and in Figure 3.10(d) for the Tornado sequence. In both motion

estimation methods and for all the image sequences, 4 −by −4 pixel blocks

have been considered. It can be observed that the optical ﬂow estimation

using block matching usually results in additional outliers when compared

to Lucas-Kanade algorithm.

A measure for assessing the ability of the smoothing algorithms is pro-

vided by the frame reconstruction accuracy when considering the smoothed

optical ﬂow. Such a measure is widely used for assessing the eﬃciency of

motion compression algorithms [1]. As it can be observed from equation

(3.1), we can predict an image frame by using the values of the current frame

I(x, t):

ˆ

I(x, t) = I(x + ∆t

ˆ

V, t + ∆t) (3.27)

50

(a) 5th frame from the Taxi sequence (b) Optical ﬂow produced by

BMA

(c) 6th frame from the Concorde sequence (d) Optical ﬂow produced by

Lucas-Kanade algorithm

(e) 173rd frame from the Fighter sequence (f) Optical ﬂow produced by

Lucas-Kanade algorithm

Figure 3.9: Frames from three image sequences and their corresponding op-

tical ﬂows.

51

(a) 474th frame from the Clouds sequence (b) Optical ﬂow produced by

BMA

(c) 341th frame from the Tornado sequence (d) Optical ﬂow produced by

Lucas-Kanade algorithm

(e) 59th frame from the Traﬃc sequence (f) Optical ﬂow produced by

BMA

Figure 3.10: Frames from additional three image sequences and their corre-

sponding optical ﬂows.

52

where

ˆ

V is the smoothed vector ﬁeld at location x and time t. Based on

equation (3.27) and by considering the image values in the ﬁrst frame as

known, we can predict all the other frames by appropriately varying ∆t.

However, the prediction of the scene based on the previous frame and the

motion ﬁeld, according to (3.27), does not take into account atmospheric

conditions, changes in illumination or the perspective distortions caused by

the 3D characteristics of the scene. The frame reconstruction accuracy us-

ing motion ﬁeld based prediction is given by the peak signal-to-noise ratio

(PSNR) which is calculated in decibels as:

PSNR = 20 log

10

_

_

_

_

_

_

255MN

¸

M

i

N

j

(

ˆ

I(x

ij

, t) −I(x

ij

, t))

2

_

_

_

_

_

_

(3.28)

where 255 is the maximum value in a greyscale image of size M − by − N

and we consider the prediction for a single frame (∆t = 1) without taking

into account the predicted frame diﬀerence that usually carries the additional

information needed for a perfect image reconstruction. A higher PSNR corre-

sponds to a better frame prediction, consequently indicating a better motion

ﬁeld smoothing algorithm. The frame prediction method ignores occlusion

and does not consider the eﬀect of multiple pixels converging to the same

location.

The experimental results are a snapshot of the conditions at convergence.

The condition of convergence for all the smoothing algorithms is given by:

1

L

L

i=1

(

ˆ

V

n+1

i

−

ˆ

V

n

)

T

(

ˆ

V

n+1

i

−

ˆ

V

n

) < τ (3.29)

53

Figure 3.11: PSNR convergence for the reconstructed frame 8 from Concorde

sequence.

where τ = 10

−2

is the convergence threshold, L is the total number of vectors

that are smoothed and n is the iteration number. The convergence results

provided by (3.29) and represented using the PSNR, as deﬁned in equation

(3.28), is shown for Concorde sequence in Figure 3.11, when the optical ﬂow

is initially estimated by LK algorithm. It can be observed that most of

the proposed robust diﬀusion algorithms smoothly converge in just a few

iterations.

In order to assess the eﬃciency of the α-trimmed mean anisotropic dif-

fusion method, we calculate the PSNR of the predicted frame using (3.28)

when smoothing the vector ﬁelds produced by either the block matching or

the Lucas-Kanade algorithms for the ATM-2DH algorithm for various win-

dow sizes and for a variety of trimming α values. The results produced by

54

these tests are shown in Figure 3.12. It can be observed that the best re-

sults are obtained when using a 3 − by − 3 window and α = 0.2, hence 4

vectors are eliminated from the diﬀusion calculation. When increasing the

neighbourhood size, the vector ﬁelds tend to be over smoothed, eventually

losing important information from the resulting optical ﬂow. Deﬁnitely, by

eliminating outliers, we improve the results produced by the diﬀusion pro-

cess, although when getting closer to the median (i.e. when α = 0.4), best

performance is not necessarily achieved. The results from Figure 3.12 are

consistent for all the image sequences under consideration.

Figure 3.12: Predicted frame PSNR evaluation, using ATM-2DH smoothed

vector ﬁelds when varying the alpha parameter for various window sizes.

The results provided by some of the proposed methods when applied on

55

the optical ﬂow extracted from the given set of image sequences are shown

in Figures 3.13, 3.14, 3.15 and 3.16. Figure 3.13 displays the results for

the Taxi sequence. The Taxi sequence displays the motion of several rigid

moving objects which is mainly translational. Figures 3.13(a), 3.13(c) and

3.13(e) show the optical ﬂow estimated between frames 3 and 5 and displayed

in Figure 3.9(b), after being smoothed using ATM-M2DH, MED-M2DH and

3DH kernels, respectively. Figures 3.13(b), 3.13(d) and 3.13(f) show the

resulting predicted frames corresponding to the smoothed motion ﬁelds. The

frame reconstruction uses the smoothed motion ﬁelds according to equation

(3.27).

Figure 3.14 shows the results obtained for the Concorde sequence. This

sequence was chosen for its complex motion characteristics such as the ro-

tational movement, turbulent air from jet thrusters, blocky artifacts from

compression and camera movement combined with a rigid moving object.

The initial optical ﬂow was extracted between frames 4 and 6 using the

Lucas-Kanade motion estimation algorithm which is shown in Figure 3.9(d).

Figures 3.14(a), 3.14(c) and 3.14(e) show the Concorde sequence optical ﬂow

from Figure 3.9(d) after being smoothed using 2DH, MED-2DH and MED-

3DH kernels, respectively. Figures 3.14(b), 3.14(d) and 3.14(f) show the

resulting predicted frames corresponding to the smoothed optical ﬂow.

The best results for the other image sequences in terms of the optical ﬂow

smoothness as well as the PSNR for the reconstructed frame are shown in

Figures 3.15 and 3.16. The optical ﬂow from Figure 3.9(f), initially estimated

using the Lucas-Kanade algorithm from the Fighter image sequence, when

smoothed by MED-M2DH, is shown in Figure 3.16(a). The frame recon-

56

structed based on this smoothed optical ﬂow is provided in Figure 3.16(b).

The optical ﬂow from Figure 3.10(b), initially estimated using the block

matching algorithm from the Clouds image sequence, when smoothed by

MED-3DH is shown in Figure 3.15(a). The frame reconstructed based on

this smoothed optical ﬂow is provided in Figure 3.15(b). The optical ﬂow

from Figure 3.10(d), initially estimated using the Lucas-Kanade algorithm

from the Tornado image sequence, when smoothed by MED-2DH is shown in

Figure 3.16(c). The frame reconstructed based on this smoothed optical ﬂow

is provided in Figure 3.16(d). The optical ﬂow from Figure 3.10(f), initially

estimated using the block matching algorithm from Traﬃc image sequence,

when smoothed by MED-2DH is shown in Figure 3.15(c). The frame recon-

structed based on this smoothed optical ﬂow is provided in Figure 3.15(d).

As it can be observed from these ﬁgures, the main motion structure is better

recovered after smoothing the initially noisy vector ﬁelds.

The PSNR results when tracking several frames from Taxi and Concorde

sequences are shown in Figures 3.17(a) and 3.17(b), respectively. These plots

prove that the observed results are consistent for all the frames from the

image sequences under analysis despite the various motion characteristics

in these image sequences. It should be noted here that for both graphs

in Figure 3.17, the diﬀusion ﬁlters that performed the best compared to

the other four kernels plotted are robust Hessian based kernels that uses

information from multiple ﬁelds. This shows that an object movement path

in predicted frames is traced more eﬀectively using prior information from

previous image frames/vector ﬁelds.

Tables 3.3 and 3.4 show the comparison between the diﬀerent diﬀusion

57

(a) ATM-M2DH smoothed ﬂow (b) Predicted frame 5 from (a)

(c) MED-M2DH smoothed ﬂow (d) Predicted frame 5 from (c)

(e) 3DH smoothed ﬂow (f) Predicted frame 5 from (e)

Figure 3.13: Smoothed optical ﬂows, initialised using BMA and the resulting

predicted 5th frame of the Taxi sequence.

58

(a) 2DH smoothed ﬂow (b) Predicted frame 6 from (a)

(c) MED-2DH smoothed ﬂow (d) Predicted frame 6 from (c)

(e) MED-3DH smoothed ﬂow (f) Predicted frame 6 from (e)

Figure 3.14: Smoothed optical ﬂows, initialised using LK and the resulting

predicted 6th frame of the Concorde sequence.

59

(a) Smoothed ﬂow from Clouds (b) Predicted 474th frame from

using MED-3DH kernel ﬂow smoothed by MED-3DH

(c) Smoothed ﬂow from Traﬃc (d) Predicted 59th frame from

using MED-2DH kernel ﬂow smoothed by MED-2DH

Figure 3.15: Results when applying optical ﬂow smoothing for the image

sequences considered in this study when the optical ﬂow has been initialised

using BMA.

algorithms, i.e. proposed set of algorithms is considered as well as other dif-

fusion algorithms such as Perona-Malik (PM) [4], Black [7], additive operator

splitting (AOS) [34], as well as Tschumperl´ e and Deriche (TD) [6]. The basis

for comparison is the PSNR of the reconstructed frames at convergence. The

results from Table 3.3 are obtained when smoothing the optical ﬂow gener-

ated by the block matching algorithm (BMA), while those from Table 3.4

60

(a) Smoothed ﬂow from Fighter (b) Predicted 173th frame from

using MED-M2DH kernel ﬂow smoothed by MED-M2DH

(c) Smoothed ﬂow from Tornado (d) Predicted 341th frame from

using MED-2DH kernel ﬂow smoothed by MED-2DH

Figure 3.16: Results when applying optical ﬂow smoothing for the image

sequences considered in this study when the optical ﬂow has been initialised

using LK.

are achieved when smoothing the motion ﬁeld which is initialised by Lucas

and Kanade (LK) approach [3]. In each of these tables, we also provide the

PSNR of the reconstruction error when no smoothing is performed, i.e. when

using the optical ﬂow calculated using BMA and LK respectively.

For each method and each image sequence, the number of iterations nec-

essary to achieve the convergence according to (3.29) is provided. As it can

61

(a) Taxi sequence

(b) Concorde sequence

Figure 3.17: PSNR of the predicted frame when tracking scene change in two

image sequences for the best ﬁve diﬀusion methods when the convergence

criterion is set to τ < 10

−1

.

62

Method Taxi Concorde Fighter Clouds Tornado Traﬃc

dB No dB No dB No dB No dB No dB No

BMA 17.72 15.78 15.52 15.40 21.39 13.21

PM 19.99 4 18.54 2 18.35 2 19.52 2 22.46 2 15.12 2

Black 20.05 4 18.54 2 18.35 2 19.52 2 22.46 2 15.13 2

2DH 21.55 8 18.98 4 18.58 3 19.33 3 21.70 2 14.53 2

M2DH 21.20 5 19.48 2 19.57 3 17.85 2 22.14 2 14.13 2

ATM-2DH 21.38 9 18.90 10 19.12 8 20.24 6 23.02 4 14.81 4

MED-2DH 22.17 6 19.59 3 20.18 3 21.21 2 23.45 2 16.93 2

ATM-M2DH24.42 1 17.52 1 16.10 1 16.49 1 23.15 1 13.64 1

MED-M2DH22.16 5 20.80 2 20.94 2 21.08 2 24.16 2 17.21 2

3DH 21.46 7 18.94 4 18.98 4 19.35 3 21.83 2 14.44 2

ATM-3DH 21.79 1 16.84 1 15.61 1 15.80 1 21.83 1 12.29 1

MED-3DH 22.25 6 19.59 3 20.18 3 21.66 2 23.34 2 16.83 2

AOS 20.11 5 18.34 1 22.26 4 18.97 1 22.93 2 15.96 2

TD 22.91 83 12.55 3 14.99 6 12.40 4 18.88 6 12.70 7

Table 3.3: PSNR (dB) of the reconstructed frame using the smoothed optical

ﬂow with the number of iterations (No) necessary to reach convergence for

each method and in the case of each image sequence when the motion ﬁeld

is initialised using BMA.

be observed, the number of iterations necessary to achieve the convergence

varies according to the method adopted and depends on the image sequence

as well. It is observed that the robust diﬀusion algorithms usually require

fewer iterations in order to achieve convergence when compared to the classi-

63

Method Taxi Concorde Fighter Clouds Tornado Traﬃc

dB No dB No dB No dB No dB No dB No

LK 20.46 14.67 16.78 17.11 21.59 15.41

PM 21.74 2 17.11 4 19.10 2 19.65 2 23.93 2 17.40 2

Black 21.74 2 17.21 4 19.10 2 19.65 2 23.93 2 17.40 2

2DH 20.50 1 19.27 10 19.34 4 19.25 2 23.11 2 15.82 1

M2DH 20.42 2 18.22 4 19.42 6 18.70 2 21.51 2 15.74 2

ATM-2DH 20.28 3 19.51 14 19.25 11 20.48 8 23.42 5 15.13 3

MED-2DH 22.09 2 20.30 8 20.78 4 22.12 3 25.13 2 18.67 2

ATM-M2DH 19.00 1 16.25 1 16.38 1 16.58 1 20.14 1 14.20 1

MED-M2DH22.19 2 19.22 4 21.36 6 20.43 2 22.48 2 17.80 2

3DH 20.50 1 19.14 10 19.31 4 19.24 2 23.25 2 15.84 1

ATM-3DH 19.08 1 16.07 1 16.18 1 16.08 1 20.06 1 13.27 1

MED-3DH 22.15 2 20.34 8 20.82 4 21.99 3 25.00 2 18.69 2

AOS 22.18 1 16.86 2 19.08 1 20.81 1 24.83 2 17.76 1

TD 19.88 8 17.92 30 17.85 12 16.02 6 20.64 10 14.77 100

Table 3.4: PSNR (dB) of the reconstructed frame using the smoothed optical

ﬂow with the number of iterations (No) necessary to reach convergence for

each method and in the case of each image sequence when the motion ﬁeld

is initialised using LK.

cal diﬀusion algorithms. The best PSNR of the frame reconstruction is high-

lighted for each image sequence and initialisation. The initial optical ﬂow

provided by the Lucas and Kanade (LK) algorithm provide better PSNR for

the frame reconstruction than the ones given by the block matching algo-

64

rithm (BMA). It is observed that MED-2DH, MED-M2DH and MED-3DH

provide the best results in terms of smoothing and reconstruction ability.

These results show clear improvement when combining diﬀusion and robust

statistics as well as when considering the motion from several consecutive

frames, according to the results from Tables 3.3 and 3.4. The smoothing re-

sults provided by the robust diﬀusion algorithms are better when the optical

ﬂow is complex, for example when the optical ﬂow represents the motion of

several objects which move in a variety of ways (including swirling motions)

or when representing ﬂuids in motion. Frame prediction can be further im-

proved by inpainting the uncovered areas in the predicted frames using the

neighbourhood information.

3.6 Conclusions

A set of robust diﬀusion algorithms have been applied for vector ﬁeld smoo-

thing. The proposed methodology is tested on optical ﬂow, when the initial

motion estimation is noisy due to various factors. The diﬀusion kernel is

multivariate Gaussian and embeds the local Hessian as its covariance ma-

trix. This type of kernel ensures that smoothing occurs along the motion

ﬁeld structure, thus preserving the moving objects borders and the main

ﬂow structure. The extension from 2D to 3D Hessian based Gaussian ker-

nels considers the temporal information from multiple frames. The analysis

has shown that following repetitive diﬀusion, outliers from data are spread

around introducing a bias in the resulting diﬀused signal. Robust statistics

algorithms such as the inter-quartile averaging and the marginal median are

employed together with the diﬀusion kernels for removing the outliers and for

65

enhancing vector smoothing. The algorithms are applied onto artiﬁcial vec-

tor ﬁelds and onto the motion ﬁelds extracted from various image sequences.

The initial motion ﬁelds are generated using the block matching algorithm

and the Lucas-Kanade algorithm. The proposed robust statistics based dif-

fusion provides clear improvements over classical diﬀusion algorithms. The

improvements are evident when dealing with complex motion ﬁelds which

are noisy. Motion ﬁeld smoothing can be used as a processing module for

various systems such as motion estimation and segmentation, tracking and

classiﬁcation of moving objects as well as in video compression systems that

rely on motion based frame prediction.

66

Chapter 4

3D Volumetric Interpolation

from Structural Flows

4.1 Research Objective

Chapter 3 described the use of robust Hessian based kernels to perform diﬀu-

sion on optical ﬂows from synthetic and real image sequences. It was thought

that the research needed to be more diverse and not restricted to regularising

motion from video sequences. Hence, the diﬀusion kernels have been tested

on structural ﬂows obtained from 3D volumetric images representing series

of sparse medical images of soft and hard tissue slices. As a result of the

work in improving the recovery of structural ﬂows, the research is expanded

to include synthesising intermediary slices and 3D volumetric visualisation

of the object in the slices.

67

4.2 Introduction

This chapter focusses on robust structural ﬂows as a methodology to model

the morphing between consecutive slices in sequences of cross-sectional im-

ages. The resulting smoothed structural ﬂows are applied to volumetric

image interpolation with the aim to reconstruct 3D shapes. The methods

considered for volumetric image interpolation can be categorised [58] into

shape-based [55, 59, 70] and grey-level based [69]. A method that considers

the grey-level information as a surface followed by interpolation and there-

after collapse it back to a slice was proposed by Grevera and Udupa [119].

In [55] the reciprocal morphing of one slice into the next one was performed

using two mathematical morphology operators.

Vector ﬁelds have been used to represent the optical ﬂow in image se-

quences [1, 14, 120]. In this case, the vector ﬁelds show the temporal warping

of one video frame into another. In the medical imaging context, optical ﬂow

has been used to model the movement of the heart [68, 121], registration of

cross-sections from the brain [122], deformations of skin tissue [123] as well

as changes in heart shape using diﬀuse tensor magnetic resonance images

(MRI) [79].

The proposed method models the variation in the internal structure rep-

resented in volumetric images. Inline with the block matching algorithm [1],

each image slice is split into blocks of pixels. The method uses correlation

to decide the correspondence between each block of pixels from one slice to

the next [14]. Variations of the block matching algorithm have been used

for registration in medical images as described in [69, 79]. In this thesis, the

proposed dual block matching algorithm (DBMA) models the structural ﬂow

68

through the volumetric image by considering matching in both directions of

the ordered slices. This algorithm provides two structural ﬂows located on a

regular 3D lattice. Each vector from the ﬁrst structural ﬂow represents the

correspondence of a set of voxels from one slice to the next one, while the

second vector ﬁeld models the reverse structural ﬂow. However, the lack of

contrast or missing data can lead to erroneous matches and unsmoothness in

the resulting dual structural ﬂows. In [14], a reliability coeﬃcient was used

to model the conﬁdence of the block matching estimates in video sequences.

In order to overcome the above mentioned problems, this study has de-

veloped a methodology that combines the advantages of two diﬀerent ap-

proaches: anisotropic diﬀusion and robust statistics. The Hessian represents

second derivatives and is known for its capability to model shape geome-

try in images [124]. In medical imaging, the local Hessian has been used

for feature localisation in [47] and for segmenting blood vessels [125]. The

anisotropic kernel functions developed in Section 3.3 and Section 3.4 are used

for automatically detecting signiﬁcant changes and smoothing accordingly

the structural ﬂow.

4.3 Volumetric Image Interpolation

The proposed methodology aims to obtain structural ﬂows from sparse med-

ical datasets. These ﬂows are then smoothed by using robust diﬀusion ﬁlters

embedding the Hessian of the local data. Once smoothed ﬂows are obtained,

they are used to reconstruct intermediary slices between the images of the

original sets. This is illustrated in Figure 4.1 on a set of images showing

cross-sections of a tooth. In the Figure 4.1, the initial slice is Slice 17 and

69

the reference slice is Slice 19. The aim is to reconstruct intermediate slices

between the two original images. The arrows depict the direction of the vol-

umetric image stacking. In between Slice 17 and Slice 19, the intermediate

structural ﬂow and its corresponding reconstructed image slice is shown. As

in [72], the slices (original and reconstructed) are stacked to reproduce a 3D

volume that appropriately represents the original 3D object. The following

sections describe the proposed framework.

Figure 4.1: An illustration of intermediate slice reconstruction.

70

4.4 Structural Flow Initialisation

In volumetric images, such as those provided by MRI, CT or by other means

[55], a set of image slices are provided, each representing a cross-section

through a volume. Vectorial ﬁelds have been used to represent the opti-

cal ﬂow in image sequences as well as to model deformations, particularly

in medical imaging [68, 79, 121, 122]. One well known method to initialise

the optical ﬂow is the block matching algorithm [14]. This algorithm ﬁnds

the best correlation between two blocks of pixels from a reference image

and an initial image, respectively, such that they have maximum correla-

tion. The equation implementing BMA is provided in equation (3.3). The

algorithm searches for the best match in a predeﬁned search region from the

reference image. Other registration methods use the gradient for ﬁnding cor-

respondences between regions of interest in medical images [69, 79]. In areas

that have similar texture or contain widespread repetition of similar features,

BMA produces erroneous decisions, thus resulting in noisy ﬂows [14, 120].

The proposed algorithm for producing structural ﬂows is called the dual

directional block matching algorithm (DBMA) and produces two vector ﬂows

that are oriented opposite to each other with respect to the ordering of slices

in the volume. It is assumed that any specular, additive or distributive noise

has been removed from the background for the ease of estimation. Each slice

is split into rectangular blocks and the best matching (highest correlation)

is sought for each block by comparing its grey-level values with those of

blocks within a search area from the reference slice, as shown in Figure 4.2.

In the matching process, only segmented foreground of the object (tooth)

is considered. Unlike BMA, DBMA estimates two structural ﬂows, forward

71

and backward respectively, each oriented in one or the other direction along

the main axis of the object structure. An example of dual structural ﬂows

is shown in Figure 4.3 for a set of cross-sectional image slices representing a

tooth.

Figure 4.2: Block matching process in DBMA.

In the example shown in Figure 4.3, structural ﬂows are computed be-

tween Slice 17 and Slice 19 of the Incisor tooth dataset [55]. For the ﬁrst

structural ﬂow (forward direction), matching from slice t to t + 1 results in

the vector ﬁeld V

1

, while for the second, the reverse matching is computed

from t +1 to t resulting in the vector ﬁeld V

2

[117]. For displaying purposes,

the total ﬂow, as shown in Figure 4.3(c) is obtained by adding forward and

reverse ﬂows, i.e. V

1

+ V

2

. This ﬂow represents the estimated deforma-

72

(a) Forward Flow, V

1

(b) Reverse Flow, V

2

(c) Combined forward and reversed ﬂows

Figure 4.3: DBMA structural ﬂows.

tion that took place between Slice 17 and Slice 19. Although edges of the

structure have been captured reasonably well, the internal structure remains

noisy.

Similar to the block matching algorithm used for estimating motion in

image sequences, the search area from the reference slice has the same centre

as the block region from the initial slice. However, due to the shape variation,

a large part of the search region may consist of the background, i.e. without

useful information for the structural ﬂows. Careful consideration of the block

size is needed, especially for locating edge correspondences. An example of

73

matching errors which occurs at the boundary of the object is shown in

Figure 4.4.

Figure 4.4: Bad edge matching.

Perfect matching against the background is lost or diminished by reduc-

ing the proportion of zeros in the pixel block containing the object boundary

considered. It is important, however, to retain the object shape information.

Therefore, it can be said that a trade-oﬀ needs to be achieved between back-

ground matching and retaining the object shape as illustrated in Figure 4.3.

Figure 4.5 presents the pseudocode of the growing algorithm. The pseu-

docode shown is a subset of the main structural ﬂow estimation program.

The input to the function GrowingRegion consists of the intensity blocks

from the original and reference slices. Line 1 of the pseudocode checks

whether the blocks are diﬀerent, i.e. there are no matching correspondences,

by checking the number of non-zeros present in the blocks. If this state-

ment is TRUE, then the variables are set in lines 3 to 6. HopX and HopY

represent the hopping distance between blocks. NewBlk is taken to be the

array of imin − NewHopX : imax + NewHopX in the x-direction and

74

1 if (nnz(BlkOrig) >= RegionSize) &&

(nnz(BlkRef) < RegionSize)

2 then

3 £ get imin,imax,jmin,jmax from main program

4 NewHopX ← HopX

5 NewHopY ← HopY

6 BoundaryBlk ← NewBlk

7 while (NoMatchingBlocks)

8 do

9 NewHopX ← NewHopX + HopX

10 NewHopY ← NewHopY + HopY

11 Length ← |NewHopX, NewHopY |

12 RegionSize ← (1.75 ∗ Length)/2

13 £ perform matching in the new search region

Figure 4.5: GrowingRegion(BlkOrig, BlkRef): Pseudocode of growing al-

gorithm used for regions of object boundaries, where nnz() is a function

that ﬁnds the number of nonzero pixels in a block.

jmin − NewHopY : jmax + NewHopX in the y-direction. The condition

NoMatchingBlock in the while loop checks the presence of shape or edge in-

formation of the object within BoundaryBlk in the corresponding reference

slice. If no such information is found, i.e. the condition is TRUE, then the

size of the block is expanded as shown in lines 9 and 10. Once the edge of

the shape is found within the reference slice, the shortest path between the

initial point and the object boundary is traced. This distance (Euclidean) is

75

(a) Initial search block (b) New search block

Figure 4.6: Visualisation of growing algorithm.

then used in line 12 to obtain a new search region. A value of 1.75, found

by experimentation, is used as a scaling constant. The new search region is

then fed back into the main program to re-estimate the structural ﬂow.

Another problem that may arise, as it can be observed in Figure 4.3, is

attributed to the boundary structure of the object. This is an interesting

problem since matching between boundary regions in the initial slice and

background at the same pixel location in the reference slice will lead to

wrong ﬂow estimation. Therefore, a growing algorithm is implemented within

DBMA, which is similar to mathematical morphology method employed in

[55]. The functioning of this algorithm is illustrated in Figure 4.6 on two slices

of the Incisor data set. With reference to Figure 4.6, a growing algorithm

is applied by progressively increasing the size of the search region as shown

in Figure 4.6(a), until a part of the object contained in the reference slice is

detected as shown in Figure 4.6(b). Then, the matching proceeds until the

best correlated block is found. For the second structural ﬂow, the algorithm

76

is applied in a reverse manner for the two slices, seeking the best correlations

for blocks from the reference slice from inside a search region deﬁned in the

initial slice. Due to the variations in the circumstances of choosing the search

region each time from a diﬀerent reference image, the two structural ﬂows are

not identical in the absolute value. While DBMA improves the modelling of

feature correspondences by creating a better representation of the variation

in the volumetric image, it also results in noisy vector ﬁelds in the same

way as BMA does. The following Section 4.5, addresses the smoothing of

structural ﬂows.

4.5 Smoothing Structural Flows

Why is smoothing necessary for structural ﬂows? As it can be observed in

Figure 4.3, the resultant ﬂows are quite noisy. If these ﬂows were to be

used for interpolating intermediate slices and hence reconstruct the 3D vol-

ume, it is more than likely that the results would not be very good. This

is because unsmoothed estimations can have detrimental inﬂuence on recon-

structed data. Hence, it is important to ﬁlter the estimations for obtaining

more accurate reconstructions. Basically, noisy (corrupted) ﬂows lead to bad

interpolations and smoothed ﬂows lead to more reliable outcomes.

The process of smoothing structural ﬂows takes place in a similar manner

to that of the optical ﬂows from image sequences. The smoothing ﬁlters used

are those developed in Section 3.3 and Section 3.4. The proposed Hessian-

based diﬀusion kernels are able to either reduce or remove noise in order to

obtain a better representation of the inner and outer volumetric structure.

The normalised updating kernel for smoothing structural ﬂows is given as

77

ˆ

V

t+1

kc

=

x

i

∈η(zc)

V

t

ki

exp[−(x

i

−z

c

)

T

H

−1

2D,c

(x

i

−z

c

)]

x

i

∈η(zc)

exp[−(x

i

−z

c

)

T

H

−1

2D,c

(x

i

−z

c

)]

(4.1)

where V

t

ki

is the vector at the location i within a neighbourhood η = 3−by−3,

t denotes the iteration number, k is the slice number, and z

c

represents the

central location of the neighbourhood. For comparative analysis, experi-

ments were carried out for smoothing using 2DH (3.12), ATM-2DH (3.20)

and MED-2DH (3.19) Hessian based kernels, which are provided in Chapter

3.

It is worth pointing out that smoothing is not conducted on the full

structural ﬂow for an intermediate slice as it would be expected. As shown

in Figure 4.7 and described in the next section, smoothing is performed on

the forward and reverse ﬂows separately. This ensures that the structural

ﬂow estimations of the object deformations are modelled reliably throughout

the whole reconstruction process.

4.6 Slice Interpolation

In many applications, particularly in the medical ﬁeld, it is essential to inter-

polate image slices in order to have an appropriate volumetric representation

[55, 58, 59, 69]. The proposed model uses smoothed structural ﬂows to in-

terpolate additional slices in between the cross-sectional slices of the existing

set. Volumetric image information is interpolated in both shape structure and

grey-level information along the structural ﬂows. The proposed intermediary

slice interpolation algorithm is presented in the chart shown in Figure 4.7.

With reference to Figure 4.7, structural ﬂows (Flow 1 and Flow 2) are

78

Figure 4.7: The slice interpolation ﬂowchart.

computed in the forward and reverse directions, respectively, as described

in Section 4.4. The main purpose of computing the structural ﬂows in this

way is to estimate the deformations from one slice to the next smoothly.

This is essential if the information of object structure from both slices are

to be used for intermediate slice reconstruction. The ﬂows model the shape

79

morphing as well as the intensity variation from one slice to another (Slice

1 and Slice 2) and consist of a 3D representation of the variation in the

volumetric shape [117]. Using this framework, up to several slices can be

generated along both, forward and backward ﬂows, between the two given

slices. Consequently, the interslice diﬀerence is reduced by

1

n −1

. As it

can be seen from Figure 4.7, the structural ﬂows interpolate the deformation

between the given pair of slices. In order to reconstruct intermediary slices,

a proportion of these ﬂows is considered for slice reconstruction along each

ﬂow. For example, in order to reconstruct the ﬁrst slice from a total number

of n = 20 slices between pairs, the corresponding forward ﬂow will be 1/20th

of Flow 1 and the reverse ﬂow will be 19/20th of Flow 2. This is appropriate

since the intermediary slice to be reconstructed is immediately adjacent to

the original Slice 1, representing the volumetric image information located

at quite a distance away from Slice 2. It should be noted that during the

intermediary slice reconstruction stage, image slices obtained by B-splines

[122] or other methods [55, 56, 60, 61, 64, 77, 119] can also be used.

During the smoothing stage, the ﬂows are independently smoothed us-

ing the diﬀusion kernels proposed and described in Section 3.3 and Section

3.4. The reconstruction of each part of the intermediary slice consists of the

contour, the texture and other features that follow the direction of the struc-

tural ﬂows in the volumetric image. Intermediary slice parts are produced

by displacing blocks of pixels along both structural ﬂows from initial slice

pairs. The resulting values from the two structural ﬂows are combined in the

last step in order to generate the interpolated slice. For areas that have ob-

ject grey-level assignments for both structural ﬂows, the resulting grey-level

80

is decided by proportionally averaging the two values. Therefore, the kth

intermediary slice between the image slicess I

1

and I

2

is generated by the

following equation :

I

1,k

(x) =

kI

1

(x +

ˆ

V

1

) + (n −k)I

2

(x +

ˆ

V

2

)

n

(4.2)

where x is the location of the interpolated result,

ˆ

V

1

and

ˆ

V

2

are the smoothed

vector ﬁelds modelling the warping along the forward and reverse ﬂows, and

I

2

corresponds conventionally to I

1,n

from a total of n slices. In the case

where there is only one data assignment from one of the structural ﬂows

while according to the reverse structural ﬂow there is no assignment, the

positive decision has priority. Results on the eﬀects of diﬀusion on structural

ﬂows and analysis are presented in the following section.

4.7 Experimental Results

The proposed methodology has been applied on real volumetric medical im-

ages. The given data sets are composed of sequences of digitised cross-

sections of medical data volumes representing both hard and soft tissue.

Three datasets are used representing an incisor

1

, a humerus bone

2

(part of

the upper arm) and an iliac bone

2

(part of the hip joint), respectively. An-

other three data sets, comprising of a sheep’s heart

3

, CT scan of a female

chest

3

and MRI of a knee

3

have been used. An example of original slices, one

1

The Incisor was used in [55]

2

The image datasets used in this experiment were from the Laboratory of Human Anatomy and Embryology,

University of Brussels (ULB), Belgium. http://isbweb.org/data/vsj/index.html

3

Obtained from The Volume Library at http://www9.informatik.uni-erlangen.de/External/vollib/

81

for each data set is shown in Figure 4.8. In Table 4.1, the technical details

of these image slices are provided.

(a) Incisor (b) Humerus bone (c) Iliac bone

(d) Sheep’s heart (e) Female chest (f) Knee

Figure 4.8: Sample slices of the data sets.

We have initialised the structural ﬂows using the DBMA (producing two

ﬂows) as explained in Section 4.4 and Lucas-Kanade algorithm [3] producing

a single ﬂow. The proposed vector smoothing methodology is applied on all

these structural ﬂows.

The ﬁrst dataset, which has also been used in [55], contains 22 slices

and represents an incisor. The slices have been obtained by mechanical

slicing followed by digitisation. After segmenting the tooth body from the

background as well as its root canal, the slices are aligned using a semi-

automatic procedure.

Three slices of the incisor sequence are shown in Figures 4.9(a), 4.9(b)

82

Data Set In-plane In-plane No. Slice

grid sizes dimensions slices spacing

(mm) (mm)

Incisor 500x500 12.40x12.40 22 2.00-2.20

Humerus 512x512 180.00x180.00 401 0.50-1.00

Iliac 512x512 250.00x250.00 260 0.50-1.00

Sheep’s Heart 352x352 1.00x1.00 256 1.00

Female Chest 384x384 1.00x1.00 240 1.00

Knee 512x512 0.25x0.25 87 1.50

Table 4.1: Summary of the slice dimensions and voxel sizes for the diﬀerent

data sets.

and 4.9(c). As it can be observed, these slices correspond to cross-sections

through the tooth in the region where the root canal emerges. The for-

ward structural ﬂow using DBMA, calculated between Slice 17 and Slice 19

is shown in Figure 4.10(a), while Slice 18 reconstructed using this struc-

tural ﬂow is displayed in Figure 4.10(b). The structural ﬂows resulted from

smoothing by 2DH kernel, Perona-Malik [4], ATM-2DH kernel, Black et. al.’s

algorithm [7] and MED-2DH are displayed in Figures 4.10(c), 4.10(e), 4.11(a),

4.11(c) and 4.11(e), respectively. Reconstructed slices corresponding to Slice

18 after using DBMA ﬂows smoothed by various algorithms are shown in

Figures 4.10(d), 4.10(f), 4.11(b), 4.11(d) and 4.11(f).

As it can be observed from these ﬁgures, all the diﬀusion based algorithms

improve the initial results provided by DBMA structural ﬂows. The direc-

tions of morphing one slice into another are captured well by the structural

ﬂows. The structural ﬂows smoothed by PM [4] and Black [7] algorithms

83

(a) Slice 17 (b) Slice 18

(c) Slice 19

Figure 4.9: Sample slices from the Incisor data set.

are noisier than the structural ﬂows smoothed by the proposed diﬀusion al-

gorithms. The robust diﬀusion algorithms provide smooth structural ﬂows

while eliminating the inﬂuence of outlying vectors. The root canal is al-

most completely closed in the reconstructions provided by 2DH, ATM-2DH

and MED-2DH smoothed structural ﬂows (as it should be according to the

ground truth slice from Figure 4.9(b)). The best visually assessed recon-

84

(a) DBMA ﬂow (b) DBMA reconstructed

(c) 2DH smoothed (d) 2DH reconstructed

(e) PM smoothed (f) PM reconstructed

Figure 4.10: Structural ﬂows and reconstructed slice of an Incisor.

85

(a) ATM-2DH smoothed (b) ATM-2DH reconstructed

(c) Black smoothed (d) Black reconstructed

(e) MED-2DH smoothed (f) MED-2DH reconstructed

Figure 4.11: Further results for the Incisor.

86

struction that also provides the most compact reconstruction of this slice is

that provided by the MED-2DH smoothed structural ﬂows.

As shown in Figures 4.10 and 4.11, the structural ﬂow is computed be-

tween Slice 17 and Slice 19 with the aim of reconstructing Slice 18. Though

it is assumed that the slices have equal distance between each other, in re-

ality this is not always the case since the results prove that the middle slice

reconstructions are not always identical to the original slices. It should be

noted that the main cause of this diﬀerence in the reconstructions is that

there is signiﬁcant shape variation from one slice to another from the data

set. If the data set is not sparse, then bad shape reconstructions would be

nulliﬁed. The focus however is to observe the eﬀects of smoothed structural

ﬂows on intermediary slice reconstructions.

Figure 4.12 shows the 3D volume visualisation of the entire set of 420

slices obtained through interpolation for structural ﬂows smoothed by all six

algorithms. This means that a total of 20 slices are interpolated between pairs

of original slices, including the originals. A low value indicates the volume is

more likely to appear squashed along the central axis, while a large value will

indicate the volume to appear elongated. It is necessary to obtain a balance

in order to obtain a suitable volume. The number of interpolated slices will

depend on the volume to be reconstructed. From the results obtained, MED-

2DH provides the smoothest surface for the 3D incisor reconstruction closely

followed by the 2DH kernel and DBMA (reconstruction based on unsmoothed

ﬂows), while PM and Black does not provide very smooth shapes.

Figure 4.13 provides numerical results for reconstructing the middle slice

for the whole Incisor data set when skipping one slice at a time and aiming

87

(a) DBMA (b) Perona-Malik (c) Black

(d) 2DH (e) ATM-2DH (f) MED-2DH

Figure 4.12: 3D Incisor reconstructions.

to reconstruct it from its two neighbouring slices. Figure 4.13(a) provides

the shape representation accuracy by evaluating the percentage of estimated

pixels that are not correctly placed in the estimated slice when compared to

the original from the initial data set. This percentage of error, ǫ is calculated

88

as

ǫ =

N

i

i

o

−

N

j

j

p

N

i

i

o

100 ∀I

o,i

, I

p,j

,= 0 (4.3)

where N is the size of the data, i

o

represent the spatial position in I

o,i

where

the object structure is represented and j

p

represent the spatial position in

I

p,j

where the estimated object structure is represented. In equation (4.3),

the percentage of pixel error is obtained by deducting the sum of correctly

placed nonzero pixels in the estimated slice from the total number of nonzero

pixels in the original slice over the sum of the nonzero pixels in the original

slice multiplied by 100. Lower percentage of error shows a higher degree of

accurate reconstruction. From the plot in Figure 4.13(a), it can be observed

that the robust Hessian based kernels perform better at correctly identifying

positions for new intensity pixels when compared to the other kernels. It can

be also observed that sudden changes in the graph generally represent major

diﬀerences between slices, i.e. large shape deformation or sharp changes in

grey-level intensity. The estimation of the structural ﬂows is based on the

assumption that the slices are equidistant, which is not always the case, par-

ticularly in this experiment where the incisor had been mechanically sliced.

BMA [1] is the classical block matching algorithm, using a single struc-

tural ﬂow for evaluating the displacement between two consecutive slices

instead of the dual structural ﬂow as in DBMA. Figure 4.13(b) evaluates the

grey-level reconstruction using various diﬀusion kernels by calculating the

peak signal-to-noise ratio (PSNR) between the estimated object/slice and

the original slice. It can be observed from Figure 4.13(b) that the recon-

89

(a) Shape reconstruction error rate

(b) Peak signal-to-noise ratio (PSNR)

Figure 4.13: Accuracy of the middle slice reconstruction considering both

shape structure and grey-level.

90

struction results for the Hessian based kernels are not very good when the

reconstructed slice has to recover signiﬁcant morphological change between

two consecutive slices.

However, the results are misleading. With BMA and DBMA, although

the shape deformation is not estimated accurately, the intensity value at cor-

rectly placed pixel matches the original, hence the high PSNR value. On

the other hand, the Hessian based kernels do the opposite. The shape defor-

mation is modelled accurately, but the grey-level reconstruction is somewhat

poor. The Perona-Malik and Black kernels are somewhere in the middle.

Furthermore, the estimation of the structural ﬂows are based on the assump-

tion that the slices are equidistant, which in reality is not, as can be observed

in Figure A.1. The higher the PSNR, the closer the slice is to the absolute

middle, while low PSNR would reﬂect the approximate distances of the real

slices. The PSNR between the original (ground truth) and the reconstructed

slice is calculated in decibels as:

PSNR(k+1) = 20 log

10

_

_

_

_

_

_

255M

¸

M

i=1

(

ˆ

I

k,k+2

(x

i

) −I

k+1

(x

i

))

2

_

_

_

_

_

_

(4.4)

where M is the number of pixels located in the foreground at x co-ordinates

in both the original k +1th frame, I

k+1

, and the predicted

ˆ

I

k,k+2

frame. The

predicted frame of

ˆ

I

k,k+2

is reconstructed from the structural ﬂows between

the k and k + 2 frames. BMA and DBMA provide quite good PSNR recon-

struction values but on the other hand they are not able to appropriately

model the variation in the shape according to the plot in Figure 4.13(a).

This is due to the fact that both BMA and DBMA rely on maximising the

91

correlation between blocks of pixels which results in a procedure ideal for

maximising the reconstruction PSNR. This is actually the main reason why

BMA is used in predictive based coding currently embedded in video coding

algorithms.

Diﬀusion based kernels, on the other hand, provide good shape recon-

structions while their grey-level restoration is not much worse than that pro-

vided by BMA and DBMA. The improvement provided by DBMA over BMA

in terms of reconstruction error is clear from the plot in Figure 4.13(a). ATM-

2DH gives the best shape reconstruction results closely followed by 2DH, PM

and MED-2DH algorithms for this 3D volume data.

Another data set that has been used for comparison purposes is the knee

MRI data set. Three slices of the knee sequence are shown in Figures 4.14(a),

4.14(b) and 4.14(c). In this case, these slices correspond to transversal-

sections through the knee and display both soft and hard human tissue.

The initialisation for this data set computed between slices 30 and 32

using the Lucas-Kanade [3] algorithm is shown in Figure 4.15(a), while slice

31 reconstructed using this structural ﬂow is displayed in Figure 4.15(b).

The structural ﬂow resulted from smoothing by 2DH kernel, Perona-Malik

[4], ATM-2DH kernel, Black’s algorithm [7] and MED-2DH are displayed in

Figures 4.15(c), 4.15(e), 4.16(a), 4.16(c) and 4.16(e), respectively. Recon-

structed slices corresponding to slice 31 after using LK ﬂows smoothed by

various algorithms are shown in Figures 4.15(d), 4.15(f), 4.16(b), 4.16(d) and

4.16(f).

92

(a) Slice 30 (b) Slice 31

(c) Slice 32

Figure 4.14: Example of Knee slices for experimentations.

As it can be observed from Figures 4.15 and 4.16, all the diﬀusion based

algorithms improve the slice reconstruction results provided by LK structural

ﬂows. The directions of morphing one slice into another are captured well

by the structural ﬂows. The structural ﬂows smoothed by PM [4] and Black

[7] algorithms are noisier than the structural ﬂows smoothed by the pro-

posed diﬀusion algorithms. The robust diﬀusion algorithms provide smooth

structural ﬂows while eliminating the inﬂuence of outlying vectors. The best

visually assessed reconstruction that provides the most compact reconstruc-

93

(a) LK ﬂow (b) LK reconstructed

(c) 2DH smoothed (d) 2DH reconstructed

(e) PM smoothed (f) PM reconstructed

Figure 4.15: Structural ﬂows and reconstructed slice of a Knee MRI.

94

(a) ATM-2DH smoothed (b) ATM-2DH reconstructed

(c) Black smoothed (d) Black reconstructed

(e) MED-2DH smoothed (f) MED-2DH reconstructed

Figure 4.16: Further results on structural ﬂows and reconstructed slices of a

Knee MRI.

95

tion of this slice is that provided by the MED-2DH smoothed structural

ﬂows. The observation shows that MED-2DH have signiﬁcantly reduced the

inﬂuence of outliers and modelled the deformation of the hard tissue of the

knee more accurately when compared to the other algorithms used in the

experiments. MED-2DH have also accurately modelled the deformation of

soft tissue in the muscular area of the knee whilst smoothing out unwanted

specular eﬀects of the image.

The small black gaps in the reconstructed slices observed in Figures 4.10,

4.11, 4.15 and 4.16 caused due to pixel block displacement can be covered

using interpolation [119], mathematical morphology operator [55], diﬀusion

algorithms for inpainting [8] or the diﬀusion methodology proposed in this

thesis, when applied on grey-level images.

Four other data sets representing a mixture of soft and hard human tis-

sues have also been used for experimental purposes. The original Humerus

and Iliac data sets are used as ground truth and are subsampled by 1:6 aiming

to replace the slices that are initially skipped for numerically assessing the

reconstruction methodology. The structural ﬂows for these slices of human

bones have been initialised using the DBMA algorithm. Figure 4.17 shows

the original 3D Humerus bone, while Figure 4.18 represent 3D volume recon-

structions using interpolated structural ﬂows smoothed by diﬀusion kernels.

Figure 4.19 shows the original Iliac bone and Figure 4.20 shows the pro-

cessed 3D volumes for the Iliac bone from the structural ﬂows smoothed by

various diﬀusion kernels.

The surfaces of the reconstructed objects are smoother for the 3D Incisor

and Humerus than those of the Iliac bone which has a more complex mor-

96

Figure 4.17: Original Humerus bone.

phology. The 3D Humerus volume reconstructed from the structural ﬂows

smoothed by MED-2DH is smoother everywhere as compared to the other

algorithms apart from the regions at the extreme end slices of the bone. At

the extreme end of the bone slices, PM [4] and Black [7] algorithms perform

better. The external surfaces of the 3D shapes can be further smoothed using

additional post-processing 3D shape smoothing algorithms.

The soft tissue data sets used in this study, female chest and sheep’s

heart MRI scans have been initialised using the LK algorithm [3]. Prior

segmentation have not been conducted on these data sets. As a consequence,

it is not possible to show a 3D rendered volume of the chest and heart. Figures

4.21 and 4.22 show some intermediate slice ﬂow and reconstruction results

for the chest and heart data sets, respectively. These results attempt to

reconstruct the middle slices, whereby ﬁve original intermediate slices have

97

(a) DBMA (b) PM (c) Black

(d) 2DH (e) ATM-2DH (f) MED-2DH

Figure 4.18: 3D Humerus bone reconstructed volumes.

98

Figure 4.19: Original Iliac bone.

been removed.

The results displayed in Figures 4.21 and 4.22 are intermediate slice recon-

structions from Black et. al. [7] and MED-2DH smoothed structural ﬂows.

As it can be observed, the MED-2DH smoothed structural ﬂows seemed to

be cleaner and less noisy when compared to the result produced by Black’s

algorithm. All the structural ﬂows computed using the LK algorithm have

been subsampled by a factor of 2 in x and y directions, for visual clarity

purposes. This is because the LK algorithm is a gradient based method and

produces dense ﬂow ﬁelds, which are diﬃcult to visualise, especially for large

slices.

The reconstruction errors are calculated by skipping one intermediary

frame for each pair of slices from the original Incisor data set and 5 slices

for the Humerus and Iliac data sets. These slices are reconstructed back as

described in this chapter using the DBMA method as the initialisation. The

diﬀerence, calculated as the percentage of wrong shape assignment between

99

(a) DBMA (b) Perona-Malik (c) Black

(d) 2DH (e) ATM-2DH (f) MED-2DH

Figure 4.20: 3D Iliac bone reconstructed volumes when skipping 5 consecu-

tive slices between the remaining 2 slices.

100

(a) Slice 54 to be reconstructed

(b) Black smoothed (c) Black reconstructed

(d) MED-2DH smoothed (e) MED-2DH reconstructed

Figure 4.21: Results on structural ﬂows and reconstructed slices of a Female

Chest MRI.

101

(a) Slice 177 to be reconstructed

(b) Black smoothed (c) Black reconstructed

(d) MED-2DH smoothed (e) MED-2DH reconstructed

Figure 4.22: Results on structural ﬂows and reconstructed Sheep’s Heart

MRI slices.

102

the original and the reconstructed data set are provided in Table 4.2. As

expected, MED-2DH is the best algorithm for volumetric reconstruction.

Also presented is the average PSNR of the intermediate slice reconstruc-

tions for the whole 3D volume of Incisor, Humerus and Iliac data sets in

Table 4.3. The numerical results in Table 4.3 have been obtained by tak-

ing the average PSNR of all slices needed to reconstruct a 3D volume of a

particular object. The average PSNR has been compared against diﬀerent

diﬀusion ﬁlters. For the Incisor, the Hessian based kernels performed bet-

ter because the dataset is sparse with large shape variations between slices.

Hence, the Hessian based kernels were able to model the shape variations

more accurately than the other methods, thereby interpolating the pixels

more accurately. However, the Humerus and Iliac datasets not being sparse,

suggested minimal variations between slices. Therefore, with shape interpo-

lation no longer an issue, DBMA (no smoothing) achieves a higher accuracy

of pixel interpolation compared to diﬀusion methods.

Table 4.4 presents the average PSNR after removing 5 slices from the

original set (for all 6 data sets), aiming to reconstruct the middle slice with

LK algorithm as the initialisation. From the result observed, it is thought

that gradient based ﬂow methods tend to be more eﬃcient than the block

based methods.

A third measure considered to validate the shape reconstructions is the

Hausdorﬀ distance measure. The Hausdorﬀ distance between two contours

f

i

and

ˆ

f

i

is deﬁned as in [126]:

H(f

i

,

ˆ

f

i

) = max¦h(f

i

,

ˆ

f

i

), h(

ˆ

f

i

, f

i

)¦ (4.5)

103

Object 3D shape reconstruction error (%)

DBMA PM Black 2DH ATM-2DH MED-2DH

Incisor 17.81 14.24 14.24 11.09 12.15 10.43

Humerus 12.12 6.59 6.60 6.97 7.52 5.85

Iliac 12.35 10.24 9.60 8.21 8.12 7.08

Table 4.2: Average percentage of reconstruction errors with DBMA as the

initialisation.

Object PSNR (dB)

DBMA PM Black 2DH ATM-2DH MED-2DH

Incisor 13.83 13.92 13.92 14.05 14.14 14.13

Humerus 13.87 13.21 13.21 13.25 13.34 13.24

Iliac 14.67 14.14 14.04 14.08 14.30 14.13

Table 4.3: Average peak signal-to-noise ratio of slice reconstructions with

DBMA as the initialisation.

where

h(f

i

,

ˆ

f

i

) = max

a∈f

i

min

b∈

ˆ

f

i

|a −b| (4.6)

where a and b are elements of the original contour f

i

and of reconstructed

contour

ˆ

f

i

, respectively, and | | denotes the Euclidean distance. The Haus-

dorﬀ distance calculated between the contours of the horizontal projections

of the 3D shapes, when skipping 1, 3, 5 and 9 slices is provided in Table 4.5

for the Humerus bone and Table 4.6 for the Iliac bone.

Based on the results from Table 4.2, MED-2DH smoothed structural ﬂows

provide the best reconstruction followed by ATM-2DH. However, from the

104

Object Peak Signal-to-Noise Ratio (dB)

LK PM Black 2DH ATM-2DH MED-2DH

Incisor 17.63 17.36 17.30 16.79 16.97 16.72

Humerus 34.92 34.52 34.52 34.79 34.50 34.62

Iliac 29.59 28.94 28.94 28.89 29.13 28.72

Knee 24.22 23.98 23.98 23.92 23.84 23.81

Chest 37.97 37.82 37.79 37.81 37.74 37.66

Heart 21.02 20.74 20.74 20.69 20.66 20.57

Table 4.4: Average PSNR of original middle slice reconstructions after re-

moving 5 intermediate slices with LK as the initialisation.

Slices Hausdorﬀ distance

skipped DBMA PM Black 2DH ATM-2DH MED-2DH

1 8.94 7 7 8.06 8 7.07

3 13.89 11.66 11.66 12.53 12.21 12.53

5 13.04 10.82 10.82 13.89 12.81 13.60

9 14.77 11.31 11.31 13.42 14.20 13.60

Table 4.5: Hausdorﬀ distance for Humerus bone.

Hausdorﬀ distance results in Table 4.5, PM and Black give better recon-

struction results compared to the Hessian based kernels. The main reason

for these results is the consistent error in the reconstruction of the end slices,

which is independent of the number of skipped slices. Relating to the Iliac

bone, it is diﬃcult to deduce a meaningful conclusion from the results in

Table 4.6. From Figure 4.19, the Iliac bone is an odd shape with lots of

indentations, a hole at the bottom and a large groove at the back which is

105

Slices Hausdorﬀ distance

skipped DBMA PM Black 2DH ATM-2DH MED-2DH

1 12.81 12.81 13.89 15.23 17.80 16.97

3 11.40 14.76 18.38 22.13 15.65 16.55

5 26.40 22.67 21.10 45.62 14.14 13.89

9 48.08 22.80 18.03 66.10 62.80 71.45

Table 4.6: Hausdorﬀ distance for Iliac bone.

expected due to its biological function. With every alternate and every third

slice being skipped, DBMA and PM (only for skip 1) are the nearest shape

match to the original contour, with the Hessian based kernels showing errors

in reconstruction around the hole. The Iliac bone reconstructed from MED-

2DH smoothed structural ﬂows is the closest match to the original contour

compared to other methods. This shows the strength of the algorithm to

reconstruct an accurate shape. For every 10th slice that is used, the results

did not favour the Hessian based kernels with Black’s method coming out

on top. The signiﬁcant diﬀerence in performance here is due to the Hessian

based kernels over compensating to reconstruct the concavity of the bone,

causing the shape error to be higher than it should really be. A probable

solution to this is to perform shape segmentation in the 3D space, which

should yield more realistic error measurements.

The 3D shape reconstruction rate in equation (4.3) is only applicable

for slices that have been segmented, i.e. noise in the background (beyond

the structure of the object in the slice) have been removed or signiﬁcantly

reduced. Since the three soft tissue data sets have not been pre-processed,

the only measurement that bears any signiﬁcance is the PSNR (4.4). The

106

comparative results for all six data sets with the structural ﬂows initialised

using the LK algorithm [3] is presented in Table 4.4. From the table, highest

PSNR is achieved by reconstructions from the initial ﬂow. This result is

expected since PSNR is intensity based measurement and the LK initial ﬂow

does not infer any intensity changes on the reconstructed slice. The results

reported here are similar to the results obtained for structural ﬂows initialised

by DBMA.

Figure 4.23 shows the stack of aligned contours for the original 3D bones

projected onto the horizontal plane together with those reconstructed using

the structural ﬂows smoothed by various diﬀusion algorithms. The contours

correspond to 3D volumes that have been reconstructed after eliminating 5 in-

termediary slices and thereafter interpolating them by using smoothed struc-

tural ﬂows. The positive eﬀect of robust diﬀusion structural ﬂow smoothing

is evident in some of these contours when compared to those obtained when

using the initial structural ﬂows as provided by DBMA. All these results

highlight the advantages of using the proposed 3D interpolation methodol-

ogy particularly when applying robust diﬀusion kernels to structural ﬂows.

4.8 Conclusion

In this chapter, a new methodology for 3D volumetric reconstruction from

sets of sparse cross-sections has been proposed. A bi-directional correlation

algorithm between pairs of image slices to construct structural ﬂows has been

used. These structural ﬂows show the correspondence in the internal struc-

ture and among features for a given 3D object. Besides the proposed DBMA

algorithm, structural ﬂows were also initialised using the LK algorithm. The

107

(a) Humerus Bones (b) Iliac Bones

Figure 4.23: Reconstruction of bone contours for Humerus and Iliac bones.

From top to bottom, the contours are from smoothed reconstructions using

original slices (blue), DBMA (red), Perona-Malik (green), Black (magenta),

2DH (yellow), ATM-2DH (cyan) and MED-2DH (black).

ﬂows are smoothed by various diﬀusion algorithms from Chapter 3. Interme-

diary slices are reconstructed using the smoothed structural ﬂows resulting

in a 3D volumetric object. Both shape structure and grey-level texture is

reconstructed according to the structural ﬂows and existing slices. The ex-

perimental results prove that the best outcome is achieved when using the

dual directional structural ﬂows smoothed by robust local Hessian based ker-

nels, provided the structural ﬂows have been initialised using feature based

method. A similar PSNR has been obtained for the interpolated slices for

all the diﬀusion methods in Chapter 3 when used on the same data set.

However, the surface of the resulting 3D shapes is not always smooth. This

problem can be resolved by using additional post-processing algorithms to

smooth the resulting 3D shape surfaces.

108

Chapter 5

Robust Physics based Diﬀusion

Solver

5.1 Research Objective

In ﬂuid dynamics, turbulence is characterised by chaotic and stochastic prop-

erty changes. This includes low momentum diﬀusion, high momentum con-

vection, and rapid variation of pressure and velocity in spatial and temporal

space. Typically, turbulence is represented by multitude of swirling motions

around either a single or multiple vortex cores. In vortex dynamics, it is

numerically assumed that at the centre of the vortex core, the pressure is

minimum in all directions [100]. Hence forth, subsequent references to com-

plex and turbulent motion in this thesis will refer to a combination of swirling

motions of the object in the scene and noise/camera movement introduced

by the video capture.

A robust Hessian based diﬀusion algorithm has been presented in Chap-

ter 3 for smoothing optical ﬂows from video sequences. Additionally, it has

109

been shown in Chapter 4 that robust and eﬃcient smoothing of structural

ﬂows can inﬂuence slice synthesisation and subsequent 3D volumetric recon-

structions. In this chapter, an improved diﬀusion methodology that is based

on the Navier-Stokes equation is developed. The proposed method is an

improvement on the Stable Fluid Solver proposed by Stam [97, 98, 127] by

robustifying the diﬀusion stage. The robust Hessian based diﬀusion devel-

oped in Chapter 3 is embedded in the proposed method. A methodology for

detecting speciﬁc movement patterns such as vortices from smoothed vector

ﬁelds is developed in this chapter as well.

5.2 Introduction

It can be safely said that modelling the optical ﬂow from motion in ﬂuid im-

age sequences can be an onerous task. The techniques that have been around

for a while, e.g. standard optical ﬂow estimation methods work well when

modelling the movement of rigid objects. However, in the case of deformable

objects and ﬂuids, existing methods perform ineﬃciently. The Navier-Stokes

equation has been frequently used to model the dynamics of ﬂuid ﬂow in

computational ﬂuid dynamics. It was used to model heart movement in

echocardiography [89], for simulating the behaviour of natural phenomena

such as air, ﬂuids, clouds, etc, in computer graphics [97, 110, 112, 128], mod-

elling the dynamics of turbulence [129], blood ﬂow analysis in angiography

[130], modelling behaviour of laminar ﬂow around a cylinder [131], modelling

the radiation of stars in dust clouds in astrophysics [132], etc.

The Navier-Stokes equations in itself are complex and are traditionally

used to model the movement of heat and ﬂuid in complex structures. Such

110

natural phenomena are more likely to exhibit nonlinear, sometimes chaotic

like behaviour. Due to the presence of nonlinearities, we are unable to model

such behaviour easily. In order to make accurate assessments on the be-

haviour of such phenomena, for example heat and ﬂuid particles, ﬁne grid

structures are a necessity to model the conﬁnement structure. Additional

constraints such as the initial and boundary conditions, explicit or implicit

ﬁnite diﬀerencing schemes are important since they increase the stability and

reliability of the computations. The reason why Navier-Stokes equations is

so important is that we are increasingly aﬀected by it in our daily life. It was

used to model injection and combustion engines, its been used to forecast cli-

mate change, model the traﬃc and crowd ﬂow, it has been used to model the

aerodynamics of wings and rotor blades, model the temperature distribution

of silicon chips and so on. However, the implementation of the Navier-Stokes

equations is application dependent. Despite the complexity involved, new

and innovative methods are continuosly being developed in order to make it

computationally more accessible and generalised.

In this chapter, an innovative smoothing method is proposed using the

Navier-Stokes equations. Together with supplemental equations, such as the

conservation of mass and well formulated boundary conditions, the proposed

methodology is used to smooth estimated optical ﬂows of complex and turbu-

lent ﬂuid motion from image sequences. The proposed method is embedded

with a robust diﬀusion algorithm, as described in Chapter 3 for better eﬃ-

ciency. Distinct features such as vortices (focus of rotation) are estimated

from turbulent optical ﬂows. The proposed technique is applied on artiﬁcial

data and image sequences representing atmospheric and solar phenomena.

111

The remainder of the chapter is presented as follows. Section 5.3 de-

scribes the stable ﬂuid model [97] and how it is being implemented in com-

puter graphics and visualisation. In Section 5.4, the methodology used to

implement the proposed robust hybrid ﬂuid solver is presented. In Section

5.5, we present methods to detect the presence of vortices in turbulent ﬁelds.

Experimental results and the analysis on simulated and real-world data is

presented in Section 5.6 while Section 5.7 concludes the chapter.

5.3 The Stable Fluid Model

Navier-Stokes methodology represents the basis for modelling a large variety

of phenomena such as those characterising weather, ocean currents, water

ﬂow in a pipe, the air ﬂow around an aircraft wing, the motion of stars

inside a galaxy, blood ﬂow, ﬁnancial derivatives, etc [91]. In engineering,

they are also used in the analysis of the eﬀects of pollution, the design of

aircraft and of power stations. Navier-Stokes methodology has been applied

in computer graphics in order to visualise and create the eﬀects given by

the complex movement of ﬂuids such as coloured gases, air, clouds, liquids,

smoke, ﬁre, etc [96, 97]. The explicit model is generally used for the precise

computation of ﬂuid dynamics and involves heavy computational complexity

[91]. The von Neumann’s stability analysis, as shown in [91], highlights

that the implicit model based on discretisation when calculating Navier-

Stokes equations is unconditionally stable, although it requires a complex

numerical implementation scheme [97, 98]. The stable ﬂuid solver (SFS)

algorithm proposed by Stam represents an implementation of the Navier-

Stokes methodology in an implicit scheme [97, 98] which has been applied

112

for complex graphics modelling.

In order to achieve visual eﬀects, the Navier-Stokes equations are used

for both density and velocity modelling in the SFS algorithm [97, 98]. Unlike

in the original SFS approach, only the modelling of motion based on the

Navier-Stokes equations are considered in this work. In SFS, the area of

investigation (in this case an image or a segmented region from an image)

is split into cells located on a grid and is assigned a particle to each grid

location. We assume that the SFS system moves the particles around in the

vector ﬁeld, where each vector corresponds to a grid location. The Navier-

Stokes equation for a given system is derived using the conservation of mass,

momentum and energy for an arbitrary control volume [91] and is given by:

∂V

∂t

= −(V ∇) V−

∇P

ρ

+ ν∇

2

V+f (5.1)

where the change of velocity V over time is represented with respect to the

advection, the gradient of the pressure P, diﬀusion and the external forcing

function f, while ν is a viscosity constant that characterises the ﬂuid and ρ is

a parameter. The pressure is assumed to be constant in the given ﬁeld and

its gradient is zero, i.e. the change in pressure from one spatial position to

another in the vector ﬁeld is negligible. Consequently, the equation employed

by the SFS method is:

∂V

∂t

= −(V ∇) V + ν∇

2

V+f (5.2)

The diﬀusion term ν∇

2

V characterises ﬂuids which are assumed incom-

pressible and Newtonian. Moreover, for incompressible ﬂuids it is important

to enforce the conservation of mass [91] condition:

113

∇ V = 0 (5.3)

which states that the divergence of velocity components is zero for inﬁnites-

imal time steps. The density of a particle is constant between iterations,

thereby the total mass of the ﬁeld is conserved within the given region.

for k ← 1 to convergence or number of iterations

do

1 add force: V

1

= V

0

+f ∆t

2 advect: V

2

(x) = adv(V

1

(x, −∆t))

3 transform:

ˆ

V

2

= FFT(V

2

)

4 diﬀuse:

ˆ

V

3

(z) =

ˆ

V

2

(z)/(1 +ν∆tk

2

)

5 conserve:

ˆ

V

4

= conserve(

ˆ

V

3

)

6 transform: V

4

= FFT

−1

(

ˆ

V

4

)

Figure 5.1: The stable ﬂuid solver algorithm.

The SFS algorithm proceeds to calculate the velocity components V as

described in Figure 5.1 [97]. For each iteration, the ﬁrst step consists of

adding the external forcing function f which determines the initial conditions

in the processing cycle. The second step represents the advection term in

equation (5.2), which corresponds to the following:

(V ∇) V =

_

V

x

∂V

x

∂x

+ V

y

∂V

x

∂y

, V

x

∂V

y

∂x

+ V

y

∂V

y

∂y

_

(5.4)

where V = (V

x

, V

y

). The analysis of the advection process in real physical

phenomena is provided in [91]. The process described by equation (5.4) is

114

known as the self-advection of velocity. The advection step from the SFS

algorithm is implemented using an implicit ﬁnite diﬀerence scheme. It is

quite impractical to model the motion from one grid location to another

using diﬀerence schemes due to constraints such as computational stability

and ill-posed system of equations. However, by assuming that the motion

vector of each grid cell is a particle, these can be traced back in time with

−∆t by backtracking the velocity ﬁeld. By doing this, the motion vector at

the backtraced location is interpolated from the neighbouring four locations.

This interpolated vector is then used as a basis for further computations.

The third step transforms the velocity ﬁeld to the frequency domain using

the Fast Fourier Transform (FFT). The requirement to set speciﬁc boundary

conditions is eliminated by extending the spatial repeatability of the area

under consideration and by applying FFT. The diﬀusion term (fourth step)

represents the decay of high spatial frequencies in the velocity ﬁeld and is

computed in the Fourier domain with a Gaussian ﬁlter processing the velocity

component V by using the time step ∆t and the ﬂuid kinematic viscosity ν.

The ﬁnite diﬀerence implicit scheme is used here to discretise the diﬀusion

term in order to obtain an unconditionally stable system [97]. The ﬁfth

step enforces the local incompressibility of the optical ﬂow which requires

that the amount of ﬂow entering a speciﬁc area should be equal to the ﬂow

exiting that area. The ﬁnal step projects the ﬂow back from the frequency

domain to the spatial-time domain using the inverse FFT transform. This

algorithm was modiﬁed in [98] by replacing the FFT transformations and the

processing in the frequency domain by deﬁning a set of boundary constraints

on a grid-based representation of the ﬂow.

115

5.4 Robust Hybrid Fluid Model

Figure 5.2: Robust hybrid solver.

The implementation of the stable ﬂuid solver [97] provided rather poor

performance in modelling turbulent optical ﬂow estimated from image se-

quences. This is mainly caused due to the uncertainty in the initial estima-

tion of the optical ﬂow which produces noise, particularly in image sequences

displaying complex motion. In order to improve the performance on optical

ﬂows, a robust anisotropic kernel [133] is embedded in the diﬀusion step of

the SFS. Figure 5.2 shows a ﬂow diagram of the proposed robust hybrid ﬂuid

solver. The initial ﬂow can be estimated using the block matching algorithm

116

as in [14] or other motion estimation algorithms [12]. Optical ﬂows provided

by block matching or by using temporal gradient estimation are invariably

noisy [14], particularly in the case of image sequences representing moving

ﬂuids or other complex phenomena.

The ﬁrst processing block corresponds to a reinforcement step and the

proposed method is implemented by adding a proportion of the velocity from

the previous iteration to the current velocity:

V

1

(t + ∆t) = (1 −w)V

0

(t) + w∆tV

5

(t) (5.5)

where V

5

(t) is the motion vector from the previous iteration t, w ∈ (0, 1)

is a weighting factor modelling the degree of the reinforcement, V

0

(t) and

V

1

(t + ∆t) represent the motion vector reinforced by force at times t and

t+∆t, respectively. At the ﬁrst iteration there is no reinforcement, i.e. w = 0.

The SFS algorithm described in Section 5.3 proposes to advect the initial ﬂow

at Step 2 from Figure 5.1.

However, the algorithm described above produces unreliable estimation

when applied to noisy vector ﬁelds. The optical ﬂow should have a degree of

smoothing before advection can be applied. In the proposed approach, the

noisy ﬂow is diﬀused before proceeding to the advection stage. The transfer

function of the original smoothing algorithm is a Gaussian function appropri-

ately deﬁned within the frequency domain [97]. The proposed method is to

implement a Hessian based diﬀusion that jointly processes the local geometry

and the statistics of the local vector ﬁeld as in [133]:

117

ˆ

V

2

(t + ∆t) =

x

i

∈η(zc)

V

1,i

(t) exp[−(x

i

−z

c

)

T

H

−1

(x

i

−z

c

)]

x

i

∈η(zc)

exp[−(x

i

−z

c

)

T

H

−1

(x

i

−z

c

)]

(5.6)

where

ˆ

V

2

(t + ∆t) is the intermediate diﬀused value, H represents the local

Hessian, V

1,i

(t) is the vector at location i within a neighbourhood η(z

c

),

centred at the location z

c

.

The eigenvector corresponding to the largest eigenvalue of the Hessian sh-

ows the local direction of the optical ﬂow. This diﬀusion kernel is anisotropic

and adapts to the local structure of the optical ﬂow. Signiﬁcant optical

ﬂow transitions are automatically detected and consequently not smoothed

over by the Hessian-based kernel. However, anisotropic diﬀusion does not

deal properly with outliers as shown in a study provided in [133] and de-

scribed in Section 3.4.1. In order to properly process the local statistics

and eliminate outliers, the median algorithm is considered for robustifying

the Hessian based diﬀusion in the neighbourhood η(z

c

), as implemented in

equation (3.19).

At the advection stage, this model is only concerned with the nonlinearity

of the advection term from equation (5.4). As mentioned in Section 5.3, the

self-advection term represents the ability of the velocity components to move

their own values from one position to another on a grid in a time step interval,

∆t in order to model complex movement usually combining rotation and

translation. This procedure involves interpolating the velocity at the grid

points, using a neighbourhood approximation, from the previous time step

back to the position in the current time step [98].

The model is dependent on the initialisation and on boundary conditions

118

of the system under study. In the proposed approach, boundary conditions

are speciﬁcally provided onto the grid in order to represent the physical limits

of the optical ﬂow. Such boundary conditions can be the result of image or

motion segmentation algorithms or of the existence of a priori information

about the image sequence. There are two boundary conditions to consider.

The ﬁrst condition is determined by the physical boundary and is represented

by the von Neumann condition which speciﬁes the normal component of the

ﬂow to the boundary surface as:

∂V

∂n

¸

¸

¸

¸

Ω

= 0 (5.7)

where Ω represents the boundary and n is its surface normal. This equa-

tion implies that the wall absorbs any ﬂow particles coming towards it. This

condition is also commonly known as the no-slip boundary condition. For

the sake of reducing the required computation complexity, the walls of the

domain, Ω are represented by zero values on a geometric grid, which are en-

forced at every stage of the computation in order to preserve the stability and

integrity of the numerical calculation. Since our proposal incorporates both

explicit and implicit ﬁnite diﬀerencing schemes, it is absolutely imperative

that the model adheres to the stability criteria, given by ∆t/(∆x)

2

≤ 1/2

[91], where ∆x represents the location change during the time interval ∆t.

The second condition relates to the conservation of mass of the velocity

ﬁeld. The conservation of mass, given by equation (5.3), should be main-

tained in order to ensure the incompressibility of the ﬂow.

In order to maintain a divergence free velocity ﬁeld for every stage of

computation, the conservation of mass is enforced after both diﬀusion and

119

advection stages. The conservation of mass stage corresponds to a data

normalisation process. This is enforced by using the Helmholtz-Hodge de-

composition [97] of the velocity ﬁeld. Using the projection method (based

on the Helmholtz-Hodge decomposition) as described in [127], the projection

of the pressure gradient onto a divergence free vector ﬁeld is thus given as

Tv = w and ∇ w = 0, Tv = v if and only if ∇ v = 0, T∇P = 0 [134].

This decomposition provides an exact solution so that the mass conserved

incompressible ﬂow can be obtained by extracting the gradient of the ﬂow

from the current vector ﬁeld. The decomposition can be shown to be

w = V+∇q (5.8)

where w is any vector ﬁeld, V is mass conserved and divergence free (5.3)

and ∇q is the gradient of a smooth scalar ﬁeld. By taking the divergence of

equation (5.8), the Poisson equation as shown in the following equation is

obtained:

∇ w = ∇ V+∇ ∇q

∇ w = ∇

2

q (5.9)

where condition (5.3) is applied. The Poisson equation obtained in equa-

tion (5.9) is sparse and is solved using the Gauss-Seidel relaxation method,

similarly described in [128]. This is because the Gauss-Seidel relaxation

method is simple to implement, achieves fast convergence and new approxi-

mations are used instantly for subsequent computations. Although there are

better relaxation schemes available, this method is suﬃcient for the current

implementation. Subsequently, the incompressible ﬁeld is obtained by

120

V = w−∇q (5.10)

as suggested by the Helmholtz-Hodge decomposition. This decomposition

maintains the incompressibility and smoothness of the estimated velocity

ﬁeld. Mass conservation is important for realistically estimating optical ﬂow

of ﬂuids. For example, the exact Helmholtz-Hodge decomposition of the

closed cavity laminar ﬂow (artiﬁcial data provided in Section 5.6) at the

thousandth iteration is shown in Figure 5.3.

Current Flow = Incompressible Flow + Gradient Flow

Figure 5.3: Helmholtz-Hodge decomposition of a closed lid driven cavity

laminar ﬂow.

5.5 Vortex Core Detection

After smoothing and modelling the vector ﬁeld as described in Chapter 2

and Section 5.4, it is now possible to extract salient patterns that are char-

acteristic to turbulent ﬂows such as vortices. Although the vector ﬁeld have

been smoothed, there is no clear process to verify whether the smoothed ﬁeld

is either good or bad ﬂow. This is especially important for fuel mixing in

jet thrusters [135]. The presence of vortices in such ﬂows could cause mas-

121

sive energy loss within the combustion chamber and subsequently cause an

unstable fuel chain reaction with potential damaging consequences. There-

fore the detection and tracking of vortices in turbulent ﬂows is of signiﬁcant

importance. Furthermore, it is possible to identify types of ﬂow ﬁelds from

the dynamics of vortices. Vortices in ﬂuid mechanics represent rotational

structures of concentrated energy around a centre [100, 103, 112, 136]. Usu-

ally, the centre is characterised by a minimum in the pressure and by zero

velocity. However, this is not the case in most turbulent ﬁelds.

A Galilean-invariant velocity ﬁeld is assumed to be of a uniformly smooth

ﬂow ﬁeld [135]. The Galilean-invariant vortex detection criteria uses the

scalar velocity gradient decomposition [100, 103]:

∇V = S +Ω (5.11)

where S is the rate of strain tensor:

S =

1

2

[∇V+ (∇V)

T

] (5.12)

and Ω is the vorticity tensor:

Ω =

1

2

[∇V−(∇V)

T

] (5.13)

The Q-criterion deﬁnes a spatial region where the Euclidean norm of the

vorticity tensor dominates that of the rate of strain:

Q =

1

2

[|Ω|

2

−|S|

2

] > 0 (5.14)

where | | represents the Euclidean norm of the tensor. This expression is

further simpliﬁed using equation (5.3) to obtain

122

Q = −

∂V

x

∂y

∂V

y

∂x

> 0 (5.15)

The Q-criterion is further decomposed into Eulerian form [100] as in equa-

tion (5.15) for ease of vortex region identiﬁcation. The equation (5.15) is

also obtained by applying the constraints for incompressibility, as described

in equation (5.3), into equation (5.14). Therefore, the criterion is dependent

on the mixed derivatives of the gradient velocity tensor. Hence, for the ex-

periments, the implementation is based on equation (5.15) since we assume

that the vector ﬁelds under study are incompressible. In addition to the Q-

criterion, the characteristic function of the deformation tensor for 2D vector

ﬁeld is given as [103]:

λ

2

+ pλ + q = 0 (5.16)

where λ is the eigenvalue of the tensor ∇V, while the co-eﬃcients depend on

its determinant and trace as: q = Det(∇V) and p = −Tr(∇V). Hence, the

discriminant of equation (5.16) is:

∆ = p

2

−4q (5.17)

where the discriminant is the root of the characteristic equation (5.16). Both

the Q-criterion and discriminant, ∆, is used to characterise critical regions

of focus, node and saddle in a topology map of a turbulent ﬂow [103]. If ∆

is positive, then the velocity particle is likely to be located within the vortex

core region. If ∆ is negative, then it is likely to be within a saddle region

outside of the vortex core boundaries.

123

These formulae are applicable for vortex detection, on the assumption

that the vortex structures are smooth and can be integrated from a suﬃ-

ciently large energy basin. The smoothness condition is ensured by using the

methodology described in the previous sections by employing the Navier-

Stokes equations with a robust diﬀusion step (Sections 5.3 and 5.4) or by

using diﬀusion algorithms proposed in Sections 3.3 and 3.4. In [100, 103], a

single point evaluation for Q was used.

The Q-criterion is notably susceptible to the presence of noise and in-

variably fails when the velocity ﬁeld is not Galilean-invariant. The current

work attempts to improve this by using a running window in which several

Q

i

using (5.15) are evaluated. A set of four vectors, symmetrically located

at the extremes of a cross, window centered, is used for calculating each Q

i

.

The resulting Q

w

is estimated as a weighted average of all Q

i

, each calculated

according to equation (5.15), from a set of four vectors located as shown in

Figure 5.4:

Q

w

=

25

i=1

w

i

Q

i

25

i=1

w

i

(5.18)

where w

i

represents the inverse of the Euclidean distance from the window’s

center to the location of each of the four vectors. For a 5−by−5 window, the

weights correspond to w

i

= ¦1,

1

√

2

,

1

2

,

1

√

5

,

1

√

5

,

1

2

√

2

¦, depending on the inverse

shortest path between pixel in a neighbourhood and its center.

The vortex structure is identiﬁed for ∆ > δ, where δ is an arbitrary

value. Our studies show that an appropriate δ should be about 0.2 from the

maximum value of ∆ topology map.

124

The weighted discriminant, ∆

w

is also computed in a similar way as to

the Q

w

-criterion as shown in equation (5.18). The computation of ∆

w

is

shown to be

∆

w

=

25

i=1

w

i

∆

i

25

i=1

w

i

(5.19)

As mentioned in [100], the discriminant, ∆ is used as another set of

constraints for identifying vortex core regions. This computation is useful

for segmenting speciﬁc regions especially when the ﬁeld is smooth. As with

the other constraints, this criterion fails in the presence of noise. Hence, the

introduction of ∆

w

, which does provide a more reliable estimation of the

vortex structure.

Vortex structures are segmented by correlating the evidence in both the

velocity ﬁelds and the vorticity maps using ∆ > δ as a constraint. This is

done by ﬁrst thresholding the topology map with the function

1(∆) =

_

_

_

1 , (∆ > 0.2)

0 , (∆ ≤ 0.2)

(5.20)

where 1() represents an indexing function. By now, it is expected that the

regions of interest (ROI), i.e. vortex regions are segmented in the index array.

Subsequently, the function 1 is convoluted with either V or [ω[ to obtain

segmented vortex regions in the vector ﬁeld or vorticity map. Additional

functions can be added to this process to allow tracking of vortices from

frame to frame. This is especially useful in weather forecasting and geo

remote sensing applications.

125

Figure 5.4: Evaluation of the Q

w

-criterion from a 5 −by −5 window, where

the location of each vector in the window is shown with a diﬀerent marker

according to its corresponding Q

i

set.

5.6 Experimental Results

The experimental results presented below show the evaluation of the pro-

posed algorithm on synthetic ﬂuid sequences and on real-world image se-

quences.

126

5.6.1 Synthetic data simulation set-up

This section describes the experimental set-up required to simulate artiﬁ-

cial sequences based on the Navier-Stokes equations. The ﬁrst part of the

research results focusses on two diﬀerent types of artiﬁcial sequences. The

ﬁrst is a closed driven-lid cavity ﬂow and the second is the ﬂow of the von

Karman’s vortex sheet.

Closed driven-lid cavity ﬂow

The ﬁrst synthetic sequence is created using the original Navier-Stokes equa-

tions [91] for a closed driven-lid cavity ﬂow. The air ﬂow within the cavity

is modelled and observed when the top wall is moved with constant velocity

as shown in Figure 5.5. The synthetic ﬂow is created using the vorticity-

stream formulation of the Navier-Stokes equations instead of the standard

velocity-pressure formulation.

Figure 5.5: Driven-lid cavity ﬂow diagram.

With reference to Figure 5.5, the walls of the cavity are marked with

A (top), B (left), C (right) and D (bottom). The initial and boundary

127

conditions for the four walls are given as follows:

• A: V

x

= V e = 5m/s, Ψ = 0,

∂P

∂n

= 0, ω = 2

Ψ

0,j

−Ψ

1,j

(∆y)

2

• B: V = 0, Ψ = 0,

∂P

∂n

= 0, ω = 2

Ψ

i,0

−Ψ

i,1

(∆y)

2

• C: V = 0, Ψ = 0,

∂P

∂n

= 0, ω = 2

Ψ

i,ny−1

−Ψ

i,ny−2

(∆y)

2

• D: V = 0, Ψ = 0,

∂P

∂n

= 0, ω = 2

Ψ

nx−1,j

−Ψ

nx−2,j

(∆y)

2

with ω being vorticity and Ψ being the stream functions of the velocity. Both

vorticity and stream functions can be represented in terms of velocity by

ω = [ω[ = [∇V[ =

∂V

y

∂x

−

∂V

x

∂y

(5.21)

and

∂Ψ

∂y

= V

x

,

∂Ψ

∂x

= −V

y

(5.22)

In addition to the given boundary conditions, there is also a constant

pressure applied at grid position (0,0) of 2bars to move the lid (top wall).

The sequence is corrupted with additive Gaussian noise of zero mean and

test sequences when varying the noise variances are obtained as follows.

Figure 5.6(a) represents the ground truth synthetic ﬁeld that visualises

the movement of air ﬂow moving inside the area of a closed driven-lid cavity

with a ﬁxed velocity. The ﬁgure shows the snapshot of the ﬂow after a thou-

sand iterations with time intervals of 1ms between iterations. Figure 5.6(b)

shows ﬂow degradation after adding Gaussian noise with variance σ

2

= 0.01,

Figure 5.6(c) is ﬂow with Gaussian noise with variance σ

2

= 0.10 and Fig-

ure 5.6(d) is ﬂow with Gaussian noise with variance σ

2

= 0.25. These are

128

(a) Ground truth ﬂow (b) σ

2

= 0.01

(c) σ

2

= 0.10 (d) σ

2

= 0.25

Figure 5.6: Synthetic closed driven-lid cavity ﬂows with noise.

examples of degraded vector ﬁelds in order to be used for testing the nonlin-

ear smoothing methodology.

von Karman’s vortex sheet

The second synthetic sequence is the von Karman ﬂow which is created

using models suggested in [92]. This model is used to simulate the creation

of artiﬁcially complex and turbulent ﬂows which is used as ground truth

for comparative analysis. The model used to create the ﬂows is shown in

Figure 5.7 and is based on the velocity-pressure formulation of the Navier-

129

Stokes equations.

Figure 5.7: Half-cylinder model for von Karman ﬂow.

With reference to Figure 5.7, the walls of the cavity are marked by A

(top), B (bottom), C (left) and D (right). The initial and boundary condi-

tions for the four walls are given as follows:

• A:

∂V

∂n

= 0, P = 0

• B:

∂V

∂n

= 0, P = 0

• C: V

x

= V e = 1m/s

• D:

∂V

∂x

= 0, P = 0

In addition to the numerical conditions at the walls, the default boundary

constraint on the half-cylinder, G is

∂V

∂n

= 0. This simple model, without

additional Dirichlet or Neumann conditions is simulated using Gerris Flow

Solver [137] with T = 9s and ∆t = 0.1.

130

5.6.2 Results on synthetic data

Closed driven-lid cavity ﬂow

Modelling results using the modiﬁed SFS (SFSM) algorithm [98] adapted for

usage on vector ﬁelds is shown in Figure 5.8(a), while vector ﬁeld smoothing

using Black’s anisotropic diﬀusion algorithm [7] is shown in Figure 5.8(b).

Figure 5.8(c) shows the eﬀects of using MED-2DH which is a robust Hessian

based diﬀusion algorithm described in [133], while the robust hybrid ﬂuid

solver embedding the median of 2D Hessian diﬀusion kernel (MedH-SFS)

algorithm, as described in Section 5.4, is shown in Figure 5.8(d). For better

visualisation, the vector from the upper-right corner of the SFSM vector ﬁeld

in Figure 5.8(a) has been rescaled.

The results in Figure 5.8 are obtained at convergence when the mean

square error diﬀerence between vector ﬁelds at two successive iterations is

less than 0.01. The number of iterations necessary to achieve convergence

is provided in the parentheses from the caption of each result of Figure 5.8.

From these results, it can be observed that the vector ﬁeld modelled by SFSM

is still noisy at convergence, while the noise has been signiﬁcantly reduced

in the other smoothed vector ﬁelds. The results also show that MedH-SFS

provides the best outcome and the vortex recovered is better located when

compared to the vortices recovered using Black and MED-2DH.

For numerical comparisons, the mean cosine error (MCE) between the

recovered smoothed ﬂow and the ground truth ﬂow is considered. The MCE

is calculated as in equation (3.26). The MCE results provided in Table 5.1 is

after one iteration of smoothing. SFS algorithm as described in Section 5.3

was adapted from [97], while SFSM has been described in [98]. Both these

131

(a) SFSM (6) (b) Black (3)

(c) MED-2DH (4) (d) MedH-SFS (5)

Figure 5.8: Closed cavity vector ﬁeld smoothing comparisons.

Gaussian Noise (σ

2

) SFSM SFS MedH-SFS Black MED-2DH

0.01 0.7525 0.6211 0.7634 0.7226 0.7383

0.10 0.6020 0.5616 0.7327 0.6554 0.6997

0.25 0.4538 0.4523 0.6849 0.5584 0.6424

0.30 0.4373 0.4624 0.6704 0.5567 0.6058

0.40 0.4005 0.4184 0.5799 0.4958 0.5556

Table 5.1: Mean cosine error of smoothed vector ﬁelds.

132

algorithms have been adapted to process vector ﬁelds. It can be observed

that SFS performs reasonably on a vector ﬁeld corrupted with low noise

variance. However, its performance deteriorates signiﬁcantly when the noise

increases, because the corrupted vector ﬁeld departs signiﬁcantly from the

Navier-Stokes underlying model. The robust diﬀusion hybrid ﬂuid algorithm

MedH-SFS provides better results than either SFS or SFSM methods in terms

of MCE when considering additive Gaussian noise, as it can be observed from

Table 5.1. MedH-SFS is also consistently better than Black [7] and MED-

2DH [133] anisotropic smoothers.

von Karman’s sheet

The velocity ﬁeld estimation from the area cropped in Figure 5.9(a) is shown

in Figure 5.9(b) and is used as ground truth information for comparative

purposes. Figure 5.9(c) shows the vector ﬁeld corrupted by adding Gaus-

sian noise distribution with variance 0.10 on each of the x and y axes. The

noisy turbulent vector ﬁeld produced by the von Karman ﬂow is smoothed

using the median of the Hessian Hybrid Fluid Solver algorithm (MedH-SFS)

described in Section 5.4, SFS (Section 5.3) provided in [97, 127], SFSM

algorithm described in [98] and the Hessian-based diﬀusion algorithm of

Tschumperl´ e and Deriche (TD) [5, 6] adapted for use on optical ﬂows. Fig-

ures 5.10(a)-(c) represent the smoothed ﬂow obtained at convergence using

the MedH-SFS, SFSM and TD, respectively (all three results have been sub-

sampled by a factor of 2 along both x and y directions).

As it can be observed, the SFSM smoothed ﬂow is visually the best

smoothed ﬁeld. However, SFSM took 4 iterations to converge and has over-

133

(a) Initial von Karman ﬂow

(b) Motion ﬂow extracted from the von Karman ﬂow

(c) Noisy von Karman ﬂow with σ

2

= 0.10

Figure 5.9: Representing von Karman ﬂows.

smoothed the ﬂow. The result shown could only be made visible after rescal-

ing by a factor of 8. In contrast, the TD smoothed ﬂow (only 1 iteration,

134

(a) Smoothed MedH-SFS ﬂow

(b) Smoothed SFSM ﬂow

(c) Smoothed TD ﬂow

Figure 5.10: Smoothing noisy von Karman ﬂows.

135

as it fails thereafter) is hardly diﬀused and remains quite noisy. In compar-

ison to SFSM, MedH-SFS does diﬀuse the velocity ﬁeld quite well within

3 iterations without rescaling. Though the vortices have been successfully

recovered (albeit not perfectly), it is possible that some regions of the ﬁeld

may have been oversmoothed.

The method described in Section 5.5 is applied for ﬁnding vortex regions

using the Q-criterion and the discriminant ∆. Figure 5.11(a) represents

the measure of the Q-criterion evaluated according to equation (5.14) for

measuring the vorticity and Figure 5.11(b) represents the locally weighted

Q

w

-criterion calculated as in equation (5.18). These results highlight the

saddle-type behaviour characteristic of the ﬂow enveloping the vortex core

[100]. The high values in the map correspond to the 2D stable manifold

characterising the regions where the Euclidean norm of the vorticity ten-

sor dominates the rate of strain (5.14), while lighter colours in the ﬁgure

correspond to smaller dispersions of the vortex energy.

From Figure 5.11, it can be observed that the measure Q

w

from (5.18)

provides a clearer identiﬁcation of the four intersections of the upper and

lower sides of the vortex rings (corresponding to hyperbolic regions around

the center of the vortex core) when compared to the locally estimated Q,

according to (5.14).

Table 5.2 presents numerical analysis results for the eﬀects of smoothing

von Karman ﬂows corrupted with a Gaussian noise of mean zero and variance

0.10 as shown in Figure 5.9(c), after one iteration for consistent comparison.

The table also presents the number of iterations required to achieve conver-

gence. The mean squared error (MSE) records the diﬀerence between the

136

(a) Q-criterion

(b) Weighted average Q

w

-criterion

Figure 5.11: Evaluating Q and Q

w

on the von Karman ﬂow using equations

(5.14) and (5.18), respectively.

ground truth, as shown in Figure 5.9(a) and the smoothed ﬂows. The MSE

plot after smoothing various von Karman ﬂows with noise of increasing vari-

137

ance, σ

2

∈ [0.01, 0.80] is displayed in Figure 5.12. From this plot it can be

clearly seen that SFS performs better than the other algorithms with respect

to the reduction of MSE.

SFSM SFS MedH-SFS TD

No. iterations 6 1 5 1

MSE 3.917 0.456 2.677 13.282

MCE 0.828 0.842 0.894 0.197

∆

E

(%) 1.37 1.77 1.29 1.37

∆

w,E

(%) 1.73 6.82 1.13 3.53

Table 5.2: Evaluation of smoothed noisy von Karman ﬂows when the noise

variance is 0.10.

0 0.01 0.05 0.10 0.15 0.20 0.25 0.30 0.50 0.60 0.80 1.00

0

2

4

6

8

10

12

14

Noise variance

M

e

a

n

S

q

u

a

r

e

E

r

r

o

r

SFSM

SFS

MedH−SFS

TD

Figure 5.12: MSE after smoothing the noisy von Karman ﬂow.

The constraints, Q, Q

w

, ∆ and ∆

w

are used to detect the regions of

vorticity, according to ∆ > δ, where δ is approximately twenty percent of

138

0 0.01 0.05 0.10 0.15 0.20 0.25 0.30 0.50 0.60 0.80 1.00

0

1

2

3

4

5

6

7

8

9

Noise variance

S

e

g

m

e

n

t

a

t

i

o

n

E

r

r

o

r

,

∆

w

,

E

(

%

)

SFSM

SFS

MedH−SFS

TD

Figure 5.13: Performance of the vorticity segmentation using ∆

w

.

the maximum discriminant (∆) value. Such vorticity regions are detected

in both the ground truth ﬂows as shown in Figure 5.14(b) as well as in the

smoothed ﬂows, including those from Figures 5.10(a)-(c). The regions of

vorticity characterised by high ∆ are segmented and compared between the

segmented regions in both the smoothed and the ground truth ﬁeld.

Consequently, the percentage of mis-classiﬁcation and of false alarms is

measured for the vorticity regions for all the tested algorithms SFS, MedH-

SFS, TD and SFSM, calculated as a percentage of the total number of vectors

in the vector ﬁeld. The vorticity error measure denoted as ∆

E

and ∆

w,E

is

provided in Table 5.2 for noise of variance 0.10. Similar to the Q and Q

w

criterions, the diﬀerence between ∆ and ∆

w

is their sensitivity to noise.

For example, Figure 5.14 shows the topology map for ∆ and ∆

w

, computed

from a MedH-SFS smoothed ﬂow after 1 iteration. In this example, the

initial ﬂow was corrupted with Gaussian noise (variance = 0.01). As it can

139

be observed, the vortex regions in Figure 5.14(a) is visibly faint. However,

the vortex regions in Figure 5.14(b) can easily be identiﬁed with the weighted

discriminant, ∆

w

.

The plot in Figure 5.13 shows the performance of vortex segmentation

for various algorithms when increasing the noise variance. Though SFS has

the least error initially, as the amount of noise is increased, its performance

gradually gets worse than MedH-SFS. The mean vortex segmentation error

measurement is calculated as

∆

E

=

1

N

2

N

2

i=1

[1(∆

O

) −1(∆)[

N

2

i=1

1(∆

O

)

100 (5.23)

where N

2

is the size of the vector ﬁeld, 1(∆

O

) is the index array of the dis-

criminant, calculated for ∆ > 20% of its maximum value for the ground truth

ﬁeld and 1(∆) represents the index array of the discriminant, calculated for

∆ > 20% of its maximum value for the smoothed ﬂows. The same equa-

tion can be used for the weighted discriminant, ∆

w,E

. However, it should

be noted that ∆

w,E

gives better results than ∆

E

when the variance is be-

tween 0.01 and 0.20. Thereafter, ∆

E

is signiﬁcantly better. This behaviour

is expected for higher noise since there is a larger likelihood of segmentation

mis-classiﬁcation due to larger ∆

w

values.

The overall observed results show that among the smoothing methods

tested, generally MedH-SFS outperforms the other methods. This method

provides smooth ﬂows and succeeds in recovering all 5 vortex core regions

(from the cropped ﬁeld) after 1 iteration. This is veriﬁed by the MCE results

140

(a) ∆

(b) ∆

w

Figure 5.14: Evaluating ∆ and ∆

w

on a smoothed von Karman ﬂow from

equations (5.17) and (5.19), respectively.

141

presented in Table 5.2, which shows that the MedH-SFS smoothed ﬁeld is

the best match to the ground truth ﬂow. Unlike the ﬂuid based methods,

TD was unable to smooth the ﬂows reliably and hence could not recover

the vortex regions properly. It is important to note that reconstructions

from TD smoothed ﬂows were very poor, hence reﬂecting the high errors

in Table 5.2. These error measurements are based on the magnitude and

direction of the vector. However, TD smoothed ﬂow does give a reasonable

value for segmentation error. This is possible due to vectors being in the

correct location for segmentation purposes, despite bad ﬂow estimations.

5.6.3 Real image sequences and remarks

The proposed methodology of hybrid ﬂuid smoothing is tested on optical

ﬂows estimated from image sequences. Figure 5.15(a) represents a frame

from the Tornado image sequence, while Figure 5.15(b) shows a frame from

the Solar Flare sequence obtained from Kanzelh¨ ohe Obervatory’s solar and

environmental research website

1

. The ﬁrst sequence represents a complex

atmospheric phenomenon while the second image sequence is used to observe

and analyse solar surface activity.

The initial optical ﬂows have been estimated using block matching algo-

rithm (BMA) and are shown in Figures 5.15(c) and (d), respectively. The

complexity of the motion in the scenes as well as the compression artefacts

inﬂuence negatively the performance of the BMA algorithm. Figure 5.15(e)

and (f) show the smoothing result when using MedH-SFS algorithm on the

optical ﬂow estimated from the Tornado sequence and from the Solar Flare

1

Sequence obtained from http://www.solobskh.ac.at/index en.php

142

(a) Original frame 341 (b) Original frame 220

(c) Initial BMA ﬂow (d) Initial BMA ﬂow

(e) MedH-SFS smoothed ﬂow (f) MedH-SFS smoothed ﬂow

Figure 5.15: Smoothing optical ﬂows in image sequences displaying turbulent

motion.

143

optical ﬂow, respectively, both after one iteration. The recovered optical ﬂow

smoothed with MedH-SFS is a signiﬁcant improvement over the initial op-

tical ﬂows. We can clearly identify moving twister and its boundaries after

using the proposed methodology as it is shown in Figure 5.15(e). Turbulent

movements of the solar surface can be properly identiﬁed in Figure 5.15(f).

The proposed methodology was also applied on real image sequences

showing atmospheric phenomena such as those that occurred during the Su-

perstorm Andrea. The satellite image sequence of the Superstorm Andrea

contains 24 images and captures the formation of the Superstorm Andrea.

2

The data set consists of 24 images acquired at 30 minute intervals per frame

covering a total period of 12 hours. Frame 22 from the Superstorm Andrea

image sequence is shown in Figure 5.16(a), while the optical ﬂow is calcu-

lated, using the block matching algorithm, between frames 20 and 22 is shown

in Figure 5.18(a). As it can be observed, the ﬂow is very noisy and is inﬂu-

enced by the map information as well as by the ﬂickering of lights (as this

is a night satellite image capture) and by additional geographic information.

The movement of clouds from frame to frame is not very clear when using the

block matching algorithm. Another possible cause of the noise is the random

bursts of clouds in the region. Figure 5.18(b) and Figure 5.18(c) show the

eﬀect of smoothing using MedH-SFS and SFSM algorithms, respectively.

Figure 5.16(b) shows the topology map of the 2D discriminant ∆ ex-

tracted from the image sequence showing the turbulent storm. Nine main

concentrations of energy (vortices) are identiﬁed on this map and their loca-

tion is marked with distinctive numbers. On the right side of the map (over

the Atlantic Ocean), a concentration of detected vortices can be observed.

2

Sequence obtained from http://www.nrlmry.navy.mil/sat products.html

144

(a) Original reference frame 22 of Superstorm Andrea

(b) Vortex detection from the smoothed Andrea ﬂow using MedH-SFS

based on ∆ topology

Figure 5.16: Finding vortices in the Superstorm Andrea sequence.

145

(a) Vortex detection from the smoothed Andrea ﬂow using Black’s

algorithm based on ∆ topology

(b) Vortex detection from the smoothed Andrea ﬂow using MED-2DH

based on ∆

w

topology

Figure 5.17: Additional results of vortex segmentation of Superstorm Andrea

sequence.

146

(a) Initial ﬂow from frame 20 to frame 22

(b) Smoothed ﬂow using MedH-SFS

(c) Smoothed ﬂow using SFSM

Figure 5.18: Smoothing optical ﬂow corresponding to cloud movement for

Superstorm Andrea.

147

This area corresponds to the region of main activity for the Superstorm

Andrea. These results show clearly that the region of principal turbulent ac-

tivity, as recorded by the movement of clouds observed in this satellite image

sequence takes place mainly above the oceans. Further results showing topol-

ogy maps of ∆ and ∆

w

are shown in Figures 5.17(a) and (b) respectively.

The segmentation shown in (a) is obtained from Black [7] smoothed ﬂow.

In contrast, the segmentation in (b) is obtained from MED-2DH smoothed

ﬂow. The main diﬀerences between the results shown is that when using ∆

w

,

the segmentation is clearer and vortex core regions can be clearly identiﬁed.

This shows that the ∆

w

produces a clear improvement over ∆.

In addition to the vortex region identiﬁcation for the Superstorm Andrea

sequence, the algorithm is also tested on the Solar Flare sequence, as shown

in Figure 5.19. The regions of high energy in Figure 5.19(b) correlates to the

turbulent motion observed in Figure 5.19(a). An interesting point to note

is that some of these high energy regions do not mimic vortex behaviour.

Instead, they are either converging or diverging from points which are the

focus of particle movements.

From these results, the beneﬁts of using a robust diﬀusion stage integrated

in the computational ﬂuid dynamics methodology such as that implemented

by Navier-Stokes equation is that it not only smoothes the vector ﬁeld, but

also contributes to detecting vortices and other regions of concentrated tur-

bulence. The proposed measure of vorticity detection using ∆

w

proved to

give reliable and accurate estimates of the areas with signiﬁcant turbulence.

148

(a) Smoothed MedH-SFS ﬂow after 6 iterations

(b) Vortex detection from the smoothed ﬂow in (a) based on ∆ topology

Figure 5.19: Finding vortices in the Solar Flare sequence.

5.7 Conclusion

A ﬂuid dynamics based model for smoothing optical ﬂow from image se-

quences representing complex and turbulent ﬂows has been presented. The

Stable Fluid Solver (SFS) method is based on the Navier-Stokes equations.

The SFS method was originally used for computer graphics and visualisa-

tion applications. In this thesis, the SFS method has been modiﬁed for the

149

speciﬁc use on optical ﬂows displaying turbulent motion.

An important point to note is that the optical ﬂows used for the ex-

periments do obey incompressible and Newtonian assumptions. Although

optical ﬂows are obtained from 3D real time scenes with changeable condi-

tions, the ﬂows themselves are constrained by the boundaries of the ﬁeld at

every snapshot. No additional physical conditions could therefore aﬀect the

ﬂows. Furthermore, the volume given by the number of vectors of the ﬂows

is conserved, since it is dependent on the motion estimation algorithm and

the boundaries are consequently deﬁned.

The motion is estimated following a set of processing steps implementing

reinforcement, diﬀusion, advection and mass conservation. An explicit dif-

fusion model is used after the reinforcement stage and before the advection

stage. The median of the local Hessian-based kernel is considered for the

diﬀusion stage. This type of kernel ensures that smoothing occurs along the

structure of the motion ﬁeld maintaining the moving objects boundaries and

the main optical ﬂow features. Moreover, the kernel embeds robust statis-

tics capability by reducing the impact of outliers and thus enhancing the

smoothness of the resulting optical ﬂow.

After modelling the optical ﬂow using the proposed method, an approach

for identifying salient patterns such as vortices (coherent structures) is used.

This method uses the Q-criterion and the discriminant ∆, as described in

Section 5.5 to identify, segment and track vortices. Vortices are spatial re-

gions characterised by rotational movement where the Euclidean norm of the

vorticity tensor dominates that of the rate of the tensor. In this thesis, we

propose to use weighted estimates of these measures, calculated from all the

150

possible combinations of four vectors distributed at the extremes of a cross

from a given window. The proposed methodology is applied on artiﬁcial

vector ﬁelds as well as on the turbulent ﬂow representing natural phenom-

ena. The proposed methodology can be applied for identifying and detecting

turbulent phenomena from satellite image sequence and it can be used for

tracking storms.

151

Chapter 6

Conclusion and Outlook

This conclusion chapter commences with a summary of the contributions of

Chapters 3 to 5 which includes a discussion of the strengths and weaknesses

of the nonlinear diﬀusion techniques. Finally, directions for future work will

be discussed.

6.1 Summary of Contributions

This thesis has three main contributions in the area of vector ﬁeld diﬀusion.

6.1.1 Robust nonlinear diﬀusion

The ﬁrst contribution to the computer vision ﬁeld is presented in Chapter

3. Here, we brieﬂy outline the our methodology for a robust anisotropic

diﬀusion algorithm to smooth optical ﬂows.

The proposed nonlinear diﬀusion algorithms in Chapter 3 are kernel based

ﬁlters and are dependent on the information of the local Hessian. The kernel

is anisotropic due to the Hessian matrix, which provides orientation infor-

152

mation of the data structure and hence copes better with identifying edges

and speciﬁc data transition patterns. The Hessian based kernels are made

statistically robust using the median or the alpha-trimmed mean algorithms

for better performance. In contrast to other known methods, the proposed

diﬀusion kernels are primarily designed for use on optical ﬂows from image

sequences displaying object deformation over time, although the kernels can

be adapted for use on images as well. The diﬀusion algorithms were tested on

synthetic vector ﬁelds and on optical ﬂows extracted from real-world image

sequences displaying various types of motion. The algorithms were compared

against various other methods [4, 6, 7, 34].

6.1.2 3D volumetric slice interpolation

3D slice interpolation technique has been around for a while and has pre-

viously been used for 3D volumetric reconstructions [14, 72]. Furthermore,

there are many diﬀerent types of techniques being used to obtain interme-

diate slices for the reconstruction. One example being the work of Weng

et. al. [72] which uses optical ﬂows.

We redeﬁne optical ﬂows as structural ﬂows, since these describe bet-

ter the deformations between image slices of soft and hard tissue of human

anatomy and can be used on medical images in general. The ﬂows are also

able to capture the direction of textural change within the inner structure

of the objects. Most ﬂow algorithms are uni-directional. In contrast, the

proposed contribution in Chapter 4 is bi-directional and thus employs addi-

tional information to appropriately model the morphing between two original

slices. Robust diﬀusion kernels, as proposed in Chapter 3, are then used to

153

remove outliers, especially around the boundaries in order to give better

and more accurate morphing. Intermediate slices are then obtained using

the information from the structural ﬂows and original slices. The collated

slices, including the originals are then stacked to reconstruct a 3D volume,

as suggested in [72]. The Hausdorﬀ distance measure is used to give a better

representation on the 3D shape reconstruction of the volume. The proposed

framework has been tested on three hard tissue data sets and three soft tissue

data sets. The data sets were a mixture of pre-segmented and unsegmented

slices. Currently, only volumes of segmented sets can be rendered, i.e. In-

cisor, Humerus and Iliac bones. Three measurements were taken to validate

the volumetric reconstructions. They are average 3D shape reconstruction

error of intermediate slices, average peak signal-to-noise ratio of intermediate

slices and the Hausdorﬀ distance of a 3D synthesised volume compared to

the ground truth information.

6.1.3 Diﬀusion based on ﬂuid dynamics

There is a huge amount of literature dedicated to using Navier-Stokes equa-

tions in the ﬁeld of ﬂuid mechanics. Navier-Stokes based methodology has

been used in computer graphics for simulating various natural eﬀects. Main

reasons for the slow adoption of this method in computer vision is due to its

nonlinear characteristics, diﬃculty in understanding, complex implementa-

tions and its heavy computational complexity.

A new approach based on Navier-Stokes equations is used for modelling

on optical ﬂows. The main purpose of using this method is for better esti-

mation of ﬂows from image sequences when complex and turbulent motion

154

is present, for example, for the modelling of vortex ﬂow, explosive sequences,

sequences showing storm motion in satellite imagery, etc. Since optical ﬂows

are constrained by construction, it is assumed that they exhibit incompress-

ible and Newtonian properties. The proposed method incorporates both

explicit and implicit diﬀerencing schemes, and a robust outlier mechanism.

The diﬀusion ﬁlter is anisotropic in design and the results obtained are com-

parable to standard diﬀusion algorithms.

An improved vortex detection method is also detailed in Chapter 5. Vor-

tex detection is an important concept, since this enables engineers and re-

searchers to identify weak structures or potentially damaging natural ele-

ments. For structures where a smooth stream ﬂow is needed, perturbations

of the ﬁeld is kept to a minimum to avoid breakdown of the overall ﬂow.

Hence, the dynamics of a vortex is of immense interest. Though there are

numerous measurements [100, 102, 136] available to identify the vortex struc-

tures, we propose a computationally eﬀective and robust method to identify

vortices in image sequences showing movement of turbulent ﬂuids.

Although the proposed algorithm is fast and eﬃcient, the resulting smoo-

thed optical ﬂows from the experiments are still fairly coarse when compared

to specialised anisotropic diﬀusion ﬁlters.

6.2 Outlook

This thesis presents three diﬀerent contributions to the ﬁeld of computer

vision. As such, continuation of the research presented is vital for continued

improvements of methods and possible contributions to future applications,

and to the advancement of the ﬁeld. It is suggested that the Hessian based

155

algorithms can be improved and optimised further for better performance.

The methodology pertaining to 3D volume reconstruction can be im-

proved for more accurate representation. Currently, this method is for seg-

mented hard tissue based structures, although there has been some degree

of success reconstructing soft tissue structures. There is scope to conduct

extensive research to soft tissue based organs.

Using Navier-Stokes equations in computer vision is still a new concept.

This is especially true in the area of motion estimation. The results presented

in the thesis are preliminary results and further detailed investigations are

needed for mainstream research. Though the proposal in the thesis is a

diﬀusion ﬁlter methodology, it is much better to develop a new optical ﬂow

technique which is based on the Navier-Stokes equations with an integrated

regularisation mechanism. There is ongoing work on this technique [24] and

is based on the energy minimisation scheme of Horn and Schunk [2]. It

is anticipated that this technique is better suited to modelling the motion

of complex and turbulent scenes such as waterfalls, hurricanes and twisters

(from satellite imagery), solar phenomena, human motion analysis and cloth

dynamics.

It is thought that ﬂuid based optical ﬂows has many potential applications

in other ﬁelds, such as climatology, weather forecasting, gait analysis, facial

feature detection and computer graphics.

156

Appendix A

Dataset

The original Incisor dataset of the 22 slices is shown in Figure A.1. Interpo-

lated structural ﬂows are obtained from the sparse dataset with segmented

foreground. 2D intermediary slices between original slices are reconstructed

from the structural ﬂows. The 2D slices are then stacked and aligned to

reconstruct 3D volumes of the Incisor, as presented in Chapter 4.

157

Figure A.1: Incisor dataset.

158

List of References

[1] A. M. Tekalp. Digital video processing. Prentice Hall, Upper Saddle

River, NJ, 1995.

[2] B. K. P. Horn and B. G. Schunck. Determining optical ﬂow. Tech-

nical Report A.I. Memo No. 572, Artiﬁcial Intelligence Laboratory,

Massachusetts Institute of Technology, 1980.

[3] B. Lucas and T. Kanade. An iterative image registration technique

with an application to stereo vision. In International Joint Conference

on Artiﬁcial Intelligence, pages 674–679, Vancouver, Canada, 1981.

[4] P. Perona and J. Malik. Scale-space and edge detection using

anisotropic diﬀusion. IEEE Transactions on Pattern Analysis and Ma-

chine Intelligence, 12(7):629–639, July 1990.

[5] D. Tschumperl´ e and R. Deriche. Orthonormal vector sets regulariza-

tion with PDE’s and applications. International Journal of Computer

Vision, 50(3):237–252, 2002.

[6] D. Tschumperl´ e and R. Deriche. Vector-valued image regularization

with PDE’s: a common framework for diﬀerent applications. In IEEE

159

Computer Vision and Pattern Recognition, Madison, United States,

June 2003.

[7] M. J. Black, G. Sapiro, D. H. Marimont, and D. Heeger. Ro-

bust anisotropic diﬀusion. IEEE Transactions on Image Processing,

7(3):421–432, 1998.

[8] M. Bertalmio, L. Vese, G. Sapiro, and S. Osher. Simultaneous structure

and texture image inpainting. IEEE Transactions on Image Processing,

12(8):882–889, 2003.

[9] I. Giakoumis and I. Pitas. Digital restoration of painting cracks. In

Proceedings of IEEE International Symposium on Circuits and Sys-

tems, volume 4, pages 269–272, California, United States, 31 May–3

June 1998.

[10] A. G. Bors, E. R. Hancock, and R. C. Wilson. Terrain analysis using

radar shape-from-shading. IEEE Transactions on Pattern Analysis and

Machine Intelligence, 25(9):974–992, 2003.

[11] B. Horn and B. Schunck. Determining optical ﬂow. Artiﬁcial Intelli-

gence, 17:185–203, 1981.

[12] J. L. Barron, D. J. Fleet, and S. Beauchemin. Performance of optical

ﬂow techniques. International Journal of Computer Vision, 12(1):43–

77, 1994.

[13] A. Bab-Hadiashar and D. Suter. Robust optic ﬂow computation. In-

ternational Journal of Computer Vision, 29(1):59–77, 1998.

160

[14] A. G. Bors and I. Pitas. Optical ﬂow estimation and moving object

segmentation based on median radial basis function network. IEEE

Transactions on Image Processing, 7(5):693–702, May 1998.

[15] A. Verri and T. Poggio. Motion ﬁeld and optical ﬂow: qualitative

properties. Technical Report A.I. Memo No. 917, Artiﬁcial Intelli-

gence Laboratory and Center for Biological Information Processing,

Whitaker College, Massachusetts Institute of Technology, December

1986. C.B.I.P. Memo No. 22.

[16] A. G. Bors and I. Pitas. Prediction and tracking of moving objects in

image sequences. IEEE Transactions on Image Processing, 9(8):1441–

1445, August 2000.

[17] N. P. de la Blanca, J. M. Fuertes, M. Lucena, and A. Garrido. Local

motion estimation from stereo image sequences. In F. J. Perales et. al.,

editor, IbPRIA, LNCS 2652, pages 740–747. Springer-Verlag, 2003.

[18] C. E. Erdem, G. Z. Karabulut, E. Yanmaz, and E. Anarim. Motion

estimation in the frequency domain using fuzzy c-planes clustering.

IEEE Transactions on Image Processing, 10(12):1873–1879, December

2001.

[19] A. M. Peacock, D. Renshaw, and J. Hannah. A fuzzy data fusion

method for improved motion estimation. In Advanced Concepts for

Intelligent Vision Systems, Baden-Baden, Germany, August 1999.

161

[20] H. Foroosh. Adaptive estimation of motion using generalized cross

validation. In 3rd International (IEEE) Workshop on Statistical and

Computational Theories of Vision, 2003.

[21] M. Elad and A. Feuer. Recursive optical ﬂow estimation - adaptive

ﬁltering approach. In The 19th IEEE Conference in Israel, Jerusalem,

November 1996.

[22] H. Liu, R. Chellappa, and A. Rosenfeld. Accurate dense optical ﬂow es-

timation using adaptive structure tensors and parametric model. IEEE

Transactions on Image Processing, 12(10):1170–1180, October 2003.

[23]

´

E. M´emin and P. Perez. Hierarchical estimation and segmentation

of dense motion ﬁelds. International Journal of Computer Vision,

46(2):129–155, 2002.

[24] N. Papenberg, A. Bruhn, T. Brox, S. Didas, and A. Weickert. Highly

accurate optic ﬂow computation with theoretical justiﬁed warping. In-

ternational Journal of Computer Vision, 67(2):141–158, 2006.

[25] S. Roth and M. J. Black. On the spatial statistics of optical ﬂow. In

IEEE International Conference on Computer Vision, volume 1, pages

42–49, Beijing, China, 2005.

[26] A. Bruhn, J. Weickert, and C. Schnorr. Lucas/Kanade meets Horn/Sc-

hunck: combining local and global optic ﬂow methods. International

Journal of Computer Vision, 61(3):211–231, 2005.

[27] J. J. Koenderink. The structure of images. Biological Cybernetics,

50:363–370, 1984.

162

[28] D. V. Widder. The heat equation. Academic Press, 1975.

[29] S.-T. Yau. Surveys in diﬀerential geometry: diﬀerential geometry in-

spired by string theory, volume 5. International Press, 1999.

[30] R. A. Hummel, B. Kimia, and S. W. Zucker. Gaussian blur and the

heat equation: forward and inverse solutions. In IEEE International

Conference on Computer Vision and Pattern Recognition, pages 668–

671, 1985.

[31] R. A. Hummel, B. Kimia, and S. W. Zucker. Deblurring gaussian blur.

Computer Vision, Graphics, Image Processing, 38(1):66–80, 1987.

[32] G. Unal, H. Krim, and A. Yezzi. Stochastic diﬀerential equations and

geometric ﬂows. IEEE Transactions on Image Processing, 11(12):1405–

1416, 2002.

[33] F. Catte, P.-L. Lions, J.-M. Morel, and T. Coll. Image selective smooth-

ing and edge detection by nonlinear diﬀusion. SIAM Journal on Nu-

merical Analysis, 29(1):182–193, 1992.

[34] J. Weickert, B. M. T. Romeny, and M. A. Viergever. Eﬃcient and

reliable schemes for nonlinear diﬀusion ﬁltering. IEEE Transactions

on Image Processing, 7(3):398–410, 1998.

[35] R. A. Hummel. Representation based on zero-crossings in scale space.

In IEEE International Conference on Computer Vision and Pattern

Recognition, June 1986.

[36] A. R. Gourlay. Implicit convolution. Image Vision Computing, 3:15–23,

1985.

163

[37] L.-D. Cai. Some notes on repeated averaging smoothing. In Proceedings

of the 4th International Conference on Pattern Recognition, pages 697–

605, London, UK, 1988. Springer-Verlag.

[38] L. Alvarez and L. Mazorra. Signal and image restoration using shock

ﬁlters and anisotropic diﬀusion. SIAM Journal on Numerical Analysis,

31(2):590–605, 1994.

[39] D. Zhao and B. Li. A new implementation of discrete multiscale ﬁlter-

ing. In IEEE International Conference on Image Processing, volume 1,

pages 383–385, Lausanne, Switzerland, September 1996.

[40] R. Deriche. Fast algorithms for low-level vision. IEEE Transactions on

Pattern Analysis and Machine Intelligence, 12(1):78–87, January 1990.

[41] R. Deriche. Recursively implementing the gaussian and its derivatives.

Technical Report 1893, INRIA, May 1993.

[42] M. Nielsen, L. Florack, and R. Deriche. Regularization, scale-space and

edge detection ﬁlters. Journal of Mathematical Imaging and Vision,

7(4), 1997.

[43] S. T. Acton, A. C. Bovik, and M. M. Crawford. Anisotropic diﬀusion

pyramids for image segmentation. In IEEE International Conference

on Image Processing, pages 478–482, 1994.

[44] J. Fr¨ ohlich and J. Weickert. Image processing using a wavelet algorithm

for nonlinear diﬀusion. Technical Report 104, Laboratory of Techno-

mathematics, University of Kaiserslautern, Kaiserslautern, Germany,

March 1994.

164

[45] B. L. Luck, K. D. Carlson, A. C. Bovik, and R. R. Richards-Kortum.

A image model and segmentation algorithm for reﬂectance confocal

images of In Vivo cervical tissue. IEEE Transactions on Image Pro-

cessing, 14(9):1265–1276, 2005.

[46] C. Wallraven, B. Caputo, and A. B. A. Graf. Recognition with lo-

cal features: the kernel recipe. In IEEE International Conference on

Computer Vision, volume 2, pages 257–264. IEEE Press, 2003.

[47] Y. Kanazawa and K. Kanatani. Do we really have to consider covari-

ance matrices for image feature points? Electronics and Communica-

tions in Japan part III - Fundamental Electronic Science, 86(1):1–10,

2003.

[48] P.-Y. Burgi. Motion estimation based on the direction of intensity

gradient. Image and Vision Computing, 22:637–653, 2004.

[49] A. Petrovic, E. O. Divorra, and P. Vandergheynst. Multiresolution

segmentation of natural images: from linear to non-linear scale-space

representations. IEEE Transactions on Image Processing, 2004.

[50] T. Preusser and M. Rumpf. Extracting motion velocities from 3D image

sequences. In SPIE Conference on Visualization and Data Analysis,

2003.

[51] R. Archibald, J. Hu, A. Gelb, and G. Farin. Improving the accuracy

of volumetric segmentation using pre-processing boundary detection

and image reconstruction. IEEE Transactions on Image Processing,

13(4):459–466, April 2004.

165

[52] K. I. Kim, M. O. Franz, and B. Sch¨ olkopf. Kernel hebbian algorithm for

single-frame super-resolution. In A. Leonardis and H. Bischof, editors,

Statistical Learning in Computer Vision, pages 135–149, 2004.

[53] S. Mika, B. Sch¨ olkopf, A. J. Smola, K.-R. M¨ uller, M. Scholz, and

G. R¨ atsch. Kernel PCA and de-noising in feature spaces. In M. S.

Kearns, S. A. Solla, and D. A. Cohn, editors, Advances in Neural In-

formation Processing Systems 11, pages 536–542. MIT Press, 1999.

[54] S. Mika, G. R¨ atsch, and K.-R. M¨ uller. A mathematical programming

approach to the kernel ﬁsher algorithm. In T. K. Leen, T. G. Dietterich,

and V. Tresp, editors, Advances in Neural Information Processing Sys-

tems 13, pages 591–597. MIT Press, 2001.

[55] A. G. Bors, L. Kechagias, and I. Pitas. Binary morphological shape-

based interpolation applied to 3D tooth reconstruction. IEEE Trans-

actions on Medical Imaging, 21(2):100–109, 2002.

[56] T.-Y. Lee and W.-H. Wang. Morphology based three dimensional in-

terpolation. IEEE Transactions on Medical Imaging, 19(7):711–721,

July 2000.

[57] D. H. Frakes et. al. Application of an adaptive control grid interpolation

technique to morphological vascular reconstruction. IEEE Transactions

on Biomedical Engineering, 50(2):197–206, February 2003.

[58] T. M. Lehmann et. al. Survey: interpolation methods in medical image

processing. IEEE Transactions on Medical Imaging, 18(11):1049–1075,

November 1999.

166

[59] G. J. Grevera and J. K. Udupa. An objective comparison of 3D im-

age interpolation methods. IEEE Transactions on Medical Imaging,

17(4):642–652, 1998.

[60] S. Schaller et. al. Spiral interpolation algorithm for multislice spiral CT

- part I: theory. IEEE Transactions on Medical Imaging, 19(9):822–834,

September 2000.

[61] T.-Y. Lee and C.-H. Lin. Feature-guided shape-based image inter-

polation. IEEE Transactions on Medical Imaging, 21(12):1479–1489,

December 2002.

[62] C. A. Johnson, J. Seidel, and A. Sofer. Interior-point methodology

for 3D PET reconstruction. IEEE Transactions on Medical Imaging,

19(4):271–285, April 2000.

[63] P. T. Fletcher, C. Lu, S. M. Pizer, and S. Joshi. Principal geodesic anal-

ysis for the study of nonlinear statistics of shape. IEEE Transactions

on Medical Imaging, 23(8):995–1005, August 2004.

[64] S. C. Joshi, S. M. Pizer, P. T. Fletcher, P. A. Yushkevich, A. Thall,

and J. S. Marron. Multi-scale deformable model segmentation and

statistical shape analysis using medial descriptions. IEEE Transactions

on Medical Imaging, 21(5):538–550, 2002.

[65] J. M. Sanches and J. S. Marques. Joint image registration and volume

reconstruction for 3D ultrasound. Pattern Recognition Letters, 24:791–

800, 2003.

167

[66] A. Pitiot, G. Malandain, E. Bardinet, and P. M. Thompson. Piece-

wise aﬃne registration of biological images. In J. C. Gee, J. B. A.

Maintz, and M. W. Vannier, editors, Second International Workshop

on Biomedical Image Registration, LNCS 2717, pages 91–101. Springer-

Verlag, 2003.

[67] Y.-P. Wang, Y. Chen, and A. A. Amini. Fast LV motion estimation us-

ing subspace approximation techniques. IEEE Transactions on Medical

Imaging, 20(6):499–513, June 2001.

[68] L. Dougherty et. al. Validation of an optical ﬂow method for tag

displacement estimation. IEEE Transactions on Medical Imaging,

18(4):359–363, April 1999.

[69] G. P. Penney et. al. Registration-based interpolation. IEEE Transac-

tions on Medical Imaging, 23(7):922–926, July 2004.

[70] W. Higgins, C. Morice, and E. Ritman. Shape-based interpolation of

tree-like structures in three-dimensional images. IEEE Transactions

on Medical Imaging, 12(3):439–450, 1993.

[71] G. Turk and J. O’Brien. Shape transformation using variational im-

plicit functions. In SIGGRAPH, pages 335–342, August 1999.

[72] N. Weng, Y.-H. Yang, and R. Pierson. Three-dimensional surface re-

construction using optical ﬂow for medical imaging. IEEE Transactions

on Medical Imaging, 16(5):630–641, 1997.

168

[73] N. Hata et. al. Three-dimensional optical ﬂow method for measure-

ment of volumetric brain deformation from intraoperative MR images.

Journal of Computer Assisted Tomography, 24:531–538, July 2000.

[74] M. D. Abramoﬀ and M. A. Viergever. Computation and visualization

of three-dimensional soft tissue motion in the orbit. IEEE Transactions

on Medical Imaging, 21(4):296–304, April 2002.

[75] L. Weruaga, J. Morales-S´anchez, L. Nunez, and R. Verd´ u. Volumetric

motion estimation in human thorax for radiotherapy planning. IEEE

Transactions on Medical Imaging, 22(6):766–772, 2003.

[76] N. Ray and S. T. Acton. Motion gradient vector ﬂow: an external force

for tracking rolling leukocytes with shape and size constrained active

contours. IEEE Transactions on Medical Imaging, 23(12):1466–1478,

2004.

[77] D. Rueckert, L. I. Sonoda, C. Hayes, D. L. G. Hill, M. O. Leach, and

D. J. Hawkes. Non-rigid registration using free-form deformations: ap-

plication to breast MR images. IEEE Transactions on Medical Imaging,

18(8):712–721, 1999.

[78] A. Rao, R. Chandrashekara, G. I. Sanchez-Ortiz, R. Mohiaddin, P. Al-

jabar, J. V. Hajnal, B. K. Puri, and D. Rueckert. Spatial transforma-

tion of motion and deformation ﬁelds using nonrigid registration. IEEE

Transactions on Medical Imaging, 23(9):1065–1076, 2004.

169

[79] Y. Cao, M. I. Miller, R. L. Winslow, and L. Younes. Large deformation

diﬀeomorphic metric mapping of vector ﬁelds. IEEE Transactions on

Medical Imaging, 24(9):1216–1230, 2005.

[80] N. G. Sharp and E. R. Hancock. Density propagation for surface track-

ing. Pattern Recognition Letters, 19(2):177–188, 1998.

[81] M. Garza-Jinich, P. Meer, and V. Medina. Robust retrieval of 3D struc-

tures from image stacks. Medical Image Analysis, 3(1):21–35, 1999.

[82] A. Sarti, K. Mikula, and F. Sgallari. Nonlinear multiscale analysis of 3D

echocardiographic sequences. IEEE Transactions on Medical Imaging,

18:453–466, 1999.

[83] D. H. Chung and G. Sapiro. Segmenting skin lesions with partial dif-

ferential equations based image processing algorithms. In IEEE Inter-

national Conference on Image Processing, 2000.

[84] K. Krissian. Flux-based anisotropic diﬀusion applied to enhancement of

3D angiogram. IEEE Transactions on Medical Imaging, 21(11):1440–

1442, 2002.

[85] K. Z. Abd-Elmoniem, A.-B. M. Youssef, and Y. M. Kadah. Real-time

speckle reduction and coherence enhancement in ultrasound imaging

via nonlinear anisotropic diﬀusion. IEEE Transactions on Biomedical

Engineering, 49(9):997–1014, 2002.

[86] J. Ling and A. C. Bovik. Smoothing low SNR molecular images via

anisotropic median-diﬀusion. IEEE Transactions on Medical Imaging,

21(4):377–384, 2002.

170

[87] M. K. Chung. Heat kernel smoothing and its application to cortical

manifolds. Technical Report TR 1090, Department of Statististics,

Universisty of Wisconsin-Madison, USA, 2004.

[88] J. M. Fitzpatrick. A method for calculating velocity in time depen-

dent images based on the continuity equation. In IEEE International

Conference on Computer Vision and Pattern Recognition, pages 78–81,

1985.

[89] S. M. Song and R. M. Leahy. Computation of 3D velocity ﬁelds from

3D cine CT images of a human heart. IEEE Transactions on Medical

Imaging, 10(1):295–306, 1991.

[90] A. J. Chorin. Numerical solution of the navier-stokes equations. Math-

ematics of Computation, 22(104):745–762, October 1968.

[91] J. D. Anderson Jr. Computational ﬂuid dynamics: the basics with

applications. McGraw Hill, 1995.

[92] Y. Nakajima, H. Inomata, H. Nogawa, Y. Sato, S. Tamura, K. Okazaki,

and S. Torii. Physics-based ﬂow estimation of ﬂuids. Pattern Recogni-

tion, 36:1203–1212, 2003.

[93] M. Bertalmio, A. L. Bertozzi, and G. Sapiro. Navier-stokes, ﬂuid dy-

namics, and image and video inpainting. In IEEE Conference on Com-

puter Vision and Pattern Recognition, volume 1, pages 355–362, 2001.

[94] T. Corpetti,

´

E. M´ emin, and P. P´ erez. Dense estimation of ﬂuid

ﬂows. IEEE Transactions on Pattern Analysis and Machine Intelli-

gence, 24(3):365–380, March 2002.

171

[95] T. Kohlberger, C. Schnorr, A. Bruhn, and J. Weickert. Domain decom-

position for variational optical-ﬂow computation. IEEE Transactions

on Image Processing, 14(8):1125–1137, August 2005.

[96] S. Premoze, T. Tasdizen, J. Bigler, A. Lefohn, and R. T. Whitaker.

Particle-based simulation of ﬂuids. In EUROGRAPHICS, Computer

Graphics Forum 22(3), pages 401–410, 2003.

[97] J. Stam. A simple ﬂuid solver based on the FFT. Journal of Graphics

Tools, 6(2):43–52, 2001.

[98] J. Stam. Real-time ﬂuid dynamics for games. In Game Developer

Conference, volume 4, pages 76–92, March 2003.

[99] A. Treuille, A. McNamara, Z. Popovic, and J. Stam. Keyframe control

of smoke simulations. ACM Transactions On Graphics, 22(3):716–723,

July 2003.

[100] J. Jeong and F. Hussain. On the identiﬁcation of a vortex. Journal of

Fluid Mechanics, 285:69–94, 1995.

[101] J. C. Hunt, A. A. Wray, and P. Moin. Eddies, stream, and convergence

zones in turbulent ﬂows. Technical Report CTR-S88, NASA Ames,

Stanford University, USA, 1988.

[102] G. Haller. An objective deﬁnition of a vortex. Journal of Fluid Me-

chanics, 525:1–26, 2005.

[103] J. Zhong, T. S. Huang, and R. J. Adrian. Extracting 3D vortices

in turbulent ﬂuid ﬂow. IEEE Transactions on Pattern Analysis and

Machine Intelligence, 20(2):193–199, 1998.

172

[104] R. M. Ford and R. N. Strickland. Representing and visualizing ﬂuid ﬂow

images and velocimetry data by nonlinear dynamical systems. Graph-

ical Models and Image Processing, 57(6):462–482, 1995.

[105] R. S. Laramee, H. Hauser, L. Zhao, and F. H. Post. Topology-based

ﬂow visualization, the state of the art. In H. Hauser, H. Hagen, and

H. Theisel, editors, Topology-Based Methods in Visualization, Visual-

ization and Mathematics, pages 1–19. Springer-Verlag, 2007.

[106] R. Peikert and M. Roth. The parallel vectors operator - a vector ﬁeld

visualization primitive. IEEE Visualization, pages 263–270, October

1999.

[107] D. Bauer and R. Peikert. Vortex tracking in scale-space. In Joint

Eurographics — IEEE TCVG Symposium on Visualization, pages 140–

147, May 2002.

[108] F. Sadlo, R. Peikert, and E. Parkinson. Vorticity based ﬂow analysis

and visualization for pelton turbine design optimization, October 2004.

[109] F. Sadlo, R. Peikert, and M. Sick. Visualization tools for vorticity

transport analysis in incompressible ﬂow. IEEE Transactions on Visu-

alization and Computer Graphics, 12(5):949–956, 2006.

[110] J. Sahner et. al. Vortex and strain skeletons in eulerian and lagrangian

frames. IEEE Transactions on Visualization and Computer Graphics,

13(5):980–990, 2007.

[111] A. Cuzol and

´

E. M´emin. Vortex and source particles for ﬂuid motion

estimation. In Scale-Space, volume 3459, pages 254–266, 2005.

173

[112] L. Zhou, C. Kambhamettu, D. B. Goldgof, K. Palaniappan, and A. F.

Hasler. Tracking nonrigid motion and structure from 2D satellite cloud

images without correspondences. IEEE Transactions on Pattern Anal-

ysis and Machine Intelligence, 23(11):1330–1336, 2001.

[113] M. Jiang, R. Machiraju, and D. Thompson. Detection and visualiza-

tion of vortices. In C. Johnson and C. Hansen, editors, Visualization

Handbook, pages 295–309. Academic Press, 2005.

[114] J. Laﬀerty and G. Lebanon. Diﬀusion kernels on statistical mani-

folds. Technical Report CMU-CS-04-101, School of Computer Science,

Carnegie Mellon University, Pittsburgh, PA, United States, January

2004.

[115] I. Pitas and A. N. Venetsaonopoulos. Nonlinear digital ﬁlters: princi-

ples and applications. Kluwer Academic, Norwell, MA, 1990.

[116] A. G. Bors and I. Pitas. Object classiﬁcation in 3D images using alpha-

trimmed mean radial basis function network. IEEE Transactions on

Image Processing, 8(12):1730–1744, December 1999.

[117] A. Doshi and A. G. Bors. Robustiﬁed heat kernel smoothing of struc-

tural ﬂows for volumetric image interpolation. In IEEE International

Conference on Image Processing, pages 1225–1228, Atlanta, GA, USA,

2006.

[118] G. James. Modern engineering mathematics. Prentince Hall, third

edition, 2001.

174

[119] G. J. Grevera and J. K. Udupa. Shape-based interpolation of multidi-

mensional grey-level images. IEEE Transactions on Medical Imaging,

15:881–892, December 1996.

[120] A. Doshi and A. G. Bors. Optical ﬂow diﬀusion with robustiﬁed ker-

nels. In LNCS 3691, International Conference on Computer Analysis

of Images and Patterns, pages 213–221, Versailles, France, 2005.

[121] I. Mikic, S. Krucinski, and J. D. Thomas. Segmentation and track-

ing in echocardiographic sequences: active contours guided by optical

ﬂow estimates. IEEE Transactions on Medical Imaging, 17(2):274–284,

1998.

[122] Z. Xie and G. E. Farin. Image registration using hierarchical b-splines.

IEEE Transactions on Visualisation and Computer Graphics, 10(1):85–

94, 2004.

[123] D. Zoccolan and V. Torre. Using optical ﬂow to characterize sensory-

motor interactions in a segment of the medicinal leech. The Journal of

Neuroscience, 22(6):2283–2298, March 2002.

[124] P. L. Worthington and E. R. Hancock. New constraints on data close-

ness and needle map consistency for shape-from-shading. IEEE Trans-

actions on Pattern Analysis and Machine Intelligence, 21(12):1250–

1267, 1999.

[125] Y. Sato et. al. Three-dimensional multi-scale line ﬁlter for segmentation

and visualization of curvilinear structures in medical images. Medical

Image Analysis, 2(2):143–168, 1998.

175

[126] A. P. Huttenlocher and W. J. Rucklidge. A multi-resolution technique

for comparing images using hausdorﬀ distance. Technical Report TR

92-1321, Department of Computer Science, Cornell University, 1992.

[127] J. Stam. Stable ﬂuids. In SIGGRAPH, pages 121–128, August 1999.

[128] N. Foster and D. Metaxas. Modeling the motion of a hot, turbulent

gas. Computer Graphics, 31(Annual Conference Series):181–188, 1997.

[129] B. Galanti, J. D. Gibbon, and M. Heritage. Vorticity alignment results

for the three-dimensional euler and navier–stokes equations. Nonlin-

earity, 10:1675–1694, 1997.

[130] J. R. Cebral, M. Hern´andez, and A. F. Frangi. Computational analysis

of blood ﬂow dynamics in cerebral aneurysms from CT and 3D rota-

tional angiography image data. In International Congress on Compu-

tational Bioengineering, Zaragoza, Spain, 2003.

[131] S. Turek, M. Sch¨afer, and R. Rannacher. Evaluation of a CFD bench-

mark for laminar ﬂows. In H. G. Bock, Y. A. Kuznetsov, R. Glowinski,

J. Periaux, and R. Rannacher, editors, Proceedings of the 2nd Euro-

pean Conference on Numerical Mathematics and Advanced Applica-

tions, pages 549–563. World Science Publishing, 1998.

[132] S. Turek. A generalized mean intensity approach for the numerical

solution of the radiative transfer equation. Computing, 54(1):27–38,

1995.

176

[133] A. Doshi and A. G. Bors. Robust diﬀusion kernels for optical ﬂow

smoothing. In IEEE Workshop on Machine Learning for Signal Pro-

cessing, pages 415–420, Maynooth, Ireland, 2006.

[134] A. J. Majda and A. L. Bertozzi. Vorticity and Incompressible Flow.

Cambridge University Press, December 2001.

[135] A. Agrawal and A. K. Prasad. Organizational modes of large-scale

vortices in an axisymmetric turbulent jet. Flow, Turbulence and Com-

bustion, 68(4):359–377, 2002.

[136] R. M. Ford. Critical point detection in ﬂuid ﬂow images using dynam-

ical system properties. Pattern Recognition, 30(12):1991–2000, 1997.

[137] S. Popinet. Gerris: a tree-based adaptive solver for the incompress-

ible euler equations in complex geometries. Journal for Computational

Physics, 190(2):572–600, 2003.

177

Copyright c by Ashish Doshi, 2008. All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage or retrieval system, without permission in writing from the author.

To. Savita .

.

The uniqueness of the proposed algorithm is that it combines the eﬀectiveness of the explicit and the speed of the implicit ﬁnite diﬀerencing schemes. Statistically robust methods such as the median and the alpha-trimmed mean are integrated within the algorithm. is carried out to verify its suitability for motion smoothing of realworld complex and turbulent image sequences and deformation of moving objects over time. the eﬀects of smoothing are visualised in reconstructed 3D volumes. An experimental analysis of the model which uses the Navier-Stokes equation. The performance of the model is compared to that of i . Although diﬀusion is primarily applied to images. The proposed algorithm is kernel based derived from the heat equation.Abstract Diﬀusion methods are fundamentally important to reduce the eﬀects of noise whilst enhancing edges and the general perception of images. As a result. In this thesis. Structural ﬂows are obtained from real-world medical data for the purpose of 3D volumetric interpolation from 2D sparse data. given an initial noisy ﬂow. The technique is evaluated on synthetic data as well as vector ﬁelds extracted from real-world image sequences. In addition to the proposed robust diﬀusion methods. Further study on the eﬀects of diﬀusion on structural ﬂows is conducted. a computational ﬂuid dynamics based nonlinear model for optical ﬂow smoothing is introduced. applying such methods on vectorial ﬁelds is an important and challenging problem under the conditions of uncertainty and corruption which exist in most vectorial ﬁeld estimation problems. a novel anisotropic diﬀusion method is developed which is based on the dynamics of the local Hessian. The result is a smoothed optical ﬂow with reduced outliers.

the robust Hessian based diﬀusion kernels developed earlier are extended to the nonlinear model for fast. Experiments show that it is possible to extract features and model the dynamics of vortices blindly. Our techniques are used to reliably estimate the presence of vortices from artiﬁcially created von Karman vortex sheet and real-time satellite imagery. ii . The work provides the foundation for the development of a robust real-time identiﬁcation and tracking of storms. Finally. smooth and eﬃcient optical ﬂow smoothing. lacking the detailed knowledge of physical properties of the event captured in each frame.other alternatives reported in the literature.

. . . . . . . .1 Optical Flow . . . . . . . . . . vii viii ix x xv xvii xix 1 1 3 5 7 7 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Contents Declaration Acknowledgements List of Publications List of Figures List of Tables List of Abbreviations List of Symbols 1 Introduction 1. . . .3 Thesis Overview . . . 1. . . . . . . . . . . . .2 Motivation . . . . . . . . . 2 Literature Review 2. . . . .2 Diﬀusion Algorithms . . . 1. . . . . . . . . . . . . . . . . . . 11 iii .1 Optical Flow Estimation . . . . . . . . . . . . .

. 38 Alpha-trimmed mean kernel . . . . . . . .5 Conclusions . . . . . . . . . . . . 49 3.4 Diﬀusion kernel . . . . . . . . . . 35 3. . . .2 Introduction . . . . . . . .2 Optical Flow Estimation . . . . . . . . . 22 3 Robust Hessian-based Anisotropic Diﬀusion 24 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 Conclusions . . . . . . . . 36 3. . . . . . . .1 3. . 34 3D Hessian kernel . . . . .3 Volumetric Image Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4. . . .2.5. . . . .4 Navier-Stokes Equations and Vortex Identiﬁcation . . . . . . . . .1 Introduction .2 3. . . . . . 31 Multiple 2D Hessian kernels .5. . . . . . . . . . . . . . . 24 3. . . . . . . . . . . . . . . . . .3 3D Volumetric Interpolations . . . . . . . . . . . . . . . . . . . . .3 Outlier robustness study . . . .2 Smoothing noisy artiﬁcial vector ﬁelds . . . 43 Smoothing motion ﬁelds . . . . 67 4. . . . . . . . . . 29 3. . . .3. . . . . . .4. . 42 3.3 Anisotropic Diﬀusion of Optical Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Robust Hessian Diﬀusion Kernels . . . . . . . . . . . . 29 Embedded Hessian diﬀusion kernel . . . . . . . . . . . . . . . . 15 2. . . . . . . . . . . . . 69 iv .4. . . . . . .5 Experimental Results . . . . . 36 Median of directional Hessians kernel .2 3. . . . . . . . . . . . . . . . 40 3. .3. . . . . . . 26 3. . . . . . . . . . . . . . 19 2. . . .1 Research Objective .3. . .3.1 3. . . .3 3. . . . . . . . .1 3. . . . . . . . . . . . 68 4. . . . . 65 4 3D Volumetric Interpolation from Structural Flows 67 4. . . .

. .1 6. .7 Experimental Results . . . . . . . . . . . . . . . . 155 A Dataset 157 v . . . . . . . . . . . . . . .5 Smoothing Structural Flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Results on synthetic data . . . . . . 154 6. . . . . . . . . . . . . . . . . . 107 5 Robust Physics based Diﬀusion Solver 109 5. . . . . . . . . . .1 Summary of Contributions . .3 Synthetic data simulation set-up . . . . . . . . . . . . . . . . 112 5. . . . . . . . . .6 Slice Interpolation . . . . . . . . . . . . . . . .4 Structural Flow Initialisation . . . . . . . . . . . . . . . . . . . . . . 131 Real image sequences and remarks . . . .5 Vortex Core Detection . . . . . . . . . . .1. . . . . . . . . . . . . . . . . .3 Robust nonlinear diﬀusion . 142 5. . . . . . . . . . . . .2 6. . . . 153 Diﬀusion based on ﬂuid dynamics . 152 6. . . . . . . . . . . . . . . .3 The Stable Fluid Model . . .8 Conclusion . . .6 Experimental Results . . . .6. . . . . . . . . . . . . . . . . . 149 6 Conclusion and Outlook 152 6. . . . . . . . . . . . . . . . . . . . . . . . . .2 5. . . . . .1 5. 78 4. . . . . 116 5. . . . . . . . . .1 Research Objective . . . . . .6. . 81 4. . . . . . . . .4 Robust Hybrid Fluid Model . . . . .7 Conclusion . . . . .1. . . . . .1. . . . . .6. . . . . . . . . . . . . . . . . .4. . . . 121 5. . 152 3D volumetric slice interpolation . . . . . . . . . . . 110 5. . . .2 Introduction . . . . 126 5. . . . . . . . .2 Outlook . . 71 4. . . . . . . . . 77 4. . . . . . . . . 109 5. . . . . . .

List of References 159 vi .

except where otherwise stated. and for the title and summary to be made available to outside organisations. I hereby give consent for my thesis. Other sources are acknowledged by explicit references. A complete list of refereed publications can be found on page ix. Some of the material in the following chapters has been previously published by the author. vii . if accepted.Declaration This thesis has not previously been accepted in substance for any degree and is not being concurrently submitted in candidature for any degree other than Doctor of Philosophy at the University of York. This thesis is the result of my own investigations. to be made available for photocopying and for inter-library loan.

coupled with his dedication towards his students is invaluable both as a supervisor and as a friend. Bors. appreciation goes to my assessors. Edwin Hancock and Prof. Secondly. wholesome support and love.Acknowledgements First and foremost. His breadth of knowledge and experience. Nishan Canagarajah for their constructive feedback on this thesis and various aspects of my research. viii . of the computer vision group for their invaluable technical discussions and brainstorming sessions. including reports and presentations. I would like to thank my supervisor. Prof. My sincere thanks go to various members. Finally. Adrian G. past and present. to my family. advice and suggestions during my research and writing up. Dr. for their encouragement. for his continued support.

pp. 8-11 October 2006. Maynooth.List of Publications Conference Papers: • A. Doshi. A. China. Bors. United Kingdom. G. In Proceedings of IEEE International Conference on Image Processing. • A. • A. A. • A. 5-8 September 2005. Bors. Bors. 222-230. Warwick. Hong Kong. A. Ireland. G. In Proceedings of 11th International Conference on Computer Analysis of Images and Patterns. In Proceedings of IEEE International Workshop on Machine Learning for Signal Processing. G. France. ix . Doshi. Structural ﬂow smoothing for shape interpolation. Atlanta. 20-24 August 2006. Doshi. Versailles. In Proceedings of 18th International Conference on Pattern Recognition. 6-8 September 2006. Doshi. Robustiﬁed heat kernel smoothing of structural ﬂows for volumetric image interpolation. A. Robust diﬀusion kernels for optical ﬂow smoothing. In Proceedings of 18th British Machine Vision Conference. USA. Lecture Notes in Computer Science 3691. Optical ﬂow diﬀusion with robustiﬁed kernels. G. Navier-Stokes formulation for modelling turbulent optical ﬂow. A. 10-13 September 2007. Bors. • A. G. Doshi. Bors.

. . . .7 Examples of Synthetic-1 vector ﬁelds after corruption with Gaussian noise with σ 2 = 0. . . . . . .3 Smoothed artiﬁcial vector ﬁelds. . . . . . . . . . . . . . . . 49 3. . . . . 31 3. 46 3.List of Figures 1. . . . . . . . . . . .2 Artiﬁcial vector ﬁeld. . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3. . . . . . . . . .8 Artiﬁcial vector ﬁeld after corruption with Poisson noise with σ 2 = 0. . . .1 Top-level research model where I is the image frame and V represents the motion ﬁeld. . . . . . .1 Block matching algorithm. . original and after being corrupted with noise. . . . .25 after ﬁve iterations of smoothing. . . . 41 3. . . . . .6 Synthetic vector ﬁelds. . . . .5 Calculation of the median of directional Hessian kernels. . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 Frames from three image sequences and their corresponding optical ﬂows. . 4 3. . . . . 52 x . . 38 3. . . . .10 Frames from additional three image sequences and their corresponding optical ﬂows. . . . . . 33 3. . . . . . . . . . . . . . . . . . . . . . . . . . .4 The eﬀect of diﬀusion on outliers. 27 3. . . . . . . . . . 51 3. . . . . . . . .1 after ﬁve iterations of smoothing. . . .

. 75 xi . . . . . . . . . . . . . . .3 DBMA structural ﬂows. . .12 Predicted frame PSNR evaluation. .2 Block matching process in DBMA. . . . . . . . . . . . . . 70 4. . . 60 3. .11 PSNR convergence for the reconstructed frame 8 from Concorde sequence. .1 An illustration of intermediate slice reconstruction. . . . . . . . . . . . .4 Bad edge matching. 74 4. . . . using ATM-2DH smoothed vector ﬁelds when varying the alpha parameter for various window sizes. . initialised using LK and the resulting predicted 6th frame of the Concorde sequence.14 Smoothed optical ﬂows. . . . . . . 73 4. . . . . . . . . . . . . . . . . BlkRef ): Pseudocode of growing algorithm used for regions of object boundaries. . where nnz(·) is a function that ﬁnds the number of nonzero pixels in a block. . . . . . . . .15 Results when applying optical ﬂow smoothing for the image sequences considered in this study when the optical ﬂow has been initialised using BMA. . . . . .16 Results when applying optical ﬂow smoothing for the image sequences considered in this study when the optical ﬂow has been initialised using LK. . . . . 54 3. . . .13 Smoothed optical ﬂows. . . . . 59 3. . . . . . .5 GrowingRegion(BlkOrig. . . . 58 3. . . . . . . . . . 62 4. . 61 3. . . . . 55 3. . . . . . . . . . . initialised using BMA and the resulting predicted 5th frame of the Taxi sequence. . . . . . . . . .17 PSNR of the predicted frame when tracking scene change in two image sequences for the best ﬁve diﬀusion methods when the convergence criterion is set to τ < 10−1 . . . . . . . . . . . . . . . . . . . . .3. . . . 72 4. . . . . . . . . . . . . . . . . . . . . . . .

4.6 Visualisation of growing algorithm. . . . . . . . . . . . . . . . 76 4.7 The slice interpolation ﬂowchart. . . . . . . . . . . . . . . . . 79 4.8 Sample slices of the data sets. . . . . . . . . . . . . . . . . . . 82 4.9 Sample slices from the Incisor data set. . . . . . . . . . . . . . 84 4.10 Structural ﬂows and reconstructed slice of an Incisor. . . . . . 85 4.11 Further results for the Incisor. . . . . . . . . . . . . . . . . . . 86 4.12 3D Incisor reconstructions. . . . . . . . . . . . . . . . . . . . . 88 4.13 Accuracy of the middle slice reconstruction considering both shape structure and grey-level. . . . . . . . . . . . . . . . . . . 90 4.14 Example of Knee slices for experimentations. . . . . . . . . . . 93 4.15 Structural ﬂows and reconstructed slice of a Knee MRI. . . . . 94 4.16 Further results on structural ﬂows and reconstructed slices of a Knee MRI. . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

4.17 Original Humerus bone. . . . . . . . . . . . . . . . . . . . . . 97 4.18 3D Humerus bone reconstructed volumes. . . . . . . . . . . . 98

4.19 Original Iliac bone. . . . . . . . . . . . . . . . . . . . . . . . . 99 4.20 3D Iliac bone reconstructed volumes when skipping 5 consecutive slices between the remaining 2 slices. . . . . . . . . . . . 100 4.21 Results on structural ﬂows and reconstructed slices of a Female Chest MRI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 4.22 Results on structural ﬂows and reconstructed Sheep’s Heart MRI slices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

xii

4.23 Reconstruction of bone contours for Humerus and Iliac bones. From top to bottom, the contours are from smoothed reconstructions using original slices (blue), DBMA (red), PeronaMalik (green), Black (magenta), 2DH (yellow), ATM-2DH (cyan) and MED-2DH (black). . . . . . . . . . . . . . . . . . . 108 5.1 The stable ﬂuid solver algorithm. . . . . . . . . . . . . . . . . 114 5.2 Robust hybrid solver. . . . . . . . . . . . . . . . . . . . . . . . 116 5.3 Helmholtz-Hodge decomposition of a closed lid driven cavity laminar ﬂow. . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 5.4 Evaluation of the Qw -criterion from a 5−by −5 window, where the location of each vector in the window is shown with a diﬀerent marker according to its corresponding Qi set. . . . . . 126 5.5 Driven-lid cavity ﬂow diagram. . . . . . . . . . . . . . . . . . 127 5.6 Synthetic closed driven-lid cavity ﬂows with noise. . . . . . . . 129 5.7 Half-cylinder model for von Karman ﬂow. . . . . . . . . . . . 130 5.8 Closed cavity vector ﬁeld smoothing comparisons. . . . . . . . 132 5.9 Representing von Karman ﬂows. . . . . . . . . . . . . . . . . . 134 5.10 Smoothing noisy von Karman ﬂows. . . . . . . . . . . . . . . 135

5.11 Evaluating Q and Qw on the von Karman ﬂow using equations (5.14) and (5.18), respectively. . . . . . . . . . . . . . . . . . . 137 5.12 MSE after smoothing the noisy von Karman ﬂow. . . . . . . . 138 5.13 Performance of the vorticity segmentation using ∆w . . . . . . 139 5.14 Evaluating ∆ and ∆w on a smoothed von Karman ﬂow from equations (5.17) and (5.19), respectively. . . . . . . . . . . . . 141

xiii

5.15 Smoothing optical ﬂows in image sequences displaying turbulent motion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 5.16 Finding vortices in the Superstorm Andrea sequence. . . . . . 145 5.17 Additional results of vortex segmentation of Superstorm Andrea sequence. . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 5.18 Smoothing optical ﬂow corresponding to cloud movement for Superstorm Andrea. . . . . . . . . . . . . . . . . . . . . . . . 147

5.19 Finding vortices in the Solar Flare sequence. . . . . . . . . . . 149 A.1 Incisor dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . 158

xiv

List of Tables

3.1 Numerical results for Synthetic-1 data after one iteration of diﬀusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.2 Numerical results for Synthetic-2 data after one iteration of diﬀusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.3 PSNR (dB) of the reconstructed frame using the smoothed optical ﬂow with the number of iterations (No) necessary to reach convergence for each method and in the case of each image sequence when the motion ﬁeld is initialised using BMA. 63 3.4 PSNR (dB) of the reconstructed frame using the smoothed optical ﬂow with the number of iterations (No) necessary to reach convergence for each method and in the case of each image sequence when the motion ﬁeld is initialised using LK. . 64 4.1 Summary of the slice dimensions and voxel sizes for the different data sets. . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.2 Average percentage of reconstruction errors with DBMA as the initialisation. . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.3 Average peak signal-to-noise ratio of slice reconstructions with DBMA as the initialisation. . . . . . . . . . . . . . . . . . . . 104

xv

.10. . 132 5. . .6 Hausdorﬀ distance for Iliac bone. . . . . .2 Evaluation of smoothed noisy von Karman ﬂows when the noise variance is 0. . . . . . . . . . . . . 105 4.5 Hausdorﬀ distance for Humerus bone. .1 Mean cosine error of smoothed vector ﬁelds. . 138 xvi . . . . . . . . . . . . . . . . .4. 106 5. . . . . . . .4 Average PSNR of original middle slice reconstructions after removing 5 intermediate slices with LK as the initialisation. . . . . . . . . . . . . . . . . 105 4. . .

al.List of Abbreviations 1D 2D 3D ATM-2DH ATM-3DH ATM-M2DH One Dimensional Two Dimensional Three Dimensional Alpha-Trimmed Mean 2D Hessian kernel Alpha-Trimmed Mean 3D Hessian kernel Alpha-Trimmed Mean Multiple ﬁeld 2D Hessian kernel Black BMA CFD CT dB DBMA L.’s method Block Matching Algorithm Computational Fluid Dynamics Computer Tomography Decibel Dual directional Block Matching Algorithm Left Hand Side Lucas-Kanade’s algorithm Multiple ﬁeld 2D Hessian kernel Median of Absolute Deviation Mean Cosine Error Median of 2D Hessian kernel Median of 3D Hessian kernel Median of Multiple ﬁeld 2D Hessian kernel MedH-SFS Robust Hybrid Fluid Solver xvii . LK M2DH MAD MCE MED-2DH MED-3DH MED-M2DH Black et.S.H.

H.MSE MRI OFE OFCE PDE PIV PM PSNR R.S SFS SVD Mean Square Error Magnetic Resonance Image Optical Flow Equation Optical Flow Constrained Equation Partial Diﬀerential Equation Particle Image Velocimetry Perona-Malik method Peak Signal-to-Noise Ratio Right Hand Side Stable Fluid Solver Singular Value Decomposition xviii .

f(·) Functions π α θ σ2 G Pi Alpha trimmed mean trimming ratio Angle Variance Gaussian function xix . l M. v H i. y t Vx . j.List of Symbols I V x. k. u Vy . N ∆ Sx Sy E λ w η(·) ψ Still video frame Vector ﬁeld Eulerian coordinates Time x component of vector ﬁeld y component of vector ﬁeld Hessian Array indices Length of array Diﬀerence Search region in the x direction Search region in the y direction Energy Lagrangian multiplier Weights Windowed neighbourhood centred around · Scalar ﬁeld ˆ f (·).

exp L (·)T τ M GK I R d Σ L ω ∇ H ∇· P ρ ν ∇2 f n Ω P S Ω Poisson function Exponential function Total number of vectors Vector or matrix transposed Convergence threshold Manifold space Heat kernel Real space Normalisation coeﬃcient Covariance matrix Laplacian matrix Vorticity function Gradient operator Hausdorﬀ distance Divergence operator Pressure Density Viscosity Laplace operator Force Surface normal Bound domain Projection operator Rate of strain tensor Vorticity tensor xx .GP e.

∆ I(·) Ψ ∇× Discriminant Indexing function Stream function Curl operator xxi .

Of particular interest has been the participation of the general public in these developments as aﬀordable computers and the incredible explosion of the World Wide Web has brought a ﬂood of instant information to an increasing percentage of homes and businesses. Representative applications of motion analysis include video coding [1]. there has been a tremendous technological breakthrough in the areas of digital computation and telecommunications.1 Optical Flow Analysing motion patterns is essential for understanding visual surroundings.Chapter 1 Introduction In modern times. maintaining and even improving the visual integrity of this information is of great interest. The rate at which information is transmitted. stored. Most of this information is designed for visual consumption in the form of text. 1 . processed and displayed in a digital visual format is increasing rapidly and thus. 1. the design of engineering methods for eﬃciently transmitting. graphics and pictures or integrated multimedia presentations.

Typically. The method employed can be either deterministic or stochastic.robotic vision. unconstrained pixel intensity and lack of local contrast variation can lead to erroneous optical ﬂow estimations. local or global. enhancement. Typically. From the point of view of image analysis. The BMA estimates the vector ﬁeld based on the correlation between each pixel microblock in one frame and the corresponding microblock within a macroblock (search region) in the subsequent frame. optical ﬂow equations are widely used to estimate motion between consecutive frames. The most widely used method for motion estimation is the block matching algorithm (BMA) due to its simple implementation scheme. In order to overcome such problems. it is assumed that all pixel intensities are locally translated from one frame to the next and that the shifted values are preserved. regularisation terms are introduced. This is also true for LK algorithms which use weighted windowed Gaussian kernels. most BMA’s use either windowed median operators or other statistical cost measures to obtain smooth vectors. This assumption is the basis of the optical ﬂow constraint equation [2]. the performance of such methods when processing 2D 2 . The focus of this research is the use of anisotropic diﬀusion for accurate estimation of the velocity ﬁelds initialised by BMA and Lucas-Kanade’s (LK) gradient based algorithm [3]. Consequently. This constraint implies that the intensity of a moving pixel in the image plane remains constant along the trajectory of that pixel in time. However. super-resolution reconstruction. hence the wide variations of BMA available within the research community. etc. However. when estimating motion. the optical ﬂow estimation algorithms can be classiﬁed as gradient-based and feature-based methods.

The robustness. A lot of this research is primarily applied for diﬀusing and segmenting colour images and image inpainting [6. to varying degrees. results can be skewed if data is uneven. hence the interest in nonlinear schemes such as partial diﬀerential equations (PDE). embedded with stochastic processes form the core basis for the remainder of this thesis. Some examples of such methods are presented in [9. in this context. Notably. These methods. This is usually regarded as the worst-case scenario in statistics. These type of kernels use rank ordering statistics for outlier rejection.motion vectors depicting spatial-temporal object deformation is questionable. 8]. tolerate the presence of data points that do not obey the assumed model. Further details of the anisotropic diﬀusion algorithms developed and their novelty is provided in Chapter 3. 7. 10]. most of it has been oriented towards understanding mathematical properties of anisotropic diﬀusion [4] and modifying the diﬀusion equations for speciﬁc applications [5. 1.2 Motivation Robust methods for motion estimation. motion compensation and regularisation are very important in image processing. This can be deﬁned as a point at which the smallest fraction of outliers can cause the estimator to produce arbitrarily bad results. Robust statistical methods are adopted in computer vision to improve the performance of feature extraction algorithms at the bottom level of the vision hierarchy. These points are typically known as outliers. These are the pre-processing 3 . can be attributed to the breakdown point. 6]. Although a considerable amount of research has been conducted in this area. These algorithms.

techniques that make standard image processing methods work. To sum it up, when dealing with image sequences, motion is estimated through the computation of the image velocity vectors. Diﬀusion schemes (ﬁlter) are applied during vector ﬁeld computation to enhance the structure of moving objects whilst smoothing out unwanted information (mostly due to noisy artifacts). When a clean optical ﬂow is obtained, this information is used in image processing ﬁlters to compensate for motion for image enhancement, segmentation, tracking and classiﬁcation. This process is highlighted in Figure 1.1.

Figure 1.1: Top-level research model where I is the image frame and V represents the motion ﬁeld.

More recently though, signiﬁcant research has been done in developing image pre-processing techniques to obtain better feature extraction results. Hence, the main aim of this research is to develop image pre-processing tech4

niques, mainly anisotropic diﬀusion using nonlinear measures such as kernel based Gaussian function and ﬁnite diﬀerencing methods from ﬂuid mechanics for feature extraction of non-rigid objects which are deformable over time. The other purpose of this research is to develop a generalised nonlinear smoothing model which can either be used on its own or as a secondary ﬁlter for optical ﬂow processing.

1.3

Thesis Overview

The structure of the thesis is organised as follows: Chapter 2 surveys signiﬁcant contributions to the ﬁeld of optical ﬂow estimation and processing, isotropic and anisotropic diﬀusion algorithms, 3D volumetric reconstructions (mainly from medical imaging) and ﬂuid mechanics methods that have been implemented in computer vision and pattern matching. Chapter 3 of the thesis presents the novel contribution of nonlinear smoothing kernel which is based on the local Hessian information of the vector ﬁeld. Included in the chapter is the formulation of the diﬀerential smoother with statistical robustness. Also this chapter provides are experimental results and analysis of techniques based on the local heat kernel and Hessian matrix of the optical ﬂow. Further discussions on the comparative performances of the proposed algorithms against known methods in the ﬁeld is also covered. In Chapter 4, the focus of research is on the utilisation of structural ﬂows to reconstruct 3D volumetric models of human tissue structures. This chapter will show how structural ﬂows are obtained and further smooth5

ed using anisotropic diﬀusion methods developed in Chapter 2 to obtain intermediate slices and subsequently reconstruct 3D volumes. The proposed methodology is shown to obtain positive results which are comparable with other known implementations. Chapter 5 presents another novel diﬀusion algorithm which is based on the Navier-Stokes equation used in computational ﬂuid dynamics (CFD). The proposed algorithm is a hybrid model which is dependent on both the explicit and implicit diﬀerencing schemes. As in prior chapters, this chapter will also describe models used to model ﬂow ﬁelds and provide experimental results from both artiﬁcial and real-world sequence based vector ﬁelds. Additionally, there is also a section on vortex dynamics which covers the identiﬁcation and segmentation of vortex cores. The methods described here can be used for vortex extraction from satellite imagery. Finally, Chapter 6 summarises the contributions to the thesis, highlighting strengths and weaknesses as well as suggestions on future research directions.

6

**Chapter 2 Literature Review
**

The literature review survey conducted for the preparation of this thesis covers numerous disciplines such as mathematics, physics, engineering, bioscience, neuroimaging, medical imaging and computational ﬂuid dynamics. This chapter is divided into four sections, each describing major contributions to the speciﬁc area of interest for the purpose of this work. Other contributions, though not signiﬁcant for the current research, are nevertheless mentioned in the review.

2.1

Optical Flow Estimation

Optical ﬂow estimation from image sequences has been identiﬁed as an illposed problem [1] which requires a regularisation methodology as shown since early eighties by Horn and Schunck [11], Lucas and Kanade [3] as well as in the comparative study from [12, 13]. In their paper, Horn and Schunck proposed a method to calculate the moving ﬁeld (u and v components) at each point using image intensity diﬀerences. Their work involved some assump7

14]. he suggests that we can only determine the motion ﬂow 8 . Secondly. They also introduced a way of implementing the derivatives digitally. They argued that the latter’s algorithm does not take into account any rotational movement and that the optical ﬂow and motion ﬁeld are generally not identical concluding that two dimensional motion ﬁeld generally cannot represent three dimensional velocity ﬁeld unless special conditions are satisﬁed. They proposed averaging four ﬁnite diﬀerences (based on 2-by-2 cube representation) [2].tions such as the surface object is ﬂat to avoid shading variations. Quite often the estimated optical ﬂow is noisy and contains outliers. Firstly. There is no deﬁnite solution for occluded/unoccluded problem. they propose the use of scene radiance and image irradiance with the aid of the Lambertian model to compute the diﬀerence between the velocity ﬁeld and motion ﬁeld. Also. they introduce minimal optical ﬂow which relates to perceived motion in the image. however in order to determine the velocity ﬁeld. there exist two unknowns (u and v velocity components) for each observation. qualitative measurements are needed. This is even more evident in the case of the block matching algorithm used in video compression algorithms [1. However. assume initial pixel intensity to be uniform across the surface with no spatial discontinuities. They suggest that feature-based matching algorithms are reliable enough to accurately recover strong structures (segmentation) from motion. a paper presented by Verri and Poggio in 1986 [15] noted some disadvantages of Horn and Schunck’s proposal. And ﬁnally. Horn and Schunck’s algorithm was among the ﬁrst that gave better smoothing of the velocity ﬁeld resulting in better segmentation of the moving objects.

there are certain assumptions that need to be made with regard to the structure of the motion ﬁeld. Bayesian based methods. subsequent image sequence frame is predicted using prior information for tracking [14]. al. There are other methods that are being researched in the ﬁeld of motion estimation. tracking was employed for moving object prediction in video frame prediction. His method implied the use of median radial basis function (MBRF) network for segmentation. More recently though. • Six motion parameters to constrain local ﬂow vectors to lie on a speciﬁc line (quasi-parametric model). Bors [14.orthogonal to the spatial image gradient (normal ﬂow) at each pixel. P´rez et. but the MBRF network has to be retrained if an object has either entered or left a scene in the frame. Simultaneously. For example. In [16]. pixel-recursive based methods (extended to Wiener ﬁlter type motion estimation). 1 (Random Sample Consensus) Algorithm 9 . Due to this problem. The RANSAC algorithm is a robust ﬁtting model in the presence of data outliers. These are: • Some sort of smoothing constraints or uniformity has to be implemented on the motion ﬁeld (non-parametric model). The process is repeated to obtain better motion estimation and frame prediction. [17] proposed using the RANSAC1 e algorithm to estimate rigid motions through the image sequence. 16] used radial basis function (RBF) networks for motion estimation and moving object segmentation [14]. Tekalp [1] identiﬁes numerous motion ﬁeld estimators. phase correlation based method and block matching methods. Among them are methods based on the optical ﬂow equation (OFE).

Roth and Black [25] proposed using spatial statistics of vector ﬁelds to obtain better accuracy in the ﬂow estimation. [18] and Peacock et. al.They then use singular value decomposition technique to estimate the scale space representation of the local motion from the data. it is limited (as suggested in [25]) due to the fact the training set used is made up of static images. Recently. 10 . 23]. This function is used within the Combined-Local-Global (CLG) [26] framework. Other examples of motion estimation techniques are fuzzy based as outlined by Erdem et. all of which are robust and eﬃcient on simulated sequences yet untested on arbitrary video sequences. Energy based minimisation frameworks have been deﬁned for estimating the optical ﬂow by taking into account various determining factors [14. al. [22]. Altough their method performs reasonably well. al. 24. 25]. blind estimation using generalized cross validation technique as proposed by Foroosh [20] and adaptive based methods using recursive optical ﬂow estimation as proposed by Elad [21] or using tensor based models as proposed by Liu et. Their method relies on learning statistical information from a synthetic training set of optical ﬂows to obtain an appropriate statistical energy function. [19]. occlusion and movement into/out-of scene is not considered. tracking in image sequences and video coding quality [1] depend on smooth optical ﬂows. 23. the research in optical ﬂow computation has focused on stochastic approaches. Diﬀerent applications such as moving object segmentation [14.

al. 28. Hummel [35] oﬀers some very important insight on the relationship between Gaussian blurring and the heat equation. 29]. 31].2. al. The solution of the heat equation provides a kernel function which is adaptive to the local manifold [30. [7] by using a diﬀerent edge detection function in order to preserve sharper boundaries in images. this is a particular case of data and most often the variation in the local manifold should be taken into account by using an anisotropic kernel. However. Their work was extended by Black et. used the additive operator splitting (AOS) [34] for smoothing volumetric images by extending the diﬀusion approach from [33]. Hummel is able to demonstrate that there is a precise relationship between the standard deviation of the Gaussian and time t of the heat diﬀusion equation. If the manifold is constant in all directions. a caveat of the heat equation is that true locations of the boundaries in very noisy images makes the boundary preservation questionable since it may be impossible to determine the existence of the boundary. However. which 11 . 34]. 33.2 Diﬀusion Algorithms Regularisation techniques using partial diﬀerential equations (PDE) have found several applications in image processing and computer vision. then an isotropic kernel can be used [27]. 32. Weickert et. The theoretical background of these approaches arises from the methodology used for describing physical phenomena such as the dissipation of heat or the movement of ﬂuids [27. Anisotropic diﬀusion was employed for smoothing images while preserving edges by Perona and Malik [4]. PDE’s used in the diﬀusion context have been applied for image smoothing while preserving main features in images such as object contours and corners [7.

al. multiplicative splittings. but using a Gaussian kernel instead of gradient of image. multigrid methods [43]. A suitable model of the local manifold geometry is indicated by the Hes- 12 . In the nonlinear diﬀusion ﬁeld. have proved that AOS schemes are at least 10 times more eﬃcient than the widely used explicit schemes. numerical schemes with wavelets as trial functions and pseudo spectral methods [44]. The AOS scheme guarantees equal treatment of all coordinate axes. Impressive results on improved eﬃciency by recursive ﬁltering can be found in [40. Weickert et. one can ﬁnd several approaches which aim to be eﬃcient alternatives to the conventional two-level explicit ﬁnitediﬀerence scheme. [7] for smoothing medical confocal images. It can be implemented easily in arbitrary dimensions. for instance three-level methods.is similar to Perona and Malik [4]. al. Implicit splitting based approaches for linear diﬀusion ﬁltering have been proposed in [36] and [37] and also in [38] and [39] where their realisation for recursive ﬁlters is suggested. [45] combined the median ﬁlter and the diﬀusion operator of Black et. The above work has been inﬂuenced by a number of related approaches. has good rotational invariance and reveal a computational complexity and memory requirement which is linear in the number of pixels. semi-implicit approaches [33]. Hardware proposals for nonlinear diﬀusion ﬁltering can be found in the literature [4]. Luck et. 41] and the close relation between recursive ﬁlters and linear scale-space approaches has been clariﬁed in [42]. The adaptivity of the Gaussian kernel to the local manifold is given by its covariance matrix [46]. ﬁnite element techniques with adaptive mesh coarsening. al.

An interesting approach is implemented by Preusser and Rumpf in [50]. They accomplish this by generating a scale-space stack and building a hierarchical tree of the scales. Burgi [48] uses diﬀusion kernel proposed in [12] for the smoothing of ﬂow obtained through intensity gradient direction method. Tschumperl´ and Deriche [6] proposed PDE based e vector diﬀusion method. Linking is then performed between the scales and diﬀusion is applied to smooth and enhance the edges. Their method has been applied mainly on still greyscale and colour images. [49] have introduced an unsupervised segmentation technique to study the multi-scale structure of images.sian matrix calculated from the local data [5. their method uses tensor and Hessian matrices which are constructed from a still image or image sequence and is then integrated within an anisotropic diﬀusion kernel of Gaussian nature for smoothing optical ﬂow ﬁeld or image intensities. On a more recent note. On the other hand. Their method utilises two dimensional optical ﬂow computation to represent three dimensional velocity vectors. marginal me- 13 . In stark contrast to applying diﬀusion based smoothing. When the Hessian is embedded in certain functions. The Hessian matrix. 47]. Petrovic et. In a nutshell. it can be used to improve the object representation as estimated from local surface normals [10]. al. represents the multidimensional second order derivates and can be used to detect local maxima and minima as well as the transition areas from images. The advantage of this method is that the Gaussian kernel used is multivariate and tends to work better when there exists multiple sources of noise or presence of complex motion. The core of their method is the anisotropic geometric diﬀusion scheme which is driven by the structure of the image and its temporal acceleration. 6.

the Gaussian diﬀusion kernel is best to de-blur. This type of method can be useful to obtain good optical ﬂows provided the structure of the moving objects are strong and the neighborhood distribution has been selected appropriately. To summarise. it also simpliﬁes the complexity of the algorithms involved thus reducing the computational cost. However. However. Interesting ones include Weibull distribution based kernel [51]. the results can be skewed if the data is uneven. The methods described above have been proposed to work individually. There are numerous diﬀusion kernels being developed with various degree of complexity. principal component analysis (PCA) based kernel [52.dian is a simple statistical ordering method [9. It relies heavily on the distribution and amount of data to be considered in the kernel. By assuming the distribution is Gaussian. it is possible to extend them to image sequences albeit some degree of complexity may be involved. The reasoning behind the distribution choice is that on most occasion of motion blur. but our opinion is that the demonstration of succesful working of a combination of methods would be an interesting concept to consider. integrate Gaussian (or its variation) distribution within the kernel. 10]. it is assumed that a Gaussian point spread function is the cause of blurring and hence. there are other types of diﬀusion kernels being researched in the ﬁeld. The data is ranked and median of the data is selected. 53] and the Fisher criterion based kernel [54]. many of the methods reviewed have direct relevance to this research. 14 . Most of the diﬀusion kernel methods reviewed. Most of which is beyond the scope of this thesis. Although some methods have been applied to image stills.

hence the problem of data extraction from sparse information. Lee [56] aligns the slices ﬁrst using object centralisation before applying morphological operators. Schaller et. However. Even with such tools at a researcher’s disposal. Lee’s [56] process only interpolate non-matching regions between slices. Comparisons between various interpolation methods are presented in [58] and [59]. it is still improbable for the machines to take suﬃcient digitised images for full 3D reconstructions.2. Due to the nature of the medical applications. For example. al. 56] or interpolation methods [57. the task to reconstruct an accurate 3D model is diﬃcult from a group of sparse image slices. Similarly. thus producing image slices which are not equidistant between slices. Other examples of reconstruction methods in use include regularised maximum-likelihood 15 . [60] proposed using spiral interpolation based on azimuthal rebinning and Lee [61] proposes shape based interpolation using warping. Traditionally. medical image slices are obtained through mechanical slicing and digitisation. Bors [55] uses reciprocal binary morphology to obtain intermediary slices to reconstruct a 3D tooth model. 58.3 3D Volumetric Interpolations Recent advances in medical imaging highlights an increasing need for 3D models of human anatomy for medical diagnostics. 59. Most methods are either a variation of morphological operations [55. it is extremely important that an accurate 3D model of the anatomy is used. However. There has been considerable amount of research devoted to reconstructing accurately the 3D models of a human body part. Modern technology provides images of slices through magnetic resonance imaging (MRI) or computer tomography (CT). 61]. 60.

In contrast to gradient based methods. al. Weng et. al. Hata et. They regularised the ﬂows using a Gaussian kernel for better estimation. 69] and shape based methods [70. Furthermore. These images tend to resemble soft tissues like optic nerves within the human or animal brain. 68. they state that it is important to segment and then remove skin from the processing stage as it can inﬂuence the accuracy of the ﬂow estimations. [68] used optical ﬂows to model myocardiac displacements within the heart. al. [72] used optical ﬂows from [2] to model the surface projection from one ultrasound image slice to another. [75] proposed a parametric approach to estimate the volumetric motion of the human thorax. Bayesian framework based on alignment [64. Turk and O’Brien [71] considered using a variational approach for shape transformation and interpolation. This would then allow them to tag and track tissue deformation over time. Ray and Acton [76] proposed using the energy minimisation method of motion vector gradient vector ﬂow to capture and track cells in microscopic imaging. 59]. 65]. which enables surface warping from one object to another. 67. [73] used the gradient based optical ﬂow formulation to measure volumetric brain deformations from MR images. al. Weruaga et. 66.(ML) method [62]. statistically based principal component analysis (PCA) method [63]. Other than standard methods being researched. matching based methods [57. Dougherty et. Abr`moﬀ and Viergever a [74] proposed a variation of Lucas and Kanade’s [3] method to visualise the movement of soft tissue in 3D volumetric Orbit images. Their method is based on an implicit implementation and uses contours of objects as boundaries. Also of interest is the myocardial motion ﬁeld captured 16 . However.

79]. al. Aside from the main focus of 3D volumetric interpolation schemes in medical imaging. The linking is done by linear interpolation between matching correspondes in the slices and alignment is made appropriately. 17 . One such method is the work of Sharp and Hancock [80] which uses probabilistic relaxation model to correlate the contour displacements of MRI surfaces with the corresponding intensity features from successive frames to track its deformation. they proceed to use B-spline method as in [77] to perform matching between slices. Garza-Jinich et. it is important that initial neighbouring slices contain similar anatomical features. [77] used a nonrigid registration approach to model deformation in breast magnetic resonance images. Rueckert et. al. otherwise the technique is unreliable. Once the above criteria of slices having similar features are met. al. They apply aﬃne transformations with spline based free form deformation methodology to achieve some degree of success. The interpolated image is obtained by using the matching information and linking it with the interpolation plane between the two slices. realises that for a registration based technique. They also test their method on modelling brain deformations. [78]. [81] used an automatic segmentation algorithm to segment regions of interest from MR image slices. In [69]. al. Variations of the block matching algorithm have been used for registration in medical images as described in [69. Their proposal is robust and the segmentation regions are then stacked to produce a 3D volume for visualisation purposes.by applying a non-rigid registration technique [77] as presented by Rao et. Penny et. there are other kinds of research in this area that has some relation to the work carried out in the thesis.

Chung [87] suggested using the heat kernel for smoothing in neuroimaging. the favoured approach of anisotropic smoothing using the heat kernel. He uses this model to investigate cortical thickness of the brain between autistic and normal children. 18 . he is able to obtain a smooth image whereby the vascular structures of the liver could be segmented for future analysis. [82] implemented an improved Perona-Malik [4] diﬀusion algorithm with a multiscale method to give a better representation of a 3D volumetric echocardiographic sequence. al.Also of interest is the implementation of PDE based models in medical imaging. Their algorithm is anisotropic in design and is applied on ultrasound images of the heart. Hence. where smoothing only occurs uniformly within a common neighbourhood. Sarti et. Also anisotropic is the diﬀusion algorithm proposed by Ling and Bovik [86]. al. Chung and Sapiro [83] used a PDE based morphological algorithm to successfully segment skin legions from images. By applying such an algorithm. which have low signal-to-noise ratio. Krissian [84] used ﬂuid based ﬂux algorithm to anisotropically diﬀuse CT images of the liver. He acknowledges that it is not appropriate to use isotropic kernels to smooth soft tissue surfaces as it is in a curvlinear space. Abd-Elmoniem et. [85] suggested using a Hessian based diﬀusion kernel to smooth speckle images. Their algorithm utilises the median of Peronal-Malik’s [4] implementation to achieve very good smoothing of structures within molecular images.

al. Corpetti et. which diﬀers radically from that of rigid bodies. used the vorticity-stream formulation to recover dense motion of water vapours [94]. which can be attributed to the image intensity-Laplacian relationship. Using Fitzpatrick’s analysis as a basis. Although the method seems to yield good results. Bertalmio et. Song and Leahy [89].4 Navier-Stokes Equations and Vortex Identiﬁcation Very often. 91]. al. Their approach uses the vorticity-stream formulation of the ﬂuid ﬂow equation. The computation of ﬂows depends largely on the speciﬁc nature of the application. Nakajima et.2. al. applied the NavierStokes equations to image and video inpainting [93]. it was only tested on simulated sequences and their algorithms is similar to that of [89]. Navier-Stokes equations have been extensively studied in ﬂuid mechanics for modelling the behaviour of ﬂuids under various conditions and constraints [90. [92] minimised the overall cost function of the constraints to obtain a better estimation of the ﬂow for dye shading movements. employed the equation of continuity as an additional constraint to Horn and Schunck’s algorithm [11] in order to obtain better motion estimation of the beating heart. The use of ﬂuid ﬂow modelling for motion estimation can be traced back to the work of Fitzpatrick [88]. Classical optical ﬂow estimation algorithms would fail in such cases. 19 . The Navier-Stokes and optical ﬂow constraint equations have been employed for modelling von Karman ﬂows in [92]. the modelling of natural phenomena involves the motion of dynamic ﬂuids. who compared optical and ﬂuid ﬂow methods.

Jeong and Hussain [100] suggest using regularisation functions to obtain a uniform and smooth ﬁeld. This is because 20 . how is it possible to verify the results obtained? One possible way is to identify the presence of vortices or coherent structures [100] in a ﬂow ﬁeld. They have identiﬁed that the vorticity magnitude measurement. Unless there is large perturbations in the ﬁeld. This would fulﬁl the properties of Galilean-invariant ﬁeld for better detection [100]. the identiﬁcation of vortices in noisy ﬁelds with small perturbations is diﬃcult. It is very well to use Navier-Stokes equations to model and smooth velocity ﬂows. especially for large scale computations. One particular work was done by Kohlberger et. whereby a vortex could easily be detected. The boundaries have been processed as a set of constraints on a grid [98]. al. diﬀusion and mass conservation. The stable ﬂuid solver (SFS) algorithm implements NavierStokes equations and consists of a set of consecutive processing steps [97]. extensive research has been conducted in optical ﬂow construction using ﬂuid based algorithms. Navier-Stokes equations have been used in computer graphics for visualising ﬂames and building animation tools based on ﬂuid-like motion [96. they also introduced a parallelisation approach to achieve fast computations. by enforcing repetition and employing the Fast Fourier Transform (FFT) [97] or by using level sets [96]. |ω| is not a reliable way to detect vortices. As such. with high frequency components removed. 97. 99]. such as: advection.In recent years. The boundary conditions are important in constraining the ﬂuid motion [91]. 98. [95] which used variational domain decomposition method to obtain a fast converging optical ﬂow construction method.

Haller also presented a secondary criteria for hyperbolic (of saddle-type) structures around the vortex centre. In computer vision. They proposed using partition based method to segment regions of interest (converging and diverging focal points) using polynomial based nonlinear least squares estimation. namely by Laramee et. in three and two dimensions respectively. al. However. a lot of work has been conducted for vortex extraction. However. compared to [101] and [100] algorithms. 109]. [106. al. Additionally. 101] for vortex detection from 3D turbulent motion ﬁelds. They also argue the reliability of Hunt et. [105] and Peikert et. al. 108. they propose the λ2 -deﬁnition as an additional constraint for better vortex identiﬁcation. Another example is the work conducted by Ford and Strickland [104]. al. 107. Recently. Zhong et. with implicit ﬁnite diﬀerencing scheme to identify and track vortices.there are other physical states where the vorticity magnitude could not detect the presence of a vortex. Haller [102] proposed MZ and QS criterions for better representation of the vortex detection from multiple frames. Peikert et. Laramee et. [103] were among the ﬁrst to implement ﬂuid mechanics constraints [100. [106. Please refer to [100] for a detailed discussion on this. al. Instead. This is then used to recover critical ﬂuid structures from streamline imageries. they only managed to test the algorithms on artiﬁcially generated 2D and 3D PIV (particle image velocimetry) turbulent ﬁelds. [105] uses topology based visualisation to extract and track vortices. In contrast.’s Q-criterion measurement [101]. 107] suggest using vector parallelism and scale space technique. al. 21 . in computer graphics.

there is no review on the eﬀects of diﬀusion on such ﬂows and how this would eﬀect 3D volumetric reconstructions. Cuzol and Memin [111] using probabilistic energy minimisation method to extract vortex from phantom satellite imagery and Zhou et. [110] using the MZ -criterion [102].Other notable research in computer graphics and vision using ﬂuid methods for vortex extraction was conducted by Sahner et. they are mostly speciﬁc on still images and not on velocity ﬁelds. volumetric reconstructions and ﬂuid methods in vision. only represents a small percentage of research in this area. Although extending optical ﬂows to structural ﬂows pertaining to anatomical organ slices is not new. This is especially true in the area of motion estimation. the motivation to focus the thesis in nonlinear models for smoothing of optical ﬂows from complex and turbulent image sequences. al. with scenes of non-rigid objects. as it is a huge ﬁeld in itself. Though there are notable review publications on diﬀusion. it is increasingly diﬃcult to ﬁnd a niche in the ﬁeld to specialise. A good review of current methods being used in the ﬁeld for vortex extraction is presented in [113]. [112] used ﬂuid methods with local and global smoothness constraints to obtain scene structures for segmentation and tracking of clouds for cloud modelling and weather prediction. diﬀusion.5 Conclusions In computer vision. there is little work in the area of non-rigid optical ﬂow estimations and regularisation techniques. The literature survey reported in this chapter. 22 . 2. Hence. namely in optical ﬂow estimations. However. al.

It is the aim of the work described in this thesis to address some gaps in the ﬁeld. we have identiﬁed an area of optical ﬂow estimation which is still in its infancy. Using ﬂuid methods in computer graphics is quite common nowadays. Though in the thesis we use Navier-Stokes equations for regularisation purposes. However. but slowly gathering pace in its level of interest by vision researchers.Furthermore. 23 . there is signiﬁcant lack of work in optical ﬂows using Navier-Stokes equations. this is viewed as a stepping stone for an optimised optical ﬂow estimation using Navier-Stokes equations.

24. 25]. tracking in image sequences and video coding quality [1] depend on smooth optical ﬂows.1 Introduction Optical ﬂow estimation from image sequences has been identiﬁed as an illposed problem which requires a regularisation methodology as shown through the works of Horn and Schunck [11] as well as Lucas and Kanade [3]. This is more evident in the case of the block matching algorithm used in video compression algorithms [1. Energy based minimisation frameworks have been deﬁned for estimating the optical ﬂow by taking into account various determining factors [14. Quite often the estimated optical ﬂow is noisy and contains outliers. The proposed 24 .Chapter 3 Robust Hessian-based Anisotropic Diﬀusion 3. The aim of this chapter is to analyse the performance of a set of algorithms that perform robust diﬀusion on vectorial ﬁelds. 23]. 14]. Diﬀerent applications such as moving object segmentation [14. 23.

while Section 3. This chapter is organised as follows: Section 3. The proposed algorithms are used for smoothing artiﬁcial vector ﬁelds as well as optical ﬂows estimated from various image sequences.6 outlines the conclusion of the study.5 presents the comparative results for the proposed methodology. Section 3. 25 . The novelty in this section of the research is the introduction of Hessian based diﬀusion kernels which diffuses data anisotropically in the presence of outliers.3 outlines the application of anisotropic diﬀusion on vector ﬁelds.algorithms combine the smoothing ability of the heat kernel with the outlier rejection mechanisms of robust statistics algorithms. The robust diﬀusion process is extended to 3D lattices by using 3D Hessians. Section 3. Section 3.2 describes the estimation algorithms of optical ﬂows that have been used in the thesis. The diﬀusion kernel is Gaussian with the covariance matrix considered as the Hessian calculated from the data located in a certain neighbourhood.4 contains a study analysing the bias introduced when diﬀusing outliers and describes the proposed set of algorithms that combine robust statistics with Hessian based diﬀusion. classical diﬀusion introduces a bias in the whole data set. Robust statistics operators are shown to improve the results provided by Hessian-based diﬀusion by rejecting outliers and by enhancing the smoothing ability of diﬀusion algorithms. Alpha-trimmed mean and median statistics have been considered for robustifying the diﬀusion kernels. The kernel implicitly embeds the local changes in the optical ﬂow. They combine anisotropic diﬀusion and robust statistics for smoothing the optical ﬂow obtained when using simple motion estimation algorithms. Under these conditions.

respectively. 11. Extending the calculation for multiple frames. Vy ) denotes the motion vector at time t. ∆x represents a translation in space and ∆t ∂I ∂I and It represent ﬁrst . y) in frame t which is part of the image sequence I. t + ∆t) ≈ I(x. while ∇I = ∂x ∂y order partial spatial and temporal derivatives.1) where I(x. Motion vectors can be initially calculated by either using gradient methods (3.3. The methodology underlying the optical ﬂow estimation and smoothing is described in the following subsection. t) and after dividing equation (3. considering the ﬁrst derivative and neglecting higher order derivatives [3. where each plane of the lattice corresponds to the motion between two consecutive frames. 12]. it can be assumed that I(x + ∆x.2) where Vt = (Vx . If the displacements are small enough. These representation are chosen to be consistent with literature within the ﬁeld of motion estimation. t + ∆t) ≈ I(x. t) + ∇I · ∆x + It ∆t (3. 11] or feature matching.2 Optical Flow Estimation The optical ﬂow extracted from a video sequence is represented as a vector ﬁeld which warps one image into another providing its changes in time [3.1) with ∆t yields the constrained optical ﬂow equation: ∇I · Vt + It = 0 (3. t) represents an area located at x = (x. a vector ﬁeld on a 3D lattice is obtained. 11]: I(x + ∆x. represents the variation in time.2) [3. The optical ﬂow can be represented using a Taylor expansion of a frame from an image sequence with respect to the other frames. for example the block matching 26 .

j. However. Figure 3. particularly in areas with constant texture and colour. t + 1)| (3. 14] used in video coding. for each block of pixels a motion vector is determined representing the displacement between the coordinates of the two blocks from the two frames as given by Vt . Sy ). Consequently.1. where the gradient is constant in all directions and the correlation is identical for several pixel block 27 .algorithm [1. The basic block matching algorithm is shown in Figure 3. l) are assumed to be inside a predeﬁned search region (Sx .3) where M ×N is the block size and (k. j + l.1: Block matching algorithm.l min i=1 j=1 |I(i. motion estimation algorithms often lead to wrong decisions. The block matching algorithm relies on the correlation between blocks from one frame and blocks from a search region from another frame of the same image sequence as in M N Vt = argk. t) − I(i + k.

which minimises the combined global smoothness and the constrained intensity gradient as shown below. t)]2 28 (3.2). Iy are ﬁrst order spatial intensity derivatives with the Lagrangian multiplier. Moreover. the resulting optical ﬂow is noisy in nature. which implements regularisation in the precomputation stage of the optical ﬂow. Although equation (3. Horn and Schunck [2] used energy minimisation technique. Due to all these causes.combinations. additional constraints are needed to improve the eﬃciency of the solution. w to corresponding centered pixels which are locally inﬂuenced by the surrounding neighbourhood before using the gradient constraint equation (3. These errors in the optical ﬂow should be smoothed out or removed altogether. λ.2). t) · u + It (x. displaying both slow varying noise as well as sudden large changes. Another popular method of computation that is often used is Lucas and Kanade’s algorithm [3]. the image noise and changes in illumination conditions produce motion vectors that are not consistent with the optical ﬂow. Rotational and complex motion is only roughly approximated by motion detection algorithms. E= 2 2 λ2 u2 + u2 + vx + vy + (Ix u + Iy v + It ) dxdy x y (3. uy . Their implementation is shown to be min x∈η w 2 (x) [∇I(x. Their algorithm assigns weights. vx .5) . vy are ﬁrst order spatial velocity derivatives and Ix .2) is constrained and is now a well-posed problem.4) where ux . Other optical ﬂow estimators used for initialisation are known as gradient based algorithms (3.

In image processing. while respecting the manifold geometry [114]. This equation assumes a given initial condition. However.6) where Vt (x) is the heat vector located at x and at time t and ∇2 Vt (x) represents the Laplacian of the vector ﬁeld which produces a tensor. 114]: ∂Vt (x) − ∇2 Vt (x) = 0 ∂t (3. 31. as in most optical ﬂow estimators. this operation was seen as being equivalent with 29 . The general solution to the heat equation from (3.1 Anisotropic Diﬀusion of Optical Flow Diﬀusion kernel Anisotropic diﬀusion underpins the modelling of complex processes in physics and chemistry.3. y) spatial locations and η is the local windowed neighbourhood. GK (x − ς) is the heat kernel. which applies the diﬀusion onto the manifold M and x ∈ M. 3.where x = (x. indiscriminate smoothing could produce undesired eﬀects in the resulting motion ﬁeld. representing a Green function. The heat kernel is the natural candidate for measuring the similarity between two points on the same manifold. The heat equation of a geometric manifold can be described as [28. 29.6) yields [31]: ˆ Vt (x) = M GK (x − ς)V(ς)dς (3. The next section explains the optical ﬂow smoothing methodology using anisotropic diﬀusion.3 3.7) where ς is an arbitrary variable.

al.6) yields: 1 ˆ Vt (zc ) = √ 4πd x∈η exp − 1 (x − zc )T Σ−1 (x − zc ) Vt (x)dx 4d (3. 114]. x is a location in a neighbourhood η ∈ I zc is the central location of the windowed neighbourhood and d is a R. More recently though.e. the presence of edges and details require an anisotropic smoothing approach. the heat kernel from equation (3. 10] and it can be used as a feature detector [47]. a multivariate approach for nonlinear diﬀusion was considered in this work.7) is the Gaussian kernel [30. The local Hessian represents the curvatures of the local manifold [6. In this case. The proposed method uses a multivariate Gaussian kernel which is based on the local Hessian information. In contrast. In the case of images. then it can be deduced that points on the manifold are appropriately correlated to points in real space. In the case of real data. kernel [30. Black et. Similar to the statistical approach in [4]. 114]. [34] proposed an additive operator splitting (AOS) method which is based on the CLMC method [33]. Tschumperl´ and Deriche [6] used oriented Laplacians for e the same purpose. Assuming that the local manifold is warped to real space. i. Weickert et. al. normalisation coeﬃcient. [7] used Tukey’s biweight function to achieve similar results as obtained by Perona and Malik [4]. 31]. The embedding of the Hessian in the heat kernel is 30 .the local convolution of an image with a kernel function [30. Isotropic smoothing using PDE’s in the context of images was ﬁrst formulated by Koenderink [27]. M ≡ I then the heat kernel is the Gaussian R. Perona and Malik [4] proposed a smoothing function based on the Lorentzian error norm. Diﬀerent to the mentioned methods. the solution to the heat equation (3.8) where Σ represents the covariance matrix.

2(a).2(b) shows the degraded ﬁeld with additive Gaussian noise of mean zero and variance σ 2 = 0.3.2 shows the artiﬁcial vector ﬁeld that is used to evaluate which of the covariance (Σ).2: Artiﬁcial vector ﬁeld. 3.9) 31 .i − Vy )T N (3.3. Figure 3.8).discussed in the following subsection. Let us consider the calculation of the following measures on a neighbourhood of the vector ﬁeld. Laplacian (L) and Hessian (H) matrices to be used in the diﬀusion kernel function in (3.i − Vx )(Vy. This results in a more diﬃcult smoothing task for the diﬀusion kernels to recover the original vector ﬁeld in Figure 3. with Vx and Vy being mean vectors of Vx and Vy .2 Embedded Hessian diﬀusion kernel (a) Original vector ﬁeld (b) Degraded vector ﬁeld Figure 3. Figure 3. respectively: N Covariance (Σ) = i=1 (Vx.

26).3 show the eﬀects of Hessian (a). On the other hand. we can automatically direct the diﬀusion along the main data features. Hence. This evaluates as to how best the smoothed ﬁelds match the original vector ﬁeld.9620 after 2 iterations. in homogeneous regions there is no preferred direction of smoothing 32 ∂ 2 Vx ∂ 2 Vx 2 ∂y 2 Laplacian (L2D ) = ∂x 2 ∂ Vy ∂ 2 Vy ∂x2 ∂y 2 2 ∂ Vx ∂ 2 Vx 2 ∂x∂y Hessian (H2D ) = ∂x 2 ∂ Vy ∂ 2 Vy ∂y∂x ∂y 2 (3.9590 after 3 iterations and the covariance smoothed ﬁeld in Figure 3. The Laplacian smoothed ﬁeld in Figure 3. with the Hessian being better. Laplacian (b) and covariance (c) kernel based smoothing on the degraded artiﬁcial vector ﬁeld shown in Figure 3.2(b).3(b) yielded a mean cosine error of 0. Mean cosine error (MCE) of the smoothed ﬁelds is calculated as in equation (3. The covariance matrix is a global measure of how diﬀerent the Vx and Vy components of the vector ﬁeld diﬀer from each other.11) .The results in Figure 3. There is only 0.10) (3.3% diﬀerence between the Hessian and Laplacian smoothed vector ﬁelds. By embedding the local data in the Hessian difusion kernel.3(a) yielded a mean cosine error of 0. the Laplacian tend to oversmooth the ﬁeld between the horizontal and vertical vectors. its relative poor performance. The Hessian smoothed ﬁeld in Figure 3.3(c) yielded a mean cosine error of 0.9566 after 5 iterations. As can be observed. which the Hessian does not. Hence the Hessian matrix is chosen to be the most suitable to be integrated into the diﬀusion kernel.

3: Smoothed artiﬁcial vector ﬁelds. The local Hessian consists of an appropriate measure of variation in the geometry of the local statistics [114] and can be viewed as the application of a moving mask that models the second derivative of the local manifold. The proposed method is based on the use of a diﬀusion kernel which adapts itself to the local manifold variations. The most common way of calculating the local Hessian is by using the local second order central diﬀerence approximation of the second derivatives.(a) Hessian smoothed (b) Laplacian smoothed (c) Covariance smoothed Figure 3. The eigenvalues of the Hessian matrix as given in (3.11) can be used as 33 . and isotropic diﬀusion can be used.

a detector of change in the direction of the optical ﬂow by indicating the localisation and orientation of the moving object boundaries. the smoothing is extended by considering consecutive vector ﬁelds on both sides of the frame k containing 34 . the updated discretised and normalised equation is: Vt exp[−(xi − zc )T H−1 (xi − zc )] ki 2D.3. This approach is extended to consider multiple frames.12).3 Multiple 2D Hessian kernels The 2D Hessian kernel from (3. the heat kernel is formulated as an adaptive anisotropic ﬁlter that implicitly considers the local manifold variation by means of its Hessian. In this instance. In the following analysis. a 3D lattice of parallel and equidistant vector ﬁelds modelling the optical ﬂow from the entire image sequence is obtained.c exp[−(xi − zc )T H−1 (xi − zc )] 2D.7). t denotes iteration number and k is the frame index. The non-singularity of the local Hessian can be enforced by using various procedures. Considering that the Gaussian kernel is a solution to the heat equation.11) is embedded as the covariance matrix in the heat kernel from (3.12) is used to perform smoothing on motion vector ﬁelds calculated between pairs of consecutive frames. local smoothing is applied in the spatial neighbourhood as in (3. Firstly. The local Hessian (3. for example by calculating its pseudo-inverse. Thereafter. 3.12) xi ∈η(zc ) where Vt is the vector at location i within a neighbourhood η(zc ). which ki deﬁnes a symmetric region centered at the location zc .c ˆ t+1 Vkc = xi ∈η(zc ) (3.

Here.3. k + K and 2K represents the number of frames under consideration for smoothing. . .c ˆ t+1 Vkc = xi ∈η3D (zc ) (3. ψyy = . .4 3D Hessian kernel The 2D Hessian kernel is extended to 3D in order to accommodate the spatiotemporal variation in the optical ﬂow. zc : Vt exp [−(xji − zjc )T H−1 (xji − zjc )] ji 2D. ψkx = . ∂x2 ∂y 2 ∂x∂y ∂ 2 Vy ∂ 2 Vx ∂ 2 Vy = .13) j xi ∈η(zc ) where j = k − K.c exp[−(xi − zc )T H−1 (xi − zc )] 3D.15) .the central location.14) xi ∈η3D (zc ) where the neighbourhood is deﬁned symmetrically in 3D as η3D (zc ) and zc is the central location in the middle frame. the optical ﬂow transitions and moving object boundaries would be better modelled by the 3D Hessian whilst diﬀusing the vector ﬁeld. .jc ˆ t+1 Vkc = j xi ∈η(zc ) (3.jc exp [−(xji − zjc )T H−1 (xji − zjc )] 2D. 3. ψky = and ψkk is ∂y∂k ∂k∂x ∂k∂y 35 = ψyx ψyy ψyk ψkx ψky ψkk (3. By processing a larger amount of data. ψyk ∂y∂x ∂x∂k ∂ 2 Vy ∂ 2 Vx ∂ 2 Vx . ψxy = . The 3D Hessian matrix is given by: ψxx ψxy ψxk H3D where the entries of the matrix are ψxx = ψyx = ∂ 2 Vy ∂ 2 Vx ψxk = . the diﬀusion process is modelled as: Vt exp[−(xi − zc )T H−1 (xi − zc )] ki 3D.

12) in 1D. This signal is used as an example to mimic the presence of noise with an overwhelming value compared to its neighbourhood.4. Vy ) is the given vector ﬁeld deﬁned on a 3D lattice and k denotes the frame index.1 Outlier robustness study Consider the following one-dimensional (1D) signal: y . x=K . 6] embed the local data geometry in the diﬀusion kernel in order to avoid over smoothing of important data features.3.16) This signal has an outlier of height M at location x = K. it is not able to identify the noise. 3.the frame index diﬀerence. Hence.2 discussed the expansion of this approach for use on vector ﬁelds. particularly outliers.4(a) when M = 100 and K = 10. the Hessian becomes the second derivative and the resulting diﬀused value at location zc 36 y+M . Consider the implementation of expression (3. It is subsequently used to test the eﬀects of diﬀusion in the neighbourhood. another novel contribution of this work is the development of statistically robust diﬀusion framework of noisy vector ﬁelds. x=K f (x) = (3. Section 3.4 Robust Hessian Diﬀusion Kernels The directional anisotropic diﬀusion approaches introduced by Tschumperl´ e and Deriche [5. (Vx . 3. and its discretised version is displayed in Figure 3. While directional anisotropic diﬀusion is able to smooth according to the local data ﬂow geometry.

17). Consequently.18) M exp −i2 The bias at a site located at a distance of i positions from that of the ˆ outlier in the signal from (3. After replacing (3. the outliers will be diﬀused in the 2D neighbourhood resulting in a bias as that indicated in (3. i + 1}.h. The fractional part on the r. It can be observed that all the second derivatives in the signal from (3. Assuming the window size is N = 3. but not the outliers. The Hessian detects the edge. The diﬀusion of the outlier will inﬂuence several signal values in its neighbourhood.16).18) represents the bias resulted from diﬀusing the outlier present in the signal from (3.16) are zero except those at the locations j = {i − 1. the following result is obtained: 2 M ˆ f (K − i) = y + 4 2 4 exp −(i − 1)2 + exp −i2 + exp −(i + 1)2 M M M (3. The diﬀusion is calculated at the center of the window which is located at zc = K −i.s. i. of equation (3.16) into (3.is: N/2 f (zc + j) exp ˆ f (zc ) = j=−N/2 N/2 −4j 2 f (zc + j + 1) − 2f (zc + j) + f (zc + j − 1) exp j=−N/2 −4j 2 f (zc + j + 1) − 2f (zc + j) + f (zc + j − 1) (3. This situation could happen when diﬀusion is performed on images where outliers are present near edges.17) where N is the size of the diﬀusion window and where the diﬀerentiating operator is approximated by central diﬀerences.16) is evaluated as |f(K − i) − y|.18). diﬀusion is applied 37 .

(a) Initial signal (b) Diﬀused signal Figure 3. Other diﬀusion functions such as those from [7.4(a). 3. After a certain number of iterations.2 Median of directional Hessians kernel Robust statistics is known for its ability to preserve edges while eliminating outliers and have been used for image ﬁltering [115]. the values of the diﬀused signal are stationary and they are always biased with respect to the original signal which in this case corresponds to y = 10. 34] are not able to distinguish between outliers and data features. 6. It can be observed that the outlier is diﬀused in the surrounding signal.repeatedly on the signal displayed in Figure 3. Two robust statisticsbased ﬁlters. However.4(b).4.4: The eﬀect of diﬀusion on outliers. 3. 12} are shown in Figure 3. The results produced by diﬀusing the signal from (3. the median and the alpha-trimmed mean have been applied together with radial basis functions for eliminating outliers from the opti38 .16) during successive iterations t = {1. the following subsections introduces a new methodology that enhances the ability of anisotropic diﬀusion with an outlier rejection mechanism.

Ranking in vectorial data can be performed marginally. but in the ﬁrst instance is evaluated at locations situated at the margins of their corresponding central window. All the directions envisaged start from the central window location zc and are spaced at intervals of π/4 from each other. The resulting diﬀused vectors are ranked. or with respect to the distance to a central data sample. In our approach. Consequently.5 for N = 3. respectively. In this case. All such extended windows are used to calculate Hessians along directions towards the center of the initial N − by − N window. the calculation of the diﬀused vectors is not done centrally as in (3. In the ﬁrst instance.cal ﬂow [14] and for segmenting volumetric images [116]. This robust diﬀusion approach consists of two stages. i. we deﬁne an additional window in order to evaluate the Hessian based diﬀusion outputs as in (3.e. We consider a N − by − N window. median estimation is applied on the results provided by directional diﬀusion. The diﬀused vector from the center of the initial N − by − N window is calculated as well. The diﬀused vectors correspond to a speciﬁc directionally oriented geometry of their neighbourhoods and this process considers a certain degree of overlapping for the diﬀusion windows. This results a total of N 2 diﬀused vectors including the one in the center. The proposal here is to combine the performance of anisotropic diﬀusion in detecting the main data features with the ability of robust statistics algorithms to eliminate outliers. The locations where the diﬀusion is calculated are indicated by crosses in Figure 3. where N is an odd number. we use the marginal 39 . For each location in this window.11) along a speciﬁc direction. the resulting neighbourhood is increased from 3 − by − 3 to 5 − by − 5 vectors. along each entry separately.11).

116].4. the extreme vectors are eliminated. This method ranks the given data and eliminates a certain percentage of data samples at the extremes of the ranked array.19) where η(zc ) represents the neighbourhood deﬁned by the window centered at zc . the orientations for the diﬀusion calculation are deﬁned in the 3D neighbourhood leading to greater reliability in data statistics [117]. The aim of this method is to remove outliers and to apply the diﬀusion algorithm only on data that are statistically consistent with each other.3. calculated according to (3. The median operator is applied onto the results produced by the directional diﬀusions resulted as described above: ˆ ˆ Vkc = Med (Vi . It can be applied to the 2D Hessian. This algorithm takes into account extended neighbourhoods aiming to reduce overlaps among local estimates.median where all the corresponding entries are ranked separately and the middle value is chosen for each entry [14].4. also known as the alpha-trimmed mean algorithm.3. containing vectors resulting from directional diﬀusions.3 Alpha-trimmed mean kernel Another robust statistics based approach is the inter-quartile averaging. η(zc )) (3. 40 . multiple frame 2D Hessian calculated in Section 3. When applied to the 3D Hessian. 3.3 or to the 3D Hessian. The inﬂuence of outliers will be diﬀused during the ﬁrst operation and eliminated during the second step of median ﬁltering.14) and as described in Section 3. After ranking the data. which is suitable for smoothing medium and long tailed data distributions [115.

. the diﬀusion algorithm corresponds to the Hessian based diﬀusion from (3.2. the alpha-trimmed mean algorithm eliminates the data which are located far away from that central data sample [116].12). When α = 0. only one vector is taken into account corresponding to the median of the ranked array. The updating equation is: N −αN ˆ t+1 Vkc = i=αN N −αN i=αN Vt exp[−(xi − zc )T H−1 (xi − zc )] i c exp[−(xi − zc )T H−1 (xi − zc )] c (3. When ranking is performed according to the distance from a central vector.20) where α ∈ [0.5] is the trimming ratio of a ranked array of N vectors from the neighbourhood η( zc ) and Vt represents the i-th ordered vector at i location xi for iteration t. while when α = 0. 0.5. . .4. . The Hessian can 41 . although the calculation in this case is diﬀerent from the one described in Section 3. This algorithm reduces the computational cost of the diﬀusion per iteration by considering diﬀusion on a reduced set of vectors.Diff Diff Diff Diff Diff MED Diff Diff Diff Diff Figure 3.5: Calculation of the median of directional Hessian kernels.

M2DH . maximum iterations = 1. All the vector neighbourhoods are assumed as η2D = 3 − by − 3.12).diﬀusion algorithm using 2D Hessian (3.Additive Operator Splitting scheme [34] with parameters T = 10s. ATM-3DH .3. [7] with parameter λ = 1/(4 ∗ t).11).alpha trimmed mean of 3D Hessian.3D Hessian (3.median of 3D Hessian. 42 . al.Black et.20).multiple frame 2D Hessian (3. MEDM2DH .alpha trimmed mean of multiple 2D Hessian. MED-3DH . H3D from (3. TD . while for multiple frames they become 3 − by − 3 − by − 4 and for 3D kernels η3D = 3 − by − 3 − by − 3. The other algorithms considered for comparisons are Black . The algorithms are denoted according to the type of kernel that has been used for smoothing: 2DH . 3.14).Perona-Malik [4] with parameters κ = 30 and λ = 1/(4 ∗ t).be H2D from (3. ATM-2DH .median of multiple frame 2D Hessian. σ = 1 and diﬀusion timesteps = 1.3.alpha trimmed mean using 2D Hessian (3.15) or calculated by using H2D on multiple frames and averaging the outputs as described in Section 3.19).median of 2D Hessian (3. MED-2DH . 3DH . ATMM2DH . λ = 0.5 Experimental Results A comparative experimental study has been performed using the proposed robust diﬀusion methodology and various other diﬀusion algorithms for smoothing artiﬁcial vector ﬁelds corrupted by noise as well as motion ﬁelds extracted from real image sequences.Tschumperl´ and Deriche [6] with parameter e dt = 20 and AOS .11).1. PM .

3.5.1

**Smoothing noisy artiﬁcial vector ﬁelds
**

The ﬁrst vector ﬁeld,

Two artiﬁcial vector ﬁelds have been considered. Synthetic-1 is modelled by:

where Vx and Vy are the velocity components in the x and y directions, c = cos(θ), s = sin(θ), θ = 0, D = 0.8 is the dilation coeﬃcient, S = 0.05 is the shear coeﬃcient, R = 0.1 is the rotation coeﬃcient, and µ = 31 is the center of the resultant ﬂow. This vector ﬁeld models a complex variation of zooming in and out, and is displayed in Figure 3.6(a). The second vector ﬁeld, Synthetic-2 is created by diﬀerentiating the following expression:

x−µ c s c −s D + S −R V x = y−µ −s c R D−S s c Vy

(3.21)

x 1 2 2 2 2 − x3 − y 5 e−x −y − e−(x+1) −y 5 3 (3.22) ∂Z ∂Z The velocity components are obtained as V = . This vec, ∂x ∂y tor ﬁeld has two attractors and two divergent centers and is displayed in Z(x, y) = 3(1 − x)2 e−x

2 −(y+1)2

− 10

Figure 3.6(b). These vector ﬁelds are corrupted by noise and independently generated for each entry. In order to model a variety of possible corruption by noise, both additive Gaussian and Poisson noise distributions were considered. The Gaussian noise is commonly used as a noise model and is represented as: 1 G(σ 2 ) = √ exp σ 2π 43 −v 2 2σ 2

(3.23)

(a) Synthetic-1

60

(b) Synthetic-2

50

40

30

20

10

0 0 10 20 30 40 50 60

(c) Gaussian σ 2 = 0.10

(d) Gaussian σ 2 = 0.10

(e) Poisson σ 2 = 0.25

(f) Poisson σ 2 = 0.25

Figure 3.6: Synthetic vector ﬁelds, original and after being corrupted with noise.

44

where v is the random variable associated with the additive noise and σ 2 is the variance of the Gaussian distribution. The Poisson distribution is characterised by a long tail distribution and can be used to contaminate data with outliers. It is modelled by: e−(σ ) (σ 2 )v GP (σ ) = v!

2

2

(3.24)

where v! is the factorial of v, the number of occurrences, assumed to be an integer, and σ 2 > 0 is the variance of the Poisson distribution [118]. The width of the Poisson distribution is controlled by its variance σ 2 , which is directly related to the percentage of outliers in data. The vector ﬁelds modelled by equations (3.21) and (3.22) corrupted by Gaussian noise with σ 2 = 0.10 are displayed in Figures 3.6(c) and 3.6(d), respectively. The same vector ﬁelds corrupted by Poisson noise with σ 2 = 0.25 are displayed in Figures 3.6(e) and 3.6(f), respectively. Five diﬀerent values for the variance have been considered in each combination of noise distribution and data set. The proposed algorithms as well as the diﬀusion algorithm of Black et. al. [7] which has been adapted for the use on vectorial data, are applied for smoothing the noisy vector ﬁelds with the aim of trying to reconstruct the original vector ﬁelds. These experiments has only been tested on the following algorithms: 2DH, ATM-2DH, MED-2DH and Black [7]. For the alphatrimmed mean smoothing algorithm in the case of the η2D , α = 0.33 and hence 6 vectors are eliminated from the diﬀusion process. The numerical results are assessed with respect to two error measures representing the diﬀerence between the original and the smoothed vector ﬁelds. The error measures consist of the mean square error (MSE) and the

45

(a) 2DH

(b) Perona-Malik

(c) ATM-2DH

(d) MED-2DH

Figure 3.7: Examples of Synthetic-1 vector ﬁelds after corruption with Gaussian noise with σ 2 = 0.1 after ﬁve iterations of smoothing. mean cosine error (MCE). The MSE is given by:

L

MSE =

i=1

ˆ ˆ (Vi − Vi)T (Vi − Vi ) L

(3.25)

where L is the total number of vectors in the given vector ﬁeld, Vi is the ˆ ground truth before considering the noise and smoothing, and Vi is the result obtained after smoothing the noisy vector ﬁeld at location i. The MCE measures the angular error in the orientation of the vector as in [10] 46

444 0.286 0.10 0.2 47 .063 0.594 MCE 0.947 0.180 0.873 0.182 0.973 0. are provided in Table 3.629 MCE 0.998 0.933 0.728 7.712 MED-2DH MSE 0.237 0.991 0. should be ideally close to 1.059 0.627 10.690 0.009 0.309 1.985 0.016 0. The numerical results obtained after smoothing the synthetic vector ﬁelds corrupted by noise.933 0.995 0.852 0. as indicated by MCE between the smoothed vector ﬁeld and the original one.931 0.860 0.036 1.982 0. The minimum error.501 6.767 MCE 0.40 0.1: Numerical results for Synthetic-1 data after one iteration of diffusion.934 0.692 0.973 0.975 0.091 20.40 20.1 for Synthetic-1 data and in Table 3.005 0.995 0.007 0. denoted as θi .649 2DH MSE 0.978 0.987 0.797 0.830 0.015 0.002 0.094 0.854 0.818 0.959 0.052 7.668 ATM-2DH MSE 0.05 0.126 0.887 0. and is given by: L MCE = i=1 ˆ Vi · Vi ˆ Vi L Vi = cos(θi ) L (3.Method Noise (σ ) 0.740 0.998 0.007 0.993 0.276 0.154 0.001 0.30 0.26) The normalised dot product between the two vectors provides the cosine of the angle between them.097 0.528 0.851 0.871 0.873 Table 3.963 0.680 0.721 0.470 MCE 0.186 0.229 0.25 2 Black MSE 0.158 0.867 0.952 0.25 0.327 0.01 Gaussian 0. after one iteration by the given algorithms.454 19.999 0.01 Poisson 0.10 0.017 0.

113 0.237 ATM-2DH MSE 0.633 0.027 0.591 Table 3.108 0.088 0.6(b).070 2.800 0.021 0.722 0.682 0.554 0.376 0.469 0.956 0. for Synthetic-2 data.012 0.805 0.8(a)-3.011 0.372 0.146 0.30 0.8(c) it can be observed that after 48 . whilst ATM-2DH is the best kernel for smoothing out Poisson noise.532 0.522 0.482 0.961 0.871 MCE 0.7(a)-3.Method Noise (σ ) 0.253 0.776 0.10 0.414 0. MED-2DH provides the best performances when smoothing out Gaussian noise.493 0.213 0.023 0.425 0.923 0.073 MCE 0.25 0.969 0.435 0.6(a) and 3.008 0.832 6.985 0. it can be observed that despite the high level of additive noise.007 0.291 1.239 0.013 0.889 0. noise distribution and noise variance in these tables.633 0. The best results are highlighted in bold for each data set.572 0.626 0.8(d) show the results after applying diﬀusion kernels onto the vector ﬁeld Synthetic-2. respectively.974 0.443 0.05 0.612 0.195 0.531 0.219 0.658 0.2: Numerical results for Synthetic-2 data after one iteration of diffusion.703 0.01 Gaussian 0.25 2 Black MSE 0.362 0.473 0.156 0.427 10.40 19.10 0. From Figure 3.428 0.364 8.482 8.546 0.994 MCE 0.106 20.240 2DH MSE 0.043 0.019 0. while Figures 3.7(d) show the results after applying diﬀusion onto the vector ﬁeld Synthetic-1. Figures 3.359 MED-2DH MSE 0.350 0.883 17.01 Poisson 0.342 0.320 0.473 0. From these results. MED-2DH provides smooth vector ﬁelds that are similar with the original vector ﬁelds provided in Figures 3.40 0.008 MCE 0.338 1.061 0.623 0.359 0.

8: Artiﬁcial vector ﬁeld after corruption with Poisson noise with σ 2 = 0.(a) 2DH (b) Perona-Malik (c) ATM-2DH (d) MED-2DH Figure 3. The image sequences that have been considered show a large variety of motion and a diversity of factors aﬀecting 49 .25 after ﬁve iterations of smoothing. there is clear identiﬁcation of the vector ﬁeld structure from Synthetic-2 data whilst most of the noise was correctly cleared out.5. 3.2 Smoothing motion ﬁelds The proposed diﬀusion algorithms have also been evaluated on motion ﬁelds extracted from real image sequences. smoothing using the ATM-2DH algorithm.

9(c) and 3. which is widely used in video compression algorithms [1].10(b) for the Clouds frame and in Figure 3. The results provided by this algorithm are shown in Figure 3. Such a measure is widely used for assessing the eﬃciency of motion compression algorithms [1]. t): ˆ ˆ I(x.9(b) for the Taxi frame.9(d) for Concorde.9(e). 4 − by − 4 pixel blocks have been considered.10(d) for the Tornado sequence.9(a).10(e). One frame from each of the sequences Taxi.3). The optical ﬂow estimated using BMA is shown in Figure 3. The second method employs the Lucas-Kanade algorithm [3] which relies on a gradient based approach for estimating the motion (3. 3.motion detection.7) while embedding certain regularisation constraints. In both motion estimation methods and for all the image sequences. t) = I(x + ∆tV. t + ∆t) (3.27) 50 . Concorde and Fighter are shown in Figures 3. in Figure 3.10(f) for the Traﬃc frame.10(c) and 3. in Figure 3. we can predict an image frame by using the values of the current frame I(x. The ﬁrst method uses the block matching algorithm (BMA) according to (3. A measure for assessing the ability of the smoothing algorithms is provided by the frame reconstruction accuracy when considering the smoothed optical ﬂow.9(f) for the Fighter and in Figure 3.10(a). The optical ﬂow is generated using two diﬀerent approaches. while frames from each of the sequences Clouds. Tornado and Traﬃc are shown in Figures 3. It can be observed that the optical ﬂow estimation using block matching usually results in additional outliers when compared to Lucas-Kanade algorithm.1). As it can be observed from equation (3. 3.

9: Frames from three image sequences and their corresponding optical ﬂows.(a) 5th frame from the Taxi sequence (b) Optical ﬂow produced by BMA (c) 6th frame from the Concorde sequence (d) Optical ﬂow produced by Lucas-Kanade algorithm (e) 173rd frame from the Fighter sequence (f) Optical ﬂow produced by Lucas-Kanade algorithm Figure 3. 51 .

(a) 474th frame from the Clouds sequence (b) Optical ﬂow produced by BMA (c) 341th frame from the Tornado sequence (d) Optical ﬂow produced by Lucas-Kanade algorithm (e) 59th frame from the Traﬃc sequence (f) Optical ﬂow produced by BMA Figure 3. 52 .10: Frames from additional three image sequences and their corresponding optical ﬂows.

t) − I(xij . according to (3.ˆ where V is the smoothed vector ﬁeld at location x and time t. A higher PSNR corresponds to a better frame prediction. does not take into account atmospheric conditions. The frame reconstruction accuracy using motion ﬁeld based prediction is given by the peak signal-to-noise ratio (PSNR) which is calculated in decibels as: M N i j where 255 is the maximum value in a greyscale image of size M − by − N and we consider the prediction for a single frame (∆t = 1) without taking into account the predicted frame diﬀerence that usually carries the additional information needed for a perfect image reconstruction.28) i=1 ˆ n+1 ˆ ˆ n+1 ˆ (V i − V n )T (V i − V n ) < τ (3. However. the prediction of the scene based on the previous frame and the motion ﬁeld.27) and by considering the image values in the ﬁrst frame as known. changes in illumination or the perspective distortions caused by the 3D characteristics of the scene. The experimental results are a snapshot of the conditions at convergence.27). The frame prediction method ignores occlusion and does not consider the eﬀect of multiple pixels converging to the same location. The condition of convergence for all the smoothing algorithms is given by: 1 L L PSNR = 20 log10 255MN ˆ (I(xij . we can predict all the other frames by appropriately varying ∆t. t)) 2 (3. Based on equation (3.29) 53 . consequently indicating a better motion ﬁeld smoothing algorithm.

The results produced by 54 . In order to assess the eﬃciency of the α-trimmed mean anisotropic diffusion method. L is the total number of vectors that are smoothed and n is the iteration number.28). where τ = 10−2 is the convergence threshold. as deﬁned in equation (3.29) and represented using the PSNR.11. It can be observed that most of the proposed robust diﬀusion algorithms smoothly converge in just a few iterations. we calculate the PSNR of the predicted frame using (3.28) when smoothing the vector ﬁelds produced by either the block matching or the Lucas-Kanade algorithms for the ATM-2DH algorithm for various window sizes and for a variety of trimming α values.11: PSNR convergence for the reconstructed frame 8 from Concorde sequence. is shown for Concorde sequence in Figure 3. when the optical ﬂow is initially estimated by LK algorithm.Figure 3. The convergence results provided by (3.

best performance is not necessarily achieved. Figure 3. we improve the results produced by the diﬀusion process. using ATM-2DH smoothed vector ﬁelds when varying the alpha parameter for various window sizes. It can be observed that the best results are obtained when using a 3 − by − 3 window and α = 0.e. The results provided by some of the proposed methods when applied on 55 . The results from Figure 3. When increasing the neighbourhood size.12.these tests are shown in Figure 3. by eliminating outliers.2.12 are consistent for all the image sequences under consideration.4).12: Predicted frame PSNR evaluation. although when getting closer to the median (i. Deﬁnitely. eventually losing important information from the resulting optical ﬂow. the vector ﬁelds tend to be over smoothed. hence 4 vectors are eliminated from the diﬀusion calculation. when α = 0.

The optical ﬂow from Figure 3.14(c) and 3. This sequence was chosen for its complex motion characteristics such as the rotational movement.14(b).13 displays the results for the Taxi sequence.13. is shown in Figure 3. 3. 3.13(f) show the resulting predicted frames corresponding to the smoothed motion ﬁelds. Figures 3. 3. The Taxi sequence displays the motion of several rigid moving objects which is mainly translational. MED-M2DH and 3DH kernels. 3.9(d) after being smoothed using 2DH. The frame recon- 56 . The initial optical ﬂow was extracted between frames 4 and 6 using the Lucas-Kanade motion estimation algorithm which is shown in Figure 3. turbulent air from jet thrusters.9(b). 3.the optical ﬂow extracted from the given set of image sequences are shown in Figures 3.15 and 3.14(a). respectively. after being smoothed using ATM-M2DH.13(a).15 and 3. Figures 3. The best results for the other image sequences in terms of the optical ﬂow smoothness as well as the PSNR for the reconstructed frame are shown in Figures 3. initially estimated using the Lucas-Kanade algorithm from the Fighter image sequence. respectively. Figures 3.14(d) and 3. 3. Figure 3.16. blocky artifacts from compression and camera movement combined with a rigid moving object.14 shows the results obtained for the Concorde sequence.13(b).9(f). MED-2DH and MED3DH kernels.9(d).14(e) show the Concorde sequence optical ﬂow from Figure 3.13(c) and 3. Figures 3.13(e) show the optical ﬂow estimated between frames 3 and 5 and displayed in Figure 3.27).14.16.16(a). The frame reconstruction uses the smoothed motion ﬁelds according to equation (3.13(d) and 3. Figure 3. when smoothed by MED-M2DH.14(f) show the resulting predicted frames corresponding to the smoothed optical ﬂow.

the main motion structure is better recovered after smoothing the initially noisy vector ﬁelds. when smoothed by MED-3DH is shown in Figure 3.3 and 3. The frame reconstructed based on this smoothed optical ﬂow is provided in Figure 3.10(f). initially estimated using the block matching algorithm from the Clouds image sequence.15(b). The optical ﬂow from Figure 3.structed based on this smoothed optical ﬂow is provided in Figure 3. The frame reconstructed based on this smoothed optical ﬂow is provided in Figure 3. The frame reconstructed based on this smoothed optical ﬂow is provided in Figure 3. initially estimated using the Lucas-Kanade algorithm from the Tornado image sequence.17(a) and 3. As it can be observed from these ﬁgures.16(d).16(b).15(a). The PSNR results when tracking several frames from Taxi and Concorde sequences are shown in Figures 3. The optical ﬂow from Figure 3.15(c). Tables 3.16(c). when smoothed by MED-2DH is shown in Figure 3. the diﬀusion ﬁlters that performed the best compared to the other four kernels plotted are robust Hessian based kernels that uses information from multiple ﬁelds.10(b). These plots prove that the observed results are consistent for all the frames from the image sequences under analysis despite the various motion characteristics in these image sequences. It should be noted here that for both graphs in Figure 3.10(d). when smoothed by MED-2DH is shown in Figure 3.15(d).17(b). respectively.17. This shows that an object movement path in predicted frames is traced more eﬀectively using prior information from previous image frames/vector ﬁelds. initially estimated using the block matching algorithm from Traﬃc image sequence. The optical ﬂow from Figure 3.4 show the comparison between the diﬀerent diﬀusion 57 .

initialised using BMA and the resulting predicted 5th frame of the Taxi sequence.(a) ATM-M2DH smoothed ﬂow (b) Predicted frame 5 from (a) (c) MED-M2DH smoothed ﬂow (d) Predicted frame 5 from (c) (e) 3DH smoothed ﬂow (f) Predicted frame 5 from (e) Figure 3.13: Smoothed optical ﬂows. 58 .

(a) 2DH smoothed ﬂow (b) Predicted frame 6 from (a) (c) MED-2DH smoothed ﬂow (d) Predicted frame 6 from (c) (e) MED-3DH smoothed ﬂow (f) Predicted frame 6 from (e) Figure 3. 59 .14: Smoothed optical ﬂows. initialised using LK and the resulting predicted 6th frame of the Concorde sequence.

4 60 .e. additive operator splitting (AOS) [34]. while those from Table 3. i. proposed set of algorithms is considered as well as other diffusion algorithms such as Perona-Malik (PM) [4]. The results from Table 3.3 are obtained when smoothing the optical ﬂow generated by the block matching algorithm (BMA). algorithms. The basis e for comparison is the PSNR of the reconstructed frames at convergence.15: Results when applying optical ﬂow smoothing for the image sequences considered in this study when the optical ﬂow has been initialised using BMA.(a) Smoothed ﬂow from Clouds using MED-3DH kernel (b) Predicted 474th frame from ﬂow smoothed by MED-3DH (c) Smoothed ﬂow from Traﬃc using MED-2DH kernel (d) Predicted 59th frame from ﬂow smoothed by MED-2DH Figure 3. Black [7]. as well as Tschumperl´ and Deriche (TD) [6].

As it can 61 . In each of these tables.e. we also provide the PSNR of the reconstruction error when no smoothing is performed. the number of iterations necessary to achieve the convergence according to (3. are achieved when smoothing the motion ﬁeld which is initialised by Lucas and Kanade (LK) approach [3].29) is provided.(a) Smoothed ﬂow from Fighter using MED-M2DH kernel (b) Predicted 173th frame from ﬂow smoothed by MED-M2DH (c) Smoothed ﬂow from Tornado using MED-2DH kernel (d) Predicted 341th frame from ﬂow smoothed by MED-2DH Figure 3.16: Results when applying optical ﬂow smoothing for the image sequences considered in this study when the optical ﬂow has been initialised using LK. when using the optical ﬂow calculated using BMA and LK respectively. i. For each method and each image sequence.

62 .(a) Taxi sequence (b) Concorde sequence Figure 3.17: PSNR of the predicted frame when tracking scene change in two image sequences for the best ﬁve diﬀusion methods when the convergence criterion is set to τ < 10−1 .

17 6 19.58 3 19.94 4 1 3 1 3 18.16 2 17.59 AOS TD 20.64 1 MED-M2DH 22.61 1 15.52 3 1 20.34 22.55 8 18.Method Taxi Concorde No Fighter Clouds Tornado No Traﬃc dB No 13.33 3 21.16 5 20.3: PSNR (dB) of the reconstructed frame using the smoothed optical ﬂow with the number of iterations (No) necessary to reach convergence for each method and in the case of each image sequence when the motion ﬁeld is initialised using BMA.21 dB No dB BMA PM Black 2DH M2DH 17.94 2 21.11 5 18.93 2 15.70 2 14.72 15.18 3 21.84 MED-3DH 22. be observed.52 2 22.18 3 21.12 2 18.02 4 14.20 5 19.08 2 24.99 4 18.10 1 16.24 6 23.98 4 19.79 1 16.55 Table 3.26 4 18.83 1 12.80 1 21.12 8 20.14 2 14.40 4 18.39 19. the number of iterations necessary to achieve the convergence varies according to the method adopted and depends on the image sequence as well.46 7 18.48 2 2 4 2 18.83 2 22.91 83 12.52 15. It is observed that the robust diﬀusion algorithms usually require fewer iterations in order to achieve convergence when compared to the classi- 63 .85 2 22.57 3 17.42 1 17.96 2 14.46 2 15.49 1 23.66 2 23.34 2 16.83 2 14.98 21.29 1 20.52 2 22.45 2 16.21 2 3DH 21.25 6 19.99 6 12.46 2 15.44 2 15.97 1 22.35 2 19.13 2 ATM-2DH 21.81 4 MED-2DH 22.90 10 19.35 2 19.88 6 12.53 2 19.13 2 18.38 9 18.21 2 23.35 3 21.54 21.80 2 20.78 dB No dB No dB 15.93 2 16.70 7 ATM-3DH 21.54 20.59 ATM-M2DH24.40 21.05 4 18.15 1 13.

07 MED-3DH 22.24 2 23.78 17.40 2 19.99 3 25.78 4 22.36 6 20.51 14 19.22 4 19. The best PSNR of the frame reconstruction is highlighted for each image sequence and initialisation.41 dB No dB LK PM Black 2DH M2DH 20.08 1 20.18 1 16.02 6 20.10 2 19.82 1 20.34 4 19.81 1 24.82 4 21.88 8 17.34 8 AOS TD 22.13 2 18.00 1 16.69 2 19.10 2 19.25 2 23.58 1 20.15 2 20.27 1 20.12 3 25.59 21.13 3 MED-2DH 22.74 2 17.80 2 20.25 2 15.28 3 19. cal diﬀusion algorithms.70 2 21.74 2 ATM-2DH 20.25 MED-M2DH22.93 2 17.11 2 15.51 2 15.30 ATM-M2DH 19.77 100 Table 3.83 2 17.76 1 ATM-3DH 19.84 1 1 16.67 dB No dB No dB 16.31 4 19.86 2 19.92 30 17.38 1 16.25 11 20.64 10 14.85 12 16.09 2 20.46 14.08 1 20.42 2 18. The initial optical ﬂow provided by the Lucas and Kanade (LK) algorithm provide better PSNR for the frame reconstruction than the ones given by the block matching algo- 64 .06 1 13.14 1 14.14 10 19.74 2 17.4: PSNR (dB) of the reconstructed frame using the smoothed optical ﬂow with the number of iterations (No) necessary to reach convergence for each method and in the case of each image sequence when the motion ﬁeld is initialised using LK.43 2 22.19 2 19.08 1 16.42 5 15.18 1 16.93 2 17.40 2 20.11 21.50 1 19.48 8 23.67 2 16.50 1 19.21 4 4 19.11 21.42 6 18.65 2 23.Method Taxi Concorde No Fighter Clouds Tornado No Traﬃc dB No 15.48 2 17.20 1 4 21.27 10 19.22 3DH 8 1 20.65 2 23.00 2 18.

outliers from data are spread around introducing a bias in the resulting diﬀused signal. thus preserving the moving objects borders and the main ﬂow structure. Frame prediction can be further improved by inpainting the uncovered areas in the predicted frames using the neighbourhood information. The diﬀusion kernel is multivariate Gaussian and embeds the local Hessian as its covariance matrix. 3.4.rithm (BMA). Robust statistics algorithms such as the inter-quartile averaging and the marginal median are employed together with the diﬀusion kernels for removing the outliers and for 65 .6 Conclusions A set of robust diﬀusion algorithms have been applied for vector ﬁeld smoothing. These results show clear improvement when combining diﬀusion and robust statistics as well as when considering the motion from several consecutive frames. when the initial motion estimation is noisy due to various factors. The analysis has shown that following repetitive diﬀusion.3 and 3. It is observed that MED-2DH. This type of kernel ensures that smoothing occurs along the motion ﬁeld structure. The proposed methodology is tested on optical ﬂow. according to the results from Tables 3. MED-M2DH and MED-3DH provide the best results in terms of smoothing and reconstruction ability. The extension from 2D to 3D Hessian based Gaussian kernels considers the temporal information from multiple frames. for example when the optical ﬂow represents the motion of several objects which move in a variety of ways (including swirling motions) or when representing ﬂuids in motion. The smoothing results provided by the robust diﬀusion algorithms are better when the optical ﬂow is complex.

Motion ﬁeld smoothing can be used as a processing module for various systems such as motion estimation and segmentation. 66 .enhancing vector smoothing. The algorithms are applied onto artiﬁcial vector ﬁelds and onto the motion ﬁelds extracted from various image sequences. tracking and classiﬁcation of moving objects as well as in video compression systems that rely on motion based frame prediction. The improvements are evident when dealing with complex motion ﬁelds which are noisy. The proposed robust statistics based diffusion provides clear improvements over classical diﬀusion algorithms. The initial motion ﬁelds are generated using the block matching algorithm and the Lucas-Kanade algorithm.

Chapter 4 3D Volumetric Interpolation from Structural Flows 4. Hence.1 Research Objective Chapter 3 described the use of robust Hessian based kernels to perform diﬀusion on optical ﬂows from synthetic and real image sequences. the research is expanded to include synthesising intermediary slices and 3D volumetric visualisation of the object in the slices. It was thought that the research needed to be more diverse and not restricted to regularising motion from video sequences. As a result of the work in improving the recovery of structural ﬂows. the diﬀusion kernels have been tested on structural ﬂows obtained from 3D volumetric images representing series of sparse medical images of soft and hard tissue slices. 67 .

59. The proposed method models the variation in the internal structure represented in volumetric images.4. Variations of the block matching algorithm have been used for registration in medical images as described in [69. A method that considers the grey-level information as a surface followed by interpolation and thereafter collapse it back to a slice was proposed by Grevera and Udupa [119]. deformations of skin tissue [123] as well as changes in heart shape using diﬀuse tensor magnetic resonance images (MRI) [79]. 79]. In [55] the reciprocal morphing of one slice into the next one was performed using two mathematical morphology operators. The resulting smoothed structural ﬂows are applied to volumetric image interpolation with the aim to reconstruct 3D shapes. In this thesis. Inline with the block matching algorithm [1]. 121]. each image slice is split into blocks of pixels. In the medical imaging context. 70] and grey-level based [69]. registration of cross-sections from the brain [122]. The methods considered for volumetric image interpolation can be categorised [58] into shape-based [55.2 Introduction This chapter focusses on robust structural ﬂows as a methodology to model the morphing between consecutive slices in sequences of cross-sectional images. In this case. optical ﬂow has been used to model the movement of the heart [68. the proposed dual block matching algorithm (DBMA) models the structural ﬂow 68 . the vector ﬁelds show the temporal warping of one video frame into another. Vector ﬁelds have been used to represent the optical ﬂow in image sequences [1. 14. The method uses correlation to decide the correspondence between each block of pixels from one slice to the next [14]. 120].

the lack of contrast or missing data can lead to erroneous matches and unsmoothness in the resulting dual structural ﬂows.1 on a set of images showing cross-sections of a tooth. In order to overcome the above mentioned problems. These ﬂows are then smoothed by using robust diﬀusion ﬁlters embedding the Hessian of the local data. they are used to reconstruct intermediary slices between the images of the original sets. while the second vector ﬁeld models the reverse structural ﬂow. In [14]. The Hessian represents second derivatives and is known for its capability to model shape geometry in images [124]. Each vector from the ﬁrst structural ﬂow represents the correspondence of a set of voxels from one slice to the next one. this study has developed a methodology that combines the advantages of two diﬀerent approaches: anisotropic diﬀusion and robust statistics. Once smoothed ﬂows are obtained. In medical imaging. the initial slice is Slice 17 and 69 .4 are used for automatically detecting signiﬁcant changes and smoothing accordingly the structural ﬂow. The anisotropic kernel functions developed in Section 3.through the volumetric image by considering matching in both directions of the ordered slices.3 and Section 3. 4. In the Figure 4. the local Hessian has been used for feature localisation in [47] and for segmenting blood vessels [125]. This algorithm provides two structural ﬂows located on a regular 3D lattice. a reliability coeﬃcient was used to model the conﬁdence of the block matching estimates in video sequences.1. However. This is illustrated in Figure 4.3 Volumetric Image Interpolation The proposed methodology aims to obtain structural ﬂows from sparse medical datasets.

the reference slice is Slice 19. The aim is to reconstruct intermediate slices between the two original images. The arrows depict the direction of the volumetric image stacking. In between Slice 17 and Slice 19, the intermediate structural ﬂow and its corresponding reconstructed image slice is shown. As in [72], the slices (original and reconstructed) are stacked to reproduce a 3D volume that appropriately represents the original 3D object. The following sections describe the proposed framework.

Figure 4.1: An illustration of intermediate slice reconstruction.

70

4.4

Structural Flow Initialisation

In volumetric images, such as those provided by MRI, CT or by other means [55], a set of image slices are provided, each representing a cross-section through a volume. Vectorial ﬁelds have been used to represent the optical ﬂow in image sequences as well as to model deformations, particularly in medical imaging [68, 79, 121, 122]. One well known method to initialise the optical ﬂow is the block matching algorithm [14]. This algorithm ﬁnds the best correlation between two blocks of pixels from a reference image and an initial image, respectively, such that they have maximum correlation. The equation implementing BMA is provided in equation (3.3). The algorithm searches for the best match in a predeﬁned search region from the reference image. Other registration methods use the gradient for ﬁnding correspondences between regions of interest in medical images [69, 79]. In areas that have similar texture or contain widespread repetition of similar features, BMA produces erroneous decisions, thus resulting in noisy ﬂows [14, 120]. The proposed algorithm for producing structural ﬂows is called the dual directional block matching algorithm (DBMA) and produces two vector ﬂows that are oriented opposite to each other with respect to the ordering of slices in the volume. It is assumed that any specular, additive or distributive noise has been removed from the background for the ease of estimation. Each slice is split into rectangular blocks and the best matching (highest correlation) is sought for each block by comparing its grey-level values with those of blocks within a search area from the reference slice, as shown in Figure 4.2. In the matching process, only segmented foreground of the object (tooth) is considered. Unlike BMA, DBMA estimates two structural ﬂows, forward 71

and backward respectively, each oriented in one or the other direction along the main axis of the object structure. An example of dual structural ﬂows is shown in Figure 4.3 for a set of cross-sectional image slices representing a tooth.

Figure 4.2: Block matching process in DBMA. In the example shown in Figure 4.3, structural ﬂows are computed between Slice 17 and Slice 19 of the Incisor tooth dataset [55]. For the ﬁrst structural ﬂow (forward direction), matching from slice t to t + 1 results in the vector ﬁeld V1 , while for the second, the reverse matching is computed from t + 1 to t resulting in the vector ﬁeld V2 [117]. For displaying purposes, the total ﬂow, as shown in Figure 4.3(c) is obtained by adding forward and reverse ﬂows, i.e. V1 + V2 . This ﬂow represents the estimated deforma72

(a) Forward Flow, V1

(b) Reverse Flow, V2

(c) Combined forward and reversed ﬂows Figure 4.3: DBMA structural ﬂows. tion that took place between Slice 17 and Slice 19. Although edges of the structure have been captured reasonably well, the internal structure remains noisy. Similar to the block matching algorithm used for estimating motion in image sequences, the search area from the reference slice has the same centre as the block region from the initial slice. However, due to the shape variation, a large part of the search region may consist of the background, i.e. without useful information for the structural ﬂows. Careful consideration of the block size is needed, especially for locating edge correspondences. An example of

73

4: Bad edge matching. Figure 4.3.e. by checking the number of non-zeros present in the blocks. Perfect matching against the background is lost or diminished by reducing the proportion of zeros in the pixel block containing the object boundary considered. however. Line 1 of the pseudocode checks whether the blocks are diﬀerent.5 presents the pseudocode of the growing algorithm. it can be said that a trade-oﬀ needs to be achieved between background matching and retaining the object shape as illustrated in Figure 4. i. The input to the function GrowingRegion consists of the intensity blocks from the original and reference slices.4. NewBlk is taken to be the array of imin − NewHopX : imax + NewHopX in the x-direction and 74 . Therefore. The pseudocode shown is a subset of the main structural ﬂow estimation program. then the variables are set in lines 3 to 6.matching errors which occurs at the boundary of the object is shown in Figure 4. Figure 4. If this statement is TRUE. to retain the object shape information. It is important. HopX and HopY represent the hopping distance between blocks. there are no matching correspondences.

imax. the condition is TRUE.jmax from main program NewHopX ← HopX NewHopY ← HopY BoundaryBlk ← NewBlk while (NoMatchingBlocks) do NewHopX ← NewHopX + HopX NewHopY ← NewHopY + HopY Length ← NewHopX. i.75 ∗ Length)/2 £ perform matching in the new search region Figure 4. If no such information is found. then the size of the block is expanded as shown in lines 9 and 10. Once the edge of the shape is found within the reference slice.1 if (nnz(BlkOrig) >= RegionSize) && (nnz(BlkRef ) < RegionSize) 2 3 4 5 6 7 8 9 10 11 12 13 then £ get imin. the shortest path between the initial point and the object boundary is traced. BlkRef ): Pseudocode of growing algorithm used for regions of object boundaries. NewHopY RegionSize ← (1. where nnz(·) is a function that ﬁnds the number of nonzero pixels in a block. jmin − NewHopY : jmax + NewHopX in the y-direction.jmin.5: GrowingRegion(BlkOrig. The condition NoMatchingBlock in the while loop checks the presence of shape or edge information of the object within BoundaryBlk in the corresponding reference slice.e. This distance (Euclidean) is 75 .

is attributed to the boundary structure of the object. This is an interesting problem since matching between boundary regions in the initial slice and background at the same pixel location in the reference slice will lead to wrong ﬂow estimation. For the second structural ﬂow.6(a). a growing algorithm is applied by progressively increasing the size of the search region as shown in Figure 4. until a part of the object contained in the reference slice is detected as shown in Figure 4. a growing algorithm is implemented within DBMA. Then.6 on two slices of the Incisor data set. The new search region is then fed back into the main program to re-estimate the structural ﬂow. A value of 1.6(b).(a) Initial search block (b) New search block Figure 4. which is similar to mathematical morphology method employed in [55]. is used as a scaling constant. the algorithm 76 . the matching proceeds until the best correlated block is found.75. Therefore.3. then used in line 12 to obtain a new search region. Another problem that may arise. With reference to Figure 4. found by experimentation.6: Visualisation of growing algorithm. as it can be observed in Figure 4. The functioning of this algorithm is illustrated in Figure 4.6.

Due to the variations in the circumstances of choosing the search region each time from a diﬀerent reference image.4. The proposed Hessianbased diﬀusion kernels are able to either reduce or remove noise in order to obtain a better representation of the inner and outer volumetric structure. The smoothing ﬁlters used are those developed in Section 3. The following Section 4. While DBMA improves the modelling of feature correspondences by creating a better representation of the variation in the volumetric image.5 Smoothing Structural Flows Why is smoothing necessary for structural ﬂows? As it can be observed in Figure 4. Hence.is applied in a reverse manner for the two slices. the resultant ﬂows are quite noisy. The process of smoothing structural ﬂows takes place in a similar manner to that of the optical ﬂows from image sequences. noisy (corrupted) ﬂows lead to bad interpolations and smoothed ﬂows lead to more reliable outcomes.5. it also results in noisy vector ﬁelds in the same way as BMA does. Basically. the two structural ﬂows are not identical in the absolute value.3. it is more than likely that the results would not be very good. The normalised updating kernel for smoothing structural ﬂows is given as 77 . 4. it is important to ﬁlter the estimations for obtaining more accurate reconstructions.3 and Section 3. seeking the best correlations for blocks from the reference slice from inside a search region deﬁned in the initial slice. addresses the smoothing of structural ﬂows. This is because unsmoothed estimations can have detrimental inﬂuence on reconstructed data. If these ﬂows were to be used for interpolating intermediate slices and hence reconstruct the 3D volume.

6 Slice Interpolation In many applications.19) Hessian based kernels. 58. As shown in Figure 4. It is worth pointing out that smoothing is not conducted on the full structural ﬂow for an intermediate slice as it would be expected.1) xi ∈η(zc ) is the vector at the location i within a neighbourhood η = 3−by−3.7 and described in the next section.7. ATM-2DH (3.7. particularly in the medical ﬁeld. k is the slice number.12). 4.ˆ t+1 Vkc = where Vt ki xi ∈η(zc ) Vt exp[−(xi − zc )T H−1 (xi − zc )] ki 2D. experiments were carried out for smoothing using 2DH (3. Volumetric image information is interpolated in both shape structure and grey-level information along the structural ﬂows. and zc represents the central location of the neighbourhood. For comparative analysis. This ensures that the structural ﬂow estimations of the object deformations are modelled reliably throughout the whole reconstruction process. With reference to Figure 4. The proposed model uses smoothed structural ﬂows to interpolate additional slices in between the cross-sectional slices of the existing set. 69]. 59. structural ﬂows (Flow 1 and Flow 2) are 78 . it is essential to interpolate image slices in order to have an appropriate volumetric representation [55. t denotes the iteration number.c (4. which are provided in Chapter 3. The proposed intermediary slice interpolation algorithm is presented in the chart shown in Figure 4.c exp[−(xi − zc )T H−1 (xi − zc )] 2D.20) and MED-2DH (3. smoothing is performed on the forward and reverse ﬂows separately.

respectively. computed in the forward and reverse directions.Figure 4. This is essential if the information of object structure from both slices are to be used for intermediate slice reconstruction.7: The slice interpolation ﬂowchart. The ﬂows model the shape 79 . The main purpose of computing the structural ﬂows in this way is to estimate the deformations from one slice to the next smoothly. as described in Section 4.4.

4. It should be noted that during the intermediary slice reconstruction stage. between the two given 1 . in order to reconstruct the ﬁrst slice from a total number of n = 20 slices between pairs. During the smoothing stage. Intermediary slice parts are produced by displacing blocks of pixels along both structural ﬂows from initial slice pairs. 61. the texture and other features that follow the direction of the structural ﬂows in the volumetric image. 119] can also be used. Using this framework. For example. As it slices. For areas that have object grey-level assignments for both structural ﬂows. 77. 56. forward and backward ﬂows. 60.3 and Section 3. 64. This is appropriate since the intermediary slice to be reconstructed is immediately adjacent to the original Slice 1. image slices obtained by B-splines [122] or other methods [55.7. The resulting values from the two structural ﬂows are combined in the last step in order to generate the interpolated slice. Consequently. the ﬂows are independently smoothed using the diﬀusion kernels proposed and described in Section 3.morphing as well as the intensity variation from one slice to another (Slice 1 and Slice 2) and consist of a 3D representation of the variation in the volumetric shape [117]. up to several slices can be generated along both. the interslice diﬀerence is reduced by n−1 can be seen from Figure 4. In order to reconstruct intermediary slices. the resulting grey-level 80 . a proportion of these ﬂows is considered for slice reconstruction along each ﬂow. the structural ﬂows interpolate the deformation between the given pair of slices. the corresponding forward ﬂow will be 1/20th of Flow 1 and the reverse ﬂow will be 19/20th of Flow 2. The reconstruction of each part of the intermediary slice consists of the contour. representing the volumetric image information located at quite a distance away from Slice 2.

one 1 2 The Incisor was used in [55] The image datasets used in this experiment were from the Laboratory of Human Anatomy and Embryology.uni-erlangen. http://isbweb. the kth intermediary slice between the image slicess I1 and I2 is generated by the following equation : ˆ ˆ kI1 (x + V1 ) + (n − k)I2 (x + V2 ) n I1. V1 and V2 are the smoothed vector ﬁelds modelling the warping along the forward and reverse ﬂows. Belgium. a humerus bone2 (part of the upper arm) and an iliac bone2 (part of the hip joint).org/data/vsj/index.de/External/vollib/ 81 . comprising of a sheep’s heart3 . Another three data sets.7 Experimental Results The proposed methodology has been applied on real volumetric medical images.k (x) = (4. 4.informatik. University of Brussels (ULB). respectively. CT scan of a female chest3 and MRI of a knee3 have been used. In the case where there is only one data assignment from one of the structural ﬂows while according to the reverse structural ﬂow there is no assignment. the positive decision has priority. Therefore. and I2 corresponds conventionally to I1.2) ˆ ˆ where x is the location of the interpolated result.html 3 Obtained from The Volume Library at http://www9.is decided by proportionally averaging the two values. Results on the eﬀects of diﬀusion on structural ﬂows and analysis are presented in the following section. An example of original slices. The given data sets are composed of sequences of digitised crosssections of medical data volumes representing both hard and soft tissue. Three datasets are used representing an incisor1 .n from a total of n slices.

which has also been used in [55]. the slices are aligned using a semiautomatic procedure. Three slices of the incisor sequence are shown in Figures 4. contains 22 slices and represents an incisor. The proposed vector smoothing methodology is applied on all these structural ﬂows. The slices have been obtained by mechanical slicing followed by digitisation.9(b) 82 . In Table 4.8: Sample slices of the data sets. We have initialised the structural ﬂows using the DBMA (producing two ﬂows) as explained in Section 4.9(a). the technical details of these image slices are provided.for each data set is shown in Figure 4.1. The ﬁrst dataset. After segmenting the tooth body from the background as well as its root canal. (a) Incisor (b) Humerus bone (c) Iliac bone (d) Sheep’s heart (e) Female chest (f) Knee Figure 4.8. 4.4 and Lucas-Kanade algorithm [3] producing a single ﬂow.

4.20 0.00x1. Perona-Malik [4].00 1. al. ATM-2DH kernel.11(d) and 4.10(c).10(a).11(a). respectively.1: Summary of the slice dimensions and voxel sizes for the diﬀerent data sets.00x250. Reconstructed slices corresponding to Slice 18 after using DBMA ﬂows smoothed by various algorithms are shown in Figures 4.40 180.25x0.’s algorithm [7] and MED-2DH are displayed in Figures 4. The directions of morphing one slice into another are captured well by the structural ﬂows.40x12.Data Set In-plane grid sizes In-plane dimensions (mm) No. As it can be observed.00-2. while Slice 18 reconstructed using this structural ﬂow is displayed in Figure 4.11(c) and 4. The forward structural ﬂow using DBMA. The structural ﬂows resulted from smoothing by 2DH kernel.00 0. these slices correspond to cross-sections through the tooth in the region where the root canal emerges. 4. 4. and 4.10(e). calculated between Slice 17 and Slice 19 is shown in Figure 4.10(b).11(b). Black et. 4.11(e).00 1.10(f).11(f).00 1.9(c).00 1. 4. slices Slice spacing (mm) Incisor Humerus Iliac Sheep’s Heart Female Chest Knee 500x500 512x512 512x512 352x352 384x384 512x512 12.00 250.50-1. 4. The structural ﬂows smoothed by PM [4] and Black [7] algorithms 83 .50 Table 4. As it can be observed from these ﬁgures.50-1.00 0.25 22 401 260 256 240 87 2.10(d).00x1.00x180.00 1. all the diﬀusion based algorithms improve the initial results provided by DBMA structural ﬂows.

The best visually assessed recon- 84 . The robust diﬀusion algorithms provide smooth structural ﬂows while eliminating the inﬂuence of outlying vectors. ATM-2DH and MED-2DH smoothed structural ﬂows (as it should be according to the ground truth slice from Figure 4.9(b)).(a) Slice 17 (b) Slice 18 (c) Slice 19 Figure 4. are noisier than the structural ﬂows smoothed by the proposed diﬀusion algorithms.9: Sample slices from the Incisor data set. The root canal is almost completely closed in the reconstructions provided by 2DH.

85 .10: Structural ﬂows and reconstructed slice of an Incisor.(a) DBMA ﬂow (b) DBMA reconstructed (c) 2DH smoothed (d) 2DH reconstructed (e) PM smoothed (f) PM reconstructed Figure 4.

11: Further results for the Incisor. 86 .(a) ATM-2DH smoothed (b) ATM-2DH reconstructed (c) Black smoothed (d) Black reconstructed (e) MED-2DH smoothed (f) MED-2DH reconstructed Figure 4.

MED2DH provides the smoothest surface for the 3D incisor reconstruction closely followed by the 2DH kernel and DBMA (reconstruction based on unsmoothed ﬂows).struction that also provides the most compact reconstruction of this slice is that provided by the MED-2DH smoothed structural ﬂows. in reality this is not always the case since the results prove that the middle slice reconstructions are not always identical to the original slices. then bad shape reconstructions would be nulliﬁed. The focus however is to observe the eﬀects of smoothed structural ﬂows on intermediary slice reconstructions. while PM and Black does not provide very smooth shapes.13 provides numerical results for reconstructing the middle slice for the whole Incisor data set when skipping one slice at a time and aiming 87 .10 and 4. while a large value will indicate the volume to appear elongated. This means that a total of 20 slices are interpolated between pairs of original slices. If the data set is not sparse. Though it is assumed that the slices have equal distance between each other. including the originals. Figure 4. It should be noted that the main cause of this diﬀerence in the reconstructions is that there is signiﬁcant shape variation from one slice to another from the data set. the structural ﬂow is computed between Slice 17 and Slice 19 with the aim of reconstructing Slice 18. A low value indicates the volume is more likely to appear squashed along the central axis. From the results obtained. As shown in Figures 4. Figure 4.12 shows the 3D volume visualisation of the entire set of 420 slices obtained through interpolation for structural ﬂows smoothed by all six algorithms.11. The number of interpolated slices will depend on the volume to be reconstructed. It is necessary to obtain a balance in order to obtain a suitable volume.

(a) DBMA (b) Perona-Malik (c) Black (d) 2DH (e) ATM-2DH (f) MED-2DH Figure 4. to reconstruct it from its two neighbouring slices. This percentage of error.12: 3D Incisor reconstructions. ǫ is calculated 88 .13(a) provides the shape representation accuracy by evaluating the percentage of estimated pixels that are not correctly placed in the estimated slice when compared to the original from the initial data set. Figure 4.

i where the object structure is represented and jp represent the spatial position in Ip. it can be observed that the robust Hessian based kernels perform better at correctly identifying positions for new intensity pixels when compared to the other kernels. io represent the spatial position in Io.13(b) that the recon89 . From the plot in Figure 4.13(b) evaluates the grey-level reconstruction using various diﬀusion kernels by calculating the peak signal-to-noise ratio (PSNR) between the estimated object/slice and the original slice.j where the estimated object structure is represented. large shape deformation or sharp changes in grey-level intensity. i. the percentage of pixel error is obtained by deducting the sum of correctly placed nonzero pixels in the estimated slice from the total number of nonzero pixels in the original slice over the sum of the nonzero pixels in the original slice multiplied by 100.3).j = 0 (4.i . It can be also observed that sudden changes in the graph generally represent major diﬀerences between slices. In equation (4.3) where N is the size of the data. It can be observed from Figure 4.13(a). BMA [1] is the classical block matching algorithm. The estimation of the structural ﬂows is based on the assumption that the slices are equidistant. Figure 4. which is not always the case. particularly in this experiment where the incisor had been mechanically sliced. using a single structural ﬂow for evaluating the displacement between two consecutive slices instead of the dual structural ﬂow as in DBMA. Ip.e. Lower percentage of error shows a higher degree of accurate reconstruction.as N N ǫ= i io − N jp j io i × 100 ∀Io.

13: Accuracy of the middle slice reconstruction considering both shape structure and grey-level.(a) Shape reconstruction error rate (b) Peak signal-to-noise ratio (PSNR) Figure 4. 90 .

the Hessian based kernels do the opposite.k+2(xi ) − Ik+1(xi )) 2 (4. the results are misleading. although the shape deformation is not estimated accurately. This is due to the fact that both BMA and DBMA rely on maximising the 91 PSNR(k+1) = 20 log10 255M ˆ (Ik. Ik+1 . hence the high PSNR value.1. The shape deformation is modelled accurately. which in reality is not. The higher the PSNR. The Perona-Malik and Black kernels are somewhere in the middle.k+2 frame. The PSNR between the original (ground truth) and the reconstructed slice is calculated in decibels as: M i=1 where M is the number of pixels located in the foreground at x co-ordinates ˆ in both the original k + 1th frame. Furthermore. while low PSNR would reﬂect the approximate distances of the real slices.13(a). the estimation of the structural ﬂows are based on the assumption that the slices are equidistant. the closer the slice is to the absolute middle. and the predicted Ik. BMA and DBMA provide quite good PSNR reconstruction values but on the other hand they are not able to appropriately model the variation in the shape according to the plot in Figure 4. However.4) . but the grey-level reconstruction is somewhat poor. On the other hand.k+2 is reconstructed from the structural ﬂows between the k and k + 2 frames. as can be observed in Figure A. the intensity value at correctly placed pixel matches the original. The ˆ predicted frame of Ik. With BMA and DBMA.struction results for the Hessian based kernels are not very good when the reconstructed slice has to recover signiﬁcant morphological change between two consecutive slices.

provide good shape reconstructions while their grey-level restoration is not much worse than that provided by BMA and DBMA. Three slices of the knee sequence are shown in Figures 4.16(d) and 4.16(e). 4.16(b). ATM2DH gives the best shape reconstruction results closely followed by 2DH.15(e). 4.16(f). Reconstructed slices corresponding to slice 31 after using LK ﬂows smoothed by various algorithms are shown in Figures 4. 4. respectively. 4.14(a).15(c).13(a).15(b).14(b) and 4. The initialisation for this data set computed between slices 30 and 32 using the Lucas-Kanade [3] algorithm is shown in Figure 4. Diﬀusion based kernels. 4. 4. while slice 31 reconstructed using this structural ﬂow is displayed in Figure 4. 4.16(c) and 4.correlation between blocks of pixels which results in a procedure ideal for maximising the reconstruction PSNR. these slices correspond to transversalsections through the knee and display both soft and hard human tissue. 92 .14(c). Perona-Malik [4]. This is actually the main reason why BMA is used in predictive based coding currently embedded in video coding algorithms. The improvement provided by DBMA over BMA in terms of reconstruction error is clear from the plot in Figure 4. PM and MED-2DH algorithms for this 3D volume data. Black’s algorithm [7] and MED-2DH are displayed in Figures 4. Another data set that has been used for comparison purposes is the knee MRI data set.15(f). on the other hand.15(a).16(a).15(d). The structural ﬂow resulted from smoothing by 2DH kernel. ATM-2DH kernel. In this case.

As it can be observed from Figures 4. all the diﬀusion based algorithms improve the slice reconstruction results provided by LK structural ﬂows.(a) Slice 30 (b) Slice 31 (c) Slice 32 Figure 4. The robust diﬀusion algorithms provide smooth structural ﬂows while eliminating the inﬂuence of outlying vectors. The best visually assessed reconstruction that provides the most compact reconstruc- 93 . The structural ﬂows smoothed by PM [4] and Black [7] algorithms are noisier than the structural ﬂows smoothed by the proposed diﬀusion algorithms.14: Example of Knee slices for experimentations. The directions of morphing one slice into another are captured well by the structural ﬂows.15 and 4.16.

94 .15: Structural ﬂows and reconstructed slice of a Knee MRI.(a) LK ﬂow (b) LK reconstructed (c) 2DH smoothed (d) 2DH reconstructed (e) PM smoothed (f) PM reconstructed Figure 4.

(a) ATM-2DH smoothed (b) ATM-2DH reconstructed (c) Black smoothed (d) Black reconstructed (e) MED-2DH smoothed (f) MED-2DH reconstructed Figure 4. 95 .16: Further results on structural ﬂows and reconstructed slices of a Knee MRI.

20 shows the processed 3D volumes for the Iliac bone from the structural ﬂows smoothed by various diﬀusion kernels. 4. mathematical morphology operator [55].11. Figure 4.15 and 4. Figure 4.19 shows the original Iliac bone and Figure 4. Four other data sets representing a mixture of soft and hard human tissues have also been used for experimental purposes. while Figure 4. The surfaces of the reconstructed objects are smoother for the 3D Incisor and Humerus than those of the Iliac bone which has a more complex mor- 96 . MED-2DH have also accurately modelled the deformation of soft tissue in the muscular area of the knee whilst smoothing out unwanted specular eﬀects of the image.16 caused due to pixel block displacement can be covered using interpolation [119]. The small black gaps in the reconstructed slices observed in Figures 4. when applied on grey-level images. The observation shows that MED-2DH have signiﬁcantly reduced the inﬂuence of outliers and modelled the deformation of the hard tissue of the knee more accurately when compared to the other algorithms used in the experiments. 4. The structural ﬂows for these slices of human bones have been initialised using the DBMA algorithm.10. diﬀusion algorithms for inpainting [8] or the diﬀusion methodology proposed in this thesis.18 represent 3D volume reconstructions using interpolated structural ﬂows smoothed by diﬀusion kernels. The original Humerus and Iliac data sets are used as ground truth and are subsampled by 1:6 aiming to replace the slices that are initially skipped for numerically assessing the reconstruction methodology.tion of this slice is that provided by the MED-2DH smoothed structural ﬂows.17 shows the original 3D Humerus bone.

The external surfaces of the 3D shapes can be further smoothed using additional post-processing 3D shape smoothing algorithms. female chest and sheep’s heart MRI scans have been initialised using the LK algorithm [3].22 show some intermediate slice ﬂow and reconstruction results for the chest and heart data sets. respectively. At the extreme end of the bone slices. The soft tissue data sets used in this study. whereby ﬁve original intermediate slices have 97 .Figure 4. The 3D Humerus volume reconstructed from the structural ﬂows smoothed by MED-2DH is smoother everywhere as compared to the other algorithms apart from the regions at the extreme end slices of the bone.21 and 4. Figures 4.17: Original Humerus bone. it is not possible to show a 3D rendered volume of the chest and heart. These results attempt to reconstruct the middle slices. PM [4] and Black [7] algorithms perform better. phology. As a consequence. Prior segmentation have not been conducted on these data sets.

98 .18: 3D Humerus bone reconstructed volumes.(a) DBMA (b) PM (c) Black (d) 2DH (e) ATM-2DH (f) MED-2DH Figure 4.

Figure 4. especially for large slices. for visual clarity purposes. This is because the LK algorithm is a gradient based method and produces dense ﬂow ﬁelds. The reconstruction errors are calculated by skipping one intermediary frame for each pair of slices from the original Incisor data set and 5 slices for the Humerus and Iliac data sets. These slices are reconstructed back as described in this chapter using the DBMA method as the initialisation.22 are intermediate slice reconstructions from Black et. been removed. All the structural ﬂows computed using the LK algorithm have been subsampled by a factor of 2 in x and y directions. al. The diﬀerence.21 and 4. The results displayed in Figures 4. [7] and MED-2DH smoothed structural ﬂows. As it can be observed. which are diﬃcult to visualise. the MED-2DH smoothed structural ﬂows seemed to be cleaner and less noisy when compared to the result produced by Black’s algorithm.19: Original Iliac bone. calculated as the percentage of wrong shape assignment between 99 .

100 .20: 3D Iliac bone reconstructed volumes when skipping 5 consecutive slices between the remaining 2 slices.(a) DBMA (b) Perona-Malik (c) Black (d) 2DH (e) ATM-2DH (f) MED-2DH Figure 4.

21: Results on structural ﬂows and reconstructed slices of a Female Chest MRI.(a) Slice 54 to be reconstructed (b) Black smoothed (c) Black reconstructed (d) MED-2DH smoothed (e) MED-2DH reconstructed Figure 4. 101 .

102 .(a) Slice 177 to be reconstructed (b) Black smoothed (c) Black reconstructed (d) MED-2DH smoothed (e) MED-2DH reconstructed Figure 4.22: Results on structural ﬂows and reconstructed Sheep’s Heart MRI slices.

MED-2DH is the best algorithm for volumetric reconstruction. aiming to reconstruct the middle slice with LK algorithm as the initialisation. h(fi .2. However.3 have been obtained by taking the average PSNR of all slices needed to reconstruct a 3D volume of a particular object. The average PSNR has been compared against diﬀerent diﬀusion ﬁlters. Humerus and Iliac data sets in Table 4. fi ). Therefore.the original and the reconstructed data set are provided in Table 4. the Humerus and Iliac datasets not being sparse.5) 103 .3. DBMA (no smoothing) achieves a higher accuracy of pixel interpolation compared to diﬀusion methods. A third measure considered to validate the shape reconstructions is the Hausdorﬀ distance measure. From the result observed. the Hessian based kernels were able to model the shape variations more accurately than the other methods. it is thought that gradient based ﬂow methods tend to be more eﬃcient than the block based methods. suggested minimal variations between slices. The Hausdorﬀ distance between two contours ˆ fi and fi is deﬁned as in [126]: ˆ ˆ ˆ H(fi. For the Incisor. Also presented is the average PSNR of the intermediate slice reconstructions for the whole 3D volume of Incisor. thereby interpolating the pixels more accurately. the Hessian based kernels performed better because the dataset is sparse with large shape variations between slices. Table 4. fi )} (4.4 presents the average PSNR after removing 5 slices from the original set (for all 6 data sets). with shape interpolation no longer an issue. fi ) = max{h(fi . Hence. As expected. The numerical results in Table 4.

2: Average percentage of reconstruction errors with DBMA as the initialisation. 3.34 14.52 8.09 6.30 MED-2DH 14.85 7. and · denotes the Euclidean distance.04 2DH 14.3: Average peak signal-to-noise ratio of slice reconstructions with DBMA as the initialisation.67 PM 13. MED-2DH smoothed structural ﬂows provide the best reconstruction followed by ATM-2DH.87 14.08 Table 4. where ˆ h(fi .13 13.24 6. Based on the results from Table 4. respectively.43 5. Object DBMA Incisor Humerus Iliac 13.83 13.24 6.2.05 13.6) where a and b are elements of the original contour fi and of reconstructed ˆ contour fi .5 for the Humerus bone and Table 4.13 Table 4.97 8.21 14. However.60 2DH 11.Object DBMA Incisor Humerus Iliac 17.14 PSNR (dB) Black 13.14 13. fi ) = max min a − b ˆ a∈fi b∈fi (4. when skipping 1.21 14.24 Black 14.25 14.24 14.60 9.21 ATM-2DH 12.35 3D shape reconstruction error (%) PM 14.92 13.12 MED-2DH 10.12 12. The Hausdorﬀ distance calculated between the contours of the horizontal projections of the 3D shapes.15 7.6 for the Iliac bone.08 ATM-2DH 14.92 13. 5 and 9 slices is provided in Table 4.59 10. from the 104 .81 12.

4: Average PSNR of original middle slice reconstructions after removing 5 intermediate slices with LK as the initialisation.94 23.92 37. the Iliac bone is an odd shape with lots of indentations.82 11.30 34.82 11.66 20.60 Table 4. PM and Black give better reconstruction results compared to the Hessian based kernels.07 12.50 29.69 ATM-2DH MED-2DH 16.Object LK Incisor Humerus Iliac Knee Chest Heart 17.36 34.97 21. a hole at the bottom and a large groove at the back which is 105 .52 28.63 34. which is independent of the number of skipped slices.53 13. it is diﬃcult to deduce a meaningful conclusion from the results in Table 4.66 10.72 23.62 28.31 Hausdorﬀ distance Black 7 11.94 23.06 12.53 13.22 37.98 37.82 20.72 34.57 Table 4.81 37.74 2DH 16.81 14.21 12.20 MED-2DH 7.89 23. Slices skipped 1 3 5 9 DBMA 8.74 20.84 37.81 20.79 20.13 23. The main reason for these results is the consistent error in the reconstruction of the end slices. Relating to the Iliac bone. From Figure 4.52 28.04 14.42 ATM-2DH 8 12.89 13.60 13.59 24.79 34.98 37.74 Black 17.92 29. Hausdorﬀ distance results in Table 4.77 PM 7 11.31 2DH 8.89 13.5: Hausdorﬀ distance for Humerus bone.97 34.79 28.6.02 Peak Signal-to-Noise Ratio (dB) PM 17.94 13.19.66 10.66 16.5.

For every 10th slice that is used.80 Hausdorﬀ distance Black 13.81 11. i.Slices skipped 1 3 5 9 DBMA 12.10 18.80 MED-2DH 16.38 21.03 2DH 15. DBMA and PM (only for skip 1) are the nearest shape match to the original contour. noise in the background (beyond the structure of the object in the slice) have been removed or signiﬁcantly reduced.3) is only applicable for slices that have been segmented.6: Hausdorﬀ distance for Iliac bone.14 62. A probable solution to this is to perform shape segmentation in the 3D space.23 22. The 3D shape reconstruction rate in equation (4.10 ATM-2DH 17.89 71. expected due to its biological function.81 14.76 22. Since the three soft tissue data sets have not been pre-processed.55 13.e.65 14.97 16.67 22. which should yield more realistic error measurements.13 45.08 PM 12. With every alternate and every third slice being skipped.40 48. The 106 .40 26.80 15. This shows the strength of the algorithm to reconstruct an accurate shape.62 66. The Iliac bone reconstructed from MED2DH smoothed structural ﬂows is the closest match to the original contour compared to other methods. causing the shape error to be higher than it should really be.89 18.45 Table 4. The signiﬁcant diﬀerence in performance here is due to the Hessian based kernels over compensating to reconstruct the concavity of the bone. the results did not favour the Hessian based kernels with Black’s method coming out on top. the only measurement that bears any signiﬁcance is the PSNR (4.4). with the Hessian based kernels showing errors in reconstruction around the hole.

This result is expected since PSNR is intensity based measurement and the LK initial ﬂow does not infer any intensity changes on the reconstructed slice. 4. The results reported here are similar to the results obtained for structural ﬂows initialised by DBMA.23 shows the stack of aligned contours for the original 3D bones projected onto the horizontal plane together with those reconstructed using the structural ﬂows smoothed by various diﬀusion algorithms. Besides the proposed DBMA algorithm.comparative results for all six data sets with the structural ﬂows initialised using the LK algorithm [3] is presented in Table 4. Figure 4. All these results highlight the advantages of using the proposed 3D interpolation methodology particularly when applying robust diﬀusion kernels to structural ﬂows. structural ﬂows were also initialised using the LK algorithm. The contours correspond to 3D volumes that have been reconstructed after eliminating 5 intermediary slices and thereafter interpolating them by using smoothed structural ﬂows. A bi-directional correlation algorithm between pairs of image slices to construct structural ﬂows has been used. These structural ﬂows show the correspondence in the internal structure and among features for a given 3D object. highest PSNR is achieved by reconstructions from the initial ﬂow. The positive eﬀect of robust diﬀusion structural ﬂow smoothing is evident in some of these contours when compared to those obtained when using the initial structural ﬂows as provided by DBMA.4.8 Conclusion In this chapter. a new methodology for 3D volumetric reconstruction from sets of sparse cross-sections has been proposed. From the table. The 107 .

Intermediary slices are reconstructed using the smoothed structural ﬂows resulting in a 3D volumetric object. However. 2DH (yellow). Perona-Malik (green). the contours are from smoothed reconstructions using original slices (blue). ATM-2DH (cyan) and MED-2DH (black). ﬂows are smoothed by various diﬀusion algorithms from Chapter 3. A similar PSNR has been obtained for the interpolated slices for all the diﬀusion methods in Chapter 3 when used on the same data set. From top to bottom. The experimental results prove that the best outcome is achieved when using the dual directional structural ﬂows smoothed by robust local Hessian based kernels.23: Reconstruction of bone contours for Humerus and Iliac bones. This problem can be resolved by using additional post-processing algorithms to smooth the resulting 3D shape surfaces. DBMA (red). the surface of the resulting 3D shapes is not always smooth. Black (magenta). provided the structural ﬂows have been initialised using feature based method.(a) Humerus Bones (b) Iliac Bones Figure 4. 108 . Both shape structure and grey-level texture is reconstructed according to the structural ﬂows and existing slices.

turbulence is represented by multitude of swirling motions around either a single or multiple vortex cores. it is numerically assumed that at the centre of the vortex core. Typically. turbulence is characterised by chaotic and stochastic property changes. the pressure is minimum in all directions [100]. Hence forth. In vortex dynamics. This includes low momentum diﬀusion. it has 109 .Chapter 5 Robust Physics based Diﬀusion Solver 5. high momentum convection. Additionally.1 Research Objective In ﬂuid dynamics. and rapid variation of pressure and velocity in spatial and temporal space. subsequent references to complex and turbulent motion in this thesis will refer to a combination of swirling motions of the object in the scene and noise/camera movement introduced by the video capture. A robust Hessian based diﬀusion algorithm has been presented in Chapter 3 for smoothing optical ﬂows from video sequences.

5. 112. 98. A methodology for detecting speciﬁc movement patterns such as vortices from smoothed vector ﬁelds is developed in this chapter as well. etc. The Navier-Stokes equation has been frequently used to model the dynamics of ﬂuid ﬂow in computational ﬂuid dynamics. 128]. for simulating the behaviour of natural phenomena such as air.2 Introduction It can be safely said that modelling the optical ﬂow from motion in ﬂuid image sequences can be an onerous task. The proposed method is an improvement on the Stable Fluid Solver proposed by Stam [97. Such 110 . 127] by robustifying the diﬀusion stage. in computer graphics [97. blood ﬂow analysis in angiography [130]. existing methods perform ineﬃciently. The Navier-Stokes equations in itself are complex and are traditionally used to model the movement of heat and ﬂuid in complex structures. in the case of deformable objects and ﬂuids. It was used to model heart movement in echocardiography [89]. The robust Hessian based diﬀusion developed in Chapter 3 is embedded in the proposed method.been shown in Chapter 4 that robust and eﬃcient smoothing of structural ﬂows can inﬂuence slice synthesisation and subsequent 3D volumetric reconstructions. However. ﬂuids. 110. an improved diﬀusion methodology that is based on the Navier-Stokes equation is developed. modelling the dynamics of turbulence [129]. standard optical ﬂow estimation methods work well when modelling the movement of rigid objects.g. In this chapter. modelling behaviour of laminar ﬂow around a cylinder [131]. clouds. modelling the radiation of stars in dust clouds in astrophysics [132]. e. etc. The techniques that have been around for a while.

ﬁne grid structures are a necessity to model the conﬁnement structure. The proposed technique is applied on artiﬁcial data and image sequences representing atmospheric and solar phenomena. for example heat and ﬂuid particles. In this chapter. we are unable to model such behaviour easily.natural phenomena are more likely to exhibit nonlinear. it has been used to model the aerodynamics of wings and rotor blades. It was used to model injection and combustion engines. However. The reason why Navier-Stokes equations is so important is that we are increasingly aﬀected by it in our daily life. 111 . Together with supplemental equations. the proposed methodology is used to smooth estimated optical ﬂows of complex and turbulent ﬂuid motion from image sequences. In order to make accurate assessments on the behaviour of such phenomena. model the traﬃc and crowd ﬂow. Distinct features such as vortices (focus of rotation) are estimated from turbulent optical ﬂows. the implementation of the Navier-Stokes equations is application dependent. its been used to forecast climate change. explicit or implicit ﬁnite diﬀerencing schemes are important since they increase the stability and reliability of the computations. model the temperature distribution of silicon chips and so on. new and innovative methods are continuosly being developed in order to make it computationally more accessible and generalised. sometimes chaotic like behaviour. as described in Chapter 3 for better eﬃciency. such as the conservation of mass and well formulated boundary conditions. Additional constraints such as the initial and boundary conditions. Despite the complexity involved. an innovative smoothing method is proposed using the Navier-Stokes equations. The proposed method is embedded with a robust diﬀusion algorithm. Due to the presence of nonlinearities.

3 describes the stable ﬂuid model [97] and how it is being implemented in computer graphics and visualisation. etc [91].3 The Stable Fluid Model Navier-Stokes methodology represents the basis for modelling a large variety of phenomena such as those characterising weather. The explicit model is generally used for the precise computation of ﬂuid dynamics and involves heavy computational complexity [91]. Navier-Stokes methodology has been applied in computer graphics in order to visualise and create the eﬀects given by the complex movement of ﬂuids such as coloured gases. etc [96. The von Neumann’s stability analysis. we present methods to detect the presence of vortices in turbulent ﬁelds. In Section 5. the design of aircraft and of power stations. water ﬂow in a pipe. In engineering. smoke.5.The remainder of the chapter is presented as follows.6 while Section 5. the methodology used to implement the proposed robust hybrid ﬂuid solver is presented. highlights that the implicit model based on discretisation when calculating NavierStokes equations is unconditionally stable. they are also used in the analysis of the eﬀects of pollution. clouds. Section 5. 98]. In Section 5. liquids. 5. as shown in [91]. The stable ﬂuid solver (SFS) algorithm proposed by Stam represents an implementation of the NavierStokes methodology in an implicit scheme [97. 97]. ﬁnancial derivatives. blood ﬂow.7 concludes the chapter. the air ﬂow around an aircraft wing. ﬁre. the motion of stars inside a galaxy. although it requires a complex numerical implementation scheme [97.4. 98] which has been applied 112 . ocean currents. air. Experimental results and the analysis on simulated and real-world data is presented in Section 5.

In order to achieve visual eﬀects. where each vector corresponds to a grid location.for complex graphics modelling. the change in pressure from one spatial position to another in the vector ﬁeld is negligible. momentum and energy for an arbitrary control volume [91] and is given by: ∇P ∂V = − (V · ∇) V − + ν∇2 V + f ∂t ρ (5. while ν is a viscosity constant that characterises the ﬂuid and ρ is a parameter. The NavierStokes equation for a given system is derived using the conservation of mass. i. Consequently.1) where the change of velocity V over time is represented with respect to the advection. only the modelling of motion based on the Navier-Stokes equations are considered in this work. Unlike in the original SFS approach. diﬀusion and the external forcing function f. 98]. the gradient of the pressure P . for incompressible ﬂuids it is important to enforce the conservation of mass [91] condition: 113 . In SFS.e. The pressure is assumed to be constant in the given ﬁeld and its gradient is zero. the Navier-Stokes equations are used for both density and velocity modelling in the SFS algorithm [97. the area of investigation (in this case an image or a segmented region from an image) is split into cells located on a grid and is assigned a particle to each grid location. the equation employed by the SFS method is: ∂V = − (V · ∇) V + ν∇2 V + f ∂t (5. We assume that the SFS system moves the particles around in the vector ﬁeld.2) The diﬀusion term ν∇2 V characterises ﬂuids which are assumed incompressible and Newtonian. Moreover.

3) which states that the divergence of velocity components is zero for inﬁnitesimal time steps. −∆t)) ˆ ˆ diﬀuse: V3 (z) = V2 (z)/(1 + ν∆tk 2 ) ˆ ˆ conserve: V4 = conserve(V3 ) ˆ transform: V4 = F F T −1 (V4 ) ˆ transform: V2 = F F T (V2) Figure 5. Vx + Vy ∂x ∂y ∂x ∂y (V · ∇) V = Vx (5.∇·V =0 (5.1 [97]. the ﬁrst step consists of adding the external forcing function f which determines the initial conditions in the processing cycle.1: The stable ﬂuid solver algorithm. For each iteration. The SFS algorithm proceeds to calculate the velocity components V as described in Figure 5.2). for k ← 1 to convergence or number of iterations do 1 2 3 4 5 6 add force: V1 = V0 + f ∆t advect: V2 (x) = adv(V1 (x. which corresponds to the following: ∂Vx ∂Vy ∂Vy ∂Vx + Vy . The analysis of the advection process in real physical phenomena is provided in [91].4) is 114 .4) where V = (Vx . The density of a particle is constant between iterations. Vy ). thereby the total mass of the ﬁeld is conserved within the given region. The process described by equation (5. The second step represents the advection term in equation (5.

by assuming that the motion vector of each grid cell is a particle. The requirement to set speciﬁc boundary conditions is eliminated by extending the spatial repeatability of the area under consideration and by applying FFT. It is quite impractical to model the motion from one grid location to another using diﬀerence schemes due to constraints such as computational stability and ill-posed system of equations. The third step transforms the velocity ﬁeld to the frequency domain using the Fast Fourier Transform (FFT). 115 . The diﬀusion term (fourth step) represents the decay of high spatial frequencies in the velocity ﬁeld and is computed in the Fourier domain with a Gaussian ﬁlter processing the velocity component V by using the time step ∆t and the ﬂuid kinematic viscosity ν. The ﬁnite diﬀerence implicit scheme is used here to discretise the diﬀusion term in order to obtain an unconditionally stable system [97]. The ﬁfth step enforces the local incompressibility of the optical ﬂow which requires that the amount of ﬂow entering a speciﬁc area should be equal to the ﬂow exiting that area. By doing this.known as the self-advection of velocity. This interpolated vector is then used as a basis for further computations. The ﬁnal step projects the ﬂow back from the frequency domain to the spatial-time domain using the inverse FFT transform. This algorithm was modiﬁed in [98] by replacing the FFT transformations and the processing in the frequency domain by deﬁning a set of boundary constraints on a grid-based representation of the ﬂow. the motion vector at the backtraced location is interpolated from the neighbouring four locations. these can be traced back in time with −∆t by backtracking the velocity ﬁeld. However. The advection step from the SFS algorithm is implemented using an implicit ﬁnite diﬀerence scheme.

Figure 5. The initial ﬂow can be estimated using the block matching algorithm 116 .4 Robust Hybrid Fluid Model Figure 5. a robust anisotropic kernel [133] is embedded in the diﬀusion step of the SFS.2: Robust hybrid solver. In order to improve the performance on optical ﬂows.5. particularly in image sequences displaying complex motion.2 shows a ﬂow diagram of the proposed robust hybrid ﬂuid solver. The implementation of the stable ﬂuid solver [97] provided rather poor performance in modelling turbulent optical ﬂow estimated from image sequences. This is mainly caused due to the uncertainty in the initial estimation of the optical ﬂow which produces noise.

In the proposed approach. The ﬁrst processing block corresponds to a reinforcement step and the proposed method is implemented by adding a proportion of the velocity from the previous iteration to the current velocity: V1 (t + ∆t) = (1 − w)V0 (t) + w∆tV5 (t) (5. the noisy ﬂow is diﬀused before proceeding to the advection stage.5) where V5 (t) is the motion vector from the previous iteration t. particularly in the case of image sequences representing moving ﬂuids or other complex phenomena. The SFS algorithm described in Section 5. Optical ﬂows provided by block matching or by using temporal gradient estimation are invariably noisy [14]. w ∈ (0. 1) is a weighting factor modelling the degree of the reinforcement. i. However. The optical ﬂow should have a degree of smoothing before advection can be applied.as in [14] or other motion estimation algorithms [12]. The proposed method is to implement a Hessian based diﬀusion that jointly processes the local geometry and the statistics of the local vector ﬁeld as in [133]: 117 .1.e. the algorithm described above produces unreliable estimation when applied to noisy vector ﬁelds. At the ﬁrst iteration there is no reinforcement. respectively.3 proposes to advect the initial ﬂow at Step 2 from Figure 5. w = 0. V0 (t) and V1 (t + ∆t) represent the motion vector reinforced by force at times t and t+∆t. The transfer function of the original smoothing algorithm is a Gaussian function appropriately deﬁned within the frequency domain [97].

This diﬀusion kernel is anisotropic and adapts to the local structure of the optical ﬂow. V1. The eigenvector corresponding to the largest eigenvalue of the Hessian shows the local direction of the optical ﬂow.4).i (t) is the vector at location i within a neighbourhood η(zc ). from the previous time step back to the position in the current time step [98].ˆ V2 (t + ∆t) = xi ∈η(zc ) V1. However. using a neighbourhood approximation. as implemented in equation (3. As mentioned in Section 5. The model is dependent on the initialisation and on boundary conditions 118 . In order to properly process the local statistics and eliminate outliers. This procedure involves interpolating the velocity at the grid points. this model is only concerned with the nonlinearity of the advection term from equation (5.4. ∆t in order to model complex movement usually combining rotation and translation.19). anisotropic diﬀusion does not deal properly with outliers as shown in a study provided in [133] and described in Section 3.i (t) exp[−(xi − zc )T H−1 (xi − zc )] exp[−(xi − zc )T H−1 (xi − zc )] (5.3. H represents the local Hessian. Signiﬁcant optical ﬂow transitions are automatically detected and consequently not smoothed over by the Hessian-based kernel. centred at the location zc .6) xi ∈η(zc ) ˆ where V2 (t + ∆t) is the intermediate diﬀused value.1. At the advection stage. the self-advection term represents the ability of the velocity components to move their own values from one position to another on a grid in a time step interval. the median algorithm is considered for robustifying the Hessian based diﬀusion in the neighbourhood η(zc ).

where ∆x represents the location change during the time interval ∆t. For the sake of reducing the required computation complexity. which are enforced at every stage of the computation in order to preserve the stability and integrity of the numerical calculation. This condition is also commonly known as the no-slip boundary condition. given by ∆t/(∆x)2 ≤ 1/2 [91]. The conservation of mass.3). it is absolutely imperative that the model adheres to the stability criteria. The ﬁrst condition is determined by the physical boundary and is represented by the von Neumann condition which speciﬁes the normal component of the ﬂow to the boundary surface as: ∂V ∂n =0 Ω (5. the conservation of mass is enforced after both diﬀusion and 119 . In the proposed approach. boundary conditions are speciﬁcally provided onto the grid in order to represent the physical limits of the optical ﬂow. given by equation (5. The second condition relates to the conservation of mass of the velocity ﬁeld. Ω are represented by zero values on a geometric grid. Such boundary conditions can be the result of image or motion segmentation algorithms or of the existence of a priori information about the image sequence.of the system under study. the walls of the domain. This equation implies that the wall absorbs any ﬂow particles coming towards it. In order to maintain a divergence free velocity ﬁeld for every stage of computation. There are two boundary conditions to consider. Since our proposal incorporates both explicit and implicit ﬁnite diﬀerencing schemes. should be maintained in order to ensure the incompressibility of the ﬂow.7) where Ω represents the boundary and n is its surface normal.

advection stages. The conservation of mass stage corresponds to a data normalisation process. the projection of the pressure gradient onto a divergence free vector ﬁeld is thus given as Pv = w and ∇ · w = 0.8) where w is any vector ﬁeld. similarly described in [128]. Pv = v if and only if ∇ · v = 0. V is mass conserved and divergence free (5. This is because the Gauss-Seidel relaxation method is simple to implement. This is enforced by using the Helmholtz-Hodge decomposition [97] of the velocity ﬁeld. By taking the divergence of equation (5. the incompressible ﬁeld is obtained by 120 .3) and ∇q is the gradient of a smooth scalar ﬁeld. The Poisson equation obtained in equation (5. This decomposition provides an exact solution so that the mass conserved incompressible ﬂow can be obtained by extracting the gradient of the ﬂow from the current vector ﬁeld. The decomposition can be shown to be w = V + ∇q (5. achieves fast convergence and new approximations are used instantly for subsequent computations. the Poisson equation as shown in the following equation is obtained: ∇ · w = ∇ · V + ∇ · ∇q ∇ · w = ∇2 q (5. this method is suﬃcient for the current implementation.8). Subsequently.9) is sparse and is solved using the Gauss-Seidel relaxation method. Although there are better relaxation schemes available. Using the projection method (based on the Helmholtz-Hodge decomposition) as described in [127]. P∇P = 0 [134].9) where condition (5.3) is applied.

10) as suggested by the Helmholtz-Hodge decomposition. Current Flow = Incompressible Flow + Gradient Flow Figure 5. there is no clear process to verify whether the smoothed ﬁeld is either good or bad ﬂow. it is now possible to extract salient patterns that are characteristic to turbulent ﬂows such as vortices.5 Vortex Core Detection After smoothing and modelling the vector ﬁeld as described in Chapter 2 and Section 5. For example.3. This is especially important for fuel mixing in jet thrusters [135]. Although the vector ﬁeld have been smoothed. The presence of vortices in such ﬂows could cause mas121 .3: Helmholtz-Hodge decomposition of a closed lid driven cavity laminar ﬂow. 5. This decomposition maintains the incompressibility and smoothness of the estimated velocity ﬁeld.V = w − ∇q (5.6) at the thousandth iteration is shown in Figure 5. Mass conservation is important for realistically estimating optical ﬂow of ﬂuids. the exact Helmholtz-Hodge decomposition of the closed cavity laminar ﬂow (artiﬁcial data provided in Section 5.4.

136]. Usually. it is possible to identify types of ﬂow ﬁelds from the dynamics of vortices. This expression is further simpliﬁed using equation (5.3) to obtain 122 . Therefore the detection and tracking of vortices in turbulent ﬂows is of signiﬁcant importance.13) The Q-criterion deﬁnes a spatial region where the Euclidean norm of the vorticity tensor dominates that of the rate of strain: 1 Q= [ Ω 2 where · 2 − S 2] > 0 (5. The Galilean-invariant vortex detection criteria uses the scalar velocity gradient decomposition [100. Furthermore. this is not the case in most turbulent ﬁelds. A Galilean-invariant velocity ﬁeld is assumed to be of a uniformly smooth ﬂow ﬁeld [135]. 112. the centre is characterised by a minimum in the pressure and by zero velocity.11) (5. 103. 103]: ∇V = S + Ω where S is the rate of strain tensor: 1 S = [∇V + (∇V)T ] 2 and Ω is the vorticity tensor: 1 Ω = [∇V − (∇V)T ] 2 (5.14) represents the Euclidean norm of the tensor. However.sive energy loss within the combustion chamber and subsequently cause an unstable fuel chain reaction with potential damaging consequences. Vortices in ﬂuid mechanics represent rotational structures of concentrated energy around a centre [100.12) (5.

Hence.16) is: ∆ = p2 − 4q (5. node and saddle in a topology map of a turbulent ﬂow [103].17) where the discriminant is the root of the characteristic equation (5. Both the Q-criterion and discriminant.15) is also obtained by applying the constraints for incompressibility. the implementation is based on equation (5. for the experiments. In addition to the Qcriterion. as described in equation (5.15) since we assume that the vector ﬁelds under study are incompressible. 123 . The equation (5.16). then the velocity particle is likely to be located within the vortex core region. ∆. the characteristic function of the deformation tensor for 2D vector ﬁeld is given as [103]: λ2 + pλ + q = 0 (5. If ∆ is negative. while the co-eﬃcients depend on its determinant and trace as: q = Det(∇V) and p = −T r(∇V). then it is likely to be within a saddle region outside of the vortex core boundaries.15) The Q-criterion is further decomposed into Eulerian form [100] as in equation (5.14). into equation (5. Therefore. the criterion is dependent on the mixed derivatives of the gradient velocity tensor.15) for ease of vortex region identiﬁcation. If ∆ is positive. the discriminant of equation (5.16) where λ is the eigenvalue of the tensor ∇V.Q=− ∂Vx ∂Vy >0 ∂y ∂x (5. Hence. is used to characterise critical regions of focus.3).

The resulting Qw is estimated as a weighted average of all Qi . 124 . the 1 1 1 1 weights correspond to wi = {1. on the assumption that the vortex structures are smooth and can be integrated from a suﬃciently large energy basin.These formulae are applicable for vortex detection. The current work attempts to improve this by using a running window in which several Qi using (5. window centered. For a 5−by −5 window. each calculated according to equation (5. 103]. In [100. The smoothness condition is ensured by using the methodology described in the previous sections by employing the NavierStokes equations with a robust diﬀusion step (Sections 5. symmetrically located at the extremes of a cross. 1 . depending on the inverse 2 shortest path between pixel in a neighbourhood and its center. √2 . is used for calculating each Qi . from a set of four vectors located as shown in Figure 5. A set of four vectors. The Q-criterion is notably susceptible to the presence of noise and invariably fails when the velocity ﬁeld is not Galilean-invariant. √5 .3 and 5. √5 .2 from the maximum value of ∆ topology map.4: 25 wi Qi Qw = i=1 25 (5.3 and 3. Our studies show that an appropriate δ should be about 0.15) are evaluated. The vortex structure is identiﬁed for ∆ > δ.15). where δ is an arbitrary value.4) or by using diﬀusion algorithms proposed in Sections 3. 2√2 }.18) wi i=1 where wi represents the inverse of the Euclidean distance from the window’s center to the location of each of the four vectors.4. a single point evaluation for Q was used.

e. As with the other constraints. (∆ ≤ 0.18). Hence. ∆ is used as another set of constraints for identifying vortex core regions. Subsequently. This computation is useful for segmenting speciﬁc regions especially when the ﬁeld is smooth. this criterion fails in the presence of noise. i.20) where I(·) represents an indexing function. Additional functions can be added to this process to allow tracking of vortices from frame to frame. ∆w is also computed in a similar way as to the Qw -criterion as shown in equation (5. By now. which does provide a more reliable estimation of the vortex structure. This is especially useful in weather forecasting and geo remote sensing applications. the introduction of ∆w . This is done by ﬁrst thresholding the topology map with the function 1 .2) I(∆) = 0 .2) (5. (∆ > 0. 125 . the function I is convoluted with either V or |ω| to obtain segmented vortex regions in the vector ﬁeld or vorticity map. it is expected that the regions of interest (ROI).The weighted discriminant.19) wi i=1 As mentioned in [100]. Vortex structures are segmented by correlating the evidence in both the velocity ﬁelds and the vorticity maps using ∆ > δ as a constraint. vortex regions are segmented in the index array. The computation of ∆w is shown to be 25 wi ∆i ∆w = i=1 25 (5. the discriminant.

126 . 5. where the location of each vector in the window is shown with a diﬀerent marker according to its corresponding Qi set.6 Experimental Results The experimental results presented below show the evaluation of the proposed algorithm on synthetic ﬂuid sequences and on real-world image sequences.4: Evaluation of the Qw -criterion from a 5 − by − 5 window.Figure 5.

The initial and boundary 127 . The ﬁrst part of the research results focusses on two diﬀerent types of artiﬁcial sequences. With reference to Figure 5. Figure 5.5.5: Driven-lid cavity ﬂow diagram. The air ﬂow within the cavity is modelled and observed when the top wall is moved with constant velocity as shown in Figure 5. The ﬁrst is a closed driven-lid cavity ﬂow and the second is the ﬂow of the von Karman’s vortex sheet. the walls of the cavity are marked with A (top).5.1 Synthetic data simulation set-up This section describes the experimental set-up required to simulate artiﬁcial sequences based on the Navier-Stokes equations. Closed driven-lid cavity ﬂow The ﬁrst synthetic sequence is created using the original Navier-Stokes equations [91] for a closed driven-lid cavity ﬂow. B (left).5. The synthetic ﬂow is created using the vorticitystream formulation of the Navier-Stokes equations instead of the standard velocity-pressure formulation.6. C (right) and D (bottom).

Ψ = 0.6(d) is ﬂow with Gaussian noise with variance σ 2 = 0.conditions for the four walls are given as follows: • A: Vx = V e = 5m/s. Both vorticity and stream functions can be represented in terms of velocity by ∂Vy ∂Vx − ∂x ∂y ω = |ω| = |∇ × V| = and (5. Figure 5. Figure 5.j = 0.0) of 2bars to move the lid (top wall).j − Ψ1. These are 128 . ∂P Ψ0. there is also a constant pressure applied at grid position (0. Ψ = 0.ny−2 = 0. The ﬁgure shows the snapshot of the ﬂow after a thousand iterations with time intervals of 1ms between iterations. Ψ = 0. ω = 2 ∂n (∆y)2 ∂P Ψnx−1. The sequence is corrupted with additive Gaussian noise of zero mean and test sequences when varying the noise variances are obtained as follows.6(c) is ﬂow with Gaussian noise with variance σ 2 = 0. Ψ = 0.01.1 ∂P = 0. ω = 2 ∂n (∆y)2 Ψi.j − Ψnx−2. ω = 2 ∂n (∆y)2 with ω being vorticity and Ψ being the stream functions of the velocity. • D: V = 0.ny−1 − Ψi. Figure 5. • B: V = 0.10 and Figure 5.6(b) shows ﬂow degradation after adding Gaussian noise with variance σ 2 = 0.6(a) represents the ground truth synthetic ﬁeld that visualises the movement of air ﬂow moving inside the area of a closed driven-lid cavity with a ﬁxed velocity.j = 0. = −Vy ∂y ∂x (5.0 − Ψi. ω = 2 ∂n (∆y)2 ∂P Ψi.25. • C: V = 0.22) In addition to the given boundary conditions.21) ∂Ψ ∂Ψ = Vx .

This model is used to simulate the creation of artiﬁcially complex and turbulent ﬂows which is used as ground truth for comparative analysis. The model used to create the ﬂows is shown in Figure 5. von Karman’s vortex sheet The second synthetic sequence is the von Karman ﬂow which is created using models suggested in [92].6: Synthetic closed driven-lid cavity ﬂows with noise.01 (c) σ 2 = 0. examples of degraded vector ﬁelds in order to be used for testing the nonlinear smoothing methodology.(a) Ground truth ﬂow (b) σ 2 = 0.7 and is based on the velocity-pressure formulation of the Navier129 .10 (d) σ 2 = 0.25 Figure 5.

The initial and boundary conditions for the four walls are given as follows: • A: • B: ∂V = 0. P = 0 ∂n ∂V = 0. Figure 5. 130 . With reference to Figure 5. P = 0 ∂n • C: Vx = V e = 1m/s • D: ∂V = 0. P = 0 ∂x In addition to the numerical conditions at the walls.7: Half-cylinder model for von Karman ﬂow. the default boundary constraint on the half-cylinder. the walls of the cavity are marked by A (top).1. without additional Dirichlet or Neumann conditions is simulated using Gerris Flow Solver [137] with T = 9s and ∆t = 0.Stokes equations. B (bottom). This simple model. C (left) and D (right).7. G is ∂V ∂n = 0.

The number of iterations necessary to achieve convergence is provided in the parentheses from the caption of each result of Figure 5. The MCE results provided in Table 5. SFS algorithm as described in Section 5. The results also show that MedH-SFS provides the best outcome and the vortex recovered is better located when compared to the vortices recovered using Black and MED-2DH.5. it can be observed that the vector ﬁeld modelled by SFSM is still noisy at convergence. From these results. is shown in Figure 5. The results in Figure 5.1 is after one iteration of smoothing.8(d). while the robust hybrid ﬂuid solver embedding the median of 2D Hessian diﬀusion kernel (MedH-SFS) algorithm.8(a). Figure 5. For numerical comparisons.2 Results on synthetic data Closed driven-lid cavity ﬂow Modelling results using the modiﬁed SFS (SFSM) algorithm [98] adapted for usage on vector ﬁelds is shown in Figure 5. while SFSM has been described in [98]. the mean cosine error (MCE) between the recovered smoothed ﬂow and the ground truth ﬂow is considered. For better visualisation. while the noise has been signiﬁcantly reduced in the other smoothed vector ﬁelds.8.4.01.3 was adapted from [97]. Both these 131 .26).6. The MCE is calculated as in equation (3. the vector from the upper-right corner of the SFSM vector ﬁeld in Figure 5.8(b).8 are obtained at convergence when the mean square error diﬀerence between vector ﬁelds at two successive iterations is less than 0. while vector ﬁeld smoothing using Black’s anisotropic diﬀusion algorithm [7] is shown in Figure 5.8(a) has been rescaled.8(c) shows the eﬀects of using MED-2DH which is a robust Hessian based diﬀusion algorithm described in [133]. as described in Section 5.

25 0.4958 MED-2DH 0.4184 Table 5.7525 0.01 0.(a) SFSM (6) (b) Black (3) (c) MED-2DH (4) (d) MedH-SFS (5) Figure 5.30 0.6997 0.7383 0.5567 0.6058 0.5616 0.4624 0.7634 0.7327 0.4538 0.4005 0.5799 Black 0.6020 0.6424 0.4373 0. Gaussian Noise (σ 2 ) SFSM 0.7226 0.1: Mean cosine error of smoothed vector ﬁelds.6554 0.6704 0.10 0.6211 0.5584 0.8: Closed cavity vector ﬁeld smoothing comparisons. 132 .40 SFS MedH-SFS 0.5556 0.6849 0.4523 0.

However. As it can be observed. Fige ures 5.algorithms have been adapted to process vector ﬁelds. The robust diﬀusion hybrid ﬂuid algorithm MedH-SFS provides better results than either SFS or SFSM methods in terms of MCE when considering additive Gaussian noise.10 on each of the x and y axes. Figure 5. The noisy turbulent vector ﬁeld produced by the von Karman ﬂow is smoothed using the median of the Hessian Hybrid Fluid Solver algorithm (MedH-SFS) described in Section 5.9(b) and is used as ground truth information for comparative purposes. 6] adapted for use on optical ﬂows.3) provided in [97. SFSM and TD.4. as it can be observed from Table 5.9(a) is shown in Figure 5. 127]. its performance deteriorates signiﬁcantly when the noise increases.10(a)-(c) represent the smoothed ﬂow obtained at convergence using the MedH-SFS.9(c) shows the vector ﬁeld corrupted by adding Gaussian noise distribution with variance 0. MedH-SFS is also consistently better than Black [7] and MED2DH [133] anisotropic smoothers.1. respectively (all three results have been subsampled by a factor of 2 along both x and y directions). SFS (Section 5. the SFSM smoothed ﬂow is visually the best smoothed ﬁeld. It can be observed that SFS performs reasonably on a vector ﬁeld corrupted with low noise variance. SFSM algorithm described in [98] and the Hessian-based diﬀusion algorithm of Tschumperl´ and Deriche (TD) [5. However. SFSM took 4 iterations to converge and has over- 133 . von Karman’s sheet The velocity ﬁeld estimation from the area cropped in Figure 5. because the corrupted vector ﬁeld departs signiﬁcantly from the Navier-Stokes underlying model.

smoothed the ﬂow.(a) Initial von Karman ﬂow (b) Motion ﬂow extracted from the von Karman ﬂow (c) Noisy von Karman ﬂow with σ 2 = 0.10 Figure 5.9: Representing von Karman ﬂows. the TD smoothed ﬂow (only 1 iteration. 134 . The result shown could only be made visible after rescaling by a factor of 8. In contrast.

10: Smoothing noisy von Karman ﬂows.(a) Smoothed MedH-SFS ﬂow (b) Smoothed SFSM ﬂow (c) Smoothed TD ﬂow Figure 5. 135 .

2 presents numerical analysis results for the eﬀects of smoothing von Karman ﬂows corrupted with a Gaussian noise of mean zero and variance 0.10 as shown in Figure 5.18) provides a clearer identiﬁcation of the four intersections of the upper and lower sides of the vortex rings (corresponding to hyperbolic regions around the center of the vortex core) when compared to the locally estimated Q. Table 5.9(c).5 is applied for ﬁnding vortex regions using the Q-criterion and the discriminant ∆.14).14). Figure 5. These results highlight the saddle-type behaviour characteristic of the ﬂow enveloping the vortex core [100]. Though the vortices have been successfully recovered (albeit not perfectly). From Figure 5.as it fails thereafter) is hardly diﬀused and remains quite noisy.14) for measuring the vorticity and Figure 5.11. The high values in the map correspond to the 2D stable manifold characterising the regions where the Euclidean norm of the vorticity tensor dominates the rate of strain (5. MedH-SFS does diﬀuse the velocity ﬁeld quite well within 3 iterations without rescaling.11(a) represents the measure of the Q-criterion evaluated according to equation (5. it is possible that some regions of the ﬁeld may have been oversmoothed. according to (5.18). after one iteration for consistent comparison. In comparison to SFSM. it can be observed that the measure Qw from (5. The mean squared error (MSE) records the diﬀerence between the 136 . The table also presents the number of iterations required to achieve convergence. The method described in Section 5.11(b) represents the locally weighted Qw -criterion calculated as in equation (5. while lighter colours in the ﬁgure correspond to smaller dispersions of the vortex energy.

11: Evaluating Q and Qw on the von Karman ﬂow using equations (5. The MSE plot after smoothing various von Karman ﬂows with noise of increasing vari137 .9(a) and the smoothed ﬂows.(a) Q-criterion (b) Weighted average Qw -criterion Figure 5.14) and (5. as shown in Figure 5. ground truth. respectively.18).

according to ∆ > δ.37 3. 0.ance.25 0.82 MedH-SFS 5 2.10.12: MSE after smoothing the noisy von Karman ﬂow.80] is displayed in Figure 5. σ 2 ∈ [0.60 0.E (%) 6 3.37 1.50 0.677 0. The constraints.10 0. SFSM No.01.2: Evaluation of smoothed noisy von Karman ﬂows when the noise variance is 0. where δ is approximately twenty percent of 138 .456 0.30 0.05 0. ∆ and ∆w are used to detect the regions of vorticity.01 0.13 TD 1 13.197 1.15 0.20 0. From this plot it can be clearly seen that SFS performs better than the other algorithms with respect to the reduction of MSE.894 1. Qw .77 6. iterations MSE MCE ∆E (%) ∆w.00 Noise variance Figure 5.12.80 1.282 0. 14 12 Mean Square Error 10 8 6 4 2 0 SFSM SFS MedH−SFS TD 0 0.29 1.842 1.53 Table 5.73 SFS 1 0.828 1.917 0. Q.

2 for noise of variance 0.13: Performance of the vorticity segmentation using ∆w . MedHSFS.05 0.20 0. For example. Such vorticity regions are detected in both the ground truth ﬂows as shown in Figure 5.E is provided in Table 5.50 0. As it can 139 .01). the diﬀerence between ∆ and ∆w is their sensitivity to noise. the initial ﬂow was corrupted with Gaussian noise (variance = 0.30 0.10 0.14 shows the topology map for ∆ and ∆w .00 Noise variance Figure 5. In this example. TD and SFSM.14(b) as well as in the smoothed ﬂows. the percentage of mis-classiﬁcation and of false alarms is measured for the vorticity regions for all the tested algorithms SFS.60 0.80 1. the maximum discriminant (∆) value. The regions of vorticity characterised by high ∆ are segmented and compared between the segmented regions in both the smoothed and the ground truth ﬁeld. ∆w.01 0.E (%) 6 5 SFSM SFS MedH−SFS TD 4 3 2 1 0 0 0. Similar to the Q and Qw criterions. Figure 5.9 8 7 Segmentation Error. calculated as a percentage of the total number of vectors in the vector ﬁeld.25 0.10.15 0. computed from a MedH-SFS smoothed ﬂow after 1 iteration. Consequently.10(a)-(c). The vorticity error measure denoted as ∆E and ∆w. including those from Figures 5.

However. This is veriﬁed by the MCE results 140 . Thereafter. The mean vortex segmentation error measurement is calculated as N2 1 ∆E = 2 N i=1 |I(∆O ) − I(∆)| N2 × 100 (5. generally MedH-SFS outperforms the other methods. the vortex regions in Figure 5. ∆w. I(∆O ) is the index array of the discriminant. calculated for ∆ > 20% of its maximum value for the smoothed ﬂows. its performance gradually gets worse than MedH-SFS. The plot in Figure 5.E gives better results than ∆E when the variance is between 0.14(b) can easily be identiﬁed with the weighted discriminant. The same equation can be used for the weighted discriminant.E .13 shows the performance of vortex segmentation for various algorithms when increasing the noise variance. ∆w . it should be noted that ∆w.be observed.01 and 0.23) i=1 I(∆O ) where N 2 is the size of the vector ﬁeld. the vortex regions in Figure 5. However. This behaviour is expected for higher noise since there is a larger likelihood of segmentation mis-classiﬁcation due to larger ∆w values. ∆E is signiﬁcantly better. Though SFS has the least error initially.14(a) is visibly faint. as the amount of noise is increased. The overall observed results show that among the smoothing methods tested. calculated for ∆ > 20% of its maximum value for the ground truth ﬁeld and I(∆) represents the index array of the discriminant. This method provides smooth ﬂows and succeeds in recovering all 5 vortex core regions (from the cropped ﬁeld) after 1 iteration.20.

(a) ∆ (b) ∆w Figure 5.19).17) and (5. respectively. 141 .14: Evaluating ∆ and ∆w on a smoothed von Karman ﬂow from equations (5.

at/index en.15(c) and (d). TD smoothed ﬂow does give a reasonable value for segmentation error.2. 5. Figure 5.15(e) and (f) show the smoothing result when using MedH-SFS algorithm on the optical ﬂow estimated from the Tornado sequence and from the Solar Flare 1 Sequence obtained from http://www.15(a) represents a frame from the Tornado image sequence. respectively. while Figure 5. However.solobskh.ac.presented in Table 5.15(b) shows a frame from the Solar Flare sequence obtained from Kanzelh¨he Obervatory’s solar and o environmental research website1 . Unlike the ﬂuid based methods. The complexity of the motion in the scenes as well as the compression artefacts inﬂuence negatively the performance of the BMA algorithm. despite bad ﬂow estimations. The ﬁrst sequence represents a complex atmospheric phenomenon while the second image sequence is used to observe and analyse solar surface activity. This is possible due to vectors being in the correct location for segmentation purposes.2. The initial optical ﬂows have been estimated using block matching algorithm (BMA) and are shown in Figures 5. It is important to note that reconstructions from TD smoothed ﬂows were very poor.3 Real image sequences and remarks The proposed methodology of hybrid ﬂuid smoothing is tested on optical ﬂows estimated from image sequences. TD was unable to smooth the ﬂows reliably and hence could not recover the vortex regions properly.php 142 . hence reﬂecting the high errors in Table 5. These error measurements are based on the magnitude and direction of the vector. which shows that the MedH-SFS smoothed ﬁeld is the best match to the ground truth ﬂow.6. Figure 5.

(a) Original frame 341 (b) Original frame 220 (c) Initial BMA ﬂow (d) Initial BMA ﬂow (e) MedH-SFS smoothed ﬂow (f) MedH-SFS smoothed ﬂow Figure 5.15: Smoothing optical ﬂows in image sequences displaying turbulent motion. 143 .

As it can be observed. Nine main concentrations of energy (vortices) are identiﬁed on this map and their location is marked with distinctive numbers. Another possible cause of the noise is the random bursts of clouds in the region. a concentration of detected vortices can be observed. The movement of clouds from frame to frame is not very clear when using the block matching algorithm.optical ﬂow.navy.15(e). Turbulent movements of the solar surface can be properly identiﬁed in Figure 5. The proposed methodology was also applied on real image sequences showing atmospheric phenomena such as those that occurred during the Superstorm Andrea. respectively. using the block matching algorithm.2 The data set consists of 24 images acquired at 30 minute intervals per frame covering a total period of 12 hours.15(f).18(b) and Figure 5. 2 Sequence obtained from http://www. Frame 22 from the Superstorm Andrea image sequence is shown in Figure 5. Figure 5.nrlmry. while the optical ﬂow is calculated.mil/sat products. the ﬂow is very noisy and is inﬂuenced by the map information as well as by the ﬂickering of lights (as this is a night satellite image capture) and by additional geographic information. both after one iteration. between frames 20 and 22 is shown in Figure 5.18(c) show the eﬀect of smoothing using MedH-SFS and SFSM algorithms.18(a).html 144 . respectively. On the right side of the map (over the Atlantic Ocean). Figure 5.16(a). We can clearly identify moving twister and its boundaries after using the proposed methodology as it is shown in Figure 5.16(b) shows the topology map of the 2D discriminant ∆ extracted from the image sequence showing the turbulent storm. The satellite image sequence of the Superstorm Andrea contains 24 images and captures the formation of the Superstorm Andrea. The recovered optical ﬂow smoothed with MedH-SFS is a signiﬁcant improvement over the initial optical ﬂows.

145 .(a) Original reference frame 22 of Superstorm Andrea (b) Vortex detection from the smoothed Andrea ﬂow using MedH-SFS based on ∆ topology Figure 5.16: Finding vortices in the Superstorm Andrea sequence.

(a) Vortex detection from the smoothed Andrea ﬂow using Black’s algorithm based on ∆ topology (b) Vortex detection from the smoothed Andrea ﬂow using MED-2DH based on ∆w topology Figure 5. 146 .17: Additional results of vortex segmentation of Superstorm Andrea sequence.

147 .18: Smoothing optical ﬂow corresponding to cloud movement for Superstorm Andrea.(a) Initial ﬂow from frame 20 to frame 22 (b) Smoothed ﬂow using MedH-SFS (c) Smoothed ﬂow using SFSM Figure 5.

An interesting point to note is that some of these high energy regions do not mimic vortex behaviour. In addition to the vortex region identiﬁcation for the Superstorm Andrea sequence. From these results. This shows that the ∆w produces a clear improvement over ∆. Further results showing topology maps of ∆ and ∆w are shown in Figures 5. the algorithm is also tested on the Solar Flare sequence. The segmentation shown in (a) is obtained from Black [7] smoothed ﬂow. as shown in Figure 5. the segmentation in (b) is obtained from MED-2DH smoothed ﬂow. the segmentation is clearer and vortex core regions can be clearly identiﬁed. they are either converging or diverging from points which are the focus of particle movements. 148 .19(a).This area corresponds to the region of main activity for the Superstorm Andrea.19(b) correlates to the turbulent motion observed in Figure 5.17(a) and (b) respectively.19. The regions of high energy in Figure 5. These results show clearly that the region of principal turbulent activity. The proposed measure of vorticity detection using ∆w proved to give reliable and accurate estimates of the areas with signiﬁcant turbulence. as recorded by the movement of clouds observed in this satellite image sequence takes place mainly above the oceans. the beneﬁts of using a robust diﬀusion stage integrated in the computational ﬂuid dynamics methodology such as that implemented by Navier-Stokes equation is that it not only smoothes the vector ﬁeld. In contrast. Instead. The main diﬀerences between the results shown is that when using ∆w . but also contributes to detecting vortices and other regions of concentrated turbulence.

19: Finding vortices in the Solar Flare sequence. 5.7 Conclusion A ﬂuid dynamics based model for smoothing optical ﬂow from image sequences representing complex and turbulent ﬂows has been presented. The SFS method was originally used for computer graphics and visualisation applications. The Stable Fluid Solver (SFS) method is based on the Navier-Stokes equations.(a) Smoothed MedH-SFS ﬂow after 6 iterations (b) Vortex detection from the smoothed ﬂow in (a) based on ∆ topology Figure 5. In this thesis. the SFS method has been modiﬁed for the 149 .

No additional physical conditions could therefore aﬀect the ﬂows. the kernel embeds robust statistics capability by reducing the impact of outliers and thus enhancing the smoothness of the resulting optical ﬂow. An explicit diffusion model is used after the reinforcement stage and before the advection stage. diﬀusion. An important point to note is that the optical ﬂows used for the experiments do obey incompressible and Newtonian assumptions. In this thesis. we propose to use weighted estimates of these measures. an approach for identifying salient patterns such as vortices (coherent structures) is used. This type of kernel ensures that smoothing occurs along the structure of the motion ﬁeld maintaining the moving objects boundaries and the main optical ﬂow features. since it is dependent on the motion estimation algorithm and the boundaries are consequently deﬁned.5 to identify. advection and mass conservation. as described in Section 5. the ﬂows themselves are constrained by the boundaries of the ﬁeld at every snapshot. The motion is estimated following a set of processing steps implementing reinforcement. the volume given by the number of vectors of the ﬂows is conserved. Vortices are spatial regions characterised by rotational movement where the Euclidean norm of the vorticity tensor dominates that of the rate of the tensor. Although optical ﬂows are obtained from 3D real time scenes with changeable conditions. Moreover. The median of the local Hessian-based kernel is considered for the diﬀusion stage. After modelling the optical ﬂow using the proposed method. This method uses the Q-criterion and the discriminant ∆.speciﬁc use on optical ﬂows displaying turbulent motion. Furthermore. segment and track vortices. calculated from all the 150 .

151 . The proposed methodology is applied on artiﬁcial vector ﬁelds as well as on the turbulent ﬂow representing natural phenomena. The proposed methodology can be applied for identifying and detecting turbulent phenomena from satellite image sequence and it can be used for tracking storms.possible combinations of four vectors distributed at the extremes of a cross from a given window.

Chapter 6 Conclusion and Outlook This conclusion chapter commences with a summary of the contributions of Chapters 3 to 5 which includes a discussion of the strengths and weaknesses of the nonlinear diﬀusion techniques. directions for future work will be discussed. 6. The proposed nonlinear diﬀusion algorithms in Chapter 3 are kernel based ﬁlters and are dependent on the information of the local Hessian.1 Robust nonlinear diﬀusion The ﬁrst contribution to the computer vision ﬁeld is presented in Chapter 3. The kernel is anisotropic due to the Hessian matrix. which provides orientation infor152 . 6. Finally. Here.1. we brieﬂy outline the our methodology for a robust anisotropic diﬀusion algorithm to smooth optical ﬂows.1 Summary of Contributions This thesis has three main contributions in the area of vector ﬁeld diﬀusion.

The diﬀusion algorithms were tested on synthetic vector ﬁelds and on optical ﬂows extracted from real-world image sequences displaying various types of motion. are then used to 153 . since these describe better the deformations between image slices of soft and hard tissue of human anatomy and can be used on medical images in general. The ﬂows are also able to capture the direction of textural change within the inner structure of the objects. 7. 34]. In contrast. The Hessian based kernels are made statistically robust using the median or the alpha-trimmed mean algorithms for better performance. as proposed in Chapter 3. [72] which uses optical ﬂows. the proposed diﬀusion kernels are primarily designed for use on optical ﬂows from image sequences displaying object deformation over time.mation of the data structure and hence copes better with identifying edges and speciﬁc data transition patterns. 6. 72].1.2 3D volumetric slice interpolation 3D slice interpolation technique has been around for a while and has previously been used for 3D volumetric reconstructions [14. there are many diﬀerent types of techniques being used to obtain intermediate slices for the reconstruction. The algorithms were compared against various other methods [4. the proposed contribution in Chapter 4 is bi-directional and thus employs additional information to appropriately model the morphing between two original slices. We redeﬁne optical ﬂows as structural ﬂows. One example being the work of Weng et. 6. Robust diﬀusion kernels. Furthermore. although the kernels can be adapted for use on images as well. al. In contrast to other known methods. Most ﬂow algorithms are uni-directional.

remove outliers. A new approach based on Navier-Stokes equations is used for modelling on optical ﬂows.1. Main reasons for the slow adoption of this method in computer vision is due to its nonlinear characteristics. Currently. The main purpose of using this method is for better estimation of ﬂows from image sequences when complex and turbulent motion 154 . The proposed framework has been tested on three hard tissue data sets and three soft tissue data sets. 6. The collated slices. only volumes of segmented sets can be rendered.e. Navier-Stokes based methodology has been used in computer graphics for simulating various natural eﬀects. The data sets were a mixture of pre-segmented and unsegmented slices. including the originals are then stacked to reconstruct a 3D volume. diﬃculty in understanding. Humerus and Iliac bones. The Hausdorﬀ distance measure is used to give a better representation on the 3D shape reconstruction of the volume. Intermediate slices are then obtained using the information from the structural ﬂows and original slices. Incisor. especially around the boundaries in order to give better and more accurate morphing.3 Diﬀusion based on ﬂuid dynamics There is a huge amount of literature dedicated to using Navier-Stokes equations in the ﬁeld of ﬂuid mechanics. Three measurements were taken to validate the volumetric reconstructions. i. as suggested in [72]. complex implementations and its heavy computational complexity. They are average 3D shape reconstruction error of intermediate slices. average peak signal-to-noise ratio of intermediate slices and the Hausdorﬀ distance of a 3D synthesised volume compared to the ground truth information.

The diﬀusion ﬁlter is anisotropic in design and the results obtained are comparable to standard diﬀusion algorithms. the resulting smoothed optical ﬂows from the experiments are still fairly coarse when compared to specialised anisotropic diﬀusion ﬁlters.2 Outlook This thesis presents three diﬀerent contributions to the ﬁeld of computer vision. Although the proposed algorithm is fast and eﬃcient. the dynamics of a vortex is of immense interest. we propose a computationally eﬀective and robust method to identify vortices in image sequences showing movement of turbulent ﬂuids. Since optical ﬂows are constrained by construction. for example. 136] available to identify the vortex structures. For structures where a smooth stream ﬂow is needed. As such. It is suggested that the Hessian based 155 . continuation of the research presented is vital for continued improvements of methods and possible contributions to future applications. Though there are numerous measurements [100. since this enables engineers and researchers to identify weak structures or potentially damaging natural elements. etc. sequences showing storm motion in satellite imagery.is present. perturbations of the ﬁeld is kept to a minimum to avoid breakdown of the overall ﬂow. and a robust outlier mechanism. it is assumed that they exhibit incompressible and Newtonian properties. 102. Vortex detection is an important concept. explosive sequences. for the modelling of vortex ﬂow. An improved vortex detection method is also detailed in Chapter 5. and to the advancement of the ﬁeld. 6. Hence. The proposed method incorporates both explicit and implicit diﬀerencing schemes.

it is much better to develop a new optical ﬂow technique which is based on the Navier-Stokes equations with an integrated regularisation mechanism. There is ongoing work on this technique [24] and is based on the energy minimisation scheme of Horn and Schunk [2]. facial feature detection and computer graphics. solar phenomena. 156 . human motion analysis and cloth dynamics. It is anticipated that this technique is better suited to modelling the motion of complex and turbulent scenes such as waterfalls. hurricanes and twisters (from satellite imagery). gait analysis. This is especially true in the area of motion estimation. such as climatology. Using Navier-Stokes equations in computer vision is still a new concept. It is thought that ﬂuid based optical ﬂows has many potential applications in other ﬁelds. weather forecasting. this method is for segmented hard tissue based structures. Currently. Though the proposal in the thesis is a diﬀusion ﬁlter methodology. The methodology pertaining to 3D volume reconstruction can be improved for more accurate representation.algorithms can be improved and optimised further for better performance. although there has been some degree of success reconstructing soft tissue structures. There is scope to conduct extensive research to soft tissue based organs. The results presented in the thesis are preliminary results and further detailed investigations are needed for mainstream research.

2D intermediary slices between original slices are reconstructed from the structural ﬂows.1. The 2D slices are then stacked and aligned to reconstruct 3D volumes of the Incisor.Appendix A Dataset The original Incisor dataset of the 22 slices is shown in Figure A. Interpolated structural ﬂows are obtained from the sparse dataset with segmented foreground. 157 . as presented in Chapter 4.

Figure A. 158 .1: Incisor dataset.

List of References [1] A. [3] B. Kanade. Perona and J. Vancouver. Upper Saddle River. [6] D. Malik. 1981. Artiﬁcial Intelligence Laboratory. Technical Report A. M. Massachusetts Institute of Technology. P. Deriche. Horn and B. 2002. 572. Orthonormal vector sets regularizae tion with PDE’s and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence. July 1990. Tschumperl´ and R. 1995. Determining optical ﬂow. [4] P. 12(7):629–639. Lucas and T. Digital video processing. NJ. Prentice Hall. Canada. In IEEE 159 . Memo No. [2] B. Tschumperl´ and R. Tekalp. K. 50(3):237–252. An iterative image registration technique with an application to stereo vision. Scale-space and edge detection using anisotropic diﬀusion. pages 674–679. G. 1980. International Journal of Computer Vision. In International Joint Conference on Artiﬁcial Intelligence. Deriche.I. Schunck. Vector-valued image regularization e with PDE’s: a common framework for diﬀerent applications. [5] D.

and S. L. California. and S. 12(8):882–889. L. 2003. United States. G. Terrain analysis using radar shape-from-shading. D. Bertalmio. Black. Fleet. Osher. [11] B. [8] M. D. Suter. 29(1):59–77. Bors. R. In Proceedings of IEEE International Symposium on Circuits and Systems. and D. 1998. J. IEEE Transactions on Image Processing. 1994. Madison. and R. Ro- bust anisotropic diﬀusion. [13] A. Determining optical ﬂow. G. C. IEEE Transactions on Pattern Analysis and Machine Intelligence. Barron. Artiﬁcial Intelligence. 7(3):421–432. Marimont. volume 4. Hancock. Simultaneous structure and texture image inpainting. Schunck. Sapiro. Performance of optical ﬂow techniques. G. [9] I. Wilson. Sapiro. International Journal of Computer Vision. 12(1):43– 77. Pitas. 1981. H. June 2003. [12] J. [10] A. Vese. [7] M. Heeger. 31 May–3 June 1998. Beauchemin. Digital restoration of painting cracks. United States. J. E. 17:185–203. pages 269–272. Horn and B.Computer Vision and Pattern Recognition. Bab-Hadiashar and D. IEEE Transactions on Image Processing. Robust optic ﬂow computation. 160 . International Journal of Computer Vision. Giakoumis and I. 2003. 1998. 25(9):974–992.

Hannah. Technical Report A. 2003. August 2000. M. Poggio. Z. de la Blanca. Whitaker College. Local motion estimation from stereo image sequences. Motion estimation in the frequency domain using fuzzy c-planes clustering. Anarim. P. and E. 7(5):693–702. IEEE Transactions on Image Processing. Massachusetts Institute of Technology.I. Germany. G. Memo No. December 1986. LNCS 2652. Erdem.[14] A. Motion ﬁeld and optical ﬂow: qualitative properties. and J. al. [15] A. August 1999. Pitas. M. IEEE Transactions on Image Processing. G. [16] A. 9(8):1441– 1445.I. 161 . Lucena. December 2001.B. Peacock. G. Verri and T. 917. Memo No. M. J. pages 740–747. Perales et. In F. [19] A. 22. IbPRIA. Artiﬁcial Intelligence Laboratory and Center for Biological Information Processing. Optical ﬂow estimation and moving object segmentation based on median radial basis function network. Bors and I. E. Garrido. In Advanced Concepts for Intelligent Vision Systems. Karabulut. [18] C. Prediction and tracking of moving objects in image sequences. Yanmaz. and A. A fuzzy data fusion method for improved motion estimation. C. Fuertes. Springer-Verlag. editor. Bors and I. D.. May 1998. 10(12):1873–1879. IEEE Transactions on Image Processing. Baden-Baden.P. Renshaw. E. Pitas. J. [17] N.

Lucas/Kanade meets Horn/Schunck: combining local and global optic ﬂow methods. [25] S. Bruhn. S. 67(2):141–158. Recursive optical ﬂow estimation . In IEEE International Conference on Computer Vision. Black. A. Brox. and A. In The 19th IEEE Conference in Israel. October 2003. [27] J. Hierarchical estimation and segmentation e of dense motion ﬁelds. T. Chellappa. Koenderink. M´min and P. On the spatial statistics of optical ﬂow. Weickert.[20] H. 2006. Weickert. International Journal of Computer Vision. IEEE Transactions on Image Processing. volume 1. International Journal of Computer Vision. 46(2):129–155. 2003. Accurate dense optical ﬂow estimation using adaptive structure tensors and parametric model. Beijing. ´ [23] E. 1984. 61(3):211–231. pages 42–49. Perez. Feuer. Elad and A. Liu. 2005. [26] A. J. Jerusalem. and A. Adaptive estimation of motion using generalized cross validation. The structure of images. and C. International Journal of Computer Vision. China. [24] N. [21] M. 162 . November 1996. 2002. Didas. Roth and M. Foroosh. Biological Cybernetics. 12(10):1170–1180. Papenberg.adaptive ﬁltering approach. Schnorr. 2005. Rosenfeld. In 3rd International (IEEE) Workshop on Statistical and Computational Theories of Vision. R. [22] H. Highly accurate optic ﬂow computation with theoretical justiﬁed warping. Bruhn. 50:363–370. J. J.

and S. [29] S. B. Gourlay. T. W. Hummel. and A. Image Processing. Kimia. Weickert. B. Surveys in diﬀerential geometry: diﬀerential geometry inspired by string theory. Zucker. IEEE Transactions on Image Processing. A. [35] R. 29(1):182–193. J. Zucker. R. Hummel. 1975. 2002.-M. V. Image Vision Computing.[28] D. Image selective smoothing and edge detection by nonlinear diﬀusion. Lions. B.-T. Kimia. [34] J. [31] R. Widder. Implicit convolution. Morel. 1987. June 1986. In IEEE International Conference on Computer Vision and Pattern Recognition. Romeny. 1992. Yau. Graphics. Catte. Viergever. Computer Vision. 3:15–23. Deblurring gaussian blur. IEEE Transactions on Image Processing. P. 1985. pages 668– 671. Coll. Representation based on zero-crossings in scale space. 1999. M. [36] A. 163 . volume 5. Hummel. [32] G. H. 7(3):398–410. 1998. Yezzi. and S. Gaussian blur and the heat equation: forward and inverse solutions. 11(12):1405– 1416. and M. Academic Press. [33] F. W. A. The heat equation.-L. Unal. A. 38(1):66–80. A. SIAM Journal on Numerical Analysis. [30] R. and T. Eﬃcient and reliable schemes for nonlinear diﬀusion ﬁltering. In IEEE International Conference on Computer Vision and Pattern Recognition. Stochastic diﬀerential equations and geometric ﬂows. Krim. International Press. 1985.

Springer-Verlag. September 1996. Li. Journal of Mathematical Imaging and Vision. [38] L. Anisotropic diﬀusion pyramids for image segmentation. Deriche. In IEEE International Conference on Image Processing. Nielsen. London. University of Kaiserslautern. L. Signal and image restoration using shock ﬁlters and anisotropic diﬀusion. IEEE Transactions on Pattern Analysis and Machine Intelligence. Switzerland. 1997. [43] S. Crawford. Laboratory of Technomathematics. Florack. [42] M. In Proceedings of the 4th International Conference on Pattern Recognition. May 1993. Germany. Regularization. and R. Fast algorithms for low-level vision. A. Deriche. 7(4). [40] R. Fr¨hlich and J. volume 1. Technical Report 104. and M. March 1994. 1988. T. [39] D. Bovik.-D. M. Weickert. 1994. [41] R. scale-space and edge detection ﬁlters. 31(2):590–605. C.[37] L. Kaiserslautern. UK. 1994. Image processing using a wavelet algorithm o for nonlinear diﬀusion. A new implementation of discrete multiscale ﬁltering. Mazorra. Zhao and B. pages 383–385. SIAM Journal on Numerical Analysis. Acton. pages 697– 605. Technical Report 1893. [44] J. 12(1):78–87. Alvarez and L. Recursively implementing the gaussian and its derivatives. In IEEE International Conference on Image Processing. Cai. pages 478–482. January 1990. Some notes on repeated averaging smoothing. Lausanne. 164 . INRIA. Deriche.

In IEEE International Conference on Computer Vision. volume 2. B. 13(4):459–466. [46] C. Luck. and G. 14(9):1265–1276.-Y. A. E. K. [47] Y. A. [50] T. 2003. Archibald. and P. Carlson. and A. C.Fundamental Electronic Science. Petrovic. Rumpf. Hu. [48] P. In SPIE Conference on Visualization and Data Analysis. IEEE Press. A image model and segmentation algorithm for reﬂectance confocal images of In Vivo cervical tissue. Gelb. 165 . Caputo. 2003. [49] A. Graf. IEEE Transactions on Image Processing. R. and R. A. Vandergheynst. IEEE Transactions on Image Processing. Kanazawa and K. Bovik. IEEE Transactions on Image Processing. Richards-Kortum. 2005. O. D. Do we really have to consider covariance matrices for image feature points? Electronics and Communications in Japan part III . Farin. Kanatani. L. Motion estimation based on the direction of intensity gradient. 2003. 2004. pages 257–264. B.[45] B. Burgi. [51] R. Recognition with local features: the kernel recipe. Divorra. J. Multiresolution segmentation of natural images: from linear to non-linear scale-space representations. 22:637–653. Wallraven. Preusser and M. Extracting motion velocities from 3D image sequences. 2004. Improving the accuracy of volumetric segmentation using pre-processing boundary detection and image reconstruction. Image and Vision Computing. April 2004. 86(1):1–10.

[57] D. H. Bischof. T. M. 1999. In M. Kernel hebbian algorithm for o single-frame super-resolution. and B. Scholz. G. K. IEEE Transactions on Biomedical Engineering. and K.-R. 2004. Kechagias. Application of an adaptive control grid interpolation technique to morphological vascular reconstruction. A mathematical programming a u approach to the kernel ﬁsher algorithm. O. Binary morphological shapebased interpolation applied to 3D tooth reconstruction. and D. L. Morphology based three dimensional interpolation. Smola.[52] K. editors. [53] S. July 2000. [56] T. S. Lehmann et. Kim. In T. and V. S. a Kearns. Frakes et. IEEE Transactions on Medical Imaging. November 1999. J.-H. 18(11):1049–1075. Wang.-R. Sch¨lkopf. al. Mika. K. al. Kernel PCA and de-noising in feature spaces. M¨ller. R¨tsch. [54] S. pages 536–542. A. IEEE Transactions on Medical Imaging. B. R¨tsch. February 2003. MIT Press. MIT Press. editors. pages 135–149. Advances in Neural Information Processing Systems 13. Tresp. 166 . M¨ller. and o u G. In A. Statistical Learning in Computer Vision. 21(2):100–109.-Y. Solla. Dietterich. Cohn. Sch¨lkopf. A. Pitas. Lee and W. Bors. pages 591–597. G. G. M. Advances in Neural Information Processing Systems 11. Franz. 2001. 2002. A. and I. 19(7):711–721. editors. Leen. Survey: interpolation methods in medical image processing. IEEE Transactions on Medical Imaging. Mika. I. 50(2):197–206. [55] A. M. [58] T. Leonardis and H.

Feature-guided shape-based image interpolation. M. and A. Fletcher. M. Joshi. Sofer. Lee and C. Pizer. 21(12):1479–1489. P. [62] C. IEEE Transactions on Medical Imaging. IEEE Transactions on Medical Imaging. August 2004. Pizer. 23(8):995–1005. Thall. IEEE Transactions on Medical Imaging. IEEE Transactions on Medical Imaging. An objective comparison of 3D image interpolation methods. December 2002. S. A. [65] J. T. Udupa. Grevera and J. M. 19(9):822–834. [61] T. Seidel.[59] G. C. 2003. Fletcher. Pattern Recognition Letters. C. J. Multi-scale deformable model segmentation and statistical shape analysis using medial descriptions. S. IEEE Transactions on Medical Imaging.part I: theory. Spiral interpolation algorithm for multislice spiral CT . Principal geodesic analysis for the study of nonlinear statistics of shape. al. Marques. Interior-point methodology for 3D PET reconstruction. 17(4):642–652. P. [64] S. and J. T. Joshi. S. 1998. K. 2002. [60] S. 19(4):271–285. September 2000. S. Lin. IEEE Transactions on Medical Imaging. Sanches and J. and S. April 2000. Yushkevich. Schaller et. Joint image registration and volume reconstruction for 3D ultrasound.-Y. Marron. A. [63] P.-H. J. 167 . 24:791– 800. Johnson. A. Lu. 21(5):538–550.

Chen. [70] W. IEEE Transactions on Medical Imaging. Shape transformation using variational implicit functions. IEEE Transactions on Medical Imaging. 12(3):439–450. pages 335–342. Y. Amini. Gee. and R.[66] A. 1997.-H. Three-dimensional surface reconstruction using optical ﬂow for medical imaging. A. IEEE Transactions on Medical Imaging. Pitiot. C. Weng. In J. Penney et. Maintz. July 2004. Yang. In SIGGRAPH. June 2001. Second International Workshop on Biomedical Image Registration. 2003. [69] G. Morice. Dougherty et.-P. Piecewise aﬃne registration of biological images. [71] G. IEEE Transactions on Medical Imaging. Pierson. C. 1993. E. al. April 1999. Registration-based interpolation. O’Brien. [67] Y. 23(7):922–926. G. and A. Shape-based interpolation of tree-like structures in three-dimensional images. B. IEEE Transactions on Medical Imaging. al. August 1999. P. 168 . Y. Higgins. 18(4):359–363. and M. LNCS 2717. Validation of an optical ﬂow method for tag displacement estimation. Vannier. Fast LV motion estimation using subspace approximation techniques. and E. Turk and J. SpringerVerlag. Bardinet. Malandain. J. editors. A. 20(6):499–513. [72] N. Ritman. [68] L. Thompson. M. Wang. pages 91–101. 16(5):630–641. W. and P.

Hill. R. Acton. P. Morales-S´nchez. Hata et. 169 . IEEE Transactions on Medical Imaging. Volumetric a u motion estimation in human thorax for radiotherapy planning. D. Verd´. D. Nunez.[73] N. and D. April 2002. IEEE Transactions on Medical Imaging. Chandrashekara. Aljabar. and D. Leach. O. R. 1999. IEEE Transactions on Medical Imaging. Abramoﬀ and M. L. 2003. Sanchez-Ortiz. Non-rigid registration using free-form deformations: application to breast MR images. K. 2004. [76] N. Rao. Rueckert. J. Journal of Computer Assisted Tomography. Weruaga. al. [77] D. 2004. Rueckert. Spatial transformation of motion and deformation ﬁelds using nonrigid registration. V. Viergever. I. July 2000. [78] A. G. J. Motion gradient vector ﬂow: an external force for tracking rolling leukocytes with shape and size constrained active contours. Hayes. Sonoda. Hawkes. A. B. Ray and S. I. L. 18(8):712–721. J. C. Hajnal. Three-dimensional optical ﬂow method for measurement of volumetric brain deformation from intraoperative MR images. Computation and visualization of three-dimensional soft tissue motion in the orbit. L. IEEE Transactions on Medical Imaging. M. IEEE Transactions on Medical Imaging. 24:531–538. T. and R. Mohiaddin. 22(6):766–772. 23(9):1065–1076. [74] M. Puri. [75] L. 21(4):296–304. 23(12):1466–1478. G.

M. 24(9):1216–1230. 170 . I. A. R. and L. Meer. Nonlinear multiscale analysis of 3D echocardiographic sequences. IEEE Transactions on Medical Imaging. 21(11):1440– 1442. 2000. Sharp and E. Kadah. 1999. Smoothing low SNR molecular images via anisotropic median-diﬀusion. In IEEE International Conference on Image Processing. Sarti. and V. 2002. [81] M.-B. M. IEEE Transactions on Medical Imaging. Bovik. 1999. 2002. IEEE Transactions on Medical Imaging. 19(2):177–188. IEEE Transactions on Biomedical Engineering. R. Mikula. Density propagation for surface tracking. Hancock. and Y. Real-time speckle reduction and coherence enhancement in ultrasound imaging via nonlinear anisotropic diﬀusion. Segmenting skin lesions with partial differential equations based image processing algorithms. and F. 3(1):21–35. Pattern Recognition Letters. Medina. 18:453–466. IEEE Transactions on Medical Imaging. Garza-Jinich. L. Large deformation diﬀeomorphic metric mapping of vector ﬁelds. Youssef. [83] D.[79] Y. [84] K. [85] K. [82] A. G. Chung and G. Abd-Elmoniem. 21(4):377–384. 49(9):997–1014. Winslow. H. Z. Ling and A. Cao. K. Sgallari. Flux-based anisotropic diﬀusion applied to enhancement of 3D angiogram. 2005. 1998. Younes. M. [86] J. Sapiro. Miller. 2002. [80] N. C. P. Robust retrieval of 3D structures from image stacks. Medical Image Analysis. Krissian.

e e Dense estimation of ﬂuid ﬂows. USA. M´min. Pattern Recognition. 22(104):745–762. IEEE Transactions on Pattern Analysis and Machine Intelligence. Department of Statististics. 36:1203–1212. 1985. D. In IEEE Conference on Computer Vision and Pattern Recognition. 10(1):295–306. Y. 171 . Okazaki. Navier-stokes. K. 2001. McGraw Hill. Chung. Corpetti. Sato. In IEEE International Conference on Computer Vision and Pattern Recognition. Torii. Technical Report TR 1090. Fitzpatrick. M. Computational ﬂuid dynamics: the basics with applications. [88] J. and G. and P. H. ﬂuid dynamics. Numerical solution of the navier-stokes equations. Heat kernel smoothing and its application to cortical manifolds. M. M. Universisty of Wisconsin-Madison. March 2002. Sapiro. Anderson Jr. volume 1. H. and image and video inpainting. pages 78–81. IEEE Transactions on Medical Imaging. [92] Y. P´rez. Computation of 3D velocity ﬁelds from 3D cine CT images of a human heart. [91] J. [90] A. Chorin. 1995. Leahy. Physics-based ﬂow estimation of ﬂuids. [93] M. 2003. J. A method for calculating velocity in time dependent images based on the continuity equation. E. 24(3):365–380. 2004. L. S. October 1968. ´ [94] T. Nogawa. and S. 1991. K. Song and R. pages 355–362.[87] M. Mathematics of Computation. Tamura. [89] S. A. Bertalmio. Inomata. Bertozzi. Nakajima.

Premoze. [103] J. J. Stam. [97] J. NASA Ames. and R. 1988. ACM Transactions On Graphics. T. A. July 2003. Lefohn. 172 . Domain decomposition for variational optical-ﬂow computation. 22(3):716–723. 14(8):1125–1137. 2001. pages 401–410. Technical Report CTR-S88. A simple ﬂuid solver based on the FFT.[95] T. Journal of Graphics Tools. Hunt. and P. stream. Extracting 3D vortices in turbulent ﬂuid ﬂow. IEEE Transactions on Pattern Analysis and Machine Intelligence. Bruhn. J. Moin. T. Stam. and J. C. A. Adrian. A. August 2005. [99] A. Tasdizen. and J. Haller. Real-time ﬂuid dynamics for games. On the identiﬁcation of a vortex. and convergence zones in turbulent ﬂows. Schnorr. Computer Graphics Forum 22(3). C. Journal of Fluid Mechanics. Popovic. March 2003. 6(2):43–52. An objective deﬁnition of a vortex. Weickert. Particle-based simulation of ﬂuids. 285:69–94. Keyframe control of smoke simulations. A. Journal of Fluid Mechanics. Huang. Jeong and F. A. Treuille. 525:1–26. [96] S. Eddies. 1995. Stanford University. Whitaker. [101] J. McNamara. [100] J. Zhong. Kohlberger. pages 76–92. USA. and R. In Game Developer Conference. volume 4. Wray. IEEE Transactions on Image Processing. Stam. [98] J. [102] G. T. 1998. 20(2):193–199. Hussain. Z. 2005. S. In EUROGRAPHICS. Bigler. 2003.

Visualization and Mathematics. Hauser. editors. 2006. H. [110] J. Peikert. [107] D. IEEE Transactions on Visualization and Computer Graphics. October 2004. Springer-Verlag. Representing and visualizing ﬂuid ﬂow images and velocimetry data by nonlinear dynamical systems. 173 . Peikert. In Joint Eurographics — IEEE TCVG Symposium on Visualization. Ford and R. 13(5):980–990. Hauser. Topology-Based Methods in Visualization. October 1999. L. 2005. In H. Peikert. and E. Vortex and strain skeletons in eulerian and lagrangian frames. [109] F. Zhao. Hagen. Sahner et. 12(5):949–956. Laramee. Post. Peikert and M. Visualization tools for vorticity transport analysis in incompressible ﬂow. IEEE Transactions on Visualization and Computer Graphics. IEEE Visualization. al. and M. Sadlo. 1995. and H. pages 1–19. Sadlo. H. Bauer and R. Vortex and source particles for ﬂuid motion e estimation. M. Sick. the state of the art. volume 3459. Theisel. [105] R. May 2002. R. R. Graphical Models and Image Processing. In Scale-Space. 57(6):462–482. The parallel vectors operator . pages 263–270. S. Cuzol and E. and F. M´min. Vortex tracking in scale-space. H. Strickland. N. Vorticity based ﬂow analysis and visualization for pelton turbine design optimization. ´ [111] A. [106] R. pages 140– 147. 2007. Topology-based ﬂow visualization. Parkinson. pages 254–266. 2007.[104] R.a vector ﬁeld visualization primitive. [108] F. Roth.

Hasler. [116] A. Venetsaonopoulos. G. [113] M. G. third edition. Kluwer Academic. Jiang. Modern engineering mathematics. pages 295–309. Thompson. Pittsburgh. James. Prentince Hall. Detection and visualization of vortices. Robustiﬁed heat kernel smoothing of structural ﬂows for volumetric image interpolation. Machiraju. Nonlinear digital ﬁlters: principles and applications. Pitas and A. K. [115] I. In C. [118] G. Laﬀerty and G. 2005. Carnegie Mellon University. Lebanon. R. 174 . Zhou. pages 1225–1228. January 2004. Object classiﬁcation in 3D images using alphatrimmed mean radial basis function network. USA. 23(11):1330–1336. 2006. D. Academic Press.[112] L. and D. 1990. Norwell. GA. F. IEEE Transactions on Image Processing. Technical Report CMU-CS-04-101. 8(12):1730–1744. 2001. PA. Visualization Handbook. Kambhamettu. editors. C. B. N. Diﬀusion kernels on statistical manifolds. Hansen. In IEEE International Conference on Image Processing. Pitas. Palaniappan. Goldgof. Atlanta. Johnson and C. Doshi and A. and A. [114] J. [117] A. School of Computer Science. MA. Bors and I. Tracking nonrigid motion and structure from 2D satellite cloud images without correspondences. 2001. Bors. IEEE Transactions on Pattern Analysis and Machine Intelligence. United States. December 1999.

G. Torre. Image registration using hierarchical b-splines. 15:881–892. [120] A. K. IEEE Transactions on Medical Imaging. 1998. and J. International Conference on Computer Analysis of Images and Patterns. [125] Y. D. pages 213–221. Shape-based interpolation of multidimensional grey-level images. France. Xie and G. Sato et. 22(6):2283–2298. 2005. Thomas. Using optical ﬂow to characterize sensorymotor interactions in a segment of the medicinal leech. 21(12):1250– 1267. J. 2004. Zoccolan and V. March 2002. 2(2):143–168. S. In LNCS 3691. IEEE Transactions on Medical Imaging. Worthington and E. December 1996. Mikic. Versailles. [123] D. Hancock. IEEE Transactions on Pattern Analysis and Machine Intelligence. Krucinski. The Journal of Neuroscience. New constraints on data closeness and needle map consistency for shape-from-shading.[119] G. al. IEEE Transactions on Visualisation and Computer Graphics. [121] I. 175 . Optical ﬂow diﬀusion with robustiﬁed kernels. E. 1999. 10(1):85– 94. Udupa. [124] P. Segmentation and tracking in echocardiographic sequences: active contours guided by optical ﬂow estimates. L. Doshi and A. Medical Image Analysis. 1998. Three-dimensional multi-scale line ﬁlter for segmentation and visualization of curvilinear structures in medical images. 17(2):274–284. Grevera and J. Bors. Farin. [122] Z. R.

10:1675–1694. and M. [131] S. 1997. Computational analysis a of blood ﬂow dynamics in cerebral aneurysms from CT and 3D rotational angiography image data. Evaluation of a CFD bencha mark for laminar ﬂows. J. 1992. In International Congress on Computational Bioengineering. J. Frangi. Y. Heritage. Stable ﬂuids. Computing. In H. Periaux. Glowinski. Hern´ndez. World Science Publishing. [128] N. In SIGGRAPH. [132] S. turbulent gas. A. Zaragoza. Galanti. 1995. A multi-resolution technique for comparing images using hausdorﬀ distance.[126] A. M. Cebral. Rucklidge. Huttenlocher and W. and A. Stam. Gibbon. Computer Graphics. Technical Report TR 92-1321. Rannacher. G. Bock. P. Vorticity alignment results for the three-dimensional euler and navier–stokes equations. [127] J. Turek. A generalized mean intensity approach for the numerical solution of the radiative transfer equation. Nonlinearity. Proceedings of the 2nd European Conference on Numerical Mathematics and Advanced Applications. Turek. pages 121–128. August 1999. pages 549–563. Kuznetsov. 54(1):27–38. and R. editors. Department of Computer Science. M. 1997. Rannacher. D. 31(Annual Conference Series):181–188. Cornell University. Metaxas. and R. F. 1998. Sch¨fer. [130] J. Modeling the motion of a hot. R. R. [129] B. Spain. Foster and D. 176 . J. 2003.

Robust diﬀusion kernels for optical ﬂow smoothing. [135] A. 2002. pages 415–420. 68(4):359–377. [134] A. [137] S. Vorticity and Incompressible Flow. [136] R. 2003. Pattern Recognition. 1997. Gerris: a tree-based adaptive solver for the incompressible euler equations in complex geometries. Doshi and A. K. Popinet. Ireland. Majda and A. Prasad. Maynooth. Cambridge University Press. 177 . Agrawal and A. Bors. J.[133] A. 30(12):1991–2000. 190(2):572–600. Turbulence and Combustion. 2006. Organizational modes of large-scale vortices in an axisymmetric turbulent jet. Ford. In IEEE Workshop on Machine Learning for Signal Processing. Bertozzi. December 2001. Critical point detection in ﬂuid ﬂow images using dynamical system properties. M. Journal for Computational Physics. G. L. Flow.

Sign up to vote on this title

UsefulNot useful- IntroKernels (1).ppt
- 2
- 937241
- 10.1.1.3.4148
- Chapter 2 Linear Transformations
- opwencv-tyxz
- The Influence of Unstable Technology on Cryptography
- Intagrate and Derivate
- 07iccv_segmMRI
- An Improvement of Web browser
- Scimakelatex.4598.Ali.mahmood
- Hand-Detection-Using-Hsv-Model.pdf
- m19n41 ai
- MCAassingmnt_270116
- Introduction to Programming Exam
- scimakelatex.56319.asldfj+we.pdf
- bridges unit 4
- Algorithm Design Foundations Analysis And
- Optimized Sensor Nodes by Fault Node Recovery Algorithm
- 16
- Self-Learning Archetypes for Rasterization
- IEICE2004
- Scimakelatex.26951.John.doe.Jane.doe
- MCA Syllabus II
- IEEE 2014 JAVA NETWORKING PROJECT Iterative Localization of Wireless Sensor NetworksAn Accurate and Robust Approach
- dtw
- 1411.6228v3
- paper
- algorithm flow chat
- Programming Contest Guide
- YCST-2008-13

Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

We've moved you to where you read on your other device.

Get the full title to continue

Get the full title to continue reading from where you left off, or restart the preview.