Instructor Dr. May Wang Point of Contact Todd Stokes
PillCam Foundation and Image Retrieval
Project Report Stephen David Conover McCann Overview Problem Solution Methodology Implementation Results What is PillCam? Wireless Capsule Endoscopy (WCE) Non-invasive Low Complication Rate Out Patient Potentially End-to-End Good diagnosis and first look http://www.nch.org/about/tech.shtml Problem Diagnosis Takes a Long Time Where are we? What are we looking at? Hours required to review video
Task Time (min) ± SD
Time to read video by physician 100 ± 14 Time for nurse to prepare patients 30 ± 7 Time for nurse to begin download 10 ± 3 Time for physician to write report 10 ± 6 * Bossa, et al. "Detection of Abnormal Lesions Recorded by Capsule Endoscopy: A Prospective Study Comparing Endoscopist's and Nurse's Accuracy." Digestive and Liver Disease 38 (Aug 2006): 599-602. Solution Provide a tool to retrieve “similar” images from a database Allow reviewer to find related frames for comparison/reference Provide general 3D model as an alternative to frames Reduce review time through search Methodology Methodology Overview Metrics Symmetry Curvature Color
3D Modeling Ranking Methodology
Symmetry Finding the “Center” of the Image Rotational Symmetry Adapted from Johansson et. al, “Detecting Rotational Symmetries using Normalized Convolution.” Only looking for standard circular rotational symmetry (no other basis functions).
Johansson, Bjorn; Knutsson, Hans; Granlund, Gosta. “Detecting Rotational Symmetries using Normalized Convolution.” International Conference on Pattern Recognition (ICPR), vol 3: p. 3500. Rotational Symmetry Transform image intensities into axial and radial components. Try to find a center point that minimizes variance in one transform and maximizes it in the other. Nominal Results Methodology
Curvature Extracting the Shape of the Image Curvature Overview
Why curvature? Good metric for features
and lumen/wall views.
A B C D Process 1. Threshold 2. Extract Curvature 3. Histogram 4. Compare Thresholding Global average of intensity Simple but effective Sample results:
1 I ij = ( R + G + B ) 3 T = Avg ( I ij ) Curvature Extraction Curvature Equations (Osher and Sethien) NormGrad = ∇ X φ 2 + ∇Y φ 2 1 where normGrad > 0.75 * max(normGrad ) wij = 0 elsewhere 2 2 ∇ Xφ ∇Y φ kappa = wij ∇X + ∇Y NormGrad NormGrad 1 C = kappa
Binary Image Gaussian Kappa 1/Kappa
50
100
150
200
250 50 100 150 200 250 Base Test Case Curvature Histogram 100
Radius of 50 Curvature Curvature is Centered Around 50! Process Development and Refinement I
Initial draft in MATLAB m-code
Ported to and refined in Java Java Application for Testing: Process Development and Refinement II
Bad
Better Masked
Best Dilated & Masked Histogram 2D Curvature Reduced 1D Array Reduced to 100 bin histogram Count is divided by # pixels for scaling
Curvature Histogram
150
100 Count
50
0 0 50 100 150 200 Curvature Comparison Sum of the squares difference in histogramed values 0:10:1000: N Comparison Score = ∑ ( Bin A [n] − BinB [n]) 2
Color Overview HSI Histogram Sum Power Compare Methodology # Metrics
Final Scoring Final Score = ∑w s
k =1 i i Methodology
3D Modeling Overview 3D OBJ Model interpolated from MRI slices Basic lighting and flat color texturing Smoothed Surface Normals The surface normal at a given vertex is the average of the surface normals of the polygons meeting at the vertex. Methodology
Summary Final Rationale: How does all this work together? Implementation Implementation Overview Completely Java OpenGL and JOGL 30 files 3,000 SLOC Class Diagram - Model Modules
Single Color Sum of
Normalize Image Load Squares Curvature Present Score All Results 3D Model Navigation Symmetry Distance Tracks Rendering
Source Metric Comparison Results
Selection Extraction Features Easy to use graphical user interface Navigate manually through model with AWSD and mouse Record camera positions to a track Save and load camera tracks Quickly scan through loaded camera track Add images and directories of images to database Compare database images to 3D model or image file Quick link back to results Save/Load feature database functionality Weaknesses Thresholding Simple Simple Ranking Need higher fidelity model Slow feature extraction Results Interface Results Video Independent Model Bad Frame Frame Image Frame Bad Results Results I: Search for Image from Video
1 2
3 4 Results II: Search for Image from Independent Frame
1 2
3 4 Results III: Search for Image from 3D Model
1 2
3 4
5 Results IV: Bad Results
1 2
3 4 Summary Metrics perform well Need deformable model Review time is improved Questions?
