StructuredLightBased Acquisition(Part2)

CS635Spring2010 DanielG. G Aliaga DepartmentofComputerScience PurdueUniversity

Acquiring q gDynamic y Scenes

Scene:object(orcamera)ismovingand/orobjectisdeforming Acquisition: q capture p asmuchinformationasp possibleinonetoa fewframes
Byexploitingcoherence By B exploiting l iti several l channels h l of fi information f ti ( (e.g.,color, l i infrared, f d etc) t )

Acquiring q gDynamic y Scenes

Capturing Capturing2DDepthandTextureofTimeVaryingScenesUsingStructured InfraredLight,FruehandZakhor,PROCAMS2005
Usesingleframesandinfraredillumination

RapidShapeAcquisitionUsingColorStructuredLightandMultipassDynamic Programming,Zhangetal.,3DPVT2002
Usesingleframesandcoloredpatterns patterns

Fast3DScanningwithAutomaticMotionCompensation,Weiseetal.,CVPR 2007
Usephaseshiftingandmotioncompensationoverafewframes

Realtime3DModelAcquisition Acquisition,Rusinkiewiczetal., al SIGGRAPH2002

Usepatternsproducinglocalcorrespondencesoverafewframesandmerge

Capturing2DDepthandTextureofTime V i Scenes Varying S Using U i Structured S dInfrared I f dLight Li h

Vlines

Line defined at middle of IR strip

How do you know which line is which? Ideas?

Hlines

H-line sweeps up/down at 2Hz and enables an ordering of (a subset of) the V-lines and thus permits their correspondence

AdditionalSteps
Grab color image Foreground segmentation, and d dense d depth th interpolation Put it all together

IR camera at 30Hz, color camera at 10Hz (probably faster today)

RapidShapeAcquisitionUsingColorStructured Light Li h and dMulti M l ipassDynamic D i Programming P i

Recall:howdowe p lines? correspond

RapidShapeAcquisitionUsingColorStructured Light Li h and dMulti M l ipassDynamic D i Programming P i

Usecolortransitionstodefinefeatures DefinelinesatthetransitionsfromcolorA tocolorB

RapidShapeAcquisitionUsingColorStructured Light Li h and dMulti M l ipassDynamic D i Programming P i

Whatisanotableproblem? Resolution.Why?

RapidShapeAcquisitionUsingColorStructured Light Li h and dMulti M l ipassDynamic D i Programming P i

Only yhavethreecolorchannels( (R,G,B) , , )andcan onlyrobustlydifferentiatestrongcolorchanges Thisreducesthenumberofcolorstouse,and Oftenresultsinambiguityinthecolorcoding

RapidShapeAcquisitionUsingColorStructured Light Li h and dMulti M l ipassDynamic D i Programming P i Challenges

Givenacolorcode,howtodobest correspondthestripes? Withtheaboveinmind,howdowedesigna goodcolorcode?

RapidShapeAcquisitionUsingColorStructured Light Li h and dMulti M l ipassDynamic D i Programming P i Challenges

Givenacolorcode,howtodobest correspondthestripes? Withtheaboveinmind,howdowedesigna goodcolorcode?

Howtobest best correspondthestripes

Solution
DynamicProgramming

Howtobest best correspondthestripes?

(rectifiedimages)

Howtobest best correspondthestripes?

Howtobest best correspondthestripes?

j1 j2 jH Multiple matchhypotheses = , ,..., i i i 1 2 H
Similarityscore(ofcolor)between edgeei andtransitionqj is s ( q j , ei )
H

S Score of fthe th entire ti match t hsequence Dynamicprogrammingobjectiveis:

f ( ) = s (q jk , eik )
arg max( f ( ))
k =1

Howtobest best correspondthestripes?

Dynamic y p programming g gobjective j is:

arg g max( ( f ( ))

isverylarge:O(MN) However,thespaceallpossible
Solution? A Assume monotonicity t i it (of ( fthe th depth d thordering): d i )

i1 i2 ... iH
Great!Butthismonotonicity doesnot holdinwhatsituation? Occlusions!Ohwell But itholdsforindividualfragments,whichwecancombine

Howtobest best correspondthestripes?

Dynamic y p programming g gobjective j is:

arg g max( ( f ( ))

becalled Letoptimal
f ( * ji ) =

0 ifj=0 ori=0 f ( * j 1,i 1 ) + s (q j , ei ) * max f ( j 1,i ) * f ( j ,i 1 )

f foundthrougharecursivesearchandsomeoptimizationsto furtherreducethesearchspace(e.g.,assumeatmostsmall depthchangesfromonecolumntoanother)

RapidShapeAcquisitionUsingColorStructured Light Li h and dMulti M l ipassDynamic D i Programming P i Challenges

Givenacolorcode,howtodobest correspondthestripes? Withtheaboveinmind,howdowedesigna goodcolorcode?

Howdowedesignagoodcolorcode?
DeBruijnsequenceB(k,n) B(k n)
isacyclicsequenceofagivenalphabetA withsize k forwhicheverypossiblesubsequenceoflength n inA appearsasasequenceofconsecutive charactersexactlyonce

B(k,n) haslengthkn Example:A={0,1} A={0 1}

B(2,3)= 00010111 or 11101000

DeBruijn sequenceB( B(k B(k,n k k,n) n) n)

Canalsobeconstructedby ytaking g aHamiltonianpathofann dimensionalDeBruijngraphover k symbols;e.g., eg ( (Hamiltonian p pathmeanseach vertexisvisitedonce)

ColorSequence
Colors={000,100,110,,111} totalof81 1=7 7because 000 isuseless Colorsequence q iscreatedby ypj+1=p pj XOR dj
XORingeffectivelyflipsbitsusingdj p0 isachoseninitialcolor(e.g.,100)

Want3letterssequencesdj tobeunique Inpracticeabout125stripesissufficient Thus,aB(5,3) isadequate

Examples

Fast3DScanningwithAutomatic MotionCompensation

Higherresolution/qualitythanprevious method Usesphase h shifting hf and dmotioncompensation

Fast3DScanningwithAutomatic MotionCompensation

Whatis(standard)phaseshifting?
(moving)object

projector

camera

MotionCompensation
Sincephaseshiftingassumesastaticscene scene, correlationbasedstereoisusedto compensateformotion A Anadditional ddi i lmodification difi i i isproposed dto handlediscontinuities(whichalsoplague standard d dphase h shifting) hif i )

RealTime3DModelAcquisition
(slidesandvideosofthissectionbySyzmon Rusinkiewicz @Princeton

RealTime3DModelAcquisition Pipeline l
3D Scanner

View Planning

Alignment

Human
Done? o e Merging eg g

Display

RealTime3DModelAcquisition Pipeline l
3D Scanner

View Planning

Alignment

Challenge Challenge: Real Time

Done? o e Merging eg g

Display

RealTime3DModelAcquisition Pipeline l
3D Scanner

View Planning

Alignment

Done? o e

Part I: StructuredStructured -Light Triangulation

Merging eg g

Display

RealTime3DModelAcquisition Pipeline l
3D Scanner

View Planning

Alignment

P t II: Part II Fast ICP

Done? o e Merging eg g

Display

RealTime3DModelAcquisition Pipeline l
3D Scanner

View Planning

Alignment

Part III: Voxel Grid

Done? o e Merging eg g

Display

Triangulation
Object Laser

(x,y)

Camera

Depthfromrayplanetriangulation

Triangulation
Fasteracquisition:projectmultiplestripes Correspondenceproblem:whichstripe iswhich?

CodesforMovingScenes
A Assign i ti timecodes d tostripeboundaries Performframetoframe trackingofcorresponding boundaries
Propagateilluminationhistory
Illumination history = (WB),(BW),(WB) [HallHolt&Rusinkiewicz,ICCV2001] Code

DesigningaCode
Wantmanyfeatures features totrack: lotsofblack/whiteedgesateachframe Trytominimizeghosts WWorBB boundariesthatcantbeseendirectly

DesigningaCode
0000 1101 1010 0111

1111 0101

0010 1000

1011 0001 1100

0110

0100

1001

1110

0011

[Hall[Hall -Holt & Rusinkiewicz, ICCV 2001]

SpaceTimeBoundaryCode

Implementation
Pipeline:
Project Code Capture Images Find Boundaries Match Boundaries Decode Compute Range

DLPprojectorilluminatesscene@60Hz. SynchronizedNTSCcameracapturesvideo Pipelinereturnsrangeimages@60Hz. Hz

RealTime3DModelAcquisition Pipeline l
3D Scanner

View Planning

Alignment

P t II: Part II Fast ICP

Done? o e Merging eg g

Display

Aligning3DData
Thisrangescannercanbeusedforany movingobjects Forrigidobjects objects,rangeimagescanbealigned toeachotherasobjectmoves

Aligning3DData
ICP(IterativeClosestPoints):foreachpointon onescan,minimizedistancetoclosestpoint onotherscan scan

Aligning3DData
and diterate it t to t find fi dalignment li t
IteratedClosestPoints(ICP)[Besl&McKay92]

ICPintheRealTimePipeline
PotentialproblemwithICP:localminima
Inthispipeline,scansclosetogether Verylikelytoconvergetocorrect(global) minimum

BasicICPalgorithmtooslow( (~seconds)
Pointtoplaneminimization Projection P j ti based b dmatching t hi Withthesetweaks,runningtime~milliseconds
[Rusinkiewicz&Levoy,3DIM2001]

RealTime3DModelAcquisition Pipeline l
3D Scanner

View Planning

Alignment

Part III: Voxel Grid

Done? o e Merging eg g

Display

MergingandRendering
Goal:visualizethemodelwellenough tobeabletoseeholes Cannotdisplayallthescanneddata accumulateslinearlywithtime Standard S d dhi high hquality li merging i methods: h d processingtime~1minuteperscan

MergingandRendering

MergingandRendering

MergingandRendering

MergingandRendering

MergingandRendering

Pointrendering, rendering usingaccumulatednormalsforlighting

Example:Photograph

18 cm.

Result

Postprocessing
Realtimedisplay
Quality/speedtradeoff Goal:letuserevaluatecoverage coverage,fillholes

Offlinepostprocessingforhighqualitymodels
Globalregistration Highqualitymerging(e.g.,usingVRIP[Curless96])

PostprocessedModel

RecapturingAlignment

Summary
3Dmodelacquisitionpipelineoptimizedfor obtainingcomplete,holefreemodels Usehumans human stimemostefficiently Piecesofpipelineselectedforrealtimeuse:
Structuredlightscannerformovingobjects FastICPvariant Simplegridbasedmerging,pointrendering

Limitations
Prototypenoisierthancommercialsystems
Couldbemadeequivalentwithcareful engineering Ultimatelimitationsonquality:focus,texture

ScantoscanICPnotperfect alignmentdrift
Duetonoise,miscalibration,degenerategeometry Reduced, R d d b but tnot teliminated, li i t d by b anchor h scans PossiblycombineICPwithseparatetrackers