Northeastern University, Fall 2005

CSG242: Computational Photography

Ramesh Raskar Mitsubishi Electric Research Labs Northeastern University
Course WebPage : http://www.merl.com/people/raskar/photo/course/

CSG242: Computational Photography
Course WebPage
http://www.merl.com/people/raskar/photo/course

Google phrase µNortheastern Computational Photography¶

Welcome
‡ Understanding Film-like Photography Film± Parameters, Nonlinearities, Ray-based concepts Ray-

‡ Image Processing and Reconstruction Tools
± Multi-image Fusion, Gradient domain, Graph Cuts Multi-

‡ Improving Camera Performance
± Better dynamic range, focus, frame rate, resolution

‡ Future Directions
± HDR cameras, Gradient sensing, Smart optics/lighting

Welcome
Topics: ‡ Digital Imaging Beyond Film-like Photography Film‡ Computational Aspects of Lenses, Image Sensors and Processing ‡ Algorithmic Solutions for Camera Sensor, Lens and Lighting Limitations ‡ Adaptive Fusion of Multiple Images for Impossible Photos ‡ Image Reconstruction from Coded Images Samples ‡ Future Directions in Smart Lighting, Optics and Sensors

Goals ± Understand computational aspects of cameras ‡ Discuss issues in traditional cameras ‡ Explore alternative imaging methods ‡ Learn vision and optics techniques ± Discuss image processing and reconstruction tools ± Review of 30+ recent papers .

Color Film Cameras. Aesthetics. Storage ‡ Novel view rendering (IBR) . Artistic filters ‡ Digital camera user manual.Goals ± What we will not cover ‡ ‡ ‡ ‡ ‡ Composition. film issues Lighting equipment Color issues Traditional image processing/editing (Photoshop) ± Histogram.

His research interests include projector-based graphics. He has published several articles on imaging and photography including multi-flash photography for depth edge detection. CVPR and many other graphics and vision conferences. MA in May 2005. IEEE Visualization. He is a panel organizer at the Symposium on Computational Photography and Video in Cambridge. computational photography and non-photorealistic rendering.merl. http://www. His papers have appeared in SIGGRAPH.html .com/people/raskar/raskar. 2003 and 2004.Instructor: Ramesh Raskar Ramesh Raskar is a Senior Research Scientist at MERL. He was a course organizer at Siggraph 2002. gradient-domain imaging and projector-camera systems. image fusion. EuroGraphics. He is a member of the ACM and IEEE.

sensor and lighting limitations Computing merges results from multiple images Computing reconstructs from coded image samples Cameras benefit from computerized light sources ± Think beyond post-capture image processing post‡ Computation well before image processing and editing ± Learn how to build your own camera-toys camera- .Opportunities ± Unlocking Photography ‡ How to expand camera capabilities ‡ Digital photography that goes beyond film-like photography film- ± Opportunities ‡ ‡ ‡ ‡ Computing corrects for lens.

Traditional Photography Detector Lens Pixels Image .

Computational Photography: Photography: Optics. Sensors and Computations Generalized Sensor Computations Ray Reconstruction Upto 4D Ray Sampler Generalized Optics 4D Ray Bender Picture .

Computational Photography Novel Cameras Generalized Sensor Generalized Processing Optics .

Computational Photography Novel Cameras Generalized Novel Illumination Light Sources Sensor Generalized Processing Optics .

Computational Photography Novel Cameras Generalized Novel Illumination Light Sources Sensor Generalized Processing Optics Scene: 8D Ray Modulator .

Computational Photography Novel Cameras Generalized Novel Illumination Light Sources Sensor Generalized Processing Optics Display Recreate 4D Lightfield Scene: 8D Ray Modulator .

Computational Photography Novel Cameras Generalized Novel Illumination Light Sources Modulators Generalized Optics Sensor Generalized Processing Ray Reconstruction Upto 4D Ray Sampler Optics 4D Ray Bender 4D Incident Lighting 4D Light Field Display Recreate 4D Lightfield Scene: 8D Ray Modulator .

A Teaser: Dual Photography Projector Photocell Scene .

A Teaser: Dual Photography Projector Photocell Scene .

A Teaser: Dual Photography Projector Photocell Scene .

A Teaser: Dual Photography Projector Photocell Camera Scene .

The 4D transport matrix: Contribution of each projector pixel to each camera pixel projector camera scene .

Siggraph 2005 .The 4D transport matrix: Contribution of each projector pixel to each camera pixel projector camera scene Sen et al.

The 4D transport matrix: Which projector pixel contribute to each camera pixel projector camera ? scene Sen et al. Siggraph 2005 .

Dual photography from diffuse reflections the camera¶s view Sen et al. Siggraph 2005 .

org/kerrmaya.famsi.Are BOTH of a µphotograph¶? http://research.html Rollout Photographs © Justin Kerr: Slide idea: Steve Seitz .

G.New Ways of Seeing the World ³Multiple-Center-of³Multiple-Center-of-Projection Images´ Rademacher. SIGGRAPH '98 .. P. Bishop.

edu/~andpph/ . RIT: RIT: http://www.rit.What rays are most expressive? Andrew Davidhazy.

1998.Thick photography: interaction What other ways better reveal shape to human viewers? (Without direct shape measurement? ) Time for space wiggle. Gasparini. Can you understand this shape better? .

Topics: ‡ Digital Imaging Beyond Film-like Photography Film‡ Computational Aspects of Lenses. Optics and Sensors . Lens and Lighting Limitations ‡ Adaptive Fusion of Multiple Images for Impossible Photos ‡ Image Reconstruction from Coded Images Samples ‡ Future Directions in Smart Lighting. Image Sensors and Processing ‡ Algorithmic Solutions for Camera Sensor.

Gemma Frisius. 1558 .A Brief History of Images 1558 Camera Obscura.

A Brief History of Images 1558 1568 Lens Based Camera Obscura. 1568 .

1837 .A Brief History of Images 1558 1568 1837 Still Life. Louis Jaques Mande Daguerre.

1970 1970 .A Brief History of Images 1558 1568 1837 Silicon Image Detector.

A Brief History of Images 1558 1568 1837 Digital Cameras 1970 1994 .

Dream of A New Photography Old ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ People and Time Each photo Lighting External Sensors µStills / Video¶ Exposure Settings Exposure Time Resolution/noise µHDR¶ range ~Cheap Precious Critical No Disjoint Pre-select Pre-select Pre-select Pre-select New Precious Free Automated* Yes Merged Post-Process Post-Process Post-Process Post-Process .

The digital photography marketplace ‡ 90+% of American households own a camera. robotics. •Marc Levoy . Webcams . 3% own only digital ‡ 57 million film cameras sold last year (down 10%) + 100+ million disposable film cameras (up 8%) + 53 million digital cameras (up 15%) + 53 million camera phones (skyrocketing) ‡ Surveillance camera. 20% own a digital camera..

Digital cameras are boring ‡ roughly the same features and controls as film cameras ± zoom and focus ± aperture and exposure ± shutter release and advance ± one shutter press = one snapshot ‡ but things are changing« •Marc Levoy .

Digital camera technology #1: real-time in-camera processing ‡ fast auto-focus systems ± sharp photographs of moving objects ‡ optical image stabilization ± long handheld exposures ‡ automatic object recognition ± adaptive metering (e.g. of faces) helps film cameras as well •Marc Levoy .

Digital camera technology #2: unusual acquisition protocols ‡ continuous auto-focusing ± sequences of sharp photographs of moving objects ‡ large DRAM buffer ± 16MB on Canon EOS-D30 = 10 JPEG images ± permits burst-mode photography ‡ free and plentiful permanent storage ± 2 GB microdrive = 1000 JPEG images ± permits extended shooting replacing film cameras for newsgathering •Marc Levoy .

stitching panoramas ± will eventually replace the view and panoramic camera ‡ around the bend ± high-X imaging (resolution. dynamic range. and other specialized shooting techniques ‡ second generation tools ± warping images. etc. focus.Digital camera technology #3: digital post-processing ‡ Photoshop ± replacing traditional darkroom techniques ± also replacing exposure compensation.) ± techniques based on multiple images •Marc Levoy . color filtering.

Survey ‡ How many of you are photographers ? ‡ How many of you are photo-artists ? photo‡ How many of you are digiphoto-artists ? digiphoto‡ How many do active programming ? ‡ Field of work: work: Academics? Industry ? Research ? Art ? ‡ Brief Introductions .

demo encouraged 15 points Final Project Individual or in groups of up to 3 20 points Discretionary credit 5 points Each student is expected to prepare notes for one lecture These notes should be prepared and emailed to the instructor no later than the following Sunday (midnight EST). 15 minute presentation.Credit (Tentative) Assignments: Five project-oriented assignments projectRequires programming in Matlab 8 points each MidMid-term Exam 20 points Term Paper Individual or a group of 2. No late submissions . Discretionary credit will be given for this. Revisions and corrections will be exchanged by email and after changes the notes will be posted to the website before class the following week. 8 to 10 pages.

csg242@ccs.neu.edu. Send email to raskar@merl. 617-621-7533. check emails. Course mailing list: Please make sure that your emailid is on the course mailing list list: ± csg242@ccs.com Instructor: Instructor: Ramesh Raskar. cellPlease do not email/IM during class.neu. Office hours: -hours: Location: Location: -- . Please keep your cell-phones switched off during class hours.Structure of class: class: 1st half 6:00PM ± 7:20PM break 7:20PM ± 7:40PM 2nd half 7:40PM ± 9:00PM Please be on time.com 617-621Email is the best way to get in touch with me. Every class has a break of 20 minutes for your convenience. etc. please use this time to take/make calls. raskar@merl.edu.