You are on page 1of 21

Digital Camera Image Processing

Konstantinos N. Plataniotis
University of Toronto kostas@dsp.utoronto.ca
• • • •

Digital Signal Processing ECE @ UofT
Image and Video Processing
Digital Camera Processing cDNA Micro-array Imaging Visual Object Compression Universal Multimedia Access

Multimedia Security
• • • • • Authentication & Fingerprinting Image and Video Encryption Visual Secret Sharing Secure Streaming Watermarking

Biometrics & Surveillance
• Face Recognition • Gait Recognition • Visual Surveillance

Professors:
• Dimitris Hatzinakos (dimitris@comm.utoronto.ca) • Kostas Plataniotis (kostas@dsp.utoronto.ca)

Common work with Rastislav

Lukac

Theme of the talk

The ‘Google Index’
• Digital imaging 25,000,000 14,100,000 6,730,000 972,000 623,000 52,800

Is it possible to compensate for the limitations of the sensor using Digital Signal Processing Solutions ?

• Digital image processing • Color image processing • Single-sensor camera • Color filter array (CFA) • Bayer CFA

Outline
• The problem • Background
K1

Digital color imaging
color image Parrots
R channel G channel B channel

image column k2

spatial position i = (k1 − 1) K1 + k2 image sample xi = (186, 48, 42) xi1 = 186 xi 2 = 48
xi 3 = 42

image row k1
(number of image rows)

• Challenges for single-sensor imaging • Applications • Performance issues • Conclusions

l=2 (image dimension)

K2 (number of image columns)

RGB image

R

G

B

RG image

RB image

GB image

Color image acquisition: • scanners

m=3 (number of color channels)

• digital cameras - most popular and widely used • synthetic (e.g. grey-scale image coloration)

The “color pixel” level
Red

Color imaging basics
Color vector is uniquely characterized by: • its magnitude (length) • direction (orientation)
R

RGB (sRGB) color space:
Yellow

Magenta White Blue Cyan

• commonly used for acquisition, storage, and displaying purposes • additive concept of color composition
1 Red xi xj Magenta

M xi = xi = ( xi1 ) 2 + ( xi 2 ) 2 + ( xi 3 ) 2

⎛ x x x ⎞ Dxi = ⎜ i1 , i 2 , i 3 ⎟ ; ⎜ Mx Mx Mx ⎟ i i ⎠ ⎝ i
R unit sphere 1

Dxi = 1
xi1 M xi xi 3 M xi

Green

Yellow

White Black 0 Blue 1 Maxwell triangle Green Cyan

( xi 2 ) 2

xi1 xi
0

( xi1 ) 2

xi
xi 2 M xi

• RGB color pixel is the vector in a three-dimensional (RGB) color space • vector components are the intensities measured in RGB color channels
1

xi 3
B

Dx

0
i

G

xi 2

( xi 3 )

2

G 1

1 B

memory. cost • optics (optical zoom). audio and video) Three-sensor imaging • sensor is a monochromatic device and the most expensive component of the digital camera (10% to 25% of the total cost) • charge-coupled device (CCD) • complementary metal oxide semiconductor (CMOS) sensor R filter + sensor optical system image scene G filter + sensor B filter + sensor color filters + image sensors (CCD/CMOS) sensor data sensor data arranged as RGB color data X3 technology-based singlesensor imaging Layered (three-layer) silicon sensor • new technology . battery. digital zoom. etc. • multimedia acquisition.Digital camera technologies • digital imaging age • images to be captured are available instantly • commercial proliferation: • imaging enabled wireless phones • pocket devices (PDAs) • surveillance and automotive applications End-user’s point of view Basic hardware architectures: • three-sensor device • single-sensor device (used in consumer-grade digital cameras) • X3 technology based device Focus on functionality vs.expensive solution for professional devices (medical & science applications) • directly captures RGB light at each spatial location in an image during a single exposure • takes advantage of the natural light absorbing characteristics of silicon + camera output • color filters are stacked vertically and ordered according to the energy of the photons absorbed by silicon • professional design • each sensor “corresponds” to a particular color channel • spectral processing . processing & transmission (image.

RGB is most widely used ii) mixed primary/complementary colors (e. A/V) camera ASIC firmware memory Diagonal Bayer CFA Pseudo-random CFA Pseudo-random CFA HVS based design • DRAM buffers & stores digital data obtained from the A/D converter • DRAM passes data to application-specific integrated circuits (ASIC) • DSP operations. YMC) systems . MGCY pattern) iii) four and more color systems (white and/or colors with shifted spectral sensitivity) • CFAs in ii) and iii) may produce more accurate hue gamut. focus aperture and shutter viewfinder image scene optical system infrared blocking.Single-sensor imaging sensor data optical system image scene CFA + sensor color filter array + image sensor (CCD/CMOS) sensor data arranged as RGB color data demosaicking camera output Color filter array (CFA) design Key factors in CFA design • immunity to color artifacts and color moiré • cost-effective reconstruction • reaction of the pattern to image sensor imperfections • immunity to optical/electrical cross talk between neighboring pixels Color filter array (CFA) • used to produce a 2-D array or mosaic of color components • produced CFA (sensor) image is a gray-scale image • full-color image is obtained by demosaicking Color systems used in CFA design i) three-color (RGB. zoom. such as demosaicking and image re-sizing may be realized in both ASIC and micro-processor levels • Bayer CFA is widely used (good performance. anti-aliasing optical filter blocking system (Bayer) CFA image sensor (CCD.g. AC) microprocessor DRAM buffer bus A/D converter stick memory media (card) color LCD display PC / TV interface (USB. costeffective color reconstruction) .CMOS) Bayer CFA Diagonal stripe CFA Vertical stripe CFA Yamanaka CFA flash user interface power supply (battery. but they limit the useful range of the darker colors RGB-based CFA proposals Single-sensor camera architecture lens.

most widely used in camera image processing outputted camera image estimation operations generalized camera solutions Vector processing • image pixels are processed as vectors • computationally expensive input color image camera image processing output color image . demosaicked image compression Using a companion personal computer (PC) • PC interfaces with the digital camera which stores the images in the raw CFA format • allows for the utilization of sophisticated solutions Digital rights management • camera (CFA) image indexing → connection to image retrieval • camera image encryption CFA data storage digital camera camera image processing storage personal computer (PC) Camera processing operations Ability to follow the spectral image characteristics • component-wise (marginal) processing (component → component) • spectral model-based processing (vector → component) • vector processing (vector → vector) Processing according to the spectral image information Component-wise processing • each color plane processed separately • omission of the spectral information results in color shifts and artifacts input color image camera image processing camera image processing camera image processing output color image Ability to follow the structural image content • non-adaptive processing • (data) adaptive processing Spectral model based processing • essential spectral information utilized during processing camera image processing camera image processing input color image camera image processing output color image Most practical camera solution • spectral model used to eliminate color shifts and artifacts input camera image • edge-sensing mechanism used to eliminate edge-blurring and to produce sharply-looking fine details Edge-sensing mechanism Spectral model • computationally very efficient . lossless compression • CFA image compression vs.Camera image processing Processing • demosaicking (spectral interpolation) • demosaicked image postprocessing (color image enhancement) • camera image zooming (spatial interpolation in CFA or fullcolor domain) Implementation Conventional digital camera • real-time constraints (computational simplicity requirements) CFA data camera image processing storage digital camera Compression • lossy (or near lossless) vs.

adaptive processing Non-adaptive processing • no data-adaptive control • often reduces to linear processing . j )∈ζ ∑ w( i . used SM both SM and ESM used (a. and spectral characteristics Edge-sensing mechanism Spectral model x processing no parameters or fixed setting y x( p. color reconstruction quality Local neighborhood area Features • approximation using a shape mask ζ • shape and size of ζ vary depending on the CFA used and processing task (demosaicking. j )∈ζ ∑ {w(′i . q + 1).easy to implement • inefficient performance (image blurring) Data-adaptive processing Construction • using spatial. ( p + 1. q ). etc.c) ζ = {( p − 1. q + 1)} . resizing. j ) ∈ ζ } z denotes the CFA image Spectral characteristics • spectral model Ψ SM and the ESM vs. ( p. j ) input camera image estimation operations generalized camera solutions outputted camera image Adaptive processing • edge-sensing weights used to follow the structural content • nonlinear processing • essential in producing high-quality. q − 1). x ( p . ( p − 1. j ) . q )} (b. used ESM + + + + + (a) (b) (c) (d) (e) omitted ESM. q − 1).e) ζ = {( p − 1. q − 1). ( p.q ) = ( i . j ) = w( i . sharply looking images parameters‘ adaptation Spatial characteristics • local neighborhood area ζ x processing y Structural characteristics • edge-sensing mechanism λ λ ( z ) → {w(i . structural.) • shape masks widely used in the demosaicking process: omitted both SM and ESM omitted SM. j ) / ( i . (i. q + 1).d. q ) )} w(′i . j ) . j ) Ψ (x (i . ( p + 1.Non-adaptive vs. ( p + 1.

x(i . q ) | + | z( p -1. q ) |) ζ = {( p − 1. q +1) = 1/(1+ | z( p . q −1) |) w( p +1. j ) ∈ ζ } for shape mask ζ = {( p − 1. q ) | + | z( p -1. q + 1). q ) | + | z( p . (i. q +1) |) w( p . q − 1). j ) = 1/(1 + ( g . q + 2) − z( p .q)3 ] pixel occupying neighboring location x(i . ( p. j ) ) • large image gradients usually indicate that the corresponding vectors are located across edges (penalized through small weights) Edge-sensing mechanism (ESM) Cost-effective.7x7) neighborhood • specialization on a particular CFA (e. j )2 . q)} w( p -1. q −1) = 1/(1+ | z( p + 2. q ) = 1/(1+ | z( p -2. q +1) = 1/(1+ | z( p + 2. j ) k + γ ) = ( x( p .q)1 . q ) k + γ ) /( x(i . j ) . q −1) |) w( p +1. q − 1). ( p + 1. q −1) |) • both structural and spatial characteristics are considered in the ESM construction Concept • ESM operator λ uses some form of inverse gradient of the samples in the CFA image for shape mask w( i .g. q + 2) − z( p . q )2 − x(i . q +1) = 1/(1+ | z( p -2. universal design • operates within the shape mask ζ • aggregation concept defined here over the four-neighborhoods only • desing suitable for any existing CFA Spectral model (SM) • takes into consideration both the spectral and the spatial characteristics of the neighboring color pixels pixel occupying location to be interpolated x( p.q)2 . q). q ) | + | z( p +1. ( p + 1. q − 1). q + 1). q ) − z( p −1. q +1) |) w( p +1.q) = [ x( p. j ) = β (1 + exp{ ( g . j ) k − x( g . q + 2) − z( p . j )2 . q ) − z( p . q +1) − z( p −1. j ) = [ x(i . h )∈ Modelling assumption in the existing SMs: CFA data demosaicking storage digital camera ∑ς | x (i . q +1) |) w( p -1. j )3 ] Fully automated solution w( i . q − 2) − z( p . j ) k − x( g . q )2 / x( i . q ) | + | z( p +1. q ) − z( p . q −1) − z( p . q ) | + | z( p . q +1) − z( p . j ) 1 = 1 + f (Δ (i . j ) k = x( p . q −1) − z( p +1. q ) = 1/(1+ | z( p + 2. q )2 + γ ) /( x(i . k = 1 or k = 3 End-user control based solution w( i . ( p − 1. q +1) − z( p +1. q ) − z( p +1. q ) | + | z( p +1. Bayer CFA): λ ( z ) → {w(i . j )2 . q −1) − z( p −1. h )∈ς storage digital camera demosaicking visual inspection or storage • normalized color ratio model (hue constancy is enforced in both in edge transitions and uniform image areas) ∑ | x( i . q − 2) − z( p . q ) k / x(i . j ) k = x( p . ( p + 1. x(i . q − 2) − z( p . q −1) = 1/(1+ | z( p . q ) k − x( i . h ) k |) CFA data • color ratio model (uniform hue modelling assumption) x( p . h ) k |}) −r parameters’ setting personal computer (PC) ( x( p . x( p. j )2 + γ ) • color difference model (constrained component-wise magnitude difference) • may better match the HVS properties x( p . ( p. q + 1)} w( p -1.Edge-sensing mechanism (ESM) Essential to produce sharply looking images • structural constraints impossed on the camera solution relate to the form of the ESM operator λ used to generate the edgesensing weights Edge-sensing mechanism (ESM) Conventional designs: • operate on large (5x5. q −1) = 1/(1+ | z( p -2. j )1 . q ) |) w( p . x( p. q ) | + | z( p -1.

for R or B component y = x( p . q )2 x( i . j )3 x(2i .q ) k x( p . j )1 + x(2i . j )3 x(2i . j ) x( p .q )2 R (or B) Δk R (or B) . j )2 + x( p . j ) k x(i . q )2 x(i . j ) k x (i . j )1 x( i .q ) x( p .g. j ) + γ I =1 for R component Geometric interpretation of 2-D case G G chromaticity line component to be calculated available components x( p . j )3 x ( p . x ( i . j ) = x ( p . j ) k x2 k =1 ( i . q ) x (i . j )1 + x(2i . j )2 x( i .q ) k via γ for B component x( p . j )2 x( i . q ) k x x(i . q )3 = x( i . j )2 x( p . q ) + γ I x(i . q )3 x(i .q )2 + γ x(i . j )2 + x( p . x(i . q ) k x(i . j ) = 0 ⇔ ∑ ∑ 3 3 k =1 ( p .q ) k x(i .q ) k via γ >> 0 =0 x( p . q )2 x( i . j ) cos x ( p . j ) x(i . q ) k = x( p . j )3 + x( p . j ) k + γ x′ (i .x (i . j )3 x(2i . q )2 = x ( p . e. q )2 x( p . j )2 x ( p . j ) x( i .q ) k + γ Δ2 x( i . q ) k ∑ x( p . q ) .q )2 Δ2 Δ′k x( p . q ) + γ I. q ) .q ) x(i . j ) k x(i . j ) + γ I ] x( p . j )1 x(i . q )3 x(i . j )3 x( p . j ) k 3 x2 k =1 ( p . j )2 x ( p . j ) ( ) =1 for G component x( p . j )2 R (or B) Vector SM Geometric interpretation • from three-component vector expression for G component R Generalized vector SM −b y = y1 = y2 = 2a Linear shifting of the input vectors • modifies their directional characteristics and normalizes their component-wise magnitude differences x( p . q )2 = x( p . j ) x′ ( p . j ) k x( p . j )2 + γ Δ′k R (or B) intermediate direction γ >>0) output direction ( original direction ( γ = 0 ) Δ2 x( p . j )2 + x(2i . q )1 = G B x( p . j ) + γ I = 0 ⇔ [x( p .Vector SM Modelling assumption • two neighboring vectors should have identical color chromaticity properties (directional characteristics) • two spatially neighboring vectors should be collinear in the RGB (vector) color space Vector SM Unique quadratic equation solution y = y1 = y2 = −b 2a (due to zero discriminant) b 2 − 4ac = 0 Geometric interpretation • from two-component vector expression G chromaticity line interpolated component available components Computational approach x ( p . j )1 x(i . q )1 x( i .q ) x( p . q ) + γ I ]. q )1 x(i . q ) . x ( i .[x (i . j )1 x( i .q ) G x( p .q )2 • any color component can be determined from the expression above by solving the quadratic equation expression ay 2 + by + c = 0 • y denotes the component to be determined. j )2 x( i .

signal processing concepts • postprocessing of demosaicked images is a novel application of great importance to both the end-users and the camera manufacturers . ⎪ = ⎨[0. 0. j ) (i . q ) ] for p even and q odd. if not identical. although may employ similar. full-color image acquired image z :Z2 → Z q p K1 eq.jq)) k = y − γ ∑ζ {w′ (i . j )∈ K2 x((ip.q ) ⎩ (1) Demosaicking (spectral interpolation) No full-color image without demosaicking • an integral and probably the most common processing step used in digital cameras • it is often connected with the demosaicked image postprocessing viewed as a correction step demosaicking process (mandatory) original R and B CFA data Bayer image G plane population R plane populated using SM B plane populated using SM Restored color image correction process (optional) original R and B CFA data G plane corrected via SM correction using R or B color components R plane corrected using SM B plane corrected using SM Demosaicked image postprocessing Full-color image enhancement • postprocessing the demosaicked image is an optional step • implemented mainly in software and activated by the end-user scene Corrected color image pleasing for viewing CFA image (gray-scale data) spectral interpolation optics CFA & sensor A/D demosaicking camera output & quality evaluation color correction & color balancing postprocessing demosaicked (full-color) image color image enhancement postprocessed demosaicked image with enhanced quality • localizes and eliminates false colors created during demosaicking • improves both the color appearance and the sharpness of the demosaicked image • unlike demosaicking. postprocessing can be applied iteratively until certain quality criteria are met • demosaicking and demosaicked image postprocessing are two processing steps are fundamentally different. q ) k x } x( p.q ) ⎧[ z( p . (γ → ∞.Generalized vector SM Features • universal solution easy to implement • tuning of both directional and magnitude characteristics during camera image processing operations • generalization of all previous spectral models: “non-shifted” vector model Demosaicking (spectral interpolation) Transforms a gray-scale CFA image into a demosaicked. j ) ( p . (1) (for Bayer CFA) color restoration colored CFA image x: Z2 → Z3 restored image y : Z2 → Z3 (γ = 0. 0.. z . two-component expression) Vector SM based data-adaptive estimator x( p . three-component expression) two-component expression) normalized color ratio model (two-component expression) color ratio model color difference model (γ = 0. q ) k = ( i . z( p . ⎪[0. q ) . 0] otherwise ( p. 0] for p odd and q even.

j N M o 2 i. j o 2 i. j ) + ( vi . or (a) • obtained from the raw CFA image stored in TIFF-EP format (when demosaicking is performed in a companion PC) (b) (c) (d) (e) (f) Universal demosaicking solution • can be used with an arbitrary RGB CFA • a canonical framework which readily accommodates a large number of application requirements and implementation constraints (a) • demosaicking/postprocessing operations are guided by vectorial fields of location flags (denoting the presence of color components in the image) (b) (c) (d) (e) (f) • control mechanism is used to denote the processing locations with sufficient amount of the available neighboring color components • demosaicked images (top rows). j − ui . j . j ) + ( ui . j Mean Square Error (MSE) K2 MSE = 1 3 NM ∑∑∑ ( 3 N M k =1 i =1 j =1 oik. j o x o x − Lx i . j − xik. j − vi . postprocessed images (bottom rows) Demosaicking: testing/evaluation procedure original image K1 Numerical criteria suitable for objective evaluation Mean Absolute Error (MAE) MAE = 1 3 NM sensor CFA data color CFA image ∑∑∑ k =1 i =1 j =1 3 N M oik. j ) 2 demosaicking Normalised Color Difference (NCD) • color chromaticity preservation • expressed in CIE LUV color space processing error restored image NCD = 1 NM ∑∑ ( L i =1 j =1 N M o i. j ) 2 2 2 1 NM ∑∑ ( L ) + ( u ) + ( v ) i =1 j =1 2 o i. j − xik.Demosaicked image postprocessing BI MFI CHI ECI SAIG Universal demosaicking Information about the arrangements of color filters in the actual CFA • available either from the camera manufacturer (when demosaicking is implemented in the camera).

q) x(p. mobile phones and PDAs -> low optical capabilities and computational resources • to improve functionality and quality of output -> increase the spatial resolution of the camera output Impact on computational complexity: • increased complexity for pseudo-random and random CFAs • Bayer CFA offers one of the simplest color reconstruction CFA solution A solution B Image zooming (spatial interpolation) Zooming in the RGB domain • conventionally used CFA image data Demosaicked (full-color) image zooming Zooming in the RGB domain • conventionally used • slower .q-1) x(p-1. demosaicking Impact on image quality: • quality significantly varies with both the CFA and the input (captured) image Digital zooming in imaging devices Motivation • technological advances -> miniaturization of single-sensor cameras • pocket devices.q+1) image lattice 3x3 supporting window .q) (central sample) Zooming in the CFA domain • novel approach CFA image data CFA image zooming • operating on noise-free samples • spectral interpolation follows spatial interpolation demosaicking zoomed image x(p+1.more samples to process • amplification of the imperfections introduced during demosaicking demosaicking color image zooming zoomed image image x(p-1.CFA selection vs.

q ) for (even p.q+1) (p+1. vector • no enough information (p+1.Demosaicked (full-color) image zooming Pixel arrangements observed during processing (p–1.q–1) (p.q) Demosaicked (full-color) image zooming Zooming methods • adaptive vs. even q ) x (2 p −1. odd q ) otherwise x (2 p −1.2 q ) ⎫ ⎪ x (2 p .q+1) • no enough information original component-wise median vector median CFA image zooming Filling CFA components • conventional approach destroys the underlying CFA structure • specially designed “filling operations” CFA image zooming G interpolation step z1 z2 z4 z3 z3 z4 z1 z2 for (odd p.q+1) (p+1.2 q −1) ⎪ ⎭ interpolator z( p .q–1) (p.2 q −1) ⎬ = b ( p .q) (p–1. q ) = ∑ w′j z j j =1 4 edge-sensing weight wi = 1 1 + ∑ zi − z j j =1 4 input Bayer CFA image conventional CFA structure preserving .q) (p. non-adaptive • component-wise vs.q+1) (p–1.q–1) (p–1.q–1) (p+1.q) (p.

q −1) + ∑ w′j z j j =1 • diagonal symmetry compared to R components z( p . q ) spectral quantities are formed using spatially shifted samples CFA image zooming B interpolation steps 4 ∑w z j =1 4 j 4 j ∑w j =1 = z( p . q ) + ∑ w′j z j j =1 4 j zi = Ri − Gi spectral quantities are formed using spatially shifted samples zi = Bi − Gi enlarged Bayer CFA image Camera image zooming: testing/evaluation procedure original image o: Z2 → Z3 Camera image zooming combined with demosaicking • original images q p processing error | o − y |: Z 2 → Z 3 restored image y : Z2 → Z3 K1 K2 color image x': Z2 → Z3 demosaicking K2 2 • conventional (demosaicked) zooming K1 2 down-sampled image o′ : Z 2 → Z Bayer image z ': Z2 → Z zoomed Bayer image z :Z2 → Z corresponding color image x: Z2 → Z3 • CFA image zooming .CFA image zooming R interpolation steps • utilizes both spatial and z = z( p . q −1) + spectral image characteristics ( p . q ) = z( p −1. q ) + ∑w z j =1 4 j 4 j j ∑w j =1 = z( p −1.

essentially attempts to reverse the demosaicking process –> disagreement with demosaicking CFA image compression Raw camera image data compression: • harder to be compressed due to the interlaced structure of red.1) z(3. spectral sensitivities.storage of details about camera setting.4) z(3.4) z(6.1) z(4.3) z(3.5) z(5.6) z(2.3) z(6.5) z(2.this effectively triples the compression ratio if measured in terms of the size of the demosaicked image CFA image display preprocessing (optional) demosaicking compression storage CFA image compression Possible preprocessing for CFA image compression: z(1.6) z(6.6) z(2.2) z(4.5) z(4.3) z(3.2) z(2.5) z(5.3) z(1.1) z(1. demosaicking performed off camera • may require preprocessing of the CFA image (spatial rearrangement of the CFA image data) .2) z(1.6) z(3.4) z(1.3) z(6. compression of demosaicked video rather than CFA images directly can be counterproductive • demosaicking triples the amount of data to be compressed by interpolating red.2) z(6.6) z(1.2) z(6.5) z(3.2) z(2. and illuminant used • consumer digital cameras store the full-color image in a compressed format using the Joint Photographic Experts Group (JPEG) standard • exchangeable image file (EXIF) format .4) z(3.2) z(5.2) z(4.5) z(4.1) z(2.6) z(3.3) z(2.1) z(6.2) z(1.1) z(3.3) z(4.Camera image compression Camera image storage: • either in a CFA format or as a full-color images • more expensive devices uses tagged image file format for electronic photography (TIFF-EP) .metadata information about the camera and the environment New possibilities: • CFA image data compression .1) z(5.4) z(5.3) z(1.allows for the storage and transmission of significantly less information compared to the full-color image compression solutions Demosaicked (full-color) image compression Standard (full-color) image compression: • although conventionally used.3) z(2.4) z(2.1) z(6.2) z(3.6) z(3.6) z(4.5) z(6.2) z(5. throughput bottleneck and power conservation: compression and demosaicking done inexpensively but also suboptimally on camera • high-end solutions: lossless or near-lossless coding. green and blue channels CFA image display demosaicking compression decompression storage • a key task of color image compression is to decorrelate the interpolated color bands .4) z(1.1) z(1.6) z(6.1) z(4.1) z(2.4) z(5.3) z(5.1) z(5.4) z(2.2) z(3.5) z(2.4) z(6. green and blue samples • only one third of the demosaicked image in size !! .6) z(5.6) z(1.5) z(4.3) z(4.3) z(5.4) z(4.6) z(5.4) z(4.5) decompression CFA image compression: • consumer electronics solutions constrained by low cost.5) z(1.5) z(6.

WWMX database) . World Wide Media eXchange .g. database software programmers. semantic information lossy demosaicked image coding lossy CFA image coding Camera image indexing Embedding procedure metadata CFA data encryption or metadata indexed demosaicked image indexed CFA data Camera image indexing • to authenticate. capturing device information satellite tracking information . and consumer electronics manufacturers Indexing: • performed in RB channel related CFA locations • HVS is more sensitive to luminance which is composed primarily of green light CFA data registration indexed CFA data metadata . organize and retrieve images in personal digital databases indexed CFA data demosaicking archive end-user Extraction procedure metadata extraction R G G B encrypted metadata R and B CFA component extraction decryption R CFA samples B CFA samples • vast number of captured images • exponentially growing market with image-enabled consumer electronics image retrieval and/or database organization digital image database + + indexed data extraction indexed data extraction Extraction tools: • either the CFA images or the demosaicked images in personal image databases using PC software commonly available by camera manufacturers indexed CFA data • conventional public image database tools (e. watermarking (robust approach) near-lossless demosaicked (full-color) image coding near-lossless CFA image coding • great importance to the end-users. mobile phone and pocket device • indexing performed by embedding metadata information • suitable embedding techniques: secret sharing (cost-effective approach).Camera image compression Camera image indexing Digital rights management in digital cameras: • captured images are directly indexed in the single sensor digital camera.

Camera image encryption Features: • confidentiality must be provided for next generation camera applications • online collaboration using PDAs or mobile phone video conferencing • request for encryption combined with compression (large volume of image data) Needs for partial encryption: • encrypt only part of a compressed bit-stream (preferably ‘the most important information’) to reduce computational complexity • compression aids encryption by removing redundancy Camera image encryption Encryption: • coder directs stream cipher to encrypt certain bits after color image coding Image C-SPIHT Coder 0 1 1 1 0 0...LIP-sig and Bk. which process separately the individual frames. JPEG-2000.0 1 1 1 0 0. SPIHT) used Video-demosaicking Essential in single-sensor VIDEO cameras • motion video or image sequences represent a 3-D image signal or a time sequence of 2-D images (frames) • motion video usually exhibits significant correlation in both the spatial and temporal sense • by omitting the essential temporal characteristics. Location of bits Bk. computational overhead can be controlled via the parameter Combined Coding and Encryption Compressed/ Encrypted .. Bitstream fD(·) Stream Cipher Decryption Function C-SPIHT Decoder Location of bits Bk. spatial processing methods. produce an output image sequence with motion artifacts Video-Demosaicking Practical solutions • spectral model used to eliminate color shifts and artifacts • edge-sensing mechanism used to eliminate edge-blurring and to produce sharply-looking fine details • spatiotemporal window placed over three consecutive frames to reduce motion artifacts Edge-sensing mechanism Spectral model Processing windows: x* x* x* input camera image estimation operations generalized camera solutions outputted camera image q t p temporal spatial spatiotemporal ..LIP-sig and Bk.LIS-sig Compressed/ Encrypted Bitstream fE(·) Stream Cipher Encryption Function • coding efficiency is unaffected since coded bits are encrypted • confidentiality vs.LIS-sig Decrypted/ Decoded Image Decryption: • user cannot decrypt the most significant information without decryption key Combined Decryption and Decoding • noise-like information varies depending on the encryption mechanism and/or coder (JPEG...

t −1) + c( r .t ) − c( r . s −1.t −1) − c( r .t +1) + c( r .t ) |) −1 w2 = (1+ | c( r .t +1) + c( r . s .t ) + c( r .t + i ) k − x( r −1.t ) ) / 2 c6 = (c( r .t −1) + c( r +1. s +1. s . s .t ) − c( r .t −1) + c( r −1. s . s .t + i )2 c( r +1.t + i ) = x( r .t −1) + c( r . s −1. s −1.t ) − c( r +1. s . s −1.t + i )2 − ( x( r .t ) + c( r . s +1. s +1.t + i )2 .t −1) − c( r +1. s −1.t + i ) k + x( r − 2.t +1) ) / 2 c4 = (c( r +1. t denotes frame index • three frames: past (i=-1). s −1.t + i )2 − ( x( r .t −1) − c( r −1.t +1) | + | c( r . s . s +1. s . s +1.t −1) − c( r . s −1. s −1.t −1) + c( r +1.t +1) |) w5 = 1/(1+ | c( r −1.Spectral input formation Spectral quantities in G plane demosaicking c( r −1. s +1. s −1.t + i ) k ) / 2 c( r +1. s −1.t ) |) −1 Spectral quantities in R and B plane demosaicking c( r −1. s +1.t ) − c( r −1.t ) ) / 4 c2 = (c( r . s +1. s − 2. s +1. s −1.t + i ) k − x( r +1. s −1. s . s .t ) ) / 4 c3 = (c( r .t ) + c( r −1. s .. s .t ) + c( r +1.t +1) | + | c( r .t +1) |) w2 = 1/(1+ | c( r .) k denotes R (k=1) or B (k=3) component Unidirectional multistage video-demosaicking • configurations used in G as well as R and B plane demosaicking • configurations used in R and B plane demosaicking Bidirectional multistage video-demosaicking • configurations used in G as well as R and B plane demosaicking • configurations used in R and B plane demosaicking (a) (b) (c) (a) (b) (c) (a) (b) (c) (d) (a) (b) (c) (d) w1 = (1+ | c( r −1. s . s . s ..t ) k = x( r . s . s .t +1) + c( r −1. s +1.t ) |) • example of unidirectional window outputs c1 = (c( r −1.t + i ) = x( r +1.)2 denotes G component x(. s −1.t + i ) = x( r . actual (i=0). s .. s −1. s +1.t )2 + ∑ {w j c j }/ ∑ w j j =1 j =1 N N • essential spectral quantities formed over the spatiotemporal neighborhood • structural content followed by spatiotemporal edge-sensing weights • color component to be outputted is obtained via weighted average operations defined over unidirectional demosaicked values • G plane demosaicking • R and B plane demosaicking x(. s . s +1.t +1) |) w4 = 1/(1+ | c( r +1. s .s) denotes spatial location.t + i ) = x( r +1. s . s −1..t −1) − c( r .t ) |) −1 (d) (e) (f) (d) (e) (f) • example of edge sensing weights • example of bidirectional window outputs c1 = (c( r −1. s .t + i ) k − x( r −1. s .. s . s . s −1. s .t −1) + c( r .t + i )2 − ( x( r .t +1) |) w3 = 1/(1+ | c( r . s +1.t + i ) = x( r −1.t +1) ) / 2 c2 = (c( r . s −1..t + i ) = x( r +1. s .t ) ) / 4 c4 = (c( r +1. s .t −1) − c( r .t + i ) = x( r −1.t + i ) k + x( r . s +1. s . s . s +1.t + i ) k ) / 2 c( r .t ) ) / 4 • example of edge sensing weights w1 = 1/(1+ | c( r −1.t ) − c( r . future (i=1) Spectral spatiotemporal solution x( r . s −1. s −1.t +1) + c( r +1.t ) |) −1 w4 = (1+ | c( r +1.t +1) | + | c( r +1.t −1) − c( r +1. s +1. s . s . s . s . s +1.t ) |) w6 = 1/(1+ | c( r . s . s −1. s +1. s .t )2 = x( r .t +1) ) / 2 c3 = (c( r . s −1. s .t + i )2 c( r +1. s ... s .t + i )2 c( r −1.t + i ) k ) / 2 c( r .t −1) − c( r −1. s . s +1.t ) + c( r .t + i ) k − x( r +1.t ) − c( r +1.t +1) ) / 2 c5 = (c( r −1. s −1.t ) + c( r +1. s +1.t + i ) = x( r −1. s +1. s + 2.t ) k + ∑ {w j c j }/ ∑ w j j =1 j =1 N N x( r .t + i ) k + x( r + 2. s +1.t −1) + c( r −1. s .t −1) + c( r .t +1) | + | c( r −1.t ) ) / 2 w3 = (1+ | c( r . s .t + i )2 − ( x( r .t + i ) k + x( r .t + i ) k ) / 2 Spatiotemporal video-demosaicking Fast video-demosaicking procedure • usage in PDAs and mobile phone imaging applications • utilization of multistage unidirectional spatiotemporal filtering concepts • to overcome the lack of spectral information • (r.

t ) k Mean square error (MSE) MSE = 1 kK1 K 2 K 3 ∑∑∑∑ ( o 3 r =1 s =1 t =1 k =1 K1 K2 K3 ( r .t ) k − x( r .75.75.50.20. • (c) Coast for t = 1.25.30. s .t ) k ) K1 K 2 K3 2 Normalized color difference (NCD) NCD = ∑∑∑ ∑ ( o 3 r =1 s =1 t =1 k =1 K1 K 2 K3 r =1 s =1 t =1 ( r . s .10. s .50. Objective criteria Mean absolute error (MAE) MAE = 1 kK1 K 2 K 3 Video-demosaicking ∑∑∑∑ o r =1 s =1 t =1 k =1 K1 K2 K3 3 ( r . s .65.t ) k ) 2 • MAE and MSE are defined in RGB space • NCD defined in perceptually uniform CIE LUV space original frames restored using spatial BI demosaicking restored using fast spatiotemporal demosaicking . s .Test image sequences Evaluation procedure original video sequence t = K3 t = 62 t=2 t =1 K1 sensor CFA data color CFA video (a) (b) K2 K3 demosaicking (c) • (a) processing error Nature for t = 1.99.t ) k ) 2 ∑∑∑ ∑ ( o k =1 ( r . s .t ) k − x( r .99 restored video sequence • (b) Bikes for t = 1.99.t ) k 3 − x( r . s .

(c) unidirectional demosaicking. (d) bidirectional demosaicking Experimental results Experimental results (a) (b) (a) (b) (c) (d) (c) (d) • (a) original image. (c) unidirectional demosaicking.Video-demosaicking Experimental results (a) (b) original frames restored using spatial BI demosaicking restored using fast spatiotemporal demosaicking (c) (d) • (a) original image. (c) unidirectional demosaicking. (d) bidirectional demosaicking • (a) original image. (b) bilinear demosaicking. (b) bilinear demosaicking. (b) bilinear demosaicking. (d) bidirectional demosaicking .

675-681." IEEE Signal Processing Magazine.0439 0.1448 0.4 28.4 174. Lukac and K.0439 0. B. 2.0545 0." IEEE Transactions on Consumer Electronics.238 3.4 355. pp.0421 0. . Venetsanopoulos.5 14. no.6 21. Smolka. vol. and A.7 NCD 0.986 1. January 2005. "Fast Video Demosaicking Solution for Mobile Phone Imaging Applications.0503 Conclusions Bikes Nature THE NUMBER OF NORMALIZING OPERATIONS PER SPATIAL LOCATION Method Criterion Additions Subtractions Multiplicat.035 MSE 177.0424 0. 22.519 6.5 24.N.Performance and computational complexity analysis OBJECTIVE EVALUATION OF VIDEO DEMOSAICKING Test color video Cost Method bilinear unidirectional bidirectional bilinear unidirectional bidirectional bilinear unidirectional bidirectional MAE 6.498 1. K.0994 0.1103 0.7 13. K. Lukac.640 2. no.1 37. 74-86. 51. vol. Plataniotis. "Vector Filtering for Color Imaging. pp.N.912 10. R. Divisions Abs.660 2.369 3. 1. Martin. Plataniotis. values unidirectional Step 1 35 18 6 25 6 Step 2 23 18 6 13 6 Total 58 36 12 38 12 Step 1 43 20 6 21 8 bidirectional Step 2 31 20 6 9 8 Total 74 40 12 30 16 Conclusions Where to learn more? R.N. May 2005.

737-745. September 2006 . K.N. Lukac and K. vol. 51. and D. “Color Image Processing: Methods & Applications. "DataAdaptive Filters for Demosaicking: A Framework. pp. R. no.N." IEEE Transactions on Circuits and Systems for Video Technology. Martin. 2.dsp. "Demosaicked Image Postprocessing Using Local Color Ratios. 2. 560-570. no. to appear. Plataniotis. R. R. Lukac and K. Plataniotis.Where to learn more? Where to learn more? R. Plataniotis." IEEE Transactions on Circuits and Systems for Video Technology." IEEE Transactions on Consumer Electronics." IEEE Transactions on Consumer Electronics.utoronto. K. "Color Image Zooming on the Bayer Pattern. 50. "Normalized Color-Ratio Modelling for CFA Interpolation. Plataniotis. May 2005. 914-920. no. Lukac. vol.” CRC Press. Hatzinakos. 14. Lukac.N. 15. June 2004.N. pp. vol. vol.N. Lukac and K. and K. Where to learn more? Color Image Processing: METHODS & APPLICATIONS Multimedia Laboratory ECE Department University of Toronto Edited by: Rastislav Lukac and Kostas Plataniotis www. pp. Plataniotis.ca Phone: 1 416 978 6845 Fax: 1 416 978 4425 R. May 2004. 2005. 6.