Saliency enhanced HDR imaging

Introduction

The visible light range is very vast, spreading from 10
-6

cd/m
2
to 10
8
cd/m
2
, while the camera's range is much lower,
at about 1 to 1000 cd/m
2
.

This of course creates a problem- how do we display the full
range of light on a limited display?

The solution- HDR imaging.

Capturing several images of the same scene at different
exposures and then merge them into 1 image- displaying the
original scene is a more accurate way.

This project is an application for compiling several multi-
exposure images into 1 HDR image, which preserves the
details from both the dark and bright sections of the
original scene.


To enhance the HDR image even further, we will also the
Saliency information in increase the contrast in the
attentive regions of the image, bringing us closer to the
actual visual scene- on LDR displays.




Algorithm
To create the saliency HDR image, the system performs the
following steps:

1. Computing the camera response curve.
2. Recovering the HDR radiance map from the LDR images.
3. Tonemapping the HDR image for display on LDR devices-
this is done by using fast bilateral filtering.
4. Applying the Saliency contrast enhancement.







Computing the camera response curve:



This is done by sampling the images in equally spaced pixels
according to the number of exposures, which currently in
this project- is 3.

We sample each of the images in the Red, Green and Blue
channels to create the following matrices:
zRed, zGreen and zBlue

Which are defined as:
Samplenumber x 3 matrices, where each row represents a
different image (since our max exposure number is 3)
And each column is the sample from the corresponding image
in the selected sample indices.
This is done by the makeImageMatrix function.


We also create the B matrix- a (samplenummber*3 x 3) matrix
that holds the exposures values of every image in log space.

Finally we solve the equations for each channel separately
using gsolve- which was taken from P. Debevec's algorithm.

This provides us with the Camera Response Curve for the 3
channels- gRed, gGreen, gBlue.














Note:
Currently the exposures vector is static- with values 4/1,
1/1 and 1/4 from brightest to darkest. I find this provides
a good result when no input exposure values are available.



Recovering the HDR radiance map:

After acquiring the gRed, gGreen, gBlue vectors, and with an
arbitrary weights vector (a linear rising and falling
function), we can compute the Radiance Map.

The radiance map is the full range HDR image- a floating
point image representation of the scene with very high
contrast.

This is done by the hdr function which performs the
following:


For every image:
Calculates the weighted value of the exposure- wij-
according to the weights vector, and adds it to the
total sum (sum).
Creates the m matrix for every color channel, using the
original image's color channels and the B matrix.
This is performed in log space.
Adds the current weighted image value (wij.*m) to the
hdr accumulated radiance map (hdr)
Finds the saturated pixels in every channel (the
'burnt' pixels)
Removes the saturated pixels from the radiance map and
the total sum.

After the loop, it then normalizes the radiance map by
dividing by the total sum, and returning from log space-
(exp(hdr./sum))

This provides us with the radiance map:














Bilateral Filtering:

The bilateral filter is similar to the Gaussian blur filter,
but with 1 crucial difference-
The effectiveness of the blur, is determined by the
similarity of the pixel to its neighbors.
If the pixel is similar, the blurring effect will be strong-
but if it's extremely different- ie, and edge- the blurring
will be minimal.




This helps preserve the hard edges while still blurring the
image and reducing its contrast.






















Tonemapping

In order to display the HDR image on LDR displays, we would
like to reduce the high contrast of the HDR image, while
preserving the colors and details.

This can be done by using fast bilateral filtering.

The basic algorithm is this:
Calculate the intensity of the HDR radiance map
Apply bilateral filtering on the intensity values
Acquire the Details layer- by subtracting the original
Intensity values from the blurred ones.
Reduce the contrast on the blurred intensity image
Add the Details layer
Add the colors


This reduces the high contrast of the HDR image while
maintaining the details and the colors of the original
scene.


At this point, we can comfortably display the HDR image on
LDR displays, while preserving most of the original scene's
display:





























But we can enhance the image further, with Saliency:


Saliency

According to several studies on the Human Visual System
(HVS), attention plays an important role in how we perceive
a visual scene.
Specifically, the areas which we focus on are perceived in
higher contrast, than the non-attentive areas- where the
contrast is actually reduced.

We would like to combine this information with our HDR
image, to enhance it further.


This can be accomplished by using the Saliency map-


The Saliency map shows the most attentive parts of the
image-the parts the eye is drawn to the most while viewing.

To acquire the Saliency map, this project uses the code
provided by the Multimedia department of the Technion.



Once the Saliency map is acquired, we can use its
information to enhance the contrast in the Salient parts of
the HDR image- giving the image a closer approximation to
the Human Visual System.






























Increased contrast image:
Saliency map:
Enhanced image:
Before Saliency enhancement:


After Saliency enhancement:




Conclusion

We presented a method for displaying a High Dynamic Range
image on a Low Dynamic Range display, while maintaining both
the saturation values and the fine details of the image, by
using the data gathered from several multi-exposure images
of the same scene.
We've also shown that by using the Saliency map it's
possible to enhance the HDR image even further, giving it a
close resembles to the actual scene as viewed by the HVS.