Temporal Modulation Imaging

In this project, we are interested in the capture of high dynamic range and multispectral images of complex scenes. Our approach is to capture multiple frames of the scene while the imaging parameters are modulated continuously; images are taken at different exposures or through different wavelength bands. A major problem associated with such a continuous modulation approach is the need for perfect synchronization between image acquisition and modulation control. In the past, this problem has been addressed by using sophisticated servo-control mechanisms. In this work, we show that the process of modulation imaging can be made much simpler by using vision algorithms to automatically relate each acquired frame to its corresponding modulation level. This correspondence is determined solely from the acquired image sequence and does not require measurement or control of the modulation. The image acquisition and the modulation work continuously, in parallel, and independently. We refer to this approach as computational synchronization. It makes the imaging process simple and easy to implement. We have developed a prototype modulation imaging system that uses computational synchronization and we have used it to acquire high dynamic range and multispectral images.

Publications

"Uncontrolled Modulation Imaging,"
Y.Y. Schechner and S.K. Nayar,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Vol.II, pp.197-204, Jun, 2004.
[PDF] [bib] [©]

Images

  Modulation Imaging System:

This system was used for demonstrating the method of uncontrolled modulation imaging. It uses a monochrome image detector. Adjacent to the detector (before the lens), there are two spatially varying filters: one with varying transmittance (the circular filter), the other with a varying spectral response (the rectangular filter).

     
  Exposure Modulation for Scene with Person:

This is the captured video for a scene with a person captured by the system for HDR imaging. Notice the continuous change in exposure.

     
  HDR Image of Scene:

This is the high dynamic range image computed from the above video clip. Notice how the whole scene is measured without any dark and saturated regions. This image was obtained by dynamic range compression of the computed HDR image to show all the details.

     
  Spectrum Modulation for Flower Scene:

This is the captured video for a scene with a bunch of flowers captured by the system for multispectral imaging. Notice the continuous change in measured spectrum which is shown here as a change in RGB color. Notice how the colors change for the different types of flowers.

     
  Multispectral Image of Flowers:

This is the multispectral image computed from the above video for the flower scene. Here the measured spectrum at each pixel is mapped to an RGB color value (for visualization purposes) based on its peak location within the visible spectrum.

     
  Spectrum Modulation for Bulbs:

This is the captured video for a scene with three different types of light bulbs captured by the system for multispectral imaging. The scene includes an incandescent bulb (left), a halogen bulb (middle) and a fluorescent bulb (right). Notice how the bulbs peak (sometimes multiple times) at different points during the scanning.

     
  Multispectral Image of Bulbs:

This is the multispectral image computed from the above video for the bulb scene. Here the measured spectrum at each pixel is mapped to an RGB color value (for visualization purposes) based on its peak location within the visible spectrum.

     

Videos

If you are having trouble viewing these .mpg videos in your browser, please save them to your computer first (by right-clicking and choosing "Save Target As..."), and then open them.

  HDR Imaging:

This video shows the capture of images during continuous change of exposure (transmittance). The exposure varies as the circular spatially varying neutral density filter rotates.

     
  Multispectral Imaging:

This video shows the capture of images during continuous change of spectral filtering. The spectral response of the camera varies as the linear interference filter translates back and forth.

     
  Multidimensional Modulation Imaging:

In this video the neutral density filter and the linear interference filter are both simultaneously in motion. If one filter completes a large number of modulation cycles for a single cycle of the other filter, then a high dynamic range multispectral image can be computed.

     

Slides

ICCV 2005 presentation     With videos (zip file)

Database

WILD: Weather and Illumination Database

High Dynamic Range Imaging: Assorted Pixels

High Dynamic Range Imaging: Multiple Exposures

Radiometric Camera Calibration

Generalized Mosaicing

Coded Rolling Shutter Photography:
Flexible Space-Time Sampling