Coded Rolling Shutter Photography:
Flexible Space-Time Sampling

Complementary Metal-Oxide Semiconductor (CMOS) image sensors are rapidly overtaking CCD sensors in a variety of imaging systems, from digital still and video cameras to mobile phone cameras to surveillance and web cameras.

In this project, we propose a novel readout architecture called coded rolling shutter for CMOS image sensors. Rolling shutter has traditionally been considered a disadvantage to image quality since it often introduces skew artifacts. We show that by controlling the readout timing and the exposure length for each row, the row-wise exposure discrepancy in rolling shutter can be exploited to flexibly sample the 3D space-time volume of scene appearance, and can thus be advantageous for computational photography. The required controls can be readily implemented in standard CMOS sensors by altering the logic of the control unit.

We propose several coding schemes and applications: (1) coded readout allows us to better sample time dimension for high-speed photography and optical flow based applications, including motion interpolation, skew compensation, and motion deblur; and (2) row-wise control enables capturing motion-blur free high dynamic range images from a single shot. While a prototype chip is currently in development, we demonstrate the benefits of coded rolling shutter via simulation using images of real scenes.

This project is done in collaboration with Yasunobu Hitomi and Tomoo Mitsunaga at Sony Corporation.

Publications

"Coded Rolling Shutter Photography: Flexible Space-Time Sampling,"
J. Gu, Y. Hitomi, T. Mitsunaga and S.K. Nayar,
IEEE International Conference on Computational Photography (ICCP),
Mar, 2010.
[PDF] [bib] [©]

Images

  CMOS Image Sensor and Coded Rolling Shutter:

A typical design of a CMOS image sensor includes the pixel array, the exposure control units, and the signal conversion units. To implement coded rolling shutter, all we need to change is the Address Generator. Instead of sending out the START and STOP signals sequentially, we can modify the Address Generator to send out them in a coded way, and therefore we can control both the readout timings and the exposure lengths for all the rows. Programmable address generators can also be designed.

     
  Coded Rolling Shutter for Flexible Space-Time Sampling:

From the perspective of space-time sampling, conventional rolling shutter samples a tilted slice of the space-time volume. The global shutter, which is often used in CCD image sensors, also samples a 2D plane from the space-time volume, parallel to the image plane. Raskar et al. proposed the flutter shutter for motion deblur, which can be viewed as a coded global shutter, because all pixels have exactly the same exposure pattern. These three camera shutters are 1-D functions. In contrast, in coded rolling shutter, both the readout timing and the exposure length for each row can be controlled, essentially giving us a 2D shutter function -- a function of both time and the image row index. This extra dimension gives us more flexibility for space-time sampling.

     
  Interlaced Readout for High Speed Photography:

Coded readout can be used for high speed photography. In interlaced readout coding, the readout time for one frame is uniformly divided into K sub-images. Both the skew and the time lag in these sub-images are reduced K times, at the cost of the vertical spatial resolution.

     
  Staggered Readout for High Speed Photography:

In staggered readout coding, we reverse the order of readout within every K rows. The skew in each of these sub-images remains almost the same as conventional rolling shutter, but the time-lag between two sub-images is very small, which is useful for capturing ultra-high speed events.

     
  Optical Flow Based Applications:

The interlaced readout can be used to compute optical flow between the two sub-images after vertical interpolation. The optical flow can be used for motion interpolation, skew compensation, and motion deblur. Please refer to the paper for details.

     
  Adaptive Row-wise Exposure:

Coded row-wise exposure can be used to implement a simple yet effective row-wise auto-exposure for better capturing the dynamic range of many scenes, especially outdoor scenes.

     
  Coded Exposure for HDR Imaging with Hand-held Cameras:

If we have control of both the exposure and the readout, we proposed a method to recover a HDR image from a single shot in the presence of camera shake. Coded rolling shutter allows us to embed multiple exposures and timings within one single frame --- multiple exposures are used to extend the dynamic range, and the sub-images sampled at different timings are used to estimate camera motion for deblurring.

     
  Noise Analysis for the HDR Imaging Application:

We evaluated the noise performance for the HDR imaging application with coded rolling shutter. Simulation results show that the proposed method will have better performance than single exposure methods. A theorectical analysis can be found here.

     

Video

If you are having trouble viewing these .mp4 videos in your browser, please save them to your computer first (by right-clicking and choosing "Save Target As..."), and then open them.

  ICCP 2010 Video:

This video is a compilation of the proposed coding schemes and some of the results of this project. (Quicktime, H.264, with narration)

     

Slides

ICCP 2010 presentation

High Dynamic Range Imaging: Assorted Pixels

Adaptive Dynamic Range Imaging

Motion Deblurring Using Hybrid Imaging

Programmable Imaging: Micro-Mirror Arrays

Programmable Imaging: Controllable Apertures

Temporal Modulation Imaging