A typical design of a CMOS image sensor includes the pixel array, the exposure control units, and the signal conversion units. To implement coded rolling shutter, all we need to change is the Address Generator. Instead of sending out the START and STOP signals sequentially, we can modify the Address Generator to send out them in a coded way, and therefore we can control both the readout timings and the exposure lengths for all the rows. Programmable address generators can also be designed.

From the perspective of space-time sampling, conventional rolling shutter samples a tilted slice of the space-time volume. The global shutter, which is often used in CCD image sensors, also samples a 2D plane from the space-time volume, parallel to the image plane. Raskar et al. proposed the flutter shutter for motion deblur, which can be viewed as a coded global shutter, because all pixels have exactly the same exposure pattern. These three camera shutters are 1-D functions. In contrast, in coded rolling shutter, both the readout timing and the exposure length for each row can be controlled, essentially giving us a 2D shutter function -- a function of both time and the image row index. This extra dimension gives us more flexibility for space-time sampling.

Coded readout can be used for high speed photography. In interlaced readout coding, the readout time for one frame is uniformly divided into K sub-images. Both the skew and the time lag in these sub-images are reduced K times, at the cost of the vertical spatial resolution.

In staggered readout coding, we reverse the order of readout within every K rows. The skew in each of these sub-images remains almost the same as conventional rolling shutter, but the time-lag between two sub-images is very small, which is useful for capturing ultra-high speed events.

The interlaced readout can be used to compute optical flow between the two sub-images after vertical interpolation. The optical flow can be used for motion interpolation, skew compensation, and motion deblur. Please refer to the paper for details.

Coded row-wise exposure can be used to implement a simple yet effective row-wise auto-exposure for better capturing the dynamic range of many scenes, especially outdoor scenes.

If we have control of both the exposure and the readout, we proposed a method to recover a HDR image from a single shot in the presence of camera shake. Coded rolling shutter allows us to embed multiple exposures and timings within one single frame --- multiple exposures are used to extend the dynamic range, and the sub-images sampled at different timings are used to estimate camera motion for deblurring.

We evaluated the noise performance for the HDR imaging application with coded rolling shutter. Simulation results show that the proposed method will have better performance than single exposure methods. A theorectical analysis can be found here.

This video is a compilation of the proposed coding schemes and some of the results of this project. (Quicktime, H.264, with narration)