Flexible Sheet Cameras With Elastic Optics

In this project, we pursue a radically different approach to imaging. Rather than seeking to capture the world from a single point in space, our goal is to explore the idea of imaging using a thin, large, flexible sheet. If such cameras can be made at a low cost (ideally, like a roll of plastic sheet), they can be used to image the world in ways that would be difficult to achieve using one or more conventional cameras. In the most general sense, such an imaging system would enable any surface in the real world to capture visual information. While there is significant ongoing work on the development of flexible image sensors, our interest here is in the design of the optics needed to form images on such sensors.

At first glance one might imagine that a simple lens array aligned with a flexible detector array would suffice - its field of view (FOV) can be varied by simply bending it. What is perhaps less apparent is the fact that, in a curved state, the FOV can end up being severely under-sampled. This under-sampling leads to a captured image that is not bandlimited. Thus, the Nyquist sampling criterion is violated and the image will suffer from aliasing artifacts when reconstructed. It is important to note that these artifacts cannot be removed via post-processing since scene information is lost during image formation.

To address the above aliasing problem over an entire range of sheet curvatures, we propose the design of a deformable (elastic) lens array. We show that, if designed carefully, the deformable lenses of the array will change shape (and hence focal length) under bending forces in a way that mitigates aliasing. A remarkable feature of our design is that the lens array can achieve aliasing compensation passively, without the use of any per-pixel actuation or control. Our optics can be combined with a flexible sensor array to obtain a complete sheet camera. This project was supported by the Office of Naval Research (ONR). The flexible lens array was prototyped using the facilities of The Columbia Laboratory for Unconventional Electronics, led by John Kymissis.

Publications

"Towards Flexible Sheet Cameras: Deformable Lens Arrays with Intrinsic Optical Adaptation,"
D.C. Sims, Y. Yue and S.K. Nayar,
IEEE International Conference on Computational Photography (ICCP),
May. 2016.
[PDF] [bib] [©]

Images

  Fixed Focal Length Lens Array:

This figure shows a lens array, where each lens has a fixed focal length, in a highly curved state. Notice the large black areas (gaps) between the fields of view (yellow regions) of adjacent lenses. These gaps lead to aliasing in the captured image. We could attempt to eliminate the gaps by increasing the field of view of each lens, but this would lead to blurring in the captured image when the lens array is flat.

     
  Adaptive Focal Length Lens Array:

Here, we see an elastic lens array where the focal length of each lens can adapt to local curvature. Notice the contiguous coverage of the scene achieved by this lens array, with no gaps between the fields of view of adjacent lenses. If the geometric and material properties of the elastic lens array are designed well, this desired adaptation of focal length will hold true over a wide range of curvatures of the array, avoiding aliasing and blurring in the captured image.

     
  Passive Optical Adaptation to Local Curvature:

This graph shows the change in field of view of a single lens of the array as a function of the local curvature. The black line (FOV des) shows the desired change in field of view of a single lens as the local curvature of the lens array increases. The blue line (FOV act) is a plot of the field of view of a single adaptive focal length lens in an elastic array as the local curvature increases. The red line (FOV fix) represents the FOV of a fixed focal length lens, which does not change as a function of local curvature.

     
  Effect of Material Properties on Optical Adaptation :

We modeled the deformable lens array using the Abaqus finite element analysis software suite. Using this model, we studied how the material properties of the lens array affect its shape when it is deformed. We found that the two properties that had the greatest impact were hardness (represented using the Shore A scale) and the compressibility (represented by the Poisson's Ratio). In this video, we show four lens arrays of different material properties being deformed. The colors within each lens array correspond to the mechanical stress. This simulation reveals that the shapes of the lenses of the array are more or less independent of hardness, even as the mechanical stress differs. Furthermore, they are also independent of the Poisson's ratio as long as the ratio is close to 0.5, which corresponds to an incompressible material.

     
  Simulated PSF as a Function of Curvature:

Using Abaqus, we simulated how the lenses of the array we designed deform when the array is bent. Then, using a ray-tracing algorithm, we simulated the object side point spread function (PSF) of the central lens of the array. As desired, the object-side PSF of that lens widens as the local curvature increases. We have found that every lens, except those close to the edge of the array, behave in the same manner.

     
  Aliasing in the Fixed and Adaptive Focal Length Cases:

In this figure we compare rendered images captured using fixed focal length and adaptive focal length lens array. The scene texture is the sum of two sinusoids, one low and one high in frequency. Using the known optical properties (PSFs and sampling period) of the two arrays, we rendered images captured by the two systems. For each curvature of the arrays, we assumed that the scene surface is also curved with the same center of curvature as the arrays. Since the FOV of the sheet camera increases with curvature, we interpolated the captured images in the horizontal direction to obtain stretched images that depict the increasing horizontal FOV of the array. For the adaptive system, due to blurring, the high frequency sinusoid decreases in magnitude with curvature, while the low frequency sinusoid is faithfully reproduced over the entire range of deformations. In contrast, due to the narrow and fixed FOV of the lenses in the non-adaptive case, the scene texture is under-sampled for the higher curvatures, which results in undesirable aliasing artifacts.

     
  Fabrication of the Elastic Lens Array:

This set of photos illustrate the procedure used to fabricate the elastic lens array. A 33 by 33 lens array mold was machined from aluminum by Contour Metrological and Manufacturing, Inc. We prepared Silopren Liquid Silicone Rubber 7005, a two-part solution, by mixing the two components together in a beaker and then removing the air bubbles that form during mixing using a vacuum chamber. The mold is cleaned and then coated with release agent to make it easy to remove the lens array from the mold after it is cured. We then pour the silicone rubber into the mold and place the mold into an oven. Once the material is cured (hardened) we remove the lens array from the mold. Next, a 33 by 33 aperture sheet made of nylon is glued to the bottom of the lens array. We then attach a diffuser on which each lens forms a spot of light. The entire grid spots formed by the lens array corresponds to a single captured image.

     
  Experimental Apparatus:

This photo shows the apparatus we have used to conduct our experiments. The deformation of the sheet camera is controlled using a vise-like mechanism - the parallel jaws of the vise push against two sides of the imaging sheet such that the distance between the jaws determines the curvature of the sheet. We used a Nikon D90 to capture an image of the 33 by 33 spots created by the lens array on the diffuser attached to the bottom of the array. An LCD monitor is placed above this apparatus to show various scenes to the imaging system.

     
  Measured PSF as a Function of Curvature:

For each curvature of the lens array, we raster scanned a small bright dot on the LCD monitor above the apparatus. As this dot was scanned, we measured the brightness of the spot on the diffuser produced by the central lens of the array. The measured brightness values are used to plot the object side PSF of the central lens. As seen this figure, the measured PSF increases in width as the local curvature increases, demonstrating that the FOV of the lens does indeed increase with curvature to mitigate image aliasing.

     
  Scenes Captured by the Flexible Sheet Camera:

We imaged various static scenes using the sheet camera. For each scene, we bent the array from the flat state to a curved one for which the angular field of view is 52 degrees. For any given sheet deformation, we know the directions of the optical axes of all the lenses of the array. This information is used to project the captured image onto a plane at a chosen distance from the sheet camera. The projected measurements are then interpolated to obtain the final image whose horizontal field of view increases with the curvature of lens array. Due to the inherent optical adaptation of our design, the images are free of aliasing effects.

     

Video

  ICCP 2016 Video:

This video introduces the concept of flexible sheet cameras, discusses several potential applications of these flexible cameras and summarizes the mechanism of passive focal length adaptation to local curvature. (With narration)

YouTube Video