Gigapixel Computational Imaging |
||||||||||||
Figure 1: A 1.7 gigapixel image captured using
the implementation shown in Figure 4. The
image dimensions are 82,000 x 20,000 pixels, and the scene occupies
a 126x32 degree FOV. From left to right, insets reveal the label
of a resistor on a PCB board, the stippling print pattern on a
dollar bill, a miniature 2D barcode pattern, and the fine ridges of
a fingerprint on a remote control. The insets are generated by
applying a 60x-200x digital zoom to the above gigapixel
image.
|
||||||||||||
Publications"Gigapixel Computational Imaging," |
||||||||||||
Project DescriptionToday's high-resolution cameras capture images with pixel counts in the tens of millions. When digital cameras can produce images with billions of pixels, they will usher in a new era for photography. A gigapixel image has such a tremendous amount of information that one can explore minute details of the scene (see Figure 1). Gigapixel images are fascinating because they capture orders of magnitude more detail than the human eye, revealing information that was completely imperceptible to the photographer at the time of capture. At present, highly specialized gigapixel imaging systems are being developed for aerial surveillance [1]. Why are there no gigapixel cameras commercially available today? CMOS and CCD technologies have improved to the point that imaging sensors with pixels in the 1µm range have been demonstrated [2]. It is certainly within the reach of manufacturing technology to produce sensors with 1 billion pixels. On the other hand, it remains a huge challenge to design and manufacture lenses which have the resolving power to match the resolution of such a sensor. This is because the number of resolvable points for a lens, referred to as the Space-Bandwidth Product (SBP) [3], is fundamentally limited by geometrical aberrations. SBP is a unit-less quantity that tells us the number of distinct points which can be measured over a given FOV. Ideally, all lenses would be diffraction-limited so that increasing the scale of a lens while keeping FOV fixed would increase SBP. Unfortunately, SBP reaches a limit due to geometrical aberrations. There are two common approaches that are taken to increase SBP in the face of this fundamental limit. The first is to increase the scale of the camera to a very large format. Two notable large format gigapixel cameras are Microsoft Research Asia's dgCam, and the Gigapxl project camera. These are special purpose cameras that are extremely large (FL > 500mm) and therefore not suitable for commercial photography. The second approach is to increase complexity as a lens is scaled up. This approach was taken by Marks and Brady, who proposed a 7-element large format monocentric lens called the Gigagon [4]. We take a different approach: we show how computations can be used to achieve the desired resolution while reducing lens complexity and camera size. Scaling Laws for Lenses. Lohmann originally observed that lenses obey certain scaling laws that determine how resolution increases as a function of lens size [5]. Consider a lens with focal length f, aperture diameter D, and image size h by w. If we scale the lens by a factor of M, then f, D, h, and w are all scaled by M, but the F/# and FOV of the lens remain unchanged. If, when we scale the lens, the minimum resolvable spot size has not also increased by a factor of M, then we have increased the total number of points that can be resolved, thus increasing SBP. Figure 2 shows three different curves for SBP as a function of lens scale. The red Rd curve shows the ideal, diffraction-limited case, where SBP increases quadratically with lens scale. Most lenses, however, are limited by geometrical aberrations. Then, as the green Rg curve shows, SBP eventually reaches a plateau and stops increasing with lens scale. Since aberrations can be reduced by stopping down the lens aperture, lens designers typically increase F/# as a lens is scaled up. Then resolution does increase beyond the aberration limit, but at the cost of reduced light intensity, introducing a degradation in Signal-to-Noise Ratio (SNR). A general rule of thumb for conventional lens design is that F/# is made to increase like the cube root of lens scale. When this rule of thumb is applied, SBP no longer plateaus at the aberration limit, as shown by the blue Rf curve. We present a different approach to increase SBP - the use of computations to correct for geometrical aberrations. In conventional lens design, resolution is limited by the spot size of the lens. For a lens with aberrations, spot size increases linearly with the scale of the lens. For a computational imaging system, resolution is related to deblurring error. We observe, however, that for a lens with spherical aberrations, deblurring error does not increase linearly with lens scale. We use this remarkable fact to derive a scaling law that shows that computational imaging can be used to develop cameras with very high resolution while maintaining low complexity and small size. The magenta Rc curve in Figure 3 shows that, for a computational imaging system with a fixed SNR (i.e. fixed deblurring error), SBP scales more quickly with lens size than it does for conventional lens designs.
Proposed Architecture. By using a large ball lens, an array of planar sensors, and deconvolution as a post processing step, we are able to capture gigapixel images with a very compact camera. The key to our architecture lays in the size of the sensors relative to the ball lens. Together, a ball lens and spherical image plane produce a camera with perfectly radial symmetry. We approximate a spherical image plane with a tessellated regular polyhedron. A planar sensor is placed on each surface of the polyhedron. Relatively small sensors are used so that each sensor occupies a small FOV and the image plane closely approximates the spherical surface. As a result, our camera produces a PSF that is not completely spatially invariant, but comes within a close approximation. The first system we demonstrate consists solely of a ball lens and an array of planar sensors. We use a 100mm acrylic ball lens and a 5 megapixel 1/2.5" Lu575 sensor from Lumenera (see Figure 4). We emulate an image captured by multiple sensors by sequentially scanning the image plane using a pan/tilt motor. With this camera, a 1 gigapixel image can be generated over a roughly 60x40 degree FOV by tiling 14x14 sensors onto a 75x50mm image surface. When acquiring images with the pan/tilt unit, we allow a small overlap between adjacent images. Our first camera system is extremely compact, but it assumes there is no dead space between adjacent sensors. Sensors require at least some packaging around the active pixel area, so they can't be packed without introducing gaps in the camera's FOV. One possible solution to this problem is to use two or three camera systems together to cover the full FOV. Another solution is to introduce a secondary optic for each sensor, changing the system magnification so that the FOVs of adjacent sensors overlap slightly. This approach is taken in the implementation, shown in Figure 5, using custom optics manufactured by Lightwave Enterprises and 5 NET-USA Foculus sensors.
A Single Element Design. The design in Figure 4 is extremely compact, but impractical because adjacent sensors must be packed without any dead space in between them. The design in Figure 5 is practical enough for an implementation using off-the-shelf components, but is much less compact. The size of this system is limited by the package size of the sensor relative to the active sensor area. Sensors with a package size that is only 1.5x larger than the active sensor area are currently commercially available. With these sensors, it is possible to build a gigapixel camera that uses only a single optical element, as shown in the Zemax raytrace of Figure 6. In this design, each sensor is coupled with a smaller acrylic relay lens that decreases the focal length of the larger acrylic ball lens. The relay lenses share a surface with the ball lens, which means that it is possible to combine the entire optical system into a single element that may be manufactured by molding a single material, drastically simplifying the complexity (and hence alignment) of the system.
Capturing the Complete Sphere. All of our proposed designs use a ball lens. A great advantage of using a ball lens is that, because it has perfect radial symmetry, a near hemispherical FOV can be captured. In fact, it can even be used to capture the complete sphere, as shown in Figure 7. This design is similar to the camera proposed by Krishnan and Nayar [6], but it uses an array of planar megapixel sensors and relay lenses instead of a custom image sensor. Light passes through the gaps on one hemisphere, forming an image on a sensor located on the opposite hemisphere. As a result, the sensors cover the complete 2π FOV at the cost of losing roughly half the incident light. 
Figure 7. A design for a gigapixel camera with a 2π
radian FOV. The design is similar to the implementation in Figures 4 and 5 with a large gap between adjacent
lens/sensor pairs. Light passes through the gaps on one hemisphere, forming an
image on a sensor located on the opposite hemisphere.
|
||||||||||||
Gallery of Captured Gigapixel ImagesClick on the thumbnails below to view the captured gigapixel images in detail at gigapan.org.
|
||||||||||||
AcknowledgementsThis research was supported in part by DARPA Award No. W911NF-10-1-0214. Oliver Cossairt wassupport by an NSF Graduate Research Fellowship. The authors thank Ravi Athale of Mitre Corporation for his comments and suggestions, Kieth Yeager of Columbia University for his help with machining the prototype components, and Lauren Kushner for her 3D modelling expertise. |
||||||||||||
Slides |
||||||||||||
Related Project |
||||||||||||
References
|
||||||||||||
