Many vision tasks such as scene segmentation, or the recognition of materials within a scene, become considerably easier when it is possible to measure the spectral reflectance of scene surfaces. In this project, we present an efficient and robust approach for recovering spectral reflectance in a scene that combines the advantages of using multiple spectral sources and a multispectral camera. We have implemented a system based on this approach using a cluster of light sources with different spectra to illuminate the scene and a conventional RGB camera to acquire images. Rather than sequentially activating the sources, we have developed a novel technique to determine the optimal multiplexing sequence of spectral sources so as to minimize the number of acquired images. We use our recovered spectral measurements to recover the continuous spectral reflectance for each scene point by using a linear model for spectral reflectance. Our imaging system can produce multispectral videos of scenes at 30fps. We demonstrate the effectiveness of our system through extensive evaluation. As a demonstration, we present the results of applying data recovered by our system to material segmentation and spectral relighting. This is a joint project with Jong-Il Park and Moon-Hyun Lee from the Mixed Reality Laboratory at Hanyang University, and Michael Grossberg at the City University of New York (CUNY).

Publications

"Multispectral Imaging Using Multiplexed Illumination,"
J. Park, M. Lee, M. D. Grossberg, and S. K. Nayar,
IEEE International Conference on Computer Vision (ICCV),
Oct, 2007.
[PDF] [bib] [©]

Images

		Illustration of the Multispectral Imaging System: The scene is lit with a set of distinct illuminations and a synchronized RGB camera captures the corresponding images. By using rapidly changing multiplexed illuminations and processing the acquired video, we obtain a multispectral video of the scene.
		Results for Macbeth Chart: Spectral reflectances of color chips on the Macbeth chart measured using multiplexed illumination. The measured spectra are shown as dotted black lines and the actual spectra (ground truth) are shown as red solid lines.
		Spectra of Illumination and Camera Channels: The spectra of the 5 types of LEDs (solid lines) and the spectral responses of the three color channels of the PointGrey Dragonfly Express camera (dashed lines) used in our system.
		Observed Spectra of Optimal Illuminations: Spectra of an optimal pair of multiplexed illuminations and the corresponding spectra received by the color camera.
		Results for Toy Scene: Images showing color cast artifacts in RGB relighting compared with multispectral relighting.
		Results for Flower Scene: Images showing color distortion artifacts in RGB relighting compared with multispectral relighting.
		Results for Fruits Scene: Images comparing RGB and multispectral relighting applied to a scene with three real fruits (on the right) and a printed image of the same fruits (on the left).

Video

		ICCV 2007 Video: This video shows our approach, a comparison of conventional and multispectral relighting, the design of our imaging system, and several results. (With narration)

Related Projects

Generalized Mosaicing

Multiplexed Illumination