Although display devices have been used for decades, they have functioned without taking into account the illumination of their environment. In this project, an initial step has been taken towards addressing this limitation. We are exploring the concept of a lighting sensitive display (LSD) -- a display that measures the surrounding illumination and modifies its content accordingly. In particular, we are interested in a system that renders a 3D scene such that it always appears to be lit by the real environment that the display resides in. In addition, we would like the appearance of the scene to quickly adapt to changes in the environmental lighting. This is a challenging rendering problem as it requires the subtle appearance details (shadings, specularities, shadows and interreflections) of the scene to be rendered with high efficiency.

We have proposed several sensing techniques for measuring the illumination field incident on a display. While it is desirable to measure the complete 4D light field incident upon the display, our current system uses a 2D measurement that is adequate when the light sources in the environment are distant from the display. A novel relighting algorithm has been developed that uses a compact representation of a large set of images of the scene that correspond to different lighting conditions. Unlike previous relighting algorithms, this one exploits not only image correlations over the illumination dimensions but also coherences over the spatial dimensions of the image. This enables the LSD to achieve high quality relighting in real time. Our prototype LSD can render 640x480 images of scenes under complex and varying illuminations at 15 frames per second using a 2GHz processor. This project was done in collaboration with Peter Belhumeur at Columbia University and Terry Boult at the University of Colorado, Colorado Springs.

Publications

"Lighting Sensitive Display,"
S.K. Nayar, P.N. Belhumeur and T.E. Boult,
ACM Transactions on Graphics,
Vol.23, No.4, pp.963-979, Oct, 2004.
[PDF] [bib] [©]

"Lighting Sensitive Display,"
S.K. Nayar, P.N. Belhumeur and T.E. Boult,
Proceedings of ACM SIGGRAPH, Technical Sketch,
Vol.23, No.4, pp.963-979,
Oct, 2004.
[PDF] [bib] [©]

Images

		Prototype Lighting Sensitive Display: A compact camera with a 170 degree field of view is embedded within the frame of the display.
		Methods for Sensing a Display's Illumination Field: (a) Dense spatial estimates of the total energy incident upon each point on the display can be obtained using a 2D array of photodetectors distributed over the surface of the display. (b) If the illumination field is very smooth, it can be interpolated using sparse spatial and directional samples obtained using optical fibers distributed over the frame of the display and pointing in different directions. (c) When the sources in the environment are distant compared to the size of the display, the illumination field at all points on the display have the same directional distribution which can be measured using a wide-angle camera placed close to the display. (d) If the sources are isotropic over the span of the display, their intensities and locations in 3D can be computed using a stereo pair of wide-angle cameras.
		Algorithm for Interactive Relighting: A fast algorithm has been developed for image based relighting of scenes in real time. The algorithm exploits not only image correlations over the illumination dimensions but also coherences over the spatial dimensions of the image.

Videos

		Lighting Sensitive Display of David: In this video the LSD displays a synthetic model of Michelangelo's David (courtesy of Marc Levoy, Stanford University). As a lamp is moved around in front of the display, the LSD changes the appearance of the synthetic David so as to keep it consistent with the illumination of the real environment.
		Lighting Sensitive Display of a Still Life Scene: This video shows the high quality of the images rendered by the relighting algorithm. The shadows cast by the pepper dispenser, the fish and the knife are sharp and always consistent with the lighting. One can also notice the way in which the highlights on the pepper dispenser, the knife, the orange and the bottle of vinegar vary with the lighting.

Software

LSD: Real-Time Relighting Algorithm

News Articles

"Getting Real: What's Next in Computer Displays?" Scientific American, December 2002.

"Lighting-Sensitive Displays Re-Create Setting," Photonics Spectra, July 2002.

Related Projects

Project Anywhere: Radiometric and Geometric Compensation

Volumetric Displays: Passive Optical Scatterers