The combination of the cornea of an eye and a camera viewing the eye form a catadioptric (mirror + lens) imaging system with a very wide field of view. We have analyzed in detail the characteristics of this corneal imaging system. Anatomical studies have shown that the shape of a normal cornea (without defects) can be approximated with an ellipsoid of fixed eccentricity and size. Using this shape model, we can determine the geometric parameters of the corneal imaging system from the image (self-calibration). Then, an environment map of the scene with a large field of view can be computed from the image. The environment map represents the illumination of the scene with respect to the eye. This use of an eye as a natural light probe is advantageous in many relighting scenarios. For instance, it enables us to insert virtual objects into an image such that they appear consistent with the illumination of the scene. The eye is a particularly useful probe when relighting faces. It can be used to sample the appearance of a face by simply waving a light source in front of the face. In the case of an already captured image, eyes could be the only direct means for obtaining illumination information. We show how illumination computed from eyes can be used to replace a face in an image with another one, while ensuring that the appearance of the new face is consistent with the lighting of the scene. The eye not only serves as a useful tool for relighting but also makes relighting possible in situations where current approaches are hard to use. The movie images used on this page are courtesy of Miramax Film Corp. (frame from the movie "Amelie") and Paramount Pictures (frame from the movie "Roman Holiday.")

Publications

"Eyes for Relighting,"
K. Nishino and S.K. Nayar,
ACM Transactions on Graphics (also Proc. of SIGGRAPH),
Vol.23, No.3, pp.704-711, Jul, 2004.
[PDF] [bib] [©]

Image

		Environment from Eyes: From an eye in a single image of a person, we can estimate an environment map which encodes the illumination distribution of the scene. Left: An image of a face. Middle: A magnified version of the image of the right eye of the person. Right: An environment map computed from the eye. This environment map tells us the illumination incident upon the person's face.

Videos

		SIGGRAPH 2004 Video: This video shows all the applications of relighting using eyes (With narration)
		Inserting Virtual Objects: Using the eye as a light probe, one can seamlessly insert virtual objects into already captured videos. (With narration)
		Relighting Faces: The eye as a light probe provides a convenient means to capture the geometry and reflectance of a face by simply waving a light source in front of it. Once this is done, the face can be rendered using any desired illumination. (With narration)
		Replacing Faces: In already taken images, the eye may be the only source of illumination information. Illumination computed from eyes can be used to alter the scene while ensuring that the appearance of the altered region remains consistent with the illumination of the rest of the scene. (With narration)

Slides

SIGGRAPH 2004 presentation     With videos (zip file)

Software

		VisualEyes: VisualEyes is a software which allows you to compute spherical panoramas, spherical environment maps and retinal perspective images from images of eyes. As a result, it allows you to look around at the world surrounding the person in the image, view what the person was seeing when the image was taken and estimate the lighting conditions of the scene.

News Articles

"Daguerreian Glimpses," The Daguerreian Society Newsletter, January 2005

"Meet the Eye Cam" (Newsweek July 2004)

"Fleeting Experience, Mirrored in Your Eyes" (New York Times July 2004)

Related Projects

The World In An Eye

Catadioptric Stereo: Planar and Curved Mirrors

Catadioptric Cameras for 360 Degree Imaging