The World In An Eye

We have conducted an analysis of exactly what visual information about the world is embedded within a single image of an eye. It turns out that the cornea of an eye and a camera viewing the eye form a catadioptric (lens + mirror) imaging system, which we refer to as a corneal imaging system. Unlike a typical catadioptric imaging system, a corneal one is flexible in that the reflector (cornea) is not rigidly attached to the camera. Using a geometric model of the cornea based on anatomical studies, its 3D location and orientation can be estimated from a single image of the eye. Once this is done, a wide-angle view of the environment of the person can be obtained from the image. In addition, we can compute the projection of the environment onto the retina with its center aligned with the gaze direction. This foveated retinal image reveals what the person is looking at. We have analyzed in detail the characteristics of the corneal imaging system including its field of view, its resolution and its locus of viewpoints. When both eyes of a person are captured in an image, we have a stereo corneal imaging system. We have analyzed the epipolar geometry of this stereo system and shown how it can be used to compute the 3D structures of objects located around the person. Our approach to interpreting eye images is passive and non-invasive. It has direct implications for several fields including visual recognition, human-machine interfaces, computer graphics and human affect studies.


"Corneal Imaging System: Environment from Eyes,"
K. Nishino and S. K. Nayar,
International Journal on Computer Vision,
Oct, 2006.
[PDF] [bib] [©]

"The World in an Eye,"
K. Nishino and S.K. Nayar,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Vol.I, pp.444-451, Jun, 2004.
[PDF] [bib] [©]

Some interactive gallery pages may require Adobe Flash in order to view the images.


  Environment Maps and Retinal Images from Eye Images:

From images of eyes (left column), we can compute wide-angle environment maps of the world surrounding the person in the images (middle column) and retinal images that reveal exactly what the person in each image is looking at (right column).



If you are having trouble viewing these .mpg videos in your browser, please save them to your computer first (by right-clicking and choosing "Save Target As..."), and then open them.

  Tracking a Bus:

From a video sequence of a person's eye, the retinal video is computed. The retinal video reveals what the person is looking at, which is the tail of the moving bus in this example. (With audio)


In this example of a casual conversation between three people, the computed retinal video shows how one person's gaze direction changes based on who is talking to him. (With audio)



CVPR 2004 presentation     With videos (zip file)



VisualEyes is a software which allows you to compute spherical panoramas, spherical environment maps and retinal perspective images from images of eyes. As a result, it allows you to look around at the world surrounding the person in the image, view what the person was seeing when the image was taken, and estimate the lighting conditions of the scene.


News Articles

"Daguerreian Glimpses", The Daguerreian Society Newsletter, January 2005.

"Fleeting Experience, Mirrored in Your Eyes" (New York Times July 2004)

"Meet the Eye Cam" (Newsweek July 2004)

Eyes for Relighting

Catadioptric Cameras for 360 Degree Imaging

Gaze Locking: Passive Eye Contact Detection
for Human–Object Interaction