The World In An Eye |
 | |
We have conducted an analysis of exactly what visual information about the
world is embedded within a single image of an eye. It turns out that the cornea
of an eye and a camera viewing the eye form a catadioptric (lens + mirror)
imaging system, which we refer to as a corneal imaging system. Unlike a typical
catadioptric imaging system, a corneal one is flexible in that the reflector
(cornea) is not rigidly attached to the camera. Using a geometric model of the
cornea based on anatomical studies, its 3D location and orientation can be
estimated from a single image of the eye. Once this is done, a wide-angle view
of the environment of the person can be obtained from the image. In addition,
we can compute the projection of the environment onto the retina with its
center aligned with the gaze direction. This foveated retinal image reveals
what the person is looking at. We have analyzed in detail the characteristics
of the corneal imaging system including its field of view, its resolution and
its locus of viewpoints. When both eyes of a person are captured in an image,
we have a stereo corneal imaging system. We have analyzed the epipolar geometry
of this stereo system and shown how it can be used to compute the 3D structures
of objects located around the person. Our approach to interpreting eye images
is passive and non-invasive. It has direct implications for several fields
including visual recognition, human-machine interfaces, computer graphics and
human affect studies.
|
Publications
"Corneal Imaging System: Environment from Eyes,"
K. Nishino and S. K. Nayar,
International Journal on Computer Vision,
Oct. 2006.
[PDF] [bib] [©]
"The World in an Eye,"
K. Nishino and S.K. Nayar,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Vol. I, pp. 444-451, Jun. 2004.
[PDF] [bib] [©]
|
Gallery
Some interactive gallery pages may require Adobe Flash in order to view the images.
|
Image
 |
|
Environment Maps and Retinal Images from Eye Images:
From images of eyes (left column), we can compute wide-angle environment maps
of the world surrounding the person in the images (middle column) and retinal
images that reveal exactly what the person in each image is looking at (right
column).
|
| |
|
|
|
Videos
 |
|
Tracking a Bus:
From a video sequence of a person's eye, the retinal video is computed. The
retinal video reveals what the person is looking at, which is the tail of the
moving bus in this example. (With audio)
|
| |
|
|
 |
|
Conversation:
In this example of a casual conversation between three people, the computed
retinal video shows how one person's gaze direction changes based on who is
talking to him. (With audio)
|
| |
|
|
|
Slides
CVPR 2004 presentation With videos (zip file)
|
Software
 |
|
VisualEyes:
VisualEyes is a software which allows you to compute spherical panoramas,
spherical environment maps and retinal perspective images from images of eyes.
As a result, it allows you to look around at the world surrounding the person
in the image, view what the person was seeing when the image was taken, and
estimate the lighting conditions of the scene.
|
| |
|
|
|
News Articles
"Daguerreian Glimpses", The Daguerreian Society Newsletter, January 2005.
"Fleeting Experience, Mirrored in Your Eyes" (New York Times July 2004)
"Meet the Eye Cam" (Newsweek July 2004)
|
Eyes for Relighting
Catadioptric Cameras for 360 Degree Imaging
Gaze Locking: Passive Eye Contact Detection
for Human–Object Interaction
|
|
|
