When an observer moves in three-dimensional space, real scene features, such as surface markings, remain stationary with respect to the surfaces they belong to. In contrast, a virtual feature, which is the specular reflection of a real feature, travels on the surface. Since algorithms for stereo and structure from motion do not distinguish between real and virtual features, they end up producing incorrect depth information in the case of specular objects. Sometimes the computed structure can be significantly different from the actual structure of the object. In this project, we are investigating techniques that enable stereo and structure from motion to detect and handle specular reflections.

The first part of the project focuses on the recovery of a specular surface using a moving camera with known motion. Based on the notion of caustics, we have developed a novel feature classification algorithm that can distinguish real and virtual features from their image trajectories due to camera motion. Next, using support functions of curves, a closed-form relation is derived between the image trajectory of a virtual feature and the geometry of the specular surface it travels on. It is shown that when the camera motion and the surface profile are coplanar, the profile is uniquely recovered by tracking just two unknown virtual features. These results have also been extended to the case of arbitrary 3D surface profiles that are traveled by virtual features when camera motion is not confined to a plane.

In the second part of the project, we have studied the the problem of accurate depth estimation using stereo in the presence of specular reflections. Since specular reflection is viewpoint dependent, it can cause large intensity differences at corresponding scene points in a stereo image pair. This results in significant errors in the computed scene structure. We have analyzed the physics of specular reflection and the geometry of stereopsis and arrived at a relationship between stereo vergence, surface roughness and the likelihood of a correct match. Given a lower bound on surface roughness, an optimal binocular stereo configuration can be determined which maximizes precision in depth estimation despite specular reflection. However, surface roughness is difficult to estimate in unstructured environments. Therefore, trinocular configurations that are independent of surface roughness are determined such that at each scene point that is visible to all three cameras, at least one stereo pair can produce correct depth. We have developed a simple algorithm to reconstruct depth from the multiple stereo pairs.

Finally, we have developed ordinal measures (which are based on rank-permutations of intensity values) for solving correspondence and tracking problems in stereo and motion. We have shown that ordinal measures are more robust to non-ideal imaging conditions than the more commonly used correlation and the sum of squared difference measures.

Publications

"On the Motion and Appearance of Specularities in Image Sequences,"
R. Swaminathan, S.B. Kang, R. Szeliski, A. Criminisi and S.K. Nayar,
European Conference on Computer Vision (ECCV),
Vol.I, pp.508-523, May, 2002.
[PDF] [bib] [©]

"Ordinal Measures for Image Correspondence,"
D.N. Bhat and S.K. Nayar,
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Vol.20, No.4, pp.415-423, Apr, 1998.
[PDF] [bib] [©]

"Stereo and Specular Reflection,"
D.N. Bhat and S.K. Nayar,
International Journal on Computer Vision,
Vol.26, No.2, pp.91-106, Feb, 1998.
[PDF] [bib] [©]

"Motion Estimation Using Ordinal Measures,"
D.N. Bhat, S.K. Nayar and A. Gupta,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp.982-987, Jun, 1997.
[PDF] [bib] [©]

"A Theory of Specular Surface Geometry,"
M. Oren and S.K. Nayar,
International Journal on Computer Vision,
Vol.24, No.2, pp.105-124, 1996.
[PDF] [bib] [©]

"Ordinal Measures for Visual Correspondence,"
D.N. Bhat and S.K. Nayar,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp.351-357, Jun, 1996.
[PDF] [bib] [©]

"A Theory of Specular Surface Geometry,"
M. Oren and S.K. Nayar,
IEEE International Conference on Computer Vision (ICCV),
pp.740-747, Jun, 1995.
[PDF] [bib] [©]

"Stereo in the Presence of Specular Reflection,"
D. Bhat and S.K. Nayar,
IEEE International Conference on Computer Vision (ICCV),
pp.1086-1092, Jun, 1995.
[PDF] [bib] [©]

"Structured Highlight Inspection of Specular Surfaces,"
A.C. Sanderson, L.E. Weiss and S.K. Nayar,
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Vol.10, No.1, pp.44-55, Jan, 1988.
[PDF] [bib] [©]

"Determining Surface Orientations of Specular Surfaces by Intensity Encoded Illumination ,"
Shree K. Nayar and Arthur C. Sanderson,
Proceedings of The International Society for Optical Engineering (SPIE),
Vol.850, pp.122-127, Nov, 1987.
[PDF] [bib] [©]

Images

		Specular Highlight Stereo: This picture shows the fundamental ambiguities related to surface shape that arise when stereo is applied to a specular surface. If an array of light sources with known directions are used, then the highlights measured by two cameras can be used to disambiguate surface normals. Once the normals have been computed, they can be integrated to obtain surface shape.
		Real and Virtual Features from Caustics: This picture shows how caustics computed from image trajectories produced by scene features can be used to classify each feature as real or virtual. Real features produce compact (a point in the ideal case) caustics, while virtual features produce larger caustics.
		Specular Surface Geometry from Motion: This picture shows experimental results on structure from motion for a highly specular surface. The image trajectories produced by just two virtual features on the surface are sufficient to compute the profile of the surface.
		Avoiding Specularities in Stereo: The vergence angles of a trinocular stereo system can be chosen such that, for any surface normal and surface roughness, at least two of the three views do not produce noticeable specular reflections. Therefore, the three views can be used to compute a good depth map of the scene.

Related Projects

Shape from Brightness

Non-Single Viewpoint Imaging: Raxels and Caustics

Photometric Invariants for Segmentation and Recognition

Radiometric Camera Calibration