Specularities in Stereo and Motion 
 When an observer moves in threedimensional space, real scene features,
such as surface markings, remain stationary with respect to the surfaces they
belong to. In contrast, a virtual feature, which is the specular reflection of
a real feature, travels on the surface. Since algorithms for stereo and
structure from motion do not distinguish between real and virtual features,
they end up producing incorrect depth information in the case of specular
objects. Sometimes the computed structure can be significantly different from
the actual structure of the object. In this project, we are investigating
techniques that enable stereo and structure from motion to detect and handle
specular reflections.
The first part of the project focuses on the recovery of a specular surface
using a moving camera with known motion. Based on the notion of caustics, we
have developed a novel feature classification algorithm that can distinguish
real and virtual features from their image trajectories due to camera motion.
Next, using support functions of curves, a closedform relation is derived
between the image trajectory of a virtual feature and the geometry of the
specular surface it travels on. It is shown that when the camera motion and the
surface profile are coplanar, the profile is uniquely recovered by tracking
just two unknown virtual features. These results have also been extended to the
case of arbitrary 3D surface profiles that are traveled by virtual features
when camera motion is not confined to a plane.
In the second part of the project, we have studied the the problem of
accurate depth estimation using stereo in the presence of specular reflections.
Since specular reflection is viewpoint dependent, it can cause large intensity
differences at corresponding scene points in a stereo image pair. This results
in significant errors in the computed scene structure. We have analyzed the
physics of specular reflection and the geometry of stereopsis and arrived at a
relationship between stereo vergence, surface roughness and the likelihood of a
correct match. Given a lower bound on surface roughness, an optimal binocular
stereo configuration can be determined which maximizes precision in depth
estimation despite specular reflection. However, surface roughness is difficult
to estimate in unstructured environments. Therefore, trinocular configurations
that are independent of surface roughness are determined such that at each
scene point that is visible to all three cameras, at least one stereo pair can
produce correct depth. We have developed a simple algorithm to reconstruct
depth from the multiple stereo pairs.
Finally, we have developed ordinal measures (which are based on
rankpermutations of intensity values) for solving correspondence and tracking
problems in stereo and motion. We have shown that ordinal measures are more
robust to nonideal imaging conditions than the more commonly used correlation
and the sum of squared difference measures. 
Publications
"On the Motion and Appearance of Specularities in Image Sequences," R. Swaminathan, S.B. Kang, R. Szeliski, A. Criminisi and S.K. Nayar, European Conference on Computer Vision (ECCV), Vol. I, pp. 508523, May. 2002. [PDF] [bib] [©]
"Ordinal Measures for Image Correspondence," D.N. Bhat and S.K. Nayar, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 4, pp. 415423, Apr. 1998. [PDF] [bib] [©]
"Stereo and Specular Reflection," D.N. Bhat and S.K. Nayar, International Journal on Computer Vision, Vol. 26, No. 2, pp. 91106, Feb. 1998. [PDF] [bib] [©]
"Motion Estimation Using Ordinal Measures," D.N. Bhat, S.K. Nayar and A. Gupta, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 982987, Jun. 1997. [PDF] [bib] [©]
"A Theory of Specular Surface Geometry," M. Oren and S.K. Nayar, International Journal on Computer Vision, Vol. 24, No. 2, pp. 105124, 1996. [PDF] [bib] [©]
"Ordinal Measures for Visual Correspondence," D.N. Bhat and S.K. Nayar, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 351357, Jun. 1996. [PDF] [bib] [©]
"A Theory of Specular Surface Geometry," M. Oren and S.K. Nayar, IEEE International Conference on Computer Vision (ICCV), pp. 740747, Jun. 1995. [PDF] [bib] [©]
"Stereo in the Presence of Specular Reflection," D. Bhat and S.K. Nayar, IEEE International Conference on Computer Vision (ICCV), pp. 10861092, Jun. 1995. [PDF] [bib] [©]
"Structured Highlight Inspection of Specular Surfaces," A.C. Sanderson, L.E. Weiss and S.K. Nayar, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 10, No. 1, pp. 4455, Jan. 1988. [PDF] [bib] [©] [Project Page]
"Determining Surface Orientations of Specular Surfaces by Intensity Encoded Illumination," Shree K. Nayar and Arthur C. Sanderson, Proceedings of The International Society for Optical Engineering (SPIE), Vol. 850, pp. 122127, Nov. 1987. [PDF] [bib] [©]

Images


Specular Highlight Stereo:
This picture shows the fundamental ambiguities related to surface shape that
arise when stereo is applied to a specular surface. If an array of light
sources with known directions are used, then the highlights measured by two
cameras can be used to disambiguate surface normals. Once the normals have been
computed, they can be integrated to obtain surface shape.






Real and Virtual Features from Caustics:
This picture shows how caustics computed from image trajectories produced by
scene features can be used to classify each feature as real or virtual. Real
features produce compact (a point in the ideal case) caustics, while virtual
features produce larger caustics.






Specular Surface Geometry from Motion:
This picture shows experimental results on structure from motion for a highly
specular surface. The image trajectories produced by just two virtual features
on the surface are sufficient to compute the profile of the surface.






Avoiding Specularities in Stereo:
The vergence angles of a trinocular stereo system can be chosen such that, for
any surface normal and surface roughness, at least two of the three views do
not produce noticeable specular reflections. Therefore, the three views can be
used to compute a good depth map of the scene.





Shape from Brightness
NonSingle Viewpoint Imaging: Raxels and Caustics
Photometric Invariants for Segmentation and Recognition
Radiometric Camera Calibration


