Motion and Transparent Objects

The perception of transparent objects from images is known to be a very hard vision problem. Given a single image, it is difficult to even detect the presence of transparent objects in the scene. In this project, we explore what can be said about transparent objects by a moving observer. We show how features that are imaged through a transparent object behave differently from those that are rigidly attached to the scene. We present a novel model-based approach to recover the shapes and the poses of transparent objects from known motion. The objects can be complex in that they may be composed of multiple layers with different refractive indices. We have applied our algorithm to real scenes that include transparent objects and recovered the shapes and dimensions of the objects with high accuracy.

Publications

"What Does Motion Reveal about Transparency?,"
M. Ben-Ezra and S.K. Nayar,
IEEE International Conference on Computer Vision (ICCV),
Vol. 2, pp. 1025-1032, Oct. 2003.
[PDF] [bib] [©]

"Transparent Grippers for Robot Vision,"
A. Nikolaev and S.K. Nayar,
IEEE International Conference on Robotics and Automation (ICRA),
Vol. 2, pp. 1644-1649, Apr. 1996.
[PDF] [bib] [©]

Images

  Refraction through Transparent Objects:

A ray of light interacts with a transparent object at two interfaces. The first interaction occurs when the ray passes from the surrounding medium having a refractive index of u1 into the transparent object having a refractive index u2. The second interaction occurs when the ray passes from the transparent object back into the surrounding medium. At each interface a portion of the ray is reflected and another portion is refracted.

     
  Recovery Method:

The set of parallel incident rays from a distant feature are scattered by a transparent object in various directions. Some of these rays are captured by a moving camera, when they pass through the optical center of the camera. Given the intrinsic parameters and motion of the camera, we can compute the location (in 3D) of each captured ray in space. Note that if we knew the shape and pose of the object, these rays could be traced backwards to form a set of (outgoing) parallel rays. This holds true regardless of the complexity of the transparent object. It also remains valid for rays that go through total internal reflections (like the one imaged by the camera in the figure) as well as rays that do not pass through the transparent object (such as the bottom most one).

     
  Initial Guess:

The initial guess shape overlaid on one of the images. The initial guess was a sphere (n=1.0).

     
  The Cube:

An image of the actual cube used in the experiment shown to convey its shape. Its refractive index u is unknown.

     
  Computed Shape:

The recovered shape of the cube (n=0.18).

     
  Initial Guess:

The initial guess shape is overlaid on one of the images. The initial diameter of the cylinder was 200mm and the initial thickness of its wall was 20mm.

     
  The Cylinder:

An image of the actual acrylic cylinder. It has a diameter of 117mm and wall thickness of 3mm.

     
  Computed Shape:

The computed diameter of the cylinder is 116.1mm and the computed thickness of its wall is 2.3mm. The results are accurate even though this is a very complex case with 4 interfaces.

     

Videos

  Transparent Cube and Moving Camera:

This video clip shows the nine views used to recover the shape of a transparent cube with rounded edges.

     
  Pipe Video:

This video clip shows the nine views used to recover the shape of an acrylic cylinder filled with water (4 interfaces).

     

Slides

ICCV 2003 presentation     With videos (zip file)

Dirty Glass: Contamination on Transparent Surfaces