We are interested in developing sensing methods and recovery algorithms for computing the reflectance and geometry of objects from one or more images. Unlike stereo or structure from motion, the images are typically captured from a single viewpoint and the object properties are computed explicitly from scene radiance values.

The structured highlight method recovers the surface normals of a highly specular object. It uses a dense array of light sources and is able to scan the set of sources efficiently by using binary coding. This system was used by Westinghouse Corporation for inspecting solder joints on surface-mount circuit boards. The photometric sampling method is an extention of photometric stereo. It uses a set of images captured under extended light sources placed in different directions and a physically-based surface reflectance model to recover the shapes and reflectances of complex objects (see the above image). Unlike traditional photometric stereo, this method can handle an entire spectrum of objects that goes from pure specular to perfectly diffuse. This method was developed based on the observation that the brightness of a scene point can be expressed as a convolution of the reflectance and the lighting of the point.

Finally, we have studied the recovery of diffuse objects in the presence of interreflections. In particular, we have analyzed the interreflections produced by concave Lambertian objects and shown that a method like photometric stereo recovers incorrect (pseudo) shape and albedo function which are invariant to the light source directions used. An algorithm is then used to recover the actual shape and albedo function of the object from the pseudo ones.

Most of this project was done at the VASC Laboratory in the Robotics Institute at Carnegie Mellon University.

Publications

"Shape Recovery Methods for Visual Inspection,"
S.K. Nayar,
IEEE Workshop on Applications of Computer Vision,
pp.136-145, Nov, 1992.
[PDF] [bib] [©]

"Colored Interreflections and Shape Recovery,"
S.K. Nayar and Y. Gong,
DARPA Image Understanding Workshop (IUW),
pp.333-343, Jan, 1992.
[PDF] [bib] [©]

"Shape from Interreflections,"
S.K. Nayar, K. Ikeuchi and T. Kanade,
International Journal on Computer Vision,
Vol.6, No.3, pp.173-195, 1991.
[PDF] [bib] [©]

"Surface Reflection: Physical and Geometrical Perspectives,"
S.K. Nayar, K.Ikeuchi and T. Kanade,
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Vol.13, No.7, pp.611-634, Jul, 1991.
[PDF] [bib] [©]

"Shape from Interreflections,"
S.K. Nayar, K. Ikeuchi and T. Kanade,
IEEE International Conference on Computer Vision (ICCV),
pp.2-11, Dec, 1990.
[PDF] [bib] [©]

"Determining Shape and Reflectance of Hybrid Surfaces by Photometric Sampling,"
S.K. Nayar, K. Ikeuchi and T. Kanade,
IEEE Transactions on Robotics and Automation,
Vol.6, No.4, pp.418-431, Aug, 1990.
[PDF] [bib] [©]

"Specular Surface Inspection using Structured Highlight and Gaussian Images,"
S.K. Nayar, L.E. Weiss, D.A. Simon and A.C. Sanderson,
IEEE Transactions on Robotics and Automation,
Vol.6, No.2, pp.208-218, Apr, 1990.
[PDF] [bib] [©]

"Shape and Reflectance from an Image Sequence Generated using Extended Sources,"
S.K. Nayar, K. Ikeuchi and T. Kanade,
IEEE International Conference on Robotics and Automation,
pp.28-35, May, 1989.
[PDF] [bib] [©]

"Structured Highlight Inspection of Specular Surfaces,"
A.C. Sanderson, L.E. Weiss and S.K. Nayar,
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Vol.10, No.1, pp.44-55, Jan, 1988.
[PDF] [bib] [©]

Images

		Structured Highlight Method: This device has 128 point sources distributed over a hemisphere. Each source is created using an LED and an optical fiber. Binary coding is used to scan the 128 sources using just seven patterns. The seven captured images are then used to compute the surface normals of the object.
		The SHINY System: The structure highlight method was used by Westinghouse Corporation in 1987 to develop the SHINY (Structured Highlight Inspection System) that was used to inspect solder joints on surface-mount circuit boards.
		Photosampler: This device uses a set of point light sources placed at the vertices of a tessellated sphere. A spherical diffuser is placed between the point sources and the object of interest to convert the point sources into extended sources. The object of interest is placed at the center of the diffuser and its images are captured while the sources are scanned. This set of images is used to compute the surface normal and the reflectance for each point on the object.
		Diffuse Interreflections: This image of a set of concave Lambertian objects shows the effects of interreflections. Once can see the increase in brightness close to concave edges that results from multiple reflections of light between surface points that are visible to each other. Shape from brightness methods will produce erroneous results if interreflections are ignored.
		The Pseudo Shape of a Concave Lambertian Object: Due to interreflections, a concave Lambertian object of given shape and albedo function behaves like an Lambertian object without interreflections but with a different shape and albedo function. This table shows a few examples of actual and pseudo shapes. In all cases the pseudo shape is shallower than the actual shape.
		Actual Surface from Pseudo Surface (2D): This picture shows shape and reflectance recovery results for two objects with translational symmetry. In each case, the pseudo shape and reflectance were measured using traditional photometric stereo. The actual shape and reflectance were then recovered from the pseudo ones using a recovery algorithm that is based on an interreflection model.
		Actual Surface from Pseudo Surface (3D): Recovery results for a 3D object. The strong interreflections between the three faces of the inverted pyramid cause the measured pseudo shape to be much shallower than the actual shape.

Related Projects

Specularities in Stereo and Motion

Photometric Invariants for Segmentation and Recognition

Structured Light in Scattering Media

Depth from Defocus

Shape from Focus