CAVE | Projects: Appearance Matching

Appearance Matching

In contrast to the traditional approach of recognizing objects based on their shapes, we formulate the recognition problem as one of matching appearances. For any given vision task, all possible appearance variations define its visual workspace. A set of images is obtained by coarsely sampling the workspace. The image set is compressed to obtain a low-dimensional subspace, called the eigenspace, in which the visual workspace is represented as a continuous appearance manifold. Given an unknown input image, the recognition system first projects the image to the eigenspace. The parameters of the vision task are recognized based on the exact position of the projection on the appearance manifold.

The proposed appearance representation has several applications in visual perception. As examples, a real-time recognition system with 100 complex objects, an illumination planning technique for robust object recognition, and a real-time visual positioning and tracking system have been developed. The simplicity and generality of the proposed ideas have led to the development of a software library for appearance modeling and matching (SLAM). SLAM has been been used by over 100 research laboratories to develop techniques for appearance matching. It has also be used by several companies to solve real-world machine vision problems.

Publications

"Appearance-Matching with Partial Data,"
E. Hadjidemetriou and S.K. Nayar,
DARPA Image Understanding Workshop (IUW),
pp.1071-1078, Nov, 1998.
[PDF] [bib] [©]

"Parametric Feature Detection,"
S. Baker, S.K. Nayar and H. Murase,
International Journal on Computer Vision,
Vol.27, No.1, pp.27-50, Mar, 1998.
[PDF] [bib] [©]

"A Simple Algorithm for Nearest Neighbour Search in High Dimensions,"
S.A. Nene and S.K. Nayar,
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Vol.19, No.9, pp.989-1003, Sep, 1997.
[PDF] [bib] [©]

"Parametric Appearance Representation,"
S.K. Nayar, H. Murase and S.A. Nene,
Early Visual Learning, Chapter 6, pp.131-160, S.K. Nayar and T. Poggio, Oxford University Press, 1996.
[PDF] [bib] [©]

"An Experimental Comparison of Appearance and Geometric Model Based Recognition,"
J.L. Mundy, A. Liu, N. Pillow, A. Zisserman, S. Abdallah, S. Utcke, S.K. Nayar and C.A. Rothwell,
Object Representation in Computer Vision (ORCV),
pp.247-269, Apr, 1996.
[PDF] [bib] [©]

"Dimensionality of Illumination in Appearance Matching,"
S.K. Nayar and H. Murase,
IEEE International Conference on Robotics and Automation (ICRA),
Vol.2, pp.1326-1332, Apr, 1996.
[PDF] [bib] [©]

"Real-Time 100 Object Recognition System,"
S.K. Nayar, S.A. Nene and H. Murase,
IEEE International Conference on Robotics and Automation (ICRA),
Vol.3, pp.2321-2325, Apr, 1996.
[PDF] [bib] [©]

"Visual Learning and Recognition of 3D Objects from Appearance,"
H. Murase and S.K. Nayar,
International Journal on Computer Vision,
Vol.14, No.1, pp.5-24, Jan, 1995.
[PDF] [bib] [©]

"SLAM: A Software Library for Appearance Matching,"
S.A. Nene, S.K. Nayar and H. Murase,
DARPA Image Understanding Workshop (IUW),
Vol.1, pp.733-737, Nov, 1994.
[PDF] [bib] [©]

"Dimensionality of Illumination Manifolds in Eigenspace,"
S.K. Nayar and H. Murase,
Technical Report, Department of Computer Science, Columbia University CUCS-021-94,
Aug, 1994.
[PDF] [bib] [©]

"Illumination Planning for Object Recognition in Structured Environments,"
H. Murase and S.K. Nayar,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp.31-38, Jun, 1994.
[PDF] [bib] [©]

"Learning, Positioning, and Tracking Visual Appearance,"
S.K. Nayar, H. Murase and S.A. Nene,
IEEE International Conference on Robotics and Automation (ICRA),
Vol.4, pp.3237-3244, May, 1994.
[PDF] [bib] [©]

"Learning and Recognition of 3D Objects from Appearance,"
H. Murase and S.K. Nayar,
IEEE Workshop on Qualitative Vision,
pp.39-50, Jun, 1993.
[PDF] [bib] [©]

Videos

If you are having trouble viewing these videos in your browser, please save them to your computer first (by right-clicking and choosing "Save Target As..."), and then open them.

		Parametric Eigenspace Representation: This video shows (shown in three dimensions) the parametric eigenspace representation of a toy car. The manifold in this case is parametrized by the pose angle of the car. The inset video shows the car moving around on a table. The car region is segmented from each frame of the video and this segmented region is projected (white dot) to the manifold representation of the car. As seen, the projection always lies on the manifold of the car and its location on the manifold reveals the pose of the car.

		100 Object Recognition System: This video shows a real-time 3D recognition system with 100 complex objects in its database. A complete recognition and pose estimation cycle is completed in less than 0.5 seconds using no more than standard workstation and a color CCD camera. A recognition cycle includes scene change detection, object segmentation, brightness and scale normalizations and appearance matching. The result is displayed in the video using a sample image of the recognized object and its pose in degrees.

		Robot Positioning: The appearance matching framework has been used for real-time positioning of a robot with respect to an object. In this video, the hand-eye system is displaced by an unknown (random) position with respect to the object. The novel image (within a fixed preselected window) is projected to the parametric eigenspace to determine the displacement of the robot with respect to the desired position. This computed distance is used to drive the robot end-effector back to the desired position. This system has been used for precise insertion of chips on circuit boards.

		Real-Time Robot Tracking: The efficiency of appearance matching makes it possible to incorporate the above positioning algorithm in a real-time robot control loop. In this video, displacement vectors computed by appearance matching are used as errors in a feedback control loop that enables the robot to track a three-dimensional object as it travels through an unknown trajectory in space.

		Temporal Appearance Model of Scanned Object: Appearance matching can be used to perform coarse inspection of complex manufactured parts. In this video, a hand-eye robot system is driven through a preselected trajectory that allows visual scanning of all the pertinent areas of a manufactured part (e.g. a printed circuit board). The sequence of images taken along the trajectory is compactly represented as a curve in eigenspace that is parametrized by robot travel time (along the trajectory).

		Temporal Inspection: In this video, the hand-eye system is driven through the same trajectory for a defective part and the images are projected to the eigenspace of the model part. The red flashing of the inset image is used to communicate large visual deviations (defects) from the stored model.