Tracking and Grasping Moving Objects

Contact: Bill Yoshimi <>
Our laboratory was one of the first laboratories to exploit data parallelism in real-time computer vision. We have been responsible for the first set of low-level image processing algorithms written for the PIPE dataflow architecture and these algorithms have been a part of our technology transfer program to companies such as Siemens, Lockheed, FMC, and North American Philips. Machine vision has become an important research topic as the systems currently being developed are finally capable of processing rates to sustain real-time control from visual feedback.

We are interested in exploring the interplay of hand-eye coordination for dynamic grasping tasks where objects to be grasped are moving. Coordination between an organism's sensing modalities and motor control system is a hallmark of intelligent behavior, and we are pursuing the goal of building an integrated sensing and actuation system that can operate in dynamic as opposed to static environments. The system we are building is a multi-sensor system that integrates work in real-time vision, robotic arm control and stable grasping of objects. Our first attempts at this have resulted in a system that can track and stably grasp a moving model train in real-time. (See figures and video below.)

The algorithms we have developed are quite general and applicable to a variety of complex robotic tasks that require visual feedback for arm and hand control. Currently, we are extending this work to tracking in a full 3-D space, and instituting 2 new control algorithms that employ multiple moving object detection and a wrist mounted camera for finer tracking and grasping. This work is important in showing 1) that the current level of computational devices for vision and real-time control are sufficient for dynamic tasks that include moving objects, 2) optic-flow is robust enough to be computed in real-time for stereo matching and 3) it can define strategies for robotic grasping with visual feedback that may be motivated by human arm movement studies.

Robot arm picking up moving train.

Click for an mpeg video of our experiment. WARNING!!! the video is 4880128 bytes long so you may have to wait a while...

Return to the Robotics Lab home page