Paul Oh Home Page Paul Oh Resume Page Paul Oh Consulting Page Paul Oh Columbia Robotics Page

Research Page

My research interests include visually servoed robotics, robot hands and sensor fusion, and control system design. My multi-disciplined background is in mechanical, electrical and computer engineering.

Demo Related

Read the following documents to visually track people or the Puma robots with the gantry

Videos

I presented S-VHS tapes of demos at the IEEE Robotics and Automation Conference (1998 Belgium and 1999 Detroit). I transfered tape contents to MPEG files. Click here to download/view and see descriptions.

Published Papers

Click to download/view Postscript or Acrobat versions of these papers. Please note that these papers are copyrighted by the authors.

    Visually Servoing

  1. "Design of a Partitioned Visual Feedback System," Paul Y. Oh, Peter K. Allen, IEEE International Conference on Robotics and Automation (ICRA), Leuven, Belgium, pp. 1360-1365, 1998 Postscript - PDF
  2. "Performance of a Partitioned Visual Feedback System," Paul Y. Oh, Peter K. Allen, IEEE International Conference on Robotics and Automation (ICRA), Detroit, Michigan, pp. 275-281, 1999 Postscript - PDF
  3. "Coupling Effects for Visually Servoed Feedback," Paul Y. Oh, Peter K. Allen, IEEE International Conference on Robotics and Automation (ICRA) San Francisco, CA, submitted for publication, 2000 Postscript - PDF

    Robot Hands/Sensor Fusion

  4. "Integration of Vision, Force and Tactile Sensing for Grasping," Peter K. Allen, Andrew Miller, Paul Y. Oh and B. Leibowitz, International Journal of Intelligent Machines, Vol. 4, No. 1, pp. 129-149, January 1999
  5. "Using Tactile and Visual Sensing with a Robotic Hand," Peter K. Allen, Andrew Miller, Paul Y. Oh and B. Leibowitz, IEEE International Conference on Robotics and Automation (ICRA), Albuquerque, New Mexico, pp. 676-681, 1997 Postscript - PDF
  6. "Integration of Vision and Force Sensors for Grasping," Peter K. Allen, Andrew Miller, Paul Y. Oh and B. Leibowitz, IEEE/SICE/RSJ International Conference on Multi-sensor Fusion and Integration for Intelligent Systems, Washington, DC, pp. 349-356, 1996 Postscript - PDF
  7. "Visual Control for Robotic Hand-eye Coordination," Peter K. Allen, B. Yoshimi, Andrew Miller, Paul Y. Oh, B. Leibowitz, Workshop on Robotics and Robot Vision, 4th International Symposium on Signal Processing and its Applications, ISSPA 96, Gold Coast, Australia, pp. 20-37, August 1996

    Stewart Platform/Control

  8. "Improved Model Reference Adaptive Control of Electro-hydraulic Servo Systems Using The Euler Operator," Paul Y. Oh, IEEE International Conference on Robotics and Automation (ICRA), Albuquerque, New Mexico, pp. 1626-1631, 1997 Postscript - PDF


Visual Servoed Robots - Hands/Sensor Fusion - Stewart Platform



Visually Servoed Robots

My specific interest in machine vision is to monitor a large assembly workcell (about the size of a classroom). I want to visually track objects like tools, workpieces and grippers as they move around in the workcell. We therefore custom built a ceiling-mounted gantry and attached a pan-tilt unit (PTU) and camera at the end-effector. The net result is a hybrid 5 degrees-of-freedom (DOF) robot that can position and orient the camera anywhere in the workspace. Our hybrid robot servos the camera to keep the moving object centered in its field-of-view and at a desired image resolution.

Approach

Traditionally researchers attacked similar problems by measuring 3-D object pose from 2-D camera images. This requires a priori knowledge of the object geometry and hence researchers typically use CAD-based models or paint fiducial marks at specific locations on the object. The 3-D object pose measurements are then used with image and manipulator Jacobians to map velocity changes in the camera's image space to the robot's joint space. The net effect is that the robot servos the camera to regulate a desired camera-to-object pose constraint.

The caveat of such a regulation technique is that the robot's motors may not have sufficient bandwidth (torque capabilities) to maintain such a constraint. Our gantry is slow because of its heavy linkages. Failure to accelerate the camera fast enough will result in loss of visual contact. Furthermore, abrupt accelerations generate endpoint vibrations which effect image acquisition. By contrast, the PTU is lightweight and fast and can quickly rotate the camera to maintain visual contact. The net effect is that tracking performance depends on which DOF are invoked in the tracking task.

My approach to the tracking problem was to design a control law that defines a joint coupling between the PTU and gantry. This idea came from casually observing human tracking behavior. People also have joints (eyes, neck, torso) of varying bandwidths and kinematic range. We synergize all of our DOF when tracking a moving object and we don't need a priori object geometry knowledge. One also notices that the eyes and neck tend to pan in the same direction as we follow an object's motion trajectory. This behavior suggests an underlying kinematic joint coupling.

Implementation

Traditional approaches rely exclusively on image data. By contrast, most shop floor robots only use kinematic joint encoder data. A joint coupling can be achieved by combining both image and kinematic data. Image data is used to control pan and tilt DOFs to keep the target centered in the camera's field-of-view. The resulting kinematic angle data is used by the gantry to transport the camera in the direction of pan and/or tilt. By defining such a joint coupling in the underlying control law we mimic the synergistic human tracking behavior mentioned previously. The net effect of partitioning the DOFs in this manner is a tracking system that (1) does not need a CAD-based model; (2) can quickly track targets by taking advantage of the PTU's fast motor bandwidth; (3) can transport the camera anywhere in the workcell by taking advantage of the gantry's large kinematic range.

Results

Our assembly workcell includes an industrial Puma robot mounted with a Toshiba multi-purpose gripper to pick up tools and workpieces. We like to visually track this gripper as it moves in the workcell.

A single sum-of-squared differences (SSD) tracker was used to monitor the image position of the hand in the camera. This text-book image processing technique uses correlation to match blocks of pixels from one image frame to the next yielding real-time (30 frames/sec) results.




 The gripper moves in a triangular trajectory; its position changes vertically, horizontally and in depth. A partitioned joint-coupling was defined between the tilt and vertical gantry DOF, and the pan and horizontal gantry DOF. SSD scale data was used to servo the remaining gantry DOF to maintain a desired image resolution (i.e. depth). The results were videotaped by both a handheld camera and the gantry-PTU camera. Image sequences from both are given below.


Handheld video camera


Gantry-PTU camera


Impact

The gripper speed ranged from 6 to 20 cm/s and was effectively tracked by the partitioned gantry-PTU system. By contrast, a traditional regulator was implemented by failed at gripper speeds greater than 2 cm/s due to the limited gantry motor bandwidth. The net effect is that the partitioned system can track faster moving objects, maintain image resolution, and does not a priori knowledge of object geometry.

By using a single SSD tracker a wide range of geometrically complex objects can be tracked using partitioning. For example, the system can track a person walking around the workcell.

Partitioning can also visually track people


Tracking geometrically complex targets significant commercial and industrial applications. The results of this research can be used in the security, manufacturing and media industries for surveillance, inspection and filming tasks.

Published Papers

The technical details of joint coupled partitioned visual servoing design have been published and presented at International Conferences. Both qualitative and quantitative results can be found in the following papers. Click to download/view Postscript or Acrobat versions of these papers. Please note that these papers are copyrighted by the authors.
  1. "Design of a Partitioned Visual Feedback System," Paul Y. Oh, Peter K. Allen, IEEE International Conference on Robotics and Automation (ICRA), Leuven, Belgium, pp. 1360-1365, 1998 Postscript - PDF
  2. "Performance of a Partitioned Visual Feedback System," Paul Y. Oh, Peter K. Allen, IEEE International Conference on Robotics and Automation (ICRA), Detroit, Michigan, pp. 275-281, 1999 Postscript - PDF
  3. "Coupling Effects for Visually Servoed Feedback," Paul Y. Oh, Peter K. Allen, IEEE International Conference on Robotics and Automation (ICRA) San Francisco, CA, submitted for publication, 2000 Postscript - PDF

Grasping and Sensor Fusion

Grasping arbitrary objects with robotic hands remains a difficult task with many open research problems. Most robotic hands are either sensorless or lack the ability to report accurate position and force information relating to contact.

By fusing finger joint encoder data, tactile sensor data, strain gage readings and vision, we can increase the capabilities of a robotic hand for grasping and manipulation tasks. The experimental hand we are using is the Barrett Hand (left photo), which is a three-fingered, four DOF hand.



The hand is covered with tactile sensors which are used to localize contacts on the surface of the hand, as well as determine contact forces. The hand also has strain gages inside each finger as seen below:

Cutaway diagram of finger reveals internal strain gages


The four strain gages form a Wheatstone bridge. The net effect is that strain readings are proportional to applied fingertip forces due to the free-moving pulley, flexible beam and cables.

My contribution to this research was to mathematically model finger curvature in response to applied forces. Vision and the tactile sensors can be used to measure the point location of the applied force and strain gages measure the magnitude of this force. I used classical beam theory to model finger deflection as a function of this sensor data.

10 different weights were hung at known positions along the finger and strain readings were collected. Using least squares, the data was fit to the model to estimate parameters such as Young's modulus and the finger's moment of inertia.

Impact

This model gives a deterministic measure of the deflection as a function of position along the finger. This information was then fused with image data from a tripod-mounted camera, and tactile sensor readings to augment grasp configuration. Experiments in using the hand to screw a canistor lid were successfully accomplished.


Barrett Hand screwing on a canister lid

Published Papers

The technical details of grasping and sensor fusion have been published and presented at International Conferences. Both qualitative and quantitative results can be found in the following papers. Click to download/view Postscript or Acrobat versions of these papers. Please note that these papers are copyrighted by the authors.
  1. "Integration of Vision, Force and Tactile Sensing for Grasping," Peter K. Allen, Andrew Miller, Paul Y. Oh and B. Leibowitz, International Journal of Intelligent Machines, Vol. 4, No. 1, pp. 129-149, January 1999
  2. "Using Tactile and Visual Sensing with a Robotic Hand," Peter K. Allen, Andrew Miller, Paul Y. Oh and B. Leibowitz, IEEE International Conference on Robotics and Automation (ICRA), Albuquerque, New Mexico, pp. 676-681, 1997 Postscript - PDF
  3. "Integration of Vision and Force Sensors for Grasping," Peter K. Allen, Andrew Miller, Paul Y. Oh and B. Leibowitz, IEEE/SICE/RSJ International Conference on Multi-sensor Fusion and Integration for Intelligent Systems, Washington, DC, pp. 349-356, 1996 Postscript - PDF
  4. "Visual Control for Robotic Hand-eye Coordination," Peter K. Allen, B. Yoshimi, Andrew Miller, Paul Y. Oh, B. Leibowitz, Workshop on Robotics and Robot Vision, 4th International Symposium on Signal Processing and its Applications, ISSPA 96, Gold Coast, Australia, pp. 20-37, August 1996

Stewart Platform and Electrohydraulic Servovalve Control

A Stewart platform (left) is a 6 degree-of-freedom mechanism that is commonly used in flight simulators. The payload rests of the top platform and the linkages extend to yield yaw, pitch, roll orientations as well as vertical, sway and heave positions. My interests in the Stewart Platform were in designing a ship motion simulator (SMS) control system for the Korean Agency of Defense Development. The end-goal of the project was to mount an automatic firing mechanism on high-speed gunboats.

The SMS linkages are electrohydraulic. The platform positioning accuracy depends on operating and ambient factors such as temperature and fluid viscosity. Thus I designed a model reference adaptive controller (MRAC) to compensate for fluctuations in these factors.




Preamble

Typically there are two approaches to digital control design. One approach is to use an analog model in the Laplace s-domain and then discretize using a zero-order-hold (ZOH). Another approach is to use a z-domain digital model from the very beginning of the design stage.

Both approaches have their advantages and disadvantages. Analog modeling gives the designer a better understanding of the real-world system, especially in terms of bandwidth and linearities. Digital modeling however readily lends itself to computer implementation but obscures real-world system understanding. This is in part due to the nature of the z-transform.

Intuitively one would think that as the sampling time approaches zero (i.e. a very fast sampling frequency) the discrete model should approach the form of the analog model. However the z-transform and ZOH does not yield this. In fact, at fast sampling frequencies, the discrete model will be non-minimum phase, that is, the discrete zeros will lie outside the unit circle. Since many controllers, including the MRAC rely on pole-zero cancellation, non-minimum phase must be eliminated to avoid instability. A stable control law thus requires using a large sampling time which leads to a loss in measurement accuracy.

The non-minimum phase phenomena is a result of using a shift operator (i.e. the z-transform and ZOH). In fact, shift operators are the reason why analog and discrete versions of the same control law (e.g. optimal, adaptive) exist. Again, intuitively one would think the discrete version of a control law should equal the analog version when the sampling time is zero.

Approach

I used the Euler operator to design a digital controller take gives advantages of analog design insight. This operator is just as easy to use as the shift operator and is consistent with intuition. As the sampling time approaches zero, the discrete Euler model approaches its analog equivalent. If the analog model is minimum phase then so will the discrete model. In fact, all continuous-time techniques can be readily applied to the discrete Euler model. For the Euler operator, the region of stability is a circle centered at -1/T with radius 1/T. As the sampling time, T, approaches zero, this stability region is the same as the Laplace s-domain. By contrast, the shift operator's stability region is always the unit circle irregardless of sampling time size.

Implementation


One leg of the Stewart Platform

Using the Euler operator I designed a MRAC to control the position of single electrohydraulic link of the 6-dof Stewart Platform. The sampling time was 25 ms (40 Hz) which would have led to non-minimum phase zeros if the shift operator was used. The control law was programmed in Pascal on a 386 PC and analog-to-digital board (in 1991).

For system identification, a fast Fourier transform (FFT) machine was used to acquire the servovalve's Bode plot. This resulted in a third order Laplace transfer function which was then discretized with the Euler operator.

A model of the servovalve, with desired performance characteristics, was then designed for the adaptive strategy. This model was nominalized for ideal ambient and operating factors. The input was fed into both this model and the actual servovalve and outputs were compared. The output differences define an error which the adaptive strategy minimizes to generate a compensated input.

To demonstrate the MRAC's effectiveness the hydraulic supply pressure and fluid temperature were manually changed while in operation. Despite these changes, the electrohydraulic servovalve positioning error was negligible.

Published Papers

The design details and results (English) of Euler-based MRAC have been published and presented at International Conferences. Click to download/view Postscript or Acrobat versions of this paper. Please note that this paper is copyrighted by the author.
  1. "Improved Model Reference Adaptive Control of Electro-hydraulic Servo Systems Using The Euler Operator," Paul Y. Oh, IEEE International Conference on Robotics and Automation (ICRA), Albuquerque, New Mexico, pp. 1626-1631, 1997 Postscript - PDF

Videos

At the IEEE International Conference on Robotics and Automation (ICRA) I presented S-VHS tapes of demos. The tapes demonstrates a working prototype - a 5-DOF hybrid gantry robot that can monitor tools, grippers, and parts moving about a large assembly workcell. I transfered tape portions to MPEG files which you can freely download/view. These files give viewers a sense of partitioning tracking performance. Note these files are large!

ICRA 1998 Leuven, Belgium

  1. icra98Scene02PtuMotions.mpg (2.8 MB - 16 second footage) Shows camera movement via the robot's 5 degrees-of-freedom (XYZ gantry and pan-tilt unit).
  2. icra98Scene04PeopleTracking.mpg (8.2 MB - 48 second footage) Tracking a person around the workcell's perimeter. The pan and tilt DOF are coupled to the horizontal and vertical gantry DOF respectively. Note tracking is robust despite non-deterministic head motions (head turns, sways, bobs).
  3. icra98Scene05PeopleTrackingCCDView.mpg (11.3 MB - 1 minute 6 second footage) Same as above, but the robot camera's point-of-view (POV). Note that XVision's sum-of-square differences (SSD) tracker is quite robust to minor occlusions!

ICRA 1999 Detroit, Michigan

  1. icra99Scene09Triangle.mpg (11.6 MB - 1 minute 8 second footage) Tracking a robot gripper (Toshiba Hand) that moves in a triangular trajectory. Scale data effectively regulates depth with a single moving camera. No a priori knowledge of hand trajectory was used.
  2. icra99Scene09Triangle.mpg (8.7 MB - 51 second footage ) Camera POV of above demo - Depth, with a single camera, is regulated despite gantry end-point vibrations.
  3. icra99Scene13PeopleSimpleDepth.mpg (6.6 MB - 38 second footage) Depth regulation technique also used to track a person.
  4. icra99Scene14PeopleSimpleDepthCCD.mpg (8.3 MB - 48 second footage) Camera POV of above demo.

Email:
paul@cs.columbia.edu


Back to Home - Resume - Consulting / Boondog - Columbia Robotics