Modeling, analysis, and animation of people has many potential applications. Examples are new paradigms in human computer interaction, visual surveillance, video database annotation, graphics, and special effects. In almost all interesting scenarios the subjects are in motion. During talking, complex configurations and subtle lip motions are generated. During gesturing, walking, and other actions, coarse articulated limb and body movements are generated. Depending on the kind of motion and application, different abstractions and resolutions are required. Some motions are very constrained, like speaking lips or walk styles; these can be learned from data. Other actions only satisfy very general constraints. Such constraints can be coded a-priori. Recognition tasks require extracting and modeling only a few discriminating features. Animation tasks require capturing every subtle detail.
The core techniques that we developed for this domain include linear subspace and nonlinear mixture models applied to learning of shape and dynamics, Hidden Markov Models for classification of visual phonetic units and gait categories, and articulated body motion estimation techniques based on product of exponential maps and twist representations. The talk will survey these techniques and their application to automatic lip-reading, photo-realistic facial animation (Video Rewrite), gait classification, and 3D articulated motion capture. The main focus will be the description of the new kinematic tracking technique, its mathematical foundation, and its application to capturing and animating the famous photo-plate sequences of Edweard Muybridge from the last century.
This work was done at U.C. Berkeley, ICSI, and Interval Research jointly with Jitendra Malik, Jerome Feldman, Steve Omohundro, Yochai Konig, Michele Covell, and Malcolm Slaney.