Description of the Action Capture Project.

Developed by the Virtual Reality and Multimedia Research Group at the Technical University of Freiberg

What is Action Capture?

Action Capture is a novel method for virtual character animation. In this approach, virtual characters learn to imitate the actions of Virtual Reality (VR) users by tracking not only the users' movements but also their interactions with scene objects. Action capture builds on conventional motion capture but differs from it in that higher-level action representations are transferred rather than low-level motion data. As an advantage, the learned actions can often be naturally applied to varying situations, thus avoiding retargetting problems of motion capture. The idea of action capture is inspired by human imitation learning; related methods have been investigated for a longer time in robotics.

From "Motion Capture" to "Action Capture"

Research on human imitation learning has long distinguished between imitation at the level of mere movements and imitation at the higher level of actions. According to M. Arbib, action = movement + goal, i.e. actions are always associated with a goal. For example, when one is shown the act of grasping an object, successful imitation involves the reenactment of the actions's effect, i.e. that the object is grasped, even if the imitiator's movements differ from the original movements, e.g. due to different spatial relationships between the actors and the object being grasped. Similarly, studies in developmental psychology on imitative abilities in infants distinguish between an earlier stage of imitation of body movements and a later stage of imitation of action on objects.
Action Capture thus represents a next step in the synthesis of life-like character animations by not only reproducing an actor's movements (motion capture) but also his or her interactions with objects in the environment. This is accomplished by placing the actor in an interactive VR environment.

Technical Components of Action Capture Systems

Action capture extends conventional motion capture by the use of interactive VR technology. Essential action capture technologies comprise:

  1. Immersive VR technology including position tracking systems and data gloves. The virtual environment should support its interactive manipulation by means of realistic, not idealized hand-object interactions.
  2. Action recognition software that analyzes the VR user's movements and manipulations of scene objects to produce high-level action descriptions.
  3. Behavioral animation techniques by which animations of virtual characters can be synthesized from high-level action commands.

Prototype Implementation

In our prototypical action capture system, the VR user can perform simple manipulations of the virtual scene which are then reproduced by a virtual human. The user's interactions are first segemented into characteristic phases of the manipulation, such as grasping an object, releasing it, and reach motions. These phases are combined into high-level, symbolic action descriptions that also describe the type of the grasp. Our work on grasp classification methods demonstrates that grasp types can be reliably recognized even with uncalibrated data gloves. The action representations or plans are then transmitted to a multi-layered behavioral animation system that reproduces the demonstrated interactions.

Team

Prof. Dr.-Ing. Bernhard Jung
Dipl.-Inform. Guido Heumer
Dipl.-Inform. Arnd Vitzthum
Dipl.-Inform. Heni Ben Amor
Maik Deininger
Frank Gommlich
Peter Menzel
Marcus Priefert
Frank Winkler

Funding

Research on Action Capture is funded by the Deutsche Forschungsgemeinschaft (DFG) in the "Virtual Workers" project