This project is my Plan B project for M.S. degree at UMN, instructed by Prof. Stephen J. Guy. It is about data-driven character animation. The data I used is daily videos, instead of motion capture data, which is a more difficult task. First task is to generate pose estimation given a video clip, and transfer the pose to artificial character and reconstruct the 3d pose. Reinforcement learning is necessary in the process to generate plausible pose.
Introduction
Physics-based simulation of passive phenomena of objects is common, while modeling motion of humans and animals is challenging.
Manually made controller could be stylized and engaging, however it has many drawbacks,
It doesn’t have generality;
It requiring considerable effort to construct, the range of motions is limited to the space of all possible reactions
How to get motions?
- Motion Capture
- In-the-wild videos
Goal: train physically simulated characters to imitating reference motion from daily videos
Pose estimation
Use OpenPose to generate 2D real-time multi-person keypoint detection.
Keypoints are stored in array and passed to another model from VideoPose3D - 3D human pose estimation in video with temporal convolutions and semi-supervised training,
3D reconstruction
Chanllenges:
- Lack of largescale ground truth 3D annotation for in-the-wild images
- Inherent ambiguities in single-view 2D-to-3D mapping
- Depth ambiguity where multiple 3D body configurations explain the same 2D projections
- …
After extracting features from images, use a temporal generation network to preserve some temporal information, also apply adversarial learning a large-scale MoCap dataset to generate kinematically plausible motion sequences.
Pose transfer:
Sequences of poses ransformed into:
root position (3D), root rotation (4D), chest rotation (4D), neck rotation (4D), right hip rotation (4D), right knee rotation (1D), right ankle rotation (4D), right shoulder rotation (4D), right elbow rotation (1D), left hip rotation (4D), left knee rotation (1D), left ankle rotation (4D), left shoulder rotation (4D), left elbow rotation (1D)
Motion imitation
Simulation environment: PyBullet
State: relative positions, rotations, linear and angular velocities of each link with respect to the root
Action: target orientations for PD controllers at each joint
Policy determines which actions should be applied at each timestep in order to reproduce the desired motion
Test on mocap data (up) and in-the-wild video (down) after training with 8 workers for 1 day.