Unified Simulation, Perception, and Generation of Human Behavior (Talk)

Understanding and modeling human behavior is fundamental to almost any computer vision and robotics applications that involve humans. In this talk, I will present a holistic approach to human behavior modeling and tackle its three essential aspects --- simulation, perception, and generation. I will show how the three aspects are deeply connected and how utilizing and improving one aspect can greatly benefit the other aspects. Since humans live in a physical world, we treat simulation as the foundation of our approach and start by developing a fundamental framework for representing human behavior in physics simulation. Specifically, we model a human using a proxy humanoid agent inside a physics simulator and treat human behavior as the result of an optimal control policy for the humanoid, which can be solved by RL. Next, we leverage the simulation-based framework to effectively tackle several behavior perception tasks, including first-person and third-person human pose estimation as well as reconstructing human-scene interaction. Furthermore, in addition to perception, we also show how the simulation-based framework, in tandem with diversity-promoting deep generative models, can yield models that produce both diverse and physically-plausible human behaviors. Finally, we also explore the synergy between perception and generation, where we leverage behavior generation models to perceive invisible global human motions from videos captured by dynamic cameras. Our approach outperforms state-of-the-art methods substantially in tackling occlusions and estimating global human motions, which marks a significant step towards global human mesh recovery in the wild.