Expressive Whole-Body 3D Multi-Person Pose and Shape Estimation from a Single Image (Talk)

Human is the most centric and interesting object in our life: many human-centric techniques and studies have been proposed from both industry and academia, such as virtual try-on, 3D personal avatar, and marker-less motion capture in the movie/game industry, including AR/VR. Recovery of accurate 3D geometry of humans (i.e., 3D human pose and shape) is a key component of the human-centric techniques and studies. In particular, the 3D pose and shape of multiple persons can deliver relative 3D location between persons. Also, the 3D pose and shape of the whole body, which includes hands and face, provides expressive and rich information, including human intention and feeling. In this talk, I'd like to present my research results on expressive whole-body 3D multi-person pose and shape estimation from a single image. The research results start from 3D multi-person pose and gradually get improved to expressive whole-body 3D multi-person pose and shape. First, a 3D multi-person pose estimation method (ICCV 2019) is introduced. It is the first learning-based framework for the 3D multi-person pose estimation from a single RGB image, which is greatly challenging due to the depth and scale ambiguity. The newly designed RootNet successfully and firstly extended previous 3D single person pose estimation methods to the multi-person case. Second, a 3D multi-person pose and shape estimation method (ECCV 2020) is introduced. It proposes I2L-MeshNet, an image-to-lixel (line+pixel) network for 3D human body and hand mesh estimation. I2L-MeshNet won first place at 3D pose in-the-wild (3DPW) challenge at ECCV 2020. Lastly, an expressive whole-body 3D multi-person pose and shape estimation method (WIP) is introduced. It proposes a positional pose-guided rotational pose prediction scheme, which largely outperforms all previous methods by a large margin.