PROX dataset: 3D people interacting with 3D scenes (ICCV 2019)

27 October 2019

05:39

Note: This video shows the PROX reference data obtained by fitting to RGB-D. This does not show the results of PROX on RGB. The goal of this dataset is to provide "reference" data that can be used to evaluate methods for 3D human pose estimation in realistic scenes with occlusion. See the paper and code for our method (PROX) for solving this problem. Abstract: To understand and analyze human behavior, we need to capture humans moving in, and interacting with, the world. Most existing methods perform 3D human pose estimation without explicitly considering the scene. We observe however that the world constrains the body and vice-versa. To motivate this, we show that current 3D human pose estimation methods produce results that are not consistent with the 3D scene. Our key contribution is to exploit static 3D scene structure to better estimate human pose from monocular images. The method enforces Proximal Relationships with Object eXclusion and is called PROX. To test this, we collect a new dataset composed of 12 different 3D scenes and RGB sequences of 20 subjects moving in and interacting with the scenes. We represent human pose using the 3D human body model SMPL-X and extend SMPLify-X to estimate body pose using scene constraints. We make use of the 3D scene information by formulating two main constraints. The interpenetration constraint penalizes intersection between the body model and the surrounding 3D scene. The contact constraint encourages specific parts of the body to be in contact with scene surfaces if they are close enough in distance and orientation. For quantitative evaluation we capture a separate dataset with 180 RGB frames in which the ground-truth body pose is estimated using a motion-capture system. We show quantitatively that introducing scene constraints significantly reduces 3D joint error and vertex error. Data and project: https://prox.is.tue.mpg.de. Code: https://github.com/MohameHassan/PROX pdf: https://ps.is.tuebingen.mpg.de/uploads_file/attachment/attachment/530/ICCV_2019___PROX.pdf Citation: @inproceedings{PROX:2019, title = {Resolving {3D} Human Pose Ambiguities with {3D} Scene Constraints}, author = {Hassan, Mohamed and Choutas, Vasileios and Tzionas, Dimitrios and Black, Michael J.}, booktitle = {International Conference on Computer Vision (ICCV)}, month = oct, year = {2019} }