Perceiving Systems, Computer Vision

Populating 3D Scenes by Learning Human-Scene Interaction

2021

Conference Paper

ps


Humans live within a 3D space and constantly interact with it to perform tasks. Such interactions involve physical contact between surfaces that is semantically meaningful. Our goal is to learn how humans interact with scenes and leverage this to enable virtual characters to do the same. To that end, we introduce a novel Human-Scene Interaction (HSI) model that encodes proximal relationships, called POSA for “Pose with prOximitieS and contActs”. The representation of interaction is body-centric, which enables it to generalize to new scenes. Specifically, POSA augments the SMPL-X parametric human body model such that, for every mesh vertex, it encodes (a) the contact probability with the scene surface and (b) the corresponding semantic scene label. We learn POSA with a VAE conditioned on the SMPL-X vertices, and train on the PROX dataset, which contains SMPL-X meshes of people interacting with 3D scenes, and the corresponding scene semantics from the PROX-E dataset. We demonstrate the value of POSA with two applications. First, we automatically place 3D scans of people in scenes. We use a SMPL-X model fit to the scan as a proxy and then find its most likely placement in 3D. POSA provides an effective representation to search for “affordances” in the scene that match the likely contact relationships for that pose. We perform a perceptual study that shows significant improvement over the state of the art on this task. Second, we show that POSA’s learned representation of body-scene interaction supports monocular human pose estimation that is consistent with a 3D scene, improving on the state of the art. Our model and code are available for research purposes at https://posa.is.tue.mpg.de.

Author(s): Mohamed Hassan and Partha Ghosh and Joachim Tesch and Dimitrios Tzionas and Michael J. Black
Book Title: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021)
Pages: 14703--14713
Year: 2021
Month: June
Publisher: IEEE

Department(s): Perceiving Systems
Research Project(s): Capturing Contact
Putting People into Scenes
Bibtex Type: Conference Paper (inproceedings)
Paper Type: Conference

DOI: 10.1109/CVPR46437.2021.01447
Event Name: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021)
Event Place: Virtual

Address: Piscataway, NJ
ISBN: 978-1-6654-4509-2
State: Published

Links: project
pdf
poster
video
Video:

BibTex

@inproceedings{Hassan:CVPR:2021,
  title = {Populating {3D} Scenes by Learning Human-Scene Interaction},
  author = {Hassan, Mohamed and Ghosh, Partha and Tesch, Joachim and Tzionas, Dimitrios and Black, Michael J.},
  booktitle = {2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021)},
  pages = {14703--14713},
  publisher = {IEEE},
  address = {Piscataway, NJ},
  month = jun,
  year = {2021},
  doi = {10.1109/CVPR46437.2021.01447},
  month_numeric = {6}
}