Research Overview

Computer vision – Our goal

Computer vision is often treated as problem of pattern recognition, 3D reconstruction, or image processing. While these all play supporting roles, our view is that the goal of computer vision is to infer what is not in the picture.  The goal is to recognize the unseen.  This is different from the Aristotelian view that “vision is knowing what is where by looking.” We see vision as the process of inferring the causes and motivations behind the images that we observe; that is, we want to infer the story behind the picture.

The most interesting stories involve people. Consequently, our research focuses on understanding humans and their actions in the world. We aim to recover human behavior in detail, including human-human interactions, and human interactions with the environment.

Development of Models and algorithms for computer training

Humans interact with each other and manipulate the world through their bodies, faces, hands and speech. If computers are to understand humans and our behavior, then they are going to have to understand much more about us than they currently do. For example, they need to recognize when we are picking up something heavy and might need help. They need to understand when we are distracted. They need to understand that changes in our behavior may signal medical or psychological changes.

To address this, we develop datasets, tools, models, and algorithms to train computers to recover human movement in unconstrained scenes at a level not previously possible. From single images or videos, we estimate full 3D body pose, including the motion of the face and the pose of the hands. We also recover the 3D structure of the world, its motion, and the objects in it so that human movement can be placed in context.

This is quite different from previous work in which the human body is treated in isolation, removed from the world around it.  We see the interesting space as the one where people are present in, and interacting with, the 3D world.  By estimating 3D models of people, scenes, and objects, we are able to place people in context and reason about their physical interactions with the world. 

Recreation of natural behavior with Computer vision

We also seek to go deeper and understand the goals behind people's movements. To this end we relate natural language descriptions of human behavior to 3D movement. The goal is to learn from images and language how people behave and then recreate natural behavior in virtual 3D scenes.

To advance this agenda, Perceiving Systems combines computer vision with machine learning and computer graphics to capture, model, and synthesize digital humans. We see the virtual human as more than a useful artifact. We see it as a tool for understanding ourselves. If we can simulate a virtual human in a virtual world behaving in ways that are indistinguishable from a real human, then we assert that we have captured something about what it means to be human.

Computer Vision – Our impact

We want to have an impact beyond the academic discipline of computer vision. Consequently, we develop applications in medicine and psychology in collaboration with medical colleagues. We have also spun off two companies that are using our 3D body model technology. One of these, Body Labs Inc., was acquired by Amazon in 2017. We also make code and data available open source or for license and our SMPL body model is now in wide use. Finally, we are responsible for, or contribute to, widely used datasets and evaluation benchmarks that help push the state of the art and provide a platform for industry to understand what works, how well, and why.