Faces are highly dynamic and deformable. Small motions convey our emotional state and intentions. Capturing natural 3D face shape in motion is the first step towards modeling and understanding facial expressions. With such models we can animate realistic avatars and analyze videos of facial motion. Our 4D face scanner employs 6 pairs of stereo cameras for computing shape and geometry with the assistance of custom speckle projectors and 6 color cameras and white-light panels for capturing texture, all of them arranged specifically to acquire the data needed to accurately reconstruct a 3D model of the human face in motion.

We process the raw 3D scan data using our co-registration algorithm to bring all the scans into correspondence with a template mesh. With this we can model facial shape and motion over time to study the dynamics of the human face in ways that were not previously possible. For example, we can obtain 3D meshes of a person having a conversation and study how different emotions are conveyed by the motion of the face. These capabilities, together with the full body 3D models that we acquire with our 4D Dynamic Scanner, enable us to observe how the expression and movement of our faces are reflected in the motion and posture of the rest of our body and vice versa, providing new insights into the relationship between face and body movement.

The system can also be reconfigured to capture 3D hand shape over time. Another 3 MCUs (Multi Camera Units) are currently being installed for this purpose.