2024
Modelling Dynamic 3D Human-Object Interactions: From Capture to Synthesis
Self- and Interpersonal Contact in 3D Human Mesh Reconstruction
Natural Language Control for 3D Human Motion Synthesis
2022
Reconstructing Expressive 3D Humans from RGB Images
2021
Skinned multi-infant linear body model
Hesse, N., Pujades, S., Romero, J., Black, M.
(US Patent 11,127,163, 2021), September 2021 (patent)
2020
Machine learning systems and methods of estimating body shape from images
Black, M., Rachlin, E., Heron, N., Loper, M., Weiss, A., Hu, K., Hinkle, T., Kristiansen, M.
(US Patent 10,679,046), June 2020 (patent)
Machine learning systems and methods for augmenting images
Black, M., Rachlin, E., Lee, E., Heron, N., Loper, M., Weiss, A., Smith, D.
(US Patent 10,529,137 B1), January 2020 (patent)
2019
Towards Geometric Understanding of Motion
The motion of the world is inherently dependent on the spatial structure of the world and its geometry. Therefore, classical optical flow methods try to model this geometry to solve for the motion. However, recent deep learning methods take a completely different approach. They try to predict optical flow by learning from labelled data. Although deep networks have shown state-of-the-art performance on classification problems in computer vision, they have not been as effective in solving optical flow. The key reason is that deep learning methods do not explicitly model the structure of the world in a neural network, and instead expect the network to learn about the structure from data. We hypothesize that it is difficult for a network to learn about motion without any constraint on the structure of the world. Therefore, we explore several approaches to explicitly model the geometry of the world and its spatial structure in deep neural networks.
The spatial structure in images can be captured by representing it at multiple scales. To represent multiple scales of images in deep neural nets, we introduce a Spatial Pyramid Network (SpyNet). Such a network can leverage global information for estimating large motions and local information for estimating small motions. We show that SpyNet significantly improves over previous optical flow networks while also being the smallest and fastest neural network for motion estimation. SPyNet achieves a 97% reduction in model parameters over previous methods and is more accurate.
The spatial structure of the world extends to people and their motion. Humans have a very well-defined structure, and this information is useful in estimating optical flow for humans. To leverage this information, we create a synthetic dataset for human optical flow using a statistical human body model and motion capture sequences. We use this dataset to train deep networks and see significant improvement in the ability of the networks to estimate human optical flow.
The structure and geometry of the world affects the motion. Therefore, learning about the structure of the scene together with the motion can benefit both problems. To facilitate this, we introduce Competitive Collaboration, where several neural networks are constrained by geometry and can jointly learn about structure and motion in the scene without any labels. To this end, we show that jointly learning single view depth prediction, camera motion, optical flow and motion segmentation using Competitive Collaboration achieves state-of-the-art results among unsupervised approaches.
Our findings provide support for our hypothesis that explicit constraints on structure and geometry of the world lead to better methods for motion estimation.
Method for providing a three dimensional body model
Loper, M., Mahmood, N., Black, M.
September 2019, U.S.~Patent 10,417,818 (patent)
2018
Method and Apparatus for Estimating Body Shape
Black, M. J., Balan, A., Weiss, A., Sigal, L., Loper, M., St Clair, T.
June 2018, U.S.~Patent 10,002,460 (patent)
Model-based Optical Flow: Layers, Learning, and Geometry
Co-Registration – Simultaneous Alignment and Modeling of Articulated 3D Shapes
Black, M., Hirshberg, D., Loper, M., Rachlin, E., Weiss, A.
February 2018, U.S.~Patent 9,898,848 (patent)
Combining Data-Driven 2D and 3D Human Appearance Models
2017
Parameterized Model of 2D Articulated Human Shape
Black, M. J., Freifeld, O., Weiss, A., Loper, M., Guan, P.
September 2017, U.S.~Patent 9,761,060 (patent)
Crowdshaping Realistic 3D Avatars with Words
Streuber, S., Ramirez, M. Q., Black, M., Zuffi, S., O’Toole, A., Hill, M. Q., Hahn, C. A.
August 2017, Application PCT/EP2017/051954 (patent)
System and method for simulating realistic clothing
Human Shape Estimation using Statistical Body Models
Learning Inference Models for Computer Vision
Capturing Hand-Object Interaction and Reconstruction of Manipulated Objects
2016
Skinned multi-person linear model
Black, M., Loper, M., Mahmood, N., Pons-Moll, G., Romero, J.
December 2016, Application PCT/EP2016/064610 (patent)
Non-parametric Models for Structured Data and Applications to Human Bodies and Natural Scenes
2015
Shape Models of the Human Body for Distributed Inference
From Scans to Models: Registration of 3D Human Shapes Exploiting Texture Information
Long Range Motion Estimation and Applications
2014
Advanced Structured Prediction
Nowozin, S., Gehler, P. V., Jancsary, J., Lampert, C. H.
Advanced Structured Prediction, pages: 432, Neural Information Processing Series, MIT Press, November 2014 (book)
Modeling the Human Body in 3D: Data Registration and Human Shape Representation
Learning People Detectors for Tracking in Crowded Scenes.
Tang, S., Andriluka, M., Milan, A., Schindler, K., Roth, S., Schiele, B.
2014, Scene Understanding Workshop (SUNw, CVPR workshop) (unpublished)
Human Pose Estimation from Video and Inertial Sensors
2013
Human Pose Calculation from Optical Flow Data
Statistics on Manifolds with Applications to Modeling Shape Deformations
Probabilistic Models for 3D Urban Scene Understanding from Movable Platforms
System and method for generating bilinear spatiotemporal basis models
Matthews, I. A. I. S. T. S. K. S. Y.
US Patent Application 13/425,369, March 2013 (patent)
A Study of X-Ray Image Perception for Pneumoconiosis Detection
2012
Virtual Human Bodies with Clothing and Hair: From Images to Animation
From Pixels to Layers: Joint Motion Estimation and Segmentation
An Analysis of Successful Approaches to Human Pose Estimation
Consumer Depth Cameras for Computer Vision - Research Topics and Applications
Fossati, A., Gall, J., Grabner, H., Ren, X., Konolige, K.
Advances in Computer Vision and Pattern Recognition, Springer, 2012 (book)
2008
GNU Octave Manual Version 3
Eaton, J. W., Bateman, D., Hauberg, S.
Network Theory Ltd., October 2008 (book)
2005
Visual motion analysis method for detecting arbitrary numbers of moving objects in image sequences
Jepson, A. D., Fleet, D. J., Black, M. J.
US Pat. 6,954,544, October 2005 (patent)
A Flow-Based Approach to Vehicle Detection and Background Mosaicking in Airborne Video
2003
Method and apparatus for generating a condensed version of a video sequence including desired affordances
Black, M. J., Ju, S., Minneman, S., Kimber, D.
US Pat. 6,560,281, May 2003 (patent)
Apparatus and method for identifying and tracking objects with view-based representations
Black, M. J., Jepson, A.
US Pat. 6,526,156, February 2003 (patent)
1995
Apparatus and method for tracking facial motion through a sequence of images
Black, M. J., Yacoob, Y.
US Pat. 5,802,220, December 1995 (patent)
Apparatus and method for recognizing facial expressions and facial gestures in a sequence of images
Black, M. J., Yacoob, Y.
US Pat. 5,774,591, December 1995 (patent)
Image segmentation using robust mixture models
Black, M. J., Jepson, A. D.
US Pat. 5,802,203, June 1995 (patent)
1992
Robust Incremental Optical Flow