Leverage Kinematic and Contact constraints for understanding hand-object interaction (Talk)

My works focus on inferring and understanding the human hand’s interaction with objects from visual inputs, which include several tasks like pose estimation, grasping pose generation, and interacting pose transfer. Unlike the single-body pose estimation task, understanding the Hand-object (multi-bodies) interactions in 3D spaces is more challenging, due to its high degree of articulations, the projection ambiguity, self or mutual occlusions, and the complicated physical constraints. Designing algorithms to tackle these challenges is my goal. We find that the mutual contact can provide rich cues to reconstruct the interaction, and the anatomical constraints of the hand skeleton can help us to control hand’s kinematic motion when inferring the interaction. In a series of our works, we model the ``contact'' as pairs of a hand and object vertices, as well as a confidence that describes the contact stability. This talk will cover three of my latest works: CPF (ICCV'21), ArtiBoost (CVPR’22), and OakInk (CVPR’22). We trained a CNN to predict contact and then facilitated the conjoint HO pose estimation in CPF (ICCV'21), leveraged contact and kinematic constraints to generate diverse interacting poses in ArtiBoost (CVPR’22), and utilized the contact and kinematic constraints to transfer hand’s interactions among similar objects in OakInk (CVPR’22).