Lighting Virtual Objects using Machine Learning (Talk)

Compositing rendered, virtual objects into photographs or videos is a fundamental technique in mixed reality, visual effects, and film production. For truly convincing and seamless composites, the subjects must be rendered with lighting that matches that of the target footage. For instance, a rendered object that is too bright, too dark, or lit from a direction inconsistent with other objects in the scene will look out of place. As such, in this talk I will introduce two recent machine learning based approaches for lighting estimation used for improving the realism of augmented reality (AR) experiences. First, I’ll describe a method for estimating 360°, high dynamic range (HDR) illumination from an unconstrained, low dynamic range background image. This technique forms the basis for Google’s ARCore Environmental HDR lighting estimation API. Next, I’ll describe a method for estimating scene illumination from an arbitrary portrait photograph, which relies on a synthetic portrait dataset generated using a “Light Stage” computational illumination system. This “lighting-from-faces” technique operates behind the scenes in Google’s latest computational photography feature, “Portrait Light,” which launched for the Google Pixel Phone and in Google Photos in October 2020.