Collaborative Control for Geometry-Conditioned PBR Image Generation (Talk)
Current diffusion models only generate RGB images. If we want to make progress towards graphics-ready 3D content generation, we need a PBR foundation model, but there is not enough PBR data available to train such a model from scratch. We introduce Collaborative Control, which tightly links a new PBR diffusion model to a pre-trained RGB model. We show that this dual architecture does not risk catastrophic forgetting, outputting high-quality PBR images and generalizing well beyond the PBR training dataset. Furthermore, the frozen base model remains compatible with techniques such as IP-Adapter.
Biography: Simon Donné is a Staff Research Scientist at Unity, focusing on Text-to-Texture and Text-to-3D. Previously, he has worked on a variety of 3D reconstruction and photogrammetry methods, as well as spatial reasoning, at the MPI in Tübingen as well as on the Amazon Scout project. He holds a Master's degree and PhD in Computer Science from the University of Ghent.
3D Generative models
Details
- 26 September 2024 • 14:00 - 15:00
- Virtual, Live stream at Max-Planck-Ring 4, N3, Aquarium
- Perceiving Systems
