DiffusionLight: Light Probes for Free by Painting a Chrome Ball

Pakkapon Phongthawee*1
Worameth Chinchuthakun*1,2
Nontaphat Sinsunthithet1
Amit Raj3
Varun Jampani4
Pramook Khungurn5
1 VISTEC
2 Tokyo Tech
3 Google Research
4 Stability AI
5 Pixiv
CVPR 2024 (Oral)
*Equal contributions

Abstract

We present a simple yet effective technique to estimate lighting in a single input image. Current techniques rely heavily on HDR panorama datasets to train neural networks to regress an input with limited field-of-view to a full environment map. However, these approaches often struggle with real-world, uncontrolled settings due to the limited diversity and size of their datasets. To address this problem, we leverage diffusion models trained on billions of standard images to render a chrome ball into the input image. Despite its simplicity, this task remains challenging: the diffusion models often insert incorrect or inconsistent objects and cannot readily generate images in HDR format. Our research uncovers a surprising relationship between the appearance of chrome balls and the initial diffusion noise map, which we utilize to consistently generate high-quality chrome balls. We further fine-tune an LDR diffusion model (Stable Diffusion XL) with LoRA, enabling it to perform exposure bracketing for HDR light estimation. Our method produces convincing light estimates across diverse settings and demonstrates superior generalization to in-the-wild scenarios.

Approach

Given an input image, we seek to estimate the scene's lighting as an HDR environment map. The idea is to inpaint a chrome ball into the image using a diffusion model and unwarp it to an environment map. We tackle two key challenges of (1) how to consistently generate chrome balls and (2) how to use an LDR diffusion model to generate HDR chrome balls. (a) We use Stable Diffusion XL with depth-conditioned ControlNet to inpaint a chrome ball. The input image, its depthmap with a white circle, and the mask are fed to the diffusion model with the prompt "a perfect mirrored reflective chrome ball sphere". (b) Our iterative inpainting algorithm helps improve generation quality and consistency by constraining the initial noise through sample averaging. (c) We train LoRA for exposure bracketing, which produces multiple LDR chrome balls with varying exposures for HDR merging.

In-the-Wild Results

Our technique can estimate lighting for a variety of input images, such as indoor and outdoor scenes, close-up shots, paintings, and photos of human faces. Here, we show our predicted chrome balls in a normally exposed version (EV0) and an underexpoed version (EV-5). These input images are from Unsplash.com.

Normally exposed

Under​exposed

Normally exposed

Under​exposed

Normally exposed

Under​exposed

Normally exposed

Under​exposed

Normally exposed

Under​exposed

Normally exposed

Under​exposed

  • Normally exposed

    Under​exposed

  • Normally exposed

    Under​exposed

  • Normally exposed

    Under​exposed

  • Normally exposed

    Under​exposed

  • Normally exposed

    Under​exposed

  • Normally exposed

    Under​exposed

Predicting Multiple Chrome Balls

Our method can generate multiple plausible chrome balls by varying the initial noise map of diffusion sampling. Note that the average of these variations captures the overall lighting reasonably well. This average is utilized by our "iterative inpainting" algorithm to enhance the quality and consistency of light estimation. Hover over these videos to pause.

Variations

Mean of the variations

Variations

Mean of the variations

Application: Virtual Object Insertion

Using our environmental light estimates, we can seamlessly insert 3D objects into an existing photograph. Hover over the video to stop time!

Input image

Without relighting

Relighting with DiffusionLight

Prelim: Spatially Varying

In our preliminary study, we demonstrate that our method can produce a spatially varying ball. We observe in this median ball that the window's reflection becomes larger and more curved as it moves to the right.

Comparison with Prior Work

BibTex

@inproceedings{Phongthawee2023DiffusionLight,
  author = {Phongthawee, Pakkapon and Chinchuthakun, Worameth and Sinsunthithet, Nontaphat and Raj, Amit and Jampani, Varun and Khungurn, Pramook and Suwajanakorn, Supasorn},
  title = {DiffusionLight: Light Probes for Free by Painting a Chrome Ball},
  booktitle = {ArXiv},
  year = {2023},
}

From our lab