We present the first system for physically based, neural inverse rendering from multi-viewpoint videos of propagating light. Our approach relies on a time-resolved extension of neural radiance caching --- a technique that accelerates inverse rendering by storing infinite-bounce radiance arriving at any point from any direction. The resulting model accurately accounts for direct and indirect light transport effects and, when applied to captured measurements from a flash lidar system, enables state-of-the-art 3D reconstruction in the presence of strong indirect light. Further, we demonstrate view synthesis of propagating light, automatic decomposition of captured measurements into direct and indirect components, as well as novel capabilities such as multi-view time-resolved relighting of captured scenes.
Below, we showcase the results of our inverse rendering system on captured datasets. After training, our system unlocks new capabilities such as lidar relighting. Below, we show relighting results with three separate light sources visualized in the different color channels. You can toggle between the sections to see particular lights, direct/indirect separation, or the normals derived from our method.
We compare our method to TransientNeRF (T-NeRF), which physically models only the direct component of light, and Flying with Photons (FWP++), which physically models the direct but bakes in the indirect component of light into a neural representation. Our method of physically modeling the global component of light gives rise to much sharper geometry reconstruction.
Our method also allows training with Indirect TOF measurements and intensity images. After training with these modalities, we can still render lidar measurements, even though the method has not seen them! Our rendered lidar measurements contain both direct and indirect components.
We compare our method to Transient NeRF (T-NeRF) and an extended version of Flying with Photons (FWP++). As we can see, the novel views rendered by our method are far closer to the ground truth than those from TransientNeRF. On the other hand, Flying with Photons' performance is similar to ours, even though it specifically optimizes for the objective of novel view synthesis.