Image-Based Relighting of 3D Objects from Flash Photographs

Abstract

Photography is a remarkable technology that allows us to capture and reproduce the appearance of the real world. Although photographs are two-dimensional and static, their capabilities have been extended into higher dimensions through the development of techniques like photogrammetry, image-based rendering, and image-based relighting. Such prior work, however, has struggled to support dynamic illumination of the subject while still representing specular reflections accurately, especially for subjects that exhibit heterogeneity in their appearance. The most successful results in this direction usually require thousands of images. This dissertation introduces a new paradigm for image-based relighting of 3D objects that requires no more than a few hundred flash photographs. The flash lighting configuration is conveniently found on most commodity cameras. After being processed using traditional photogrammetry, the flash images serve as a collection of virtual light sources in a relighting system, through the power of Cook-Torrance microfacet theory. This approach produces new images of 3D objects that effectively retain the photographic accuracy of the subject's color appearance from the original flash photos. The same flash images can also be used to estimate reflectance parameters that improve the accuracy of the relighting technique. This rendering method can even be used to emulate lighting conditions -- both outdoor and indoor -- that are very different than flash. This work proceeds to show how the intensities of point light sources derived from the collection of flash images can be chosen to effectively emulate the intended environment. Although similar results have been previously achieved using thousands of images, the method shown here can be effective with a few hundred images or less. However, this reduction is not without limitation; a shinier object requires more photographs to avoid discontinuity between the intended environment and its reflection. To address this one limitation, this work ultimately develops a fidelity metric for assessing whether reflections of individual light sources are likely to be discernible when the object is relit. This metric is a heuristic solution which estimates the magnitude of this issue by quantifying the degree of overlap in specular reflections between similar images in the dataset. It is believed to be the first work which assesses the fidelity of how highlights are depicted when an object is relit using a limited number of views. This solution is shown to be generally effective and serves as a foundation which can be built upon by future metrics.