Shape, Reflectance and Illumination Estimation From Mobile Devices

Inferring shape, reflectance and illumination from an image enables us to better understand a scene and has opened up a wide range of socially-compelling applications ranging from virtual reality, entertainment and home surveillance. However, this is a particularly challenging problem since all the factors are combined into a single observation. At the same time, the ubiquity of mobile devices has raised immense opportunities for wide spread adoption of such techniques. Toward this end, this dissertation addresses the problem for the shape, reflectance and illumination estimation using sensors and illuminants commonly found on commodity devices, such as smart phones and tablets. We focus on two subsets of the problem. We first address the
problem of estimating the shape and reflectance of objects that exhibit spatially-varying reflectance under known single illuminant. Our contributions here are two-fold. First, we provide a non-iterative technique for per-pixel shape and reflectance, that is able to outperform the state-of-the-art methods on a wide range of real scenes. Second, we translate our technique to mobile devices and demonstrate capabilities in estimating as well as editing reflectance in spite of the ash unit and camera sensor being collocated.
Next, for a Lambertian scene being illuminated with
multiple light sources, we propose a method to separate and manipulate the scene illuminants based on their spectral differences. As before, we make two contributions. First, we derive physics based constraints for the ash/no-ash image pairs and provide identifiable analysis with respect to the number of light sources in the scene. We show that this separation can be used to support applications like white balancing, lighting editing, and RGB photometric
stereo, where we demonstrate results that outperform
state-of-the-art techniques on a wide range of images. Second, to address the limitations of the
ash light for the mobile devices, such as the presence of strong ambient light as well as the scenes
with large depth variations, we further demonstrate the ability
to separate the image by simulating the
ash image with a deep neutral network. We show that we are able to produce high quality outputs that match the performance of previous methods that required a ash/no-ash pair, while being more practical in requiring only a single image. We believe that, all together, the techniques developed in this dissertation makes a significant advance in our ability to not just estimate a scene's shape, reflectance, and illumination but also enable subsequent inference and post-processing capabilities.