Spectrally-Programmable Cameras for Imaging and Inference

Understanding light and its interactions with materials in a scene forms the bedrock of modern computer vision. In this context, the spectral properties of light play a very important role, especially when we seek to study the material composition of a scene. As a consequence, spectral measurements ?find applications across a wide range of scienti?fic fi?elds such as medical diagnostics, microscopy, geospace intelligence, remote sensing, and computer vision.
Several vision tasks benefit immensely from capturing spectra at all spatial locations in a scene. This requires an optical system called hyperspectral camera, which captures images across ?finely spaced wavelengths. Despite its wide applicability, measuring a high resolution hyperspectral image is inherently a hard task. Sampling a scene over million of spatial locations, and across hundreds of spectral
bands results in diminishing photon count at each spatio-spectral voxel, leading to extremely low signal
to noise ratios (SNR). This is often compensated with long exposure times, which precludes imaging of dynamic scenes. Further, the giga-pixels of data associated with each scan places immense burden on capture and processing hardware. The work in this thesis seeks to simplify the process of capturing spectral information of a scene
with design of novel imaging systems. This thesis relies on two key observations. First, despite the high dimensional nature of hyperspectral images, the number of distinct materials in any given scene is very small; this leads to a concise low-dimensional representation of the hyperspectral image. Second, owing to this low diversity, capturing a small set of spectrally-fi?ltered images of the scene su?ccess for most sensing and inference tasks. Exploiting these two observations, this thesis builds novel and effi?cient
optical systems for imaging and inference. Central to the contributions of this thesis is an optical system that can provide programmable spectral filtering, by attenuating intensity of light at each wavelength arbitrarily and capturing the resultant image. The fi?rst contribution of this thesis shows that capturing sharp images with arbitrarily high resolution spectral fi?ltering is not possible – a property that arises due to the shape of the pupil function of the camera. This fundamental limit is provided in the form of the space-spectrum uncertainty principle, which sets a lower bound on product of spectral and spatial spreads. We then show that the resolutions can be enhanced computationally, if the pupil function is carefully engineered to introduce invertible
spatial and spectral blurs. Armed with the insights of a spectrally-programmable setup, we show that such cameras can be used to effi?ciently sense hyperspectral images. Since the true complexity of sensing hyperspectral images lies
not in high resolution space or spectrum, but only the diversity of materials, the hyperspectral image can be represented using a low-rank matrix model. This thesis provides a novel adaptive sensing strategy to optically compute this low-rank model. We note that the dominant spatial and spectral singular vectors can be sensed by building two optical operators, namely a spatially-coded spectrometer, and a spectrally programmable camera. By alternating between the two operators, and using output of one operator as input to the second, we can measure a low-rank approximation with as few as ten measurements –
contrasted with several hundreds of measurements for fully scanning the hyperspectral image. Finally, the thesis builds on spectral-programmability and optical computing to enable per-pixel material classi?fication. This is achieved by capturing images of the scene with learned, discriminative
spectral fi?lters and then using the images to classify materials. This enables a per-pixel classifi?cation
strategy with a small set of high SNR measurements – thereby leading to real-time vision capabilities.
At its culmination, this thesis lays groundwork for making hyperspectral cameras more practical by introducing computing into the sensing pipeline, and moving most of computational burden into the optical domain. This successfully decouples the number of measurements and SNR, thereby allowing future optical systems to achieve very high resolution along spatial and spectral axes.