Improving the generalization of a snapshot spectral imaging system with system characterization, and Deep-Learning based solutions

Abstract

As part of the research field of Computational Imaging, Snapshot Spectral Imaging (SSI) systems aim to capture the 3-Dimensional (3D) spatial and spectral scene information using a co-design of hardware and software. So far, many efforts had been invested in developing optical systems that can capture such an encoded version of the scene information, and corresponding reconstruction algorithms for recovering a 3D hyperspectral “cube” (hyperspectral refers to spectral information of some approximately continuous wavelengths range) from the encoded measurements.

In line with this, our research group had previously demonstrated a cost-effective, portable, and simple SSI system that utilizes a single diffractive element (“diffuser”) and produces Dispersed and Diffused (DD) monochromatic images, which are encoded measurements of the scene.  

Recently, our group also presented a Machine-Learning based algorithms called “DD-Net”, that achieved state-of-the-art results for the reconstruction of HS cubes from the DD images. However, experiments showed that the recovery algorithms tend to be sensitive and unsatisfactory for data

that is significantly different than the data they were optimized on, in what being called the “Generalization problem” of data-driven solutions. In the case of supervised learning, this problem is present both in the data’s input samples domain and in the output labels domain, independently.

This thesis work focuses on decreasing the generalization error the system obtains, which is the error obtained from DD-Net evaluation on test datasets, without additional physical data acquisition process. Based on how the generalization problem manifests differently in the input and output algorithm’s domains, a slightly different approach was taken. 

The results of the two parallel research directions led to a significant contribution to the system's HS reconstruction quality, as well as important and beneficial insights regarding the system capabilities. As part of this work, we developed the “SHS-GAN”, which is a novel general-purpose, end-to-end framework for enlarging HS datasets based on RGB samples that can easily be adapted and contribute to a variety of HS-related tasks.

* A scientific paper describing the “SHS-GAN” implement and utilization will be soon published in: “IEEE Transactions In Computational Imaging” journal.

.

.