Image Processing for Quanta Image Sensors

Omar A Elgendy, Purdue University

Abstract

Since the birth of charge coupled devices (CCD) and the complementary metaloxide-semiconductor (CMOS) active pixel sensors, pixel pitch of digital image sensors has been continuously shrinking to meet the resolution and size requirements of the cameras. However, shrinking pixels reduces the maximum number of photons a sensor can hold, a phenomenon broadly known as the full-well capacity limit. The drop in full-well capacity causes drop in signal-to-noise ratio and dynamic range.The Quanta Image Sensor (QIS) is a class of solid-state image sensors proposed by Eric Fossum in 2005 as a potential solution for the limited full-well capacity problem. QIS is envisioned to be the next generation image sensor after CCD and CMOS since it enables sub-diffraction-limit pixels without the inherited problems of pixel shrinking. Equipped with a massive number of detectors that have single-photon sensitivity, the sensor counts the incoming photons and triggers a binary response “1” if the photon count exceeds a threshold, or “0” otherwise. To acquire an image, the sensor oversamples the space and time to generate a sequence of binary bit maps. Because of this binary sensing mechanism, the full-well capacity, signal-to-noise ratio and the dynamic range can all be improved using an appropriate image reconstruction algorithm. The contribution of this thesis is to address three image processing problems in QIS: 1) Image reconstruction, 2) Threshold design and 3) Color filter array design.Part 1 of the thesis focuses on reconstructing the latent grayscale image from the QIS binary measurements. Image reconstruction is a necessary step for QIS because the raw binary measurements are not images. Previous methods in the literature use iterative algorithms which are computationally expensive. By modeling the QIS binary measurements as quantized Poisson random variables, a new non-iterative image reconstruction method based on the Transform-Denoise framework is proposed. Experimental results show that the new method produces better quality images while requiring less computing time.Part 2 of the thesis considers the threshold design problem of a QIS. A spatiallyvarying threshold can significantly improve the reconstruction quality and the dynamic range. However, no known method of how to achieve this can be found in the literature. The theoretical analysis of this part shows that the optimal threshold should match with the underlying pixel intensity. In addition, the analysis proves the existence of a set of thresholds around the optimal threshold that give asymptotically unbiased reconstructions. The asymptotic unbiasedness has a phase transition behavior. A new threshold update scheme based on this idea is proposed. Experimentally, the new method can provide good estimates of the thresholds with less computing budget compared to existing methods.Part 3 of the thesis extends QIS capabilities to color imaging by studying how a color filter array should be designed. Because of the small pixel pitch of QIS, crosstalk between neighboring pixels is inevitable and should be considered when designing the color filter arrays. However, optimizing the light efficiency while suppressing aliasing and crosstalk in a color filter array are conflicting tasks. A new optimization framework is proposed to solve the problem. The new framework unifies several mainstream design criteria while offering generality and flexibility. Extensive experimental comparisons demonstrate the effectiveness of the framework.

Degree

Ph.D.

Advisors

Chan, Purdue University.

Subject Area

Operations research

COinS