Dynamic, Pixelwise Image Sensing and Systems

This opportunity includes a suite of inventions including a fully-functional coded two-bucket (C2B) prototype, a novel programmable CMOS sensor, and a method for adaptive adjustment of exposure of every pixel, based on changes in the brightness of the scene for that pixel to extend dynamic range at native sensor resolution.

Coded two-bucket (C2B) imaging, is a new operating principle for computational sensors with applications in active 3D shape estimation and coded-exposure imaging. A C2B sensor modulates the light arriving at each pixel by controlling which of the pixel’s two “buckets” should integrate it. C2B sensors output two images per video frame—one per bucket—and allow rapid, fully-programmable, per-pixel control of the active bucket. Using these properties as a starting point, we:

  1. develop an image formation model for these sensors,
  2. couple them with programmable light sources to acquire illumination mosaics, i.e., images of a scene under many different illumination conditions whose pixels have been multiplexed and acquired in one shot, and
  3. show how to process illumination mosaics to acquire live disparity or normal maps of dynamic scenes at the sensor’s native resolution

OPPORTUNITY

Modern computational photography applications such as 3D sensing, gesture analysis, and robotic navigation drive the growing need for programmability, or coding, of the camera exposure at the individual-pixel level. Unlike conventional cameras, which record all light incidents onto a pixel, the emerging class of coded-exposure-pixel (CEP) cameras can be programmed to selectively detect only some of that light or, better, sort all of the light, depending on the pixel code. In conjunction with a concurrently coded illumination, this enables a wide range of new coded multi-exposure single-readout-frame imaging capabilities at video rates.

The light efficiency and electronic per-pixel coding capabilities of C2B cameras and methods open up a range of applications that go well beyond what is possible today. This potentially includes compressive acquisition of high-speed video with optimal light efficiency; simultaneous acquisition of both epipolar-only and non-epipolar video streams; fully-electronic acquisition of high-dynamic-range AC-flicker videos; conferring EpiScan3D-like functionality to non-rectified imaging systems; and performing many other coded-exposure imaging tasks with a compact camera platform.

STATUS

  • Patent applications are pending for imaging methods and apparatus

ID:

P2006

Keywords:

3D imaging , Artificial Intelligence (AI) , Cameras , Circuit design , Computer Graphics , Computer Hardware , Software , Computer Vision , Image and video processing , Semiconductors