8. Digital Image Correlation with a Neuromorphic Event-Based Imager

Digital image correlation is a well-established method for estimating the full-field displacement of two-dimensional surfaces by comparing pairs of images. The basic idea is to attach a texture or speckle pattern to the surface, and track the features of this pattern through the grayscale values of the image pixels. All this assumes a conventional camera that records image intensity at every pixel in every frame, which is inefficient for sharp transient events. In watching a balloon burst, for example, one needs a high frame rate to capture the details of the burst, but storing such a frame rate for the longer, slower expansion of the balloon before it bursts costs a lot of memory. It would be convenient, in applications such as monitoring structures, to store dense information only during dynamic events, not between them.

Silicon retinas are an alternative sensor type that record events rather than pixels. The raw data for such a device are a sequence of 4-tuples: the time at which each event occurred, the horizontal and vertical pixel coordinates containing the event, and whether the event involved increasing or decreasing intensity. Events can be reassembled into buckets corresponding to each pixel and a time interval corresponding to any desired camera shutter to produce frames analogous to those of a conventional camera. This project assesses whether it is feasible to combine digital image correlation or a similarly developed computer vision technique with the efficient storage of a silicon retina, via such converted frames.

The test article for this experiment was a latex band with a painted speckled pattern, mounted into the stationary and moving ends of a frame and subject to cyclical stretching. The image of this band was recorded simultaneously on conventional and silicon retina detectors through a beam splitter. Analysis of the resulting data showed that the silicon retina frames allow feature tracking with close to the quality of a conventional camera, but the computed displacements are consistently smaller. The surface has to accelerate before there are enough changes to register on the silicon retina, and this initial motion at either end of the oscillation is not recorded in the converted frames. This work demonstrates that silicon retina imagers have potential for persistent surveillance applications where there is a need to record sparsely occurring transient deformations over long time periods.