Closing the loop with blur

Abstract

A great variety of systems use image sensors to provide measurements for closed loop operation. A drawback of using image sensors in real-time feedback is that they provide measurements at slower sampling rates as compared to the actuators, typically around 30 Hz for CCD cameras, hence acting as the bottleneck for closed loop control bandwidths. While high speed cameras exist, higher frame rates imply an upper bound on exposures which lowers the signal-to-noise-ratio (SNR), reducing measurements accuracy. The integrative nature of image sensors though offers the opportunity to prolong the exposure window and collect motion blurred measurements. This research describes how to exploit the dynamic information of observed system outputs, encoded in motion blur, to control fast systems at the fast rate through slow rate image sensors.

This work proves the utility of exploiting motion blurred measurements in a variety of classic control problems, departing from the traditional idea of considering motion blur an undesired artifact to be avoided, mitigated, or actively removed.

Depending on the application, this model varies, e.g., it is a sine wave with undetermined amplitude and phase in a system identification problem, or the output of a linear time invariant system with unknown initial conditions in an estimation one. Even when no prior assumption on the signal source is made nor information on its parametric model is available, it is still possible to reasonably approximate the motion. In the signal reconstruction problem the signal can be modeled, e.g., as a Taylor series expansion within the exposure window, provided that attention is paid to representing the signal to a desired threshold of approximation.

Therefore in order to ``close the loop with blur", this work describes how to pose and solve the problems of, namely: system identification, state estimation, closed loop control and signal reconstruction from motion blur. By modeling the image sensor as a time to pixel domain integrative transformation, these problems are recast in nonlinear least square ones where a spatial error metric, i.e., the pixel domain error between measurements and image predictions, is minimized to obtain the desired solutions. Such operation corresponds to the inversion of the image sensor integral transformation, which is an ill posed problem that requires some form of regularization to avoid alias solutions. This regularization is provided in the form of different parametric models of the unknown time domain signal with different constraints on the parameters depending on the studied problem.

In order to achieve this objective it is necessary to (a) design a controller providing fast rate input to the system based on the slow image measurements. Ideally such a controller would require a fast rate estimate of the system’s state variables in order to provide the necessary control action, therefore an (b) image blur based estimator is to be developed. State estimators typically need a model of the system in order to provide their estimates, so (c) a system identification problem has to be addressed, where a reliable model describing the frequency content of the system, up to frequencies corresponding to the fast rate, has to be determined through slow rate image sensor measurements. Alternatively when such a procedure is not possible for lack, e.g., of knowledge of the input to the system, then (d) a method to reconstruct the output signal frequency content up to frequencies above those set by the limitations of the sampling theorem is to be devised.