Motion Understanding

The velocity field that represents the motion of object points across an image is called the optical flow field. Optical flow results from relative motion between a camera and objects in the scene. One class of techniques for the estimation of optical flow utilizes a relationship between the motion of surfaces and the derivatives of image brightness (Limb and Murphy, 1975; Cafforio and Rocca, 1976; Fennema and Thompson, 1979; Netravali and Robbins, 1979; Schalkoff, 1979; Lucas and Kanade, 1981; Schunck and Horn, 1981; Thompson and Barnard, 1981; and Schalkoff and McVey, 1982). The major difficulty with gradient-based methods is their sensitivity to conditions commonly encountered in real imagery. Highly textured regions, motion boundaries, and depth discontinuities can all be troublesome for gradient-based methods. Fortunately, the areas characterized by these difficult conditions are usually small and localized.

This chapter describes work toward understanding the fundamentals of image flow and presents algorithms for estimating the image flow field. Image flow is the velocity field in the image plane that arises due to the projection of moving patterns in the scene onto the image plane. The motion of patterns in the image plane may be due to the motion of the observer, the motion of objects in the scene, or both. The motion may also be apparent motion where a change in the image between frames gives the illusion of motion. The image flow field can be used to solve important vision problems provided that it can be accurately and reliably computed. Potential applications are discussed in Section 2.1.2.

Vision research in fields as diverse as computer science, psychology, and neurophysiology, has led to the emergence of stereopsis and kineopsis as the two principal views which explain some of the mechanisms of space perception.

There have been important accomplishments in the computational analysis of the recovery of information about three-dimensional relationships in the environment from dynamic two-dimensional images. As an achievement in artificial intelligence and as a significant application to robotics, these results stand by themselves. There has been increasing interest, however, in applying these analyses to the study of human visual perception. The question of applicability of a particular computational analysis to human vision is an empirical one, and one that can only be answered through rigorous empirical research. In this chapter, I will review the current status of empirical research that addresses four issues of relevance to motion understanding. The first issue is whether there are two paths to the recovery of three-dimensional structure from motion: (a) one that proceeds from the primal sketch to a viewer-centered 2 1/2 dimensional sketch to an object-centered 3-dimensional model as proposed by Marr (1982), and (b) a direct path from the primal sketch to an object-centered representation, with viewer-centered information added from separate sources. The second issue is whether the solution to the correspondence problem must precede the extraction of structure from motion. The third issue is the now familiar controversy in the perception literature about whether a rigidity constraint plays a central role in the recovery of structure from motion.

The idea of using three or more consecutive frames for motion analysis has been mentioned by many researchers (Ullman, 1979; Lawton, 1980; Yasumoto and Medioni, 1985). However, most approaches formulate the motion problem in such a way that they do not use the time flow information content of the image sequence optimally. The majority of techniques work with only two frames at a time, and even then, they are treated no differently than a stereo pair. Consequently, they do not make use of the fact that these frames are from a time sequence, that is, the third frame was taken T seconds after the second frame and 2T seconds after the first frame. (T being the time between any two successive frames.) As a result, most of the current approaches to motion analysis are formulated in such a way that they must treat a matched set of n features in three frames, as a set of 2n features in two frames. By doing so, they treat a three-frame sequence as two sets of two-frame sequences, and hence, under-utilize the information available to them.

The human visual system is capable of extracting three-dimensional shape information from two-dimensional transformations in the image. Experiments employing shadow projections of moving objects and computer generated displays have established that the three-dimensional shape of objects in motion can be perceived when their changing projection is observed, even when each static view is completely devoid of three-dimensional information.

An important trend in the visual sciences is the emerging convergence between psychophysical and computational approaches to visual information processing (Beck, Hope, and Rosenfeld, 1983). Each field is concerned with similar issues and problems; however, each applies a sufficiently different approach to provide complementary lines of investigation. One point of convergence is found in current “bootstrapping” procedures that analyze visual information into minimal stimulus conditions and then seek to model processes by which these conditions can be transformed into relevant environmental properties.

The goal in the motion estimation problem is to determine the transformation between two three-dimensional positions of a rigid body that is undergoing some arbitrary motion. Throughout this discussion it will be assumed that it is possible to acquire three-dimensional positional information of points located on the rigid body at two separate time instances. This may be accomplished through the use of a stereo camera setup (Barnard and Fischler, 1982). Alternatively, three-dimensional positional data can be obtained explicitly from a laser range finder. In general, points selected from the rigid body will tend to correspond to special geometrical features, such as corners or identifiable surface markings, and must often be extracted by special low level processing tasks (Gu and Huang, 1984). Such feature points will be invariant to the particular position sensing techniques used, and thus may be reliably located independent of object position and orientation.