Several simplifying assumptions are made to the general problem of 3D models from 2D imagery to formulate the Structure from Motion task. The figure above shows a standard SfM setup where a camera is viewing a scene. One key assumption is that objects in the scene are moving rigidly or, equivalently, only the camera is allowed to move in the environment.
An additional simplification is that there exists a module which pre-processes the camera's images to consistently extract, locate and label 2D features in the scene. Such 2D features could include salient points in the image, corners of objects, lines along their edges or curves around their contours. In each frame, the features are detected and associated to their corresponding instantiations in the other frames. These (usually noisy and error-prone) 2D measurements are the inputs to the SfM problem. The availability of corresponded features restricts the SfM problem to the so-called Corresponded Structure from Motion geometric task which will be the focus herein. It should be noted, however, that matching and detecting feature points is a fundamental and decidedly difficult computer vision problem which can not be dismissed so easily in practical implementations.
The locations of the 2D features in the images depend on 1) their coordinates in 3D space, 2) the relative 3D motion between the camera and the scene and 3) the camera's internal geometry. We assume that we have no prior knowledge of these three causes and wish to recover their parameters only from 2D point coordinate measurements over several frames or views. Of course, there exist many alternative problem statements in the SfM community with various twists ranging from the types of input features (i.e. curves or line features are alternatives), to the algorithm's required output, and so on. We shall focus primarily on the task stated above. Other SfM overviews can be seen in [31] [14] [41] [18].
The paper motivates the SfM approaches by describing some current practical applications. This is followed by a brief discussion of the background of the field. Then, several techniques are outlined that show various important approaches and paradigms to the SfM problem. Critical issues, advantages and disadvantages are pointed out. Subsequently, we present our SfM approach for recursive estimation of motion, structure and camera geometry in a nonlinear dynamic system framework. Results are given for synthetic and real imagery. These are used to assess the accuracy and stability of the technique. We then discuss some practical and real-time applications we have encountered and the reliability and flexibility of the approach in those settings. Finally, we conclude with results from an independent evaluation study conducted by industry where the proposed SfM algorithm compared favorably to alternative approaches. Our SfM software is available for public ftp at:
ftp whitechapel.media.mit.edu /pub/sfm