Visual Correspondence, the Lambert-Ambient Shape Space and the Systematic Design of Feature Descriptors

In this expository article, we justify the use of sparse local descriptors for correspondence, and illustrate a systematic method for their design. Correspondence is the process that allows using image data to infer properties of the “scene,” where the scene can refer to a specific object or landscape, or can be abstracted into a category label to take into account intra-class variability. As the generality increases, the complexity of nuisance factors does too, so global pixel-level correspondence is not viable, and one has to settle instead for sparse descriptors. These should be co-designed with the classifier, and for a given classifier family, one can design the descriptors to be invariant to uninformative nuisances that are explicitly modeled, insensitive to other nuisances that are not explicitly modeled, and maximally discriminative, relative to the chosen family of classifiers. Existing descriptors are interpreted in this framework, where their limitations are illustrated, together with pointers on how to improve them.