Synthesizing Real World Stereo Challenges

Synthetic datasets for correspondence algorithm benchmarking recently gained more and more interest. The primary aim in its creation commonly has been to achieve highest possible realism for human observers which is regularly assumed to be the most important design target. But datasets must look realistic to the algorithm, not to the human observer. Therefore, we challenge the realism hypothesis in favor of posing specific, isolated and non-photorealistic problems to algorithms. There are three benefits: (i) Images can be created in large numbers at low cost. This addresses the currently largest problem in ground truth generation. (ii) We can combinatorially iterate through the design space to explore situations of highest relevance to the application. With increasing robustness of future stereo algorithms, datasets can be modified to increase matching challenges gradually. (iii) By isolating the core problems of stereo methods we can focus on each of them in turn. Our aim is not to produce a new dataset. Instead, we contribute with a new perspective on synthetic vision benchmark generation and show encouraging examples to validate our ideas. We believe that the potential of using synthetic data for evaluation in computer vision has not yet been fully utilized. Our first experiments demonstrate it is worthwhile to setup purpose designed datasets, as typical stereo failure can readily be reproduced, and thereby be better understood. Datasets are made available online [1].