Top

International Journal of Computer Vision

Published in:

07-11-2019

SceneFlowFields++: Multi-frame Matching, Visibility Prediction, and Robust Interpolation for Scene Flow Estimation

Authors: René Schuster, Oliver Wasenmüller, Christian Unger, Georg Kuschk, Didier Stricker

Published in: International Journal of Computer Vision | Issue 2/2020

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

State-of-the-art scene flow algorithms pursue the conflicting targets of accuracy, run time, and robustness. With the successful concept of pixel-wise matching and sparse-to-dense interpolation, we shift the operating point in this field of conflicts towards universality and speed. Avoiding strong assumptions on the domain or the problem yields a more robust algorithm. This algorithm is fast because we avoid explicit regularization during matching, which allows an efficient computation. Using image information from multiple time steps and explicit visibility prediction based on previous results, we achieve competitive performances on different data sets. Our contributions and results are evaluated in comparative experiments. Overall, we present an accurate scene flow algorithm that is faster and more generic than any individual benchmark leader.

previous article Hallucinating Unaligned Face Images by Multiscale Transformative Discriminative Networks

next article Statistical Modeling of Craniofacial Shape and Texture

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Bailer, C., Taetz, B., & Stricker, D. (2015). Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation. In International conference on computer vision (ICCV).

Bailer, C., Taetz, B., & Stricker, D. (2019). Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 41, 1879–1892.CrossRef

Basha, T., Moses, Y., & Kiryati, N. (2013). Multi-view scene flow estimation: A view centered variational approach. International Journal of Computer Vision (IJCV), 101, 6–21.MathSciNetCrossRef

Behl, A., Jafari, O. H., Mustikovela, S. K., Alhaija, H. A., Rother, C., & Geiger, A. (2017). Bounding boxes, segmentations and object coordinates: How important is recognition for 3D scene flow estimation in autonomous driving scenarios? In Conference on computer vision and pattern recognition (CVPR).

Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In European conference on computer vision (ECCV).

Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012). A naturalistic open source movie for optical flow evaluation. In European conference on computer vision (ECCV).

Chen, Q., & Koltun, V. (2016). Full flow: Optical flow estimation by global optimization over regular grids. In Conference on computer vision and pattern recognition (CVPR).

Dollár, P., & Zitnick, C. L. (2013). Structured forests for fast edge detection. In International conference on computer vision (ICCV).

Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016). Virtual worlds as proxy for multi-object tracking analysis. In Conference on computer vision and pattern recognition (CVPR).

Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. In Conference on computer vision and pattern recognition (CVPR).

He, K., & Sun, J. (2012). Computing nearest-neighbor fields via propagation-assisted kD-trees. In Conference on computer vision and pattern recognition (CVPR).

Herbst, E., Ren, X., & Fox, D. (2013). RGB-D flow: Dense 3-D motion estimation using color and depth. In International conference on robotics and automation (ICRA).

Hirschmüller, H. (2008). Stereo processing by semiglobal matching and mutual information. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 30, 328–341.CrossRef

Hu, Y., Li, Y., & Song, R. (2017). Robust interpolation of correspondences for large displacement optical flow. In Conference on computer vision and pattern recognition (CVPR).

Hu, Y., Song, R., & Li, Y. (2016). Efficient coarse-to-fine patchmatch for large displacement optical flow. In Conference on computer vision and pattern recognition (CVPR).

Huguet, F., & Devernay, F. (2007). A variational method for scene flow estimation from stereo sequences. In International conference on computer vision (ICCV).

Jaimez, M., Souiai, M., Gonzalez-Jimenez, J., & Cremers, D. (2015). A primal-dual framework for real-time dense RGB-D scene flow. In International conference on robotics and automation (ICRA).

Liu, C., Yuen, J., & Torralba, A. (2011). SIFT flow: Dense correspondence across scenes and its applications. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 33, 978–994.CrossRef

Lv, Z., Beall, C., Alcantarilla, P. F., Li, F., Kira, Z., & Dellaert, F. (2016). A continuous optimization approach for efficient and accurate scene flow. In European conference on computer vision (ECCV).

Lv, F., Lian, Q., Yang, G., Lin, G., Jialin Pan, S., & Duan, L. (2018). Domain adaptive semantic segmentation through structure enhancement. In European conference on computer vision (ECCV).

Ma, W. C., Wang, S., Hu, R., Xiong, Y., & Urtasun, R. (2019). Deep Rigid Instance Scene Flow. In Conference on computer vision and pattern recognition (CVPR).

Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In International conference on computer vision (ICCV).

Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., & Brox, T. (2016). A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Conference on computer vision and pattern recognition (CVPR).

Menze, M., & Geiger, A. (2015). Object scene flow for autonomous vehicles. In Conference on computer vision and pattern recognition (CVPR).

Menze, M., Heipke, C., & Geiger, A. (2018). Object scene flow. Journal of Photogrammetry and Remote Sensing (JPRS), 140, 60–76.CrossRef

Neoral, M., & Šochman, J. (2017). Object scene flow with temporal consistency. In Computer vision winter workshop (CVWW).

Ošep, A., Hermans, A., Engelmann, F., Klostermann, D., Mathias, M., & Leibe, B. (2016). Multi-scale object candidates for generic object tracking in street scenes. In International conference on robotics and automation (ICRA).

Ren, Z., Sun, D., Kautz, J., & Sudderth, E. B. (2017). Cascaded scene flow prediction using semantic segmentation. In International conference on 3D vision (3DV).

Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015). EpicFlow: edge-preserving interpolation of correspondences for optical flow. In Conference on computer vision and pattern recognition (CVPR).

Ros, G., Ramos, S., Granados, M., Bakhtiary, A., Vazquez, D., & Lopez, A. M. (2015). Vision-based offline–online perception paradigm for autonomous driving. In Winter conference on applications of computer vision (WACV).

Saxena, R., Schuster, R., Wasenmüller O, & Stricker D (2019) PWOC-3D: Deep occlusion-aware end-to-end scene flow estimation. In Intelligent vehicles symposium (IV).

Schuster, R., Bailer, C., Wasenmüller, O., & Stricker, D. (2018a) Combining stereo disparity and optical flow for basic scene flow. In Commercial vehicle technology symposium (CVT).

Schuster, R., Bailer, C., Wasenmüller, O., & Stricker, D. (2018b). FlowFields++: Accurate optical flow correspondences meet robust interpolation. In International conference on image processing (ICIP).

Schuster, R., Wasenmüller, O., Kuschk, G., Bailer, C., & Stricker, D. (2018c). SceneFlowFields: Dense interpolation of sparse scene flow correspondences. In Winter conference on applications of computer vision (WACV).

Taniai, T., Sinha, S. N., & Sato, Y. (2017). Fast multi-frame stereo scene flow with motion segmentation. In Conference on computer vision and pattern recognition (CVPR).

Vedula, S., Baker, S., Rander, P., Collins, R., & Kanade, T. (1999). Three-dimensional scene flow. In International conference on computer vision (ICCV).

Vogel, C., Roth, S., & Schindler, K. (2014). View-consistent 3d scene flow estimation over multiple frames. In European conference on computer vision (ECCV).

Vogel, C., Schindler, K., & Roth, S. (2013). Piecewise rigid scene flow. In International conference on computer vision (ICCV).

Vogel, C., Schindler, K., & Roth, S. (2015). 3D scene flow estimation with a piecewise rigid scene model. International Journal of Computer Vision (IJCV), 115, 1–28.MathSciNetCrossRef

Wang, M., & Deng, W. (2018). Deep visual domain adaptation: A survey. Neurocomputing, 312, 135–153.CrossRef

Wannenwetsch, A. S., Keuper, M., & Roth, S. (2017). ProbFlow: Joint optical flow and uncertainty estimation. In International conference on computer vision (ICCV).

Wedel, A., Rabe, C., Vaudrey, T., Brox, T., Franke, U., & Cremers. D. (2008). Efficient dense scene flow from sparse or dense stereo data. In European conference on computer vision (ECCV).

Xu, P., Davoine, F., Bordes, J. B., Zhao, H., & Denœux, T. (2016). Multimodal information fusion for urban scene understanding. Machine Vision and Applications (MVA), 27, 331–349.CrossRef

Yamaguchi, K., McAllester, D., & Urtasun, R. (2014). Efficient joint segmentation, occlusion labeling, stereo and flow estimation. In European conference on computer vision (ECCV).

Yoshida, T., Wasenmüller, O., & Stricker, D. (2017). Time-of-flight sensor depth enhancement for automotive exhaust gas. In International conference on image processing (ICIP).

Zweig, S., & Wolf, L. (2017). InterpoNet, a brain inspired neural network for optical flow dense interpolation. In Conference on computer vision and pattern recognition (CVPR).

Title: SceneFlowFields++: Multi-frame Matching, Visibility Prediction, and Robust Interpolation for Scene Flow Estimation
Authors: René Schuster
Oliver Wasenmüller
Christian Unger
Georg Kuschk
Didier Stricker
Publication date: 07-11-2019
Publisher: Springer US
Published in: International Journal of Computer Vision / Issue 2/2020
Print ISSN: 0920-5691
Electronic ISSN: 1573-1405
DOI: https://doi.org/10.1007/s11263-019-01258-1

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Other articles of this Issue 2/2020

Semantically Coherent 4D Scene Flow of Dynamic Scenes

Dual L1-Normalized Context Aware Tensor Power Iteration and Its Applications to Multi-object Tracking and Multi-graph Matching

Deep Insights into Convolutional Networks for Video Recognition

Adaptive Importance Learning for Improving Lightweight Image Super-Resolution Network

Hallucinating Unaligned Face Images by Multiscale Transformative Discriminative Networks

Recognizing Profile Faces by Imagining Frontal View

Premium Partner