Top

International Journal of Computer Vision

Published in:

01-12-2016

DeepMatching: Hierarchical Deformable Dense Matching

Authors: Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui, Cordelia Schmid

Published in: International Journal of Computer Vision | Issue 3/2016

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

We introduce a novel matching algorithm, called DeepMatching, to compute dense correspondences between images. DeepMatching relies on a hierarchical, multi-layer, correlational architecture designed for matching images and was inspired by deep convolutional approaches. The proposed matching algorithm can handle non-rigid deformations and repetitive textures and efficiently determines dense correspondences in the presence of significant changes between images. We evaluate the performance of DeepMatching, in comparison with state-of-the-art matching algorithms, on the Mikolajczyk (Mikolajczyk et al. A comparison of affine region detectors, 2005), the MPI-Sintel (Butler et al. A naturalistic open source movie for optical flow evaluation, 2012) and the Kitti (Geiger et al. Vision meets robotics: The KITTI dataset, 2013) datasets. DeepMatching outperforms the state-of-the-art algorithms and shows excellent results in particular for repetitive textures. We also apply DeepMatching to the computation of optical flow, called DeepFlow, by integrating it in the large displacement optical flow (LDOF) approach of Brox and Malik (Large displacement optical flow: descriptor matching in variational motion estimation, 2011). Additional robustness to large displacements and complex motion is obtained thanks to our matching approach. DeepFlow obtains competitive performance on public benchmarks for optical flow estimation.

previous article A Riemannian Bayesian Framework for Estimating Diffusion Tensor Images

next article Multiview Differential Geometry of Curves

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

This amounts to the cross-correlation of the patch and \(I'\).

Note that \(I'_{N,\varvec{p}'}\) only roughly corresponds to a \(N \times N\) square patch centered at \(2^\ell \varvec{p}'\) in \(I'\), due to subsampling and possible deformations.

Although the backtracking is conceptually close to the back-propagation training algorithm, it differs in term of how the scores are accumulated for each path.

We implemented this method ourselves.

We used the online code.

We report results from the original paper.

Note that this systematically improves the endpoint error compared to using the raw dense correspondence fields as flow.

Bailer, C., Taetz, B., & Stricker, D. (2015). Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation.

Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011). A database and evaluation methodology for optical flow. In IJCV.

Barnes, C., Shechtman, E., Goldman, D. B., & Finkelstein, A. (2010). The generalized PatchMatch correspondence algorithm. In ECCV.

Bengio, Y. (2009). Learning deep architectures for AI. Foundations and Trends in Machine Learning, 1(1), 1–127.MathSciNetCrossRefMATH

Black, M. J., & Anandan, P. (1996). The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding, 63, 75–104.CrossRef

Braux-Zin, J., Dupont, R., & Bartoli, A. (2013). A general dense image matching framework combining direct and feature-based costs. In ICCV.

Brox, T., & Malik, J. (2011). Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Transactions on PAMI, 33, 500–513.CrossRef

Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In ECCV.

Bruhn, A., Weickert, J., Feddern, C., Kohlberger, T., & Schnörr, C. (2005). Variational optical flow computation in real time. IEEE Transactions on Image Processing, 15, 608–615.MathSciNetCrossRefMATH

Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012). A naturalistic open source movie for optical flow evaluation. In ECCV.

Chen, Z., Jin, H., Lin, Z., Cohen, S., & Wu, Y. (2013). Large displacement optical flow from nearest neighbor fields. In CVPR.

Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In CVPR.

Demetz, O., Stoll, M., Volz, S., Weickert, J., & Bruhn, A. (2014). Learning brightness transfer functions for the joint recovery of illumination changes and optical flow. In ECCV.

Ecker, A., & Ullman, S. (2009). A hierarchical non-parametric method for capturing non-rigid deformations. Image and Vision Computing, 27, 87–98.CrossRef

Forsyth, D., & Ponce, J. (2011). Computer vision: A modern approach. New York: Pearson Education.

Furukawa, Y., Curless, B., Seitz, S. M., & Szeliski, R. (2010). Towards internet-scale multi-view stereo. In CVPR.

Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013). Vision meets robotics: The KITTI dataset. IJRR, 32, 1231–1237.

HaCohen, Y., Shechtman, E., Goldman, D. B., & Lischinski, D. (2011). Non-rigid dense correspondence with applications for image enhancement. In SIGGRAPH.

Hassner, T., Mayzels, V., & Zelnik-Manor, L. (2012). On sifts and their scales. In CVPR.

Horn, B. K. P., & Schunck, B. G. (1981). Determining optical flow. Artificial Intelligence, 17, 185–203.CrossRef

Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014). Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093.

Kennedy, R., & Taylor, C. J. (2015). Optical flow with geometric occlusion estimation and fusion of multiple frames. In EMMCVPR.

Keysers, D., Deselaers, T., Gollan, C., & Ney, H. (2007). Deformation models for image recognition. IEEE Transactions on PAMI, 29, 1422–1435.CrossRef

Kim, J., Liu, C., Sha, F., & Grauman, K. (2013). Deformable spatial pyramid matching for fast dense correspondences. In CVPR.

Korman, S., & Avidan, S. (2011). Coherency sensitive hashing. In ICCV.

LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998a). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86, 2278–2324.CrossRef

LeCun, Y., Bottou, L., Orr, G., & Muller, K. (1998b). Efficient backprop. In Neural Networks: Tricks of the trade.

Leordeanu, M., Zanfir, A., & Sminchisescu, C. (2013). Locally affine sparse-to-dense matching for motion and occlusion estimation. In ICCV.

Liu, C., Yuen, J., & Torralba, A. (2011). SIFT flow: Dense correspondence across scenes and its applications. IEEE Transactions on PAMI, 33, 978–994.CrossRef

Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. IJCV, 60, 91–110.CrossRef

Malik, J., & Perona, P. (1990). Preattentive texture discrimination with early vision mechanisms. Journal of the Optical Society of America A, 7, 923–932.CrossRef

Menze, M., Heipke, C., & Geiger, A. (2015). Discrete optimization for optical flow. In GCPR.

Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., et al. (2005). A comparison of affine region detectors. IJCV, 65, 43–72.CrossRef

Muja, M., & Lowe, D.G. (2009). Fast approximate nearest neighbors with automatic algorithm configuration. In International Conference on Computer Vision Theory and Application VISSAPP’09). INSTICC Press.

Papenberg, N., Bruhn, A., Brox, T., Didas, S., & Weickert, J. (2006). Highly accurate optic flow computation with theoretically justified warping. IJCV, 67, 141–158.CrossRef

Philbin, J., Isard, M., Sivic, J., & Zisserman, A. (2010). Descriptor learning for efficient retrieval. In ECCV.

Ranftl, R., Bredies, K., & Pock, T. (2014). Non-local total generalized variation for optical flow estimation. In ECCV.

Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015). EpicFlow: Edge-preserving interpolation of correspondences for optical flow. In CVPR.

Stoll, M., Volz, S., & Bruhn, A. (2012). Adaptive integration of feature matches into variational optical flow methods. In ACCV.

Sun, D., Liu, C., & Pfister, H. (2014a). Local layering for joint motion estimation and occlusion detection. In CVPR.

Sun, D., Roth, S., & Black, M. (2014b). A quantitative analysis of current practices in optical flow estimation and the principles behind them. IJCV, 106, 115–137.CrossRef

Sun, J. (2012). Computing nearest-neighbor fields via propagation-assisted kd-trees. In CVPR.

Szeliski, R. (2010). Computer vision: Algorithms and applications. New York: Springer.MATH

Timofte, R., & Van Gool, L. (2015). Sparse flow: Sparse matching for small to large displacement optical flow. In Applications of Computer Vision (WACV).

Tola, E., Lepetit, V., & Fua, P. (2008). A fast local descriptor for dense matching. In CVPR.

Tola, E., Lepetit, V., & Fua, P. (2010). DAISY: An efficient dense descriptor applied to wide baseline stereo. IEEE Transactions on PAMI, 32, 815–830.CrossRef

Uchida, S.,&Sakoe, H. (1998). A monotonic and continuous two-dimensional warping based on dynamic programming. In ICPR.

Vogel, C., Roth, S., & Schindler, K. (2013a). An evaluation of data costs for optical flow. In GCPR.

Vogel, C., Schindler, K., & Roth, S. (2013b). Piecewise rigid scene flow. In ICCV.

Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009). Structure- and motion-adaptive regularization for high accuracy optic flow. In ICCV.

Weinzaepfel, P., Revaud, J., Harchaoui, Z., & Schmid, C. (2013). Deepflow: Large displacement optical flow with deep matching. In ICCV.

Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., & Bischof, H. (2009). Anisotropic Huber-L1 optical flow. In BMVC.

Wills, J., Agarwal, S., & Belongie, S. (2006). A feature-based approach for dense segmentation and estimation of large disparity motion. IJCV.

Xu, L., Jia, J., & Matsushita, Y. (2012). Motion detail preserving optical flow estimation. IEEE Transactions on PAMI, 34, 1744–1757.CrossRef

Yang, H., Lin, W., & Lu, J. (2014). DAISY filter flow: A generalized discrete approach to dense correspondences. In CVPR.

Young, D. M., & Rheinboldt, W. (1971). Iterative solution of large linear systems. New York: Academic Press.

Zimmer, H., Bruhn, A., & Weickert, J. (2011). Optic flow in harmony. IJCV.

Title: DeepMatching: Hierarchical Deformable Dense Matching
Authors: Jerome Revaud
Philippe Weinzaepfel
Zaid Harchaoui
Cordelia Schmid
Publication date: 01-12-2016
Publisher: Springer US
Published in: International Journal of Computer Vision / Issue 3/2016
Print ISSN: 0920-5691
Electronic ISSN: 1573-1405
DOI: https://doi.org/10.1007/s11263-016-0908-3

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Other articles of this Issue 3/2016

Multiview Differential Geometry of Curves

A Riemannian Bayesian Framework for Estimating Diffusion Tensor Images

Visualizing Deep Convolutional Neural Networks Using Natural Pre-images

Occlusion-Aware Stereo Matching

Premium Partner