Skip to main content
Erschienen in: International Journal of Computer Vision 3/2016

01.12.2016

DeepMatching: Hierarchical Deformable Dense Matching

verfasst von: Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui, Cordelia Schmid

Erschienen in: International Journal of Computer Vision | Ausgabe 3/2016

Einloggen

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

We introduce a novel matching algorithm, called DeepMatching, to compute dense correspondences between images. DeepMatching relies on a hierarchical, multi-layer, correlational architecture designed for matching images and was inspired by deep convolutional approaches. The proposed matching algorithm can handle non-rigid deformations and repetitive textures and efficiently determines dense correspondences in the presence of significant changes between images. We evaluate the performance of DeepMatching, in comparison with state-of-the-art matching algorithms, on the Mikolajczyk (Mikolajczyk et al. A comparison of affine region detectors, 2005), the MPI-Sintel (Butler et al. A naturalistic open source movie for optical flow evaluation, 2012) and the Kitti (Geiger et al. Vision meets robotics: The KITTI dataset, 2013) datasets. DeepMatching outperforms the state-of-the-art algorithms and shows excellent results in particular for repetitive textures. We also apply DeepMatching to the computation of optical flow, called DeepFlow, by integrating it in the large displacement optical flow (LDOF) approach of Brox and Malik (Large displacement optical flow: descriptor matching in variational motion estimation, 2011). Additional robustness to large displacements and complex motion is obtained thanks to our matching approach. DeepFlow obtains competitive performance on public benchmarks for optical flow estimation.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Fußnoten
1
This amounts to the cross-correlation of the patch and \(I'\).
 
2
Note that \(I'_{N,\varvec{p}'}\) only roughly corresponds to a \(N \times N\) square patch centered at \(2^\ell \varvec{p}'\) in \(I'\), due to subsampling and possible deformations.
 
3
Although the backtracking is conceptually close to the back-propagation training algorithm, it differs in term of how the scores are accumulated for each path.
 
4
We implemented this method ourselves.
 
5
We used the online code.
 
6
We report results from the original paper.
 
7
Note that this systematically improves the endpoint error compared to using the raw dense correspondence fields as flow.
 
Literatur
Zurück zum Zitat Bailer, C., Taetz, B., & Stricker, D. (2015). Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation. Bailer, C., Taetz, B., & Stricker, D. (2015). Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation.
Zurück zum Zitat Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011). A database and evaluation methodology for optical flow. In IJCV. Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011). A database and evaluation methodology for optical flow. In IJCV.
Zurück zum Zitat Barnes, C., Shechtman, E., Goldman, D. B., & Finkelstein, A. (2010). The generalized PatchMatch correspondence algorithm. In ECCV. Barnes, C., Shechtman, E., Goldman, D. B., & Finkelstein, A. (2010). The generalized PatchMatch correspondence algorithm. In ECCV.
Zurück zum Zitat Black, M. J., & Anandan, P. (1996). The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding, 63, 75–104.CrossRef Black, M. J., & Anandan, P. (1996). The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding, 63, 75–104.CrossRef
Zurück zum Zitat Braux-Zin, J., Dupont, R., & Bartoli, A. (2013). A general dense image matching framework combining direct and feature-based costs. In ICCV. Braux-Zin, J., Dupont, R., & Bartoli, A. (2013). A general dense image matching framework combining direct and feature-based costs. In ICCV.
Zurück zum Zitat Brox, T., & Malik, J. (2011). Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Transactions on PAMI, 33, 500–513.CrossRef Brox, T., & Malik, J. (2011). Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Transactions on PAMI, 33, 500–513.CrossRef
Zurück zum Zitat Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In ECCV. Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In ECCV.
Zurück zum Zitat Bruhn, A., Weickert, J., Feddern, C., Kohlberger, T., & Schnörr, C. (2005). Variational optical flow computation in real time. IEEE Transactions on Image Processing, 15, 608–615.MathSciNetCrossRefMATH Bruhn, A., Weickert, J., Feddern, C., Kohlberger, T., & Schnörr, C. (2005). Variational optical flow computation in real time. IEEE Transactions on Image Processing, 15, 608–615.MathSciNetCrossRefMATH
Zurück zum Zitat Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012). A naturalistic open source movie for optical flow evaluation. In ECCV. Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012). A naturalistic open source movie for optical flow evaluation. In ECCV.
Zurück zum Zitat Chen, Z., Jin, H., Lin, Z., Cohen, S., & Wu, Y. (2013). Large displacement optical flow from nearest neighbor fields. In CVPR. Chen, Z., Jin, H., Lin, Z., Cohen, S., & Wu, Y. (2013). Large displacement optical flow from nearest neighbor fields. In CVPR.
Zurück zum Zitat Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In CVPR. Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In CVPR.
Zurück zum Zitat Demetz, O., Stoll, M., Volz, S., Weickert, J., & Bruhn, A. (2014). Learning brightness transfer functions for the joint recovery of illumination changes and optical flow. In ECCV. Demetz, O., Stoll, M., Volz, S., Weickert, J., & Bruhn, A. (2014). Learning brightness transfer functions for the joint recovery of illumination changes and optical flow. In ECCV.
Zurück zum Zitat Ecker, A., & Ullman, S. (2009). A hierarchical non-parametric method for capturing non-rigid deformations. Image and Vision Computing, 27, 87–98.CrossRef Ecker, A., & Ullman, S. (2009). A hierarchical non-parametric method for capturing non-rigid deformations. Image and Vision Computing, 27, 87–98.CrossRef
Zurück zum Zitat Forsyth, D., & Ponce, J. (2011). Computer vision: A modern approach. New York: Pearson Education. Forsyth, D., & Ponce, J. (2011). Computer vision: A modern approach. New York: Pearson Education.
Zurück zum Zitat Furukawa, Y., Curless, B., Seitz, S. M., & Szeliski, R. (2010). Towards internet-scale multi-view stereo. In CVPR. Furukawa, Y., Curless, B., Seitz, S. M., & Szeliski, R. (2010). Towards internet-scale multi-view stereo. In CVPR.
Zurück zum Zitat Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013). Vision meets robotics: The KITTI dataset. IJRR, 32, 1231–1237. Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013). Vision meets robotics: The KITTI dataset. IJRR, 32, 1231–1237.
Zurück zum Zitat HaCohen, Y., Shechtman, E., Goldman, D. B., & Lischinski, D. (2011). Non-rigid dense correspondence with applications for image enhancement. In SIGGRAPH. HaCohen, Y., Shechtman, E., Goldman, D. B., & Lischinski, D. (2011). Non-rigid dense correspondence with applications for image enhancement. In SIGGRAPH.
Zurück zum Zitat Hassner, T., Mayzels, V., & Zelnik-Manor, L. (2012). On sifts and their scales. In CVPR. Hassner, T., Mayzels, V., & Zelnik-Manor, L. (2012). On sifts and their scales. In CVPR.
Zurück zum Zitat Horn, B. K. P., & Schunck, B. G. (1981). Determining optical flow. Artificial Intelligence, 17, 185–203.CrossRef Horn, B. K. P., & Schunck, B. G. (1981). Determining optical flow. Artificial Intelligence, 17, 185–203.CrossRef
Zurück zum Zitat Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014). Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014). Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:​1408.​5093.
Zurück zum Zitat Kennedy, R., & Taylor, C. J. (2015). Optical flow with geometric occlusion estimation and fusion of multiple frames. In EMMCVPR. Kennedy, R., & Taylor, C. J. (2015). Optical flow with geometric occlusion estimation and fusion of multiple frames. In EMMCVPR.
Zurück zum Zitat Keysers, D., Deselaers, T., Gollan, C., & Ney, H. (2007). Deformation models for image recognition. IEEE Transactions on PAMI, 29, 1422–1435.CrossRef Keysers, D., Deselaers, T., Gollan, C., & Ney, H. (2007). Deformation models for image recognition. IEEE Transactions on PAMI, 29, 1422–1435.CrossRef
Zurück zum Zitat Kim, J., Liu, C., Sha, F., & Grauman, K. (2013). Deformable spatial pyramid matching for fast dense correspondences. In CVPR. Kim, J., Liu, C., Sha, F., & Grauman, K. (2013). Deformable spatial pyramid matching for fast dense correspondences. In CVPR.
Zurück zum Zitat Korman, S., & Avidan, S. (2011). Coherency sensitive hashing. In ICCV. Korman, S., & Avidan, S. (2011). Coherency sensitive hashing. In ICCV.
Zurück zum Zitat LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998a). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86, 2278–2324.CrossRef LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998a). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86, 2278–2324.CrossRef
Zurück zum Zitat LeCun, Y., Bottou, L., Orr, G., & Muller, K. (1998b). Efficient backprop. In Neural Networks: Tricks of the trade. LeCun, Y., Bottou, L., Orr, G., & Muller, K. (1998b). Efficient backprop. In Neural Networks: Tricks of the trade.
Zurück zum Zitat Leordeanu, M., Zanfir, A., & Sminchisescu, C. (2013). Locally affine sparse-to-dense matching for motion and occlusion estimation. In ICCV. Leordeanu, M., Zanfir, A., & Sminchisescu, C. (2013). Locally affine sparse-to-dense matching for motion and occlusion estimation. In ICCV.
Zurück zum Zitat Liu, C., Yuen, J., & Torralba, A. (2011). SIFT flow: Dense correspondence across scenes and its applications. IEEE Transactions on PAMI, 33, 978–994.CrossRef Liu, C., Yuen, J., & Torralba, A. (2011). SIFT flow: Dense correspondence across scenes and its applications. IEEE Transactions on PAMI, 33, 978–994.CrossRef
Zurück zum Zitat Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. IJCV, 60, 91–110.CrossRef Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. IJCV, 60, 91–110.CrossRef
Zurück zum Zitat Malik, J., & Perona, P. (1990). Preattentive texture discrimination with early vision mechanisms. Journal of the Optical Society of America A, 7, 923–932.CrossRef Malik, J., & Perona, P. (1990). Preattentive texture discrimination with early vision mechanisms. Journal of the Optical Society of America A, 7, 923–932.CrossRef
Zurück zum Zitat Menze, M., Heipke, C., & Geiger, A. (2015). Discrete optimization for optical flow. In GCPR. Menze, M., Heipke, C., & Geiger, A. (2015). Discrete optimization for optical flow. In GCPR.
Zurück zum Zitat Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., et al. (2005). A comparison of affine region detectors. IJCV, 65, 43–72.CrossRef Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., et al. (2005). A comparison of affine region detectors. IJCV, 65, 43–72.CrossRef
Zurück zum Zitat Muja, M., & Lowe, D.G. (2009). Fast approximate nearest neighbors with automatic algorithm configuration. In International Conference on Computer Vision Theory and Application VISSAPP’09). INSTICC Press. Muja, M., & Lowe, D.G. (2009). Fast approximate nearest neighbors with automatic algorithm configuration. In International Conference on Computer Vision Theory and Application VISSAPP’09). INSTICC Press.
Zurück zum Zitat Papenberg, N., Bruhn, A., Brox, T., Didas, S., & Weickert, J. (2006). Highly accurate optic flow computation with theoretically justified warping. IJCV, 67, 141–158.CrossRef Papenberg, N., Bruhn, A., Brox, T., Didas, S., & Weickert, J. (2006). Highly accurate optic flow computation with theoretically justified warping. IJCV, 67, 141–158.CrossRef
Zurück zum Zitat Philbin, J., Isard, M., Sivic, J., & Zisserman, A. (2010). Descriptor learning for efficient retrieval. In ECCV. Philbin, J., Isard, M., Sivic, J., & Zisserman, A. (2010). Descriptor learning for efficient retrieval. In ECCV.
Zurück zum Zitat Ranftl, R., Bredies, K., & Pock, T. (2014). Non-local total generalized variation for optical flow estimation. In ECCV. Ranftl, R., Bredies, K., & Pock, T. (2014). Non-local total generalized variation for optical flow estimation. In ECCV.
Zurück zum Zitat Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015). EpicFlow: Edge-preserving interpolation of correspondences for optical flow. In CVPR. Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015). EpicFlow: Edge-preserving interpolation of correspondences for optical flow. In CVPR.
Zurück zum Zitat Stoll, M., Volz, S., & Bruhn, A. (2012). Adaptive integration of feature matches into variational optical flow methods. In ACCV. Stoll, M., Volz, S., & Bruhn, A. (2012). Adaptive integration of feature matches into variational optical flow methods. In ACCV.
Zurück zum Zitat Sun, D., Liu, C., & Pfister, H. (2014a). Local layering for joint motion estimation and occlusion detection. In CVPR. Sun, D., Liu, C., & Pfister, H. (2014a). Local layering for joint motion estimation and occlusion detection. In CVPR.
Zurück zum Zitat Sun, D., Roth, S., & Black, M. (2014b). A quantitative analysis of current practices in optical flow estimation and the principles behind them. IJCV, 106, 115–137.CrossRef Sun, D., Roth, S., & Black, M. (2014b). A quantitative analysis of current practices in optical flow estimation and the principles behind them. IJCV, 106, 115–137.CrossRef
Zurück zum Zitat Sun, J. (2012). Computing nearest-neighbor fields via propagation-assisted kd-trees. In CVPR. Sun, J. (2012). Computing nearest-neighbor fields via propagation-assisted kd-trees. In CVPR.
Zurück zum Zitat Szeliski, R. (2010). Computer vision: Algorithms and applications. New York: Springer.MATH Szeliski, R. (2010). Computer vision: Algorithms and applications. New York: Springer.MATH
Zurück zum Zitat Timofte, R., & Van Gool, L. (2015). Sparse flow: Sparse matching for small to large displacement optical flow. In Applications of Computer Vision (WACV). Timofte, R., & Van Gool, L. (2015). Sparse flow: Sparse matching for small to large displacement optical flow. In Applications of Computer Vision (WACV).
Zurück zum Zitat Tola, E., Lepetit, V., & Fua, P. (2008). A fast local descriptor for dense matching. In CVPR. Tola, E., Lepetit, V., & Fua, P. (2008). A fast local descriptor for dense matching. In CVPR.
Zurück zum Zitat Tola, E., Lepetit, V., & Fua, P. (2010). DAISY: An efficient dense descriptor applied to wide baseline stereo. IEEE Transactions on PAMI, 32, 815–830.CrossRef Tola, E., Lepetit, V., & Fua, P. (2010). DAISY: An efficient dense descriptor applied to wide baseline stereo. IEEE Transactions on PAMI, 32, 815–830.CrossRef
Zurück zum Zitat Uchida, S.,&Sakoe, H. (1998). A monotonic and continuous two-dimensional warping based on dynamic programming. In ICPR. Uchida, S.,&Sakoe, H. (1998). A monotonic and continuous two-dimensional warping based on dynamic programming. In ICPR.
Zurück zum Zitat Vogel, C., Roth, S., & Schindler, K. (2013a). An evaluation of data costs for optical flow. In GCPR. Vogel, C., Roth, S., & Schindler, K. (2013a). An evaluation of data costs for optical flow. In GCPR.
Zurück zum Zitat Vogel, C., Schindler, K., & Roth, S. (2013b). Piecewise rigid scene flow. In ICCV. Vogel, C., Schindler, K., & Roth, S. (2013b). Piecewise rigid scene flow. In ICCV.
Zurück zum Zitat Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009). Structure- and motion-adaptive regularization for high accuracy optic flow. In ICCV. Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009). Structure- and motion-adaptive regularization for high accuracy optic flow. In ICCV.
Zurück zum Zitat Weinzaepfel, P., Revaud, J., Harchaoui, Z., & Schmid, C. (2013). Deepflow: Large displacement optical flow with deep matching. In ICCV. Weinzaepfel, P., Revaud, J., Harchaoui, Z., & Schmid, C. (2013). Deepflow: Large displacement optical flow with deep matching. In ICCV.
Zurück zum Zitat Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., & Bischof, H. (2009). Anisotropic Huber-L1 optical flow. In BMVC. Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., & Bischof, H. (2009). Anisotropic Huber-L1 optical flow. In BMVC.
Zurück zum Zitat Wills, J., Agarwal, S., & Belongie, S. (2006). A feature-based approach for dense segmentation and estimation of large disparity motion. IJCV. Wills, J., Agarwal, S., & Belongie, S. (2006). A feature-based approach for dense segmentation and estimation of large disparity motion. IJCV.
Zurück zum Zitat Xu, L., Jia, J., & Matsushita, Y. (2012). Motion detail preserving optical flow estimation. IEEE Transactions on PAMI, 34, 1744–1757.CrossRef Xu, L., Jia, J., & Matsushita, Y. (2012). Motion detail preserving optical flow estimation. IEEE Transactions on PAMI, 34, 1744–1757.CrossRef
Zurück zum Zitat Yang, H., Lin, W., & Lu, J. (2014). DAISY filter flow: A generalized discrete approach to dense correspondences. In CVPR. Yang, H., Lin, W., & Lu, J. (2014). DAISY filter flow: A generalized discrete approach to dense correspondences. In CVPR.
Zurück zum Zitat Young, D. M., & Rheinboldt, W. (1971). Iterative solution of large linear systems. New York: Academic Press. Young, D. M., & Rheinboldt, W. (1971). Iterative solution of large linear systems. New York: Academic Press.
Zurück zum Zitat Zimmer, H., Bruhn, A., & Weickert, J. (2011). Optic flow in harmony. IJCV. Zimmer, H., Bruhn, A., & Weickert, J. (2011). Optic flow in harmony. IJCV.
Metadaten
Titel
DeepMatching: Hierarchical Deformable Dense Matching
verfasst von
Jerome Revaud
Philippe Weinzaepfel
Zaid Harchaoui
Cordelia Schmid
Publikationsdatum
01.12.2016
Verlag
Springer US
Erschienen in
International Journal of Computer Vision / Ausgabe 3/2016
Print ISSN: 0920-5691
Elektronische ISSN: 1573-1405
DOI
https://doi.org/10.1007/s11263-016-0908-3

Weitere Artikel der Ausgabe 3/2016

International Journal of Computer Vision 3/2016 Zur Ausgabe