nach oben

International Journal of Computer Vision

Erschienen in:

04.03.2020

Deep Image Prior

verfasst von: Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky

Erschienen in: International Journal of Computer Vision | Ausgabe 7/2020

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Deep convolutional networks have become a popular tool for image generation and restoration. Generally, their excellent performance is imputed to their ability to learn realistic image priors from a large number of example images. In this paper, we show that, on the contrary, the structure of a generator network is sufficient to capture a great deal of low-level image statistics prior to any learning. In order to do so, we show that a randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, super-resolution, and inpainting. Furthermore, the same prior can be used to invert deep neural representations to diagnose them, and to restore images based on flash-no flash input pairs. Apart from its diverse applications, our approach highlights the inductive bias captured by standard generator network architectures. It also bridges the gap between two very popular families of image restoration methods: learning-based methods using deep convolutional networks and learning-free methods based on handcrafted image priors such as self-similarity (Code and supplementary material are available at https://dmitryulyanov.github.io/deep_image_prior).

Vorheriger Artikel Is There Anything New to Say About SIFT Matching?

Nächster Artikel Light Structure from Pin Motion: Geometric Point Light Source Calibration

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Equation (2) can also be thought of as a regularizer R(x) in the style of (1), where \(R(x)=0\) for all images that can be generated by a deep ConvNet of a certain architecture with the weights being not too far from random initialization, and \(R(x)=+\infty \) for all other signals.

http://www.cs.tut.fi/~foi/GCF-BM3D/index.html#ref_results.

Athar, S., Burnaev, E., & Lempitsky, V. S. (2018). Latent convolutional models. In CoRR.

Bahat, Y., Efrat, N., & Irani, M. (2017). Non-uniform blind deblurring by reblurring. In Proceedings of CVPR (pp. 3286–3294). IEEE Computer Society.

Bevilacqua, M., Roumy, A., Guillemot, C., & Alberi-Morel, M. (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC (pp. 1–10).

Bojanowski, P., Joulin, A., Lopez-Paz, D., & Szlam, A. (2017). Optimizing the latent space of generative networks. In CoRR.

Boominathan, L., Maniparambil, M., Gupta, H., Baburajan, R., & Mitra, K. (2018). Phase retrieval for fourier ptychography under varying amount of measurements. In CoRR.

Bristow, H., Eriksson, A. P., & Lucey, S. (2013). Fast convolutional sparse coding. In CVPR (pp. 391–398). IEEE Computer Society.

Buades, A. (2005). NLM demo. Retrieved December 2017 from http://demo.ipol.im/demo/bcm_non_local_means_denoising/.

Buades, A., Coll, B., & Morel, J. M. (2005). A non-local algorithm for image denoising. In Proceedings of CVPR (Vol. 2, pp. 60–65). IEEE Computer Society.

Burger, M., Osher, S. J., Xu, J., & Gilboa, G. (2005). Nonlinear inverse scale space methods for image restoration. In Variational, geometric, and level set methods in computer vision, third international workshop, VLSM (pp. 25–36).

Burger, H. C., Schuler, C. J., & Harmeling, S. (2012). Image denoising: Can plain neural networks compete with bm3d? In CVPR (pp. 2392–2399).

Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8), 2080–2095.MathSciNetCrossRef

Dong, C., Loy, C.C., He, K., & Tang, X. (2014). Learning a deep convolutional network for image super-resolution. In Proceedings of ECCV (pp. 184–199).

Dosovitskiy, A., & Brox, T. (2016a). Generating images with perceptual similarity metrics based on deep networks. In NIPS (pp. 658–666).

Dosovitskiy, A., & Brox, T. (2016b). Inverting convolutional networks with convolutional networks. In CVPR. IEEE Computer Society.

Dosovitskiy, A., Tobias Springenberg, J., & Brox, T. (2015). Learning to generate chairs with convolutional neural networks. In Proceedings of CVPR (pp. 1538–1546).

Erhan, D., Bengio, Y., Courville, A., & Vincent, P. (2009). Visualizing higher-layer features of a deep network. Tech. Rep. Technical Report 1341, University of Montreal.

Field, D. J. (1987). Relations between the statistics of natural images and the response properties of cortical cells. Josa A, 4(12), 2379–2394.CrossRef

Glasner, D., Bagon, S., & Irani, M. (2009). Super-resolution from a single image. In Proceedings of ICCV (pp. 349–356).

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In Proceedings of NIPS (pp. 2672–2680).

Grosse, R. B., Raina, R., Kwong, H., & Ng, A. Y. (2007). Shift-invariance sparse coding for audio classification. In UAI (pp. 149–158). AUAI Press.

Gu, S., Zuo, W., Xie, Q., Meng, D., Feng, X., & Zhang, L. (2015). Convolutional sparse coding for image super-resolution. In ICCV (pp. 1823–1831). IEEE Computer Society.

He, K., Sun, J., & Tang, X. (2013). Guided image filtering. T-PAMI, 35(6), 1397–1409.CrossRef

He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In CVPR (pp. 1026–1034). IEEE Computer Society.

Heide, F., Heidrich, W., & Wetzstein, G. (2015). Fast and flexible convolutional sparse coding. In CVPR (pp. 5135–5143). IEEE Computer Society.

Huang, J., & Mumford, D. (1999). Statistics of natural images and models. In CVPR (pp. 1541–1547). IEEE Computer Society.

Huang, J.B., Singh, A., & Ahuja, N. (2015). Single image super-resolution from transformed self-exemplars. In CVPR (pp. 5197–5206). IEEE Computer Society.

Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2017). Globally and locally consistent image completion. ACM Transactions on Graphics (Proceedings of SIGGRAPH)36(4), 107:1–107:14 (2017)

Ilyas, A., Jalal, A., Asteri, E., Daskalakis, C., & Dimakis, A. G. (2017). The robust manifold defense: Adversarial training using generative models. In CoRR.

Kim, J., Lee, J. K., & Lee, K. M. (2016). Accurate image super-resolution using very deep convolutional networks. In CVPR (pp. 1646–1654). IEEE Computer Society.

Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. In CoRR.

Kingma, D. P., & Welling, M. (2014). Auto-encoding variational bayes. In Proceedings of ICLR.

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, K. Q. Weinberger (Eds.) Advances in neural information processing systems (Vol. 25, pp. 1097–1105). New York:Curran Associates, Inc.

Lai, W. S., Huang, J. B., Ahuja, N., & Yang, M. H. (2017). Deep laplacian pyramid networks for fast and accurate super-resolution. In CVPR. IEEE Computer Society.

Lebrun, M. (2011). BM3D code. Retrieved December 2017 from https://github.com/gfacciol/bm3d.

Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., et al. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In CVPR. IEEE Computer Society.

Lefkimmiatis, S. (2016). Non-local color image denoising with convolutional neural networks. In CVPR. IEEE Computer Society.

Mahendran, A., & Vedaldi, A. (2015). Understanding deep image representations by inverting them. In CVPR. IEEE Computer Society.

Mahendran, A., & Vedaldi, A. (2016). Visualizing deep convolutional neural networks using natural pre-images. In IJCV.

Mairal, J., Bach, F., Ponce, J., & Sapiro, G. (2010). Online learning for matrix factorization and sparse coding. Journal of Machine Learning Research, 11(Jan), 19–60.MathSciNetMATH

Mallat, S. G. (1989). A theory for multiresolution signal decomposition: The wavelet representation. PAMI, 11(7), 674–693.CrossRef

Marquina, A. (2009). Nonlinear inverse scale space methods for total variation blind deconvolution. SIAM Journal on Imaging Sciences, 2(1), 64–83.MathSciNetCrossRef

Papyan, V., Romano, Y., & Elad, M. (2017). Convolutional neural networks analyzed via convolutional sparse coding. Journal of Machine Learning Research, 18(83), 1–52.MathSciNetMATH

Papyan, V., Romano, Y., Sulam, J., & Elad, M. (2017). Convolutional dictionary learning via local processing. In ICCV. IEEE Computer Society.

Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M. F., Hoppe, H., & Toyama, K. (2004). Digital photography with flash and no-flash image pairs. ACM Transactions on Graphics, 23(3), 664–672.CrossRef

Plotz, T., & Roth, S. (2017). Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1586–1595).

Ren, J. S. J., Xu, L., Yan, Q., & Sun, W. (2015). Shepard convolutional neural networks. In NIPS (pp. 901–909).

Roth, S., & Black, M. J. (2009). Fields of experts. CVPR, 82(2), 205–229.

Ruderman, D. L., & Bialek, W. (1993). Statistics of natural images: Scaling in the woods. In NIPS (pp. 551–558). Morgan Kaufmann.

Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. In Proceedings of the eleventh annual international conference of the center for nonlinear studies on experimental mathematics : Computational issues in nonlinear science: Computational issues in nonlinear science (pp. 259–268). New York, NY, USA: Elsevier North-Holland, Inc.

Sajjadi, M. S. M., Scholkopf, B., & Hirsch, M. (2017). Enhancenet: Single image super-resolution through automated texture synthesis. In The IEEE international conference on computer vision (ICCV).

Scherzer, O., & Groetsch, C. W. (2001). Inverse scale space theory for inverse problems. In Scale-space and morphology in computer vision, third international conference (pp. 317–325).

Shedligeri, P. A., Shah, K., Kumar, D., & Mitra, K. (2018). Photorealistic image reconstruction from hybrid intensity and event based sensor. In CoRR.

Shocher, A., Cohen, N., & Irani, M. (2018). “Zero-shot” super-resolution using deep internal learning. In CVPR: IEEE Computer Society.CrossRef

Simoncelli, E. P., & Adelson, E. H. (1996). Noise removal via bayesian wavelet coring. In ICIP (1) (pp. 379–382). IEEE Computer Society.

Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. In CoRR.

Tai, Y., Yang, J., & Liu, X. (2017). Image super-resolution via deep recursive residual network. In CVPR. IEEE Computer Society.

Turiel, A., Mato, G., Parga, N., & Nadal, J. (1997). Self-similarity properties of natural images. In NIPS (pp. 836–842). The MIT Press.

Turkowski, K. (1990). Filters for common resampling-tasks. In A. S. Glassner (Ed.), Graphics gems (pp. 147–165). Cambridge: Academic Press.CrossRef

Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2018). Deep image prior. In CVPR. IEEE Computer Society.

Veen, D. V., Jalal, A., Price, E., Vishwanath, S., & Dimakis, A. G. (2018). Compressed sensing with deep image prior and learned regularization. In CoRR.

Zeiler, M. D., Krishnan, D., Taylor, G. W., & Fergus, R. (2010). Deconvolutional networks. In Proceedings of CVPR (pp. 2528–2535). IEEE Computer Society.

Zeyde, R., Elad, M., & Protter, M. (2010). On single image scale-up using sparse-representations. In J. D. Boissonnat, A. Chenin, P. Cohen, C. Gout, T. Lyche, M.-L. Mazure, & L. L. Schumaker (Eds.), Curves and Surfaces (Vol. 6920, pp. 711–730)., Lecture Notes in Computer Science Berlin: Springer.CrossRef

Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In ICLR.

Zhu, S. C., & Mumford, D. (1997). Prior learning and gibbs reaction–diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(11), 1236–1250.CrossRef

Titel: Deep Image Prior
verfasst von: Dmitry Ulyanov
Andrea Vedaldi
Victor Lempitsky
Publikationsdatum: 04.03.2020
Verlag: Springer US
Erschienen in: International Journal of Computer Vision / Ausgabe 7/2020
Print ISSN: 0920-5691
Elektronische ISSN: 1573-1405
DOI: https://doi.org/10.1007/s11263-020-01303-4

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Weitere Artikel der Ausgabe 7/2020

Light Structure from Pin Motion: Geometric Point Light Source Calibration

Exploiting Semantics for Face Image Deblurring

Is There Anything New to Say About SIFT Matching?

Semi-online Multi-people Tracking by Re-identification

Enhanced Balanced Min Cut

The Open Images Dataset V4

Premium Partner