Skip to main content

Über dieses Buch

This unique text/reference presents a detailed review of noise removal for photographs and video. An international selection of expert contributors provide their insights into the fundamental challenges that remain in the field of denoising, examining how to properly model noise in real scenarios, how to tailor denoising algorithms to these models, and how to evaluate the results in a way that is consistent with perceived image quality. The book offers comprehensive coverage from problem formulation to the evaluation of denoising methods, from historical perspectives to state-of-the-art algorithms, and from fast real-time techniques that can be implemented in-camera to powerful and computationally intensive methods for off-line processing.

Topics and features: describes the basic methods for the analysis of signal-dependent and correlated noise, and the key concepts underlying sparsity-based image denoising algorithms; reviews the most successful variational approaches for image reconstruction, and introduces convolutional neural network-based denoising methods; provides an overview of the use of Gaussian priors for patch-based image denoising, and examines the potential of internal denoising; discusses selection and estimation strategies for patch-based video denoising, and explores how noise enters the imaging pipeline; surveys the properties of real camera noise, and outlines a fast approximation of nonlocal means filtering; proposes routes to improving denoising results via indirectly denoising a transform of the image, considering the right noise model and taking into account the perceived quality of the outputs.

This concise and clearly written volume will be of great value to researchers and professionals working in image processing and computer vision. The book will also serve as an accessible reference for advanced undergraduate and graduate students in computer science, applied mathematics, and related fields.

"The relentless quest for higher image resolution, greater ISO sensitivity, faster frame rates and smaller imaging sensors in digital imaging and videography has demanded unprecedented innovation and improvement in noise reduction technologies. This book provides a comprehensive treatment of all aspects of image noise including noise modelling, state of the art noise reduction technologies and visual perception and quantitative evaluation of noise.”

Geoff Woolfe, Former President of The Society for Imaging Science and Technology.

"This book on denoising of photographic images and video is the most comprehensive and up-to-date account of this deep and classic problem of image processing. The progress on its solution is being spectacular. This volume therefore is a must read for all engineers and researchers concerned with image and video quality."

Jean-Michel Morel, Professor at Ecole Normale Supérieure de Cachan, France.



Chapter 1. Modeling and Estimation of Signal-Dependent and Correlated Noise

The additive white Gaussian noise (AWGN) model is ubiquitous in signal processing. This model is often justified by central-limit theorem (CLT) arguments. However, whereas the CLT may support a Gaussian distribution for the random errors, it does not provide any justification for the assumed additivity and whiteness. As a matter of fact, data acquired in real applications can seldom be described with good approximation by the AWGN model, especially because errors are typically correlated and not additive. Failure to model accurately the noise leads to inaccurate analysis, ineffective filtering, and distortion or even failure in the estimation. This chapter provides an introduction to both signal-dependent and correlated noise and to the relevant models and basic methods for the analysis and estimation of these types of noise. Generic one-parameter families of distributions are used as the essential mathematical setting for the observed signals. The distribution families covered as leading examples include Poisson, mixed Poisson–Gaussian, various forms of signal-dependent Gaussian noise (including multiplicative families and approximations of the Poisson family), as well as doubly censored heteroskedastic Gaussian distributions. We also consider various forms of noise correlation, encompassing pixel and readout cross-talk, fixed-pattern noise, column/row noise, etc., as well as related issues like photo-response and gain nonuniformity. The introduced models and methods are applicable to several important imaging scenarios and technologies, such as raw data from digital camera sensors, various types of radiation imaging relevant to security and to biomedical imaging.
Lucio Azzari, Lucas Rodrigues Borges, Alessandro Foi

Chapter 2. Sparsity-Based Denoising of Photographic Images: From Model-Based to Data-Driven

What makes photographic images different from random noise? It has been hypothesized that sparsity is a key factor separating the class of photographic images from noise observations. Accordingly, sparse representations have been widely studied in the literature of image denoising in the past decades. In this chapter, we present a critical review of the most important ideas/insights behind sparsity-based image denoising algorithms. In the first-generation (model-based) approaches, we will highlight the evolution from local wavelet-based image denoising in 1990s–2000s to nonlocal and patch-based image denoising from 2006 to 2015. In the second-generation (data-driven) approaches, we have opted to review several latest advances in the field of image denoising since 2016 such as learning parametric sparse models (for heavy noise removal) and deep learning -based approaches (including deep residue learning). The overarching theme of our review is to provide a unified conceptual understanding of why and how sparsity-based image denoising works—in particular, the evolving role played by models and data. Based on our critical review, we will discuss a few open issues and promising directions for future research.
Xin Li, Weisheng Dong, Guangming Shi

Chapter 3. Image Denoising—Old and New

Image Denoising is among the most fundamental problems in image processing, not only for the sake of improving the image quality, but also as the first proof-of-concept for the development of virtually any new regularization term for inverse problems in imaging. While variational methods have represented the state of the art for several decades, they are recently being challenged by (deep) learning-based approaches. In this chapter, we review some of the most successful variational approaches for image reconstruction and discuss their structural advantages and disadvantages in comparison to learning-based approaches. Furthermore, we present a framework to incorporate deep learning approaches in inverse problem formulations, so as to leverage the descriptive power of deep learning with the flexibility of inverse problems’ solvers. Different algorithmic schemes are derived from replacing the regularizing subproblem of common optimization algorithms by neural networks trained on image denoising. We conclude from several experiments that such techniques are very promising but further studies are needed to understand to what extent and in which settings the power of the data-driven network transfers to a better overall performance.
Michael Moeller, Daniel Cremers

Chapter 4. Convolutional Neural Networks for Image Denoising and Restoration

With the tremendous progress of convolutional neural networks (CNNs), recent years have witnessed a dramatic upsurge of exploiting CNN toward solving image denoising. Compared to traditional model-based methods, CNN enjoys the principal merits of fast inference and good performance. In this chapter, brief survey and discussions are also given to CNN-based denoising methods from the aspects of effectiveness, interpretability, modeling ability, efficiency, flexibility, and applicability. Then, we provide a gentle introduction of CNN-based denoising methods by presenting and answering the following three questions: (i) can we learn a deep CNN for effective image denoising, (ii) can we learn a single CNN for fast and flexible non-blind image denoising, and (iii) can we leverage CNN denoiser prior to versatile image restoration tasks. Finally, we point out that image denoising remains far from solved. The real image noise is much more sophisticated than additive white Gaussian noise, making the existing CNN denoisers generally perform poorly on real noisy images. As a result, it is still very challenging and valuable to study the issues such as noise modeling, acquisition of noisy-clean image pairs and unsupervised CNN learning for real image denoising.
Wangmeng Zuo, Kai Zhang, Lei Zhang

Chapter 5. Gaussian Priors for Image Denoising

This chapter is dedicated to the study of Gaussian priors for patch-based image denoising. In the last 12 years, patch priors have been widely used for image restoration. In a Bayesian framework, such priors on patches can be used for instance to estimate a clean patch from its noisy version, via classical estimators such as the conditional expectation or the maximum a posteriori. As we will recall, in the case of Gaussian white noise, simply assuming Gaussian (or Mixture of Gaussians) priors on patches leads to very simple closed-form expressions for some of these estimators. Nevertheless, the convenience of such models should not prevail over their relevance. For this reason, we also discuss how these models represent patches and what kind of information they encode. The end of the chapter focuses on the different ways in which these models can be learned on real data. This stage is particularly challenging because of the curse of dimensionality. Through these different questions, we compare and connect several denoising methods using this framework.
Julie Delon, Antoine Houdard

Chapter 6. Internal Versus External Denoising—Benefits and Bounds

Image denoising has been a popularly studied problem for several decades in image processing and low-level computer vision communities. Many effective denoising approaches, such as BM3D, utilize spatial redundancy of patches (relatively small, cropped windows) either within a single natural image, or within a large collection of natural images. In this chapter, we summarize our previous finding that “Internal-Denoising” (based on internal noisy patches) can outperform “External Denoising” (based on external clean patches), especially in the presence of high noise levels. We explain this phenomenon in terms of “Patch Signal-to-Noise Ratio” (PatchSNR), an inherent characteristic of a noisy patch that determines its preference of either internal or external denoising. We further experiment with the recent state-of-the-art convolutional residual neural network for Gaussian denoising. We show that it closes the gap on the previously reported external denoising bounds. We further compare its performance to internal local multi-scale Oracle (that has the same receptive field as the network). We show that for patches with low PatchSNR, the network does not manage to reconstruct the best “clean” patch that resides in the network’s receptive field. This suggests that the future challenge of denoising community is to train an image-specific CNN that will exploit local recurrence of patches, without relying on external examples, as was recently successfully done for super-resolution task. Combining such a model with external-based models may push PSNR bounds further up and improve denoising by \(\sim \)1–2 dB, especially for higher noise levels.
Maria Zontak, Michal Irani

Chapter 7. Patch-Based Methods for Video Denoising

Video denoising is an important and open problem, which is less treated than the single-image case. Most image sequence denoising techniques rely on still image denoising algorithms; however, it is possible to take advantage of the redundant information contained in the sequence to improve the denoising results. Most recent algorithms are patch based. These methods have two clearly differentiated steps: select similar patches to a reference one and estimate a noise-free version from this group. We review selection and estimation strategies. In particular, we show that the performance is improved by introducing motion compensation. We use as example a recent video denoising technique inspired by fusion algorithms that use motion compensation by regularized optical flow methods, which permits robust patch comparison in a spatiotemporal volume. The use of principal component analysis ensures the correct preservation of fine texture and details, provided that the noise is Gaussian and white, with known variance. Video acquired by any video camera or mobile phone undergoes several processings from the sensor to the final output. This processing, including at least demosaicking, white balance, gamma correction, filtering, and compression, makes a white noise model unrealistic. Indeed, real video captured in dark environments has a very poor quality, with severe spatially and temporally correlated noise. We discuss a denoising framework including realistic noise estimation, multiscale processing, variance stabilization, and white noise removal algorithms. We illustrate the performance of such a chain with real dark and compressed movie sequences.
A. Buades, J. L. Lisani

Chapter 8. Image and Video Noise: An Industry Perspective

Images and video are increasingly becoming a part of our everyday lives and with this growth, we find an increasing number of industrial and commercial applications of imagery. In this chapter, we will examine the problem of image noise from an industrial and commercial viewpoint. We will consider how noise enters the imaging chain in these settings and how noise is measured and quantified for later removal. We will also discuss standards and standardisation activities that relate to noise measurement in a commercial or industrial setting.
Stuart Perry

Chapter 9. Noise Characteristics and Noise Perception

Denoising is a traditional but still challenging problem in signal processing. To reduce the noise in images and videos captured by a digital sensor receives more and more attention also due to the shrinking size of today’s image sensors and striving for even higher resolutions. A vast amount of research has been conducted to solve the complex problem of separating noise from the true signal. The widespread assumption of additive white Gaussian noise (AWGN) in readily processed image data, however, has led to algorithms that fail on real camera data. This shows how crucial the underlying assumptions and the considered quality metrics are to reach results that are convincing on real data and for real people. In this chapter, we will discuss the properties of real camera noise from sensor data up to human perception. First, we will address how test data is generated and review the noise characteristics of a real single sensor camera. Real camera noise is fundamentally different from AWGN: it is spatially and chromatically correlated, signal dependent, and its probability distribution is not necessarily Gaussian. Second, the challenging aspects of evaluating denoising results based on metrics will be addressed. Instead of rating an algorithm based on a metric like PSNR, which is still the metric the latest benchmarks are based on, a more meaningful metric is required. We show our results of different perception tests that investigated the visibility of spatiotemporal noise as it occurs in digital video. Including these results into a perceptual metric could enable a reliable denoising evaluation with respect to the human perception of visual quality.
Tamara Seybold

Chapter 10. Pull-Push Non-local Means with Guided and Burst Filtering Capabilities

Non-local means filtering (NLM) has cultivated a large amount of work in the computational imaging community due to its ability to use the self-similarity of image patches in order to more accurately filter noisy images. However, non-local means filtering has a computational complexity that is the product of three different factors, namely, \(O(NPK)\), where K is the number of filter kernel taps (e.g., search window size), \(P\) is the number of taps in the patches used for comparison, and \(N\) is number of pixels in the image. We propose a fast approximation of non-local means filtering using the multiscale methodology of the pull-push scattered data interpolation method. By using NLM with a small filter kernel to selectively propagate filtering results and noise variance estimates from fine to coarse scales and back, the process can be used to provide comparable filtering capability to brute force NLM but with algorithmic complexity that is decoupled from the kernel size, K. We demonstrate that its denoising capability is comparable to NLM with much larger filter kernels, but at a fraction of the computational cost. In addition to this, we demonstrate extensions to the approach that allows for guided filtering using a reference image as well as motion compensated multi-image burst denoising. The motion compensation technique is notably efficient and effective in this context since it reuses the multiscale patch comparison computations required by the pull-push NLM algorithm.
John R. Isidoro, Peyman Milanfar

Chapter 11. Three Approaches to Improve Denoising Results that Do Not Involve Developing New Denoising Methods

Image denoising has been a topic extensively investigated over the last three decades and, as repeatedly shown in this book, denoising algorithms have become incredibly good, so much so that many researchers have started questioning the need to further pursue this line of research. In this chapter, we argue that there is indeed room for improvement of denoising results, and we propose three different avenues to explore, none of which requires the development of new denoising methods. First, we describe how it can be better to denoise a transform of the noisy image rather than denoise the noisy image directly. We mention several possible transforms, and an open problem is to find a transform that is optimal for denoising, according to a proper image quality metric. Next, we point out the importance of having a proper noise model for JPEG pictures, so that a variance stabilization transform can be developed that transforms noise in JPEG images into additive white Gaussian noise, enabling existing denoising methods to be properly applied to the JPEG case. Finally, we highlight the fact that while virtually all denoising methods are optimized and validated in terms of the PSNR or SSIM measures, these metrics are not well correlated with perceived image quality, and therefore, it could be best to optimize the parameter values of denoising methods according to subjective testing. A remaining challenge is to develop perceptually based image quality metrics that match observer preference.
Gabriela Ghimpeteanu, Thomas Batard, Stacey Levine, Marcelo Bertalmío


Weitere Informationen

Premium Partner