Reproducing Kernel Hilbert Spaces in Probability and Statistics

A Reproducing Kernel Hilbert Space (RKHS) is first of all a Hilbert space, that is, the most natural extension of the mathematical model for the actual space where everyday life takes place (the Euclidean space ℝ3). When studying elements of some abstract set S it is convenient to consider them as elements of some other set S′ on which is already defined a structure relevant to the problem to be treated. It can be for instance an order structure, a vector structure, a metric structure or a mixing of algebraic and topological structures. For this we need an “imbedding theorem” or a “representation theorem”. Through this kind of theorem the study of elements of S is transferred to their “representers” in S′ and can be carried out using the structure on S′. For their richness and simplicity Hilbert spaces are introduced as often as possible when a vector structure and an inner product can be exploited. They provide powerful mathematical tools and geometric concepts on which our intuition can rest. The phrase “RKHS method” is generic to name a method based on the embedding of the abstract set S into some RKHS S′.

In Chapter 1, we have studied the relationships between reproducing kernels and positive definite functions. In this chapter, the central result due to Loève is that the class of covariance functions of second order-stochastic processes coincide with the class of positive definite functions. This link has been used to translate some problems related to stochastic processes into functional ones. Such equivalence results are potentially interesting either to use functional methods for solving stochastic problems but also to use stochastic methods for improving algorithms for functional problems, as will be illustrated in Section 4.2, and belong to the large field of interactions between approximation theory and statist ics. Bayesian numerical analysis for example is surveyed in Diaconis (1988) who traces it back to Poincaré (1912). We will come back to this topic later on in Chapter 3. In the 1960’s, Parzen popularized the use of Mercer and Karhunen representation t heorems to write formal solutions to best linear prediction problems for stochastic processes. This lead Wahba in the 1970’s to reveal the spline nature of the solution of some filtering problems.

Reproducing kernels are often found in nonpararnetric curve estimation in connection with the use of spline functions, which were popularized by Wahba in the statistics literature in the 1970s. A brief introduction to the theory of splines is presented in Section 2. Sections 4 and 5 are devoted to the use of splines in nonparametric estimation of density and regression functions. Sections 6 and 7 shortly present the application of reproducing kernels to the problem of shape constraints and unbiasedness. For different purposes kernels (in the sense of Parzen and Rosenblatt) with vanishing moments have been introduced in the literature. In Section 8 we state the link between those kernels (called higher order kernels) and reproducing kernels. Section 9 provides some background on local approximation of functions in view of application to local polynomial smoothing of statistical functionals presented in Section 10. A wide variety of functionals and their derivatives can be treated (density, hazard rate, mean resid ual time, Lorenz curve, spectral density, quantile function, . . .). The examples show the practical interest of kernels of order (m, p) (kernels of order p for estimating derivatives of order m). Their properties and the definition of hierarchies of higher order kernels are further developed in Section 11. Indeed hierarchies of kernels offer large possibilities for optimizing smoothers such as in Cross-Validation techniques, double or multiple kernel procedures, multiparameter kernel estimation, reduction of kernel complexity and many others which are far from being fully investigated. They can also be used as dictionaries of kernels in Support Vector Functional Estimation (see Chapter 5).

Since its foundation by Borel and Lebesgue around the year 1900 the modern theory of measure, generalizing the basic notions of length, area and volume, has become one of the major fields in Pure and Applied Mathematics. In all human activities one collects measurements subject to variability and leading to the classical concepts of Probability and Mathematical Statistics that can modelize “observations” : random variables, samples, point processes. All of them belong to Measure Theory. The study of point processes and more generally of random measures has recently known a large development. It requires sophisticated mathematical tools. The very definition of random measures raises delicate problems, just as the need for a notion of closeness between them.

The theory of reproducing kernel Hilbert spaces interacts with so many subjects in Probability and Mathematical Statistics that it is impossible to deal with all of them in this book. Besides topics that we were willing to develop and to which a chapter is devoted we have selected a few themes gathered in the present chapter.

In many applications, the choice of Hilbert space and norm is governed by context related modeling reasons and one has to face the problem of computing the corresponding reproducing kernel. Symmetrically, it is of interest to characterize the Hilbert space H _K associated with a given kernel K by the Moore-Aronszajn theorem and in particular to give necessary and sufficient conditions for a function to belong to 1iK. Gu and Wahba (1992) say: “T he norm and t he reproducing kernel in a RKHS determine each other uniquely, but like other duals in mathematical structures, the interpretability, and the availability of an explicit form for one part is often at the expense of the same for the other part”. Gu (2000) argues that it can be viewed as an inversion problem: “Just as the inverse J+ of a matrix J can rarely be seen through the entries of J, the “inverse” R(x ₁, x ₂) of J(f) = ∫₀ ¹ f”² .... is not to be per ceived intuitively. For the first question , there is a debate on whether closed form expressions are necessary versus efficient numerical algorithms. Besides the artistic interest one may have for such formulas, t he right choice is certainly dependent on the ultimate use of the kernel. Due to the diversity of spaces and norms, there are few systematic principles for the derivation of a kernel formula. Nevertheless, we also present the study of a number of interesting ad hoc constructions

New reproducing kernels with interesting applications continually appear in the literature. In Section 4 of the present chapter we list major examples for which the kernel and the associated norm and space are explicitly described. They can be used to illustrate aspects of the theory or to practically implement some of the tools presented in the book. In Section 2 and 3 we give examples of effective constructions of kernels. In Section 2 we apply the general characterization theorem proved in Chapter 1. In Section 3 we consider the class of factorizable kernels to which belong markovian kernels defined in Chapter 2, and show how to construct them from functions of one single variable. The present chapter does not end with exercises as others but we would like to encourage the reader to use it as a basis to construct new kernels, norms and spaces with amazing features and applications!