Incorporating Dependencies in Spectral Kernels for Gaussian Processes

Gaussian processes (GPs) provide regression models where a posterior distribution over the unknown function is maintained as evidence is accumulated. This allows GPs to learn complex functions when a large amount of evidence is available, and it makes them robust against overfitting in the presence of little evidence. GPs can model a large class of phenomena through the choice of the kernel, which characterizes one’s assumption on how the unknown function autocovaries [17, 18]. The choice of the kernel is a core aspect of a GP design, since the posterior distribution can significantly vary for different kernels. In particular, in [24] a flexible kernel called Spectral Mixture (SM) was defined, by modelings the kernel’s spectrum with a mixture of Gaussians. An SM kernel can be represented by a sum of components, and can be derived from Bochner’s theorem as the inverse Fourier Transform (FT) of its corresponding spectral density. SM kernels assume mutually independence of its components [24‐26].

Here we propose a generalization of SM kernels that explicitly incorporates dependencies between components. We use cross convolution to model dependencies between components, and derive a new kernel called Generalized Convolution Spectral Mixture (GCSM) kernel. The number of hyper-parameters remains equal to that of SM, and there is no increase in computational complexity. A stochastic variational inference technique is used to perform scalable inference. In the proposed framework, GCSM without cross components (that is, by only considering auto-convolution of base components) reduces to the SM kernel.

We assess the performance of GCSM kernels through extensive experiments on real-life datasets. The results show that GCSM is able to capture dependence structure in time series and multi-dimensional data containing correlated patterns. Furthermore, we show the benefits of the proposed kernel for reducing uncertainty, overestimation and underestimation in extrapolation tasks. Our main contributions can be summarized as follows:

The remainder of this paper is organized as follows. We start by giving a background on GPs, SM kernels, and we briefly describe related work. Next, we introduce the GCSM kernel, and discuss the differences between the GCSM and SM kernels. Then we describe the experimental setting and show results on synthetic and real-world datasets. We conclude with a summary and discussion on future work.

A GP is any distribution over functions such that any finite set of function values has a joint Gaussian distribution. A GP model, before conditioning on the data, is completely specified by its mean function \(m({{\mathbf {x}}})= \mathbb {E}(f({\mathbf {x}}))\) and its covariance function (also called kernel) \(k({{\mathbf {x}}}, {{{\mathbf {x}}}{'}})= {{\text {cov}}}(f({{\mathbf {x}}}), f({{{\mathbf {x}}}{'}}))\) for input vectors \({{\mathbf {x}},{\mathbf {x}}'}\in {{\mathbb {R}}}^{P}\). It is common practice to assume that the mean function is simply zero everywhere, since uncertainty about the mean function can be taken into account by adding an extra term to the kernel (cf. e.g. [18]).

The kernel induces a positive definite covariance matrix \(K=k(X, X)\) of the training locations set X. For a regression task [18], by choosing a kernel and inferring its hyper-parameters \(\varTheta \), we can predict the unknown function value \(\tilde{y}^{*}\) and its variance \(\mathbb {V}[\tilde{y}^{*}]\) (the uncertainty) for a test point \({{\mathbf {x}}}^{*}\) as follows: where \(k^{**}=k({{\mathbf {x}}}^{*}, {{\mathbf {x}}}^{*})\), \({{\mathbf {k}}^{*}}^{\top }\) is the vector of covariances between \({{\mathbf {x}}}^{*}\) and X, and \({{\mathbf {y}}}\) are the observed values at training locations in X. The hyper-parameters can be optimized by minimizing the Negative Log Marginal Likelihood (NLML) \(-\log \ p({{\mathbf {y}}}|{{\mathbf {x}}},{\varTheta })\). Smoothness and generalization properties of GPs depend on the kernel function and its hyper-parameters \({\varTheta }\) [18]. In particular, the SM kernel [26], here denoted by \(k_\text {SM}\), is derived by modeling the empirical spectral density as a Gaussian mixture, using Bochner’s Theorem [2, 22], resulting in the following kernel: where \(\tau ={{\mathbf {x}}}-{{{\mathbf {x}}}{'}}\), Q denotes the number of components, \(k_{{\text {SM}}i}\) is the i-th component, P denotes the input dimension, and \(w_i\), \({\varvec{\mu }}_{i}=\left[ \mu _{i,1},...,\mu _{i,P}\right] \), and \({{\varSigma }}_{i}=\text {diag}\left( \left[ \sigma _{i,1}^{2},...,\sigma _{i,P}^{2}\right] \right) \) are the weight, mean, and variance of the i-th component in the frequency domain, respectively. The variance \({\sigma }_{i}^{2}\) can be thought of as an inverse length-scale, \(\mu _{i}\) as a frequency, and \(w_i\) as a contribution. For SM kernel, we have \({{\hat{k}}_{{\text {SM}}i}}({{\mathbf {s}}}) = [{\varphi }_{{\text {SM}}i}({{\mathbf {s}}}) + {\varphi }_{{\text {SM}}i}(-{{\mathbf {s}}})]/2\) where \({\varphi }_{{\text {SM}}i}({{\mathbf {s}}})={{\mathcal {N}}}({{\mathbf {s}}};{\varvec{\mu }}_{i},{{\varSigma }}_{i})\) is a symmetrized scale-location Gaussian in the frequency domain.

The SM kernel does not consider dependencies between components, because it is a linear combination of \(\{k_{{\text {SM}}i}\}_{i=1}^{Q}\) (see Eq. 3). Therefore its underlying assumption is that such components are mutually independent. One should not confuse the spectral mixture components that make up the spectral density of the SM kernel with the base components of the Fourier Transform (FT): (1) FT components are periodic trigonometric functions, such as sine and cosine functions, while SM kernel components are quasi-periodic Gaussian functions; (2) FT components are orthogonal (i.e. the product of an arbitrary pair of Fourier series components is zero) while the product of two arbitrary SM components is not necessarily equal to zero; (3) the SM component in the frequency domain is a Gaussian function covering wide frequency range while an FT component is just a sharp peak at a single frequency, which is covered by multiple SM components.

Various kernel functions have been proposed [18], such as Squared Exponential (SE), Periodic (PER), and general Matérn (MA). Recently, Spectral Mixture (SM) kernels have been proposed in [24]. Additive GPs have been proposed in [4], a GP model whose kernel implicitly sums over all possible products of one-dimensional base kernels. Extensions of these kernels include the spectral mixture product kernel (SMP) [25] \(k_\text {SMP}(\tau |\varTheta )=\prod _{p=1}^{P}k_\text {SM}(\tau _{p}|\varTheta _{p})\), which uses multi-dimensional SM kernels, and extends the application scope of SM kernels to image data and spatial time data. Other interesting families of kernels include non-stationary kernels [7, 10, 19, 21], which are capable to learn input-dependent covariances between inputs. All these mentioned kernels do not consider dependencies between components. To the best of our knowledge, our proposed kernel is the first attempt to explicitly model dependencies between components.

The problem of expressing structure present in the data being modeled with kernels has been investigated also in the context of kernel composition. For instance, in [3] a framework was introduced for composing kernel structures. A space of kernel structures is defined compositionally in terms of sums and products of a small number of base kernel structures. Then an automatic search over this space of kernel structures is performed using marginal likelihood as search criterion. Although composing kernels allows one to produce kernels combining several high-level properties, they depend on the choice of base kernel families, composition operators, and search strategy. Instead, here we directly enhance SM kernels by incorporating dependency between components.

Since the SM kernel is additive, any \(f \sim \mathcal {GP}(0,k_{\text {SM}})\) can be expressed as where each \(f_i \sim \mathcal {GP}(0,w_i k_{{\text {SM}}i})\) is drawn from a GP with kernel \(w_i k_{{\text {SM}}i}\). With a slight abuse of notation we denote by \({{\varvec{f}}}_i\) the function values at training locations X, and by \({{\varvec{f}}}_{i}^{*}\) the function values at some set of query locations \(X^*\).

From the additivity of the SM kernel it follows that the \(f_i\)’s are a priori independent. Then, by using the formula for Gaussian conditionals we can give the conditional distribution of a GP-distributed function \({{\varvec{f}}}_{i}^{*}\) conditioned on its sum with another GP-distributed function \({{\varvec{f}}}_{j}\) : where \({{\varvec{f}}}_{i+j}={{\varvec{f}}}_{i}+{{\varvec{f}}}_{j}\) and \(K_{i+j}=K_{i}+K_{j}\). The reader is referred to [3] (Sect. 2.4.5) for the derivation of these results. The Gaussian conditionals express the model’s posterior uncertainty about the different components of the signal, integrating over the possible configurations of the other components.

In particular, we have: In general \(\mathbb {V}({{\varvec{f}}}_{i}^{*}|{{\varvec{f}}}_{i})\ne {\mathbb {V}({{\varvec{f}}}_{i}^{*}|{{\varvec{f}}}_{i}, {{\varvec{f}}}_{j})}\) when dependencies between components are present. We can also compute the posterior covariance between the height of any two functions, conditioned on their sum [3]: We define posterior correlation \(\rho _{ij}^{*}\) as normalized posterior covariance: We can use \(\rho _{ij}^{*}\ne {0}\) as indicator of statistical dependence between components i and j. In our experiments, we will use the normalized posterior covariance to illustrate the presence of dependencies between components in SM kernels for GPs.

We propose to generalize SM kernels by incorporating cross component terms. To this aim we use versions of the seminal Convolution theorem, which states that under suitable conditions the Fourier transform of a convolution of two signals is the pointwise product of their Fourier transforms. In particular, convolution in the time domain equals point-wise multiplication in the frequency domain. The construction of our kernel relies on the fact that any stationary kernel \(k(\mathbf {x}, \mathbf {x}')\) can be represented as a convolution form on \({{\mathbb {R}}}^P\) (see e.g. [5, 6, 13]) By applying a Fourier transformation to the above general convolution form of the kernel we obtain \(\hat{k}({\mathbf {s}})=(\hat{g}({\mathbf {s}}))^2 \) in the frequency domain. For each weighted component \(w_{i}{k}_{\text {SM}i}(\tau )\) in the SM kernel, we can define the function \(\hat{g}_{\text {SM}i}({\mathbf {s}})\) as which is the basis function of the i-th weighted spectral density. We use cross-correlation, which is similar in nature to the convolution of two functions. The cross-correlation of functions \(f(\tau )\) and \(g(\tau )\) is equivalent to the convolution of \(\overline{f}(-\tau )\) and \(g(\tau ) \) [1]. we have that the cross-correlation between two components \(f_{i}\sim {{\mathcal {GP}}}(0, w_{i}k_{{\text {SM}}i})\) and \(f_{j}\sim {{\mathcal {GP}}}(0, w_{j}k_{{\text {SM}}j})\) is as where \({{{\mathcal {F}}}_{s\rightarrow \tau }^{-1}}\), \(\star \), and \(\overline{(-)}\) denote the inverse FT, the cross-correlation operator, and the complex conjugate operator, respectively. Here \({\varphi }_{{\text {SM}}i}({{\mathbf {s}}})={{\mathcal {N}}}({{\mathbf {s}}};{\varvec{\mu }}_{i},{{\varSigma }}_{i})\) is a symmetrized scale-location Gaussian in the frequency domain (\({\varphi }_{{\text {SM}}i}({{\mathbf {s}}})=\overline{{\varphi }_{{\text {SM}}i}}({{\mathbf {s}}})\)). The product of Gaussians \({\varphi }_{{\text {SM}}i}({{\mathbf {s}}})\) and \({\varphi }_{{\text {SM}}j}({{\mathbf {s}}})\) is also a Gaussian. Therefore, the cross-correlation term in the frequency domain has also a Gaussian form and must be greater than zero, which implies the presence of dependencies between \(f_{i}\) and \(f_{j}\).

The cross-correlation term \({{k}_{\text {GCSM}}^{{i}\times {j}}(\tau )}\) of our new kernel, obtained as cross-correlation of the i-th and j-th base components in SM, corresponds to the cross spectral density term in the frequency domain. From (12) and (14) we obtain The parameters for the cross spectral density term \({{\hat{k}_{\text {GCSM}}^{{i}\times {j}}({\mathbf {s}})}}\) corresponding to the cross convolution component \({{k}_{\text {GCSM}}^{{i}\times {j}}(\tau )}\) are:

Parameters \({\varvec{\mu }}_{ij}\) and \(\varSigma _{ij}\) can be interpreted as frequency and inverse length-scale of the cross component \(k_{\text {GCSM}}^{{i}\times {j}}(\tau )\), respectively. Cross amplitude \(a_{ij}\) is a normalization constant which does not depend on \({\mathbf {s}}\).

Observe that when \(\hat{g}_{{\text {SM}i}}({\mathbf {s}})\) is equal to \(\hat{g}_{{\text {SM}j}}({\mathbf {s}})\), \(w_{ij}\) \({a}_{ij}\), \({\varvec{\mu }}_{ij}\), and \({\varSigma }_{ij}\) reduce to \(w_{i}\), 1, \({\varvec{\mu }}_{i}\), and \({\varSigma }_{i}\), respectively. In this case, the cross spectral density \(\hat{k}_{\text {GCSM}}^{{i}\times {j}}({\mathbf {s}})\) is equal to \(\hat{k}_{\text {SM}{i}}({\mathbf {s}})\). We can observe that the closer the frequencies \({\varvec{\mu }}_i\) and \({\varvec{\mu }}_j\) are and as closer the scales \(\varSigma _{i}\) and \(\varSigma _{j}\) between components i and j in the SM kernel are, the higher the cross convolution components contribution in GCSM will be.

Using the inverse FT, by the distributivity of the convolution operator and by the symmetry of the spectral density, we can obtain the GCSM kernel with Q (auto-convolution) components as: where \(c_{ij}={w_{ij}}{a_{ij}}\) is the cross contribution incorporating cross weight and cross amplitude to quantify the dependency between components in the GCSM kernel. The proof that GCSM is positive semi-definite is given in the Appendix. The auto-convolution cross-terms in GCSM correspond to the components in SM since \({k}_{\text {GCSM}}^{{i}\times {i}}(\tau )=k_{{\text {SM}}i}(\tau )\). It is a mixture of periodic cosine kernels and their dependencies, weighted by exponential weights.

Figure 1 illustrates the difference between SM and GCSM, where each connection represents a convolution component of the kernel. SM is an auto-convolution spectral mixture kernel that ignores the cross-correlation between base components. The figure also shows that SM is a special case of GCSM since the latter involves both cross convolution and auto-convolution of base components. In GCSM, dependencies are explicitly modeled and quantified. In the experiment illustrated in Fig. 2, SM and GCSM have the same initial parameters the same noise term. The observations are sampled from a \({\mathcal {GP}}(0, K_{\text {SM}}+K_{\text {GCSM}})\). From Fig. 2 we can observe clear differences (in terms of amplitude, peak, and trend from SM) for the kernel functions (SM: top, in dashed red; GCSM: bottom, in dashed blue). For the corresponding spectral densities, the dependence (in magenta) modeled by GCSM is also a Gaussian in the frequency domain, which yields a spectral mixture with different magnitude. The posterior distribution and sampling are obtained from GCSM and SM conditioned on six observations (black crosses). One can observe that the predictive distribution of GCSM has a tighter confidence interval (in blue shadow) than SM (in red shadow).

Exact inference for GPs is prohibitively slow for more than a few thousand datapoints, as it involves inverting the covariance matrix \((K+\sigma ^{2}_{n}I)^{-1}\) and computing the determinant of the covariance \(|K+\sigma ^{2}_{n}I|\). This issues are addressed by covariance matrix approximation [16, 20, 23] and inference approximation [8, 9].

Here we employ stochastic variational inference (SVI) which provides a generalized framework for combining inducing points \({\mathbf {u}}\) and variational inference yielding impressive efficiency and precision. Specifically, SVI approximates the true GP posterior with a GP conditioned on a small set of inducing points \({\mathbf {u}}\), which as a set of global variables summarise the training data and are used to perform variational inference. The variational distribution \(P({\mathbf {u}})={\mathcal {N}}({\mathbf {u}}; {\varvec{\mu }}_{{\mathbf {u}}}, \varSigma _{{\mathbf {u}}})\) gives a variational lower bound \({{\mathcal {L}}_{3}({\mathbf {u}}; {\varvec{\mu }}_{{\mathbf {u}}}, \varSigma _{{\mathbf {u}}})}\), also called Evidence Lower Bound (ELBO) of the quantity \(p({\mathbf {y}}|X)\). From [9], the variational distribution \({\mathcal {N}}({\mathbf {u}}; {\varvec{\mu }}_{{\mathbf {u}}}, \varSigma _{{\mathbf {u}}})\) contains all the information in the posterior approximation, which represents the distribution on function values at the inducing points \({\mathbf {u}}\). From \(\frac{\partial {{\mathcal {L}}_{3}}}{\partial {\varvec{\mu }}_{{\mathbf {u}}}}=0\) and \(\frac{\partial {{\mathcal {L}}_{3}}}{\partial \varSigma _{{\mathbf {u}}}}=0\), we can obtain an optimal solution of the variational distribution. The posterior distribution of testing data can be written as where \({\mathbf {k}}^{*}_{{\mathbf {u}}}\) is the GCSM covariance vector between \({\mathbf {u}}\) and test point \({\mathbf {x}}^{*}\). The complexity of SVI is \({\mathcal {O}}(m^{3})\) where m is the number of inducing points.

We comparatively assess the performance of GCSM on real-world datasets. Three of these datasets have been used in the literature of GP methods. The other is a relative new dataset which we use to illustrate the capability of GPs with the considered kernels to model irregular long term increasing trends. We use Mean Squared Error (\({\mathrm {MSE} ={\frac{1}{n}\sum _{i=1}^{n}\big (y_{i}-\tilde{y}_{i}\big )^{2}}}\)) as the performance metric for all tasks. We used the 95% confidence interval (instead of, e.g., error bar) to quantify uncertainty (see Eq. (2)). In addition to these performance metrics, we also consider the posterior correlation \(\rho _{ij}^{*}\) (see Eq. (10)) to illustrate the underlying dependency between SM components. Moreover, to illustrate the dependency between components captured by the cross-components in our GCSM kernel, we use the normalized cross-correlation term: We call \(\gamma _{ij}\) learned dependency between component i and j. Note that \(\gamma _{ij}=1\) when \(i=j\). In our experiments we will analyze dependency between components in SM kernel for GPs as expressed by the posterior covariance, and dependency modeled by GCSM kernels for GPs as expressed by \(\gamma _{ij}\)’s. We compare GCSM with ordinary SM for prediction tasks on four real-life datasets: monthly average atmospheric CO\(_2\) concentrations [12, 18], monthly ozone concentrations, air revenue passenger miles, and the larger multidimensional alabone dataset.

As baselines for comparison we consider the popular kernels implemented in the GPML toolbox [18]: linear with bias (LIN), SE, polynomial (Poly), PER, rational quadratic (RQ), MA, Gabor, fractional Brownian motion covariance (FBM), underdamped linear Langevin process covariance (ULL), neural network (NN) and SM kernels. For the considered multidimensional dataset, we use automatic relevance determination (ARD) for other kernels to remove irrelevant input. FBM and ULL kernels are only available for time series type of data, thus they are not applied to this dataset. We use the GPML toolbox [17] and GPflow [14] for ordinary and scalable inference, respectively. For GCSM, we calculate the gradient of the parameters using an analytical derivative technique. In all experiments we use the hyper-parameter initialization previously described for SM and GCSM kenels.

We proposed the generalized convolution spectral mixture (GCSM) kernel, a generalization of SM kernels with an expressive closed form to modeling dependencies between components using cross convolution in the frequency domain.

Experiments on real-life datasets indicate that the proposed kernel, when used in GPs, can identify and model the complex structure of the data and be used to perform long-term trends forecasting. Although here we do not focus on non-stationary kernels, GCSM can be transformed into a non-stationary GCSM, through parameterizing weights \(w_{i}(x)\), means \(\mu _{i}(x)\), and \(\sigma _{i}(x)\) as kernel matrices by means of a Gaussian function. Future work includes the investigation of more generalized non-stationary GCSM.

An issue that remains to be investigated is efficient inference. This is a core issue in GP methods which needs to be addressed also for GPs with GCSM kernels. Lev́y process priors as proposed in [11] present a promising approach for tackling this problem, by regularizing spectral mixture for automatic selection of the number of components and pruning of unnecessary components.