Smoothing spline regression estimation based on real and artificial data

Abstract

In this article we introduce a smoothing spline estimate for fixed design regression estimation based on real and artificial data, where the artificial data comes from previously undertaken similar experiments. The smoothing spline estimate gives different weights to the real and the artificial data. It is investigated under which conditions the rate of convergence of this estimate is better than the rate of convergence of the ordinary smoothing spline estimate applied to the real data only. The finite sample size performance of the estimate is analyzed using simulated data. The usefulness of the estimate is illustrated by applying it in the context of experimental fatigue tests.

Appendix

1.1 A deterministic lemma

Lemma 5

Let $d\ge 1, t > 0, w_1, \ldots , w_N \in \mathrm{I R}_+, x_1, \ldots , x_N \in \mathrm{I R}^d$ and $z_1, \bar{z}_1, \ldots , z_N, \bar{z}_N \in \mathrm{I R}$. Let $m$ be a function $m: \mathrm{I R}^d \rightarrow \mathrm{I R}$. Let $\mathcal{F}_n$ be a set of functions $f:\mathrm{I R}^d \rightarrow \mathrm{I R}$ and for $f \in \mathcal{F}_n$ let

$$\begin{aligned} pen^2\left( f\right) \ge 0 \end{aligned}$$

be a penalty term. Define

$$\begin{aligned} \bar{m}_n = \arg \min _{f \in \mathcal{F}_n} \left( \sum _{i=1}^N w_i\cdot |f (x_i) - \bar{z}_i|^2 + pen^2\left( f\right) \right) \end{aligned}$$

and

$$\begin{aligned} m_n^* = \arg \min _{f \in \mathcal{F}_n} \left( \sum _{i=1}^N w_i\cdot |f (x_i) - m(x_i) |^2 + pen^2\left( f\right) \right) \end{aligned}$$

and assume that both minima exist. Then

$$\begin{aligned}&\sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i) - m(x_i)|^2 + pen^2\left( \bar{m}_n\right) \nonumber \\&\quad \ge 3 \min _{f \in \mathcal{F}_n} \left( \sum _{i=1}^N w_i\cdot |f (x_i) - m(x_i) |^2 + pen^2\left( f\right) \right) + 128 \sum _{i=1}^N w_i\cdot |z_i - \bar{z}_i|^2 +t^2 \nonumber \\ \end{aligned}$$

(30)

implies

$$\begin{aligned}&\sum _{i=1}^N w_i\cdot \left( \bar{m}_n(x_i)-m_n^*(x_i)\right) \cdot (z_i - m(x_i))\nonumber \\&\quad \ge \frac{1}{24}\left( \sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i) - m_n^*(x_i)|^2 + pen^2\left( \bar{m}_n\right) \right) + \frac{t^2}{6}. \end{aligned}$$

(31)

Proof

The result can be proven by modifying a proof in Kohler and Krzyżak (2012). For the sake of completeness we give nevertheless in the sequel a complete proof.

By definition of the estimate we have

$$\begin{aligned} \sum _{i=1}^N w_i\cdot |\bar{z}_{i}-\bar{m}_n(x_i)|^2 + pen^2\left( \bar{m}_n\right) \le \sum _{i=1}^N w_i\cdot |\bar{z}_{i} - m_n^*(x_i)|^2 + pen^2\left( m_n^*\right) , \end{aligned}$$

hence

$$\begin{aligned}&\sum _{i=1}^N w_i\cdot |\bar{z}_i- m(x_i)|^2 + 2 \sum _{i=1}^N w_i\cdot ( m(x_i)-\bar{m}_n(x_i)) \cdot (\bar{z}_i - m(x_i))\\&\quad + \sum _{i=1}^N w_i\cdot |m(x_i) - \bar{m}_n(x_i)|^2 +pen^2\left( \bar{m}_n\right) \\&\le \sum _{i=1}^N w_i\cdot |\bar{z}_i- m(x_i)|^2 + 2 \sum _{i=1}^N w_i\cdot ( m(x_i)-m^*_{n}(x_i)) \cdot (\bar{z}_i - m(x_i))\\&\quad + \sum _{i=1}^N w_i\cdot |m(x_i) - m^*_{n}(x_i)|^2 +pen^2\left( m^*_{n}\right) , \end{aligned}$$

which implies

$$\begin{aligned}&\sum _{i=1}^N w_i\cdot |m(x_i) - \bar{m}_n(x_i)|^2 \!+\! pen^2\left( \bar{m}_n\right) \!-\! \sum _{i=1}^N w_i\cdot |m(x_i) - m^*_{n}(x_i)|^2 \!-\! pen^2(m^*_{n})\\&\quad \le 2\sum _{i=1}^N w_i \cdot (\bar{z}_i - m(x_i)) \cdot \left( \bar{m}_n(x_i)-m^*_{n}(x_i)\right) \\&\quad = 2 \sum _{i=1}^N w_i \cdot (\bar{z}_i - z_i) \cdot \left( \bar{m}_n(x_i)-m^*_{n}(x_i)\right) \nonumber \\&\qquad + 2\sum _{i=1}^N w_i \cdot (z_i - m(x_i)) \cdot \left( \bar{m}_n(x_i)-m^*_{n}(x_i)\right) \\&\quad =: T_1 + T_2. \end{aligned}$$

We show next that $T_1 \le T_2$. Assume to the contrary that this is not true. Then

$$\begin{aligned}&\sum _{i=1}^N w_i\cdot |m(x_i) - \bar{m}_n(x_i)|^2 + pen^2\left( \bar{m}_n\right) - \sum _{i=1}^N w_i\cdot |m(x_i) - m^*_{n}(x_i)|^2 - pen^2(m^*_{n})\\&\quad < 4 \sum _{i=1}^N w_i \cdot (\bar{z}_i - z_i) \cdot \left( \bar{m}_n(x_i)-m^*_{n}(x_i)\right) \\&\quad \le 4 \cdot \sqrt{ \sum _{i=1}^N w_i \cdot (\bar{z}_i - z_i)^2 } \cdot \sqrt{ \sum _{i=1}^N w_i \cdot \left( \bar{m}_n(x_i)-m^*_{n}(x_i)\right) ^2}\\&\quad \le 4 \cdot \sqrt{ \sum _{i=1}^N w_i \cdot (\bar{z}_i - z_i)^2}\\&\qquad \cdot \sqrt{ 2\sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i)\!-\!m(x_i)|^2 \!+\! 2pen^2\left( \bar{m}_n\right) \!+\! 2\sum _{i=1}^N w_i\cdot |m^*_{n}(x_i)-m(x_i)|^2 \!+\!2 pen^2(m^*_{n}) }. \end{aligned}$$

Using (30) we see that

$$\begin{aligned}&\sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i)-m(x_i)|^2 \!+\! pen^2\left( \bar{m}_n\right) - \sum _{i=1}^N w_i\cdot |m^*_{n}(x_i)-m(x_i)|^2 - pen^2(m^*_{n})\\&\quad \ge \frac{1}{2} \cdot \left( \sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i) - m(x_i)|^2 \!+\! pen^2\left( \bar{m}_n\right) \right) \\&\qquad + \frac{1}{2} \!\cdot \! \left( 3 \left( \sum _{i=1}^N w_i\cdot |m^*_{n}(x_i)\!-\!m(x_i)|^2 \!+\! pen^2(m^*_{n}) \right) \!+\! 128 \!\cdot \! \sum _{i=1}^N w_i \cdot |z_i - \bar{z}_i|^2 \!+\! t^2\right) \\&\qquad - \sum _{i=1}^N w_i\cdot |m^*_{n}(x_i)-m(x_i)|^2 - pen^2(m^*_{n})\\&\quad \ge \frac{1}{2} \cdot \left( \sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i)-m(x_i)|^2 + pen^2\left( \bar{m}_n\right) \right. \nonumber \\&\left. \qquad + \sum _{i=1}^N w_i\cdot |m^*_{n}(x_i)-m(x_i)|^2 + pen^2(m^*_{n}) \right) , \end{aligned}$$

which implies

$$\begin{aligned}&\frac{1}{2} \cdot \sqrt{ \sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i)-m(x_i)|^2 \!+\! pen^2\left( \bar{m}_n\right) \!+\! \sum _{i=1}^N w_i\cdot |m^*_{n}(x_i)-m(x_i)|^2 \!+\! pen^2(m^*_{n})}\\&\quad < 4 \cdot \sqrt{2} \cdot \sqrt{ \sum _{i=1}^N w_i \cdot |z_i - \bar{z}_i|^2 } \end{aligned}$$

i.e.,

$$\begin{aligned}&\sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i)-m(x_i)|^2 + pen^2\left( \bar{m}_n\right) + \sum _{i=1}^N w_i\cdot |m^*_{n}(x_i)-m(x_i)|^2 + pen^2(m^*_{n})\\&\quad < 128 \cdot \sum _{i=1}^N w_i \cdot |z_i - \bar{z}_i|^2 \end{aligned}$$

But this is a contradiction to (30), so we have indeed proved $T_1 \le T_2$. As a consequence we can conclude from (30)

$$\begin{aligned}&4 \sum _{i=1}^N w_i\cdot (\bar{m}_n(x_i)-m_n^*(x_i)) \cdot (z_i - m(x_i))\\&\quad \ge \sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i)-m(x_i)|^2 \!+\! pen^2\left( \bar{m}_n\right) \!-\! \sum _{i=1}^N w_i \!\cdot \! |m^*_{n}(x_i)-m(x_i)|^2 \!-\! pen^2(m^*_{n})\\&\quad \ge \frac{1}{3}\left( \sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i)-m(x_i)|^2 + pen^2\left( \bar{m}_n\right) \right) \\&\qquad +\frac{2}{3}\left( 2\sum _{i=1}^N w_i\cdot |m^*_{n}(x_i)-m(x_i)|^2 + 2pen^2(m^*_{n}) + t^2 \right) \\&\qquad - \sum _{i=1}^N w_i\cdot |m^*_{n}(x_i)-m(x_i)|^2 - pen^2(m^*_{n})\\&\quad = \frac{1}{3}\sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i)-m(x_i)|^2 + \frac{1}{3} pen^2\left( \bar{m}_n\right) \\&\qquad +\frac{1}{3}\sum _{i=1}^N w_i\cdot |m^*_{n}(x_i)-m(x_i)|^2 + \frac{1}{3} pen^2(m^*_{n}) + \frac{2}{3}t^2\\&\quad = \frac{1}{3}\sum _{i=1}^N w_i\cdot |\left( \bar{m}_n(x_i)- m^*_{n}(x_i)\right) - \left( m(x_i)-m^*_{n}(x_i)\right) |^2\\&\qquad + \frac{1}{3} pen^2\left( \bar{m}_n\right) +\frac{1}{3} \sum _{i=1}^N w_i\cdot |m^*_{n}(x_i)-m(x_i)|^2 + \frac{1}{3} pen^2(m^*_{n}) + \frac{2}{3}t^2\\&\quad \ge \frac{1}{6}\sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i)- m^*_{n}(x_i)|^2 - \frac{1}{3}\sum _{i=1}^N w_i\cdot |m(x_i)-m^*_{n}(x_i)|^2\\&\qquad + \frac{1}{3} pen^2\left( \bar{m}_n\right) +\frac{1}{3} \sum _{i=1}^N w_i\cdot |m^*_{n}(x_i)-m(x_i)|^2 + \frac{1}{3} pen^2(m^*_{n}) + \frac{2}{3}t^2\\&\quad \ge \frac{1}{6}\left( \sum _{i=1}^N w_i\cdot |\bar{m}_n(x_i)- m^*_{n}(x_i)|^2 + pen^2\left( \bar{m}_n\right) \right) + \frac{2}{3}t^2. \end{aligned}$$

In the next to last inequality we have used, that $a^2/2-b^2 \le (a-b)^2$ $(a,b \in \mathrm{I R})$ with $a=\bar{m}_n(x_i)- m^*_{n}(x_i)$ and $b=m(x_i)- m^*_{n}(x_i)$.$\square$

Proof (Proof of Lemma 1)

Set $N=n+N_n$, and for $i \in \{1, \ldots , N\}$ choose

$$\begin{aligned} \bar{z}_i = y_i \quad \text{ for } \quad i\le n \quad \text{ and } \quad \bar{z}_i = \hat{y}_i \quad \text{ for } \quad i > n \end{aligned}$$

and

$$\begin{aligned} z_i = y_i \quad \text{ for } \quad i\le n \quad \text{ and } \quad z_i = m(x_i) \quad \text{ for } \quad i > n \end{aligned}$$

in Lemma 5. Then we immediately get the assertion of Lemma 1.$\square$

1.2 Auxiliary results

Lemma 6

Assume that the sub-Gaussion condition (9) ist satisfied. Then there exist constants $c_{28},t_0 \in \mathrm{I R}_{+}$ depending only on $K$ and $\sigma _0$ such that for all $k \in \mathrm{I N}, t,\sigma >0$ with $t>2^{k}\sigma ^{\frac{1}{2k}}t_0$ and with $t\ge 2^kt_0$

$$\begin{aligned} \mathbf{P}\left[ \sup _{f: \Vert f-m\Vert _n\le \sigma } \frac{\frac{1}{n}\sum _{i=1}^{n}\left( f\left( x_i\right) \!-\!m\left( x_i\right) \right) \left( Y_i-m\left( x_i\right) \right) }{n^{-\frac{1}{2}}\Vert f-m\Vert _n^{1-\frac{1}{2k}}\left( 1\!+\!J_k\left( f\right) \!+\!\Vert f\Vert _{\infty }\right) ^{\frac{1}{2k}}}\!>\!t\right] \!\le \! 2 \exp \left( -\frac{c_{28}t^2}{4^k\sigma ^{\frac{1}{k}}}\right) \end{aligned}$$

Proof

The proof is a modification of the proof of Lemma 6.1 in van de Geer (1990).   $\square$

Lemma 7

Let $k \in \mathrm{I N}$ and $c_{29}>0$. Then there exists a constant $c_{30} \in \mathrm{I R}_{+}$ (independent of $k$ and $c_{29}$) such that

$$\begin{aligned} \mathcal{N} _{\infty } \Bigg ( u, \{f: f \in W^k\left( \left[ 0,1\right] \right) , \Vert f\Vert _{\infty } + J_k\left( f\right) \le c_{29}\} \Bigg ) \le \exp \left( c_{30} \left( \frac{c_{29}}{u}\right) ^{\frac{1}{k}}\right) . \end{aligned}$$

Proof

See Birman and Solomjak (1967), Theorem 5.2.$\square$

Lemma 8

Let $H_n$ be a set of functions $h:[0,1] \rightarrow \mathrm{I R}$ and let $R>0$. Suppose that $\sup _{h\in H_n} \Vert h\Vert _n \le R$ and that the sub-Gaussian condition (9) is satisfied. Then for some constant $c_{31}$ depending only on $K$ and $\sigma _0$, and for $\delta >0$ and $\sigma >0$ satisfying $R>\delta /\sigma$ and

$$\begin{aligned} \sqrt{n}\delta \ge 2c_{31} \max \Bigg \{ \int _{ \delta / (8\sigma )}^{R} \log \mathcal{N} ^{1/2}_2 \left( u,H_n,x_1^n\right) du , R\Bigg \} \end{aligned}$$

we have

$$\begin{aligned} \mathbf{P}\left\{ \sup _{h \in H_n} \left| \frac{1}{n} \sum _{i=1}^n h(x_i) \cdot W_i \right| \ge \delta , \frac{1}{n} \sum _{i=1}^n W_i^2 \le \sigma ^2 \right\} \le c_{31}\exp \left( -\frac{n\delta ^2}{4c^2_{31}R^2}\right) . \end{aligned}$$

Proof

See van de Geer (2000), Corollary 8.3. $\square$