Hotelling tubes, confidence bands and conformal inference

One of my indelible memories of Peter Schmidt was a conversation we had in my kitchen at a party for Midwest Econometrics Group participants in 1993 about the uneasy relationship between statistics and econometrics. “If a statistical tree falls in the forest, but no econometrician sees it,” Peter said matter-of-factly, “then it never happened.” In 1939 Harold Hotelling, arguably one of the most eminent statisticians and econometricians of the twentieth century witnessed such an event and wrote about in Hotelling (1939). The paper inspired Hermann Weyl to write a highly influential paper, Weyl (1939) generalizing it. Hotelling’s idea has attracted a small coterie of admirers in statistics, but it is fair to say that it remains almost unknown in econometrics.

My quixotic aim in this paper is to rescue Hotelling’s idea from econometric obscurity. I will begin by describing a simple setting in which the idea can be employed to construct a confidence interval for a scalar parameter that enters awkwardly in a standard regression problem. Then, I will describe how it can be used to construct uniform confidence bands for nonparametric regression using penalty methods, and finally I will compare performance with confidence bands constructed with recently developed methods of conformal inference.

Consider the nonlinear regression modelwhere \(\alpha , \beta , \tau \) are unknown parameters, \(\lambda _i(\cdot )\) are known functions and \(\varepsilon _i\sim {{\mathcal {N}}}(0, \sigma ^2)\). For the sake of concreteness, we might interpret \(\lambda _i (\tau )\) as a Box-Cox transformation of another covariate, say \((z_i^\tau -1)/\tau \). We would like to test \(H_0: \beta =0\). Under the null, the Box-Cox parameter \(\tau \) is not identified, so we need to consider strategies that properly account for this.¹

By the familiar (Frisch and Waugh 1933) trickery, we can eliminate the \(\alpha \) effect.² Redefining the notation and assuming for convenience that \(\sigma ^2 = 1\), we are left with the likelihood ratio statisticNow, denoting the n-vectors, \(Y = (Y_i)\), \(\lambda = (\lambda _i)\), and the Euclidean norm by \(\Vert \cdot \Vert \), \({\hat{\beta _\tau }} = Y^\top \lambda (\tau )/ \Vert \lambda (\tau ) \Vert ^2\) so we can rewrite,Now \( U =Y/ \Vert Y \Vert \) is uniformly distributed on the sphere \(S^{n-1}\) and \(\gamma (\tau ) = \lambda (\tau )/ \Vert \lambda (\tau ) \Vert \) is a curve in \(S^{n-1}\). Thus, the test rejects when \(W=\sup _\tau \gamma (\tau )^\top U \) exceeds some value \(w=\cos \theta \) which is equivalent toNote that the original definition of L is such that we reject for small values, so \(L<c, \) implies we reject for \(\sup _\tau \gamma (\tau )^\top U > w = \cos \theta \) for some critical value of \(\theta \). This is illustrated in Fig. 1 of Johansen and Johnstone (1990), reproduced here as Fig. 1. They call this the “angular or geodesic radius \(\theta \) about \(\gamma \):”So when the distance \(d(u,\gamma )\) is small, U falls inside tube, and we reject. This may seem a bit counter-intuitive, but is nonetheless correct. There are probably many ways to it sound more intuitive. Here is one possibility. Since it all boils down to a cosine, that is the simple correlation between \(\lambda (\tau )\) and Y, we want to reject \(H_0: \beta =0\) if this correlation/cosine is too large, but Y’s that make it too large are the \(Y's\) that fall inside the tube.

So how do we compute the critical w or equivalently the critical \(\theta \)? Since \(W>w \equiv \cos \theta \) is equivalent to U being in the tube, we need the volume of the tube. Let \(| \gamma |\) denote the length of the arc \(\gamma (\tau )\) on the sphere. This can be approximated by the finite difference formula,and the \(\tau \)’s are chosen on some relatively fine grid of m points. Note that in the finite difference approximation the \(\tau _i - \tau _{i-1}\) that would normally appear in the denominator of the difference quotient inside the norm expression cancels with the contribution of the \(\text {d}\tau \).

We ignore pathological complications involving self-intersections of the curve, \(\gamma \). This follows from a result of Hotelling (1939), as does the next theorem.

Heuristically, the formula is,Recall that the volume of the unit ball in dimension d is \(V=\pi ^{d/2}/\Gamma ((d+2)/2)\). When \(\theta \) is larger, or \(\gamma \) is twisty, then the tube may intersect itself and the formula would need some refinement. Figure 2 is a crude attempt to depict tube on the 2-sphere, those with enhanced geometric imagination may try to visualize a three dimensional tube on the 3-sphere embedded in 4-space.

When the curve is not closed, then it needs “caps” on each end. These caps are given bywhere \(w_{d-2} = 2\pi ^{(d-1)/2} /\Gamma ((d-1)/2)\) is the \((d-2)\)-volume of \(S^{d-2}\). Note that the volume of the sphere, \(V(S^{d-1}) = 2 \pi ^{d/2} /\Gamma (d/2)\), is not the same as the volume of the ball. Note also that \((1-z^2)^{1/2}\) is again the radius and integrating out the \(r^{d-3}\) yields a \(d-2\) dimensional volume. A useful reference for this sort of geometry is Kendall (1961).

How do we get from (2) to (1)? Recall that U is uniform on the \((d-1)\) sphere so we need to divide by the volume of that sphere to evaluate the probability of being in the tube, so for closed curves,To include caps, we also need to divide by the volume of the sphere. Note thatChanging variables \(x\rightarrow y^2\), we haveIt remains to show that \(B^{-1} = w_{d-2}/\text{V(sphere) }\), which follows after a little simplification and recalling that \(\Gamma (1/2) = \sqrt{\pi }\).

To check how the Hotelling tube procedure performs in moderate sample sizes, Table 1 reports results of a small simulation experiment. Data are generated with iid \(x_i\) standard log-normal andThree values of \(\tau \) are considered \(\tau \in \{ -0.5, 0, 0.5 \}\). Local alternatives, \(\beta _n = \beta _0/\sqrt{n}\), are considered with \(\beta _0 \in \{ 0, 1, 2\}\). The nominal level of the Hotelling test is taken to be 0.05. and 1000 replications of the experiment are made for each parametric setting. When \(\beta = 0\) so the null is true, the test delivers quite accurate size for all of the sample sizes considered, and power is respectable when \(\beta \) deviates from zero.

Consider the series expansion modelwith \(\varepsilon _i\sim {{\mathcal {N}}}(0, \sigma ^2)\) as before and \(t\in I \subset {\mathbb {R}}\). Our objective is to find a positive c such thatuniformly in \(\beta , \sigma .\) Johansen and Johnstone (1990) write this as, \( P_{\beta ,\sigma , \Sigma } (T < c\sigma ) \) where

Now consider \(X\sim {{\mathcal {N}}}(\xi , \Sigma )\), so X plays the role of \({\hat{\beta }}\) and \(\xi \) of \(\beta .\) We’d like to make a confidence statement about \(\{a^\top \xi |a\in C\}\) and C is some sort of “curve.” So now we write,We want the distribution of T, so we can obtain the confidence setwhere \(P_{\xi , \Sigma } (T< c_{1-\varepsilon }) = 1-\varepsilon .\) Write \(T=RW\) where,andNow to put things back into the earlier framework of \(\gamma \) and U we set,So as before, \(\gamma =\gamma (C) \subset S^{d-1}\), and U is uniform on \(S^{d-1}\). R and W do not depend on \(\xi , \Sigma \) or they do, but only via \(\gamma \). \(R^2\) is independent of W and \(R^2\sim \chi _d^2\) so,The random variable W has the same form as in the simple example so,(Naiman 1986) bounds this probability by,and Knowles (1987) suggests ignoring the \(b_\gamma <1\) constraint and then integrates the bound to obtain,This integration may appear somewhat miraculous, but does actually work out provided that one carefully observes the \({{\mathcal {P}}}(R\in \text {d}r)\) term. Since \(R^2\sim \chi _d^2\), letting F denote the distribution function of \(\chi _d^2\), we have,so the corresponding density of R isOnce one has this bound then various other things fall into place. For example,(Johansen and Johnstone 1990) give further details on the accuracy of the bounds and applications.

In Koenker (2011), I have described a general approach to estimation and inference for additive nonparametric quantile regression models of the form,The components \(g = (g_1, \cdots , g_J)\) can be univariate or bivariate. Their smoothness can be controlled by penalizing total variation of the functions themselves or their gradients. Estimation is carried out by solving the linear program,where \(\rho _\tau (u) = u (\tau - \mathbb {1}(u < 0))\) is the usual quantile objective function, \(\Vert \theta _0 \Vert _1 = \sum _{k=1}^{\scriptscriptstyle K} |\theta _{0k}|\) and \(\bigvee (\nabla g_j)\) denotes the total variation of the derivative or gradient of the function g. Recall that for g with absolutely continuous derivative \(g'\) we can express the total variation of \(g':{\mathcal R} \rightarrow {{\mathcal {R}}}\) aswhile for \(g:{\mathbb {R}}^2 \rightarrow {\mathbb {R}}\) with absolutely continuous gradient,where \(\nabla ^2g(z)\) denotes the Hessian of g and \(\Vert \cdot \Vert \) denotes the Hilbert–Schmidt norm for matrices. In contrast, total variation penalization of the component functions themselves yields piecewise constant solutions.

Adapting the Hotelling tube idea to construct uniform confidence bands for these components is also described in Koenker (2011), as is selection of the smoothing parameters \(\lambda _j, j = 0, 1, \cdots J\). It should be stressed that all of this machinery relies on the validity of Gaussian approximations for the fitted parameters and estimated functions and is conditional on selected tuning parameters. This is in accord with a large strand of earlier literature including (Wahba 1983; Nychka 1983), and Krivobokova et al. (2010); however, there are inevitable questions that can be raised about both aspects. To explore this, we consider some recent proposals for strengthening coverage guarantees based on conformal inference in the next section.

Conformal prediction, and conformal inference more generally, has grown out of work by Vladimir Volk and colleagues, see, e.g., Shafer and Vovk (2008) for an overview. It has emerged as an essential tool in uncertainty quantification throughout statistics and machine learning. An essential feature of the conformal inference approach in regression is a sample splitting device that allows one to adjust a confidence band constructed with training data based on its performance on a validation sample. Strong finite sample performance guarantees can be proven based on seemingly rather weak exchangeability assumptions. In regression settings, early work presumed conventional iid error structure when constructing the initial bands from the training data; however, (Romano et al. 2019) noted that in more heterogeneous settings narrower bands could be constructed using quantile regression methods. This approach has been further developed in Lei and Candès (2022). In high-dimensional regression, this typically would involve some form of random forest or neural network model for the initial bands, but the same methods can be used in simpler models like the additive models described above.

Construction of conformal prediction bands for additive quantile regression models can be described briefly as follows:

Note that the conformal adjustment of the initial band can make it wider or narrower. When \(Q<0\), it indicates that the validation sample fell well inside the initial band indicating that it is safe to shrink the width of the initial band.

There are several potential difficulties with the foregoing recipe.

We conclude this section by illustrating the use of the conformal method in an artificial data example taken from Romano et al. (2019). Simulated data are generated as,There are 7000 observations plotted in grey. The Poisson contribution to the response produces a banded structure to the scatterplot with pronounced heteroscedasticity. There are a small number of extreme outliers many of which lie outside the frame of the figure; such outliers are harmless since we are estimating conditional quantile functions. Penalizing total variation of \(g'\) yields a piecewise linear fit that does not fit the scatter as well as the piecewise constant estimate obtained by penalizing the total variation of g itself. It is striking here that the conformal adjustment in both figures is almost imperceptible. Thus, if interest focuses on prediction intervals for the response, the initial estimates provided by the penalized quantile regression estimates are fine, even though they are based on only half the original sample.

Prediction bands for Y are fine as far as they go, but what if we wanted confidence bands for the conditional quantile functions? Some might argue, e.g., Geiser (1993); Clarke and Clarke (2018), that it is pointless to predict quantities that can never be observed, but I subscribe to the principle: every decent estimate deserves a standard error. Figure 5 illustrates confidence bands for the lower, \(\tau = 0.05\) and upper, \(\tau = 0.95\) conditional quantile functions as estimated using penalization of \(g'\). The dark grey bands are the pointwise bands, while the lighter grey bands are those based on the Hotelling tube approach. Note that the bands for the 0.05 estimate are extremely narrow since the data are very concentrated in this region, so the \(\tau = 0.05\) conditional quantile is very precisely estimated.

The large literature in econometrics about stochastic frontier models is mostly concerned with parametric models of the tail behavior of the response “near the production frontier.” Nonparametric quantile regression offers yet another perspective on estimating such models. It would be extremely foolish to make any claims for alternative methodology described here on the basis of the flimsy evidence offered, let me conclude simply by saying that it might be worthy of further consideration.