Time Series: Theory and Methods

In this chapter we introduce some basic ideas of time series analysis and stochastic processes. Of particular importance are the concepts of stationarity and the autocovariance and sample autocovariance functions. Some standard techniques are described for the estimation and removal of trend and seasonality (of known period) from an observed series. These are illustrated with reference to the data sets in Section 1.1. Most of the topics covered in this chapter will be developed more fully in later sections of the book. The reader who is not already familiar with random vectors and multivariate analysis should first read Section 1.6 where a concise account of the required background is given. Notice our convention that an n-dimensional random vector is assumed (unless specified otherwise) to be a column vector X = (X ₁, X ₂, ... , X _n )′ of random variables. If S is an arbitrary set then we shall use the notation S ⁿ to denote both the set of n-component column vectors with components in S and the set of n-component row vectors with components in S.

Although it is possible to study time series analysis without explicit use of Hilbert space terminology and techniques, there are great advantages to be gained from a Hilbert space formulation. These are largely derived from our familiarity with two- and three-dimensional Euclidean geometry and in particular with the concepts of orthogonality and orthogonal projections in these spaces. These concepts, appropriately extended to infinite-dimensional Hilbert spaces, play a central role in the study of random variables with finite second moments and especially in the theory of prediction of stationary processes.

In this chapter we introduce an extremely important class of time series {X _t , t = 0, ± 1, ± 2,...} defined in terms of linear difference equations with constant coefficients. The imposition of this additional structure defines a parametric family of stationary processes, the autoregressive moving average or ARMA processes. For any autocovariance function γ(·) such that lim _h→∞ γ(h) = 0, and for any integer k > 0, it is possible to find an ARMA process with autocovariance function γ _X (·) such that γ _X (h) = γ(h), h = 0, 1,...., k. For this (and other) reasons the family of ARMA processes plays a key role in the modelling of time-series data. The linear structure of ARMA processes leads also to a very simple theory of linear prediction which is discussed in detail in Chapter 5.

The spectral representation of a stationary process {X _t , t = 0, ± 1, ... } essentially decomposes {X _t } into a sum of sinusoidal components with uncorrelated random coefficients. In conjunction with this decomposition there is a corresponding decomposition into sinusoids of the autocovariance function of {X _t }. The spectral decomposition is thus an analogue for stationary stochastic processes of the more familiar Fourier representation of deterministic functions. The analysis of stationary processes by means of their spectral representations is often referred to as the “frequency domain” analysis of time series. It is equivalent to “time domain” analysis, based on the autocovariance function, but provides an alternative way of viewing the process which for some applications may be more illuminating. For example in the design of a structure subject to a randomly fluctuating load it is important to be aware of the presence in the loading force of a large harmonic with a particular frequency to ensure that the frequency in question is not a resonant frequency of the structure. The spectral point of view is particularly advantageous in the analysis of multivariate stationary processes (Chapter 11) and in the analysis of very large data sets, for which numerical calculations can be performed rapidly using the fast Fourier transform (Section 10.7).

In this chapter we investigate the problem of predicting the values {X _t , t ≥ n + 1} of a stationary process in terms of {X ₁,..., X _n }. The idea is to utilize observations taken at or before time n to forecast the subsequent behaviour of {X _t }. Given any closed subspace ℳ of L ²(Ω, ℱ, P), the best predictor in ℳ of X _{n +h} is defined to be the element of ℳ with minimum mean-square distance from X _{n +h.} This of course is not the only possible definition of “best”, but for processes with finite second moments it leads to a theory of prediction which is simple, elegant and useful in practice. (In Chapter 13 we shall introduce alternative criteria which are needed for the prediction of processes with infinite second-order moments.) In Section 2.7, we showed that the projections are respectively the best function of X ₁,..., X _n and the best linear combination of 1, X ₁,..., X _n for predicting X _n+h . For the reasons given in Section 2.7 we shall concentrate almost exclusively on predictors of the latter type (best linear predictors) instead of attempting to work with conditional expectations.

In order to carry out statistical inference for time series it is necessary to be able to derive the distributions of various statistics used for the estimation of parameters from the data. For finite n the exact distribution of such a statistic f _n (X ₁,...,X _n ) is usually (even for Gaussian processes) prohibitively complicated. In such cases, we can still however base the inference on large-sample approximations to the distribution of the statistic in question. The mathematical tools for deriving such approximations are developed in this chapter. A comprehensive treatment of asymptotic theory is given in the book of Serfling (1980). Chapter 5 of the book by Billingsley (1986) is also strongly recommended.

If {X _t } is a real-valued stationary process, then from a second-order point of view it is characterized by its mean μ and its autocovariance function γ(•). The estimation of μ, γ(•) and the autocorrelation function ρ(•) = γ(•)/γ(0) from observations of X ₁, ..., X _n , therefore plays a crucial role in problems of inference and in particular in the problem of constructing an appropriate model for the data. In this chapter we consider several estimators which will be used and examine some of their properties.

The determination of an appropriate ARMA(p, q) model to represent an observed stationary time series involves a number of inter-related problems. These include the choice of p and q (order selection), and estimation of the remaining parameters, i.e. the mean, the coefficients {φ _i , θ _j : i = 1,..., p; j = 1,..., q} and the white noise variance σ ², for given values of p and q. Goodness of fit of the model must also be checked and the estimation procedure repeated with different values of p and q. Final selection of the most appropriate model depends on a variety of goodness of fit tests, although it can be systematized to a large degree by use of criteria such as the AICC statistic discussed in Chapter 9.

In this chapter we shall examine the problem of selecting an appropriate model for a given set of observations n{X _t , t = 1,…, n} data (a) exhibits no apparent deviations from stationarity and (b) has a rapidly decreasing autocorrelation function, we shall seek a suitable ARMA process to represent the mean-corrected data. If not, then we shall first look for a transformation of the data which generates a new series with the properties (a) and (b). This can frequently be achieved by differencing, leading us to consider the class of ARIMA (autoregressive-integrated moving average) processes which is introduced in Section 9.1. Once the data has been suitably transformed, the problem becomes one of finding a satisfactory ARMA(p, q) model, and in particular of choosing (or identifying) p and q. The sample autocorrelation and partial autocorrelation functions and the preliminary estimators \({\hat \phi _m}\) and \({\hat \theta _m}\) of Sections 8.2 and 8.3 can provide useful guidance in this choice. However our prime criterion for model selection will be the AICC, a modified version of Akaike’s AIC, which is discussed in Section 9.3. According to this criterion we compute maximum likelihood estimators of φ, θ and σ² for a variety of competing p and q values and choose the fitted model with smallest AICC value. Other techniques, in particular those which use the R and S arrays of Gray et al. (1978), are discussed in the recent survey of model identification by de Gooijer et al. (1985).

In this chapter we consider problems of statistical inference for time series based on frequency-domain properties of the series. The fundamental tool used is the periodogram, which is defined in Section 10.1 for any time series {x ₁,..., x _n }. Section 10.2 deals with statistical tests for the presence of “hidden periodicities” in the data. Several tests are discussed, corresponding to various different models and hypotheses which we may wish to test. Spectral analysis for stationary time series, and in particular the estimation of the spectral density, depends very heavily on the asymptotic distribution as n → ∞ of the periodogram ordinates of the series {X ₁,..., X _n }. The essential results are contained in Theorem 10.3.2. Under rather general conditions, the periodogram ordinates I _n (λ _i ) at any set of frequencies λ ₁,..., λ _m , 0 < λ ₁ < ⋯ < λ _m < π, are asymptotically independent exponential random variables with means 2πf(λ _i ), were f is the spectral density of {X _t }. Consequently the periodogram I _n is not a consistent estimator of 2πf. Consistent estimators can however be constructed by applying linear smoothing filters to the periodogram. The asymptotic behaviour of the resulting discrete spectral average estimators can be derived from the asymptotic behaviour of the periodogram as shown in Section 10.4. Lag-window estimators of the form \({\left( {2\pi } \right)^{ - 1}}\sum\nolimits_{\left| h \right| \leqslant r} {w\left( {h/r} \right)\hat \gamma \left( h \right)} {e^{ - ih\omega }}\), where w(x), −1 ≤ x ≤ 1, is a suitably chosen weight function, are also discussed in Section 10.4 and compared with discrete spectral average estimators. Approximate confidence intervals for the spectral density are given in Section 10.5. An alternative approach to spectral density estimation, based on fitting an AR MA model to the data and computing the spectral density of the fitted process, is discussed in Section 10.6. An important role in the development of spectral analysis has been played by the fast Fourier transform algorithm, which makes possible the rapid calculation of the periodogram for very large data sets. An introduction to the algorithm and its application to the computation of autocovariances is given in Section 10.7. The chapter concludes with a discussion of the asymptotic behaviour of the maximum likelihood estimators of the coefficients of an ARMA(p, q) process.

Many time series arising in practice are best considered as components of some vector-valued (multivariate) time series {X _t } whose specification includes not only the serial dependence of each component series {X _tj } but also the interdependence between different component series {X _ti } and {X _tj }. From a second order point of view a stationary multivariate time series is determined by its mean vector, μ = E X, and its covariance matrices Γ(h) = E(X _{t +h} X_t′) — μμ′, h = 0, ± 1,.... Most of the basic theory of univariate time series extends in a natural way to multivariate series but new problems arise. In this chapter we show how the techniques developed earlier for univariate series are extended to the multivariate case. Estimation of the basic quantities μ and Γ(·) is considered in Section 11.2. In Section 11.3 we introduce multivariate ARMA processes and develop analogues of some of the univariate results in Chapter 3. The prediction of stationary multivariate processes, and in particular of ARMA processes, is treated in Section 11.4 by means of a multivariate generalization of the innovations algorithm used in Chapter 5. This algorithm is then applied in Section 11.5 to simplify the calculation of the Gaussian likelihood of the observations {X ₁, X ₂,..., X _n } of a multivariate ARMA process. Estimation of parameters using maximum likelihood and (for autoregressive models) the Yule—Walker equations is also considered. In Section 11.6 we discuss the cross spectral density of a bivariate stationary process {X _t } and its interpretation in terms of the spectral representation of {X _t }. (The spectral representation is discussed in more detail in Section 11.8.) The bivariate periodogram and its asymptotic properties are examined in Section 11.7 and Theorem 11.7.1 gives the asymptotic joint distribution for a linear process of the periodogram matrices at frequencies λ ₁, λ ₂,..., λ _m ∈ (0, π). Smoothing of the periodogram is used to estimate the cross-spectrum and hence the cross-amplitude spectrum, phase spectrum and squared coherency for which approximate confidence intervals are given. The chapter ends with an introduction to the spectral representation of an m-variate stationary process and multivariate linear filtering.

In recent years, state-space representations and the associated Kalman recursions have had a profound impact on time series analysis and many related areas. The techniques were originally developed in connection with the control of linear systems (for accounts of this subject, see the books of Davis and Vinter (1985) and Hannan and Deistler (1988)). The general form of the state-space model needed for the applications in this chapter is defined in Section 12.1, where some illustrative examples are also given. The Kalman recursions are developed in Section 12.2 and applied in Section 12.3 to the analysis of ARMA and ARIMA processes with missing values. In Section 12.4 we examine the fundamental concepts of controllability and observability and their relevance to the determination of the minimal dimension of a state-space representation. Section 12.5 deals with recursive Bayesian state estimation, which can be used (at least in principle) to compute conditional expectations for a large class of not necessarily Gaussian processes. Further applications of the Bayesian approach can be found in the papers of Sorenson and Alspach (1971), Kitagawa (1987) and Grunwald, Raftery and Guttorp (1989).

In this final chapter we touch on a variety of topics of special interest. In Section 13.1 we consider transfer function models, designed to exploit, for predictive purposes, the relationship between two time series when one leads the other. Section 13.2 deals with long-memory models, characterized by very slow convergence to zero of the autocorrelations ρ(h) as h → ∞oo. Such models are suggested by numerous observed series in hydrology and economics. In Section 13.3 we examine linear time-series models with infinite variance and in Section 13.4 we briefly consider non-linear models and their applications.