A comparative simulation study of AR(1) estimators in short time series

Time series analysis has been valuable for achieving insight into the nature of longitudinal processes. Especially the autoregressive moving average (ARMA) model (Box and Jenkins 1976) has gained enormous popularity in various research areas. The autoregressive part models the serial dependence between consecutive measurements. The moving average part models the serial dependence between consecutive error terms. The ARMA(p, q) model is given by:with \(y_t\) the score at time \(t\;(t=1,2,\ldots ,T)\), μ the population mean, \(\phi _i\) the autocorrelation for lag \(i\;(i=1,2,\ldots ,p)\), \(\theta _j\) the moving average parameter at lag \(j\; (j=1,2,\ldots ,q)\) and \(e_t\) the residual.

One of the simplest versions of the ARMA(p, q) is the AR(1) model:where, for simplicity, the subscript 1 is omitted from ϕ. Several estimation methods have been proposed to estimate the AR(1) model. These estimation methods include closed form estimation methods, such as the r ₁ estimator (Yule 1927; Walker 1931; Box and Jenkins 1976), C-statistic (Young 1941) and Ordinary Least Squares (OLS) estimator, and iterative estimation methods, such as frequentist Maximum Likelihood Estimation (MLE) and Bayesian Markov Chain Monte Carlo (MCMC) estimation. The performance of the closed form estimation methods in terms of efficiency have been examined and compared in some simulation studies (Huitema and McKean 1991; DeCarlo and Tryon 1993; Arnau and Bono 2001; Solanas et al. 2010). Generally, in particular for shorter time series (e.g., length \(T \le 50\)), the closed form estimation methods have been shown to have biased autocorrelation estimates and/or high variability. Because the closed form and iterative estimation methods have not been mutually compared so far, it is unclear which estimation methods perform better in terms of having a low bias and variability under relevant conditions for empirical practice. Further, little is known about the robustness of the specific estimation methods towards misspecification of the model. This knowledge is important to optimize a time series research design, and to select a low-variability, low-bias, and robust method for estimating an AR(1) model in empirical practice.

In this paper, we discuss two studies to assess the relative performance of several estimators of the AR(1) model. We focus on short time series, with a length T between 10 and 100. Even though these lengths are relevant, for example in psychological research, they are not thoroughly studied yet for all estimators we compare. For the autocorrelation we use values between −1 and 1, and hence consider stationary time series. Our first study provides the information needed to make an informed choice between the estimation methods for the AR(1) model. To this end, we selected five popular and/or promising estimation methods. In a simulation study, we compare these methods with regard to bias, standard error, the bias of the standard error, the rejection rate for \(\phi =0\), the power for \(\phi \ne 0\), and the point and 95 % interval estimates.

Our second study focuses on the issue of robustness. Robustness, as used in this paper, is the resilience to misspecification with regard to the number of parameters. The effects of misspecification of the ARMA(1,1), AR(1) and AR(2) model have been studied for the least squares estimator. For an underspecified model, the parameters become more biased when the unspecified parameters are further from zero (Tanaka and Maekawa 1984). Overspecification of the model gives a larger prediction mean squared error for the estimation of the score at \(y_t\) (Kunitomo and Yamamoto 1985). To study the robustness with regard to misspecification, we use the two estimation methods that showed the lowest bias in the first study. In the misspecification study we generate the data using an ARMA(1,1) model, but estimate the parameters as if the data was generated using the same AR(1) = ARMA(1,0) model as used in Study 1.

In the next section, we describe the selection process of the estimation methods used in this paper, followed by a short introduction to the estimators. Then, we present the design, performance criteria and the results of the first simulation study, which aims at comparing various estimators when applied to short time series following an AR(1) model. We continue with the design and results of the second simulation study, which aims at exploring the effect of underspecifying a short time series following an ARMA(1,1) model as data following an AR(1) model. We conclude our paper with a discussion of the simulation studies and the implications of the results.

To start, we performed a literature search towards estimation methods for AR(1) models. Our selection criteria for the papers were as follows: (1) it must discuss one or more simulation studies that compare different estimators of the AR(1) model; (2) it must include conditions with less than 50 time points and a range of values between r ₁ and 1 for the autocorrelation ϕ. This literature search revealed the five papers shown in Table 1.

The earliest discussed estimator is the r ₁ estimator (Walker 1931), as implemented in the Yule–Walker model (Yule 1927; Box and Jenkins 1976). However, since several studies have found that the bias of r ₁ for small samples is large, especially for data with a positive autocorrelation, various alternatives were proposed (Huitema and McKean 1991, 1994; DeCarlo and Tryon 1993; Arnau and Bono 2001; Solanas et al. 2010). A selection of these is given by name in Table 1. Note that most alternatives are based on the original r ₁, as can be deduced from the names using ‘r’ or ‘r ₁’ and a sub- or superscript. In general, the modifications of r ₁ showed a smaller bias than r ₁ itself, but a larger variability of the estimated autocorrelation (Huitema and McKean 1991, 1994; Arnau and Bono 2001; Solanas et al. 2010), except for the estimators \(r^+_1\) and the C-statistic. In direct comparisons between \(r^+_1\) and the C-statistic, it was shown that the C-statistic had a smaller average bias and a smaller average mean square error, thus a smaller variability, over different values of ϕ than the \(r^+_1\) estimation method.

Apart from the modifications of r ₁, another closed form solution may be used. The ordinary least squares (OLS) estimator is used in many different applications, most notably in regression analysis. Since the autocorrelation may be interpreted as a special kind of regression parameter, OLS can be used to find the autocorrelation. In comparisons, the OLS estimator showed a smaller bias than most derivations from the r ₁ estimators (Huitema and McKean 1991; Solanas et al. 2010). However, the OLS estimator also showed a slightly larger mean squared error than most r ₁ derivations. These comparisons between estimators reveal a bias-variance tradeoff in the autocorrelation estimator.

Two important methods that are not found in the comparisons listed in Table 1, are the frequentist MLE and Bayesian MCMC estimation. Though simulation studies using MLE have been done, those studies did not include the conditions of our primary interest. For example, the studies that considered different ARMA(p, q)-models (Stoica et al. 1986; Pantula and Fuller 1985; Garcia-Hiernaux et al. 2009 ), had no condition with less than 100 time points (Cox and Llatas 1991) or were aimed at examining other parts of the estimation process, such as deciding on which ARMA(p, q)-model to use (Watson and Nicholls 1992). This was the same for papers using Bayesian MCMC estimation. Examples of this are studies that have no systematic comparison using different estimators (Price 2012), use AR(2) models (West and Wilcox 1996) or use lagged cross-correlation (Zhang and Nesselroade 2007). The MLE and Bayesian MCMC have become often-used methods of analysis in different fields and applications.

In the next paragraphs we will describe the five different estimation methods used in this paper.

To compare the various estimators for the autocorrelation (ϕ), we simulate according to an AR(1) model (see Eq. 2). In the generation of the data we vary the length of the time series T and the autocorrelation ϕ. For T we use five different sizes, namely 10, 25, 40, 50 and 100. For \(\phi\), we use an autocorrelation of −0.90 to 0.90 inclusive, taking steps of 0.10. Earlier studies show that there is a difference between the bias for the negative and positive ϕ for several estimators, including r ₁ and the C-statistic (DeCarlo and Tryon 1993; Solanas et al. 2010). This indicates that a thorough test is required to include both positive and negative autocorrelations. Finally, the number of replications must be set. All of the studies in Table 1 have a minimum of 10,000 replications per condition. However, a pilot study showed that the maximum standard deviation of the mean \(\hat{\phi }\) over 5000–10,000 replications was 0.0007, when T = 10 and \(\phi =0.7\), for all estimators. Therefore we use N = 2000 replications per condition. Considering a fully crossed experimental design, this yields 19 × 5 × 5000 = 475,000 simulated data sets.

Across all conditions, μ is set to zero and \(\sigma ^2_e\) to one, which can be done without loss of generality. This results in a standard normal distribution for \(y_t\) given ϕ.

Priors We performed a small simulation study to decide on the values for the hyperparameters of the priors in our Bayesian analyses. In the model we use, only the prior distributions for μ and \(\sigma _e\) have such hyperparameters. We used 3 conditions, with \(\phi =-0.50,\;0\) and 0.50, using 1000 replications per condition and 2000 iterations per analysis. We set T = 10, since shorter series provide less data, and will therefore be more strongly influenced by the choice of the prior. For μ we used a normal prior with mean and standard deviation as given, and for \(\sigma _e\) we used a γ prior with shape and rate as given in the top part of Table 2.

As can be seen in Table 2, the differences in the estimated parameters are small, especially when taking into account the uncertainty added by the small T. As a result, we based our choice of priors on theoretical grounds. To reduce the influence of the priors, we choose our priors close to the distributions used for the data generation: \(\mu \sim N(0,2)\) and \(\sigma _e \sim \Gamma (2,2).\)

Outcome measures For each data set we obtain different estimators: r ₁, C-statistic, OLS, MLE, B_f and B_sr. To compare the estimators, we consider the bias of the various estimators of ϕ, their empirical standard error, the bias of the estimated standard error, the rejection rate for \(\phi =0\), power for \(\phi \ne 0\), and the point and 95 % interval estimates of ϕ. All outcome measures are calculated for each condition and each estimation method.

The OLS estimator rendered estimates of ϕ that were higher than one in absolute value, and thus non-stationary, as expected. The highest percentage of non-stationary estimates, 15.1 %, was found for the shortest series, r = 10 and the highest autocorrelation, \(\phi =0.90\). For r = 10 and \(\phi =-0.90 \; \text { to } \; \phi =0.80\), up to 6.8 % of the estimates per condition were non-stationary, with higher percentages associated with higher values of \(|\phi |\). For T = 25 to 50 and \(\phi = 0.50\) to 0.90 in absolute value, up to 2.3 % of the estimates were non-stationary. However, the difference in the results was quite small. Thus we will discuss only the OLS-A results for the OLS, which includes all measurements, unless the OLS-S shows a strong deviation from OLS-A.

For the Bayesian analysis, non-convergence is expressed in the potential scale reduction factor, \(\hat{R}\). The potential scale reduction factor shows the ratio of how much the estimation may change when the number of iterations is doubled, with a perfect 1 indicating that no change is expected (Gelman and Rubin 1992; Stan Development Team 2014). For each estimated parameter \(\phi\), μ and \(\sigma _e\), less than 0.39 % of the estimates showed a \(\hat{R}\) above 1.02. Furthermore, a maximum of \(2.8\,\%\), found for μ as estimated with B_f, showed a \(\hat{R}\) above 1.01.

To study the robustness of the estimation methods, we misspecify the model. The data is still analysed as if they stem from an AR(1) model, but we generate the data using an ARMA(1,1) model. We generate data sets for two different sizes of T, namely 25 and 50. For ϕ and θ, we use parameters of −0.90 to 0.90 inclusive, taking steps of 0.15. Every condition consists of 5000 replications. Considering a fully crossed design, this yields 13 × 13 × 2 × 5000 = 1,690,000 datasets.

Again, across all conditions, μ is set to zero and \(\sigma _e^2\) to one. We consider the same outcome measures for Study 2 as we did for Study 1.

As with the first study, we checked the \(\hat{R}\) of the estimated parameters ϕ, μ and \(\sigma _e\) of B_sr. For each of the parameters, less than \(0.14\,\%\) showed an \(\hat{R}\) above 1.02, and less than \(1.69\,\%\) showed an \(\hat{R}\) above 1.01.

We compared five estimation methods for the autocorrelation: the r ₁, C-statistic, ordinary least squares, maximum likelihood estimation and Bayesian MCMC estimation. For the Bayesian MCMC estimation we used both a flat prior and a symmetrized reference prior, giving a total of six autocorrelation estimators. We compared these estimators with regard to bias, variability, rejection rate, power and point and 95 % estimation interval estimates. After this comparison, we selected the Bayesian estimation with symmetrized reference prior and the maximum likelihood estimator to use in a second study. In this small study we assessed the robustness of the methods against underspecification.

The results we found in the first study largely complied with the results from previous studies. For the bias for positive values of ϕ, we found the bias of the C-statistic and the OLS to be smaller than the bias of r ₁, as did previous studies (DeCarlo and Tryon 1993; Huitema and McKean 1991; Solanas et al. 2010). For the empirical standard error, we found smaller values for r ₁ than for OLS, as did Huitema and McKean (1991). The low rejection rate we found for the r ₁ and the C-statistic confirms to earlier studies (Huitema and McKean 1991; DeCarlo and Tryon 1993; Solanas et al. 2010). The power we found for the C-statistic is similar to the power found by Arnau and Bono (2001). However, the results of Solanas et al. (2010) with regard to power were only partly confirmed by our study: for negative ϕ we indeed found a higher power for OLS followed r ₁, than for the C-statistic. But for positive ϕ, we found a higher power for the C-statistic than for r ₁.

The first study showed a strong improvement in all measures for all methods between T = 10 and T = 25. This improvement continued, be it not as strong, for higher values of T. When comparing methods, B_sr showed the smallest bias. For the frequentist methods, this was MLE followed by the C-statistic. The smallest empirical standard error is shown by r ₁, the smallest bias of the estimated standard error is shown by the C-statistic, the OLS and the MLE. We further found that a small bias is often paired with a high empirical standard error. The power was rather low for all methods at the lengths of time series we considered. For T = 25, the power is below 80 % for all methods for ϕ between −0.5 and 0.5, for T = 100, the power is below 80 % for ϕ between −0.2 and 0.2. The differences between methods with regard to power are negligible. In research areas where effect sizes are small, this may pose a problem. Some studies use moving windows to assess the stability of parameter estimates over time. For these moving windows, these results indicate that a moving window of at least 50 time points is advisable, especially when the differences in parameters over time are small.

The first study was conducted to explore the differences between estimation methods for the autocorrelation in a single subject design. However, this is only a small step in a large research area. The next step may be to explore these results in multilevel or group analyses, thus when there is not one but multiple subjects per dataset. Another issue may be how the different methods respond to non-stationary data, i.e., \(|\phi |> 1\).

In the second study, the robustness of the MLE and B_sr to underspecification was examined. In general, we confirmed the notion that the further the unmodelled parameter is from zero, the larger the influence of this parameter is on \(\hat{\phi }\) (Tanaka and Maekawa 1984). As with the first study, the empirical standard error decreased when T became larger. However, the bias reacted differently for both methods: for the MLE, the bias became slightly smaller for most conditions, where the bias of B_sr became slightly larger for a larger T. The difference in performance for all measurements between the MLE and the B_sr was small for both values of T. It was shown that the bias, variability, rejection rate and power were all highly dependent on the value of the non-modeled parameter in the data, θ. This can be related to the fact that the autocorrelation of the error also has an influence on the autocorrelation of the total score.

The robustness study was rather small and specific, looking into only one possible way to misspecify the ARMA (1, 1) model. More options within misspecification should be explored to find how robust the estimation methods are with regard to under-, over- and misspecification. Important points are the influence of a misspecified error distribution or overspecification of the model.

In conclusion, we found that the best performing methods for autocorrelation estimation are the Bayesian estimator with symmetrized reference prior and the maximum likelihood estimator. The difference in performance between these two is, for all practical purposes, negligible. The results for the measurements improving greatly between T = 10 and T = 25 and continue to do so, but in a less spectacular fashion. For the misspecification study, we found the size of θ, the non-modelled parameter, to be vital for the performance of the estimation methods. The differences between lengths of the series and estimation methods was of lesser influence on the results.