A common goodness-of-fit framework for neural population models using marked point process time-rescaling

Define an observation interval [0, T] and let 0 ≤ s₁ < s₂ <,...,< s_n− 1 < s_n ≤ T be a set of event (spike) times. Let N(t) be the number of spikes up to time t, which will increase by 1 at times when a spike occurs and will remain constant otherwise. Any point process N(t) describing neural spiking can be fully characterized by its conditional intensity function (Daley and Vere-Jones 2003) where H_t = {0 ≤ s₁ < s₂ <,...,< s_N(t) ≤ t} is the history of spiking activity up to time t. The conditional intensity function expresses the instantaneous likelihood of observing a spike at time t, and implicitly defines a complete probability model for the point process. It therefore serves as the fundamental building block for constructing the likelihoods and probability distributions needed for the point process data analysis.

The basic idea of the time-rescaling theorem is to transform a general temporal point process to a constant-intensity Poisson process by rescaling the spike times.

Note that u_j, j = 1,..., N(T), will be independent, identically uniformly distributed on the observation interval [0, \({\int _{0}^{T}} \lambda (t|H_{t}) dt]\) (Ross 1996). Once a point process model is fitted, we can integrate the estimated conditional intensity between the observed spike times s_j− 1 to s_j to get a set of rescaled interspike intervals, \(\hat {z}_{j} = \hat {u}_{j} - \hat {u}_{j-1}\), which should be independent, and follow an exponential distribution with rate equal to 1 if the fitted model is correct. We can then use well studied methods for assessing whether the rescaled times are well fit by a unit rate Poisson process model. For example, the Kolmogorov-Smirnov (KS) plot, which plots an empirical distribution from data against a model distribution, can be used to compare the rescaled interspike intervals to the exponential distribution. Similarly an autocorrelation analysis of the rescaled interspike intervals should show no significant structure at any lag if the estimated conditional intensity from the fitted model accurately describes the spiking observations (Brown et al. 2002; Truccolo et al. 2005).

We describe spike data from a neural population using a combination of the spike time, and another variable, m, called the mark, which can provide information about the spike waveform or the identity of the neuron to which that spike is associated (Kloosterman et al. 2014; Deng et al. 2015). This mark may be discrete (e.g. Neuron 1 vs Neuron 2) or continuous (e.g. spike amplitude); it may be univariate, a vector (e.g. spike amplitude from each channel in a tetrode), or even a function (e.g. a continuous waveform function). The population spiking activity is then given by the set of observations (s₁, m₁),(s₂, m₂),...,(s_n, m_n).

A marked point process is completely defined by its joint mark intensity function such that: where M is a subset of the mark space \(\mathscr{M}\) and H_t is the history of spiking activity, including all the marks, up to time t. Here λ(t, m|H_t) characterizes the instantaneous likelihood of observing a spike with mark m at time t. For fixed value m and t, λ(t, m|H_t) may depend on the past history of spikes with similar marks (corresponding to the intrinsic history dependence of each neuron), on the history of spikes with dissimilar marks (corresponding to functional connectivity between neurons), and on the extrinsic covariates that the neural population is encoding (for example, place and movement coding in rat hippocampus).

Taking an integral of λ(t, m|H_t) over the entire mark space \(\mathscr{M}\), gives the conditional intensity of observing a spike at time t regardless of the mark value. Λ(t|H_t) is often called the ground intensity of the marked point process (Daley and Vere-Jones 2003).

Marked point process modeling has been successfully applied to multi-unit spiking data (Kloosterman et al. 2014; Deng et al. 2015) and to the analysis of simultaneously recorded spike events (Ba et al. 2014). While some theoretical results related to time-rescaling of the marked point processes have been developed (Vere-Jones and Schoenberg 2004), a complete goodness-of-fit paradigm for population spiking models over fixed observation intervals has yet to be established. Here, we present a general time-rescaling theorem for marked point processes observed on a finite observation interval [0, T], with marks that could be either continuous or discrete.

A heuristic proof of this theorem arises from a simple change of variables. Consider the joint probability distribution of all of the spike times and marks, which is given by the product of the joint mark intensity function, \(\lambda (s_{j}, \mathbf {m}_{j}|H_{s_{j}})\), at the spike locations and the exponential of the negative integral of λ(t, m|H_t) over the whole time-mark space: Note that these marked spikes completely specify the joint mark intensity (which is history dependent) everywhere in the observation interval and therefore also specify the extent of the stochastic region R. By the multivariate change-of-variables formula (Port 1994), the joint distribution of the rescaled times and marks is given by the expression: The elements of \(\left [\frac {\partial \tau _{i}}{\partial s_{j}} \right ]\) are equal to \(\lambda \left (s_{j}, \mathbf {m}_{j}| H_{s_{j}}\right )\) if i = j, and are 0 if i < j. Therefore \(\left [\frac {\partial \tau }{\partial s} \right ]\) is a lower triangular matrix, and its determinant is given by the product of its diagonal terms, \(|\frac {\partial \tau }{\partial s}| = \prod _{j = 1}^{n} \lambda (s_{j} , \mathbf {m}_{j} |H_{s_{j}})\), so that where |R| is the volume of region R. This is equivalent to the joint distribution of a marked point process with constant unit joint mark intensity over the region R.

We can further conclude that the number of spikes in region R follows a Poisson distribution with mean equal to |R|. Thus the conditional joint distribution of rescaled spike times given that there are n spikes in the region R is This is exactly the joint density function of a temporally ordered set of independent uniformly distributed events in the rescaled stochastic region R.

Here, we presented a heuristic proof of the marked point process time-rescaling theorem based on a change-of-variables argument with the intension of providing intuition about the effect of rescaling. A complete proof requires a few additional details to ensure that the resulting process is well behaved, and more technical proofs are available in the literature (Meyer 1971; Brown and Nair 1988; Vere-Jones and Schoenberg 2004).

Based on the time-rescaling theorem result above, we can also derive the spike rate for the ground process of all the rescaled spikes across all marks.

The focus of the remainder of this paper is on using the generalized time-rescaling theorem to enable methods for evaluating the goodness-of-fit of a marked point process model to recorded population spiking data. Before delving into that topic, it is worth noting that the generalized time-rescaling theorem can also be used to generate simulated spiking data from any joint mark intensity function for a marked point process. This method parallels the well-known method for generating a spike train from the conditional intensity of a univariate point process using the classic time-rescaling theorem.

For a deterministic joint mark intensity function, λ(t, m), the simulation procedure is relatively simple. First, compute the total integrated intensity, \(\boldsymbol {{\Lambda }} = \int _{\mathscr{M}}^{} {\int _{0}^{T}} \lambda (t, \mathbf {m})dt d\mathbf {m}\), and sample the total number of spikes from a Poisson distribution with parameter Λ. For each of these spikes, sample its location uniformly from the region enclosed by the boundary, \(b(\mathbf {m}) = {\int _{0}^{T}} \lambda (t, \mathbf {m}) dt\). This can be achieved by sampling uniformly a rectangular volume encasing this region and only accepting samples that are within the boundary. If the i^th uniformly sampled spike has mark m_i and time τ_i, compute the spike time as \(s_{i} = \min \{s:\tau _{i} = {\int _{0}^{s}} \lambda (t, \mathbf {m}) dt\}\), where min gives the minimum (technically the infimum) value of s that satisfies thins integral. The resulting set of spike times and marks, (s_i, m_i) represents a sample from the marked point process with joint mark intensity λ(t, m).

When the joint mark intensity depends on its own history, λ(t, m|H_t), the simulation procedure is slightly more complicated, as each spike can influence the future values of the joint mark intensity as well as the computations of the total integrated intensity, the boundary, and the scaling of other spikes. In this case, the above procedure can be performed iteratively. Start by initializing with a history that includes no spikes, and generate a set of rescaled spikes. Fix the earliest occurring spike, update the joint mark intensity to include this spike time, and simulate a new set of spikes from this time forward. Continue iterating, fixing the earliest new spike, until you obtain a sample of zero new spikes from the Poisson sample. This procedure will produce a sample consistent with λ(t, m|H_t), where H_t is the history of the sampled spikes.

There are a variety of well studied approaches for assessing goodness-of-fit based on this rescaled process. These multiple methods are complimentary in that one method may detect lack of fit due to particular structure in the data that may not be detected by another method. A number of these are discussed in the discussion section, but here we focus on two relatively simple approaches that are easy to interpret and highlight multiple ways in which the model may fit the data well or poorly.

The first approach is based on Pearson’s chi-square statistic. To implement this, we divide the region R into M smaller subregions, R_i, each with volume |R_i|, and count the number of rescaled spikes, r_i, in each of these subregions. The test statistic is where \(p_{i}=\frac {|R_{i}|}{|R|}\) and n is the total number of points in R. We select the subregions such that np_i is sufficiently large (say, above 5) in each. If our marked point process model is correct and the rescaled spikes are uniform in this region, then X² will follow a chi-square distribution with M − 1 degrees of freedom. We will reject the null hypothesis that the points are uniformly distributed in region R if \(X^{2}> \chi ^{2}_{M-1,1-\alpha }\), where \(\chi ^{2}_{M-1,1-\alpha }\) is the critical value of the chi-square distribution with M − 1 degrees at a level of significance α.

Another approach for assessing the goodness-of-fit for the rescaled process is based on the Kolmogorov-Smirnov (KS) plot. For a univariate (unmarked) point process, if the model is correct, the rescaled process should be a homogeneous Poisson process with interspike intervals that have independent exponential distributions with mean 1. A KS plot then simply plots the empirical cumulative distribution function (CDF) of the rescaled times against the model CDF of an exponential distribution to visualize the deviation from the 45 degree line (Johnson and Kotz 1970). For a marked point process, the set of rescaled spike times (ignoring the mark values) should be an inhomogeneous Poisson process with rate \(\tilde {\lambda }(\tau )\), as defined in Section 2.2. We can therefore rescale this process one more time, based on the univariate time-rescaling theorem, construct KS plots, and make inferences from them. Additionally, we can perform KS tests to the supposed spikes over any subspace of the full mark space. That is, the set of rescaled spike time in a subspace R_s ⊂ R should be an inhomogeneous Poisson process with rate \(\lambda (\tau ) = \int _{\mathscr{M}}^{} I_{(\tau , \mathbf {m}) \in R_{s}} d \mathbf {m}\). This provides additional methods for detecting model misfit around particular marks, even when the marginal spike times over the full mark space is not rejected by the full KS test.

An alternative approach for constructing KS plots is first to normalize the rescaled spike times, τ_j, at each mark by the value of the boundary at the mark b(m). The following corollary gives the distribution of the resulting unordered rescaled spike times.

Note that the resulting transformed marked process is no longer a spatiotemporal homogeneous Poisson process. On the other hand, all the normalized rescaled spike times will lie in the cubic region [0,1]^p, where p is the dimension of joint time-mark space. The KS plot would then compare the empirical CDF of the interspike intervals of these rescaled spike times against the model CDF, the exponential distribution with parameter λ = N(T), without the need for an additional rescaling step. This approach is a general marked point process analogue of the method described in Gerhard et al. (2011) for multiple univariate point processes.

We will demonstrate the time-rescaling theorem as well as these two goodness-of-fit approaches to simulated data in Section 3, and to real neural population spiking data recorded from a rat performing a memory-guided spatial navigation task in Section 4.

The first simulation scenario comprises two neurons with spiking tuned to a single covariate, x_t, with coordinated, history dependent firing and overlapping mark distributions. We can think of x_t as a one-dimensional position variable, and our neurons as place cells with distinct place fields. Each neuron has a history dependent structure leading to a brief refractory period, and neuron 2 has an excitatory influence on neuron 1 at a lag of 10 time steps.

The position variable, x_t, is modeled as a stationary autoregressive (AR(1)) process. Mathematically, we define the state update equation for x_t as: where α = 0.98 and 𝜖_t is a zero mean white noise process, with standard deviation 0.3. The top panel of Fig. 1 shows a realization of x_t, over 10,000 time steps.

Spiking data was simulated according to a marked point process model with a joint mark intensity function with two modes, each corresponding to place fields with different locations in space and mark values. Both the spike time and mark are generated as a function of the process x_t and the mark can be thought of as a waveform amplitude. The two peaks are centered at 2 and − 2 in position and 11 and 12 in mark space. These peaks are each modeled as Gaussian functions with peak values of 0.15 spikes per time step and covariance matrix [0.5, 0; 0, 0.09]. This leads to moderate overlap between the peaks in the mark space (making perfect spike sorting impossible) but minimal overlap in the place coding. Finally, each neuron has a refractory period defined by the negative of a Gaussian function, centered at zero lag after a spike and with a standard deviation of 14 time steps, and neuron 2 has an excitatory influence on neuron 1 defined by a positive Gaussian function, centered at lag 10 time steps after a spike and with a standard deviation of 2 time steps.

Mathematically, the population spiking model is given by the joint mark intensity function where and represent the place fields for neurons 1 and 2 respectively, represents the excitatory influence of neuron 2 on neuron 1, for j = 1,2 represents the refractoriness of neuron j, and \(N(m; \mu _{mj}, \sigma ^{2}_{mj})\) expresses the normal distribution of marks for neuron j.

Here, N(t⁻) is the total number of spikes up to, but not including, time t, and for neuron index j = 1,2, S_j are the sets of spike times. The parameters a_j are the numeric values for the peak firing rates, the μ_xj, μ_mj are centers and the \(\sigma ^{2}_{xj}\), \(\sigma ^{2}_{ml}\) are variances in location and mark space for these 2 place cells, and the \({\sigma ^{2}_{j}}\) is the variance of excitatory influence and refractoriness. a₃ is the peak excitatory influence from neuron 2 on the firing rate of neuron 1. The numeric values for these constants used in the simulation can be found in Table 1.

Figure 1 shows the simulated spiking from this population as a function of the simulated x_t trajectory. There are a total of 383 spikes in this example. In panel A, spikes are shown as a function of time and position as red and blue dots. The red and blue coloration indicate whether a spike comes from neuron 1 or neuron 2, respectively. We can see a set of red spikes that tend to occur whenever x_t is near -2, and a set of both blue and red spikes that occur whenever x_t is near 2. This is due to the place field of neuron 2 and its excitatory influence on neuron 1. Note that the purpose of this simulation is not to mimic actual place field populations accurately and find the best model to fit, but to generate data that will provide intuition and highlight the ability of the general time-rescaling theorem to assess the goodness-of-fit in data with different types of dependence structures.

Using the simulated data, we performed goodness-of-fit analysis using the time-rescaling theorem we developed above on three possible spiking models. The first uses the true model that generated the data from Eq. 15, including the correct structure for the place fields and the mark distribution, and the full history dependence capturing the refractoriness of each neuron and the excitatory influence of neuron 2 on neuron 1.

The second model uses the correct place and mark structure of the spiking, but omits the history dependent structure completely. Mathematically, this is given by the joint mark intensity function

The third model uses a crude spike sorting procedure based on whether each mark value is above or below 11.5, to fit individual intensity models for each of the two sorted neurons. Each neuron has the correct place field structure and history dependent structure, but some spikes are mis-sorted due to the overlap in the mark distribution. Mathematically, the pair of the intensity models for these neurons are given by the following equations: and where the excitatory and refractory history dependent component now use the sorted spike identities: and

The parameters for the centers and variances, including μ_x1, μ_x2, μ_m1, μ_m2, σ_x1, σ_x2, σ_m1 and σ_m2, were fit from the simulated data using a maximum-likelihood estimate (MLE). The excitatory influence parameter, r, was estimated using the method of moments from the cross-correlation coefficients. The remaining model parameters, including a₁, a₂, a₃, σ₁ and σ₂, were assumed to be known.

Figure 2 shows the results of the time-rescaling analysis applied to each of the proposed models, using the same simulated spike data. The leftmost panels (Fig. 2a) show the goodness-of-fit assessment based on the true model and the true parameters used to generate the data given by Eq. 15. The next pair of panels (Fig. 2b) to the right show the analysis using the complete model form in Eq. 15 with parameters estimated by maximum likelihood. The next pair of panels (Fig. 2c) to the right show the goodness-of-fit for the marked point process model in Eq. 16 with correct mark and state dependence, but missing the history dependent component and with estimated parameter values. The rightmost panels (Fig. 2d) show the goodness-of-fit based on crudely sorted spikes given by the models in Eq. 17 with the correct state and history dependence structure and estimated parameters. The top panels show the rescaled spike times for each model. For the top-right panel, this is just the rescaled spikes for the two sorted neurons. For the other top panels, the rescaled spike times are given by blue dots, and the rescaled values of the end of the observation interval, τ(m, T), are shown as a function of m as a solid red line. The bottom panels show KS plots for all of the rescaled spike times under each of these models.

For the true model with the true parameter values, the value of τ(m, T) has local peaks around mark values of 11 and 12, corresponding to the two peaks in the joint mark-intensity function at these values. The peak around m = 11 is larger because of the excitatory influence in the history dependence from neuron 2 to neuron 1. Visually, the rescaled spike times appear to fill out this rescaled time-mark subspace uniformly. To perform the Chi-square test, we partitioned the mark space into a grid with M equal segments, and partitioned the rescaled region R up into M subregions R_i. The number M was chosen such that the expected number of spikes in each subregion is above 5. We then counted the number of rescaled spikes, r_i, in each of these subregions and calculated the Chi-square statistic by Eq. 13. The Pearson chi-square test for homogeneity of the rescaled times in this interval yields a p-value of 0.37, suggesting no clear evidence of inhomogeneity. The KS plot everywhere stays within its 95% significance bounds, suggesting no clear lack of fit among the full set of rescaled spike times.

The middle-left panels (Fig. 2b) show the goodness-of-fit analysis results using this same model, with parameters estimated by maximum likelihood. It is immediately evident that the quality of fit differs for the model between the true and estimated parameter values. The two clear modes in the boundary region under the true parameters are not present in the boundary using the estimated parameters, and there are evident regions where the density of the rescaled spikes in reduced. The Pearson test for uniformity now gives a p-value of p = 1.1 × 10^− 10. Similarly, the KS plot now slightly departs from its 95% bounds, suggesting some lack of fit due to the imprecision of the parameter estimates.

For the marked point process model missing the history dependent structure (middle-right panel (Fig. 2c)), the peak around m = 12 is larger because x_t stays near the place field of neuron 2 more often than neuron 1, while the missing history dependence does not affect the intensity. By eye, it seems that the rescaled times for mark values below 11.5 occur more densely than those for mark values above 11.5. This is borne out by the Pearson chi-square test (p < 10^− 10), which suggests inhomogeneity on the rescaled times, and therefore lack of fit between the model and the original spike data. The lack of fit is also visible in the KS plot, where the observed rescaled interspike intervals are consistently significantly larger than the model estimates.

The panel on the top right shows the rescaled times based on two sorted clusters. As a population model, this could be considered as a marked point process where the marks represent the cluster assignment. In that case, rescaling each spike according to its mark is equivalent to rescaling based on the intensity for whichever neuron the spike is clustered into. Missorted spikes therefore tend to be incorrectly scaled, leading to lack of fit, as observed through the KS plot.

We performed a second simulation to illustrate how the KS plot and chi-square test highlight different aspects of the goodness-of-fit. We consider again the same two neurons tuned to a single covariate x_t, and remove the history dependence of spiking, so that two neurons are simply inhomogeneous Poisson spiking units. In this case, the true joint mark intensity model, from which we generate the data, has the same form as Eq. 16, with parameters given in Table 1.

Figure 3 shows a goodness-of-fit analysis on the resulting data for four different candidate models we propose, Fig. 3a, the true model, Fig. 3b and c, two models whose λ(t, m) are uniformly scaled by 0.56 and 1.6, respectively, and Fig. 3d, a non-uniformly scaled model, with a_c (from Table 1) scaled separately by 0.56 and 1.6, for c = 1,2. Rescaling of the spikes according to the true model λ(t, m) produces good fits according to both tests, while the uniformly scaled candidate models pass the Pearson chi-square test with p-value 0.35, but the KS plot are far from being in the 95% confidence bounds. The deviation direction from the 45 degree line can be used to determine that the misspecified models underestimate and overestimate the intensity, respectively. Utilizing 523 spikes, the non-uniformly scaled candidate model, where each neuron has been scaled separately while keeping the overall firing rate close to that generated by the true model, passes the KS test, but the chi-square p-value is very small at 1.5 × 10^− 6, highlighting the complementarity of the tests.

While the overall pattern of rescaled interspike intervals doesn’t show lack of fit, the rescaled spikes are more concentrated at low mark values and less concentrated at high mark values. This example illustrates the importance of having multiple goodness-of-fit approaches to characterize different features of the data that may be captured or misspecified by a model. In this example both the KS analysis and the Pearson chi-square test are enabled by time-rescaling of the marked point process.

In this subsection, we perform a third simulation to illustrate how the sample size N(T) affects the statistical power for chi-square and KS tests in each of our simulation models. We consider again the same two neurons with spiking tuned to a single covariates, x_t, with history dependent firing and overlapping mark distributions, as described in Eq. 15. In this case we compare the ability to detect model misfit in three models, the true model that generated the data, the misspecified model that is missing history dependence, and the crude sorted spike model.

We simulated spike trains with increasing duration ranging from .5 to 20 seconds, and performed a goodness-of-fit analysis using the time-rescaling theorem and Pearson and KS tests we described above on these three possible spiking models. We repeated each simulation 100 times and recorded the resulting Pearson uniformity test and KS test p-values. Figure 4 shows the power as a function of the expected number of spikes on the rejection rate of Pearson (left panel) and KS (right panel) tests for the three different models, the true model (black), the missing history model (red), and the sorted spike model (blue). The rejection rate is calculated as the proportion of p-values smaller than a significance level of 0.05.

From both panels, we see that when the expected number of spikes is small, both the Pearson uniformity test and KS test reject the hypothesis of a well-fit model a relatively small fraction of the time. For example, as seen in the left panel of Fig. 4, the rejected fraction of simulations using the Pearson uniformity test for the misspecified missing history model is around 0.4 when the expected number of spike is about 20, and increases to 1 as the expected spike count increases to 800. We can also see similar trend for the KS test for both the missing history model and the sorted spike model. Note that the rejection rate of KS test for the sorted spike model increases more slowly than that of the missing history model, due to the misspecification of the latter. Also, as the expected spike count becomes large, both tests will reject with a probability approaching 1.

In this paper, we developed a general toolbox for assessing statistical models of neural populations based on a generalization of the time-rescaling theorem. Given technological advances in neural data acquisition, experiments involving multiple electrodes have now become standard in the practice of neuroscience, making these neural population models of great interest. Understanding these network structures sheds light on how groups of neurons interact with, react and respond to one another and help define possible functions of regions of the brain (Chen et al. 2011; Macke et al. 2011). In addition, the prevalence of multiunit data has brought into question the necessity of spike sorting in every neural population analysis. While many population analyses begin with a spike sorting step and a characterization of the receptive field properties of each sorted neuron, multiple recent experiments have explored the power of clusterless population models (Kloosterman et al. 2014; Deng et al. 2015). Therefore it is valuable to have goodness-of-fit tools that can apply equivalently to both sorted and clusterless population models. A fundamental challenge in assessing the goodness-of-fit of models of spiking systems is that the timing of each spike has its own distribution, based on many factors that can include coding of dynamic biological and behavioral variables, past spiking history, network effects, and adaptation. The time-rescaling theorem allows us to take any candidate model, and all the dependence structures it describes, and rescale the spikes in such a way that, if the model is correct, they should become samples from a simple uniform distribution. We can then use well-established methods for assessing uniformity to assess the quality of the original model used for rescaling. Furthermore, by taking only the rescaled spike times and disregarding the marks, we can generate a new univariate spike train and use the many existing goodness-of-fit tools for individual spike trains to assess the quality of the joint mark intensity model. An important feature of this general time-rescaling theorem is that not only are the spike times rescaled, but the observation interval [0, T] is also rescaled for each possible mark value. Since a joint mark intensity model can depend on other stochastic processes (such as its own history or the biological and behavioral variables encoded by the population), the intensity is itself a stochastic process, and therefore the rescaled observation region is also stochastic. Therefore, the assessment of uniformity is based both on the rescaled spike times and the rescaled region.

We illustrated our approach via a series of simulation analyses as well as an application involving place cell spiking activity from the CA3 region of the hippocampus in a rat performing memory guided navigation task on a W-shaped maze. In the first simulation, we implemented the goodness-of-fit tests in three different model fits: the true model, a model intentionally missing history-dependence, and a model for which the mark corresponds to one of two labels given by spike sorting. As expected, the results indicated proper fit with the true model, and a lack-of-fit in both the model missing history dependence as well as the sorted model. This demonstrated the ability of the approach to discern different reasons for lack of fit. Importantly, we could assess the quality of fit for both sorted and clusterless spiking models and determine the degree to which sorting affected the model fit. In our second simulation, we demonstrated that distinct goodness-of-fit measures, both based on the same time-rescaling approach, could be used to determine different aspects of the model fit quality. Incorrectly scaling the intensity uniformly over all marks led to lack of fit evidenced by the KS plot but not the assessment of uniformity; differentially scaling subsets of mark values, as might occur with a model that misspecified the receptive fields of particular neurons, led to lack of fit evidenced by lack of uniformity.

In our real-data example, we used a 4-dimensional mark representing the waveform peak amplitudes across a tetrode to exhibit the ability to generalize to more complicated mark spaces. The fit of a Gaussian mixture model with no history dependence captured much of the temporal structure, as evidenced by the KS plot, but perhaps fit the spikes in some mark regions better than others, as suggested by the analysis of uniformity in disjoint subsets of the mark space. A follow-up analysis, suggested that the model may not be capturing the hash spikes as well as the higher amplitude spikes.

In this paper we focused on two goodness-of fit measures that could be applied to the rescaled spike times and marks, a Pearson chi-square test comparing the expected and observed number of spikes in subsets of the rescaled observation region, and a KS plot analysis based on rescaling the rescaled spike times again based on the expected rescaled spiking rate. However, there are a variety of other goodness-of-fit tools available after rescaling that could also be used, either instead of, or to compliment these analyses. For example, a well-studied statistical approach for assessing uniformity is based on Ripley’s K-function (Ripley 1977). This function, K(x, r), is defined as the expected number of points within a ball b(r) with radius r centered at x. For uniform rescaled spikes, this function should grow as r^d, where d is the dimension of the mark-time space. We can compute the empirical K function, \(\hat K(r)=\frac {1}{n}\sum _{i\neq j}I_{d_{ij}}<r/n\), where d_ij is the Euclidean distance between rescaled spikes i and j, and \(I_{d_{ij}}<r\) is equal to 1 if that distance is less than r, and otherwise 0. We can then compare the empirical K function to the theoretical one under a uniform model to assess the quality of our original model. We can also construct confidence intervals for the estimated function and compute a corresponding p-value via Monte Carlo simulations (Baddeley et al. 2005). A variety of other well-documented and tested methods are also available (Petrie and Willemain 2013) and could be used interchangeably with those we specifically mention in this paper. A few examples include those that perform a two-sample test on a subsample of points in a high-density region and a subsample in a low-density region (Jain et al. 2002), or those that consider the distribution of distances from points to the boundary of support, both in the case of known support (Berrendero et al. 2006) and unknown support (Berrendero et al. 2012).

It is important to note that no single goodness-of-fit technique can demonstrate that a model completely captures the statistical structure present in a data. Thorough model assessment requires applying multiple goodness-of-fit tools to characterize the ways in which a model fails to capture different features of the data. Here, we focus on goodness-of-fit approaches that are enabled by time-rescaling, which have been used extensively for models of sorted spike train data. These approaches, along with complementary methods, such as those based on the model deviance, cross-validation, residual analysis, and auto and cross-correlation estimation, provide a powerful toolbox for model assessment (Ogata 1988; Truccolo et al. 2005; Kass et al. 2014).

There are a number of extensions and avenues for future exploration for this goodness-of-fit framework. In the simulations, we provided examples of how assessments based on time-rescaling could be used to help identify areas of lack of fit, and to suggest refinements to population spiking models. The ways in which different measures might be used for model refinement should be explored in more detail, and specific recommendations could be made about the best measures to use to identify particular features that should be added or altered in a model. Also, in our examples, we limit the standard point process goodness-of-fit analysis to KS plots but with the appropriate adaptations and generalizations, one could also employ other common techniques such as the QQ plot, autocorrelations of rescaled wait times, or a Fano Factor analysis to assess dispersion.

Another possible extension might focus on mark rescaling rather than time-rescaling. We could retain the observed times of each spike, and modify each spike mark to produce uniform spikes over a stochastic region with a fixed temporal extent, but random mark boundaries. For a one-dimensional mark, this could be achieved by replacing the mark of the ith spike, m_i, with the integral \(\int _{0}^{m_{i}}\lambda (t_{i},m)dm\) (Merzbach and Nualart 1986). An advantage of such an approach would be that the spike times would remain the same and be interpretable. For example a cluster of points at a particular time point might suggest model lack of fit specific to that time. However, since the temporal pattern of spikes would be unchanged, it would still retain all the temporal dependence structure in the original data. Additionally, how to best rescale in general mark spaces is still unknown.

Additional research could also be done on improving the computational burden of these methods in high dimensional mark spaces. While the rescaling of times is based only on the number of spikes, not the dimensionality of the marks, the computation of the boundary of the stochastic region will grow in complexity with the mark dimension, and it can be challening to determine which model features lead to lack-of-fit. There may be multiple ways to deal with this, including methods of efficiently approximating the boundary assuming smoothness of the intensity, or goodness-of-fit measures that are less sensitive or do not require direct knowledge of the full boundary.

With this method, we provide model assessment tools that can be used appropriately for both population models and sorted models and and help collect more detailed information on their respective fits. In this way, researchers can better understand the advantages and disadvantages posed by population and single-unit modeling. Ultimately, this could provide significant insights into the question of when neural network structures can be better understood with spike sorting or direct ensemble modeling. Additionally, as experiments head in more complex directions and datasets become richer, modeling methods will need to improve and develop alongside them. For researchers to maintain confidence in any conclusions drawn from the application of a particular modeling approach, a corresponding goodness-of-fit toolset is essential. Here, we present a general goodness-of-fit approach that can assess and indicate areas of lack-of-fit for a wide variety of population spiking models, enabling researchers to gain more understanding and insight into the increasingly complex data structures being made available in neuroscience. We have made the MATLAB and Python code used for the analyses in this paper publicly available through our Github repository at We have made the MATLAB and Python code used for the analyses in this paper publicly available through our Github repository at https://​github.​com/​Eden-Kramer-Lab/​popTRT.