A Hamilton-Jacobi-Bellman approach for termination of seizure-like bursting

Epilepsy affects as many as 3 million people in the United States alone and annually costs approximately 12.5 billion dollars (CDC 1994; Kobau et al. 2007). For many of these people, seizures remain poorly controlled despite the use of anti-convulsive medication. This has prompted researchers to search for other therapies to help mitigate seizure frequency and duration. Among these treatments, Deep Brain Stimulation (DBS), a method by which a high frequency, pulsatile stimulus is periodically injected into the anterior nucleus of the thalamus, has been successful in clinical trials in suppressing seizure frequency and severity (Fisher et al. 2010; Lee et al. 2006; Lim et al. 2007). Furthermore, brief pulses of electrical stimulation have been shown to suppress seizure-like cortical afterdischarges, raising the possibility that DBS could terminate seizures during the ictal phase (Motamedi et al. 2002; Kinoshita et al. 2004). While these medical interventions are undeniably effective in some patients, these methods are ad hoc, are administered in an open loop manner, do not take into account the underlying seizure dynamics, and are most likely far from optimal. This has led researchers to develop alternative control strategies to better control epileptic seizures. For instance, Gluckman et al. (2001) used a proportional feedback algorithm to control epiliptic activity in brain slices with an electric field. Methods for control of periodicity in chaotically bursting neural systems have been proposed and have proven successful in brain slices (Schiff et al. 1994; Slutzky et al. 2003). Also, Ching et al. (2012) investigated the feasibility of grid-based brain electrical stimulation to suppress seizure propagation with a proportional control strategy.

The exact mechanism by which seizures are created and sustained is unknown, but many studies are beginning to investigate the role of the extracellular microenvironment in pathological neural bursting behavior (Kager et al. 2000; Bazhenov et al. 2004; Park and Durand 2006; Fröhlich et al. 2010). In this work, we present a control strategy for terminating bursting behavior for the model presented in Cressman et al. (2009), which includes a conductance-based neural model as well as local intra- and extracellular ion concentration dynamics. For certain parameters, this model displays periodic, recurrent seizure-like activity, and our control strategy seeks to find both time-optimal and energy-optimal DBS stimuli which will terminate seizure-like bursting behavior by driving each pathological neuron to a sufficiently refractory target set. We illustrate this strategy for a single neuron and for a homogeneous network of pathologically bursting neurons with the control objective of terminating a seizure once it begins. This novel approach to DBS has the potential to significantly improve the management of seizures in patients with medically intractable epilepsy.

We consider a six-dimensional conductance-based model for neural activity with intracellular and extracellular ion concentration dynamics (Cressman et al. 2009). We choose this model because it exhibits periodic, seizure-like bursting behavior. The equations for this model are as follows:

Here, V represents the transmembrane voltage of the neuron, C represents the cell membrane capacitance, n and h represent gating variables, and [Ca]_i, [Na]_i, and [K]_o represent intracellular calcium, intracellular sodium, and extracellular potassium concentrations, respectively. We have augmented the voltage equation by additively including DBS input, u(t)=I(t)/C. For a full explanation of all model functions and parameters we refer the reader to Appendix ; code for this model is available from ModelDB.¹ For the parameters used in this paper, the model exhibits periodic seizure-like bursting behavior due to the slow dynamics of [K]_o and [Na]_i. Figure 1 displays the periodic behavior of the slow ion dynamics. The extracellular potassium rises slowly, increasing the excitability of the neuron until it reaches a level where the neuron is depolarized beyond its spiking threshold, leading to bursting behavior which rapidly increases the extracellular potassium concentration. Finally, the intracellular sodium concentrations grow until bursting behavior terminates, allowing [K]_o and [Na]_i to recover.

In order to terminate seizure-like bursting in this model, we must first identify the regimes for which Eqs. (1)–(6) exhibit bursting behavior. To produce bursting, a model must have a mechanism to generate spiking behavior and a separate mechanism with slow dynamics (Ermentrout and Terman 2010). For the model presented in Section 2, Eqs. (1)–(4) (fast variables) describe the neural spiking behavior, with variables that change on a much shorter time scale than [K]_o and [Na]_i (slow variables). Treating the slow variables as constants, we perform a bifurcation analysis on the full model to quantitatively analyze the bursting and quiescent regimes of the model. Using MATCONT (Dhooge et al. 2003), we are able to follow a curve of Saddle-Node Infinite Periodic orbit (SNIPER) bifurcations (Guckenheimer and Holmes 2010), a codimension one bifurcation which gives rise to a stable limit cycle. The left panel of Fig. 2 shows the curve of SNIPER bifurcations for the fast spiking variables from Eqs. (1)–(4). For ion concentrations to the right of the dotted curve, the neuron is in a bursting regime, \(\mathcal {B}\), and for concentrations to the left of the curve, the neuron is in a quiescent regime, \(\mathcal {Q}\). The right panel of Fig. 2 shows the trajectory of the slow variables from Fig. 1 with the line of SNIPER bifurcations shown for reference.

A naive approach to terminating bursting behavior in this model is to drive a neuron from \(\mathcal {B}\) to anywhere in \(\mathcal {Q}\). This objective can be accomplished, for instance, by simply giving an inhibitory stimulus until the neuron reaches the quiescent regime. In Fig. 3 we employ this strategy to drive a bursting neuron to a quiescent regime, but because the ion concentrations have not been significantly altered, the neuron begins bursting soon after the inhibitory control is removed.

In order to further refine a target set, we define a refractory index for quiescent neurons, which measures the time it takes for a quiescent neuron to begin bursting in the absence of a control input. We note that the refractory index is largely independent of the initial conditions of the fast variables, provided they do not cause the neuron to spike. Figure 4 shows the results of a numerical calculation of the refractory index as well as the line which separates the boundary of the target set \(\mathcal {T} = \{ x | \mathcal {R}(x) \geq 15 , \quad d(x,\mathcal {B}) \geq 0.2 \} \), where x=[[K]_o,[Na]_i] ^T , and \(d(x,\mathcal {B})\) is the minimum distance from point x to the bursting regime using the 2-norm. We recognize that in an experimental setting, it may be difficult to measure the ion concentrations precisely, and include the second condition, \( d(x,\mathcal {B}) \geq 0.2\) as a margin of safety to ensure that the neuron has reached \(\mathcal {Q}\).

We note that for this particular model, only the intracellular and sodium extracellular potassium dynamics fluctuate on a slow time scale to give a bifurcation boundary that exists in a two dimensional space. For a different model with extra slow variables, for instance, if the glial buffering capacity also varied on a slow time scale, the bifurcation boundary and subsequent target set will exist in a higher dimensional space.

We consider a single neuron with DBS input u(t) where z=[V,n,h,[Ca]_i,[K]_o,[Na]_i] ^T , B=[1,0,0,0,0,0] ^T , and Implementing an optimal control strategy for the full model would be difficult given the fast-slow dynamics of the system, and would likely require a control feedback system with knowledge of each of the state variables to be implemented effectively. In order to simplify the control problem, we first reduce the system by eliminating the fast-time-scale dynamics. Notice that f _K and f _Na only depend on the fast variables in the terms I _K and I _Na. This allows us to reduce (8) by averaging out the fast dynamics, resulting in where x=[[K]_o,[Na]_i] ^T , \(I(x,u) = [ -0.33\overline {I_{\mathrm {K}}}(x,u) , \frac {0.33}{\beta } \overline {I_{\text {Na}}}(x,u)]^{T}\), and

Here, \(\overline {I_{\mathrm {K}}}\) and \(\overline {I_{\text {Na}}}\) are determined by fixing the external potassium concentration, internal sodium concentration, and control input, then allowing the model cell to reach either a steady state or a periodic orbit, and finally time-averaging I _K and I _Na over one second. The terms \(\overline {I_{\mathrm {K}}}(x,u)\) and \(\overline {I_{\text {Na}}}(x,u)\) are calculated by interpolating the numerically time-averaged data.

We first solve for \(\mathcal {V}_{t}\) using the minimum-time methodology presented in Section 4.1. Using this information, we determine an optimal control policy based on Eq. (16). The cost function \(\mathcal {V}_{t}(x)\) and optimal control policy are shown in Fig. 5. The numerics suggest that for our particular choice of parameters in Eq. (15), \(\text {argmin} (\nabla \mathcal {V}_{t}^{T}(x)I(x,u))\) is unique for all x, thus Eq. (16) would imply that the resulting minimum-time control policy is unique. Notice that the control policy is not strictly of the bang-bang type as is common for minimum time problems. This happens because, unlike problems for which the control input is simply added to the right-hand-side of the dynamic equations, the influence of the applied control is a function of the time-averaged neural dynamics.

In order to test the validity of the reduction, we apply the control policy shown in Fig. 5 to both the reduced and full dynamics, Eqs. (10) and (8) respectively. The external control is set to zero until the neuron exhibits seizure-like behavior (i.e. bursting), and to calculate the control we use an initial condition that is just to the right of the SNIPER bifurcation along the orbit shown in the right panel of Fig. 2. Figure 6 shows the result of this simulation. The left panel shows the trajectory for the reduced model and full model as thin, grey and black lines, respectively. The top-right panel gives the control input applied to the full and reduced model as a black and grey line, respectively. Note that for the reduced model, the control and associated state trajectory are time optimal. The bottom right panel gives the voltage trace for the neuron from the full model. The control policy applied to the reduced model gives t _{m i n} =1.61 while the same control policy applied to the full model gives t _{m i n} =1.65. We see good agreement between the solutions obtained from full and reduced models, and conclude that the reduction by elimination of fast variables is a useful approach to solving this problem. Note that even though it takes more than one second to reach the target set, bursting activity terminates as soon as the stimulus switches from positive to negative. When we apply this control policy to the full model, bursting activity terminates after 0.53 seconds while, without any control input, bursting activity lasts for 6.36 seconds. The control policy applied to the full model yields a 92 % decrease in the duration of bursting activity. Note that the positive stimulus induces bursting that is more rapid than when the control is not applied, but there are approximately half as many spikes as compared to when control is not applied. Also, it is possible to further improve the reduction in bursting time by reducing the restriction on the size of the external input, particularly, by increasing the value of u _{m a x} .

Next, we compare the minimum-time stimulus to other similar, non-optimal stimuli, with results shown in Fig. 7. The total time to reach the target set for u _{o p t} ,u ₂,u ₃, and u ₄ is 1.651, 1.664, 1.653, and 1.799 units of time, respectively. The stimuli u ₂ and u ₃ use signals with similar amplitude to u _{o p t} , but change the time at which the transition from positive and negative control occurs. We find that these two non-optimal stimuli only marginally increase the time to reach the target set. The stimulus u ₄ varies from the u _{o p t} in the magnitude of the positive control used. We find that this has a relatively larger effect on the overall time required to reach the target set. It is worth noting that neural spiking ends when each stimulus switches from an excitatory (positive) to an inhibitory (negative) stimulus, but spiking will return if the inhibitory stimulus is not applied for a sufficient amount of time. We also note that each stimulus reaches the target set in a different location and, if desired, the cost function in the optimal control problem could be reformulated to balance the trade off between the time to reach the target set and the refractory index at the end time, \(\mathcal {R}(x(t_{end}))\).

For simulations of multiple neurons, we consider a network model consisting of two layers of one-dimensional networks comprised of 60 excitatory pyramidal cells (PC’s) and 60 inhibitory interneurons (IC’s), which is inspired by Ullah et al. (2009) and Gutkin et al. (2001). Both neuron types are modeled as conductance-based cells with ionic concentration gradients. Neurons within the same layer are aligned in a ring and coupled through spatially dependent synapses as well as lateral diffusion of potassium through extracellular space. For further details of the equations and parameters used in the network model, we refer the reader to Appendices A and B.

We apply control to the network model assuming that each neuron receives an identical control input. The population average potassium, \(\overline {[\mathrm {K}]}_{\mathrm {o}}\), and sodium, \(\overline {[\text {Na}]}_{\mathrm {i}}\), levels of the excitatory neurons in the network are monitored, and the control is applied based on the strategy from Fig. 5. Network results are shown in Fig. 8. The average value of the ion concentrations reaches the target set approximately 1.6 seconds after onset of bursting activity, and the bursting lasts approximately one half-second, whereas in the absence of control, network bursting lasts for approximately 6.4 seconds. Note that the results for the noisy network simulation are similar to the results for the single neuron from Fig. 6.

For most of the duration of the minimum-time stimulus, the controller is operating at or close to the positive and negative limits of the applied stimulus. Thus, we expect that if we relax the minimum time constraint, to give solutions that reach the target set at times that are larger but still close to the minimum time, we could save a significant amount of energy. With this in mind, we use the minimum-energy methodology presented in Section 4.2 to solve for \(\mathcal {V}_{e}(x,t)\), and calculate the optimal stimuli for the full model for different choices of t _{e n d} . The resulting energy-optimal stimuli and the associated trajectories in the ([K]_o,[Na]_i) plane are shown in the right and left panels of Fig. 9, respectively. For the energy-optimal stimuli in Fig. 9, the overall energy used, \({\int }_{0}^{t_{end}}u^{2}dt\), in order of decreasing values of t _{e n d} is 5.65, 6.29, 7.42, and 55.3 units. We find that by slightly relaxing the minimum time constraint, it is possible to find stimuli which use an order of magnitude less energy than the minimum-time stimulus, which may be attractive from a clinical perspective.

As with all control methods, this methodology is not without limitations. Safety concerns such as Faradaic charge injection (Merill et al. 2005) become important when DBS is implemented in the long term. Also, positive charge injection during the application of the optimal control serves to temporarily increase the bursting activity of the neuron, which may be harmful. These and other biologically relevant safety issues must be carefully considered before experiments can be performed on real neurons, but because of the generality of the Hamilton-Jacobi-Bellman approach, they can be addressed through modification to parameters in the calculation of the optimal control. This implementation of the time-optimal control strategy would require a model of real epileptic neurons that is accurate enough for control purposes, as well as a way to estimate the intra- and extracellular ion concentrations in real time. These challenges are beginning to be addressed through Kalman filtering, and may be feasible in the future (Ullah and Schiff 2010; Schiff 2010).

While this particular analysis was applied to a single-neuron model of seizure-like bursting activity, this methodology can be adapted to include different considerations for different models. For instance, this particular control strategy was applied to a model of periodic recurrent seizure-like activity, which does not accurately reflect all types of seizures. For a different model which exhibits seizure-like activity spontaneously or as the result of an external input, this methodology could still be implemented by defining an appropriate, non-pathological target set without consideration of the refractory period, as the seizure-like activity does not occur periodically. Furthermore, the single-neuron mechanism of bursting activity in this model is caused by pathological fluctuations of cellular ion concentrations, but at a network level, seizures are thought to occur as the result of an imbalance of synaptic excitation and inhibition (Cossart et al. 2001; Wendling et al. 2002; Gnatkovsky et al. 2008; Avoli and de Curtis 2011). This framework can still be handled for an appropriate model of seizure-like activity with fast-slow dynamics by averaging the fast, spiking dynamics for any general control input and applying the HJB control framework to the remaining slow dynamics.

We have described a method for driving a periodically bursting neuron to a sufficiently refractory target set in minimum time using electrical stimuli. Also, the total energy consumption for the minimum-time stimulus has been compared to the energy-optimal stimuli obtained for stimuli of slightly larger duration than the minimum time. We find when an entire network exhibits pathological, seizure-like bursting, this control methodology can reduce the amount of time the network spends in the bursting state by an order of magnitude. While the problem formulation is relatively complex, the resulting control strategy is quite simple.

The specific model used in this study examines the dynamic relationship between cellular sodium, potassium and calcium concentrations and their effect on the qualitative behavior of inhibitory and excitatory neurons. In this study, we find that once bursting begins, an initial excitatory stimulus increases the firing rate of bursting neurons, quickly increasing the extracellular sodium and extracellular potassium levels. Once a sufficiently large concentration of intracellular sodium has been reached, an inhibitory stimulus suppresses neural firing allowing ion exchange pumps, glial cells, and diffusive mechanisms to remove excess potassium, ending the bursting activity. Interestingly, when the intracellular sodium concentration is larger, ion exchange pumps work faster to remove excess potassium, causing the refractory index of x(t _{e n d} ) for the minimum time to reach control strategy to be much larger than required. For a different model, such as a network model for seizure-like behavior with large scale network dynamics such as network excitation and inhibition, different mechanisms of seizure termination could be exploited to find minimum-time, or optimal-energy control inputs.

We emphasize that the specific control strategy employed in this paper is not meant to be a definitive treatment for epileptic seizures. The model we have used is not perfect, as each neuron in this model spends much time close to a bifurcation point, and cannot spike more than a few times before reaching bursting state, which does not reflect the physiological need to propagate information through neurological spikes. The preceding method is meant as a proof of concept that more sophisticated methods than proportional feedback control have the potential to be successfully implemented given an accurate model of epilepsy and a clear control objective. Further investigation of models and mechanisms of seizure initiation and termination are needed, but this method shows promise in improving existing DBS strategies for termination of medically intractable epileptic seizures.