Determining Confidence Intervals, and Convergence, for Parameters in Stochastic Evacuation Models

A stochastic approach to evacuation simulation [1, 2] is employed in many models [3‐8] to reflect uncertainty in human behaviour [9]. If an (experimental or simulated) evacuation trial is repeated using the same population and the same initial conditions, it is possible that the evacuation will progress differently and may result in differences in total evacuation time (TET) or other statistics of interest. In addition to that uncertainty there is also uncertainty related to the initial conditions where the distribution of agents (e.g. number of agents and starting locations) and their attributes (e.g. walking speeds and response times) may also cause differences between trials. These uncertainties can be simulated by randomly sampling suitable distribution functions for those attributes. A simple random approach is adopted in Monte Carlo egress simulation although other possible sampling methods could be utilised [10].

Many simulations are needed to represent the range of potential behaviours which may be expected for a particular scenario. However, this is set against the potentially significant computational cost of running a stochastic egress model which an end user would wish to minimise. Thus, a major issue facing stochastic egress model users is determining when enough simulations have been performed to give an accurate representation of the modelled scenario and the statistics of interest. In evacuation modelling literature [11‐20] that examines this issue there are broadly two approaches suggested. The first approach is based on running a fixed number of simulations. The second approach is based on some dynamic assessment of the behaviour of the statistic(s) of interest.

When the first approach is used, a wide range for the required number of simulations has been suggested. For high speed train evacuation analysis, Li et al. [11] suggested that 10 simulations would suffice provided certain conditions were met. For large passenger ship analysis, the International Maritime Organisation (IMO) recommends that a minimum of 500 simulations should be used to determine the 95th percentile TET [12]. They additionally stated that a lower number of simulations would be allowable provided a suitable convergence methodology was followed [13]. For aircraft evacuation analysis, Galea [14] noted that 1000 simulations could typically be utilised to determine the 95th percentile TET. Meacham et al. [15] utilised 2000 simulations to construct a Cumulative Density Function (CDF) of the TETs for building scenarios. There is no consensus on the number of simulations used and there is generally little justification for the number of simulations proposed. The disadvantage of this approach is that it is unknown whether the number of simulations is sufficient to represent the range of behaviours that may exist. Furthermore, the number of simulations performed could be excessive, for the intended purpose, which could limit the number of potential scenarios or alternative designs that could be explored. This approach does have the advantage of requiring little additional modeller expertise but is also limited as some models may be very sensitive to the chosen number of simulations [16].

In the second approach, additional egress simulations are performed until some indicators of convergence have been satisfied. With this approach, the number of simulations performed is unknown a priori. The eventual number of simulations performed is based on the behaviour of the statistics of interest and should ensure that sufficient, and not excessive, computation being performed. Compared to running an arbitrary number of simulations this requires some expertise in determining suitable convergence tolerances for the scenario being run. Two types of indicators have been proposed, the first type is based on confidence intervals (CIs) [21, 22] and the second type is based on examining successive differences in the statistics of interest with additional simulation.

CI based techniques have been successively used in egress modelling [13, 20], and various other fields of study [23‐28], to define convergence. For large passenger ship analysis, Grandison et al. [13] calculate the CI of the 95th percentile TET, using a binomial distribution, and incrementally test, with additional simulations, the limits of the CI against a pass/fail criterion [12]. This technique was found to accurately determine whether a ship design had passed or failed, whilst minimising the computational effort required. Furthermore, they noted that this technique could be adapted to determine convergence based on a required precision and that the methodology could be extended to other suitable statistics of interest and types of egress analysis. A disadvantage of this method is that it can be difficult to determine CIs for some statistics. However, an advantage of this scheme is that the convergence indicators are directly linked to the precision of the statistics of interest.

A successive difference technique has been proposed by Ronchi et al. [17] and Ronchi and Nilsson [18]. Ronchi et al. [17] define convergence (i.e. sufficient repeat simulations have been performed) when their five statistics of interest (MT, SD and three functional analysis measures (FAMs) of the average total egress curve (AC)) were all changing by less than their specified tolerances per simulation over a set number of simulations. It was suggested that these tolerances could be based on the uncertainty of the available safe egress time from a fire modelling analysis. Additionally, Lovreglio et al. [19] proposed that those tolerances could be based on values determined from a set of repeated evacuation experiments when used to validate a stochastic egress model. An advantage of the successive difference approach is the simplicity in calculating the convergence indicators. However, Ronchi et al. noted that ‘The first limitation of the method is that it uses the concepts of convergence in mean and the central limit theorem rather than a statistical estimation of the expected values…’. This means that their convergence indicators do not directly relate to standard errors or CIs of the statistics of interest.

From the previous examples, the parameter of most interest is a representative TET. However, an analysis based solely on the TET may miss important details of the evacuation [16‐20, 29], e.g., a set of egress trials could have similar representative TETs, but the individual evacuation dynamics could be very different. A more detailed analysis of the evacuation can be performed by additionally examining the exit time of certain percentages of the agents [18, 20]. However, an even greater level of detail can be achieved by examining the entire agent egress curve (an ordered vector of the individual agent’s exit time) via FAMs [16, 17, 19, 29]. Furthermore, egress curves are a typical output from many evacuation models. The convergence behaviour of the AC was used to represent behavioural uncertainty, the uncertainty associated with the stochastic nature of human behaviour, by Ronchi et al.

CIs have been demonstrated to accurately and efficiently determine convergence for statistics of interest in egress modelling and other fields of study. Ronchi et al.’s use of the AC was an important innovation in studying egress variability. It is therefore desirable to extend their original method by applying CIs to the AC, thereby giving a more standard statistical interpretation of behavioural uncertainty. However, it is not intuitively obvious how to determine the CIs for this curve and no solution has been previously published. The key contribution of this paper is the development of FAM-based CIs for the AC (Sect. 3) with an accurate overall confidence level. This was achieved using the novel application of bootstrapping (see Sect. 2.1), FAMs (see Sect. 2.2), and a bisection algorithm (see Sect. 3.1). This addresses the first limitation of Ronchi et al.’s method and allows the benefits of the CI approach to convergence to be combined with the AC within a common framework. It is also advantageous to use the FAMs, to form the CIs, as they are somewhat familiar to the fire and evacuation research communities. FAMs were introduced to the fire safety field in 1999 [30] and the first reported use for evacuation was in 2012 [29]. A further contribution of this paper is the development of a convergence scheme. The convergence scheme is based on the same variables used by Ronchi et al. [17], i.e. AC, MT, and SD, but utilises CIs. The CI for the MT is based on the standard t-distribution expression (see Sect. 2) and the CI for the SD is obtained using a standard application of bootstrapping (see Sect. 2.1.1). This convergence scheme will efficiently determine when all the statistics of interest have reached their required statistical precisions. Utilising convergence of the AC helps to ensure that variability, between simulations, which is not expressed by examining TETs alone, is also captured.

This section provides a brief background on the necessary terminology and methods used in this paper. Initially, some standard statistical terminology will be described; this initial section can be skipped over by readers with a grounding in statistics. This is followed by an overview of “bootstrapping” with some worked examples (Sect. 2.1). Finally, a summary of functional analysis, used to compare egress curves, is given (Sect. 2.2).

A sample statistic is the value of an estimator based on a sample from the population of results. A population parameter can be considered as the value of an estimator based on the entire population of results. Examples of estimators include the mean TET (MT) (Eq. 1), the 95th percentile TET, and the standard deviation (SD) of TET. Ideally, one would want to know the parameter but practically it is (generally) only possible to obtain the statistic of a sample of results and a range of potential values for the parameter. The sample statistic would display variability depending on the nature of the sample obtained although the sample statistic tends to the value of the parameter as the size of the sample increases (Law of Large Numbers [22]).

The “expected value” of a sample statistic is the mean value of a series of infinitely repeated independent instances of that sample statistic with the same sample size [22]. This will equate to the corresponding population parameter if the estimator is unbiased. This is the case for many estimators although the sample SD is slightly biased. Whilst the standard variance bias correction has been applied (Eq. 2), i.e. dividing by (n − 1) rather than n, there is still some residual bias. It is reasonable to neglect this bias given the large scale of the variability associated with the SD compared to the small size of the bias and the complexity of calculating that bias. The small bias that does exist also diminishes as the sample size increases.

Although it is generally impossible to exactly know the population parameter it is possible to estimate a range of possible values for the population parameter based on the sample of results. This is a CI, which will differ between different samples but will frequently include the parameter of interest. Ideally, this frequency is determined by the confidence level (CL). However, the actual frequency, known as the coverage probability, may differ in practice. A set of CIs for 50 different samples is depicted in Fig. 1. This is illustrated for a 95% CL in Fig. 2 where the vast majority, 47 (~ 95%), of the sample CIs contain MT^(∞) but 3 (~ 5%) of the sample CIs do not. The superscript brackets indicate the number of samples used to calculate the statistic, when the number is ∞ the statistic is equivalent to the population parameter. In general, only a single sample of results will be generated, although that single sample could contain hundreds of simulations, so only one CI will be generated per statistic of interest.

The calculation of the CI for certain parameters is straightforward due to the availability of standard expressions. For the MT, the CI with a 100(1 − α)% CL can be calculated using Eq. 3 where T⁻¹(p, df) is the inverse t-distribution for percentile p with df degrees of freedom [22]. This expression assumes that the population SD of the TETs is unknown, which is generally the case in egress models/experiments, and is estimated with the sample SD. If the original data is normally distributed, then the expression is valid for n > 1. For non-normally distributed data, the expression is generally valid due to the central limit theorem for a larger n but is dependent on the nature of the original data distribution. Some sources advocate a minimum n of 30 although for skewed data a minimum n of 40 [31] has been suggested. It is further assumed that the data is independent and identically distributed.

In Eq. 3, the CI is quoted as a lower limit and an upper limit of the population parameter value, e.g. 150.0 with a 95% CI [135.0, 165.0]. The CI could also be quoted as the difference between the limits and the sample statistic, e.g. 150.0 with a 95% CI [− 15.0, + 15.0]. Those differences could also be quoted as percentages, e.g. 150.0 with a 95% CI [− 10%, + 10%]. If the CI is symmetric then it can be expressed in shortened form with a plus-minus limit, e.g. 150 ± 10% (95% CI).

A CI is a trade-off between its width, which should ideally be as narrow as possible, and its CL, which should ideally be as high as possible. Ideally, one would want a 100% confidence, but this leads to an infinitely wide CI which is not very useful. For a statistic calculated from a fixed number of data points, the CI width will increase with increasing CL. Having a very high CL will result in a very wide CI whilst a “low” CL with a correspondingly narrow CI is also unhelpful as there is little confidence that the CI contains the parameter of interest. A 95% CL is typically chosen as it is very likely to contain the population parameter whilst maintaining a relatively narrow width. When calculating the CI_MT (Eq. 3) with varying CLs (see Table 1), a 99% CL gains an extra 4% in confidence but requires a 33% increase in CI width compared to a 95% CL. A higher 99.9% CL gains an extra 4.9% in confidence but requires a 71% increase in CI width compared to a 95% CL. Using a lower 68% CL (equivalent to 2 × standard error widths) reduces the CI width by 49% but there is a reduction in confidence of 27% compared to a 95% CL. From Eq. 3, the width (W_MT) of the CI_MT with a fixed CL will also tend to narrow with increasing n (see Eq. 4).

In order to further discuss CIs, it is necessary to describe sampling distributions. A sampling distribution is the probability distribution of a statistic for a fixed sample size for all the possible samples that could be selected [22]. For example, a set of five TETs are obtained from repeated trials and the statistic of interest is the mean of those TETs. If this set of five trials is repeated, a further set of TETs can be obtained, and another mean of those values is calculated. Each time the process is repeated a different estimate for the mean value will be obtained, see Table 2.

If this could be repeated a very large number of times, then it would be possible to produce a distribution of the sample mean values, i.e. the sampling distribution. Assuming the TETs are normally distributed with an unknown population SD then it has been shown theoretically that the sampling distribution for the example can be represented by a t-distribution [22]. The mean of the sampling distribution is equal to the expected value and the population mean of the original TET distribution. From the central limit theorem, the standard deviation of the sampling distribution, also known as the standard error, is the standard deviation of the original TET distribution divided by √n.

From this sampling distribution, the interval that contains the innermost 100(1 − α)% of sample means can be described by Eq. 5 (see Fig. 2).

Equation 5 is not particularly useful as in general the value of the MT⁽ⁿ⁾ is known and the MT^(∞) is unknown. However, Eq. 5 can be algebraically recast [22] to make the MT^(∞) the subject of the inequality leading to the probability that the MT^(∞) lies within a CI with a CL of 100(1 − α)% (Eq. 6). This is a probabilistic statement of the CI given in Eq. 3.

For some statistics, such as the mean value (Eq. 3) and the 95th percentile value [13], a theoretical solution exists for determining the CI. However, for many statistics a theoretical solution may not exist or is difficult to derive. For those cases, it may be possible to use bootstrapping to approximate the sampling distribution and hence derive the CI.

The average egress curve AC⁽ⁿ⁾ = {ac₁, ac_k, …, ac_m} is calculated by averaging all n generated egress curves c_i (Eq. 26 shows the calculation for the kth component of AC⁽ⁿ⁾) from the simulations performed. A bootstrap average curve AC^* is derived by randomly resampling, with replacement, those n generated curves n times, and then averaging. This process is repeated B times and leads to a bootstrap sampling distribution of the average egress curve. Unfortunately, it is difficult to directly assign a CI to the sampling distribution of curves. However, it is possible to create proxies that represent the precision of the AC indirectly. This can be achieved by transforming the AC sampling distribution of curves into usable sampling distributions of point values, via the FAMs, that can be assigned CIs. Each bootstrapped average curve AC^* is compared to the average egress curve AC⁽ⁿ⁾ with the FAMs to create sampling distributions of those measures, i.e. ERD^*(AC^*, AC⁽ⁿ⁾), EPC^*(AC^*, AC⁽ⁿ⁾), and SC^*(AC^*, AC⁽ⁿ⁾). This approximates the (generally) unknowable sampling distributions of all possible sample average egress curves AC⁽ⁿ⁾ compared to the population (or expected) average egress curve AC^(∞) using the FAMs, i.e., ERD(AC⁽ⁿ⁾, AC^(∞)), EPC(AC⁽ⁿ⁾, AC^(∞)), and SC(AC⁽ⁿ⁾, AC^(∞)). The CIs of the FAM-based sampling distributions represent the precision of the AC.

The ERD sampling distribution can be represented by a semi-infinite distribution between zero to infinity as the ERD values are greater than or equal to zero. It is anticipated that the distribution would be concentrated near zero with the frequency tending to zero as the bootstrapped ERDs tend to infinity. When AC^(∞) is unknown the best estimate of ERD^*(AC^(∞), AC⁽ⁿ⁾) is zero as the best estimate of AC^(∞) is AC⁽ⁿ⁾. The 0th percentile of the ERD^*,o sampling distribution is zero. The CI is obtained by using the 0th and 100(1−α)th percentiles of the ordered bootstrapped values as the lower and upper limits for a confidence level of 100(1−α)%. The small sample size correction (Eq. 8) is applied to the 100(1−α)th percentile; this is achieved by modelling the bootstrapped ERD measures using a folded normal distribution from zero to infinity with the (unfolded and folded) mode at zero. The corrected 100(1−α)th percentile bootstrap index b_ERD of the ordered bootstrapped ERD^*,o values is given by Eq. 27 where Eqs. 28 and 29 map the percentile from the folded distribution to the full distribution and back again. The CI_ERD is [0, \(ERD_{{b_{\text{ERD}} }}^{*,o}\)].

The SC sampling distribution can be represented by a distribution between zero and one where it is expected that the values will be concentrated near one and the frequency will tend to zero as the SC values tend to zero. The best estimate of SC(AC^(∞), AC⁽ⁿ⁾) is one and the 100th percentile of the SC sampling distribution is one. The CI, with a confidence level of 100(1−α)%, is given by the limits of the corrected (100α)th and 100th percentile of the ordered bootstrapped SC^*,o measures. The corrected 100αth percentile bootstrap index b_SC is modelled, in a similar fashion to b_ERD, using Eq. 30 where Eqs. 31 and 32 map the percentile from the folded distribution to the full distribution and back again. The CI_SC is [\(SC_{{b_{\text{SC}} }}^{*,o}\), 1].

The EPC sampling distribution will vary between zero and infinity where it is expected that the values will be concentrated about one and the frequency tends to zero as EPC tends to either zero or infinity. This can be represented by a two-tailed distribution and the CI of the ordered bootstrapped EPC measures is calculated using the BCa method (Sect. 2.4). The best estimate of EPC(AC^(∞), AC⁽ⁿ⁾) is one. The specific equations for calculating the BCa limits for the EPC are obtained by substituting EPC into Eqs. 9–14 leading to Eqs. 33–37.where \({\text{AC}}_{ - i}^{\left( n \right)}\) is an average egress curve calculated using the original egress curves barring the egress curve from the ith simulation.

The CI_EPC is given by [\(EPC_{{b_{{{\text{l}},{\text{EPC}}}} }}^{*,o}\), \(EPC_{{b_{{{\text{u}},{\text{EPC}}}} }}^{*,o}\)] where b_l,EPC and b_u,EPC are given by Eqs. 36 and 37 respectively.

Now that CIs have been developed for the statistics it is possible to determine convergence w.r.t. the estimate of the population parameters. The CI_MT is calculated using Eq. 3, the CI_SD is calculated using bootstrapping (Sect. 2.1.1), and the FAM-based CIs of the AC are calculated using the methodology described in Sect. 3. The width of a CI narrows (converges) with increasing sample size, e.g. Eq. 4. This enables a convergent approach based on incremental testing where the number of trials can be progressively increased until the width of the CI is sufficiently reduced that it is less than the specified precision tolerance. The convergence of the MT, the SD, and the FAMs are determined by comparing the normalised width (W) of the CIs against a specified tolerance (Tol) (Eqs. 46–50) for a 100(1 − α)% CL.

The convergence algorithm is depicted as a flowchart in Fig. 5. The user needs to specify the above tolerances (Tol_MT, Tol_SD, Tol_ERD, Tol_EPC, and Tol_SC) as well as the minimum number of simulations (MIN_SIM), the maximum number of simulations (MAX_SIM), the number of bootstrap-resamples (B), the CL, and the number of simulations performed between testing (K). Convergence is determined when all the CI widths are less than their specified tolerances. Convergence may not be achieved within the MAX_SIM simulations, in that case the MAX_SIM simulations will be performed with the final CIs reported. If the user has no specific convergence requirements, then a set of values for the convergence attributes are suggested in Table 6. The tolerances are discussed in Sect. 6.

A theoretical case study model, similar to Ronchi et al. [17], was used to create multiple fictitious egress curves of 120 agents. The data is generated by pseudo-randomly [44] sampling a log-normal distribution with a mean of 12 s and standard deviation of 13.4 s (√180 s). A set of 120 values are generated {l₁, …, l₁₂₀} with the egress time (t) of the kth agent given by Eq. 51. Figure 6 depicts ten randomly generated egress curves using the case study model.

Using the case study model has two major advantages over using simulated or real data. Firstly, the generally unknowable population parameters are defined for the case study model so the methods can be tested and verified. The population MT^(∞) is 1440 s (120 × 12 s), as the mean of the sum is the sum of the means. The population SD^(∞) is 147.0 s (√120√180 s), as the variance of the sum is the sum of the variances and the standard deviation is the square root of the variance. The exit time of the kth agent of AC^(∞) is given by Eq. 52.

Secondly, there is very little computational cost in generating this data allowing a vast number of curves to be generated in a short timeframe. Virtually all the computational cost of the case study is incurred from the bootstrapping process rather than curve generation. In practice, it is anticipated that the computational cost of egress simulations (curve generation) will dwarf the computational costs of the bootstrapping process. In this test case study, the step size (s) used for the SC (Eq. 24) is one. This gives the most conservative value for the difference in shape between the curves.

The testing of the CIs is performed in three parts. The first part examines the behaviour of the CIs, within a single set of simulations, relative to the sample estimates of the statistics and the population parameters (Sect. 5.1). The second part examines the average behaviour of the CIs over multiple sets of simulations including the measured coverage probability and average CI width for a parameter at a specified sample size (Sect. 5.2). The third part examines the behaviour of the convergence scheme based on the CIs (Sect. 5.3).

The advantage of the convergence scheme developed here compared to Ronchi et al.’s scheme is the convergence is directly based on the required precision of the statistics of interest. The most intuitive tolerance to set is Tol_MT. It has been previously suggested that the tolerance for Ronchi et al.’s original method could be based on uncertainty from either a fire modelling study determining available safe egress time or uncertainty based on experimental egress trials [19]. Those techniques could also be applied to the CI based method provided those uncertainties are calculated using CIs. In lieu of that type of information, a typical choice for Tol_MT could be 0.02, the equivalent of ± 1% error, but the choice depends on the user’s requirements. However, the choice of tolerances for the other statistics SD, ERD, EPC and, SC are less apparent.

Prescribing appropriate tolerances for the ERD, EPC and SC FAM-based CIs is problematic, but the case study suggests a potential way forward. It was found that the W_EPC, with a 95% confidence level, and W_MT have a similar magnitude, W_EPC is approximately 10% wider than W_MT, for the same sample size (see Table 10) and the behaviour of the corrected CI limits of MT and EPC (see Figs. 7 and 10) are also similar. It was also shown that under certain conditions the true error measured by EPC and MT are approximately equivalent (see appendix). The ERD also has a behaviour and magnitude related to MT under certain circumstances. W_ERD is roughly half the magnitude of W_MT for the same sample size (see Table 10) and the upper confidence limit of the ERD and MT CIs are similar in magnitude and trend (see Figs. 7 and 10). It was also shown (see appendix) that the true error measured by ERD and MT are also equivalent under certain circumstances. In lieu of any other requirements, it is reasonable to base the tolerances of the ERD and EPC CIs on the specified tolerance of MT (Eqs. 66 and 67). Although there is no obvious relationship between MT and SC convergence behaviour, it is also suggested that Tol_SC is also based on the specified tolerance of MT (Eq. 68) unless the user has a specific requirement.

Setting the Tol_SD in relation to Tol_MT is not recommended. From Table 8, even with 4000 simulations the average W_SD is 0.049. By assuming that the TETs follow a normal distribution it can be calculated [22] that it would take ~ 10,000 simulations to achieve convergence for a Tol_SD of 0.02. Unless the user specifically requires a certain precision for the SD then it is suggested setting Tol_SD between 0.4 and 0.2.

There are circumstances where a user would wish to set the MT tolerance based on time, e.g. ± 10 s, Tol_MT,seconds = 20 s. The tolerance of the FAM-based CIs cannot be directly expressed in seconds, but this is not a particular problem as the Tol_MT,seconds is easily converted to a decimal (Eq. 69). This needs to be calculated at runtime as MT is generally unknown a priori.

In the previous case study, W_EPC/2 and W_ERD are slightly larger than W_MT/2. As W_ERD is greater than W_MT/2 then there is a greater relative variability expressed in the AC compared to the MT (see appendix). This is due to the nature of the theoretical case study model. The coefficient of variation (CV) of the exit time of the kth agent, for the case study model, decreases with increasing k (see Eq. 70). Thus, the earlier exiting agents exhibit the most relative variability. However, the actual variability of the kth agent, i.e. SD_k, increases with increasing k. The ERD gives greater weight to larger differences due to the squaring term (Eq. 22) leading to a small increase in W_ERD compared to W_MT/2. When performing practical egress simulations, there is potentially larger variation in the exit time of agents other than the last agent to exit which would be reflected in the FAM-based CIs of the AC.

Although the use of CIs addresses Ronchi et al.’s first noted limitation, the other limitations they discussed still apply to the work performed here. For instance, it is possible that egress curves can appear identical even when the dynamics of the evacuation are different. Furthermore, differing egress curves compared to the AC can return the same ERD, EPC or SC measures. The convergent behaviours of the FAM-based CIs of the AC are therefore imperfect proxies for ensuring all the variability in an egress simulation is adequately represented. However, these measures are superior to using the convergence of the MT alone when trying to determine whether enough simulations have been performed to represent the true variability of the evacuation scenario.

Ronchi et al. [17] pioneered the use of the convergence behaviour of the AC to better represent the convergence of the evacuation scenario in general. In this paper, their original approach was modified to use CIs. The CI approach has been shown to give reliable convergence with reference to the estimate interval of the population parameters when applied to the mean TET, standard deviation of TETs and the AC. The FAM-based CIs for the AC have been calculated using a novel application of bootstrapping, FAMs, and bisection algorithm. Although the methods described in this paper are more complex than Ronchi et al.’s original convergence indicators, the resultant CIs have a more standard statistical interpretation than their original convergence indicators. The choice of convergence tolerances is therefore more straightforward as it is just the required statistical precision of the statistics of interest. The case study and algebraic study (see appendix) of the ERD and EPC true errors showed there can be equivalency between those errors and the error in MT. These studies provided guidance for setting the tolerances of ERD and EPC.

The convergence scheme is easily adapted to include other statistics of interest. There are many more parameters that may be of interest, e.g. measures of congestion, that may not have a known CI. For example, Smedburg [45] suggested several statistics, such as egress exit flow rates, which were represented by time series data; FAM-based CIs and convergence for those statistics could be determined using the methods described in this paper. The calculation of the CIs and convergence should be implemented in software so that end users can have easy access to the technology.

The CIs can also be easily applied to experimental or simulated trial data without the convergence scheme suggested here where the number of trials performed is limited by other criteria.