In situ and in-transit analysis of cosmological simulations

The backdrop for the data post-processing methods described in this work is Nyx, a compressible gasdynamics/ N-body particle code for cosmological simulations of baryonic and cold dark matter (CDM) (Almgren et al. 2013; Lukić et al. 2015). Nyx characterizes the expanding universe using the Friedman equation, where \(a \equiv(1+z)^{-1}\), where z is the redshift, \(H_{0}\) is the Hubble constant, \(\varOmega _{\varLambda }\) is the cosmological constant, and \(\varOmega _{0}\) is the total matter content in the universe at \(z=0\). The continuity equation for the baryonic gas satisfies where \(\rho_{b} \equiv a^{3} \rho_{\mathrm{proper}}\) is the co-moving baryon density, and U is the proper baryonic velocity. The baryon momentum equation is where \(p \equiv a^{3} p_{\mathrm{proper}}\) is the co-moving pressure, and g is the gravitational acceleration vector. Nyx uses a dual-energy formalism to evolve the baryonic energy (Bryan et al. 1995): where \(E \equiv e + |\mathbf {U}|^{2}/2\) is the total energy, e is the internal energy, \(\gamma\equiv C_{P}/C_{V}\), and \(\varLambda _{\mathrm{HC}}\) represents all heating and cooling terms. This dual-energy formalism is necessary due to the numerical misbehavior inherent to hypersonic flows, where \(E/e \gg1\). At the end of each time step, Nyx synchronizes E and e, the method for which is determined by their relative values. The chemical equation of state for a mixture of hydrogen and helium, as well as the heating and cooling terms in \(\varLambda _{\mathrm{HC}}\), are described in Lukić et al. (2015).

The cold dark matter is treated as a pressureless, non-relativistic fluid, characterized by the Vlasov equation: where f is the phase-space distribution of dark matter, m is its mass, p is its momentum, and ϕ is its gravitational potential. Nyx solves the Vlasov equation using a collisionless N-body treatment of dark matter particles. Finally, the self-gravity of the simulation is treated using where G is the gravitational constant, and \(\rho_{0}\) the average of \(\rho_{b} + \rho_{\mathrm{dm}}\) over the whole domain. Both baryonic and dark matter contribute to and are affected by the total self-gravity of the system.

Nyx is based on BoxLib, an \(\mbox{MPI}+\mbox{OpenMP}\) parallelized, block-structured, adaptive mesh refinement (AMR) framework (BoxLib 2016). It evolves the two-component (hydrogen and helium) baryonic gas equations with a second-order accurate piecewise-parabolic method, using an unsplit Godunov scheme with full corner coupling (Colella 1990; Almgren et al. 2010; Almgren et al. 2013). It solves the Riemann problem iteratively using a two-shock approximation (Colella and Glaz 1985). The dark matter particles interact with the AMR hierarchy using a ‘cloud-in-cell’ method (Hockney and Eastwood 1988). The details of the numerical methods used to solve the above equations are provided in Almgren et al. (2013).

The Nyx code is fully implemented with AMR capabilities, including subcycling in time. The effectiveness of AMR has been validated in simulations of pure dark matter, as well as the ‘Santa Barbara cluster’ problem (Frenk et al. 1999; Almgren et al. 2013). However, the simulations presented in this work focus on the Lyman-α forest, which consists of large-scale systems in the intergalactic medium (IGM) which absorb radiation, preferentially Lyman-α, from distant quasars (Lukić et al. 2015). The IGM spans virtually the entire problem domain in these simulations, and as a result the signals of interest (optical depth, fluxes, etc.) are rarely localized to a specific region. As a results, it is generally impractical from a code performance perspective to use AMR at all in Lyman-α simulations, and as a results, all of our simulations here use a single level, with no AMR.

A typical Nyx simulation evolves a cosmology from high redshift (\(z > 100\)) to the present (\(z = 0\)) in many thousands of time steps. The size of each time step is limited by the standard CFL condition for the baryonic fluid, as well as by a quasi-CFL constraint imposed on the dark matter particles, and additionally by the evolution of the scale factor a in the Friedmann equation; details of these constraints are described in Almgren et al. (2013). Currently the largest Nyx simulations are run on a \(4\text{,}096^{3}\) grid, and at this scale each plotfile at a single time step is \({\sim}4~\mathrm{TiB}\), with checkpoint files being even larger; a complete data set for a single simulation therefore reaches well into the petascale regime. A single simulation can therefore fill up a typical user allocation of scratch disk space (\(\mathcal{O}(10)~\mathrm{TiB}\)) in just a few time steps. We see then that modern simulation data sets represent a daunting challenge for both analysis and storage using current supercomputing technologies.

Nyx simulation data lends itself to a variety of popular post-processing cosmological analysis tasks. For example, in galaxies and galaxy clusters, observations have indicated that dark matter is distributed in roughly spherical ‘halos’ that surround visible baryonic matter (Davis et al. 1985). These halos provide insight into the formation of the largest and earliest gravitationally bound cosmological structures. Thus a common task performed on cosmological simulation data sets is determining the distribution, sizes, and merger histories of dark matter halos, which are identified as regions in simulations where the dark matter density is higher than some prescribed threshold. A recent review (Knebe et al. 2013) enumerates 38 different algorithms commonly used to find halos. To process data from Nyx (an Eulerian code), we use a topological technique based on iso-density contours, as discussed in Section 4.1. The approach produces results similar to the ‘friends-of-friends’ (FOF) algorithm used for particle data (Davis et al. 1985).

A second common data post-processing task in cosmological simulations is calculating statistical moments of different fields, like matter density, or Lyman-α flux. The first two moments - the PDF and power spectrum - are often of most interest in cosmological simulation analysis. Indeed, it is fair to say that modern cosmology is essentially the study of the statistics of density fluctuations, whether probed by photons, or by more massive tracers, such as galaxies. The power spectrum of these fluctuations is the most commonly used statistical measure for constraining cosmological parameters (Palanque-Delabrouille et al. 2013; Anderson et al. 2014; Planck Collaboration et al. 2014), and is one of the primary targets for numerical simulations. In addition to cosmology, one may be interested in predictions for astrophysical effects from these simulations, like the relationship between baryonic density \(\rho_{b}\) and temperature T in the intergalactic medium, or details of galaxy formation.

To implement a simulation analysis tool in situ in BoxLib codes such as Nyx, one appends to the function Amr::CoarseTimeStep() ¹ a call to the desired analysis routine. All MPI processes which participate in the simulation execute the data analysis code at the conclusion of each time step, operating only on their own sets of grid data. As discussed earlier, the advantages of this execution model are that it is minimally invasive to the existing code base, and that it requires no data movement (except that inherent to the analysis itself). One potential disadvantage of in situ analysis is that if the analysis algorithm does not scale as well as the simulation itself, the execution time of the entire code (\(\mathrm{simulation} + \mathrm{analysis}\)) will suffer. Indeed, we encounter exactly this bottleneck when calculating power spectra, which we discuss in Section 4.2.

The in-transit implementation of data analysis codes in Nyx is more complex than the in situ approach, due to the necessary data movement and the asynchrony of the calculation. During initialization, BoxLib splits its global MPI communicator into two disjoint communicators, m_comm_comp for the group which executes the simulation, and m_comm_sidecar for the ‘sidecar’ group which performs the analysis. The user prescribes the sizes of each group at runtime, and the sizes are fixed for the duration of code execution. Upon reaching a time step at which analysis is requested, Nyx transfers via MPI the requisite data from the compute group to the sidecar group. Some of this data is copied to every sidecar process, e.g., the geometric information of the problem domain and how Boxes are arranged on the domain; to communicate such data, Nyx performs an intergroup MPI_Bcast(). The bulk of the data to be communicated consists of the floating-point state field stored in each Box; in BoxLib these data are called Fortran Array Boxes (FABs). Because we have two MPI groups and two communicators when executing in-transit (as well as an intergroup communicator connecting the two), we generate two ‘distribution maps’ for the simulation data, one describing the distribution of FABs across processes in the compute group, and the other in the sidecar group. This provides BoxLib with a bijective mapping of the FAB data distribution between the two groups, allowing us to perform point-to-point intergroup MPI_Send()s and MPI_Recv()s to transfer the data between corresponding processes in the two groups. We summarize the method for sending and receiving this data in Algorithm 1.

The distribution of data over each MPI group need not be the same, since each group can have arbitrary size. For example, if the simulation contains 4,096 Boxes and the compute group has 2,048 processes, each process will own 2 Boxes; however, if the sidecar group has only 256 processes, each process will own 16 Boxes. After the FABs have been sent to the analysis group, that group executes the desired analysis code, while the compute group continues with the simulation. A schematic of this Box movement across MPI groups is depicted in Figure 1.

The receipt of FABs onto a single MPI process is an inherently serial process. This property can affect code performance adversely if a large number of compute processes send their data to a small number of sidecar processes, because a compute process cannot continue with the simulation until all of its FABs have been sent, and each sidecar process can receive only one FAB at a time. In the example shown in Figure 1, process 5 receives FABs from processes 1 and 3; if, by coincidence, process 5 receives all four of process 3’s FABs in order, process 3 can continue with its next task before process 1; however, process 5 cannot continue until it has received all FABs from both processes 1 and 3. In this example the ratio of sending to receiving processes, \(R \equiv N_{s} / N_{r} = 2\), is relatively small; the serialization of this data transfer will have only a minor effect on aggregate code performance. However, if \(R \sim\mathcal{O}(100)\) or \(\mathcal{O}(1\text{,}000)\), the effect will be more pronounced.

Reeber is a topological analysis code, which constructs merge trees of scalar fields. A merge tree describes the relationship among the components of super-level sets, i.e., regions of the data set with values above a given threshold. The leaves of a merge tree represent maxima; its internal nodes correspond to saddles where different components merge; its root corresponds to the global minimum (Morozov and Weber 2013). In the case of Reeber operating on Nyx data sets, the points x and y correspond to distinct grid points \(\mathbf {r}_{1}\) and \(\mathbf {r}_{2}\), and the function f corresponds to the density \(\rho(\mathbf {r})\).

An illustration of Reeber’s halo-finding algorithm is shown in Figure 2(a). To identify iso-density-based halos efficiently, we traverse the merge tree upwards from the root (i.e., from the global minimum density), finding all edges that cross the value \(t_{\mathrm{boundary}}\). This operation corresponds to drawing the merge tree such that the height of each node corresponds to its function value and identifying all sub-trees above a line at the height of \(t_{\mathrm{boundary}}\). We then traverse each sub-tree to identify its highest maximum. If this maximum exceeds \(t_{\mathrm{halo}}\), the sub-tree represents a halo, and we compute its position as the centroid of all grid points belonging to the sub-tree as well as its mass as the cell-size-weighted sum of all grid points belonging to the sub-tree. The merge tree corresponding to the halo-finding parameters in Figure 2(a) is illustrated in Figure 2(b).

As a topological descriptor, a merge tree is primarily a means to an end: its utility comes from providing an efficient way to query a field for interesting topological information, e.g., are two regions of a field connected above a given threshold, or what is the number of connected components at density ρ whose density maximum is higher than \(\rho'\)? In Nyx simulations, one requires the answer to exactly these questions when identifying the locations, distribution, and merger histories of dark matter halos, as discussed in Section 2. Given the dark matter density field and its boundary conditions, Reeber first constructs its merge tree, and then uses it to identify halos based on user-defined density thresholds (Morozov et al., in preparation). The merge tree itself does not depend on any parameters, only on the input function. Accordingly, one can repeatedly query the same tree with different halo thresholds without reconstructing it each time. A recent result applying Reeber to various Nyx simulation data sets is shown in Figure 3.

Given that so many models and methods for generating halo mass functions already exist (Knebe et al. 2013), it is prudent to validate the iso-density approach used by Reeber by comparing with others. We have found close agreement with the FOF halo mass function (see Figure 3), although the two approaches for finding halos are quite different. Even more useful for comparison would be analytic or semi-analytic halo mass function results, such as the Press-Schechter formalism (Press and Schechter 1974); however, as shown in Lukić et al. (2009), the FOF model and the spherical collapse model used in the Press-Schechter function are incompatible.

Implementing a scalable representation of merge trees on distributed memory systems is a challenging but critical endeavor. Because the merge tree is a global representation of the entire field, traditional approaches for distributing the tree across independent processes inevitably require communication-intensive reduction to construct the final tree, which in turn lead to poor scalability. Furthermore, modern simulations operate on data sets that are too large for all topological information to fit on a single compute node. Reeber’s ‘local-global’ representation addresses this problem by distributing the merge tree, so that each node stores detailed information about its local data, together with information about how the local data fits into the global merge tree. The overhead from the extra information is minimal, yet it allows individual processors to globally identify components of super-level sets without any communication (Morozov and Weber 2013). As a result, the merge trees can be queried in a distributed way, where each processor is responsible for answering the query with respect to its local data, and a simple reduction is sufficient to add up contribution from different processes. A detailed description of merge trees, contour trees, and their ‘local-global’ representation, which allows Reeber to scale efficiently on distributed memory systems, is given in Morozov and Weber (2013), Morozov and Weber (2014) and its application to halo finding in Morozov et al. (in preparation).

Gimlet calculates a variety of quantities relevant for the intergalactic medium studies, which are derived from different fields in Nyx, including: We calculate the optical depth τ of Lyman-α radiation in the optically thin limit (Lukić et al. 2015): where \(n_{X}\) is the number density of neutral hydrogen, \(\sigma_{\nu}\) is the opacity to Lyman-α radiation, e is the elementary charge, \(m_{e}\) is the electron mass, c is the speed of light, \(f_{12}\) is the oscillator strength of the Lyman-α transition, \(\varDelta \nu_{D} \equiv(b/c) \nu_{0}\) is the Doppler width of the Lyman-α transition with the Doppler parameter \(b = b_{{\mathrm{thermal}}} \equiv \sqrt{2 k_{B} T/m_{H}}\), T is the gas temperature, and \(m_{H}\) is the mass of the hydrogen atom. In the optically thin limit, and absent scattering effects, τ becomes a purely local quantity (Mihalas 1978). Similarly, the flux due to Lyman-α radiation is also a local quantity, being a simple rescaling of the neutral hydrogen number density \(n_{X}\) (Lukić et al. 2015):

The most interesting quantities calculated by Gimlet are power spectra of the Lyman-α flux and matter density. Gimlet uses FFTW3 (Frigo and Johnson 2005) to calculate the discrete Fourier transformation (DFT) of the grid data required for power spectra. We note here also that FFTW3’s domain decomposition strategy for 3-D DFTs has implications which are especially relevant for a study of in situ and in-transit analysis performance. We discuss these in Section 5.

Given a 3-D scalar field, Gimlet calculates its DFT in two different ways:

As an example of Gimlet application, we show in Figure 4 a comparison between the observed Lyman-α flux power spectrum, and predictions from 3 Nyx simulations. We plot a dimensionless power spectrum calculated along the line of sight versus the wavelength mode k. We show here one redshift only using data from Viel et al. (2013), and we demonstrate how power spectra differ when changing the thermal velocity dispersion of the dark matter. The black line is the cold dark matter model which has no thermal velocity component in the initial state. The blue and red lines correspond to two different warm dark matter (WDM) models, \(m_{\mathrm{DM}} = 0.85~{\mathrm{keV}}\) and \(m_{\mathrm{DM}} = 2.1~{\mathrm{keV}}\), respectively. The main task of Nyx simulations with the Gimlet analysis pipeline is to determine which cosmological model and reionization history fits the best existing observational data, and Figure 4 is a simple example when we vary only one parameter out of ∼10.

In situ and in-transit methods attempt to circumvent the limitations of not only disk capacity, but also disk bandwidth. It is therefore of interest to this study to measure the time a code would normally spend saving the required data to disk in the traditional post-processing workflow. To do so, we measured the time to write grids of various sizes, each containing 10 state variables,² to the Lustre file system on Edison. BoxLib writes simulation data in parallel using std::ostream::write to individual files; it does not write to shared files. The user specifies the number of files over which to distribute the simulation data, and BoxLib in turn divides those files among its MPI processes. Each process writes only its own data, and only one process writes to a given file at a time, although a single file may ultimately contain data from multiple processes. The maximum number of files allowable is the number of processes, such that each process writes its own data to a separate file.

We varied the size of the simulation grid from 128³ to \(2\text{,}048^{3}\), divided among Boxes of size 32³. This is a small Box size for Lyman-α simulations which typically do not use mesh refinement; however, it is entirely typical for many other BoxLib-based applications which perform AMR. The maximum number of processes for each test was 8,192, although for tests which had fewer than 8,192 total Boxes (namely, those with 128³ and 256³ simulation domains), we set the number of processes such that each process owned at least 1 Box. We also varied the number of total files used, from 32 up to 8,192, except in cases where there were fewer than 8,192 total Boxes.

We illustrate the file write performance on Lustre in Figure 5. We find that, for the largest grid tested (\(2\text{,}048^{3}\)), the highest achievable write bandwidth on the Edison Lustre file system is \({\sim} 20~\mathrm{GiB}\mbox{/}\mathrm{s}\), about \((\sim)45\%\) of the peak bandwidth available on this file system. For the 512³ grid, which serves as our test case when we explore the performance of MPI traffic when running Nyx analysis codes in-transit in Section 5, the highest write bandwidth is \({\sim}5~\mathrm{GiB}\mbox{/}\mathrm{s}\), about 11% of peak bandwidth.

Before exploring the analysis workflow performance study which is presented in Section 5.4, here we first perform two simple studies which measure the time required to move grid data from one MPI group to another. In one test we used a 10-component grid of size \(1\text{,}024^{3}\), which has a total memory footprint of \({\sim}80~\mathrm{GiB}\). The grid was divided into Boxes of size 32³, yielding 32,768 total Boxes. We then fixed the total number of MPI processes at 8,192, and varied the number of analysis processes from 64 to 4,096 (with the size of the compute group varying from 8,128 to 4,096 processes). The results for this test are shown in Figure 6. In the second test, we fixed the number of total processes at 8,192 and also fixed the number of analysis processes at 1,024, leaving 7,168 processes in the compute group. We then varied the size of the grid to be transferred from 128³ (64 Boxes with total size 156 MiB) to \(2\text{,}048^{3}\) (262,144 Boxes with total size 640 GiB). The results are presented in Figure 7.

We see from this figure that the fastest bandwidth we achieve across the interconnect is \({\sim}58~\mathrm{GiB}\mbox{/}\mathrm{s}\). The peak bandwidth for the entire Aries interconnect on Edison is 23.7 TB/s (\({\sim}21.6~\mathrm{TiB}\mbox{/}\mathrm{s}\)) distributed across 5,576 compute nodes, indicating a peak bandwidth per node of \({\sim}4.0~\mathrm{GiB}\mbox{/}\mathrm{s}\). From our test, the highest bandwidth per node was \({\sim}116~\mathrm{MiB}\mbox{/}\mathrm{s}\), only about 3% of peak. Since several different configurations in this test appear to plateau at the same bandwidth, this limit may be due to high latency costs associated with sending and receiving so many small Boxes. Were the Boxes much larger, as they often are in Lyman-α simulations, the MPI traffic would consist of fewer and larger messages, which may achieve higher per-node bandwidth. However, even performing so far below the peak, the bandwidth of moving Boxes across the interconnect for the 512³ grid is still a factor of 10 faster than the fastest bandwidth achieved saving it to disk on the Edison Lustre file system (\({\sim}45\%\) of peak; cf. Figure 5).

From this simple study we can estimate that, in terms of pure transfer speed (neglecting analysis or simulation scaling behavior), the optimal ratio of compute to analysis on Edison is \(R \sim9\). For analysis algorithms which strong scale efficiently, this should be a useful metric. The poor bandwidth for the small grids in Figure 7 arises because there are more MPI processes than Boxes, so many processes do not participate in the transfer process; the simulation therefore has access to a smaller portion of the aggregate available bandwidth on the interconnect. In all cases, the total transfer time is on the order of seconds. If analysis is performed infrequently, this comprises a small component of the total run time of most Nyx simulations, which often take \({\gtrsim}\mathcal{O}(10^{5})~\mathrm{s}\). However as the frequency of analysis increases, the total time spent moving data can become considerable, and an in situ approach may become more attractive.

In-transit analysis introduces an additional layer of load balancing complexity, in that an optimally performant simulation maximizes the asynchrony of computation and analysis. To illustrate this point, suppose a simulation for evolving a system forward in time reserves C MPI processes for computation and A for analysis. Denote by \(\tau_{c}\) the time required for the compute processes to complete one time step of the simulation (without analysis), and \(\tau _{a}\) the time for the analysis processes to complete analysis for one set of data. If the user requests that the analysis execute every n time steps, then an optimal configuration of compute and analysis groups would have \(n \tau_{c} \simeq\tau_{a}\). If the former is larger, then the analysis group finishes too early and has no work to do while it waits for the next analysis signal; the converse is true if the latter is larger. If one finds that \(n \tau_{c} > \tau_{a}\), then one option is to decrease A, the number of analysis processes. This will increase the \(\tau_{a}\) but will simultaneously decrease \(\tau _{c}\) since we assume that \(C + A = \mathrm{const}\). A second option to equilibrate the two time scales is to decrease n, although this option may not be useful in all cases, since n is likely driven by scientific constraints and not code performance. If one relaxes the restriction that \(C + A = \mathrm{const.}\), then one can adjust the size of the analysis group arbitrarily while keeping the compute group fixed in order to balance the two time scales \(\tau_{c}\) and \(\tau_{a}\).

The risk of load imbalance described above traces its roots to the static nature of the roles of compute and analysis processes which we have assumed for this example, and which has traditionally been characteristic of large-scale simulations. Much of it could be ameliorated using dynamic simulation ‘steering,’ in which the simulation periodically analyzes its own load balance and adjusts the roles of its various processes accordingly. Returning to the above example, one may request an additional task from the analysis group: every m time steps, it measures the time spent in MPI calls between the compute and analysis groups (such calls remain unmatched while one group is still working). If \(n \tau_{c} > \tau_{a}\), then before the next time step the simulation re-sizes the MPI groups to increase C and decrease M, by an amount commensurate with the length of the time spent waiting for both groups to synchronize. An even more exotic solution would be to subsume all resources into the compute group until the nth time step, at which point the simulation spawns the analysis group on the spot, performs the analysis, and re-assimilates those processes after analysis is complete. Another possibility would be to decompose the tasks of simulation evolution and post-processing within a node, using OpenMP, for example. This may alleviate the data movement penalty, requiring data to move only across NUMA domains within a compute node (or perhaps not at all), the speed of which would be significantly faster than moving across the interconnect. The disadvantage of this approach would be that all codes involved in the simulation/post-processing workflow would need to contain a hybrid \(\mbox{MPI}+\mbox{OpenMP}\) implementation; for codes which are old or large, or which have a large ecosystem of already existing ‘companion’ codes for post-processing, this can be a laborious process. If OpenMP is not an option for controlling process placement on compute nodes, some MPI libraries provide mechanisms for placing processes on nodes by hand, although we are unaware of any truly portable method of doing so. The heuristics used to adjust the simulation configuration will likely be problem-dependent and will need to draw from a statistical sample of simulation performance data; we are actively pursuing this line of research.

Having established some synthetic benchmarks for disk and interconnect bandwidths, we now turn to the application of these workflows on science problems in Nyx. Here we present an exploratory performance analysis of in situ and in-transit implementations of both Reeber and Gimlet. We evolved a Lyman-α forest problem (Lukić et al. 2015) uniform grid (no mesh refinement) for 30 time steps, resuming a previous simulation which stopped at redshift \(z \sim3\). We chose this cosmological epoch because this is often when the most ‘post-processing’ is performed, due the wealth of observational data available for comparison (Heitmann et al. 2015; Lukić et al. 2015). The domain boundaries were periodic in all three dimensions. We ran the simulation in three different configurations: with no analysis, with Reeber, and with Gimlet.³ In simulations which perform analysis, the analysis code executed every 5 time steps, starting at step 1. We choose this configuration such that the last time step during which analysis is performed is number 26; this gives the sidecar group a ‘complete’ window of 5 simulation time steps (26-30) in which to perform the last analysis. Finally, we repeated this calculation on two different grid sizes, 512³ and \(1\text{,}024^{3}\), in order to explore the performance of these algorithms at different scales.

When running in situ, we ran the simulation and analysis on a fixed number of MPI processes (2,048 processes for the 512³ problem and 4,096 for the \(1\text{,}024^{3}\) problem). For the in-transit mode, we chose the number of MPI processes in two different ways. First, we fixed the number of total processes at the same number used for the in situ run, and varied the number of processes allocated to either the simulation or sidecars. This approach shows the optimal balance of each if the goal is to fit the simulation into a desired queue on a computational system which has a restriction on the maximum number of total processes. In the text which follows, we label this configuration ‘CT.’ Our second approach was to fix the number of processes devoted to the simulation at the same number used for in situ, and to vary the number of additional processes devoted to analysis. The total number of processes was then larger than the number used for the in situ run. This approach has in mind the case that the user wishes for the grid data to be distributed among the simulation cores in a particular way to preserve load balance, and that one is less concerned with the total number of processes being used. We denote this configuration as ‘CS.’ The details of each configuration are listed in Table 1. An example illustrates the influence of load imbalance across MPI processes: if we have 4,096 Boxes spanning the domain but only 2,047 processes evolving the simulation instead of 2,048, two of those 2,047 process must each operate on 3 Boxes, while the other 2,045 processes operate on only 2. However, all processes must wait for the two processes which are computing 3 Boxes. Thus, decreasing the computational resources by 0.05% increases the total run time by 50%.

In all simulations, we ran Nyx, Reeber, and Gimlet using pure MPI, with no OpenMP. We used version 3.3.4.6 of FFTW3, and compiled all codes with the GNU compiler suite, version 5.2.0, with ‘-O3’ optimization. In all DFT calculations, we used FFTW3’s default ‘plan’ flag, FFTW_MEASURE, to determine the optimal FFTW3 execution strategy.

We summarize our performance results in Figures 8 through 11. There we compare various components of the simulation running both Reeber and Gimlet in situ and in-transit. Specifically, we plot the total end-to-end run times in solid lines, the time for each post-processing step (every 5th time step) as dashed lines, and the time to evolve the simulation 5 time steps with dot-dashed lines. For visual clarity, the in-transit runs have markers at each data point, while the in situ lines are unmarked, as they represent a single data point. Each data point for the post-processing execution time is an average over 6 iterations; the standard deviations illustrated with error bars would be smaller than the line thickness and are thus not indicated in either figure. Each total run time is a single data point since each complete configuration was run only once. The lines labeled ‘CT’ used a constant number of total processes, such that, e.g., when using 2,048 total processes with 16 sidecar processes, the remaining 2,032 are running the simulation. Those labeled ‘CS’ have a constant number of processes working on the simulation alone, such that when using 16 sidecar processes and additional 2,048 for evolving the simulation, a total of 2,064 processes are running. The in situ lines are horizontal and independent of the number of sidecar processes; they use a fixed number of 2,048 total processes, all of which perform both simulation and analysis. The CT and in situ configurations, therefore, always use the same number of total processes, while the CS configurations use more. We also plot with the short-dashed purple line the total time to run the Nyx simulation without any analysis, running on 2,048 processes.

The times to post-process (dashed lines in each of the four figures) illustrate the strong scaling behavior of both analysis codes. The interplay among the different scalabilities presented here - those of the analysis suite, simulation code itself, and the combination of the two - are complex and require a nuanced interpretation. In Figure 8 we see that Reeber strong scales efficiently, although its performance begins to plateau at large numbers of MPI processes, due to the relatively small problem size (a 512³ problem domain). We also note that Reeber running in situ on 2,048 processes is actually slower than on 512 or 1,024 processes in-transit, which is due to having too many processes operating on too small a problem.

Figure 8 also illustrates the relationship between the scalability of the analysis code alone and that of the entire code suite (simulation+analysis). In particular, we see that for ≤64 sidecar processes performing analysis, a single Reeber analysis call takes longer than the 5 intervening simulation time steps. As a result, the strong scaling behavior of the complete simulation+analysis suite mirrors that of Reeber almost exactly. However, for ≥128 sidecar processes, Reeber is equal to or faster than the 5 simulation time steps, such that the scalability of the entire code suite begins to decouple from that of Reeber alone. This behavior has different effects for the CS and CT in-transit configurations. For the CS configuration, the scalability of the code becomes asymptotically flat for ≥128 sidecar processes, because Reeber completes before the 5 simulation time steps are done. Any number of additional sidecar processes above ∼256 is wasted in this mode. For the CT configuration, the end-to-end wall clock time begins to increase for ≥128 sidecar processes, because even though Reeber is executing faster and faster, increasing number of sidecar processes decreases the number of simulation processes, which slows the entire code down.

In light of this behavior, it is critical to find the configuration which yields the best performance for in-transit analysis. Deviating from this configuration leads either to wasted compute resources, or to a degradation of overall code performance. We note that the optimal setup for Reeber in both the CS and CT configurations for this particular Nyx simulation - using 128 to 256 sidecar processes for Reeber and the remainder for simulation - is faster than running the entire code suite in situ. In particular, the CS mode with \(2\text{,}048+256\) processes is faster than the in situ mode by \({\sim}30\%\) (\({\sim}70~\mathrm{s}\)). The CT mode is especially appealing, as it uses the same number of MPI resources as the in situ mode (2,048 processes total), and less than the corresponding CS configuration (2,176 to 2,304 processes).

Finally, we note that even the fastest post-processing configuration (256 sidecars in the ‘CS’ in-transit configuration) is still about \({\sim}35\%\) (\({\sim}40~\mathrm{s}\)) slower than running Nyx with no analysis at all. We identify at least two factors which contribute to this overhead cost. Firstly, the workload between the simulation MPI partition and the post-processing partition is not perfectly balanced; one is always waiting for the other. Secondly, when we run in-transit, we do not control the physical placement of MPI processes on the machine, whereas when running with no analysis, all processes do the same work and are more likely to be optimally placed together. Ideally, when post-processing, one would gather the compute partition as closely together as possible, and similarly for the post-processing partition. However, this is not guaranteed to occur, and indeed the data suggest that it indeed does not.

Gimlet’s scaling in Figure 9, however, is more complex. This is because Gimlet performs a variety of tasks, calculating both PDFs, which strong scale very well, and power spectra, which do not. Specifically, while BoxLib can decompose the problem domain into an arbitrary block structure, the only domain decomposition strategy which FFTW3 supports is to stripe along the first dimension in row-major array indexing, or the last dimension in column-major indexing. (FAB data in BoxLib uses the latter.) Therefore, the Nyx problem domain must be divided into chunks which span the entire x-y plane. Furthermore, the maximum number of chunks we can create is the number of grid points of the domain along the z-axis (512 in this case). Since FFTW3 allows only one MPI process to work on each chunk, we are therefore limited to a total of 512 processes which can participate in the DFT calculation; the remaining processes (1,536 when running in situ) are idle. This load imbalance becomes worse as the problem size grows: if using \({\sim}{100\text{,}000}\) processes to simulate a \(4\text{,}096^{3}\) grid, then \({\sim} {96\text{,}000}\) cores will be idle during the DFT calculation. This chunk decomposition problem has inspired the development of ‘pencil’-based decomposition strategies for 3-D DFTs which provide better scaling behavior (Habib et al. 2012; Habib et al. 2016). If we run Gimlet in-transit instead of in situ, however, we can address this problem by choosing a relatively small number of processes to participate in the DFT, leaving the rest to continue with the simulation. Similarly to the Reeber, case, when running with no analysis, Nyx is \({\sim}35\%\) faster than the best post-processing configuration (256 processes running in the ‘CS’ in-transit mode). We expect this gap to be the same because in the best CT configurations the code execution is limited by the simulation, and is independent of the post-processing algorithm.

The aggregate performance of Gimlet therefore represents a convolution of the scaling properties of both PDFs and power spectra. Although the power spectrum calculation scaling behavior quickly saturates as discussed above, we expect nearly ideal strong scaling behavior for the calculation of PDFs. Therefore, when we see in Figure 9 that the time for analysis increases between 256 and 512 analysis processes, this is due to the DFT calculations. In particular, when calculating the DFT of a 512³ grid on 512 MPI processes, each process has exactly one x-y plane of data. The ratio of work-to-communication may be low for such an extreme decomposition, leading to worse performance with 512 processes than with 256. Despite the jump in analysis time between 256 and 512 processes, however, the time decreases once again between 512 and 1,024 processes. When executing the DFT on 1,024 processes, we leave 512 idle, so the time for that component is likely exactly the same as it was with 512; in fact, the DFT calculation time will never decrease for >512 processes.⁴ The decrease in total analysis time is instead due to the continued strong scaling of the PDF calculations.

The relationship between the scalability of Gimlet and the entire code suite is different than for Reeber, chiefly because Gimlet execution takes significantly longer than does Reeber. For almost any number of sidecar processes in Figure 9, the time to execute Gimlet is longer than the corresponding 5 simulation time steps. (For 256 sidecar processes the two times are roughly equivalent.) As a result, Gimlet dominates the total code execution time, and the simulation+analysis strong scaling behavior follows that of Gimlet alone almost exactly.

As was the case with Reeber, we see in Figure 9 that some CS and CT configurations lead to faster end-to-end run times than in situ. Coincidentally, the threshold at which CT and CS runs become faster is between 64 and 128 sidecar processes, the same as Reeber. When using 256 sidecar processes, the CS configuration is \({\sim}65 \%\) faster than the in situ run, significantly larger than the 35% speedup seen with the CS mode when running Reeber. These values are functions both of the scalability of the analysis codes being used, as well as the total time they take to execute. The key point is that for both analysis codes, one can construct an in-transit MPI configuration which is significantly faster than running in situ.

Figures 10 and 11 show results for the same analyses performed on a larger \(1\text{,}024^{3}\) grid. When running with no analysis, we executed the code on 4,096 processes. In these tests we increased the number of MPI processes used in the in situ and CT in-transit configurations from 2,048 to 4,096, and the CS configuration used up to 8,192 total processes. The scaling behavior for Reeber running in all three modes is similar to that of the 512³ problem illustrated in Figure 8. When ≳512 processes are devoted to Reeber, it becomes faster than the 5 time steps of simulation, and the overall performance of the code plateaus, being bound by the simulation and not by post-processing. Gimlet running on the larger \(1\text{,}024^{3}\) problem also scales similarly to that of the 512³ problem shown in Figure 9. The characteristic jump in post-processing wall clock time when going from 512 to 1,024 processes running Gimlet is due to the slowdown of FFTW when decreasing the number of x-y chunks from 2 per process to 1. (In Figure 9 the jump occurred when going from 256 to 512 processes.) Because the FFTW scales poorly, we also find, as in the smaller run, that most in-transit configurations are faster than running in situ since fewer processes are wasted computing the DFT. Reeber’s fastest in-transit configuration - CS with \(2\text{,}048+1\text{,}024\) processes - is \({\sim}35\%\) faster than in situ. Gimlet’s fastest - CS with \(2\text{,}048+512\) - is also \({\sim}35\%\) faster than the corresponding run in situ.

We note also that for these larger problems the gap between the fastest post-processing configurations and the case with no post-processing is wider than for the 512³ problem. For the 512³ problems it was about \({\sim}35\%\), but for these \(1\text{,}024^{3}\) problems it is now about \({\sim}55\%\) (\({\sim}500~\mathrm{s}\)). This widening is likely due to exacerbation of the effects described above, particularly that the compute and post-processing MPI partitions are now spread even more widely across the machine, leading to larger MPI communication costs.