Machine learning applied to simulations of collisions between rotating, differentiated planets

Each collision is uniquely defined by 12 pre-impact parameters (Table 1). Together, these parameters define the geometry of the impact and the physical and rotational characteristics of the bodies involved in the collision. This set of parameters allows us to investigate the role of collisions in terrestrial planet formation, critically including the role of core mass fraction, rotation, and mutual orientation. The ranges of these parameters were chosen with two constraints in mind. First, the datasets should be focused on terrestrial planet formation. Second, and foremost for this work, the datasets should allow for a fair and robust comparison between distinct emulation strategies.

In order to satisfy the first constraint, we simulated collisions with total masses (\(M_{\mathrm{tot}}\)) between 0.1–2 Earth masses, which is of interest to late-stage terrestrial planet formation. The ratio of projectile mass to target mass (γ) was allowed to range from 0.1 up to equal-mass collisions (\(\gamma = 1\)). The resulting models range in mass from roughly a lunar mass up to nearly twice that of Earth.

The bodies involved in the collisions—referred to in this work as the target and projectile—are fully differentiated planets composed of an iron core and granite mantle. The mass fraction of the core relative to the body’s total mass is defined by \(F^{\mathrm{core}}_{\mathrm{body}}\), where the body subscript can refer to the target, projectile, largest post-impact remnant (LR), or second largest post-impact remnant (SLR). The core mass fractions of the target and projectile range from 0.1–0.9 (i.e., iron cores ranging from 10–90% by mass).

The target and projectile in the collisions are allowed to rotate. The rotation rates range from non-rotating to rotation at 90% the estimated breakup rate (\(\Omega _{\mathrm{crit}}\)). The estimated breakup rate is calculated according to Maclaurin’s formula for a self-gravitating fluid body of uniform density, where G is the gravitational constant and ρ is the bulk density of the body (Chandrasekhar 1969). Here, we calculate the bulk density of the body by using the mass and radius of the non-rotating model. Because the Maclaurin formula assumes a uniform density, the estimated breakup rate is more accurate for lower mass bodies and bodies with small core mass fractions. For high-mass bodies and those bodies with large core mass fractions, where the density profile strongly deviates from uniformity, the estimated breakup rate will be a lower bound. While the Maclaurin formula is a somewhat blunt approximation, it serves as a good estimate of the permissible rotation rates and therefore provides an upper limit for rotation rates in the pre-impact parameter space. We set the maximum rotation at 90% of the critical rate in order to avoid borderline unstable cases at lower masses. While it would be better to use empirically derived breakup rates for each model, such a study would require significant computational resources that were beyond the scope of this work.

The orientations of the target and projectile are uniquely defined by the obliquity (θ) and azimuth (ϕ) of their angular momentum vectors (i.e., rotation axes). These angles are allowed to vary between 0–180^∘ and 0–360^∘, respectively, where the obliquity is measured relative to the unit vector normal to the collision plane (ẑ) and the azimuth relative to a pre-defined reference direction (ŷ) in the collision plane. This allows for every possible mutual orientation between the target and projectile prior to impact.

In defining the pre-impact geometry of the collision, we depart from previous work by specifying the asymptotic impact parameter (\(b_{\infty }\)) and asymptotic relative velocity (\(v_{\infty }\)). In contrast, previous studies have generally used the associated quantities at the moment of impact (\(b_{\mathrm{imp}}\) and \(v_{\mathrm{imp}}\), respectively). However, this latter parameterization can result in unphysical initial conditions. Indeed, prior to impact, the mutual gravitational interaction between the target and projectile can alter their shapes, rotation rates, and relative orientations. This also alters the pre-impact trajectory and subsequent collision. This is due to the fact that both the target and projectile act as reservoirs of energy, whereby some fraction of the orbital energy in the pre-impact trajectory is transferred into the tidal deformation and rotational energy of the bodies. The simulations in this work therefore begin with the target and projectile separated by 10 critical radii, where the critical radius is given by \(R_{\mathrm{crit}} = R_{\mathrm{targ}} + R_{\mathrm{proj}}\). Note that we use the non-rotating radii of the target and projectile in calculating the critical radius. This parameterization avoids the degeneracy introduced by arbitrary mutual orientations of rotating bodies. Indeed, rapidly rotating bodies can take on significantly oblate shapes, increasing their radii and making a clear definition of the critical radius problematic when the orientations are taken into account. With respect to the data-driven models, this parameterization is ideal because it does not introduce any additional colinearity into the pre-impact parameter space. A parameterization of \(b_{\infty }\) that takes into account the orientations (θ and ϕ) and rotation rates (Ω) would introduce significant colinearity and was therefore avoided.

The parameter space investigated in this work is larger than any extant collision dataset known to the authors at the time of writing. Nonetheless, the parameter space is limited by computational resources and sampling requirements. It therefore does not yet include the full range of collisions relevant to planet formation, but does serve as a good training, test, and validation space for the emulators in this work. The emulation strategy developed in the work that follows easily allows for the parameter space to be expanded as computational resources become available.

In order to make a robust comparison between different emulation strategies, the underlying datasets must be well-sampled and well-behaved. However, generating a well-sampled training dataset in a high-dimensional parameter space is not a trivial task. The large number of dimensions quickly renders many approaches computationally infeasible. Indeed, a uniform grid sample would require \(n^{d}\) simulations, where d is the number of dimensions and n is the desired number of samples in each dimension. A low resolution 12-dimensional dataset with 10 samples in each dimension would then require 10¹² simulations, which is roughly eight orders of magnitude beyond current practical computational limits.

In order to overcome this problem while maintaining flexibility in the dataset requirements, we used a Latin hypercube sample (LHS) based version of the adaptive response surface method (LHS-ARSM) in order to sample a series of LHS (Wang 2003). Latin hypercube sampling is a statistical method for generating a near-random sample of parameter values from a d-dimensional distribution (McKay et al. 1979). LHS works on a function of d parameters by dividing each parameter into n equally probable intervals. The samples generated in this fashion are then distributed such that there is only one sample in each axis-aligned hyperplane. The advantage of this scheme is that it does not require additional samples for additional dimensions. LHS techniques have been used to considerable success in other high-dimensional astrophysical applications (Knabenhans et al. 2019).

In this study, the training dataset sizes required to reach optimal accuracies were not known a priori. Therefore, a procedure was needed to expand an existing dataset while maintaining certain properties, such as Latin hypercube, space-filling, and stratification properties. LHS-ARSM achieves this by sequentially generating sample points while preserving these distributional properties as the sample size grows. Unlike LHS, LHS-ARSM generates a series of smaller subsets that exhibit the following properties: the first subset is a Latin hypercube, the progressive union of subsets remains a LHS (and achieves maximum stratification in any one-dimensional projection), and the entire sample set at any time is a Latin hypercube. Benchmarking tests show that LHS-ARSM leads to improved efficiency of sampling-based analyses over older versions of ARSM (Wang 2003).

For the 12D_LHS10K dataset, we generated an initial LHS of 1000 collisions using the standard maximin distance criterion in order to guarantee space-filling properties. We then used LHS-ARSM to progressively enrich the sample in steps of 1000 collisions until we reached a total sample size of 10,000. We separately generated a 12D LHS sample of 500 collisions, designated 12D_LHS500, and a 12D LHS of 200 collisions designated 12D_LHS200. We subsequently used the 12D_LHS200 dataset to study the temporal convergence of the post-impact parameters, as a convergence study on the larger datasets was computationally infeasible.

In addition to the 12D datasets introduced above, we simulated two datasets of 500 collisions each, but with fewer dimensions. In the 6D_LHS500 and 4D_LHS500 datasets, the target and projectile are non-rotating, therefore fixing the rotational input parameters (Ω, θ, and ϕ for each body). In the 4D_LHS500 dataset, the core mass fractions of the target and projectile (\(F_{\mathrm{targ}}^{\mathrm{core}}\) and \(F_{\mathrm{proj}}^{\mathrm{core}}\), respectively) are additionally held constant at 0.33.

The 12D_LHS10K, 12D_LHS500, and 12D_LHS200, as well as the 6D_LHS500 and 4D_LHS500 datasets share the same parameter ranges (for those parameters that are varied) and therefore represent a composite sample in a shared parameter space. Superimposing LHS of the same dimension has the effect of increasing the resolution of the sample uniformly. However, superimposing LHS of different dimensions increases the resolution along specific hyperplanes of the higher dimension sample. Indeed, the resolution of the training dataset increases for values of the paramaters that are not varied in the lower dimension LHS (e.g., for core mass fractions of 0.33 in the case of 4D_LHS500).

While this causes the training dataset to deviate from truly uniform sampling, we found that the additional simulations provided a net improvement to the performance of our models. Moreover, the additional simulations provide increased resolution in the regions of the parameter space in which collisions are most likely to occur, which is a desirable feature in a training dataset. The choice to include the lower dimension LHS was therefore justified and tends to improve the performance of the models in the regions of the parameter space which are critical for planet formation.

In training the data-driven models in this work, it became clear that a large number of additional simulations were required at lower velocities. These additional collisions are necessary to sample both sides of the boundary between merging and hit-and-run collisions, which represents a relatively sharp discontinuity in the parameter space. We therefore simulated an additional 3384 simulations with asymptotic relative velocities between 0.1–1 \(\mathrm{v}_{\mathrm{esc}}\). The number of simulations in this region was chosen to match the resolution in the parameter space at higher velocities.

The relatively large number of simulations at lower velocities is due to the increased gravitational focusing radius, which grows rapidly at these velocities. Because we sample the asymptotic impact parameter (\(b_{ \infty }\)) in units of critical radii (\(R_{\mathrm{crit}}\)), more simulations are required at low velocities in order to maintain the desired resolution. As with the superimposed samples at higher velocities, the resulting increase in resolution at lower velocities is desirable because this is the region of the parameter space where collisions are most likely to occur. Moreover, it has the intended benefit of increasing the resolution of the training dataset around the transition region between merging and hit-and-run, which is a difficult transition to capture.

The datasets used in this work are summarized in Table 2 and were combined to create a composite training dataset. This dataset was used to train and test the data-driven classification and regression models in the work that follows.

The collisions in this work are pairwise collisions between a target and projectile, where the target is the more massive of the two bodies. In order to simulate collisions between these bodies using a particle-based method such as SPH, we had to first create suitable particle representations (i.e., models) of each body. We used ballic (Reinhardt and Stadel 2017) to generate non-rotating, low-noise particle representations of each body. The ballic code solves the equilibrium internal structure equations using the Tillotson equation of state (EOS) and can generate models with distinct compositional layers. In this work we investigated fully differentiated two-layer bodies with iron cores and granite mantles.

In order to facilitate collisions between rotating planets, we developed a method to induce rotation in the non-rotating models generated by ballic. The planets were first generated as non-rotating spherical models, after which a linearly increasing centrifugal force was applied to the particles in the rotating frame. The maximum centrifugal force applied to each particle is that which is required to achieve the desired rotation rate, \(F_{c} = m_{p} r_{xy} \Omega ^{2}\), where \(m_{p}\) is the particle mass and \(r_{xy}\) is the particle’s distance from the rotational axis. Once the maximum centrifugal force has been reached, \(F_{c}\) is held constant and the model is allowed to relax to a low-noise state. The particles are then transformed into the non-rotating frame and allowed to relax again. This method can spin-up a body up to its critical rotation rate (and beyond if not careful) and therefore allows us to probe collisions between rotating planets at any mutual orientation. An example of a model before and after the spin-up procedure is shown in Fig. 1. This represents a significant improvement over previous work, which has generally only considered collisions between non-rotating bodies.

The collisions in the datasets reported here have been simulated with Gasoline (Wadsley et al. 2004), a massively-parallel SPH code. The version of Gasoline used in this work has been modified specifically to handle planetary collisions and has been used in previous work to study the origin of the Moon, Mercury’s large core (Chau et al. 2018), and the ice giant dichotomy (Reinhardt et al. 2019). These modifications are described in detail in previous papers (Reinhardt and Stadel 2017; Reinhardt et al. 2019). Gasoline uses the Tillotson EOS (Tillotson 1962; Brundage 2013), which allows us to simulate collisions between differentiated planets with iron cores and granite mantles.

The resolution of the collisions in this work ranges from 20,000 to 110,000 particles. The resolution of each collision is set by the pre-impact mass ratio, whereby the smaller body (the projectile) is required to have \(N_{\mathrm{proj}} = 10\text{,}000\) particles. The particle mass is constant and therefore the larger body (the target) has \(N_{\mathrm{targ}} = 10\text{,}000/\gamma \) particles. The minimum mass ratio that we consider is \(\gamma =0.1\) and therefore the maximum resolution is 110,000 particles.

The simulations used in this work were simulated at the Swiss National Supercomputing Center (CSCS) and are publicly available in the Dryad repository: https://​doi.​org/​10.​5061/​dryad.​j6q573n94.

In this work, we consider a wide range of pre-impact conditions. This diversity of pre-impact conditions leads to a diverse set of post-impact states. The post-impact states for a subset of collisions in this work are shown in Fig. 2, wherein the collisions are roughly ordered by their pre-impact geometry. Collisions near the top left are high-velocity head-on impacts, whereas collisions near the bottom right are low-velocity grazing impacts. The range of collision outcomes required a robust script to retrieve the desired post-impact properties.

Every collision was evaluated for more than a hundred post-impact properties. We focus on a subset of these properties that are likely to prove important for N-body studies of terrestrial planet formation. These properties are listed in Table 3. In particular, we focus on the properties of the LR, SLR, and the debris field.

Collisions were simulated for a time equal to 100 times the timescale of the collision (100τ). The collision timescale τ is equivalent to the crossing time of the encounter and is given by, where \(v_{\mathrm{imp}}\) is the velocity at impact (see Appendix A) and we reiterate that \(R_{\mathrm{crit}}\) depends on the non-rotating radii of the colliding bodies.

In order to identify the post-impact LR, SLR, and debris field we used the SKID group finder (Stadel 2001). SKID identifies coherent, gravitationally bound clumps of material. It does this by identifying regions which are bounded by a critical surface in the density gradient (akin to identifying watershed regions). Then it removes the most unbound particles one-by-one from the resulting structure until all particles are self-bound. In this work, the minimum number of particles in a SKID clump was set to 10. This usually produces a much larger number of clumps than just the two that would correspond to the first and second largest remnants. For this reason the analysis routine checks if these clumps are further bound to either of the first or second largest clumps, if not, they are identified as part of the debris field of the collision.

In addition, in order to qualify as a remnant, the two largest SKID clumps are required to meet a minimum mass requirement. The largest clump only qualifies as the LR if its mass is greater than 10% of the target mass (\(M_{\mathrm{LR}} > 0.1~M_{\mathrm{targ}}\)). Similarly, the second largest clump only qualifies as the SLR if its mass is greater than 10% of the projectile mass (\(M_{\mathrm{SLR}} > 0.1~M_{\mathrm{proj}}\)). If one or both of the clumps does not meet the relevant mass requirement, then it is considered to be part of the debris.

A number of the post-impact properties investigated here do not have obvious definitions and require some explanation. We define or provide explanations for the post-impact parameters in Appendix A. In addition, we investigate the normalized masses to determine whether or not such normalization leads to improved regression performance for the data-driven techniques.

We evaluated the convergence of all post-impact properties considered in this work (Table 3) using the 12D_LHS200 dataset.² Convergence was measured relative to the post-impact quantity’s value at 100τ (the value used to train the emulators). In order to quantify the convergence, we calculated the absolute relative error E at uniformly sampled intervals of τ, where \(y(\tau )\) is the value of the post-impact parameter at τ and \(y_{100}\) is the value used in the training dataset. For a single post-impact quantity, this yields 200 measurements of E at each evaluated step of τ. The median of these relative errors is plotted as a function of τ in Fig. 3.

Most post-impact parameters have converged to within 1% of their training value by 50τ, however the radii (R), rotation rates (Ω), and debris angular momentum (\(J_{\mathrm{deb}}\)) are still converging at 100τ. We note that the non-convergence of the radii and rotation rates is the result of both numerical considerations and ongoing physical processes post-impact (e.g., differentiation, thermal equilibration, etc.). The choice of EOS in the SPH simulations is thought to a significant role in the convergence of the post-impact radii and rotation rates, which are coupled. However, at the time of writing the magnitude of this effect is not well understood.

The debris field generally provides a much smaller reservoir for angular momentum than either the LR or SLR. Therefore, ongoing exchange of angular momentum with one or more massive remnants generally has a large effect on the debris angular momentum budget, while at the same time having a negligible effect on the LR and SLR angular momenta. These properties (i.e., the post-impact radii and debris angular momentum) are therefore not suitable for training data-driven methods on account of their non-convergence. Future datasets based on longer simulations are required to determine the convergence timescale of these properties and may subsequently allow these parameters to be used in emulation tasks.

In order to track the rotation of planets during N-body simulations, a substitute for the rotation rate is therefore needed. We investigated the convergence of the rotational angular momenta of the remnants (\(J_{\mathrm{LR}}\) and \(J_{\mathrm{SLR}}\)) and found that they converge quickly following the impact. Indeed, following an impact, the angular momentum is quickly partitioned between the surviving bodies and has largely converged within a few tens of τ. While the debris angular momentum does not show the same convergence, it can instead be calculated implicitly from the angular momenta of the remnants and initial total angular momentum. We therefore suggest that N-body studies should utilize the angular momenta budget of the remnants to track rotation, rather than the rotation rates themselves.

A classification step is necessary to determine which post-impact properties need to be predicted by the regression models. The classification step must therefore determine which post-impact remnants are produced by the collision. In this work, we consider at most two post-impact remnants; the resulting classes are 0 (no remnants), 1 (one remnant; the LR), and 2 (two remnants; the LR and SLR).

The approach to collision emulation introduced here produces a single classifier and a set of single-target regression models, whereby each regression model is optimized for a specific post-impact property. With this strategy, the regression models are simple and achieve optimal accuracy for each individual post-impact property. However, the drawback of decoupling the post-impact quantities from one another is that the resulting regression models are agnostic to the underlying physical relationships and constraints between the quantities (e.g., mass conservation). It’s therefore not guaranteed that the emulator predictions will be physically self-consistent. In this paper, we focus on comparing the accuracy of regression strategies and in a forthcoming paper, we introduce a method for imposing physical constraints and self-consistency on the regression models.

PIM is an analytic method in which all collisions are treated as perfectly inelastic mergers.³ In a perfectly inelastic merger, the masses and momenta of the colliding bodies are conserved in a single post-impact remnant. There is no net conversion of kinetic energy into other forms such as heat, noise, or potential energy during the impact. This is the simplest possible model for emulating the outcome of a pairwise collision while maintaining physical self-consistency (but not accuracy).

The outcome of a perfectly inelastic merger is always a single remnant, which we refer to here as the LR for consistency. There are no additional remnants or debris. PIM can predict the mass and core mass fraction of the LR, and can additionally make naïve predictions of certain rotational parameters for the LR. PIM has been employed in the vast majority of N-body simulations to date. Details of our implementation of PIM can be found in Appendix B.

The impact-erosion model (IEM) is a semi-analytic model for gravity-dominated planetesimals (Genda et al. 2017). IEM predicts the normalized mass of the debris (\(M^{\mathrm{norm}}_{\mathrm{deb}}\)) as a function of the specific impact energy (\(Q_{R}\)) scaled to the catastrophic disruption threshold (\(Q^{\prime \star }_{\mathrm{RD}}\)). The normalized mass of the debris \(M^{\mathrm{norm}}_{\mathrm{deb}}\) is expressed as, where \(\phi = Q_{R} / Q^{\prime \star }_{\mathrm{RD}}\). IEM assumes that only a single remnant is produced by the collision (referred to as the LR for consistency) and therefore \(M^{\mathrm{norm}}_{\mathrm{LR}}\) can be determined via a straightforward relation, \(M^{\mathrm{norm}}_{\mathrm{LR}} = 1 - M^{\mathrm{norm}}_{\mathrm{deb}}\). For consistency, we use the same values of \(Q_{R}\) and \(Q^{\prime \star }_{\mathrm{RD}}\) in the calculations of IEM and EDACM. Details of the calculation of \(Q_{R}\) and \(Q^{\prime \star }_{\mathrm{RD}}\) used here and in EDACM can be found in Appendix C.

EDACM is a set of analytic relations that predict the masses of the LR, SLR, and debris, as well as the core mass fraction of the LR (Leinhardt and Stewart 2012) via a mantle stripping law (Marcus et al. 2010). In order to evaluate and compare the performance of EDACM to the other emulators developed in this work, we implemented EDACM as prescribed in Leinhardt and Stewart (2012). EDACM has been used in several recent N-body studies of planet formation (Carter et al. 2015; Quintana and Lissauer 2017). Most notably, EDACM allows for collision outcomes with more than one remnant (referred to as fragmentation) and is thus capable of predicting a larger set of post-impact parameters than either PIM or IEM. We give a brief overview of EDACM in Appendix C and explain where our implementation differs from that used in previous studies.

PCE is a popular technique in the field of UQ, where it is typically used to replace a computable-but-expensive computational model with an inexpensive-to-evaluate polynomial function (Ghanem and Spanos 1991). In this work, we use a PCE based on tensor products of Legendre polynomials (Benner et al. 2017). Recent work has demonstrated that data-driven PCE models can yield point-wise predictions with accuracies comparable to that of other machine learning regression models (e.g., neural networks) (Torre et al. 2019). In this work, we use UQLab (Marelli and Sudret 2014) to train and evaluate all PCE models. The documentation for UQLab is freely available at https://​www.​uqlab.​com/​documentation. An overview PCE as used in this work is provided in Appendix D.

GPs are a generic supervised learning method designed to solve regression and probabilistic classification problems (Rasmussen and Williams 2005). They are a non-parametric method that finds a distribution over the possible functions \(f(x)\) that are consistent with the observed data. ML algorithms that involve a GP use a measure of the similarity between points (the kernel function) to predict a value for an unseen point from training data. The Gaussian radial basis function (RBF) kernel is commonly used, however in this work we test multiple kernels, including the constant, Matérn (\(\nu =3/2\)), rational quadratic, and RBF kernels (see Table 4).

A potential downside of GPs is that they are not sparse (i.e., they use all of the sample and features information to perform the prediction) and they lose efficiency in high dimensional spaces (Rasmussen and Williams 2005). While our 12-dimensional space is relatively small for GPs, the number of training examples is much larger than that for which GPs are generally employed. More advanced algorithms have been suggested to improve the scaling of GPs, such as bagging and enforced sparsity, but we have not attempted to implement these here. A brief mathematical introduction to GPs is provided in Appendix E.

XGBoost (XGB) is an open-source decision-tree-based ensemble ML algorithm that uses a gradient boosting framework (Chen and Guestrin 2016). It has become one of the most popular ML techniques in the previous years and is well documented. Gradient boosting is a machine learning technique for regression and classification problems which produces a prediction model in the form of an additive expansion of simple parameterized functions h (typically called weak or base learners) (Friedman 2001). These base learners are usually simple classification and regression trees (CART). In gradient boosting, the base learners are generated sequentially in such a way that the present base learner is always more effective than the previous one. Thus, the overall model improves sequentially with each iteration. A detailed overview of the XGB models used here is available in Appendix F.

MLPs are a class of feed-forward deep neural network that consist of multiple, fully-connected (i.e., dense) hidden layers. In MLPs, the mapping f between the pre- and post-impact parameters is defined by a composition of functions \(g_{1}, g_{2}, \ldots, g_{n}\) (n being the number of layers in the network), yielding, where each function \(g_{i}(w_{i},b_{i},h_{i}(\cdot ))\) is parameterized by a weights matrix (\(w_{i}\)), a bias vector (\(b_{i}\)), and an activation function (\(h_{i}(\cdot )\)). The weights matrix and bias vector are the parameters of the network that are tuned by minimizing a loss function which measures how well the mapping f performs on a given dataset. In this work, the MLPs are implemented with Python’s Keras library and models consist of an input layer with 12 nodes, one to three hidden layers with up to 24 nodes each, and an output layer with a single node (i.e., a scalar output). All activation functions in the resulting network are the Rectified Linear Unit (ReLU). A detailed overview of the MLPs used in this work is provided in Appendix G.

In order to provide training and test labels for the classification models, we encode collision outcomes as integers. These labels depend on whether the model is a binary classifier or multiclass classifier. In binary classification, the labels encode whether the remnant (LR or SLR, depending on the task) exists or not, whereby the labels are 0 (does not exist) or 1 (exists). In multiclass classification, the outcomes are encoded as 0, 1, or 2, where the label corresponds directly to the number of post-impact remnants. These labels are defined for all collisions and the classification models will therefore always leverage the full training dataset.

Of the post-impact properties that we consider in this work, the mass and angular momentum properties are always defined as either zero (in the case where the associated remnant doesn’t exist) or a finite number. However, in the case of all other post-impact properties, the property’s value will be undefined if the associated remnant does not exist. Therefore, before training the regression models, it is necessary to first remove entries from the dataset where the target value is undefined.

Undefined entries occur when either the LR or SLR was not produced by a collision. This is often the case in head-on, high-velocity impacts, after which only debris is present, and in the case of mergers, in which no SLR survives. Because collision outcomes with an LR are more common than those with both an LR and SLR, the resulting training and test set sizes for the regression models will differ between LR, SLR, and debris properties. The training set size will therefore be largest for debris (which is produced in all collisions) properties, smaller for LR properties, and smallest for SLR properties. However, to reiterate, this is only the case for properties not related to the mass or angular momentum, which are defined in all cases.

In both classification and regression tasks, the pre-impact quantities are standardized to improve training efficiency and performance. For regression tasks, the post-impact quantities are standardized in the same manner as the pre-impact quantities.

The procedure for standardizing the input data differs between PCE and the other data-driven methods. In the case of PCE, the input parameters are linearly mapped into a hypercube \([-1,1]^{12}\), within which the distribution of the transformed features is still uniform.

For the other methods, the pre- and post-impact parameters are scaled using the standard scaling method. The result of standardization (a.k.a. Z-score normalization) is that the features will be rescaled such that they evince the properties of a standard normal distribution, \(\mu = 0\) and \(\sigma = 1\), where μ and σ are the mean and standard deviation of the distribution, respectively. The z-values are then calculated as,

Standardization is a general requirement for many ML algorithms. The only family of algorithms that are scale-invariant are tree-based methods (e.g., XGB). However, since we are comparing several different ML algorithms here, some of which depend strongly on standardization, we standardize the input and output features for all techniques (except as noted above for PCE).

The classification and regression performances reported in Tables 5 and 6, respectively, are for models trained on the full training dataset (\(N=11\text{,}884\)). However, for the purpose of investigating performance as a function of dataset size, we have sub-sampled the training dataset to create a series of smaller datasets. These subsets were generated by drawing random samples from the training dataset while the holdout test dataset remains unchanged.

We created training subsets with set sizes increasing in steps of 100 up to 1000 and from thereon in steps of 1000 up to 11,000. Note that there is a difference between the training set size (TSS) and the effective TSS on which the regression models are actually trained. Because we remove undefined values in the pre-processing step, the effective TSS is dependent on the post-impact property in question. The effective TSS is therefore generally lower than the TSS for LR quantities and even lower for SLR quantities. To reiterate, this is because the number of remnants depends on the initial conditions of the collision. Outcomes with an LR are more common than outcomes with both an LR and SLR. This also affects the holdout test dataset. This is important because the effective test set size (\(N_{\mathrm{test}}\)) determines the expected variance σ of the performance measures,

In order to evaluate the performance of our classification models, we consider two metrics. The first, the accuracy score, is simply the fraction of correct predictions over the total number of predictions, where predictions are either true positives (TP), true negatives (TN), false positives (FP), or false negative (FN). As we are not more concerned by either false positives (FP) or false negatives (FN), this metric is well suited evaluating our classification models.

However, while the accuracy quantifies the rate of correct predictions, it does not give any information as to the nature of the incorrect predictions. We therefore also consider the distribution of mass residuals resulting from the incorrect predictions (FP and FN predictions). Given two classification models with identical accuracy, the model with the lower mean and standard deviation in its residual distribution is preferred.

There are several commonly employed regression metrics that are not suitable for collision emulation due to the range of the post-impact properties. For example, mean squared error (MSE) is not scale invariant and relative error metrics are ill-suited to the many parameters that can take on null values. For this reason, we use the coefficient of determination, known as the \(r^{2}\)-score, to measure the quality of the regressors, where \(\mathrm{SS}_{\mathrm{res}} = \sum_{i} (y_{i} - \hat{y}_{i})^{2}\) is the residual sum of squares and \(\mathrm{SS}_{\mathrm{tot}} = \sum_{i} (y_{i} - \bar{y})^{2}\) is the total sum of squares. Here, \(y_{i}\) is the ith expected value, ȳ is the mean of the expected distribution, and \(\hat{y}_{i}\) is the ith predicted value. The \(r^{2}\)-score has been used as the performance metric in similar work (Cambioni et al. 2019) and is therefore a prudent choice in order to make comparisons to other studies.

In addition to the \(r^{2}\)-score, which quantifies the regression performance globally, we also consider the residuals as a function of each individual pre-impact property. Because we consider 12 pre-impact properties, 27 post-impact properties, and four data-driven models, the number of residual plots is in excess of a thousand. We therefore provide the residuals for a single post-impact property (accretion efficiency) at the end of the paper (Figs. 8–11) and provide the remaining residual plots as Additional file 1.

Ideally, in order to evaluate the performance of our emulation method, we would evaluate the performance of the classification and regression models together, as a unified emulation pipeline. However, for many of the post-impact properties, false positive (FP) and false negative (FN) predictions in the classification stage result in meaningless regression predictions which cannot be evaluated by the regression metric. When evaluating the regression models, we must therefore be careful to distinguish which models reflect the classification performance in their \(r^{2}\)-scores and which do not.

Sobol’ indices (Sobol’ 1993; Le Gratiet et al. 2016) measure how sensitive a given post-impact parameter is to each of the individual pre-impact parameters, as well as to any of their interactions. The indices quantify the relative contribution of variance explained by one variable—or group of variables—to the total variance, where \(S_{i_{1}\dots i_{s}}\) is the Sobol’ index of order s. The first order Sobol’ indices are the values \(S_{i}\) which characterize the variance explained by the variable \(x_{i}\). The higher order Sobol’ indices (second order \(S_{ij}\) with \(i\neq j\) etc.) quantify how much variance is explained not by single variables but rather by their interactions.

The Sobol’ indices are a particularly useful sensitivity measurement tool in the context of PCE because a Sobol’ decomposition can be computed directly from a PCE by employing a simple reordering of terms. Hence the computation of Sobol’ indices from a PCE is analytic and exact. For a more thorough introduction to Sobol’ sensitivity analysis we refer to the following references (Marelli et al. 2017; Le Gratiet et al. 2016).

To understand how our models are making certain predictions we use the SHAP framework proposed in Lundberg and Lee (2017). This is based on Shapley values (Roth 1988), introduced in a game theory context as a solution to fairly distributing gains and costs of a given game v to a set of collaborating players N. The Shapley value ϕ of one player i is the average expected marginal contribution of player i after all possible combinations of other players (denoted as S) have been considered:

Analogously, in the context of model interpretability, the game v is how well the model output is represented (for a fixed input x) and the set of players N are the features. In Lundberg and Lee (2017), the game \(v(S)\) is defined as the conditional expectation \(E(f(x)|x_{S})\), for model f, observation x and \(x_{S}\), the observation in which the features coincide with observation x on the set of features S. To avoid the necessity to train many models that include or exclude features to evaluate \(v(S)\), specific model based approximations can be used. In our work, SHAP values are computed from the gradient boosting models as described in Lundberg et al. (2018).

The analytic and semi-analytic methods investigated in this work achieved relatively poor \(r^{2}\)-scores relative to the data-driven methods. While limited to a narrow set of parameters, IEM is the most accurate of these methods for LR properties, where EDACM performs significantly worse. PIM performs worst, with the notable exception that it excels at predicting the core mass fraction of the LR.

The analytic and semi-analytic methods’ regression performances on \(M_{\mathrm{LR}}\) are significantly below that of the data-driven methods, achieving \(r^{2}\)-scores of 0.7698 and 0.6932 for IEM and EDACM, respectively. Their relative performance is somewhat surprising, as EDACM uses an explicit relationship to predict \(M_{\mathrm{LR}}\), whereas IEM only predicts \(M_{\mathrm{deb}}\) and provides no explicit relation for \(M_{\mathrm{LR}}\). PIM does poorly when predicting \(M_{\mathrm{LR}}\). This latter result is perhaps not surprising, as PIM assumes all collisions result in perfect accretion and studies have shown that this is not the case in most collisions (Quintana et al. 2016).

Of the analytic and semi-analytic methods, only EDACM is capable of making explicit (non-zero) prediction for \(M_{\mathrm{SLR}}\) (\(M^{\mathrm{norm}}_{\mathrm{SLR}}\)). The resulting \(r^{2}\)-score, 0.0773 (−1.3057), is much worse than the associated score for its prediction of \(M_{\mathrm{LR}}\) (\(M^{\mathrm{norm}}_{\mathrm{LR}}\)). EDACM’s significantly worse performance when predicting the mass of the SLR as opposed to the LR is likely influenced by two important aspects of the EDACM algorithm. First, EDACM delineates collisions into multiple regimes (e.g., perfect merging, hit-and-run), in which different analytic relations are used. Second, the calculation of \(M_{\mathrm{SLR}}\) uses \(M_{\mathrm{LR}}\) as an input (via \(M^{\mathrm{norm}}_{\mathrm{LR}}\); see Eq. (25) in Appendix C). Thus, any error in the prediction of \(M_{\mathrm{LR}}\) will propagate to the prediction of \(M_{\mathrm{SLR}}\). Note that the data-driven models do not suffer from this issue, as the prediction of the post-impact properties are entirely decoupled from each other.

In the case of the debris properties, only IEM explicitly predicts the mass. IEM predicts \(M^{\mathrm{norm}}_{\mathrm{deb}}\), from which \(M^{\mathrm{norm}}_{\mathrm{LR}}\) is subsequently derived. IEM’s prediction of \(M^{\mathrm{norm}}_{\mathrm{deb}}\) is surprisingly good with an \(r^{2}\)-score of 0.8448, but still approximately 10% lower than that of the data-driven methods. This reverse approach taken by IEM, first predicting the \(M^{\mathrm{norm}}_{\mathrm{deb}}\), allows it to make an accurate, if implicit, prediction of \(M_{\mathrm{LR}}\), relative to the other analytic and semi-analytic methods.

We additionally compared the ability of the analytic and semi-analytic methods to predict the normalized mass quantities. In the case of the LR, this resulted in significantly worse performance for these methods. Similarly for IEM and EDACM, the \(r^{2}\)-scores are significantly lower when predicting the normalized masses of the LR and SLR, but are similar for the debris mass. The poor performance of the analytic and semi-analytic methods on the normalized quantities is expected as a side-effect of how the \(r^{2}\)-score is calculated. Because the normalized quantities are scaled by the total mass of the collision (\(M_{\mathrm{tot}}\), which is different for each collision), the distribution of \(M_{\mathrm{tot}}\) skews the predicted distribution of \(M_{\mathrm{LR}}\). Thus, the normalized quantities are only of interest to the data-driven methods, which predict the normalized masses directly and therefore don’t suffer from this issue.

The core mass fraction of the LR (\(F^{\mathrm{core}}_{\mathrm{LR}}\)) is predicted by both PIM and EDACM (via a mantle stripping formula (Marcus et al. 2010)). Here, PIM performs unexpectedly well, yielding an \(r^{2}\)-score of 0.5549. PIM’s unexpected performance on \(F^{\mathrm{core}}_{\mathrm{LR}}\) provides physical insight into the processes that determine \(F^{\mathrm{core}}_{\mathrm{LR}}\), suggesting that the cores of pre-impact bodies often merge. In contrast, EDACM yields an objectively poor \(r^{2}\)-score of −0.0792 for \(F^{\mathrm{core}}_{\mathrm{LR}}\), despite utilizing a more complicated formulation.

For both \(F^{\mathrm{core}}_{\mathrm{LR}}\) and \(M_{\mathrm{SLR}}\), a large factor in EDACM’s poor performance are the collisions that comprise the super-catastrophic disruption (SCD) regime (Leinhardt and Stewart 2012) (see Appendix C). In Fig. 5, it’s clear that \(M_{\mathrm{LR}}\) is systematically under-predicted for a subset of collisions, which corresponds to the SCD regime. The poor predictions in this subset of collisions are propagated to the calculations of both \(F^{\mathrm{core}}_{\mathrm{LR}}\) and \(M_{\mathrm{SLR}}\), causing the former to be systematically over-predicted and the latter to be under-predicted.

In addition to the data-driven methods, only PIM makes any prediction of rotational properties. These predictions were not expected to be very accurate, given the assumptions of the model (see Appendix B). Indeed, the resulting regression performances are exceptionally poor. As Fig. 13 illustrates, PIM tends to greatly overestimate the angular momentum budget of the LR (\(J_{\mathrm{LR}}\)), which results in similar overestimates of its rotation rate (\(\Omega _{\mathrm{LR}}\)). This has the opposite effect on \(\theta _{\mathrm{LR}}\), which is systematically underpredicted by PIM. The obliquities are predicted to be low because the angular momentum delivered by the impact tends to dominate the resulting angular momentum budget.

The method for handling debris in the N-body implementation of EDACM (Chambers 2013) performs poorly relative to the data-driven methods as well. This is unsurprising given the simplifying assumptions of the debris model (see Appendix C). This would suggest that more accurate models for handling debris within N-body simulations are sorely needed.

The data-driven methods universally evince better accuracy than the analytic and semi-analytic methods. Of the data-driven methods, the MLP and XGB models generally achieve the best performance, but are often matched by the PCE models. The GP models, on the other hand, generally perform significantly worse than the other data-driven methods.

The data-driven predictions for each post-impact property are plotted relative to their true (i.e., simulated) values in Fig. 5 for LR properties, Fig. 6 for SLR properties, and Fig. 7 for debris properties (and the accretion efficiency).

In Figs. 8–11, we show the prediction residuals for accretion efficiency resulting from each of the four data-driven methods. The distribution of residuals is an important consideration in addition to the \(r^{2}\)-score, as it can reveal dependencies of the residuals on individual pre-impact properties. The most common residual dependence revealed by these plots (see Additional file 1) is that which corresponds to the boundary between the merging and hit-and-run regimes. This manifests as increased residual values at low velocities (\({\approx}1~\mathrm{v}_{\mathrm{esc}}\)). This dependence tends to be particular pronounced for GP models, which are not able to capture the relative sharp transition between these regimes.

Given the large number of plots required to illustrate the residuals, we show only those for a single post-impact property (accretion efficiency) here and provide the remaining residuals in Additional file 1.

In many cases, the performance of the GP models is below that of the other data-driven models. The lower \(r^{2}\)-scores for GPs are likely, at least in part, a result of the limitations on HPO for GPs. Recall that HPO is only carried out for GP models on training datasets with sizes of \(N \leq 1000\). Due to these limitations, the GP models are not fully optimized on the full training dataset, while the other data-driven methods are.

For different post-impact properties, the best achieved accuracy can differ significantly. Given that the different data-driven techniques are able to achieve the indistinguishable accuracy, this suggests that the difficultly in reaching higher accuracy lies not with the emulation methodology, but rather with the data or the underlying physical processes that determine the post-impact quantity. In the former case, this may be due to insufficient fidelity of the simulations, insufficient resolution of the training dataset, or ill-defined parameterizations of the post-impact properties.

A known source of uncertainty in the post-impact quantities is the post-impact group finding step. In subsequent steps, the group finding algorithm can assign particles to a group to which they were previously not a part of. While the number of these particles is almost always small (on the order of a few), this can have a large effect on the calculation of post-impact quantities, especially for remnants or debris fields composed of a small number of particles.

Parameters whose accuracy are likely affected by the underlying physical process are, for example, the obliquities. In this case, the limitation on performance may be a result of the obliquity (via the angular momentum vector) being highly variable at low rotation rates. Another set of parameters affected in this way are likely those related to the debris field spatial distribution (e.g., \(\bar{\theta }_{\mathrm{deb}}\) and \(\bar{\phi }_{\mathrm{deb}}\)). It may be that these quantities are inherently noisy as a result of being sensitive to small changes in the impact geometry. Parameters such as these may benefit from being separated into distinct outcome regimes.

The Sobol’ indices in Fig. 13 suggest that, for most post-impact properties, the geometry and energy of the impact—determined by γ, \(b_{\infty }\), and \(v_{\infty }\)—are the strongest factors in deciding the outcome of a collision. However, for some post-impact properties, other pre-impact parameters are important. This is true for the obliquities and core mass fractions, which are generally dependent on the pre-impact values of the associated body—i.e., the target for the LR and projectile for the SLR. Pointedly, the Sobol’ analysis also shows that the azimuthal orientation (ϕ) of the pre-impact bodies tend to play an insignificant role in the outcome of collisions.

As opposed to the global view provided by the Sobol’ indices in the previous section, the SHAP values provide a local view of feature importance for each post-impact property. In Fig. 14, we show SHAP values on the test set for a selected subset of post-impact properties of the LR and SLR.

In predicting the masses of the remnants (\(M_{\mathrm{LR}}\) and \(M_{\mathrm{SLR}}\)), the XGB models leverage the total (\(M_{\mathrm{tot}}\)) and impact geometry (γ, \(b_{\infty }\), and \(v_{\infty }\)). The other pre-impact properties—those related to the internal and rotational properties of the target and projectile—appear to play little role in the predictions. The feature importance metrics are largely intuitive in this case, indicating that lower total masses (\(M_{\mathrm{tot}}\)) lead to lower predictions of the remnant masses. Similarly, head-on, high-velocity impacts drive the predictions toward lower remnant masses, which is expected in disruptive collisions.

In the case of the remnant core mass fractions, the SHAP values indicate that the most important pre-impact property is the core mass fraction of the associated pre-impact body (i.e., the target for the LR and projectile for the SLR). This is also somewhat intuitive and matches the results of the Sobol’ analysis.

The remnant obliquities (\(\theta _{\mathrm{LR}}\) and \(\theta _{\mathrm{SLR}}\)) are relatively difficult properties to regress. These post-impact properties show a strong dependence on the pre-impact rotation rate (Ω) and obliquity (θ) of the associated body. Once again, the Sobol’ indices indicate the same feature importance. For the obliquity of the LR (\(\theta _{\mathrm{LR}}\)), the impact velocity (\(v_{\infty }\)) is also important.