A Comparative Study of the Efficacy of Local/Global and Parametric/Nonparametric Machine Learning Methods for Establishing Structure–Property Linkages in High-Contrast 3D Elastic Composites

Advances in materials science and engineering have served as critical enablers for technology leaps in multiple industries, including information technology, aerospace, automobile, and energy [1‐4]. As specific examples, nickel-based superalloys have increased efficiency in gas turbines [3, 5, 6], amorphous soft magnetic materials have lowered transformation losses in the electrical power distribution systems [4], and Li-ion batteries have made possible zero-emission electric vehicles [7]. However, the successful deployment of advanced materials in commercial products following their initial discovery in the laboratory often takes one to two decades. In part, this is due to the lack of a rigorous mathematical framework for the design of new and improved materials customized for selected applications. This current deficiency has been recognized in multiple recent strategic initiatives aimed at realizing advanced materials with reduced cost and time [1, 8].

Rigorous approaches to materials design require computational strategies that efficiently solve inverse problems. More specifically, materials design aims to identify the combinations of material chemistries and manufacturing/process conditions that would result in a targeted combination of material properties in the final product. In most cases, the design space that needs to be explored to facilitate these inverse solutions spans an extremely large and high-dimensional space. Formally, in order to accomplish this grand challenge, one needs to first establish robust and reliable process-structure-property (PSP) linkages [9‐14]. It is very important that these PSP linkages are established in forms that allow application of established design methodologies (i.e., inverse solutions) [15‐20].

In this work, we will focus our attention on the establishment of structure–property linkages. Generally speaking, process–structure linkages are significantly harder to establish as they demand attention to the details of material structure evolution during the imposed process conditions. We will also limit our attention here to the material structure at the mesoscale, generally referred to as the microstructure. Composite theories [21‐25] offer an important avenue to formulate computationally efficient microstructure-property linkages. However, their success has been largely limited to highly idealized microstructures that can be effectively quantified using a small number of parameters such as phase volume fractions and average shape factors. Most microstructures encountered in advanced materials defy such simplistic descriptions. Composite theories have also encountered significant challenges in arriving at sufficiently accurate predictions of the homogenized properties in high-contrast composites (these are composites whose microscale constituents exhibit widely varying local properties) [26‐29].

Finite element (FE) approaches offer a sophisticated toolset for incorporating the intricate details of 3D microstructures along with highly complex details of the governing physics in arriving at estimates of homogenized (macroscale) properties [21, 30‐33]. However, the use of this toolset demands significant computational resources. At the present time, these toolsets are not practically viable for addressing directly the material design problems demanding inverse solutions. Therefore, the most logical approach is to generate a sufficiently large collection of simulation datasets covering a wide variety of microstructures using the FE toolsets, and then extract computationally low-cost, reduced-order models from such datasets. Although this strategy requires modest computational resources in order to conduct the FE simulations on a variety of microstructures of interest, it incurs a one-time only computational effort. When properly designed and executed, this strategy has the potential to produce major time and cost savings in addressing the inverse problems central to the materials design efforts.

The approach described above based on reduced-order models has enjoyed remarkable success in low- to moderate-contrast composites [31, 34‐36]. The overall effort involved in establishing reduced-order microstructure–property linkages can be broken down conveniently into two main subtasks. The first subtask involves rigorous statistical quantification of the material microstructure (often referred to as feature engineering in data science terminology). Recent work [31, 35, 37, 38] has demonstrated that this subtask can be effectively handled through principal component analysis (PCA) of n-point spatial correlations computed for each microstructure included in the study. The second subtask involves model building using various machine learning approaches. For these models, the descriptors of the microstructure (e.g., the PC representations of their spatial correlations) serve as inputs and the effective macroscale properties of interest serve as outputs. The simplest of the model building approaches employs a large suite of easily accessible linear regression toolsets [31, 35, 39, 40] with or without regularization (especially needed when only a small number of data points are available). On the other hand, it is also possible to deploy highly sophisticated approaches based on convolution neural networks (CNN) [41‐44] that completely skip the feature engineering step (the first subtask mentioned above) and rely mainly on the power of a sophisticated network of models in capturing the intricate nonlinearities present in the model being built. However, CNN approaches require a fairly large number of data points before arriving at a reliable model with good predictive accuracy for new microstructures [41, 42].

The selection of the appropriate machine learning approach is often a nontrivial task, especially in addressing materials design problems. As already mentioned, the number of available data points for training and testing the model is one of the major considerations in this selection. Most regression techniques need a large number of training data points, and this requirement usually grows exponentially with the number of inputs (i.e., microstructure descriptors). Consequently, whenever possible, it is desirable to seek a low-dimensional representation of the inputs (e.g., the PC representations of spatial correlations used to represent the material microstructure). A second consideration in the selection of an appropriate machine learning approach is the availability of a suitable model form that could be justified by pre-existing domain knowledge (or the governing physics for the phenomena being studied). A third consideration in the selection of the appropriate machine learning approach lies in the need for rigorous quantification of the uncertainty in the prediction. In this regard, it should be noted that the error resulting from the use of the reduced-order model in place of the rigorous physics-based numerical simulation represents only one of the contributors to the uncertainty in the model predictions. This is because uncertainty in the prediction can also come from uncertainty in the governing physics or in the values of the parameters used in the physics-based model. For situations where it is important to quantify rigorously the uncertainty in the predictions, one generally prefers the use of approaches employing Bayesian inference [30, 36, 37, 45]. Based on these considerations, one typically selects a global/local (i.e., the predictions for new inputs are made either using all available data points or a selection of available data points in close proximity to the prediction data point), parametric/nonparametric approach (i.e., using a specified model form or not using any specified model form), or a Bayesian/non-Bayesian approach to model building. Given the complexities associated with each of the resulting choices, the selection of an appropriate machine learning toolset is often nontrivial in most applications. In the present application for extracting microstructure–property linkages in high-contrast elastic composites, there has not yet been a systematic study of the relative merits of the different approaches on the resulting models.

The difficulty in establishing reliable and robust structure–property linkages for high-contrast composites stems from the highly nonlinear and complex interactions that occur between the microscale constituents for any imposed macroscale loading condition. There currently do not exist any validated model forms that can accurately capture these complex interactions. Furthermore, the microstructure space (i.e., the complete set of all microstructures of interest) is an extremely large space. As a result, global approaches (a single model addressing the complete microstructure space) to model building have not yet yielded satisfactory results. Although CNN approaches [41, 42, 46] have shown significant promise, they place extremely high demands on computational resources. The computational burden involved in training a sophisticated CNN becomes even more dramatic if one considers exploring different combinations of layers used in CNN and retraining the network in the event of adding additional data points to the training ensemble.

The goal of this work is to compare objectively the efficacy of the different machine learning methods that adopt different combinations of local/global, parametric/nonparametric, and Bayesian/non-Bayesian approaches for establishing structure–property linkages. The remainder of this paper will be organized as follows. We first describe the dataset including an ensemble of microstructures with a wide range of morphological diversity and their analyses using previously established finite element methods. Then, the generation of microstructure features in the form of reduced-order representation of the 2-point spatial correlations of the microstructures in the ensemble studied here will be summarized. We will identify and review the specific modeling approaches that will be compared in this study. Finally, the relative benefits from the use of the different model building strategies for predicting the effective elastic properties of high-contrast composites will be critically assessed for accuracy and robustness.

The 3D material microstructure and its effective elastic stiffness predicted by the FE method constitute the input and output for the structure–property linkage to be built in this work. In other words, one microstructure and its FE-predicted effective stiffness constitute one data point for building and validating the desired reduced-order model. In order to facilitate this model building process, a large ensemble of voxelized 3D microstructures with broad morphological diversity were synthetically generated; these will be referred to as microscale volume elements (MVEs; [42, 46, 47]) in this paper. The workflow used to generate this ensemble of synthetic microstructures is depicted in Fig. 1. For simplicity, the workflow is illustrated with 2D microstructures. The process illustrated in this figure is trivially extended to the 3D microstructures generated for this study. The microstructure generation process starts by assigning a random number within the range of [0,1] to every voxel in the MVE (2D or 3D). Then, a Gaussian filter is applied on this random number field while treating boundaries as periodic to produce periodic (discretized) fields (i.e., the values wrap around the boundaries without any marked discontinuities). The desired discretized MVEs are obtained by simply thresholding the filtered image to attain a pre-selected volume fraction.

For the MVE generation process described above, the mean and covariance of the Gaussian filter determine the microstructure morphology obtained in the generated MVE. A 3D Gaussian filter with a mean μ (a 3X1 vector) and a covariance Σ (a 3 × 3 symmetric positive definite matrix) can be expressed mathematically as:

The mean vector can be selected arbitrarily and is taken as a zero vector. The components of Σ play a critical role in controlling the shapes of the constituents (e.g., elliptical phase regions) in the microstructure. The off-diagonal components of Σ were assigned zero values for all microstructures generated in this study. This essentially implies that the principal frame of the microstructure is fully aligned with the sample axes of the MVEs. However, if one desires to produce microstructures whose principal axes are misaligned with the sample frame, it can be accomplished by using nonzero values for the off-diagonal terms in Σ. In the 2D example shown in Fig. 1, the Gaussian filter has a larger variance in the y direction. Hence, the filtered and the segmented images show an elongated feature in the y direction. Sixty-four different 3D Gaussian filters (using different values for the diagonal terms in Σ) were generated to include a wide range of microstructure morphologies in the generated MVEs. A total of 140 different 3D random number fields were generated and the 64 Gaussian filters were applied on each to provide a total of ≈ 8900 3D filtered images. Each filtered image (see top right plot of Fig. 1) was binarized using a threshold value designed to yield a selected volume fraction of the white phase sampled from a uniform distribution in the 25–75% range.

The material system of interest in this study is a high-contrast elastic 3D composite. The higher contrast in the individual properties of the microscale constituents in such composites is expected to lead to longer range complex and nonlinear interactions at the microscale. In prior works involving low- to moderate-contrast composites [48, 49], MVE sizes of 21 × 21 × 21 voxels were found to be adequate for capturing these microscale interactions accurately. For the high-contrast composites, significantly larger MVEs of size 51 × 51 × 51 voxels were employed. This will ensure that the MVEs can be treated as RVEs (representative volume elements) in the FE prediction of their effective stiffness. The microscale constituents (i.e., depicted as white and black phases in the segmented MVE) are both assumed to exhibit isotropic elastic response. A contrast of 50 was obtained by setting the Young’s moduli of white and black phases as E₁ = 120 GPa and E₂ = 2.4 GPa, respectively. On the other hand, Poisson ratios were kept the same for both phases, i.e., ν₁ = ν₂ = 0.3. The targeted property of interest for each MVE was selected as the effective stiffness parameter, C_11,eff. The value of C_11,eff for each MVE was obtained by executing a FE simulation imposing suitable periodic boundary conditions that would render all macroscale strain components except 〈ε₁₁〉 0 (angular brackets represent the volume average). Other details of the FE simulations can be seen in prior reports [48, 49].

FE simulation of each MVE takes around 15 min with 4 CPUs and 32 GB of memory on a supercomputer. Since the number of distinct microstructural configurations is an extremely large set, it is not practical to explore the microstructure space with FE simulations. The only practical approach is therefore to first train a high-fidelity reduced-order model, and then to use that model to explore the large microstructure design space.

We briefly present the details of the microstructure quantification framework employed in this work, which includes the use of the 2-point spatial correlations (often simply referred to as 2-point statistics) and PCA.

In this work, we seek to study critically the efficacy of different combinations of local/global, parametric/ nonparametric, and Bayesian/non-Bayesian strategies in arriving at accurate and robust reduced-order structure–property linkages for two-phase high-contrast elastic composites. We will briefly review next the relevant model building approaches.

Four distinct reduced-order models were generated in this work for capturing the elusive linkages between the microstructure and the effective stiffness in high-contrast elastic composites. In building each model, both the dimensionality and the size of the dataset were systematically varied to compare and contrast the efficacy of each model building strategy. In the present work, the dimensionality is controlled by the number of PC weights (R^∗) utilized. The size of the dataset is varied according to the fraction utilized (F). For instance, F = 0.2 implies that the reduced-order model is built using a randomly selected 20% fraction of the full dataset.

Model efficacy was quantified by considering a 10-fold cross-validation strategy. The employed validation strategy was established such that each of the reduced-order models was tested on unseen data over all N observations. In this way, we mitigate against certain validation realizations producing anomalous results due to chance predictions at “easy” or “hard” regions in \(\mathcal {X}\). The following strategy was employed: (1) the data is randomly partitioned into F and 1 − F fractions, (2) each of these fractions is divided into 10 folds, (3) the model is trained on 9 folds of the F fraction, (4) predictions are made on the left-out fold in F and one of the folds in the 1 − F fraction, and (5) the process is repeated until predictions are made at all N points. Note that some of the considered models require the estimation of hyperparameters using cross-validation (e.g., the regularization terms in the EN model). In these cases, the training folds of the F fraction are further “internally” split into cross-validation folds for parameter estimation.

The following four modeling strategies were selected for this study.

In this work, our choice of GP nonparametric regression models is motivated by our desire to not only make predictions but also derive interpretable physical knowledge. As will be shown, the values of the inferred hyperparameters will yield physical information about underlying spatial relationships. There are many other choices of nonparametric regression models, which we have considered in this work. Random forest models are ubiquitous [80], yet challenging to interpret. Bayesian adaptive regression trees are similar to random forest but include the use of a Bayesian regularization criterion, which bolsters predictiveness [81], but yet again is difficult to interpret. Therefore, although there are many classes of nonparametric regression models, we focus on GPs as they enable both future predictions and interpretations of existing data.

For this study, we selected the MSE and mean absolute error (MAE) as the error metric to evaluate predictive performance. MSE can be expressed as:

where \(\hat {y}(\boldsymbol {x}_{i})\) is a prediction made on unseen data using the previously described cross-validation strategy. MAE is similar to Eq. 12; however, squared quantities are replaced by absolute values.

The MSE and MAE from the best of each of the models obtained in this study as well as results from a recent paper using 3D convolutional neural network (CNN)[42] are summarized in Table 1. The prior CNN work, however, only reported MAE values and therefore there is no MSE value for comparison. MSE is useful if one is interested in putting additional emphasis on the predictions with higher residuals. If there are a few predictions with significantly high residuals, MSE will quickly shift to larger values due to the squared residual term. MSE can be especially misleading in noisy observations. On the other hand, MAE weighs all predictions equally, meaning it does not penalize the predictions considered outliers as bad as MSE. MAE is easier to interpret since it provides the direct residuals while MSE gives a sense of spread of the predictions in terms of residuals and helps focus on outlier predictions. Note that the 3D CNN does not utilize 2-point statistics and instead directly considers the entire 51 × 51 × 51 voxel microstructure representation. This approach enables the possibility of inferring higher order statistics through the filter weights which the CNN estimates from the data [41]. Hence, the improvement in predictive accuracy obtained through the 3D CNN is evidence that perhaps important information is available in the higher order statistics. It should however be noted that deep learning approaches such as CNN require significantly larger amounts of training data compared to standard machine learning applications. Furthermore, the computational effort involved in CNN training is also significantly higher, and often requires GPU computing. This is because the CNN accepts the entire 3D microstructure as the input, not the low-dimensional measures of the microstructure as employed in this work. It is also important to recognize that the extracted CNN is not easily interpretable.

Note that there are varying degrees of correlation scale among the differing principal components as shown in Fig. 4. Consider that the median roughness parameter in the GPR model for PC₁ is 𝜃₁ ≈ 0.78. Therefore, for any two points in PC space, (x₁,x₂), the correlation function expressed as a product of exponentials is:

The correlation between points differing by Δ₁ = 0.78^− 1/2 ≈ 1.13 in PC₁ will be penalized multiplicatively by exp(− 1) ≈ 0.368. Consider now as an example the influence of PC₈. Solving \(\theta _{8}{{\Delta }_{8}^{2}}= 1\) illustrates that to contribute the same amount of correlation decay \({\Delta }_{8} = \theta _{*}^{-1/2} = 3.1\). In other words, the data is much more sensitive to changes in PC₁ than PC₈. This agrees with the visually clear global trend in Fig. 3 where PC₁ appears to have the greatest impact on the response variable when compared to PC₂ and PC₃. The significance of this observations is that predictions of y (x) will favor “borrowing“ information from points spatially close in PC₁, PC₂, PC₃, PC₆, etc. This inference indicates that those particular PC features contain important physical insight about the system. Analyzing the roughness parameters associated with each individual input variable for their relative importance to output is called automatic relevance determination (ARD) [57, 69]. ARD can be effectively used to eliminate the uncorrelated variables from the input space, which can help avoid overfitting.

The first, second, third, and sixth basis vectors are shown in Fig. 5. All basis vectors are scaled to the same color limits displayed by the color bar at the right side. The first basis vector exhibits an almost uniform distribution with a peak at the origin (corresponding to zero vector; this peak is too small to be seen in the figure). This is an indication that PC₁ score correlates strongly with the volume fraction information. On the other hand, PC₂ basis map has negative values in the short vector region indicating that it captures a decrease in the particle size. The PC₃ basis map exhibits slightly positive values in the x direction, indicating an expansion of the particle shapes in this direction. The PC₆ basis map exhibits high negative values in the x direction and high positive values in both y and z directions. This captures shape changes in particles which are being shrunk in the x direction and expanded in the y − z plane. Since the effective property of interest is the normal component in x direction in the stiffness tensor (i.e., C_11,eff), the morphological features identified in these PC scores seem to be the correct ones needed for building high-fidelity reduced-order models.

The efficiencies of different model building strategies explored in this study are summarized in Fig. 6. More quantitatively interpretable results are also shown in Fig. 7. The MSPE computed from the 10-fold cross-validation is shown as a function of the training fraction and number of considered principle components. One trend is clear across all considered models—there is a sharp increase in predictive accuracy when including PC₆ − PC₁₅ into the model. In Fig. 4, the correlation length scales suggest that the model is equally sensitive to PC₆ (loading direction morphology) and PC₁ (volume fraction). In addition, PC₁₁–PC₁₃ also have some nonnegligible influence on the response. Therefore, the improvements obtained by including additional PC’s is not simply a matter of “including more data” but instead can be attributed to “including better physics.” Recall that PCA is an unsupervised method where the PC bases are established without considering the response variable. Therefore, it is possible that higher order PC’s, which are believed to be less important when establishing a PC representation, may actually be rather influential on the response.

Among the considered models, the best predictive performance was obtained by the laGP model. Furthermore, the laGP model is able to significantly outperform the other models at lower R^∗. This indicates that the high-contrast elasticity dataset considered is both highly nonlinear and may be plagued by nonstationarity. A single structure–property reduced-order mapping over the entire domain \(\mathcal {X}\) is difficult to obtain and instead local nonparametric relationships better describe the linkage. Interestingly, the laGP model performs worse with increasing number of PC’s. In this case, we believe that this is due to overfitting. Recall that, for all laGP evaluations, the number of neighbors considered for establishing the local model was fixed to 200 points. For R^∗ = 60, this corresponds to about three data points per dimension. This is considered rather low and therefore we believe that prediction deteriorates with increasing R^∗ since the “true” hyperparameters cannot be inferred. However, scaling the number of neighbors with increasing R^∗ was observed to increase the computational burden.

Interestingly, the EN model appeared to perform well at a large R^∗. Unfortunately, regression is iterative and requires inversion of an extremely large X^TX matrix (X is the design matrix) whose size is determined by the number of features considered (columns in X). The number of features grows exponentially with increasing R^∗, especially when considering cross-terms of polynomials up to the fourth order. Therefore, for larger values of R^∗, the EN was not able to train in a timely manner. Compare this difficulty instead to the global GPR model which is bounded instead to the dataset size (N) and thus at worst requires inversion of a N × N matrix.

The LOESS model performs the worst from the considered models. The limitation in LOESS is twofold: (1) it does not utilize regularization and hence, to mitigate against overfitting, we limited the model to only consider linear terms; and (2) for high-dimensional data, it is nontrivial to compute distance-based weights. In the GP setting, distance is utilized to compute correlations, which can be thought of as weights, but there are correlation length-scale hyperparameters which allow for flexibility in establishing feature importance. In the standard LOESS implementation, there exists no analogue. For instance in LOESS, differences in PC₁ are equally as impactful as differences in PC₈. However, from Fig. 4, it is clear that PC₁ is more “important.”

A number of parametric, nonparametric, local, and global models were considered for performing regression on a large, numerically generated, high-contrast, 3D elastic composite MVE dataset. Hyperparameters obtained from Gaussian Process regression aided in identifying important PC components that were otherwise deemed relatively unimportant by PCA. The PC basis for one such component corresponded intuitively with problem-specific structure–property-coupled physics. A locally approximate Gaussian Process model was found to yield the best predictive structure–property model. This local nonparametric model was found to significantly outperform other models particularly when considering more compact microstructure representations. This indicates that the nonparametric model is adept at exploiting information-rich features and that the high-contrast elasticity dataset likely exhibits nonstationary or heteroskedastic behavior which cannot be captured accurately using global approaches. The source of this anomalous behavior is likely associated with the highly complex response of the microstructures considered. Nonparametric methods were found to outperform parametric methods while avoiding difficulties associated with the curse of dimensionality.

The dataset is available at https://​matin.​gatech.​edu/​resources/​309 [82].