Analysis of three-dimensional potential problems in non-homogeneous media with physics-informed deep collocation method using material transfer learning and sensitivity analysis
In this work, we present a deep collocation method (DCM) for three-dimensional potential problems in non-homogeneous media. This approach utilizes a physics-informed neural network with material transfer learning reducing the solution of the non-homogeneous partial differential equations to an optimization problem. We tested different configurations of the physics-informed neural network including smooth activation functions, sampling methods for collocation points generation and combined optimizers. A material transfer learning technique is utilized for non-homogeneous media with different material gradations and parameters, which enhance the generality and robustness of the proposed method. In order to identify the most influential parameters of the network configuration, we carried out a global sensitivity analysis. Finally, we provide a convergence proof of our DCM. The approach is validated through several benchmark problems, also testing different material variations.
Hinweise
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Abkürzungen
\(k({{\varvec{x}}})\)
Position-oriented material function
\(\phi\)
Potential function
q
Flux of potential field
\(\varvec{n}\)
Unit normal vector to a surface
\(w_{jk}^{l}\)
Weight between neuron k in hidden layer \(l-1\) and neuron j in hidden layer l
\(b^l_j\)
Bias of neuron j in layer l
\(\sigma\)
Activation function
\(\theta\)
Hyperparameters including all weights and biases
\(\mathrm{{Loss}}\left( \theta \right)\)
Loss function for training
\({{\varvec{x}}}\,_\Omega\)
Collocation points to discretize the physical domain
\({{\varvec{x}}}\,_\Gamma\)
Collocation points to discretize the boundaries
\(\mathrm{{MSE}}\)
Mean square error loss form
\(\phi ^h (\varvec{x};\theta )\)
Potential function approximated by Neural networks
\(G\left( {{\varvec{x}}}\right)\)
Governing equation
\({\tilde{\phi }}, \tilde{ \textit{q}}\)
Potential field and flux prescribed at boundaries
\(\eta _i\)
Learning rate
\(\mathrm{{EE}}_i\)
Elementary effect for each input factor
\(\mu _{i}^{*}\)
Mean of the distribution of the elementary effects of each input
\(\sigma _{i}\)
Standard deviation of the distribution of the elementary effects of each input
\(\Lambda _j\)
Spectral curve of the Fourier progression
\(S_{i}^\mathrm{{FAST}}\)
First-order FAST sensitivity indices
\(S_{T_{i}}\)
Total order FAST sensitivity indices
e
Relative error to measure the model accuracy
\(E_\mathrm{a}\)
Analytical solution
\(E_\mathrm{a}\)
Predicted solution
\(\left\| \cdot \right\|\)
\(L_2\)-norm
1 Introduction
Recent years have witnessed the vast growing application of neural networks in physics, this is partly due to the fact that by training the neural network, high-dimensional raw data can be converted to low-dimensional codes [1], and thus the high-dimensional PDEs can be directly solved using a ‘meshfree’ deep learning algorithm, which improves computing efficiency and reduces the complexity of problems. The deep learning method deploys a deep neural network architecture with nonlinear activation functions that introduces the nonlinearity that the system as a whole needs for learning nonlinear patterns. This lends some credence to the application of a physics informed machine learning method in discovering the physics behind the potential problems in non-homogeneous media, which is a wide range of problems in physics and engineering.
The current wave of deep learning started around 2006, when Hinton et al. [2, 3] introduced deep belief nets and unsupervised learning procedures that could create layers of feature detectors without needs of labelled data. Equipped with deep learning model, information can be extracted from complicated raw input data with multiple levels of abstraction through a layer-by-layer process [4]. Various variants, such as multilayer perceptron (MLP), convolutional neural networks (CNN) and recurrent/recursive neural networks (RNN) [5], have been developed and applied to, e.g. image processing [6, 7], object detection [8, 9], speech recognition [10, 11], biology [12, 13] and even finance [14, 15]. Over the past decade, it has been widely used in applications due to high performance demonstrated. Deep learning can learn features from data automatically, and the features can be used to get the approximation of solutions to differential equations [16], which cast light on the possibility of using deep learning as functional approximators.
Anzeige
Artificial neural networks (ANN) stand at the center of the deep learning revolution, it can be traced back to the 1940s [17], but they became especially popular in the past few decades due to the vast development in computational power and sophisticated machine learning algorithms, such as backpropagation technique and advances in deep neural networks. Due to the simplicity and feasibility of ANNs to deal with nonlinear and multi-dimensional problems, they were applied in inference and identification by data scientists [18]. They were also adopted to solve partial differential equations (PDEs) [19‐21], but shallow ANNs are unable to learn the complex nonlinear patterns effectively. With improved theories incorporating unsupervised pre-training, stacks of auto-encoder variants, and deep belief nets, deep learning with enhanced learning abilities can also serve as an interesting alternative to classical methods such as FEM.
According to the universal approximation theorem [22, 23], any continuous function can be approximated by a feedforward neural network with one single hidden layer. However, the number of neurons of the hidden layer tends to increase exponentially with increasing complexity and non-linearity of a model. Recent studies show that DNNs render better approximations for nonlinear functions [24]. Some researchers employed deep learning for the solution of PDEs. E et al. developed a deep learning-based numerical method for high-dimensional parabolic PDEs and back-forward stochastic differential equations [25, 26]. Raissi et al. [27] introduced physics-informed neural networks for supervised learning of nonlinear partial differential equations. Beck et al. [28] employed deep learning to solve nonlinear stochastic differential equations and Kolmogorov equations. Sirignano and Spiliopoulos [29] provided a theoretical proof for deep neural networks as PDE approximators, and concluded that they converge as the number of hidden layers tend to infinity. Karniadakis et al. presented physics-informed neural networks for various applications including fluid mechanics [30]. For problems in solid mechanics, we presented a Deep Collocation Method (DCM) in [31, 32], which has been the basis for a stochastic deep collocation method with neural architecture search strategy for stochastic flow analysis in heterogeneous media. We found that physics-informed deep learning model can account for stochastic disturbance/uncertainties efficiently and stably in [33]. An alternative to physics informed neural networks based on the strong form, such as the DCM, the Deep Energy Method (DEM) [34‐37] takes advantage of the total potential energy in the loss instead of a BVP.
The problems of potential represent a category of physical and engineering problems. For some physical parameters in potential problems, for example, heat conductivity, permeability, permittivity, resistivity, magnetic permeability, tends to have a spatial distribution, and they can vary with respect to one or more coordinates. To deal with these problems, the non-homogeneous problems are translated into homogeneous problems with some classes of material variations. The steady-state heat conduction analysis of FGMs analysis is a representative of potential problems. Due to the inherent mathematical difficulties, closed-form solutions exist in a few simple cases. Some traditional powerful methods, such as the finite element method (FEM), the boundary element method (BEM), and the method of fundamental solutions (MFS) and the dual reciprocity method (DRM) were used to solve the potential problems [38, 39]. The ‘meshfree’ physics-informed neural networks offered a novel and robust approach in discovering the nonlinear patterns behind the potential patterns, especially for higher dimensions.
The learning ability of deep neural networks strongly relies on the optimization algorithm and the neural network configurations, such as the activation function, number of neurons and layers, weight initialization methods, number of iterations, and so on. In this paper, we therefore compare different parameters to offer suggestions on the choice of a favourable configuration for the physics-informed neural network. Moreover, to increase the generality and robustness of the physics-informed deep learning based collocation method, the material transfer learning technique is integrated in the model, which will reduce the computation costs for different material variation types and help to improve the numerical results. Further, to unveil those influencing parameters for the proposed model, a global sensitive analysis is supplemented in the paper, which will be instructive for setting up physics-informed neural networks.
Anzeige
The paper is organised as follows: First, the three-dimensional potential problem with in-homogeneous media is presented. Then we introduce the physics-informed deep learning-based collocation method, which includes the neural network architecture, activation functions, sampling methods, a convergence proof, the material transfer learning and sensitivity analysis. Subsequently, a sufficient survey of numerical examples is presented, which investigated different neural network configurations, material transfer learning and model sensitivity analysis. Finally, the effectiveness of the deep learning method is demonstrated for solving three-dimensional potential problems in non-homogeneous media.
2 The governing equation for 3D problems of potential
The general partial differential equation for potential function \(\phi\) defined on a region \(\Omega\) bounded by surface \(\tau\), with an outward normal \(\varvec{n}\), can be written as:
where k is a position-oriented material function. Equation (1) is the field equation for a wide range of problems in physics and engineering, such as heat transfer, incompressible flow, gravity field, shaft torsion, electrostatics and magnetostatics, some of which are shown in Table 1 [40].
Table 1
Problems belong to the category of problems of potential
where \(\varvec{n}\) is the unit outward normal to \(\tau _N\). The material properties of functionally graded materials (FGMs) vary gradually in space. Classical variations of \(k(\varvec{x})\) take the form \(k(\varvec{x})=k_0f(\varvec{x})\), \(k_0\) denoting a reference value and \(f(\varvec{x})\) is the material property variation function. Among the most common variation functions are the quadratic, exponential and trigonometric:
3 Physics-informed deep learning-based collocation method
3.1 Feed forward neural network
The basic architecture of a fully connected feedforward neural network is shown in Fig. 1. It comprises multiple layers: an input layer, one or more hidden layers and an output layer. Each layer consists of one or more nodes called neurons, shown in Fig. 1 by the small colored circles. For an interconnected structure, every two neurons in neighboring layers have a connection, where the weights between neuron k in hidden layer \(l-1\) and neuron j in hidden layer l is denoted by \(w_{jk}^{l}\), see Fig. 1. No connection exists among neurons in the same layer as well as in the non-neighboring layers. Input data, defined from \(x_{1}\) to \(x_{N}\), flow through this neural network via connections between neurons, starting from the input layer, through the hidden layers \(l-1\), l, to the output layer, which eventually outputs data from \(y_{1}\) to \(y_{M}\).
Fig. 1
Architecture of a fully connected feedforward back-propagation neural network
×
The activation function is defined for an output of each neuron in order to introduce a non-linearity into the neural network and make the back-propagation possible, where gradients are supplied along with an error to update weights and biases. The activation function in layer l will be denoted by \(\sigma\) here.
There are many activation functions \(\sigma\) proposed for inference and identification with neural networks, such as sigmoids function [41], hyperbolic tangent function \(\left( Tanh \right)\) [41], Rectified linear units \(\left( Relu \right)\), to name a few. And some recent smooth activation functions, such as Swish [42], LeCuns Tanh [41], Bipolar sigmoid [41], Mish [42], Arctan [43], listed in Appendix B Table 9 have been studied and compared in the numerical example section. All selected activation functions must be smooth enough to avoid gradient vanishing during backpropagation, since the governing equation is introduced in the loss which includes the second-order derivatives of the field variable. Afterward, the value on each neuron in the hidden layers and output layer can be yielded by adding the weighted sum of values of output values from previous layer to basis. An intermediate quantity for neuron j on hidden layer l is defined as
where \(y^{l-1}_k\) is the output from previous layer.
Based on the previous derivation and description, we can draw a definition which will be used in Section 3.3:
Definition 3.1
(Feedforward Neural Network) A generalized neural network with activation function can be written in a tuple form \(\left( (f_1,\sigma _1), \ldots ,(f_n,\sigma _n)\right)\), \(f_i\) referring to an affine-line function \((f_i = W_i{{\varvec{x}}}+b_i)\) that mapps \(R^{i-1} \rightarrow R^{i}\). The tuple formed neural network in all defines a continuous bounded function mapping \(R^{d}\) to \(R^{n}\):
where d indicates the dimension of the inputs, n the number of field variables, \(\varvec{\theta }=\{ \varvec{W};\varvec{b} \}\) consisting of hyperparameters such as weights and biases and \(\circ\) denotes the element-wise operator.
The universal approximation theorem [22, 23] states that this continuous bounded function F with nonlinear activation \(\sigma\) can be adopted to capture the nonlinear property of the system, in our case the potential problem. With this definition, we can define [44]:
Theorem 1
If \(\sigma ^i \in C^m(R^i)\) is non-constant and bounded, then \(F^n\) is uniformly m-dense in \(C^m(R^n)\).
3.2 Backpropagation
Backpropagation \(\left( backward\;propagation \right)\) can be used to train multilayer feed-forward networks by calculating the gradient of a loss function and finding the minimum value of the loss function. The backward (output-to-input) flow determines how to adjust each weight as shown in Fig. 2.
Fig. 2
The ‘compute graph’ for the feedforward neural network
×
Backpropagation is based on the chain rule, which is used to calculate the derivative of loss function with regard to the weight in the network. The governing equation in our problem requires the second partial derivatives of the potential function \(\phi \left( {{\varvec{x}}}\right)\). To find the weights and biases, a loss function \({\text {Loss}}\left( \textit{f},\theta \right)\) is defined. The backpropagation algorithm for computing the gradient of this loss function \({\text {Loss}}\left( f,\theta \right)\), the weight coefficients \({{\varvec{w}}}\) and thresholds of neurons \({{\varvec{b}}}\) can be written as follow:
×
3.3 Physics-informed deep collocation method
To train the network, we place collocation points in the physical domain and at the boundaries denoted by \({{\varvec{x}}}\,_\Omega =(x_1, \ldots ,x_{N_\Omega })^T\) and \({{\varvec{x}}}\,_\Gamma (x_1, \ldots ,x_{N_\Gamma })^T\), respectively. Then the potential function \(\phi\) is approximated with the aforementioned deep feedforward neural network \(\phi ^h (\varvec{x};\varvec{\theta })\). Thus, a loss function related to the underlying BVP is constructed. Substituting \(\phi ^h \left( \varvec{x}\,_\Omega ;\varvec{\theta }\right)\) into governing equation, we obtain
which results in a physics-informed deep neural network \(G\left( {{\varvec{x}}}\,_\Omega ;\varvec{\theta }\right)\). The boundary conditions illustrated in Section 2 can also be expressed by the neural network approximation \(\phi ^h \left( {{\varvec{x}}}\,_\Gamma ;\varvec{\theta }\right)\) as: On \(\Gamma _{D}\), we have
where \(\textit{q}^h \left( {{\varvec{x}}}\,_{\Gamma _N};\varvec{\theta }\right)\) can be obtained from Eq. (2) by combing \(\phi ^h \left( {{\varvec{x}}}\,_{\Gamma _N};\varvec{\theta }\right)\). Note the induced physics-informed neural network \(G\left( {{\varvec{x}}};\varvec{\theta }\right)\), \(q\left( {{\varvec{x}}};\varvec{\theta }\right)\) share the same parameters as \(\phi ^h \left( {{\varvec{x}}};\varvec{\theta }\right)\). Considering the generated collocation points in domain and on boundaries, they can all be learned by minimizing the mean square error loss function [45]:
where \(x\,_\Omega \in {R^N}\), \(\varvec{\theta } \in {R^K}\) are the neural network parameters. \(\mathrm{{Loss}}\left( \varvec{\theta }\right) = 0\), \(\phi ^h \left( {{\varvec{x}}};\varvec{\theta }\right)\) is then a solution to potential function. Here, the defined loss function measures how well the approximation satisfies the physical law (governing equation), boundaries conditions. Our goal is to find a set of parameters \(\varvec{\theta }\) that the approximated potential \(\phi ^h \left( {{\varvec{x}}};\varvec{\theta }\right)\) minimizes the loss Loss. If Loss is a very small value, the approximation \(\phi ^h \left( {{\varvec{x}}};\varvec{\theta }\right)\) is very closely satisfying governing equations and boundary conditions, namely
The solution of heat conduction problems by deep collocation method can be reduced to an optimization problem. In the deep learning Tensorflow framework, a variety of optimizers are available. One of the most widely used optimization methods is the Adam optimization algorithm, which is also adopted in the numerical study. The idea is to take a descent step at collocation point \({{\varvec{x}}}_{i}\) with Adam-based learning rate \(\eta _i\),
and then the process in Eq. (13) is repeated until a convergence criterion is satisfied.
3.4 Convergence of deep collocation method for non-homogeneous PDEs
With the universal approximation theorem of neural networks, a feedforward neural network is used to approximate the potential function as \(\phi ^h \left( {{\varvec{x}}};\varvec{\theta }\right)\). The approximation power of neural networks for a quasilinear parabolic PDEs is shown by Sirignano et al. [29]. For non-homogeneous elliptic PDEs, the convergence study can be boiled down to:
The non-homogeneous PDEs has a unique solution, s.t. \(\phi \in C^2(\Omega )\) with its derivatives uniformly bounded. Also, the conductivity function \(k({{\varvec{x}}})\) is assumed to be \(C^{1,1}\) (\(C^1\) with Lipschitz continuous derivative).
Theorem 2
Assume that \(\Omega\) is compact with measures \(\ell _1\), \(\ell _2\), and \(\ell _3\) whose supports are constrained in \(\Omega\), \(\Gamma _D\), and \(\Gamma _N\). Furthermore, the governing Eq. (1) subject to 2 has a unique classical solution and material function \(k({{\varvec{x}}})\) being \(C^{1,1}\) (\(C^1\) with Lipschitz continuous derivative). Then, \(\forall \;\; \varepsilon >0\), \(\exists \;\; K>0\), which may dependent on \(sup_{\Omega }\left\| \phi _{ii}\right\|\) and \(sup_{\Omega }\left\| \phi _{i}\right\|\), s.t. \(\exists \;\; \phi ^h\in F^n\), satisfies \(\mathrm{{Loss}}(\varvec{\theta })\le K\varepsilon\)
Proof
For governing Eq. (1) subject to 2, according to Theorem 1, \(\forall\)\(\varepsilon \;\; >0\), \(\exists \;\; \phi ^h\;\; \in \;\; F^n\), s.t.
Let us consider the \(C^{1,1}\) conductivity function \(k({{\varvec{x}}})\), \(\exists \;\;M_1>0,\;\;M_2>0\), \(\exists \;\; x \in \;\Omega\), \(\left\| k({{\varvec{x}}})\right\| \leqslant M_1\), \(\left\| k_{,i}({{\varvec{x}}})\right\| \leqslant M_2\). From Eq. (15), we can then obtain:
With Theorem 2 and the condition that \(\Omega\) is a bounded open subset of R, \(\forall n\in N_+\), \(\phi ^h\in \;F^n \;\in L^2(\Omega )\), it can be concluded from Sirignano et al. [29] that:
Theorem 3
\(\forall \;p<2\), \(\phi ^h\in \;F^n\) converges to \(\phi\) strongly in \(L^p(\Omega )\) as \(n\rightarrow \infty\) with \(\phi\) being the unique solution to the potential problems.
In summary, for feedforward neural networks \(F^n \in L^p\) space (\(p<2\)), the approximated solution \(\phi ^h\in F^n\) will converge to the solution to the non-homogeneous PDE.
3.5 Collocation points generation
Model training is an important process in machine learning and the quality of training datasets determines the reliability of the machine learning model to a large extent. The deep collocation method (DCM) utilizes physics-informed neural networks for solving PDEs with randomly generated training points in the physical domain. To test the influence of training points on the stability and accuracy, different sampling methods are compared. The Halton and Hammersley sequences generate random points by a constructing the radical inverse [46]. They are both low discrepancy sequences. The method of Korobov Lattice creates samples from Korobov lattice point sets [47]. Sobol Sequence is a quasi-random low-discrepancy sequence to generate sampling points [48]. Latin hypercube sampling (LHS) is a statistical method, where a near-random sample of parameter values is generated from a multidimensional distribution [49]. Monte Carlo methods can create points by repeated random sampling [50]. The distribution plots of different sampling points inside a cube is listed in Appendix B Table 10 (Fig. 3).
Fig. 3
Schematic diagram of physics-informed neural networks
×
3.6 Material transfer learning
To improve the generality and robustness of the DCM, transfer learning is exploited, which makes use of the information from an already trained model yielding to training with less data and a reduced training time. The basic idea can be found in Fig. 4. For different material variations in nonhomogeneous media, the ‘knowledge’ of one material model can be exploited as the pretrained model resulting in a two-stage paradigm. The material transfer learning model is divided into two parts, i.e. pretraining, where the network is trained on a large dataset and longer iterations for one material variation type. The remaining part is the fine-tuning, where the pretrained model is trained on other material variations with few data and number of epochs. Consequently, the weights and biases and network configurations from a trained model are passed to other relevant models.
Fig. 4
Transfer learning
×
There are still some unresolved limitations in the literature. Most importantly, physics-informed deep learning algorithms lack a more systematic procedure to prevent overfitting and finding global minima.
4 Sensitivity analysis
Algorithm-specific parameters, such as the neural architecture configurations, parameters related to optimizers and number of collocation points significantly influence the model’s accuracy. To quantify their influence on the accuracy, a global sensitivity analysis (GSA) is performed. Classical GSA including regression methods, screening approaches, such as Morris method [51], the variance-based measures, such as Sobol’s method [52], and the Fourier amplitude sensitivity test (FAST) [53], or the extended FAST (EFAST) [54].
Variance-based methods are usually more computationally expensive than the derivative-based methods as well as the regression methods. If the model or the parameters in analysis is large, the use of variance-based method can be costly. The method of Morris is generally robust to correctly screen the most and least sensitive parameters for a highly parameterized model with 300 times fewer model evaluations than the Sobol’ method [55]. Therefore, the computational cost of a sensitivity analysis can potentially be reduced by first performing parameter screening using the Morris method to identify non-influential parameters, reducing the dimension of the parameter space to be studied in further analysis, then filter them again, but with the eFAST method. In this way, we can quantifying the effects of inputs more accurately with a relatively small amount of time.
4.1 Method of Morris
The method of Morris [56] is a screening technique used to rank the importance of parameters by averaging coarse difference relations termed elementary effects. Given a model with n parameters, \(\varvec{X}={X_1,X_2, \ldots X_n}\) denoting a vector of parameter values, we can specify an objective function \(y(x)=f(X_1,X_2, \ldots X_n)\), change the variables \(X_i\) by specific ranges and then calculate the distribution of elementary effects (EE) of each input factor with respect to the model outputs, i.e.
where f(x) represents the prior point in the trajectory. Using the single trajectory shown in Eq. (20), the elementary effects of each parameter can be calculated with \(p+1\) model evaluations. After sampling the trajectories, the resulting sets of elementary effects are then averaged to obtain the total-order sensitivity of the i-th parameter \(\mu _{i}^{*}\):
The mean value \(\mu ^*\) quantifies the individual effect of the parameters on an output while the variance \(\sigma ^2\) indicates the influence of parameter interactions. We rank the parameters according to \(\sqrt{\sigma ^2+{\mu ^*}^2}\).
4.2 eFAST method
The eFAST method [54] is based on Fourier transformations. The spectrum is obtained by each parameter and the output variance of model results due to interactions. Employing a suitable search function, the model \(y(x)=f(X_1,X_2, \ldots X_n)\) can be transformed by the Fourier transform into \(y= f(s)\)
The spectral curve of the Fourier progression is defined as \(\Lambda _j=A_j^2+B_j^2\). The variance of the model results due to the uncertainty in the parameter \(X_i\) is given by
with the parametric frequency \(\omega _1\), the spectrum of the Fourier transform \(\Lambda\), and the non-zero integers \(Z_0\). The total variance can be obtained by cumulatively summing the spectra at all frequencies
To find the total sensitivity of \(X_i\), the frequency of \(X_i\) is set to \(\omega _i\), while a different frequency \(\omega '\) is set for all other parameters. By calculating the frequency \(\omega _i\) and its higher harmonics \(p\omega _i\) spectra, the output variance \(D_{-i}\) due to the influence of all parameters except \(X_i\) and their interrelationships can be obtained. Thus, the total-order sensitivity indices can be obtained:
In this section, several cases are considered testing the accuracy and efficiency of our DCM including the influence of suitable NN configurations, sampling methods and optimizers taking advantage of GSA. Also, different material variations using material transfer learning are studied. The accuracy is measured in the relative error between the predicted solution and the analytical solution:
where \(E_{a}\) is the analytical solution and \(E_{pred}\) is the predicted solution while \(\left\| \cdot \right\|\) refers to the \(L_2\)-norm. All simulations are done on a 64-bit macOS Catalina computer with Intel(R) Core(TM) i7-8850H CPU, 32GB memory. The parametric settings for training are summarised in Table 3.
Table 3
Hyper-parameters settings in training
Model
Hyper-parameters
Values
Adam optimizer
Learning rate
0.001
L-BFGS-B optimizer
Maximum number of iterations to perform
50,000
Maximum number of function evaluations
50,000
Maximum number of variable metric corrections
50
Maximum number of line search steps (per iteration)
50
5.1 Case 1: Sensitivity analysis
First, we perform a SA to determine the key parameters of the deep collocation method.
5.1.1 Parameters screening with Morris method
The sensitivity indices computed by the Morris screening method with 30 trajectories and 4 grid levels are listed in Figs. 5 and 6, showing the effect of the numbers of neurons, layers, iterations and collocation points on the loss values. Figure 5 depicts the horizontal barplot of the GSA measure \(\mu ^*\). The highest \(\mu ^*\) value is found for the numbers of layers and neurons. The numbers of collocation points barely have an effect on the loss value. According to a classification scheme proposed by Garcia Sanchez et al. [57], the ratio \(\sigma /\mu ^*\) allows the characterisation of the model parameters in terms of (non-)linearity \((\sigma /\mu ^*< 0.1)\), (non-) monotony \((0.1<\sigma /\mu ^*< 0.5)\) or possible parameter interactions \((1<\sigma /\mu ^*)\), see also Fig. 6. For our test models, all parameters are in the range \(\sigma /\mu ^*>1\) suggesting that most parameters exhibit either non-linear behaviour, interaction effects with each other or both. The plot of the mean value and standard deviation \((\sigma , \mu ^*)\) in Fig. 6 reveals that the most influential parameter with largest \(\sqrt{\sigma ^2+{\mu ^*}^2}\) is the numbers of layers. The number of neurons and iterations is less important. The collocation points inside the physical domain and on the surface do not have a significant impact neither. Thus, while tuning the parameters of the model, more attention should be paid on the numbers of layers, neurons and iterations.
Fig. 5
Horizontal barplot of the mean absolute elementary effects \(\mu ^*\)
Fig. 6
\(\sigma\) versus \(\mu ^*\) for parameter screening with Morris method
×
×
5.1.2 Variance-based sensitivity indices
We now take advantage of the variance-based eFAST method to compute sensitivity indices. The independent first-order sensitivity indices \(S_i\) and dependent total order sensitivity indices \(S_{T_i}\) can be found in Fig. 8. Due to the high computational costs, with 3000 simulations run and 1000 generated samples, no analyses concerning the variation of in \(S_i\) and \(S_{T_i}\) with different sample sizes were performed.
The associated scatter plots are shown in Fig. 7. The more randomly the loss values are distributed, the less sensitive the parameters is. According to Fig. 7, the number of layers is the most influential parameter, followed by the number of neurons and number of iterations.
Fig. 7
Scatter plot of loss value against parameter values with 3000 runs
×
The first-order sensitivity index \(S_i\) represents the parameter importance. The number of layers affects the model most, followed by the numbers of neurons and the least influential parameter is the number of iterations, which agrees well with the results of Morris Method. However, the first-order indices are all beyond 0.01, which manifest that those algorithm-specific parameters individually do not have too much influence on the loss value of the model. The total effects index \(S_{T_i}\) greater than 0.8 can be regarded very important parameters. Again, the number of layers and neurons is greater than 0.8. For the number of iterations, it is between 0.5 to 0.8. However, there is a big difference between the value of total and first-order sensitivity indices, which quantifies the effects of the parameter’s interactions. It can be concluded that the output variance can be attributed to their interactions with other parameters rather than their nonlinear effects and all interactions between these three parameters are noteworthy.
Fig. 8
Results for first-order and total indices with eFAST method
×
5.2 Case 2. Cube with material gradation along the z-axis
Let us consider a unit cube (L = 1) with prescribed constant temperature on two sides. The top surface of the cube at z = 1 is maintained at a temperature of T = 100 while the bottom temperature at z = 0 is zero. The remaining four faces are insulated (zero normal flux). Three different classes of variations shown in Table 4 are considered [58]. The profiles of the thermal conductivity k(z) of the three material variation cases are illustrated in Fig. 9, and the boundary conditions of the unit cube can be found in Fig. 10. For each nonhomogeneous thermal conductivity, the analytical solution is summarized in Table 4.
Fig. 9
Thermal conductivity variation along the z direction
Fig. 10
The boundary conditons of the unit cube
Table 4
Analytical solutions for various forms of thermal conductivity \(k(\varvec{x})\)
First, different NN configurations are investigated. Figure 11 shows the relative error for various activation functions and layers. The arctan function yields the most stable and accurate results. Both arctan and Tanh function outperform the other activation functions. Figure 12 depicts the influence of different sampling methods on the relative error. Random sampling method obtained most stable and accurate potentials with increasing layers. Korobov, Hammersley, LatinHypercube sampling methods also provide reasonable results.
Next, we focus on various material variations, see Fig. 13. All material variations can be predicted accurately, but the most accurate results are obtained for the exponential conductivity. The results from Figs. 11, 12 and 13 suggest that 2 hidden layers are a good choice for the underlying problem.
Fig. 11
Comparison of results predicted by DCM with different activation functions
Fig. 12
Comparison of results predicted by DCM with different sampling methods
Fig. 13
Comparison of predicted results for different material variations
Fig. 14
Comparison of predicted results for different collocation points in the cube
Fig. 15
Comparison of predicted results with FEM versus numbers of points
Fig. 16
Comparison of predicted results for different collocation points on each boundary
×
×
×
×
×
×
We study now different numbers of collocation points (inside the cube and on its surface). The relative error in the temperature is depicted in Figs. 14 and 16. We also compared our results to results from FEM in Fig. 15. The temperature profiles along the z-axis for three material variations are plotted with the corresponding analytical solutions in Fig. 17.
Fig. 17
Temperature profile in the Z direction for different material variations
×
The predicted temperature and flux distributions for three material variations inside the cube are shown in Figs. 18, 19 and 20. The heat distribution varies with graded variation in the z coordinates which is consistent with the material property of the FGMs.
Fig. 18
a Predicted temperature and b Predicted flux for exponential material variation for the functionally graded unit cubic
Fig. 19
a Predicted temperature and b Predicted flux for trigonometric material variation for the functionally graded unit cubic
Fig. 20
a Predicted temperature and b Predicted flux for quadratic material variation for the functionally graded unit cubic
×
×
×
Let us now test the influence of the optimizer on the results. First-order methods minimize the function using its gradient, while second-order methods minimize the loss function using the second-order derivatives (Hessian information). In this application, a combination of these two optimizers is employed. The used first-order method is the Adam algorithm while L-BFGS is the tested second-order method. The convergence history for different optimizers is illustrated in Fig. 21. Although the first-order optimizer can be faster, they require more iterations. The L-BFGS optimizer needs less iterations, but there is the risk in being trapped in local minima. Using the combined optimizers, the loss reaches a significant smaller value with acceptable number of iterations and simultaneously ensures the solution being close to the global minima. The results for different number of layers are illustrated in Fig. 22.
Fig. 21
Comparison of different optimizers for deep collocation method
Fig. 22
Convergence graph for DCM with increasing hidden layers
×
×
5.2.2 Material transfer learning
The loss vs number of iterations is shown in Fig. 23. After funetuning, the loss decreases to a smaller value in less iterations for all three material variations. The numerical results are summarized in Table 5 demonstrating that the computational effort can be drastically reduced with transfer learning.
Fig. 23
Loss vs iteration using transfer learning with different material variations
Table 5
Relative error and training time for material variation with transfer learning
Results
Material variation
Exponential
Exponential
Quadratic
Trigonometric
Without TL
With TL
With TL
With TL
Relative error
4.2846e-06
3.9015e-06
3.7033e-06
3.6562e-06
Training time
45.5s
9.1s
22.4s
18.3s
×
Figure 24 shows the loss vs iteration using transfer learning for different material parameters while Tables 6 and 7 list the accuracy and CPU time with and without transfer learning.
Fig. 24
Loss vs iteration using transfer learning with different material parameters
Table 6
Relative error of temperature with varying material parameters
\(k_0\)
\(\beta\)
3
2
1
Without TL
With TL
Without TL
With TL
Without TL
With TL
6
1.9416e-05
8.6204e-06
1.7244e-05
8.2445e-06
3.8324e-06
3.1974e-06
5
1.8445e-05
1.9075e-05
6.8346e-06
9.9521e-06
1.9358e-06
2.7892e-06
4
1.4026e-05
8.0790e-06
6.8956e-06
1.6358e-05
4.8229e-06
2.3579e-06
Table 7
Computation time with varying material parameters (s or sec)
\(k_0\)
\(\beta\)
3
2
1
Without TL
With TL
Without TL
With TL
Without TL
With TL
6
6.9715e+01
1.8325e+01
4.6163e+01
1.0890e+01
4.7989e+01
5.4962e+00
5
5.6954e+01
1.2305e+01
4.0428e+01
1.0409e+01
4.3479e+01
4.9966e+00
4
6.4699e+01
1.3876e+01
5.3908e+01
6.9735e+00
3.8583e+01
6.1923e+00
×
5.3 Case 3: Cube with a 3D material gradation
Now, we consider a cube with the following three-dimensional thermal conductivity variation:
Representation of iso-surfaces for the three-dimensional variation of thermal conductivity k(x,y,z)
×
The boundary conditions at the six faces of the cube are listed in Table 8.
Table 8
The boundary conditons of cube with a 3D material gradation
Boundary condition
Dirichlet
Neumann
\(\phi (0,y,z)=0\)
\(q(1,y,z)=-0.2zy(25+2y+3z+zy)\)
\(\phi (x,0,z)=0\)
\(q(x,1,z)=-0.1xz(50+2x+6z+3xz)\)
\(\phi (x,y,0)=0\)
\(q(x,y,1)=-0.1xy(50+2x+4y+xy)\)
The predicted temperature and flux distributions are shown in Fig. 26. The predicted relative error of the temperature across the cube is 5.215360e-03, see also Fig. 27.
Fig. 26
Distributions of a temperature; and \(\left( b\right)\)flux for the cube with 3D material variation
Fig. 27
Temperature profile in diagonal line for the cube with a 3D material gradation
×
×
5.4 Case 4: Irregular-shaped annular sector
Next, we present results for an irregular-shaped annular sector as depicted in Fig. 29. The inner radius is 0.3, the outer radius is 0.5, the top surface is at Z = 0.1 and the thermal conductivity for the geometry varies exponentially according to
$$\begin{aligned} k(z)=5{ e }^{ (3z) } \end{aligned}$$
(33)
The variation of the thermal conductivity k(z) is illustrated in Fig. 28. The temperature is specified along the inner radius as \({T}_\mathrm{{inner}}=0\), and outer radius as \({T}_\mathrm{{outer}}=100\); all other surfaces are insulated. The boundary conditions of the geometry are shown in Fig. 29.
Fig. 28
Profile of thermal conductivity in z direction. The exponential variation of the conductivity is \(k(z)=5{ e }^{ (3z) }\)
Fig. 29
Annular sector subjected to thermal boundary conditions
×
×
The results of the predicted temperature can be found in Figs. 30 and 31 and is compared to a FEM solution using the commercial software package ABAQUS, as no analytical solution is available for this problem. The temperature along the radial direction at the edge is plotted and compared with results obtained by ABAQUS in Fig. 32.
Fig. 30
Temperature distribution for irregular-shaped FGMs obtained by a deep collocation method; and b ABAQUS
Fig. 31
Flux distribution for irregular-shaped FGMs obtained by a deep collocation method; and b ABAQUS
Fig. 32
Temperature comparison along the right top edge (indicated by the arrow)
×
×
×
6 Conclusion
We presented a transfer learning-based deep collocation method (DCM) for solving the problems of potential in non-homogeneous media. It avoids classical discretization methods, such as FEM, and treats the problem as minimization problem, minimizing a loss function which is related to the underlying governing equation. Thanks to the nonlinear activation function, the approach enables us to discover complex nonlinear pattern. The DCM requires sampling inside the physical domain. Therefore, we obtained a suitable sampling method for selected problems. To find the most favorable configuration of the neural network for specific problems, we carried out a sensitivity analysis quantifying the influence of algorithm-specific parameters on specific outputs such as the relative error in the L2 norm. For different material variation forms and material parameters, a material transfer learning is embedded into the framework to enhance the robustness and generality of this deep collocation method. To demonstrate the performance of the proposed DCM, various benchmark problems including the heat transfer and a representative potential problem are studied.
Acknowledgements
The authors extend their appreciation to the Distinguished Scientist Fellowship Program (DSFP) at King Saud University for funding this work.
Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The data flow inside all three modules is depicted in Fig. 33. The first module is based on the DCM. The second module includes a parametric study on the influence of algorithm-specific parameters including numbers of collocation points and parameters for the deep learning configurations on predictive accuracy, which in turn provides guidance for further applications of the DCM. For the parametric analysis, a two-step Morris screening method and EFAST method are adapted providing qualitative and quantitative measures of importance and interaction of DCM-specific parameters. To facilitate and improve the generality and robustness of the presented model, data are finally imported into a material transfer learning model to transfer and expand the learned knowledge between different material variations.
Fig. 33
Data flow in physics-informed deep collocation method and sensitivity analysis
×
Appendix B: Activation function and sampling method for comparison
The following table shows a list of classical activation functions and its graphs that studied in this application, which will help to choose a suitable activation for physics-informed neural networks (Table 9).
Table 9
Activation function
Various sampling methods are used to generate sequence of points within a cube. The purpose of the sampling method is to generate training datasets for the DCM and improve the training of the network. Proper sampling will help in case the neural network is only trained on fixed points and prevent a biased trained model, which may have a better prediction on random new data (Table 10).
Analysis of three-dimensional potential problems in non-homogeneous media with physics-informed deep collocation method using material transfer learning and sensitivity analysis
