Domain adaptation based transfer learning approach for solving PDEs on complex geometries

The idea to approximate PDEs using NNs was first proposed by Dissanayake et al [26]. In this paper we will also provide a strong theoretical formulation regarding the approximation power of NN for some specific family of PDEs. Consider the family,where \({\mathcal {L}}\) and \({\mathcal {B}}\) is a differential and boundary operator (quasilinear or non-linear), respectively, domain of interest \(\Omega \subset {\mathbb {R}}^n\) is bounded , boundary \(\partial \Omega\) is at least Lipschitz continuous unless mentioned otherwise and \(\gamma\) is its subset on which the boundary conditions (BCs) are imposed. We intend to train the DNN approximate solution \(U_{{\varvec{\theta }}}:= U({\varvec{x}},{\varvec{\theta }})\) We proceed by approximating u and \({\mathcal {L}}\) using Deep Neural Networks (DNNs) \(U_{\theta } = U(\mathbf {x},\theta )\). Any prior knowledge on the exact analytical solution u is redundant. As previously mentioned, our approach adopts the deep collocation method, which assumes a certain discretization of the domain \(\Omega\) and the boundary \(\gamma\) into a collection of points \(\pi _{\Omega }\) and \(\pi _{\gamma }\), respectively. The goal is to learn the parameters of NNs. These along with parameters of the operator \({\mathcal {L}}\) are learned by minimizing the error function under mean squared error (MSE) norm. For \({\varvec{x}}_i \in \pi _{\Omega }\) and \(\mathbf{s} _i \in \pi _{\gamma }\), we define:The points are uniformly sampled from the domain and the given boundary. Basically this equation is a Monte Carlo approximation ofwhere \(U(\Omega )\) and \(U(\gamma )\) represent the uniform distribution over the domain and given boundary. This error function measures how accurately our model approximates the original solution and satisfies the boundary value problem (BVP) with respect to the defined norm. Our main objective is to construct such a neural network function \(U_{\theta }\) by fine tuning the entire model so that \({\mathcal {E}}\) is as close to 0 as possible.

DNN is comprised of multiple hidden layers and single input–output layers, where each layer is consisting of numerous neurons. For example if we have N neurons in the input layer, and M neurons in the output layer then the neural network is simply a mapping: \({\mathbb {R}}^N \mapsto {\mathbb {R}}^M\). The neurons in each layer are connected by weights-bias parameters and they are used to compute a weighted sum of the input neurons from the preceding layers to which they are connected. For example two adjacent layers are coupled as follows :where \(y_{\ell }\) is the output of layer \(\ell\), \({\mathcal {W}}_{\ell }\) are affine mappings and \(\sigma _{\ell }\) is a fixed element wise activation function. The connection strength between neurons are solely dependent upon the weights and bias terms associated with these. The entire signal is now summed and used as an input for the layers activation function. The role of activation function is to introduce non linearity into the network. This is a crucial component for modeling nonlinear responses. The structure of the network can be complex. Once the network is fixed, the training is initiated and the task is to update the parameters appropriately. The network is trained using back-propagation which ultimately becomes the optimization problem of finding the parameters that minimize the loss function or the output errors. Gradually, as the parameters are adjusted, the network evolves and predicts the output with minimal errors. In short, this is the “learning pattern” of NN. In transfer learning, the goal is to store and access this previously gained knowledge from source data to reuse for second workflow. One needs to add a few high-level new layers see Fig. 1 on top of pre-trained fully connected layers in order to utilize the off-the-shelf representations from preceding deep layers. The core idea of transfer learning is inherent in the fact that neural networks are made up of layers which can be seen as interchangeable building blocks. Basically, in this approach we use well-trained, well-constructed models that have gained sufficient knowledge over the training process through larger or more generic data sets and apply them to boost the performance on smaller or more specific data sets. Fine-tuning begins, i.e, retraining of the entire model with a suitable learning rate and gradually the new layers learn to turn the old features into predictions on a target data set. In practice, differential learning rate a.k.a discriminative fine-tuning can be an effective strategy to customize the model. In this process, one may start with a fixed learning rate \(\alpha \in [0.001,0.1]\) to instantiate the base or inception model with pre-trained weights and once the model starts to converge on source data, in the second workflow one needs to drop \(\alpha\) preferably by a factor of \(\kappa \in (2,20)\) to avoid larger weight updates while embedding fine-tuning to the pretrained layers. In general, this is the guiding principle to update the parameters progressively from top-to-bottom to encourage the target model parameters to stay close to the parameters of pre-trained model. Freezing (when, \(\alpha =0\)) has not been investigated in detail in the present work. For a relatively tiny neural network model this setting can be deemed robust. However, if the network is too large, for example ResNet, it is recommended to partition the entire network into small groups of layers and set different learning rates to each group during the training, otherwise the whole process would be very slow and memory intensive. Some excellent literature on this topic can be found in [27, 28].

The success of a neural network completely depends on its architecture. There is no exact formula for selecting an optimal architecture, and different problems demand different architectures. For example, activation functions are one of the core components and are used as gates to filter between “useful” and “not so useful” from the plethora of information. There exist numerous works in the literature [25, 29, 30] and the references therein, where authors have conducted comparative experiments to obtain the best possible results from an activation function. One thing to note is that the final layer should always be the linear function of its preceding layers. Apart from that, optimizers also have a significant influence for fast and easy convergence of the network. The optimizer shapes and molds the model into its most accurate possible form by updating the parameters. In our experiments, we have explored different optimizers, configuring the key hyper-parameters (momentum, learning rate etc.) to improve much as possible the accuracy of the model.

We now provide the implementation details of the algorithm and have highlighted some typical steps that one would have to follow for a Python based Tensorflow [31], an open source well documented and currently one of the most popular, fastest growing deep learning library. The idea is to demonstrate the general procedure. The full source code is available on Github.

Using the Xavier Initialization technique [32] the parameters are initialized in the following manner:

\(u(\mathbf {x})\) is defined as follows:

The neural network is defined using an activation function:

Define the base model and network learning:

Transfer learning model and fine-tuning:

Motivated by the results in [33, 34] we are also going to provide an existence theorem ensuring a feed forward multi-layer network U capable to approximate the solution of (1) along with optimizing (2). For the convenience sake we have assumed the existence of classical solution and consider,and \({\mathcal {B}}[\cdot ]\) as a linear combinations of \(\partial ^{{\varvec{m}}}_x\), where \(|{\varvec{m}}| \le 2\). Therefore, (1) can re written as,In simplified form,For any activation function \(\sigma\) ( preferably “non-linear” and mandatorily “bounded”) consider the set of NNs with a single hidden layer and \(\ell\) neurons,and the class of all functions implemented in such network for \(\ell \in {\mathbb {N}}\)

A significant improvement over the performance of the base model, i.e, a positive transfer is an indication of a successful algorithm. The choice of pretraining and target tasks is intertwined closely and, therefore, for the sake of best target performance, it is beneficial to opt for similar pretraining tasks. The whole algorithm is tailored for an accurate and efficient approximation to obtain \(\Gamma _{TL}<\Gamma _{CL}\). The general practice is to set a termination criteria by enforcing a Tol at the pretraining phase. Thus if the condition is not satisfied then this can serve as an error indicator which is employed to guide the parameters in the neural network architecture. The parameters get updated accordingly and this process continues iteratively. To the end summarizing all steps, we obtain the basic scheme shown in Fig. 2 and detailed in Algorithm 1.

Consider,Define,

and the problem,To obtain the exact solution we take, for exampleand plug it into the problem to compute f. In the numerical model we consider the following domains : a square plate of edge length 2 and a similar length plate with a circular hole of half-unit radius at its center. We generate N uniformly distributed collocation points inside the domain and M uniformly distributed points on the boundary. Once the NNs have been trained, we can then evaluate on any number of points. Initially, the model is trained on a square plate which effectively serve as an inception model. The training process on a square plate (i.e., the source task) is less expensive relative to the training process on a plate with hole (i.e., the target task). To relate these two tasks more closely and improve the learning, the source domain has been subdivided into two different data sets. The data set represented by blue dots is trained to minimize the loss function evaluated over \(\Omega\), while the data set represented by green dots is trained to match the boundary conditions. Once the learning process is invoked and the loss reaches a certain tolerance, we stop the process. The stopping criteria is again based on trial-error so that the algorithm never fails to converge. Now we utilize this previously constructed model architecture and most of the learned parameters, and then using standard training methods to learn the remaining, non-reused parameters of the new model corresponding to the plate with hole.

In the source task, we have used 3 hidden layers of 40 neurons each and the entire domain is discretized with \(N\approx 5600\) interior, \(M\approx 1800\) boundary training points as shown in Fig 4. On the target task, we have considered 2 more hidden layers on top of preceding trained layers, comprising of 35 neurons in each hidden layer, and the domain contains \(N=2000\) interior, \(M=500\) boundary training points. In both cases, the swish activation function has been used and SGD optimizer with \(\alpha \in (0.0001,0.1]\) and \(\zeta \in (0.3,1]\). In the source task, a relatively larger learning rate was used and once the layers have been trained to converge we then retrain the whole model end-to-end with layer specific learning rate. In particular, we set a moderate learning rate for the pretrained layers to avoid the time-cost. For the sake of experiment, we consider three different training schedules viz. “Lower learning rate”, “Larger learning rate” and “Differential learning rate”. We emphasize that all the schedules require the same amount of training iterations. From our findings, we conclude that larger learning rate puts the model under a higher risk of exploding gradients and failure to converge, while a lower learning rate consumes excessive amounts of time. Therefore, following standard practice, we attempt to predict the regime of learning rates, where the optimal performance can be achieved. In this phase we choose \(\alpha _1=0.001~,~\alpha _2=0.008\) and \(\alpha _3=(0.0076,0.08)\) and continue to use the same set of hyperparameters across tasks. The evolution plots are depicted in Fig 5. We find that for a fixed computational budget the best performance is always achieved in differential learning rate. Nevertheless there is a trade-off and one needs to go over trial-error process depending on the model. The error plots from a conventional learning and transfer learning are also depicted in Fig. 5. In addition, performance comparisons between a model trained from scratch (i.e., CL model) and a pretrained model (i.e., TL model) are demonstrated by learning curves. Since the exact solution is known, we also compare relative errors, which are defined as:where \({\mathcal {N}}_{\text {pred}}\) is the number of testing points and \({\mathcal {U}}_{\text {err}}(x,y)= {\mathcal {U}}(x,y)-U_{ \theta }(x,y)\). Together this is a clear cut evidence how transfer learning boosts the computation performance with smaller data set, possessing the benefits of less training time for a NN model and resulting in a lower generalization error in comparison to conventional learning, where the model needs to train from scratch on a larger data set to reach a desired level of accuracy.

More recently, researchers are focusing on SGD with momentum instead of vanilla SGD. With the goal of training faster and more accurate neural nets, our empirical results demonstrates that SGD \(+~\zeta\) converges much better than Adam. Despite its widespread popularity, under specific circumstances Adam sometimes fails to converge to an optimal solution. Exploratory studies [41‐43] in this direction highlight the possible inabilities of adaptive optimization technique, such as Adam compared to SGD. We conducted an experiment with Adam and SGD. On the same problem setup, we train our model on the data set obtained from the hemispherical domain. We follow the same learning rate scheme. Figure 8 illustrates the efficacy of the respective optimizers. Evidently, this experiment suggests that SGD achieves a better accuracy compared to Adam, not only in faster decaying the loss values but also with respect to the training time and number of epochs necessary to attain that performance.

In this example we consider p Laplacian problem with homogeneous Dirichlet condition and right hand side f is chosen to match the exact solution. However, the problem in this case has low regularity. This example is somewhat motivated by an example in [44]. The degenerate nature of the p type problems makes the study of their regularity properties difficult, and in general, even with smooth problem data, high regularity for the solution u is not guaranteed. It has been shown in [45] that under some additional assumptions on the problem data, one can obtain a desired level of regularity which is sufficient to ensure an optimal convergence of the loss function. In this case, our domain is an annular ring,For the manufactured solution, we consider:

Following previous techniques, the source task here is to train the model on a unit disc filled with \(N=6400\) interior points and \(M\approx 2000\) boundary points, see Fig 9. We found for this model the hyperparameters : 3 hidden layers each with 50 neurons and “swish” activation were effective. Swish is bounded below but unbounded above and non-monotonic in nature; however, the evaluation cost per iteration is higher. Therefore, whenever possible, it should be combined with tanh or ReLU layers. This is what we also follow in our targeted task. Furthermore, here the data set contains \(N\approx 2500\) interior points and \(M=900\) boundary points, we have 2 more hidden layers of equal width of 30 neurons, and as mentioned earlier SGD followed by BFGS is applied in both the training process. Finally, the comparison with respect to the exact solution and also loss function evaluation determines how well our algorithm models the data set. By taking a look at the figures, Fig. 10, constructed by our algorithm, one can conclude that even in the case for less regularized PDE and also involving a complex domain, the transfer learning approximation reaches an error closer to \(\approx 10^{-7}\) while leading to a significant speedup in comparison to conventional learning.

In the final example, we turn our attention to an important class of partial differential equation that describes the dynamics of an incompressible Newtonian fluid flow. Because of the regularity issues, here we have studied only the stationary version of the problem in two dimensional bounded domains. The domains are described in Fig. 11. To explore the effectiveness of task-to-task learning, in this particular example we have conducted the experiment of implementing the pretrained model directly without any target task specific modification. In general, for a better target performance, it is always beneficial to choose a similar pretraining task. Therefore, in our experiment, we consider the square plate as source domain consisting of uniformly distributed \(N \approx 6700\) interior and \(M=1050\) boundary collocation points for training the source task model. We then use this pretrained model for initialization and evaluate its performance on a geometrically different target task model, i.e., plate with multiple holes.

The problem is motivated by an example in [46].Here, \(\mu =0.01\) is the kinemetic viscosity coefficient, \(\mathbf{v} (x,y)=[v_1(x,y)~~v_2(x,y)]\) & p(x, y) are the Eulerian velocity and pressure fields. We choose the boundary conditions and \(\tau\) so that the exact solution is given by,As mentioned earlier, our goal here is to train the model on a simpler regular domain, e.g., square plate and then we are generalizing the gained knowledge by applying this model on the targeted domain, e.g., plate with multiple holes of different dimensions. We avoid the training process on such domain which is quite challenging. Nevertheless, transfer learning allows to deal with these scenarios by leveraging the pre-built model from source task. The basic architecture consists of 3-hidden layer neural network containing 60 neurons in each layer. We have trained one neural network with 3 outputs but, it can also be done individually to approximate \(\mathbf{u}\) and p. This seems to be much harder, because training different architectures and attaining optimal hyperparameters is itself a daunting task and it also requires a lot of computational time. Besides, the former task is more suited for grasping the underlying equations. The weights are initialized using a Xavier initialization, while the biases are generated using a normal distribution with mean 0 and standard deviation 1. They are trained using an SGD optimizer with the fine tuned learning rate and momentum followed by BFGS. The first two hidden layers are connected with “swish” activation and the final layer is with “tanh” activation. The resulting prediction error is validated against the test data. Figures 12, 13, 14 provide a comparison between the exact solution and the transfer learning outcome. It can be observed that the predicted outputs and exact solutions are quite close on the testing sets, which evidently supports the success of the pretrained model over the targeted task. Therefore, from these empirical findings, we conclude that even without learning every task specific features, TL implementations are still adaptable enough to achieve reasonable accuracy with only a pretrained model. Thus by learning quite generic feature of the target domain, the model is capable of capturing the intricate non linear behavior of N–S equation in that domain. Moreover, as an addendum, we have achieved such performances with zero training duration on target model. To quantify the accuracy of this novel approach we also compute the relative error as:where \({\mathcal {N}}_{\text {pred}}\) is the no. of testing points and \({\mathcal {U}}_{\text {err}}(x,y)= {\mathcal {U}}(x,y)-U_{\sim \theta }(x,y)\). Similarly, \({\mathcal {L}}_2^{v}\) and \({\mathcal {L}}_2^{p}\) are also defined. To the end an experimental assessment of performance is demonstrated in Fig. 15.