Skip to main content
Erschienen in:

08.04.2024 | Original Article

Novel approaches for hyper-parameter tuning of physics-informed Gaussian processes: application to parametric PDEs

verfasst von: Masoud Ezati, Mohsen Esmaeilbeigi, Ahmad Kamandi

Erschienen in: Engineering with Computers | Ausgabe 5/2024

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Today, Physics-informed machine learning (PIML) methods are one of the effective tools with high flexibility for solving inverse problems and operational equations. Among these methods, physics-informed learning model built upon Gaussian processes (PIGP) has a special place due to provide the posterior probabilistic distribution of their predictions in the context of Bayesian inference. In this method, the training phase to determine the optimal hyper parameters is equivalent to the optimization of a non-convex function called the likelihood function. Due to access the explicit form of the gradient, it is recommended to use conjugate gradient (CG) optimization algorithms. In addition, due to the necessity of computation of the determinant and inverse of the covariance matrix in each evaluation of the likelihood function, it is recommended to use CG methods in such a way that it can be completed in the minimum number of evaluations. In previous studies, only special form of CG method has been considered, which naturally will not have high efficiency. In this paper, the efficiency of the CG methods for optimization of the likelihood function in PIGP has been studied. The results of the numerical simulations show that the initial step length and search direction in CG methods have a significant effect on the number of evaluations of the likelihood function and consequently on the efficiency of the PIGP. Also, according to the specific characteristics of the objective function in this problem, in the traditional CG methods, normalizing the initial step length to avoid getting stuck in bad conditioned points and improving the search direction by using angle condition to guarantee global convergence have been proposed. The results of numerical simulations obtained from the investigation of seven different improved CG methods with different angles in angle condition (four angles) and different initial step lengths (three step lengths), show the significant effect of the proposed modifications in reducing the number of iterations and the number of evaluations in different types of CG methods. This increases the efficiency of the PIGP method significantly, especially when the traditional CG algorithms fail in the optimization process, the improved algorithms perform well. Finally, in order to make it possible to implement the studies carried out in this paper for other parametric equations, the compiled package including the methods used in this paper is attached.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Liu X, Yao W, Peng W, Zhou W (2023) Bayesian physics-informed extreme learning machine for forward and inverse PDE problems with Noisy data. Neurocomputing 549:126425CrossRef Liu X, Yao W, Peng W, Zhou W (2023) Bayesian physics-informed extreme learning machine for forward and inverse PDE problems with Noisy data. Neurocomputing 549:126425CrossRef
2.
Zurück zum Zitat Donnelly J, Daneshkhah A, Abolfathi S (2024) Forecasting global climate drivers using Gaussian processes and convolutional autoencoders. Eng Appl Artif Intell 128:107536CrossRef Donnelly J, Daneshkhah A, Abolfathi S (2024) Forecasting global climate drivers using Gaussian processes and convolutional autoencoders. Eng Appl Artif Intell 128:107536CrossRef
3.
Zurück zum Zitat Dwivedi V, Srinivasan B (2020) Physics informed extreme learning machine (PIELM)-a rapid method for the numerical solution of partial differential equations. Neurocomputing 391:96–118CrossRef Dwivedi V, Srinivasan B (2020) Physics informed extreme learning machine (PIELM)-a rapid method for the numerical solution of partial differential equations. Neurocomputing 391:96–118CrossRef
4.
Zurück zum Zitat Xiang Z, Peng W, Liu X, Yao W (2022) Self-adaptive loss balanced Physics-informed neural networks. Neurocomputing 496:11–34CrossRef Xiang Z, Peng W, Liu X, Yao W (2022) Self-adaptive loss balanced Physics-informed neural networks. Neurocomputing 496:11–34CrossRef
5.
Zurück zum Zitat Escapil-Inchauspe P, Ruz GA (2023) Hyper-parameter tuning of physics-informed neural networks: application to Helmholtz problems. Neurocomputing 561:126826CrossRef Escapil-Inchauspe P, Ruz GA (2023) Hyper-parameter tuning of physics-informed neural networks: application to Helmholtz problems. Neurocomputing 561:126826CrossRef
6.
Zurück zum Zitat Chatrabgoun O, Esmaeilbeigi M, Cheraghi M, Daneshkhah A (2022) Stable likelihood computation for machine learning of linear differential operators with Gaussian processes. Int J Uncertain Quantif 12(3):75–99MathSciNetCrossRef Chatrabgoun O, Esmaeilbeigi M, Cheraghi M, Daneshkhah A (2022) Stable likelihood computation for machine learning of linear differential operators with Gaussian processes. Int J Uncertain Quantif 12(3):75–99MathSciNetCrossRef
7.
Zurück zum Zitat Donnelly J, Daneshkhah A, Abolfathi S (2024) Physics-informed neural networks as surrogate models of hydrodynamic simulators. Sci Total Environ 912:168814CrossRef Donnelly J, Daneshkhah A, Abolfathi S (2024) Physics-informed neural networks as surrogate models of hydrodynamic simulators. Sci Total Environ 912:168814CrossRef
8.
Zurück zum Zitat Hao Z, Liu S, Zhang Y, Ying C, Feng Y, Su H, Zhu J (2022) Physics-informed machine learning: a survey on problems, methods and applications. arXiv preprint arXiv:2211.08064 Hao Z, Liu S, Zhang Y, Ying C, Feng Y, Su H, Zhu J (2022) Physics-informed machine learning: a survey on problems, methods and applications. arXiv preprint arXiv:​2211.​08064
9.
Zurück zum Zitat Asrav T, Aydin E (2023) Physics-informed recurrent neural networks and hyper-parameter optimization for dynamic process systems. Comput Chem Eng 173:108195CrossRef Asrav T, Aydin E (2023) Physics-informed recurrent neural networks and hyper-parameter optimization for dynamic process systems. Comput Chem Eng 173:108195CrossRef
10.
Zurück zum Zitat Yang X, Tartakovsky G, Tartakovsky A (2018) Physics-informed kriging: a physics-informed Gaussian process regression method for data-model convergence. arXiv preprint arXiv:1809.03461 Yang X, Tartakovsky G, Tartakovsky A (2018) Physics-informed kriging: a physics-informed Gaussian process regression method for data-model convergence. arXiv preprint arXiv:​1809.​03461
11.
Zurück zum Zitat Alvarez MA, Luengo D, Lawrence ND (2013) Linear latent force models using Gaussian processes. IEEE Trans Pattern Anal Mach Intell 35(11):2693–2705CrossRef Alvarez MA, Luengo D, Lawrence ND (2013) Linear latent force models using Gaussian processes. IEEE Trans Pattern Anal Mach Intell 35(11):2693–2705CrossRef
12.
Zurück zum Zitat Raissi M, Perdikaris P, Em Karniadakis G (2017) Machine learning of linear differential equations using Gaussian processes. J Comput Phys 348(1):683–693MathSciNetCrossRef Raissi M, Perdikaris P, Em Karniadakis G (2017) Machine learning of linear differential equations using Gaussian processes. J Comput Phys 348(1):683–693MathSciNetCrossRef
13.
Zurück zum Zitat Williams CK, Rasmussen CE (2006) Gaussian processes for machine learning. MIT Press, Cambridge Williams CK, Rasmussen CE (2006) Gaussian processes for machine learning. MIT Press, Cambridge
14.
15.
Zurück zum Zitat Narayan A, Yan L, Zhou T (2021) Optimal design for kernel interpolation: applications to uncertainty quantification. J Comput Phys 430:1–20MathSciNetCrossRef Narayan A, Yan L, Zhou T (2021) Optimal design for kernel interpolation: applications to uncertainty quantification. J Comput Phys 430:1–20MathSciNetCrossRef
16.
Zurück zum Zitat Qin T, Chen Z, Jakeman JD, Xiu D (2021) Deep learning of parameterized equations with applications to uncertanity quantification. Int J Uncertain Quantif 11(2):63–82MathSciNetCrossRef Qin T, Chen Z, Jakeman JD, Xiu D (2021) Deep learning of parameterized equations with applications to uncertanity quantification. Int J Uncertain Quantif 11(2):63–82MathSciNetCrossRef
17.
Zurück zum Zitat Nocedal J, Wright SJ (eds) (1999) Numerical optimization. Springer, New York Nocedal J, Wright SJ (eds) (1999) Numerical optimization. Springer, New York
18.
19.
Zurück zum Zitat Polak E, Ribiere G (1969) Note sur la convergence de methodes de directions conjuguees. Revue francaise dinformatique et de recherche operationnelle. Serie Rouge 3(16):35–43 Polak E, Ribiere G (1969) Note sur la convergence de methodes de directions conjuguees. Revue francaise dinformatique et de recherche operationnelle. Serie Rouge 3(16):35–43
20.
Zurück zum Zitat Hestenes MR, Stiefel E (1952) Methods of conjugate gradients for solving linear systems. J Res Natl Bur Stand 49(6):409–436MathSciNetCrossRef Hestenes MR, Stiefel E (1952) Methods of conjugate gradients for solving linear systems. J Res Natl Bur Stand 49(6):409–436MathSciNetCrossRef
21.
22.
Zurück zum Zitat Liu Y, Storey C (1991) Efficient generalized conjugate gradient algorithms, part 1: theory. J Optim Theory Appl 69:129–137MathSciNetCrossRef Liu Y, Storey C (1991) Efficient generalized conjugate gradient algorithms, part 1: theory. J Optim Theory Appl 69:129–137MathSciNetCrossRef
23.
Zurück zum Zitat Dai YH, Yuan Y (1999) A nonlinear conjugate gradient method with a strong global convergence property. SIAM J Optim 10(1):177–182MathSciNetCrossRef Dai YH, Yuan Y (1999) A nonlinear conjugate gradient method with a strong global convergence property. SIAM J Optim 10(1):177–182MathSciNetCrossRef
24.
Zurück zum Zitat Hager WW, Zhang H (2005) A new conjugate gradient method with guaranteed descent and an efficient line search. SIAM J Optim 16(1):170–192MathSciNetCrossRef Hager WW, Zhang H (2005) A new conjugate gradient method with guaranteed descent and an efficient line search. SIAM J Optim 16(1):170–192MathSciNetCrossRef
25.
26.
Zurück zum Zitat Hager WW, Zhang H (2006) A survey of nonlinear conjugate gradient methods. Pac J Optim 2(1):35–58MathSciNet Hager WW, Zhang H (2006) A survey of nonlinear conjugate gradient methods. Pac J Optim 2(1):35–58MathSciNet
27.
Zurück zum Zitat Raissi M, Perdikaris P, Em Karniadakis G (2018) Numerical Gaussian processes for time-dependent and nonlinear partial differential equations. SIAM J Sci Comput 40(1):172–198MathSciNetCrossRef Raissi M, Perdikaris P, Em Karniadakis G (2018) Numerical Gaussian processes for time-dependent and nonlinear partial differential equations. SIAM J Sci Comput 40(1):172–198MathSciNetCrossRef
28.
Zurück zum Zitat Perkins TJ, Jaeger J, Reinitz J, Glass L (2006) Reverse engineering the gap gene network of Drosophila melanogaster. PLoS Comput Biol 2(5):e51CrossRef Perkins TJ, Jaeger J, Reinitz J, Glass L (2006) Reverse engineering the gap gene network of Drosophila melanogaster. PLoS Comput Biol 2(5):e51CrossRef
29.
Zurück zum Zitat Poustelnikova E, Pisarev A, Blagov M, Samsonova M, Reinitz J (2004) A database for management of gene expression data in situ. Bioinformatics 20(14):2212–2221CrossRef Poustelnikova E, Pisarev A, Blagov M, Samsonova M, Reinitz J (2004) A database for management of gene expression data in situ. Bioinformatics 20(14):2212–2221CrossRef
Metadaten
Titel
Novel approaches for hyper-parameter tuning of physics-informed Gaussian processes: application to parametric PDEs
verfasst von
Masoud Ezati
Mohsen Esmaeilbeigi
Ahmad Kamandi
Publikationsdatum
08.04.2024
Verlag
Springer London
Erschienen in
Engineering with Computers / Ausgabe 5/2024
Print ISSN: 0177-0667
Elektronische ISSN: 1435-5663
DOI
https://doi.org/10.1007/s00366-024-01970-8