1 Introduction
2 Kriging
3 Gradient-enhanced kriging
4 Noise regularisation
5 Hyper-parameter tuning
6 Obtaining gradients
6.1 Gradients of the condensed likelihood function w.r.t. the correlation matrix
6.2 Gradients of the condition number w.r.t. the correlation matrix
6.3 Gradients of the correlation matrix w.r.t. regularisation parameters
6.4 Gradients of the correlation matrix w.r.t. the hyper parameters
7 Computational performance
Variables | Description | Time [s] |
---|---|---|
Objective and constraint functions
| ||
R,B,f
| Pre-processing | 0.3 |
LL
T
| Cholesky decomposition | 1.7 |
|R| | Determinant | <0.1 |
\(\hat {\mu }\)
| System mean | <0.1 |
\(\hat {\sigma }^{2}\)
| System variance | <0.1 |
ϕ
| Condensed likelihood function | <0.1 |
R
−1
| Inverse of R
| 4.7 |
κ
| Condition number of R
| 0.1 |
Total | 6.9 | |
Derivatives of the objective and constraint functions
| ||
∂
ϕ/∂
R
| Partial derivatives of ϕ w.r.t. R
| 0.7 |
∂
κ/∂
R
| Partial derivatives of κ w.r.t. R
| 20.8 |
∂
R/∂
𝜃
| Partial derivatives of R w.r.t. 𝜃
| 23.9 |
∂
R/∂
λ
| Partial derivatives of R w.r.t. λ
| <0.1 |
Total | 45.4 |
8 Proposed optimisation approach
8.1 Finding a suitable starting point
8.2 Gradient based optimisation
9 Comparative study of optimisation approaches
Abbreviation | Optimisation method |
---|---|
GS | Golden search |
R-MFD | Random start MFD |
R-SQP | Random start SQP |
GS-MFD | MFD starting from GS result |
GS-SQP | SQP starting from GS result |
M-MFD | Multi-start MFD |
M-SQP | Multi-start SQP |
GA | Genetic algorithm |
GA-MFD | MFD starting from GA result |
GA-SQP | SQP starting from GA result |
9.1 Two dimensional benchmark study
Function name | Equation |
---|---|
Six-hump |
\(f(\textbf {x}) = \left (4 - 2.1 {x_{1}^{2}} + \frac {{x^{4}_{1}}}{3}\right ){x^{2}_{1}} + x_{1} x_{2} + (4{x^{2}_{2}} -4){x_{2}^{2}}\)
|
Branin-Hoo |
\(f(\textbf {x}) = \left (x_{2} - \frac {5.1{x_{1}^{2}}}{4\pi ^{2}} + \frac {5x_{1}}{\pi } - 6 \right )^{2}\)
|
\(\qquad \quad \,\,+ 10\left (1- \frac {1}{8\pi }\right ) cos(x_{1}) + 10\)
| |
Himmelblau |
\(f(\textbf {x}) = \left ({x_{1}^{2}} + x_{2} - 11\right )^{2} + \left (x_{1} + {x_{2}^{2}} - 7 \right )^{2}\)
|
Ursem |
f(x)=−s
i
n(2x
1−0.5π)−3c
o
s(x
2)−0.5x
1
|
Adjiman |
\(f(\textbf {x}) = cos(x_{1})sin(x_{2}) - \frac {x_{1}}{{x_{2}^{2}} +1}\)
|
Keane |
\(f(\textbf {x}) = \frac {sin^{2}(x_{1}-x_{2})sin^{2}(x1+x2)}{\sqrt {{x_{1}^{2}} + {x^{2}_{2}}}}\)
|
Time | Mean | RMSE | Time | Mean | RMSE | Time | Mean | RMSE | |
---|---|---|---|---|---|---|---|---|---|
(ms) |
ϕ
|
%
| (ms) |
ϕ
|
%
| (ms) |
ϕ
|
%
| |
Six Hump | Branin Hoo | Himmelblau | |||||||
GS | 4.8 | −41.05 | 12.89 % | 6.2 | −33.21 | 12.78 % | 3.8 | −27.29 | 3.69 % |
R-MFD | 3.5 | −35.20 | 10.90 % | 3.7 | −25.13 | 10.46 % | 3.6 | −29.68 | 5.23 % |
R-SQP | 34.2 | −34.49 | 10.78 % | 28.6 | −25.18 | 10.37 % | 20.2 | −32.38 | 6.26 % |
GS-MFD | 7.5 | −33.04 | 9.86 % | 9.9 | −23.39 | 9.57 % | 7.8 | −26.86 | 3.72 % |
GS-SQP | 12.1 | −33.38 | 10.07 % | 12.4 | −23.56 | 9.67 % | 8.6 | −26.86 | 3.71 % |
M-MFD | 19.7 | −33.27 | 9.81 % | 15.5 | −23.36 | 9.69 % | 15.4 | −25.88 | 2.94 % |
M-SQP | 373.8 | −33.04 | 9.91 % | 253.9 | −23.24 | 9.64 % | 156.6 | −26.48 | 3.10 % |
GA | 376.4 | −33.04 | 9.92 % | 376.3 | −23.24 | 9.64 % | 376.2 | −25.42 | 2.50 % |
GA-MFD | 379.9 | −33.03 | 9.92 % | 380.0 | −23.24 | 9.64 % | 379.1 | −25.39 | 2.49 % |
GA-SQP | 383.3 | −33.03 | 9.92 % | 382.6 | −23.24 | 9.64 % | 382.2 | −25.39 | 2.49 % |
Ursem | Adjiman | Keane | |||||||
GS | 4.6 | −2.26 | 1.71 % | 4.3 | 51.20 | 0.19 % | 4.9 | −31.25 | 10.10 % |
R-MFD | 4.6 | 8.70 | 4.29 % | 4.1 | 12.86 | 5.49 % | 3.5 | −31.11 | 10.18 % |
R-SQP | 20.6 | 17.27 | 1.60 % | 12.2 | 34.68 | 2.56 % | 16.5 | −32.65 | 11.44 % |
GS-MFD | 8.8 | 23.19 | 0.68 % | 7.1 | 51.97 | 0.19 % | 6.5 | −31.11 | 10.17 % |
GS-SQP | 10.4 | 22.28 | 0.70 % | 9.4 | 52.02 | 0.18 % | 7.7 | −31.11 | 10.17 % |
M-MFD | 16.5 | 23.19 | 0.68 % | 20.7 | 47.86 | 0.47 % | 15.3 | −31.09 | 9.77 % |
M-SQP | 160.6 | 23.19 | 0.68 % | 113.7 | 52.04 | 0.19 % | 137.3 | −31.11 | 10.17 % |
GA | 373.9 | 23.19 | 0.68 % | 374.5 | 52.02 | 0.19 % | 378.4 | −31.11 | 10.17 % |
GA-MFD | 377.3 | 23.19 | 0.68 % | 377.7 | 52.04 | 0.19 % | 381.9 | −31.11 | 10.17 % |
GA-SQP | 378.9 | 23.19 | 0.68 % | 379.9 | 52.03 | 0.19 % | 385.2 | −31.11 | 10.17 % |
9.2 Dimensionally scalable benchmark study
10 Training Points | 20 Training Points | 50 Training Points | |||||||
---|---|---|---|---|---|---|---|---|---|
Time | Mean | RMSE | Time | Mean | RMSE | Time | Mean | RMSE | |
(h
h : m
m : s
s) |
ϕ
| % | (h
h : m
m : s
s) |
ϕ
| % | (h
h : m
m : s
s) |
ϕ
| % | |
GS | <00:00:01 | 39.12 | 16.32 % | <00:00:01 | 107.50 | 14.93 % | <00:00:01 | 428.49 | 10.76 % |
R-MFD | <00:00:01 | 66.90 | 16.44 % | 00:00:02 | 182.21 | 12.73 % | 00:00:10 | 618.52 | 9.77 % |
R-SQP | <00:00:01 | 54.22 | 16.05 % | <00:00:01 | 142.08 | 15.17 % | 00:00:03 | 548.38 | 12.61 % |
GS-MFD | <00:00:01 | 76.97 | 13.88 % | 00:00:01 | 202.91 | 10.66 % | 00:00:04 | 729.28 | 8.40 % |
GS-SQP | <00:00:01 | 77.19 | 13.59 % | <00:00:01 | 197.48 | 10.99 % | 00:00:03 | 729.72 | 8.39 % |
M-MFD | 00:00:01 | 71.72 | 15.57 % | 00:00:11 | 193.84 | 11.28 % | 00:01:13 | 671.56 | 8.86 % |
M-SQP | <00:00:01 | 74.74 | 13.96 % | 00:00:01 | 171.87 | 13.62 % | 00:00:31 | 718.10 | 8.58 % |
GA | 00:00:03 | 74.19 | 14.47 % | 00:00:13 | 183.26 | 12.07 % | 00:01:06 | 656.52 | 8.84 % |
GA-MFD | 00:00:03 | 74.74 | 14.37 % | 00:00:14 | 199.83 | 10.85 % | 00:01:10 | 728.86 | 8.39 % |
GA-SQP | 00:00:03 | 77.81 | 13.44 % | 00:00:13 | 193.35 | 11.39 % | 00:01:08 | 729.68 | 8.39 % |
10 Training Points | 20 Training Points | 50 Training Points | |||||||
---|---|---|---|---|---|---|---|---|---|
Time | Mean | RMSE | Time | Mean | RMSE | Time | Mean | RMSE | |
(h
h : m
m : s
s) |
ϕ
| % | (h
h : m
m : s
s) |
ϕ
| % | (h
h : m
m : s
s) |
ϕ
| % | |
GS | <00:00:01 | 519.24 | 15.67 % | 00:00:01 | 1153.24 | 14.60 % | 00:00:11 | 3190.31 | 13.85 % |
R-MFD | 00:00:05 | 610.67 | 17.44 % | 00:00:23 | 1328.32 | 17.19 % | 00:03:07 | 3647.07 | 16.38 % |
R-SQP | 00:00:01 | 566.24 | 17.51 % | 00:00:12 | 1329.35 | 17.22 % | 00:03:01 | 3937.76 | 12.99 % |
GS-MFD | 00:00:06 | 664.50 | 15.54 % | 00:00:25 | 1435.77 | 14.31 % | 00:03:10 | 3945.74 | 12.87 % |
GS-SQP | 00:00:03 | 669.91 | 15.98 % | 00:00:12 | 1437.64 | 14.51 % | 00:01:18 | 3946.86 | 12.92 % |
M-MFD | 00:00:49 | 632.29 | 17.38 % | 00:03:43 | 1350.05 | 17.14 % | 00:29:34 | 3697.19 | 14.91 % |
M-SQP | 00:00:13 | 603.41 | 17.40 % | 00:02:10 | 1355.04 | 16.83 % | 00:30:59 | 3946.38 | 12.92 % |
GA | 00:00:41 | 622.84 | 17.48 % | 00:03:00 | 1321.30 | 17.26 % | 00:20:47 | 3564.98 | 17.01 % |
GA-MFD | 00:00:43 | 633.20 | 17.41 % | 00:03:14 | 1347.24 | 17.18 % | 00:23:43 | 3696.73 | 14.97 % |
GA-SQP | 00:00:44 | 659.21 | 16.42 % | 00:03:04 | 1352.33 | 16.84 % | 00:22:39 | 3933.32 | 13.07 % |
10 Training Points | 20 Training Points | 50 Training Points | |||||||
---|---|---|---|---|---|---|---|---|---|
Time | Mean | RMSE | Time | Mean | RMSE | Time | Mean | RMSE | |
(h
h : m
m : s
s) |
ϕ
| % | (h
h : m
m : s
s) |
ϕ
| % | (h
h : m
m : s
s) |
ϕ
| % | |
GS | <00:00:01 | 926.49 | 15.03 % | 00:00:03 | 2048.53 | 14.13 % | 00:00:26 | 5487.08 | 13.58 % |
R-MFD | 00:00:12 | 1040.72 | 16.91 % | 00:01:03 | 2291.84 | 16.66 % | 00:07:54 | 6133.60 | 16.20 % |
R-SQP | 00:00:07 | 1041.60 | 16.88 % | 00:00:43 | 2329.69 | 16.59 % | 00:10:27 | 6562.21 | 13.00 % |
GS-MFD | 00:00:12 | 1137.31 | 14.98 % | 00:01:07 | 2463.50 | 14.05 % | 00:08:04 | 6559.71 | 12.92 % |
GS-SQP | 00:00:09 | 1156.69 | 16.00 % | 00:00:36 | 2471.26 | 14.38 % | 00:04:13 | 6563.42 | 13.00 % |
M-MFD | 00:02:12 | 1082.86 | 16.87 % | 00:10:42 | 2332.23 | 16.63 % | 01:21:39 | 6167.31 | 15.79 % |
M-SQP | 00:01:16 | 1101.63 | 16.62 % | 00:07:33 | 2376.14 | 15.71 % | 01:45:10 | 6563.21 | 13.00 % |
GA | 00:01:30 | 1050.56 | 16.95 % | 00:07:09 | 2241.87 | 16.70 % | 00:54:27 | 5898.04 | 16.56 % |
GA-MFD | 00:01:40 | 1080.82 | 16.92 % | 00:09:47 | 2320.88 | 16.67 % | 01:03:26 | 6126.89 | 16.27 % |
GA-SQP | 00:01:36 | 1093.91 | 16.59 % | 00:08:55 | 2309.32 | 16.68 % | 01:02:19 | 6562.34 | 13.00 % |
10 Case study: aircraft wing example
10.1 The wing model
10.2 Study set-up
10.3 Results
No. points | GS | R-MFD | R-SQP | GS-MFD | GS-SQP |
---|---|---|---|---|---|
(a) Condensed log likelihood (ϕ) | |||||
5 | 732 | 798 | 1082 | 1731 | 1780 |
10 | 2197 | 2173 | 4069 | 3915 | 4241 |
20 | 5406 | 5113 | 6434 | 8275 | 8725 |
50 | 14696 | 13912 | 17415 | 21737 | 21853 |
(b) Generalisation error (RMSE) | |||||
5 | 14.77 % | 14.78 % | 14.78 % | 13.27 % | 10.69 % |
10 | 6.56 % | 12.57 % | 10.84 % | 5.17 % | 7.66 % |
20 | 6.34 % | 12.64 % | 12.59 % | 5.01 % | 6.66 % |
50 | 5.82 % | 12.37 % | 12.32 % | 6.33 % | 7.26 % |
No. points | GS | R-MFD | R-SQP | GS-MFD | GS-SQP |
---|---|---|---|---|---|
(a) Condensed log likelihood (ϕ) | |||||
5 | 1360 | 1384 | 1500 | 2091 | 2157 |
10 | 2768 | 2621 | 3682 | 3949 | 3910 |
20 | 5982 | 5293 | 6371 | 7824 | 7916 |
50 | 15998 | 13494 | 16003 | 19123 | 19236 |
(b) Generalisation error (RMSE) | |||||
5 | 10.10 % | 8.16 % | 8.16 % | 9.07 % | 9.06 % |
10 | 4.50 % | 7.80 % | 7.64 % | 4.81 % | 6.32 % |
20 | 4.55 % | 7.62 % | 7.60 % | 3.91 % | 4.16 % |
50 | 4.07 % | 7.18 % | 7.18 % | 3.49 % | 3.68 % |