1 Introduction
-
an extension of affine-invariant ensemble transform approaches, such as the EnKF, to logistic regression,
-
an affine-invariant generalisation of the FPF and its application to logistic regression,
-
an extension of data sub-sampling (mini-batches) to Bayesian homotopy methods.
-
a derivative-free and affine-invariant SDE-based sampling methods for logistic regression.
2 Mathematical Problem Formulation
3 The Homotopy Approach to Bayesian Inference
3.1 Affine-Invariance
3.2 Data Sub-sampling
4 Ensemble Transform Algorithms
4.1 Diffusion Maps and the FPF
4.2 Second-Order Methods
4.3 Ensemble Kalman–Bucy Filter
4.4 Dropout and Time-Stepping
5 Ensemble Transform Langevin Dynamics
5.1 Asymptotic Behaviour
5.2 Numerical Implementation
6 Numerical Example
Ensemble size/method |
\(M=50\)
|
\(M=100\)
|
\(M=200\)
|
\(M=400\)
|
---|---|---|---|---|
McKean–Vlasov SDE |
\(\left( \begin{array}{c}-3.34\\ -3.39 \\ 3.22 \end{array}\right) \)
|
\(\left( \begin{array}{c} -3.35\\ -3.39 \\ 3.21 \end{array}\right) \)
|
\(\left( \begin{array}{c}-3.35\\ -3.39 \\ 3.21 \end{array}\right) \)
|
\(\left( \begin{array}{c} -3.35\\ -3.39 \\ 3.22 \end{array}\right) \)
|
FPF |
\(\left( \begin{array}{c} -3.31\\ -3.35 \\ 3.20 \end{array}\right) \)
|
\(\left( \begin{array}{c} -3.33\\ -3.37 \\ 3.21 \end{array}\right) \)
|
\(\left( \begin{array}{c} -3.33\\ -3.37 \\ 3.21 \end{array}\right) \)
|
\(\left( \begin{array}{c} -3.33\\ -3.37 \\ 3.21 \end{array}\right) \)
|
Second-order |
\(\left( \begin{array}{c} -3.36\\ -3.40 \\ 3.22 \end{array}\right) \)
|
\(\left( \begin{array}{c} -3.36\\ -3.41 \\ 3.22 \end{array}\right) \)
|
\(\left( \begin{array}{c} -3.36\\ -3.41 \\ 3.22 \end{array}\right) \)
|
\(\left( \begin{array}{c} -3.36\\ -3.41 \\ 3.22 \end{array}\right) \)
|
EnKBF |
\(\left( \begin{array}{c} -3.27\\ -3.31 \\ 3.20 \end{array}\right) \)
|
\(\left( \begin{array}{c} -3.27\\ -3.32 \\ 3.20 \end{array}\right) \)
|
\(\left( \begin{array}{c} -3.27\\ -2.31 \\ 3.19 \end{array}\right) \)
|
\(\left( \begin{array}{c} -3.27\\ -3.31 \\ 3.19 \end{array}\right) \)
|
Ensemble size/method |
\(M=50\)
|
\(M=100\)
|
\(M=200\)
|
\(M=400\)
|
---|---|---|---|---|
McKean–Vlasov SDE | 0.89 | 0.83 | 0.80 | 0.78 |
FPF | 0.98 | 0.90 | 0.86 | 0.83 |
Second-order | 0.75 | 0.75 | 0.74 | 0.74 |
EnKBF | 0.80 | 0.79 | 0.78 | 0.78 |
Ensemble size/method |
\(M=50\)
|
\(M=100\)
|
\(M=200\)
|
\(M=400\)
|
---|---|---|---|---|
McKean–Vlasov SDE |
\(\left( \begin{array}{c} -2.61\\ -2.65 \\ 2.15 \end{array}\right) \)
|
\(\left( \begin{array}{c} -2.60\\ -2.64 \\ 2.15 \end{array}\right) \)
|
\(\left( \begin{array}{c} -2.59\\ -2.62 \\ 2.15 \end{array}\right) \)
|
\(\left( \begin{array}{c} -2.59\\ -2.62 \\ 2.15 \end{array}\right) \)
|
FPF |
\(\left( \begin{array}{c} -2.16\\ -2.15 \\ 1.64 \end{array}\right) \)
|
\(\left( \begin{array}{c} -2.28\\ -2.30 \\ 1.83 \end{array}\right) \)
|
\(\left( \begin{array}{c} -2.39\\ -2.41 \\ 1.95 \end{array}\right) \)
|
\(\left( \begin{array}{c} -2.44\\ -2.47 \\ 2.00 \end{array}\right) \)
|
Second-order |
\(\left( \begin{array}{c} -2.29\\ -2.32 \\ 1.82 \end{array}\right) \)
|
\(\left( \begin{array}{c} -2.30\\ -2.32 \\ 1.83 \end{array}\right) \)
|
\(\left( \begin{array}{c} -2.31\\ -2.33 \\ 1.83 \end{array}\right) \)
|
\(\left( \begin{array}{c} -2.31\\ -3.34 \\ 1.84 \end{array}\right) \)
|
EnKBF |
\(\left( \begin{array}{c} -2.14\\ -2.16 \\ 1.73 \end{array}\right) \)
|
\(\left( \begin{array}{c} -2.16\\ -2.18 \\ 1.74 \end{array}\right) \)
|
\(\left( \begin{array}{c} -2.17\\ -2.19 \\ 1.75 \end{array}\right) \)
|
\(\left( \begin{array}{c} -2.17\\ -2.19 \\ 1.76 \end{array}\right) \)
|
Ensemble size/method |
\(M=50\)
|
\(M=100\)
|
\(M=200\)
|
\(M=400\)
|
---|---|---|---|---|
McKean–Vlasov SDE | 1.36 | 1.14 | 1.03 | 0.97 |
FPF | 3.61 | 2.58 | 2.01 | 1.66 |
Second-order | 0.46 | 0.45 | 0.45 | 0.45 |
EnKBF | 0.60 | 0.59 | 0.59 | 0.59 |
Ensemble size/method | \(M=20\) | \(M=60\) | \(M=100\) |
---|---|---|---|
EnKBF | 6.26 ± 0.75 | 2.67 ± 0.68 | 1.69 ± 0.48 |
EnKBF (with dropout) | 1.29 ± 0.29 | 1.28 ± 0.30 | 1.39 ± 0.46 |
EnKBF (with dropout & mini-batch) | 2.14 ± 0.46 | 1.42 ± 0.25 | 1.35 ± 0.21 |
Ensemble size/method | \(M=20\) | \(M=60\) | \(M=100\) |
---|---|---|---|
EnKBF | 0.014 ± 0.004 | 0.058 ± 0.012 | 0.097 ± 0.015 |
EnKBF (with dropout) | 0.043 ± 0.012 | 0.088 ± 0.022 | 0.109 ± 0.025 |
EnKBF (with dropout and mini-batch) | 0.041 ± 0.012 | 0.084 ± 0.021 | 0.105 ± 0.025 |
ensemble size/method |
\(M=20\)
|
\(M=60\)
|
\(M=100\)
|
---|---|---|---|
EnKBF (with dropout) | 3.30±1.01 | 1.26±0.23 | 1.12±0.36 |
EnKBF (with dropout and mini-batch) | 3.39±0.86 | 1.38±0.25 | 1.19±0.27 |