Introduction
Complex-step derivative approximation (CSDA)
Illustrative example
Implementation of CSDA in feed-forward neural networks
Illustrative example
@ x = pi/4 | Exact | CSDA | FDA | CFDA |
---|---|---|---|---|
Output | 3.10176 | 3.10167 | 3.55271 | 3.10862 |
Error | – | 2.9e−5 | 0.1454 | 2.2e−3 |
Regression
Datasets and FFNN Configurations
Function | Exact derivatives | |
---|---|---|
\({R}_{1}:\,y={x}_{1}^{4}+2{x}_{2}^{3}+3\sqrt{{x}_{3}}\) | \(\frac{\partial y}{\partial {x}_{1}}=4{x}_{1}^{3}; \frac{\partial y}{\partial {x}_{2}}=6{x}_{2}^{2}; \frac{\partial y}{\partial {x}_{3}}=\frac{3}{2\sqrt{{x}_{3}}}\) | |
\({R}_{2}:\,y=\mathrm{sin}(\pi {x}_{1})+{e}^{{x}_{2}}+{x}_{3}^{2}\): | \(\frac{\partial y}{\partial {x}_{1}}=\pi \mathrm{cos}\left(\pi {x}_{1}\right); \frac{\partial y}{\partial {x}_{2}}={e}^{{x}_{2}}; \frac{\partial y}{\partial {x}_{3}}=2{x}_{3}\) | |
\({R}_{3}:\,y=\mathrm{sin}(\pi {x}_{1})+{e}^{{x}_{2}}+{x}_{3}^{2}+0.00001{x}_{4}\) | \(\frac{\partial y}{\partial {x}_{4}}=1e-5\) |
Function | Range of input features |
---|---|
\({R}_{1}\) | \({x}_{1}\sim U\left(0, 1\right); {x}_{2}\sim U\left(1, 2\right); {x}_{3}\sim U\left(0.5, 5\right)\) |
\({R}_{2}\) | \({x}_{1}\sim U\left(-1, 1\right); {x}_{2}\sim U\left(0, 5\right); {x}_{3}\sim U\left(0, 3\right)\) |
\({R}_{3}\) | \({x}_{1}\sim U\left(-1, 1\right); {x}_{2}\sim U\left(0, 5\right); {x}_{3}\sim U\left(0, 3\right);{x}_{4}\sim U\left(0, 2\right)\) |
Comparison of CSDA-FFNN output and the exact analytical derivative
Classification
Dataset and CSDA implementation
Input feature, \(j=\) | 1 | 2 | 3 |
---|---|---|---|
\(\frac{\partial {\Sigma }_{{o}_{1}}}{\partial {x}_{j}}\) | 0.5009 | 0.4935 | 0.0056 |
\(\frac{\partial {\Sigma }_{{o}_{2}}}{\partial {x}_{j}}\) | − 0.5009 | − 0.4935 | − 0.0056 |
Predicted | ||
---|---|---|
Class label 1 | Class label 2 | |
Actual | ||
Class label 1 | 0.99 | 0.01 |
Class label 2 | 0.02 | 0.98 |