Reservoir Sizes and Feedback Weights Interact Non-linearly in Echo State Networks

In this paper we investigate parameter dependencies in the echo state network (ESN). In particular, we investigate the interplay between reservoir sizes and the choice of the average absolute output feedback connection weight values (

W

OFB

). We consider the multiple sine wave oscillator problem and the powered sine problem. The results show that somewhat contrary to basic intuition (1) smaller reservoir sizes often yield better networks with higher probability; (2) large

W

OFB

values paired with comparatively large reservoirs may strongly decrease the likelihood of generating effective networks; (3) the likelihood of generating an effective ESN depends non-linearly on the choice of

W

OFB

: very small and large weight values often yield higher likelihoods of generating effective ESNs than networks resulting from intermediate

W

OFB

choices. While the considered test problems are rather simple, the insights gained need to be considered when designing effective ESNs for the problem at hand. Nonetheless, further studies appear necessary to be able to explain the actual reasons behind the observed phenomena.