Investigating Neural Activation Effects on Deep Belief Echo-State Networks for Prediction Toward Smart Ocean Environment Monitoring

Ocean sensor data prediction has become a promising means for smart ocean monitoring. In alternative solutions, deep neural networks (DNNs) are considered as a good choice. The determination of activation functions in DNNs has a significant effect on training speed and nonlinear approximation. In this paper, the effect of activation functions on a deep computing model called deep belief echo-state network (DBEN) is studied in the scenario of ocean time series prediction. Here, different forms, including hyperbolic tangent, rectified linear unit, exponential linear unit, swish, softplus and their variants, are considered. The purpose is to investigate, from the perspectives of accuracy and training efficiency, whether certain activation function in DBEN is completely universal for the different tasks of ocean sensor data processing or not. On a great deal of real-world ocean time series of different characteristics, the results show that the selection of activation functions in DBEN is task-related. Specially, these newly introduced activation functions are more beneficial to the accurate predictions for conventional and chemical data sets compared with sigmoid benchmark. The statistical analysis further verifies this finding.