Advertisement
Published in:
2014 | OriginalPaper | Chapter
Improving the Convergence Property of Soft Committee Machines by Replacing Derivative with Truncated Gaussian Function
Authors : Kazuyuki Hara, Kentaro Katahira
Published in: Artificial Neural Networks and Machine Learning – ICANN 2014
Publisher: Springer International Publishing
Activate our intelligent search to find suitable subject content or patents.
Select sections of text to find matching patents with Artificial Intelligence. powered by
Select sections of text to find additional relevant content using AI-assisted search. powered by
In online gradient descent learning, the local property of the derivative of the output function can cause slow convergence. This phenomenon, called a
plateau
, occurs in the learning process of a multilayer network. Improving the derivative term, we propose a simple method replacing the derivative term with a truncated Gaussian function that greatly increases the convergence speed. We then analyze a soft committee machine trained by proposed method, and show how proposed method breaks a plateau. Results showed that the proposed method eventually led to break the symmetry between hidden units.