5. Recurrent Neural Networks

We applied the generic neural network framework from Chap. 3 to specific network structures in the previous chapter. Multilayer Perceptrons and Convolutional Neural Networks fit squarely into that framework, and we were also able to modify it to capture Deep Auto-Encoders. We now extend the generic framework even further to handle Recurrent Neural Networks (RNNs), the sequence-parsing network structure containing a recurring latent, or hidden, state that evolves at each layer of the network. This involves the development of new notation, but we remain as consistent as possible with previous chapters. The specific layout of this chapter is as follows. We first formulate a generic, feed-forward recurrent neural network. We calculate gradients of loss functions for these networks in two ways: Real-Time Recurrent Learning (RTRL) and Backpropagation Through Time (BPTT). Using our notation for vector-valued maps, we derive these algorithms directly over the inner product space in which the parameters reside. We then proceed to formally represent a vanilla RNN, which is the simplest form of RNN, and we formulate RTRL and BPTT for that as well. At the end of the chapter, we briefly mention modern RNN variants in the context of our generic framework.