4. Specific Network Descriptions

We developed an algebraic framework for a generic layered network in the preceding chapter, including a method to express error backpropagation and loss function derivatives directly over the inner product space in which the network parameters are defined. We dedicate this chapter to expressing three common neural network structures within this generic framework: the Multilayer Perceptron (MLP), the Convolutional Neural Network (CNN), and the Deep Auto-Encoder (DAE). The exact layout of this chapter is as follows. We first explore the simple case of the MLP, deriving the canonical vector-valued form of backpropagation along the way. Then, we shift our attention to the CNN. Here, our layerwise function is far more complicated, as our inputs and parameters are in tensor product spaces, and thus we require more complex operations to combine the inputs and the parameters. CNNs still fit squarely in the framework developed in the previous chapter. The final network that we consider in this chapter, the DAE, does not fit as easily into that framework, as the parameters at any given layer have a deterministic relationship with the parameters at exactly one other layer, violating the assumption of parametric independence across layers. We overcome this issue, however, with a small adjustment to the framework. An algorithm for one step of gradient descent is derived for each type of network.