Memory-Efficient Backpropagation for Recurrent Neural Networks

Recurrent Neural Networks (RNN) process sequential data to capture the time-dependency in the input signal. Training a deep RNN conventionally involves segmenting the data sequence to fit the model into memory. Increasing the segment size permits the model to better capture long-term dependencies at the expense of creating larger models that may not fit in memory. Therefore, we introduce a technique to allow designers to train a segmented RNN and obtain the same model parameters as if the entire data sequence was applied regardless of the segment size. This enables an optimal capturing of long-term dependencies. This technique can increase the computational complexity during training. Hence, the proposed technique grants designers the flexibility of balancing memory and runtime requirements. To evaluate the proposed method, we compared the total loss achieved on the testing dataset after every epoch while varying the size of the segments. The results we achieved show matching loss graphs irrespective of the segment size.