Recurrent neural networks (RNNs) provide state-of-the-art performance in processing sequential data but are memory intensive to train, limiting the flexibility
of RNN models which can be trained. Reversible RNNs—RNNs for which the
hidden-to-hidden transition can be reversed—offer a path to reduce the memory
requirements of training, as hidden states need not be stored and instead can be
recomputed during backpropagation. We first show that perfectly reversible RNNs,
which require no storage of the hidden activations, are fundamentally limited because they cannot forget information from their hidden state. We then provide a
scheme for storing a small number of bits in order to allow perfect reversal with
forgetting. Our method achieves comparable performance to traditional models
while reducing the activation memory cost by a factor of 10–15. We extend our
technique to attention-based sequence-to-sequence models, where it maintains
performance while reducing activation memory cost by a factor of 5–10 in the
encoder, and a factor of 10–15 in the decoder.
