What is the primary difficulty encountered when training standard (vanilla) Recurrent Neural Networks (RNNs) on very long sequences?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Imagine you're trying to read a long book, but by the time you reach page 50, you've completely forgotten what happened on page 1. That's a standard RNN in a nutshell. When you train them on long sequences, you have to use a process called Backpropagation Through Time. If the weights are small, multiplying them repeatedly causes the gradients to shrink to zero—vanishing gradient. If the weights are large, they blow up to infinity—exploding gradient. Either way, the network stops learning or goes completely off the rails. It's a huge headache, and it's why we moved on to LSTMs, GRUs, and eventually Transformers.
Full explanation below image
Full Explanation
Vanilla Recurrent Neural Networks (RNNs) process sequences token-by-token, passing a hidden state forward through time. While this architecture is theoretically capable of modeling sequential dependencies of arbitrary length, in practice, training standard RNNs on sequences longer than a few dozen steps is extremely difficult due to unstable gradients.
During the backward pass of Backpropagation Through Time (BPTT), the gradient of the loss at a given time step is calculated with respect to parameters at much earlier steps. This computation involves a product of Jacobian matrices (derivatives of the hidden state transition). If the eigenvalues of the weight matrices are less than 1, the gradients decay exponentially as they propagate backward, leading to the vanishing gradient problem. The network fails to learn how early inputs affect late outputs. Conversely, if the eigenvalues are greater than 1, the gradients grow exponentially, leading to the exploding gradient problem, which causes numerical instability (e.g., NaN values) and weight updates that are too large.
Let's analyze the incorrect options: - Option B is incorrect because vanishing and exploding gradients make the model difficult to train, often resulting in underfitting, but not because the computation 'decreases too rapidly.' - Option C is incorrect because RNNs are specifically designed for sequential data; they are just limited by sequence length in practice. - Option D is incorrect. While sorting sequences by length (padding/packing) is an optimization technique used to speed up training in frameworks like PyTorch, it is not a structural limitation of standard RNNs.