During the training of a deep neural network, you monitor the logs and notice that the loss suddenly becomes NaN (Not a Number) and the model's weight updates are swinging wildly. When you look closer at the backpropagation step, you see that the gradient values are multiplying exponentially as they flow backward through the layers. What is the name of this classic deep learning issue?

The exploding gradient problem occurs during the training of deep neural networks when the gradients of the loss function accumulate and grow exponentially as they are backpropagated through the network. This mathematical phenomenon is primarily caused by multiplying gradients that are greater than one across many layers during backpropagation, according to the chain rule of calculus.

When gradients explode, they cause massive updates to the network's weights. This leads to unstable training, where the model fails to converge, and the loss function swings wildly or outputs NaN (Not a Number) values because the numbers exceed the floating-point representation limits of the hardware. This is particularly common in Recurrent Neural Networks (RNNs) due to the sharing of weights over long temporal sequences.

To mitigate exploding gradients, engineers use techniques like gradient clipping (which caps the gradients at a maximum threshold), weight regularization, batch normalization, or specialized architectures like Long Short-Term Memory (LSTM) networks.

Let's look at the distractors: Vanishing gradients are the exact opposite, where gradients shrink toward zero, causing the early layers of the network to stop learning. Overfitting refers to a model learning the noise in the training data too well, while underfitting means the model is too simple to capture the underlying patterns. Neither of these latter terms directly describes the physical explosion of derivative values during the backpropagation calculation.