You are training a very deep feedforward neural network using sigmoid activation functions. After several epochs, you observe that the weights of the layers closest to the input are barely changing, while the outer layers are training normally. What common neural network problem are you experiencing, and what is its primary cause?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Pay close attention here, because this one bites people all the time, both in production and on the exam! When you use backpropagation in a really deep network—especially with activations like sigmoid or tanh—you have to multiply a bunch of small numbers together as you pass the error back from the output to the input. Think about it: if you multiply 0.1 by 0.1 by 0.1, the number gets tiny fast. By the time that gradient signal reaches the first few layers, it's practically zero! Those early layers don't learn a thing, and your network stalls out. That's the vanishing gradient problem. Exploding gradients do the opposite (they blow up to infinity). Underfitting happens when a model is too simple, not too complex, and overfitting is when a complex model memorizes the training data. Remember: vanishing gradients keep early layers in the dark. Let's keep rolling!
Full explanation below image
Full Explanation
The correct answer is Vanishing Gradients (Option C). During the backpropagation phase in a deep neural network, gradients are calculated using the chain rule, multiplying derivatives from the output layer back to the input layer. When using activation functions whose derivatives are less than 1 (such as the sigmoid function, which has a maximum derivative of 0.25), multiplying these small values repeatedly across many layers causes the gradient to decrease exponentially. Consequently, the gradient updates for weights in the early layers become virtually zero, halting their training and preventing the network from learning deep features. Option A is incorrect because exploding gradients occur when weights or derivatives are large (greater than 1), causing gradients to grow exponentially and lead to numerical instability, not slow learning due to small values. Option B is incorrect because overfitting is typically a symptom of a highly complex model memorizing the training data, rather than a simple model. Option D is incorrect because underfitting is a symptom of a model that is too simple (or undertrained) to capture the underlying pattern of the data, not a highly complex model.