During the training of a deep convolutional neural network, you notice that as weights in early layers change, the distribution of inputs to later layers shifts dramatically, causing training to be slow and unstable. Which technique should you implement to standardize the inputs of each layer, thereby stabilizing and accelerating the training process?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: when you train a deep network, each layer's inputs keep shifting around because the previous layers are constantly updating their weights. We call this internal covariate shift, and it makes training a slow, painful crawl. Think of it like trying to hit a moving target while standing on a skateboard. Batch normalization solves this by stepping in and saying, 'Hey, let's normalize the inputs to this layer for each mini-batch so they always have a mean of zero and variance of one.' This stabilizes the whole network, meaning you can crank up the learning rate and train way faster. Trust me on this, it's one of the best tricks in deep learning. Dropout is for regularization, learning rate decay adjusts the optimizer step size over time, and gradient clipping is just a band-aid for exploding gradients. Batch normalization is what actually keeps the layers steady. Got it? Sweet.
Full explanation below image
Full Explanation
The correct answer is Batch Normalization (Option C). During training, the distribution of each layer's inputs changes as the parameters of the preceding layers change. This phenomenon, known as internal covariate shift, requires subsequent layers to continuously adapt to new distributions, slowing down training. Batch normalization addresses this by normalizing the activations of a layer across each mini-batch. Specifically, it scales and shifts the activations to have a mean of zero and a variance of one, using learned parameters (gamma and beta) to preserve the network's representational power. This stabilization allows for higher learning rates, reduces sensitivity to initialization, and acts as a mild regularizer, significantly accelerating training. Option A (Dropout) is incorrect because dropout is a regularization technique that randomly deactivates neurons during training to prevent overfitting, rather than normalizing layer inputs. Option B (Learning Rate Decay) is incorrect because it gradually reduces the learning rate over time to help the model converge to a local minimum, but does not standardize layer inputs. Option D (Gradient Clipping) is incorrect because it caps gradients at a maximum threshold to prevent exploding gradients, rather than standardizing activations within layers.