To combat overfitting in a deep neural network, an engineer decides to implement a regularization technique that randomly deactivates a specified percentage of neurons during each training step. What is this technique called?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Okay, let's dive into regularization. Think of it like this: if you're preparing for a big exam and you always study with the exact same group of friends, you might start relying on one super-smart friend to answer all the hard questions. If they get sick on exam day, you're in big trouble! Dropout does something similar to a neural network. During training, it randomly 'drops out'—or deactivates—a fraction of the neurons in a layer. This forces the remaining neurons to step up, learn the features independently, and prevents the network from becoming co-dependent on a few specific pathways. It's one of the absolute best ways to stop overfitting. So, C is the correct answer. Option A, batch normalization, is about scaling inputs to layers, not deactivating neurons. Option B is an evaluation method, and Option D is halting training early. By the way, if you ever see a weird translation like 'give up' in a test, now you know they actually meant Dropout!
Full explanation below image
Full Explanation
Dropout is a widely used regularization technique designed to prevent overfitting in deep neural networks. During training, dropout randomly deactivates a pre-defined fraction of neurons (along with their incoming and outgoing connections) at each update step. This forces the neural network to learn redundant representations and prevents co-adaptation of features, where certain neurons rely heavily on the outputs of specific neighboring neurons. During inference (evaluation and testing), dropout is turned off, and all neurons are active; however, their weights are scaled down by the dropout rate to ensure the expected values remain consistent with the training phase.
Let's look at why the other options are incorrect: - Option A is incorrect because Batch Normalization is a technique that standardizes the inputs to a layer for each mini-batch. While it has some weak regularizing side effects, its primary purpose is to stabilize and accelerate training. - Option B is incorrect because Cross-validation is an evaluation methodology used to estimate how well a model generalizes to unseen data by partitioning the dataset, not a regularization technique applied to network layers during training. - Option D is incorrect because Early Stopping is a regularization method that monitors validation loss and halts the entire training process when performance stops improving, rather than modifying the active neurons within layers.