You are training a deep feedforward neural network and notice that as you add more hidden layers, the model stops learning because the gradients propagating back to the early layers become extremely small. To resolve this, you decide to replace the Sigmoid activation functions in the hidden layers with Rectified Linear Unit (ReLU) functions. What is the primary advantage of ReLU that addresses this issue?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out: in the early days of neural networks, everyone used activation functions like Sigmoid or Tanh. But here's the catch—if you look at a Sigmoid curve, it flattens out at the very high and very low ends. When your inputs land in those flat zones, the gradient drops almost to zero. If you have a deep network with ten or twenty layers, those tiny gradients get multiplied together during backpropagation. By the time they reach the first few layers, they vanish completely, and your network stops learning! ReLU, or Rectified Linear Unit, is a game-changer. For any positive input, the slope is a constant 1. It doesn't saturate or flatten out, which means those gradients keep flowing backward loud and clear. That makes training run way, way faster. Trust me, for hidden layers, ReLU is your default choice.
Full explanation below image
Full Explanation
The activation function in a neural network introduces non-linearity, enabling the model to learn complex patterns. The Sigmoid function, defined as $f(x) = \frac{1}{1 + e^{-x}}$, maps inputs to a range between 0 and 1. However, for very large positive or negative inputs, the gradient of the Sigmoid function approaches zero (saturation). During backpropagation, these small gradients are multiplied across multiple layers, causing the gradients to diminish exponentially (vanishing gradient problem), which prevents early layers from learning. The Rectified Linear Unit (ReLU) activation function, defined as $f(x) = \max(0, x)$, solves this issue for positive inputs. Since the derivative of ReLU is 1 for any input $x > 0$, the gradient does not saturate or fade as it travels backward through the network. This constant gradient allows deep networks to train much faster and more stably. Sigmoid, not ReLU, maps outputs strictly between 0 and 1. ReLU is computationally very simple and cheap to compute (a simple threshold at zero), unlike Sigmoid which requires exponential calculations. Finally, while ReLU does output zero for negative inputs (which introduces sparsity), its goal is not to force the network into only learning linear relationships, as the overall network remains highly non-linear due to the active neurons. Therefore, ReLU's main advantage is preventing the vanishing gradient problem for positive inputs.