You are constructing a deep neural network to classify network traffic into one of five mutually exclusive protocols. Which activation function should you apply to the output layer of this network to obtain a valid probability distribution?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: if you're building a neural network to classify images into three categories—say, routers, switches, and firewalls—your output layer is going to spit out raw scores called logits. But raw numbers like 4.2, 1.1, and -0.5 don't help you much. You need probabilities. That's where the Softmax function comes in. It takes those raw scores and squashes them into a neat probability distribution where every value is between 0 and 1, and—crucially—they all add up to exactly 1.0 (or 100%). If you only had a binary choice (yes/no), you'd use Sigmoid, but for multi-class, Softmax is your absolute must-have.
Full explanation below image
Full Explanation
In neural network design, the activation function chosen for the output layer is determined by the type of machine learning task being performed:
1. Multi-class Classification (Softmax): When a model must classify an input into one of three or more mutually exclusive classes, the output layer uses the Softmax activation function. Softmax normalizes the output values (logits) of the final layer into a probability distribution over the classes. Mathematically, it exponentiates each output and divides it by the sum of all exponentiated outputs. This ensures that: - Each output value is a probability between $0$ and $1$. - The sum of all output probabilities equals exactly $1$. 2. Binary Classification (Sigmoid): When there are only two possible classes, a Sigmoid function is used, outputting a single probability value between $0$ and $1$. 3. Hidden Layers (ReLU, Tanh): Functions like ReLU (Rectified Linear Unit) and Tanh (Hyperbolic Tangent) are used in hidden layers to introduce non-linearity, but they are not suitable for output layers of multi-class classifiers because they do not output normalized probabilities that sum to 1.
- Option A (ReLU) is incorrect because it outputs values from 0 to infinity, which cannot represent a probability distribution. - Option B (Tanh) is incorrect because it outputs values between -1 and 1, which cannot represent probabilities. - Option D (Sigmoid) is incorrect because it is designed for binary classification, mapping a single output to a range between 0 and 1, and does not coordinate multiple output nodes to sum to 1.