You're fine-tuning a machine learning model that has hundreds of input features, but you suspect a lot of them are just noise. You want to apply regularization that not only prevents overfitting but also acts as a built-in feature selector by driving the weights of the useless features all the way to zero. Which regularization technique should you implement, and how does its behavior compare to the alternative?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out — if you're dealing with a massive dataset with tons of features, you don't want to carry around dead weight. L1 regularization (we call it Lasso) is your best friend here. It penalizes the absolute value of the weights, which actually drives the weights of the useless features all the way to absolute zero. That means you get feature selection built right in! L2 regularization (Ridge) is different. It squares the weights, so it shrinks them down close to zero, but it never actually crosses the finish line to make them zero. Think of it like this: L1 uses a pair of scissors to cut out features, while L2 just puts them on a strict diet.
Full explanation below image
Full Explanation
Regularization is a fundamental technique used in machine learning to prevent overfitting by adding a penalty term to the loss function. The two most common types are L1 regularization (Lasso) and L2 regularization (Ridge).
The core mathematical difference lies in the penalty function. L1 regularization adds a penalty equal to the sum of the absolute values of the weights (L1 norm). Because of the geometry of the absolute value function, L1 optimization often leads to corner solutions where some weights are driven to exactly zero. This makes L1 highly useful for feature selection, as it produces sparse models where only a subset of features is actively used.
On the other hand, L2 regularization adds a penalty equal to the sum of the squared values of the weights (L2 norm). The squared penalty term heavily penalizes large weights but becomes extremely small as the weights approach zero. As a result, L2 regularization shrinks all weights toward zero but rarely, if ever, makes them exactly zero. It retains all features in the model, distributing the weights more evenly, which is ideal when you have many collinear or moderately important features.
Understanding this distinction is critical for architecture and feature engineering. If your boss walks in and asks you to simplify a model running on low-resource devices, L1 is your go-to because it trims the fat by dropping features entirely. L2 is better suited when you want to keep all features but keep their influence in check. Distractors claiming that one is only for regression or that they behave identically fail to capture these fundamental mathematical and practical differences.