A team deploys an ML classifier to detect fraudulent credit card transactions, which make up only 0.5% of all transactions. The test accuracy is reported as 99.5%, but in production, the model fails to block any actual fraudulent transactions. Why is relying solely on accuracy a major issue in this scenario?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Think about this: if you have 100 emails, and 99 of them are spam, you could write a super simple program that just says 'everything is spam' and boom—you've got 99% accuracy! Pretty cool, right? Well, not really, because that one important email from your boss gets tossed in the trash! That's the classic class imbalance trap. When one class dominates the dataset, accuracy becomes a completely useless metric. It makes you feel good with a high percentage, but in the real world, your model is blind to the rare events you actually care about—like credit card fraud or network intrusion. You need metrics like precision, recall, or the F1-score to tell you the real story here. Trust me, relying only on accuracy will bite you in production.
Full explanation below image
Full Explanation
The primary issue with using classification accuracy in an imbalanced dataset is that accuracy measures the ratio of correct predictions to total predictions without considering the distribution of classes. In scenarios where a single class dominates (known as class imbalance), a naive classifier that simply predicts the majority class for every instance will achieve a very high accuracy rate. For example, if 99% of the test samples belong to Class A and only 1% belong to Class B, a model that constantly predicts Class A will achieve 99% accuracy. However, this model has zero predictive power for Class B, which is often the minority class of interest (such as fraud detection, medical diagnoses, or equipment failure).
To address this, machine learning practitioners use alternative evaluation metrics: 1. Precision: Measures the ratio of true positives to the total predicted positives, indicating how reliable the positive predictions are. 2. Recall (Sensitivity): Measures the ratio of true positives to the actual positive instances, indicating the model's ability to find all positive instances. 3. F1-Score: The harmonic mean of precision and recall, providing a balanced metric that is robust to class imbalance. 4. ROC-AUC (Receiver Operating Characteristic - Area Under Curve): Evaluates the model's ability to distinguish between classes across various threshold settings.
Using accuracy alone hides the model's failure to learn the underlying characteristics of the minority class, leading to severe failures in real-world deployment where identifying the minority class is critical.