An engineer building a sentiment classifier notices that highly frequent but uninformative words (such as 'really', 'extremely', and 'very') are dominating the model's feature weights and skewing the classification outputs. Which preprocessing step should the engineer implement to address this issue?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Imagine you're trying to figure out if a movie review is good or bad, but your model keeps focusing on words like 'very' or 'really'. Those words don't actually tell you what the customer is feeling—they just add emphasis to other words. In NLP, we call these common, low-value words 'stop words'. Standard libraries like NLTK or spaCy have built-in lists of stop words (like 'the', 'is', 'at'), but sometimes you need to add your own custom words to that list to stop them from skewing your model's training. Option A is exactly what you need to do here. The other options are about formatting words, not filtering them out.
Full explanation below image
Full Explanation
Stop words are words that occur frequently in a language but carry minimal semantic information (such as 'the', 'a', 'is', 'and', 'with'). In many text mining and classification tasks, these words can dominate frequency counts and feature representations without contributing useful signal, which can skew the performance of algorithms like Naive Bayes or TF-IDF.
To solve this, developers perform stop-word removal during the text preprocessing phase. While standard NLP libraries provide default stop-word lists, these lists may not cover all domain-specific or task-specific uninformative words. In this scenario, words like 'very' and 'really' are causing noise. The developer should modify the default stop-word list to include these custom terms so they are filtered out before vectorization and model training.
Let's review the incorrect options: - Option B, lemmatization, reduces words to their dictionary root form (e.g., 'running' -> 'run'), but does not filter out or remove words from the text. - Option C, stemming, trims prefixes or suffixes using heuristic rules (e.g., 'really' -> 'real'), but again, does not discard the words. - Option D, tokenization, is the basic step of splitting a text string into individual words or tokens, which is necessary before any filtering or normalization can occur, but it does not remove noise words.
For the exam, remember that filtering out frequent, uninformative words to improve model performance is called stop-word removal, and it can be customized with a user-defined list.