When preprocessing textual data for a Natural Language Processing (NLP) pipeline, what is the primary goal of applying lemmatization to the corpus?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out: in English, we use different versions of the same word depending on the context. You might say "runs," "running," or "ran," but deep down, they all point to the same action: "run." If you're building an NLP system, you want the computer to know they're the same thing. You've got two ways to do this: stemming or lemmatization. Stemming is like using a machete—it just chops off word endings, which often leaves you with gibberish like "runn." But lemmatization is smart. It uses vocabulary and grammatical context to look up the actual dictionary form—the "lemma." So "better" becomes "good," and "saw" (as a verb) becomes "see." That's why Option D is correct!
Full explanation below image
Full Explanation
Lemmatization is an essential text normalization technique in NLP. Its objective is to reduce a word to its lemma, which is its canonical, dictionary-defined base form. Unlike stemming—which applies simple, rule-based heuristics to slice off suffixes (e.g., transforming "flies" and "flying" to "fli" or "fly" regardless of context)—lemmatization uses vocabulary databases (such as WordNet) and part-of-speech (POS) tagging to perform a full morphological analysis. For example, the word "meeting" could be lemmatized to "meet" if tagged as a verb, or remain "meeting" if tagged as a noun. Similarly, the word "was" is lemmatized to "be" because it recognizes the underlying lemma of the verb.
Let's review the incorrect options: - Option A defines stop word removal, which eliminates highly frequent words that lack unique semantic value for tasks like classification. - Option B describes stemming, which is computationally faster but linguistically crude and often yields non-words. - Option C refers to basic text cleaning processes such as lowercasing or punctuation stripping.
For the exam, make sure you distinguish between stemming (crude slicing that may create non-words) and lemmatization (linguistically accurate reduction to a dictionary form).