When preprocessing text for a Natural Language Processing application, what is the primary objective of performing lemmatization on the corpus?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out. In NLP, we want our models to understand that 'ran', 'running', and 'runs' are all variations of the same action: 'run'. If we don't clean this up, our model has to learn three separate words instead of one. There are two ways to fix this: stemming and lemmatization. Stemming is like a brute-force lumberjack that just chops the ends off words, which often leaves you with gibberish like 'runn'. Lemmatization, however, is smart. It uses a dictionary and checks the context of the sentence to convert 'geese' to 'goose' and 'better' to 'good'. It always gives you a real, valid dictionary word. That's why Option C is the winner here.
Full explanation below image
Full Explanation
Lemmatization is a text normalization technique in Natural Language Processing (NLP) that reduces words to their lemma, which is their canonical, dictionary, or base form. Unlike stemming, which relies on simple heuristic rules to chop off the ends of words, lemmatization uses vocabulary databases and morphological analysis to determine the correct base form.
For example, when lemmatizing the verb 'saw' in the sentence 'He saw a movie', the lemmatizer uses Part-of-Speech (POS) tagging to recognize it as a verb and resolves it to 'see'. If the sentence was 'He bought a saw', it would identify it as a noun and leave it as 'saw'. This contextual awareness ensures that the resulting lemma is always a valid word in the target language. This is crucial for search engines, topic modeling, and text classification where preserving grammatical accuracy and semantic meaning is important.
Let's look at why the other options are incorrect: - Option A describes term frequency counting, which is a statistical step in feature extraction (like bag-of-words or TF-IDF), not a word reduction technique. - Option B describes stemming (e.g., using a Porter Stemmer), which often trims words to non-real root forms (e.g., 'studying' -> 'studi'). - Option D describes stop-word removal, which filters out high-frequency words that carry little semantic weight, such as 'is', 'the', and 'at'.
Remember for the exam: lemmatization uses morphological and contextual rules to reduce words to their actual dictionary base form (lemma).