When evaluating static word embedding techniques for natural language processing, what is the primary difference in how Word2Vec and GloVe (Global Vectors) generate their word representations?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Okay, let's dive in. Word embeddings are how we turn words into math so computers can understand them, but Word2Vec and GloVe go about this in two completely different ways. Think of Word2Vec like a local detective. It slides a small window across a sentence and tries to predict a word based on its immediate neighbors (or vice-versa, using CBOW or Skip-gram). It's a predictive model. GloVe, on the other hand, is like a global census taker. It looks at the entire corpus of documents all at once, builds a massive grid showing how often words appear next to each other (a co-occurrence matrix), and then performs matrix factorization to extract the embeddings. One is local and predictive (Word2Vec), while the other is global and count-based (GloVe). Hopefully you answered C, because understanding this distinction is a favorite topic on certification exams!
Full explanation below image
Full Explanation
Both Word2Vec and GloVe are static word embedding techniques that represent words as dense vectors in a continuous vector space, capturing semantic and syntactic relationships. However, their optimization objectives and underlying training methodologies differ significantly:
1. Word2Vec (Predictive Model): Developed by Google, Word2Vec utilizes a local context window approach. It processes text sequentially using one of two architectures: - Continuous Bag of Words (CBOW): Predicts a target word given its surrounding context words. - Skip-gram: Predicts surrounding context words given a single target word. The model uses a shallow, two-layer neural network trained via optimization techniques like negative sampling or hierarchical softmax. It adjusts vector weights based on local prediction errors.
2. GloVe (Global Vectors for Word Representation - Count-Based Model): Developed by Stanford, GloVe combines the advantages of global matrix factorization (like Latent Semantic Analysis) and local context window methods. It constructs a global word-word co-occurrence matrix $X$, where entry $X_{ij}$ represents the number of times word $i$ appears in the context of word $j$ across the entire corpus. The model then minimizes a weighted least-squares objective function to factorize this matrix, directly utilizing the global statistical ratios of word co-occurrences to construct the vector space.
Let's look at why the other options are distractors: - Option A is incorrect because both Word2Vec and GloVe generate unsupervised word embeddings that can be used for any downstream task, including both classification and regression. - Option B is incorrect because it reverses the definitions: GloVe is the count-based model using global co-occurrences, while Word2Vec is the predictive local window model. - Option D is incorrect because both Word2Vec and GloVe generate static embeddings, meaning a word has the same vector representation regardless of its context in a new sentence (unlike modern contextual embeddings like BERT or GPT).