What major capability made the Transformer architecture the dominant foundation for modern natural language processing (NLP) models compared to traditional Recurrent Neural Networks (RNNs)?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out: back in the day, RNNs and LSTMs had to process text one word at a time, like reading a book through a tiny straw. By the time they reached the end of a long sentence, they completely forgot what happened at the start. Transformers changed the game with the self-attention mechanism. They look at the entire sentence all at once, linking words together no matter how far apart they are. It's fast, it's smart, and it's the engine behind almost every modern LLM you use today.
Full explanation below image
Full Explanation
The Transformer architecture, introduced by Vaswani et al. in 2017, replaced recurrent neural networks (RNNs, LSTMs, GRUs) as the state-of-the-art framework for sequential data. Its primary breakthrough is the self-attention mechanism, which allows the model to process all tokens in a sequence in parallel rather than sequentially. This parallelization eliminates the bottleneck of sequential step-by-step processing and enables the network to capture dependencies and relationships between words regardless of their distance from one another in the text.
To review the distractors: - Option A refers to recurrent layers (like those in RNNs), which process data step-by-step. This serial dependency makes training slow and limits their ability to capture long-range context due to vanishing gradients. - Option B is incorrect because while Transformers are highly parallelizable and run efficiently on GPUs, their computational complexity actually scales quadratically with sequence length, making them computationally intensive. - Option D is incorrect because Transformers are notoriously parameter-heavy, requiring massive amounts of memory and compute to train and deploy.
By utilizing multi-head self-attention and positional encodings, Transformers can dynamically weigh the relevance of every token in a prompt relative to every other token, capturing deep semantic context across very long sequences.