During the implementation of a custom training loop in PyTorch, why is it necessary to call optimizer.zero_grad() at the start of each training iteration?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: PyTorch is designed to accumulate gradients by default. That means if you don't clear the slate before calling loss.backward(), PyTorch will take the new gradients and just add them to whatever was left over from the last batch. Not very efficient, and it'll completely wreck your training! By calling optimizer.zero_grad(), you're telling PyTorch to reset all those gradient buffers to zero. Think of it like wiping the whiteboard clean before you start solving the next math problem. If you don't, your new math is going to get jumbled up with your old work, and you'll end up with a total mess. Trust me on this one, forgetting to zero out your gradients is a classic mistake that catches people in production and on the exam all the time.
Full explanation below image
Full Explanation
In PyTorch, the gradients of the model's parameters (weights and biases) are calculated during the backward pass (when loss.backward() is called) and stored in each parameter's .grad attribute. Crucially, PyTorch does not automatically overwrite these gradient values; instead, it accumulates (adds) them. This accumulation is highly useful for certain architectures, such as recurrent neural networks (RNNs) or when implementing virtual batch sizes that exceed hardware memory limits.
However, in standard mini-batch gradient descent, gradients must be computed independently for each individual mini-batch. If optimizer.zero_grad() is omitted, the gradients calculated for the current batch will be added to the gradients from all previous batches. As a result, the parameter updates performed by the optimizer (via optimizer.step()) will use an accumulation of gradients across multiple steps, causing the model to update parameters in incorrect directions and ultimately preventing convergence.
Let's break down why the other options are incorrect: - Option A is incorrect because resetting the learning rate is not the purpose of zero_grad(). The learning rate is a hyperparameter and remains constant unless adjusted by a learning rate scheduler. - Option C describes resetting the model's parameters (weights and biases) to zero. Doing this would destroy all learned patterns and cause the model to fail to train, as neural networks must be initialized with non-zero (often randomized) weights to break symmetry. - Option D describes purging GPU memory. While memory management is important in deep learning, zero_grad() does not free GPU memory or delete tensors from the computational graph; it merely sets the values within existing gradient tensors to zero.