When training a deep neural network, you must configure a key hyperparameter called "batch size" before initiating the training loop. What is the specific purpose of this setting?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Think of training a neural network like studying for a big exam. You wouldn't try to memorize the entire textbook in one massive breath—that's a quick way to crash your brain (and your GPU's memory). Instead, you read a few pages, check your understanding, and adjust what you know. In deep learning, that group of pages is your batch size. It's the exact number of training examples the network looks at before it does a backpropagation step and updates its weights. Get it too large, and your hardware runs out of memory. Get it too small, and your training wanders around and takes forever. Option B is what you want here.
Full explanation below image
Full Explanation
In the training of deep neural networks, optimization algorithms such as Mini-batch Gradient Descent calculate the gradient of the loss function to update the model's internal weights. The batch size is a critical hyperparameter that determines how many training examples are processed during a single optimization iteration.
When batch size is set to the entire dataset (Batch Gradient Descent), the calculated gradients are highly stable and accurate representation of the entire dataset, but the calculation requires immense memory and processing power, often resulting in out-of-memory errors on modern accelerators. Conversely, setting the batch size to 1 (Stochastic Gradient Descent) updates the weights after every single sample. While this is memory-efficient, the updates are highly noisy and unstable, which can prevent the model from smoothly converging. Modern deep learning practitioners therefore choose a mini-batch size (typically between 32 and 512) to strike an optimal balance: utilizing the parallel processing capabilities of GPUs while maintaining stable and efficient gradient updates.
Let's look at the other options to understand why they are incorrect: - Option A refers to the depth of the network, which is determined by the number of hidden layers, not the batch size. - Option C describes the concept of an epoch, which is one complete pass through the entire training dataset. A single epoch is made up of multiple iterations (steps) determined by the batch size. - Option D refers to the output layer size, which is dictated by the number of target classes or values in your specific classification or regression task.
For the exam, make sure you associate 'batch size' with the number of training examples processed in a single weight-update iteration.