You are assisting in troubleshooting a distributed training job for a large neural network across an 8-GPU node. You notice that while GPU 0 is pinned at 100% utilization, the other seven GPUs (GPUs 1-7) are frequently sitting idle or showing very low utilization. What is the most likely cause of this poor scaling performance?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Pay close attention here, because this one bites people in production all the time. When you run distributed training, you're trying to share the load. Imagine you have a team of eight workers, but you only give the instructions to one guy, and the other seven just stand around waiting for him to finish. That's exactly what happens when you don't shard your dataset properly! In distributed training, you need to divide your data into distinct, non-overlapping shards and feed one shard to each GPU. If you fail to do this, or if your data loader is bottlenecked sending all the data to GPU 0 first, your training speed hits a brick wall. The other GPUs will sit there twiddling their thumbs. Always make sure your distributed sampler is properly splitting the data!
Full explanation below image
Full Explanation
Distributed training on multiple GPUs relies on parallel processing techniques, most commonly Data Parallelism (DP) or Distributed Data Parallelism (DDP). In these paradigms, the model is replicated across all GPUs, and the training dataset must be partitioned so that each GPU processes a unique subset (shard) of the data during each step.
If the dataset is not sharded properly, the pipeline suffers from severe workload imbalance. Typically, this manifests as a single GPU (often GPU 0, which acts as the coordinator) performing the bulk of the data loading, preprocessing, or gradient aggregation, while the remaining GPUs wait idle. A proper implementation uses a distributed data sampler (like PyTorch's DistributedSampler) to ensure that each GPU processes an equal-sized, unique chunk of the batch simultaneously. The gradients are then synchronized across all GPUs using collective communication operations (such as All-Reduce) via NCCL. Without proper sharding, you lose the linear speedup expected from multi-GPU scaling.
Let's evaluate the incorrect choices: Option A is incorrect because if the global batch size were set too high, it would typically trigger an Out-of-Memory (OOM) error on the GPUs, crashing the training script entirely rather than causing low utilization on the other GPUs. Option B is incorrect because learning rate schedulers are mathematical functions that update the learning rate hyperparameter. While they must be synchronized to ensure mathematical correctness, desynchronization does not physically prevent GPUs from executing their compute kernels or cause them to sit idle. Option D is incorrect because a model being 'too simple' would result in high data transfer overhead relative to computation, but it would affect all GPUs relatively equally (high communication overhead causing overall low utilization across all GPUs, not one GPU at 100% and others idle).