Multi-GPU Training with DataParallel

Modern models and batches outgrow a single GPU memory and compute. Using multiple GPUs lets you train larger models or process bigger batches in less wall-clock time. PyTorch offers several ways to do this; the simplest is DataParallel.

Data parallelism replicates the model on every GPU and splits each input batch across them. Every GPU computes on its shard, then gradients are combined so all replicas stay in sync.