DistributedDataParallel (DDP)
Process groups, dist.init_process_group, DistributedSampler, gradient synchronization.
DistributedDataParallel (DDP) is a free Learn AI with Python lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What is DDP
DistributedDataParallel (DDP) is PyTorch high-performance approach to data-parallel training. It launches one process per GPU, each holding a full model replica, and synchronizes gradients efficiently. DDP scales near-linearly across GPUs and machines.
The Process Group
DDP coordinates processes through a process group. Each process gets a unique rank and knows the total world size. They communicate over a backend; on NVIDIA GPUs that backend is nccl.
init_process_group
Every process starts by joining the group. dist.init_process_group with backend="nccl" sets up GPU-to-GPU communication.
import torch.distributed as dist
import os
dist.init_process_group(backend="nccl")
rank = dist.get_rank()
world_size = dist.get_world_size()Pinning the Device
Each process should own one GPU. Use the local rank to set the device so process 0 uses cuda:0, process 1 uses cuda:1, and so on.
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)
device = torch.device("cuda", local_rank)Wrapping the Model
Move the model to its GPU, then wrap it in DDP with device_ids=[local_rank]. DDP registers hooks that will synchronize gradients during backprop.
from torch.nn.parallel import DistributedDataParallel as DDP
model = MyModel().to(device)
model = DDP(model, device_ids=[local_rank])The DistributedSampler
Each process must see a different slice of the data, with no overlap. The DistributedSampler partitions the dataset across ranks so the union covers the whole dataset exactly once per epoch.
from torch.utils.data import DataLoader
from torch.utils.data.distributed import DistributedSampler
sampler = DistributedSampler(dataset)
loader = DataLoader(dataset, batch_size=32, sampler=sampler)Shuffling Per Epoch
Call sampler.set_epoch(epoch) at the start of each epoch. This reseeds the shuffle consistently across processes so every rank shuffles the same way and partitions stay disjoint.
for epoch in range(epochs):
sampler.set_epoch(epoch)
for x, y in loader:
...Gradient All-Reduce
During loss.backward(), DDP performs an all-reduce: every process sends its gradients and receives the averaged result, so all replicas apply identical updates. All-reduce overlaps with backprop, hiding communication cost.
No GPU 0 Bottleneck
Unlike DataParallel, DDP has no central GPU gathering outputs. Each process computes its own loss and gradients; only gradients are exchanged via all-reduce. This symmetry is why DDP scales so much better.
Launching with torchrun
torchrun launches the per-GPU processes and sets the rank environment variables. --nproc_per_node=4 starts 4 processes (one per GPU) on this node.
torchrun --nproc_per_node=4 train.pySaving Only on Rank 0
All ranks hold identical weights, so only one should write the checkpoint to avoid clobbering. Guard the save with a rank check.
if rank == 0:
torch.save(model.module.state_dict(), "model.pt")
dist.barrier()Quick Check
Test your DDP knowledge.
Recap
You learned DistributedDataParallel:
dist.init_process_group(backend="nccl")joins the process group- DistributedSampler gives each rank a disjoint data slice
DDP(model, device_ids=[rank])syncs gradients via all-reduce- Launch with
torchrun --nproc_per_node=4 - Save only on rank 0
Frequently asked questions
Is the “DistributedDataParallel (DDP)” lesson free?
Yes — the full text of “DistributedDataParallel (DDP)” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “DistributedDataParallel (DDP)”?
Process groups, dist.init_process_group, DistributedSampler, gradient synchronization. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “DistributedDataParallel (DDP)” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Multi-GPU Training with DataParallel
- DistributedDataParallel (DDP)
- Mixed Precision Training with AMP
- Efficient Training with Hugging Face Accelerate