0Pricing
Learn AI with Python · Lesson

Multi-GPU Training with DataParallel

nn.DataParallel, GPU memory balancing, bandwidth bottlenecks, when DDP is better.

Multi-GPU Training with DataParallel is a free Learn AI with Python lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Multiple GPUs

Modern models and batches outgrow a single GPU memory and compute. Using multiple GPUs lets you train larger models or process bigger batches in less wall-clock time. PyTorch offers several ways to do this; the simplest is DataParallel.

Data Parallelism

Data parallelism replicates the model on every GPU and splits each input batch across them. Every GPU computes on its shard, then gradients are combined so all replicas stay in sync.

nn.DataParallel

nn.DataParallel wraps a model in one line. You pass the GPU ids to use, and PyTorch handles scattering inputs and gathering outputs automatically.

import torch
import torch.nn as nn

model = MyModel().cuda()
model = nn.DataParallel(model, device_ids=[0, 1])

Automatic Batch Split

During the forward pass, DataParallel splits the batch along dimension 0 across the listed GPUs. A batch of 64 on two GPUs becomes two sub-batches of 32. Each GPU runs the same model on its sub-batch in parallel.

# batch of 64 with device_ids=[0, 1]
# GPU 0 processes samples 0..31
# GPU 1 processes samples 32..63

Gradient Averaging

In the backward pass, the per-GPU gradients are gathered to the primary device and averaged (effectively summed and scaled) so the model update reflects the whole batch. The replicas are then re-synced for the next step.

A Normal Training Loop

The best part: your training loop barely changes. You move data to the primary GPU and call the wrapped model as usual; DataParallel does the distribution under the hood.

for x, y in loader:
    x, y = x.cuda(), y.cuda()
    optimizer.zero_grad()
    out = model(x)            # auto-split across GPUs
    loss = criterion(out, y)
    loss.backward()           # gradients averaged
    optimizer.step()

The GPU 0 Imbalance

DataParallel has a well-known flaw: the primary GPU (usually GPU 0) gathers all outputs and computes the loss, so it carries extra memory and compute. With many GPUs, GPU 0 becomes a bottleneck and may run out of memory before the others.

Single Process, Many Threads

DataParallel runs in a single Python process and uses threads to drive the GPUs. Python global interpreter lock and the scatter/gather overhead limit scaling efficiency, especially beyond 2 to 4 GPUs.

Why DDP is Preferred

For these reasons, DistributedDataParallel (DDP) is recommended for serious multi-GPU work. DDP runs one process per GPU, syncs gradients with efficient all-reduce, and avoids the GPU 0 bottleneck, giving near-linear scaling.

When DataParallel is Still Fine

DataParallel is acceptable for quick experiments on a single machine with 2 GPUs, where its one-line simplicity outweighs the inefficiency. For multi-node training or many GPUs, reach for DDP instead.

Common Pitfall: Saving

A wrapped model state dict has a module. prefix. To save a clean checkpoint, save model.module.state_dict() so it loads correctly into an unwrapped model later.

torch.save(model.module.state_dict(), "model.pt")

Quick Check

Test your DataParallel knowledge.

Recap

You learned multi-GPU training with DataParallel:

  • nn.DataParallel(model, device_ids=[0, 1]) replicates the model and splits batches
  • Gradients are averaged across GPUs each step
  • The GPU 0 imbalance and single-process design limit scaling
  • Prefer DDP for many GPUs or multi-node training

Frequently asked questions

Is the “Multi-GPU Training with DataParallel” lesson free?

Yes — the full text of “Multi-GPU Training with DataParallel” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.

What will I learn in “Multi-GPU Training with DataParallel”?

nn.DataParallel, GPU memory balancing, bandwidth bottlenecks, when DDP is better. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Learn AI with Python?

No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Multi-GPU Training with DataParallel” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Learn AI with Python lesson?

Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Multi-GPU Training with DataParallel
  2. DistributedDataParallel (DDP)
  3. Mixed Precision Training with AMP
  4. Efficient Training with Hugging Face Accelerate
← Back to Learn AI with Python