Deep Learning Academy · 课时

DistributedDataParallel 基础

标准的多 GPU 训练路径

DistributedDataParallel 基础 是 CoddyKit 上的免费 Deep Learning Academy 课时。这是第 2 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Deep Learning Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Deep Learning Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Meet DDP

DistributedDataParallel, or DDP, is PyTorch's go-to tool for multi-GPU training. It runs one process per GPU and keeps every model copy in sync.

One Process per GPU

Unlike the older DataParallel, DDP spawns a separate process for each GPU. This avoids Python's GIL and scales far more cleanly.

Rank and World Size

Each process gets a rank (its id) and shares the world size (total processes). Rank 0 is usually the one that logs and saves.

import torch.distributed as dist
rank = dist.get_rank()
world = dist.get_world_size()

Init the Process Group

Before any communication you call init_process_group. The nccl backend is the fast choice for GPUs.

import torch.distributed as dist
dist.init_process_group(backend="nccl")

Pin Each Process to a GPU

Use the local rank to set the device so every process owns exactly one GPU. This keeps work from piling onto a single card.

import torch
torch.cuda.set_device(local_rank)
model = model.to(local_rank)

Wrap Your Model

The magic is one line: wrap your model in DDP. From then on, gradients sync automatically during the backward pass.

from torch.nn.parallel import DistributedDataParallel as DDP
model = DDP(model, device_ids=[local_rank])

Gradients Sync Themselves

During backward(), DDP performs an all-reduce to average gradients across GPUs. You write normal training code and it just stays in sync. ✨

Use a DistributedSampler

So each GPU sees different data, give your DataLoader a DistributedSampler. It hands every process a non-overlapping slice of the dataset.

from torch.utils.data.distributed import DistributedSampler
sampler = DistributedSampler(dataset)

Reshuffle Every Epoch

Call sampler.set_epoch(epoch) at the top of each epoch. Without it, every GPU reshuffles the same way and you lose real shuffling.

for epoch in range(epochs):
    sampler.set_epoch(epoch)
    train_one_epoch()

Save Only on Rank 0

All copies are identical, so checkpoint from rank 0 only. Saving from every process just writes the same file many times.

if rank == 0:
    torch.save(model.module.state_dict(), "ckpt.pt")

Clean Up at the End

When training finishes, call destroy_process_group to release the group cleanly and avoid hanging processes.

import torch.distributed as dist
dist.destroy_process_group()

Quick Check

Think about how DDP keeps copies in sync.

Recap

You set up DDP: init the process group, wrap the model, feed it a DistributedSampler, and save from rank 0. Gradients sync for free.

常见问题解答

「DistributedDataParallel 基础」课时是免费的吗？

是的 — 「DistributedDataParallel 基础」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Deep Learning Academy 课程的其余内容，请升级到 CoddyKit PRO。 Deep Learning Academy 课程共包含 4 节课。

「DistributedDataParallel 基础」这节课中我会学到什么？

标准的多 GPU 训练路径你通过在浏览器中直接运行的动手代码来练习 Deep Learning Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Deep Learning Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Deep Learning Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 2 节课，共 4 节。

「DistributedDataParallel 基础」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Deep Learning Academy 课中编写并运行代码吗？

能。每节 Deep Learning Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Deep Learning Academy