Multi-GPU with NCCL
Collective communication for scaling.
Multi-GPU with NCCL is a free CUDA Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the CUDA Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Hand-Rolled Gets Hard
Wiring peer copies by hand across many GPUs becomes messy fast. For real scaling you want a library built for collective communication.
Meet NCCL
NVIDIA's answer is NCCL, the collective communications library. It moves data among GPUs using the fastest links it can find, automatically.
What a Collective Is
A collective is one operation that every GPU joins together, like summing a value held on each card. All devices cooperate in a single call.
The Star Collective: AllReduce
The workhorse is AllReduce. It combines a buffer from every GPU, say by adding, and hands the same result back to all of them.
ncclAllReduce(send, recv, n, ncclFloat, ncclSum, comm, stream);More Collectives
NCCL also offers Broadcast to share one GPU's data with all, plus AllGather and ReduceScatter for other common exchange patterns.
The Communicator
The GPUs that talk together form a group tracked by a communicator. Every collective call passes this handle so NCCL knows the participants.
Ranks Identify GPUs
Inside a communicator each GPU gets a number called its rank, from 0 up. Ranks let you say who sends, who receives, and who is the root.
Building the Communicators
For GPUs in one process, ncclCommInitAll sets up every communicator at once from the list of device ids you provide.
ncclCommInitAll(comms, nGpus, devs);Stream Aware
Every NCCL call takes a stream. Operations queue there and run alongside your kernels, so communication overlaps computation.
Grouping Calls
To issue collectives on many GPUs without deadlock, wrap them between ncclGroupStart and ncclGroupEnd so NCCL launches them together.
ncclGroupStart();
// per-GPU calls
ncclGroupEnd();Why Training Loves It
Deep learning syncs gradients across GPUs every step. AllReduce is exactly that, which is why NCCL powers nearly all multi-GPU training.
Quick Check
Recall the NCCL collective that sums a buffer across GPUs and returns the result to all of them.
Recap
NCCL runs collectives like AllReduce over fast links, organized by a communicator of ranked GPUs. It is the backbone of multi-GPU training. Course complete. ✨
Frequently asked questions
Is the “Multi-GPU with NCCL” lesson free?
Yes — the full text of “Multi-GPU with NCCL” is free to read here on the web, and the CUDA Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the CUDA Academy course, upgrade to CoddyKit PRO.
What will I learn in “Multi-GPU with NCCL”?
Collective communication for scaling. You practise CUDA Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start CUDA Academy?
No prior experience is required. CUDA Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Multi-GPU with NCCL” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this CUDA Academy lesson?
Yes. Every CUDA Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.