Launch Jobs with torchrun
Spawn and coordinate worker processes.
Launch Jobs with torchrun is a free Deep Learning Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Who Starts the Processes
DDP needs one process per GPU, but who spawns them? You do not launch them by hand. torchrun is the tool that does it for you.
Meet torchrun
torchrun is PyTorch's launcher. You give it your script and how many processes to start, and it handles the coordination.
torchrun --nproc_per_node=4 train.pyProcesses per Node
The nproc_per_node flag sets how many processes to spawn on this machine. Usually it equals the number of GPUs you have.
torchrun --nproc_per_node=8 train.pyEnvironment Variables Appear
torchrun injects key values as environment variables: RANK, LOCAL_RANK, and WORLD_SIZE. Your script reads them to know who it is.
import os
local_rank = int(os.environ["LOCAL_RANK"])No Manual Address Wiring
Older launchers made you pass ranks by hand. With torchrun, the rendezvous is automatic, so your script stays clean and portable.
Init Reads the Env
Because torchrun sets the env vars, your init_process_group call needs no arguments beyond the backend. It picks everything up automatically.
import torch.distributed as dist
dist.init_process_group(backend="nccl")Scale to Many Machines
To go multi-node, add nnodes and a node rank. Each machine runs the same command with its own node id.
torchrun --nnodes=2 --node_rank=0 --nproc_per_node=4 train.pyPoint to the Master
Across nodes, all processes must find a meeting point. The rdzv_endpoint gives the host and port they rendezvous on.
torchrun --rdzv_endpoint=host0:29500 train.pySurvive a Worker Crash
torchrun supports elastic training: set a min and max node count, and the job can recover if a worker drops out. 💪
torchrun --nnodes=1:4 --max_restarts=3 train.pyLog From One Rank
Every process prints, so logs get noisy. Gate prints behind a rank check so only rank 0 reports progress.
if int(os.environ["RANK"]) == 0:
print("epoch done")The Whole Recipe
The pattern is simple: write a normal DDP script, then launch it with torchrun. The launcher and DDP handle the rest together.
Quick Check
Recall what torchrun hands to your script.
Recap
You learned to launch jobs with torchrun: set processes per node, read the injected env vars, and scale to many nodes with elastic recovery.
Frequently asked questions
Is the “Launch Jobs with torchrun” lesson free?
Yes — the full text of “Launch Jobs with torchrun” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Launch Jobs with torchrun”?
Spawn and coordinate worker processes. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Launch Jobs with torchrun” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Data vs Model Parallelism
- DistributedDataParallel Basics
- Sync Batch Norm & Sharded State
- Launch Jobs with torchrun