torchrun으로 작업 실행하기

DDP needs one process per GPU, but who spawns them? You do not launch them by hand. torchrun is the tool that does it for you.

torchrun is PyTorch's launcher. You give it your script and how many processes to start, and it handles the coordination.

The nproc_per_node flag sets how many processes to spawn on this machine. Usually it equals the number of GPUs you have.

torchrun injects key values as environment variables: RANK, LOCAL_RANK, and WORLD_SIZE. Your script reads them to know who it is.

Older launchers made you pass ranks by hand. With torchrun, the rendezvous is automatic, so your script stays clean and portable.

Because torchrun sets the env vars, your init_process_group call needs no arguments beyond the backend. It picks everything up automatically.

To go multi-node, add nnodes and a node rank. Each machine runs the same command with its own node id.

Across nodes, all processes must find a meeting point. The rdzv_endpoint gives the host and port they rendezvous on.

torchrun supports elastic training: set a min and max node count, and the job can recover if a worker drops out. 💪

Every process prints, so logs get noisy. Gate prints behind a rank check so only rank 0 reports progress.

The pattern is simple: write a normal DDP script, then launch it with torchrun. The launcher and DDP handle the rest together.

Recall what torchrun hands to your script.

You learned to launch jobs with torchrun: set processes per node, read the injected env vars, and scale to many nodes with elastic recovery.