Batch Norm แบบซิงก์และสถานะแบ่งส่วน

Batch norm computes stats from each GPU's local batch. Split across many GPUs, each per-device batch shrinks and those stats get noisy.

SyncBatchNorm fixes this by computing mean and variance across all GPUs together, as if it saw the full global batch.

You do not rewrite layers by hand. One helper converts every BatchNorm in your model to its synced version.

Order matters: call the conversion before you wrap the model in DDP, so the synced layers are the ones DDP manages.

Syncing stats means an extra all-reduce at every batch norm layer. Use it when small per-GPU batches hurt accuracy, not by default.

Plain DDP copies the full model, gradients, and optimizer state onto every GPU. For huge models that redundancy wastes memory fast.

Sharding splits those tensors across GPUs so each device stores only a slice. Together the GPUs still hold the whole model.

PyTorch's FullyShardedDataParallel shards parameters, gradients, and optimizer state. It lets you train models far larger than one GPU's memory.

FSDP gathers each layer's full weights only when it is needed for compute, then frees them again. That keeps peak memory low.

Sharding comes in levels, called ZeRO stages: shard optimizer state, then gradients, then parameters. More sharding saves more memory.

If your model fits per GPU, plain DDP is simplest. When it does not, reach for FSDP to shard state and keep going.

Decide what each technique is really for.

You met two consistency tools: SyncBatchNorm keeps stats global across GPUs, and FSDP shards state so giant models fit. Use each only when you need it.