สะสมเกรเดียนต์สำหรับแบตช์ขนาดใหญ่

Large batches often train more smoothly, but they also need lots of GPU memory. A small card simply cannot hold a giant batch at once.

Gradient accumulation splits one big batch into small chunks. You add up their gradients and update once, as if the whole batch ran together.

PyTorch adds each backward pass into .grad rather than replacing it. This default behavior is exactly what accumulation relies on.

Choose how many mini-batches make one update. With accum_steps of four, four small batches behave like one batch four times larger.

The key change is timing: zero_grad only at the start of an accumulation cycle, not after every single mini-batch.

Divide each mini-batch loss by accum_steps before backward. This keeps the average gradient identical to running the full batch at once.

After enough mini-batches pile up, call optimizer.step. The accumulated gradients now reflect the whole large batch.

Put it together: scale the loss, backward every step, but only step and zero once per cycle. The loop stays simple.

You only ever hold one mini-batch in memory at a time. That is why a modest GPU can mimic a batch many times its real capacity.

Accumulation trades time for memory: more forward and backward passes per update mean each effective batch takes a little longer to finish.

Batch norm still sees only the small mini-batch, so its statistics are noisier than a true large batch would produce. Keep that in mind.

You accumulate over 4 mini-batches. When should you call optimizer.step()?

Split a big batch into chunks, divide the loss by accum_steps, backward every chunk, and step only once per cycle to fake a large batch on small memory. 🧮