ความแม่นยำผสมด้วย autocast และ GradScaler

By default PyTorch does math in 32-bit floats. Mixed precision runs many operations in 16-bit instead, which is faster and uses far less memory.

Modern GPUs have special tensor cores tuned for 16-bit math. Feeding them half-precision data can speed up training two or three times with little accuracy loss.

Half precision has a tiny range, so very small gradient values can round down to zero. That silent underflow stalls learning if you do nothing about it.

Wrap your forward pass in autocast and PyTorch picks a safe precision per operation automatically. You never cast tensors by hand.

Put only the forward pass and loss inside autocast. The backward call stays outside the block, where PyTorch handles precision for you.

A GradScaler fixes underflow by multiplying the loss by a large factor before backward, so small gradients stay big enough to survive in 16-bit.

Instead of loss.backward(), call scaler.scale(loss).backward(). The scaler inflates the loss first so the resulting gradients do not vanish.

Run the optimizer with scaler.step, which quietly unscales the gradients back to normal size before applying the update.

Finish each step with scaler.update(). It grows the scale when things are stable and shrinks it if it ever spots an overflow.

Together these calls form one clean AMP iteration: zero grads, autocast forward, scaled backward, scaler step, then update.

Mixed precision shines on a recent NVIDIA GPU with large batches. On a plain CPU it gives little benefit, so save it for real training runs.

You enabled autocast but tiny gradients keep vanishing. What tool fixes that?

Wrap the forward in autocast for fast 16-bit math, then use a GradScaler to scale, step, and update so tiny gradients survive. ⚡