Mixed Precision: FP16, BF16, TF32

Tensor Cores get their speed from mixed precision: feeding smaller number formats in so the hardware can push far more math per cycle. ⚡

Standard FP32 uses 32 bits per value. It is accurate but heavy, so moving and multiplying many FP32 numbers eats bandwidth and time.

FP16 uses just 16 bits: a smaller exponent and fewer mantissa bits. Half the size means more values move and multiply at once.

FP16 saves space but its narrow exponent gives a tiny dynamic range. Very large or very small values can overflow or vanish to zero.

BF16 is also 16 bits but keeps FP32's full exponent, trading mantissa bits instead. It matches FP32's range with less precision.

Because BF16 shares FP32's wide range, gradients rarely overflow. That makes BF16 a favorite for training large neural networks safely.

TF32 is a 19-bit Tensor Core format: FP32's exponent with a trimmed mantissa. It speeds up math while looking like FP32 to your code.

No matter the input format, Tensor Cores usually accumulate partial sums in FP32. The speed is in the inputs; the safety is in the running total.

The big idea: FP16 and BF16 cost the same 16 bits but split them differently. One favors precision, the other favors range.

Use BF16 or TF32 when range matters, FP16 when you can manage scaling. The right format depends on your data, not just on speed.

In CUDA C++ the FP16 type has a short name. This line declares one half value ready for Tensor Core math.

Which format keeps FP32's full exponent range in only 16 bits?

You saw how mixed precision trades bits for throughput: FP16 favors precision, BF16 favors range, and TF32 speeds FP32-style math. Wide accumulation keeps results safe. ✨