梯度消失与梯度爆炸

Through many layers, backprop multiplies many local derivatives together. The size of that long product decides whether learning works or breaks.

If each link's derivative is below one, the product shrinks fast. After many layers the gradient becomes tiny, almost zero by the time it reaches early layers.

When gradients shrink to near zero, early layers barely update and stop learning. This is the vanishing gradient problem that long stalled deep nets.

If each derivative is above one, the product blows up instead. Gradients grow huge as they travel back, and the weights lurch wildly.

Runaway gradients are the exploding gradient problem. Loss often jumps to NaN as updates overshoot far past any useful value. 💥

Sigmoid and tanh squash inputs, so their derivatives stay well below one. Stacking them multiplied small numbers and made vanishing gradients common.

ReLU has a derivative of exactly one for positive inputs, so it does not shrink the gradient. Switching to ReLU was a key early fix.

Smart schemes like Xavier and He initialization scale starting weights so the gradient product stays near one across many layers.

For explosions, gradient clipping caps the gradient's size before the update, keeping a single huge step from wrecking the model.

ResNet adds skip connections that let gradients flow straight back, sidestepping the long chain of multiplications that causes vanishing.

Keeping the gradient product near one is what lets very deep nets train at all. Every fix here exists to protect that balance.

Let's check the failure modes.

Long products of derivatives can vanish or explode. ReLU, careful init, clipping, and skip connections keep gradients healthy through deep nets. 💥