Das Problem des verschwindenden Gradienten

Plain RNNs look powerful, but they struggle to remember long-range context. The cause is the vanishing gradient problem.

Networks learn by sending error signals, called gradients, backward through time to nudge each weight in the right direction.

For an RNN this backward pass unrolls across every step, so a long sentence means a very long chain of multiplications.

When you multiply many values below one together, the result rushes toward zero. The gradient fades long before it reaches early words.

Because the signal vanishes, the network barely updates weights tied to the first tokens, so distant context is effectively lost.

Imagine multiplying 0.5 by itself twenty times. The value almost disappears, and that is what happens to deep-step gradients.

Gradients can also explode when values exceed one, growing huge and destabilizing training with wild weight swings.

Exploding gradients have an easy patch: gradient clipping caps their size so a single step cannot blow up your model.

Vanishing gradients resist simple fixes, so plain RNNs keep forgetting. We need a smarter cell to hold memory across many steps.

Special architectures with gates, like LSTM and GRU, control what to keep and forget, preserving long-range signals.

Understanding this limit explains why modern sequence models exist at all: they were designed to keep gradients alive over distance. 💡

Why do plain RNNs forget early words in long sequences?

In long sequences plain RNN gradients vanish, erasing distant context. Gated cells like LSTM and GRU are the fix we explore next. 🎯