The Vanishing Gradient Problem in Deep Time Steps

To learn from long-range dependencies in a sequence, gradients from the loss at the final timestep must travel backward through every timestep to update the parameters that processed the early inputs. For a sequence of length T, this means multiplying the same weight matrix W_hh by itself T times during BPTT. This repeated multiplication is the root cause of both vanishing gradients (exponential decay) and exploding gradients (exponential growth).

During BPTT, the gradient of the loss with respect to the hidden state at timestep t involves the product of the Jacobian matrices of h with respect to h at each step from t to T. The Jacobian at each step involves diag(f'(h_t)) * W_hh where f' is the derivative of the activation. For tanh, f' is bounded by 1, and typical random weights have spectral radius less than 1 — so this product of T matrices drives gradients to zero exponentially fast with T.

You can directly observe vanishing gradients by logging the gradient norm at each timestep during BPTT. Register backward hooks on each RNN step's hidden state to capture gradient magnitudes. In a 50-step vanilla RNN, the gradient at timestep 1 will typically be 1e-6 or smaller — effectively zero — meaning the first several tokens of a sequence have almost no influence on the model parameters. The model cannot learn that the subject at the start of a long sentence determines the verb at the end.

Exploding gradients occur when the spectral radius of W_hh exceeds 1 — gradient norms grow exponentially with sequence length. The symptom is NaN loss values or parameters updating to infinity. Unlike vanishing gradients (which cause silent learning failure), exploding gradients crash training visibly. The standard fix is gradient clipping: rescale the gradient vector to have maximum L2 norm of 1.0 before the optimizer step. This prevents catastrophic updates without removing the gradient signal.

A practical debugging technique is to log gradient norms for all parameters after each backward pass and plot them over training. For vanilla RNNs, the recurrent weight matrix W_hh typically shows much smaller gradients than the input weight W_xh, confirming that long-range information is not reaching the earlier parameters. This visualisation often reveals that only the last few timesteps contribute meaningfully to learning, motivating the switch to gated architectures.

The tanh activation function is bounded between -1 and 1 with derivative 1 - tanh^2(x). When the input is large (saturated), the derivative approaches 0 — cutting the gradient to nearly zero at that step. Multiplying many near-zero derivatives through BPTT compounds the vanishing problem. ReLU has derivative 1 for positive inputs (no saturation), which helps gradient flow in feedforward networks, but in RNNs the repeated multiplication of W_hh still dominates and can cause explosions with ReLU.

Truncated BPTT limits gradient propagation to a fixed window of K timesteps instead of the full sequence length. Gradients are propagated back K steps, then the hidden state is detached from the computation graph (becoming a constant). This prevents memory and gradient explosion for very long sequences (audio, text corpora) at the cost of not learning dependencies spanning more than K steps. K=20-50 is typical for language modelling with vanilla RNNs.

Consider the sentence: 'The trophy that the man won was big.' The verb 'was' must agree with 'trophy', not with 'man'. This requires carrying information about 'trophy' across 5 words to where 'was' appears. A vanilla RNN trained via BPTT essentially cannot do this reliably for gaps longer than 5-10 tokens. This is the core limitation that motivated the development of LSTM (1997) and later Transformers (2017), both of which have explicit mechanisms for maintaining long-range information.

Several initialisation tricks improve vanilla RNN training on moderate-length sequences. Initialising W_hh as an orthogonal matrix (spectral radius exactly 1) prevents early vanishing/explosion. Adding a skip connection from input directly to output bypasses several matrix multiplications. Identity matrix initialisation for W_hh with ReLU activation (IRNN) was shown to match LSTM on some tasks, proving initialisation alone can partially address the vanishing gradient problem.

The vanishing gradient problem in RNNs was documented by Hochreiter in 1991. His solution, the Long Short-Term Memory (LSTM) network, introduced in 1997, replaces the single hidden state with a cell state that is protected by gates. The cell state flows through time with only additive modifications (not multiplicative), creating a gradient highway that allows gradients to flow backward indefinitely without vanishing. This single architectural innovation unlocked practical training of sequences with 100+ timestep dependencies.

The difference in training stability between vanilla RNNs and LSTMs becomes dramatic for sequences longer than 20-30 timesteps. On the classic copy task (reproduce the input sequence after a long delay), vanilla RNNs fail completely for delays above 10 steps, while LSTMs succeed for delays of 100+ steps. This practical benchmark concretely demonstrates that the vanishing gradient problem fundamentally limits vanilla RNNs, and that LSTM's architectural solution is necessary for real sequence modelling.

Test your understanding of Machine Learning with Python concepts from this lesson.

In this lesson you learned: vanishing gradients occur when repeated multiplication by W_hh (with spectral radius < 1) drives gradients to zero exponentially over long sequences, exploding gradients occur when spectral radius > 1 and are fixed with gradient clipping, and LSTM was invented specifically to solve the vanishing gradient problem through an additive cell state update that provides a gradient highway. Next up we examine the LSTM cell architecture in detail.