0Pricing
Machine Learning Academy · Lesson

Vanilla RNNs: Hidden State and Sequence Unrolling

Learners will implement a one-step RNN cell manually, unroll it across a short sequence, and visualise how the hidden state accumulates context.

Vanilla RNNs: Hidden State and Sequence Unrolling is a free Machine Learning Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Machine Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why We Need Recurrent Networks

Standard feedforward networks treat each input independently — they have no memory of previous inputs. But many real-world problems involve sequential data where context from the past matters: predicting the next word in a sentence, forecasting tomorrow's stock price from historical data, or classifying a gesture from a video frame sequence. Recurrent Neural Networks (RNNs) solve this by maintaining a hidden state that carries information across timesteps.

# Examples of sequential data:
sequences = {
    'NLP': 'The cat sat on the ___  (predict next word)',
    'Time Series': '[1.2, 1.5, 1.3, 1.8, ???]',
    'Speech': '[audio_t0, audio_t1, ..., audio_tN]',
    'Video': '[frame_1, frame_2, ..., frame_T]',
    'DNA': 'ATCGATCG... (biological sequence)',
}
for name, example in sequences.items():
    print(f'{name}: {example}')

The Vanilla RNN Cell

A vanilla RNN cell takes two inputs: the current input x_t and the previous hidden state h_{t-1}. It produces the next hidden state h_t using the formula: h_t = tanh(W_hh * h_{t-1} + W_xh * x_t + b). The same weight matrices W_hh and W_xh are used at every timestep — this is weight sharing across time, analogous to how CNNs share weights across space. The hidden state carries the network's memory of all past inputs.

import torch

def rnn_cell(x_t, h_prev, W_xh, W_hh, b):
    '''One step of a vanilla RNN cell'''
    # x_t: (batch, input_size)
    # h_prev: (batch, hidden_size)
    h_t = torch.tanh(
        x_t @ W_xh.T +    # input contribution
        h_prev @ W_hh.T +  # hidden-to-hidden contribution
        b                  # bias
    )
    return h_t

# Example: input_size=4, hidden_size=8
batch = 3
x_t   = torch.randn(batch, 4)
h_prev = torch.zeros(batch, 8)
W_xh  = torch.randn(8, 4) * 0.01
W_hh  = torch.randn(8, 8) * 0.01
b     = torch.zeros(8)
h_t = rnn_cell(x_t, h_prev, W_xh, W_hh, b)
print(h_t.shape)   # (3, 8)

Unrolling the RNN Through Time

To process a sequence of length T, the RNN cell is applied T times in a loop — this is called unrolling (or unfolding) through time. The hidden state from step t-1 is passed to step t, connecting all timesteps. During backpropagation, gradients must flow back through every timestep — this is called Backpropagation Through Time (BPTT). The depth of this unrolled graph equals the sequence length, creating challenges for long sequences.

import torch
import torch.nn as nn

# Unroll RNN manually over a sequence
batch_size, seq_len, input_size, hidden_size = 4, 10, 8, 16

rnn_cell = nn.RNNCell(input_size, hidden_size)
sequence = torch.randn(seq_len, batch_size, input_size)

h = torch.zeros(batch_size, hidden_size)  # initial hidden state
hidden_states = []

for t in range(seq_len):
    x_t = sequence[t]   # (batch, input_size)
    h = rnn_cell(x_t, h)  # apply RNN cell
    hidden_states.append(h)

print('Final hidden state:', h.shape)     # (4, 16)
print('All hidden states:', len(hidden_states), 'steps')

Using nn.RNN: The Module Version

nn.RNN handles the unrolling automatically. It takes inputs of shape (seq_len, batch, input_size) (or (batch, seq_len, input_size) with batch_first=True) and returns all hidden states and the final hidden state. Key parameters: num_layers stacks multiple RNN layers; bidirectional=True processes the sequence in both directions; dropout applies dropout between layers in multi-layer RNNs.

import torch
import torch.nn as nn

rnn = nn.RNN(
    input_size=16,
    hidden_size=32,
    num_layers=2,
    batch_first=True,    # input: (batch, seq, features)
    dropout=0.2          # between layers
)

X = torch.randn(8, 20, 16)   # batch=8, seq=20, features=16
output, h_n = rnn(X)

print('Output shape:', output.shape)   # (8, 20, 32) all steps
print('h_n shape:', h_n.shape)         # (2, 8, 32) last hidden

Hidden State Initialisation

The initial hidden state h_0 is passed as the second argument to nn.RNN. If not provided, it defaults to zeros. For sequence classification, the final hidden state h_n summarises the entire sequence. For sequence-to-sequence tasks, all intermediate hidden states in output are used. Initialising h_0 with zeros is standard; some applications learn the initial state as a parameter for improved performance on short sequences.

import torch
import torch.nn as nn

rnn = nn.RNN(8, 16, batch_first=True)
X = torch.randn(4, 10, 8)   # batch=4, seq=10, input=8

# Default: h_0 = zeros
output, h_n = rnn(X)
print('h_n with zero init:', h_n.shape)  # (1, 4, 16)

# Custom initial hidden state
h_0 = torch.randn(1, 4, 16)  # (num_layers, batch, hidden)
output, h_n = rnn(X, h_0)
print('h_n with custom init:', h_n.shape)  # (1, 4, 16)

Sequence Classification with Final Hidden State

A common RNN application is sequence classification: given a sequence, predict a single class label. The standard approach is to use only the final hidden state h_n as the input to a linear classifier, since it has seen all the sequence so far. For a bidirectional RNN, concatenate the final forward and backward hidden states to capture both past and future context.

import torch
import torch.nn as nn

class SequenceClassifier(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super().__init__()
        self.rnn = nn.RNN(input_size, hidden_size,
                          batch_first=True)
        self.fc  = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        _, h_n = self.rnn(x)
        # h_n shape: (1, batch, hidden) -> squeeze to (batch, hidden)
        h_n = h_n.squeeze(0)
        return self.fc(h_n)

model = SequenceClassifier(16, 32, 5)
X = torch.randn(8, 20, 16)
logits = model(X)
print(logits.shape)   # (8, 5) -- 8 samples, 5 classes

Many-to-Many: Sequence Labelling

In sequence labelling tasks (POS tagging, named entity recognition, time-series anomaly detection), you need a prediction at every timestep, not just the last one. Use the full output tensor from nn.RNN (shape: batch x seq_len x hidden_size) and apply a linear layer to each timestep independently. The linear layer weights are shared across timesteps — one more example of weight sharing in sequence models.

import torch
import torch.nn as nn

class SequenceLabeler(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super().__init__()
        self.rnn = nn.RNN(input_size, hidden_size,
                          batch_first=True)
        self.fc  = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        output, _ = self.rnn(x)  # (batch, seq, hidden)
        return self.fc(output)   # (batch, seq, num_classes)

model = SequenceLabeler(8, 16, 3)
X = torch.randn(4, 10, 8)       # 4 seqs of 10 timesteps
logits = model(X)
print(logits.shape)              # (4, 10, 3) per-step labels

Stacked and Bidirectional RNNs

Stacked RNNs (num_layers > 1) feed the output of one RNN layer as input to the next, learning progressively more abstract temporal representations. Bidirectional RNNs process the sequence both forward (left to right) and backward (right to left) simultaneously, then concatenate the hidden states. Bidirectional processing allows each output to incorporate context from both past and future — useful for sentence understanding where future words clarify the meaning of past words.

import torch
import torch.nn as nn

# Stacked bidirectional RNN
brnn = nn.RNN(
    input_size=16,
    hidden_size=32,
    num_layers=3,          # 3 stacked layers
    batch_first=True,
    bidirectional=True     # forward + backward
)

X = torch.randn(4, 10, 16)
output, h_n = brnn(X)

# Output: (batch, seq, hidden*2) because bidirectional
print('Output shape:', output.shape)  # (4, 10, 64)
# h_n: (num_layers*2, batch, hidden) -- 2 dirs x 3 layers
print('h_n shape:', h_n.shape)        # (6, 4, 32)

Backpropagation Through Time (BPTT)

Backpropagation Through Time unrolls the RNN and applies standard backpropagation through the unrolled graph. For a sequence of length T, gradients are computed at every timestep and propagated backward. The gradient of the loss with respect to parameters involves products of the same weight matrix W_hh taken T times. When the spectral radius of W_hh is less than 1, these products vanish; when greater than 1, they explode — this is the fundamental challenge of training RNNs on long sequences.

import torch
import torch.nn as nn

# Demonstrate gradient flow through different sequence lengths
rnn = nn.RNNCell(4, 8)
h = torch.zeros(1, 8, requires_grad=True)

# Short sequence: gradients flow back relatively well
for t in range(5):
    x = torch.randn(1, 4)
    h = rnn(x, h)
loss = h.sum()
loss.backward()
print('h.grad (seq=5):', h.grad.norm().item())

# For long sequences (T=100+), vanilla RNN gradients
# typically vanish (near zero) or explode (very large)
# This is why LSTM/GRU were invented

Handling Variable-Length Sequences

In practice, sequences in a batch have different lengths (sentences of different word counts). PyTorch handles this with packed sequences: torch.nn.utils.rnn.pack_padded_sequence removes the padding from a batch and packs sequences efficiently. After the RNN, pad_packed_sequence restores the padded format. Without packing, the RNN processes padding tokens unnecessarily and may pollute the hidden state with meaningless padding information.

import torch
import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

rnn = nn.RNN(4, 8, batch_first=True)

# Padded sequences: length 5, 3, 2
X = torch.zeros(3, 5, 4)  # (batch=3, max_seq=5, features=4)
X[0, :, :] = torch.randn(5, 4)  # full seq
X[1, :3, :] = torch.randn(3, 4) # length 3
X[2, :2, :] = torch.randn(2, 4) # length 2
lengths = torch.tensor([5, 3, 2])

packed = pack_padded_sequence(X, lengths, batch_first=True)
out_packed, h_n = rnn(packed)
out, _ = pad_packed_sequence(out_packed, batch_first=True)
print(out.shape)  # (3, 5, 8) -- back to padded form

When to Use Vanilla RNNs

Vanilla RNNs are rarely used in practice because they suffer severely from vanishing gradients on sequences longer than 10-20 timesteps. They are primarily useful for educational purposes and very short sequences. For any real application with sequences longer than 20 timesteps, use LSTM or GRU, which have gating mechanisms specifically designed to preserve information over long distances. For sequences where order is less critical, Transformer architectures often outperform both.

# When to use each sequence model:
use_cases = {
    'Vanilla RNN (nn.RNN)':  'Short sequences (<20 steps), learning/demos',
    'LSTM':                  'Long sequences, NLP, time-series (general)',
    'GRU':                   'Similar to LSTM but faster, fewer params',
    'Transformer':           'Parallelisable, long documents, state-of-art NLP',
    'Temporal Conv (TCN)':   'Long sequences, strong parallelism, audio',
}
for model, use in use_cases.items():
    print(f'{model}: {use}')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: vanilla RNNs maintain a hidden state that carries memory across timesteps using the formula h_t = tanh(W_xh * x_t + W_hh * h_{t-1} + b), unrolling applies the same cell at every timestep with shared weights, and BPTT propagates gradients back through the unrolled graph, causing vanishing/exploding gradient problems on long sequences. Next up we examine the vanishing gradient problem in depth and understand why LSTM was designed to solve it.

Frequently asked questions

Is the “Vanilla RNNs: Hidden State and Sequence Unrolling” lesson free?

Yes — the full text of “Vanilla RNNs: Hidden State and Sequence Unrolling” is free to read here on the web, and the Machine Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Machine Learning Academy course, upgrade to CoddyKit PRO.

What will I learn in “Vanilla RNNs: Hidden State and Sequence Unrolling”?

Learners will implement a one-step RNN cell manually, unroll it across a short sequence, and visualise how the hidden state accumulates context. You practise Machine Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Machine Learning Academy?

No prior experience is required. Machine Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Vanilla RNNs: Hidden State and Sequence Unrolling” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Machine Learning Academy lesson?

Yes. Every Machine Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Vanilla RNNs: Hidden State and Sequence Unrolling
  2. The Vanishing Gradient Problem in Deep Time Steps
  3. LSTM Cell: Input, Forget, and Output Gates
  4. Sequence-to-One: Sentiment Analysis with an LSTM
← Back to Machine Learning Academy