Machine Learning Academy · 课时

基础 RNN：隐藏状态与序列展开

您将手动实现单步 RNN 单元，在短序列上展开它，并直观展示隐藏状态如何累积上下文信息。

基础 RNN：隐藏状态与序列展开 是 CoddyKit 上的免费 Machine Learning Academy 课时。这是第 1 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Machine Learning Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Machine Learning Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Why We Need Recurrent Networks

Standard feedforward networks treat each input independently — they have no memory of previous inputs. But many real-world problems involve sequential data where context from the past matters: predicting the next word in a sentence, forecasting tomorrow's stock price from historical data, or classifying a gesture from a video frame sequence. Recurrent Neural Networks (RNNs) solve this by maintaining a hidden state that carries information across timesteps.

# Examples of sequential data:
sequences = {
    'NLP': 'The cat sat on the ___  (predict next word)',
    'Time Series': '[1.2, 1.5, 1.3, 1.8, ???]',
    'Speech': '[audio_t0, audio_t1, ..., audio_tN]',
    'Video': '[frame_1, frame_2, ..., frame_T]',
    'DNA': 'ATCGATCG... (biological sequence)',
}
for name, example in sequences.items():
    print(f'{name}: {example}')

The Vanilla RNN Cell

A vanilla RNN cell takes two inputs: the current input x_t and the previous hidden state h_{t-1}. It produces the next hidden state h_t using the formula: h_t = tanh(W_hh * h_{t-1} + W_xh * x_t + b). The same weight matrices W_hh and W_xh are used at every timestep — this is weight sharing across time, analogous to how CNNs share weights across space. The hidden state carries the network's memory of all past inputs.

import torch

def rnn_cell(x_t, h_prev, W_xh, W_hh, b):
    '''One step of a vanilla RNN cell'''
    # x_t: (batch, input_size)
    # h_prev: (batch, hidden_size)
    h_t = torch.tanh(
        x_t @ W_xh.T +    # input contribution
        h_prev @ W_hh.T +  # hidden-to-hidden contribution
        b                  # bias
    )
    return h_t

# Example: input_size=4, hidden_size=8
batch = 3
x_t   = torch.randn(batch, 4)
h_prev = torch.zeros(batch, 8)
W_xh  = torch.randn(8, 4) * 0.01
W_hh  = torch.randn(8, 8) * 0.01
b     = torch.zeros(8)
h_t = rnn_cell(x_t, h_prev, W_xh, W_hh, b)
print(h_t.shape)   # (3, 8)

Unrolling the RNN Through Time

To process a sequence of length T, the RNN cell is applied T times in a loop — this is called unrolling (or unfolding) through time. The hidden state from step t-1 is passed to step t, connecting all timesteps. During backpropagation, gradients must flow back through every timestep — this is called Backpropagation Through Time (BPTT). The depth of this unrolled graph equals the sequence length, creating challenges for long sequences.

import torch
import torch.nn as nn

# Unroll RNN manually over a sequence
batch_size, seq_len, input_size, hidden_size = 4, 10, 8, 16

rnn_cell = nn.RNNCell(input_size, hidden_size)
sequence = torch.randn(seq_len, batch_size, input_size)

h = torch.zeros(batch_size, hidden_size)  # initial hidden state
hidden_states = []

for t in range(seq_len):
    x_t = sequence[t]   # (batch, input_size)
    h = rnn_cell(x_t, h)  # apply RNN cell
    hidden_states.append(h)

print('Final hidden state:', h.shape)     # (4, 16)
print('All hidden states:', len(hidden_states), 'steps')

Using nn.RNN: The Module Version

nn.RNN handles the unrolling automatically. It takes inputs of shape (seq_len, batch, input_size) (or (batch, seq_len, input_size) with batch_first=True) and returns all hidden states and the final hidden state. Key parameters: num_layers stacks multiple RNN layers; bidirectional=True processes the sequence in both directions; dropout applies dropout between layers in multi-layer RNNs.

import torch
import torch.nn as nn

rnn = nn.RNN(
    input_size=16,
    hidden_size=32,
    num_layers=2,
    batch_first=True,    # input: (batch, seq, features)
    dropout=0.2          # between layers
)

X = torch.randn(8, 20, 16)   # batch=8, seq=20, features=16
output, h_n = rnn(X)

print('Output shape:', output.shape)   # (8, 20, 32) all steps
print('h_n shape:', h_n.shape)         # (2, 8, 32) last hidden

Hidden State Initialisation

The initial hidden state h_0 is passed as the second argument to nn.RNN. If not provided, it defaults to zeros. For sequence classification, the final hidden state h_n summarises the entire sequence. For sequence-to-sequence tasks, all intermediate hidden states in output are used. Initialising h_0 with zeros is standard; some applications learn the initial state as a parameter for improved performance on short sequences.

import torch
import torch.nn as nn

rnn = nn.RNN(8, 16, batch_first=True)
X = torch.randn(4, 10, 8)   # batch=4, seq=10, input=8

# Default: h_0 = zeros
output, h_n = rnn(X)
print('h_n with zero init:', h_n.shape)  # (1, 4, 16)

# Custom initial hidden state
h_0 = torch.randn(1, 4, 16)  # (num_layers, batch, hidden)
output, h_n = rnn(X, h_0)
print('h_n with custom init:', h_n.shape)  # (1, 4, 16)

Sequence Classification with Final Hidden State

A common RNN application is sequence classification: given a sequence, predict a single class label. The standard approach is to use only the final hidden state h_n as the input to a linear classifier, since it has seen all the sequence so far. For a bidirectional RNN, concatenate the final forward and backward hidden states to capture both past and future context.

import torch
import torch.nn as nn

class SequenceClassifier(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super().__init__()
        self.rnn = nn.RNN(input_size, hidden_size,
                          batch_first=True)
        self.fc  = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        _, h_n = self.rnn(x)
        # h_n shape: (1, batch, hidden) -> squeeze to (batch, hidden)
        h_n = h_n.squeeze(0)
        return self.fc(h_n)

model = SequenceClassifier(16, 32, 5)
X = torch.randn(8, 20, 16)
logits = model(X)
print(logits.shape)   # (8, 5) -- 8 samples, 5 classes

Many-to-Many: Sequence Labelling

In sequence labelling tasks (POS tagging, named entity recognition, time-series anomaly detection), you need a prediction at every timestep, not just the last one. Use the full output tensor from nn.RNN (shape: batch x seq_len x hidden_size) and apply a linear layer to each timestep independently. The linear layer weights are shared across timesteps — one more example of weight sharing in sequence models.

import torch
import torch.nn as nn

class SequenceLabeler(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super().__init__()
        self.rnn = nn.RNN(input_size, hidden_size,
                          batch_first=True)
        self.fc  = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        output, _ = self.rnn(x)  # (batch, seq, hidden)
        return self.fc(output)   # (batch, seq, num_classes)

model = SequenceLabeler(8, 16, 3)
X = torch.randn(4, 10, 8)       # 4 seqs of 10 timesteps
logits = model(X)
print(logits.shape)              # (4, 10, 3) per-step labels

Stacked and Bidirectional RNNs

Stacked RNNs (num_layers > 1) feed the output of one RNN layer as input to the next, learning progressively more abstract temporal representations. Bidirectional RNNs process the sequence both forward (left to right) and backward (right to left) simultaneously, then concatenate the hidden states. Bidirectional processing allows each output to incorporate context from both past and future — useful for sentence understanding where future words clarify the meaning of past words.

import torch
import torch.nn as nn

# Stacked bidirectional RNN
brnn = nn.RNN(
    input_size=16,
    hidden_size=32,
    num_layers=3,          # 3 stacked layers
    batch_first=True,
    bidirectional=True     # forward + backward
)

X = torch.randn(4, 10, 16)
output, h_n = brnn(X)

# Output: (batch, seq, hidden*2) because bidirectional
print('Output shape:', output.shape)  # (4, 10, 64)
# h_n: (num_layers*2, batch, hidden) -- 2 dirs x 3 layers
print('h_n shape:', h_n.shape)        # (6, 4, 32)

Backpropagation Through Time (BPTT)

Backpropagation Through Time unrolls the RNN and applies standard backpropagation through the unrolled graph. For a sequence of length T, gradients are computed at every timestep and propagated backward. The gradient of the loss with respect to parameters involves products of the same weight matrix W_hh taken T times. When the spectral radius of W_hh is less than 1, these products vanish; when greater than 1, they explode — this is the fundamental challenge of training RNNs on long sequences.

import torch
import torch.nn as nn

# Demonstrate gradient flow through different sequence lengths
rnn = nn.RNNCell(4, 8)
h = torch.zeros(1, 8, requires_grad=True)

# Short sequence: gradients flow back relatively well
for t in range(5):
    x = torch.randn(1, 4)
    h = rnn(x, h)
loss = h.sum()
loss.backward()
print('h.grad (seq=5):', h.grad.norm().item())

# For long sequences (T=100+), vanilla RNN gradients
# typically vanish (near zero) or explode (very large)
# This is why LSTM/GRU were invented

Handling Variable-Length Sequences

In practice, sequences in a batch have different lengths (sentences of different word counts). PyTorch handles this with packed sequences: torch.nn.utils.rnn.pack_padded_sequence removes the padding from a batch and packs sequences efficiently. After the RNN, pad_packed_sequence restores the padded format. Without packing, the RNN processes padding tokens unnecessarily and may pollute the hidden state with meaningless padding information.

import torch
import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

rnn = nn.RNN(4, 8, batch_first=True)

# Padded sequences: length 5, 3, 2
X = torch.zeros(3, 5, 4)  # (batch=3, max_seq=5, features=4)
X[0, :, :] = torch.randn(5, 4)  # full seq
X[1, :3, :] = torch.randn(3, 4) # length 3
X[2, :2, :] = torch.randn(2, 4) # length 2
lengths = torch.tensor([5, 3, 2])

packed = pack_padded_sequence(X, lengths, batch_first=True)
out_packed, h_n = rnn(packed)
out, _ = pad_packed_sequence(out_packed, batch_first=True)
print(out.shape)  # (3, 5, 8) -- back to padded form

When to Use Vanilla RNNs

Vanilla RNNs are rarely used in practice because they suffer severely from vanishing gradients on sequences longer than 10-20 timesteps. They are primarily useful for educational purposes and very short sequences. For any real application with sequences longer than 20 timesteps, use LSTM or GRU, which have gating mechanisms specifically designed to preserve information over long distances. For sequences where order is less critical, Transformer architectures often outperform both.

# When to use each sequence model:
use_cases = {
    'Vanilla RNN (nn.RNN)':  'Short sequences (<20 steps), learning/demos',
    'LSTM':                  'Long sequences, NLP, time-series (general)',
    'GRU':                   'Similar to LSTM but faster, fewer params',
    'Transformer':           'Parallelisable, long documents, state-of-art NLP',
    'Temporal Conv (TCN)':   'Long sequences, strong parallelism, audio',
}
for model, use in use_cases.items():
    print(f'{model}: {use}')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: vanilla RNNs maintain a hidden state that carries memory across timesteps using the formula h_t = tanh(W_xh * x_t + W_hh * h_{t-1} + b), unrolling applies the same cell at every timestep with shared weights, and BPTT propagates gradients back through the unrolled graph, causing vanishing/exploding gradient problems on long sequences. Next up we examine the vanishing gradient problem in depth and understand why LSTM was designed to solve it.

常见问题解答

「基础 RNN：隐藏状态与序列展开」课时是免费的吗？

是的 — 「基础 RNN：隐藏状态与序列展开」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Machine Learning Academy 课程的其余内容，请升级到 CoddyKit PRO。 Machine Learning Academy 课程共包含 4 节课。

「基础 RNN：隐藏状态与序列展开」这节课中我会学到什么？

您将手动实现单步 RNN 单元，在短序列上展开它，并直观展示隐藏状态如何累积上下文信息。你通过在浏览器中直接运行的动手代码来练习 Machine Learning Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Machine Learning Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Machine Learning Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 1 节课，共 4 节。

「基础 RNN：隐藏状态与序列展开」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Machine Learning Academy 课中编写并运行代码吗？

能。每节 Machine Learning Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Machine Learning Academy