Sel LSTM: Gerbang Input, Lupa, dan Output
Peserta akan membuat diagram sel LSTM, menelusuri aliran informasi melalui setiap gerbang, dan menerapkan pengklasifikasi teks LSTM menggunakan nn.LSTM.
Sel LSTM: Gerbang Input, Lupa, dan Output adalah pelajaran Machine Learning Academy gratis di CoddyKit. Ini adalah pelajaran 3 dari 4. Kamu bisa membaca pelajaran lengkapnya di bawah secara gratis — lalu praktikkan langsung di browser dengan editor kode bawaan dan tutor AI 24/7. Ini adalah bagian dari jalur belajar Machine Learning Academy, dan progresmu tersinkronisasi di web dan aplikasi CoddyKit. Kursus Machine Learning Academy mencakup 4 pelajaran total.
Bagian dari pelajaran ini belum diterjemahkan dan ditampilkan dalam bahasa Inggris.
Why RNNs Need Gating Mechanisms
Standard Vanilla RNNs struggle with long-range dependencies because gradients vanish during backpropagation through time. The hidden state h_t = tanh(W_h * h_{t-1} + W_x * x_t) overwrites previous context with each new input, causing the network to forget information from many steps back.
The Long Short-Term Memory (LSTM) was designed in 1997 by Hochreiter and Schmidhuber to solve this problem. It introduces a cell state — a separate memory conveyor belt — alongside gating mechanisms that explicitly control what information is added, removed, or passed forward.
The LSTM Cell State Concept
The LSTM has two internal states: the cell state C_t and the hidden state h_t. The cell state runs like a conveyor belt through the entire sequence, with gates controlling minor linear interactions. This makes gradients flow more easily through time.
The hidden state h_t is the output at each timestep, computed from the cell state. Think of C_t as long-term memory and h_t as working memory that gets passed to the next layer and returned as output.
import torch
import torch.nn as nn
# An LSTM processes sequences and returns (output, (h_n, c_n))
lstm = nn.LSTM(input_size=10, hidden_size=32, batch_first=True)
print('LSTM parameters:', sum(p.numel() for p in lstm.parameters()))The Forget Gate
The forget gate is the first gate the input passes through. It reads the previous hidden state h_{t-1} and the current input x_t, and outputs a value between 0 and 1 for each element in the cell state.
A value of 1 means 'keep everything', while 0 means 'forget completely'. The formula is: f_t = sigmoid(W_f · [h_{t-1}, x_t] + b_f). For example, when processing a new sentence in an NLP task, the forget gate would reset any stored subject information from the previous sentence.
import torch
import torch.nn as nn
# Manual forget gate illustration
batch_size, hidden_size, input_size = 1, 4, 3
h_prev = torch.zeros(batch_size, hidden_size)
x_t = torch.randn(batch_size, input_size)
W_f = torch.randn(hidden_size, hidden_size + input_size)
b_f = torch.zeros(hidden_size)
combined = torch.cat([h_prev, x_t], dim=1)
f_t = torch.sigmoid(combined @ W_f.T + b_f)
print('Forget gate output:', f_t) # Values between 0 and 1The Input Gate and Candidate Cell
The input gate decides which new values to store in the cell state. It has two parts working together: the input gate i_t = sigmoid(W_i · [h_{t-1}, x_t] + b_i) determines how much to update, and the candidate cell g_t = tanh(W_g · [h_{t-1}, x_t] + b_g) creates candidate values to potentially add.
The sigmoid controls the gate (0 = closed, 1 = open) while tanh creates values in the range [-1, 1]. The new information that gets added to the cell state is i_t * g_t — the gate filtered by the candidate values.
# Input gate and candidate values
W_i = torch.randn(hidden_size, hidden_size + input_size)
b_i = torch.zeros(hidden_size)
W_g = torch.randn(hidden_size, hidden_size + input_size)
b_g = torch.zeros(hidden_size)
i_t = torch.sigmoid(combined @ W_i.T + b_i) # Input gate: what to update
g_t = torch.tanh(combined @ W_g.T + b_g) # Candidate values to add
print('Input gate:', i_t.detach())
print('Candidate cell:', g_t.detach())Updating the Cell State
Once we have the forget gate f_t, input gate i_t, and candidate cell g_t, updating the cell state is straightforward: C_t = f_t * C_{t-1} + i_t * g_t.
The first term f_t * C_{t-1} applies the forget gate — selectively erasing information from the previous cell state. The second term i_t * g_t adds new information selectively. This additive structure is what allows gradients to flow back through many timesteps without vanishing, since the gradient flows directly through the addition operation.
# Update cell state
C_prev = torch.zeros(batch_size, hidden_size) # Previous cell state
C_t = f_t * C_prev + i_t * g_t # Element-wise operations
print('Updated cell state C_t:', C_t.detach())
# The additive update is key: gradients flow through + easily
# Compare to vanilla RNN: h_t = tanh(W * h_{t-1} + U * x_t)
# where gradients must flow through the tanh compression each stepThe Output Gate
The output gate controls what part of the cell state gets exposed as the hidden state output h_t. First, the output gate decides which parts of the cell state to output: o_t = sigmoid(W_o · [h_{t-1}, x_t] + b_o).
Then, the cell state is passed through tanh (to push values between -1 and 1) and multiplied by the output gate: h_t = o_t * tanh(C_t). The hidden state h_t serves as both the output at this timestep and the input to the next LSTM step alongside the new cell state.
# Output gate and hidden state
W_o = torch.randn(hidden_size, hidden_size + input_size)
b_o = torch.zeros(hidden_size)
o_t = torch.sigmoid(combined @ W_o.T + b_o) # Output gate
h_t = o_t * torch.tanh(C_t) # New hidden state
print('Output gate:', o_t.detach())
print('New hidden state h_t:', h_t.detach())LSTM Parameter Count and Weight Matrices
An LSTM has four weight matrices (forget, input, candidate, output), each of size (hidden_size, hidden_size + input_size) plus bias. The total parameter count is 4 * hidden_size * (hidden_size + input_size) + 4 * hidden_size.
PyTorch's nn.LSTM packs all four gates into combined weight matrices weight_ih_l0 and weight_hh_l0 for efficiency. This is why LSTM training can be slower than a GRU (Gated Recurrent Unit), which uses only two gates and fewer parameters.
import torch.nn as nn
hidden = 64
input_s = 32
lstm = nn.LSTM(input_size=input_s, hidden_size=hidden, batch_first=True)
# PyTorch stores weight_ih (input-hidden) and weight_hh (hidden-hidden)
print('weight_ih_l0 shape:', lstm.weight_ih_l0.shape) # (4*hidden, input)
print('weight_hh_l0 shape:', lstm.weight_hh_l0.shape) # (4*hidden, hidden)
print('Total params:', sum(p.numel() for p in lstm.parameters()))Using nn.LSTM in PyTorch
PyTorch's nn.LSTM accepts a sequence tensor of shape (batch, seq_len, input_size) when batch_first=True, and optionally an initial state tuple (h_0, c_0). It returns the output tensor (all hidden states) and a tuple of the final states.
For sequence classification, you typically only need the last hidden state h_n from the final timestep. For sequence labelling tasks (like named entity recognition), you use all timestep outputs. Always initialise hidden states to zero unless you have a reason to pass context between batches.
import torch
import torch.nn as nn
batch_size, seq_len, input_size = 16, 20, 32
hidden_size = 64
lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size, batch_first=True)
x = torch.randn(batch_size, seq_len, input_size)
# Forward pass
output, (h_n, c_n) = lstm(x)
print('Output shape (all timesteps):', output.shape) # (16, 20, 64)
print('h_n shape (final hidden):', h_n.shape) # (1, 16, 64)
print('c_n shape (final cell):', c_n.shape) # (1, 16, 64)Multi-Layer and Bidirectional LSTMs
A stacked LSTM feeds the output of one LSTM layer as input to the next, allowing the network to learn hierarchical temporal representations. Set num_layers=2 or more in nn.LSTM.
A bidirectional LSTM runs one LSTM forward through the sequence and another backward, concatenating their hidden states. This gives each timestep context from both the past and the future. Set bidirectional=True; the output dimension becomes 2 * hidden_size. Bidirectional LSTMs are very effective for NLP tasks where both left and right context matter.
import torch
import torch.nn as nn
# Bidirectional stacked LSTM
lstm = nn.LSTM(
input_size=32,
hidden_size=64,
num_layers=2,
batch_first=True,
bidirectional=True,
dropout=0.3 # Dropout between layers
)
x = torch.randn(16, 20, 32)
output, (h_n, c_n) = lstm(x)
print('Bidirectional output shape:', output.shape) # (16, 20, 128)
print('h_n shape:', h_n.shape) # (4, 16, 64): 2 layers * 2 directionsGRU: A Simpler Alternative to LSTM
The Gated Recurrent Unit (GRU) simplifies the LSTM by merging the forget and input gates into a single update gate, and combining the cell and hidden states. It uses only two gates: the reset gate (how much past to forget) and the update gate (how much to update).
GRU has fewer parameters and trains faster than LSTM while often achieving comparable performance. Use LSTM when you need maximum expressiveness on complex sequences, and GRU when speed and simplicity matter. In practice, empirical testing on your specific task determines the better choice.
import torch
import torch.nn as nn
# GRU is simpler: only 3 weight matrices instead of 4
gru = nn.GRU(input_size=32, hidden_size=64, batch_first=True)
lstm = nn.LSTM(input_size=32, hidden_size=64, batch_first=True)
print('GRU params:', sum(p.numel() for p in gru.parameters()))
print('LSTM params:', sum(p.numel() for p in lstm.parameters()))
# GRU has 25% fewer parameters than LSTM for same hidden sizeLSTM Intuition Through a Language Example
Consider the sentence: 'The author, who lived in Paris for many years, wrote a novel.' An LSTM needs to remember 'author' (singular) when it reaches 'wrote' many words later.
The forget gate keeps 'author' in the cell state throughout the long relative clause. The input gate adds new relevant information like 'wrote' to the cell state. The output gate uses the cell state at the right moment to produce the correct hidden state for downstream tasks like predicting the next word or classifying the sentence.
- Forget gate: Keep the subject through the relative clause
- Input gate: Store the verb information when encountered
- Output gate: Use stored information at prediction time
Quick Check
Test your understanding of LSTM gates from this lesson.
Lesson Recap
In this lesson you learned: the LSTM cell state acts as long-term memory that flows through the sequence with minimal modification, three gating mechanisms (forget, input, output) control information flow using sigmoid-gated multiplication, and the additive cell state update C_t = f_t * C_{t-1} + i_t * g_t allows gradients to flow without vanishing. Next up we apply LSTMs to a real sentiment analysis task, building an end-to-end text classifier.
Pertanyaan yang Sering Diajukan
Apakah pelajaran “Sel LSTM: Gerbang Input, Lupa, dan Output” gratis?
Ya — teks lengkap “Sel LSTM: Gerbang Input, Lupa, dan Output” gratis dibaca di sini di web. Untuk praktiknya secara interaktif (editor kode bawaan dan tutor AI 24/7) dan buka sisa kursus Machine Learning Academy, upgrade ke CoddyKit PRO. Kursus Machine Learning Academy mencakup 4 pelajaran total.
Apa yang akan aku pelajari di “Sel LSTM: Gerbang Input, Lupa, dan Output”?
Peserta akan membuat diagram sel LSTM, menelusuri aliran informasi melalui setiap gerbang, dan menerapkan pengklasifikasi teks LSTM menggunakan nn.LSTM. Kamu berlatih Machine Learning Academy dengan kode praktik yang langsung kamu jalankan di browser, dan tutor AI 24/7 menjawab pertanyaanmu saat kamu mengerjakan pelajaran ini.
Apakah aku perlu pengalaman untuk memulai Machine Learning Academy?
Tidak diperlukan pengalaman sebelumnya. Machine Learning Academy di CoddyKit dirancang untuk pemula hingga pelajar tingkat lanjut, jadi kamu bisa memulai di sini atau dari awal dan belajar sesuai kecepatan kamu sendiri. Ini adalah pelajaran 3 dari 4.
Berapa lama pelajaran “Sel LSTM: Gerbang Input, Lupa, dan Output” memakan waktu?
Sebagian besar pelajaran CoddyKit memakan waktu sekitar 5–10 menit. Setiap pelajaran ringkas dan interaktif, jadi kamu membuat kemajuan stabil dan melanjutkan dari tempat kamu tinggalkan di web dan aplikasi.
Bisakah aku menulis dan menjalankan kode dalam pelajaran Machine Learning Academy ini?
Ya. Setiap pelajaran Machine Learning Academy menyertakan editor kode bawaan, jadi kamu menulis dan menjalankan kode nyata langsung di browser dan mendapatkan umpan balik AI instan — tidak diperlukan penyiapan lokal.
Semua pelajaran dalam kursus ini
- RNN Vanilla: Keadaan Tersembunyi dan Penguraian Urutan
- Masalah Gradien Menghilang pada Langkah Waktu yang Dalam
- Sel LSTM: Gerbang Input, Lupa, dan Output
- Urutan-ke-Satu: Analisis Sentimen dengan LSTM