Pack Sequences & Handle Padding
Train efficiently on variable lengths.
Pack Sequences & Handle Padding is a free Deep Learning Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Deep Learning Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Batches Want Rectangles
To process many sequences at once, a batch must be a neat rectangle. But real sentences have different lengths, so they don't line up.
Padding to a Common Length
The fix is padding: add filler tokens, usually zeros, to short sequences so every row in the batch is the same length.
pad_sequence Helper
PyTorch's pad_sequence stacks a list of variable-length tensors into one padded batch in a single call.
from torch.nn.utils.rnn import pad_sequence
batch = pad_sequence(seqs, batch_first=True)Padding Wastes Compute
Those filler tokens are fake. If the RNN processes them anyway, it burns compute and can let the padding pollute the hidden state.
Track the Real Lengths
Before padding, record each sequence's true length. You'll hand these lengths to PyTorch so it knows where the real data ends.
lengths = [len(s) for s in seqs]Packing the Batch
pack_padded_sequence turns the padded batch plus lengths into a compact form the RNN can run without touching the padding.
packed = pack_padded_sequence(batch, lengths, batch_first=True)Run the RNN on Packed Data
Feed the packed object straight into nn.LSTM or nn.GRU. The cell skips padded steps, so memory stays clean and clear.
out_packed, h_n = lstm(packed)Unpack the Output
To read per-step outputs again, call pad_packed_sequence to restore the rectangular padded shape after the RNN finishes.
out, lengths = pad_packed_sequence(out_packed, batch_first=True)Sort by Length
Classic packing wants sequences sorted longest first. Set enforce_sorted to False and PyTorch handles unsorted batches for you.
Mask the Loss
Even with packing, ignore padded positions when scoring. A mask or ignore_index keeps fake tokens out of the loss.
loss_fn = nn.CrossEntropyLoss(ignore_index=PAD_ID)Wire It Into the DataLoader
Do the padding inside a custom collate_fn so every batch arrives already padded, with lengths ready for packing.
Quick Check
Why pack a padded batch before feeding it to an RNN?
Recap
Pad to align a batch, track true lengths, then pack so the RNN ignores filler. Mask padded tokens in the loss too. ✅
Frequently asked questions
Is the “Pack Sequences & Handle Padding” lesson free?
Yes — the full text of “Pack Sequences & Handle Padding” is free to read here on the web, and the Deep Learning Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Deep Learning Academy course, upgrade to CoddyKit PRO.
What will I learn in “Pack Sequences & Handle Padding”?
Train efficiently on variable lengths. You practise Deep Learning Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Deep Learning Academy?
No prior experience is required. Deep Learning Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Pack Sequences & Handle Padding” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Deep Learning Academy lesson?
Yes. Every Deep Learning Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Why Sequences Need Memory
- The Vanilla RNN Cell
- LSTM & GRU Gates
- Pack Sequences & Handle Padding