Sequence-to-One: Sentiment Analysis with an LSTM

In sequence-to-one tasks, the model reads an entire sequence and produces a single output label. Sentiment analysis is the classic example: given a full movie review ('I loved every minute of this film'), predict a single label — positive or negative.

This contrasts with sequence-to-sequence (translation, where every input token produces an output token) and one-to-sequence (image captioning). In sequence-to-one, we discard intermediate LSTM outputs and use only the final hidden state h_T, which theoretically encodes the meaning of the entire input sequence.

The IMDB dataset contains 50,000 movie reviews (25,000 train, 25,000 test) labelled as positive or negative. Each review is a variable-length string of text. Before feeding it to an LSTM, we need three preprocessing steps: tokenisation (split text into words), vocabulary building (map words to integer indices), and padding (make all sequences the same length).

We also need to handle reviews that are very long — LSTM training slows quadratically with sequence length, so truncating to 256 or 512 tokens is common practice without significantly hurting accuracy.

Padding is necessary because PyTorch DataLoaders require tensors of the same shape within a batch. We pad shorter sequences with a special <pad> token (usually index 0) and truncate longer ones to a maximum length.

The key insight is that the LSTM should ignore padding tokens. PyTorch's pack_padded_sequence and pad_packed_sequence utilities let you efficiently skip padding during the LSTM computation, which is important for correctness — otherwise the model will update its hidden state based on meaningless padding, distorting the final h_T.

Integer word indices cannot be fed directly into an LSTM — we need to convert them to continuous vectors. An embedding layer nn.Embedding(vocab_size, embedding_dim) is a lookup table that maps each word index to a dense vector of floats.

Common embedding dimensions are 100, 200, or 300. The embeddings are learnable parameters that get updated during training alongside the LSTM weights. Alternatively, you can initialise them from pre-trained word vectors like GloVe or Word2Vec, which dramatically speeds up training on small datasets by providing pre-learned semantic relationships between words.

The complete model architecture has three parts: an embedding layer, an LSTM, and a fully connected head. The embedding converts token indices to vectors, the LSTM processes the sequence, and the linear layer maps the final hidden state to a single logit.

We add dropout after the embedding and before the linear layer to regularise the model. For the output, we apply a sigmoid to get a probability between 0 and 1. Training uses Binary Cross-Entropy loss since this is binary classification.

For binary classification we use nn.BCEWithLogitsLoss, which combines sigmoid activation and binary cross-entropy loss in a single numerically stable operation. This is preferred over applying sigmoid manually followed by nn.BCELoss.

We use the Adam optimiser with a learning rate of 1e-3. Adam adapts learning rates per parameter, making it well-suited for sparse gradients that arise from embeddings (only the embeddings for words that appear in the batch receive gradient updates). Use gradient clipping with torch.nn.utils.clip_grad_norm_ to prevent exploding gradients, a common issue in RNN training.

The training loop for sequence-to-one classification follows the standard PyTorch pattern: forward pass, compute loss, backpropagate, clip gradients, update parameters. We also compute accuracy by comparing predicted class (sigmoid output > 0.5) to ground truth labels.

One important consideration is moving both the model and batch tensors to the same device (CPU or GPU). Using model.to(device) and tensor.to(device) ensures consistent computation. On a GPU, training a two-layer LSTM on IMDB typically takes 2-5 minutes per epoch.

During evaluation we call model.eval() and wrap forward passes in torch.no_grad() to disable dropout and gradient tracking. This gives us an unbiased view of generalisation performance.

We track four numbers across epochs: train loss, validation loss, train accuracy, and validation accuracy. If train accuracy climbs but validation accuracy stagnates, the model is overfitting — increase dropout or reduce model size. If both are low, the model is underfitting — add LSTM layers or increase hidden size.

A well-tuned LSTM sentiment classifier on IMDB typically achieves 85-90% test accuracy. Common mistakes that hurt performance include: not setting model.eval() during evaluation (dropout remains active), not zeroing gradients before backprop, and training for too few epochs before the embeddings converge.

Another pitfall is using the same vocabulary index for unknown words and padding. Separate <unk> (index 0) and <pad> (index 1) tokens and pass padding_idx=1 to the embedding layer so pad embeddings are always zero and their gradients are suppressed during training.

To classify new reviews at inference time, we apply the same preprocessing pipeline used during training: tokenise, convert to indices, truncate, and add a batch dimension. The model returns a logit; applying sigmoid converts it to a probability.

A prediction score above 0.5 indicates positive sentiment. You can adjust this threshold: lowering it to 0.3 increases recall for positive reviews at the cost of precision. Always apply model.eval() and torch.no_grad() during inference to disable dropout and avoid unnecessary gradient computation.

While LSTMs were state-of-the-art for NLP until around 2018, transformer-based models like BERT now dominate most NLP benchmarks. BERT achieves ~93-95% on IMDB, compared to LSTM's ~87-90%.

However, LSTMs remain relevant for several reasons: they are far more computationally efficient for very long sequences (transformers scale quadratically with sequence length), they work well on time-series data where the sequential nature is intrinsic, and they have a much smaller memory footprint. Understanding LSTMs also provides the conceptual foundation needed to understand how attention mechanisms improve upon them.

Test your understanding of LSTM-based sequence classification from this lesson.

In this lesson you learned: sequence-to-one architecture reads an entire input sequence and maps it to a single label using the final LSTM hidden state, text preprocessing pipeline requires tokenisation, vocabulary building, and padding before data enters the model, and embedding layers convert discrete word indices into dense learnable vectors that capture semantic meaning. Next up we explore transfer learning, where pre-trained model weights replace training from scratch.