Transformer Architecture: Attention, Tokens, and Context

The Transformer is a neural network architecture introduced in 2017 that replaced recurrent networks for most NLP tasks. Unlike RNNs that process tokens one at a time, Transformers process the entire sequence in parallel using a mechanism called self-attention. This parallel processing makes training much faster and allows the model to capture long-range dependencies more effectively.

Self-attention allows each token in a sequence to attend to every other token simultaneously. For the sentence 'The bank by the river was steep', the word 'bank' can attend strongly to 'river' to resolve its meaning. Each token produces three vectors: Query (Q), Key (K), and Value (V), which are used to compute weighted relationships between all token pairs.

The attention score between token i and token j is computed as the dot product of Q_i and K_j, divided by the square root of the key dimension to prevent vanishingly small gradients. The formula is: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) * V. The scaling factor sqrt(d_k) keeps gradients stable for large embedding dimensions.

Instead of one set of Q, K, V projections, Transformers use multi-head attention: h parallel attention heads, each learning different aspects of token relationships. One head might learn syntactic dependencies (subject-verb), another semantic ones (synonyms). The outputs of all heads are concatenated and projected to produce the final representation.

BERT (Bidirectional Encoder Representations from Transformers) reads the entire sequence at once, attending to both left and right context simultaneously. Earlier models like GPT read left-to-right only. This bidirectionality lets BERT understand that 'bank' in 'river bank' differs from 'bank' in 'bank account' by seeing all surrounding words at once.

BERT introduces two special tokens. The [CLS] (classification) token is prepended to every input; after processing, its final hidden state aggregates sentence-level information and is used for classification tasks. The [SEP] token separates two sentences in tasks like question answering or next-sentence prediction. Understanding these tokens is essential when building BERT pipelines.

Because Transformers process all tokens in parallel, they have no inherent sense of token order. Positional encodings are added to each token embedding to inject position information. BERT uses learned positional embeddings while the original Transformer used sinusoidal functions. Without positional encoding, 'cat bites dog' and 'dog bites cat' would produce identical representations.

Each BERT encoder layer consists of two sub-layers: multi-head self-attention followed by a position-wise feed-forward network (two linear layers with a GELU activation). Each sub-layer has a residual connection and layer normalisation. BERT-base stacks 12 such layers; BERT-large uses 24. Deeper stacks capture more abstract linguistic structure.

BERT tokenises text using WordPiece subword tokenisation. Rare words are split into frequent sub-units: 'unbelievable' might become ['un', '##believe', '##able']. The ## prefix indicates a continuation subword. This approach handles out-of-vocabulary words gracefully and uses a vocabulary of ~30,000 tokens, balancing coverage and embedding table size.

When processing batches of variable-length sentences, shorter sentences are padded with [PAD] tokens to match the longest sequence. The attention mask is a binary tensor (1 for real tokens, 0 for padding) that tells the model to ignore padding positions in the attention computation. Without this mask, the model would attend to meaningless padding tokens and corrupt its representations.

BERT was pre-trained on two tasks. Masked Language Modelling (MLM) randomly masks 15% of tokens and trains BERT to predict the original token from context, forcing bidirectional understanding. Next Sentence Prediction (NSP) trains BERT to determine whether two sentences are consecutive, helping sentence-pair tasks. Fine-tuning then adapts these rich representations to downstream tasks with minimal additional training.

Test your understanding of Machine Learning with Python concepts from this lesson.

In this lesson you learned: Transformers use parallel self-attention instead of sequential recurrence, BERT reads bidirectional context using masked language modelling pre-training, and special tokens [CLS] and [SEP] structure BERT's inputs for classification and sentence-pair tasks. Next up we explore how Hugging Face tokenizers encode raw text into the tensor format BERT expects.