트랜스포머 인코더 블록 쌓기

A transformer is just one encoder block repeated. Learn the block and you understand the whole tower, from tiny models to giant ones.

Each block has two parts: a multi-head attention sub-layer, then a small feedforward network. Both wrapped with residuals and normalization.

The block starts with self-attention, letting every token mix in context from the rest of the sequence before any further processing.

A residual adds the sub-layer's input back to its output. This shortcut lets gradients flow and keeps deep stacks trainable.

After adding the residual, layer norm rescales each token's features to a stable distribution, steadying training across layers.

Next comes a position-wise feedforward network: expand to a wider hidden size, apply a nonlinearity, then project back down.

The feedforward layer treats each token independently. Attention shared information; this step refines each token on its own.

The feedforward output gets the same treatment: a residual add plus another layer norm, finishing the block.

Stack many identical blocks and the model builds richer representations layer by layer. Depth is where transformer power comes from.

Many modern models apply layer norm before each sub-layer instead of after. Pre-norm trains more stably in very deep stacks.

PyTorch gives you nn.TransformerEncoderLayer and nn.TransformerEncoder, so you can assemble a full stack in just a couple of lines.

You assembled an encoder block: attention, residual, norm, feedforward, residual, norm. Stack it deep and you have a transformer. Amazing work!