nn.Embedding：可学习的词向量

Token ids are just labels. The id 5 is not greater than 2 in any meaningful way, so feeding raw ids to a net misleads it.

One way to represent ids is a giant one-hot vector, but with thousands of words it is huge, sparse, and carries no similarity.

An embedding maps each id to a short dense vector of floats. These few numbers can encode rich meaning about the word.

Think of nn.Embedding as a lookup table: row i holds the vector for token id i. Indexing the table fetches that row.

You set num_embeddings to your vocab size and embedding_dim to how many numbers each word vector holds.

Pass a tensor of ids and the layer returns their vectors. A batch of ids becomes a batch of dense vectors instantly.

The embedding adds an extra dimension. An input of shape (batch, length) becomes (batch, length, embedding_dim).

At first the embedding rows are random. They hold no meaning yet, just noise waiting to be shaped by training.

Embedding weights are parameters. Gradient descent nudges each word vector during training, so the table learns over time.

A larger embedding_dim can capture more nuance but costs memory. Small tasks use 16 to 100; large language models use far more.

The embedding is usually the first layer. It converts ids to vectors, then later layers process those vectors as usual.

How does nn.Embedding turn a token id into a vector?

nn.Embedding is a trainable lookup table mapping ids to dense vectors. It starts random and learns meaning through training. ✅