From Sparse Counts to Dense Vectors

Bag-of-words and TF-IDF turned text into long count vectors. They work, but those sparse vectors hide a real weakness you are about to see.

A bag-of-words vector has one slot per vocabulary word, so a short sentence is almost all zeros. We call this a sparse representation.

With 50,000 vocabulary words, every document becomes a 50,000-long vector. That is a lot of dimensions for just a few real signals.

In bag-of-words, cat and kitten sit in totally separate slots. The model sees no similarity between them at all, even though we do.

Sparse vectors treat every word as unrelated to every other. They capture presence but completely miss meaning and word relationships.

An embedding represents each word as a short list of real numbers, maybe 100 values. This compact form is a dense vector.

Instead of 50,000 mostly-zero slots, you get maybe 100 packed numbers. Each dimension quietly encodes some learned aspect of meaning. ✨

The magic is geometry: words with similar meaning land near each other in space. Closeness in this space now means closeness in meaning.

We compare two dense vectors with cosine similarity, which scores how aligned their directions are. Higher means more alike in meaning.

Nobody hand-writes these numbers. An algorithm reads huge amounts of text and learns each word vector from how words are actually used.

Dense vectors are smaller, smarter, and capture relationships sparse counts never could. This single shift powers most modern NLP.

What is the key advantage of dense word vectors over sparse counts?

Sparse counts are huge and treat words as unrelated. Dense embeddings fix this with compact vectors where similar words sit close. ✅