حقيبة الكلمات: CountVectorizer وTfidfVectorizer

Machine learning models require numerical inputs, but text is inherently unstructured. Converting text documents into a format a model can understand requires a systematic approach. The Bag of Words (BoW) model is the simplest and most widely used method: it treats a document as an unordered collection of words, ignoring grammar and word order, and counts how many times each word appears. The result is a numeric vector — one number per word in the vocabulary. Despite its simplicity, BoW is surprisingly effective for text classification, spam filtering, and sentiment analysis.

Scikit-learn's CountVectorizer automates the Bag of Words process. It: (1) tokenises each document by splitting on whitespace and punctuation, (2) builds a vocabulary from all unique tokens seen during fit(), and (3) converts each document to a sparse vector of word counts. The result is a document-term matrix where rows are documents and columns are vocabulary words. Sparse format is used because most words appear in only a small fraction of documents — most entries are zero.

CountVectorizer returns a sparse matrix because most word counts are zero — a document about Python will not mention 'elephant' or 'galaxy'. Storing all zeros would waste enormous memory. A sparse matrix stores only the non-zero values and their positions. For a vocabulary of 100,000 words and 10,000 documents, the full matrix would be 8 GB; the sparse version might be only 50 MB. Always keep text feature matrices in sparse format until you are certain your downstream operations support it — converting to dense can exhaust memory on large text corpora.

CountVectorizer offers built-in text preprocessing options. lowercase=True (default) ensures 'Python' and 'python' are the same token. stop_words='english' removes common words like 'the', 'is', 'a' that carry little meaning. ngram_range=(1,2) includes both single words (unigrams) and consecutive word pairs (bigrams), capturing phrases like 'not good' that a unigram model would misinterpret as 'not' and 'good' separately. Bigrams significantly improve classification quality for sentiment analysis where negation matters.

Raw word counts have a fundamental flaw: very common words dominate the vector even though they carry little discriminative information. The word 'the' might appear 50 times in a news article but tells you nothing about the article's topic. Conversely, a rare technical term like 'eigenvalue' that appears only twice strongly suggests the document is about mathematics. TF-IDF (Term Frequency-Inverse Document Frequency) addresses this by down-weighting words that appear in many documents and up-weighting words that appear in few documents but frequently in the current one.

TF-IDF adjusts word importance by two factors: Term Frequency (TF) — how often the word appears in this document (locally important), and Inverse Document Frequency (IDF) — the logarithm of the ratio of total documents to documents containing the word (globally rare = more distinctive). TF-IDF(w, d) = TF(w,d) * IDF(w) where IDF(w) = log((N+1)/(df+1)) + 1 (scikit-learn uses smoothed IDF). A word in only 1 of 1000 documents has very high IDF; a word in every document has IDF near 0.

TfidfVectorizer combines tokenisation, count vectorisation, and TF-IDF weighting into a single transformer. It applies L2 normalisation by default, so each document vector has unit length — making documents of different lengths comparable. The result is still a sparse matrix. TfidfVectorizer almost always outperforms CountVectorizer on text classification tasks, especially when the corpus contains documents of varying lengths or when common stopwords were not explicitly removed.

For large corpora with millions of documents, the vocabulary can grow to hundreds of thousands of terms. max_features=N keeps only the top N words by frequency, reducing memory and speeding up training. min_df ignores words appearing in fewer than this many documents (removes very rare terms that may be typos or unique identifiers). max_df ignores words appearing in more than this fraction of documents (removes de facto stopwords that appear everywhere). Typical production settings: max_features=50,000, min_df=5, max_df=0.95.

Use CountVectorizer when: (1) your model already handles frequency weighting internally (e.g., Multinomial Naive Bayes expects raw counts), or (2) document length is uniform and frequency differences are meaningful. Use TfidfVectorizer when: (1) documents vary in length, (2) common words are not filtered by stopwords, or (3) you use algorithms that assume normalised feature vectors (SVM, logistic regression). For Naive Bayes specifically, CountVectorizer with MultinomialNB often works best. For SVM or logistic regression, TfidfVectorizer is the standard choice.

After vectorisation, you can inspect the document-term matrix to understand what was learned. Converting to a Pandas DataFrame with feature names as column headers makes it easy to see how each word is represented. This debugging step is essential: it reveals whether tokenisation worked correctly, whether stopwords were removed, whether the vocabulary captured the expected terms, and whether the TF-IDF weights seem sensible. Always inspect a sample of the matrix before training your model to catch preprocessing bugs early.

Word-level tokenisation breaks when text contains typos, abbreviations, or morphological variants. Character-level n-grams treat overlapping sequences of characters as features instead of whole words. Setting analyzer='char_wb' in TfidfVectorizer with ngram_range=(3,5) generates 3-to-5-character substrings with word boundaries. This makes the model robust to misspellings and multilingual text. It is especially useful for language detection, authorship analysis, and social media text where spelling is inconsistent. Character n-grams are also the foundation of sub-word tokenisation used in transformer models like BERT.

In this lesson you learned: Bag of Words converts text to word count vectors using CountVectorizer, TF-IDF down-weights common words and up-weights distinctive rare words using TfidfVectorizer, and key preprocessing parameters including stop_words, ngram_range, max_features, and min_df. Next up we train a Multinomial Naive Bayes classifier on text data.