Machine Learning Academy · 课时

词袋模型：CountVectorizer 与 TfidfVectorizer

您将对文本进行分词，构建词汇表，将文档转换为计数向量，并应用 TF-IDF 加权来降低常见词的权重

词袋模型：CountVectorizer 与 TfidfVectorizer 是 CoddyKit 上的免费 Machine Learning Academy 课时。这是第 2 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Machine Learning Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Machine Learning Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

The Text-to-Numbers Problem

Machine learning models require numerical inputs, but text is inherently unstructured. Converting text documents into a format a model can understand requires a systematic approach. The Bag of Words (BoW) model is the simplest and most widely used method: it treats a document as an unordered collection of words, ignoring grammar and word order, and counts how many times each word appears. The result is a numeric vector — one number per word in the vocabulary. Despite its simplicity, BoW is surprisingly effective for text classification, spam filtering, and sentiment analysis.

# Bag of Words: ignore order, just count words
doc1 = 'the cat sat on the mat'
doc2 = 'the cat ate the rat'

# Vocabulary: all unique words across documents
vocab = sorted(set(doc1.split() + doc2.split()))
print('Vocabulary:', vocab)

# Count vectors
vec1 = [doc1.split().count(w) for w in vocab]
vec2 = [doc2.split().count(w) for w in vocab]
print('doc1 vector:', vec1)
print('doc2 vector:', vec2)

CountVectorizer: Building the Vocabulary

Scikit-learn's CountVectorizer automates the Bag of Words process. It: (1) tokenises each document by splitting on whitespace and punctuation, (2) builds a vocabulary from all unique tokens seen during fit(), and (3) converts each document to a sparse vector of word counts. The result is a document-term matrix where rows are documents and columns are vocabulary words. Sparse format is used because most words appear in only a small fraction of documents — most entries are zero.

from sklearn.feature_extraction.text import CountVectorizer

corpus = [
    'I love Python and machine learning',
    'Machine learning is awesome',
    'Python is great for data science',
    'I hate bugs in Python code'
]

vec = CountVectorizer()
X = vec.fit_transform(corpus)  # Returns sparse matrix

print('Vocabulary size:', len(vec.vocabulary_))
print('Matrix shape:', X.shape)  # (4 docs, N vocabulary words)
print('Vocabulary:', sorted(vec.vocabulary_.keys()))

Sparse Matrices and Memory Efficiency

CountVectorizer returns a sparse matrix because most word counts are zero — a document about Python will not mention 'elephant' or 'galaxy'. Storing all zeros would waste enormous memory. A sparse matrix stores only the non-zero values and their positions. For a vocabulary of 100,000 words and 10,000 documents, the full matrix would be 8 GB; the sparse version might be only 50 MB. Always keep text feature matrices in sparse format until you are certain your downstream operations support it — converting to dense can exhaust memory on large text corpora.

from sklearn.feature_extraction.text import CountVectorizer
from scipy.sparse import issparse
import numpy as np

corpus = ['Python is great', 'Machine learning rocks', 'Data science rules']
vec = CountVectorizer()
X_sparse = vec.fit_transform(corpus)

print('Is sparse:', issparse(X_sparse))        # True
print('Shape:', X_sparse.shape)
print('Non-zero entries:', X_sparse.nnz)       # Only non-zero values stored
print('Density:', X_sparse.nnz / np.prod(X_sparse.shape))  # Very low

# Dense (RAM-expensive for large corpora)
X_dense = X_sparse.toarray()
print('Dense shape:', X_dense.shape)

Preprocessing: Stopwords, Lowercase, and n-grams

CountVectorizer offers built-in text preprocessing options. lowercase=True (default) ensures 'Python' and 'python' are the same token. stop_words='english' removes common words like 'the', 'is', 'a' that carry little meaning. ngram_range=(1,2) includes both single words (unigrams) and consecutive word pairs (bigrams), capturing phrases like 'not good' that a unigram model would misinterpret as 'not' and 'good' separately. Bigrams significantly improve classification quality for sentiment analysis where negation matters.

from sklearn.feature_extraction.text import CountVectorizer

corpus = ['not good at all', 'very good movie', 'bad experience']

# Unigrams only (default)
uni_vec = CountVectorizer(stop_words='english')
print('Unigram features:', uni_vec.fit(corpus).get_feature_names_out())

# Bigrams too
bigram_vec = CountVectorizer(ngram_range=(1,2), stop_words='english')
bigram_features = bigram_vec.fit(corpus).get_feature_names_out()
print('Unigram+Bigram features:', bigram_features)
# 'not good' appears as a bigram -- captures negation

The Problem with Raw Counts: Common Words

Raw word counts have a fundamental flaw: very common words dominate the vector even though they carry little discriminative information. The word 'the' might appear 50 times in a news article but tells you nothing about the article's topic. Conversely, a rare technical term like 'eigenvalue' that appears only twice strongly suggests the document is about mathematics. TF-IDF (Term Frequency-Inverse Document Frequency) addresses this by down-weighting words that appear in many documents and up-weighting words that appear in few documents but frequently in the current one.

from sklearn.feature_extraction.text import CountVectorizer
import numpy as np

corpus = [
    'the cat sat on the mat the cat is fat',
    'the eigenvalue decomposition is powerful math'
]

vec = CountVectorizer()
X = vec.fit_transform(corpus).toarray()
features = vec.get_feature_names_out()

for i, doc in enumerate(corpus[:1]):
    counts = sorted(zip(features, X[i]), key=lambda x: -x[1])
    print('Top words by count in doc 1:')
    for word, count in counts[:5]:
        print(f'  {word}: {count}')
# 'the' dominates even though it is meaningless

TF-IDF: Term Frequency Inverse Document Frequency

TF-IDF adjusts word importance by two factors: Term Frequency (TF) — how often the word appears in this document (locally important), and Inverse Document Frequency (IDF) — the logarithm of the ratio of total documents to documents containing the word (globally rare = more distinctive). TF-IDF(w, d) = TF(w,d) * IDF(w) where IDF(w) = log((N+1)/(df+1)) + 1 (scikit-learn uses smoothed IDF). A word in only 1 of 1000 documents has very high IDF; a word in every document has IDF near 0.

import numpy as np

# Manual TF-IDF
documents = [
    'cat sat mat cat',   # 'cat' appears twice here
    'dog ran park',
    'cat dog park'
]

N = len(documents)  # 3 documents

# IDF for 'cat': appears in documents 0 and 2 (df=2)
df_cat = 2
idf_cat = np.log((N + 1) / (df_cat + 1)) + 1

# TF for 'cat' in doc0: 2 out of 4 words
tf_cat_doc0 = 2 / 4

tfidf_cat_doc0 = tf_cat_doc0 * idf_cat
print(f'TF-IDF(cat, doc0) = {tfidf_cat_doc0:.4f}')

TfidfVectorizer: One-Step Text Transformation

TfidfVectorizer combines tokenisation, count vectorisation, and TF-IDF weighting into a single transformer. It applies L2 normalisation by default, so each document vector has unit length — making documents of different lengths comparable. The result is still a sparse matrix. TfidfVectorizer almost always outperforms CountVectorizer on text classification tasks, especially when the corpus contains documents of varying lengths or when common stopwords were not explicitly removed.

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    'I love Python and machine learning',
    'Machine learning is awesome',
    'Python is great for data science',
    'I hate bugs in Python code'
]

tfidf = TfidfVectorizer(stop_words='english', max_features=20)
X = tfidf.fit_transform(corpus)

print('Shape:', X.shape)
print('Features:', tfidf.get_feature_names_out())

# Inspect TF-IDF weights for document 0
import numpy as np
weights = zip(tfidf.get_feature_names_out(), X.toarray()[0])
for word, weight in sorted(weights, key=lambda x: -x[1])[:5]:
    print(f'  {word}: {weight:.4f}')

max_features and vocabulary Parameters

For large corpora with millions of documents, the vocabulary can grow to hundreds of thousands of terms. max_features=N keeps only the top N words by frequency, reducing memory and speeding up training. min_df ignores words appearing in fewer than this many documents (removes very rare terms that may be typos or unique identifiers). max_df ignores words appearing in more than this fraction of documents (removes de facto stopwords that appear everywhere). Typical production settings: max_features=50,000, min_df=5, max_df=0.95.

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = ['...']  # assume large corpus

tfidf = TfidfVectorizer(
    max_features=50000,   # Top 50k words by frequency
    min_df=5,            # Ignore words in fewer than 5 documents
    max_df=0.95,         # Ignore words in more than 95% of docs
    ngram_range=(1, 2),  # Include bigrams
    stop_words='english',
    sublinear_tf=True    # Replace TF with 1+log(TF) to dampen outliers
)

print('CountVectorizer vs TfidfVectorizer settings configured')
print('sublinear_tf=True: dampens high-frequency words further')

CountVectorizer vs TfidfVectorizer: When to Use Each

Use CountVectorizer when: (1) your model already handles frequency weighting internally (e.g., Multinomial Naive Bayes expects raw counts), or (2) document length is uniform and frequency differences are meaningful. Use TfidfVectorizer when: (1) documents vary in length, (2) common words are not filtered by stopwords, or (3) you use algorithms that assume normalised feature vectors (SVM, logistic regression). For Naive Bayes specifically, CountVectorizer with MultinomialNB often works best. For SVM or logistic regression, TfidfVectorizer is the standard choice.

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

corpus = ['spam message buy now', 'hello friend how are you',
          'click here to buy', 'good morning everyone']
labels = [1, 0, 1, 0]  # 1=spam, 0=ham

# Naive Bayes: works well with CountVectorizer
nb_pipe = Pipeline([('vec', CountVectorizer()), ('clf', MultinomialNB())])

# Logistic regression: works better with TF-IDF
lr_pipe = Pipeline([('vec', TfidfVectorizer()), ('clf', LogisticRegression())])

print('CountVectorizer + MultinomialNB: standard for text Naive Bayes')
print('TfidfVectorizer + LogisticRegression: standard for text linear models')

Inspecting the Document-Term Matrix

After vectorisation, you can inspect the document-term matrix to understand what was learned. Converting to a Pandas DataFrame with feature names as column headers makes it easy to see how each word is represented. This debugging step is essential: it reveals whether tokenisation worked correctly, whether stopwords were removed, whether the vocabulary captured the expected terms, and whether the TF-IDF weights seem sensible. Always inspect a sample of the matrix before training your model to catch preprocessing bugs early.

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd

corpus = [
    'Python machine learning tutorial',
    'Deep learning neural networks',
    'Python data analysis pandas numpy'
]

tfidf = TfidfVectorizer()
X = tfidf.fit_transform(corpus)

# Convert to DataFrame for inspection
df = pd.DataFrame(
    X.toarray(),
    columns=tfidf.get_feature_names_out()
)

print('Document-Term Matrix (TF-IDF weights):')
print(df.to_string())

Character-Level N-grams for Robust Tokenisation

Word-level tokenisation breaks when text contains typos, abbreviations, or morphological variants. Character-level n-grams treat overlapping sequences of characters as features instead of whole words. Setting analyzer='char_wb' in TfidfVectorizer with ngram_range=(3,5) generates 3-to-5-character substrings with word boundaries. This makes the model robust to misspellings and multilingual text. It is especially useful for language detection, authorship analysis, and social media text where spelling is inconsistent. Character n-grams are also the foundation of sub-word tokenisation used in transformer models like BERT.

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    'python programming language',
    'pythn programmng (typo)',  # Misspellings
    'java programming'
]

# Word-level (default): typos create unseen tokens
word_vec = TfidfVectorizer(analyzer='word')
word_features = word_vec.fit(corpus).get_feature_names_out()
print('Word features:', word_features)

# Character n-gram: handles typos gracefully
char_vec = TfidfVectorizer(analyzer='char_wb', ngram_range=(3, 4))
X_char = char_vec.fit_transform(corpus)
print('Char n-gram features:', len(char_vec.get_feature_names_out()), 'total')
print('Typos share substrings with correct words -> similar vectors')

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: Bag of Words converts text to word count vectors using CountVectorizer, TF-IDF down-weights common words and up-weights distinctive rare words using TfidfVectorizer, and key preprocessing parameters including stop_words, ngram_range, max_features, and min_df. Next up we train a Multinomial Naive Bayes classifier on text data.

常见问题解答

「词袋模型：CountVectorizer 与 TfidfVectorizer」课时是免费的吗？

是的 — 「词袋模型：CountVectorizer 与 TfidfVectorizer」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Machine Learning Academy 课程的其余内容，请升级到 CoddyKit PRO。 Machine Learning Academy 课程共包含 4 节课。

「词袋模型：CountVectorizer 与 TfidfVectorizer」这节课中我会学到什么？

您将对文本进行分词，构建词汇表，将文档转换为计数向量，并应用 TF-IDF 加权来降低常见词的权重你通过在浏览器中直接运行的动手代码来练习 Machine Learning Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Machine Learning Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Machine Learning Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 2 节课，共 4 节。

「词袋模型：CountVectorizer 与 TfidfVectorizer」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Machine Learning Academy 课中编写并运行代码吗？

能。每节 Machine Learning Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Machine Learning Academy