Bag of Words：CountVectorizerとTfidfVectorizer
テキストをトークン化し、語彙を構築して文書をカウントベクトルに変換し、一般的な単語の重みを下げるTF-IDFを適用します。
「Bag of Words：CountVectorizerとTfidfVectorizer」はCoddyKit上の無料Machine Learning Academyレッスンです。 これはレッスン2/4です。 下記で完全なレッスンを無料で読むことができます。その後、ブラウザ内の組み込みコードエディタと24時間対応のAIチューターでハンズオン演習できます。 これはMachine Learning Academy学習パスの一部であり、ウェブとCoddyKitアプリ全体で進捗が同期されます。 Machine Learning Academyコースには全4レッスンが含まれています。
このレッスンの一部はまだ翻訳されておらず、英語で表示されています。
The Text-to-Numbers Problem
Machine learning models require numerical inputs, but text is inherently unstructured. Converting text documents into a format a model can understand requires a systematic approach. The Bag of Words (BoW) model is the simplest and most widely used method: it treats a document as an unordered collection of words, ignoring grammar and word order, and counts how many times each word appears. The result is a numeric vector — one number per word in the vocabulary. Despite its simplicity, BoW is surprisingly effective for text classification, spam filtering, and sentiment analysis.
# Bag of Words: ignore order, just count words
doc1 = 'the cat sat on the mat'
doc2 = 'the cat ate the rat'
# Vocabulary: all unique words across documents
vocab = sorted(set(doc1.split() + doc2.split()))
print('Vocabulary:', vocab)
# Count vectors
vec1 = [doc1.split().count(w) for w in vocab]
vec2 = [doc2.split().count(w) for w in vocab]
print('doc1 vector:', vec1)
print('doc2 vector:', vec2)CountVectorizer: Building the Vocabulary
Scikit-learn's CountVectorizer automates the Bag of Words process. It: (1) tokenises each document by splitting on whitespace and punctuation, (2) builds a vocabulary from all unique tokens seen during fit(), and (3) converts each document to a sparse vector of word counts. The result is a document-term matrix where rows are documents and columns are vocabulary words. Sparse format is used because most words appear in only a small fraction of documents — most entries are zero.
from sklearn.feature_extraction.text import CountVectorizer
corpus = [
'I love Python and machine learning',
'Machine learning is awesome',
'Python is great for data science',
'I hate bugs in Python code'
]
vec = CountVectorizer()
X = vec.fit_transform(corpus) # Returns sparse matrix
print('Vocabulary size:', len(vec.vocabulary_))
print('Matrix shape:', X.shape) # (4 docs, N vocabulary words)
print('Vocabulary:', sorted(vec.vocabulary_.keys()))Sparse Matrices and Memory Efficiency
CountVectorizer returns a sparse matrix because most word counts are zero — a document about Python will not mention 'elephant' or 'galaxy'. Storing all zeros would waste enormous memory. A sparse matrix stores only the non-zero values and their positions. For a vocabulary of 100,000 words and 10,000 documents, the full matrix would be 8 GB; the sparse version might be only 50 MB. Always keep text feature matrices in sparse format until you are certain your downstream operations support it — converting to dense can exhaust memory on large text corpora.
from sklearn.feature_extraction.text import CountVectorizer
from scipy.sparse import issparse
import numpy as np
corpus = ['Python is great', 'Machine learning rocks', 'Data science rules']
vec = CountVectorizer()
X_sparse = vec.fit_transform(corpus)
print('Is sparse:', issparse(X_sparse)) # True
print('Shape:', X_sparse.shape)
print('Non-zero entries:', X_sparse.nnz) # Only non-zero values stored
print('Density:', X_sparse.nnz / np.prod(X_sparse.shape)) # Very low
# Dense (RAM-expensive for large corpora)
X_dense = X_sparse.toarray()
print('Dense shape:', X_dense.shape)Preprocessing: Stopwords, Lowercase, and n-grams
CountVectorizer offers built-in text preprocessing options. lowercase=True (default) ensures 'Python' and 'python' are the same token. stop_words='english' removes common words like 'the', 'is', 'a' that carry little meaning. ngram_range=(1,2) includes both single words (unigrams) and consecutive word pairs (bigrams), capturing phrases like 'not good' that a unigram model would misinterpret as 'not' and 'good' separately. Bigrams significantly improve classification quality for sentiment analysis where negation matters.
from sklearn.feature_extraction.text import CountVectorizer
corpus = ['not good at all', 'very good movie', 'bad experience']
# Unigrams only (default)
uni_vec = CountVectorizer(stop_words='english')
print('Unigram features:', uni_vec.fit(corpus).get_feature_names_out())
# Bigrams too
bigram_vec = CountVectorizer(ngram_range=(1,2), stop_words='english')
bigram_features = bigram_vec.fit(corpus).get_feature_names_out()
print('Unigram+Bigram features:', bigram_features)
# 'not good' appears as a bigram -- captures negationThe Problem with Raw Counts: Common Words
Raw word counts have a fundamental flaw: very common words dominate the vector even though they carry little discriminative information. The word 'the' might appear 50 times in a news article but tells you nothing about the article's topic. Conversely, a rare technical term like 'eigenvalue' that appears only twice strongly suggests the document is about mathematics. TF-IDF (Term Frequency-Inverse Document Frequency) addresses this by down-weighting words that appear in many documents and up-weighting words that appear in few documents but frequently in the current one.
from sklearn.feature_extraction.text import CountVectorizer
import numpy as np
corpus = [
'the cat sat on the mat the cat is fat',
'the eigenvalue decomposition is powerful math'
]
vec = CountVectorizer()
X = vec.fit_transform(corpus).toarray()
features = vec.get_feature_names_out()
for i, doc in enumerate(corpus[:1]):
counts = sorted(zip(features, X[i]), key=lambda x: -x[1])
print('Top words by count in doc 1:')
for word, count in counts[:5]:
print(f' {word}: {count}')
# 'the' dominates even though it is meaninglessTF-IDF: Term Frequency Inverse Document Frequency
TF-IDF adjusts word importance by two factors: Term Frequency (TF) — how often the word appears in this document (locally important), and Inverse Document Frequency (IDF) — the logarithm of the ratio of total documents to documents containing the word (globally rare = more distinctive). TF-IDF(w, d) = TF(w,d) * IDF(w) where IDF(w) = log((N+1)/(df+1)) + 1 (scikit-learn uses smoothed IDF). A word in only 1 of 1000 documents has very high IDF; a word in every document has IDF near 0.
import numpy as np
# Manual TF-IDF
documents = [
'cat sat mat cat', # 'cat' appears twice here
'dog ran park',
'cat dog park'
]
N = len(documents) # 3 documents
# IDF for 'cat': appears in documents 0 and 2 (df=2)
df_cat = 2
idf_cat = np.log((N + 1) / (df_cat + 1)) + 1
# TF for 'cat' in doc0: 2 out of 4 words
tf_cat_doc0 = 2 / 4
tfidf_cat_doc0 = tf_cat_doc0 * idf_cat
print(f'TF-IDF(cat, doc0) = {tfidf_cat_doc0:.4f}')TfidfVectorizer: One-Step Text Transformation
TfidfVectorizer combines tokenisation, count vectorisation, and TF-IDF weighting into a single transformer. It applies L2 normalisation by default, so each document vector has unit length — making documents of different lengths comparable. The result is still a sparse matrix. TfidfVectorizer almost always outperforms CountVectorizer on text classification tasks, especially when the corpus contains documents of varying lengths or when common stopwords were not explicitly removed.
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
'I love Python and machine learning',
'Machine learning is awesome',
'Python is great for data science',
'I hate bugs in Python code'
]
tfidf = TfidfVectorizer(stop_words='english', max_features=20)
X = tfidf.fit_transform(corpus)
print('Shape:', X.shape)
print('Features:', tfidf.get_feature_names_out())
# Inspect TF-IDF weights for document 0
import numpy as np
weights = zip(tfidf.get_feature_names_out(), X.toarray()[0])
for word, weight in sorted(weights, key=lambda x: -x[1])[:5]:
print(f' {word}: {weight:.4f}')max_features and vocabulary Parameters
For large corpora with millions of documents, the vocabulary can grow to hundreds of thousands of terms. max_features=N keeps only the top N words by frequency, reducing memory and speeding up training. min_df ignores words appearing in fewer than this many documents (removes very rare terms that may be typos or unique identifiers). max_df ignores words appearing in more than this fraction of documents (removes de facto stopwords that appear everywhere). Typical production settings: max_features=50,000, min_df=5, max_df=0.95.
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = ['...'] # assume large corpus
tfidf = TfidfVectorizer(
max_features=50000, # Top 50k words by frequency
min_df=5, # Ignore words in fewer than 5 documents
max_df=0.95, # Ignore words in more than 95% of docs
ngram_range=(1, 2), # Include bigrams
stop_words='english',
sublinear_tf=True # Replace TF with 1+log(TF) to dampen outliers
)
print('CountVectorizer vs TfidfVectorizer settings configured')
print('sublinear_tf=True: dampens high-frequency words further')CountVectorizer vs TfidfVectorizer: When to Use Each
Use CountVectorizer when: (1) your model already handles frequency weighting internally (e.g., Multinomial Naive Bayes expects raw counts), or (2) document length is uniform and frequency differences are meaningful. Use TfidfVectorizer when: (1) documents vary in length, (2) common words are not filtered by stopwords, or (3) you use algorithms that assume normalised feature vectors (SVM, logistic regression). For Naive Bayes specifically, CountVectorizer with MultinomialNB often works best. For SVM or logistic regression, TfidfVectorizer is the standard choice.
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
corpus = ['spam message buy now', 'hello friend how are you',
'click here to buy', 'good morning everyone']
labels = [1, 0, 1, 0] # 1=spam, 0=ham
# Naive Bayes: works well with CountVectorizer
nb_pipe = Pipeline([('vec', CountVectorizer()), ('clf', MultinomialNB())])
# Logistic regression: works better with TF-IDF
lr_pipe = Pipeline([('vec', TfidfVectorizer()), ('clf', LogisticRegression())])
print('CountVectorizer + MultinomialNB: standard for text Naive Bayes')
print('TfidfVectorizer + LogisticRegression: standard for text linear models')Inspecting the Document-Term Matrix
After vectorisation, you can inspect the document-term matrix to understand what was learned. Converting to a Pandas DataFrame with feature names as column headers makes it easy to see how each word is represented. This debugging step is essential: it reveals whether tokenisation worked correctly, whether stopwords were removed, whether the vocabulary captured the expected terms, and whether the TF-IDF weights seem sensible. Always inspect a sample of the matrix before training your model to catch preprocessing bugs early.
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
corpus = [
'Python machine learning tutorial',
'Deep learning neural networks',
'Python data analysis pandas numpy'
]
tfidf = TfidfVectorizer()
X = tfidf.fit_transform(corpus)
# Convert to DataFrame for inspection
df = pd.DataFrame(
X.toarray(),
columns=tfidf.get_feature_names_out()
)
print('Document-Term Matrix (TF-IDF weights):')
print(df.to_string())Character-Level N-grams for Robust Tokenisation
Word-level tokenisation breaks when text contains typos, abbreviations, or morphological variants. Character-level n-grams treat overlapping sequences of characters as features instead of whole words. Setting analyzer='char_wb' in TfidfVectorizer with ngram_range=(3,5) generates 3-to-5-character substrings with word boundaries. This makes the model robust to misspellings and multilingual text. It is especially useful for language detection, authorship analysis, and social media text where spelling is inconsistent. Character n-grams are also the foundation of sub-word tokenisation used in transformer models like BERT.
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
'python programming language',
'pythn programmng (typo)', # Misspellings
'java programming'
]
# Word-level (default): typos create unseen tokens
word_vec = TfidfVectorizer(analyzer='word')
word_features = word_vec.fit(corpus).get_feature_names_out()
print('Word features:', word_features)
# Character n-gram: handles typos gracefully
char_vec = TfidfVectorizer(analyzer='char_wb', ngram_range=(3, 4))
X_char = char_vec.fit_transform(corpus)
print('Char n-gram features:', len(char_vec.get_feature_names_out()), 'total')
print('Typos share substrings with correct words -> similar vectors')Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: Bag of Words converts text to word count vectors using CountVectorizer, TF-IDF down-weights common words and up-weights distinctive rare words using TfidfVectorizer, and key preprocessing parameters including stop_words, ngram_range, max_features, and min_df. Next up we train a Multinomial Naive Bayes classifier on text data.
よくある質問
「Bag of Words：CountVectorizerとTfidfVectorizer」レッスンは無料ですか？
はい。「Bag of Words：CountVectorizerとTfidfVectorizer」の完全なテキストはこのウェブで無料で読めます。インタラクティブに演習し（組み込みコードエディタと24時間対応のAIチューター）、Machine Learning Academyコースの残りをアンロックするには、CoddyKit PROにアップグレードしてください。 Machine Learning Academyコースには全4レッスンが含まれています。
「Bag of Words：CountVectorizerとTfidfVectorizer」で何を学びますか？
テキストをトークン化し、語彙を構築して文書をカウントベクトルに変換し、一般的な単語の重みを下げるTF-IDFを適用します。 ブラウザで直接実行するハンズオンコードでMachine Learning Academyを演習し、24時間対応のAIチューターがレッスンを進める中での質問に答えます。
Machine Learning Academyを始めるのに経験は必要ですか？
事前経験は必要ありません。CoddyKitのMachine Learning Academyは初級者から上級者向けに構成されているため、ここから始めるか最初から始めて、自分のペースで進むことができます。 これはレッスン2/4です。
「Bag of Words：CountVectorizerとTfidfVectorizer」レッスンにはどのくらい時間がかかりますか？
ほとんどのCoddyKitレッスンは約5～10分かかります。各レッスンはコンパクトでインタラクティブなので、着実に進歩し、ウェブとアプリ全体で正確に前回の場所から再開できます。
このMachine Learning Academyレッスンでコードを書いて実行できますか？
はい。すべてのMachine Learning Academyレッスンに組み込みコードエディタが含まれているため、ブラウザでリアルコードを書いて実行し、即座のAIフィードバックを取得できます。ローカル設定は不要です。
このコースのすべてのレッスン
- 平易な言葉で学ぶベイズの定理
- Bag of Words：CountVectorizerとTfidfVectorizer
- 多項ナイーブベイズ分類器の訓練
- ラプラス平滑化とゼロ確率問題