Bag of Words: CountVectorizer y TfidfVectorizer
Tokenice texto, cree un vocabulario, convierta documentos en vectores de recuento y aplique ponderación TF-IDF para reducir el peso de las palabras frecuentes.
Bag of Words: CountVectorizer y TfidfVectorizer es una lección gratuita de Machine Learning Academy en CoddyKit. Esta es la lección 2 de 4. Puedes leer la lección completa abajo gratuitamente — luego la practicas en el navegador con un editor de código integrado y un tutor de IA 24/7. Forma parte de la ruta de aprendizaje de Machine Learning Academy, y tu progreso se sincroniza en la web y la app de CoddyKit. El curso de Machine Learning Academy incluye 4 lecciones en total.
Partes de esta lección aún no han sido traducidas y se muestran en inglés.
The Text-to-Numbers Problem
Machine learning models require numerical inputs, but text is inherently unstructured. Converting text documents into a format a model can understand requires a systematic approach. The Bag of Words (BoW) model is the simplest and most widely used method: it treats a document as an unordered collection of words, ignoring grammar and word order, and counts how many times each word appears. The result is a numeric vector — one number per word in the vocabulary. Despite its simplicity, BoW is surprisingly effective for text classification, spam filtering, and sentiment analysis.
# Bag of Words: ignore order, just count words
doc1 = 'the cat sat on the mat'
doc2 = 'the cat ate the rat'
# Vocabulary: all unique words across documents
vocab = sorted(set(doc1.split() + doc2.split()))
print('Vocabulary:', vocab)
# Count vectors
vec1 = [doc1.split().count(w) for w in vocab]
vec2 = [doc2.split().count(w) for w in vocab]
print('doc1 vector:', vec1)
print('doc2 vector:', vec2)CountVectorizer: Building the Vocabulary
Scikit-learn's CountVectorizer automates the Bag of Words process. It: (1) tokenises each document by splitting on whitespace and punctuation, (2) builds a vocabulary from all unique tokens seen during fit(), and (3) converts each document to a sparse vector of word counts. The result is a document-term matrix where rows are documents and columns are vocabulary words. Sparse format is used because most words appear in only a small fraction of documents — most entries are zero.
from sklearn.feature_extraction.text import CountVectorizer
corpus = [
'I love Python and machine learning',
'Machine learning is awesome',
'Python is great for data science',
'I hate bugs in Python code'
]
vec = CountVectorizer()
X = vec.fit_transform(corpus) # Returns sparse matrix
print('Vocabulary size:', len(vec.vocabulary_))
print('Matrix shape:', X.shape) # (4 docs, N vocabulary words)
print('Vocabulary:', sorted(vec.vocabulary_.keys()))Sparse Matrices and Memory Efficiency
CountVectorizer returns a sparse matrix because most word counts are zero — a document about Python will not mention 'elephant' or 'galaxy'. Storing all zeros would waste enormous memory. A sparse matrix stores only the non-zero values and their positions. For a vocabulary of 100,000 words and 10,000 documents, the full matrix would be 8 GB; the sparse version might be only 50 MB. Always keep text feature matrices in sparse format until you are certain your downstream operations support it — converting to dense can exhaust memory on large text corpora.
from sklearn.feature_extraction.text import CountVectorizer
from scipy.sparse import issparse
import numpy as np
corpus = ['Python is great', 'Machine learning rocks', 'Data science rules']
vec = CountVectorizer()
X_sparse = vec.fit_transform(corpus)
print('Is sparse:', issparse(X_sparse)) # True
print('Shape:', X_sparse.shape)
print('Non-zero entries:', X_sparse.nnz) # Only non-zero values stored
print('Density:', X_sparse.nnz / np.prod(X_sparse.shape)) # Very low
# Dense (RAM-expensive for large corpora)
X_dense = X_sparse.toarray()
print('Dense shape:', X_dense.shape)Preprocessing: Stopwords, Lowercase, and n-grams
CountVectorizer offers built-in text preprocessing options. lowercase=True (default) ensures 'Python' and 'python' are the same token. stop_words='english' removes common words like 'the', 'is', 'a' that carry little meaning. ngram_range=(1,2) includes both single words (unigrams) and consecutive word pairs (bigrams), capturing phrases like 'not good' that a unigram model would misinterpret as 'not' and 'good' separately. Bigrams significantly improve classification quality for sentiment analysis where negation matters.
from sklearn.feature_extraction.text import CountVectorizer
corpus = ['not good at all', 'very good movie', 'bad experience']
# Unigrams only (default)
uni_vec = CountVectorizer(stop_words='english')
print('Unigram features:', uni_vec.fit(corpus).get_feature_names_out())
# Bigrams too
bigram_vec = CountVectorizer(ngram_range=(1,2), stop_words='english')
bigram_features = bigram_vec.fit(corpus).get_feature_names_out()
print('Unigram+Bigram features:', bigram_features)
# 'not good' appears as a bigram -- captures negationThe Problem with Raw Counts: Common Words
Raw word counts have a fundamental flaw: very common words dominate the vector even though they carry little discriminative information. The word 'the' might appear 50 times in a news article but tells you nothing about the article's topic. Conversely, a rare technical term like 'eigenvalue' that appears only twice strongly suggests the document is about mathematics. TF-IDF (Term Frequency-Inverse Document Frequency) addresses this by down-weighting words that appear in many documents and up-weighting words that appear in few documents but frequently in the current one.
from sklearn.feature_extraction.text import CountVectorizer
import numpy as np
corpus = [
'the cat sat on the mat the cat is fat',
'the eigenvalue decomposition is powerful math'
]
vec = CountVectorizer()
X = vec.fit_transform(corpus).toarray()
features = vec.get_feature_names_out()
for i, doc in enumerate(corpus[:1]):
counts = sorted(zip(features, X[i]), key=lambda x: -x[1])
print('Top words by count in doc 1:')
for word, count in counts[:5]:
print(f' {word}: {count}')
# 'the' dominates even though it is meaninglessTF-IDF: Term Frequency Inverse Document Frequency
TF-IDF adjusts word importance by two factors: Term Frequency (TF) — how often the word appears in this document (locally important), and Inverse Document Frequency (IDF) — the logarithm of the ratio of total documents to documents containing the word (globally rare = more distinctive). TF-IDF(w, d) = TF(w,d) * IDF(w) where IDF(w) = log((N+1)/(df+1)) + 1 (scikit-learn uses smoothed IDF). A word in only 1 of 1000 documents has very high IDF; a word in every document has IDF near 0.
import numpy as np
# Manual TF-IDF
documents = [
'cat sat mat cat', # 'cat' appears twice here
'dog ran park',
'cat dog park'
]
N = len(documents) # 3 documents
# IDF for 'cat': appears in documents 0 and 2 (df=2)
df_cat = 2
idf_cat = np.log((N + 1) / (df_cat + 1)) + 1
# TF for 'cat' in doc0: 2 out of 4 words
tf_cat_doc0 = 2 / 4
tfidf_cat_doc0 = tf_cat_doc0 * idf_cat
print(f'TF-IDF(cat, doc0) = {tfidf_cat_doc0:.4f}')TfidfVectorizer: One-Step Text Transformation
TfidfVectorizer combines tokenisation, count vectorisation, and TF-IDF weighting into a single transformer. It applies L2 normalisation by default, so each document vector has unit length — making documents of different lengths comparable. The result is still a sparse matrix. TfidfVectorizer almost always outperforms CountVectorizer on text classification tasks, especially when the corpus contains documents of varying lengths or when common stopwords were not explicitly removed.
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
'I love Python and machine learning',
'Machine learning is awesome',
'Python is great for data science',
'I hate bugs in Python code'
]
tfidf = TfidfVectorizer(stop_words='english', max_features=20)
X = tfidf.fit_transform(corpus)
print('Shape:', X.shape)
print('Features:', tfidf.get_feature_names_out())
# Inspect TF-IDF weights for document 0
import numpy as np
weights = zip(tfidf.get_feature_names_out(), X.toarray()[0])
for word, weight in sorted(weights, key=lambda x: -x[1])[:5]:
print(f' {word}: {weight:.4f}')max_features and vocabulary Parameters
For large corpora with millions of documents, the vocabulary can grow to hundreds of thousands of terms. max_features=N keeps only the top N words by frequency, reducing memory and speeding up training. min_df ignores words appearing in fewer than this many documents (removes very rare terms that may be typos or unique identifiers). max_df ignores words appearing in more than this fraction of documents (removes de facto stopwords that appear everywhere). Typical production settings: max_features=50,000, min_df=5, max_df=0.95.
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = ['...'] # assume large corpus
tfidf = TfidfVectorizer(
max_features=50000, # Top 50k words by frequency
min_df=5, # Ignore words in fewer than 5 documents
max_df=0.95, # Ignore words in more than 95% of docs
ngram_range=(1, 2), # Include bigrams
stop_words='english',
sublinear_tf=True # Replace TF with 1+log(TF) to dampen outliers
)
print('CountVectorizer vs TfidfVectorizer settings configured')
print('sublinear_tf=True: dampens high-frequency words further')CountVectorizer vs TfidfVectorizer: When to Use Each
Use CountVectorizer when: (1) your model already handles frequency weighting internally (e.g., Multinomial Naive Bayes expects raw counts), or (2) document length is uniform and frequency differences are meaningful. Use TfidfVectorizer when: (1) documents vary in length, (2) common words are not filtered by stopwords, or (3) you use algorithms that assume normalised feature vectors (SVM, logistic regression). For Naive Bayes specifically, CountVectorizer with MultinomialNB often works best. For SVM or logistic regression, TfidfVectorizer is the standard choice.
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
corpus = ['spam message buy now', 'hello friend how are you',
'click here to buy', 'good morning everyone']
labels = [1, 0, 1, 0] # 1=spam, 0=ham
# Naive Bayes: works well with CountVectorizer
nb_pipe = Pipeline([('vec', CountVectorizer()), ('clf', MultinomialNB())])
# Logistic regression: works better with TF-IDF
lr_pipe = Pipeline([('vec', TfidfVectorizer()), ('clf', LogisticRegression())])
print('CountVectorizer + MultinomialNB: standard for text Naive Bayes')
print('TfidfVectorizer + LogisticRegression: standard for text linear models')Inspecting the Document-Term Matrix
After vectorisation, you can inspect the document-term matrix to understand what was learned. Converting to a Pandas DataFrame with feature names as column headers makes it easy to see how each word is represented. This debugging step is essential: it reveals whether tokenisation worked correctly, whether stopwords were removed, whether the vocabulary captured the expected terms, and whether the TF-IDF weights seem sensible. Always inspect a sample of the matrix before training your model to catch preprocessing bugs early.
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
corpus = [
'Python machine learning tutorial',
'Deep learning neural networks',
'Python data analysis pandas numpy'
]
tfidf = TfidfVectorizer()
X = tfidf.fit_transform(corpus)
# Convert to DataFrame for inspection
df = pd.DataFrame(
X.toarray(),
columns=tfidf.get_feature_names_out()
)
print('Document-Term Matrix (TF-IDF weights):')
print(df.to_string())Character-Level N-grams for Robust Tokenisation
Word-level tokenisation breaks when text contains typos, abbreviations, or morphological variants. Character-level n-grams treat overlapping sequences of characters as features instead of whole words. Setting analyzer='char_wb' in TfidfVectorizer with ngram_range=(3,5) generates 3-to-5-character substrings with word boundaries. This makes the model robust to misspellings and multilingual text. It is especially useful for language detection, authorship analysis, and social media text where spelling is inconsistent. Character n-grams are also the foundation of sub-word tokenisation used in transformer models like BERT.
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
'python programming language',
'pythn programmng (typo)', # Misspellings
'java programming'
]
# Word-level (default): typos create unseen tokens
word_vec = TfidfVectorizer(analyzer='word')
word_features = word_vec.fit(corpus).get_feature_names_out()
print('Word features:', word_features)
# Character n-gram: handles typos gracefully
char_vec = TfidfVectorizer(analyzer='char_wb', ngram_range=(3, 4))
X_char = char_vec.fit_transform(corpus)
print('Char n-gram features:', len(char_vec.get_feature_names_out()), 'total')
print('Typos share substrings with correct words -> similar vectors')Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: Bag of Words converts text to word count vectors using CountVectorizer, TF-IDF down-weights common words and up-weights distinctive rare words using TfidfVectorizer, and key preprocessing parameters including stop_words, ngram_range, max_features, and min_df. Next up we train a Multinomial Naive Bayes classifier on text data.
Preguntas frecuentes
¿La lección «Bag of Words: CountVectorizer y TfidfVectorizer» es gratis?
Sí — el texto completo de «Bag of Words: CountVectorizer y TfidfVectorizer» es gratis para leer aquí en la web. Para practicarla de forma interactiva (editor de código integrado y tutor de IA 24/7) y desbloquear el resto del curso de Machine Learning Academy, actualiza a CoddyKit PRO. El curso de Machine Learning Academy incluye 4 lecciones en total.
¿Qué aprenderé en «Bag of Words: CountVectorizer y TfidfVectorizer»?
Tokenice texto, cree un vocabulario, convierta documentos en vectores de recuento y aplique ponderación TF-IDF para reducir el peso de las palabras frecuentes. Practicas Machine Learning Academy con código real que ejecutas directamente en el navegador, y un tutor de IA 24/7 responde tus preguntas mientras trabajas en la lección.
¿Necesito experiencia previa para empezar Machine Learning Academy?
No se requiere experiencia previa. Machine Learning Academy en CoddyKit está estructurado para principiantes hasta estudiantes avanzados, así que puedes empezar aquí o desde el inicio y avanzar a tu ritmo. Esta es la lección 2 de 4.
¿Cuánto tiempo toma la lección «Bag of Words: CountVectorizer y TfidfVectorizer»?
La mayoría de las lecciones de CoddyKit toman alrededor de 5–10 minutos. Cada una es compacta e interactiva, así que avanzas constantemente y retomas exactamente por donde dejaste en la web y la app.
¿Puedo escribir y ejecutar código en esta lección de Machine Learning Academy?
Sí. Cada lección de Machine Learning Academy incluye un editor de código integrado, así que escribes y ejecutas código real directamente en tu navegador y obtienes retroalimentación instantánea de IA — sin configuración local necesaria.
Todas las lecciones de este curso
- El teorema de Bayes en lenguaje sencillo
- Bag of Words: CountVectorizer y TfidfVectorizer
- Entrenamiento de un clasificador Naive Bayes multinomial
- Suavizado de Laplace y el problema de la probabilidad cero