Bag of Words: CountVectorizer dan TfidfVectorizer
Peserta didik akan melakukan tokenisasi teks, membangun kosakata, mengubah dokumen menjadi vektor hitungan, dan menerapkan pembobotan TF-IDF untuk mengurangi bobot kata yang umum.
Bag of Words: CountVectorizer dan TfidfVectorizer adalah pelajaran Machine Learning Academy gratis di CoddyKit. Ini adalah pelajaran 2 dari 4. Kamu bisa membaca pelajaran lengkapnya di bawah secara gratis — lalu praktikkan langsung di browser dengan editor kode bawaan dan tutor AI 24/7. Ini adalah bagian dari jalur belajar Machine Learning Academy, dan progresmu tersinkronisasi di web dan aplikasi CoddyKit. Kursus Machine Learning Academy mencakup 4 pelajaran total.
Bagian dari pelajaran ini belum diterjemahkan dan ditampilkan dalam bahasa Inggris.
The Text-to-Numbers Problem
Machine learning models require numerical inputs, but text is inherently unstructured. Converting text documents into a format a model can understand requires a systematic approach. The Bag of Words (BoW) model is the simplest and most widely used method: it treats a document as an unordered collection of words, ignoring grammar and word order, and counts how many times each word appears. The result is a numeric vector — one number per word in the vocabulary. Despite its simplicity, BoW is surprisingly effective for text classification, spam filtering, and sentiment analysis.
# Bag of Words: ignore order, just count words
doc1 = 'the cat sat on the mat'
doc2 = 'the cat ate the rat'
# Vocabulary: all unique words across documents
vocab = sorted(set(doc1.split() + doc2.split()))
print('Vocabulary:', vocab)
# Count vectors
vec1 = [doc1.split().count(w) for w in vocab]
vec2 = [doc2.split().count(w) for w in vocab]
print('doc1 vector:', vec1)
print('doc2 vector:', vec2)CountVectorizer: Building the Vocabulary
Scikit-learn's CountVectorizer automates the Bag of Words process. It: (1) tokenises each document by splitting on whitespace and punctuation, (2) builds a vocabulary from all unique tokens seen during fit(), and (3) converts each document to a sparse vector of word counts. The result is a document-term matrix where rows are documents and columns are vocabulary words. Sparse format is used because most words appear in only a small fraction of documents — most entries are zero.
from sklearn.feature_extraction.text import CountVectorizer
corpus = [
'I love Python and machine learning',
'Machine learning is awesome',
'Python is great for data science',
'I hate bugs in Python code'
]
vec = CountVectorizer()
X = vec.fit_transform(corpus) # Returns sparse matrix
print('Vocabulary size:', len(vec.vocabulary_))
print('Matrix shape:', X.shape) # (4 docs, N vocabulary words)
print('Vocabulary:', sorted(vec.vocabulary_.keys()))Sparse Matrices and Memory Efficiency
CountVectorizer returns a sparse matrix because most word counts are zero — a document about Python will not mention 'elephant' or 'galaxy'. Storing all zeros would waste enormous memory. A sparse matrix stores only the non-zero values and their positions. For a vocabulary of 100,000 words and 10,000 documents, the full matrix would be 8 GB; the sparse version might be only 50 MB. Always keep text feature matrices in sparse format until you are certain your downstream operations support it — converting to dense can exhaust memory on large text corpora.
from sklearn.feature_extraction.text import CountVectorizer
from scipy.sparse import issparse
import numpy as np
corpus = ['Python is great', 'Machine learning rocks', 'Data science rules']
vec = CountVectorizer()
X_sparse = vec.fit_transform(corpus)
print('Is sparse:', issparse(X_sparse)) # True
print('Shape:', X_sparse.shape)
print('Non-zero entries:', X_sparse.nnz) # Only non-zero values stored
print('Density:', X_sparse.nnz / np.prod(X_sparse.shape)) # Very low
# Dense (RAM-expensive for large corpora)
X_dense = X_sparse.toarray()
print('Dense shape:', X_dense.shape)Preprocessing: Stopwords, Lowercase, and n-grams
CountVectorizer offers built-in text preprocessing options. lowercase=True (default) ensures 'Python' and 'python' are the same token. stop_words='english' removes common words like 'the', 'is', 'a' that carry little meaning. ngram_range=(1,2) includes both single words (unigrams) and consecutive word pairs (bigrams), capturing phrases like 'not good' that a unigram model would misinterpret as 'not' and 'good' separately. Bigrams significantly improve classification quality for sentiment analysis where negation matters.
from sklearn.feature_extraction.text import CountVectorizer
corpus = ['not good at all', 'very good movie', 'bad experience']
# Unigrams only (default)
uni_vec = CountVectorizer(stop_words='english')
print('Unigram features:', uni_vec.fit(corpus).get_feature_names_out())
# Bigrams too
bigram_vec = CountVectorizer(ngram_range=(1,2), stop_words='english')
bigram_features = bigram_vec.fit(corpus).get_feature_names_out()
print('Unigram+Bigram features:', bigram_features)
# 'not good' appears as a bigram -- captures negationThe Problem with Raw Counts: Common Words
Raw word counts have a fundamental flaw: very common words dominate the vector even though they carry little discriminative information. The word 'the' might appear 50 times in a news article but tells you nothing about the article's topic. Conversely, a rare technical term like 'eigenvalue' that appears only twice strongly suggests the document is about mathematics. TF-IDF (Term Frequency-Inverse Document Frequency) addresses this by down-weighting words that appear in many documents and up-weighting words that appear in few documents but frequently in the current one.
from sklearn.feature_extraction.text import CountVectorizer
import numpy as np
corpus = [
'the cat sat on the mat the cat is fat',
'the eigenvalue decomposition is powerful math'
]
vec = CountVectorizer()
X = vec.fit_transform(corpus).toarray()
features = vec.get_feature_names_out()
for i, doc in enumerate(corpus[:1]):
counts = sorted(zip(features, X[i]), key=lambda x: -x[1])
print('Top words by count in doc 1:')
for word, count in counts[:5]:
print(f' {word}: {count}')
# 'the' dominates even though it is meaninglessTF-IDF: Term Frequency Inverse Document Frequency
TF-IDF adjusts word importance by two factors: Term Frequency (TF) — how often the word appears in this document (locally important), and Inverse Document Frequency (IDF) — the logarithm of the ratio of total documents to documents containing the word (globally rare = more distinctive). TF-IDF(w, d) = TF(w,d) * IDF(w) where IDF(w) = log((N+1)/(df+1)) + 1 (scikit-learn uses smoothed IDF). A word in only 1 of 1000 documents has very high IDF; a word in every document has IDF near 0.
import numpy as np
# Manual TF-IDF
documents = [
'cat sat mat cat', # 'cat' appears twice here
'dog ran park',
'cat dog park'
]
N = len(documents) # 3 documents
# IDF for 'cat': appears in documents 0 and 2 (df=2)
df_cat = 2
idf_cat = np.log((N + 1) / (df_cat + 1)) + 1
# TF for 'cat' in doc0: 2 out of 4 words
tf_cat_doc0 = 2 / 4
tfidf_cat_doc0 = tf_cat_doc0 * idf_cat
print(f'TF-IDF(cat, doc0) = {tfidf_cat_doc0:.4f}')TfidfVectorizer: One-Step Text Transformation
TfidfVectorizer combines tokenisation, count vectorisation, and TF-IDF weighting into a single transformer. It applies L2 normalisation by default, so each document vector has unit length — making documents of different lengths comparable. The result is still a sparse matrix. TfidfVectorizer almost always outperforms CountVectorizer on text classification tasks, especially when the corpus contains documents of varying lengths or when common stopwords were not explicitly removed.
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
'I love Python and machine learning',
'Machine learning is awesome',
'Python is great for data science',
'I hate bugs in Python code'
]
tfidf = TfidfVectorizer(stop_words='english', max_features=20)
X = tfidf.fit_transform(corpus)
print('Shape:', X.shape)
print('Features:', tfidf.get_feature_names_out())
# Inspect TF-IDF weights for document 0
import numpy as np
weights = zip(tfidf.get_feature_names_out(), X.toarray()[0])
for word, weight in sorted(weights, key=lambda x: -x[1])[:5]:
print(f' {word}: {weight:.4f}')max_features and vocabulary Parameters
For large corpora with millions of documents, the vocabulary can grow to hundreds of thousands of terms. max_features=N keeps only the top N words by frequency, reducing memory and speeding up training. min_df ignores words appearing in fewer than this many documents (removes very rare terms that may be typos or unique identifiers). max_df ignores words appearing in more than this fraction of documents (removes de facto stopwords that appear everywhere). Typical production settings: max_features=50,000, min_df=5, max_df=0.95.
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = ['...'] # assume large corpus
tfidf = TfidfVectorizer(
max_features=50000, # Top 50k words by frequency
min_df=5, # Ignore words in fewer than 5 documents
max_df=0.95, # Ignore words in more than 95% of docs
ngram_range=(1, 2), # Include bigrams
stop_words='english',
sublinear_tf=True # Replace TF with 1+log(TF) to dampen outliers
)
print('CountVectorizer vs TfidfVectorizer settings configured')
print('sublinear_tf=True: dampens high-frequency words further')CountVectorizer vs TfidfVectorizer: When to Use Each
Use CountVectorizer when: (1) your model already handles frequency weighting internally (e.g., Multinomial Naive Bayes expects raw counts), or (2) document length is uniform and frequency differences are meaningful. Use TfidfVectorizer when: (1) documents vary in length, (2) common words are not filtered by stopwords, or (3) you use algorithms that assume normalised feature vectors (SVM, logistic regression). For Naive Bayes specifically, CountVectorizer with MultinomialNB often works best. For SVM or logistic regression, TfidfVectorizer is the standard choice.
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
corpus = ['spam message buy now', 'hello friend how are you',
'click here to buy', 'good morning everyone']
labels = [1, 0, 1, 0] # 1=spam, 0=ham
# Naive Bayes: works well with CountVectorizer
nb_pipe = Pipeline([('vec', CountVectorizer()), ('clf', MultinomialNB())])
# Logistic regression: works better with TF-IDF
lr_pipe = Pipeline([('vec', TfidfVectorizer()), ('clf', LogisticRegression())])
print('CountVectorizer + MultinomialNB: standard for text Naive Bayes')
print('TfidfVectorizer + LogisticRegression: standard for text linear models')Inspecting the Document-Term Matrix
After vectorisation, you can inspect the document-term matrix to understand what was learned. Converting to a Pandas DataFrame with feature names as column headers makes it easy to see how each word is represented. This debugging step is essential: it reveals whether tokenisation worked correctly, whether stopwords were removed, whether the vocabulary captured the expected terms, and whether the TF-IDF weights seem sensible. Always inspect a sample of the matrix before training your model to catch preprocessing bugs early.
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
corpus = [
'Python machine learning tutorial',
'Deep learning neural networks',
'Python data analysis pandas numpy'
]
tfidf = TfidfVectorizer()
X = tfidf.fit_transform(corpus)
# Convert to DataFrame for inspection
df = pd.DataFrame(
X.toarray(),
columns=tfidf.get_feature_names_out()
)
print('Document-Term Matrix (TF-IDF weights):')
print(df.to_string())Character-Level N-grams for Robust Tokenisation
Word-level tokenisation breaks when text contains typos, abbreviations, or morphological variants. Character-level n-grams treat overlapping sequences of characters as features instead of whole words. Setting analyzer='char_wb' in TfidfVectorizer with ngram_range=(3,5) generates 3-to-5-character substrings with word boundaries. This makes the model robust to misspellings and multilingual text. It is especially useful for language detection, authorship analysis, and social media text where spelling is inconsistent. Character n-grams are also the foundation of sub-word tokenisation used in transformer models like BERT.
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
'python programming language',
'pythn programmng (typo)', # Misspellings
'java programming'
]
# Word-level (default): typos create unseen tokens
word_vec = TfidfVectorizer(analyzer='word')
word_features = word_vec.fit(corpus).get_feature_names_out()
print('Word features:', word_features)
# Character n-gram: handles typos gracefully
char_vec = TfidfVectorizer(analyzer='char_wb', ngram_range=(3, 4))
X_char = char_vec.fit_transform(corpus)
print('Char n-gram features:', len(char_vec.get_feature_names_out()), 'total')
print('Typos share substrings with correct words -> similar vectors')Quick Check
Test your understanding of Machine Learning with Python concepts from this lesson.
Lesson Recap
In this lesson you learned: Bag of Words converts text to word count vectors using CountVectorizer, TF-IDF down-weights common words and up-weights distinctive rare words using TfidfVectorizer, and key preprocessing parameters including stop_words, ngram_range, max_features, and min_df. Next up we train a Multinomial Naive Bayes classifier on text data.
Pertanyaan yang Sering Diajukan
Apakah pelajaran “Bag of Words: CountVectorizer dan TfidfVectorizer” gratis?
Ya — teks lengkap “Bag of Words: CountVectorizer dan TfidfVectorizer” gratis dibaca di sini di web. Untuk praktiknya secara interaktif (editor kode bawaan dan tutor AI 24/7) dan buka sisa kursus Machine Learning Academy, upgrade ke CoddyKit PRO. Kursus Machine Learning Academy mencakup 4 pelajaran total.
Apa yang akan aku pelajari di “Bag of Words: CountVectorizer dan TfidfVectorizer”?
Peserta didik akan melakukan tokenisasi teks, membangun kosakata, mengubah dokumen menjadi vektor hitungan, dan menerapkan pembobotan TF-IDF untuk mengurangi bobot kata yang umum. Kamu berlatih Machine Learning Academy dengan kode praktik yang langsung kamu jalankan di browser, dan tutor AI 24/7 menjawab pertanyaanmu saat kamu mengerjakan pelajaran ini.
Apakah aku perlu pengalaman untuk memulai Machine Learning Academy?
Tidak diperlukan pengalaman sebelumnya. Machine Learning Academy di CoddyKit dirancang untuk pemula hingga pelajar tingkat lanjut, jadi kamu bisa memulai di sini atau dari awal dan belajar sesuai kecepatan kamu sendiri. Ini adalah pelajaran 2 dari 4.
Berapa lama pelajaran “Bag of Words: CountVectorizer dan TfidfVectorizer” memakan waktu?
Sebagian besar pelajaran CoddyKit memakan waktu sekitar 5–10 menit. Setiap pelajaran ringkas dan interaktif, jadi kamu membuat kemajuan stabil dan melanjutkan dari tempat kamu tinggalkan di web dan aplikasi.
Bisakah aku menulis dan menjalankan kode dalam pelajaran Machine Learning Academy ini?
Ya. Setiap pelajaran Machine Learning Academy menyertakan editor kode bawaan, jadi kamu menulis dan menjalankan kode nyata langsung di browser dan mendapatkan umpan balik AI instan — tidak diperlukan penyiapan lokal.
Semua pelajaran dalam kursus ini
- Teorema Bayes dalam Bahasa Sederhana
- Bag of Words: CountVectorizer dan TfidfVectorizer
- Melatih Pengklasifikasi Multinomial Naive Bayes
- Penghalusan Laplace dan Masalah Probabilitas Nol