Machine Learning Academy · Pelajaran

Tokenizer Hugging Face: Mengodekan Teks untuk BERT

Peserta akan memuat BertTokenizer, melakukan tokenisasi pada sekumpulan kalimat, memeriksa tensor input_ids dan attention_mask, serta menangani pemotongan dan padding.

Tokenizer Hugging Face: Mengodekan Teks untuk BERT adalah pelajaran Machine Learning Academy gratis di CoddyKit. Ini adalah pelajaran 2 dari 4. Kamu bisa membaca pelajaran lengkapnya di bawah secara gratis — lalu praktikkan langsung di browser dengan editor kode bawaan dan tutor AI 24/7. Ini adalah bagian dari jalur belajar Machine Learning Academy, dan progresmu tersinkronisasi di web dan aplikasi CoddyKit. Kursus Machine Learning Academy mencakup 4 pelajaran total.

Bagian dari pelajaran ini belum diterjemahkan dan ditampilkan dalam bahasa Inggris.

Why Tokenisation Matters for BERT

Before BERT can process text, each character sequence must be converted into numerical IDs the model understands. Tokenisation is the process of splitting text into sub-word units (tokens) and mapping them to integer IDs from a fixed vocabulary. Getting tokenisation right is critical: the wrong padding strategy, missing attention masks, or incorrect truncation can silently corrupt your model's input and hurt accuracy.

Installing Hugging Face Transformers

The Hugging Face Transformers library provides pre-trained models and tokenizers for hundreds of architectures. Install it along with datasets for data loading and torch as the backend. The library follows a consistent API: instantiate a tokenizer with from_pretrained, pass it text, and receive ready-to-use tensors.

# Install dependencies
# pip install transformers datasets torch

from transformers import BertTokenizer
import torch

# Load the pre-trained BERT tokenizer (downloads ~200 KB vocab file)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
print('Vocabulary size:', tokenizer.vocab_size)  # 30522
print('Max length:', tokenizer.model_max_length)  # 512

Tokenizing a Single Sentence

Calling the tokenizer on a string returns a dictionary containing input_ids (integer token IDs), attention_mask (1 for real tokens), and optionally token_type_ids (segment IDs). The tokenizer automatically adds [CLS] at the start and [SEP] at the end. Setting return_tensors='pt' returns PyTorch tensors ready for the model.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = 'Machine learning is transforming the world.'
encoding = tokenizer(
    text,
    return_tensors='pt',
    truncation=True,
    max_length=128
)

print('input_ids:', encoding['input_ids'])
print('attention_mask:', encoding['attention_mask'])
print('Token count:', encoding['input_ids'].shape[1])

WordPiece Subword Tokenisation

BERT uses WordPiece tokenisation: rare or unknown words are split into frequent sub-word pieces from the vocabulary. A word like 'tokenisation' might become ['token', '##isation'] where ## marks a continuation subword. This handles out-of-vocabulary words without an unknown token fallback, preserving morphological information that whole-word tokenisation loses.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Inspect raw tokens before converting to IDs
word = 'tokenisation'
pieces = tokenizer.tokenize(word)
print('WordPiece pieces:', pieces)  # ['token', '##isation']

ids = tokenizer.convert_tokens_to_ids(pieces)
print('IDs:', ids)

# Decode back to text
decoded = tokenizer.decode(ids)
print('Decoded:', decoded)

Batch Tokenisation with Padding

Real training uses mini-batches of sentences of different lengths. The tokenizer's padding=True option pads shorter sequences with [PAD] tokens to match the longest in the batch. The attention_mask ensures the model ignores these padding positions during attention computation. Always pad to the batch maximum rather than the model maximum to avoid wasting computation.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

sentences = [
    'Short.',
    'This is a much longer sentence for demonstration purposes.'
]

encoded = tokenizer(
    sentences,
    padding=True,        # pad to longest in batch
    truncation=True,
    max_length=64,
    return_tensors='pt'
)
print('input_ids shape:', encoded['input_ids'].shape)
print('attention_mask:', encoded['attention_mask'])

Truncation: Handling Long Text

BERT accepts at most 512 tokens per input (including [CLS] and [SEP]). Longer documents must be truncated. Setting truncation=True with max_length=512 cuts at the token limit. For tasks like document classification, common strategies include using only the first 512 tokens, the last 512 tokens (often more informative), or splitting the document into overlapping windows and aggregating predictions.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

long_text = 'word ' * 600  # 600 words -- too long for BERT

# Strategy 1: truncate to first 512 tokens
encoded_first = tokenizer(long_text, truncation=True, max_length=512)
print('First 512 length:', len(encoded_first['input_ids']))

# Strategy 2: only take the last 510 tokens + [CLS] + [SEP]
tokens = tokenizer.tokenize(long_text)[-510:]
encoded_last = [tokenizer.cls_token] + tokens + [tokenizer.sep_token]
print('Last-window length:', len(encoded_last))

Sentence-Pair Inputs for BERT

Tasks like question answering and natural language inference require feeding two sentences to BERT simultaneously. The tokenizer accepts two arguments: tokenizer(sentence_a, sentence_b). It automatically builds the input as [CLS] A [SEP] B [SEP] and populates token_type_ids with 0 for sentence A tokens and 1 for sentence B tokens, letting BERT distinguish the two segments.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

question = 'Where was Marie Curie born?'
context = 'Marie Curie was born in Warsaw, Poland in 1867.'

encoded = tokenizer(
    question,
    context,
    truncation=True,
    max_length=128,
    return_tensors='pt'
)

print('token_type_ids:', encoded['token_type_ids'])
# 0 = question tokens, 1 = context tokens

AutoTokenizer: Provider-Agnostic Loading

Instead of importing model-specific classes, use AutoTokenizer from Hugging Face. It automatically selects the correct tokenizer class based on the model checkpoint name. This makes your code portable: swapping 'bert-base-uncased' for 'roberta-base' or 'distilbert-base-uncased' requires changing only one string, with all tokenizer logic handled automatically.

from transformers import AutoTokenizer

# Works for BERT, RoBERTa, DistilBERT, GPT-2, and more
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

# Exact same API regardless of model family
text = 'Transfer learning is powerful.'
encoded = tokenizer(text, return_tensors='pt')
print(encoded['input_ids'])

# Switching models is trivial:
# tokenizer = AutoTokenizer.from_pretrained('roberta-base')

Decoding: IDs Back to Text

You can convert token IDs back to human-readable text with tokenizer.decode(ids). This is useful for debugging tokenisation and for sequence-to-sequence tasks (translation, summarisation) where the model outputs token IDs that must be decoded to text. Use skip_special_tokens=True to strip [CLS], [SEP], and [PAD] from the output.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = 'Hello, world!'
ids = tokenizer.encode(text)
print('Encoded IDs:', ids)  # includes [CLS]=101 and [SEP]=102

decoded = tokenizer.decode(ids, skip_special_tokens=True)
print('Decoded text:', decoded)  # 'hello, world!'

decoded_with_special = tokenizer.decode(ids)
print('With special tokens:', decoded_with_special)  # '[CLS] hello, world! [SEP]'

Building a DataLoader for Fine-Tuning

During fine-tuning, sentences are tokenised in batches using a PyTorch DataLoader. A custom Dataset stores raw texts and labels; tokenisation happens in __getitem__ or via a collate function. Tokenising inside the DataLoader enables on-the-fly processing and avoids storing large pre-tokenised tensors in memory for large datasets.

import torch
from torch.utils.data import Dataset, DataLoader
from transformers import BertTokenizer

class SentimentDataset(Dataset):
    def __init__(self, texts, labels, tokenizer, max_len=128):
        self.texts = texts
        self.labels = labels
        self.tokenizer = tokenizer
        self.max_len = max_len

    def __len__(self):
        return len(self.texts)

    def __getitem__(self, idx):
        enc = self.tokenizer(
            self.texts[idx],
            truncation=True, padding='max_length',
            max_length=self.max_len, return_tensors='pt'
        )
        return {
            'input_ids': enc['input_ids'].squeeze(),
            'attention_mask': enc['attention_mask'].squeeze(),
            'label': torch.tensor(self.labels[idx], dtype=torch.long)
        }

Fast Tokenizers and Offset Mapping

Hugging Face provides fast tokenizers (backed by Rust) that are 10-100x faster than the pure Python versions. They also support offset mapping: for each token, you get the character start and end positions in the original string. This is essential for token-level tasks like named entity recognition and question answering where you need to map model outputs back to character spans in the source text.

from transformers import BertTokenizerFast

tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')

text = 'Paris is lovely.'
encoded = tokenizer(text, return_offsets_mapping=True)

for token_id, offset in zip(encoded['input_ids'], encoded['offset_mapping']):
    token = tokenizer.convert_ids_to_tokens([token_id])[0]
    print(f'{token:15s} -> chars {offset}')
# paris           -> chars (0, 5)
# is              -> chars (6, 8)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: BertTokenizer converts raw text into input_ids, attention_mask, and token_type_ids tensors, WordPiece subword splitting handles out-of-vocabulary words gracefully, and padding with attention masks and truncation to 512 tokens are required when batching variable-length inputs. Next up we fine-tune BertForSequenceClassification on a real sentiment dataset.

Pertanyaan yang Sering Diajukan

Apakah pelajaran “Tokenizer Hugging Face: Mengodekan Teks untuk BERT” gratis?

Ya — teks lengkap “Tokenizer Hugging Face: Mengodekan Teks untuk BERT” gratis dibaca di sini di web. Untuk praktiknya secara interaktif (editor kode bawaan dan tutor AI 24/7) dan buka sisa kursus Machine Learning Academy, upgrade ke CoddyKit PRO. Kursus Machine Learning Academy mencakup 4 pelajaran total.

Apa yang akan aku pelajari di “Tokenizer Hugging Face: Mengodekan Teks untuk BERT”?

Peserta akan memuat BertTokenizer, melakukan tokenisasi pada sekumpulan kalimat, memeriksa tensor input_ids dan attention_mask, serta menangani pemotongan dan padding. Kamu berlatih Machine Learning Academy dengan kode praktik yang langsung kamu jalankan di browser, dan tutor AI 24/7 menjawab pertanyaanmu saat kamu mengerjakan pelajaran ini.

Apakah aku perlu pengalaman untuk memulai Machine Learning Academy?

Tidak diperlukan pengalaman sebelumnya. Machine Learning Academy di CoddyKit dirancang untuk pemula hingga pelajar tingkat lanjut, jadi kamu bisa memulai di sini atau dari awal dan belajar sesuai kecepatan kamu sendiri. Ini adalah pelajaran 2 dari 4.

Berapa lama pelajaran “Tokenizer Hugging Face: Mengodekan Teks untuk BERT” memakan waktu?

Sebagian besar pelajaran CoddyKit memakan waktu sekitar 5–10 menit. Setiap pelajaran ringkas dan interaktif, jadi kamu membuat kemajuan stabil dan melanjutkan dari tempat kamu tinggalkan di web dan aplikasi.

Bisakah aku menulis dan menjalankan kode dalam pelajaran Machine Learning Academy ini?

Ya. Setiap pelajaran Machine Learning Academy menyertakan editor kode bawaan, jadi kamu menulis dan menjalankan kode nyata langsung di browser dan mendapatkan umpan balik AI instan — tidak diperlukan penyiapan lokal.

Semua pelajaran dalam kursus ini

Arsitektur Transformer: Attention, Token, dan Konteks
Tokenizer Hugging Face: Mengodekan Teks untuk BERT
Fine-Tuning BertForSequenceClassification
Evaluasi dan Inferensi: Dari Logit ke Label Prediksi

← Kembali ke Machine Learning Academy