0Pricing
Machine Learning Academy · Ders

Hugging Face Belirteçleyicileri: BERT için Metin Kodlama

BertTokenizer yükleyecek, bir cümleler yığınına belirteçlere ayırma uygulayacak, input_ids ve attention_mask tensörlerini inceleyecek, kesme ve doldurma işlemlerini yöneteceksiniz.

Hugging Face Belirteçleyicileri: BERT için Metin Kodlama, CoddyKit'te ücretsiz bir Machine Learning Academy dersidir. Bu, 4 dersinin 2. dersidir. Aşağıdan dersin tamamını ücretsiz okuyabilir, sonra tarayıcıda yerleşik kod editörü ve 7/24 yapay zeka koçu ile uygulamalı olarak pratik yapabilirsin. Bu, Machine Learning Academy öğrenme yolunun bir parçasıdır ve ilerlemeniz web ve CoddyKit uygulaması arasında senkronize olur. Machine Learning Academy kursu toplamda 4 dersten oluşur.

Bu dersin bazı bölümleri henüz çevrilmemiş olup İngilizce olarak gösterilmektedir.

Why Tokenisation Matters for BERT

Before BERT can process text, each character sequence must be converted into numerical IDs the model understands. Tokenisation is the process of splitting text into sub-word units (tokens) and mapping them to integer IDs from a fixed vocabulary. Getting tokenisation right is critical: the wrong padding strategy, missing attention masks, or incorrect truncation can silently corrupt your model's input and hurt accuracy.

Installing Hugging Face Transformers

The Hugging Face Transformers library provides pre-trained models and tokenizers for hundreds of architectures. Install it along with datasets for data loading and torch as the backend. The library follows a consistent API: instantiate a tokenizer with from_pretrained, pass it text, and receive ready-to-use tensors.

# Install dependencies
# pip install transformers datasets torch

from transformers import BertTokenizer
import torch

# Load the pre-trained BERT tokenizer (downloads ~200 KB vocab file)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
print('Vocabulary size:', tokenizer.vocab_size)  # 30522
print('Max length:', tokenizer.model_max_length)  # 512

Tokenizing a Single Sentence

Calling the tokenizer on a string returns a dictionary containing input_ids (integer token IDs), attention_mask (1 for real tokens), and optionally token_type_ids (segment IDs). The tokenizer automatically adds [CLS] at the start and [SEP] at the end. Setting return_tensors='pt' returns PyTorch tensors ready for the model.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = 'Machine learning is transforming the world.'
encoding = tokenizer(
    text,
    return_tensors='pt',
    truncation=True,
    max_length=128
)

print('input_ids:', encoding['input_ids'])
print('attention_mask:', encoding['attention_mask'])
print('Token count:', encoding['input_ids'].shape[1])

WordPiece Subword Tokenisation

BERT uses WordPiece tokenisation: rare or unknown words are split into frequent sub-word pieces from the vocabulary. A word like 'tokenisation' might become ['token', '##isation'] where ## marks a continuation subword. This handles out-of-vocabulary words without an unknown token fallback, preserving morphological information that whole-word tokenisation loses.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Inspect raw tokens before converting to IDs
word = 'tokenisation'
pieces = tokenizer.tokenize(word)
print('WordPiece pieces:', pieces)  # ['token', '##isation']

ids = tokenizer.convert_tokens_to_ids(pieces)
print('IDs:', ids)

# Decode back to text
decoded = tokenizer.decode(ids)
print('Decoded:', decoded)

Batch Tokenisation with Padding

Real training uses mini-batches of sentences of different lengths. The tokenizer's padding=True option pads shorter sequences with [PAD] tokens to match the longest in the batch. The attention_mask ensures the model ignores these padding positions during attention computation. Always pad to the batch maximum rather than the model maximum to avoid wasting computation.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

sentences = [
    'Short.',
    'This is a much longer sentence for demonstration purposes.'
]

encoded = tokenizer(
    sentences,
    padding=True,        # pad to longest in batch
    truncation=True,
    max_length=64,
    return_tensors='pt'
)
print('input_ids shape:', encoded['input_ids'].shape)
print('attention_mask:', encoded['attention_mask'])

Truncation: Handling Long Text

BERT accepts at most 512 tokens per input (including [CLS] and [SEP]). Longer documents must be truncated. Setting truncation=True with max_length=512 cuts at the token limit. For tasks like document classification, common strategies include using only the first 512 tokens, the last 512 tokens (often more informative), or splitting the document into overlapping windows and aggregating predictions.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

long_text = 'word ' * 600  # 600 words -- too long for BERT

# Strategy 1: truncate to first 512 tokens
encoded_first = tokenizer(long_text, truncation=True, max_length=512)
print('First 512 length:', len(encoded_first['input_ids']))

# Strategy 2: only take the last 510 tokens + [CLS] + [SEP]
tokens = tokenizer.tokenize(long_text)[-510:]
encoded_last = [tokenizer.cls_token] + tokens + [tokenizer.sep_token]
print('Last-window length:', len(encoded_last))

Sentence-Pair Inputs for BERT

Tasks like question answering and natural language inference require feeding two sentences to BERT simultaneously. The tokenizer accepts two arguments: tokenizer(sentence_a, sentence_b). It automatically builds the input as [CLS] A [SEP] B [SEP] and populates token_type_ids with 0 for sentence A tokens and 1 for sentence B tokens, letting BERT distinguish the two segments.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

question = 'Where was Marie Curie born?'
context = 'Marie Curie was born in Warsaw, Poland in 1867.'

encoded = tokenizer(
    question,
    context,
    truncation=True,
    max_length=128,
    return_tensors='pt'
)

print('token_type_ids:', encoded['token_type_ids'])
# 0 = question tokens, 1 = context tokens

AutoTokenizer: Provider-Agnostic Loading

Instead of importing model-specific classes, use AutoTokenizer from Hugging Face. It automatically selects the correct tokenizer class based on the model checkpoint name. This makes your code portable: swapping 'bert-base-uncased' for 'roberta-base' or 'distilbert-base-uncased' requires changing only one string, with all tokenizer logic handled automatically.

from transformers import AutoTokenizer

# Works for BERT, RoBERTa, DistilBERT, GPT-2, and more
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

# Exact same API regardless of model family
text = 'Transfer learning is powerful.'
encoded = tokenizer(text, return_tensors='pt')
print(encoded['input_ids'])

# Switching models is trivial:
# tokenizer = AutoTokenizer.from_pretrained('roberta-base')

Decoding: IDs Back to Text

You can convert token IDs back to human-readable text with tokenizer.decode(ids). This is useful for debugging tokenisation and for sequence-to-sequence tasks (translation, summarisation) where the model outputs token IDs that must be decoded to text. Use skip_special_tokens=True to strip [CLS], [SEP], and [PAD] from the output.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = 'Hello, world!'
ids = tokenizer.encode(text)
print('Encoded IDs:', ids)  # includes [CLS]=101 and [SEP]=102

decoded = tokenizer.decode(ids, skip_special_tokens=True)
print('Decoded text:', decoded)  # 'hello, world!'

decoded_with_special = tokenizer.decode(ids)
print('With special tokens:', decoded_with_special)  # '[CLS] hello, world! [SEP]'

Building a DataLoader for Fine-Tuning

During fine-tuning, sentences are tokenised in batches using a PyTorch DataLoader. A custom Dataset stores raw texts and labels; tokenisation happens in __getitem__ or via a collate function. Tokenising inside the DataLoader enables on-the-fly processing and avoids storing large pre-tokenised tensors in memory for large datasets.

import torch
from torch.utils.data import Dataset, DataLoader
from transformers import BertTokenizer

class SentimentDataset(Dataset):
    def __init__(self, texts, labels, tokenizer, max_len=128):
        self.texts = texts
        self.labels = labels
        self.tokenizer = tokenizer
        self.max_len = max_len

    def __len__(self):
        return len(self.texts)

    def __getitem__(self, idx):
        enc = self.tokenizer(
            self.texts[idx],
            truncation=True, padding='max_length',
            max_length=self.max_len, return_tensors='pt'
        )
        return {
            'input_ids': enc['input_ids'].squeeze(),
            'attention_mask': enc['attention_mask'].squeeze(),
            'label': torch.tensor(self.labels[idx], dtype=torch.long)
        }

Fast Tokenizers and Offset Mapping

Hugging Face provides fast tokenizers (backed by Rust) that are 10-100x faster than the pure Python versions. They also support offset mapping: for each token, you get the character start and end positions in the original string. This is essential for token-level tasks like named entity recognition and question answering where you need to map model outputs back to character spans in the source text.

from transformers import BertTokenizerFast

tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')

text = 'Paris is lovely.'
encoded = tokenizer(text, return_offsets_mapping=True)

for token_id, offset in zip(encoded['input_ids'], encoded['offset_mapping']):
    token = tokenizer.convert_ids_to_tokens([token_id])[0]
    print(f'{token:15s} -> chars {offset}')
# paris           -> chars (0, 5)
# is              -> chars (6, 8)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: BertTokenizer converts raw text into input_ids, attention_mask, and token_type_ids tensors, WordPiece subword splitting handles out-of-vocabulary words gracefully, and padding with attention masks and truncation to 512 tokens are required when batching variable-length inputs. Next up we fine-tune BertForSequenceClassification on a real sentiment dataset.

Sıkça Sorulan Sorular

“Hugging Face Belirteçleyicileri: BERT için Metin Kodlama” dersi ücretsiz mi?

Evet — “Hugging Face Belirteçleyicileri: BERT için Metin Kodlama” dersin tüm metni burada web'de ücretsiz olarak okunabilir. Etkileşimli olarak pratik yapmak (yerleşik kod editörü ve 7/24 yapay zeka koçu) ve Machine Learning Academy kursunun geri kalanını açmak için CoddyKit PRO'ya yükselt. Machine Learning Academy kursu toplamda 4 dersten oluşur.

“Hugging Face Belirteçleyicileri: BERT için Metin Kodlama” dersinde ne öğreneceğim?

BertTokenizer yükleyecek, bir cümleler yığınına belirteçlere ayırma uygulayacak, input_ids ve attention_mask tensörlerini inceleyecek, kesme ve doldurma işlemlerini yöneteceksiniz. Machine Learning Academy ile uygulamalı kodu tarayıcıda doğrudan çalıştırarak pratik yaparsın ve 7/24 yapay zeka koçu dersi çalışırken sorularını yanıtlar.

Machine Learning Academy öğrenmeye başlamak için deneyim gerekli mi?

Önceden deneyim gerekmez. CoddyKit'te Machine Learning Academy, başlangıçtan ileri seviyeye kadar yapılandırıldığı için buradan başlayabilir veya başından başlayıp kendi hızında ilerleme yapabilirsin. Bu, 4 dersinin 2. dersidir.

“Hugging Face Belirteçleyicileri: BERT için Metin Kodlama” dersi ne kadar sürer?

Çoğu CoddyKit dersi yaklaşık 5–10 dakika sürer. Her biri kısa ve etkileşimli olduğu için sabit ilerleme yaparsın ve web ile uygulama arasında tam olarak bıraktığın yerden devam edebilirsin.

Bu Machine Learning Academy dersinde kod yazıp çalıştırabilir miyim?

Evet. Her Machine Learning Academy dersi yerleşik bir kod editörü içerir, bu sayede tarayıcıda gerçek kod yazıp çalıştırabilir ve anlık yapay zeka geri bildirimi alırsın — yerel kurulum gerekli değildir.

Bu kursun tüm dersleri

  1. Transformer Mimarisi: Dikkat, Belirteçler ve Bağlam
  2. Hugging Face Belirteçleyicileri: BERT için Metin Kodlama
  3. BertForSequenceClassification için İnce Ayar
  4. Değerlendirme ve Çıkarım: Logitlerden Tahmin Edilen Etiketlere
← Machine Learning Academy Sayfasına Dön