Machine Learning Academy · درس

مجزئات Hugging Face: ترميز النص لـBERT

سيحمّل المتعلمون BertTokenizer، ويجزّئون دفعة من الجمل إلى رموز، ويفحصون موترات input_ids وattention_mask، ويتعاملون مع الاقتطاع والحشو.

الدرس 2 من 413 خطوة

مجزئات Hugging Face: ترميز النص لـBERT درس مجاني في Machine Learning Academy على CoddyKit. هذا هو الدرس 2 من أصل 4. يمكنك قراءة الدرس كاملاً أدناه مجاناً — ثم تمرن عليه مباشرة في المتصفح باستخدام محرر أكواد مدمج ومدرس ذكاء اصطناعي متاح 24/7. هذا الدرس جزء من مسار التعلم في Machine Learning Academy، وتقدمك يتزامن عبر الويب وتطبيق CoddyKit. تتضمن دورة Machine Learning Academy 4 دروس في المجموع.

بعض أجزاء هذا الدرس لم تُترجم بعد وتظهر باللغة الإنجليزية.

Why Tokenisation Matters for BERT

Before BERT can process text, each character sequence must be converted into numerical IDs the model understands. Tokenisation is the process of splitting text into sub-word units (tokens) and mapping them to integer IDs from a fixed vocabulary. Getting tokenisation right is critical: the wrong padding strategy, missing attention masks, or incorrect truncation can silently corrupt your model's input and hurt accuracy.

Installing Hugging Face Transformers

The Hugging Face Transformers library provides pre-trained models and tokenizers for hundreds of architectures. Install it along with datasets for data loading and torch as the backend. The library follows a consistent API: instantiate a tokenizer with from_pretrained, pass it text, and receive ready-to-use tensors.

# Install dependencies
# pip install transformers datasets torch

from transformers import BertTokenizer
import torch

# Load the pre-trained BERT tokenizer (downloads ~200 KB vocab file)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
print('Vocabulary size:', tokenizer.vocab_size)  # 30522
print('Max length:', tokenizer.model_max_length)  # 512

Tokenizing a Single Sentence

Calling the tokenizer on a string returns a dictionary containing input_ids (integer token IDs), attention_mask (1 for real tokens), and optionally token_type_ids (segment IDs). The tokenizer automatically adds [CLS] at the start and [SEP] at the end. Setting return_tensors='pt' returns PyTorch tensors ready for the model.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = 'Machine learning is transforming the world.'
encoding = tokenizer(
    text,
    return_tensors='pt',
    truncation=True,
    max_length=128
)

print('input_ids:', encoding['input_ids'])
print('attention_mask:', encoding['attention_mask'])
print('Token count:', encoding['input_ids'].shape[1])

WordPiece Subword Tokenisation

BERT uses WordPiece tokenisation: rare or unknown words are split into frequent sub-word pieces from the vocabulary. A word like 'tokenisation' might become ['token', '##isation'] where ## marks a continuation subword. This handles out-of-vocabulary words without an unknown token fallback, preserving morphological information that whole-word tokenisation loses.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Inspect raw tokens before converting to IDs
word = 'tokenisation'
pieces = tokenizer.tokenize(word)
print('WordPiece pieces:', pieces)  # ['token', '##isation']

ids = tokenizer.convert_tokens_to_ids(pieces)
print('IDs:', ids)

# Decode back to text
decoded = tokenizer.decode(ids)
print('Decoded:', decoded)

Batch Tokenisation with Padding

Real training uses mini-batches of sentences of different lengths. The tokenizer's padding=True option pads shorter sequences with [PAD] tokens to match the longest in the batch. The attention_mask ensures the model ignores these padding positions during attention computation. Always pad to the batch maximum rather than the model maximum to avoid wasting computation.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

sentences = [
    'Short.',
    'This is a much longer sentence for demonstration purposes.'
]

encoded = tokenizer(
    sentences,
    padding=True,        # pad to longest in batch
    truncation=True,
    max_length=64,
    return_tensors='pt'
)
print('input_ids shape:', encoded['input_ids'].shape)
print('attention_mask:', encoded['attention_mask'])

Truncation: Handling Long Text

BERT accepts at most 512 tokens per input (including [CLS] and [SEP]). Longer documents must be truncated. Setting truncation=True with max_length=512 cuts at the token limit. For tasks like document classification, common strategies include using only the first 512 tokens, the last 512 tokens (often more informative), or splitting the document into overlapping windows and aggregating predictions.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

long_text = 'word ' * 600  # 600 words -- too long for BERT

# Strategy 1: truncate to first 512 tokens
encoded_first = tokenizer(long_text, truncation=True, max_length=512)
print('First 512 length:', len(encoded_first['input_ids']))

# Strategy 2: only take the last 510 tokens + [CLS] + [SEP]
tokens = tokenizer.tokenize(long_text)[-510:]
encoded_last = [tokenizer.cls_token] + tokens + [tokenizer.sep_token]
print('Last-window length:', len(encoded_last))

Sentence-Pair Inputs for BERT

Tasks like question answering and natural language inference require feeding two sentences to BERT simultaneously. The tokenizer accepts two arguments: tokenizer(sentence_a, sentence_b). It automatically builds the input as [CLS] A [SEP] B [SEP] and populates token_type_ids with 0 for sentence A tokens and 1 for sentence B tokens, letting BERT distinguish the two segments.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

question = 'Where was Marie Curie born?'
context = 'Marie Curie was born in Warsaw, Poland in 1867.'

encoded = tokenizer(
    question,
    context,
    truncation=True,
    max_length=128,
    return_tensors='pt'
)

print('token_type_ids:', encoded['token_type_ids'])
# 0 = question tokens, 1 = context tokens

AutoTokenizer: Provider-Agnostic Loading

Instead of importing model-specific classes, use AutoTokenizer from Hugging Face. It automatically selects the correct tokenizer class based on the model checkpoint name. This makes your code portable: swapping 'bert-base-uncased' for 'roberta-base' or 'distilbert-base-uncased' requires changing only one string, with all tokenizer logic handled automatically.

from transformers import AutoTokenizer

# Works for BERT, RoBERTa, DistilBERT, GPT-2, and more
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

# Exact same API regardless of model family
text = 'Transfer learning is powerful.'
encoded = tokenizer(text, return_tensors='pt')
print(encoded['input_ids'])

# Switching models is trivial:
# tokenizer = AutoTokenizer.from_pretrained('roberta-base')

Decoding: IDs Back to Text

You can convert token IDs back to human-readable text with tokenizer.decode(ids). This is useful for debugging tokenisation and for sequence-to-sequence tasks (translation, summarisation) where the model outputs token IDs that must be decoded to text. Use skip_special_tokens=True to strip [CLS], [SEP], and [PAD] from the output.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = 'Hello, world!'
ids = tokenizer.encode(text)
print('Encoded IDs:', ids)  # includes [CLS]=101 and [SEP]=102

decoded = tokenizer.decode(ids, skip_special_tokens=True)
print('Decoded text:', decoded)  # 'hello, world!'

decoded_with_special = tokenizer.decode(ids)
print('With special tokens:', decoded_with_special)  # '[CLS] hello, world! [SEP]'

Building a DataLoader for Fine-Tuning

During fine-tuning, sentences are tokenised in batches using a PyTorch DataLoader. A custom Dataset stores raw texts and labels; tokenisation happens in __getitem__ or via a collate function. Tokenising inside the DataLoader enables on-the-fly processing and avoids storing large pre-tokenised tensors in memory for large datasets.

import torch
from torch.utils.data import Dataset, DataLoader
from transformers import BertTokenizer

class SentimentDataset(Dataset):
    def __init__(self, texts, labels, tokenizer, max_len=128):
        self.texts = texts
        self.labels = labels
        self.tokenizer = tokenizer
        self.max_len = max_len

    def __len__(self):
        return len(self.texts)

    def __getitem__(self, idx):
        enc = self.tokenizer(
            self.texts[idx],
            truncation=True, padding='max_length',
            max_length=self.max_len, return_tensors='pt'
        )
        return {
            'input_ids': enc['input_ids'].squeeze(),
            'attention_mask': enc['attention_mask'].squeeze(),
            'label': torch.tensor(self.labels[idx], dtype=torch.long)
        }

Fast Tokenizers and Offset Mapping

Hugging Face provides fast tokenizers (backed by Rust) that are 10-100x faster than the pure Python versions. They also support offset mapping: for each token, you get the character start and end positions in the original string. This is essential for token-level tasks like named entity recognition and question answering where you need to map model outputs back to character spans in the source text.

from transformers import BertTokenizerFast

tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')

text = 'Paris is lovely.'
encoded = tokenizer(text, return_offsets_mapping=True)

for token_id, offset in zip(encoded['input_ids'], encoded['offset_mapping']):
    token = tokenizer.convert_ids_to_tokens([token_id])[0]
    print(f'{token:15s} -> chars {offset}')
# paris           -> chars (0, 5)
# is              -> chars (6, 8)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

أجب عليها في الدرس التفاعلي

Lesson Recap

In this lesson you learned: BertTokenizer converts raw text into input_ids, attention_mask, and token_type_ids tensors, WordPiece subword splitting handles out-of-vocabulary words gracefully, and padding with attention masks and truncation to 512 tokens are required when batching variable-length inputs. Next up we fine-tune BertForSequenceClassification on a real sentiment dataset.

البدء مجانًا

تعلم Python مع معلم ذكاء اصطناعي — مجانًا

اكتب وقم بتشغيل أكوادك الفعلية في المتصفح، واحصل على مساعدة فورية من معلم ذكاء اصطناعي متاح 24/7، واستمر من حيث توقفت على الويب أو في التطبيق.

الدورات: 30
الدروس: 120

ابدأ التعلم مجانًا عرض الدورة الكاملة

الأسئلة الشائعة

هل درس «مجزئات Hugging Face: ترميز النص لـBERT» مجاني؟

نعم — نص درس «مجزئات Hugging Face: ترميز النص لـBERT» كامل متاح مجاناً هنا على الويب. لتمرينه بشكل تفاعلي (محرر أكواد مدمج ومدرس ذكاء اصطناعي متاح 24/7) وفتح باقي دورة Machine Learning Academy، انتقل إلى CoddyKit PRO. تتضمن دورة Machine Learning Academy 4 دروس في المجموع.

ماذا ستتعلم في «مجزئات Hugging Face: ترميز النص لـBERT»؟

سيحمّل المتعلمون BertTokenizer، ويجزّئون دفعة من الجمل إلى رموز، ويفحصون موترات input_ids وattention_mask، ويتعاملون مع الاقتطاع والحشو. تتمرن على Machine Learning Academy مع أكواد عملية تشغلها مباشرة في المتصفح، ومدرس ذكاء اصطناعي متاح 24/7 يجيب على أسئلتك أثناء عملك.

هل أحتاج إلى خبرة سابقة لأبدأ Machine Learning Academy؟

لا تُشترط خبرة سابقة. Machine Learning Academy على CoddyKit منظم للمبتدئين حتى المتقدمين، لذا يمكنك البدء من هنا أو من البداية والتقدم بسرعتك الخاصة. هذا هو الدرس 2 من أصل 4.

كم من الوقت يستغرق درس «مجزئات Hugging Face: ترميز النص لـBERT»؟

معظم دروس CoddyKit تستغرق حوالي 5–10 دقائق. كل منها موجز وتفاعلي، لذا تحرز تقدماً مستمراً وتستأنف من حيث توقفت عبر الويب والتطبيق.

هل يمكنني كتابة وتشغيل أكواد في درس Machine Learning Academy هذا؟

نعم. كل درس في Machine Learning Academy يتضمن محرر أكواد مدمج، لذا تكتب وتشغل أكواداً حقيقية مباشرة في متصفحك وتحصل على تعليقات فورية من الذكاء الاصطناعي — بدون إعداد محلي.

جميع الدروس في هذه الدورة

بنية Transformer: الانتباه والرموز والسياق
مجزئات Hugging Face: ترميز النص لـBERT
الضبط الدقيق لـBertForSequenceClassification
التقييم والاستدلال: من القيم اللوغاريتمية إلى التسميات المتوقعة

← العودة إلى Machine Learning Academy