0Pricing
Machine Learning Academy · Aula

Tokenizadores do Hugging Face: codificação de texto para BERT

Os alunos carregarão BertTokenizer, tokenizarão um lote de frases, examinarão os tensores input_ids e attention_mask e lidarão com truncamento e preenchimento.

Tokenizadores do Hugging Face: codificação de texto para BERT é uma aula grátis de Machine Learning Academy no CoddyKit. Esta é a aula 2 de 4. Você pode ler a aula completa abaixo gratuitamente — depois pratica ao vivo no navegador com um editor de código integrado e um tutor de IA 24/7. Faz parte do caminho de aprendizado de Machine Learning Academy, e seu progresso é sincronizado entre a web e o app CoddyKit. O curso de Machine Learning Academy inclui 4 aulas no total.

Partes desta aula ainda não foram traduzidas e aparecem em inglês.

Why Tokenisation Matters for BERT

Before BERT can process text, each character sequence must be converted into numerical IDs the model understands. Tokenisation is the process of splitting text into sub-word units (tokens) and mapping them to integer IDs from a fixed vocabulary. Getting tokenisation right is critical: the wrong padding strategy, missing attention masks, or incorrect truncation can silently corrupt your model's input and hurt accuracy.

Installing Hugging Face Transformers

The Hugging Face Transformers library provides pre-trained models and tokenizers for hundreds of architectures. Install it along with datasets for data loading and torch as the backend. The library follows a consistent API: instantiate a tokenizer with from_pretrained, pass it text, and receive ready-to-use tensors.

# Install dependencies
# pip install transformers datasets torch

from transformers import BertTokenizer
import torch

# Load the pre-trained BERT tokenizer (downloads ~200 KB vocab file)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
print('Vocabulary size:', tokenizer.vocab_size)  # 30522
print('Max length:', tokenizer.model_max_length)  # 512

Tokenizing a Single Sentence

Calling the tokenizer on a string returns a dictionary containing input_ids (integer token IDs), attention_mask (1 for real tokens), and optionally token_type_ids (segment IDs). The tokenizer automatically adds [CLS] at the start and [SEP] at the end. Setting return_tensors='pt' returns PyTorch tensors ready for the model.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = 'Machine learning is transforming the world.'
encoding = tokenizer(
    text,
    return_tensors='pt',
    truncation=True,
    max_length=128
)

print('input_ids:', encoding['input_ids'])
print('attention_mask:', encoding['attention_mask'])
print('Token count:', encoding['input_ids'].shape[1])

WordPiece Subword Tokenisation

BERT uses WordPiece tokenisation: rare or unknown words are split into frequent sub-word pieces from the vocabulary. A word like 'tokenisation' might become ['token', '##isation'] where ## marks a continuation subword. This handles out-of-vocabulary words without an unknown token fallback, preserving morphological information that whole-word tokenisation loses.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Inspect raw tokens before converting to IDs
word = 'tokenisation'
pieces = tokenizer.tokenize(word)
print('WordPiece pieces:', pieces)  # ['token', '##isation']

ids = tokenizer.convert_tokens_to_ids(pieces)
print('IDs:', ids)

# Decode back to text
decoded = tokenizer.decode(ids)
print('Decoded:', decoded)

Batch Tokenisation with Padding

Real training uses mini-batches of sentences of different lengths. The tokenizer's padding=True option pads shorter sequences with [PAD] tokens to match the longest in the batch. The attention_mask ensures the model ignores these padding positions during attention computation. Always pad to the batch maximum rather than the model maximum to avoid wasting computation.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

sentences = [
    'Short.',
    'This is a much longer sentence for demonstration purposes.'
]

encoded = tokenizer(
    sentences,
    padding=True,        # pad to longest in batch
    truncation=True,
    max_length=64,
    return_tensors='pt'
)
print('input_ids shape:', encoded['input_ids'].shape)
print('attention_mask:', encoded['attention_mask'])

Truncation: Handling Long Text

BERT accepts at most 512 tokens per input (including [CLS] and [SEP]). Longer documents must be truncated. Setting truncation=True with max_length=512 cuts at the token limit. For tasks like document classification, common strategies include using only the first 512 tokens, the last 512 tokens (often more informative), or splitting the document into overlapping windows and aggregating predictions.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

long_text = 'word ' * 600  # 600 words -- too long for BERT

# Strategy 1: truncate to first 512 tokens
encoded_first = tokenizer(long_text, truncation=True, max_length=512)
print('First 512 length:', len(encoded_first['input_ids']))

# Strategy 2: only take the last 510 tokens + [CLS] + [SEP]
tokens = tokenizer.tokenize(long_text)[-510:]
encoded_last = [tokenizer.cls_token] + tokens + [tokenizer.sep_token]
print('Last-window length:', len(encoded_last))

Sentence-Pair Inputs for BERT

Tasks like question answering and natural language inference require feeding two sentences to BERT simultaneously. The tokenizer accepts two arguments: tokenizer(sentence_a, sentence_b). It automatically builds the input as [CLS] A [SEP] B [SEP] and populates token_type_ids with 0 for sentence A tokens and 1 for sentence B tokens, letting BERT distinguish the two segments.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

question = 'Where was Marie Curie born?'
context = 'Marie Curie was born in Warsaw, Poland in 1867.'

encoded = tokenizer(
    question,
    context,
    truncation=True,
    max_length=128,
    return_tensors='pt'
)

print('token_type_ids:', encoded['token_type_ids'])
# 0 = question tokens, 1 = context tokens

AutoTokenizer: Provider-Agnostic Loading

Instead of importing model-specific classes, use AutoTokenizer from Hugging Face. It automatically selects the correct tokenizer class based on the model checkpoint name. This makes your code portable: swapping 'bert-base-uncased' for 'roberta-base' or 'distilbert-base-uncased' requires changing only one string, with all tokenizer logic handled automatically.

from transformers import AutoTokenizer

# Works for BERT, RoBERTa, DistilBERT, GPT-2, and more
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

# Exact same API regardless of model family
text = 'Transfer learning is powerful.'
encoded = tokenizer(text, return_tensors='pt')
print(encoded['input_ids'])

# Switching models is trivial:
# tokenizer = AutoTokenizer.from_pretrained('roberta-base')

Decoding: IDs Back to Text

You can convert token IDs back to human-readable text with tokenizer.decode(ids). This is useful for debugging tokenisation and for sequence-to-sequence tasks (translation, summarisation) where the model outputs token IDs that must be decoded to text. Use skip_special_tokens=True to strip [CLS], [SEP], and [PAD] from the output.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = 'Hello, world!'
ids = tokenizer.encode(text)
print('Encoded IDs:', ids)  # includes [CLS]=101 and [SEP]=102

decoded = tokenizer.decode(ids, skip_special_tokens=True)
print('Decoded text:', decoded)  # 'hello, world!'

decoded_with_special = tokenizer.decode(ids)
print('With special tokens:', decoded_with_special)  # '[CLS] hello, world! [SEP]'

Building a DataLoader for Fine-Tuning

During fine-tuning, sentences are tokenised in batches using a PyTorch DataLoader. A custom Dataset stores raw texts and labels; tokenisation happens in __getitem__ or via a collate function. Tokenising inside the DataLoader enables on-the-fly processing and avoids storing large pre-tokenised tensors in memory for large datasets.

import torch
from torch.utils.data import Dataset, DataLoader
from transformers import BertTokenizer

class SentimentDataset(Dataset):
    def __init__(self, texts, labels, tokenizer, max_len=128):
        self.texts = texts
        self.labels = labels
        self.tokenizer = tokenizer
        self.max_len = max_len

    def __len__(self):
        return len(self.texts)

    def __getitem__(self, idx):
        enc = self.tokenizer(
            self.texts[idx],
            truncation=True, padding='max_length',
            max_length=self.max_len, return_tensors='pt'
        )
        return {
            'input_ids': enc['input_ids'].squeeze(),
            'attention_mask': enc['attention_mask'].squeeze(),
            'label': torch.tensor(self.labels[idx], dtype=torch.long)
        }

Fast Tokenizers and Offset Mapping

Hugging Face provides fast tokenizers (backed by Rust) that are 10-100x faster than the pure Python versions. They also support offset mapping: for each token, you get the character start and end positions in the original string. This is essential for token-level tasks like named entity recognition and question answering where you need to map model outputs back to character spans in the source text.

from transformers import BertTokenizerFast

tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')

text = 'Paris is lovely.'
encoded = tokenizer(text, return_offsets_mapping=True)

for token_id, offset in zip(encoded['input_ids'], encoded['offset_mapping']):
    token = tokenizer.convert_ids_to_tokens([token_id])[0]
    print(f'{token:15s} -> chars {offset}')
# paris           -> chars (0, 5)
# is              -> chars (6, 8)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: BertTokenizer converts raw text into input_ids, attention_mask, and token_type_ids tensors, WordPiece subword splitting handles out-of-vocabulary words gracefully, and padding with attention masks and truncation to 512 tokens are required when batching variable-length inputs. Next up we fine-tune BertForSequenceClassification on a real sentiment dataset.

Perguntas Frequentes

A aula “Tokenizadores do Hugging Face: codificação de texto para BERT” é grátis?

Sim — o texto completo de “Tokenizadores do Hugging Face: codificação de texto para BERT” é grátis para ler aqui na web. Para praticá-la interativamente (um editor de código integrado e um tutor de IA 24/7) e desbloquear o restante do curso de Machine Learning Academy, atualize para CoddyKit PRO. O curso de Machine Learning Academy inclui 4 aulas no total.

O que vou aprender em “Tokenizadores do Hugging Face: codificação de texto para BERT”?

Os alunos carregarão BertTokenizer, tokenizarão um lote de frases, examinarão os tensores input_ids e attention_mask e lidarão com truncamento e preenchimento. Você pratica Machine Learning Academy com código prático que executa diretamente no navegador, e um tutor de IA 24/7 responde suas dúvidas enquanto trabalha na aula.

Preciso ter experiência prévia para começar Machine Learning Academy?

Nenhuma experiência prévia é necessária. Machine Learning Academy no CoddyKit é estruturado para alunos iniciantes até avançados, então você pode começar aqui ou desde o início e aprender no seu ritmo. Esta é a aula 2 de 4.

Quanto tempo leva a aula “Tokenizadores do Hugging Face: codificação de texto para BERT”?

A maioria das aulas CoddyKit leva cerca de 5–10 minutos. Cada uma é compacta e interativa, então você faz progresso constante e retoma exatamente de onde parou entre web e app.

Posso escrever e executar código nesta aula de Machine Learning Academy?

Sim. Cada aula de Machine Learning Academy inclui um editor de código integrado, então você escreve e executa código real direto no navegador e recebe feedback de IA instantaneamente — nenhuma configuração local necessária.

Todas as aulas deste curso

  1. Arquitetura Transformer: atenção, tokens e contexto
  2. Tokenizadores do Hugging Face: codificação de texto para BERT
  3. Ajuste fino de BertForSequenceClassification
  4. Avaliação e inferência: dos logits aos rótulos previstos
← Voltar para Machine Learning Academy