0Pricing
Machine Learning Academy · Lección

Tokenizadores de Hugging Face: codificación de texto para BERT

Cargará BertTokenizer, tokenizará un lote de oraciones, examinará los tensores input_ids y attention_mask y gestionará el truncamiento y el relleno.

Tokenizadores de Hugging Face: codificación de texto para BERT es una lección gratuita de Machine Learning Academy en CoddyKit. Esta es la lección 2 de 4. Puedes leer la lección completa abajo gratuitamente — luego la practicas en el navegador con un editor de código integrado y un tutor de IA 24/7. Forma parte de la ruta de aprendizaje de Machine Learning Academy, y tu progreso se sincroniza en la web y la app de CoddyKit. El curso de Machine Learning Academy incluye 4 lecciones en total.

Partes de esta lección aún no han sido traducidas y se muestran en inglés.

Why Tokenisation Matters for BERT

Before BERT can process text, each character sequence must be converted into numerical IDs the model understands. Tokenisation is the process of splitting text into sub-word units (tokens) and mapping them to integer IDs from a fixed vocabulary. Getting tokenisation right is critical: the wrong padding strategy, missing attention masks, or incorrect truncation can silently corrupt your model's input and hurt accuracy.

Installing Hugging Face Transformers

The Hugging Face Transformers library provides pre-trained models and tokenizers for hundreds of architectures. Install it along with datasets for data loading and torch as the backend. The library follows a consistent API: instantiate a tokenizer with from_pretrained, pass it text, and receive ready-to-use tensors.

# Install dependencies
# pip install transformers datasets torch

from transformers import BertTokenizer
import torch

# Load the pre-trained BERT tokenizer (downloads ~200 KB vocab file)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
print('Vocabulary size:', tokenizer.vocab_size)  # 30522
print('Max length:', tokenizer.model_max_length)  # 512

Tokenizing a Single Sentence

Calling the tokenizer on a string returns a dictionary containing input_ids (integer token IDs), attention_mask (1 for real tokens), and optionally token_type_ids (segment IDs). The tokenizer automatically adds [CLS] at the start and [SEP] at the end. Setting return_tensors='pt' returns PyTorch tensors ready for the model.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = 'Machine learning is transforming the world.'
encoding = tokenizer(
    text,
    return_tensors='pt',
    truncation=True,
    max_length=128
)

print('input_ids:', encoding['input_ids'])
print('attention_mask:', encoding['attention_mask'])
print('Token count:', encoding['input_ids'].shape[1])

WordPiece Subword Tokenisation

BERT uses WordPiece tokenisation: rare or unknown words are split into frequent sub-word pieces from the vocabulary. A word like 'tokenisation' might become ['token', '##isation'] where ## marks a continuation subword. This handles out-of-vocabulary words without an unknown token fallback, preserving morphological information that whole-word tokenisation loses.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Inspect raw tokens before converting to IDs
word = 'tokenisation'
pieces = tokenizer.tokenize(word)
print('WordPiece pieces:', pieces)  # ['token', '##isation']

ids = tokenizer.convert_tokens_to_ids(pieces)
print('IDs:', ids)

# Decode back to text
decoded = tokenizer.decode(ids)
print('Decoded:', decoded)

Batch Tokenisation with Padding

Real training uses mini-batches of sentences of different lengths. The tokenizer's padding=True option pads shorter sequences with [PAD] tokens to match the longest in the batch. The attention_mask ensures the model ignores these padding positions during attention computation. Always pad to the batch maximum rather than the model maximum to avoid wasting computation.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

sentences = [
    'Short.',
    'This is a much longer sentence for demonstration purposes.'
]

encoded = tokenizer(
    sentences,
    padding=True,        # pad to longest in batch
    truncation=True,
    max_length=64,
    return_tensors='pt'
)
print('input_ids shape:', encoded['input_ids'].shape)
print('attention_mask:', encoded['attention_mask'])

Truncation: Handling Long Text

BERT accepts at most 512 tokens per input (including [CLS] and [SEP]). Longer documents must be truncated. Setting truncation=True with max_length=512 cuts at the token limit. For tasks like document classification, common strategies include using only the first 512 tokens, the last 512 tokens (often more informative), or splitting the document into overlapping windows and aggregating predictions.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

long_text = 'word ' * 600  # 600 words -- too long for BERT

# Strategy 1: truncate to first 512 tokens
encoded_first = tokenizer(long_text, truncation=True, max_length=512)
print('First 512 length:', len(encoded_first['input_ids']))

# Strategy 2: only take the last 510 tokens + [CLS] + [SEP]
tokens = tokenizer.tokenize(long_text)[-510:]
encoded_last = [tokenizer.cls_token] + tokens + [tokenizer.sep_token]
print('Last-window length:', len(encoded_last))

Sentence-Pair Inputs for BERT

Tasks like question answering and natural language inference require feeding two sentences to BERT simultaneously. The tokenizer accepts two arguments: tokenizer(sentence_a, sentence_b). It automatically builds the input as [CLS] A [SEP] B [SEP] and populates token_type_ids with 0 for sentence A tokens and 1 for sentence B tokens, letting BERT distinguish the two segments.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

question = 'Where was Marie Curie born?'
context = 'Marie Curie was born in Warsaw, Poland in 1867.'

encoded = tokenizer(
    question,
    context,
    truncation=True,
    max_length=128,
    return_tensors='pt'
)

print('token_type_ids:', encoded['token_type_ids'])
# 0 = question tokens, 1 = context tokens

AutoTokenizer: Provider-Agnostic Loading

Instead of importing model-specific classes, use AutoTokenizer from Hugging Face. It automatically selects the correct tokenizer class based on the model checkpoint name. This makes your code portable: swapping 'bert-base-uncased' for 'roberta-base' or 'distilbert-base-uncased' requires changing only one string, with all tokenizer logic handled automatically.

from transformers import AutoTokenizer

# Works for BERT, RoBERTa, DistilBERT, GPT-2, and more
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

# Exact same API regardless of model family
text = 'Transfer learning is powerful.'
encoded = tokenizer(text, return_tensors='pt')
print(encoded['input_ids'])

# Switching models is trivial:
# tokenizer = AutoTokenizer.from_pretrained('roberta-base')

Decoding: IDs Back to Text

You can convert token IDs back to human-readable text with tokenizer.decode(ids). This is useful for debugging tokenisation and for sequence-to-sequence tasks (translation, summarisation) where the model outputs token IDs that must be decoded to text. Use skip_special_tokens=True to strip [CLS], [SEP], and [PAD] from the output.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = 'Hello, world!'
ids = tokenizer.encode(text)
print('Encoded IDs:', ids)  # includes [CLS]=101 and [SEP]=102

decoded = tokenizer.decode(ids, skip_special_tokens=True)
print('Decoded text:', decoded)  # 'hello, world!'

decoded_with_special = tokenizer.decode(ids)
print('With special tokens:', decoded_with_special)  # '[CLS] hello, world! [SEP]'

Building a DataLoader for Fine-Tuning

During fine-tuning, sentences are tokenised in batches using a PyTorch DataLoader. A custom Dataset stores raw texts and labels; tokenisation happens in __getitem__ or via a collate function. Tokenising inside the DataLoader enables on-the-fly processing and avoids storing large pre-tokenised tensors in memory for large datasets.

import torch
from torch.utils.data import Dataset, DataLoader
from transformers import BertTokenizer

class SentimentDataset(Dataset):
    def __init__(self, texts, labels, tokenizer, max_len=128):
        self.texts = texts
        self.labels = labels
        self.tokenizer = tokenizer
        self.max_len = max_len

    def __len__(self):
        return len(self.texts)

    def __getitem__(self, idx):
        enc = self.tokenizer(
            self.texts[idx],
            truncation=True, padding='max_length',
            max_length=self.max_len, return_tensors='pt'
        )
        return {
            'input_ids': enc['input_ids'].squeeze(),
            'attention_mask': enc['attention_mask'].squeeze(),
            'label': torch.tensor(self.labels[idx], dtype=torch.long)
        }

Fast Tokenizers and Offset Mapping

Hugging Face provides fast tokenizers (backed by Rust) that are 10-100x faster than the pure Python versions. They also support offset mapping: for each token, you get the character start and end positions in the original string. This is essential for token-level tasks like named entity recognition and question answering where you need to map model outputs back to character spans in the source text.

from transformers import BertTokenizerFast

tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')

text = 'Paris is lovely.'
encoded = tokenizer(text, return_offsets_mapping=True)

for token_id, offset in zip(encoded['input_ids'], encoded['offset_mapping']):
    token = tokenizer.convert_ids_to_tokens([token_id])[0]
    print(f'{token:15s} -> chars {offset}')
# paris           -> chars (0, 5)
# is              -> chars (6, 8)

Quick Check

Test your understanding of Machine Learning with Python concepts from this lesson.

Lesson Recap

In this lesson you learned: BertTokenizer converts raw text into input_ids, attention_mask, and token_type_ids tensors, WordPiece subword splitting handles out-of-vocabulary words gracefully, and padding with attention masks and truncation to 512 tokens are required when batching variable-length inputs. Next up we fine-tune BertForSequenceClassification on a real sentiment dataset.

Preguntas frecuentes

¿La lección «Tokenizadores de Hugging Face: codificación de texto para BERT» es gratis?

Sí — el texto completo de «Tokenizadores de Hugging Face: codificación de texto para BERT» es gratis para leer aquí en la web. Para practicarla de forma interactiva (editor de código integrado y tutor de IA 24/7) y desbloquear el resto del curso de Machine Learning Academy, actualiza a CoddyKit PRO. El curso de Machine Learning Academy incluye 4 lecciones en total.

¿Qué aprenderé en «Tokenizadores de Hugging Face: codificación de texto para BERT»?

Cargará BertTokenizer, tokenizará un lote de oraciones, examinará los tensores input_ids y attention_mask y gestionará el truncamiento y el relleno. Practicas Machine Learning Academy con código real que ejecutas directamente en el navegador, y un tutor de IA 24/7 responde tus preguntas mientras trabajas en la lección.

¿Necesito experiencia previa para empezar Machine Learning Academy?

No se requiere experiencia previa. Machine Learning Academy en CoddyKit está estructurado para principiantes hasta estudiantes avanzados, así que puedes empezar aquí o desde el inicio y avanzar a tu ritmo. Esta es la lección 2 de 4.

¿Cuánto tiempo toma la lección «Tokenizadores de Hugging Face: codificación de texto para BERT»?

La mayoría de las lecciones de CoddyKit toman alrededor de 5–10 minutos. Cada una es compacta e interactiva, así que avanzas constantemente y retomas exactamente por donde dejaste en la web y la app.

¿Puedo escribir y ejecutar código en esta lección de Machine Learning Academy?

Sí. Cada lección de Machine Learning Academy incluye un editor de código integrado, así que escribes y ejecutas código real directamente en tu navegador y obtienes retroalimentación instantánea de IA — sin configuración local necesaria.

Todas las lecciones de este curso

  1. Arquitectura Transformer: atención, tokens y contexto
  2. Tokenizadores de Hugging Face: codificación de texto para BERT
  3. Ajuste fino de BertForSequenceClassification
  4. Evaluación e inferencia: de los logits a las etiquetas predichas
← Volver a Machine Learning Academy