0Pricing
AI Engineering Academy · Leçon

Qu’est-ce qu’un token ?

Utilisez la bibliothèque tiktoken pour tokeniser du texte réel et découvrez comment les mots, la ponctuation et les espaces correspondent à des séquences de tokens dans différents modèles.

Qu’est-ce qu’un token ? est une leçon AI Engineering Academy gratuite sur CoddyKit. Ceci est la leçon 1 sur 4. Tu peux lire la leçon complète ci-dessous gratuitement — puis la pratiquer en direct dans le navigateur avec un éditeur de code intégré et un tuteur IA 24/7. Elle fait partie du parcours d'apprentissage AI Engineering Academy, et ta progression se synchronise sur le web et l'application CoddyKit. Le cours AI Engineering Academy comprend 4 leçons au total.

Certaines parties de cette leçon n'ont pas encore été traduites et s'affichent en anglais.

LLMs Process Tokens, Not Words

When you send text to an LLM, the model does not see characters or words — it sees tokens. A token is a chunk of text that the model's vocabulary maps to a single integer ID. Tokens can be whole words, parts of words, punctuation, or whitespace depending on the tokenizer.

Understanding tokens is practically important because OpenAI's pricing is per token, the context window is measured in tokens, and the maximum response length is controlled by max_tokens. Surprises in cost and behavior almost always trace back to misunderstanding how your text tokenizes.

How Tokenization Works: BPE

GPT models use Byte Pair Encoding (BPE) tokenization. BPE starts with a vocabulary of individual bytes and iteratively merges the most frequent adjacent pairs until it reaches the desired vocabulary size. OpenAI's GPT models use a vocabulary of about 100,000 tokens.

The result is that common words become single tokens (hello is one token), while rare or made-up words are split into multiple subword tokens (subaqueous might become three tokens: sub, aque, ous). This lets the model handle any text, even words it has never seen, by composing them from smaller familiar pieces.

Using tiktoken to Count Tokens

OpenAI provides the tiktoken library to tokenize text locally without making an API call. This is essential for counting tokens before sending a request to estimate cost and verify you are within the context window.

import tiktoken

# Get the encoding for a specific model
enc = tiktoken.encoding_for_model('gpt-4o')

text = 'Hello! How many tokens does this sentence use?'
tokens = enc.encode(text)

print(f'Text: {text}')
print(f'Token count: {len(tokens)}')
print(f'Token IDs: {tokens}')

# You can also decode tokens back to text
decoded = enc.decode(tokens)
print(f'Decoded: {decoded}')

# Inspect individual token strings
for token_id in tokens:
    token_str = enc.decode([token_id])
    print(f'  Token {token_id}: "{token_str}"')

Practical Token Counts

A useful rule of thumb: 1 token ≈ 4 characters of English text, or about 0.75 words. So 1,000 tokens is roughly 750 words or 3-4 pages of text. However, this ratio varies significantly:

  • Common English words: ~1 token each
  • Uncommon technical terms: 2-4 tokens each
  • Numbers: often 1 digit per token (so '12345' is 5 tokens)
  • Code: varies widely, but Python is typically efficient at ~1-2 tokens per symbol
  • Non-Latin scripts (Chinese, Arabic): often 1 token per character, making them much more expensive per word than English

Tokenizing Different Content Types

Let us explore how token counts differ dramatically between content types using tiktoken.

import tiktoken

enc = tiktoken.encoding_for_model('gpt-4o')

examples = {
    'English sentence': 'The quick brown fox jumps over the lazy dog.',
    'Number sequence': '1234567890',
    'Python code': 'def fibonacci(n):\n    if n <= 1:\n        return n\n    return fibonacci(n-1) + fibonacci(n-2)',
    'Technical jargon': 'autoregressive transformers tokenization subword BPE',
    'URL': 'https://api.openai.com/v1/chat/completions',
    'Chinese text': '大型语言模型使用令牌处理文本',
}

for label, text in examples.items():
    count = len(enc.encode(text))
    ratio = len(text) / count
    print(f'{label}: {count} tokens ({ratio:.1f} chars/token)')

How Chat Messages Are Tokenized

The total tokens in a chat API call include not just the text content of your messages but also the formatting overhead added by the chat template. Every message has a few extra tokens for the role label and delimiters. OpenAI's documentation specifies the formula:

  • Each message adds approximately 4 tokens for formatting overhead
  • Every reply starts with 3 additional tokens for the assistant role priming

For short conversations this overhead is negligible, but for systems managing long conversations it adds up. The tiktoken library provides helper functions to count tokens for a full message array correctly.

import tiktoken

def count_messages_tokens(messages, model='gpt-4o'):
    enc = tiktoken.encoding_for_model(model)
    # 4 tokens per message (role + content structure), 3 for reply priming
    total = 3
    for msg in messages:
        total += 4
        for key, value in msg.items():
            total += len(enc.encode(str(value)))
    return total

messages = [
    {'role': 'system', 'content': 'You are a helpful assistant.'},
    {'role': 'user', 'content': 'What is the capital of France?'},
    {'role': 'assistant', 'content': 'The capital of France is Paris.'},
    {'role': 'user', 'content': 'And what is the population?'},
]

print(f'Total tokens: {count_messages_tokens(messages)}')

Token Limits by Model

Each model has a maximum context window measured in tokens. As of 2025, representative limits include:

  • gpt-4o: 128,000 tokens context, up to 16,384 output
  • gpt-4o-mini: 128,000 tokens context, up to 16,384 output
  • Claude 3.5 Sonnet: 200,000 tokens context
  • Gemini 1.5 Pro: 1,000,000 tokens context

The sum of input tokens plus output tokens must not exceed the context window. If you exceed it, you get a context_length_exceeded error. Always verify token count before sending requests for long documents.

Tokens and Pricing

OpenAI charges separately for input tokens (the prompt you send) and output tokens (the response you receive). Output tokens are typically 3-4x more expensive than input tokens because they require sequential generation. As of early 2025, illustrative pricing for gpt-4o-mini is approximately $0.15 per million input tokens and $0.60 per million output tokens.

This means a 2,000-token prompt with a 500-token response costs approximately ($0.15 × 2/1000 + $0.60 × 0.5/1000) = $0.00060 per request. At 10,000 requests per day that is $6/day — manageable, but it scales with usage. Caching, model routing, and token minimization all become important at production scale.

Reducing Token Count Without Losing Meaning

Shorter prompts cost less and leave more room for the model's response. Common token-reduction techniques include:

  • Remove redundant instructions: 'Please be so kind as to...' → 'Please...'
  • Compress examples: Use the minimum number of words in each few-shot example
  • Abbreviate field names in structured prompts: Use 'Q:' and 'A:' instead of 'Question:' and 'Answer:'
  • Use JSON rather than prose for structured context: JSON is more token-efficient than prose descriptions of the same data

Run tiktoken before and after any compression to verify you actually saved tokens — some 'compressions' counterintuitively increase token count.

Special Tokens and Control Tokens

In addition to text tokens, the model's vocabulary includes special tokens used to structure inputs. Common examples are <|endoftext|> (marks the end of a document), <|im_start|> and <|im_end|> (chat message delimiters in the ChatML format used internally).

You do not need to manage these directly when using the OpenAI API — the SDK handles them for you. But awareness of their existence explains why sometimes a request with 'empty' messages still consumes a few tokens. Special tokens are also why you should never construct raw strings with <|...|> patterns in user input without sanitizing, since they could interfere with the model's input formatting.

Always Count Before You Send

The practical takeaway from this lesson: always count tokens before sending any request that involves dynamically assembled prompts. Write a small helper function that wraps tiktoken, and call it at the point where you assemble the messages array. Log the token count so you can monitor it over time.

For RAG systems, this is especially important: the retrieved chunks you inject into the prompt can vary widely in size, and you need to ensure the total stays within the model's context window. We will build exactly this kind of token-aware context assembly in the RAG pipeline lessons.

import tiktoken

def token_count(text, model='gpt-4o'):
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

def safe_assemble_prompt(system, user_content, max_tokens=120000):
    combined = system + user_content
    count = token_count(combined)
    if count > max_tokens:
        raise ValueError(
            f'Prompt too long: {count} tokens (limit {max_tokens})'
        )
    return [{'role': 'system', 'content': system},
            {'role': 'user', 'content': user_content}]

Quick Check

Test your understanding of AI Engineering concepts from this lesson.

Lesson Recap

In this lesson you learned: LLMs process text as tokens, not words, using BPE tokenization with a vocabulary of ~100,000 entries, the tiktoken library lets you count tokens locally before making API calls to estimate cost and check context limits, and pricing is per token with output tokens costing significantly more than input tokens. Next up we explore context windows: what they are, how they limit your application, and how different models compare.

Questions Fréquemment Posées

La leçon « Qu’est-ce qu’un token ? » est-elle gratuite ?

Oui — le texte complet de « Qu’est-ce qu’un token ? » est gratuit à lire ici sur le web. Pour la pratiquer de manière interactive (un éditeur de code intégré et un tuteur IA 24/7) et déverrouiller le reste du cours AI Engineering Academy, passe à CoddyKit PRO. Le cours AI Engineering Academy comprend 4 leçons au total.

Qu'est-ce que j'apprendrai dans « Qu’est-ce qu’un token ? » ?

Utilisez la bibliothèque tiktoken pour tokeniser du texte réel et découvrez comment les mots, la ponctuation et les espaces correspondent à des séquences de tokens dans différents modèles. Tu pratiques AI Engineering Academy avec du code pratique que tu exécutes directement dans le navigateur, et un tuteur IA 24/7 répond à tes questions au fur et à mesure que tu avances dans la leçon.

Dois-je avoir de l'expérience pour commencer AI Engineering Academy ?

Aucune expérience préalable n'est requise. AI Engineering Academy sur CoddyKit est structuré pour les débutants jusqu'aux apprenants avancés, donc tu peux commencer ici ou depuis le début et avancer à ton rythme. Ceci est la leçon 1 sur 4.

Combien de temps prend la leçon « Qu’est-ce qu’un token ? » ?

La plupart des leçons CoddyKit prennent environ 5–10 minutes. Chacune est courte et interactive, tu progresses régulièrement et tu repiques exactement où tu t'es arrêté sur le web et l'app.

Peux-tu écrire et exécuter du code dans cette leçon AI Engineering Academy ?

Oui. Chaque leçon AI Engineering Academy inclut un éditeur de code intégré, tu écris et exécutes du vrai code directement dans ton navigateur et tu reçois des retours IA instantanés — aucune configuration locale requise.

Toutes les leçons de ce cours

  1. Qu’est-ce qu’un token ?
  2. Fenêtres de contexte : taille et conséquences
  3. Calculer et prévoir les coûts de l’API
  4. Stratégies pour rester dans les limites du contexte
← Retour à AI Engineering Academy