Tokenizing Input
Turn text into tokens.
What Is a Tokenizer?
An interpreter starts by turning raw text into tokens — the smallest meaningful units. For the string 3 + 4 * 2, the tokenizer (or lexer) emits numbers and operators.
This stage strips whitespace and classifies each character group, so the parser never deals with raw bytes.
A Token Type
We model each token with an enum tag and a payload. Numbers carry an integer value; operators and parentheses just need their kind.
Keeping the value inside the struct avoids re-scanning the source later.
typedef enum {
TOK_NUM, TOK_PLUS, TOK_MINUS,
TOK_STAR, TOK_SLASH,
TOK_LPAREN, TOK_RPAREN, TOK_EOF
} TokKind;
typedef struct {
TokKind kind;
int value; /* used when kind == TOK_NUM */
} Token;All lessons in this course
- Tokenizing Input
- Parsing Expressions
- Evaluating the Tree
- Adding Variables