Tokenizing Input
Turn text into tokens.
Tokenizing Input is a free C Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the C Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is a Tokenizer?
An interpreter starts by turning raw text into tokens — the smallest meaningful units. For the string 3 + 4 * 2, the tokenizer (or lexer) emits numbers and operators.
This stage strips whitespace and classifies each character group, so the parser never deals with raw bytes.
A Token Type
We model each token with an enum tag and a payload. Numbers carry an integer value; operators and parentheses just need their kind.
Keeping the value inside the struct avoids re-scanning the source later.
typedef enum {
TOK_NUM, TOK_PLUS, TOK_MINUS,
TOK_STAR, TOK_SLASH,
TOK_LPAREN, TOK_RPAREN, TOK_EOF
} TokKind;
typedef struct {
TokKind kind;
int value; /* used when kind == TOK_NUM */
} Token;Scanning Position
The lexer walks the source with a cursor pointer. A small helper peeks at the current character without consuming it, returning '\0' at the end.
Pointer arithmetic keeps the scanner fast and simple.
static const char *src;
static char peek(void) {
return *src;
}
static char advance(void) {
return *src++;
}Skipping Whitespace
Before reading a token we discard spaces and tabs. The standard isspace from <ctype.h> handles every blank character.
Newlines count as whitespace here, so the expression can span lines.
#include <ctype.h>
static void skip_ws(void) {
while (isspace((unsigned char)peek()))
advance();
}Lexing a Number
When the cursor sits on a digit we accumulate consecutive digits into an integer. Multiplying by ten and adding each digit builds the value left to right.
The loop stops at the first non-digit, leaving the cursor ready for the next token.
static int lex_number(void) {
int n = 0;
while (isdigit((unsigned char)peek())) {
n = n * 10 + (advance() - '0');
}
return n;
}The next_token Function
The core routine skips whitespace, then dispatches on the current character. Digits become a TOK_NUM; each operator maps to its own kind.
Reaching the terminating NUL yields TOK_EOF, the signal to stop.
static Token next_token(void) {
skip_ws();
char c = peek();
if (c == '\0') return (Token){TOK_EOF, 0};
if (isdigit((unsigned char)c))
return (Token){TOK_NUM, lex_number()};
advance();
switch (c) {
case '+': return (Token){TOK_PLUS, 0};
case '-': return (Token){TOK_MINUS, 0};
case '*': return (Token){TOK_STAR, 0};
case '/': return (Token){TOK_SLASH, 0};
case '(': return (Token){TOK_LPAREN, 0};
case ')': return (Token){TOK_RPAREN, 0};
}
return (Token){TOK_EOF, 0};
}Running the Lexer
Here is a full program that tokenizes an expression and prints each token kind. Numbers also print their value.
Notice the loop ends when it reads TOK_EOF.
#include <stdio.h>
#include <ctype.h>
typedef enum { TOK_NUM, TOK_PLUS, TOK_STAR, TOK_EOF } TokKind;
typedef struct { TokKind kind; int value; } Token;
static const char *src;
static char peek(void){ return *src; }
static char advance(void){ return *src++; }
static Token next_token(void){
while (isspace((unsigned char)peek())) advance();
char c = peek();
if (c=='\0') return (Token){TOK_EOF,0};
if (isdigit((unsigned char)c)){
int n=0; while(isdigit((unsigned char)peek())) n=n*10+(advance()-'0');
return (Token){TOK_NUM,n};
}
advance();
if (c=='+') return (Token){TOK_PLUS,0};
return (Token){TOK_STAR,0};
}
int main(void){
src = "12 + 3 * 4";
Token t;
do {
t = next_token();
if (t.kind==TOK_NUM) printf("NUM %d\n", t.value);
else if (t.kind==TOK_PLUS) printf("PLUS\n");
else if (t.kind==TOK_STAR) printf("STAR\n");
else printf("EOF\n");
} while (t.kind != TOK_EOF);
return 0;
}One Token of Lookahead
Parsers usually need to inspect the upcoming token before consuming it. We store one token in a global current and refill it after each match.
This single-token lookahead is enough for our LL(1) grammar.
static Token current;
static void init_lexer(const char *s) {
src = s;
current = next_token();
}
static Token cur(void) { return current; }
static void bump(void) { current = next_token(); }Reporting Lexical Errors
An unknown character — say $ or @ — should not silently vanish. A robust lexer reports the offending byte and aborts.
Failing fast at the lexer keeps later stages from seeing garbage tokens.
#include <stdio.h>
#include <stdlib.h>
static void lex_error(char c) {
fprintf(stderr, "lex error: unexpected '%c'\n", c);
exit(1);
}Multi-Character Operators
Real languages have tokens like == or <=. To lex them we peek one extra character after the first.
If the next byte completes the operator we consume both; otherwise we emit the single-character form.
/* fragment: distinguish '=' from '==' */
if (peek() == '=') {
advance();
if (peek() == '=') { advance(); /* TOK_EQ */ }
else { /* TOK_ASSIGN */ }
}Why Tokens Matter
By collapsing characters into tokens, the parser works with a clean, typed stream. Operator precedence, grouping, and errors all become easier to reason about.
Next we feed these tokens into a recursive-descent parser.
Quick Check
Think about what the lexer produces for grouping symbols.
Recap
You built a lexer: a token type, a scanning cursor, whitespace skipping, number lexing, and a next_token dispatcher with one-token lookahead.
These tokens are the input to parsing, which builds a syntax tree from them.
Frequently asked questions
Is the “Tokenizing Input” lesson free?
Yes — the full text of “Tokenizing Input” is free to read here on the web, and the C Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the C Academy course, upgrade to CoddyKit PRO.
What will I learn in “Tokenizing Input”?
Turn text into tokens. You practise C Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start C Academy?
No prior experience is required. C Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Tokenizing Input” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this C Academy lesson?
Yes. Every C Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Tokenizing Input
- Parsing Expressions
- Evaluating the Tree
- Adding Variables