Tokenizing Input
Turn text into tokens.
Tokenizing Input is a free C Academy lesson on CoddyKit. This is lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the C Academy learning path, and your progress syncs across the web and the CoddyKit app. The C Academy course includes 4 lessons in total.
What Is a Tokenizer?
An interpreter starts by turning raw text into tokens — the smallest meaningful units. For the string 3 + 4 * 2, the tokenizer (or lexer) emits numbers and operators.
This stage strips whitespace and classifies each character group, so the parser never deals with raw bytes.
A Token Type
We model each token with an enum tag and a payload. Numbers carry an integer value; operators and parentheses just need their kind.
Keeping the value inside the struct avoids re-scanning the source later.
typedef enum {
TOK_NUM, TOK_PLUS, TOK_MINUS,
TOK_STAR, TOK_SLASH,
TOK_LPAREN, TOK_RPAREN, TOK_EOF
} TokKind;
typedef struct {
TokKind kind;
int value; /* used when kind == TOK_NUM */
} Token;Scanning Position
The lexer walks the source with a cursor pointer. A small helper peeks at the current character without consuming it, returning '\0' at the end.
Pointer arithmetic keeps the scanner fast and simple.
static const char *src;
static char peek(void) {
return *src;
}
static char advance(void) {
return *src++;
}Skipping Whitespace
Before reading a token we discard spaces and tabs. The standard isspace from <ctype.h> handles every blank character.
Newlines count as whitespace here, so the expression can span lines.
#include <ctype.h>
static void skip_ws(void) {
while (isspace((unsigned char)peek()))
advance();
}Lexing a Number
When the cursor sits on a digit we accumulate consecutive digits into an integer. Multiplying by ten and adding each digit builds the value left to right.
The loop stops at the first non-digit, leaving the cursor ready for the next token.
static int lex_number(void) {
int n = 0;
while (isdigit((unsigned char)peek())) {
n = n * 10 + (advance() - '0');
}
return n;
}The next_token Function
The core routine skips whitespace, then dispatches on the current character. Digits become a TOK_NUM; each operator maps to its own kind.
Reaching the terminating NUL yields TOK_EOF, the signal to stop.
static Token next_token(void) {
skip_ws();
char c = peek();
if (c == '\0') return (Token){TOK_EOF, 0};
if (isdigit((unsigned char)c))
return (Token){TOK_NUM, lex_number()};
advance();
switch (c) {
case '+': return (Token){TOK_PLUS, 0};
case '-': return (Token){TOK_MINUS, 0};
case '*': return (Token){TOK_STAR, 0};
case '/': return (Token){TOK_SLASH, 0};
case '(': return (Token){TOK_LPAREN, 0};
case ')': return (Token){TOK_RPAREN, 0};
}
return (Token){TOK_EOF, 0};
}Running the Lexer
Here is a full program that tokenizes an expression and prints each token kind. Numbers also print their value.
Notice the loop ends when it reads TOK_EOF.
#include <stdio.h>
#include <ctype.h>
typedef enum { TOK_NUM, TOK_PLUS, TOK_STAR, TOK_EOF } TokKind;
typedef struct { TokKind kind; int value; } Token;
static const char *src;
static char peek(void){ return *src; }
static char advance(void){ return *src++; }
static Token next_token(void){
while (isspace((unsigned char)peek())) advance();
char c = peek();
if (c=='\0') return (Token){TOK_EOF,0};
if (isdigit((unsigned char)c)){
int n=0; while(isdigit((unsigned char)peek())) n=n*10+(advance()-'0');
return (Token){TOK_NUM,n};
}
advance();
if (c=='+') return (Token){TOK_PLUS,0};
return (Token){TOK_STAR,0};
}
int main(void){
src = "12 + 3 * 4";
Token t;
do {
t = next_token();
if (t.kind==TOK_NUM) printf("NUM %d\n", t.value);
else if (t.kind==TOK_PLUS) printf("PLUS\n");
else if (t.kind==TOK_STAR) printf("STAR\n");
else printf("EOF\n");
} while (t.kind != TOK_EOF);
return 0;
}One Token of Lookahead
Parsers usually need to inspect the upcoming token before consuming it. We store one token in a global current and refill it after each match.
This single-token lookahead is enough for our LL(1) grammar.
static Token current;
static void init_lexer(const char *s) {
src = s;
current = next_token();
}
static Token cur(void) { return current; }
static void bump(void) { current = next_token(); }Reporting Lexical Errors
An unknown character — say $ or @ — should not silently vanish. A robust lexer reports the offending byte and aborts.
Failing fast at the lexer keeps later stages from seeing garbage tokens.
#include <stdio.h>
#include <stdlib.h>
static void lex_error(char c) {
fprintf(stderr, "lex error: unexpected '%c'\n", c);
exit(1);
}Multi-Character Operators
Real languages have tokens like == or <=. To lex them we peek one extra character after the first.
If the next byte completes the operator we consume both; otherwise we emit the single-character form.
/* fragment: distinguish '=' from '==' */
if (peek() == '=') {
advance();
if (peek() == '=') { advance(); /* TOK_EQ */ }
else { /* TOK_ASSIGN */ }
}Why Tokens Matter
By collapsing characters into tokens, the parser works with a clean, typed stream. Operator precedence, grouping, and errors all become easier to reason about.
Next we feed these tokens into a recursive-descent parser.
Quick Check
Think about what the lexer produces for grouping symbols.
Recap
You built a lexer: a token type, a scanning cursor, whitespace skipping, number lexing, and a next_token dispatcher with one-token lookahead.
These tokens are the input to parsing, which builds a syntax tree from them.
Learn C with an AI tutor — free
Write and run real code in your browser, get instant help from a 24/7 AI tutor, and pick up where you left off on the web or in the app.
- Courses
- 39
- Lessons
- 144
Frequently Asked Questions
Is the “Tokenizing Input” lesson free?
Yes — the full text of “Tokenizing Input” is free to read here on the web. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the C Academy course, upgrade to CoddyKit PRO. The C Academy course includes 4 lessons in total.
What will I learn in “Tokenizing Input”?
Turn text into tokens. You practise C Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start C Academy?
No prior experience is required. C Academy on CoddyKit is structured for beginners through advanced learners, so you can start here or from the beginning and move at your own pace. This is lesson 1 of 4.
How long does the “Tokenizing Input” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this C Academy lesson?
Yes. Every C Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Tokenizing Input
- Parsing Expressions
- Evaluating the Tree
- Adding Variables