C Academy · Lesson

strtok and Tokenizing

Split strings.

Lesson 3 of 413 steps

strtok and Tokenizing is a free C Academy lesson on CoddyKit. This is lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the C Academy learning path, and your progress syncs across the web and the CoddyKit app. The C Academy course includes 4 lessons in total.

What is tokenizing?

Tokenizing means splitting a string into smaller pieces (tokens) separated by delimiter characters.

For example, splitting "a,b,c" on commas gives the tokens a, b, and c. C provides strtok for this.

#include <stdio.h>
#include <string.h>

int main(void) {
    char text[] = "a,b,c";
    char *token = strtok(text, ",");
    printf("First token: %s\n", token);
    return 0;
}

The strtok signature

char *strtok(char *str, const char *delim);

On the first call you pass the string. On subsequent calls you pass NULL so it continues where it left off. It returns the next token, or NULL when done.

#include <stdio.h>
#include <string.h>

int main(void) {
    char text[] = "red green blue";
    char *tok = strtok(text, " ");
    while (tok != NULL) {
        printf("%s\n", tok);
        tok = strtok(NULL, " ");
    }
    return 0;
}

strtok modifies the string

Important: strtok changes the original string. It replaces each delimiter with a '\0' and returns pointers into the same buffer.

This means you cannot tokenize a string literal like "a,b", which is read-only.

#include <stdio.h>
#include <string.h>

int main(void) {
    char data[] = "x:y:z";
    strtok(data, ":");
    printf("data[1] is now: %d (the null byte)\n", data[1]);
    return 0;
}

Make a writable copy

Because strtok destroys its input, copy the string first if you need the original later.

Use a local char array or strcpy into a buffer.

#include <stdio.h>
#include <string.h>

int main(void) {
    const char *original = "one-two-three";
    char copy[64];
    strcpy(copy, original);
    char *tok = strtok(copy, "-");
    while (tok) {
        printf("%s\n", tok);
        tok = strtok(NULL, "-");
    }
    printf("Original intact: %s\n", original);
    return 0;
}

Multiple delimiters

The delim argument is a set of characters, not a sequence. Any character in the set acts as a separator.

So " ,;" splits on spaces, commas, and semicolons all at once.

#include <stdio.h>
#include <string.h>

int main(void) {
    char text[] = "a,b;c d";
    char *tok = strtok(text, ",; ");
    while (tok) {
        printf("[%s]\n", tok);
        tok = strtok(NULL, ",; ");
    }
    return 0;
}

Consecutive delimiters

strtok treats runs of delimiters as a single separator and skips empty tokens.

So "a,,b" yields just a and b, not an empty token in between. Keep this in mind for CSV parsing where empty fields matter.

#include <stdio.h>
#include <string.h>

int main(void) {
    char text[] = "a,,b";
    char *tok = strtok(text, ",");
    while (tok) {
        printf("token: %s\n", tok);
        tok = strtok(NULL, ",");
    }
    return 0;
}

Counting tokens

A common task is to count how many tokens a string contains. Increment a counter inside the loop.

#include <stdio.h>
#include <string.h>

int main(void) {
    char sentence[] = "the quick brown fox";
    int count = 0;
    char *tok = strtok(sentence, " ");
    while (tok) {
        count++;
        tok = strtok(NULL, " ");
    }
    printf("Word count: %d\n", count);
    return 0;
}

strtok keeps hidden state

strtok stores its position in a static internal variable. That is why NULL continues the previous string.

The downside: it is not reentrant and not thread-safe. You cannot tokenize two strings at once.

#include <stdio.h>
#include <string.h>

int main(void) {
    char a[] = "1 2";
    char *t = strtok(a, " ");
    printf("%s\n", t);
    t = strtok(NULL, " ");
    printf("%s\n", t);
    return 0;
}

Storing tokens

To use tokens after splitting, store the returned pointers in an array. They point into the (now modified) source buffer, which must stay alive.

#include <stdio.h>
#include <string.h>

int main(void) {
    char csv[] = "name,age,city";
    char *fields[10];
    int n = 0;
    char *tok = strtok(csv, ",");
    while (tok && n < 10) {
        fields[n++] = tok;
        tok = strtok(NULL, ",");
    }
    for (int i = 0; i < n; i++)
        printf("Field %d: %s\n", i, fields[i]);
    return 0;
}

The reentrant alternative

POSIX offers strtok_r, which takes an explicit char **saveptr instead of hidden state. This makes it safe to nest and use across threads.

For portable standard C, plain strtok is fine for single-pass parsing.

#include <stdio.h>
#include <string.h>

int main(void) {
    char text[] = "p:q:r";
    char *save;
    char *tok = strtok_r(text, ":", &save);
    while (tok) {
        printf("%s\n", tok);
        tok = strtok_r(NULL, ":", &save);
    }
    return 0;
}

Practical: split a path

Tokenizing shines when parsing structured text like file paths. Splitting on / reveals each path component.

#include <stdio.h>
#include <string.h>

int main(void) {
    char path[] = "usr/local/bin";
    char *part = strtok(path, "/");
    while (part) {
        printf("-> %s\n", part);
        part = strtok(NULL, "/");
    }
    return 0;
}

Quick Check

Test your understanding of how strtok works.

Answer it in the interactive lesson

Recap

You learned to split strings with strtok:

First call takes the string; later calls take NULL.
It modifies the input, replacing delimiters with '\0' — copy read-only strings first.
The delimiter argument is a set; consecutive delimiters are merged.
It uses static state and is not thread-safe; use strtok_r when needed.

Free to start

Learn C with an AI tutor — free

Write and run real code in your browser, get instant help from a 24/7 AI tutor, and pick up where you left off on the web or in the app.

Courses: 39
Lessons: 144

Start learning free View the full course

Frequently Asked Questions

Is the “strtok and Tokenizing” lesson free?

Yes — the full text of “strtok and Tokenizing” is free to read here on the web. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the C Academy course, upgrade to CoddyKit PRO. The C Academy course includes 4 lessons in total.

What will I learn in “strtok and Tokenizing”?

Split strings. You practise C Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start C Academy?

No prior experience is required. C Academy on CoddyKit is structured for beginners through advanced learners, so you can start here or from the beginning and move at your own pace. This is lesson 3 of 4.

How long does the “strtok and Tokenizing” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this C Academy lesson?

Yes. Every C Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to C Academy