C Academy · Lesson

strtok and Tokenizing

Split strings.

strtok and Tokenizing is a free C Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the C Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What is tokenizing?

Tokenizing means splitting a string into smaller pieces (tokens) separated by delimiter characters.

For example, splitting "a,b,c" on commas gives the tokens a, b, and c. C provides strtok for this.

#include <stdio.h>
#include <string.h>

int main(void) {
    char text[] = "a,b,c";
    char *token = strtok(text, ",");
    printf("First token: %s\n", token);
    return 0;
}

The strtok signature

char *strtok(char *str, const char *delim);

On the first call you pass the string. On subsequent calls you pass NULL so it continues where it left off. It returns the next token, or NULL when done.

#include <stdio.h>
#include <string.h>

int main(void) {
    char text[] = "red green blue";
    char *tok = strtok(text, " ");
    while (tok != NULL) {
        printf("%s\n", tok);
        tok = strtok(NULL, " ");
    }
    return 0;
}

strtok modifies the string

Important: strtok changes the original string. It replaces each delimiter with a '\0' and returns pointers into the same buffer.

This means you cannot tokenize a string literal like "a,b", which is read-only.

#include <stdio.h>
#include <string.h>

int main(void) {
    char data[] = "x:y:z";
    strtok(data, ":");
    printf("data[1] is now: %d (the null byte)\n", data[1]);
    return 0;
}

Make a writable copy

Because strtok destroys its input, copy the string first if you need the original later.

Use a local char array or strcpy into a buffer.

#include <stdio.h>
#include <string.h>

int main(void) {
    const char *original = "one-two-three";
    char copy[64];
    strcpy(copy, original);
    char *tok = strtok(copy, "-");
    while (tok) {
        printf("%s\n", tok);
        tok = strtok(NULL, "-");
    }
    printf("Original intact: %s\n", original);
    return 0;
}

Multiple delimiters

The delim argument is a set of characters, not a sequence. Any character in the set acts as a separator.

So " ,;" splits on spaces, commas, and semicolons all at once.

#include <stdio.h>
#include <string.h>

int main(void) {
    char text[] = "a,b;c d";
    char *tok = strtok(text, ",; ");
    while (tok) {
        printf("[%s]\n", tok);
        tok = strtok(NULL, ",; ");
    }
    return 0;
}

Consecutive delimiters

strtok treats runs of delimiters as a single separator and skips empty tokens.

So "a,,b" yields just a and b, not an empty token in between. Keep this in mind for CSV parsing where empty fields matter.

#include <stdio.h>
#include <string.h>

int main(void) {
    char text[] = "a,,b";
    char *tok = strtok(text, ",");
    while (tok) {
        printf("token: %s\n", tok);
        tok = strtok(NULL, ",");
    }
    return 0;
}

Counting tokens

A common task is to count how many tokens a string contains. Increment a counter inside the loop.

#include <stdio.h>
#include <string.h>

int main(void) {
    char sentence[] = "the quick brown fox";
    int count = 0;
    char *tok = strtok(sentence, " ");
    while (tok) {
        count++;
        tok = strtok(NULL, " ");
    }
    printf("Word count: %d\n", count);
    return 0;
}

strtok keeps hidden state

strtok stores its position in a static internal variable. That is why NULL continues the previous string.

The downside: it is not reentrant and not thread-safe. You cannot tokenize two strings at once.

#include <stdio.h>
#include <string.h>

int main(void) {
    char a[] = "1 2";
    char *t = strtok(a, " ");
    printf("%s\n", t);
    t = strtok(NULL, " ");
    printf("%s\n", t);
    return 0;
}

Storing tokens

To use tokens after splitting, store the returned pointers in an array. They point into the (now modified) source buffer, which must stay alive.

#include <stdio.h>
#include <string.h>

int main(void) {
    char csv[] = "name,age,city";
    char *fields[10];
    int n = 0;
    char *tok = strtok(csv, ",");
    while (tok && n < 10) {
        fields[n++] = tok;
        tok = strtok(NULL, ",");
    }
    for (int i = 0; i < n; i++)
        printf("Field %d: %s\n", i, fields[i]);
    return 0;
}

The reentrant alternative

POSIX offers strtok_r, which takes an explicit char **saveptr instead of hidden state. This makes it safe to nest and use across threads.

For portable standard C, plain strtok is fine for single-pass parsing.

#include <stdio.h>
#include <string.h>

int main(void) {
    char text[] = "p:q:r";
    char *save;
    char *tok = strtok_r(text, ":", &save);
    while (tok) {
        printf("%s\n", tok);
        tok = strtok_r(NULL, ":", &save);
    }
    return 0;
}

Practical: split a path

Tokenizing shines when parsing structured text like file paths. Splitting on / reveals each path component.

#include <stdio.h>
#include <string.h>

int main(void) {
    char path[] = "usr/local/bin";
    char *part = strtok(path, "/");
    while (part) {
        printf("-> %s\n", part);
        part = strtok(NULL, "/");
    }
    return 0;
}

Quick Check

Test your understanding of how strtok works.

Recap

You learned to split strings with strtok:

First call takes the string; later calls take NULL.
It modifies the input, replacing delimiters with '\0' — copy read-only strings first.
The delimiter argument is a set; consecutive delimiters are merged.
It uses static state and is not thread-safe; use strtok_r when needed.

Frequently asked questions

Is the “strtok and Tokenizing” lesson free?

Yes — the full text of “strtok and Tokenizing” is free to read here on the web, and the C Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the C Academy course, upgrade to CoddyKit PRO.

What will I learn in “strtok and Tokenizing”?

Split strings. You practise C Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start C Academy?

No prior experience is required. C Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “strtok and Tokenizing” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this C Academy lesson?

Yes. Every C Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to C Academy