正規表現によるトークン化のテクニック

You can tokenize text by describing what a token looks like, then letting regex find every piece that fits your definition.

The pattern \w+ with findall grabs every run of word characters, giving you a clean list of words and ignoring the spaces between them.

The re.split function breaks text wherever a pattern appears. Splitting on whitespace turns a sentence into a list of rough tokens fast.

With a character class, re.split can break on many separators at once, like spaces, commas, and semicolons in a single sweep.

Sometimes punctuation matters. A pattern like \w+|[^\w\s] captures words and lone symbols separately, so nothing is silently dropped.

The pipe means or. The pattern cat|dog matches either word, letting one regex describe several token shapes you care about.

Numbers need their own rule. The pattern \d+\.?\d* matches whole numbers and decimals, so prices and amounts stay together as one token.

Naive splitting wrecks words like do not in its short form. A smarter token pattern can keep apostrophes inside a word where they belong.

The \b anchor marks a word boundary, the edge between a word and a non-word. It helps you grab whole words without grabbing neighbors.

A practical tokenizer often combines rules with alternation: match URLs, then numbers, then words, then symbols, in priority order.

If you reuse a pattern a lot, re.compile turns it into a reusable object. It reads cleaner and runs faster across many strings.

You want to break a string anywhere one or more spaces appear. Which function fits best?

You used findall, split, alternation, and compile to turn raw text into exactly the tokens you want. Regex gives you full control. 🎯