空白で分割する方法とその限界

The easiest way to tokenize is to split on spaces. Python's split() method does exactly that with zero setup. ✂️

Called with no arguments, split() breaks on any run of whitespace: spaces, tabs, or newlines. Empty gaps are ignored automatically.

For clean, simple sentences, whitespace splitting is fast and good enough. It is a fine first step while you are learning the basics.

Here is the first crack: punctuation glues to words. Splitting cats! gives cats! as one token, not the clean word cats you wanted.

Now cats and cats! count as two different tokens. Your word counts split apart, and that quietly corrupts every later measurement.

Whitespace splitting keeps don't as one piece. It cannot see the hidden not inside, so negation gets lost completely.

A word like state-of-the-art stays glued as a single token. Sometimes that is right, but split() gives you no choice in the matter.

Many languages, like Chinese, write without spaces between words. Here whitespace splitting fails entirely, handing back one giant token.

Splitting alone keeps Cats and cats apart because of the capital. Whitespace splitting does no normalization for you at all.

These cracks add up. Dirty tokens mean wrong counts, and wrong counts mean a weaker model later in your pipeline.

Whitespace splitting is a great teacher but a poor finish. Real projects reach for a proper tokenizer that handles these edge cases.

Spot the weakness of naive splitting.

Whitespace split() is simple and fast, but it glues punctuation, misses contractions, and fails without spaces. Great to learn, not enough to ship.