NLTK でトークン化する

Time to upgrade. NLTK is a classic Python library that gives you a proper tokenizer for messy, real-world text. 🛠️

You install it once with pip, then import it in your script. From there, NLTK's word_tokenize is one function call away.

NLTK ships extra data separately. Before tokenizing, you download the punkt model once, and then it just works.

Now pass any string to word_tokenize. It returns a clean list of word and punctuation tokens, ready to count or filter.

Notice the win: the exclamation mark is now its own token. So cats and cats! finally count as the same word.

NLTK is smart about contractions. It splits don't into do and n't, keeping the hidden negation visible to your code.

Under the hood, NLTK follows linguistic rules learned from real text. That is why it beats a plain whitespace split every time.

NLTK also segments sentences. Pair sent_tokenize with word_tokenize to split a document into sentences, then each into words.

A common pattern loops over sentences and tokenizes each. This gives you a tidy list of lists, one token list per sentence.

NLTK is great for learning, but it is not alone. Libraries like spaCy offer faster tokenizers you will meet later on.

You now have the full move: raw text in, a clean token list out. This is the foundation every later NLP step builds on.

How does NLTK improve on naive splitting?

You used NLTK to tokenize real text: install, download punkt, then call word_tokenize. It splits punctuation and contractions cleanly for you.