文分割の基本

Sometimes you need whole sentences, not words. Sentence segmentation finds where one sentence ends and the next begins. 📑

The strongest signal is end punctuation: a period, a question mark, or an exclamation point usually closes a sentence.

You might just split on the period. For simple text it works, turning two clean sentences into a tidy list.

Here is the trap: not every period ends a sentence. Abbreviations like Dr. or U.S.A. use periods that fool a naive splitter.

Numbers like 3.14 contain a period too. Split blindly and you slice a single number into two broken fragments.

Dialogue makes it harder. A sentence can end inside quotation marks, so the closing quote may come after the period.

To decide if a period ends a sentence, you must look at the context around it: the next character, capitalization, and known abbreviations.

Many tasks work sentence by sentence. Summarizers, translators, and sentiment tools all rely on clean boundaries to work well.

Because the rules are subtle, real code uses a trained tool. NLTK's sent_tokenize handles abbreviations and edge cases for you.

A common order is sentences first, then words. You segment into sentences, then tokenize each one into its own word list.

Sentence rules differ by language. Some scripts use their own end marks, so a good tool is language-aware rather than hardcoded to English.

Why is splitting on every period risky?

Sentence segmentation finds sentence boundaries. Periods are clues but not proof, so abbreviations and decimals trip naive splits. Tools like sent_tokenize handle it.