適切な N-Gram の範囲を選ぶ

Picking an n-gram range is a balancing act. More context helps the model, but it can flood you with features and noise.

Each step up in n can multiply your vocabulary. This feature blow-up means slower training and a hungrier memory footprint.

Long n-grams often appear once and never again. These rare features rarely generalize and can nudge the model toward overfitting.

A great first move is (1, 2): unigrams plus bigrams. It catches negation and short phrases without exploding the feature count.

Reach for (1, 3) only when fixed three-word phrases matter, like a product name. Otherwise the extra columns rarely pay off.

Set min_df so a gram must appear in several documents to count. This quietly removes the one-off phrases that only add noise.

You can also hand the vectorizer a budget. max_features keeps only the top terms by frequency, holding the matrix to a fixed width.

Do not guess forever. Try a few ranges and compare validation scores. The data will tell you where the gains flatten out.

Drop ngram_range into a grid search and let scikit-learn test ranges for you, picking the option with the best cross-validated score.

If training accuracy soars but validation lags, your range may be too wide. Shrinking n often generalizes better on new text.

For most text tasks, ngram_range (1, 2) with a small min_df is a strong, fast baseline. Reach higher only when evidence demands it.

Your trigram model overfits and trains slowly. Which single change most directly fights the feature blow-up?

Start at (1, 2), prune with min_df, cap with max_features, and let validation pick the winner. You can now model context with n-grams. 🚀