R Academy · Lesson

Tokenization and Stop Word Removal

Break text into tokens and filter uninformative words with anti_join().

What Is Text Mining?

Text mining (or text analytics) transforms unstructured text into structured data that can be analysed statistically. The tidytext package enables a tidy approach: each row is one token (word, bigram, sentence), making it compatible with dplyr and ggplot2.

library(tidytext)
library(dplyr)

# A simple text corpus
text_df <- tibble::tibble(
  doc_id = 1:3,
  text   = c(
    'The quick brown fox jumps over the lazy dog.',
    'Text mining with R is powerful and fun.',
    'Natural language processing enables many applications.'
  )
)

cat('Input: ', nrow(text_df), 'documents\n')
cat('Columns:', names(text_df), '\n')

unnest_tokens: Word Tokenisation

unnest_tokens(output, input) splits text into one-row-per-token format. By default it tokenises by word, converting to lowercase and stripping punctuation. The token argument supports 'words', 'ngrams', 'sentences', and more.

library(tidytext)
library(dplyr)

text_df <- tibble::tibble(
  doc_id = 1:2,
  text   = c(
    'R is a great language for data analysis.',
    'Text mining reveals hidden patterns in documents.'
  )
)

# Tokenise into words
tokens <- text_df |>
  unnest_tokens(word, text)

cat('Tokens extracted:', nrow(tokens), '\n')
print(tokens)

All lessons in this course

← Back to R Academy