R Academy · Lesson

Tokenization and Stop Word Removal

Break text into tokens and filter uninformative words with anti_join().

Tokenization and Stop Word Removal is a free R Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the R Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Is Text Mining?

Text mining (or text analytics) transforms unstructured text into structured data that can be analysed statistically. The tidytext package enables a tidy approach: each row is one token (word, bigram, sentence), making it compatible with dplyr and ggplot2.

library(tidytext)
library(dplyr)

# A simple text corpus
text_df <- tibble::tibble(
  doc_id = 1:3,
  text   = c(
    'The quick brown fox jumps over the lazy dog.',
    'Text mining with R is powerful and fun.',
    'Natural language processing enables many applications.'
  )
)

cat('Input: ', nrow(text_df), 'documents\n')
cat('Columns:', names(text_df), '\n')

unnest_tokens: Word Tokenisation

unnest_tokens(output, input) splits text into one-row-per-token format. By default it tokenises by word, converting to lowercase and stripping punctuation. The token argument supports 'words', 'ngrams', 'sentences', and more.

library(tidytext)
library(dplyr)

text_df <- tibble::tibble(
  doc_id = 1:2,
  text   = c(
    'R is a great language for data analysis.',
    'Text mining reveals hidden patterns in documents.'
  )
)

# Tokenise into words
tokens <- text_df |>
  unnest_tokens(word, text)

cat('Tokens extracted:', nrow(tokens), '\n')
print(tokens)

The stop_words Dataset

tidytext ships a built-in stop_words tibble containing 1,149 common English stop words from three lexicons: SMART, Snowball, and onix. These words ("the", "is", "and"…) carry little semantic meaning and are typically removed before analysis.

library(tidytext)
library(dplyr)

# Inspect the stop_words dataset
cat('Total stop words:', nrow(stop_words), '\n')
cat('Lexicons:', paste(unique(stop_words$lexicon), collapse = ', '), '\n')

# First few stop words from each lexicon
stop_words |>
  group_by(lexicon) |>
  slice_head(n = 3) |>
  print()

anti_join: Removing Stop Words

anti_join(tokens, stop_words, by = 'word') removes all rows whose word matches any entry in stop_words. This is the standard tidytext pattern for stop word removal — clean, readable, and easily extendable.

library(tidytext)
library(dplyr)

text_df <- tibble::tibble(
  doc_id = 1:3,
  text   = c(
    'The quick brown fox jumps over the lazy dog',
    'A stitch in time saves nine important words',
     'To be or not to be that is the question'
  )
)

tokens <- text_df |>
  unnest_tokens(word, text)

cat('Before removing stop words:', nrow(tokens), '\n')

tokens_clean <- tokens |>
  anti_join(stop_words, by = 'word')

cat('After removing stop words:', nrow(tokens_clean), '\n')
print(tokens_clean)

count: Term Frequency

After tokenising and removing stop words, use count(word, sort = TRUE) to compute term frequency. This is the foundation of many text mining analyses — the most frequent meaningful words characterise the document's topics.

library(tidytext)
library(dplyr)

# Use Jane Austen's novels from janeaustenr package
# For demo: simulate a small corpus
corpus <- tibble::tibble(
  text = c(
    'data science involves statistics programming analysis',
    'machine learning algorithms data patterns training',
    'statistics probability distributions random variables data',
    'programming languages python r julia statistics',
    'analysis patterns distributions algorithms science'
  )
)

word_counts <- corpus |>
  unnest_tokens(word, text) |>
  anti_join(stop_words, by = 'word') |>
  count(word, sort = TRUE)

cat('Top 8 words:\n')
print(head(word_counts, 8))

Custom Stop Words

Domain-specific corpora often contain high-frequency words that are irrelevant (e.g. "patient", "doctor" in medical notes). Create a custom stop words tibble and bind it to the built-in lexicon using bind_rows().

library(tidytext)
library(dplyr)

# Domain-specific words to remove
custom_stops <- tibble::tibble(
  word   = c('data', 'analysis', 'r', 'using', 'also'),
  lexicon = 'custom'
)

# Combine with built-in stop words
all_stops <- bind_rows(stop_words, custom_stops)
cat('Total stop words after custom:', nrow(all_stops), '\n')

corpus <- tibble::tibble(
  text = c(
    'Using R for data analysis and machine learning models',
    'Statistical data analysis also involves data visualisation'
  )
)

clean_tokens <- corpus |>
  unnest_tokens(word, text) |>
  anti_join(all_stops, by = 'word') |>
  count(word, sort = TRUE)

print(clean_tokens)

Bigram Tokenisation

Set token = 'ngrams', n = 2 to tokenise into consecutive word pairs (bigrams). Bigrams capture phrases like "machine learning" or "data science" that individual words miss, giving richer context.

library(tidytext)
library(dplyr)

corpus <- tibble::tibble(
  doc_id = 1:2,
  text = c(
    'machine learning and deep learning are powerful techniques',
    'natural language processing uses machine learning methods'
  )
)

bigrams <- corpus |>
  unnest_tokens(bigram, text, token = 'ngrams', n = 2)

cat('Bigrams extracted:', nrow(bigrams), '\n')
print(bigrams)

# Count most common bigrams
bigram_counts <- bigrams |> count(bigram, sort = TRUE)
cat('\nTop bigrams:\n')
print(head(bigram_counts, 5))

Filtering Bigrams for Quality

Most frequent bigrams include stop word pairs like "of the". Separate the bigram into two columns with tidyr::separate(), filter out rows where either word is a stop word, then re-unite to get meaningful phrase pairs.

library(tidytext)
library(dplyr)
library(tidyr)

corpus <- tibble::tibble(
  text = c(
    'the field of machine learning and deep learning is growing',
    'natural language processing is a subfield of artificial intelligence'
  )
)

bigrams_filtered <- corpus |>
  unnest_tokens(bigram, text, token = 'ngrams', n = 2) |>
  separate(bigram, into = c('word1', 'word2'), sep = ' ') |>
  filter(
    !word1 %in% stop_words$word,
    !word2 %in% stop_words$word
  ) |>
  unite(bigram, word1, word2, sep = ' ') |>
  count(bigram, sort = TRUE)

print(bigrams_filtered)

Visualising Word Frequencies

Plot word frequencies as a bar chart using ggplot2. Sort bars with reorder(word, n) and use coord_flip() for horizontal labels. This is one of the most communicative outputs in text mining.

library(tidytext)
library(dplyr)
library(ggplot2)

corpus <- tibble::tibble(
  text = c(
    'statistics probability machine learning algorithms patterns',
    'data science programming analysis visualisation models',
    'machine learning deep learning neural networks patterns',
    'probability statistics hypothesis testing distributions',
    'algorithms optimisation gradient descent models training'
  )
)

top_words <- corpus |>
  unnest_tokens(word, text) |>
  anti_join(stop_words, by = 'word') |>
  count(word, sort = TRUE) |>
  slice_head(n = 10)

ggplot(top_words, aes(reorder(word, n), n)) +
  geom_col(fill = 'steelblue') +
  coord_flip() +
  labs(x = 'Word', y = 'Count', title = 'Top 10 Terms') +
  theme_minimal()

Per-Document Word Counts

When your corpus has multiple documents, add a document identifier column and group by doc_id, word to compute per-document term frequencies. This feeds directly into TF-IDF and topic model analyses.

library(tidytext)
library(dplyr)

docs <- tibble::tibble(
  doc_id = c(1, 1, 1, 2, 2, 2, 3, 3, 3),
  text   = c(
    'machine learning models training',
    'neural networks deep learning',
    'algorithms gradient descent optimisation',
    'statistical analysis probability distributions',
    'hypothesis testing confidence intervals regression',
    'bayesian inference prior posterior likelihood',
    'data visualisation plots charts dashboards',
    'ggplot2 ggvis plotly interactive graphics',
    'colour scales aesthetics themes layers'
  )
)

word_counts <- docs |>
  unnest_tokens(word, text) |>
  anti_join(stop_words, by = 'word') |>
  count(doc_id, word, sort = TRUE)

cat('Word-doc pairs:', nrow(word_counts), '\n')
print(head(word_counts, 10))

Sentence Tokenisation

Set token = 'sentences' to split text at sentence boundaries. Sentence-level tokens are useful for sentiment analysis (score each sentence), summarisation (select representative sentences), and question-answering tasks.

library(tidytext)
library(dplyr)

doc <- tibble::tibble(
  id   = 1L,
  text = paste(
    'R is a statistical programming language.',
    'It is widely used in data science and machine learning.',
    'The tidyverse makes data manipulation easy.',
    'Text mining with tidytext is elegant and powerful.'
  )
)

sentences <- doc |>
  unnest_tokens(sentence, text, token = 'sentences')

cat('Sentences found:', nrow(sentences), '\n')
print(sentences$sentence)

Quick Check

You have tokenised a corpus into words using unnest_tokens() and want to remove common English words. Which operation achieves this using the built-in stop_words dataset?

Recap: Tokenisation and Stop Words

Key takeaways:

unnest_tokens(word, text) converts raw text into one-row-per-word tidy format
Default: lowercase, strip punctuation; supports 'words', 'ngrams', 'sentences'
stop_words is a built-in tibble with 1,149 common English words from 3 lexicons
anti_join(tokens, stop_words, by = 'word') removes stop words
Combine with bind_rows() to add custom domain-specific stop words
count(word, sort = TRUE) computes term frequency from tidy tokens
Bigrams: unnest_tokens(bigram, text, token = 'ngrams', n = 2)

library(tidytext)
library(dplyr)

tibble::tibble(text = 'The quick brown fox jumps over the lazy dog') |>
  unnest_tokens(word, text) |>
  anti_join(stop_words, by = 'word') |>
  count(word, sort = TRUE) |>
  print()

Frequently asked questions

Is the “Tokenization and Stop Word Removal” lesson free?

Yes — the full text of “Tokenization and Stop Word Removal” is free to read here on the web, and the R Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the R Academy course, upgrade to CoddyKit PRO.

What will I learn in “Tokenization and Stop Word Removal”?

Break text into tokens and filter uninformative words with anti_join(). You practise R Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start R Academy?

No prior experience is required. R Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Tokenization and Stop Word Removal” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this R Academy lesson?

Yes. Every R Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to R Academy