R Academy · Lesson

TF-IDF and Term Frequency Analysis

Score term importance across documents with bind_tf_idf().

What Is TF-IDF?

TF-IDF (Term Frequency–Inverse Document Frequency) measures how distinctive a word is to a specific document within a corpus. A high TF-IDF score means the word is frequent in that document but rare across all documents — a good indicator of the document's key topics.

# TF-IDF formula
# TF(w, d)  = count of w in document d / total words in d
# IDF(w)    = log(N / number of docs containing w)
# TF-IDF(w, d) = TF(w, d) * IDF(w)

# Manual example
N <- 4  # total documents
docs_with_word <- 1  # 'neural' appears in only 1 doc
tf <- 5 / 100  # word appears 5 times in 100-word doc
idf <- log(N / docs_with_word)
tfidf <- tf * idf

cat('TF:     ', round(tf, 4), '\n')
cat('IDF:    ', round(idf, 4), '\n')
cat('TF-IDF: ', round(tfidf, 4), '\n')

Preparing the Token Count

Before computing TF-IDF, you need a data frame with columns: document (document identifier), word (token), and n (count). Use unnest_tokens() followed by count(document, word).

library(tidytext)
library(dplyr)

# Simulate a 4-document corpus on tech topics
docs <- tibble::tibble(
  document = c(rep('ML', 3), rep('Stats', 3), rep('DB', 3), rep('Web', 3)),
  text = c(
    'machine learning neural networks training',
    'deep learning models gradient descent',
    'reinforcement learning reward policy agent',
    'probability distributions hypothesis testing',
    'bayesian inference regression analysis variance',
    'statistics sampling confidence intervals normal',
    'database sql joins indexes queries transactions',
    'relational tables primary keys foreign constraints',
    'nosql document store mongodb redis cassandra',
    'html css javascript frontend react components',
    'dom events browser api fetch requests',
    'responsive design flexbox grid layout'
  )
)

word_counts <- docs |>
  unnest_tokens(word, text) |>
  count(document, word, sort = TRUE)

cat('Document-word pairs:', nrow(word_counts), '\n')
print(head(word_counts, 8))

All lessons in this course

← Back to R Academy