R Academy · Lesson

Handling Pagination and Multiple Pages

Loop over paginated results and combine scraped data into a single dataset.

Handling Pagination and Multiple Pages is a free R Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the R Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Understanding Pagination

Many websites split content across multiple pages. URL-based pagination appends a page number to the URL (e.g., ?page=2 or /page/2/). Identify the pattern to construct URLs programmatically.

library(rvest)

# Common URL pagination patterns:
# Pattern 1: query parameter
# https://site.com/products?page=1
# https://site.com/products?page=2

# Pattern 2: path segment
# https://site.com/products/page/1/
# https://site.com/products/page/2/

# Build URLs for pages 1 to 5
base_url <- 'https://books.toscrape.com/catalogue/page-'
page_urls <- paste0(base_url, 1:5, '.html')
page_urls
# [1] 'https://books.toscrape.com/catalogue/page-1.html'
# [2] 'https://books.toscrape.com/catalogue/page-2.html' ...

Building URL Patterns with sprintf

sprintf() and paste0() both build URL strings, but sprintf() gives more control over zero-padding and formatting. Use whichever matches the site's URL structure.

# Various URL building approaches

# paste0 for simple concatenation
pages <- 1:5
urls_v1 <- paste0('https://site.com/list?p=', pages)
urls_v1[1]  # 'https://site.com/list?p=1'

# sprintf for formatted strings
urls_v2 <- sprintf('https://site.com/page/%02d/', pages)
urls_v2[1]  # 'https://site.com/page/01/'

# With multiple parameters
urls_v3 <- sprintf('https://site.com/items?page=%d&size=20', pages)
urls_v3[3]  # 'https://site.com/items?page=3&size=20'

# glue package alternative
# glue('https://site.com/page/{pages}')

Scraping Multiple Pages with map()

purrr::map() applies a scraping function over a vector of URLs, returning a list of results. Combine with dplyr::bind_rows() to merge into one data frame.

library(rvest)
library(purrr)

# Define a function to scrape one page
scrape_page <- function(url) {
  # In real code: page <- read_html(url)
  # Here we simulate with local HTML
  page <- read_html(sprintf(
    '<ul><li class="item">Item %d-A</li><li class="item">Item %d-B</li></ul>',
    url, url
  ))
  data.frame(
    title = html_text2(html_elements(page, '.item')),
    page  = url
  )
}

# Apply over pages 1 and 2 (simulated with numbers)
results <- map(1:2, scrape_page)
do.call(rbind, results)

Polite Scraping: Sys.sleep()

Sending rapid requests to a server is rude and may trigger rate limiting or IP bans. Add Sys.sleep() between requests to respect the server. A delay of 1-2 seconds is typically sufficient.

library(rvest)

# Polite scraping with delays between requests
scrape_with_delay <- function(urls, delay = 1.5) {
  results <- vector('list', length(urls))
  for (i in seq_along(urls)) {
    cat(sprintf('Scraping page %d of %d\n', i, length(urls)))
    # results[[i]] <- read_html(urls[[i]])
    Sys.sleep(delay)  # Wait before next request
  }
  results
}

# Also: randomize delay to appear more human-like
random_delay <- function(min = 1, max = 3) {
  Sys.sleep(runif(1, min = min, max = max))
}
random_delay()
cat('Always be polite to servers you scrape')

Setting a User Agent

Identify your scraper politely by setting a descriptive User-Agent header. Servers use this to identify bots. A good User-Agent includes your name and contact info so site owners can reach you.

library(rvest)

# Default user agent reveals R/rvest
# Better: set a descriptive, honest UA
ua <- paste0(
  'MyResearchBot/1.0 ',
  '(Academic research; ',
  'contact: researcher@university.edu)'
)

# With httr/httr2 for UA control:
# library(httr2)
# response <- request(url) |>
#   req_headers('User-Agent' = ua) |>
#   req_perform()
# html <- read_html(resp_body_string(response))

# Or use polite package for full compliance:
# library(polite)
# session <- bow('https://example.com', user_agent = ua)
cat('Honest UA: better for everyone')

Respecting robots.txt

robots.txt tells crawlers which paths are off-limits. The robotstxt package lets you check programmatically whether a URL is allowed before scraping it.

library(rvest)
# library(robotstxt)  # uncomment to use

# Check if scraping is allowed:
# rtxt <- robotstxt(domain = 'books.toscrape.com')
# rtxt$check(paths = '/catalogue/', bot = '*')
# TRUE means allowed

# Robots.txt typically looks like:
# User-agent: *
# Disallow: /admin/
# Disallow: /private/
# Allow: /catalogue/
# Crawl-delay: 2

# The polite package combines UA + robots.txt + delays:
# library(polite)
# session <- bow('https://books.toscrape.com')
# page <- scrape(session)
cat('Always check robots.txt before large-scale scraping')

Following Pagination Links

Instead of guessing URLs, follow the actual 'Next' link on each page. html_attr(html_element(page, 'a.next'), 'href') finds the next page link dynamically — handles non-sequential pagination patterns.

library(rvest)

# Pattern: follow 'next' links dynamically
scrape_all_pages <- function(start_url, max_pages = 10) {
  all_data <- list()
  url <- start_url
  i <- 1
  while (!is.null(url) && i <= max_pages) {
    # page <- read_html(url)
    # data <- extract_data(page)
    # all_data[[i]] <- data
    # next_link <- html_attr(html_element(page, 'li.next a'), 'href')
    # url <- if (!is.na(next_link)) xml2::url_absolute(next_link, url)
    #        else NULL
    cat(sprintf('Would scrape page %d: %s\n', i, url))
    url <- NULL  # Stop in this demo
    i <- i + 1
  }
  do.call(rbind, all_data)
}
scrape_all_pages('https://books.toscrape.com')

session_follow_link() with rvest

rvest's session() creates a stateful browsing session that maintains cookies and history. session_follow_link() navigates to a link — useful for sites requiring login or maintaining state across pages.

library(rvest)
# Stateful session with rvest
# s <- session('https://example.com/login')
# s <- session_submit(s,
#   html_form(read_html(s))[[1]],
#   list(username='me', password='pw')
# )
# After login, navigate:
# page2 <- session_follow_link(s, 'Products')
# page3 <- session_back(page2)   # go back
# session_history(page2)         # show history

# For simple cases, follow href directly:
# href <- html_attr(html_element(page, 'a.next'), 'href')
# next_page <- session_jump_to(s, href)
cat('session() maintains cookies across requests')

Detecting Last Page

Know when to stop: check if the 'Next' button exists, if the current page's item count is less than expected, or if the page number exceeds a known total. Handle edge cases gracefully.

library(rvest)
# Strategy 1: Check if 'Next' link exists
has_next <- function(page) {
  next_link <- html_element(page, 'li.next a')
  !is.na(html_attr(next_link, 'href'))
}

# Strategy 2: Compare item count to expected page size
page_size <- 20
check_done <- function(items_on_page) {
  items_on_page < page_size
}

# Strategy 3: Read total from pagination text
# e.g., 'Page 1 of 50' -> extract 50
extract_total_pages <- function(page) {
  text <- html_text2(html_element(page, '.current'))
  as.integer(gsub('.*of (\\d+).*', '\\1', text))
}
cat('Always plan your stopping condition')

Handling Scraping Errors

Network errors, timeouts, and 404s will happen. Wrap requests in tryCatch() to handle failures gracefully and log which pages failed, so you can retry them later.

library(rvest)

safe_scrape <- function(url) {
  tryCatch({
    # page <- read_html(url)
    # data <- extract_data(page)
    # return(data)
    cat(sprintf('Scraping: %s\n', url))
    data.frame(url = url, status = 'ok')
  }, error = function(e) {
    cat(sprintf('FAILED: %s -> %s\n', url, e$message))
    data.frame(url = url, status = 'error')
  })
}

urls <- c('https://example.com/page/1',
          'https://example.com/page/2')
results <- lapply(urls, safe_scrape)
do.call(rbind, results)

Caching Scraped Pages

Save downloaded HTML to disk before parsing. This way, if parsing fails, you don't need to re-download. Re-reads come from disk, not the network — polite and fast during development.

library(rvest)

scrape_cached <- function(url, cache_dir = 'cache') {
  dir.create(cache_dir, showWarnings = FALSE)
  # Create a filename from URL hash
  fname <- file.path(cache_dir,
    paste0(digest::digest(url), '.html'))
  if (file.exists(fname)) {
    cat('Loading from cache\n')
    return(read_html(fname))
  }
  cat('Downloading...\n')
  # page <- read_html(url)
  # xml2::write_html(page, fname)
  # return(page)
  cat('Cached to:', fname, '\n')
}

# Without digest: use URLencode or gsub
url <- 'https://example.com/page/1'
fname <- gsub('[^a-z0-9]', '_', tolower(url))
cat(fname)

Quick Check

Test your understanding of polite and robust multi-page scraping.

Recap: Pagination and Polite Scraping

Key takeaways: Build page URLs with paste0() or sprintf(). Use purrr::map() to apply scraping across many URLs. Always add Sys.sleep() between requests. Check robots.txt with the robotstxt package. Follow 'Next' links dynamically with html_attr() or session_follow_link(). Detect the last page by checking for a missing Next link. Wrap requests in tryCatch() for error resilience. Cache HTML to disk during development.

# Multi-page scraping template:
# urls <- paste0(base, 1:n_pages)
# results <- purrr::map(urls, function(url) {
#   Sys.sleep(runif(1, 1, 2))  # polite delay
#   page <- read_html(url)
#   extract_data(page)          # your parser
# })
# df <- dplyr::bind_rows(results)

# Key principles:
# 1. Respect robots.txt
# 2. Add delays (Sys.sleep)
# 3. Set honest User-Agent
# 4. Cache HTML during development
# 5. Handle errors with tryCatch
cat('Polite scraping = sustainable scraping')

Frequently asked questions

Is the “Handling Pagination and Multiple Pages” lesson free?

Yes — the full text of “Handling Pagination and Multiple Pages” is free to read here on the web, and the R Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the R Academy course, upgrade to CoddyKit PRO.

What will I learn in “Handling Pagination and Multiple Pages”?

Loop over paginated results and combine scraped data into a single dataset. You practise R Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start R Academy?

No prior experience is required. R Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Handling Pagination and Multiple Pages” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this R Academy lesson?

Yes. Every R Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to R Academy