R Academy · Lesson

Handling Pagination and Multiple Pages

Loop over paginated results and combine scraped data into a single dataset.

Understanding Pagination

Many websites split content across multiple pages. URL-based pagination appends a page number to the URL (e.g., ?page=2 or /page/2/). Identify the pattern to construct URLs programmatically.

library(rvest)

# Common URL pagination patterns:
# Pattern 1: query parameter
# https://site.com/products?page=1
# https://site.com/products?page=2

# Pattern 2: path segment
# https://site.com/products/page/1/
# https://site.com/products/page/2/

# Build URLs for pages 1 to 5
base_url <- 'https://books.toscrape.com/catalogue/page-'
page_urls <- paste0(base_url, 1:5, '.html')
page_urls
# [1] 'https://books.toscrape.com/catalogue/page-1.html'
# [2] 'https://books.toscrape.com/catalogue/page-2.html' ...

Building URL Patterns with sprintf

sprintf() and paste0() both build URL strings, but sprintf() gives more control over zero-padding and formatting. Use whichever matches the site's URL structure.

# Various URL building approaches

# paste0 for simple concatenation
pages <- 1:5
urls_v1 <- paste0('https://site.com/list?p=', pages)
urls_v1[1]  # 'https://site.com/list?p=1'

# sprintf for formatted strings
urls_v2 <- sprintf('https://site.com/page/%02d/', pages)
urls_v2[1]  # 'https://site.com/page/01/'

# With multiple parameters
urls_v3 <- sprintf('https://site.com/items?page=%d&size=20', pages)
urls_v3[3]  # 'https://site.com/items?page=3&size=20'

# glue package alternative
# glue('https://site.com/page/{pages}')

All lessons in this course

← Back to R Academy