0Pricing
AI Agents · Lesson

Handling Pagination and Dynamic Content

Following next-page links and using Playwright for JavaScript-rendered pages.

Handling Pagination and Dynamic Content is a free AI Agents lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Pagination Problem

Most real websites don't serve all their data on a single page. Results are split across multiple pages to reduce load. An agent must detect and follow pagination to collect complete datasets.

There are two common pagination patterns: next-page links and page number URL patterns.

Detecting Next-Page Links

Many sites have a 'Next' button or arrow link. Find the anchor tag with rel='next' or with text like 'Next', extract its href, and follow it in a loop until none exists.

from bs4 import BeautifulSoup
import httpx

def scrape_all_pages(start_url: str) -> list:
    results = []
    url = start_url

    while url:
        response = httpx.get(url, timeout=10.0)
        soup = BeautifulSoup(response.text, 'html.parser')

        # Collect data from this page
        for item in soup.select('.result-item'):
            results.append(item.text.strip())

        # Find the next page link
        next_link = soup.find('a', rel='next') or soup.find('a', string='Next')
        url = next_link.get('href') if next_link else None

    return results

Page Number URL Patterns

Many APIs and sites use predictable URL patterns: /results?page=1, /results?page=2, etc. An agent can iterate these directly without parsing the DOM for a next link.

import httpx

def scrape_paginated_api(base_url: str, max_pages: int = 10) -> list:
    all_items = []

    for page in range(1, max_pages + 1):
        response = httpx.get(
            base_url,
            params={'page': page, 'per_page': 50},
            timeout=10.0
        )
        response.raise_for_status()
        data = response.json()

        items = data.get('results', [])
        if not items:
            break  # No more data

        all_items.extend(items)
        print(f'Page {page}: fetched {len(items)} items')

    return all_items

Cursor-Based Pagination

Modern APIs (GitHub, Slack, Twitter) use cursor-based pagination. Each response includes a cursor token for the next page. Pass the cursor in the next request until it is absent or null.

import httpx

def fetch_with_cursor(url: str, headers: dict) -> list:
    all_data = []
    cursor = None

    while True:
        params = {'cursor': cursor} if cursor else {}
        response = httpx.get(url, headers=headers, params=params, timeout=10.0)
        response.raise_for_status()
        data = response.json()

        all_data.extend(data.get('items', []))

        cursor = data.get('next_cursor')  # None when done
        if not cursor:
            break

    return all_data

Static vs. Dynamic Content

Some pages load content via JavaScript after the initial HTML is delivered. A normal HTTP client receives only the empty shell — the actual data is injected by JS in the browser.

For these pages, tools like Playwright run a real browser engine and wait for JavaScript to execute before extracting content.

Installing and Importing Playwright

Install Playwright with pip install playwright then run playwright install chromium to download the browser. Use async_playwright() for async agent workflows.

# Installation (run in terminal):
# pip install playwright
# playwright install chromium

from playwright.async_api import async_playwright
import asyncio

async def fetch_dynamic(url: str) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url)
        html = await page.content()  # Full rendered HTML
        await browser.close()
    return html

# html = asyncio.run(fetch_dynamic('https://example.com'))

Waiting for Dynamic Content to Load

JavaScript pages often load data asynchronously. Tell Playwright to wait until a specific element appears before extracting content, ensuring the data you need is actually present in the DOM.

from playwright.async_api import async_playwright

async def scrape_dynamic_table(url: str) -> list:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url)

        # Wait for the table to appear in the DOM
        await page.wait_for_selector('table.results', timeout=10000)

        html = await page.content()
        await browser.close()

    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, 'html.parser')
    return [td.text for td in soup.select('table.results td')]

Clicking and Interacting with Pages

Playwright can simulate user interactions: clicks, form fills, scrolling, and keyboard input. This lets an agent navigate through login flows, dropdowns, or infinite scroll pages.

from playwright.async_api import async_playwright

async def click_load_more(url: str) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url)

        # Click the 'Load More' button
        load_more = page.locator('button:text("Load More")')
        if await load_more.count() > 0:
            await load_more.click()
            await page.wait_for_timeout(2000)  # wait 2s for content

        html = await page.content()
        await browser.close()
    return html

Handling Infinite Scroll

Some sites load more content as you scroll down. Playwright can simulate scrolling to trigger these loads. Scroll repeatedly and check if new content appeared each time.

from playwright.async_api import async_playwright

async def scrape_infinite_scroll(url: str, scroll_count: int = 5) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url)

        for _ in range(scroll_count):
            await page.evaluate('window.scrollTo(0, document.body.scrollHeight)')
            await page.wait_for_timeout(1500)  # wait for new content

        html = await page.content()
        await browser.close()
    return html

Combining Playwright with BeautifulSoup

After Playwright renders the page and returns the full HTML, feed it into BeautifulSoup for easy extraction. This combines the strengths of both libraries.

from playwright.async_api import async_playwright
from bs4 import BeautifulSoup

async def extract_js_rendered_data(url: str) -> list:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url)
        await page.wait_for_selector('.product-card')
        html = await page.content()
        await browser.close()

    soup = BeautifulSoup(html, 'html.parser')
    products = []
    for card in soup.select('.product-card'):
        products.append({
            'name': card.select_one('.name').text.strip(),
            'price': card.select_one('.price').text.strip()
        })
    return products

Choosing the Right Tool

Not every page needs Playwright. Use the simplest approach that works:

  • Static HTML: httpx + BeautifulSoup (fast, no browser overhead)
  • JSON API with pagination: httpx with page/cursor params
  • JS-rendered content: Playwright async (slower, more powerful)

Always start with the static approach and only reach for Playwright when needed.

Knowledge Check: Pagination and Dynamic Content

Test your understanding of pagination and dynamic content handling.

Recap: Pagination and Dynamic Content

Your agents can now handle the full spectrum of web content:

  • Follow next-page links or iterate URL page parameters
  • Use cursor tokens for modern API pagination
  • Use async_playwright() for JavaScript-rendered pages
  • Wait for elements with page.wait_for_selector()
  • Combine Playwright with BeautifulSoup for easy extraction

Always start with the simplest approach and escalate to Playwright only when the static approach falls short.

Frequently asked questions

Is the “Handling Pagination and Dynamic Content” lesson free?

Yes — the full text of “Handling Pagination and Dynamic Content” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Handling Pagination and Dynamic Content”?

Following next-page links and using Playwright for JavaScript-rendered pages. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Handling Pagination and Dynamic Content” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. HTTP Clients for Agents: httpx and requests
  2. Parsing HTML with BeautifulSoup
  3. Handling Pagination and Dynamic Content
  4. Respectful Scraping Practices
← Back to AI Agents