0Pricing
Learn AI with Python · Lesson

Paginating and Collecting Large Datasets

Handling pagination, next_cursor patterns, rate limiting with time.sleep, progress bars.

Paginating and Collecting Large Datasets is a free Learn AI with Python lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Pagination Exists

APIs rarely return millions of records in one response. They split results into pages so each response stays small and fast.

To collect a full dataset you must loop through every page and combine the results. This lesson covers the common pagination styles plus rate limiting and progress tracking.

Page-Number Pagination

The simplest scheme uses a page parameter. You increment it until the API returns an empty page.

import requests

all_items = []
page = 1
while True:
    resp = requests.get(url, params={"page": page, "per_page": 100}, timeout=10)
    resp.raise_for_status()
    items = resp.json()
    if not items:
        break
    all_items.extend(items)
    page += 1
print(len(all_items), "items collected")

Cursor / Next-URL Pagination

Many modern APIs return a next URL or cursor token pointing to the following page. You loop while a next link exists.

This is robust because the server controls where the next page starts.

all_items = []
next_url = "https://api.example.com/items?limit=100"
while next_url:
    resp = requests.get(next_url, timeout=10)
    resp.raise_for_status()
    data = resp.json()
    all_items.extend(data["results"])
    next_url = data.get("next")   # None when no more pages
print(len(all_items))

Respecting Rate Limits with time.sleep

APIs cap how many requests you may send per second or minute. Hammering them returns 429 Too Many Requests or gets you blocked.

Insert a small time.sleep between requests to stay polite and under the limit.

import time

while next_url:
    resp = requests.get(next_url, timeout=10)
    resp.raise_for_status()
    data = resp.json()
    all_items.extend(data["results"])
    next_url = data.get("next")
    time.sleep(0.5)   # 2 requests per second

Reading Rate-Limit Headers

Well-behaved APIs report your remaining quota in headers like X-RateLimit-Remaining and X-RateLimit-Reset.

You can read them to slow down only when you are close to the limit instead of always sleeping.

remaining = int(resp.headers.get("X-RateLimit-Remaining", 1))
if remaining < 5:
    reset = int(resp.headers.get("X-RateLimit-Reset", 1))
    print("Near limit, sleeping", reset, "s")
    time.sleep(reset)

Handling 429 with Backoff

If you do hit a 429, the response often includes a Retry-After header telling you how long to wait. Honor it before retrying.

resp = requests.get(url, timeout=10)
if resp.status_code == 429:
    wait = int(resp.headers.get("Retry-After", 5))
    time.sleep(wait)
    resp = requests.get(url, timeout=10)   # retry once

Progress Bars with tqdm

Long collection jobs feel stuck without feedback. tqdm wraps any iterable and prints a live progress bar with rate and ETA.

from tqdm import tqdm

for page in tqdm(range(1, 101), desc="Fetching"):
    resp = requests.get(url, params={"page": page}, timeout=10)
    all_items.extend(resp.json())
    time.sleep(0.2)

tqdm with Unknown Totals

For cursor pagination you do not know the total count up front. Use tqdm as a manual counter and call update() each loop.

from tqdm import tqdm

pbar = tqdm(desc="Pages")
while next_url:
    resp = requests.get(next_url, timeout=10)
    data = resp.json()
    all_items.extend(data["results"])
    next_url = data.get("next")
    pbar.update(1)
pbar.close()

Incremental Collection

For huge datasets, do not hold everything in memory. Incremental collection writes each page to disk as it arrives, so a crash does not lose all progress.

import json

with open("items.jsonl", "w") as f:
    while next_url:
        resp = requests.get(next_url, timeout=10)
        data = resp.json()
        for item in data["results"]:
            f.write(json.dumps(item) + "\n")
        next_url = data.get("next")

Resumable Collection

Save the last cursor or page number so a re-run can resume instead of starting over. This makes long jobs fault-tolerant.

import os

start_page = 1
if os.path.exists("checkpoint.txt"):
    start_page = int(open("checkpoint.txt").read()) + 1

for page in range(start_page, 1000):
    # ... fetch and store ...
    open("checkpoint.txt", "w").write(str(page))

Putting It Together

A production collection loop combines all the pieces: paginate, sleep for rate limits, show progress, write incrementally, and checkpoint.

from tqdm import tqdm
import time, json

with open("out.jsonl", "a") as f:
    next_url = "https://api.example.com/items?limit=100"
    pbar = tqdm(desc="Pages")
    while next_url:
        r = requests.get(next_url, timeout=10)
        r.raise_for_status()
        d = r.json()
        for it in d["results"]:
            f.write(json.dumps(it) + "\n")
        next_url = d.get("next")
        pbar.update(1)
        time.sleep(0.3)

Quick Check: Cursor Pagination

An API returns a "next" field that is a URL, or None on the last page.

Recap: Collecting Large Datasets

You can now gather datasets that span many pages:

  • Page-number and cursor (while next_url) pagination
  • time.sleep and rate-limit headers to avoid 429s
  • Retry-After backoff on throttling
  • tqdm progress bars for long jobs
  • Incremental, resumable writing for fault tolerance

Next: storing the collected data efficiently.

Frequently asked questions

Is the “Paginating and Collecting Large Datasets” lesson free?

Yes — the full text of “Paginating and Collecting Large Datasets” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.

What will I learn in “Paginating and Collecting Large Datasets”?

Handling pagination, next_cursor patterns, rate limiting with time.sleep, progress bars. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Learn AI with Python?

No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Paginating and Collecting Large Datasets” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Learn AI with Python lesson?

Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. REST API Fundamentals for Data Collection
  2. Paginating and Collecting Large Datasets
  3. Storing Collected Data Efficiently
  4. Working with Public Data APIs
← Back to Learn AI with Python