0Pricing
Python Academy · Lesson

Handling JavaScript and Anti-scraping Measures

Use Playwright/Selenium for JS-rendered content and rate limiting.

Handling JavaScript and Anti-scraping Measures is a free Python Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Python Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why JavaScript Matters

Modern websites render content with JavaScript after page load. Static HTML scrapers like requests+BS4 see the skeleton, not the rendered DOM.

# Static scraper sees:
# <div id="app"></div>

# Rendered page has:
# <div id="app">
#   <h1>Product List</h1>
#   <ul>... dynamically loaded items ...</ul>
# </div>

Playwright for Python

Playwright controls a real browser (Chromium, Firefox, WebKit) programmatically. It waits for JavaScript to execute before extracting content.

# pip install playwright
# playwright install
import asyncio
from playwright.async_api import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("https://example.com")
        title = await page.inner_text("h1")
        print(title)
        await browser.close()

asyncio.run(main())

Waiting for Elements

Use page.wait_for_selector() to wait until an element appears in the DOM before extracting data.

from playwright.async_api import async_playwright
import asyncio

async def scrape():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("https://spa-example.com")
        await page.wait_for_selector(".product-list")
        items = await page.query_selector_all(".product-item")
        texts = [await i.inner_text() for i in items]

Selenium

Selenium is an older browser automation library. Playwright is preferred for new projects due to better async support and speed.

# pip install selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://example.com")
elem = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, "h1"))
)
print(elem.text)
driver.quit()

Headless Mode

Run browsers in headless mode (no visible window) for server-side scraping.

from playwright.async_api import async_playwright
import asyncio

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)  # no window
        page = await browser.new_page()
        await page.goto("https://example.com")
        html = await page.content()
        await browser.close()

Intercepting Network Requests

Playwright can intercept and mock network requests, useful for blocking ads/trackers or capturing API responses directly.

from playwright.async_api import async_playwright
import asyncio

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()

        await page.route("**/*.{png,jpg,css}", lambda route: route.abort())
        await page.goto("https://example.com")   # loads 3x faster

Rate Limiting and Delays

Mimic human behaviour: add random delays between requests and limit the rate to avoid triggering anti-bot systems.

import time, random

def polite_scrape(urls):
    for url in urls:
        fetch(url)
        time.sleep(random.uniform(1, 3))  # 1-3 s random delay

User-Agent Rotation

Rotate User-Agent strings to avoid being blocked based on the default scraper UA.

import requests, random

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ...",
]

def fetch(url):
    return requests.get(url, headers={"User-Agent": random.choice(USER_AGENTS)})

Cookies and Sessions

Maintain session cookies to stay logged in across requests. Both requests Session and Playwright handle cookies automatically.

import requests

with requests.Session() as s:
    s.post("https://example.com/login", data={"user":"me","pass":"pw"})
    # session cookie kept automatically:
    dashboard = s.get("https://example.com/dashboard")

CAPTCHA and Legal Considerations

CAPTCHAs are anti-bot measures. Solving them programmatically is technically possible (2captcha, anti-captcha) but raises ethical and legal concerns. Always review the site's robots.txt and Terms of Service before scraping.

# Before scraping:
# 1. Check robots.txt: https://example.com/robots.txt
# 2. Review the Terms of Service for "scraping" or "automated access" clauses
# 3. Only scrape public data
# 4. Honour Crawl-delay directives
# 5. Cache responses to avoid repeated hits

Playwright Screenshot and PDF

Playwright can screenshot or print pages to PDF — useful for visual testing and archiving.

from playwright.async_api import async_playwright
import asyncio

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("https://example.com")
        await page.screenshot(path="page.png", full_page=True)
        await page.pdf(path="page.pdf")
        await browser.close()

Quick Check

Why does requests + BeautifulSoup fail on many modern websites?

Recap

Use Playwright or Selenium for JavaScript-rendered pages. Run in headless mode for servers. Add delays and rotate User-Agents to avoid blocks. Intercept network requests to block trackers. Always respect robots.txt and Terms of Service.

Frequently asked questions

Is the “Handling JavaScript and Anti-scraping Measures” lesson free?

Yes — the full text of “Handling JavaScript and Anti-scraping Measures” is free to read here on the web, and the Python Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Python Academy course, upgrade to CoddyKit PRO.

What will I learn in “Handling JavaScript and Anti-scraping Measures”?

Use Playwright/Selenium for JS-rendered content and rate limiting. You practise Python Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Python Academy?

No prior experience is required. Python Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Handling JavaScript and Anti-scraping Measures” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Python Academy lesson?

Yes. Every Python Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. HTTP Requests with requests and httpx
  2. Parsing HTML with BeautifulSoup
  3. Building a Scrapy Spider
  4. Handling JavaScript and Anti-scraping Measures
← Back to Python Academy