Web Scraping & Bots · 课时

轮换用户代理与请求头

实施动态用户代理和 HTTP 请求头轮换，模拟合法浏览器流量并避免被检测。

轮换用户代理与请求头 是 CoddyKit 上的免费 Web Scraping & Bots 课时。这是第 1 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Web Scraping & Bots 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Web Scraping & Bots 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Avoiding Detection

When you scrape websites, they often try to detect if you're a human or a bot. If they think you're a bot, they might block you!

One common way websites spot bots is by looking at your HTTP headers. These headers contain information about your request.

Understanding HTTP Headers

Every time your browser (or a script) makes a request to a website, it sends HTTP headers. Think of them as metadata attached to your request.

User-Agent: Identifies your browser/OS.
Accept: What content types you prefer.
Referer: The previous page you were on.
Accept-Language: Your preferred language.

Your Digital Fingerprint

The User-Agent header is super important. It tells the web server what kind of client is making the request.

For example, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" identifies a Chrome browser on Windows.

Bots often use a generic User-Agent, or none at all, which is a big red flag!

Spotting Suspicious Patterns

If a website sees many requests from the same IP address, all using the exact same, generic User-Agent, it's easy to tell it's a bot.

Web servers can also analyze other headers. A real browser sends a rich set of headers, while a simple scraping script might send very few.

Building a User-Agent List

To mimic real browsers, you need a collection of diverse User-Agent strings. You can find these online!

Search for "list of user agents".
Extract them from browser requests.
Use libraries that maintain such lists.

Aim for a mix of different browsers (Chrome, Firefox, Safari) and operating systems.

Your First Custom Header

In Python, using the requests library, you can easily set custom headers for your requests. Let's start with a single User-Agent.

Try running this example:

import requests

url = "https://httpbin.org/headers"
headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
}

response = requests.get(url, headers=headers)
print(response.json()['headers']['User-Agent'])

Dynamic User-Agent Selection

To make your requests appear more natural, you should rotate your User-Agent. This means picking a different one for each request (or after a few requests).

We can use Python's random module to select a User-Agent from a list.

Try running this example:

import requests
import random

user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
]

url = "https://httpbin.org/headers"
random_ua = random.choice(user_agents)
headers = {"User-Agent": random_ua}

response = requests.get(url, headers=headers)
print(f"Used UA: {random_ua}")
print(response.json()['headers']['User-Agent'])

Beyond User-Agent

While User-Agent is crucial, real browsers send many other headers. Including a few more can make your bot look even more legitimate.

Accept-Language: e.g., en-US,en;q=0.9
Accept-Encoding: e.g., gzip, deflate, br
Connection: e.g., keep-alive

You can rotate these too, or pick a consistent set that matches your chosen User-Agent's browser type.

A More Complete Header Set

Let's combine random User-Agent selection with a few other common headers to create a more robust request.

This makes your scraping bot harder to distinguish from a regular browser.

Try running this example:

import requests
import random

user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
    "Mozilla/5.0 (iPhone; CPU iPhone OS 13_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1"
]

url = "https://httpbin.org/headers"
random_ua = random.choice(user_agents)

headers = {
    "User-Agent": random_ua,
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
}

response = requests.get(url, headers=headers)
print(f"Used UA: {random_ua}")
print("Received headers:")
for key, value in response.json()['headers'].items():
    print(f"  {key}: {value}")

Header Rotation Check

You've learned how User-Agents and other HTTP headers are used in web requests. Rotating these can help avoid detection.

Which of the following are good reasons to rotate User-Agent and other HTTP headers when scraping?

Recap: Blending In

Great job! You've learned how critical HTTP headers, especially the User-Agent, are for web scraping.

Websites use headers to identify clients.
Consistent, generic headers flag bots.
Rotating User-Agents and adding other common headers makes your bot harder to detect.

This is a powerful technique for blending in. Next, we'll explore proxy management to hide your IP address!

常见问题解答

「轮换用户代理与请求头」课时是免费的吗？

是的 — 「轮换用户代理与请求头」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Web Scraping & Bots 课程的其余内容，请升级到 CoddyKit PRO。 Web Scraping & Bots 课程共包含 4 节课。

「轮换用户代理与请求头」这节课中我会学到什么？

实施动态用户代理和 HTTP 请求头轮换，模拟合法浏览器流量并避免被检测。你通过在浏览器中直接运行的动手代码来练习 Web Scraping & Bots，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Web Scraping & Bots 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Web Scraping & Bots 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 1 节课，共 4 节。

「轮换用户代理与请求头」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Web Scraping & Bots 课中编写并运行代码吗？

能。每节 Web Scraping & Bots 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Web Scraping & Bots