Web Scraping & Bots · 课时

模拟复杂用户流程

设计机器人遵循复杂的用户路径，在多个页面和交互之间导航以实现目标。

模拟复杂用户流程 是 CoddyKit 上的免费 Web Scraping & Bots 课时。这是第 2 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 Web Scraping & Bots 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Web Scraping & Bots 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Understanding User Journeys

Imagine a human browsing a website: they click links, fill forms, scroll, and wait for pages to load. This sequence of actions is a user journey.

For bots, simulating these journeys means programming a series of steps to achieve a specific goal, just like a human would.

Why Simulate Journeys?

Simulating complex user journeys is vital for:

Automated Testing: Ensuring multi-step processes (like checkout) work correctly.
Advanced Data Collection: Scraping data that's only accessible after several interactions.
Task Automation: Performing repetitive tasks that require navigating multiple pages.

It allows bots to go beyond simple page visits.

Selenium: Your Journey Tool

For simulating human-like interactions and navigating complex journeys, Selenium is our go-to tool. It allows your bot to control a real web browser.

Remember, we covered Selenium basics in an earlier lesson. Now we'll apply it to multi-step workflows.

Starting the Path

Every journey begins by navigating to the first page. You use driver.get() to tell Selenium which URL to open.

Let's open a sample website:

from selenium import webdriver
from selenium.webdriver.common.by import By

def main():
    driver = webdriver.Chrome()
    try:
        driver.get("http://quotes.toscrape.com/")
        print("Page Title:", driver.title)
    finally:
        driver.quit()

if __name__ == "__main__":
    main()

Clicking & Typing

Once on a page, your bot needs to interact. This often involves clicking buttons or links, and typing into input fields.

We use find_element() with locators (like By.LINK_TEXT or By.ID) to target elements, then click() or send_keys().

from selenium import webdriver
from selenium.webdriver.common.by import By
import time

def main():
    driver = webdriver.Chrome()
    try:
        driver.get("http://quotes.toscrape.com/")
        # Find the 'Login' link and click it
        login_link = driver.find_element(By.LINK_TEXT, "Login")
        login_link.click()
        print("Navigated to login page.")
        # Wait a bit to see the action
        time.sleep(2)
    finally:
        driver.quit()

if __name__ == "__main__":
    main()

Following the Flow

A complex journey involves moving between several pages. After clicking a link, Selenium automatically loads the new page. You can then continue interacting with elements on that new page.

Let's log in and then click the 'Quotes' link to go back to the main page.

from selenium import webdriver
from selenium.webdriver.common.by import By
import time

def main():
    driver = webdriver.Chrome()
    try:
        driver.get("http://quotes.toscrape.com/")
        # Go to login page
        driver.find_element(By.LINK_TEXT, "Login").click()
        time.sleep(1) # For demonstration

        # Fill login form (using dummy credentials)
        driver.find_element(By.NAME, "username").send_keys("test_user")
        driver.find_element(By.NAME, "password").send_keys("test_pass")
        driver.find_element(By.CSS_SELECTOR, "input[type='submit']").click()
        time.sleep(1) # For demonstration

        print("Logged in (or attempted). Current URL:", driver.current_url)
        # Now click 'Quotes' to go back home
        driver.find_element(By.LINK_TEXT, "Quotes").click()
        time.sleep(1)
        print("Back to home page. Title:", driver.title)
    finally:
        driver.quit()

if __name__ == "__main__":
    main()

Waiting for Elements

Web pages often load content dynamically. If your bot tries to interact with an element before it appears, it will fail. This is where explicit waits come in.

WebDriverWait combined with expected_conditions tells Selenium to wait for a specific element to be visible or clickable before proceeding.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def main():
    driver = webdriver.Chrome()
    try:
        driver.get("http://quotes.toscrape.com/login")
        # Wait until the username input field is visible
        username_field = WebDriverWait(driver, 10).until(
            EC.visibility_of_element_located((By.NAME, "username"))
        )
        print("Username field is ready.")
        username_field.send_keys("user")
        
        # We can also wait for a button to be clickable
        login_button = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, "input[type='submit']"))
        )
        print("Login button is clickable.")
        login_button.click()
        
    finally:
        driver.quit()

if __name__ == "__main__":
    main()

Branching Your Journey

Not all user journeys are linear. Sometimes your bot needs to make decisions, like checking if an item is in stock or if a certain message appears.

You can use if/else statements based on whether an element is present, its text content, or other attributes.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
import time

def main():
    driver = webdriver.Chrome()
    try:
        driver.get("http://quotes.toscrape.com/login")
        # Attempt to find an error message element
        try:
            error_message = driver.find_element(By.CLASS_NAME, "error")
            print("Error message found:", error_message.text)
            # If error, maybe go back or try again
        except NoSuchElementException:
            print("No error message initially.")
            # Proceed with login
            driver.find_element(By.NAME, "username").send_keys("user")
            driver.find_element(By.NAME, "password").send_keys("pass")
            driver.find_element(By.CSS_SELECTOR, "input[type='submit']").click()
            time.sleep(1)
            # After login attempt, check for error again
            try:
                error_message = driver.find_element(By.CLASS_NAME, "error")
                print("Login failed:", error_message.text)
            except NoSuchElementException:
                print("Login successful (or no error shown).")
    finally:
        driver.quit()

if __name__ == "__main__":
    main()

End-to-End Search Journey

Let's simulate a more complete journey: navigating to a search page, typing a query, clicking search, and then interacting with the results.

This example combines several techniques we've learned.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

def main():
    driver = webdriver.Chrome()
    try:
        driver.get("http://quotes.toscrape.com/")
        print("Starting at:", driver.title)

        # 1. Click on 'About' (simulating navigating to a feature)
        about_link = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.LINK_TEXT, "About"))
        )
        about_link.click()
        print("Navigated to About page. Title:", driver.title)
        time.sleep(1) # Observe the page

        # 2. Go back to home page (simulating returning to main task)
        driver.find_element(By.LINK_TEXT, "Quotes").click()
        print("Returned to Home page. Title:", driver.title)
        time.sleep(1) # Observe the page

        # 3. Find a quote by an author (simulating a search/filter interaction)
        # Let's try to click on a tag, e.g., 'love'
        love_tag = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href='/tag/love/']"))
        )
        love_tag.click()
        print("Filtered by 'love' tag. Title:", driver.title)
        time.sleep(2)
        
        # 4. Extract some data from the filtered results
        quotes = driver.find_elements(By.CLASS_NAME, "quote")
        print(f"Found {len(quotes)} quotes with 'love' tag.")
        if quotes:
            print("First quote text:", quotes[0].find_element(By.CLASS_NAME, "text").text)

    finally:
        driver.quit()

if __name__ == "__main__":
    main()

Journey Best Practices

To build robust and maintainable bot journeys:

Use Explicit Waits: Always wait for elements to be ready.
Descriptive Locators: Use IDs, names, or unique CSS selectors. Avoid fragile XPath if possible.
Error Handling: Use try-except blocks for unexpected elements or network issues.
Modularize Code: Break down complex journeys into smaller, reusable functions.

Journey Challenge

You are building a bot to navigate a multi-page checkout process. Which of the following are crucial techniques for ensuring your bot successfully completes the journey?

Journey Summary

In this lesson, you learned how to design bots that simulate complex user journeys. We covered:

Navigating multiple pages with clicks.
Handling dynamic content using explicit waits.
Implementing conditional logic for branching paths.

These skills are fundamental for building sophisticated and reliable web automation bots!

常见问题解答

「模拟复杂用户流程」课时是免费的吗？

是的 — 「模拟复杂用户流程」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Web Scraping & Bots 课程的其余内容，请升级到 CoddyKit PRO。 Web Scraping & Bots 课程共包含 4 节课。

「模拟复杂用户流程」这节课中我会学到什么？

设计机器人遵循复杂的用户路径，在多个页面和交互之间导航以实现目标。你通过在浏览器中直接运行的动手代码来练习 Web Scraping & Bots，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Web Scraping & Bots 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Web Scraping & Bots 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 2 节课，共 4 节。

「模拟复杂用户流程」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Web Scraping & Bots 课中编写并运行代码吗？

能。每节 Web Scraping & Bots 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 Web Scraping & Bots