0Pricing
Web Scraping & Bots · Lesson

Logging and Error Handling for Bots

Build resilient simple bots by adding structured logging and graceful error handling so failures are visible and recoverable.

Logging and Error Handling for Bots is a free Web Scraping & Bots lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Web Scraping & Bots learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Bots Need Logs

A bot runs unattended, often on a schedule. When something breaks, you are not watching. Logging leaves a trail so you can diagnose what happened after the fact.

Without logs, a silent failure can go unnoticed for days.

print() Is Not Logging

Beginners use print(), but it has no levels, no timestamps, and no easy way to redirect to a file. Python's logging module solves all of this.

print('something happened')  # no level, no time, hard to manage

Setting Up Logging

Configure logging once at startup. Set a format with a timestamp and a level, and choose where output goes.

import logging

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s %(levelname)s %(message)s'
)
log = logging.getLogger('bot')
log.info('Bot started')

Log Levels

Levels let you control verbosity:

  • DEBUG detailed tracing.
  • INFO normal progress.
  • WARNING something odd but recoverable.
  • ERROR an operation failed.
  • CRITICAL the bot cannot continue.
log.debug('fetched 12 rows')
log.warning('retrying after timeout')
log.error('login failed')

Logging to a File

For unattended bots, write logs to a file so you can review them later. A rotating handler caps file size.

from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler('bot.log', maxBytes=1_000_000, backupCount=3)
log.addHandler(handler)

Catching Exceptions

Wrap risky operations in try/except so one failure does not crash the whole run. Log the error with context.

try:
    submit_form(data)
except Exception as e:
    log.error('form submission failed: %s', e)

Logging Full Tracebacks

Use log.exception inside an except block to capture the full stack trace, which is invaluable for debugging.

try:
    risky()
except Exception:
    log.exception('unexpected error in risky()')

Retrying Transient Failures

Network blips are common. Catch them and retry a few times before giving up, logging each attempt.

for attempt in range(3):
    try:
        fetch()
        break
    except ConnectionError:
        log.warning('attempt %d failed, retrying', attempt + 1)
else:
    log.error('all attempts failed')

Failing Loudly

Silent failures are dangerous. For critical errors, escalate: send an alert email, write a marker file, or exit with a non-zero code so a scheduler notices.

import sys
if not data:
    log.critical('no data scraped, aborting')
    sys.exit(1)

Clean Shutdown

Always release resources even on error. A finally block (or context managers) guarantees browsers close and files flush regardless of what went wrong.

try:
    run_bot()
finally:
    driver.quit()
    log.info('bot finished')

Structured Logging Context

Attach context to each log line, such as the current URL or item id, so when you scan a long log you immediately know which record a message refers to.

log.info('scraped item %s from %s', item_id, url)

Quick Check

Test your understanding of bot logging and error handling.

Recap

You learned to make simple bots resilient: use the logging module instead of print, choose appropriate levels, log to rotating files, catch and log exceptions with tracebacks, retry transient errors, fail loudly on critical issues, and shut down cleanly.

Frequently asked questions

Is the “Logging and Error Handling for Bots” lesson free?

Yes — the full text of “Logging and Error Handling for Bots” is free to read here on the web, and the Web Scraping & Bots course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Web Scraping & Bots course, upgrade to CoddyKit PRO.

What will I learn in “Logging and Error Handling for Bots”?

Build resilient simple bots by adding structured logging and graceful error handling so failures are visible and recoverable. You practise Web Scraping & Bots with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Web Scraping & Bots?

No prior experience is required. Web Scraping & Bots on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Logging and Error Handling for Bots” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Web Scraping & Bots lesson?

Yes. Every Web Scraping & Bots lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Defining Bot Objectives
  2. Automating Simple Form Submissions
  3. Scheduling Basic Tasks
  4. Logging and Error Handling for Bots
← Back to Web Scraping & Bots