Integrating with APIs
Connect your bots with external APIs to enrich data, trigger actions, or interact with other services.
Integrating with APIs is a free Web Scraping & Bots lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Web Scraping & Bots learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Bots & APIs: A Powerful Combo
You've learned to automate browser interactions and scrape web pages. But what if you need structured data, or to trigger actions on another service?
This is where APIs (Application Programming Interfaces) come in! APIs provide a direct, organized way for your bot to communicate with other software systems.
APIs vs. Web Scraping
While web scraping involves parsing HTML from web pages, APIs offer data in a more machine-readable format, typically JSON or XML.
- Web Scraping: Best for public data on websites without APIs, or complex UI interactions.
- API Integration: Ideal for structured data, performing actions (like creating a post), and when an API is explicitly provided.
Often, the most powerful bots combine both!
API Basics: Endpoints & Methods
APIs are accessed via endpoints, which are specific URLs. You interact with them using standard HTTP methods:
- GET: To retrieve data (like reading a post).
- POST: To send new data (like creating a post).
- PUT/PATCH: To update existing data.
- DELETE: To remove data.
For bots, GET and POST are the most common.
Making a Simple GET Request
Let's use Python's requests library to fetch some data from a public test API. We'll use jsonplaceholder.typicode.com which provides fake data.
Try running this example:
import requests
def main():
api_url = "https://jsonplaceholder.typicode.com/todos/1"
response = requests.get(api_url)
print(f"Status Code: {response.status_code}")
print(response.text)
if __name__ == "__main__":
main()Parsing JSON Responses
APIs commonly return data in JSON (JavaScript Object Notation) format. It's a lightweight, human-readable data interchange format.
The requests library can automatically parse JSON for you, turning it into a Python dictionary. Let's extract specific fields from our previous example.
import requests
import json
def main():
api_url = "https://jsonplaceholder.typicode.com/todos/1"
response = requests.get(api_url)
if response.status_code == 200:
todo_item = response.json() # Parse JSON into a Python dict
print(f"User ID: {todo_item['userId']}")
print(f"Title: {todo_item['title']}")
print(f"Completed: {todo_item['completed']}")
else:
print("Failed to fetch data.")
if __name__ == "__main__":
main()API Authentication Concepts
Many APIs require authentication to ensure only authorized users or bots can access their data or services. Common methods include:
- API Keys: A unique string sent with each request.
- Tokens: Often generated after a login, valid for a certain period.
- OAuth: A more complex standard for secure delegation of access.
For simple bots, API keys are often used.
Using API Keys in Requests
API keys are typically passed as a query parameter in the URL or as an HTTP header. requests makes this easy with the params argument.
This example shows how you would pass a key (using a fictional API for demonstration).
import requests
def main():
# This is a placeholder for demonstration.
# In a real scenario, use a valid API key.
api_key = "YOUR_DUMMY_API_KEY_HERE"
search_term = "bot automation"
# Fictional API endpoint requiring a key
api_url = "https://api.example.com/search"
params = {
"q": search_term,
"apiKey": api_key # API key as a query parameter
}
response = requests.get(api_url, params=params)
print(f"Request URL: {response.url}")
print(f"Status Code: {response.status_code}")
# In a real app, you'd process response.json()
if __name__ == "__main__":
main()Sending Data with POST Requests
If your bot needs to submit data, like creating a new entry or updating information, you'll use a POST request. The data payload is typically sent in the request body.
requests handles sending JSON data in the body automatically using the json parameter.
import requests
import json
def main():
api_url = "https://jsonplaceholder.typicode.com/posts"
new_post_data = {
"title": "Bot-Generated Post",
"body": "This is content created by our bot!",
"userId": 1
}
# The 'json' parameter automatically sets Content-Type
response = requests.post(api_url, json=new_post_data)
print(f"Status Code: {response.status_code}")
if response.status_code == 201: # 201 Created is common for successful POST
print("Post created successfully!")
print(response.json()) # API returns the created item
else:
print("Failed to create post.")
print(response.text)
if __name__ == "__main__":
main()Robust Error Handling for APIs
API calls can fail due to network issues, incorrect requests, or server errors. Your bot should handle these gracefully.
- Check
response.status_code(e.g.,200for OK,404for Not Found,500for Server Error). - Use
response.raise_for_status()to automatically raise anHTTPErrorfor bad responses. - Wrap calls in
try-exceptblocks to catch network-related errors.
import requests
def main():
# Intentional bad URL to show error handling
api_url = "https://jsonplaceholder.typicode.com/nonexistent_endpoint"
try:
response = requests.get(api_url, timeout=5) # Set a timeout
response.raise_for_status() # Raises HTTPError for 4xx/5xx responses
print(f"Status Code: {response.status_code}")
print("Successfully fetched data.")
except requests.exceptions.HTTPError as err:
print(f"HTTP Error occurred: {err}")
print(f"Status Code: {response.status_code}")
except requests.exceptions.ConnectionError as err:
print(f"Connection Error occurred: {err}")
except requests.exceptions.Timeout as err:
print(f"Timeout Error occurred: {err}")
except requests.exceptions.RequestException as err:
print(f"An unexpected error occurred: {err}")
if __name__ == "__main__":
main()Quick Check: API Interaction
Which of the following are common ways bots interact with APIs?
Recap & Next Steps
Great job! You've learned how to integrate your bots with APIs:
- APIs provide structured data and allow bots to trigger actions.
- You use GET for fetching and POST for sending data.
- Python's
requestslibrary simplifies API calls. - JSON is the common data format, easily parsed into Python dictionaries.
- Authentication (like API keys) is crucial for many APIs.
- Robust error handling is key for reliable bot operations.
Combining web scraping with API integration allows your bots to perform more complex and powerful workflows!
Frequently asked questions
Is the “Integrating with APIs” lesson free?
Yes — the full text of “Integrating with APIs” is free to read here on the web, and the Web Scraping & Bots course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Web Scraping & Bots course, upgrade to CoddyKit PRO.
What will I learn in “Integrating with APIs”?
Connect your bots with external APIs to enrich data, trigger actions, or interact with other services. You practise Web Scraping & Bots with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Web Scraping & Bots?
No prior experience is required. Web Scraping & Bots on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Integrating with APIs” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Web Scraping & Bots lesson?
Yes. Every Web Scraping & Bots lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.