Managing Sessions and Cookies
Learn how bots maintain state across requests using sessions, cookies, and tokens to power authenticated, multi-step workflows.
Managing Sessions and Cookies is a free Web Scraping & Bots lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Web Scraping & Bots learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
HTTP Is Stateless
Each HTTP request is independent: the server forgets you between calls. To build multi-step workflows, a bot must carry state itself.
Cookies and sessions are how that state travels between requests.
What a Cookie Is
A cookie is a small key-value pair the server sets via a Set-Cookie header. The client returns it on every subsequent request, letting the server recognize the same client.
Set-Cookie: session_id=abc123; Path=/; HttpOnlySession Objects
The requests.Session object automatically stores and resends cookies across requests, so a login persists for the rest of your workflow.
import requests
session = requests.Session()
session.post('https://site.com/login', data={'user': 'a', 'pass': 'b'})
# cookies now persist on subsequent calls
profile = session.get('https://site.com/profile')Inspecting Cookies
You can read what cookies a session holds. This helps debug why an authenticated request unexpectedly redirects to a login page.
for c in session.cookies:
print(c.name, '=', c.value)Setting Cookies Manually
Sometimes you obtain a cookie elsewhere (a browser export) and inject it into your session to skip the login step.
session.cookies.set('session_id', 'abc123', domain='site.com')CSRF Tokens
Forms often embed a hidden CSRF token that must be sent back on submit. Scrape it from the page first, then include it in your POST.
page = session.get('https://site.com/form')
token = extract_hidden(page.text, 'csrf_token')
session.post('https://site.com/form', data={'csrf_token': token, 'msg': 'hi'})Bearer Tokens
API-driven sites issue a token after login that you send in an Authorization header rather than as a cookie. Store it and attach it to every call.
token = login_resp.json()['access_token']
session.headers.update({'Authorization': 'Bearer ' + token})Persisting Sessions to Disk
To resume a workflow later without re-authenticating, serialize the cookie jar and reload it on the next run.
import pickle
with open('cookies.pkl', 'wb') as f:
pickle.dump(session.cookies, f)
# later
with open('cookies.pkl', 'rb') as f:
session.cookies.update(pickle.load(f))Handling Expiry
Sessions and tokens expire. Detect a redirect to login or a 401 response, then re-authenticate transparently before retrying the original request.
resp = session.get(url)
if resp.status_code == 401:
relogin(session)
resp = session.get(url)Sessions in Selenium
In a browser-driven bot, Selenium manages cookies for you. You can still read or transplant them between a requests.Session and the browser to share authentication.
for c in driver.get_cookies():
session.cookies.set(c['name'], c['value'])Security of Stored Sessions
A saved cookie jar is as sensitive as a password: anyone with it can act as you. Store session files with restricted permissions and never commit them to version control.
import os
os.chmod('cookies.pkl', 0o600) # owner read/write onlyQuick Check
Test your understanding of session management.
Recap
You learned how bots maintain state: cookies, requests.Session, CSRF and bearer tokens, persisting cookies to disk, handling expiry with re-login, and sharing auth between Selenium and requests.
Managing sessions is essential for any authenticated multi-step workflow.
Frequently asked questions
Is the “Managing Sessions and Cookies” lesson free?
Yes — the full text of “Managing Sessions and Cookies” is free to read here on the web, and the Web Scraping & Bots course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Web Scraping & Bots course, upgrade to CoddyKit PRO.
What will I learn in “Managing Sessions and Cookies”?
Learn how bots maintain state across requests using sessions, cookies, and tokens to power authenticated, multi-step workflows. You practise Web Scraping & Bots with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Web Scraping & Bots?
No prior experience is required. Web Scraping & Bots on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Managing Sessions and Cookies” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Web Scraping & Bots lesson?
Yes. Every Web Scraping & Bots lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Handling User Authentication
- Simulating Complex User Journeys
- Integrating with APIs
- Managing Sessions and Cookies