Vision Models for Screen Understanding
Send screenshots to a multimodal model so the agent 'sees' what a human would see.
Vision Models for Screen Understanding is a free AI Agents lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Vision?
Some web actions are hard to express with selectors:
- Captchas (well, attempts)
- Visually-positioned elements (chat bubbles, dynamic UIs)
- Apps with no stable DOM (Canvas / WebGL)
Multimodal LLMs let the agent SEE the screen.
Take a Screenshot, Send to LLM
import base64
page.screenshot(path='screen.png')
with open('screen.png', 'rb') as f:
b64 = base64.b64encode(f.read()).decode()
response = client.chat.completions.create(
model='gpt-4o',
messages=[{
'role': 'user',
'content': [
{'type': 'text', 'text': 'What buttons are visible? Return JSON.'},
{'type': 'image_url', 'image_url': {'url': f'data:image/png;base64,{b64}'}}
]
}]
)Anthropic Vision
response = client.messages.create(
model='claude-sonnet-4-5',
max_tokens=1024,
messages=[{
'role': 'user',
'content': [
{'type': 'image', 'source': {'type': 'base64', 'media_type': 'image/png', 'data': b64}},
{'type': 'text', 'text': 'Describe what you see.'}
]
}]
)Vision Quality
GPT-4o and Claude Sonnet 4.5 can:
- Describe screenshots accurately
- Read text in images (OCR)
- Identify UI elements by position
- Spot patterns and anomalies
They still struggle with very small text, complex tables, and precise pixel coordinates.
Locating Elements by Description
Ask the model "where is the Submit button?" and it returns approximate coordinates or instructions:
prompt = 'Look at the screenshot. Where is the Submit button? Return the approximate (x, y) coordinates of its center.'
# Response: {"x": 420, "y": 530}Coordinate Precision Caveat
Models are not pixel-perfect. Treat their coordinates as hints. For precise clicks, combine with DOM selectors when possible.
SoM (Set of Marks)
Annotate the screenshot with numbered boxes around interactive elements. Ask the model "which number do you want to click?":
- Identify interactive elements via DOM
- Draw numbered overlays
- Send annotated screenshot to LLM
- LLM responds with a number; you click that element
Way more reliable than free-form coordinates.
Code for SoM
elements = page.query_selector_all('button, a, input, [role="button"]')
annotated = draw_numbered_overlays(screenshot, elements)
idx = ask_llm_for_choice(annotated, prompt='Click the Submit button')
elements[idx].click()Multi-Frame Reasoning
For dynamic pages, take screenshots at multiple moments and send all to the model. The model can reason about temporal changes.
Latency and Cost
Each screenshot is ~85k tokens at full resolution. Costs:
- gpt-4o: $0.85 / 100 images
- claude-sonnet-4-5: ~$1.50 / 100 images
Resize before sending; use vision only when DOM-based selection fails.
Resize for Cheaper Calls
from PIL import Image
img = Image.open('screen.png')
img.thumbnail((1024, 1024))
img.save('screen-small.png')OCR Fallback
For pure text extraction, OCR (Tesseract, PaddleOCR) is much cheaper and often better than vision LLMs.
SoM Pattern
What is the Set-of-Marks (SoM) prompting pattern?
Recap
Send screenshots to GPT-4o or Claude for screen understanding. Use SoM annotations for reliable interaction. OCR for pure text. Resize to save cost.
Frequently asked questions
Is the “Vision Models for Screen Understanding” lesson free?
Yes — the full text of “Vision Models for Screen Understanding” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.
What will I learn in “Vision Models for Screen Understanding”?
Send screenshots to a multimodal model so the agent 'sees' what a human would see. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Agents?
No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Vision Models for Screen Understanding” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Agents lesson?
Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Browser Automation with Playwright
- Vision Models for Screen Understanding
- Computer-Use Patterns (Anthropic Computer-Use)
- Building a Reliable Form-Filling Agent