Image + Text Agents with Claude Vision and GPT-4V
Sending images in API calls, visual grounding, and image-aware tool use.
Image + Text Agents with Claude Vision and GPT-4V is a free AI Agents lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Multimodal Agents: Images + Text
A multimodal agent can see as well as read. By sending images alongside text in the same API call, the agent can answer questions about photos, analyse charts, read screenshots, and describe diagrams — all within the same conversation loop.
Sending Images with the OpenAI API
OpenAI's vision models (GPT-4o, GPT-4V) accept a content array instead of a plain string. Each element is either a text object or an image_url object. The image can be a public URL or a base64-encoded data URL.
from openai import OpenAI
client = OpenAI(api_key='YOUR_OPENAI_API_KEY')
# Using a public URL
response = client.chat.completions.create(
model='gpt-4o',
messages=[
{
'role': 'user',
'content': [
{
'type': 'image_url',
'image_url': {
'url': 'https://example.com/chart.png'
}
},
{
'type': 'text',
'text': 'What trend does this chart show?'
}
]
}
],
max_tokens=512
)
print(response.choices[0].message.content)Base64 Image Encoding
For images that are not publicly accessible (local files, screenshots, user uploads), encode them as base64 and embed them directly in the API call using the data:image/jpeg;base64,... URL scheme.
import base64
from pathlib import Path
Path('screenshot.png').write_bytes(b'\x89PNG\r\n\x1a\n' + bytes(range(40)))
def encode_image_to_base64(image_path: str) -> str:
with open(image_path, 'rb') as f:
return base64.b64encode(f.read()).decode('utf-8')
def build_image_message(image_path: str, question: str, mime: str = 'jpeg') -> dict:
b64 = encode_image_to_base64(image_path)
data_url = f'data:image/{mime};base64,{b64}'
return {
'role': 'user',
'content': [
{'type': 'image_url', 'image_url': {'url': data_url}},
{'type': 'text', 'text': question}
]
}
message = build_image_message('screenshot.png', 'What error is shown?', mime='png')
print('Message built, image size:', len(message['content'][0]['image_url']['url']))Claude Vision API
Anthropic's Claude models also support vision. The content block uses a source field with type: base64 and media_type. The structure is slightly different from OpenAI's but equally powerful.
import anthropic
import base64
client = anthropic.Anthropic(api_key='YOUR_ANTHROPIC_API_KEY')
with open('diagram.png', 'rb') as f:
image_data = base64.standard_b64encode(f.read()).decode('utf-8')
response = client.messages.create(
model='claude-opus-4-5',
max_tokens=512,
messages=[
{
'role': 'user',
'content': [
{
'type': 'image',
'source': {
'type': 'base64',
'media_type': 'image/png',
'data': image_data
}
},
{
'type': 'text',
'text': 'Describe the architecture shown in this diagram.'
}
]
}
]
)
print(response.content[0].text)Detecting MIME Type from File Extension
Different image formats (JPEG, PNG, GIF, WebP) use different MIME types. Autodetect the MIME type from the file extension so you never need to hard-code it in your agent code.
import os
import base64
with open('photo.jpg', 'wb') as f:
f.write(b'\xff\xd8\xff' + bytes(range(30)))
MIME_MAP = {
'.jpg': 'image/jpeg',
'.jpeg': 'image/jpeg',
'.png': 'image/png',
'.gif': 'image/gif',
'.webp': 'image/webp'
}
def get_mime_type(image_path: str) -> str:
ext = os.path.splitext(image_path)[1].lower()
mime = MIME_MAP.get(ext)
if not mime:
raise ValueError(f'Unsupported image format: {ext}')
return mime
def load_image_as_base64(image_path: str) -> tuple:
"""Returns (base64_string, mime_type)"""
mime = get_mime_type(image_path)
with open(image_path, 'rb') as f:
b64 = base64.b64encode(f.read()).decode('utf-8')
return b64, mime
b64, mime = load_image_as_base64('photo.jpg')
print(f'MIME: {mime}, Size: {len(b64)} bytes base64')Image Analysis in an Agent Loop
In an agent loop, image analysis is a tool. The agent decides when to invoke it, just like any other tool. Define it in the tool schema and return structured data (JSON) so the agent can reason over it programmatically.
def analyze_image_tool(image_path: str, query: str) -> dict:
"""
Agent tool: analyse an image and return structured findings.
"""
import anthropic, base64
client = anthropic.Anthropic(api_key='YOUR_API_KEY')
b64, mime = load_image_as_base64(image_path)
structured_query = (
query + '\n\nRespond with JSON only: '
'{"description": str, "objects": [str], "text_found": str, "confidence": float}'
)
response = client.messages.create(
model='claude-opus-4-5',
max_tokens=512,
messages=[{
'role': 'user',
'content': [
{'type': 'image', 'source': {'type': 'base64',
'media_type': mime, 'data': b64}},
{'type': 'text', 'text': structured_query}
]
}]
)
import json
return json.loads(response.content[0].text)Controlling Image Detail Level (OpenAI)
OpenAI's vision API accepts a detail parameter: low (fast, cheap, 85 tokens), high (detailed, tiles the image, up to 1105 tokens), or auto (model decides). Use low for simple yes/no questions and high for fine-grained analysis like reading small text.
def query_image_openai(
image_path: str,
question: str,
detail: str = 'auto' # 'low', 'high', or 'auto'
) -> str:
import base64
from openai import OpenAI
client = OpenAI(api_key='YOUR_OPENAI_API_KEY')
b64, mime = load_image_as_base64(image_path)
data_url = f'data:{mime};base64,{b64}'
response = client.chat.completions.create(
model='gpt-4o',
max_tokens=512,
messages=[{
'role': 'user',
'content': [
{
'type': 'image_url',
'image_url': {'url': data_url, 'detail': detail}
},
{'type': 'text', 'text': question}
]
}]
)
return response.choices[0].message.contentMulti-Image Conversations
You can send multiple images in a single message, or across turns in a conversation. This enables comparison tasks: 'Compare these two screenshots and describe what changed.'
def compare_two_images(
image_path_1: str,
image_path_2: str,
comparison_question: str
) -> str:
import anthropic, base64
client = anthropic.Anthropic(api_key='YOUR_API_KEY')
b64_1, mime_1 = load_image_as_base64(image_path_1)
b64_2, mime_2 = load_image_as_base64(image_path_2)
response = client.messages.create(
model='claude-opus-4-5',
max_tokens=512,
messages=[{
'role': 'user',
'content': [
{'type': 'image', 'source': {'type': 'base64',
'media_type': mime_1, 'data': b64_1}},
{'type': 'image', 'source': {'type': 'base64',
'media_type': mime_2, 'data': b64_2}},
{'type': 'text', 'text': comparison_question}
]
}]
)
return response.content[0].textImage Agent: Screenshot Reader
A practical use case: an agent that reads application screenshots to extract form field values, error messages, or UI state. This is useful for testing automation and monitoring pipelines.
SCREENSHOT_READER_PROMPT = (
'You are a UI analyser. Given this application screenshot, extract:\n'
'1. The current page/screen name\n'
'2. Any error messages\n'
'3. Key UI elements visible (buttons, form fields, text)\n'
'4. The overall app state\n\n'
'Return JSON: {"screen": str, "errors": [str], '
'"elements": [str], "state": str}'
)
def read_screenshot(screenshot_path: str) -> dict:
import anthropic, base64, json
client = anthropic.Anthropic(api_key='YOUR_API_KEY')
b64, mime = load_image_as_base64(screenshot_path)
response = client.messages.create(
model='claude-opus-4-5',
max_tokens=512,
messages=[{
'role': 'user',
'content': [
{'type': 'image', 'source': {'type': 'base64',
'media_type': mime, 'data': b64}},
{'type': 'text', 'text': SCREENSHOT_READER_PROMPT}
]
}]
)
return json.loads(response.content[0].text)Cost Optimisation for Vision Calls
Vision API calls are significantly more expensive than text-only calls. Optimise costs by: resizing images before encoding (most models accept 512×512–2048×2048), using detail=low for simple tasks, caching analysis results for unchanged images, and batching multiple questions into one call.
from PIL import Image
import io, base64
def resize_and_encode(
image_path: str,
max_dim: int = 1024
) -> tuple:
img = Image.open(image_path)
# Resize maintaining aspect ratio
ratio = min(max_dim / img.width, max_dim / img.height)
if ratio < 1.0:
new_w = int(img.width * ratio)
new_h = int(img.height * ratio)
img = img.resize((new_w, new_h), Image.LANCZOS)
print(f'Resized to {new_w}x{new_h} (from {img.width}x{img.height})')
# Convert to JPEG for smaller size
buf = io.BytesIO()
img.save(buf, format='JPEG', quality=85)
b64 = base64.b64encode(buf.getvalue()).decode('utf-8')
return b64, 'image/jpeg'Vision Limitations and Fallbacks
Vision models have limitations: they cannot read very small text reliably, struggle with highly compressed images, and may hallucinate details. Always validate critical extracted data (e.g., numbers from charts) with a fallback prompt asking for confidence scores.
def analyze_with_confidence(
image_path: str,
question: str,
confidence_threshold: float = 0.7
) -> dict:
import anthropic, json
client = anthropic.Anthropic(api_key='YOUR_API_KEY')
b64, mime = load_image_as_base64(image_path)
prompt = (
question + '\n\nAlso rate your confidence (0.0-1.0) in the answer.\n'
'Return JSON: {"answer": str, "confidence": float, '
'"uncertainty_reason": str}'
)
response = client.messages.create(
model='claude-opus-4-5', max_tokens=256,
messages=[{'role': 'user', 'content': [
{'type': 'image', 'source': {'type': 'base64',
'media_type': mime, 'data': b64}},
{'type': 'text', 'text': prompt}
]}]
)
result = json.loads(response.content[0].text)
if result['confidence'] < confidence_threshold:
result['action'] = 'human_review_needed'
return resultKnowledge Check
Which parameter controls image analysis detail level and cost in the OpenAI vision API?
Recap: Image-Text Agents
Great work! Here is what you covered:
- OpenAI vision: content array with
image_url+textobjects; supports URL and base64 - Claude vision:
source.type: base64withmedia_type - Base64 encoding: read file → base64 encode → embed as data URL
- Detail level: use
lowfor fast/cheap,highfor fine-grained analysis - Cost optimisation: resize images, batch questions, cache results
Next: audio-text agent workflows with Whisper transcription and TTS responses.
Frequently asked questions
Is the “Image + Text Agents with Claude Vision and GPT-4V” lesson free?
Yes — the full text of “Image + Text Agents with Claude Vision and GPT-4V” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.
What will I learn in “Image + Text Agents with Claude Vision and GPT-4V”?
Sending images in API calls, visual grounding, and image-aware tool use. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Agents?
No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Image + Text Agents with Claude Vision and GPT-4V” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Agents lesson?
Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Image + Text Agents with Claude Vision and GPT-4V
- Audio + Text Agent Workflows
- Video Understanding in Agents
- Cross-Modal Reasoning Patterns