AI Agents · Lesson

Computer-Use Patterns (Anthropic Computer-Use)

Anthropic's Computer Use exposes screenshot, mouse, and keyboard as tools to Claude.

Computer-Use Patterns (Anthropic Computer-Use) is a free AI Agents lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Is Computer Use?

Anthropic's "Computer Use" capability (in Claude 3.5+) exposes screenshot, mouse, and keyboard as tools. The model can drive a virtual desktop end-to-end.

The Tool Set

Three core tools:

computer — screenshot, click, type, scroll
bash — run shell commands
text_editor — read/write files

Quickstart

from anthropic import Anthropic
client = Anthropic()

response = client.beta.messages.create(
    model='claude-sonnet-4-5',
    max_tokens=4096,
    tools=[{'type': 'computer_20241022', 'name': 'computer', 'display_width_px': 1280, 'display_height_px': 720}],
    messages=[{'role': 'user', 'content': 'Open Firefox and search for the weather.'}]
)

Tool Calls Returned

The model emits tool_use blocks like:

{
    'type': 'tool_use',
    'name': 'computer',
    'input': {
        'action': 'screenshot'  # or 'left_click', 'type', 'mouse_move'
    }
}

Action Types

screenshot — take and return a screenshot
left_click {coordinate: [x, y]} — click
type {text: "hello"} — type text
key {text: "Return"} — press a key
mouse_move {coordinate: [x, y]}
scroll {coordinate: [...], scroll_direction: "down"}

You Build the Executor

Anthropic provides the tool DEFINITION; you implement the execution. Typically via Playwright, X11, or a headless desktop in Docker:

from playwright.sync_api import sync_playwright
p = sync_playwright().start()
browser = p.chromium.launch()
page = browser.new_page(viewport={'width': 1280, 'height': 720})

def execute(action_input):
    action = action_input['action']
    if action == 'screenshot':
        return page.screenshot()
    if action == 'left_click':
        x, y = action_input['coordinate']
        page.mouse.click(x, y)
        return page.screenshot()
    if action == 'type':
        page.keyboard.type(action_input['text'])
        return page.screenshot()
    # ... etc.

Returning the Result

tool_result_content = [
    {'type': 'image', 'source': {'type': 'base64', 'media_type': 'image/png', 'data': screenshot_b64}}
]

messages.append({
    'role': 'user',
    'content': [{
        'type': 'tool_result',
        'tool_use_id': tc.id,
        'content': tool_result_content
    }]
})

Sandbox Required

Computer Use lets the model click ANYWHERE. Run it in a sandboxed VM (Docker + Xvfb + a headless browser) — never on your laptop.

Anthropic Reference Docker Image

Anthropic provides a reference Docker image that bundles Xvfb, Firefox, a desktop environment, and the executor. Great starting point.

Speed and Cost

Each step takes 1-5 seconds (screenshot + LLM + action)
Each screenshot adds tokens to context
Long workflows can cost $0.50 - $5 per run

Failure Modes

Misclicked elements (off by 10px)
Getting stuck on popups/cookie banners
Page hasn't loaded yet (wait!)
Captchas

OpenAI Operator

OpenAI launched a similar product (Operator) — same concept, different implementation. Both have minor differences but similar capabilities.

Use Cases

Filling forms on legacy intranet apps
Test automation across multiple browsers
QA exploratory testing
"Use any web app for me" assistant

Computer-Use Tools

What tools does Anthropic's Computer Use expose to the model?

Recap

Anthropic's Computer Use lets Claude drive a desktop. You implement screenshot/click/type via Playwright or X11. Run only in a sandbox. Slow but flexible.

Frequently asked questions

Is the “Computer-Use Patterns (Anthropic Computer-Use)” lesson free?

Yes — the full text of “Computer-Use Patterns (Anthropic Computer-Use)” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Computer-Use Patterns (Anthropic Computer-Use)”?

Anthropic's Computer Use exposes screenshot, mouse, and keyboard as tools to Claude. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Computer-Use Patterns (Anthropic Computer-Use)” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Browser Automation with Playwright
Vision Models for Screen Understanding
Computer-Use Patterns (Anthropic Computer-Use)
Building a Reliable Form-Filling Agent

← Back to AI Agents