Claude Video: The Open-Source Tool That Lets AI Coding Agents Watch and Analyze Any Video — 5,400+ GitHub Stars
Claude Video gives AI coding agents like Claude Code, Cursor, and Codex the ability to watch videos, extract frames, transcribe audio, and answer questions grounded in what is actually on screen.
Quick Answer: Claude Video is an open-source tool (5,400+ GitHub stars) that lets AI coding agents like Claude Code, Cursor, and Codex actually watch videos. Type /watch [URL] [question] and it downloads the video, extracts scene-aware frames, transcribes audio, and gives Claude full visual+audio context to answer questions, debug recordings, or summarize content — all in one command.
Why Your AI Coding Agent Needs Video Understanding
AI coding agents can read code, browse repos, and run scripts. But there's a glaring blind spot: they can't watch videos. When you paste a YouTube link, Claude guesses from the title. When a teammate sends a screen recording of a bug, you have to watch it yourself.
Claude Video fixes this. It's an open-source plugin that gives your AI agent the ability to literally watch any video — see the frames, hear the audio via transcription, and answer questions grounded in what's actually happening on screen.
With 5,400+ stars and nearly 1,000 new stars in a single day, this is one of the fastest-growing AI developer tools on GitHub right now.
How Claude Video Works Under the Hood
The architecture is elegant in its simplicity. When you run /watch with a video URL and a question, here's what happens:
- Smart download: yt-dlp checks for existing captions first. If captions exist, no video download needed — instant, free.
- Frame extraction: ffmpeg extracts frames using one of three strategies: keyframes only (fastest), scene-change detection (balanced), or uncapped scene detection (token-burner mode).
- Deduplication: A clever algorithm compares each frame to the last kept frame (not the previous one), catching slow fades that frame-to-frame comparison misses. Default threshold: 2.0 brightness difference on a 0-255 scale.
- Transcription: Free captions when available; otherwise extracts mono 16kHz audio and sends to Whisper (Groq's whisper-large-v3 preferred for speed/cost, OpenAI whisper-1 as fallback).
- Claude processes everything: Frames arrive as images in Claude's context with timestamps. Transcript arrives with timestamps. Claude answers based on what it actually saw and heard.
The Three Detail Modes Explained
Claude Video offers three detail levels that trade token cost for visual fidelity:
| Mode | Engine | Speed | Best For |
|---|---|---|---|
transcript |
Captions only (no frames) | ~4.5s | Quick summaries, cheapest option |
efficient |
Keyframes only | ~0.5s extraction | Fast visual scan, 40x faster than scene modes |
balanced |
Scene-change detection | ~21s extraction | Good coverage, capped at 100 frames |
token-burner |
Scene-change, uncapped | ~21s extraction | Full fidelity, high-motion videos |
For a 49-minute screen recording, efficient mode pulls 50 frames in half a second. balanced detects 100+ scene changes and samples down. token-burner keeps every single one — at the cost of your token budget.
Real-World Use Cases That Actually Matter
1. Debugging From Screen Recordings
A teammate sends a .mov file showing a UI bug. Instead of watching it yourself:
/watch bug-repro.mov what's going wrong at the 12-second mark?Claude watches the recording, identifies the frame where the issue appears, describes what's on screen, and often catches the root cause without you opening a single file.
2. Analyzing Content Strategy
/watch https://youtu.be/<competitor-video> what hook did they open with?Claude examines the opening frames, reads the transcript, and breaks down the structure — useful for ad creative, competitor launches, podcast intros, or anything where how matters as much as what.
3. Cutting Through Marketing Hype
/watch https://youtu.be/<launch-video> what's actually new — skip the hypeStrip a 30-minute product launch down to the 3 things that actually matter. No more sitting through intros and overselling.
4. Turn a Playlist Into Searchable Notes
/watch https://youtu.be/<video> summarize this to a noteRun it across a course or tutorial series and build a searchable knowledge base instead of watching hours of content.
Token Cost: What You Need to Know
Token cost is dominated by frames — each frame is an image, and image tokens add up fast. Here's the automatic frame budget:
| Video Duration | Default Frame Budget | Coverage |
|---|---|---|
| ≤ 30 seconds | ~30 frames | Dense — every key moment |
| 30s - 1 min | ~40 frames | Still dense |
| 1 - 3 min | ~60 frames | Comfortable |
| 3 - 10 min | ~80 frames | Sparse but workable |
| > 10 min | 100 frames (capped) | Sparse scan — use --start/--end to focus |
Pro tip: For long videos, use --start and --end flags to focus on specific timestamps. Focused mode gets denser per-second budgets (capped at 2 fps), which is far more useful than a sparse pass over the whole video.
Installation: One Command for Any AI Agent
Claude Video works with 50+ AI coding agents out of the box:
# Claude Code (recommended — auto-updates)
/plugin marketplace add bradautomates/claude-video
/plugin install watch@claude-video
# Codex, Cursor, Copilot, Gemini CLI, and 50+ others
npx skills add bradautomates/claude-video -gZero config to start — yt-dlp and ffmpeg auto-install on first run via brew on macOS (Linux/Windows get exact commands). Captions cover most public videos for free. A Whisper API key is only needed when a video has no captions.
Key Benefits
- True multimodal understanding: Claude doesn't guess from titles — it sees frames and reads transcripts
- Works with any video source: YouTube, Loom, TikTok, X, Instagram, local files (.mp4, .mov, .mkv, .webm)
- Smart token management: Auto-fps, deduplication, and detail modes keep costs predictable
- Universal compatibility: Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and 50+ agents
- Privacy-first: Local processing with optional Whisper API fallback
- Free for most videos: Native captions are used when available — no API costs
- Battle-tested dedup: Drops near-identical frames before they hit your token budget
Frequently Asked Questions
Does Claude Video work with local video files?
Yes. Pass any local path — .mp4, .mov, .mkv, or .webm — and Claude Video processes it the same way as a URL. This is especially useful for debugging screen recordings sent by teammates.
How much does it cost in tokens per video?
It depends on the detail mode and video length. A 49-minute video in efficient mode costs ~9,800 image tokens for 50 frames plus transcript tokens. In transcript mode (captions only), it's ~26,600 text tokens with zero image cost. Short videos under 1 minute are very affordable at ~30-40 frames.
Do I need a Whisper API key?
Only when the video has no captions (manual or auto-generated). Most YouTube videos have auto-captions, so you'll rarely need Whisper. When you do, Groq's whisper-large-v3 is recommended for speed and cost.
Which AI coding agents are supported?
Claude Video works with Claude Code (via plugin), and 50+ other agents via the Agent Skills protocol: Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, and more. Installation is a single npx skills add command.
Can I analyze specific timestamps instead of the whole video?
Yes. Use --start and --end flags like /watch https://youtu.be/video what happens around 2:30 --start 2:00 --end 3:00. Focused mode gives denser frame coverage within your window and is far more efficient than scanning the entire video.
Is the video data sent to any third-party service?
Video download and frame extraction happen locally. Only the audio (when captions aren't available) is sent to Whisper API for transcription. If captions exist, nothing leaves your machine except Claude's API calls with the extracted frames.