AI Agents · Lesson

Video Understanding in Agents

Frame extraction, video summarization, and temporal reasoning over clips.

Video Understanding in Agents is a free AI Agents lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Video Understanding for Agents

Most LLMs cannot directly process video files, but agents can analyse video by extracting representative frames and sending them as images. The agent then reasons over the visual sequence to detect events, changes, and patterns over time.

Installing OpenCV for Frame Extraction

OpenCV (cv2) is the standard library for video processing in Python. It lets you open video files, read their metadata (frame rate, resolution, frame count), and extract individual frames as NumPy arrays.

# pip install opencv-python-headless
import cv2

def get_video_metadata(video_path: str) -> dict:
    cap = cv2.VideoCapture(video_path)
    if not cap.isOpened():
        raise IOError(f'Cannot open video: {video_path}')

    fps = cap.get(cv2.CAP_PROP_FPS)
    frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    duration_s = frame_count / fps if fps > 0 else 0
    cap.release()

    return {
        'fps': round(fps, 2),
        'frame_count': frame_count,
        'width': width,
        'height': height,
        'duration_seconds': round(duration_s, 2)
    }

meta = get_video_metadata('recording.mp4')
print(meta)

Extracting Frames Every N Seconds

For most video understanding tasks, you do not need every frame. Sampling one frame every N seconds gives a representative summary of the video content. At 1 frame/second, a 60-second video produces 60 frames.

import cv2
import os

def extract_frames(
    video_path: str,
    output_dir: str,
    every_n_seconds: float = 5.0
) -> list:
    cap = cv2.VideoCapture(video_path)
    fps = cap.get(cv2.CAP_PROP_FPS)
    frame_interval = int(fps * every_n_seconds)
    os.makedirs(output_dir, exist_ok=True)

    saved_frames = []
    frame_idx = 0
    while True:
        ret, frame = cap.read()
        if not ret:
            break
        if frame_idx % frame_interval == 0:
            timestamp = round(frame_idx / fps, 2)
            fname = f'{output_dir}/frame_{frame_idx:06d}_{timestamp}s.jpg'
            cv2.imwrite(fname, frame)
            saved_frames.append({'path': fname, 'timestamp': timestamp})
        frame_idx += 1

    cap.release()
    print(f'Extracted {len(saved_frames)} frames')
    return saved_frames

Scene Change Detection

Instead of uniform sampling, extract frames at scene boundaries — moments when the visual content changes significantly. This gives more informative frames per API call. Use frame differencing: if the mean absolute difference between consecutive frames exceeds a threshold, a scene change occurred.

import cv2
import numpy as np

def extract_scene_change_frames(
    video_path: str,
    output_dir: str,
    diff_threshold: float = 30.0
) -> list:
    cap = cv2.VideoCapture(video_path)
    fps = cap.get(cv2.CAP_PROP_FPS)
    os.makedirs(output_dir, exist_ok=True)
    prev_gray = None
    saved = []
    frame_idx = 0

    while True:
        ret, frame = cap.read()
        if not ret:
            break
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        if prev_gray is not None:
            diff = np.mean(np.abs(gray.astype(float) - prev_gray.astype(float)))
            if diff > diff_threshold:
                ts = round(frame_idx / fps, 2)
                fname = f'{output_dir}/scene_{frame_idx:06d}_{ts}s.jpg'
                cv2.imwrite(fname, frame)
                saved.append({'path': fname, 'timestamp': ts, 'diff': round(diff, 2)})
        prev_gray = gray
        frame_idx += 1

    cap.release()
    return saved

Sending a Frame Sequence to Vision API

Send multiple extracted frames to the vision API in a single message. Include each frame's timestamp so the model can reason about temporal order and duration. Ask explicitly for temporal reasoning in your prompt.

import base64
import anthropic

def describe_video_frames(frame_info_list: list, query: str) -> str:
    client = anthropic.Anthropic(api_key='YOUR_API_KEY')
    content = []
    for fi in frame_info_list:
        with open(fi['path'], 'rb') as f:
            b64 = base64.standard_b64encode(f.read()).decode('utf-8')
        content.append({
            'type': 'text',
            'text': f'Frame at {fi["timestamp"]}s:'
        })
        content.append({
            'type': 'image',
            'source': {'type': 'base64', 'media_type': 'image/jpeg', 'data': b64}
        })
    content.append({'type': 'text', 'text': query})

    response = client.messages.create(
        model='claude-opus-4-5',
        max_tokens=1024,
        messages=[{'role': 'user', 'content': content}]
    )
    return response.content[0].text

Temporal Reasoning Prompts

Temporal reasoning requires specific prompt framing. Ask the model to compare frames explicitly, note what changed, estimate duration of activities, and identify causal relationships between frames.

TEMPORAL_QUERY = (
    'You are analysing a video sequence. The frames above are in chronological order '
    'with their timestamps shown. Please:\n'
    '1. Describe what changed between the first and last frame\n'
    '2. Identify any notable events and when they occurred (timestamp)\n'
    '3. Summarise the overall activity shown in the video\n'
    '4. Estimate the total duration of the main activity\n'
    'Be specific about timestamps.'
)

# More targeted temporal query:
CHANGE_DETECTION_QUERY = (
    'Compare frame at {start_ts}s and frame at {end_ts}s. '
    'List all visible differences in bullet points. '
    'Categorise each change as: object_moved, object_added, '
    'object_removed, lighting_change, or camera_move.'
)

# Usage:
query = CHANGE_DETECTION_QUERY.format(start_ts=0.0, end_ts=20.0)

if __name__ == '__main__':
    print('Temporal reasoning query:')
    print(TEMPORAL_QUERY)
    print()
    print('Change detection query:')
    print(query)

Video Summary Pipeline

A complete video summary pipeline: extract frames → send to vision model in batches → collect per-batch summaries → synthesise final summary. Batching prevents exceeding the context window limit.

def summarise_video(
    video_path: str,
    every_n_seconds: float = 10.0,
    batch_size: int = 5
) -> str:
    import anthropic
    client = anthropic.Anthropic(api_key='YOUR_API_KEY')

    # Extract frames
    frames = extract_frames(video_path, '/tmp/video_frames', every_n_seconds)

    # Process in batches
    batch_summaries = []
    for i in range(0, len(frames), batch_size):
        batch = frames[i:i + batch_size]
        ts_range = f'{batch[0]["timestamp"]}s - {batch[-1]["timestamp"]}s'
        query = f'Summarise what happens in this video segment ({ts_range}).'
        summary = describe_video_frames(batch, query)
        batch_summaries.append(f'[{ts_range}] {summary}')

    # Synthesise
    all_summaries = '\n\n'.join(batch_summaries)
    result = client.messages.create(
        model='claude-opus-4-5',
        max_tokens=512,
        messages=[{'role': 'user', 'content':
            f'Here are segment-level video summaries:\n\n{all_summaries}\n\n'
            'Write a single coherent summary of the entire video.'
        }]
    )
    return result.content[0].text

Object Tracking Across Frames

Ask the vision model to track specific objects across a frame sequence. For example: 'In each frame, describe the position of the red car: top-left, top-right, centre, bottom-left, bottom-right.' This creates a coarse motion trajectory without computer vision tracking algorithms.

def track_object_across_frames(
    frames: list,
    object_description: str
) -> list:
    import anthropic, base64
    client = anthropic.Anthropic(api_key='YOUR_API_KEY')
    trajectory = []

    for fi in frames:
        with open(fi['path'], 'rb') as f:
            b64 = base64.standard_b64encode(f.read()).decode('utf-8')
        response = client.messages.create(
            model='claude-opus-4-5',
            max_tokens=128,
            messages=[{
                'role': 'user',
                'content': [
                    {'type': 'image', 'source': {'type': 'base64',
                      'media_type': 'image/jpeg', 'data': b64}},
                    {'type': 'text', 'text':
                      f'Where is the {object_description} in this frame? '
                      'Answer with one of: top-left, top-right, centre, '
                      'bottom-left, bottom-right, not-visible.'}
                ]
            }]
        )
        trajectory.append({
            'timestamp': fi['timestamp'],
            'position': response.content[0].text.strip()
        })
    return trajectory

Handling Frame Count Limits

Vision API context windows have limits on how many images can be sent per request (typically 5–20 images). For long videos, use a hierarchical approach: analyse segments independently, then merge segment summaries into a final summary using a second LLM call.

MAX_FRAMES_PER_REQUEST = 8

def hierarchical_video_analysis(frames: list, final_query: str) -> str:
    import anthropic
    client = anthropic.Anthropic(api_key='YOUR_API_KEY')

    # Level 1: analyse each chunk
    chunk_summaries = []
    for i in range(0, len(frames), MAX_FRAMES_PER_REQUEST):
        chunk = frames[i:i + MAX_FRAMES_PER_REQUEST]
        ts = f'{chunk[0]["timestamp"]}s-{chunk[-1]["timestamp"]}s'
        summary = describe_video_frames(
            chunk, f'What happens in this segment ({ts})?'
        )
        chunk_summaries.append(f'Segment {ts}: {summary}')

    # Level 2: synthesise all chunk summaries
    combined = '\n'.join(chunk_summaries)
    result = client.messages.create(
        model='claude-opus-4-5',
        max_tokens=512,
        messages=[{
            'role': 'user',
            'content': f'Video segments:\n{combined}\n\n{final_query}'
        }]
    )
    return result.content[0].text

Combining Audio and Video Analysis

For the richest understanding, combine frame-level visual analysis with Whisper audio transcription. The agent can correlate what is seen with what is said at each timestamp, enabling powerful tasks like meeting analysis or tutorial indexing.

def full_video_analysis(video_path: str) -> dict:
    import subprocess

    # Step 1: Extract audio track
    audio_path = '/tmp/video_audio.mp3'
    subprocess.run([
        'ffmpeg', '-i', video_path, '-q:a', '0', '-map', 'a',
        audio_path, '-y'
    ], capture_output=True)

    # Step 2: Transcribe audio
    transcript_data = transcribe_with_timestamps(audio_path)

    # Step 3: Extract key frames
    frames = extract_frames(video_path, '/tmp/key_frames', every_n_seconds=15.0)

    # Step 4: Visual summary
    visual_summary = hierarchical_video_analysis(
        frames, 'Summarise the visual content.'
    )

    return {
        'transcript': transcript_data['full_text'],
        'transcript_segments': transcript_data['segments'],
        'visual_summary': visual_summary,
        'frame_count': len(frames)
    }

Cleanup and Frame Management

Extracted frames can accumulate quickly. Always clean up temporary frame files after processing, implement a frame cache keyed by (video_path, frame_rate) hash to avoid re-extraction on repeated analysis of the same video.

import os
import shutil
import hashlib

FRAME_CACHE_DIR = '/tmp/frame_cache'

def get_cache_key(video_path: str, every_n_seconds: float) -> str:
    content = f'{video_path}:{every_n_seconds}'
    return hashlib.md5(content.encode()).hexdigest()[:12]

def get_or_extract_frames(video_path: str, every_n_seconds: float) -> list:
    key = get_cache_key(video_path, every_n_seconds)
    cache_dir = os.path.join(FRAME_CACHE_DIR, key)

    if os.path.exists(cache_dir):
        print(f'Cache hit: {key}')
        files = sorted(os.listdir(cache_dir))
        return [
            {'path': os.path.join(cache_dir, f),
             'timestamp': float(f.split('_')[-1].replace('s.jpg', ''))}
            for f in files if f.endswith('.jpg')
        ]
    return extract_frames(video_path, cache_dir, every_n_seconds)

def clear_frame_cache():
    if os.path.exists(FRAME_CACHE_DIR):
        shutil.rmtree(FRAME_CACHE_DIR)
        print('Frame cache cleared')

Knowledge Check

Why is scene change detection preferable to uniform frame sampling for video analysis?

Recap: Video Understanding in Agents

Great job! Key takeaways from this lesson:

OpenCV: extract frames with cap.read(); sample every N seconds or at scene changes
Frame sequences: send in chronological order with timestamps for temporal context
Hierarchical analysis: chunk into batches → segment summaries → final synthesis
Combined analysis: ffmpeg extracts audio; Whisper transcribes; cross-reference with visual frames
Frame cache: avoid redundant extraction with hash-keyed cache directory

Next: cross-modal reasoning — when text says one thing and the image shows another.

Frequently asked questions

Is the “Video Understanding in Agents” lesson free?

Yes — the full text of “Video Understanding in Agents” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Video Understanding in Agents”?

Frame extraction, video summarization, and temporal reasoning over clips. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Video Understanding in Agents” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Image + Text Agents with Claude Vision and GPT-4V
Audio + Text Agent Workflows
Video Understanding in Agents
Cross-Modal Reasoning Patterns

← Back to AI Agents