AI Prompt Engineering · Lesson

Multimodal Output Control

Shaping mixed-media responses.

Multimodal Output Control is a free AI Prompt Engineering lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Controlling Mixed-Media Output

Output control is the discipline of shaping what form the response takes when the answer spans text, structured data, and references to generated or selected media. The model will default to prose; production systems need parseable, renderable, composable artifacts.

You are specifying a render contract, not just asking a question.
Format reliability beats format richness — predictable shapes win.

Separate Content From Presentation

Have the model emit structured content, and render media on your side. Asking a text model to emit raw image bytes or markup-heavy layouts is fragile; asking it for a typed description that your renderer turns into media is robust.

The model decides what; your pipeline decides how it looks.

block = {
  'type': 'figure',
  'caption': 'Quarterly revenue',
  'chart': {'kind': 'bar', 'x': ['Q1','Q2'], 'y': [10, 14]},
  'alt_text': 'Bar chart rising from 10 to 14.'
}

Block-Typed Response Schemas

Model rich responses as an ordered list of typed blocks: text, code, image_ref, table, chart_spec. Each block carries only the fields its type needs. Your renderer walks the list and emits the right component per type.

This is how chat UIs, reports, and slide generators stay reliable across thousands of responses.

schema = {
  'blocks': [
    {'type': 'text', 'value': '...'},
    {'type': 'code', 'lang': 'python', 'value': '...'},
    {'type': 'image_ref', 'id': 'fig1', 'alt': '...'},
    {'type': 'table', 'columns': [...], 'rows': [...]}
  ]
}

Referencing vs Embedding Media

Prefer references over embeds. The model emits an id or a generation spec; your system resolves it to an actual asset. This decouples the language step from the media step and lets you cache, regenerate, or swap assets without re-prompting.

Embed: response carries the asset inline (heavy, fragile).
Reference: response carries a pointer or recipe (light, composable).

image_block = {
  'type': 'image_ref',
  'spec': {'prompt': 'minimal line icon of a gear', 'size': '512x512'},
  'alt': 'Settings gear icon'
}
# Pipeline calls the image model with spec, fills in the URL.

Constraining Generation Specs

When the model writes a spec for a downstream generator (image, chart, TTS), constrain that spec tightly. Open-ended specs drift; enumerated options stay on-brand and on-budget.

Give the model a controlled vocabulary: allowed chart kinds, allowed image styles, allowed voices — and forbid free-form deviation.

chart_spec = {
  'kind': 'one of [bar, line, scatter]',   # enumerated
  'palette': 'brand_default',               # not a hex free-for-all
  'max_series': 4
}

Mandatory Accessibility Fields

Every media block should carry an alt text or transcript field, and you should make it non-optional in the schema. This is both an accessibility requirement and a verification handle — the alt text reveals what the model intended the media to convey, which you can check against the spec.

Interleaving Order and Layout Intent

The order of blocks is the layout. If the model returns a chart before the paragraph that explains it, the rendered document reads wrong. State layout intent: 'explanatory text precedes the figure it describes; a figure is always followed by a one-line takeaway.'

Encode reading-order rules so the block list renders into a coherent narrative.

Length and Density Budgets per Block

Control verbosity at the block level, not globally. A caption is one line; a section intro is a short paragraph; a table is capped at N rows. Per-block budgets prevent the model from ballooning one component while starving another.

'Captions: max 12 words.'
'Tables: max 8 rows; summarize the rest as a final row.'

limits = {'caption_words': 12, 'table_rows': 8, 'intro_sentences': 3}

Validation and Repair Loops

A mixed-media schema is only as good as your validator. Parse the block list, validate each block against its type schema, and on failure issue a targeted repair prompt naming the exact violation.

Repair beats regenerate: re-asking the whole response wastes tokens and may break the parts that were correct.

def validate(blocks):
    for b in blocks:
        if b['type'] == 'image_ref' and 'alt' not in b:
            return f'Block {b} missing required alt text'
    return None  # ok

Streaming Mixed Media

When streaming, blocks must be individually parseable so the UI can render each as it completes rather than waiting for the whole response. Emit one well-formed JSON object per block (newline-delimited) instead of one giant array that is invalid until the final bracket.

Block-at-a-time streaming gives responsive UIs and early validation.

{'type': 'text', 'value': 'Revenue grew this quarter.'}
{'type': 'chart_spec', 'kind': 'bar', 'x': ['Q1','Q2'], 'y': [10,14], 'alt': '...'}
{'type': 'text', 'value': 'The bulk came from new accounts.'}

An Output-Control Contract

A complete mixed-media contract specifies: the typed block vocabulary, references over embeds, enumerated generation specs, mandatory alt/transcript, reading-order rules, per-block budgets, and a streaming-friendly serialization. Bundle it as a reusable system block and your renderer becomes a stable target across many tasks.

Quick Check

You are building a report generator whose responses mix paragraphs, tables, and figures, rendered by your own front end.

Recap: Shaping Mixed-Media Responses

Treat rich output as a render contract: a vocabulary of typed blocks, media as references or constrained generation specs rather than embeds, mandatory alt/transcript fields, explicit reading-order and per-block budgets, and a streaming-friendly serialization with validation-and-repair. Separate what the model decides from how your pipeline renders it, and mixed-media responses become predictable, auditable, and easy to compose.

Frequently asked questions

Is the “Multimodal Output Control” lesson free?

Yes — the full text of “Multimodal Output Control” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Multimodal Output Control”?

Shaping mixed-media responses. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Multimodal Output Control” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering