AI Prompt Engineering · Lesson

Grounding Across Modalities

Referencing visual evidence in text.

Grounding Across Modalities is a free AI Prompt Engineering lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Grounding Means

Grounding is the requirement that every textual claim be traceable to specific evidence in another modality. An ungrounded multimodal answer is a fluent guess; a grounded one is a defensible assertion with a pointer back to the pixels (or audio frame) that justify it.

Grounding converts opaque output into auditable output.
It is the single most effective lever against confident visual hallucination.

Evidence-First Reasoning Order

Force the model to extract evidence before it concludes. If the conclusion is generated first, the 'evidence' becomes post-hoc rationalization that matches the (possibly wrong) answer.

Structure the response so observed regions come first, interpretation second, and final answer last.

schema = {
  'observations': '[{region: str, text_read: str}]',
  'inference': 'str — reasoning over observations only',
  'answer': 'str'
}
# Order in the prompt enforces order in generation.

Bounding-Box and Region Citations

Ask the model to emit approximate normalized coordinates for each visual claim. Even imperfect boxes are valuable: a downstream system can crop and re-verify, and a wildly wrong box exposes a hallucination immediately.

Use normalized [x0,y0,x1,y1] in 0..1 so resolution is irrelevant.
Treat boxes as localization hints, not pixel-perfect detection.

instruction = (
  'For each extracted field, return '
  '{"field": str, "value": str, "box": [x0,y0,x1,y1]}. '
  'Coordinates normalized 0..1. If you cannot locate it, omit the field.'
)

Quote-the-Pixels Verification

For any text visible in the image, require a verbatim quote of what the model reads. A verbatim quote is checkable: you can OCR-spot-check it, and the model is far less likely to invent a value it must also transcribe exactly.

This 'read it back' constraint is cheap and disproportionately effective for documents, charts, and signage.

Cross-Modal Consistency Checks

When the same fact appears in two modalities (a slide image and its spoken narration), prompt the model to cross-check them. Agreement raises confidence; disagreement is itself signal worth surfacing.

'Does the figure's caption match what the chart shows?'
'Does the narration agree with the on-screen number?'

prompt = (
  'You have a slide image and its transcript. '
  'For each numeric claim, state: value_on_slide, value_in_transcript, agree(bool). '
  'Flag any disagreement explicitly.'
)

Refusal as a Grounding Outcome

A grounded system must be allowed to say 'not visible'. Without an explicit escape hatch, the model fills gaps with plausible fabrication. Provide one and reward it.

Instruct: 'If the evidence does not exist or is illegible, return NOT_FOUND rather than guessing.' Then make NOT_FOUND a first-class, non-penalized output in your pipeline.

Grounding Spatial Relationships

Relational claims ('the valve is left of the gauge') are harder to ground than object presence. Ask the model to ground both referents independently with boxes, then derive the relationship from coordinates rather than asserting it directly.

This decomposition turns a fragile relational judgment into two simpler localization tasks plus arithmetic.

# Derive relation from grounded boxes, not from vibes
# left_of(a, b) := a.x1 < b.x0
def left_of(a, b):
    return a['box'][2] < b['box'][0]

Audio and Temporal Grounding

Grounding generalizes beyond images. For audio or video, require timestamps as evidence: 'cite the [mm:ss] where this was said.' Temporal pointers let you spot-check the claim against the source segment.

Timestamps anchor transcription and diarization claims.
They expose summarization that drifts from what was actually said.

The Text-Override Trap

A confident assertion in the text channel can suppress correct visual evidence — the model defers to the prior you supplied. If your context says 'the invoice total is $400' and the image shows $450, an ungrounded model may parrot $400.

Defense: instruct the model to derive purely from the image first, then compare to supplied text, and flag mismatches rather than reconciling silently.

Verifying Grounding Downstream

Grounding is only useful if you act on it. Build a verifier that consumes the structured evidence:

Re-crop each box and re-OCR the quoted text; mismatch -> reject.
Replay the cited timestamp through a second ASR pass.
Treat NOT_FOUND and conflict flags as routing signals to human review.

The prompt produces evidence; your system enforces it.

def verify(field):
    crop = image.crop(field['box'])
    read = ocr(crop)
    return similar(read, field['value']) > 0.9

A Grounding Contract Template

A reusable grounding contract bundles every technique:

Observations before conclusions.
Box + verbatim quote per claim.
NOT_FOUND escape hatch, never guess.
Image-first derivation, then reconcile with supplied text and flag conflicts.
Timestamps for any audio/video claim.

Codify it once and reuse across every multimodal task.

Quick Check

Your context metadata states the order total is $400, but you suspect the scanned image may differ.

Recap: Grounding Across Modalities

Grounding makes multimodal output auditable: observations before conclusions, boxes and verbatim quotes per claim, timestamps for audio/video, a NOT_FOUND escape hatch, and image-first derivation that flags conflicts with supplied text. Pair the grounding contract with a downstream verifier that re-crops, re-reads, and routes flags to review. Evidence in the prompt, enforcement in the system — together they neutralize confident hallucination.

Frequently asked questions

Is the “Grounding Across Modalities” lesson free?

Yes — the full text of “Grounding Across Modalities” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.

What will I learn in “Grounding Across Modalities”?

Referencing visual evidence in text. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Prompt Engineering?

No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Grounding Across Modalities” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Prompt Engineering lesson?

Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to AI Prompt Engineering