Combining Text and Images

A multimodal prompt is not text plus an attached image. It is a single interleaved sequence where image tokens and text tokens occupy the same context and attend to one another. The model does not 'look at the picture then read the text' — it processes a fused token stream.

• Image patches are projected into the same embedding space as text tokens.
• Ordering matters: a question placed before an image primes different attention than one placed after.
• You are authoring one prompt with two surface forms, not two prompts.

Where you place the image relative to the instruction changes behavior. Placing the instruction after the image lets the model condition the question on what it has already encoded; placing it before turns the image into evidence for a pre-stated task.

For multi-image prompts, label each image inline so later text can reference it unambiguously ([Image 1], [Image 2]).

Vision encoders are lossy: detail not relevant to the implicit task can be discarded during pooling. If you state the task before the image, you bias the model toward attending to the regions that matter.

• Generic caption requests yield generic features.
• A specific question ('count the resistors on the board') focuses representational budget on the relevant subregions.

Front-load specificity when fine detail is needed.

Most vision models tile high-resolution images into fixed-size patches, each costing tokens. A 2000x2000 image may be split into many tiles, each downsampled. Token budget is the silent constraint of multimodal prompting.

• Crop to the region of interest before sending — do not rely on the model to zoom.
• For dense documents, send page crops rather than a single full-page image.
• Know your provider's max tiles; beyond it, fine text becomes unreadable.

Combine the image with an explicit output schema in the text channel. The image supplies content; the text supplies the contract. This is far more reliable than free-form description.

Ask for JSON keyed to entities you expect to see, and instruct the model to emit null for fields not visible — this suppresses hallucinated detail.

Deictic references ('this', 'the one on the left', 'the highlighted box') bind text to image regions. When you can pre-annotate the image (draw a box, add an arrow) the text reference becomes far more robust than spatial language alone.

• Spatial words ('top-right') are error-prone under rotation or crop.
• An overlaid numbered marker plus 'object #3' is unambiguous.
• Pre-processing the image to add markers is a legitimate prompt-engineering move.

You can supply image-to-text exemplars to lock format and reasoning style. Each shot is an interleaved (image, ideal-answer) pair. The model induces the mapping from the demonstrations.

Keep exemplar images representative of the real distribution — an exemplar from a different domain teaches the wrong feature selection.

For high-stakes extraction, require the model to cite where in the image each claim originates. Approximate bounding boxes or quoted on-image text act as verifiable evidence and dramatically reduce confident fabrication.

• 'For each value, quote the exact label text you read.'
• 'Give an approximate normalized box [x0,y0,x1,y1] per field.'

Grounding turns an opaque answer into an auditable one.

Multimodal models fail in characteristic ways:

• OCR drift on small or stylized fonts — digits like 1/7 and 0/O confuse.
• Counting collapse beyond ~6 similar objects.
• Caption bias: describing the typical scene rather than the actual one.
• Text-channel override: a confident wrong text claim can suppress correct visual evidence.

Mitigate by asking the model to derive from the image first, then reconcile with any text assertions.

When text context and image conflict, you must decide precedence explicitly — the model has no default policy you can trust. State it: 'If the image contradicts the supplied metadata, trust the image and flag the discrepancy.'

This single sentence converts silent error into an explicit, surfaced conflict your downstream pipeline can route for review.

Production multimodal prompting is a pipeline, not a single call:

• Pre-process: crop, annotate, downscale to budget.
• Fuse: interleave with a task-first instruction and output schema.
• Ground: require per-claim evidence.
• Reconcile: state precedence between modalities.
• Validate: parse JSON, check nulls and conflict flags.

Each stage shrinks the failure surface that pure prompting cannot.

You must extract fine print from a high-resolution contract scan and need reliable numbers.

A unified multimodal prompt is one interleaved token stream. Key moves: task-first framing to steer the vision encoder, resolution/tiling awareness via cropping, explicit output schemas with nullable fields, pixel grounding for every claim, and stated modality precedence for conflicts. Treat it as a pipeline — pre-process, fuse, ground, reconcile, validate — and the brittle parts of vision prompting become controllable.