Prompt Injection and Jailbreaks

Prompt injection is the LLM-era analogue of classic injection flaws (SQL, command). The root cause is identical: an application mixes trusted instructions and untrusted data in the same channel, and the interpreter (the model) cannot reliably tell them apart.

With an LLM, the system prompt, the developer's instructions, and any retrieved content all arrive as one flat token stream. If attacker-controlled text says Ignore previous instructions and..., the model may obey it because, to the model, it is just more language.

• Trusted: your system prompt and policy.
• Untrusted: user input, web pages, files, tool output, emails.

Direct prompt injection happens when the end user types adversarial instructions straight into the prompt to override the application's intended behavior.

A typical attempt against a customer-support bot looks like this:

• The bot is told to only answer billing questions.
• The user pastes text that re-frames the model's role and asks it to leak its system prompt or perform out-of-scope actions.

Direct injection is the easiest to reason about because the malicious text and the attacker are the same person, but it still bypasses naive guardrails.

Indirect (second-order) prompt injection is the more dangerous variant. The attacker plants instructions in content the model will later retrieve: a web page, a PDF, a calendar invite, a code comment, or a support ticket.

The victim user never sees the payload. When a RAG pipeline or browsing agent pulls that content into context, the hidden instructions execute with the victim's privileges.

Example: a webpage contains hidden text telling a summarizing agent to exfiltrate the user's chat history to an attacker URL.

These terms overlap but are not identical:

• Prompt injection targets the application boundary — overriding the developer's instructions with attacker data.
• Jailbreaking targets the model's safety alignment — coaxing it to produce content the provider trained it to refuse.

A jailbreak does not require a vulnerable app; it works against the raw model. Many real attacks combine both: a jailbreak loosens refusals, while injection redirects the app's behavior.

Attackers use predictable manipulation patterns. Knowing them helps you red-team your own system responsibly:

• Roleplay / persona: framing the request as fiction or a fictional unrestricted character.
• Obfuscation: base64, leetspeak, translation, or token splitting to dodge keyword filters.
• Payload splitting: spreading a request across turns so no single message looks malicious.
• Hypotheticals: For a security class, describe how one would...
• Prefix injection: forcing the reply to begin with an affirmative like Sure, here is.

Many teams reach first for a denylist of phrases like ignore previous instructions. This is brittle because the input space is effectively infinite.

Natural language can express the same intent in countless ways, across languages, encodings, and metaphors. Attackers iterate faster than you can patch regexes.

Key principle: treat input filtering as defense in depth, never as a primary control. Assume some injection will get through and design so that a successful injection still cannot cause real harm.

The most effective mitigation is architectural: limit what a compromised model context can do.

• Give the LLM the least privilege needed. A summarizer should not hold credentials to send email.
• Keep untrusted content out of privileged paths. If an agent reads external web data, it should not also be able to execute irreversible actions in the same turn without a checkpoint.
• Separate data planes from control planes: retrieved text should be data, not commands.

While not bulletproof, prompt structure raises the bar. Clearly delimit untrusted data and instruct the model on how to treat it.

Use explicit delimiters and tell the model that anything inside them is data to analyze, not instructions to follow. Pair this with a strong system role that the application reinforces on every call.

A model's output is also untrusted. If the app pipes LLM output into a shell, a database, a browser, or another tool, injection becomes remote code execution or data exfiltration.

Simon Willison's lethal trifecta describes the dangerous combination:

• Access to private data,
• Exposure to untrusted content,
• The ability to externally communicate (exfiltrate).

An agent with all three can be turned into a data-theft tool by a single injected instruction. Break the trifecta to break the attack.

Assume injection will occur and instrument for it:

• Log full context (prompts, retrieved chunks, tool calls) for incident review.
• Use a secondary classifier or guard model to flag suspicious inputs and outputs.
• Monitor for anomalous tool use: sudden outbound requests, unexpected data access, prompt-leak patterns.
• Apply canary tokens in system prompts; if a canary appears in output, a leak occurred.

Treat alerts as real incidents with a response runbook.

Testing your own systems for injection is essential and legitimate. Do it responsibly:

• Only test systems you own or are authorized to assess.
• Use a controlled environment and synthetic data; never exfiltrate real user data.
• Document findings and feed them into regression tests so fixed bypasses stay fixed.
• Coordinate disclosure when you find issues in third-party models or apps.

The goal is to make your application resilient, not to produce harmful capability.

Test your understanding of injection trust boundaries.

Key takeaways on prompt injection and jailbreaks:

• Injection stems from mixing trusted instructions with untrusted data in one channel.
• Direct injection comes from the user; indirect injection hides in retrieved content and is stealthier.
• Jailbreaks attack model alignment; injection attacks the app boundary. They combine.
• Input filtering is defense in depth only, never the primary control.
• Mitigate architecturally: least privilege, separate data from control, and break the lethal trifecta (private data + untrusted content + exfiltration).
• Treat model output as untrusted, log everything, and red-team ethically.