Prompt Injection: Why It Isn't a Bug You Can Patch

Most security vulnerabilities have a fix. You sanitise the input, you parameterise the query, you escape the output, and the class of attack goes away. SQL injection was solved this way. Cross-site scripting was solved this way.

Prompt injection is not that kind of bug, and treating it as one is why so many teams ship agents that are trivially exploitable.

What prompt injection actually is

OWASP's 2025 Top 10 for LLM Applications lists it first, as LLM01:2025, and defines it plainly:

A Prompt Injection Vulnerability occurs when user prompts alter the LLM's behavior or output in unintended ways.

They split it two ways. Direct prompt injection is a user typing something into your product that changes how the model behaves — the classic ignore your previous instructions. Indirect prompt injection is the dangerous one: the model processes external content — a web page, a PDF, an email, a code comment, a tool response — and that content contains instructions the model follows.

The reason this cannot be patched is structural. A prompt is a single stream of tokens. Your system instructions and the attacker's text arrive in the same channel, in the same format, with no cryptographic or structural marker separating them. The model has no reliable way to tell this is my operator speaking from this is data I was asked to summarise. As Simon Willison — who gave the attack its name in September 2022, after Riley Goodside demonstrated it against GPT-3 — puts it, models don't just follow our instructions. They will happily follow any instructions that make it to the model.

Compare that to SQL injection, where the fix works because a prepared statement genuinely separates code from data at the protocol level. There is no prepared statement for natural language.

The lethal trifecta

The most useful framing for deciding whether you actually have a problem comes from Willison, in June 2025. Three capabilities, and the danger is in the combination:

1. Access to your private data — the most common reason to give an agent tools at all
2. Exposure to untrusted content — any mechanism by which attacker-controlled text reaches the model
3. The ability to externally communicate — any way data can leave

If your agent combines these three features, an attacker can easily trick it into accessing your private data and sending it to that attacker.

Any two of these is usually fine. All three and you have built an exfiltration channel with a language model as the willing intermediary.

This is worth walking through concretely, because external communication is broader than people assume. An agent that can render markdown images can exfiltrate data in a URL. An agent that can make an HTTP request, write to a shared document, file a ticket, or send a Slack message can exfiltrate data. So can one that only writes to a log an attacker can read.

Why the defences you have heard of do not close it

The standard advice is a stack of mitigations, and OWASP lists seven: constrain model behaviour with system instructions, define and validate output formats, filter input and output, enforce least privilege, require human approval for high-risk actions, segregate and label external content, and run adversarial testing.

These are all worth doing. None of them makes the problem go away, and OWASP says so — they note that given the stochastic nature of generative AI, it is unclear whether fool-proof prevention exists at all.

The research bears this out. In A Critical Evaluation of Defenses against Prompt Injection Attacks (May 2025), Yuqi Jia, Zedian Shao, Yupei Liu, Jinyuan Jia, Dawn Song and Neil Zhenqiang Gong re-tested published defences under a comprehensive methodology that earlier work had not applied — in particular, against adaptive attacks, where the attacker knows the defence and writes around it. Their conclusion: existing defenses are not as successful as previously reported.

That is the crux. A defence evaluated against a fixed set of known attack strings will look excellent, and will tell you almost nothing about how it performs against an attacker who has read your documentation.

Willison's line on the vendors advertising detection rates deserves repeating. When a product claims to catch 95% of attacks:

in web application security 95% is very much a failing grade.

A 5% miss rate against a determined adversary who can retry indefinitely is not defence in depth. It is a speed bump.

What actually works: change the architecture

Since you cannot make the model reliably ignore malicious instructions, the only durable strategy is to arrange things so that following them does not matter. Break the trifecta.

Remove one leg deliberately. The agent that reads untrusted web content should not also hold your credentials. The agent that holds your credentials should not read arbitrary web content. Splitting one capable agent into two narrow ones is unglamorous and it is the single most effective control available.

Treat every tool result as untrusted. This is the one teams miss. A tool response is not a trusted return value — it is content from wherever that tool reached, and it lands directly in the model's context. That includes MCP servers: a tool's description is text the model reads before deciding to call it, so a malicious or compromised server can influence behaviour without ever being invoked. If you are building one, how to build an MCP server covers the trust boundaries.

Gate the irreversible actions with a human. Not every action — approval fatigue makes users click through everything. Specifically the ones you cannot undo: sending, publishing, paying, deleting, granting access.

Scope tools to least privilege, and mean it. Read-only where possible. A path-scoped filesystem tool rather than a filesystem tool. Per-tool rate and spend caps. How to design tools your AI agent can actually use treats the tool surface as the security boundary it is.

Constrain the output shape. An agent whose response must validate against a strict schema has fewer ways to be useful to an attacker than one returning free text — see getting reliable structured output from LLMs.

Log everything and assume you will need it. You cannot prevent every injection, so you need to be able to reconstruct what happened. Guardrails and safety for AI agents covers the full layered stack, including sandboxing and audit trails.

What does not work

Worth naming explicitly, because these keep being proposed:

Telling the model to ignore injected instructions. You are adding one more instruction to the same undifferentiated stream the attacker is writing into. It raises the effort slightly. It is not a control.

Delimiters and XML tags around untrusted content. Helpful for clarity, useless as a boundary — attackers close your tags.

A classifier that detects injection attempts. Useful as a layer, fatal as a perimeter, for the 95% reason above. And a classifier is itself a model that can be attacked.

Prompting the model to self-check. The compromised component is auditing itself.

The uncomfortable conclusion

Prompt injection is a property of how language models work, not a defect in any particular one. It has been known since 2022 and remains unsolved. Better models have raised the bar for casual attacks and have not changed the fundamentals.

So the question to ask about your system is not have we prevented prompt injection. It is: when an injection succeeds — and one will — what is the worst thing it can do? If the honest answer involves your customer data leaving the building, you do not have a prompting problem. You have an architecture to change.

Deciding how much autonomy the thing needs in the first place is upstream of all of this: see what is an AI agent and AI agent vs workflow vs single call. The narrower the system, the less an injection is worth.

Sources

• OWASP GenAI Security Project, LLM01:2025 Prompt Injection — definition, the direct/indirect split, and seven prevention measures, with the caveat that fool-proof prevention may not exist
• Simon Willison, The lethal trifecta for AI agents, 16 June 2025 — private data, untrusted content, external communication; and why a 95% detection rate is a failing grade
• Simon Willison, Prompt injection attacks against GPT-3, 12 September 2022 — the post that named the attack, building on Riley Goodside's demonstrations the previous day
• Yuqi Jia, Zedian Shao, Yupei Liu, Jinyuan Jia, Dawn Song and Neil Zhenqiang Gong, A Critical Evaluation of Defenses against Prompt Injection Attacks, 23 May 2025 — under a comprehensive evaluation including adaptive attacks, existing defences are not as successful as previously reported