What Is Prompt Injection, and How Do You Prevent It in AI Agents?
By Devansh Goel, November 12, 2026
- Prompt injection happens when an LLM treats text it reads as instructions to follow. No model can fully block it today.
- In an agent, an injected instruction can call tools, read private data and send that data out.
- Assume the model will sometimes be tricked, and limit what it can reach and send when it is.
- Test for it in CI and log every tool call in production.
Why prompt injection is the agent problem
Prompt injection tops the OWASP Top 10 for LLM Applications (LLM01:2025), and its agent form, Agent Goal Hijack, leads the OWASP Top 10 for Agentic Applications for 2026 (ASI01). Every LLM app has this weakness by default, and no model vendor has shipped a complete fix.
When a chatbot is injected, the damage is usually limited to what it says. Agents connected to email, databases and internal APIs can also act on the attacker's instructions.
What is prompt injection?
Prompt injection is an attack where untrusted text changes what an LLM does, overriding its developer's instructions. It works because a model receives instructions and data in one stream of tokens and can't reliably tell them apart. SQL injection was fixed by keeping code and data separate; LLMs have no equivalent yet.
| Direct injection | Indirect injection | |
|---|---|---|
| Who writes it | The user, in the chat box | A third party, in content the agent reads |
| Where it hides | The prompt | Emails, web pages, tickets, code, tool descriptions, RAG documents |
| Main risk | Policy bypass and jailbreaks | An outsider steering an agent that holds someone else's permissions |
Indirect injection is the bigger threat for agents: the attacker never talks to your agent at all.
Prompt injection in shipped products
Security researchers have found prompt injection in agents from Microsoft, Salesforce, GitHub, Supabase and Slack. Six of their disclosures:
| Date | Incident | Payload hid in | What the attacker got |
|---|---|---|---|
| Sep 2025 | ForcedLeak, Salesforce Agentforce | A public web-to-lead form | CRM data, via an image sent to an expired allowlisted domain (CVSS 9.4) |
| Aug 2025 | CVE-2025-53773, GitHub Copilot in VS Code | Code, web pages, issues | Remote code execution: the agent switched on its own auto-approve setting |
| Jul 2025 | Supabase MCP with Cursor | A support ticket | A private tokens table, written back into the ticket |
| Jun 2025 | EchoLeak, Microsoft 365 Copilot | One unopened email | Copilot context data, with zero clicks (CVSS 9.3) |
| May 2025 | GitHub MCP | A public issue | Private repo contents, published in a public PR |
| Aug 2024 | Slack AI | A public channel message | Secrets from private channels |
In every case the payload arrived as ordinary content the agent had been asked to read, and the agent held more access than its task required. In three of them (ForcedLeak, EchoLeak and Slack AI) the data left through a rendered image or link. Most of the vendor fixes changed what the agent could load or reach, for example by blocking external images or enforcing URL allowlists.
Why the obvious fixes fall short
| Common fix | Why it isn't enough |
|---|---|
| Telling the model to ignore instructions in documents | The defense and the attack are written in the same language and read by the same model |
| Filtering for "ignore previous instructions" | Attackers paraphrase, translate, encode, or hide text in HTML comments |
| A second LLM that flags malicious input | A useful signal, but probabilistic and injectable, and attackers get unlimited retries |
| Waiting for a stronger model | Newer models resist more attacks, but published attacks still succeed against them |
All four try to stop the model from being fooled. The controls below start from the opposite assumption: the model will sometimes follow an attacker, so the system should limit what it can reach and send when that happens.
Six controls that limit the damage
Put controls where the model's decisions turn into actions, so they hold even when the model is fooled.
- Remove at least one condition of the lethal trifecta for each task. An agent can leak data only when it has private data, untrusted content and a way to send data out at the same time. A coding agent working on a public issue, for example, has no reason to reach the network.
- Run tools as the end user, read-only by default and scoped to one repo, mailbox or table. A summarization step has no use for
send_email. - Check every tool call in code before it runs, outside the model, so a persuasive prompt can't change the outcome. There's an example below.
- Ask a human before high-impact actions, and show the exact arguments and where they came from. Avoid "allow all" buttons on sensitive tools.
- Treat model output as untrusted input. Block images and links to unapproved domains, set a strict CSP and an egress allowlist, and keep output away from shells, eval and raw SQL.
- Run an injection test suite in CI whenever the model, prompt or tools change. In production, log every tool call and alert when a sensitive read is followed by an outbound request.
A minimal version of the check in item 3:
def authorize(call, session):
if call.tool not in session.task.allowed_tools:
return deny("tool not in task scope")
if call.tool in SENSITIVE and session.context_tainted:
return require_human_approval(call) # untrusted text is in context
if call.tool == "http_request" and host(call.args["url"]) not in ALLOWED_DOMAINS:
return deny("egress to unapproved domain")
return allow()
Two other measures reduce risk but shouldn't be relied on alone. Marking untrusted content cut attack success from over 50% to under 2% in Microsoft's spotlighting tests. Design patterns such as plan-then-execute and dual LLM (six are catalogued here), along with Google DeepMind's CaMeL, stop injected text from adding new actions, at some cost to flexibility. MCP servers need supply-chain hygiene as well: keep an inventory, pin versions and alert when a tool description changes.
How NuGuard helps
NuGuard is an AI trust platform for behavioral validation of agentic AI. It helps teams prove that an agent does only what it is supposed to do before it reaches production.
- An AI-SBOM lists each agent's tools, permissions and identity, which shows where all three trifecta conditions meet.
- Automated red-teaming runs more than 100 adversarial scenarios, prompt injection included, against the agent's real tools.
- A CI/CD gate blocks a deployment when an agent's risk score rises, and reports map results to the OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF and the EU AI Act.
NuGuard's core is open source. Try the CLI on GitHub or book a demo to see it run against your own agents.
Sources
- OWASP, LLM01:2025 Prompt Injection and Top 10 for Agentic Applications for 2026
- Simon Willison, The lethal trifecta for AI agents (June 2025)
- Beurer-Kellner et al., Design Patterns for Securing LLM Agents against Prompt Injections (2025)
- Debenedetti et al., Defeating Prompt Injections by Design (CaMeL) (2025)
- Hines et al., Defending Against Indirect Prompt Injection Attacks With Spotlighting (2024)
- Incident write-ups are linked in the attack table above.
Related Articles
How Do You Secure AI Agents Before Production?
A 10-point release gate for AI agents, mapped to the OWASP Top 10 for Agentic Applications, covering identity, tool access, prompt injection testing, sandboxing, human approval and a tested kill switch.
AI Red-Teaming in 2026: NuGuard vs. Promptfoo, PyRIT, and Garak
A practical comparison of Promptfoo, PyRIT, Garak, and NuGuard across the capabilities security teams need to test modern AI applications before they ship.