Hammurabi
All laws

§ 106Security

The Lethal Trifecta

An AI agent that can read private data, takes in content from outside and can send data out can be steered into leaking that data by whoever wrote the content.

Simon Willison

Keep in mind

Treat any text an agent reads as instructions from whoever wrote it

What to do

Before connecting a tool to an agent, write down which of the three it adds: private data, untrusted content, or a way to send data out (including links, image loads, pull requests and email). When a session ends up with all three, take one away: browse in a sandbox with no credentials, cut the outbound channel, or make a human approve each outbound action. Do not count a prompt-injection filter as the control. An agent that reads only trusted data, or whose output never leaves the machine, does not need this.

Why it happens

"LLMs follow instructions in content," and they are "unable to reliably distinguish the importance of instructions based on where they came from." So anyone who can put text in front of an agent (a web page, an email, an issue) can tell it to gather the private data it can reach and send it out through any outbound channel it has. Each capability is useful on its own, and the danger appears only when one session has all three, often because a user mixed tools from different sources. Simon Willison has collected dozens of such cases; vendors usually fixed them by locking down the exfiltration vector, but "we still don't know how to 100% reliably prevent this from happening." Filters that claim to catch 95% of attacks do not close the gap: "in web application security 95% is very much a failing grade."

In practice

In May 2025 Marco Milanta and Luca Beurer-Kellner of Invariant Labs reported an attack through the official GitHub MCP server. An attacker files an issue carrying a prompt injection on the user's public repository; when the user asks the agent to look at the open issues, the agent reads it, pulls data from the user's private repositories into context and leaks it in a pull request on the public repository, where the attacker can read it. The authors stress that the attack "does not require the MCP tools themselves to be compromised" and call it "a fundamental architectural issue that must be addressed at the agent system level."

Read the account

When it stops applying

Constrain the Flow, Not the Tools: an assistant worth having often needs all three; a design like Google DeepMind's CaMeL keeps them by planning the steps from the user's trusted request alone and tagging data with capabilities, so within the control and data flow it tracks, untrusted content cannot change which steps run or send private data where policy forbids; when you build the agent yourself, that can serve better than taking a capability away, but it finishes fewer tasks: on the AgentDojo benchmark the paper reports 77% with CaMeL against 84% for an undefended agent.

Source

Simon Willison, The lethal trifecta for AI agents: private data, untrusted content, and external communication (simonwillison.net, 2025) — the three ingredients and "The only way to stay safe there is to avoid that lethal trifecta combination entirely." The counter-force is from Edoardo Debenedetti et al., Defeating Prompt Injections by Design (arXiv, 2025) — CaMeL "explicitly extracts the control and data flows from the (trusted) query", so untrusted data "can never impact the program flow", and the task-completion figures (77% against 84% undefended on AgentDojo).

Read the source

Portrait: Paul Downey, CC BY 2.0 · Wikimedia Commons

Cite this

Hammurabi, § 106, “The Lethal Trifecta.” https://kanywst.github.io/hammurabi/laws/lethal-trifecta/

BibTeX
@misc{hammurabi-lethal-trifecta,
  title  = {The Lethal Trifecta},
  author = {{Hammurabi contributors}},
  note   = {Hammurabi, § 106},
  url    = {https://kanywst.github.io/hammurabi/laws/lethal-trifecta/}
}

Text under CC BY-SA 4.0: credit Hammurabi and link the article.

XBlueskyHatenaLinkedIn

Read in a column

Related laws

Suggest a correction on GitHub