Posted in Incident Response
When the email becomes the instruction An AI agent built to automate invoice processing was manipulated into…
Schomer — Principal Security Solutions Architect & Founder
When the email becomes the instruction
An AI agent built to automate invoice processing was manipulated into changing vendor bank
details - by a malicious email.
No exploit chain. No traditional security alert fired. The agent read the email, treated its
contents as work to do, and used the access it already had to make a state-changing update:
where the money goes.
That is the part worth sitting with. The failure was not that the model believed something false.
The failure was architectural:
The agent had an identity with permission to change payment details.
Its tools let it make that change directly.
Untrusted input - an inbound email - was allowed to drive a trusted action.The same pattern shows up in the support-ticket version reported this week: a ticket contains
instructions, the agent follows them, and records get exported to an unfamiliar destination -
again, because the identity could and the tools allowed it.
Prompt filtering does not fix that. You cannot filter your way to safety when the job of the agent
is to read untrusted content and act on it.
What fixes the shape of the problem is boring and specific:
Separate reading from acting. Reading an email or a ticket should never, by itself,
authorize a payment change or an export.
Escalate state-changing steps. Bank details, exports, deletions, permission changes - a
human approves those first, every time.
Log the decision, not just the action. Which rule allowed it, which input drove it, who
approved it.
That is how I built my own agent work: state-changing steps pause for approval instead of
running straight through. It is slower on purpose. Changing where money goes should be
slow.
Give an agent a mailbox and a payment tool and you have not automated accounts payable.
You have built a new way for an email to spend money - unless the boundary sits at execution.
- Justin Schomer, Eternal Boundaries
4 reactions · 4 comments
Comments
Victor Font: Schomer, this is an excellent illustration of why AI governance must begin with decision authority rather than model behavior. The agent didn't need to bypass a security control if the architecture already permitted an untrusted instruction to trigger an authorized transaction. That's a fundamental failure in the separation of information, authority, and execution. Your distinction between reading
Schomer: Appreciate that, Victor. "Moving the vulnerability one step downstream" hits the nail on the head. In my own builds, I’ve found that the approval prompt itself has to surface the raw trigger and the verification context—otherwise, human-in-the-loop just becomes an expensive rubber stamp. Governance has to live in the execution layer, not a policy doc.
Ben N.: great insight on AI, Justin
Schomer: thank you ben Nguyen