Posted in Incident Response

When the email becomes the instruction An AI agent built to automate invoice processing was manipulated into…

Schomer — Principal Security Solutions Architect & Founder

When the email becomes the instruction

An AI agent built to automate invoice processing was manipulated into changing vendor bank

details - by a malicious email.

No exploit chain. No traditional security alert fired. The agent read the email, treated its

contents as work to do, and used the access it already had to make a state-changing update:

where the money goes.

That is the part worth sitting with. The failure was not that the model believed something false.

The failure was architectural:

The agent had an identity with permission to change payment details.

Its tools let it make that change directly.

Untrusted input - an inbound email - was allowed to drive a trusted action.The same pattern shows up in the support-ticket version reported this week: a ticket contains

instructions, the agent follows them, and records get exported to an unfamiliar destination -

again, because the identity could and the tools allowed it.

Prompt filtering does not fix that. You cannot filter your way to safety when the job of the agent

is to read untrusted content and act on it.

What fixes the shape of the problem is boring and specific:

Separate reading from acting. Reading an email or a ticket should never, by itself,

authorize a payment change or an export.

Escalate state-changing steps. Bank details, exports, deletions, permission changes - a

human approves those first, every time.

Log the decision, not just the action. Which rule allowed it, which input drove it, who

approved it.

That is how I built my own agent work: state-changing steps pause for approval instead of

running straight through. It is slower on purpose. Changing where money goes should be

slow.

Give an agent a mailbox and a payment tool and you have not automated accounts payable.

You have built a new way for an email to spend money - unless the boundary sits at execution.

- Justin Schomer, Eternal Boundaries

4 reactions · 4 comments

Comments

Victor Font: Schomer, this is an excellent illustration of why AI governance must begin with decision authority rather than model behavior. The agent didn't need to bypass a security control if the architecture already permitted an untrusted instruction to trigger an authorized transaction. That's a fundamental failure in the separation of information, authority, and execution. Your distinction between reading

Schomer: Appreciate that, Victor. "Moving the vulnerability one step downstream" hits the nail on the head. In my own builds, I’ve found that the approval prompt itself has to surface the raw trigger and the verification context—otherwise, human-in-the-loop just becomes an expensive rubber stamp. Governance has to live in the execution layer, not a policy doc.

Ben N.: great insight on AI, Justin

Schomer: thank you ben Nguyen

Read and join the discussion on Cyberr