CASE STUDY · AI AGENT SECURITY & GOVERNANCE
Catching a prompt-injection vulnerability before it shipped.
- 1212 injection test cases now run on every release
01 · PROJECT OVERVIEW
The client and the brief.
An anonymised mid-market B2B services company. Name withheld under confidentiality.
A scoped penetration test, input handling fixes, an approval step for risky actions, and the test cases added to CI.
- 1A task agent read untrusted input from a source system and could act on instructions hidden in it.
- 2The agent processed requests that arrived from an external source system and then updated records in an internal tool.
- 3The text in those requests came from outside the company.
- 4That meant anyone who could write into the source could, in principle, write instructions the agent might follow.
- 5The team had tested that the agent did its job.
02 · WHAT WENT WRONG
What went wrong, and how we fixed it.
Every engagement hits problems. These were the ones that mattered here, and what we changed so they did not come back.
- PROBLEM 01Our first round of fixes only cleaned the input text
A subtler attack still got through: the instruction was split across two fields that the agent read together. Cleaning text was the wrong layer to rely on. The durable fix was to limit what the agent could do, so that even a successful injection could not cause harm.
- PROBLEM 02We narrowed the tool permissions, added the approval step, and added the split-field case to the tests
03 · THE PROCESS
How we ran it.
The engagement was a short, scoped test ahead of a planned launch.
- 01Scoping
we agreed the target, the allowed test methods and the stop conditions in writing.
- 02Testing
we built a set of malicious inputs, from crude ("ignore your instructions") to subtle (instructions disguised as a customer note or a field value), and ran them against a staging copy.
- 03Fix and retest
the team fixed issues as we reported them, and we retested each fix.
- 04Hand-over
we turned the successful attacks into regression tests and trained the team to add more.
We shared findings the same day we confirmed them, so the engineers could fix them while the context was fresh.
04 · RESULTS
What changed.
Measured against the way the work ran before the engagement began.
- NOW 01
The team treats any text from outside the company as hostile by default.
- NOW 02
Every new tool given to an agent is reviewed for what damage it could do if the agent were tricked.
- NOW 03
When someone finds a new attack pattern, it becomes a test case the same day.
The vulnerability found and fixed before launch
- 12
12 injection test cases now run on every release
05 · WHAT WE BUILT
How it works.
- Untrusted text is now passed to the model as clearly marked data, separate from the instructions.
- The agent's tools were narrowed.It could previously update any field on a record. Now it can change only the fields its task requires.
- Actions that change records outside a safe set require approval from a person, who sees the original input next to the proposed action.
- Outputs are checked against an allow-list of actions before they run.A model output that asks for an action outside the list is rejected and logged.
- The attack inputs live in the test suite and run on every release.
- Monitoring and controls
- Handover
- We ran a half-day session with the engineers on how to write new injection cases, and left a short guide listing the attack patterns we used and why each worked. Two engineers added new cases of their own before we finished.
06 · QUICK ANSWERS
The short version.
- What problem did the client have?
- A task agent read untrusted input from a source system and could act on instructions hidden in it.
- What did Brandsmashers build?
- A scoped penetration test, input handling fixes, an approval step for risky actions, and the test cases added to CI.
- How was the work run?
- The engagement was a short, scoped test ahead of a planned launch.
- What were the results?
- The vulnerability found and fixed before launch; 12 injection test cases now run on every release.
YOUR TURN
Have a similar requirement?
Tell us what is stuck. We will come back with a plan and matched, pre-vetted engineers.
