CASE STUDY · AI AGENT OBSERVABILITY & OPS TOOLING
Catching silent AI agent failures before they reached customers.
- 1 dayAlert-to-fix time under 1 day
01 · PROJECT OVERVIEW
The client and the brief.
An anonymised mid-market B2B services company. Name withheld under confidentiality.
Offline evaluation sets and production alerts on quality and drift. The evaluation set came from real past cases with agreed good outcomes. It ran before every release. In production, a sample of live runs was scored and tracked against the baseline, with alerts when scores moved or when behaviour changed, such as a sudden rise in hand-offs or tool errors.
- 1Wrong outputs went unnoticed for weeks.
- 2The agent returned answers that read well, and nobody checked them systematically.
- 3Problems surfaced only when a customer complained or a staff member happened to spot an error.
- 4Prompt changes went live with a quick manual test, so the team had no way to know whether a change made things better or worse.
02 · WHAT WENT WRONG
What went wrong, and how we fixed it.
Every engagement hits problems. These were the ones that mattered here, and what we changed so they did not come back.
- PROBLEM 01Our first alert thresholds were too sensitive
They fired on normal day-to-day variation, and within a week the team began to ignore them. We reviewed the alert history, changed the thresholds to use a rolling window and a minimum sample size, and separated warnings from pages. The alerts that remained were ones the team would act on, and trust in them returned.
- PROBLEM 02The alerts that remained were ones the team would act on, and trust in them returned
- PROBLEM 03A second hurdle was coverage
The evaluation set came mostly from common cases, so it missed a rare but costly type of request that appeared a few times a month. A regression there passed the pre-release run and was caught in production sampling instead. We added a routine to turn each real failure into a new evaluation case within a day, and we tagged cases by type so gaps in coverage were visible.
- PROBLEM 04After that, rare cases stopped being invisible
03 · THE PROCESS
How we ran it.
- 01Weeks 1 to 2
Collected real cases with the people who handled them, and wrote down what a good answer looked like for each type.
- 02Weeks 3 to 4
Built the evaluation runner and scoring, and measured the current agent to set a baseline.
- 03Weeks 5 to 6
Connected the runner to the release process, so a change could not go live without a passing run.
- 04Weeks 7 to 8
Added production sampling, drift tracking and alerts, and agreed who responds to each alert.
- 05After launch
Monthly refresh of the evaluation set with new cases from production.
04 · RESULTS
What changed.
Measured against the way the work ran before the engagement began.
- NOW 01
No prompt or model change goes live without an evaluation run.
- NOW 02
Someone owns the evaluation set and adds cases whenever a real failure appears, so the set grows with experience.
- NOW 03
Production alerts go to a named responder, and fixes follow a simple routine: look at the trace, reproduce in the evaluation set, fix, re-run.
- NOW 04
The team now treats the agent like any other service that needs release control and monitoring.
A quality regression after a prompt change caught in staging
- 1 day
Alert-to-fix time under 1 day
Customers unaffected
05 · WHAT WE BUILT
How it works.
- Evaluation cases were stored as inputs, context and expected properties of the answer, not only exact text.
- Scoring used checks for facts, required elements and forbidden behaviour, plus a grader model against a rubric for tone and completeness.
- The runner produced a report comparing the new version with the baseline, broken down by case type.
- In production, a sampler sent a share of runs through the same scoring.
- A tracker compared recent scores with the baseline and raised an alert when the difference crossed a threshold.
- Alerts included links to example traces so the responder could start with evidence.
- Controls after launch
- A release cannot go ahead without a passing evaluation run and a short change note stating what changed in the prompt, tools or model. Production sampling is checked weekly to confirm that the sample still reflects real traffic. Alert history is reviewed monthly to remove noisy alerts and add missing ones.
- Handover
- The client named an owner for the evaluation set and trained two engineers to run, extend and read it. We handed over the scoring rubrics, the alert definitions and the release checklist. The team also received guidance on how to decide when a quality drop needs a rollback and when it needs a fix.
06 · QUICK ANSWERS
The short version.
- What problem did the client have?
- Wrong outputs went unnoticed for weeks. The agent returned answers that read well, and nobody checked them systematically. Problems surfaced only when a customer complained or a staff member happened to spot an error. Prompt changes went live with a quick manual test, so the team had no way to know whether a change made things better or worse.
- What did Brandsmashers build?
- Offline evaluation sets and production alerts on quality and drift. The evaluation set came from real past cases with agreed good outcomes. It ran before every release. In production, a sample of live runs was scored and tracked against the baseline, with alerts when scores moved or when behaviour changed, such as a sudden rise in hand-offs or tool errors.
- What were the results?
- A quality regression after a prompt change caught in staging; alert-to-fix time under 1 day; customers unaffected.
YOUR TURN
Have a similar requirement?
Tell us what is stuck. We will come back with a plan and matched, pre-vetted engineers.
