CASE STUDY · AI AGENT OBSERVABILITY & OPS TOOLING
Building a debugging dashboard for a client's multi-agent AI system.
- 3 hoursTime to diagnose a failure down from 3 hours to 45 minutes
01 · PROJECT OVERVIEW
The client and the brief.
An anonymised mid-market B2B services company. Name withheld under confidentiality.
Tracing of every step and tool call, cost and latency views, and quality scoring. Each run received an identifier that followed it across agents. The dashboard showed the full chain for a run: inputs, prompts, tool calls, outputs, time and cost at each step. A scoring layer rated results so engineers could filter for the bad ones.
- 1Three task-specific agents and failures that were hard to trace.
- 2Each agent handled a different job, and some tasks passed from one agent to another.
- 3When a result was wrong, engineers had to piece together the story from scattered logs, prompts in code and database records.
- 4Diagnosing a single failure could take half a day, and sometimes the cause was never found.
02 · WHAT WENT WRONG
What went wrong, and how we fixed it.
Every engagement hits problems. These were the ones that mattered here, and what we changed so they did not come back.
- PROBLEM 01At first the automated grader scored fluent but wrong answers highly
It liked confident language, and some bad results looked good on the dashboard. Human review of a sample exposed the gap. We rewrote the rubric to check facts against the source data and required the grader to point to evidence.
- PROBLEM 02We also kept a human-rated sample running as a check on the grader, so drift in the scoring would be noticed
- PROBLEM 03A second hurdle was volume
Once every step was traced, storage grew quickly, and the dashboard slowed down when engineers searched across weeks of runs. We kept full detail for a short window, stored summaries and scores for longer, and kept full traces permanently only for runs that scored low or were flagged by an engineer. Search became fast again, and the cost of storage stayed predictable.
- PROBLEM 04Search became fast again, and the cost of storage stayed predictable
03 · THE PROCESS
How we ran it.
- 01Week 1
Read the code and logs for the three agents, listed known failure types and agreed what a single trace should contain.
- 02Weeks 2 to 3
Added tracing to the first agent, then the second and third. Checked that traces from different agents linked together.
- 03Weeks 4 to 5
Built the dashboard with engineers, using their last real incidents as test cases.
- 04Weeks 6 to 7
Added quality scoring and a review queue for low scores.
- 05After launch
Fortnightly session to review the worst runs and tune scoring.
04 · RESULTS
What changed.
Measured against the way the work ran before the engagement began.
- NOW 01
Engineers open a trace first, not the logs.
- NOW 02
Incident reviews begin from the recorded run and focus on the cause.
- NOW 03
The team reviews the lowest-scoring runs on a regular schedule and uses what they find to improve prompts and tools.
- NOW 04
Cost per task is visible, so an expensive loop gets noticed in days, not at the end of the month.
- 3 hours
Time to diagnose a failure down from 3 hours to 45 minutes
05 · WHAT WE BUILT
How it works.
- We wrapped model calls and tool calls with a thin tracing layer that recorded inputs, outputs, timings and token use under a shared run identifier.
- Spans were stored in a trace store with search by run, agent, tool, time and score.
- Sensitive fields were masked before storage, and access was limited by role.
- Quality scoring combined simple rules, such as output format and required fields, with a model-based grader against a rubric.
- We compared the grader's scores with human ratings on a sample, and adjusted the rubric until they agreed on the main failure types.
- Cost and latency were rolled up per run and per agent.
- Controls after launch
- Alerts cover cost per run, loop detection and a sudden rise in failed tool calls. Each alert names an owner and links to an example trace. Access to traces follows role, and masked fields stay masked in exports. Retention periods are written down and enforced by the store.
- Handover
- The client's engineers learned to add tracing to a new agent from a short template, to adjust scoring rubrics and to read a trace during an incident. We left a runbook with the common failure patterns found during the project, such as an agent passing the wrong identifier to a tool, and how each looks in a trace.
06 · QUICK ANSWERS
The short version.
- What problem did the client have?
- Three task-specific agents and failures that were hard to trace. Each agent handled a different job, and some tasks passed from one agent to another. When a result was wrong, engineers had to piece together the story from scattered logs, prompts in code and database records. Diagnosing a single failure could take half a day, and sometimes the cause was never found.
- What did Brandsmashers build?
- Tracing of every step and tool call, cost and latency views, and quality scoring. Each run received an identifier that followed it across agents. The dashboard showed the full chain for a run: inputs, prompts, tool calls, outputs, time and cost at each step. A scoring layer rated results so engineers could filter for the bad ones.
- What were the results?
- Time to diagnose a failure down from 3 hours to 45 minutes.
YOUR TURN
Have a similar requirement?
Tell us what is stuck. We will come back with a plan and matched, pre-vetted engineers.
