The new engineering discipline nobody's hiring for yet: AI Ops

01.The gap

When a company ships a conventional service, it knows who runs it. There are on-call engineers, release processes, monitoring and incident reviews. Over decades, the industry has built roles and habits around reliability.

Agents have none of this yet. The team that builds an agent often ships it and moves on. Prompts are edited in place. Models are swapped when a cheaper one appears. Nobody owns a list of what good looks like, and nobody reviews failures on a schedule. The agent behaves as well as its last test, and no better.

02.What an AI Ops engineer does

The role has five main parts.

• Builds eval sets. Collects real cases, defines what a good outcome looks like with domain experts, and keeps the set up to date as the product changes.
• Monitors quality. Watches scores, hand-off rates and drift in production. Knows what normal looks like and notices when it moves.
• Investigates incidents. When an agent does something wrong, reads the trace, finds the cause, writes it up and proposes a fix.
• Controls change. Treats prompts, tools and model versions as releases. Requires an evaluation run before anything goes live, and keeps a record of what changed and when.
• Manages cost. Tracks spend per task, finds loops and wasteful calls, and weighs cheaper models against quality.

03.A day in the role

A typical week might look like this. On Monday, the engineer reviews the lowest-scoring runs from the weekend and tags the causes. On Tuesday, a product team proposes a prompt change, and the engineer runs the evaluation set and reports two regressions. On Wednesday, a cost alert shows one workflow has doubled its spend, and a trace shows a retry loop. On Thursday, the engineer adds three new cases from real failures to the evaluation set. On Friday, they write a short note for the team on what changed and what to watch.

Nothing here is glamorous. It is the work that keeps an agent trustworthy over time.

04.Why now

Teams are shipping agents faster than they are building the ability to run them. That gap will produce incidents. Some will be small and embarrassing. Some will reach customers or regulators.

Early hires in this area shape how the company operates agents for years. The habits set now, such as requiring evaluations before release, or reviewing failures weekly, become part of the culture. Companies that build the discipline early will ship changes with confidence. Companies that do not will move slowly, because every change feels risky.

05.Skills to look for

• Software engineering skills, especially testing and debugging.
• Comfort with data: sampling, labelling and reading distributions.
• Clear writing, since incident notes and eval criteria need to be precise.
• Judgment about risk, cost and when to involve a person.
• Curiosity about how models fail.

You may not find all of this in one person. Many teams start by giving an existing engineer a share of the work and clear ownership.

06.Why existing roles do not quite fit

Teams often ask whether an existing role can absorb this work. Sometimes it can, but each has a gap.

• Software engineers build the agent, but may not have the habit of sampling outputs and judging them. They are used to deterministic tests.
• Data scientists are good at evaluation, but may not own production incidents or release processes.
• Site reliability engineers know how to run services, but agent quality is not a matter of uptime and latency alone.
• Support and operations staff see the failures first, but rarely have the tools to investigate them.

An AI Ops engineer sits between these groups. They take the evaluation instincts of data work, the discipline of operations and the practical skill of engineering, and aim them at agent behaviour.

07.An incident, step by step

Imagine an agent that routes invoices for approval. After a model update, it begins to send high-value invoices to the wrong approver for one department. No error is raised. A finance manager notices a delay and reports it three days later.

A team with AI Ops practice would handle it like this:

• An alert fires on a change in routing patterns, or the evaluation run before the model update flags the regression.
• The on-call engineer opens traces for the affected runs and sees the wrong approver chosen after a change in how the model reads department names.
• They roll back the model version or the prompt, and confirm with a targeted test.
• They add the failing cases to the evaluation set.
• They write a short note on cause, impact and prevention.

A team without it finds out from the finance manager, spends a day guessing at the cause and fixes it in production with no test.

08.Building the practice in a small team

You do not need a new department. Start small.

• Give one person named ownership of agent quality, even if it is a part of their job.
• Keep a living document of what each agent should and should not do.
• Review a sample of runs together each week, so that everyone sees real behaviour.
• Treat prompt, tool and model changes as releases, with notes and rollback plans.
• Measure a few things well: quality score, cost per task, hand-off rate and incident count.

As volume grows, split the work: someone owns evaluations, someone owns monitoring, someone owns change control. The habits matter more than the titles.

09.Common mistakes

• Assigning ownership to everyone. If everyone owns agent quality, nobody does.
• Treating evaluation as one-off. The set must grow with real failures.
• Separating ops from builders. The best results come when the people who build agents and the people who run them talk daily.
• Waiting for an incident. Setting up basics after a customer-facing failure is harder than before.

10.What to do next

• Name an owner for each production agent.
• Create a first evaluation set from 30 real cases.
• Require an evaluation run before any prompt or model change.
• Hold a weekly 30-minute review of the worst runs.
• Track cost per task and set a limit.

Write the answers on one page and keep it where the on-call engineer can find it. Teams that settle these questions before the first serious failure tend to handle that failure calmly. Teams that do not tend to argue about them during it.

Brandsmashers supplies engineers who set up tracing, evaluation and alerting, and who can help your team build this discipline from the start. If you are deciding how to staff it, book a call and we will talk through the options.