CASE STUDY · DEVOPS / PLATFORM ENGINEERING & SRE AUGMENTATION
Standing up a dedicated SRE pod that hit 99.9% uptime in 90 days.
- 99.2%Uptime up from 99.2% to 99.9% over 90 days
- 3 hoursMean time to recover down from 3 hours to 40 minutes
- 4Outages down from 4 a month to 1
01 · PROJECT OVERVIEW
The client and the brief.
An anonymised mid-market SaaS company. Name withheld under confidentiality.
A 4-person pod defining SLOs and error budgets, building monitoring and alerting, and setting up an on-call rota and incident reviews.
- 1The product was stable enough to win customers but not stable enough to keep them happy.
- 2The company saw about 4 outages a month.
- 3There was no on-call process: whoever noticed a problem fixed it, often a developer in the middle of other work.
- 4Customers found issues before the team did.
- 5Nobody had agreed what "reliable enough" meant, so reliability lost every argument against new features.
02 · WHAT WENT WRONG
What went wrong, and how we fixed it.
Every engagement hits problems. These were the ones that mattered here, and what we changed so they did not come back.
- PROBLEM 01In the first weeks the new alerts paged far too often
We had set thresholds from a short window of data, and normal evening traffic crossed them. Engineers started to mute the channel, which defeated the point of building it.
- PROBLEM 02We stopped, reviewed every alert that fired in a week and asked one question: did a human need to act?
Alerts that failed the test were removed or turned into tickets. We rebuilt the remaining ones around error budget burn. Pages dropped to a level the rota could sustain, and trust in the pager came back.
- PROBLEM 03Pages dropped to a level the rota could sustain, and trust in the pager came back
- PROBLEM 04The first incident reviews turned into long meetings with vague actions
People were wary of saying what had really happened. We shortened the format, wrote the timeline before the meeting, and began each review by saying that the aim was to fix the system, not to find a person. Actions got one owner and a date.
03 · THE PROCESS
How we ran it.
The engagement ran for 90 days, in three stages.
- 01Weeks 1 to 3, baseline
we read past incidents, mapped the main user journeys and measured current availability. We interviewed engineers about what woke them up.
- 02Weeks 4 to 8, instrument and agree
we set SLOs for the main journeys with product and engineering leads, then built dashboards and alerts against them. We started the on-call rota with shadow shifts.
- 03Weeks 9 to 13, operate and improve
the rota ran for real, every incident got a review, and the error budget started to steer priorities.
The pod joined the client's standup daily. A weekly reliability review with engineering and product leads tracked incidents, actions and budget burn.
04 · RESULTS
What changed.
Measured against the way the work ran before the engagement began.
- NOW 01
The team has a written on-call rota with escalation and fair handover.
- NOW 02
Every significant incident gets a review within days, with actions tracked to completion.
- NOW 03
Product and engineering use the error budget to decide when to slow feature work.
- NOW 04
Alerts are tied to user impact, and noisy ones are removed on a routine basis.
- 99.2%
Uptime up from 99.2% to 99.9% over 90 days
- 3 hours
Mean time to recover down from 3 hours to 40 minutes
- 4
Outages down from 4 a month to 1
05 · WHAT WE BUILT
How it works.
- We chose a small number of SLOs tied to what customers feel: availability and latency on the key journeys.
- Alerts fired when the error budget was burning too fast, not on every spike in CPU.
- This cut noisy pages sharply.
- Monitoring covered metrics, logs and traces, with a single dashboard per service.
- Runbooks linked from each alert described the first checks and the usual fixes.
- Incident reviews were blameless and ended with owned actions.
- What went wrong, part two
- The first incident reviews turned into long meetings with vague actions. People were wary of saying what had really happened. We shortened the format, wrote the timeline before the meeting, and began each review by saying that the aim was to fix the system, not to find a person. Actions got one owner and a date. After a few reviews, engineers began to bring up near misses on their own.
- Handover
- The client's senior engineers took over as incident commanders in the final weeks, with our pod shadowing. We left dashboards, runbooks, severity definitions and a template for reviews.
06 · QUICK ANSWERS
The short version.
- What problem did the client have?
- The product was stable enough to win customers but not stable enough to keep them happy. The company saw about 4 outages a month. There was no on-call process: whoever noticed a problem fixed it, often a developer in the middle of other work. Customers found issues before the team did. Nobody had agreed what "reliable enough" meant, so reliability lost every argument against new features.
- What did Brandsmashers build?
- A 4-person pod defining SLOs and error budgets, building monitoring and alerting, and setting up an on-call rota and incident reviews.
- How was the work run?
- The engagement ran for 90 days, in three stages.
- What were the results?
- Uptime up from 99.2% to 99.9% over 90 days; mean time to recover down from 3 hours to 40 minutes; outages down from 4 a month to 1.
YOUR TURN
Have a similar requirement?
Tell us what is stuck. We will come back with a plan and matched, pre-vetted engineers.
