CASE STUDY · DEVOPS / PLATFORM ENGINEERING & SRE AUGMENTATION
Building a Kubernetes platform from scratch for a scaling startup.
- 2Deploys down from a 2-hour manual process to 15 minutes
01 · PROJECT OVERVIEW
The client and the brief.
An anonymised SaaS startup. Name withheld under confidentiality.
A managed Kubernetes cluster, infrastructure as code, automated CI/CD, autoscaling and monitoring.
- 1Deployments were a manual process that depended on one or two people being available.
- 2Each release was slow and stressful.
- 3Environments had drifted apart, so a change that worked in staging could behave differently in production.
- 4At peak load, the application timed out.
- 5The founders were about to sign larger customers and knew the platform would not cope.
02 · WHAT WENT WRONG
What went wrong, and how we fixed it.
Every engagement hits problems. These were the ones that mattered here, and what we changed so they did not come back.
- PROBLEM 01In an early load test, autoscaling worked but the application still timed out
New pods started quickly, yet the database ran out of connections because each pod opened its own pool. The cluster scaled up and took the database down with it.
- PROBLEM 02We found it by watching the database metrics during the test, not just the application ones
The fix had three parts: smaller pools per pod, a connection pooler in front of the database, and a cap on maximum replicas until the database could cope. We repeated the test until it passed cleanly. The lesson stayed with the team: scale the whole path, not one layer.
- PROBLEM 03The lesson stayed with the team: scale the whole path, not one layer
- PROBLEM 04After the first production deploy through the new pipeline, one service failed to start because a secret had been set by hand in the old environment and never recorded
The deploy rolled back cleanly, which was the point of the setup. We moved all secrets into a managed store, referenced from code, and added a check that fails a build if a required secret is missing.
03 · THE PROCESS
How we ran it.
We worked in short stages so the team always had something usable.
- 01Assessment
we reviewed the application, its dependencies and the manual deploy steps. We wrote the steps down, since many only existed in people's memory.
- 02Foundation
infrastructure as code for networking, the cluster and supporting services. A second environment was built from the same code to prove it was repeatable.
- 03Pipeline and migration
automated build, test and deploy, then services moved to the cluster one at a time.
- 04Scale and harden
autoscaling, monitoring, load tests and a handover to the client's engineers.
We took part in the client's standup, and held a weekly demo of working changes.
04 · RESULTS
What changed.
Measured against the way the work ran before the engagement began.
- NOW 01
Any engineer can deploy, because the process is automated and documented.
- NOW 02
Staging and production come from the same code, so surprises are rare.
- NOW 03
Load tests run before major launches and big customer onboardings.
- NOW 04
Infrastructure changes go through review like application changes.
- 2
Deploys down from a 2-hour manual process to 15 minutes
No peak-load timeouts in load tests
Environments reproducible from code
05 · WHAT WE BUILT
How it works.
- A managed Kubernetes service removed the burden of running the control plane.
- Everything else, including networks, cluster settings, secrets handling and deployments, was defined in code and kept in version control.
- A pull request was the only way to change infrastructure.
- The pipeline built a container image, ran tests and scans, and deployed to staging automatically.
- Promotion to production used the same artefact and needed one approval.
- Rollback was a single command to redeploy the previous version.
- What went wrong, part two
- After the first production deploy through the new pipeline, one service failed to start because a secret had been set by hand in the old environment and never recorded. The deploy rolled back cleanly, which was the point of the setup. We moved all secrets into a managed store, referenced from code, and added a check that fails a build if a required secret is missing.
- Controls
- Cluster access was granted by role and logged. Images were scanned in the pipeline. Production changes needed review and one approval, and every change was traceable to a commit.
- Handover
- We ran two sessions with the client's engineers: one on day-to-day deploys, one on cluster upgrades and recovery. They rebuilt a test environment from code during the second session without our help.
06 · QUICK ANSWERS
The short version.
- What problem did the client have?
- Deployments were a manual process that depended on one or two people being available. Each release was slow and stressful. Environments had drifted apart, so a change that worked in staging could behave differently in production. At peak load, the application timed out. The founders were about to sign larger customers and knew the platform would not cope.
- What did Brandsmashers build?
- A managed Kubernetes cluster, infrastructure as code, automated CI/CD, autoscaling and monitoring.
- How was the work run?
- We worked in short stages so the team always had something usable.
- What were the results?
- Deploys down from a 2-hour manual process to 15 minutes; no peak-load timeouts in load tests; environments reproducible from code.
YOUR TURN
Have a similar requirement?
Tell us what is stuck. We will come back with a plan and matched, pre-vetted engineers.
