CASE STUDY · DATA ENGINEERING & MLOPS SERVICES
Rebuilding a client's data pipeline to support real-time AI features.
- 24 hoursData latency down from about 24 hours to under 5 minutes
- 15%About 15% lower avoidable operating cost
01 · PROJECT OVERVIEW
The client and the brief.
An anonymised mid-market B2B software company. Name withheld under confidentiality.
Kafka-based event streaming with managed processing, a versioned event data model, and schema, freshness and duplicate checks.
- 1Data arrived in daily batches, but the product needed near-real-time status updates.
- 2Customers saw yesterday's picture in a product that implied it was live.
- 3Support tickets asked why numbers had not moved.
- 4The batch jobs were also fragile.
- 5A failure at night often meant a manual re-run the next morning, and nobody knew about bad data until a customer reported it.
02 · WHAT WENT WRONG
What went wrong, and how we fixed it.
Every engagement hits problems. These were the ones that mattered here, and what we changed so they did not come back.
- PROBLEM 01Soon after the first cutover, status values for some customers flickered between two states
Events from two upstream systems for the same customer could arrive out of order, and our consumer applied them as received. We added event timestamps and sequence rules, so older events cannot overwrite newer state, and extended the parallel run to confirm the old and new outputs matched. The change delayed the next cutover by about a week.
- PROBLEM 02The change delayed the next cutover by about a week
- PROBLEM 03A second concern was cost and capacity
In early tests, a few very active customers produced much more traffic than the rest, and some partitions ran hot while others sat idle. We changed the partitioning key, set sensible retention periods per topic and added alerts on consumer lag, so slowdowns were seen before customers saw them. This work is part of why avoidable operating cost fell.
- PROBLEM 04This work is part of why avoidable operating cost fell
03 · THE PROCESS
How we ran it.
We ran the work in phases, with a weekly planning call and a short demo every second week. First, we agreed which status events mattered to customers and mapped where each originated. Second, we designed the event model and built a single stream end to end, with the old batch job running in parallel. Third, we moved the remaining sources one at a time, comparing old and new outputs for each before cutting over. Last, we tuned costs, wrote runbooks and trained the on-call engineers. The batch path was retired only after several clean weeks side by side.
- 01Step 1
We ran the work in phases, with a weekly planning call and a short demo every second week.
- 02Step 2
First, we agreed which status events mattered to customers and mapped where each originated.
- 03Step 3
Second, we designed the event model and built a single stream end to end, with the old batch job running in parallel.
- 04Step 4
Third, we moved the remaining sources one at a time, comparing old and new outputs for each before cutting over.
- 05Step 5
Last, we tuned costs, wrote runbooks and trained the on-call engineers.
- 06Step 6
The batch path was retired only after several clean weeks side by side.
04 · RESULTS
What changed.
Measured against the way the work ran before the engagement began.
- NOW 01
Schema changes go through review, with compatibility checks.
- NOW 02
Freshness is monitored as closely as uptime.
- NOW 03
Failed events are inspected and replayed, not lost.
- NOW 04
Product teams design features assuming data can be current.
- 24 hours
Data latency down from about 24 hours to under 5 minutes
A near-real-time customer-status feature launched
- 15%
About 15% lower avoidable operating cost
05 · WHAT WE BUILT
How it works.
- Event streaming: source changes publish events to Kafka topics, partitioned by customer or entity to preserve ordering where it matters.
- Managed processing: stream processing on a managed service, so the client's team does not run clusters.
- Versioned event model: each event type has a schema in a registry, with compatibility rules.Producers cannot ship a breaking change without review.
- Quality checks: schema validation at the edge, freshness monitors on every topic and duplicate detection keyed on event identifiers.Failures are routed to a dead-letter topic for inspection and replay.
- Idempotent consumers: writing the same event twice does not corrupt state.
- Serving: the status feature reads from a fast store updated by the stream.
06 · QUICK ANSWERS
The short version.
- What problem did the client have?
- Data arrived in daily batches, but the product needed near-real-time status updates. Customers saw yesterday's picture in a product that implied it was live. Support tickets asked why numbers had not moved. The batch jobs were also fragile. A failure at night often meant a manual re-run the next morning, and nobody knew about bad data until a customer reported it.
- What did Brandsmashers build?
- Kafka-based event streaming with managed processing, a versioned event data model, and schema, freshness and duplicate checks.
- How was the work run?
- We ran the work in phases, with a weekly planning call and a short demo every second week. First, we agreed which status events mattered to customers and mapped where each originated. Second, we designed the event model and built a single stream end to end, with the old batch job running in parallel. Third, we moved the remaining sources one at a time, comparing old and new outputs for each before cutting over. Last, we tuned costs, wrote runbooks and trained the on-call engineers. The batch path was retired only after several clean weeks side by side.
- What were the results?
- Data latency down from about 24 hours to under 5 minutes; a near-real-time customer-status feature launched; about 15% lower avoidable operating cost.
YOUR TURN
Have a similar requirement?
Tell us what is stuck. We will come back with a plan and matched, pre-vetted engineers.
