CASE STUDY · CUSTOMER EXPERIENCE (CX) & PERSONALIZATION
A personalization engine that lifted e-commerce conversion by 18%.
- 18%Conversion up 18% in a controlled A/B test over 4 weeks
01 · PROJECT OVERVIEW
The client and the brief.
An anonymised retailer. Name withheld under confidentiality.
Event data, recommendation models and controlled A/B tests. We captured browsing and purchase events cleanly, built recommendation models for the homepage and product pages, and ran each change against a control group.
- 1A generic experience for every visitor.
- 2Returning customers saw the same homepage as first-time visitors.
- 3Category pages were sorted by a fixed rule, and emails went to broad lists.
- 4The team had a lot of browsing and purchase data but no way to use it on the site.
- 5They also did not trust claims of uplift that were not backed by a proper test.
02 · WHAT WENT WRONG
What went wrong, and how we fixed it.
Every engagement hits problems. These were the ones that mattered here, and what we changed so they did not come back.
- PROBLEM 01In the first week of the test, the personalized group looked worse
The cause was a caching rule that served some visitors stale recommendations, including items already sold out. We found it through the test's own monitoring: click rates on recommendations dropped for one segment. We fixed the cache, added an out-of-stock filter, restarted the test period and did not count the faulty week.
- PROBLEM 02The cost was a delay
The benefit was a result we could trust.
- PROBLEM 03A second hurdle was the cold start
New visitors had no history, so early recommendations were weak and the test showed little change for them. We did not hide that. We used session context, such as the category being viewed, and a curated best-seller list as the fallback, and reported new and returning visitors separately.
- PROBLEM 04The returning group drove most of the gain, which told the team where to invest next
03 · THE PROCESS
How we ran it.
- 01Weeks 1 to 2
audited the existing analytics events, found gaps and duplicates, and agreed the success measure with the commercial lead.
- 02Weeks 3 to 5
fixed event tracking, built a session and customer history store, and trained a first recommendation model on past data.
- 03Weeks 6 to 7
built the serving layer and the on-site components, with a fallback to the existing best-seller lists.
- 04Test period
a controlled A/B test over 4 weeks, with a random split and a pre-agreed way of reading the result.
- 05Step 5
A weekly review with product, merchandising and analytics throughout.
04 · RESULTS
What changed.
Measured against the way the work ran before the engagement began.
- NOW 01
New site changes are tested against a control group before they are rolled out to everyone.
- NOW 02
Merchandisers have rules and boosts they can apply on top of the models, such as promoting seasonal items.
- NOW 03
The team reviews recommendation quality and stock handling every week.
- NOW 04
Event tracking changes go through a checklist, since a broken event now affects revenue.
- 18%
Conversion up 18% in a controlled A/B test over 4 weeks
05 · WHAT WE BUILT
How it works.
- Event collection from the site and app into a central store, with schema checks to catch bad or missing events.
- Features built from recent browsing, category affinity and purchase history.Anonymous visitors got context-based recommendations from the current session.
- A mix of simple and learned approaches: popularity as a baseline, then collaborative and content-based models, chosen by offline tests before going live.
- A low-latency serving service with caching, and a timeout that falls back to the default page so the site is never held up.
- Consent flags respected at collection and at serving time.Visitors who did not consent saw non-personalized content.
- Randomisation by visitor ID, with a check that the groups were balanced before reading any result.
- Monitoring and handover
- We handed over a dashboard showing recommendation click rate, add-to-basket rate, coverage and latency, with alerts when the service slowed or fell back to defaults too often. A runbook describes how to retrain the models, add a rule and switch the feature off safely. The analytics lead was trained to design and read the next test, including how to avoid stopping early when a result looks good.
06 · QUICK ANSWERS
The short version.
- What problem did the client have?
- A generic experience for every visitor. Returning customers saw the same homepage as first-time visitors. Category pages were sorted by a fixed rule, and emails went to broad lists. The team had a lot of browsing and purchase data but no way to use it on the site. They also did not trust claims of uplift that were not backed by a proper test.
- What did Brandsmashers build?
- Event data, recommendation models and controlled A/B tests. We captured browsing and purchase events cleanly, built recommendation models for the homepage and product pages, and ran each change against a control group.
- What were the results?
- Conversion up 18% in a controlled A/B test over 4 weeks.
YOUR TURN
Have a similar requirement?
Tell us what is stuck. We will come back with a plan and matched, pre-vetted engineers.
