CASE STUDY · EDTECH PLATFORMS
Automating course content personalization for an online education provider.
- 20%Engagement up 20%
- 9090-day retention up 6 points
01 · PROJECT OVERVIEW
The client and the brief.
An anonymised online education provider. Name withheld under confidentiality.
Content tagging and a recommendation engine.
We tagged the catalogue by topic, skill, difficulty and format. The tagging was a mix of a review tool for editors and automated suggestions that editors approved or corrected. The recommendation engine then used each learner's activity and the tags to propose the next lesson or course. Recommendations appeared on the home screen and at the end of every lesson.
- 1One-size-fits-all content.
- 2Every learner saw the same catalogue in the same order.
- 3Beginners were shown advanced material and advanced learners saw basics.
- 4The provider had plenty of content, but finding the right next item was left to the learner.
- 5Many gave up, and the team could see it in weak engagement and in learners not coming back after the first few weeks.
02 · WHAT WENT WRONG
What went wrong, and how we fixed it.
Every engagement hits problems. These were the ones that mattered here, and what we changed so they did not come back.
- PROBLEM 01The first ranking model pushed popular courses to everyone
New and niche content almost never surfaced, and recommendations began to look the same for all learners. Review of the "why shown" records made the pattern plain. We added an exploration share for newer content and a cap on how often one item could appear.
- PROBLEM 02Variety returned, and editors started to see their newer work being used
- PROBLEM 03A second hurdle was tagging quality
The automated suggestions were right for broad topics and weak for skill level, so beginner and advanced items were sometimes mixed. Editors corrected a sample each week, and we fed the corrections back so suggestions improved. We also added a simple agreement check between two editors on a random sample, which showed where the tag definitions were unclear.
- PROBLEM 04The team rewrote those definitions in plain language, and disagreement fell
03 · THE PROCESS
How we ran it.
We worked in two-week sprints with a product owner from the provider. Editors joined the sprint reviews, because the tags were theirs.
- 01Step 1
Weeks 1 to 3: catalogue audit, tag scheme agreed with editors, analytics baseline set up.
- 02Step 2
Weeks 4 to 7: tagging tool built, backlog tagged, simple rule-based recommendations released to a small share of learners.
- 03Step 3
Weeks 8 to 11: behaviour-based ranking added, with an A/B test against the old home screen.
- 04Step 4
Weeks 12 onward: gradual rollout and tuning, with a weekly review of what was being recommended.
04 · RESULTS
What changed.
Measured against the way the work ran before the engagement began.
- NOW 01
Editors tag every new course before launch and review recommendation quality weekly.
- NOW 02
Product decisions are tested against a control group rather than judged by opinion.
- NOW 03
The team plans the catalogue around gaps that learner behaviour reveals, not around what is easiest to produce.
- 20%
Engagement up 20%
- 90
90-day retention up 6 points
05 · WHAT WE BUILT
How it works.
- Content metadata lives in a tagging service with versioned schemes, so a change in the taxonomy does not break old recommendations.
- Events such as lesson started, finished and abandoned flow into a store used for ranking.
- The first version used rules: same topic, next level, skip what is done.
- The second blended those rules with a ranking model trained on completed journeys.
- We kept the rules as a fallback for new learners and new content.
- Every recommendation records why it was shown, which made review simple.
06 · QUICK ANSWERS
The short version.
- What problem did the client have?
- One-size-fits-all content. Every learner saw the same catalogue in the same order. Beginners were shown advanced material and advanced learners saw basics. The provider had plenty of content, but finding the right next item was left to the learner. Many gave up, and the team could see it in weak engagement and in learners not coming back after the first few weeks.
- What did Brandsmashers build?
- Content tagging and a recommendation engine. We tagged the catalogue by topic, skill, difficulty and format. The tagging was a mix of a review tool for editors and automated suggestions that editors approved or corrected. The recommendation engine then used each learner's activity and the tags to propose the next lesson or course. Recommendations appeared on the home screen and at the end of every lesson.
- How was the work run?
- We worked in two-week sprints with a product owner from the provider. Editors joined the sprint reviews, because the tags were theirs.
- What were the results?
- Engagement up 20%; 90-day retention up 6 points.
YOUR TURN
Have a similar requirement?
Tell us what is stuck. We will come back with a plan and matched, pre-vetted engineers.
