CASE STUDY · CUSTOM NLP & DOCUMENT INTELLIGENCE
Building a document classification system for a legal back office.
- 91%91% classification accuracy
- 55%Triage time down 55%
01 · PROJECT OVERVIEW
The client and the brief.
An anonymised law firm. Name withheld under confidentiality.
12 document categories, a classifier and rules-based routing. The classifier reads each document, assigns one of the categories and passes it to a routing layer that applies the firm's rules, such as matter type, urgency markers and named client teams.
- 1Thousands of documents sorted by hand.
- 2Incoming files from clients, courts and counterparties arrived by email and upload, and support staff opened each one to decide what it was and who should handle it.
- 3Peaks caused delays, and misfiled documents sometimes reached the wrong team late.
- 4The firm wanted a consistent first sort without letting software make unchecked decisions about sensitive material.
02 · WHAT WENT WRONG
What went wrong, and how we fixed it.
Every engagement hits problems. These were the ones that mattered here, and what we changed so they did not come back.
- PROBLEM 01Two categories were routinely confused, and the confusion was not a model fault
Staff themselves classed similar documents differently depending on the matter. Our first labelled set therefore carried inconsistent labels. We held a calibration session, wrote clear tie-break rules, relabelled the disputed items and retrained.
- PROBLEM 02After that, the confusion fell and the written definitions became useful training material for new staff
- PROBLEM 03A second hurdle came with volume
During a busy week, a large batch of scanned court bundles arrived as single long files that combined many documents. The classifier treated each bundle as one item and gave poor answers. We added a splitting step that detects document boundaries by layout cues such as cover pages and headers, and routes unsplittable bundles to a person.
- PROBLEM 04We also set a size limit and a clear message for files that exceed it
03 · THE PROCESS
How we ran it.
- 01Weeks 1 to 2
worked with senior support staff to define the 12 categories, with written descriptions and examples. We merged two categories that people could not reliably tell apart.
- 02Weeks 3 to 5
labelled a sample of documents, built the classifier and tested on a held-out set.
- 03Weeks 6 to 8
designed the routing rules with partners and practice managers and tested them on a back-catalogue.
- 04Weeks 9 to 10
live pilot with all results visible to staff, who could override any decision.
- 05Step 5
Short weekly reviews with the practice support lead.
04 · RESULTS
What changed.
Measured against the way the work ran before the engagement began.
- NOW 01
The first sort is automatic, and staff concentrate on low-confidence documents and exceptions.
- NOW 02
Override decisions are reviewed monthly, and used to refine categories and rules.
- NOW 03
The routing rules are owned by practice management, not buried in code.
- NOW 04
New document types go through the same process of definition, examples and testing before being added.
- 91%
91% classification accuracy
- 55%
Triage time down 55%
05 · WHAT WE BUILT
How it works.
- Ingestion from the firm's email and document management system, with OCR for scans.
- A text classifier fine-tuned on the firm's labelled documents, using content and a small set of metadata such as sender type.
- Calibrated confidence scores.Documents below the threshold go to a manual triage queue.
- A rules engine on top for routing, kept separate from the model so partners could read and change rules without retraining.
- Strict access control, since documents were privileged and confidential.Processing ran inside the firm's own environment, with no external calls.
- An audit trail of the category, confidence, route and any human override.
- Monitoring and handover
- Each month, the practice support lead reviews a sample of automatic decisions and all overrides. A short dashboard shows volume by category, share sent to manual triage and average time to first assignment. We trained two staff to edit routing rules and add a category, and wrote down the steps for retraining. Any new category needs examples, a definition and approval from a partner before it is released.
06 · QUICK ANSWERS
The short version.
- What problem did the client have?
- Thousands of documents sorted by hand. Incoming files from clients, courts and counterparties arrived by email and upload, and support staff opened each one to decide what it was and who should handle it. Peaks caused delays, and misfiled documents sometimes reached the wrong team late. The firm wanted a consistent first sort without letting software make unchecked decisions about sensitive material.
- What did Brandsmashers build?
- 12 document categories, a classifier and rules-based routing. The classifier reads each document, assigns one of the categories and passes it to a routing layer that applies the firm's rules, such as matter type, urgency markers and named client teams.
- What were the results?
- 91% classification accuracy; triage time down 55%.
YOUR TURN
Have a similar requirement?
Tell us what is stuck. We will come back with a plan and matched, pre-vetted engineers.
