CASE STUDY · LEGALTECH AI SOLUTIONS
Automating clause extraction across 10,000+ legal documents.
- 12,00012,000 documents processed in the first month
- 93%93% extraction accuracy on a reviewed sample
01 · PROJECT OVERVIEW
The client and the brief.
An anonymised legal services organisation. Name withheld under confidentiality.
An extraction pipeline for 12 clause types, with sampling-based validation. Documents came in from the organisation's storage, were converted to text, and passed through extraction. Each extracted field carried a confidence value and a link to its source passage. Items below the confidence threshold went to a lawyer queue instead of the database.
- 1Manual review of about 25,000 records a month.
- 2Staff opened each document, found the relevant clauses and keyed the details into a tracking system.
- 3The work was slow and tiring, and quality varied from person to person.
- 4Backlogs grew whenever volumes rose.
- 5The organisation needed structured data from its documents without adding headcount.
02 · WHAT WENT WRONG
What went wrong, and how we fixed it.
Every engagement hits problems. These were the ones that mattered here, and what we changed so they did not come back.
- PROBLEM 01Scanned documents with skewed pages and stamps produced poor text, and extraction on those files was unreliable
Early sampling showed that most errors came from a small set of document sources. We added an image clean-up step, flagged files with low text quality for a separate path, and raised the review rate on those sources until accuracy recovered. Without the sampling, the problem would have stayed hidden inside an average.
- PROBLEM 02Without the sampling, the problem would have stayed hidden inside an average
- PROBLEM 03A second hurdle appeared in the second week of the parallel run
The same agreement type from two different sources used different headings for the same clause, and the extractor often attached the wrong date to the termination clause. The sampling showed a pattern: most errors involved one clause type and one source. We added examples of that source's layout to the instructions, changed the clause definition so it asked for the notice period and the term separately, and added a rule that rejected a termination date earlier than the signing date.
- PROBLEM 04Accuracy for that clause moved in line with the others
03 · THE PROCESS
How we ran it.
- 01Weeks 1 to 2
Agreed the 12 clause types and the definition of a correct extraction for each. Pulled a varied sample, including poor scans.
- 02Weeks 3 to 5
Built ingestion and extraction, and labelled a reviewed set with the organisation's staff.
- 03Weeks 6 to 8
Tuned thresholds, built the lawyer queue and the sampling process, and ran the pipeline in parallel with manual work.
- 04Cut-over
Production run with daily volume checks and a weekly accuracy review.
04 · RESULTS
What changed.
Measured against the way the work ran before the engagement began.
- NOW 01
Staff no longer key in clause details by hand.
- NOW 02
They work the lawyer queue, which holds the hard cases, and they review the weekly sample.
- NOW 03
Managers read accuracy by clause type and by document source, so they know where to look when something slips.
- NOW 04
When a new document type arrives, the team adds it to the evaluation set first and checks results before it enters the main flow.
- 12,000
12,000 documents processed in the first month
- 93%
93% extraction accuracy on a reviewed sample
Low-confidence items sent to lawyers
05 · WHAT WE BUILT
How it works.
- The pipeline had four stages: text conversion with optical character recognition for scanned files, document type detection, clause extraction, and validation.
- Extraction used structured prompts that returned the clause text, the normalised value and a confidence score.
- Validation had two parts.
- Rules checked basic sense, such as dates in a valid range and parties present.
- A random sample of every batch went to lawyers, and their corrections fed a running accuracy figure.
- We kept raw files, extracted text and outputs separate so any result could be traced back.
- Controls after launch
- Every batch has a sample reviewed by lawyers, and the reviewed share rises automatically when a source or clause type shows more errors. A weekly report shows volume, share routed to lawyers and accuracy by clause type. If accuracy falls below the agreed floor for any clause type, extraction for that type pauses and the documents go to the manual queue until the cause is found.
- Handover
- The organisation's operations team learned to read the weekly report, adjust thresholds with a second person's sign-off and add a new document source through a short checklist. Their IT staff received the deployment notes and a list of what to check if volumes spike. We stayed available for questions for a few weeks, then stepped back.
06 · QUICK ANSWERS
The short version.
- What problem did the client have?
- Manual review of about 25,000 records a month. Staff opened each document, found the relevant clauses and keyed the details into a tracking system. The work was slow and tiring, and quality varied from person to person. Backlogs grew whenever volumes rose. The organisation needed structured data from its documents without adding headcount.
- What did Brandsmashers build?
- An extraction pipeline for 12 clause types, with sampling-based validation. Documents came in from the organisation's storage, were converted to text, and passed through extraction. Each extracted field carried a confidence value and a link to its source passage. Items below the confidence threshold went to a lawyer queue instead of the database.
- What were the results?
- 12,000 documents processed in the first month; 93% extraction accuracy on a reviewed sample; low-confidence items sent to lawyers.
YOUR TURN
Have a similar requirement?
Tell us what is stuck. We will come back with a plan and matched, pre-vetted engineers.
