CASE STUDY · CUSTOM NLP & DOCUMENT INTELLIGENCE
Automated contract clause extraction using custom NLP.
- 92%92% extraction accuracy on the evaluated sample
- 45%Review time down 45%
01 · PROJECT OVERVIEW
The client and the brief.
An anonymised mid-market B2B services company. Name withheld under confidentiality.
A custom model for the client's clause types. It reads a contract, finds each clause, extracts the key details and links them back to the exact passage. Reviewers see the highlighted text and confirm or correct it, rather than hunting for it.
- 1Manual clause review.
- 2Staff read contracts end to end to find a defined list of clauses: term and renewal, liability limits, payment terms, termination and a few others.
- 3The work was slow, tiring and uneven.
- 4Two reviewers could mark the same contract differently.
- 5Backlogs built up at quarter end, and nobody could easily search across the contract archive for a particular term.
02 · WHAT WENT WRONG
What went wrong, and how we fixed it.
Every engagement hits problems. These were the ones that mattered here, and what we changed so they did not come back.
- PROBLEM 01The first model did well on the client's standard templates and poorly on counterparty paper, where clauses were numbered differently and sometimes split across pages
We had under-sampled those contracts. We added more of them to the labelled set, improved the page-break handling and re-ran the test. We also made the evaluation report show results by contract source, so the gap could not hide inside an average again.
- PROBLEM 02We also made the evaluation report show results by contract source, so the gap could not hide inside an average again
- PROBLEM 03A second hurdle was scanned contracts
Several older agreements were low-quality scans with skewed pages and faint print, and the text recognition step introduced errors that the model then extracted faithfully. We added an image clean-up step, flagged documents with poor recognition scores for manual review, and recorded recognition quality next to each result. Reviewers could then see at a glance when a wrong answer came from a bad scan and not from the model.
- PROBLEM 04Reviewers could then see at a glance when a wrong answer came from a bad scan and not from the model
03 · THE PROCESS
How we ran it.
- 01Weeks 1 to 2
collected a sample of contracts across templates and counterparties, and ran a workshop with the reviewers to define each clause type with examples and edge cases.
- 02Weeks 3 to 5
reviewers labelled a training and test set. A second reviewer checked a portion to measure agreement.
- 03Weeks 5 to 8
built and tuned the model, compared it with a general tool on the same test set and iterated on errors.
- 04Weeks 9 to 10
piloted with a small group of reviewers, with a feedback button on each result.
- 05Step 5
Weekly check-ins with the legal operations lead.
04 · RESULTS
What changed.
Measured against the way the work ran before the engagement began.
- NOW 01
Reviewers start from highlighted clauses and spend their time on judgement, not searching.
- NOW 02
Corrections are captured and used in periodic retraining, so the model improves with use.
- NOW 03
The team can search the archive for a clause type and see the exact wording.
- NOW 04
New clause types are added through a defined process: definition, labelled examples, test, release.
- 92%
92% extraction accuracy on the evaluated sample
- 45%
Review time down 45%
05 · WHAT WE BUILT
How it works.
- Text extraction from PDFs and scans, with OCR for image-only pages and layout handling for headings and numbered clauses.
- Documents split into sections, then each passage scored against the client's clause types by a fine-tuned language model.
- Extraction of structured fields for each clause, such as notice periods or caps, with the source span stored.
- Confidence scores per prediction.Low-confidence items are routed to full human review.
- A review screen that shows the highlight beside the extracted value and lets staff correct it.
- Deployment inside the client's own cloud account, with access limited to the legal team and an audit log of corrections.
- Monitoring and handover
- We set up a monthly accuracy check on a fresh sample, scored by reviewers, so a drop would show before it caused harm. The legal operations lead received a guide to adding a clause type and a short runbook for reprocessing a batch. We also agreed a rule that any change to the standard contract templates is shared with the team, since new wording can affect extraction.
06 · QUICK ANSWERS
The short version.
- What problem did the client have?
- Manual clause review. Staff read contracts end to end to find a defined list of clauses: term and renewal, liability limits, payment terms, termination and a few others. The work was slow, tiring and uneven. Two reviewers could mark the same contract differently. Backlogs built up at quarter end, and nobody could easily search across the contract archive for a particular term.
- What did Brandsmashers build?
- A custom model for the client's clause types. It reads a contract, finds each clause, extracts the key details and links them back to the exact passage. Reviewers see the highlighted text and confirm or correct it, rather than hunting for it.
- What were the results?
- 92% extraction accuracy on the evaluated sample; review time down 45%.
YOUR TURN
Have a similar requirement?
Tell us what is stuck. We will come back with a plan and matched, pre-vetted engineers.
