Custom NLP vs. off-the-shelf AI tools: when you need custom

01.Where off-the-shelf works well

General tools do well on tasks where the language is ordinary and the cost of a small mistake is low.

• General summaries: a long email thread or report boiled down to the main points.
• Drafting: first versions of letters, replies or internal notes.
• Common document types: standard invoices, receipts and basic forms that look much like millions of others.
• One-off or low-volume work: if you do the task twice a week, setup effort for a custom system will never pay back.

If these describe your need, use a general tool, set some usage rules and move on. Building custom here is a waste of money.

02.When custom starts to pay off

Custom work earns its place when one or more of these conditions are true.

• Specialist language. Your documents use terms, abbreviations and structures that general tools handle unevenly. Think of the difference between "termination for convenience" and "termination for cause" in a contract, or the way a claim form encodes a procedure. Small misreadings carry real consequences.
• Measurable accuracy is required. If someone will ask "how often is this right?", you need a labelled test set and a score per field. General tools can be scored too, but you cannot easily improve them on your specific cases.
• Data cannot leave your environment. Client confidentiality, contract terms or regulation may rule out sending documents to a third-party service. A model deployed in your own account or on your own servers solves that.
• Volume makes small gains add up. At a few thousand documents a month, a few percentage points of accuracy, or a few minutes saved per document, becomes a lot of staff time.

You do not need all four. Two is usually enough to justify a serious look.

03.A worked example

Take a team that reviews supplier contracts. A general tool summarises each contract nicely. But the team needs the renewal date, notice period and liability cap in a table, every time, with the page reference. The general tool gets the dates right most of the time, and occasionally picks up the wrong figure from a nearby sentence. In a summary this is not noticed. In a table that feeds a renewal calendar, it is a missed deadline.

That team has a measurable need, a repeated pattern and a cost of error. That is the profile where custom extraction, scored on their own documents, makes sense.

04.The test to run first

Run a sample of your documents through a general tool and score the result. If it clears your bar, stop there.

Here is how:

• Pick 50 to 100 real documents that reflect normal variety, including the messy ones.
• Decide what you need from each: fields, categories or answers.
• Have a knowledgeable person produce the correct answers for the sample.
• Run the general tool and compare, field by field.
• Decide your threshold before you look at results, for example "wrong values are acceptable if flagged, but not if they pass silently".
• Note where the errors cluster. That tells you whether the problem is fixable with better prompts or needs a custom model.

This costs a few days, and it gives you evidence instead of opinions.

05.What custom actually involves

Custom does not mean building a language model from scratch. It usually means a pipeline: text extraction and OCR, a model tuned or prompted on your labelled examples, confidence scores, a review step for uncertain cases, and integration with your systems. The labelled data is the most valuable piece. It is also the thing you keep if you change tools later.

06.Cost and effort, honestly

Custom has costs that a subscription does not. You need labelled data, which takes expert time. You need engineering to build and connect the pipeline. You need someone to monitor it. In return you get measured accuracy, control over where data lives and the ability to improve on your own cases. Compare that with the subscription cost plus the staff time lost to errors and rechecking. Include the cost of a mistake: a missed clause or a misfiled notice may cost far more than the tool.

07.Hybrid setups

You do not have to pick one side. Many teams use a general model for drafting and summaries, and a custom component for the steps that need precision, such as pulling renewal dates or classifying incoming mail. Others start with a general model behind a thin layer that checks outputs against rules, and move to custom pieces only where the checks keep failing. This keeps spending tied to evidence.

08.What the test tells you

Read the results of your sample test in three ways. If the general tool clears your bar on every field, adopt it and add basic controls. If it clears the bar on some fields and not others, consider a hybrid, with custom work only on the weak fields. If it fails widely, look at why. Poor scans, unusual layouts and specialist terms point towards custom. Vague instructions point towards better prompting first. Write the findings down, because they will also guide how you brief any supplier.

09.Keeping a custom system healthy

Documents change. A supplier updates its template, a regulator revises a form, a new product line adds terms. A custom system needs a feedback loop: reviewers correct errors, corrections are stored, and the model is retrained from time to time. Check accuracy on fresh samples every month or so. Without that, a system that was good at launch slowly slips, and people stop trusting it before anyone notices the numbers.

10.A note on language and formats

Check early which languages and file types your documents use. Mixed English and regional language text, scanned stamps and handwriting notes in margins are common in real archives. They change what is feasible and what the test should include. Put a few of the worst examples in your sample on purpose.

11.Common mistakes

• Choosing custom because it sounds more serious.
• Skipping the baseline test.
• Judging on a handful of clean documents.
• Measuring accuracy once and never again.
• Having no plan for errors, such as a review queue.
• Forgetting that the documents will change.

12.What to do next

Run the test above on one document type. If the general tool clears your bar, deploy it with sensible controls and revisit in six months. If it does not, you now have a labelled set, a baseline and a clear list of failures, which is exactly what a custom project needs as its starting point.

Brandsmashers builds custom extraction, classification and search for sensitive document sets, and we are glad to look at your test results and tell you honestly which path makes sense.