Why every AI initiative depends on boring data engineering

01.The gap between demo and production

A data scientist takes a sample of data, cleans it by hand and trains a model. The results look good. The demo goes well. Then the team tries to run it on live data and the problems begin.

Fields are missing. Dates arrive in three formats. A source system changes a column name on a Tuesday. The same customer appears under two IDs. The data that arrives is a day late. None of these is a modelling problem. All of them stop the model working.

Gartner expects organisations to abandon 60% of AI projects that lack AI-ready data through 2026. The model is rarely the problem. The problem is that nobody budgeted for the pipeline underneath it.

02.What "AI-ready data" actually means

• It arrives when the model needs it. A fraud model fed yesterday's data is a report, not a defence. Decide how fresh the data must be for the decision at hand, and build to that.
• It means the same thing everywhere. If "customer" has three definitions across three systems, the model learns all three badly. This is what master data management fixes. It gives each customer, product or supplier one agreed identity and one set of rules.
• It is tested like code. Schema checks, freshness checks and volume alerts catch a broken feed before it quietly ruins a model.
• Someone owns it. Every important dataset needs an owner who answers when it breaks.

These four are not exciting. They are also the difference between a model that runs for years and one that is switched off in six months.

03.How failure looks in practice

Consider a retailer that builds a demand forecasting model. In testing it works well. In production, accuracy falls. After weeks of investigation the team finds that one store system changed how it records returns, and the feed now counts a return as a negative sale on the day of the return rather than the day of the sale. No test caught it. No one owned the feed. The model kept producing forecasts, and people kept acting on them.

A schema check would not have caught this, since the format was valid. A distribution check would have. Daily volume and range checks on key fields, with an alert to a named owner, would have flagged it the same morning.

This is typical. Data does not fail loudly. It drifts, and the damage arrives later.

04.Where to spend first

Before you hire another data scientist, check three things:

• Can you trace where a model's input data came from?
• Do you know within an hour when a feed breaks?
• Can a new engineer find the right table in a day?

If any answer is no, that is your first project.

05.What a first project looks like

Do not start with a company-wide platform. Pick one important use case and build it properly end to end.

• Choose the use case. One decision, one dataset family, one owner.
• Map the sources. Write down where each input comes from and who runs the source.
• Agree definitions. Settle what each key term means and write it down.
• Build the pipeline with tests. Schema, freshness, volume and duplicate checks from the start.
• Add monitoring and an alert route. An alert nobody receives is decoration.
• Document it where people look. A short page beside the data, not a document in a forgotten folder.

Then repeat for the next use case. Each pass leaves the shared platform a little better.

06.Common mistakes

• Buying a platform before agreeing the problem. Tools do not fix unclear ownership or definitions.
• Building everything for real-time. Real-time costs more. Use it where freshness changes the outcome.
• Treating data quality as a clean-up project. It is an ongoing practice, like testing code.
• Skipping lineage. When a number is challenged, you need to show where it came from.
• Starting with the largest dataset. Start with the most valuable decision.
• Leaving governance to the end. Access control and documentation are cheaper early.

07.What to do next

Pick the AI initiative you care about most. Ask the three questions above about its data. Write down the answers honestly, then fix the gaps in order of risk.

08.Pipelines are products, not scripts

Many data teams treat pipelines as scripts that someone wrote once. That works until the author leaves or the source changes. A better habit is to treat each important pipeline as a small product with a version, tests, an owner and users.

What this looks like in practice: the pipeline code lives in version control and goes through review. Changes run through tests before they reach production. Each pipeline has a short page that says what it does, who owns it, what depends on it and what to do when it fails. Downstream users know where to report problems. It sounds like overhead. It is far cheaper than the alternative, which is an urgent hunt through old scripts the day a dashboard goes wrong.

09.The cost of definitions

Master data work is often seen as bureaucratic, until the first meeting where finance, sales and support each present a different customer count. All three are right by their own rules. The model trained on one of them will quietly disagree with the other two.

A practical way to start is to pick the five most-used business entities, such as customer, product, supplier, order and location. For each, write down the agreed definition, the system that is the source of truth, the identifier and who decides when rules change. Then build matching and merging rules on top. Do not try to cover every entity at once.

10.Common mistakes

• Treating data quality as a one-off clean-up. Without ongoing checks, it decays again.
• Skipping ownership. An alert with no recipient is noise.
• Over-building for scale you do not have. Solve today's volume well first.
• Hiring for models only. A team of modellers without data engineers will spend most of its time wrangling files.
• Ignoring access control. Sensitive data in a wide-open lake is a risk, and fixing it later is harder.