Enterprise AI pilots almost never fail because somebody picked the wrong model, they fail because four unglamorous preconditions were missing before anyone wrote a line of code, and each one could have been checked in an afternoon. The first is practical data access: The pilot team cannot reach real production records without filing a ticket and waiting on an approval. The second is measurement: there is no evaluation harness, so nobody can demonstrate that the system got better at anything. The third is ownership: no single person’s job actually changes if the thing ships, so nobody fights for it when the timeline slips. The fourth is placement: The tool lands beside the workflow it was meant to improve instead of inside it. Miss two of those four and the pilot is already dead, even though it usually takes another nine months to stop moving.

That is the pattern behind nearly every rescue engagement I have been pulled into. The demo works, and the demo was never the problem. What is missing is the unglamorous scaffolding of proof and ownership that should have existed before the first prompt was written.

You have seen the statistic. Some enormous share of enterprise generative AI pilots — the widely circulated MIT Project NANDA figure put it at 95% — deliver no measurable profit-and-loss impact.

I would treat that number carefully, and not because I think pilots are secretly succeeding. There are two problems with leaning on it. First, the report’s primary PDF is no longer served from MIT’s own domain, which is an awkward position to be in when you are citing something as fact. Second, and more importantly, several people picked at the methodology when it landed, and the criticism was not unreasonable.

The exact figure doesn’t change the advice by one degree, though. Whether the real number is 95% or 70%, the interesting question was never how many fail. It is what separates the ones that don’t.

The separator is not the model. I have never once been called into a stalled pilot where the honest diagnosis was “you picked the wrong foundation model.” Not once. The diagnosis is always some combination of the same four missing preconditions, and each one is checkable in an afternoon.

“Our case is different, we have good data.”

Maybe so. But good data sitting in a warehouse your pilot team has to request access to through a ticketing queue is not access, it is a promise of access, and the gap between them is where most of the first quarter goes.

The test is embarrassingly simple. Can one engineer, working alone, pull a representative sample today, without opening a ticket and without a meeting? If the answer is “well, they’d need to ask Sandra, and Sandra’s on leave,” you don’t have data access. You have a dependency with a human bottleneck in it, and every experiment you want to run will be paced by that bottleneck for the life of the project.

Volume is rarely the problem. Teams obsess over whether they have enough rows, and in my experience you can learn most of what you need from a few thousand well-chosen examples. What kills you is round-trip time — when checking a hypothesis takes four days instead of forty minutes, you stop checking hypotheses.

Then there is what “representative” has to mean. It means the ugly rows: the nulls, the free-text field where three regional offices invented their own conventions, the records from before the 2019 migration that still carry the old encoding. Those are not edge cases you will handle later. In most enterprise data they are a meaningful fraction of the corpus, and they are precisely the inputs a demo quietly avoided.

The classic failure here is the pre-cleaned warehouse export. Somebody helpfully hands the pilot team a tidy extract — deduplicated, normalised, six weeks stale. The model does beautifully on it. Then it meets production, the accuracy falls off a cliff, and nobody can explain why, because the clean extract was never the thing.

This is the one almost everybody skips, and skipping it is what kills budget renewal.

An evaluation harness is three things: a frozen test set, a scoring method, and a baseline number. That is the whole of it. It is not a platform, it does not need procurement approval, and a competent engineer can stand one up in under a week.

The frozen test set is 50 to 200 labelled cases, and small is genuinely fine. What is not optional is the freezing. The moment people start adding cases when results look bad and quietly dropping the ones that seem “unfair,” you have lost the ability to compare anything to anything.

The scoring method has to be decided before you see results, for the obvious reason.

The baseline is usually the human doing the job today, and you should measure that first. If you don’t know how accurate your current process is, you can never claim the model improved on it. You can only claim the model produced output, which is not an achievement anyone should pay for.

I was brought into a pilot a while back that had a working prototype and eighteen months of runway already burned, and no frozen test set. The first week was not modelling at all; it was building the harness and labelling a couple hundred historical cases. By Thursday we had the baseline, and the baseline was already outperforming the model on two of the five case types. Nobody in that building knew. They had been iterating on prompts for months against a vibe check.

That is not a failure of the team, it is a failure of sequencing. They built the thing before they built the instrument that tells you whether the thing works — and once you are six months in with no instrument, the political cost of finding out becomes higher than the cost of not knowing.

This is the precondition people believe they have satisfied when they name a sponsor or stand up a steering committee. Neither one is an owner.

The test I use is to ask who gets a materially worse week if this pilot is cancelled tomorrow. If the honest answer is “nobody, really, we’d all just go back to our normal work,” then what you have is theatre — well-funded and well-intentioned, but theatre.

A sponsor approves budget and a steering committee reviews slides, and neither of them has to live with the output. An owner is the person whose queue gets shorter, whose error rate goes down, whose Tuesday is different. That person will fight for integration work that a committee will happily defer, because a committee experiences deferral as prudence and an owner experiences it as their week not improving.

It is also the cheapest precondition to check and the most uncomfortable, because the answer is usually known before you ask.

“The pilot worked but nobody used it.” I hear this one constantly, and it is almost never an adoption problem. It is a placement problem.

If the output arrives in a new tab, a new portal, or a separate dashboard someone has to remember exists, usage decays to near zero within about three weeks. It happens reliably enough that I would almost call it a law.

The pilots that stick put the output inside the ticket queue the agent already has open, inside the review step that already exists in the approval chain, inside the screen where the decision was already being made. The user should not have to choose to use the AI; it should be sitting in the place they were going anyway.

Integration work is unglamorous, it is usually the largest single chunk of engineering in the whole project, and it is the first thing cut when timelines slip. Cutting it does not save the project. It converts a working pilot into an unused one, which scores identically on the only measure that matters.

One more thing, said plainly: if the underlying workflow is broken, automating it makes the breakage arrive faster. AI is not a process fix. I have watched a team try to automate an approval chain that had four redundant sign-offs nobody could justify, and the result was four redundant sign-offs happening in ninety seconds instead of nine days. The bottleneck moved, but it did not go away.

Score each one present, partial, or absent. Be harsh; a generous score here is a bill you pay in month seven.

1. Data access. Can an engineer pull a representative sample today, alone, without a ticket — including the ugly records?
2. Evaluation harness. Is there a frozen test set, an agreed scoring method, and a measured baseline for the human process?
3. Named owner. Is there one person whose week gets worse if this is cancelled?
4. Workflow placement. Is there a specific existing screen or queue where the output will appear, and has someone scoped that integration?

The decision rule: two or more absent, don’t start. Fix the precondition first. Each of the four is buildable in days rather than quarters, and each one can kill the project early, which is the entire point. Killing a bad pilot in week two costs almost nothing. Killing it in month nine costs a budget cycle.

That last part is the real damage. Failed pilots don’t just waste money; they make the next proposal harder to fund.

If you score two or more absent and you are not sure how to close the gap, that’s a conversation worth having before the next sprint rather than after it. And if the gap is precondition 2 or 4 — the harness or the integration — that is production engineering rather than model work, and it is worth asking honestly whether the pilot team does that kind of production engineering day to day.

Everyone says start small. As the advice is usually given, it is wrong.

The standard version means pick a low-stakes workflow so failure is cheap, which sounds prudent right up until you watch it play out. Low stakes means nobody measures it, because measurement costs effort and nobody spends effort on something that doesn’t matter, and it means nobody owns it, because ownership is a burden you only accept for things you care about. So you get a pilot that satisfies preconditions 1 and 4 and structurally cannot satisfy 2 and 3. It “succeeds,” it proves nothing, and the follow-up funding request has no evidence attached to it.

Pick a workflow that matters enough to be measured. Then reduce scope on the surface rather than the stakes: one team instead of twelve, one queue instead of the whole department, one case type instead of all five. The stakes stay the same and the blast radius gets smaller. That version gives you a real owner, a real baseline, and a result someone will act on.

Small stakes and small surface sound like the same advice. They produce opposite outcomes.

This objection comes up constantly and deserves a straight answer, because it is usually said apologetically, as though it were an admission of a hiring mistake.

It isn’t one. Production agents are a distributed-systems problem wearing an ML hat, and the work involved is familiar to anyone who has run services in anger: retries, idempotency, cost ceilings that don’t blow up when someone pastes a novel into the input box, observability good enough to reconstruct what happened at 2am on a Sunday, and graceful fallback for the hour when the provider is having a bad day.

None of that is machine learning. It is the same engineering discipline you would want around a payments integration, and expecting one person to cover both that surface and the modelling work is a scoping error rather than a talent problem.

Why do most enterprise AI pilots fail? Not because of model quality. They fail because the team lacked practical data access, had no evaluation harness to prove improvement, had no owner whose job depended on the outcome, or built something that lived outside the workflow it was meant to improve. Two or more of those missing is close to fatal, and all four are checkable before you write code.

How long should an AI pilot take before you kill it? If you have the four preconditions, you should have a defensible signal in six to eight weeks. If you don’t have them, the honest answer is that the clock hasn’t started. A pilot with no baseline can run indefinitely without producing evidence, which is exactly why so many of them do.

Do we need an AI engineer to run a pilot, or can our existing team do it? Your existing team can usually run the pilot. What they often can’t do alone is production hardening — retries, cost controls, observability, fallback behaviour. That’s distributed-systems work, not modelling work. Scope the two separately rather than expecting one hire to cover both.

What’s the difference between an AI pilot and a proof of concept? A proof of concept answers “is this technically possible.” A pilot answers “does this improve a real workflow with real users and real data.” Most enterprise AI proof of concept mistakes come from teams that built the first and reported it as the second.

How do you measure ROI on an AI pilot that hasn’t shipped? Against the baseline you measured before starting. If the human process handles 100 cases a week at a known error rate and known cost, and the assisted process moves either number on a frozen test set, you have a defensible projection. Without that baseline, there’s nothing to compute ROI against — which is the position most stalled pilots are in.