A pilot answers one question: can this work? Production answers a harder one: does it work every day, for people who weren't in the room when it was built, with data nobody cleaned for the occasion?
Many pilots that never reach production didn't fail. They succeeded, and then nothing happened. The demo impressed management, everyone agreed it was promising and the project drifted, because nobody had decided in advance what success meant or what would happen next.
My view: the gap between pilot and production is rarely technical. It's a set of decisions that belong before the pilot starts, not after.
What a pilot hides
A pilot is designed to succeed. That isn't a criticism; it's how pilots work. They usually run on:
- a clean or hand-picked data sample
- a few motivated users, often the people who asked for it
- a standalone setup, outside the systems people use every day
- close attention: someone is watching and fixing problems as they appear
A pilot is a training session in perfect conditions; production is race day, in the rain. Production removes all four. The data is messy and keeps changing. The users are busy and didn't ask for anything. The tool has to fit into the ERP, the CRM or the email client people already work in. And after a few weeks, nobody is watching closely any more.
Five things that separate a deployment from a demo
1. Success criteria set before the pilot. Write down what result would justify moving to production, and what result would justify stopping. Both. A pilot without a stop criterion never really ends; it just fades.
2. Real data, early. Test on the data you'll actually use, with its gaps and inconsistencies, as early as possible. If the pilot only works on a cleaned sample, you've learned something about the sample, not about your operations.
3. Integration where the work happens. A tool that lives in a separate window is a tool people forget. Plan from the start how results will reach the systems and screens people already use, and who will build that link.
4. An owner and a run plan. Someone on the business side has to own the result once it's live: who supports users, who watches quality, who decides on changes, who pays for it next year. Without that person, the tool becomes an orphan the day the project team moves on.
5. A production budget, decided before the pilot. Pilots are cheap on purpose. Production costs more: integration, security, support, licences at full scale, training. If the money for that stage isn't at least roughly planned before the pilot starts, a successful pilot puts you in a bind: the organization has said yes to the idea, but not to the cost.
For AI, two more
AI tools add two requirements that classic systems don't have to the same degree.
First, quality changes over time. Data shifts, processes change, the vendor updates the model. Decide how quality will be measured in production, how often and what level triggers a review.
Second, errors have to be caught. Decide which outputs a person reviews before they're used, and make that review part of someone's job, not a good intention. Where an error is cheap, review less. Where it's expensive, review more. Either way, decide it deliberately.
The rule
Don't start a pilot unless you can answer three questions. If it succeeds, who owns it? Where will it run? Roughly what will production cost? If any of those answers is "we'll see", the pilot is a demo with a longer schedule.
Go/no-go checklist for moving to production
- The success criteria set before the pilot were met, and documented
- The tool was tested on real, unfiltered data
- Integration with the systems people use is designed and budgeted
- A named business owner accepts responsibility for the tool in production
- Support, monitoring and updates are assigned
- The production budget is approved
- Users outside the pilot group are trained, or a training plan exists
- For AI: quality measures, review rules and a threshold for action are defined
- There is a way to roll back if production goes badly
If your pilot is already stuck
It isn't too late to apply the same logic. Write the success criteria now, as if the pilot were starting today, and measure the pilot against them honestly. Then make one of three decisions, with a date:
- Scale it: name the owner, approve the production budget and plan the integration.
- Fix it: define what must change, by when and what result would settle the question.
- Stop it: write down what was learned and close it properly.
Stopping is a legitimate outcome. A pilot that ends with a clear "no" has done its job. The only result to avoid is the one where nobody decides.
A last word
A pilot is a good tool when it answers a real question. It's a poor one when it postpones a decision. If you have a pilot that's stuck, or one about to launch, a short conversation around these questions usually shows quickly which situation you're in.