How to scope an AI pilot that can actually reach production
A good pilot ends with a decision, not a demo. Six choices to make before anyone picks a model, so the pilot has a real path to production.
· 5 min read
An AI pilot has one job: to show whether a specific use of AI is worth putting into production. That job is easy to lose sight of. A pilot that proves the technology can do something impressive in a meeting room is not the same as a pilot that tells the budget owner, with evidence, whether to go ahead.
Much of that difference is decided before any code is written, in the scope. Below are six decisions we recommend making up front. None of them is technical, yet each one shapes what the technical work will be able to prove.
1. Choose one process, not a platform
The narrower the pilot, the clearer the answer. Instead of “an assistant for the whole company”, pick a single process with a defined start and end. A good candidate usually has these properties:
- It runs often enough that improving it matters: daily or weekly, not once a quarter.
- It has one owner who can say what a good result looks like and who will actually use the outcome.
- Its inputs are mostly digital already: emails, tickets, forms, documents or records in a system.
- A mistake can be caught and corrected before it reaches a customer, at least while the pilot runs.
Be wary of processes that cross several departments, depend on rules people still disagree about, or happen so rarely that you will not collect enough real cases within a few weeks.
2. Write down the success criteria before you build
If success is defined after the results are in, it tends to be defined around them. Agree on the measures first and have the process owner sign them off. Three values are enough for most pilots:
- Baseline: how the process performs today, measured on real cases rather than estimated. For example, the time spent per case or the share of cases that need rework.
- Target: the result that would justify the cost of building and running the solution in production.
- Quality floor: the level below which an output cannot be used, however much time it saves. A draft reply with the wrong refund amount is not a time saving.
If nobody can measure the baseline today, measuring it becomes the first task of the pilot. It is still worth doing: without a baseline, any result is an opinion.
3. Sort out data access in week one
A pilot run on invented or hand-picked data tells you little about production. Real data needs access, and access needs approvals, which take time. Ask these questions before the kickoff, not when the build is already waiting:
- Which systems hold the inputs, and who can grant read access to them?
- Can we work with the real data as it is, or does it need to be masked, sampled or reduced first?
- Where may the data be processed: which region, which cloud account, which model provider?
- Who signs off on security and privacy, and what do they need from us to do it?
4. Put a person in the loop from day one
During a pilot, a person should review what the system produces before it has any effect: before the email goes out, the record is updated or the case is closed. Decide early who reviews, how they approve, edit or reject an output, and how their corrections are recorded.
Those corrections are valuable. They show where the system fails, and they become the test cases you use to measure every later change. They also give you an honest basis for deciding which steps could later run without review, and which should keep a person in charge.
5. Agree on what “done” means
“The demo works” is not a definition of done. For a pilot, done should mean all of the following:
- It has run on real inputs from the actual process, not on a curated set of examples.
- The people who do the work have used it as part of their normal day.
- The results have been measured against the baseline, with the method agreed at the start.
- The known failure cases are written down, together with an idea of how production would handle them.
- There is a rough estimate of what production would take: integration work, running costs, monitoring and support.
6. Hold a go/no-go decision after four to six weeks
Give the pilot a fixed end date. Four to six weeks is usually enough for one well-chosen process to show whether it works on real data, and short enough that the question being answered is still the original one. At the end, there are three legitimate outcomes:
- Go: the targets are met. Plan production, including integration, a security review and monitoring.
- Adjust: the result is promising, but there is a specific gap. Run one more short cycle aimed at that gap, with a new end date.
- Stop: the target is out of reach, or the value is smaller than expected. Stopping after six weeks is a good and inexpensive result.
Decide in advance who makes the call and which evidence they will look at. That keeps the decision about the business case, not about how much work has already gone in.
Put the scope on one page
A useful test is whether the whole scope fits on a single page: the process and its owner, the baseline and the targets, the data sources and who grants access, the review step, the definition of done, and the date and owner of the go/no-go decision. If any of those lines is still empty, filling it is the first piece of work. It costs far less than discovering the gap in week five.
Before you build an internal knowledge assistant: a practical checklist
An assistant that answers from your own documents is only as good as those documents, and only as safe as its access rules. What to check before the first line of code.
The EU AI Act for business teams: what changes when you deploy AI
A plain-language overview of the risk-based rules, the duty to tell people when they are dealing with an AI, and how it all fits with GDPR and KVKK. General information, not legal advice.
Reading is good. Building is better.
Bring a problem. We will bring a plan.
Or write to us: hello@nbmcore.com