Your AI pilot works. Why hasn’t it shipped?

A good pilot can become strangely difficult to ship. It handles the sample cases and reviewers like the output. Then the work stalls over integration, security, ownership, or another prompt change.

The pilot may be fine. What is missing is evidence that the whole service can run under ordinary conditions without its builders standing beside it. A demonstration answers, “Can the model perform this task?” Production asks a harder question: “Can the organization depend on this system, detect when it is wrong, and recover without improvising?”

Treat the delay as a diagnostic problem. Find the unproven part, assign an owner, and decide what evidence would close it. Another polished demo rarely does.

A pilot proves capability, not a service

Pilot conditions are forgiving. A developer can retry a failed call, repair malformed data, or explain a confusing result. The system may also sit outside the identity, logging, approval, and records policies that apply to production software. The test was built to answer a narrower question.

A production service has to handle the rest of the workflow. It receives incomplete inputs, waits on slow dependencies, respects access rules, and gives an operator enough information to understand a failure. It has to work after a model, prompt, retrieval index, policy, or connected API changes.

Start by locating the gap instead of treating “production readiness” as one large concern.

Area	Evidence needed before release	Common warning sign
Outcome	A baseline, a target, and a method for measuring both	The result is described as “better”
Ownership	Named people for the workflow and the technical service	The pilot team owns everything collectively
Quality	Tests that represent normal cases, exceptions, and costly failures	Reviewers have seen only handpicked examples
Integration	Production-shaped data, permissions, interfaces, and failure handling	The demo depends on copying data between tools
Operations	Actionable alerts, recovery procedures, and a known safe fallback	There is a dashboard but no response owner
Adoption	Users can complete the changed workflow and report a bad result	Training and review effort are assumed to be small

One empty cell can delay a responsible launch. It may also reveal that more model tuning will not resolve the blocker.

Name the workflow owner

An executive sponsor can fund the work and remove obstacles, but sponsorship is not day-to-day ownership. Production needs someone who can define a correct business outcome, resolve exceptions, and decide whether the service should continue operating when its results deteriorate.

There is usually a separate technical owner for the application, dependencies, deployment path, and recovery controls. Other teams may own specific approvals. Write down the boundaries. A shared channel does not tell an operator who can authorize a rollback at 2 a.m.

The NIST AI Risk Management Framework Core calls for documented responsibilities, lines of communication, ongoing monitoring, and executive responsibility for deployment risk. Accountability has to survive the handoff from builders to operators.

Before release, name who accepts the quality threshold, reviews reports, approves behavior changes, pauses automated actions, and owns the manual queue when the AI path is unavailable.

Write a release contract

A team cannot decide whether the pilot is ready until it agrees on what ready means. Write a short release contract for one unit of work. It should state the input, required output, permitted actions, fallback, success measure, and conditions that force escalation.

Consider a hypothetical support-ticket pilot. A useful contract might require the system to reduce median routing time by 30 percent without increasing the verified misroute rate. Security and billing requests must always enter specialist queues. Tickets without enough evidence must remain unassigned and ask for review. The numbers are illustrative. Speed cannot hide a drop in correctness, and an average score cannot hide a costly category of error.

Measure the current workflow before choosing the target. Include the time people spend preparing inputs, checking outputs, correcting records, and resolving exceptions. A model response that arrives in two seconds is not a two-second workflow if an employee needs six minutes to verify it.

The contract should also identify the release boundary. Drafting a response for approval is a different service from sending it. Recommending a refund is different from issuing one. A pilot may be ready for the first boundary while the second remains unjustified.

Test the system on production-shaped cases

Ten successful demonstrations do not estimate production quality. Build an evaluation set from historical work, with the necessary privacy controls, and preserve the distribution that the service will encounter. Include routine cases, missing or conflicting data, prior failures, and inputs that should be refused or escalated.

The OpenAI evaluation best-practices guide recommends task-specific tests that reflect real traffic, human calibration of automated scores, and continuous evaluation as the system changes. Those principles are provider-independent. The test should represent the job, not flatter the model.

Score the whole path. Check whether ingestion preserved the source correctly, retrieval supplied the right evidence, tool calls used valid arguments, and approval rules stopped prohibited actions. For an agent, inspect the route it took and where it stopped. A strong final answer can still conceal an unsafe tool call or a lucky recovery from bad retrieval.

Define release thresholds before running the final comparison. Separate ordinary quality misses from failures that violate policy or create material harm. Averages are useful for broad comparison, but a critical class may need its own threshold. Keep failed cases. They are the start of the regression suite for the next prompt, model, or workflow change.

Close the integration gap

If the pilot depends on a spreadsheet export, a developer’s credentials, or manual copy and paste, integration remains untested. The production path needs the same identity model, data freshness, network boundaries, and record-keeping rules as the systems around it.

Exercise failure behavior deliberately. What happens when a provider rate-limits requests, a document index is stale, an API returns half a record, or a queue backs up? Retries need limits. Actions that may repeat need idempotency controls or another way to prevent duplicate effects. Timeouts should route work somewhere visible instead of leaving it in an ambiguous state.

Version the pieces that can change behavior: application code, model choice, instructions, tool schemas, retrieval configuration, policy rules, and reference data. A production result should be traceable to the configuration that created it. Without that record, the team cannot reproduce a failure or know what to roll back.

Prepare the operating controls

Track demand, latency, errors, saturation, dependency failures, and cost. Add workflow measures such as correction rate, approval burden, stale retrieval, and the outcome named in the release contract.

The Google Cloud operational-readiness guidance puts roles, observability, service disruption, tooling, and service levels in the same readiness model. Model quality cannot be separated from the system and people responsible for it.

Every alert needs a threshold, an owner, and a first action. Add a feature flag or other tested way to stop the AI path. Keep a known-good configuration. Document how work moves to a manual queue and how operators prevent duplicate actions during recovery. A dashboard can show that something changed; a runbook tells the operator what to do about it.

Access should match the release boundary. If the service only drafts, it does not need permission to send. If it reads four systems, avoid granting a broad service account because it was convenient in the pilot. Set request, retry, time, and spend limits close to the action they constrain.

Move through real work gradually

Begin with shadow operation when the workflow allows it. Give the system live inputs, prevent external effects, and compare its result with the completed human process. This reveals production data and timing problems while the existing workflow remains authoritative.

Next, let users see the recommendation and decide whether to accept it. Measure how often they edit, reject, or ignore it, along with the time required for review. Those signals expose whether the pilot has moved work or merely added another screen.

When the evidence holds, release to a limited group or a small share of eligible work. Google’s guidance on canary releases explains why partial, time-limited exposure can reveal problems that test environments miss while reducing the impact of a bad change. For AI systems, evaluate the workflow outcome and policy failures alongside HTTP errors and latency.

Expand only after a defined review window. Keep the fallback staffed until the team has observed enough real volume, including exceptions, to trust the operating process.

Treat adoption as production work

People need to know what the service may do, when they must verify it, and where a disputed result goes. Train with awkward cases. Show source evidence and system status when reviewers need them, and make feedback part of the workflow.

Watch for responsibility without authority. If an employee remains accountable for every result but cannot inspect the evidence or override the system, adoption will be rationally cautious. If review is mandatory, include that work in the measured process. The goal is not a high acceptance rate. It is a better workflow with known limits.

Set a post-release review date before launch. Bring the workflow owner, technical owner, and frequent users. Compare the production result with the baseline, review incidents and corrections, and decide whether to expand, change the boundary, or return work to the previous process.

Run a readiness review with evidence

A useful readiness review is short because the work happened beforehand. Bring links, test results, and named owners rather than a presentation about confidence.

• The release contract defines the workflow outcome and authority boundary.
• Historical cases cover normal work, exceptions, refusals, and costly errors.
• The candidate configuration passes its quality and policy thresholds.
• Production identity, data, interfaces, and audit records have been exercised.
• Operators can detect a bad result and trace it to a versioned configuration.
• A tested procedure pauses the AI path and routes unfinished work safely.
• Users have completed the changed workflow and their review time is measured.
• The workflow owner accepts the remaining risk and the first review date is booked.

If one item is missing, assign the evidence needed, its owner, and a decision date. If nobody will own the service after launch, stop. The pilot has found a capability, but the organization has not chosen to operate it. If the owners and evidence are in place, the next step is a controlled release, not another demo.