Measure the workflow before you automate it

An AI demo can finish a task in seconds and still make the surrounding workflow slower. The output may need a new review step. Exceptions may bounce between teams. A person may spend ten minutes finding the source that the system should have cited. None of that appears in a prompt test.

Before deciding what to automate, measure the work as people perform it today. The baseline gives the project a job: improve a named outcome without breaking the limits that matter. It also gives the team permission to stop when the proposed system cannot do that.

This is not a request for a six-month process study. A useful baseline can be modest. It needs a clear boundary, consistent definitions, enough cases to expose normal variation, and an owner who trusts the numbers.

Start with the decision

Write the decision that the measurement must support. “Explore AI for operations” is too loose. “Decide whether to pilot automated invoice intake for North American suppliers” is specific enough to bound the work, the population, and the people involved.

Then name the outcome in plain language. Perhaps invoices should reach approval sooner. Perhaps the team needs to absorb seasonal volume without adding another queue. Perhaps the current correction rate is creating duplicate work downstream. Pick the outcome the workflow owner would recognize, not the metric that makes the model look good.

The GAO AI accountability framework makes the same connection at a broader level: system performance metrics should map to the program’s intended outcomes. Model accuracy may contribute to an outcome, but it is rarely the outcome itself.

Set a decision threshold before anyone tunes a prompt. A threshold can be directional at first: reduce median elapsed time by at least 20 percent while keeping the downstream correction rate at or below its current level. The exact figure should come from the economics and risk of the workflow, not from what seems achievable in a demo.

Draw the workflow people actually run

Start and end points are where baselines quietly go wrong. If the clock starts when a file reaches the automation but today’s clock starts when an email enters a shared inbox, the comparison has already been tilted. Use the same trigger and the same definition of done for both versions.

Follow several cases through the work. Record queue time as well as hands-on time. Note each handoff, system lookup, approval, correction, and return to an earlier step. Ask what happens when an input is incomplete or arrives in the wrong format. The written procedure will usually describe the normal path. The baseline must include the path that people use when the normal one fails.

Existing logs can answer part of this. Ticket timestamps, audit histories, and status changes often show volume and elapsed time. They may not show why a case waited or how much effort a workaround required. A short sample of direct observation or time recording can fill that gap. Keep the method light enough that people will use it consistently.

Do not average away the queue. If most cases finish in a day and a small group waits two weeks, the mean can improve while those costly cases remain untouched. Record a median and a high percentile, such as the 90th percentile, when volume allows. Keep exception cases visible as their own cohort.

Use a small metric set with hard definitions

A baseline needs both an outcome measure and diagnostic measures. The outcome decides whether the change helped. Diagnostics explain why it helped or failed.

Questions, practical measures, and definitions for a workflow baseline

Question	Practical measure	Definition to settle in advance
Did work finish sooner?	Median and 90th-percentile elapsed time	Exact start, finish, business-hours rule, and paused states
Did it consume less effort?	Human touch minutes per accepted case	Which review, correction, and support activities count
Did quality hold?	Downstream corrections per 100 completed cases	What qualifies as a correction and how long the observation window lasts
Did capacity change?	Accepted cases per constrained staff hour	Which role is the constraint and what “accepted” means
Did the economics work?	Total operating cost per accepted case	Labor rate, software, model use, monitoring, and exception costs
Did risk stay inside limits?	Count and severity of prohibited outcomes	Severity levels, escalation rules, and who adjudicates an event

An “accepted case” matters because raw throughput rewards systems that produce fast, unusable work. Similarly, automation rate can hide a growing exception queue. Report straight-through completion alongside the volume sent for review, then count how often reviewers change the result.

Write a short data dictionary. If one manager treats a returned invoice as complete and another does not, the dashboard will look precise while comparing different events. Definitions should cover missing data too. Track missing observations separately instead of entering zero.

Count the work the pilot creates

Project teams tend to count model calls because those costs are easy to export. The larger cost may sit elsewhere: preparing inputs, checking outputs, maintaining integrations, investigating alerts, and helping users recover from an error.

Use a full numerator:

Operating cost per accepted case unit cost = labor + software + model use + review + rework + monitoring + incident cost

Divide that total by cases that reached the agreed definition of done. Keep one-time build cost separate from recurring operating cost, but report both. A cheap run is not cheap if a specialist has to rescue one case in five.

Review time deserves its own measurement. Teams sometimes add mandatory review during a pilot and assume it will disappear later. That may be reasonable, but it is still an assumption. Report the pilot with review included, then model any future reduction as a separate scenario.

A worked example

Consider a hypothetical accounts-payable team assessing invoice intake. This is an example, not a reported client result.

The team receives 2,400 invoices in a typical month. About 18 percent enter an exception path. Clean invoices require a median of four minutes of human touch time; exception cases require 22 minutes. Three and a half percent of completed invoices receive a downstream correction within 30 days. The team’s main complaint, however, is not touch time. It is the age of the shared queue before someone opens the invoice.

A prototype extracts fields with high accuracy on 50 clean PDFs. That result says little about the business case. The baseline suggests a different evaluation. The pilot should include scans, credit notes, duplicate invoice numbers, partial purchase-order matches, and documents missing a required field. It should measure time from inbox arrival through entry into the existing approval process.

Suppose the system cuts extraction effort but every invoice now receives a two-minute review. Clean cases save time; some exception cases take longer because reviewers compare the draft against source records. Whether the pilot wins depends on the mix of cases and what happens to the queue. The team can calculate the result without pretending that extraction accuracy settles it.

The same baseline may point to a simpler first change. If most elapsed time sits before assignment, better routing or queue ownership could produce more value than AI extraction. Finding that is a good outcome for the assessment, even when it means dropping the proposed tool.

Compare like with like

A before-and-after chart is vulnerable to changes in volume, staffing, policy, and case mix. If possible, compare the pilot with concurrent work handled under the current process. Assign cases in a way the operating team considers fair, and record the attributes that affect difficulty.

When a concurrent comparison is impractical, use matched time windows and explain the differences. Separate routine and exception cohorts. Keep the same clock, completion rule, and correction window. Do not compare a supervised pilot team during a quiet week with a normal team at quarter end.

Small samples need restraint. Report the counts and the spread, not just a percentage with two decimal places. Read the failed cases. A modest pilot can support a decision about the next test; it usually cannot prove that every future case will behave the same way.

This is consistent with NIST’s guidance to test systems under conditions similar to deployment and to document test sets, metrics, and limitations. The Measure section of the AI Risk Management Framework also calls for field data to show measurable improvements or declines, rather than relying only on development results.

Keep a baseline after launch

The baseline becomes the first production comparison. Preserve its definitions and raw counts. Add live measures for case mix, review rate, corrections, latency, and incidents. NIST’s Measure playbook recommends revisiting metrics as the operational setting and data change. A metric that was useful during a pilot can lose meaning after policy, staffing, or upstream systems change.

Assign one person to review the measures on a schedule and one workflow owner to decide what happens when a limit is crossed. Decide in advance whether a breach triggers case review, rollback, narrower scope, or a new evaluation case based on the failure.

The project is ready to build when the team can state what starts the workflow, what counts as done, what outcome must improve, which limits cannot move, and how the new version will be compared with the old one. If those answers are still disputed, another prototype will produce more screenshots, not better evidence.