# What to measure after an AI system goes live

> A launch is the beginning of operational evidence. These measures show whether the system is becoming more reliable.

Published: 2026-04-14
Updated: 2026-04-14
Category: Operations

An AI system can pass a test set, perform well in a demonstration, and still create trouble in daily work. Launch introduces real volume, incomplete inputs, changing customer behavior, edge cases, and the ordinary pressure of a team trying to get through the day.

That is why deployment should begin a measurement cycle, not end the project.

## Measure the operating outcome

Start with the result the workflow is meant to produce. That result should already be visible in the assessment, because [the work should decide where AI belongs](/journal/the-work-should-decide-where-ai-belongs). If the system answers calls, the goal is not simply “calls answered.” The useful outcome may include qualified bookings, correct handoffs, complete records, and confirmations sent before the interaction ends.

Good operating measures answer questions such as:

- Did the work finish?
- Was it correct enough for its consequence?
- Did the right person receive the exceptions?
- How much time did the system remove or add?
- What happened to the customer or employee experience?

Model-level measures still matter, but they do not replace these outcomes. A system can generate fluent answers and still fail to complete the work.

## Track completion and correction together

Completion rate is attractive because it produces one clean number. On its own, it can reward the wrong behavior. A system that attempts everything may report high completion while creating a large correction burden later.

Pair completion with measures such as:

- correction rate;
- reopened or repeated requests;
- records changed after review;
- commitments reversed by a person; and
- downstream work caused by an incorrect action.

This pairing helps distinguish useful autonomy from work that only appears finished.

> A completed task is not evidence of value if someone has to quietly repair it later.

## Treat escalation as a signal

Measure how often the system hands work to a person, why the handoff happened, where it went, and whether the receiving person had enough context. These measures are only useful when [escalation is designed as a product feature](/journal/escalation-is-a-product-feature).

Do not begin with the assumption that fewer escalations are always better. A high escalation rate may reveal a missing source or an overly narrow permission. It may also show that the system is correctly refusing work with significant consequences.

The target is appropriate escalation: the right cases, at the right time, to the right owner.

## Watch the distribution, not only the average

Averages can hide the cases that damage trust. An average response time of ten seconds says little if a small group of requests waits several minutes or never receives an answer.

Review the range:

- the fastest and slowest completion times;
- outcomes by request type or channel;
- performance during busy periods;
- differences across sources, locations, or teams; and
- the small set of cases with the highest consequence.

This is where operational patterns appear. The average may stay stable while one category begins to fail.

## Measure the review burden

Human oversight is real work. It should be planned and measured rather than treated as free capacity.

Track how many items require review, how long review takes, what reviewers change, and whether the same issue appears repeatedly. If review volume rises faster than useful throughput, the system may be shifting work instead of reducing it.

Review data can also justify improvement. Repeated corrections may point to an unclear instruction, a weak source, a missing field, or a task that should be removed from the system’s scope.

## Record changes to autonomy

Permissions should not drift silently. When the team allows the system to handle more—or requires it to handle less—record the reason, the evidence, the date, and the person who approved the change. This is the operating connection between measurement and a broader [digital transformation strategy](/journal/digital-strategy-vs-digital-transformation).

This creates an operating history. It helps a team understand whether performance improved because the system changed, the workflow changed, or the boundary moved.

A simple autonomy review can ask:

- Which cases consistently succeed without correction?
- Which cases still require judgment?
- Which failure modes repeat?
- Is the escalation path working?
- What evidence would justify the next change?

## Use a regular review rhythm

Early after launch, review should be frequent. As the system and workflow stabilize, the rhythm can slow, but it should not disappear.

The meeting does not need to be large. It needs the people who own the workflow, the people who handle exceptions, and the people who can change the system. A useful review follows four steps:

1. Review a small set of operating measures.
2. Inspect representative failures and exceptions.
3. Agree on the changes the evidence supports.
4. Name the owner of each change.

## Autonomy is earned after launch

The most responsible systems are not frozen at their initial settings, and they are not allowed to expand on their own. They operate inside defined permissions, produce evidence, and change through deliberate review.

Launch creates the first real evidence of how the system behaves inside the work. Measurement turns that evidence into decisions. That is how autonomy becomes something a system earns—not something a team assumes because the demo went well.