Agent programs get measured badly for a predictable reason: the artifacts they emit - missions run, tokens burned, tool calls made - are all activity metrics, and activity is precisely the thing agents make cheap. Measuring an agent program by missions run is measuring a factory by electricity consumed. The corrective is to measure agents the way you would measure a team that produces work: what did the work cost, how much ran without help, was it right, and how fast did it arrive. Four metrics carry that - this article defines each, shows the instrumentation, and names the traps.
Measure work, not activity
The unit that matters is the delivered outcome: an audit with ranked fixes, a verified account list in the CRM, a published article, a launched campaign, a teardown a decision was made on. Outcomes are what teams used to buy with hours or agency retainers, which is what makes them comparable across the before and after. Every metric below is a property of outcomes - their cost, their independence, their correctness, their latency - because a program's value is the stream of outcomes it ships, and nothing else survives contact with a budget review.
Metric one: cost per outcome
All-in cost - model spend, tool metering, the human review minutes - divided by delivered outcomes, computed per mission type. The number is only meaningful against its counterfactual, priced honestly: a teardown mission at a few dollars replaces either two days of a strategist (hundreds of dollars), an agency line item (more), or the teardown not existing and the decision being made blind (priced in regret). The complete AI marketing agents guide makes this argument at program level; the KPI version is just that argument with a denominator.
Watch the trend as much as the level: cost per outcome should fall as mission types mature - better plans, fewer retries, cheaper model routing per the stack guide - and a rising curve is an early smell of scope drift or degraded data sources.
Metric two: autonomy rate
The share of missions that complete without a human touching them mid-flight - excluding designed touchpoints like approval gates, and tracked per mission type. It is the program's maturity curve made visible: new mission types start low (everything gated, per the governance ladder for autonomous AI agents), and rise as types graduate. Two readings matter. Level: a mature read-only type below ~90% has a reliability problem worth debugging in the trail. Slope: a program whose blended autonomy rate is flat for a quarter is accumulating gated types without promoting any - governance theater, usually.
per-typethe only honest way to read autonomy rate
Metric three: evidence quality
The quality claim of agentic work is that findings cite sources and numbers come from computation - so audit the claim. The protocol: sample deliverables monthly, follow each sampled citation to its source, check the number against the tool output, and record the pass rate. It is an hour of work that keeps everyone honest, catches data- source rot early (a vendor quietly degrading shows up here first), and produces the calibration that makes the rest of the organization trust the program. A platform where this sampling is hard - where provenance is not two clicks away - has told you something important about itself.
Metric four: cycle time
Objective-to-deliverable latency, per mission type. Its value is mostly comparative: the before-state was measured in days-to-weeks (the audit that took a sprint, the teardown that took two days, the list that took a week of SDR time), and the after-state is minutes-to-hours. Cycle time is also where queue health shows up - a type whose latency is dominated by approval wait needs either a faster reviewer rotation or a promotion review, and the metric is what surfaces which.
Instrumenting all four from the audit trail
| Metric | Source fields | Cadence |
|---|---|---|
| Cost per outcome | Settled mission cost ÷ delivered outcomes, per type | Monthly |
| Autonomy rate | Missions without mid-flight intervention ÷ total, per type | Monthly |
| Evidence quality | Sampled citation pass rate | Monthly sample |
| Cycle time | Deliverable timestamp - objective timestamp, split by queue wait | Monthly |
The load-bearing observation: all four fall out of the mission record described in the execution walkthrough - objective, costs, interventions, gates, timestamps, citations. If computing them requires interviewing the team, the platform lacks a real trail, and that absence is itself the finding.
The vanity traps
The monthly review that keeps programs honest
The operating ritual is one meeting: the four numbers per mission type, month over month, with three standing questions. Which types earn promotion on the governance ladder (queue evidence says boring)? Which types earn investigation (cost rising, evidence slipping)? Which candidate missions from the backlog earn a pilot (the playbook library is the menu)? Thirty minutes, evidence on screen, ladder moved or not - that cadence, more than any dashboard, is what separates programs that compound from programs that plateau at demo.
Frequently asked questions
What KPIs should an AI agent program track?
Four: cost per delivered outcome (against the honest counterfactual), autonomy rate per mission type, evidence quality from citation spot-checks, and objective-to-deliverable cycle time. All four compute from the mission audit trail.
How do you measure agent ROI?
Price each delivered outcome against what it replaced - hours at loaded cost, agency fees, or a decision made blind - and compare with the all-in mission cost. The comparison is per outcome, never per token.
What is a good autonomy rate?
It depends on type and maturity: mature read-only missions should exceed ~90% untouched completion, while newly gated types are correctly near zero. The slope across the program matters more than any single level.
How is evidence quality actually checked?
Monthly sampling: pull a handful of deliverables, follow each citation to its source, verify each number against the tool output, and record the pass rate. An hour of work that catches both model drift and data-source rot.
Which agent metrics are vanity?
Missions run, tokens consumed, tool-call counts, self-reported hours saved and vendor benchmark scores - all can improve while the program degrades. If a metric cannot get worse when the program gets worse, it is decoration.
Sources
- NIST AI Risk Management Framework - measurement and monitoring vocabulary
- Google - the HEART framework paper lineage for outcome-over-activity metrics
Every playbook on this blog ships as a runnable mission.
Open a workspace and the playbook library is waiting - describe the outcome and the agents carry it end to end, on your plan's monthly credits.