Autonomy's value and its risk are the same property: the system acts while nobody watches. That is why "should we let agents do things?" is the wrong question - the right one is "under what guarantees?", and it has a concrete, buildable answer. Three mechanisms - cost caps enforced by a ledger, approval queues on consequential writes, and audit trails on everything - convert agent risk from an anxiety into an engineering property. This article covers each mechanism, the graduated-autonomy habit that ties them together, and the policy template that survives contact with a real team.
The real question governance answers
When a leadership team hesitates on agents, the hesitation decompresses into three specific fears: it will spend money we did not approve; it will say something to a customer we would never say; and if something goes wrong, we will not be able to reconstruct what happened. Notice that none of these is "the model might be wrong" - wrongness is caught by review, like a junior hire's wrongness. The fears are about unbounded consequence, and each maps to exactly one mechanism. Bound the spend, gate the writes, record everything - and the residual risk profile looks like any other delegated work.
Mechanism one: hard cost caps
The distinction that matters is enforcement location. A limit the agent is told about is a request. A limit checked at day's end is a report. A limit the ledger enforces - where the mission reserves its worst case before executing and physically cannot exceed the envelope - is a guarantee, and only guarantees govern. The full pattern lives in Designing budgets for autonomous agents: reserve-then-settle per mission, stacked ceilings per period, structured stops that park work safely with an explicit gap instead of dying silent.
The governance payoff is worst-case arithmetic. "What is our maximum exposure if every scheduled mission runs wild tonight?" has a number, computable from the envelopes - and a program whose worst case is a known number is a program finance can approve. 1number your CFO actually needs
Mechanism two: approval queues
The read-write divide does the philosophical work: reads (research, audits, analysis) are safe at any scale and run unattended; writes that touch the world - spending, publishing, contacting humans, mutating records - carry consequence and wait for sign-off. The engineering that keeps this from becoming bureaucracy is described in Approval queues that keep autonomy fast, and its critical property deserves restating: the queue parks the exact call with its exact arguments. Reviewing means reading precisely what will execute; approving executes precisely that. No paraphrase, no drift between what was shown and what was done.
Well-run queues are fast because review is editing, not creation - a finished campaign read in minutes, a drafted send skimmed and approved from a phone. The queue is also where the evidence for promotion accumulates, which is what the graduated model below runs on.
Mechanism three: audit trails
Every mission records its objective, plan, tool calls with arguments and results, costs, gates hit, decisions taken, and deliverable. The trail serves three distinct audiences. Operators debugging a weird deliverable walk the steps instead of guessing. Compliance answers "who approved this send and on what evidence" with a record instead of a meeting. And the team's own trust-building runs on it: spot-check citations, compare claims to sources, and calibration replaces faith. A useful test of any platform's trail: pick a number in a deliverable and demand its provenance in two clicks.
Graduated autonomy: the operating habit
| Stage | Writes | Promotion evidence |
|---|---|---|
| 1 · Fully gated | Every write queued | n/a - the starting state for every new mission type |
| 2 · Spot-checked | Routine writes auto-approve; samples reviewed | Weeks of queue history with near-zero edits |
| 3 · Unattended | Runs on schedule; exceptions queue | Sustained clean record + bounded blast radius |
Two rules make the ladder trustworthy. Promotion is per mission type, never global - the weekly search-term audit earning stage three says nothing about outbound sends. And demotion is instant on incident: one bad outcome returns the type to stage one, with the trail explaining what happened. Teams that run this ladder stop having the abstract "do we trust AI" debate, because trust became a property of specific mission types with specific records.
A policy template that survives contact
The whole policy fits on a page, which is why it gets followed. Per mission type: its budget envelope; its gate status (stage one, two or three); who approves (role, not name); response-time expectation for the queue; and the demotion trigger. Plus three global lines: no autonomous external contact below stage three; no destructive operations (deletes, bulk mutations) above stage one without a named exception; and quarterly review of the ladder against the audit trail. Everything else - model choice, tool grants, schedules - is operations, not policy, and lives with the team. The stack guide covers where each piece is enforced.
Governance anti-patterns
The common repair is the same: put the rule where it is enforced, and let the evidence - never the sentiment - move the ladder. Measured this way, governance is not overhead on the agent program; it is the KPI system wearing its work clothes.
Frequently asked questions
What does governing an AI agent actually mean?
Three enforced mechanisms: hard cost caps reserved before execution, approval queues that park consequential writes with their exact arguments, and audit trails recording every call, cost and decision. Everything else is operations.
Why are prompt instructions not enough?
Instructions are requests the model usually honors; governance requires guarantees. Only enforcement below the agent - in the ledger and the tool layer - holds under retries, edge cases and adversarial inputs.
How fast should approval queues be?
Fast enough that review is editing, not rework - typically same-day, often minutes from a phone. Queues stay fast when they park finished artifacts and when routine, proven mission types graduate to spot-checking.
When should a mission type run unattended?
After weeks of queue history with near-zero edits, a bounded blast radius, and a demotion trigger agreed in advance. Promotion is per mission type and reverses instantly on incident.
Who should own agent governance?
The team running the missions owns operations; a named approver role owns each gated type; leadership owns the one-page policy and its quarterly review against the audit trail.
Sources
- NIST AI Risk Management Framework - the reference vocabulary for AI controls
- Anthropic - Building effective agents (bounding agent actions)
Every playbook on this blog ships as a runnable mission.
Open a workspace and the playbook library is waiting - describe the outcome and the agents carry it end to end, on your plan's monthly credits.