AI agent governance: designing for the day the agent gets it wrong

Board discussions about agents reach the same question: what happens when the agent gets it wrong? A policy document describes intentions but does not constrain an action. Governance must also be implemented in the software: what the agent may touch, how far one action can reach, what trace it leaves and when a person must approve. These decisions are cheaper to make before the first deployment than to retrofit later.

Design for errors

Agents use probabilistic models, so some decisions will be wrong, just as some human decisions are wrong. A design that assumes zero errors cannot define how to detect, contain or correct them.

The target is three properties: bounded, so a single mistake has a known ceiling; detectable, so it surfaces within an agreed period; and reversible, so it can be corrected. Permissions, action limits, traces and human approval implement those properties.

Permissions: narrower than the humans they assist

A common shortcut is to run an agent with the credentials of the person it assists. That gives the agent every permission of that account, including permissions unrelated to its task.

The design rule is least privilege per task, applied more strictly than you would apply it to people. An agent that drafts supplier emails needs no access to send them. An agent that reads contracts to extract terms needs no write access to anything. Each agent gets its own identity, so the systems it touches record it as itself rather than as the human it assists. The log can then distinguish a change made by the person from one made by the agent.

Narrow permissions make the maximum scope of an error calculable in advance. The boundary is enforced by access controls rather than by an instruction in the prompt.

Action limits: deciding the ceiling in advance

Permissions say what an agent may touch. Limits say how hard. This is the layer most deployments skip, and it is the one your CFO will care about most.

Imagine an agent handling supplier payments at a company doing 200 million. Controls can include a cap per payment, a cumulative daily cap, human approval for a new payee and escalation for deviations from contract terms. Rate limits also matter: an agent can repeat the same error hundreds of times before a person notices. Throttling limits the number and value of affected cases.

Classify which actions are irreversible: sending money, deleting records or making commitments to customers. Require human approval for those actions until evidence supports a different rule, and retain prohibitions where no error rate is acceptable.

Traceability: every action reconstructible

When something goes wrong, the response depends on whether the action can be reconstructed.

The standard I hold agent systems to: for every action, the trace records what the agent saw, what context it was given, what it decided, what it did, and under whose authority. And the trace must be readable by a non-engineer, because the person who will need it is an auditor, a director, or a regulator, none of whom will accept “let me get an engineer to interpret the logs” as an answer.

A useful test is to select a supplier payment from seven months ago and ask why it was made. The system should return the invoice, matching logic, approval trail and acting identity within an agreed time. If those elements require manual reconstruction, the trace is incomplete.

The same trace can support the auditor and measure what each operated process costs, which is the basis of an auditable ROI calculation. One set of records serves both controls and measurement.

Human-in-the-loop by design, not by exhortation

“Human in the loop” needs an implemented routing rule. A training slide that asks staff to review outputs does not enforce review, and repeated correct cases can lead people to approve without checking.

The escalation logic should be code. Triggers may include low confidence, new counterparties, amounts above thresholds or exception categories reserved for people. Other cases can receive sampled review at a rate adjusted from observed errors. Reviewing every case removes much of the economic benefit. Reviewing none leaves no check on model error. Each process needs its own rule, as part of how agents operate.

Why governance must adapt to the operation

Governance requirements also affect the sourcing decision.

With an opaque product, each control is limited to what the vendor exposes. The vendor defines the permission model, available thresholds, log format and retention. If the board requires a guarantee that every new supplier receives approval, the client must verify that the product enforces and records that rule. Configuration alone is not proof.

In a managed operation, thresholds, exception categories and audit formats are defined with the client and encoded with the rest of the process logic. I describe that approach in the operating model as code. Arkatai remains our technical responsibility while the process owner retains authority over its limits. The required level of control is one input to deciding whether the operation is custom-phase work.

A product for a commodity function may have stronger security engineering than an internal implementation, and buying it can be the right decision. I compare that option with internal build and managed service in buy, build or contract the outcome. The governance question is visibility and fit, not whether product companies care about security. Where agents execute company-specific rules, the controls must express those rules.

Questions boards ask me

Who is accountable when the agent gets it wrong?

The process needs a named accountable person before the first agent runs. The agent is a tool. Accountability remains with the owner of the process it operates. If nobody accepts that role, the company is not ready to run the agent.

Do we need an AI governance committee?

A committee can set risk appetite and cross-company policy. Operating governance still needs a person who can change the process and answer for its results. Assigning that owner is part of preparing the company for agents.

Can we let agents touch money at all?

Yes, inside the mechanisms described here: hard caps, gated irreversibles, full traces, humans on the exceptions. Start with actions that are reversible and bounded, let the trace accumulate evidence, and widen limits as the evidence justifies. What I would refuse is the inverse path: broad authority first, controls after the first incident.

What will regulators make of this?

Regimes differ and continue to change, so this is not a legal summary. The system should be able to show what happened, which rule applied and who authorized it. That evidence is useful across regulatory frameworks even when the specific reporting duties differ.