Designing an accountable agent architecture

Part 2 of 2 — the architecture. Part 1 set out four production realities. This is how they are answered.

Part one described four problems: telemetry that quietly becomes a copy of every customer conversation, models that change without notice, human review that degrades into rubber stamping, and customer-facing obligations arriving unevenly across jurisdictions. None of them is solved in isolation. They are answered by one architecture, and the shape of that architecture follows directly from the shape of the problems.

Governance is not the architecture. It is one responsibility within it. An agent cannot be held accountable by prompts, dashboards, or an approval queue, because none of those sit in the path of the action. Accountability has to exist where the work happens, which is why what follows is organized around what an agent must be able to answer rather than around what a platform can display.

Four questions an agent has to answer

Delegating work to a capable person is safe not because they never get anything wrong, but because they can be asked. The same four questions apply to software that acts on a business's behalf, and each one has a different audience and a different clock.

One instrumentation layer, forked by purpose. Every consequential action is recorded, including the ones that were blocked. Validated outcomes become the record a tenant can show; failures and exceptions enter remediation, where judgment, root cause and governed changes feed the system forward.

Where enforcement actually sits

The most common design mistake is treating governance as something that reports after the fact. A budget that appears in a report once the money is spent is not a budget, and a consent check that runs after a document is requested has not protected anything.

Enforcement therefore sits at three controlled chokepoints, with each operation passing through the relevant one or more before it executes. The third is the one most often missing.

That third checkpoint is the difference between governing language and governing consequences. An agent that has been permitted to use a model has not thereby been permitted to issue a refund.

Capture once, fork by purpose

Instrumentation happens once. Every step emits a span through one open standard, tagged with its tenant at the moment of emission. That single capture then forks, because the four questions make contradictory demands on the same event.

The same moment has to be both disposable and durable. That is why keeping more logs does not solve the problem, and why the filter between the two stores is the whole design. If every event were promoted to evidence it would not be evidence, it would be expensive logs. If none were, there would be nothing to produce when a customer asks.

What counts as proof

A quality score is a signal, not proof. An evaluation model can indicate that an answer looked well grounded; it cannot establish that a refund reached the customer, that a record actually updated, or that consent existed before a document was requested.

Telemetry tells you what the system reported. Evidence establishes what the business can substantiate.

Promotion into the evidence record therefore depends on validation rather than scoring: acknowledgement from the system of record, deterministic reconciliation against a business rule, delivery confirmation, or human approval of the authorization decision where policy requires it.

The record also has to include what did not happen. A policy block, a denied action, an override, an abandoned workflow: these are frequently the entries that matter most in an investigation, and an audit trail containing only successes is not an audit trail. Validated outcomes become the proof a tenant can show. Failures, blocks and overrides route into remediation, and both remain in the record.

Where judgment enters

Everything above produces one output a person actually touches: a queue. Its design decides whether human oversight stays real or becomes ceremony — the failure mode Part 1 traces in detail. The architectural response to a degrading queue is to reduce what arrives rather than to add reviewers. Anything a rule can decide never reaches the queue at all. What should arrive is what genuinely needed judgment: the held action, the low-confidence answer, the case outside policy. Each item then has to be decidable in one sitting, carrying what happened, why it stopped, what the agent proposed, and a link to the full sequence behind it.

Five conditions raise an item, and naming them is what stops the queue becoming a catch-all.

The operator then has five moves, and the fifth is the one that keeps the system honest: approve as proposed, edit and approve, decline with a reason drawn from a defined list, escalate to someone with more authority, or mark the flag itself as wrong. That last action is the feedback that tunes a threshold. Without it, the only available correction is to keep raising the bar until the queue empties for the wrong reason.

Every resolution closes two loops. It writes an evidence record, because a human decision to proceed carries the same weight as the agent's would have. And it returns a data point to measurement about whether the rule that raised the item was calibrated correctly. That return path is what separates remediation from a support queue. A ticket ends when it is closed. A remediation event feeds the system that raised it.

Human review resolves an exception. Remediation changes the system that produced it.

From exception to fix

Resolving an item and preventing its recurrence are different acts. An override fixes one instance. A test case catches the next one. Only a change to the system stops it happening, and which change depends entirely on what the root cause turned out to be.

Most of that is configuration rather than retraining. Knowledge, instructions, thresholds and routing account for the large majority of corrections, which matters because it puts remediation within reach of a team that has no machine-learning function. Each of those changes is itself a governed change: a revised instruction, a new threshold, an updated document set each gets a version, and that version is recorded against every action taken under it. Without that, the next investigation cannot tell whether behaviour changed because the agent drifted or because somebody fixed it.

Autonomous correction sits deliberately outside all of this. A system that acts on its own findings needs observation to be complete and enforcement to be reliable first, which is why it is the last thing to build rather than the first thing to demonstrate.

Across both surfaces

An action taken on a business's behalf carries the same obligation wherever it originates, so the architecture applies uniformly across two categories of work. Conversations are a two-way exchange, opened by either party, with a customer or an employee, across chat, messaging and voice. Workflows and tasks are orchestrated multi-step work that reasons through exceptions, scheduled or event-triggered, customer-facing or internal — and a workflow may open a conversation as one of its steps. A scheduled run at nine in the morning produces the same trace, the same enforcement, and the same evidence record as a customer message at eleven at night. That uniformity is the point of a single spine.

The channel itself is a control point rather than a transport. Two obligations that Part 1 identified belong here rather than in a prompt, because a prompt can be edited and a channel setting cannot drift silently.

Architectural dependencies

What follows is a dependency order rather than a delivery schedule. Actual implementation sequencing depends on team capacity and commercial priority. What does not vary is which capabilities depend on which, and one of them cannot be added retroactively at all.

The instrumentation contract

One detail decides whether any of the above works: what every span is required to carry. Agreeing this before instrumentation begins is what allows four different consumers to read one capture. OpenTelemetry generative-AI semantic conventions cover the model call, and a small namespace carries multi-tenancy. The tool-call row is the one most often omitted, and it is the row that makes an action provable.

The evidence record is a different artifact rather than another span: a stable identifier, the tenant, the event type, the actor, the model and prompt version, a link back to the trace that survives the trace being purged, the payload of what changed, an authoritative timestamp, and a chain link making later alteration detectable.

An agent that cannot account for itself is not autonomous. It is unsupervised.