Operations and Governance

For an active incident, start with the recovery lifecycle. This chapter establishes the ongoing ownership and controls that support those actions.

Operational systems earn reliability through structure.

Once an AI system is in use, it carries operating responsibilities, including during a live experiment. It has users who depend on it, costs that accumulate over time, decisions that propagate beyond their point of origin, and consequences that persist after any single release. Operations and governance are how those realities are handled deliberately.

I have seen capable systems struggle because no one was clearly responsible for deciding when to slow down, change course, or stop. Governance places that authority before it is needed.

Well-run governance gives systems a stable shape as they grow. Reviews and controls consume time and effort; their value is in making the commitment clear and keeping its consequences manageable.

In practice, operational governance answers a small set of questions that never fully go away:

• Who is accountable for this system’s behavior?
• How are decisions reviewed, changed, or reversed?
• What signals tell us the system is healthy, drifting, or creating risk?
• When intervention is required, who acts, and with what authority?

These structures are part of how the system functions day to day.

Governance as an operating discipline#copy

Exercise governance through ordinary operating decisions as well as incident response.

In systems that operate well, governance shows up in ordinary moments: during deployment decisions, when expanding scope, when evaluating a change request, or when deciding whether the system should act with more autonomy than it did yesterday.

In those environments, governance is experienced less as restriction and more as clarity. Teams know what is allowed, who decides, and what evidence matters.

That clarity usually comes from a small number of concrete practices:

• Clear ownership boundaries across models, data access, interfaces, and outcomes
• Explicit rules for deployment, rollback, escalation, and scope expansion
• Regular review of system behavior under real usage alongside evaluation results
• Instrumentation that surfaces signals operators can act on within the available intervention time

I have found that when these practices are present, teams spend far less time debating responsibility after the fact and far more time making informed adjustments before problems escalate.

Where governance actually lives#copy

Interfaces are where authority enters the system, outputs acquire downstream meaning, and controls can intercept action. Their contracts should make permitted behavior, failure, and escalation explicit.

The operating boundary states the commitment. The agency envelope describes the delegation and controls through which it is implemented, following the paper's six dimensions. For each workflow, record and test:

1. Permissions: allowed tools, data, side effects, and acting identity. Enforce the grant at the tool or service boundary and test attempts outside it.
2. Horizon: time, action, and retry limits; stopping conditions; and points requiring renewed approval. Include timed-out and resumed runs in the test.
3. State: what may be read, retained, retrieved, or changed across runs. Specify purpose, provenance, access, retention, expiry, and the response to corrupted or stale state.
4. Delegation/concurrency: permitted subtasks, parallel paths, and aggregate budgets. Ensure delegated work inherits the allowed authority, remains traceable, and responds to cancellation.
5. Reversibility/containment: which effects can be isolated, safely retried, reversed, or reconciled. Gate high-consequence irreversible actions and test partial failure across downstream systems.
6. Oversight capacity: the evidence, staffing, control coverage, and intervention time required at the permitted rate of action. Test interruption with queued and in-flight work, including what happens if monitoring becomes unavailable.

These dimensions interact. Increasing concurrency can overwhelm review at unchanged permissions; extending a horizon can let an early state error reach more actions. Evaluate the combination at the proposed load. Use the Flywheel monitorability definition to determine whether the record supports the required intervention.

Accountability as a design choice#copy

When systems misbehave, the instinct to search for a single cause is strong.

In practice, failures emerge from interactions: between automation and judgment, between generation and verification, or between system output and downstream action.

Operational governance places responsibility before the system acts and preserves evidence for investigation afterward.

Responsibility means deciding, in advance:

• how much authority the system has,
• how outcomes will be evaluated,
• and what happens when reality diverges from expectations.

It also means being clear about who can change those parameters and under what conditions. Review effort and restrictions on flexibility belong in the cost of a control; assess them alongside the clarity and protection it provides.

Make that responsibility executable through named roles:

• The workflow owner owns acceptance criteria, outcome quality, and the decision to propose a changed commitment.
• The control owners maintain permissions, data and state controls, evaluations, and interface contracts, with authority to block changes that invalidate them.
• The response owner and backup can contain the system, coordinate recovery, and account for downstream effects.
• The business owner assesses full costs, benefits, and the capacity of affected teams to absorb the work.
• The boundary approver authorizes changed work or authority and the evidence required for re-entry after contraction.

One person may hold several roles in a small team. Record coverage, conflicts, and escalation explicitly; maintain any independent approval required by the organization for consequential actions.

Governance that evolves with capability#copy

As systems become more capable, governance has to evolve alongside them.

New work, longer action sequences, and greater exposure can invalidate assumptions that supported the earlier boundary.

Operational governance adapts by revisiting:

• trust boundaries as new workflows are introduced,
• escalation paths as autonomy expands,
• and evaluation coverage as the system’s role grows.

I have seen governance break when it stayed static while the system changed underneath it.

Require revalidation when a changed model, tool, schema, permission, or process affects a control's assumptions, even if entrusted work stays fixed. For expanded work or authority, preserve the evidence and approval in the Execution decision record. When testing whether retained learning supports expansion, establish the Helix entry conditions before the attempt.

Operator takeaways#copy

If you are responsible for operating an AI system, governance should help you answer, clearly and confidently:

• What is this system allowed to do today?
• Under what conditions does that change?
• What signals would tell us to pause, constrain, or roll back behavior?
• Who has the authority to make that call?
• What evidence would show we waited too long?

Review those answers against operating evidence. A control document earns its place when the people and mechanisms it names can act under the conditions the system encounters.