Why Are We Still Governing AI Agents Like Software? with Tim Tyler, Gravitee

As enterprises deploy AI agents as a new "digital workforce," traditional software governance is insufficient; organizations must apply century-old principle

Why Are We Still Governing AI Agents Like Software?

Abstract

Enterprise deployment of autonomous AI agents has outpaced the governance models available to control them. This synthesis examines the proposition that agents constitute a digital workforce rather than conventional software, and that the appropriate governance substrate is therefore human workforce management practice rather than application security alone. The Agent Accountability Framework is presented as a three-stage model - assignment, alignment, and administration - comprising underlying principles and auditable controls. Key contributions include a non-expansive authority propagation rule for delegation chains, a two-tier policy model separating deterministic rules from probabilistic intents, a confidence-threshold mapping from probabilistic judgment to deterministic recourse, and a division of oversight labor between automated guardian agents and human sponsors. The practical implication is that oversight must operate at machine speed, since periodic human review cadences lag agent decision volume by orders of magnitude.

1. Introduction

Enterprises are deploying autonomous agents in production without the organizational infrastructure that would be mandatory for any human employee performing equivalent work: no unique identifier, shared credentials, no designated supervisor, continuous operation, and no mechanism for accountability when outcomes are poor. No organization would extend such terms to a human hire, yet this describes a substantial share of current agent deployments.

This governance gap originates in a categorization failure. Leaders responsible for AI initiatives report being routed between organizational functions - engineering declines ownership on the grounds that an agent "is not really a software application," while human resources declines on the grounds that it is. The entity falls outside both the software development lifecycle and the employment lifecycle, inheriting the controls of neither.

The central thesis is that large language model (LLM)-driven agents should be governed as workforce rather than as software artifacts. Organizations have spent roughly a century developing mechanisms for directing entities that act with partial autonomy, exercise judgment, make errors, and require supervision: identity, reporting lines, job descriptions, performance review, and progressive discipline. The analysis proceeds through the three stages of the resulting framework - assignment, alignment, and administration - extracts the technical controls each implies, and discusses the operational model those controls produce.

2. Background and Related Work

The framework draws on three distinct intellectual traditions. From human workforce governance it inherits the sequence of onboarding, role definition, delegated authority, performance measurement, and escalation. From information security it inherits zero trust, applied not to network position but to agent authority: an agent begins heavily restricted and accumulates latitude only through repeatedly demonstrated success. From authorization engineering it inherits fine-grained access (FGA) policies, which supply the deterministic enforcement floor beneath otherwise probabilistic agent behavior.

The framework also assumes a pre-existing alignment deficit rather than a solvable one. Frontier model developers have publicly stated that their models exhibit misalignment. Any governance model that treats agent alignment as a property fully configurable by the deploying organization, rather than as a condition partially inherited and partially outside organizational control, is built on an unsound premise.

3. Core Analysis

3.1 Assignment: Establishing Organizational Presence

Assignment answers the question "who are you?" and establishes four properties. Identity requires each agent to hold exactly one unique identifier in a known, shared registry that functions as the single point of truth to which authorization, logging, cost attribution, and performance measurement all refer. Team membership requires agents to receive and provide context on goals, decisions, and history, positioning them as teammates rather than digital clones of the human on whose behalf they act, even when the "team" consists of one human and one agent.

Sponsorship assigns each agent a named human accountable for it - analogous to a pool safety convention in which an adult states "I have the kids." The sponsor defines purpose, owns cost and spend, and answers for the agent if outcomes go wrong, producing a traceable line of accountability that integrates into the existing org chart. Role supplies purpose, responsibilities, and budget, functioning as a job description, since vague expectations - occasionally tolerable for a new human hire - do not function for agents. Role is split into two layers: a static onboarding scope set once and changed rarely, and a dynamic runtime "mission" describing moment-to-moment activity, such as an individual tool call. The agent may define its own mission, but the sponsor always defines the role. Finally, authority grants the agent explicit, sponsor-approved access to specific data and tools, sometimes mediated by IT or the teams owning the relevant systems.

3.2 Alignment: Scope and Authority Boundaries

Alignment governs how authority behaves once agents delegate to one another. The framework specifies that delegated authority always resolves to the lowest common scope: a sub-agent can never exceed the access scope of the party delegating to it, and any data returned up a delegation chain is constrained to the intersection of access shared by all parties in that chain. In the framework's illustrative case, an agent holding access to items A-D delegates to a second agent holding access to D-F; the second agent may operate across D, E, and F, but can return only D up the chain, since D is the only item common to both.

A second principle is that alignment must never be assumed. Agents inherit alignment from sources outside the sponsor's control: the underlying model's own built-in alignment characteristics (which frontier labs have acknowledged as imperfect), context supplied by other parts of the team, and third-party agents, which should be treated like contractors requiring their own identity and authorization rather than implicit trust. Sponsors can govern only the alignment they directly define for their own agent; inherited alignment is a boundary condition, not a controllable variable.

The framework distinguishes two categories of alignment rule. Deterministic rules form a hard floor - fine-grained access policies that are either satisfied or violated. Probabilistic intents are goals that must be measured and judged rather than mechanically enforced. Together, the sponsor-defined set of rules and intents constitutes the enforceable contract governing the agent.

3.3 Administration: Oversight, Scorecards, and Recourse

Administration addresses a distinction the framework treats as central: behaving within one's authority is not equivalent to succeeding at one's role, and the two must be measured separately. Oversight is therefore structured as a scorecard rather than a single metric, combining deterministic rule-checks tied to authority (for example, a hard constraint such as never approving invoices above a fixed dollar threshold) with judged, confidence-scored evaluations of intent tied to role (for example, invoice match accuracy or reconciliation cycle time). Rules are checked; intents are judged.

Because agents operate at a volume and velocity no human review cycle can match, the framework calls for continuous, automated oversight performed by "guardian agents" monitoring the underlying model, the context supplied to it, and its output - in contrast to quarterly human review, which would arrive tens of thousands to millions of decisions too late. Recourse mirrors human management escalation - correction, authority restriction, then suspension or termination - but is calibrated to confidence levels rather than applied uniformly: in the framework's example thresholds, greater than 90% confidence permits automated redaction of out-of-alignment content, below 50% confidence triggers escalation to a human, and below 20% confidence results in the action being blocked outright. This converts probabilistic judgment into deterministic, actionable steps. The framework recommends a zero-trust starting posture - heavier initial restriction, loosened over time as success is demonstrated - with labor divided between automated guardian agents acting as enforcement ("cop") and the human sponsor acting as a "coach" who adjusts rules and authority based on accumulated judgment.

4. Technical Insights

Several implementation-relevant patterns emerge from the framework. The non-expansive delegation rule - authority and returned data shrink to the intersection of all parties' access along a chain - provides a concrete, auditable constraint that authorization systems can enforce mechanically, independent of any single agent's stated intent. The two-tier policy model separating FGA-style deterministic rules from probabilistic intent scoring clarifies which violations should be treated as binary failures versus which require confidence-weighted judgment, a distinction with direct implications for how alerting and automated remediation pipelines are designed.

The confidence-threshold recourse model (redact above 90%, escalate below 50%, block below 20%, in the framework's example) is notable as a general pattern for converting probabilistic model outputs into deterministic operational actions, though the specific thresholds are illustrative rather than empirically derived and would require calibration against an organization's actual risk tolerance and false-positive costs. The split between static onboarding scope and dynamic runtime mission offers a practical schema for separating slow-changing authorization decisions (which can bear human review latency) from fast-changing task execution (which cannot). A limitation worth noting is that the framework does not specify how guardian agents themselves are governed, authorized, or audited - an infinite-regress concern common to any oversight-of-oversight architecture.

5. Discussion

The framework's central move - treating agents as workforce rather than software - has implications beyond the specific controls it proposes. It reframes AI governance debates currently dominated by model-level alignment research toward organizational and procedural controls that do not depend on solving alignment at the model layer. This is consistent with the framework's explicit premise that model alignment is partially unknowable and must be treated as an inherited boundary condition rather than a configurable parameter.

A notable gap is empirical validation: the framework is presented as a set of principles and auditable controls, but the source material does not report deployment data, incident rates, or comparative outcomes against software-centric governance baselines. The confidence thresholds offered as examples (90%/50%/20%) are illustrative rather than derived from a validated calibration methodology, and organizations adopting this model would need to establish their own thresholds against measured false-positive and false-negative costs.

The framework also surfaces a tension relevant to current industry trends toward multi-agent and agent-to-agent systems: as delegation chains lengthen, the non-expansive authority rule becomes increasingly important as a safety property, but also increasingly difficult to audit without tooling that can trace scope intersection across many hops in real time.

6. Conclusion

The Agent Accountability Framework's core contribution is a structural argument: agents behave more like a workforce than like software, and governance should therefore borrow from a century of human workforce management rather than from application security conventions alone. Its three stages - name every agent (assignment), bound every action (alignment), trace every outcome (administration) - translate into concrete controls: unique identity in a shared registry, named sponsorship, scoped and non-expansive delegated authority, dual deterministic/probabilistic policy enforcement, and confidence-calibrated automated recourse.

The practical takeaway for organizations currently deploying agents without these controls is that identity assignment is the lowest-cost, highest-leverage starting point. Subsequent work - defining sponsorship, authority scope, and the oversight scorecard - builds on that foundation but cannot substitute for it.

Sources

• Why Are We Still Governing AI Agents Like Software? - Tim Tyler, Gravitee - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub