Your Sub-Agents Are a Distributed System. Treat Them Like One. with Girish Konda, Microsoft

Sub agent systems are distributed systems in disguise - they exhibit classic distributed systems failures (partial failure, dirty reads, unbounded fanout, un

By Sean Weldon

Sub Agent Systems as Distributed Systems: A Consistency Model for Orchestrator-Worker Handoffs

Abstract

Sub agent architectures - an orchestrator decomposing a task across concurrently executing worker agents - are typically exposed through an interface that accepts a string and returns a string. This synthesis, drawn from Girish Konda's analysis, argues that this interface conceals a distributed system, and that the failures practitioners encounter in production (partial failure, dirty reads, unbounded fanout, unsafe retries) are classical distributed systems failures rather than artifacts specific to AI model behavior. Established results - the FLP impossibility result, Chandra-Toueg failure detectors, SQL isolation-level critiques, weighted voting, and remote procedure call (RPC) delivery semantics - are applied to multi-agent orchestration. A consistency model for handoffs, SAFE (Source feeds, At-most-once effects, Fanout budgets, Explicit outcomes), is presented alongside empirical evidence from an MCP Pacemaker bridge implementation. Limitations are stated explicitly: SAFE constrains what can be verified and repeated but does not make claims true, guarantee exactly-once delivery, or provide rollback.

1. Introduction

A sub agent system is one in which an orchestrator splits a task and hands pieces to multiple workers that run concurrently. These workers share mutable state - conversation history, memory files, repositories, working trees, issue trackers, vector stores - and issue tool calls against real systems, including databases, APIs, continuous integration (CI) tooling, and deployment pipelines.

Every structural property of a distributed system is present in this arrangement: concurrency, shared mutable state, partial failure, and externally visible side effects. What is absent is the interface discipline normally applied to such systems. The function-call abstraction - string in, string out - hides status, provenance, and side-effect identity, forcing the orchestrator to infer system state from prose rather than read it from structured fields.

The central thesis is that sub agent systems should be engineered with explicit status, provenance, idempotency, flow control, and verified outcomes, rather than treated as simple function calls. This analysis proceeds as follows: Section 2 establishes the theoretical background drawn from distributed systems literature. Section 3 analyzes three failure classes observed in production - partial failure and detection limits, dirty reads in shared context, and the absence of a default consistency model. Section 4 synthesizes technical insights and implementation tradeoffs. Sections 5 and 6 discuss broader implications and conclude with practical takeaways.

2. Background and Related Work

Three bodies of prior work are load-bearing for this analysis. Failure detection theory establishes hard limits on what can be known about a non-responding process. The FLP impossibility result (1985) shows that in an asynchronous system with even one faulty process, agreement can fail to terminate. Chandra and Toueg (1996) formalized failure detectors and their essential property: a detector suspects failure, it does not know. No timeout mechanism escapes this limit, a point with direct bearing on how sub agent orchestrators should interpret a non-responding worker.

Consistency and isolation theory addresses the second failure class. The 1995 critique of SQL isolation levels defines a dirty read as reading a value before the writer commits or rolls back. The weighted voting quorum model (1979) shows that agreement among replicas is meaningful only when read quorum plus write quorum exceeds the total, guaranteeing overlap with the latest write - a condition sub agents polling shared context do not satisfy, since they are not independent sources. Ordering and delivery theory completes the foundation: Lamport's 1978 happened-before relation provides causal ordering without a global clock, and the 1984 RPC semantics paper establishes that a returned call ran precisely once, whereas an exception means the call ran once or not at all - with the caller left uninformed as to which. Contemporary agent protocols partially anticipate these concerns: A2A tasks can terminate as completed, failed, canceled, rejected, or request input, and MCP tool results carry an is_error flag and an idempotent hint, though the latter defaults to false and requires explicit implementation.

3. Core Analysis

3.1 Partial Failure and the Limits of Detection

Crash-stop failure - a correctly running process that stops and stays stopped - is tractable to handle. The harder case is no reply, which is indistinguishable from several underlying states: the worker is still working, the worker crashed and the reply was lost, or the call is genuinely slow. A third mode resembles Byzantine-style failure: a half-finished sub agent returns fluent, confident text that looks like success but is neither malicious nor crashed. In one documented case, three different execution paths - full analysis, pattern match, and no work performed at all - all returned the verdict "benign" with identical confidence, indistinguishable downstream. As stated in the source material, "the orchestrator can read the conclusion. It cannot read how much work was behind it."

Empirical evidence from the MCP Pacemaker implementation illustrates both directions of this ambiguity. A self-downloading server hit a 30-second timeout and appeared broken when it was merely slow; the fix was to give the initialize call its own independent timeout rather than sharing a budget with subsequent operations. The reverse case also occurred: a server crash during a call produced a timeout that overrode the exit code explaining the actual crash, destroying diagnostic information; the fix was to record a timeout only when the process was confirmed still alive. The general principle: treat a timeout as a suspicion, record it as unknown, and preserve the real cause when it is known.

3.2 Dirty Reads in Shared Context

A documented failure case shows one sub agent inferring an unmeasured 80% error-rate drop and writing that claim to shared context. Two other sub agents subsequently read and built upon it, and the orchestrator merged all three outputs as corroborating agreement. In fact, "three artifacts agree" was "one unsupported sentence read three different times." This is a dirty read by the 1995 isolation-level definition: a value read before its provenance was checked or verified. Shared context, as typically implemented, is readable the instant it is written, with nothing marking the entry unchecked or provisional. Quorum voting works in databases because read quorum plus write quorum exceeding the total guarantees overlap with the latest committed write; polling multiple sub agents against the same shared context does not satisfy this guarantee, because the agents are not independent sources - they are reading and amplifying the same unverified claim.

3.3 The Absence of a Default Consistency Model

Standard database consistency models - linearizable, sequential, causal, and session guarantees such as read-your-writes and monotonic reads, down to eventual consistency - exist on a spectrum of availability-versus-guarantee tradeoffs. A shared context used by sub agents provides none of these guarantees by default. The strongest model, linearizability, cannot remain available during network partitions; causal and session guarantees can remain available, provided each client sticks to one consistent vantage point. For sub agent architectures, that vantage point is the orchestrator itself, which motivates a purpose-built handoff model rather than reliance on ambient consistency properties.

4. Technical Insights

The proposed SAFE handoff model addresses the three failure classes with four concrete mechanisms:

A key limitation applies throughout: "restore context, not side effects" - client or agent restarts should rehydrate conversational state but never replay uncertain external effects.

5. Discussion

The broader implication of this analysis is that multi-agent orchestration failures currently attributed to model unreliability are frequently architectural gaps that distributed systems theory has addressed for decades. This reframing matters practically: engineering teams debugging "hallucination-like" agent failures may be better served applying RPC delivery semantics, quorum theory, and isolation-level analysis than adjusting prompts or increasing model capability.

A notable gap is tooling support. MCP's idempotent hint defaults to false, and A2A task states exist but require active enforcement loops that read structured status rather than text. The CAP and PACELC theorems remain directly relevant: the documented bridge implementation chose availability for cold starts but consistency (refusal over guessing) for writes, and every freshness check or refusal introduces additional latency. As multi-agent systems proliferate in production CI/CD and deployment pipelines, these tradeoffs will recur, and treating them as explicit engineering decisions - rather than implicit defaults - appears to be the primary lever available to practitioners today.

6. Conclusion

This analysis establishes that sub agent systems exhibit the full structural profile of distributed systems - concurrency, shared mutable state, partial failure, externally visible side effects - and that their common failure modes map directly onto established results in distributed systems theory. The SAFE model (Source feeds, At-most-once effects, Fanout budgets, Explicit outcomes) offers a concrete, protocol-compatible framework for engineering these handoffs explicitly rather than relying on free-text consensus.

Practically, teams building sub agent orchestration should instrument provenance on every shared fact, assign identity to every side effect, bound fanout before synchronous starts rather than after, and validate agent output in code as untrusted input. The model's explicit boundaries matter equally: it does not make claims true, guarantee exactly-once delivery, or provide rollback, and it is poorly suited to low-stakes drafting-only sub agents or peer-agent architectures lacking a single orchestrator vantage point. "If the same model can skip a check and declare it passed, you have a request, not a boundary" - a conclusion that locates the fix for agent reliability in system design rather than model behavior.


Sources


About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub

Sean Weldon builds apps and custom software at 22Atlas.

Start your build

More about Sean