Your AI Agents Shouldn't Care Where They Run with Nikhil Harithas, Factory

Enterprises deploying autonomous coding agents need agent identity and agent compute decoupled from individual user identity and from any single infrastructu

Your AI Agents Shouldn't Care Where They Run: Decoupling Agent Identity and Compute from Infrastructure

Abstract

Enterprise deployment of autonomous coding agents surfaces two recurring architectural questions: on whose behalf does an agent act, and where does it execute? This synthesis examines how Factory addressed both questions during a deployment at Blackstone, a trillion-dollar financial institution whose regulatory posture invalidated every default product assumption. The resulting system, Open Compute, is a soon-to-be open-sourced abstraction layer that manages the compute life cycle of arbitrary agents across heterogeneous targets including EC2, Kubernetes, and third-party sandbox providers. The design introduces a dedicated compute plane alongside the conventional control and data planes, guarantees egress-only connectivity into customer environments, and decouples agent identity and compute from individual user credentials and any single infrastructure vendor. Findings indicate that compute life cycle management is better treated as independent infrastructure than as a feature bolted onto an agent product, and that enterprises will increasingly demand sovereignty over the entire agent stack.

1. Introduction

The rapid commercialization of agentic software over the past eighteen months has outpaced the maturation of its deployment substrate. Many agentic products reached market by attaching ephemeral sandboxes to an individual user's personal permissions, inheriting that user's identity for all downstream actions. This synthesis treats that pattern as a transitional expedient rather than a durable architecture, since it conflates a human's authority with an autonomous process's authority and leaves enterprises unable to audit, scope, or contain agent behavior independently of the user who happened to launch it.

Two definitions frame the analysis. Agent identity refers to the principal on whose authority an agent acts, which may or may not correspond to a human user. Agent compute refers to the execution environment in which agent processes run, including its provisioning, suspension, and destruction. The central thesis is that both concerns must be decoupled from individual user identity and from any single infrastructure provider, and that the compute concern in particular warrants its own abstraction layer rather than remaining embedded in the product that happens to use it.

The analysis proceeds in four stages: the enterprise constraint set that exposed the need for this abstraction, the identity argument underlying agent-as-principal, the compute plane architecture itself, and the life cycle sovereignty model that follows from it. A final section isolates engineering trade-offs encountered in building the system.

2. Background and Related Work

Distributed systems practice conventionally separates a control plane, which decides what should happen, from a data plane, which carries the work. Agentic systems inherit this decomposition but introduce a concern that neither plane models cleanly: the dynamic provisioning and teardown of general-purpose execution environments on a per-session basis, often spanning hours or days and crossing infrastructure boundaries.

Commercial sandbox providers such as Daytona and Modal supply managed environments for this purpose, and hyperscaler primitives including EC2 micro VMs and Kubernetes/EKS offer lower-level alternatives. A common vendor response to enterprise constraints has been a "bring your own compute" tier, in which the customer supplies infrastructure. This proved insufficient because it transferred the full compute life cycle burden to the customer without tooling to manage it, trading vendor lock-in for operational load. On the identity side, decades of software engineering practice have produced mature models for service principals, delegated authority, and scoped permissions; the position taken here is that agents require no novel identity primitives, since, as stated in the source material, "agents are just slightly non-deterministic applications and we don't need to reinvent the wheel here."

3. Core Analysis

3.1 The Enterprise Constraint Set

The Blackstone engagement is characterized as a "fractally complex enterprise" whose regulatory and security posture invalidated nearly every product default Factory offered. The customer could not run agents on laptops and required bring-your-own compute, bring-your-own inference, bring-your-own identity, and single-tenancy at every layer. Mid-engagement, a chosen sandbox provider changed key operating constraints, forcing an unplanned migration to EC2. This pattern recurred across multiple customers wanting Kubernetes, EC2, or GCP targets, indicating that no single compute backend would satisfy the enterprise market and that the product needed to move organically across all of them.

Two questions recurred across these engagements: who is an agent acting on behalf of, and where does it run? These questions proved separable from any single customer's specific infrastructure choice, which is what justified building a general abstraction rather than a point solution for Blackstone alone.

3.2 Compute Plane Architecture

Open Compute introduces a third architectural layer, the compute plane, alongside the traditional control and data planes. Its components include a compute gateway and relay in Factory's backend, an Open Compute driver/controller running inside the customer's VPC, and drivers - compute-type-specific wrappers for EC2, Kubernetes/EKS, or sandbox providers such as Daytona and Modal.

The provisioning flow proceeds as follows: a client requests a new agent session, which reaches Factory's backend compute gateway, which forwards the request to the Open Compute control plane running in the customer's VPC. The relevant driver then provisions actual compute - for example, a new EC2 instance - and that machine initiates an outbound websocket connection back to Factory's relay, which bridges client and sandbox communication. The system's defining guarantee is egress-only connectivity: "once set up, we never need to make an inward connection. It's always the customer that's making an outward connection and establishing [a] web socket connection." This directly addresses enterprise unease about inbound access into regulated environments. LLM inference can similarly be routed through BYOK (bring your own key) arrangements, such as AWS Bedrock, keeping model calls inside the customer's VPC alongside compute and data.

3.3 Life Cycle Sovereignty

Open Compute is explicitly scoped to "manage computers and not sessions," a design choice that decouples it from any Factory-specific session concept and allows customers to reuse it for other products. Core life cycle operations include pause/resume, destroy/start, suspend, snapshot, and fork. Fork is identified as increasingly important for multi-agent swarm and sub-agent delegation patterns, where a parent session must spawn independent, forkable copies of its execution state.

Customers can own snapshot storage - in S3 or their own database - and hydrate compute instances with secrets drawn from their own secret store. Templates control exact build requirements, including SHAs and container specifications, for reproducible compute. This yields a broader implication: enterprises will increasingly want sovereignty over the entire agent stack, extending beyond model choice to evals, the compute layer, and hosting, all governed on the customer's own terms rather than the vendor's.

4. Technical Insights

Several implementation-level findings are worth isolating for practitioners building similar systems.

• Three-plane architecture. Treating compute provisioning as a first-class plane, distinct from control and data planes, clarifies ownership boundaries and makes it tractable to swap compute backends without touching the product logic that consumes them.
• Egress-only connectivity as a security default. Requiring the customer environment to always initiate the connection - rather than accepting inbound traffic - removes a major objection in security reviews and should be treated as a design constraint from the outset rather than a retrofit.
• Driver abstraction. Standardizing create/suspend/snapshot/fork/destroy operations behind a driver interface allows new compute targets to be added incrementally. Factory reports generating new drivers by handing the driver spec directly to its own coding agent, Droid, which implements and self-validates against that spec - a spec-driven development loop that lowers the marginal cost of supporting a new backend.
• Session duration constraints. AWS micro VMs impose an 8-hour session cap, which conflicts with multi-day, multi-agent orchestration ("missions"). This was addressed through snapshot-and-swap techniques that rehydrate state onto a fresh instance without disrupting the session from the client's perspective - a pattern applicable to any system built on time-bounded compute primitives.
• Bookkeeping under partial failure. Dropped or unacknowledged provisioning requests required careful tracking of which sandbox belongs to which user/session to enable safe retries, a reminder that compute orchestration layers must be designed for idempotency from the start.
• Distribution before open-sourcing. During private preview, Factory versioned and shared Terraform packages so customers could stand up the system themselves, illustrating a practical interim distribution model for infrastructure not yet publicly released.

5. Discussion

The Blackstone case illustrates a more general dynamic: enterprise security and compliance requirements function as a forcing mechanism that exposes architectural debt in agentic products built primarily for individual or small-team use. Products that borrow user identity and rely on a single compute vendor work until they meet an enterprise whose constraints are incompatible with either assumption, at which point the product either generalizes or fails the engagement.

This suggests that agent infrastructure is converging toward patterns already established in distributed systems and identity management, rather than requiring new primitives. The explicit rejection of novel identity models for agents, combined with the introduction of a compute plane that mirrors existing control/data plane thinking, supports the view that agentic systems are best served by extending established infrastructure patterns rather than inventing parallel ones.

An open question is how driver standardization will hold up as the diversity of compute backends grows, particularly for GPU-bound or highly specialized sandbox environments where the create/suspend/snapshot/fork/destroy interface may need extension. Similarly, the long-term implications of customer-owned secret and snapshot storage for auditability and incident response remain unexplored in the source material and merit further investigation as more enterprises adopt this model.

6. Conclusion

This synthesis identifies agent identity and agent compute as two concerns that enterprise deployments force apart from individual user identity and single-vendor infrastructure. Factory's response, Open Compute, operationalizes this separation through a dedicated compute plane, a driver abstraction over heterogeneous compute backends, and an egress-only connectivity guarantee that addresses enterprise security objections directly.

The practical takeaway for teams building agentic products is that compute life cycle management should be designed as independent, reusable infrastructure from early on, rather than retrofitted once an enterprise customer's constraints invalidate existing assumptions. As Open Compute moves toward open source, its driver spec and life cycle model offer a reusable reference point for teams facing similar enterprise sovereignty requirements across model inference, compute, and evaluation layers.

Sources

• Your AI Agents Shouldn't Care Where They Run - Nikhil Harithas, Factory - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub