The Spatial Harness: Bringing Agents to the Canvas - Max Drake, tldraw

The canvas is an underexplored but powerful medium for working with AI agents, offering spatial, collaborative, multiplayer primitives that make agent-human ...

By Sean Weldon

The Spatial Harness: Bringing Agents to the Canvas

Abstract

This synthesis examines the infinite canvas as an alternative medium for agentic computing, drawing on prototypes built atop the tldraw Infinite Canvas SDK. The central argument is that canvases supply spatial, multiplayer primitives - shared cursors, viewports, and selections - that render agent reasoning and multi-agent coordination more legible than chat transcripts. Evidence is drawn from five successive artifacts: a canvas-comprehension experiment (Teach), an agentic harness (Agent Starter Kit), a multi-agent avatar system (Fairies), a dependency-graph task manager (Tech Tree), and a desktop application exposing an editor server for arbitrary scripting. Findings indicate that large language models require substantial engineering scaffolding to reason in two dimensions, that dual screenshot-plus-JSON representations partially close this gap, and that sandbox boundaries remain the principal limitation on canvas-native agents. Practical implications extend to multi-agent orchestration design and real-world action grounding.

1. Introduction

Chat interfaces dominate contemporary human-agent interaction, imposing a linear, textual bottleneck on work that is frequently spatial in nature. The infinite canvas - an unbounded workspace in which content exists as positioned, resizable shapes rather than as sequential tokens - offers an alternative substrate. This synthesis considers whether such a medium can support agentic work as effectively as, or more effectively than, the chat-based paradigm.

The motivating observation concerns medium alignment: coding agents such as Claude Code succeed in part because their working medium (text) matches their training medium (text in, text out). No representational translation is required between what the model was optimized to predict and what a coding task demands it produce. This raises the central question addressed here: what happens when agents operate in a medium not aligned with their training distribution, namely a two-dimensional spatial canvas?

The thesis advanced is that despite this misalignment, the canvas remains a powerful medium for agentic work because its spatial and multiplayer primitives - originally designed for human remote collaboration - make both agent-human and agent-agent coordination more intuitive and legible than chat. The analysis proceeds by first establishing the technical foundation of the tldraw SDK and the nature of the spatial reasoning deficit in current models, then tracing five prototypes of increasing autonomy and environmental scope, before extracting transferable technical patterns and discussing broader implications.

2. Background and Related Work

tldraw exists both as a freely available infinite-canvas whiteboarding application and as a software development kit (SDK) that powers canvas experiences inside third-party products, including Replit's agent canvas. The SDK originated to solve a recurring engineering failure mode: developers building canvas applications become indefinitely occupied implementing primitives - shape resizing, selection handling, matrix transformation mathematics - rather than the application logic that differentiates their product.

Critically, the SDK ships with multiplayer and live-synchronization features as first-class concerns: cursors, selections, and viewports are shared state, propagated across all collaborators in a room. This infrastructure, designed originally for human co-presence during remote work, supplies exactly the observability substrate that multi-agent systems require, a connection made explicit in the framing that "the canvas can be a place where we collaborate with agents" much as it already is for human collaborators.

3. Core Analysis

3.1 The Spatial Reasoning Deficit

Empirical experience with canvas-resident agents reveals a pronounced weakness: "agents are really really bad at working in 2D space and understanding 2D space and actually requires a lot of engineering work to get them to do it." The canonical failure case is UI element alignment, a task trivial for humans but unreliable for models trained predominantly on text. This deficit motivated the Teach project, which taught LLMs to interpret canvas state via a dual representation - screenshots combined with JSON data describing shapes. This combination allowed a single-shot prompt ("make the mouse blow out the candle") to produce a correct wind-and-smoke animation using only primitive shapes, notably without any special-cased "mouse" object: "This is not a special mouse shape. These are just shapes on the canvas." The result demonstrates that the harder problem is not merely interpreting canvas state but understanding how actions causally affect that state.

3.2 From Single-Shot Understanding to Agentic Autonomy

The Agent Starter Kit, MIT licensed and distributed via the tldraw website, extends single-shot canvas comprehension into a full agentic harness capable of autonomous goal-setting and multi-step execution. In demonstration, an agent formed its own to-do list and altered its viewport to search the canvas for conceptual "friends" of a cat shape - a spatial analogue to codebase search performed by coding agents. This suggests that canvas navigation via viewport manipulation can substitute for the file-traversal patterns familiar from text-based agentic tooling.

3.3 Multi-Agent Coordination via Spatial Avatars

The Fairies system introduces customizable agent avatars (varied by hat, color) directly onto the canvas, addressing the practical need to distinguish multiple simultaneously active agents at a glance. Fairies can be grouped into a "chat" in which one agent acts as orchestrator: assigning tasks, waiting for peer fairies to complete them, and reviewing outputs. Because task state is rendered visibly on the canvas, participants - human or agent - are not required to parse chat logs to understand coordination status. This represents a direct application of the spatial-legibility thesis to multi-agent orchestration. A structural limitation is noted directly: "the fairies are trapped in the canvas," meaning the system currently cannot integrate agent actions with effects in the world outside the canvas sandbox.

3.4 Dependency Graphs and Real-World Escape

The Tech Tree app, inspired by tldraw's internal practice of replacing conventional task-tracking software with a dependency graph during product launches, converts each graph node into an autonomous coding-agent task. It supports multimodal input, in-canvas pull request review and merging, and multiplayer collaboration on task management, positioning it alongside tools such as Conductor and OpenAI Symphony as an abstracted interface for multi-agent coordination.

The final prototype, the tldraw desktop application, addresses the sandbox limitation directly by exposing the editor instance through a local server, enabling agents such as Claude Code to write plain JavaScript against editor state - termed "code mode." Because Claude Code retains access to the underlying computer, agents are no longer confined to the canvas: a colleague demonstration used the desktop app as a window manager, with the agent generating rectangles and corresponding AppleScript to move real application windows, and a further demonstration implemented Pong using canvas windows as game elements.

4. Technical Insights

Several transferable patterns emerge from these prototypes. First, dual representation - pairing screenshots with structured JSON shape data - appears necessary for models to reconcile visual and symbolic understanding of spatial environments; neither modality alone was sufficient in the Teach experiment. Second, single-shot spatial competence does not automatically generalize to multi-step autonomy; an explicit agentic harness (to-do formation, viewport control) was required to convert comprehension into planning. Third, the orchestrator pattern observed in Fairies - one agent assigning and reviewing, others executing - depends on shared, visible task state rather than message-passing logs, reducing the coordination overhead associated with parsing conversational history. Fourth, the boundary between sandboxed and real-world agent action is a first-order design decision: the desktop app's "code mode" demonstrates that exposing an editor server and permitting arbitrary script generation (including OS-level scripting such as AppleScript) is one viable mechanism for escaping sandbox constraints, though this was not fully realized in a live demonstration attempting to build a fluid simulation from a linked Notion document within the talk's timeframe.

5. Discussion

These findings position the canvas as a plausible complement to, rather than replacement for, chat-based agent interfaces. The persistent spatial reasoning deficit indicates that current LLMs remain fundamentally text-oriented, and canvas competence is achieved through engineering scaffolding rather than native model capability. This raises an open question about whether future model training regimes might incorporate spatial-visual objectives directly, reducing dependence on dual-representation workarounds.

The multi-agent orchestration patterns demonstrated by Fairies and Tech Tree are relevant to broader industry efforts in multi-agent systems, where legibility of coordination state is a recognized bottleneck. The comparison to Conductor and OpenAI Symphony suggests convergent interest in visual or structural abstractions over raw conversational logs for managing concurrent agent tasks.

The sandbox limitation identified in Fairies and partially addressed in the desktop application suggests that the value of canvas-native agents may ultimately depend on their capacity to act beyond the canvas itself, using spatial representations as a planning and coordination layer while executing effects in external systems.

6. Conclusion

This synthesis has traced a progression from canvas comprehension to autonomous, multi-agent, and ultimately world-affecting agentic systems built on the tldraw SDK. The core contribution is evidence that spatial, multiplayer primitives - originally designed for human collaboration - can be repurposed to render agent behavior and multi-agent coordination more legible than chat transcripts, provided sufficient engineering investment is made to compensate for models' native weakness in two-dimensional reasoning. Practically, developers building agentic tools may benefit from adopting dual screenshot-JSON representations, visible task-state artifacts for orchestration, and explicit mechanisms for bridging sandboxed and real-world action.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub