Why We Rebuilt Microsoft Office From Scratch for AI Agents with Yijia Liang, Compound AI

Institutional finance requires a fundamentally different agent architecture than consumer personal agents - one built on multiplayer collaboration (shared in

Why We Rebuilt Microsoft Office From Scratch for AI Agents

Abstract

Consumer personal agents have advanced on the strength of reliable background execution, proactivity, and integration with systems of context. This synthesis examines why that capability profile fails to transfer to institutional finance and documents an alternative architecture built for buy-side deal teams. Three design commitments are analyzed: multiplayer embedded agents exposing a per-agent inbox for shared deal-team access; a headless office engine replicating Excel, PowerPoint, and Word from scratch; and a context-management strategy combining agent swarms with citation grounding. The engineering effort totaled approximately 1.5 trillion input-output tokens and over 8 million lines of code. Reported outcomes include 4x faster recalculation of 20MB workbooks relative to desktop Excel and compression of investment committee preparation from roughly four weeks to several days. The analysis argues that institutional finance requires vertical integration down to the document engine, not merely orchestration atop existing tools.

1. Introduction

Agentic systems that gained traction in consumer and developer markets share a recognizable shape: a single user delegates a task, an agent runs in the background, and it draws context from a small, well-structured set of personal sources - email, calendar, a code repository. In institutional finance, that shape does not hold. The central question this synthesis addresses is what architecture replaces it when the unit of work is a deal team rather than an individual, and the unit of context is a 15,000-file data room rather than an inbox.

Several terms require definition at the outset. A data room is the document corpus assembled for a transaction, frequently exceeding 15,000 files and including individual files larger than 20 megabytes. An IC memo is the diligence artifact, often a 50-page presentation, prepared for an investment committee. The copilot-to-autopilot distinction separates line-by-line assistance, where a human verifies each output, from delegation of entire tasks with trusted, checked results.

The thesis advanced is that institutional finance requires a fundamentally different agent architecture than consumer personal agents: multiplayer by default at the interaction layer, vertically integrated down to the document engine rather than layered atop it, and scalable across both the breadth and depth of institutional data. Section 2 situates this claim against the recent consumer-agent inflection. Section 3 analyzes the three architectural commitments in turn. Section 4 consolidates technical findings, Section 5 discusses broader implications, and Section 6 concludes.

2. Background and Related Work

Three concurrent developments are credited with the recent inflection in personal agent adoption: background agents became reliable enough to run unsupervised, agents became proactive rather than purely reactive, and connectors to systems of context matured. Each has only a partial analog in finance. The background-task analog is clear: compressing the path from a raw data room to complete IC models, slides, and memos, a process that conventionally consumes four weeks, into a few days. Proactivity, by contrast, does not translate - a managing director does not want an agent "rogue running around doing a bunch of things and burning tokens" without first soliciting input, an authority and accountability constraint rather than a capability one. The context dimension differs not merely in degree but in kind: a firm's institutional memory spans SharePoint or Box repositories holding petabytes of unstructured material accumulated over a forty-year history.

Prior art for document manipulation was evaluated and found insufficient. Desktop plugins, computer-use virtual machines, open-source libraries such as OpenPyXL, and commercial spreadsheet engines each failed to meet the accuracy, speed, and fidelity requirements described below, motivating the from-scratch rebuild analyzed in Section 3.2.

3. Core Analysis

3.1 Multiplayer Collaboration for Deal Teams

Plugins and single-user web applications encode an implicit assumption that one person owns a task. Deal work violates this assumption: outputs must circulate among an entire deal team, and the conventional workaround - downloading files, re-uploading them to email or Teams, and manually requesting feedback - is slow and lossy. The architectural response is to give every agent a built-in inbox, so deal team members interact with the same shared agent through ordinary Outlook email threads rather than through a single-user interface.

Implementing agentic email surfaces two non-trivial design problems. The first is authentication: the system must verify that a given sender has read/write access to a deal under NDA before acting on their request. The second is UX calibration for a non-human actor operating in a human channel - for example, deliberately adding latency before replying, so that when several deal team members email in quick succession the agent does not fire off redundant acknowledgements that read as spam. Both problems are specific to multiplayer, human-in-the-loop operation and have no clear analog in single-user consumer agents.

3.2 Rebuilding Office From Scratch

Having rejected desktop plugins, computer-use VMs, and off-the-shelf libraries, the team built headless replicas of Excel, PowerPoint, and Word using coding agents. The Excel engine began from the open-source IronCalc project and was extended, using Microsoft Office itself as an oracle for correctness and benchmarking against both real-world and internally generated files, into an 800,000-line Rust codebase. Across all three applications, the total effort reached approximately 1.5 trillion input-output tokens and over 8 million lines of code.

Two properties motivate owning the engine rather than orchestrating around Office. First, performance: desktop Office is CPU-bound to a single running instance, whereas a headless engine parallelizes work, recalculating a 20MB Excel file roughly 4x faster. Second, fidelity: firms "only trust what they can download and ... send," which requires pixel-perfect rendering parity with native Office output, achieved via Rust-based renderers. Owning the engine also enables a write-render-check-fix iterative loop - the agent writes an edit, renders it, visually checks the result, and corrects discrepancies - a loop unavailable to tools like OpenPyXL, which has no visual render to check against, and unavailable within Microsoft's own, more limited programmatic interface. The same ownership supports richer asynchronous collaboration, such as building a slide deck through dropped-in formatting comments visible to the whole team.

3.3 Managing Breadth and Depth of Institutional Data

Data rooms present a breadth problem distinct from depth. For breadth - corpora exceeding 15,000 files - the chosen approach borrows from coding-agent practice: a lead agent slices the file directory and dispatches swarms of sub-agents that read files via grep-based search rather than vector-database indexing, iterating until output quality is satisfactory, with every finding grounded in citations to its source document. This design choice favors direct, auditable retrieval over semantic indexing for most use cases, trading some recall flexibility for traceability and lower infrastructure overhead.

Depth presents a separate problem: large individual workbooks, exemplified by a 30MB Excel model containing 2 million formulas and five sensitivity tables requiring five minutes to recalculate, are slow and expensive to process through code-and-openpyxl pipelines. The custom Excel engine addresses this by laying out workbook content in an easy-to-read, easy-to-process internal format, exposing richer APIs and tools to the agent than a cell-by-cell library interface allows, and thereby reducing both cost and latency for depth-heavy analysis.

4. Technical Insights

Several implementation details generalize beyond this specific system. Building on an existing open-source kernel (IronCalc) rather than starting from zero reduced the scope of the rewrite while still permitting extension to 800,000 lines of Rust. Using the incumbent commercial product (Office) as an oracle for correctness testing provided a ready-made ground truth for benchmarking a from-scratch reimplementation, a pattern applicable to other domains where a trusted legacy tool exists. The 4x recalculation speedup over desktop Excel stemmed specifically from removing the single-instance CPU bottleneck inherent to desktop applications, not from algorithmic improvements to formula evaluation alone.

On the retrieval side, the decision to favor grep-based swarm reading over vector-database indexing for most data-room queries is a notable trade-off: it sacrifices some semantic search capability in exchange for citation-level traceability, which matters disproportionately in a regulated, audit-sensitive domain. A key limitation acknowledged in the source material is scope: the rebuilt engines target the specific editing and analysis patterns of deal work, not general-purpose office productivity, and the 1.5-trillion-token, 8-million-line build cost reflects a bet that document-engine ownership is a durable moat rather than a one-time convenience.

5. Discussion

The broader implication is that vertical integration - owning the document engine, not just the orchestration layer atop it - may be a prerequisite for agentic autonomy in domains where output fidelity and auditability carry contractual or regulatory weight. This contrasts with the prevailing pattern in consumer agent development, where thin orchestration over existing APIs (email, calendar, file storage) has been sufficient. Institutional finance's requirement that firms "only trust what they can download and send" suggests that similarly document-fidelity-sensitive domains - legal, healthcare, insurance - may face the same architectural pressure toward rebuilding rather than wrapping incumbent tools.

A related open question concerns generalizability of the grep-over-vector-DB finding: whether citation-grounded swarm retrieval scales to corpora substantially larger than 15,000 files, or whether hybrid indexing becomes necessary, is not resolved in the source material. The analogy drawn to Opus 4.5's effect on coding workflows - where developers shifted from line-by-line review to delegating entire tasks once sufficient checks existed - frames the current state of finance agents as pre-inflection: well-served for narrow tasks like spreading comps, but not yet trusted for longer-running deliverables like a full IC presentation.

6. Conclusion

This synthesis documents three interlocking architectural commitments - multiplayer embedded agents, a from-scratch headless office engine, and swarm-based context management with citation grounding - developed to close the gap between consumer-grade agent capability and institutional finance requirements. The practical takeaway is that document-fidelity-sensitive, multi-stakeholder domains may not be served by orchestration alone; they may require rebuilding the underlying editing and rendering engine to achieve the speed, accuracy, and trust needed for autonomous operation. The stated endpoint - deal teams dropping in a data room and scaling deal review tenfold - remains aspirational, with current deployment concentrated in well-defined copilot tasks rather than full autopilot delegation.

Sources

• Why We Rebuilt Microsoft Office From Scratch for AI Agents - Yijia Liang, Compound AI - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub