Connect AI to Billions of Legal Documents - Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora
Legora's evolution of search infrastructure - from Elasticsearch to Postgres to Turbopuffer - shows that an object-storage-native, memory-hierarchy-aware databas...
By Sean WeldonConnect AI to Billions of Legal Documents: An Analysis of Legora's Search Infrastructure Evolution
Abstract
This synthesis examines the evolution of retrieval infrastructure at Legora, a collaborative artificial intelligence platform for legal work, and the architectural properties of Turbopuffer, the object-storage-native search database adopted to support it. Three successive architectures are analyzed: a multi-tenant Elasticsearch deployment, a consolidated Postgres system using pgvector with DiskANN indexing, and a namespace-per-project deployment on Turbopuffer. Evidence indicates that partition-based bin-packing in Postgres produced cache thrashing that degraded P99 latency from 100 milliseconds to 20 seconds, while migration to an object-storage-native design at approximately 400 million documents yielded order-of-magnitude improvements in median latency and larger gains at the tail. The analysis further characterizes the memory-hierarchy-aware design enabling this behavior, including clustering-based vector search, hashmap-structured full-text search, and namespace-level isolation supporting data residency and customer-managed encryption keys. Practical implications for infrastructure teams building compliant, large-scale retrieval systems are discussed.
1. Introduction
Retrieval infrastructure has become a determinant of output quality in applied AI systems, particularly in domains where reasoning is grounded in large, heterogeneous document corpora. Legal work exemplifies this condition: contract review, contract creation, and legal research all depend on locating precise passages within corpora that may span billions of units of text. Errors or latency in retrieval propagate directly into the reliability of downstream legal reasoning, making the underlying database architecture a first-order product concern rather than a purely operational one.
Legora serves law firms and in-house legal teams with two distinct retrieval workloads. Project search operates within the bounds of a single matter or project, ranging from tens of documents to several million. Legal research performs deep search across statutes, case law, and regulations, and is scaling toward 10 billion vectors with high and spiky query-per-second (QPS) load driven by query fan-out. These workloads differ not only in scale but in access distribution, isolation requirements, and latency tolerance.
The central thesis advanced here is that an object-storage-native database - one that treats object storage as the system of record and manages DRAM and NVMe SSD as caches within an explicit memory hierarchy - resolves the joint scaling, cost, and regulatory constraints that defeated both cluster-per-region and partition-based relational approaches. Sections 2 and 3 trace the architectural history and its failure modes; Section 4 examines the algorithmic and systems-level mechanisms; Sections 5 and 6 discuss implications and limitations.
2. Background and Related Work
Three technical traditions inform this analysis. The first is BM25, the probabilistic relevance function underpinning classical lexical retrieval and implemented natively in inverted-index engines such as Elasticsearch. The second is approximate nearest neighbour (ANN) search, represented here by two families: graph-based methods, exemplified by DiskANN as exposed through the pgvector extension, and clustering-based methods that organize the vector space hierarchically. The third is memory hierarchy optimization, the discipline of placing data across registers, DRAM, local NVMe SSD, and remote object storage according to access frequency and cost.
Regulatory frameworks constitute an equally important constraint. Data residency obligations require that tenant data remain within specified geographies, while enterprise clients increasingly demand full physical isolation and customer-managed encryption keys (CMEK), in which the tenant rather than the vendor controls the key material protecting their data. These requirements interact directly with database architecture, because the granularity at which a system can assign storage location and encryption key determines the operational cost of compliance.
3. Core Analysis
3.1 The Elasticsearch and Postgres Eras
Legora's initial deployment used a single Elasticsearch cluster serving all tenants and workloads. This design was operationally simple but failed to satisfy regional data residency requirements, prompting a migration to region-specific clusters (US, EU, Asia Pacific). Enterprise clients subsequently demanded full physical isolation and CMEK, requirements that a cluster-per-region topology could not efficiently satisfy without proliferating clusters per tenant.
To consolidate with existing OLTP infrastructure, Legora migrated to Postgres, using pgvector with DiskANN for vector search and ts_vector for lexical matching rather than true BM25. To manage multi-tenancy at scale, the team implemented aggressive partitioning - 4,000 partitions bin-packed by project key. This approach failed under production load: hot and cold projects collided within the same partitions, producing cache thrashing. P99 latency spiked from 100 milliseconds to 20 seconds as scale increased, indicating that the partitioning scheme did not preserve locality between frequently and infrequently accessed data.
3.2 Migration to Turbopuffer and Namespace-Based Isolation
At approximately 400 million documents, Legora migrated to Turbopuffer, adopting a one-namespace-per-project model. This architectural shift yielded real BM25 scoring, improved relevancy, lower latency, and lower cost, alongside simplified single-cluster operations. Post-migration, median latencies improved by an order of magnitude, with P99 improving even further - directly reversing the tail-latency pathology observed in the Postgres deployment.
The namespace functions as the atomic unit of isolation in Turbopuffer, permitting per-namespace encryption keys and bucket placement. This granularity allows data residency and CMEK requirements to be satisfied natively, without requiring separate physical clusters per tenant - a capability that directly addresses the regulatory pressures that drove the earlier Elasticsearch fragmentation. Legora reports managing 70 to 200-plus tenants under this model without maintaining isolated infrastructure per client.
3.3 Memory Hierarchy and Legal Research at Scale
For legal research, corpus size is approaching 10 billion vectors with spiky QPS from query fan-out, and retrieval must accommodate hierarchical jurisdiction structures (city, county, state, federal), temporal validity (overruled decisions), and exemption logic. Turbopuffer maps jurisdictions to namespaces, allowing frequently queried jurisdictions (e.g., EU law) to remain cached while infrequently queried ones (e.g., Danish law) persist cheaply on object storage. A 500-millisecond latency for cold blob fetches is deemed acceptable given the deep-research nature of these queries, illustrating a deliberate trade-off between cost and latency tolerance calibrated to workload characteristics.
4. Technical Insights
Turbopuffer's design is described as fundamentally about "puffing" data across memory hierarchy levels - DRAM, NVMe SSD, and object storage - based on access frequency. Several implementation details merit attention:
Write path: Writes go directly to object storage (e.g., S3) with no disk replication or Paxos consensus, using a write-ahead log while vector, text, and columnar indexes build asynchronously. This is characterized as "the cheapest way that you can run a database period," though it implies eventual rather than immediate index consistency.
Query path: Queries check node affinity, then memory cache, then NVMe SSD cache, then object storage, with design emphasis on minimizing round trips - ideally around three - since S3 P99 latency for a 1MB blob is approximately 200 milliseconds. Excessive round trips directly translate into user-facing latency at this baseline cost.
Vector search: Implemented via hierarchical clustering (clusters of clusters) forming a tree structure analogous to a complex B-tree, rather than a navigable graph. Graph-based approaches such as
DiskANNstruggle on object storage because their random access patterns require many round trips; clustering-based tree search aligns better with the memory hierarchy, since root centroids can be cached in DRAM while leaf data resides on SSD.Full-text search: Implemented as a hashmap - tokens as keys, document ID sets as values - intersected to compute BM25-style scores. This requires multiple round trips (dictionary segments, then posting lists) and careful attention to memory bandwidth during compression and intersection. Notably, text search at web scale is described as more computationally expensive than vector search, a counterintuitive finding given the typical emphasis on vector search complexity in AI system design.
Encryption trade-off: Legora disabled the disk cache due to encryption requirements on volatile memory, yet performance remained strong relying on memory cache alone, suggesting that DRAM caching of hot data can compensate for the loss of an intermediate SSD tier under specific compliance constraints.
5. Discussion
The Legora case illustrates a broader pattern in retrieval infrastructure: architectures optimized for single-tenant or homogeneous access patterns tend to fail as multi-tenancy and regulatory granularity requirements increase. The Postgres partitioning failure is instructive because it was not a failure of the underlying index algorithms but of the locality assumptions embedded in the partitioning strategy - hot and cold data sharing physical resources degraded cache behavior at the system level, independent of query correctness.
The namespace-as-isolation-unit model addresses a structural mismatch between compliance requirements (per-tenant encryption and residency) and traditional cluster-based deployment (per-region or per-tenant clusters), suggesting that isolation granularity should be a first-class architectural decision rather than an operational afterthought. This has implications beyond legal technology for any regulated, multi-tenant AI application.
An open question concerns the generalizability of clustering-based vector search over graph-based methods specifically for object-storage-backed systems; the source material suggests this trade-off but does not quantify recall differences relative to DiskANN-based approaches at comparable scale. Further empirical comparison would clarify whether this represents a universal architectural principle or one specific to Turbopuffer's implementation.
6. Conclusion
Legora's infrastructure evolution demonstrates that object-storage-native, memory-hierarchy-aware database design can jointly resolve scaling, cost, and regulatory constraints that defeat both cluster-based and partition-based alternatives. The transition from Elasticsearch to Postgres to Turbopuffer reduced P99 latency from 20 seconds back to sub-second ranges while simultaneously simplifying multi-tenant compliance through namespace-level isolation. Practically, teams building large-scale, regulated retrieval systems should evaluate isolation granularity, memory hierarchy placement, and round-trip minimization as primary architectural criteria, rather than treating them as downstream optimizations after index selection.
Sources
- Connect AI to Billions of Legal Documents - Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora - Original Creator (YouTube)
- Analysis and summary by Sean Weldon using AI-assisted research tools
About the Author
Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.