Lessons from Generating 12 Trillion Synthetic Tokens - Bogdan Gaza, DatologyAI

Scaling synthetic data to trillion-token pre-training requires rethinking infrastructure - merging research and production pipelines across Kubernetes, Spark, ...

Lessons from Generating 12 Trillion Synthetic Tokens: A Systems Engineering Analysis

Abstract

This synthesis examines a production case study in trillion-token synthetic data generation for language model pre-training, drawing on the experience of DatologyAI scaling output from approximately 30 billion to 12 trillion tokens within one year. The central thesis is that synthetic data generation at this magnitude is fundamentally a systems engineering challenge rather than a prompting challenge: capturing the gains of a seed-document-grounded rephrasing recipe (Beyond Web) required unifying previously disconnected research and production infrastructure across Kubernetes, Spark, Ray, and batch inference frameworks. Documented interventions include batched object-store metadata retrieval, checkpointed generation jobs with idempotent recovery, dedicated per-workload resource pools, and inference hyperparameter sweeps. These changes reduced a 9-11 day metadata operation to roughly two hours and yielded approximately 40% throughput gains. The findings have direct implications for any organization attempting to operationalize synthetic data at scale.

1. Introduction

Neural scaling laws establish that exponential increases in compute and training data produce only linear improvements in model quality. This relationship, combined with a finite supply of high-quality natural text, produces what practitioners term a data wall: the usable public internet contains an estimated 30 trillion tokens, a figure that becomes a binding constraint as frontier training runs approach it. Comparable multimodal corpora, such as DataComp, exhibit similar scarcity at roughly 12.8 billion image-text pairs.

Synthetic data - text generated by existing models for the purpose of training subsequent models - is one proposed route past this constraint, provided the generation process introduces genuine information rather than recycling the generator's own distribution. However, the practical challenge of producing synthetic data at trillion-token scale is distinct from the conceptual question of whether synthetic data is useful. This synthesis argues that the dominant bottleneck is infrastructural: research pipelines built for small-scale experimentation on isolated compute (e.g., Slurm clusters) do not transfer to production-scale generation without substantial re-architecture.

This analysis covers the Beyond Web synthetic data recipe and its empirical performance, the evolution of infrastructure from a two-track research/production split to unified Kubernetes-based orchestration, and four specific engineering bottlenecks encountered at scale - metadata fetching, GPU instability, cross-cluster scheduling, and inference efficiency - along with their resolutions.

2. Background and Related Work

The operating metaphor for this work is industrial refinement: synthetic data generation is framed as a transformation process applied to raw inputs, described as functioning like "the oil refinery for your pre-training and mid-training data sets." This framing is deliberately contrasted with naive generative approaches, in which a model is prompted directly for training examples without any grounding document. The limitation of direct prompting is distributional in nature - prompting a model for a data point causes it to "just learn the modes of the distribution the data was originally trained on," collapsing output diversity toward high-probability regions of the generator's own training set.

Several public synthetic corpora provide comparative reference points, including Nemotron-Synth, Cosmopedia, QuRating-derived sets, and RedPajama. These serve as baselines against which the token-efficiency of the Beyond Web recipe is evaluated in Section 3.

3. Core Analysis

3.1 The Beyond Web Recipe: Grounded Rephrasing

The Beyond Web recipe generates synthetic documents by rephrasing and reframing existing high-quality documents rather than prompting a model from a blank context. By grounding generation in seed documents, the recipe avoids the mode-collapse problem associated with pure prompting while still introducing variation absent from the original text. Empirically, Beyond Web enables a 3B parameter model to match the benchmark performance of an 8B model across evaluations including Neotron Synth, Cosmopedia, QuARP, and RedPajama. Relative to Nemotron-Synth and Cosmopedia specifically, Beyond Web achieves matching performance using 2.7x and 5.3x fewer tokens, respectively - indicating that the limiting factor for quality is not raw token volume but the information density of the generation method.

3.2 Infrastructure Evolution: From Two-Track to Unified Orchestration

The initial infrastructure ran synthetic data research on a separate Slurm cluster, disconnected from the Kubernetes-based production pipeline. This two-track arrangement produced slow, error-prone workflows and chronically underutilized GPUs, since research and production workloads could not share scheduling or compute resources. The resolution was a unified orchestration layer in which a control cluster, a Spark cluster, and an H100 GPU cluster are all coordinated through Kubernetes. H100 clusters are provisioned via AWS Hyperpod (EKS on Hyperpod) on a shared network spine, and an internal job scheduler was built to coordinate Spark/Ray CPU jobs alongside GPU jobs. This consolidation allows a full workflow - curate, generate synthetic data, train, evaluate - to execute as a single chained pipeline rather than as disconnected manual stages.

3.3 Engineering Bottlenecks at Trillion-Token Scale

Four specific bottlenecks emerged as generation volume scaled, each with a corresponding infrastructural solution.

Metadata retrieval. Fetching S3 object metadata across millions of partitions originally required 9-11 days, driven by per-object request rate limits and slow single-object requests. Batching list requests - submitting up to 1,000 objects per S3 list call - reduced this operation to approximately two hours, a reduction of roughly two orders of magnitude.

GPU instability. Long-running rephrasing jobs are vulnerable to node failures, which previously caused lost progress, wasted compute, and partial outputs. The solution combines checkpointing of job progress with right-sizing of partition sizes, limiting the compute lost on any single node failure. Because the pipeline maintains a one-to-one input-output partition mapping, oversized partitions risk substantial time loss if a node fails near job completion; smaller partitions bound this risk.

Cross-cluster orchestration. Scheduling CPU-bound Ray/Spark jobs alongside GPU-bound H100 jobs across multiple clusters constitutes a bin-packing problem, with risk of scheduling deadlock when workloads compete for shared resources. The adopted solution assigns dedicated resource pools per job type - separate pools for Spark, Ray head nodes, and Ray worker nodes - enabling atomic scheduling within each pool rather than contention across a shared pool.

Inference efficiency. Throughput of the VLM/SGLang batch inference layer was improved through systematic hyperparameter tuning, specifically grid search over batch size and speculative decoding configuration, yielding approximately 40% throughput gains without architectural changes.

4. Technical Insights

Several implementation-level findings generalize beyond this specific pipeline:

• Batch API calls for object-store metadata. Single-object metadata requests do not scale past a few million partitions; batching requests up to the API-supported limit (1,000 objects per call for S3) is a low-effort, high-impact intervention.
• Checkpointing with idempotent file overrides. Enabling seamless recovery from GPU or node failures requires that retried work be safely re-executable without duplicating or corrupting output - idempotency is a precondition for checkpointing to be useful at scale.
• Partition sizing as a risk-management decision. Because partition mapping is one-to-one between input and output, partition size directly determines the worst-case compute loss from a late-stage node failure; this is a tunable trade-off against job overhead.
• Resource pool isolation prevents deadlock. Shared resource pools across heterogeneous job types (CPU orchestration vs. GPU inference) risk scheduling deadlock; dedicated pools per job type trade some utilization flexibility for scheduling reliability.
• Inference hyperparameters require sweeping at the target scale. Default batch size and speculative decoding settings are not necessarily optimal; empirical sweeps produced material (~40%) throughput improvements, suggesting this tuning step should not be skipped when inference cost dominates job runtime.

5. Discussion

The broader implication of this case study is that the marginal return on synthetic data research is increasingly gated by infrastructure rather than by recipe design. The Beyond Web recipe itself - grounded rephrasing rather than blank prompting - is conceptually straightforward; the harder problem documented here is operationalizing it at a volume (12 trillion tokens) that makes naive infrastructure choices (unbatched metadata calls, non-checkpointed jobs, shared scheduling pools) prohibitively slow or unreliable. This suggests that organizations evaluating synthetic data strategies should budget engineering effort for orchestration and fault tolerance comparable to the effort spent on recipe design.

A remaining gap is the extent to which these infrastructural lessons generalize across cloud providers and inference stacks; the solutions described are specific to AWS Hyperpod/EKS and SGLang, and the magnitude of gains from analogous interventions elsewhere is not established. Additionally, the synthesis does not report ablations isolating the contribution of each infrastructural fix to overall token throughput, making it difficult to prioritize among them for a team with constrained engineering resources.

6. Conclusion

This analysis demonstrates that scaling synthetic pre-training data generation from tens of billions to trillions of tokens required reconstructing the underlying infrastructure as much as refining the data recipe. Key interventions - batched metadata retrieval, checkpointed and idempotent generation jobs, dedicated resource pools, and inference hyperparameter sweeps - collectively transformed a slow, fragile research pipeline into a production system capable of generating 12 trillion tokens (7 trillion web, 5 trillion math/code) within a year. For practitioners, the practical takeaway is that synthetic data pipelines should be designed from the outset with production-scale failure modes in mind, since the engineering cost of retrofitting fault tolerance and cross-cluster orchestration at scale is substantial.

Sources

• Lessons from Generating 12 Trillion Synthetic Tokens - Bogdan Gaza, DatologyAI - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub