'From Tokens to Cells: Foundation Models for Single-Cell Biology - Akram Baharlouei, Altos Labs'

Foundation models for single-cell biology face significant challenges in data quality and model architecture; flow matching-based approaches show promise ove...

By Sean Weldon

From Tokens to Cells: Architectural Considerations for Single-Cell Foundation Models

Abstract

Foundation models for single-cell biology represent a critical frontier in computational biology, with applications spanning cellular reprogramming, drug development, and digital twin technologies. This analysis examines the fundamental architectural tension between transformer-based compression approaches and flow matching-based distribution modeling in single-cell foundation models. Evidence from recent benchmarking studies reveals that transformer architectures, despite massive computational investments, suffer from significant information loss during latent vector compression and frequently underperform simple linear baselines. In contrast, flow matching models such as PrimeFlow demonstrate superior performance by preserving distributional characteristics of cellular states. Critical challenges in data quality, measurement heterogeneity, and the balance between scaling and biological realism are identified. These findings suggest that architectural choices emphasizing distribution matching over compression, combined with improved measurement methodologies across multiple modalities, offer more promising paths toward clinically relevant single-cell models capable of accelerating therapeutic development.

1. Introduction

The field of single-cell biology underwent a paradigm shift following the 2006 discovery of Yamanaka factors - four transcription factors (Oct4, Sox2, Klf4, c-Myc) capable of reprogramming aged somatic cells to embryonic stem cell states. This breakthrough demonstrated that cellular identity is not immutably fixed but programmable, establishing the foundation for cellular rejuvenation therapies. Notably, partial programming techniques can modulate cellular age without altering cell type, potentially restoring youthful function and addressing age-related pathologies. The first reprogramming medicine utilizing these factors is scheduled to enter human trials in 2026, marking a twenty-year trajectory from initial discovery to clinical application.

The development of foundation models for single-cell data aims to accelerate therapeutic translation by enabling comprehensive modeling across biological scales. This hierarchical approach progresses from individual cells to tissues, organs, and ultimately complete organismal systems, supporting the vision of virtual cells, virtual tissues, and digital twins - computational representations capable of simulating biological responses to interventions. Such capabilities are particularly urgent given the current state of pharmaceutical development: drug acceptance rates remain below 5%, development timelines exceed ten years, and costs reach billions of dollars per approved therapy. AI-driven single-cell models promise to reduce these development cycles across the entire pipeline.

This synthesis examines the engineering and methodological challenges confronting single-cell foundation models, with particular emphasis on the comparative performance of transformer-based versus flow matching-based architectures. The analysis reveals fundamental limitations in compression-based approaches and identifies distribution matching as a more promising paradigm for capturing cellular state complexity.

2. Background and Related Work

2.1 Single-Cell Measurement Landscape

Single-cell analysis encompasses multiple measurement modalities, each capturing distinct aspects of cellular state. The genome (DNA sequence) remains nearly identical across cells within an organism, providing limited information about functional differences. RNA sequencing (RNA-seq), or transcriptomics, measures gene expression profiles as a matrix of dimensions cells × genes, where the gene dimension typically encompasses approximately 20,000 genes. This modality reveals which genes are actively expressed and their relative abundance, providing a functional snapshot of cellular state at the time of measurement.

Proteomics measures protein abundance and function, representing the most biologically relevant layer since proteins execute cellular functions. However, proteomic measurements remain technically challenging and low-throughput, limiting their scalability. Morphological imaging captures cellular shape and spatial organization within tissues, providing complementary structural information useful for understanding tissue architecture. Among these modalities, RNA-seq dominates foundation model training due to its scalability through polymerase chain reaction (PCR) technologies, enabling datasets ranging from tens of millions to one billion cells.

2.2 Existing Modeling Paradigms

Current single-cell foundation models predominantly adopt transformer-based architectures inspired by natural language processing. These models treat cells as sentences and genes as tokens, applying BERT-style masking to predict gene expression counts. The architecture employs bidirectional attention across genes to capture gene-gene relationships and compresses single-cell data into latent vectors for downstream tasks including cell type prediction and perturbation response modeling. Alternative approaches include autoencoder-based models such as CPADR and emerging flow matching models like PrimeFlow, which represent fundamentally different strategies for capturing cellular state distributions.

3. Core Analysis

3.1 Data Quality and Heterogeneity Challenges

A fundamental challenge in single-cell modeling stems from measurement variability and biological dynamics. Identical cells measured twice produce different RNA-seq readings due to both biological and technical noise sources. Biological heterogeneity arises because cells are living organisms undergoing continuous changes, including growth cycles, RNA copying, and cell division with burst-like expression patterns. Current sequencing technologies capture only snapshots from what is effectively a continuous temporal process, analogous to extracting individual frames from a motion picture rather than recording the complete temporal dynamics.

Technical heterogeneity compounds these challenges, as different laboratories and sequencing platforms produce varying measurements for identical samples. This multi-source variability creates high-dimensional, multi-state data where quality considerations become as critical as scale. The heterogeneity problem is particularly acute given that foundation models must generalize across diverse experimental conditions, tissue types, and perturbation contexts. Consequently, naive scaling of existing measurement approaches without addressing these fundamental quality issues may yield diminishing returns.

3.2 Transformer Architecture Limitations

Transformer-based models for single-cell data exhibit systematic performance limitations despite substantial computational investments. These architectures compress cellular information into fixed-dimensional latent vectors, introducing unavoidable information loss that prevents accurate reconstruction of original data distributions. Benchmarking studies presented at NeurIPS, examining both multimodal imaging combined with RNA-seq and perturbation response prediction, demonstrate that transformer models frequently underperform compared to domain-specific approaches and even simple linear baselines.

The compression bottleneck proves particularly problematic for capturing the distributional characteristics of cellular states. When evaluated against ground truth data, transformer-based models and autoencoders such as CPADR successfully predict mean values but fail to capture the shape and variance of underlying distributions. This limitation is critical because cellular responses to perturbations or developmental signals are inherently distributional rather than deterministic - individual cells within a population exhibit variable responses that must be modeled accurately to predict tissue-level or organism-level outcomes.

Furthermore, the computational expense of training large transformer models on single-cell data is not justified by performance gains. Simple linear models often match or exceed the predictive accuracy of complex transformer architectures on standard benchmarking tasks, raising fundamental questions about the appropriateness of compression-based approaches for this domain.

3.3 Flow Matching as an Alternative Paradigm

Flow matching-based models represent a fundamentally different approach that addresses the distributional modeling limitations of compression-based architectures. Rather than encoding cellular states into fixed latent vectors, flow matching models predict the complete data distribution by learning transformations from a Gaussian noise distribution to the target cellular state distribution. The PrimeFlow model exemplifies this approach, demonstrating superior performance compared to both autoregressive and autoencoder-based alternatives.

Ground truth validation studies reveal the critical advantage of flow matching: while compression-based models predict only mean values, PrimeFlow accurately captures distributional shape, variance, and higher-order statistical properties. Performance is quantified using Maximum Mean Discrepancy (MMD) scores, which measure the quality of distribution matching. The superior MMD scores achieved by flow matching models reflect their ability to preserve the complex, multi-modal distributions characteristic of cellular populations responding to perturbations or developmental signals.

This architectural distinction has profound implications for downstream applications. Accurate distribution modeling is essential for predicting population-level cellular responses, identifying rare cell states, and simulating the stochastic nature of biological processes. The success of flow matching suggests that future single-cell foundation models should prioritize distributional fidelity over compression efficiency.

3.4 Scaling Considerations and Future Trajectories

While massive data scaling to billions of cells is necessary for comprehensive single-cell modeling, the analysis reveals that scale alone is insufficient. The quality and realism of measurement methodologies are equally critical. Current datasets, ranging from tens of millions to one billion cells, primarily consist of RNA-seq measurements that capture only transcriptional state. Comprehensive understanding of cellular function requires integration of proteomic, morphological, and spatial information alongside temporal dynamics.

The path forward requires balancing three considerations: data scale, measurement quality, and biological realism. Rather than exclusively scaling existing RNA-seq approaches, technological advances must expand to multiple modalities that more accurately reflect actual organismal biology. Models must demonstrate generalization to unseen data, tissues, and perturbation contexts - a requirement that demands both increased scale and improved measurement fidelity across diverse experimental conditions.

4. Technical Insights

4.1 Architectural Design Principles

The comparative analysis of transformer-based and flow matching architectures reveals several actionable design principles for single-cell foundation models. First, architectures should prioritize distribution preservation over compression efficiency. The demonstrated information loss in latent vector representations fundamentally limits model utility for applications requiring accurate prediction of cellular response variability. Flow matching approaches that model complete distributions rather than point estimates better align with the stochastic nature of biological systems.

Second, model evaluation must extend beyond mean prediction accuracy to include distributional metrics such as MMD scores, which capture the full statistical properties of cellular populations. Standard evaluation frameworks that focus exclusively on point predictions may inadvertently favor architectures unsuitable for biological applications requiring distributional fidelity.

4.2 Implementation Trade-offs

Flow matching models introduce computational trade-offs relative to compression-based approaches. While transformer models enable efficient encoding of cellular states into compact latent representations suitable for large-scale retrieval and comparison, flow matching requires sampling from learned distributions, which is computationally more intensive. However, the performance gap between these approaches - with simple linear models often matching transformer performance while flow matching models substantially exceed it - suggests that the computational investment in transformer architectures is misallocated.

Implementation of flow matching models requires careful consideration of the noise distribution initialization and the matching objective formulation. The choice of base distribution (typically Gaussian) and the parameterization of the flow transformation significantly impact both training stability and final performance. These architectural decisions represent areas requiring further investigation and optimization.

4.3 Data Integration Challenges

Foundation models must ultimately integrate multiple measurement modalities - transcriptomics, proteomics, morphology, and spatial organization - to achieve comprehensive cellular understanding. Current models predominantly train on RNA-seq data due to its scalability, but this unimodal focus limits biological realism. The technical challenge lies in developing architectures capable of learning joint distributions across modalities with vastly different dimensionalities, measurement scales, and noise characteristics.

The benchmarking results showing that domain-specific models outperform general-purpose transformers on multimodal tasks suggest that naive concatenation or attention-based fusion may be insufficient. Alternative approaches that explicitly model cross-modal dependencies and preserve modality-specific distributional properties warrant investigation.

5. Discussion

The findings synthesized in this analysis reveal a critical juncture in single-cell foundation model development. The predominant paradigm of adapting natural language processing architectures through compression-based encoding has not yielded performance commensurate with computational investment. The success of flow matching models suggests that the biological domain requires architectures fundamentally aligned with distributional modeling rather than information compression. This insight has broader implications for computational biology: biological systems are inherently stochastic, and models that preserve distributional characteristics will more accurately capture cellular behavior than those optimizing for compact representations.

Several knowledge gaps require attention. First, the temporal dynamics of cellular state transitions remain poorly captured by current snapshot-based measurements. Developing models that can infer temporal trajectories from static observations or integrating emerging time-resolved measurement technologies represents a critical frontier. Second, the generalization capabilities of flow matching models across diverse tissues, perturbation contexts, and disease states require systematic evaluation. The superior performance demonstrated on specific benchmarks must be validated across the full scope of biological applications.

Third, the integration of mechanistic biological knowledge with data-driven modeling remains underdeveloped. Current foundation models operate primarily as black boxes, learning patterns from data without incorporating known regulatory networks, pathway structures, or physical constraints. Hybrid approaches that combine the distributional modeling capabilities of flow matching with structured biological priors may yield models that are both more accurate and more interpretable.

The trajectory toward clinically relevant applications - including the 2026 human trials of reprogramming medicine - underscores the urgency of addressing these challenges. Models capable of accurately predicting cellular responses to therapeutic interventions could dramatically accelerate drug development pipelines, potentially reversing the declining trend in new drug approvals despite technological advances. However, achieving this vision requires moving beyond naive scaling of existing approaches toward architectures and measurement technologies that capture biological reality with greater fidelity.

6. Conclusion

This analysis demonstrates that architectural choices in single-cell foundation models have profound implications for their utility in biological and therapeutic applications. Transformer-based compression approaches, despite their success in natural language processing, exhibit systematic limitations when applied to cellular data, frequently underperforming simple linear baselines while incurring substantial computational costs. Flow matching models such as PrimeFlow offer a more promising paradigm by preserving distributional characteristics essential for accurate modeling of stochastic biological processes.

The practical takeaway for researchers and engineers developing single-cell models is clear: prioritize distribution matching over compression, evaluate models using metrics that capture full distributional properties rather than point estimates alone, and recognize that data quality and measurement realism are as critical as scale. The path toward virtual cells, virtual tissues, and digital twins capable of accelerating therapeutic development requires architectural innovations aligned with the fundamental stochastic nature of biological systems, combined with technological advances that expand measurement capabilities across multiple modalities and temporal scales. These considerations will determine whether single-cell foundation models fulfill their promise of transforming drug development and enabling precision medicine approaches that address age-related diseases and other complex pathologies.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub