Robot Demos Are Easy. Reliability Is Hard - Jason Ma, Dyna Robotics

Covariant (Diana) is building generalist robotic manipulation foundation models that combine broad pre-training with targeted active learning and reward mode...

By Sean Weldon

Robot Demos Are Easy, Reliability Is Hard: An Analysis of Covariant's Approach to Generalist Manipulation

Abstract

Generalist robotic manipulation policies have advanced rapidly, yet most remain confined to demonstration settings because per-task success rates of 80-90% collapse under the compounding demands of continuous commercial operation. This synthesis examines the approach taken by Covariant (Diana), a Series A robotics company founded in September 2024 with approximately $120 million in funding, which pairs frontier foundation-model research with more than five active commercial deployment sites. The central methodological contribution is the augmentation of standard pre-training and post-training with reward models that score manipulation progress from video, enabling a human-in-the-loop active learning loop termed scalable supervision. Applied to deformable-object manipulation, this pipeline raised napkin-folding success from approximately 80% to 99.4% across four independent 24-hour continuous trials. A complementary data recipe enabled zero-fine-tuning transfer to novel environments, validated at public venues. Implications for reliability engineering in embodied AI are discussed.

1. Introduction

The deployment of learned manipulation policies into economically meaningful settings is constrained less by average capability than by tail reliability. A policy that succeeds on 80-90% of attempts appears competent in isolated demonstrations, but the probability of ten consecutive successes at that rate falls below 0.1%. Commercial tasks - folding napkins in a restaurant, folding towels in a laundromat, opening cans at a public event - require thousands of consecutive successes with minimal human intervention, a regime that ordinary benchmark evaluation does not stress.

Two established paradigms address this tension imperfectly. Specialist pipelines, engineered for a single task, achieve high reliability but do not scale: each new task demands a new engineering effort. Generalist foundation models transfer across tasks but plateau at reliability levels insufficient for unattended commercial use. The thesis examined here is that these paradigms need not be mutually exclusive: a broadly pre-trained backbone, combined with targeted post-training driven by learned reward signals, can yield policies that are simultaneously general-purpose and near-100% reliable, while also generalizing to unseen deployment environments without additional fine-tuning.

This synthesis proceeds by establishing organizational and data context, analyzing the model architecture and the napkin-folding case study in detail, extracting actionable technical insights, and discussing broader implications for reliability engineering in embodied AI.

2. Background and Related Work

Covariant's operating thesis is the research and deployment flywheel: frontier model development and commercial deployment are treated as mutually reinforcing rather than sequential. Deployment sites serve a dual function - commercially, they generate revenue and distribution; scientifically, they function as adversarial test environments that surface model and hardware weaknesses invisible in laboratory evaluation, sharpening research prioritization.

"Our thesis is that to actually bring robots into the real world, being at commercial grade doing useful tasks, the company needs to combine doing frontier research with a lot of commercial deployments."

The data strategy is organized as a pre-training data pyramid with three tiers: off-robot data (egocentric human video, public datasets, simulation), on-robot diverse task data collected across embodiments, and high-quality deployment data from live commercial sites. This pyramid, comprising more than 200,000 hours of data, underlies a two-part architecture pairing a high-level reasoning model with a low-level world action model responsible for fine-grained, high-frequency dexterous control. This separation of semantic planning from low-level control echoes broader trends in embodied AI toward hierarchical policy decomposition, though the specific integration of reward-model-driven active learning distinguishes this approach from purely imitation-based pipelines.

3. Core Analysis

3.1 The Reliability Gap in Generalist Models

Current generalist manipulation models achieve approximately 80-90% success per task out of the box. Given the multiplicative nature of sequential task execution, this ceiling is incompatible with commercial deployment: at 85% success, ten consecutive attempts succeed less than 0.1% of the time. This gap explains why specialist, hand-engineered pipelines remain prevalent in industry despite their lack of scalability - they trade generality for the reliability that commercial operation demands. Covariant's stated objective is to close this gap without sacrificing generality, positioning reliability itself as the primary engineering target rather than an emergent byproduct of scale.

3.2 Reward Models and Scalable Supervision

The mechanism used to close the reliability gap is a reward model trained to score robot task progress from video on a continuous scale and to detect non-monotonic dips that indicate errors. This capability is significant because deformable-object manipulation - the napkin-folding case - presents near-infinite configuration states, making exhaustive data collection infeasible. Rather than attempting to cover the state space directly, the reward model identifies where the current policy fails, and human-in-the-loop annotators collect targeted correction data at those specific failure points. This loop, termed scalable supervision, converts a data collection problem of unbounded scope into a tractable, iterative refinement process.

"Once we have this reward model, we can actually do what I consider scalable supervision."

3.3 Case Study: Napkin Folding with Dyna-1

The Dyna-1 model provides a quantitative demonstration of this pipeline. Standard pre-training and post-training alone yielded approximately 80% success, with the model frequently becoming stuck once an error occurred - for example, when a parallel jaw gripper failed to isolate a single napkin from a stack. Applying the reward-model-driven active learning loop raised success to 99.4% across four independent 24-hour continuous trials, evaluated against fold-quality grading criteria (e.g., a grade-five fold acceptable versus a grade-three fold rejected, differing by approximately one inch in seam placement).

Notably, the resulting model generalized to error-recovery behaviors that were not explicitly present in the collected training data, including recovering from accidentally pulling an entire napkin stack over. This suggests that targeted post-training on a subset of failure modes can interpolate to a broader family of recovery behaviors, rather than merely memorizing corrections for observed failures.

3.4 Generalization to Novel Deployment Environments

A separate but related contribution is a data recipe enabling deployment to new physical sites without additional fine-tuning. This was validated publicly at CoRL 2025 in Korea, where a robot folded t-shirts continuously for three days despite active interference from conference attendees, and at Red Bull-sponsored events where a robot opened cans amid the variable and uncontrolled lighting conditions of a music festival. These deployments, alongside restaurant napkin-folding and a Sacramento laundromat towel-folding installation, indicate that the reliability gains observed in controlled settings persist under environmental distribution shift, a property not guaranteed by reward-model refinement alone.

4. Technical Insights

Several implementation-level findings emerge from this approach:

5. Discussion

These findings suggest that the reliability barrier in generalist manipulation is not solely a function of model scale or pre-training volume but of the presence or absence of a targeted feedback mechanism connecting deployment failures back to training data collection. The reward-model-based active learning loop functions analogously to reinforcement learning from human feedback in language models, substituting a learned progress estimator for a preference model, and applying it specifically to the long-horizon, partially observable setting of physical manipulation.

An open question concerns the boundaries of generalization: while error-recovery behaviors transferred to unseen failure modes within napkin folding, and the zero-fine-tuning data recipe transferred across physical sites, it remains unclear how far this generalization extends across fundamentally different task categories, object classes, or embodiments. The organization's stated caution around consumer deployment - citing polish and privacy/safety concerns - indicates that current generalization claims are bounded by the enterprise task distribution studied thus far.

The emphasis on hardware-software co-design as the primary bottleneck, rather than modeling capacity, is a notable departure from narratives that treat scale as the dominant lever in embodied AI, and points toward proprietary full-stack integration as a deliberate strategic choice over open developer ecosystems.

6. Conclusion

This analysis demonstrates that near-100% reliability in generalist robotic manipulation is achievable through the combination of broad pre-training, a reward model capable of scoring long-horizon task progress, and a targeted active learning loop that converts observed failures into corrective training data. The napkin-folding case study, moving from approximately 80% to 99.4% success across sustained 24-hour trials, provides concrete evidence that this combination addresses the reliability gap that has historically confined generalist policies to demonstration settings. Practically, this suggests that organizations pursuing commercial-grade autonomy should prioritize investment in progress-estimation and failure-detection infrastructure alongside raw data scale, and that hardware-software co-design remains a first-order constraint on further progress.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub