"Where is all the automation?" with Diogo Almeida, TypeSafe AI
Modern AI (specifically RLHF-trained LLMs) is optimized for human-pleasing assistance rather than reliable automation, which explains why it can be superhuma
By Sean WeldonMachine-Native Intelligence and the Limits of RLHF: Why Benchmark-Leading AI Fails at Automation
Abstract
This synthesis examines an argument that the jaggedness of contemporary AI capability - superhuman benchmark performance alongside minimal real-world automation - stems from an objective-selection artifact rather than a capability deficit. The central claim is that Reinforcement Learning from Human Feedback (RLHF) embeds a human-in-the-loop requirement into every modern Large Language Model (LLM), optimizing the deployed population for assistance rather than automation. The analysis extends Sutton's bitter lesson, arguing task selection dominates data and compute as a source of capability gains, and grounds this in the original RLHF finding that a GPT-2-scale model matched instruction-following gains that pre-training scaling alone would require roughly 100x effective compute to reach. Drawing on this framework, the discussion develops a case for narrow, reliable, "machine-native" intelligence - exemplified by the system Jev - as a distinct architectural category, with implications for agent memory, software design, and the economics of AI-generated low-quality output ("slop").
1. Introduction
Contemporary discourse on AI capability has split into two largely incompatible readings of the same evidence. One camp observes benchmark results in which LLMs now exceed human performance across broad task distributions, including competition-level mathematics. A second camp observes that software, business processes, and everyday automation remain largely unchanged in structure. This analysis holds that both observations are correct simultaneously: AI is transformative in raw capability and underwhelming in deployed effect.
Automation is defined here as task execution in which a decision is made and control is returned to a machine process without human review. Assistance, by contrast, denotes tasks that are intrinsically human-in-the-loop - deep research, drafting, exploratory analysis - where a human remains the final arbiter of output. The central thesis is that this distinction, not task difficulty, explains why a system can contribute to advanced mathematics while failing at routine accounting or drive-through order processing. Because automation is expressed in code, automation capability requires integration with code rather than conversational interaction with a human reviewer.
The analysis proceeds in three stages: it establishes the theoretical basis for the assistance/automation partition via RLHF's structural properties; it develops the architectural consequences for agent design, particularly constraints imposed by the KV cache; and it considers the societal and economic consequences, including the production of AI-generated "slop" as a function of unreliability combined with externalized cost.
2. Background and Related Work
Three prior results anchor the argument. Sutton's bitter lesson holds that general methods leveraging scaled computation reliably outperform methods encoding hand-designed structure. This analysis proposes two extensions. The first - termed the "bitterest lesson" - asserts that data matters more than compute, consistent with current practice in which internet-scale text has become a binding constraint, motivating large-scale digitization of books as a supplementary data source. The second, stronger extension asserts that task selection dominates both compute and data as a driver of capability gains.
The transition from BERT-era masked language modeling (MLM) to autoregressive next-token prediction is presented as supporting evidence. These objectives are not ordered by quality on a shared metric; next-token prediction is not objectively superior at the MLM task. It is superior for generating strings, which is the property that made GPT-3 possible. The claim is counterfactual: no increase in compute or algorithmic refinement applied to MLM would have produced an artifact comparable to GPT-3, because MLM solves a different task. InstructGPT and the original RLHF paper provide the quantitative anchor developed in Section 3.2, while RL with Verifiable Rewards (RLVR) enters as a contrasting post-training regime whose products are reported to be difficult to interact with absent a subsequent RLHF stage.
3. Core Analysis
3.1 The Assistance/Automation Partition
The argument rests on a structural claim about RLHF: the "HF" component requires human feedback during training, which embeds a human-in-the-loop assumption into the resulting model's behavior. The consequence is that 100% of LLMs deployed today are optimized for assistance rather than automation, regardless of their benchmark performance on unrelated axes. This produces a model that is, in the phrase used in the source material, "too good to be true" at assistance while being unsuitable for unsupervised background cognitive work - because it was never optimized for that objective. RLVR models, trained without an HF stage, are described as functioning poorly in interactive use unless RLHF is subsequently applied, reinforcing the claim that the human-pleasing property is a distinct, separately-trained capability rather than an emergent byproduct of raw capability.
3.2 The RLHF Scaling Result
The quantitative basis for prioritizing task selection over scaling is drawn from the original RLHF paper, in which a GPT-2-sized model trained with comparatively simple algorithms achieved instruction-following performance comparable to a hypothetical much larger model ("GPT-9," using unadjusted internal performance figures) scaled via pre-training alone. Matching this gain through pre-training scaling alone is estimated to require on the order of 100x effective compute scaling - a factor characterized as abandoned by the field as infeasible. This result is presented as the clearest evidence that selecting the correct training objective (instruction-following via RLHF) produced a larger capability leap than scaling model size, supporting the claim that "doing the right task" is the dominant lever for capability gains.
3.3 Reliability as a Design Objective: The Jev Case
The system Jev is presented as a counter-architecture built around narrow, reliable, machine-native intelligence rather than RLHF-style human-pleasing optimization. The design rationale is summarized by the heuristic that an unreliable system produces "Einstein 95% of the time, Mr. Bean 5% of the time," and that automation is infeasible if the two outcomes cannot be distinguished at inference time. The claim is that reliability, rather than speed, cost, or interface quality, is the attribute driving adoption, and that this property is not captured by standard benchmarks, which the material characterizes as actively misleading the practitioners most dependent on them.
3.4 Architectural Constraints: The KV Cache Bottleneck
Agent architectures are described as constrained by the KV cache, which forces context to grow in proportion to history length (O(N) memory) and treats accumulated context as effectively global, undifferentiated state. This is identified as the mechanism preventing more modular agent designs, including non-O(N) memory via map-reduce-style history summarization, shared state across multiple agents to avoid redundant rediscovery of information, and non-naive routing among sub-agents. The repeated reversal among large AI labs on whether sub-agent architectures are effective is attributed to this constraint undermining the structural elegance such designs would otherwise offer.
4. Technical Insights
Several implementation-relevant findings follow from the above analysis:
- Objective mismatch over capability deficit: A model's failure at automation tasks should not be interpreted as a general capability limitation; it may reflect that the model's training objective (human approval) is orthogonal to the task's success criterion (correct unsupervised action).
- RLVR requires RLHF for usability: Verifiable-reward training alone produces models reportedly poor to interact with, suggesting post-training stages are not interchangeable and that human-interaction quality is a separately optimized property.
- Scaling versus task selection trade-off: The cited 100x effective compute gap between pre-training scaling and RLHF-driven instruction-following suggests that, for a fixed compute budget, selecting an appropriate training objective can outperform further scaling of an existing objective.
- KV cache as an architectural limitation: Treating the KV cache as global state limits agent memory to linear growth and complicates state-sharing across sub-agents; map-reduce-style context management and explicit shared-state designs are proposed as mitigations, though no implementation details or empirical validation are provided in the source material.
- Benchmarks as an unreliable proxy: Reliability, framed as the central differentiator for automation-suitable systems, is explicitly described as unmeasured by current benchmark suites, limiting their utility for evaluating automation readiness.
A limitation of this analysis is that it is argument-driven rather than empirically demonstrated within the source material; the RLHF scaling comparison and the Jev reliability claims are presented without accompanying quantitative benchmarks or reproducible methodology.
5. Discussion
The partition between assistance and automation offers a parsimonious explanation for an otherwise puzzling pattern in deployed AI systems: capability on open-ended, human-evaluated tasks coexisting with fragility on bounded, unsupervised tasks. This reframes a common narrative of "uneven" AI progress as a predictable consequence of the dominant post-training methodology rather than an unexplained inconsistency.
The economic implication concerns the production of low-quality AI output. The argument holds that because current systems cannot be trusted with costly, consequential decisions, organizations default to human review, and the resulting costs are frequently externalized onto third parties rather than absorbed internally - an asymmetry proposed as the structural cause of AI-generated "slop." This suggests that reliability improvements, rather than raw capability improvements, are the more direct lever for addressing output quality at scale.
A broader implication concerns software architecture itself. The analogy drawn between machine-native reliable intelligence and the historical addition of a fourth logic gate to binary computing frames this shift as a change in the available primitives of software construction, not merely an added feature. This motivates the proposed research agenda of "software research" - investigating decomposition, verifiable intermediate stages, and non-KV-cache-bound state management as open architectural questions rather than settled engineering practice.
6. Conclusion
This synthesis argues that the apparent contradiction between benchmark-superhuman AI and underwhelming real-world automation is resolved by recognizing that RLHF structurally optimizes LLMs for human-approved assistance rather than autonomous task completion. Task selection, exemplified by the outsized capability gain RLHF produced relative to pre-training scaling alone, is presented as a more consequential lever than either data or compute scaling. The practical takeaway is that automation-oriented AI development may require distinct training objectives and architectures - such as the reliability-first design exemplified by Jev and KV-cache-aware agent memory management - rather than further scaling of assistance-optimized systems. Future work should focus on reliability-centric evaluation methodologies, since current benchmarks do not measure the property identified here as decisive for automation suitability.
Sources
- "Where is all the automation?" - Diogo Almeida, TypeSafe AI - Original Creator (YouTube)
- Analysis and summary by Sean Weldon using AI-assisted research tools
About the Author
Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.
Sean Weldon builds apps and custom software at 22Atlas.