Engineering Agency out of the Happy Path with Matthew Jewkes, Standard Cybernetics

Full automations - systems that remove humans entirely from the "happy path" and reserve human attention only for genuine exceptions - outperform

Engineering Agency out of the Happy Path: Full Automation as an Alternative to Human-in-the-Loop Agentic Systems

Abstract

This synthesis examines the claim that full automations - systems that remove humans from the "happy path" entirely and reserve human attention for genuine exceptions - outperform human-in-the-loop agentic designs in enterprise settings. As the marginal cost of white-collar execution falls with cheaper AI capacity, the binding constraint shifts to organizational judgment, which is characterized as currently underpriced. Evidence comes from a production document-indexing system deployed at Clarivate across a bio-pharma drug-development ontology spanning tens of millions of documents, which processed tens of thousands of monthly tasks and reached 99.95% accuracy on business-critical work. The central methodological finding is that accuracy gains derived less from model improvement than from case routing - a three-bucket failure taxonomy separating contentious, critical, and exceptional cases. Practical implications include eval-driven development with domain experts, explicit investment criteria for automation candidates, and the aggregation of automation "exhaust" as a form of organizational refactoring.

1. Introduction

Enterprise deployment of large language models has been dominated by agentic architectures, in which a model plans, calls tools, and iterates, typically with a human reviewing intermediate outputs. This analysis presents a contrasting position, summarized under the heading "engineering agency out of the happy path."

Two terms require precise definition. The happy path denotes the modal, well-specified route through a business process, excluding anomalies and edge cases. Full automation, as used here, means removing humans from that path completely - humans are reserved for exceptions, not inserted as reviewers of routine transactions. This is explicitly distinguished from human-in-the-loop (HITL) designs, in which human attention is consumed on every transaction regardless of whether that transaction is routine.

The central question this analysis addresses is not whether frontier models are sufficiently capable, but how enterprise systems should be architected so that scarce human judgment is conserved and redirected rather than diffused across routine decisions. The underlying economic premise is that cheap execution does not reduce organizational burden; it increases the relative load on judgment, since judgment was already the scarcer resource. Section 2 establishes this economic framing and its intellectual lineage. Section 3 analyzes the architectural argument through a bio-pharma case study. Section 4 extracts technical findings on accuracy and routing. Sections 5 and 6 discuss implications and investment criteria for organizations considering similar deployments.

2. Background and Related Work

The framing rests on a simple economic observation: the cost of raw white-collar execution is dropping rapidly as AI capacity becomes cheaper. Consequently, cheap execution increases, rather than relieves, the load on judgment - the determination of what to do - which was already in short supply within organizations.

"Cheap execution increases the load and the burden on judgment on deciding what to do. Organizational judgment was already in short supply."

The observed failure mode is that an organization's strongest domain experts have attention smeared across multitasking and routine decision-making, leaving little residual capacity to move the business forward. This positions full automation not as a novel technological frontier but as organizationally familiar, inheriting from decades of deterministic software automation in marketing operations and DevOps, and from centuries of industrial automation more broadly. Its defining property is legibility: the system accepts an input, produces an output, and encapsulates internal complexity so that the organization can reason about it the way it reasons about a machine.

Agentic systems are treated as a distinct and, in enterprise contexts, more problematic category. They are described as difficult to maintain, prone to failure in unusual ways, and - critically - as increasing rather than decreasing the decision burden on the humans who must monitor their notification streams. This is not a wholesale dismissal of agents:

"Agents have permanently changed and expanded my idea of what I think is possible with software. But in enterprise, agents have problems."

The distinction is architectural rather than evaluative: agents expand the space of what software can do, but their flexibility is precisely what makes them organizationally illegible when deployed as the default execution mechanism for routine work.

3. Core Analysis

3.1 Case Study Context and Initial Prototype

The case study derives from a deal-enablement company serving bio-pharmaceutical M&A and licensing, sold to Clarivate in 2023. Clarivate maintains an annotated history of the drug development industry comprising tens of millions of documents and hundreds of thousands of edits per year across a rich ontology. The automation task was document indexing: tens of thousands of new indexing tasks per month, triggered by document edits or insertions, needed to be classified and routed correctly.

An initial 2024 prototype, built using GPT-4 and o3-era models in a research-and-assignment agent style, reached 80-83% accuracy after several sprints. This performance plateau motivated a shift in methodology: rather than continuing to iterate on the agent's reasoning process, approximately 1,000 evals were constructed and reviewed directly with domain experts. Notably, experts were presented with pre-populated examples rather than blank spreadsheets to fill out - a design choice that built buy-in and focused expert attention specifically on failure cases rather than on open-ended annotation work.

3.2 The Three-Bucket Failure Taxonomy

Expert review of the evals surfaced three distinct categories of failure, each requiring a different engineering response rather than a uniform accuracy-improvement effort:

• Bucket 1 - Undecided/contentious cases: situations where experts disagree or the organization has not settled an answer. One example involved whether colchicine targets interleukin-1 beta, a question on which GPT-5 and Opus 4.5 disagreed with each other.
• Bucket 2 - Critical errors: mistakes that must not occur because they threaten material accounts. Even top models failed at a baseline rate of 2-3% on multi-step workflows in this category.
• Bucket 3 - Exceptions/escalations: roughly 3-5% of cases involving genuinely novel situations, such as a new drug (Verve 102) with mechanisms new to science, requiring human ontology work that could not be automated by definition.

The routing logic built around this taxonomy worked by detecting AI confusion or unusual case shape and categorizing such cases away from the main pipeline, identifying business-critical cases and sending them through a robustly tested sub-pipeline, and shipping remaining uncertain cases back to humans as structured exceptions rather than silent failures.

3.3 From 95% to 99.95%: The Role of Case Routing

The most significant methodological finding is that the final accuracy gain - from approximately 95% to 99.95% (three-and-a-half nines) - was achieved primarily through case routing rather than through further model or prompt improvement. Removing ambiguous and escalation cases from the main pipeline allowed standard hill-climbing techniques to be applied effectively to a narrowed, well-defined problem set. At least as much engineering time was invested in case routing as in error-climbing for the critical set, indicating that routing logic, not model capability, was the binding constraint on production accuracy. The system was deployed only after a random sample demonstrated performance exceeding historic human work on the same tasks.

4. Technical Insights

Several implementation-level findings generalize beyond the specific case study. First, scale matters for eval construction: approximately 1,000 evals were required to surface the failure taxonomy with sufficient granularity, and this process demanded significant direct time investment reading failure cases alongside domain experts rather than delegating eval creation entirely. Second, model disagreement on ambiguous domain facts persisted even among frontier models (GPT-5, Opus 4.5), and individual models exhibited flip-flopping behavior on repeated queries of the same ambiguous question - a finding with direct implications for any system treating model output as ground truth on contentious facts. Third, the 2-3% baseline failure rate on multi-step LLM workflows, observed even with top models, suggests that routing around critical-error cases is not an optional refinement but a structural necessity for reaching enterprise-grade reliability. Fourth, the AI system itself was used as part of the pre-filtering mechanism - detecting confusion, categorizing unusual shapes, and assessing business-criticality - meaning the routing layer was not a separate deterministic rule set but partially AI-driven. A limitation worth noting is that this architecture presupposes a sufficiently mature, specifiable business process; the approach was not evaluated against early-stage or exploratory workflows where the happy path itself is not yet stable.

5. Discussion

The broader implication of this case study is that the primary engineering challenge in enterprise AI deployment may not be model capability but organizational routing - the ability to separate the specifiable majority of a workflow from its ambiguous and exceptional minority. This reframes "AI engineering" as substantially an exercise in taxonomy construction and failure-mode routing, conducted in close collaboration with domain experts, rather than purely a model-selection or prompt-engineering exercise.

A second implication concerns the aggregation of what is termed automation "exhaust" - the ambiguous, exception, and uncertain critical cases removed from the main pipeline. Rather than being a byproduct to be discarded, this exhaust is reframed as valuable organizational signal. Ambiguous or undecided cases, when aggregated, can prompt an organization to formally deprioritize marginally valuable unresolved questions, reducing cognitive load. Rare exception cases, individually too scattered to prioritize, become visible as patterns once aggregated, enabling experts to form investment opinions proactively - as illustrated by an expert forming a view on an 8-figure account's problem before it fully materialized. This function is described as a form of organizational refactoring: reducing complexity and routing talent toward higher-value problems.

This analysis also surfaces a notable claim about human-AI performance comparison: the automated system is reported to have outperformed PhD-level human experts in aggregate, though not necessarily on any individual case, attributable to human fatigue, misclicks, and time pressure under volume. This distinction - aggregate versus case-level superiority - is an important nuance often elided in capability comparisons, and merits further empirical scrutiny in other domains before being generalized.

6. Conclusion

The central contribution of this analysis is a reframing of enterprise AI engineering: the primary lever for reliability is not incremental model improvement but the construction of a routing layer that separates specifiable, high-volume work from ambiguous and exceptional cases. Full automation outperforms human-in-the-loop agentic design in mature, well-specified enterprise processes precisely because it preserves legibility while freeing expert attention for aggregated, pattern-level judgment rather than per-transaction review.

Practically, organizations considering this approach should prioritize scaled, already-specified business processes tied to material ROI, invest heavily in eval construction conducted jointly with domain experts, and treat exception aggregation as a deliberate organizational design output rather than an incidental byproduct. The approach is explicitly not recommended for early-stage or discovery-oriented business processes where flexibility, not legibility, remains the priority.

Sources

• Engineering Agency out of the Happy Path - Matthew Jewkes, Standard Cybernetics - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub