Your Old Prompts Are Making Claude Worse with Lance Martin, Anthropic

Many AI applications underperform the cost/quality frontier that frontier models like Claude Opus and Sonnet are actually capable of, not because of model li

Your Old Prompts Are Making Claude Worse: An Analysis of Configuration-Driven Cost/Quality Deficits in Production LLM Systems

Abstract

This synthesis examines a recurring failure mode in production deployments of large language model (LLM) applications: systems that operate well inside, rather than on, the cost/quality frontier their underlying models can support. The analysis consolidates evidence from internal benchmarks and case studies covering four intervention classes - prompt anti-pattern auditing, prompt-cache structuring, evaluation design, and autonomous hill climbing - alongside an architectural account of harness design for long-duration agents. Reported results include a 72% cost reduction attributable solely to corrected prompt-cache usage with no measured performance change, an approximately 90% cost reduction on a customer support benchmark with concurrent performance gains, and an improvement from a 66% baseline on a documentation-grounded skill evaluation. The findings suggest that most observable cost/quality deficits in contemporary agentic systems are configuration artifacts rather than model limitations, and that the primary engineering constraint is evaluation quality rather than model capability.

1. Introduction

Practitioners typically treat cost and quality as a strict trade-off: better outcomes are assumed to require larger models, higher reasoning effort, or more tokens. The central claim examined in this synthesis is that this framing misdescribes the majority of deployed systems. As one practitioner source puts it, "we typically think of cost and quality as a trade-off... but many times you're not actually on this frontier. Your application is actually sitting inside this frontier that you could be on." An application positioned inside the frontier can, in principle, improve on both axes simultaneously by removing the configuration artifacts that constrain it.

The term unhobbling describes this removal process - eliminating stale instructions, cache-invalidating prompt structures, misdesigned evaluations, and brittle agent harnesses that prevent a frontier model from performing at its native capability level. This synthesis defines several terms used throughout: prompt debt, accumulated instructions that no longer serve the current model generation and may actively confuse it; hill climbing, iterative, automated optimization of prompts and configurations against a fixed evaluation set; and effort level, a controllable parameter governing how much reasoning a model expends per task.

The analysis proceeds through four intervention classes - prompt auditing, prompt-cache discipline, evaluation design, and hill climbing - followed by an examination of harness architecture requirements for agents operating over multi-day horizons. Each section draws on reported case studies, quantitative benchmarks, and architectural descriptions to substantiate the claim that most cost/quality deficits are fixable rather than inherent.

2. Background and Related Work

Three strands of context frame this analysis. First, task-duration benchmarking: the METER benchmark measures the length of human-equivalent task a model can complete autonomously, with a reported trajectory from roughly four minutes of sustained autonomy (Opus 3) to multiple hours in 2025-era models to, potentially, a day or more in current frontier systems - a horizon that now exceeds METER's measurable range. This trajectory implies that harness requirements change qualitatively as task duration extends, since failure recovery and state persistence become dominant concerns relative to raw model capability.

Second, established evaluation methodology - LLM-as-judge grading, rubric construction, and pairwise comparison - provides the baseline toolkit referenced here, with the contribution being a set of diagnostic properties that distinguish a climbable evaluation from one that merely appears plausible. Third, the Recursive Language Model (RLM) conception of context, in which the context object exists independently of the model and harness and is accessed through tools rather than held wholly within the prompt, underlies the Managed Agents API architecture discussed in Section 3.4.

3. Core Analysis

3.1 Prompt Debt and Anti-Pattern Removal

Prompting conventions developed for earlier, weaker model generations persist in production codebases long after they cease to provide value. "Prompts accumulate debt just [like] code." Anti-patterns in this category include mandated step-by-step decomposition, verification rituals, scratch-pad scaffolding, and thoroughness boosters - instructions that weaker models once ignored harmlessly but that frontier models now actively attend to, sometimes in contradictory ways that degrade output quality. As one source states, "frontier models are very good at thinking, choosing when to think. They don't need to be told what to do and how to verify their work."

A documented case study migrated a 50-ticket customer support evaluation from Opus 4.8 to Opus 5.5. The model upgrade alone produced cost and performance gains; running a prompt audit skill afterward to strip anti-patterns produced compounding additional improvements. Anti-pattern removal guidance is reported as most relevant from Opus 4.5 onward, with earlier or smaller models (e.g., Haiku) potentially still benefiting from some booster instructions - indicating the intervention is model-generation-dependent rather than universally applicable.

3.2 Prompt Cache Structuring

Prompt cache tokens are priced at roughly one-tenth the rate of base tokens and process faster, but the caching mechanism requires strict prefix matching: any byte-level change to a cached prefix invalidates the entire cache for that request. This creates a structural incentive to organize prompts by volatility tier - system instructions and tool definitions (global, lowest volatility), CLAUDE.md/memory content (per project), session state (per session), and message history (incremented per turn, highest volatility).

Running a prompt cache audit skill against internal benchmarks identified cost reduction opportunities of up to 72%, with no change in task performance, attributable entirely to harnesses that were not structuring prompts to preserve cache hits. This result isolates cache misconfiguration as a pure efficiency loss - unlike anti-pattern removal, it carries no performance trade-off, making it a comparatively low-risk, high-yield intervention.

3.3 Evaluation Design as the Binding Constraint

A recurring finding is that frontier Claude models are effective at self-correction and autonomous hill climbing, but only when given a well-structured, climbable evaluation. The diagnostic properties of a climbable eval include: performance increasing with model size, performance increasing with thinking/effort, measurable headroom remaining at the frontier rather than saturation, and low run-to-run variance. Many widely used evaluations fail these criteria because they contain ambiguous tasks that falsely appear to have headroom, causing optimization effort to be wasted on cases that are effectively unsolvable or mis-specified.

Recommended grading practice favors binary or pairwise comparisons over arbitrary numeric scales (e.g., 0-10), with rubrics constructed so a human can reason through the grading logic directly. Consistency should be validated across multiple model families (Sonnet, Opus, Fable) and across repeated runs, and adversarial sampling that overfits an eval set to one model's specific failure modes should be avoided, since this produces an eval that measures idiosyncrasy rather than general capability.

3.4 Hill Climbing and Overfitting Control

A hill climbing skill autonomously improves performance on a constructed evaluation by iterating on prompts, skills, and configuration. The principal risk is overfitting to the eval distribution - for instance, introducing bespoke tools or distinctive phrasing that improves eval scores without generalizing to production traffic. The reported mitigation is a test/train split: the hill climber reflects only on training-set failures while test-set performance is checked each iteration, surfacing a train-improves/test-flat signal as an overfitting indicator.

Two case studies illustrate outcomes. A Claude API skill evaluation improved from a 66% baseline through iterative hill climbing against documentation text. A customer support benchmark hill-climbed for cost reduced spend by approximately 90% while simultaneously improving performance, via a cascade from Opus 5 at high effort to Opus 5 at low effort, then a test of Haiku at low effort (which degraded performance and was discarded), then prompt tuning to recover quality at the cheaper configuration. This generalizes to the heuristic that "the stronger model at lower effort is more token efficient and therefore cheaper than a weaker model at high effort." When hill climbing stalls, a fallback procedure clusters remaining failure cases to detect ambiguous or faulty eval examples, which can then be rewritten rather than optimized against directly.

3.5 Harness Architecture for Long-Duration Agents

As model task-duration capability extends toward and beyond a day, harness design must decouple the "brain" (agent harness/reasoning loop) from the "hands" (execution sandboxes), so that either component can fail and recover independently without losing session state. The Managed Agents API implements this via an append-only, recoverable session log stored independently of both harness and sandbox, with vaults holding secrets so the model never observes them directly, only references them. This reflects the RLM principle that context should exist as an independently queryable object rather than being wholly embedded in the model's working prompt, enabling recovery after compaction events that would otherwise discard state.

4. Technical Insights

Several implementation considerations follow from the evidence above. Cache-structuring interventions are the lowest-risk starting point, since the reported 72% cost reduction carried no performance cost - this suggests auditing cache hit rate (available via Claude Console diagnostics for API-based applications) should precede more invasive prompt or harness changes. Effort-level selection should default to testing the strongest available model at low effort before escalating, since high or max effort on simple problems can burn tokens on unnecessary verification without improving outcomes. Anti-pattern removal is not universally safe to apply; its benefit is contingent on model generation, with Opus 4.5 and later being the primary beneficiaries. Finally, multi-objective hill climbing (jointly optimizing cost, performance, and speed) is reported as feasible but harder to interpret than serial, single-objective optimization passes, suggesting a practical preference for sequencing objectives rather than optimizing them simultaneously.

5. Discussion

The evidence collectively repositions evaluation quality, rather than model capability, as the binding constraint on production system performance. If an eval lacks headroom, exhibits high variance, or contains ambiguous ground truth, hill climbing cannot reliably improve the underlying system regardless of how capable the model is - as noted, non-convergence in hill climbing "usually [is] because of failures in your eval or misconfigurations in eval," not the model. This reframes much of the practical work of LLM application engineering as evaluation engineering.

A second implication concerns the shift from static to dynamic harness design as autonomous task duration grows. The reported trajectory from minutes to potential multi-day autonomy implies that architectural choices such as brain/hands decoupling and recoverable session logs will become default requirements rather than advanced optimizations, mirroring broader software engineering lessons about statelessness and fault isolation in distributed systems.

A plausible production loop emerges from the case studies: online evaluators score live traces, poor-scoring traces are promoted to an offline evaluation set with human review, hill climbing runs against that set, and passing cases become regression tests. This closes the loop between live system behavior and the evaluation artifacts used to improve it, though it depends on sustained human-in-the-loop review to prevent eval drift - a gap the source material flags but does not fully resolve, particularly regarding how to scale human review as trace volume grows.

6. Conclusion

This synthesis demonstrates that a substantial share of cost and quality deficits observed in production LLM applications stem from fixable configuration issues - stale prompt anti-patterns, cache-invalidating prompt structures, poorly designed evaluations, and brittle harnesses - rather than from inherent model limitations. Quantitative evidence spanning a 72% cache-driven cost reduction, a 90% hill-climbed cost reduction with performance gains, and model-migration-plus-audit improvements on a customer support benchmark supports the broader claim that most applications operate inside, not on, their achievable cost/quality frontier.

The practical takeaway is a prioritized intervention order: audit cache structure first (low risk, measurable gain), audit prompts for anti-patterns appropriate to the model generation in use, validate that evaluations meet climbability criteria before investing in optimization, and only then apply hill climbing with explicit train/test separation to guard against overfitting. As task-duration capability continues to extend, harness architectures that decouple reasoning from execution and persist session state independently will likely become a prerequisite rather than an optimization for reliable long-running agentic deployments.

Sources

• Your Old Prompts Are Making Claude Worse - Lance Martin, Anthropic - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub

At 22Atlas, the studio I run, we put Claude to work for clients, and 22Atlas on Claude covers how.