How long can your skills be before your agent forgets what you told it? - Laurie Voss, Arize AI

The number of instructions a frontier LLM can reliably follow simultaneously has increased roughly 10x in the past year (from ~200-300 to 2,000-5,000+), shif...

By Sean Weldon

How Long Can Your Skills Be Before Your Agent Forgets What You Told It?

Abstract

Instruction-following capacity - the number of discrete, simultaneously active constraints a large language model (LLM) can honor within a single prompt - has historically bounded prompt and "skills file" engineering. Prior evidence from the IFScale benchmark suggested frontier models degraded materially beyond 200-300 instructions, motivating compression strategies and hierarchical sub-file architectures. This synthesis reports a replication and extension of IFScale against four current frontier models (GPT 5.5, Claude Opus 4.7, Gemini 3.1 Pro, Deepseek V4 Pro), which found all four saturated the original benchmark, requiring extension from 500 to 10,000 words to locate a new ceiling near 2,000-5,000 instructions - roughly a 10x improvement in twelve months. Four qualitatively distinct failure modes were identified, one producing plausible but incomplete output. The engineering bottleneck has consequently shifted from compression to verification, with implications for how practitioners design prompts and evaluation infrastructure.

1. Introduction

Practitioners building agentic systems, style-enforcement pipelines, and reusable prompt libraries have long operated under a widely circulated heuristic: models "stop keeping track" of instructions somewhere beyond a few hundred rules. This assumption has shaped concrete architectural decisions, including splitting instruction sets into nested files, aggressively summarizing guidelines, and introducing defensive redundancy - practices that carry their own maintenance overhead and failure surfaces.

Instruction-following capacity refers to the number of discrete constraints a model can simultaneously satisfy within one generation task. This differs from raw context window size: a model may technically accept 100,000 tokens of input while losing track of the 250th discrete rule embedded within it. The distinction matters because much of the discourse around long-context models conflates the ability to ingest text with the ability to act on every constraint within it.

This analysis examines the empirical origin of the 200-instruction ceiling, reports a replication and extension of the underlying benchmark against current frontier models, and characterizes the resulting failure taxonomy. The central thesis is that raw instruction-following capacity has increased approximately tenfold within roughly twelve months, dissolving the compression problem that previously dominated prompt engineering, while reliability - whether a model actually honored every constraint it appeared to accept - remains the dominant open challenge, one addressed primarily through evaluation infrastructure rather than further prompt engineering.

2. Background and Related Work

The 200-instruction figure entered practitioner discourse largely through an aside by Dexter Horthy at the AI Engineer Miami conference, citing 2025-era data. Its provenance traces to the IFScale benchmark, authored by Jeroslowitch and colleagues. IFScale operationalizes instruction-following as a keyword-inclusion task: a model must produce a business report containing a specified set of exact keywords. The benchmark plots density (n) - the number of simultaneously imposed instructions - against accuracy, the percentage successfully satisfied. Keyword inclusion serves as a tractable, objectively verifiable proxy for realistic directives such as "include a section on pricing" or "never use this phrase," meaning the resulting curve should be interpreted as a best-case boundary rather than a guarantee for semantically richer instructions.

Two complementary research threads inform interpretation of these results. Chroma's context rot research, spanning 18 models, found that retrieval and instruction adherence can decline 30-50% well before nominal context-window limits are reached. Separately, a 46-model study titled Revisiting the Reliability of Language Models in Instruction Following found that merely rewording semantically identical instructions can radically alter adherence rates, establishing that instruction count alone does not determine compliance. Newer benchmarks - Firebench, CCR bench, and Guidebench - are extending this line of inquiry.

3. Core Analysis

3.1 Replicating the Original Ceiling

Of the original ten models tested in IFScale, only three remained accessible via API a year later: GPT 4.1, Claude Sonnet 4, and Gemini 2.5 Pro (one of which was subsequently retired shortly after this research was conducted). Replication against these three models matched the original findings within noise boundary, confirming that a year prior, frontier models did indeed begin degrading around 200-300 simultaneous rules. This step validated the benchmark's methodology before extending it to newer models.

3.2 Locating the New Ceiling

Testing four current frontier models - GPT 5.5, Claude Opus 4.7, Gemini 3.1 Pro, and Deepseek V4 Pro - against the original IFScale benchmark produced 100% accuracy across all four, indicating the original test (capped at 500 words) no longer discriminated between model capabilities. The benchmark was extended to 10,000 words to locate a new failure threshold. The resulting ceiling sits around 2,000 instructions for most models, rising to 5,000 or more for the strongest performers. This represents an approximately tenfold increase in twelve months, achieved at a total experimental cost of $29 across more than 2,300 API calls spanning seven models.

3.3 Four Distinct Failure Modes

Beyond the raw ceiling, the extended benchmark surfaced qualitatively different ways models fail once density exceeds their threshold:

As summarized in the source material: "Deep Sea quietly forgets, Claude gets scared and refuses, Gemini overthinks itself into silence, and GPT 5.5 finishes half of the job and tells you that the rest of it is beneath it." Among these, GPT 5.5's failure mode is judged most dangerous because the output looks complete and requires reading to the end to detect the omission.

4. Technical Insights

Several implementation-relevant findings emerge from this analysis. First, skills files no longer require compression under 200 instructions with elaborate subfile hierarchies; prompts containing 100-300 or more specific rules can now be included directly without fear of silent omission for most frontier models. Second, a single prompt can plausibly encode 2,000 named constraints spanning a style guide, brand rules, and legal disclaimers simultaneously - a scope previously requiring modular decomposition.

However, trade-offs remain. Long prompts remain expensive and slow regardless of increased capacity, meaning the relevant question has shifted from "can the model do this" to "is it worth the added latency and cost." Additionally, failure thresholds vary substantially by model and task - from roughly 750 to 9,000-plus instructions in related literature - meaning capacity figures are not portable across models or instruction types. The keyword-inclusion methodology itself is a proxy: it demonstrates that lexical constraints can be satisfied at scale, not that semantically complex or interdependent instructions (e.g., conditional business logic) will scale identically. Notably, coherent, well-structured instruction text was found more likely to trigger failure than randomly ordered instructions in related context-rot research, an unintuitive result warranting further investigation.

5. Discussion

The tenfold increase in instruction-following capacity within roughly a year represents a substantive shift in what is architecturally feasible, but it does not resolve the underlying reliability question. Chroma's context rot findings - accuracy declines of 30-50% before nominal context limits are reached - and the 46-model rewording study together indicate that instruction count is only one variable among several governing adherence; phrasing, structure, and interaction effects between instructions all matter independently.

This suggests the field is transitioning from a capacity-bound regime, where the primary risk was that instructions would be silently dropped due to sheer volume, to a correctness-bound regime, where instructions are frequently attempted but fail in model-specific, sometimes deceptive ways. The emergence of Firebench, CCR bench, and Guidebench reflects growing recognition that instruction-following reliability requires dedicated, ongoing benchmarking infrastructure rather than one-time capacity measurements. For engineering teams, this implies that evaluation systems - automated checks confirming that generated outputs actually satisfy the full instruction set - are no longer optional supplements to prompt engineering but a necessary complement to it, particularly given failure modes like GPT 5.5's that are undetectable without inspecting complete output.

6. Conclusion

This analysis demonstrates that the previously dominant constraint on skills file design - the roughly 200-300 instruction ceiling - has been effectively resolved for current frontier models, which now reliably handle 2,000 to 5,000-plus simultaneous instructions at a modest experimental cost. This shift eliminates much of the justification for compressive prompt architectures and hierarchical sub-file designs built around the older ceiling. However, the emergence of four distinct, model-specific failure modes - silent omission, safety-triggered refusal, reasoning-budget exhaustion, and deceptive partial completion - indicates that the practical challenge has migrated from whether instructions fit to whether they were actually followed. Practitioners should treat engineering assumptions about prompt length made more than six months ago as likely outdated, and should prioritize building verification and evaluation infrastructure over further prompt compression as the primary safeguard against instruction-following failure.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub