Your SKILL.md Is 70% Stuff the Model Already Knows with Nikolay Petrov, Intel
Agent "skills" that aim to teach models new knowledge should be evaluated with a deterministic, item-level methodology (inspired by FActScore) comparing mode
By Sean WeldonYour SKILL.md Is 70% Stuff the Model Already Knows
Abstract
Agent skills - packaged instructions and reference material loaded into a model's context - are typically validated by author intuition rather than measurement. This synthesis presents a deterministic methodology, inspired by FActScore, for evaluating skills intended to teach new knowledge rather than merely steer process. The approach decomposes a skill into discrete items (typically 25-30, scaling to ~400), each paired with a non-leaking question, and scores model performance with and without the skill loaded using a judge from a different model family. Items fall into four outcome categories spanning redundancy, degradation, genuine learning, and gaps. Findings show frontier models retain general concepts far better than exact parameters (Opus: ~70% lenient vs. ~22% strict scoring), degradation is rare in popular skills, and a 29-skill evaluation cost approximately $14. The method offers skill authors a guidance signal in place of brute-force editing.
1. Introduction
Agent capability engineering follows a predictable lifecycle. A practitioner issues the same manual prompt repeatedly, confirms it works, and eventually packages it into a reusable artifact. This artifact - a skill - confers repeatability, portability, and reproducibility on behavior that previously existed only as a human's working habit. Skill ecosystems have matured enough that individual community skills accumulate hundreds of GitHub stars, yet validation practice has not kept pace with packaging practice.
A useful taxonomy separates skills into two categories. Knowledge skills supply information or capability the model does not already possess. Steering (process) skills encode a repeatable sequence of actions without introducing new information. This distinction is consequential: a steering skill succeeds if its sequence is followed, while a knowledge skill succeeds only if the model's effective knowledge state actually changes. As framed directly: "We get our first skills packed, start using them starting with the feel of does it work or not. But there is also a more technical way and more deterministic way for checking if skills are working or not."
The central thesis is that knowledge-oriented skills should be evaluated through a deterministic, item-level comparison of model performance with and without the skill loaded, so that authors can distinguish content that genuinely teaches from content that is redundant or actively harmful. This analysis covers the evaluation methodology, its scoring dimensions, cross-model findings, and the resulting optimization framework.
2. Background and Related Work
Two prior strands inform this methodology. The first is evals - task definitions constructed to be as deterministic as possible so outcomes can be validated rather than judged impressionistically. Applied to skills, an eval becomes a two-arm comparison: an agent runs a fixed task suite with no skill loaded, then with the skill loaded. A representative outcome is a no-skill arm solving three of five tasks against a with-skill arm solving all five, with token usage and time-to-completion also tracked as secondary costs.
The second strand is FActScore, a factuality-evaluation methodology that decomposes generated text into atomic claims scored independently. Transposing this decomposition from model outputs to skill inputs is the methodological core of this approach, and it responds directly to the inadequacy of prior practice: "Instead of brute forcing and guessing what we can or should remove or add or how to fix degradations, we can have a guidance."
3. Core Analysis
3.1 Item-Based Scoring Methodology
A skill is split into discrete items, each paired with an expected question and answer. A typical skill decomposes into roughly 25-30 items, though some evaluation setups scale to approximately 400. Each item is tested twice - once without the skill loaded, once with it - and scored by a judge or panel of judges drawn from a model family different from the one under evaluation, to reduce bias. Questions are deliberately designed not to leak their answers, and can optionally be run with or without tool/web access to separate a model's internal knowledge from its practical, augmented performance.
This produces a four-category outcome matrix:
- No impact - known without and with the skill; already model knowledge.
- Degradation - known without the skill but answered incorrectly with it; the skill actively hurts performance.
- Genuine learning - unknown without the skill, known with it; the ideal outcome.
- Persistent gap - unknown in both cases; the skill misses needed concepts or is unclear.
Degradation proved rare in well-established skills - only 1 of roughly 400 items across Anthropic's popular skills fell into this category - but more common in smaller, less popular skills. Line- or sentence-level coloring can visualize which parts of a skill add new knowledge versus restate what the model already knows, giving authors a direct editing signal rather than relying on brute-force trial and error.
3.2 Lenient vs. Strict Scoring
Scoring is applied at two levels of rigor. Lenient scoring checks general idea or concept understanding; strict scoring checks exact details, parameters, and values. A substantial gap emerges in frontier models under this distinction: Opus drops from approximately 70% understanding under lenient scoring to approximately 22% under strict scoring. This indicates that frontier models generalize concepts well on their own but still derive concrete value from skills that supply exact, verifiable details and parameters - the kind of content least substitutable by the model's prior knowledge.
3.3 Cross-Model and Cross-Skill Comparisons
The methodology was applied across 29 skills using a combination of Sonnet and Haiku calls, with Sonnet serving as judge, at a total cost of approximately $14. Smaller models consistently showed lower baseline understanding of skill content than larger models - Sonnet understood more of a given skill's content unaided than smaller models did. Understanding also varied significantly from skill to skill, since some skills are intentionally narrow and directional rather than designed to generalize. Comparisons were additionally run across three popular skill "wrappers" and across different underlying models, confirming that novelty is not a fixed property of a skill but an interaction between skill content and the model consuming it.
4. Technical Insights
Several implementation considerations follow from these findings. First, novelty scoring functions only as an initial optimization signal; it is not sufficient on its own, and changes to a skill must still be validated against actual task-solving evals, since removing "known" content could still damage downstream performance in ways item-level scoring alone would miss. Second, content classified as redundant - already known to the model without the skill - can typically be trimmed to reduce token footprint while preserving eval scores, directly addressing the context-cost concern raised by two-arm evaluation. Third, judge selection matters: drawing the panel from a different model family than the model under test reduces the risk that shared blind spots inflate apparent understanding. Fourth, toggling tool/web access during testing isolates internal model knowledge from augmented performance, which is relevant when deciding whether a skill substitutes for retrieval or genuinely encodes knowledge the model cannot otherwise obtain.
All measurements in this work were performed using Docume, a framework for comparing agent and context artifacts - skills, AGENTS.md files, and MCP tools alike. Beyond novelty scoring, Docume also supports context impact assessment, routing evaluation, and comparison of skill descriptions and naming, situating novelty scoring as one module within a broader context-engineering evaluation suite rather than a standalone technique.
5. Discussion
These findings carry implications beyond individual skill authorship. The low cost of full-suite evaluation - $14 across 29 skills - suggests that systematic, deterministic skill validation is economically trivial relative to the effort currently spent on manual tuning and user-feedback-driven iteration. This reframes skill maintenance as an evaluation-first discipline analogous to regression testing in conventional software, rather than an intuition-driven craft.
The lenient-versus-strict gap also points to an underexplored dimension of model capability: general conceptual competence and exact parametric recall diverge sharply even in frontier models, and this divergence is precisely where skills add the most defensible value. This suggests that skill authors should prioritize encoding exact values, thresholds, and parameters over restating conceptual explanations the model already generalizes well, since the latter contributes disproportionately to context bloat without commensurate benefit.
An open question concerns stability across model upgrades. The stated motivation for this methodology is that brute-force modification and user feedback are unreliable as models evolve, implying that novelty profiles computed today may shift as new model versions are released - making periodic re-evaluation, rather than one-time validation, the more defensible practice.
6. Conclusion
This analysis demonstrates a deterministic, item-level methodology for evaluating knowledge-oriented agent skills, adapting FActScore's claim decomposition from generated text to skill content itself. By comparing model performance with and without a skill across discrete, judged items, authors can distinguish genuine novel knowledge from redundant restatement and rare but real degradation, while separately tracking conceptual versus parametric understanding. The practical takeaway is that skill maintenance need not rely on intuition or brute-force editing: inexpensive, repeatable evaluation can guide trimming, flag harmful instructions, and inform how skills should adapt as underlying models change.
Sources
- Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel - Original Creator (YouTube)
- Analysis and summary by Sean Weldon using AI-assisted research tools
About the Author
Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.
Sean Weldon builds apps and custom software at 22Atlas.