Don't Ship Skills Without Evals - Philipp Schmid, Google DeepMind

Skills should never be shipped to production without comprehensive evaluations, as non-deterministic agent behavior makes it impossible to determine if failu...

By Sean Weldon

Don't Ship Skills Without Evals: A Systematic Framework for Evaluating Agentic AI Skills

Abstract

This synthesis examines the critical gap between skill deployment and evaluation in agentic AI systems, arguing that skills should never reach production without comprehensive testing frameworks. Analysis of over 50,000 indexed skills reveals that most lack proper evaluations, creating uncertainty about whether task failures stem from poor skill design or fundamental model limitations. The research presents a systematic evaluation methodology encompassing skill architecture patterns, best practices for skill authoring, and rigorous testing protocols. Key findings demonstrate that human-written skills outperform AI-generated alternatives by approximately 15% across diverse tasks, while proper evaluation frameworks combining regex-based assertions and LLM-as-judge approaches achieve 90% validation accuracy. The work establishes actionable guidelines for skill lifecycle management, from creation through retirement, with implications for production AI agent reliability and performance optimization.

1. Introduction

The proliferation of skills - modular instructions that augment large language model (LLM) capabilities - has created a fundamental challenge in agentic AI systems: the absence of systematic evaluation practices. Skills function as contextual enhancements that guide model behavior for specific tasks or domains, yet empirical evidence from Skill Bench reveals a troubling reality. Among over 50,000 indexed skills, nearly all lack formal evaluations, with the majority being AI-generated and untested. This evaluation deficit represents a critical infrastructure vulnerability in production AI systems.

The challenge is compounded by the non-deterministic nature of agent behavior, which obscures the root causes of task failures. When an agent fails to complete a task, engineers face an attribution problem: did the failure occur because the skill provided inadequate guidance, because the underlying model lacks necessary capabilities, or because the skill triggered inappropriately for the given context? Without rigorous evaluation frameworks, these questions remain unanswerable, leading to unreliable production deployments.

Furthermore, a critical distinction exists between agents used by engineers who understand the available skills and agents built for consumers who lack such awareness. This asymmetry in skill visibility necessitates different evaluation approaches and quality standards. The central thesis posits that skills represent critical infrastructure components requiring evaluation rigor comparable to traditional software testing, with formalized testing protocols governing their entire lifecycle from creation through retirement.

2. Background and Related Work

2.1 Skill Architecture and Information Layering

Skills employ a progressive disclosure pattern that layers information according to computational cost and contextual relevance. The architecture consists of three hierarchical components: title and description metadata, the skill body containing detailed instructions, and reference files providing deep contextual information. This layering reflects the economic reality that description tokens incur costs on every model invocation, typically consuming 100-200 tokens per call regardless of whether the skill activates. Consequently, skill bodies are only processed when triggered, while reference files provide optional deep context for complex scenarios.

2.2 Skill Taxonomy and Functional Categories

Two primary skill categories emerge from functional analysis. Capability skills teach models new abilities or compensate for current model limitations. These skills are inherently temporary, designed for retirement as base model capabilities improve through training and architectural advances. In contrast, preference skills encode domain-specific workflows, organizational standards, and company-specific preferences. These skills exhibit greater durability and require protection through comprehensive evaluation frameworks to prevent degradation during model updates.

A secondary classification distinguishes model-triggered skills (activated based on contextual analysis) from user-invoked skills (explicitly called for workflow execution). This distinction carries significant implications for evaluation strategy, as model-triggered skills require negative case testing to prevent inappropriate activation and over-triggering scenarios.

3. Core Analysis

3.1 The Evaluation Gap and Attribution Problem

Empirical analysis of Skill Bench data reveals a fundamental disconnect between skill usage and quality assurance. The platform indexed over 50,000 skills, yet comprehensive evaluations remain virtually nonexistent. This gap creates an attribution problem that undermines production reliability: when agent performance degrades, teams cannot determine whether failures stem from skill design deficiencies or fundamental model capability limitations.

The non-deterministic nature of agent behavior exacerbates this challenge. Unlike traditional software where deterministic execution enables straightforward debugging, agent behavior varies across identical inputs due to model sampling, context window dynamics, and skill triggering logic. This variability necessitates statistical testing approaches with multiple trial runs per test case, typically 2-6 iterations to account for stochastic behavior patterns.

3.2 Skill Authoring Best Practices and Performance Impact

Empirical evidence demonstrates that human-written skills consistently outperform AI-generated alternatives. Data from Skills Bench 1.1 indicates that skills improve performance by approximately 15% on average across 100 different tasks covering coding and productivity domains. However, AI-generated skills can negatively impact performance, suggesting that automated skill generation without human oversight introduces systematic quality problems.

Effective skill authoring requires adherence to several critical principles. First, skill files should remain below 500 lines to minimize context overhead and processing costs. Second, descriptions must employ directive language rather than passive information presentation. For example, "use the Interactions API if working on chat" proves more effective than "recommended for multi-chat applications." This directive approach provides clear actionable guidance rather than encyclopedic reference material.

Third, appropriate freedom levels must be established. Skills should define goals and constraints rather than prescribing step-by-step workflows, allowing models to leverage their reasoning capabilities while maintaining guardrails. Fourth, negative cases must be explicitly included in skill descriptions to prevent over-triggering. Without explicit guidance on when not to activate, model-triggered skills may fire inappropriately, degrading overall agent performance.

3.3 Comprehensive Evaluation Methodology

The recommended evaluation framework combines multiple testing approaches to achieve robust validation. Each skill requires 10-20 test prompts distributed across happy path scenarios (when to use) and negative cases (when not to use). Production traces provide invaluable test data, capturing real-world usage patterns that synthetic examples may miss.

For validation, the methodology employs two complementary approaches. Regex-based assertions handle simple checks including correct SDK usage, appropriate model versions, correct method invocation, and pattern compliance. This approach offers computational efficiency and deterministic validation for straightforward requirements. For complex skills requiring trace analysis, LLM-as-judge evaluation with explicit rubrics provides nuanced assessment of agent behavior and output quality.

A practical implementation example demonstrates this methodology's effectiveness. The Gemini Interactions API skill employed 117 test cases derived from real user data and synthetic examples, achieving 90% performance for generating valid API code. The test structure included prompt specification, language parameters (TypeScript or Python), a should_trigger boolean flag, and an expected_checks array for validation assertions. A basic Python script running the Gemini CLI validated outputs through regex assertions checking SDK correctness, model versions, and method usage patterns.

3.4 Continuous Integration and Regression Testing

Production deployment requires continuous evaluation infrastructure. At Google DeepMind, evaluations are stored alongside every skill with multiple test cases executed in isolated workspaces featuring defined environments and startup commands. This isolation prevents agent "cheating" by accessing skill definitions during testing, ensuring that evaluations measure actual skill effectiveness rather than meta-reasoning about test structure.

Critical to this approach is regression testing on every skill modification. Changes are blocked if they fail to improve test performance or add new evaluation coverage. This policy ensures that skill evolution proceeds only when demonstrable improvements occur, preventing degradation through incremental changes. Testing must span multiple harnesses including Claude, Cursor, Gemini, and Codex rather than optimizing for a single platform, ensuring cross-platform reliability.

The evaluation philosophy emphasizes measuring outcomes rather than execution paths. Skills may trigger at different points during task execution, and agents may accomplish goals through various approaches. Consequently, evaluations should focus on task achievement regardless of skill trigger timing or specific implementation paths chosen by the agent.

4. Technical Insights

4.1 Implementation Considerations

Several technical parameters emerge as critical for effective skill evaluation. Description tokens represent a constant cost paid on every model invocation, necessitating careful optimization of this high-frequency component. Skill bodies should leverage the progressive disclosure pattern, providing detailed instructions only when triggered rather than front-loading all information into descriptions.

For statistical reliability, 2-6 trial runs per test case account for model non-determinism while balancing computational costs. This range provides sufficient samples to detect consistent patterns without excessive resource consumption. Test case construction should include both the should_trigger flag to validate appropriate activation and expected_checks arrays to verify output correctness through programmatic assertions.

4.2 Skill Lifecycle Management and Retirement

Skills exhibit finite lifespans, particularly capability skills that compensate for model limitations. The recommended approach employs ablation testing, continuously evaluating performance with and without skill enablement. When models achieve comparable performance without triggering the skill, retirement becomes appropriate. Critically, evaluations should be retained after skill retirement to detect performance degradation, enabling skill reintroduction if subsequent model updates reduce capability.

This lifecycle management reflects the empirical observation that models improve faster than initially expected. Skills developed six months prior may become obsolete as base model capabilities advance through training improvements and architectural innovations. However, preference skills encoding organizational workflows and domain-specific standards require longer retention, protected by comprehensive evaluation frameworks that ensure consistency across model updates.

5. Discussion

The analysis reveals that skill evaluation represents a fundamental requirement for production AI agent reliability rather than an optional quality enhancement. The 15% average performance improvement demonstrated by properly implemented skills across diverse tasks justifies the evaluation investment, particularly given the potential for AI-generated skills to negatively impact performance without proper validation.

The distinction between capability skills and preference skills carries important implications for organizational strategy. Capability skills require continuous monitoring for retirement opportunities as models improve, while preference skills demand protection through rigorous evaluation to preserve organizational knowledge and workflow optimization. This taxonomy suggests different governance models and lifecycle management approaches for each category.

Several areas warrant further investigation. The optimal ratio of happy path to negative case test prompts remains an open question, as does the appropriate balance between regex-based assertions and LLM-as-judge evaluation for different skill types. Additionally, the cross-platform testing requirement raises questions about standardization efforts and whether platform-agnostic skill specifications might reduce evaluation overhead while maintaining reliability guarantees.

6. Conclusion

This synthesis establishes that comprehensive evaluation frameworks represent a prerequisite for production skill deployment in agentic AI systems. The empirical evidence demonstrates that human-written skills with proper evaluation achieve significant performance improvements, while unevaluated or AI-generated skills risk degrading agent reliability. The methodology presented - combining 10-20 test prompts across happy path and negative cases, regex-based assertions for simple validation, LLM-as-judge for complex scenarios, and continuous regression testing - provides a practical framework for organizations deploying agentic systems.

Key practical takeaways include maintaining skill files below 500 lines, employing directive rather than passive language, including explicit negative cases to prevent over-triggering, and implementing ablation testing for retirement decisions. Organizations should establish continuous integration pipelines that block skill modifications failing to improve evaluation metrics, ensuring that skill evolution proceeds only with demonstrable quality improvements. As model capabilities continue advancing, the skill evaluation infrastructure established today will enable organizations to systematically retire obsolete capability skills while preserving valuable preference skills that encode domain expertise and organizational workflows.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub