How I automate my own job at Hugging Face using agents - Niels Rogge, Hugging Face

AI agents can effectively automate research artifact discovery and outreach workflows at scale, with open models now providing sufficient capability to repla...

By Sean Weldon

Automating Research Artifact Discovery Through AI Agents: A Production Case Study

Abstract

This synthesis examines the deployment of AI agents for automating research artifact discovery and outreach workflows at scale within the academic machine learning community. The work addresses the challenge of connecting published research with centralized repositories, specifically migrating models and datasets from distributed third-party services to Hugging Face's unified platform. The methodology evolved from deterministic workflow automation using predefined LLM API pipelines to fully autonomous agents employing tool-based reasoning with command-line interfaces. Key findings demonstrate that open-source models, particularly GLM 5.2, now match or exceed closed-source alternatives on relevant benchmarks while reducing costs. The system successfully generated thousands of automated GitHub issues with minimal negative feedback (two negative responses total), facilitating major dataset migrations exceeding 400GB and improving research discoverability across multiple organizations including Paddle OCR, Apple, and Google DeepMind.

1. Introduction

The contemporary machine learning research landscape produces hundreds of publications daily on platforms such as arXiv, creating substantial challenges in artifact management and discoverability. While peer-reviewed papers document methodological innovations, the associated computational artifacts - model weights, datasets, and implementation code - frequently reside on disparate third-party services including Google Drive, GitHub releases, Dropbox, and Zenodo. This fragmentation fundamentally impedes reproducibility and constrains the research community's capacity to systematically build upon published work.

Centralized artifact repositories address this challenge by providing unified platforms with structured metadata, standardized documentation through model cards and dataset cards, and filtering capabilities organized by task type, language support, and library compatibility. These platforms transform research artifacts from isolated files into discoverable, reusable components within a broader ecosystem. However, the migration of existing research artifacts to centralized repositories requires substantial manual effort in discovery, technical evaluation, and author outreach - a process that becomes fundamentally unscalable given contemporary publication volumes, particularly during major conference cycles such as NeurIPS.

This work presents a systematic approach to automating research artifact discovery and migration through AI agents, examining the technical evolution from deterministic workflows to autonomous agents, the transition from closed-source to open-source language models, and the critical evaluation practices necessary to prevent automated spam while maintaining outreach quality. The analysis demonstrates that properly evaluated AI agents can perform community-scale coordination tasks while achieving acceptance rates comparable to manual human effort.

2. Background and Related Work

2.1 Research Infrastructure and Documentation Standards

The Model Cards for Model Reporting framework established standardized documentation practices for machine learning models, enabling systematic communication of model characteristics, intended use cases, performance metrics, and limitations. Dataset cards provide analogous functionality for training data documentation. These structured approaches facilitate discovery through metadata tags that enable filtering by technical specifications such as depth estimation capabilities, language support, and library compatibility, transforming artifact discovery from ad-hoc web searches to systematic queries against structured metadata.

Centralized platforms leverage these documentation standards to create searchable repositories with paper pages directly linked to computational artifacts. This infrastructure substantially reduces friction in research replication and extension by providing unified access points with consistent metadata schemas. However, the voluntary nature of artifact submission creates a coordination problem: researchers optimize for publication acceptance rather than post-publication discoverability, resulting in artifact fragmentation across convenience-oriented storage services.

2.2 Agent Design Philosophies

Recent guidance from Anthropic's "Building Effective Agents" framework advocates for simplicity in initial agent implementations, recommending that developers avoid complex frameworks and begin with single LLM API calls in predefined pipeline steps - a deterministic workflow approach. This methodology prioritizes transparency and debuggability over architectural sophistication, enabling developers to identify failure modes and refine prompts before introducing autonomous decision-making capabilities.

The framework distinguishes between deterministic workflows, where LLM outputs feed into predetermined subsequent steps, and fully autonomous agents, where language models operate in iterative loops with tool access and self-directed reasoning. The evolution from workflows to autonomous agents represents a capability threshold: as model performance improves on reasoning benchmarks, the reliability of autonomous tool use increases sufficiently to warrant deployment in production systems.

3. Core Analysis

3.1 Workflow Architecture Evolution

The initial automation implementation employed a deterministic workflow replicating the manual research artifact discovery process. The pipeline executed five sequential steps: identifying GitHub repository URLs from arXiv papers, parsing README files for artifact locations, verifying whether artifacts existed on Hugging Face, programmatically opening GitHub issues or pull requests when artifacts were absent, and conducting follow-up communication with authors. This workflow operated as a nightly cron job using GitHub Actions' free tier, executing a Python script with LLM API calls at each decision point.

The system utilized LangFuse for observability, providing tracing capabilities for LLM inputs, outputs, prompts, associated costs, and latency metrics. This instrumentation proved critical for identifying failure modes and optimizing prompt engineering. The deterministic workflow successfully generated hundreds of GitHub issues nightly, demonstrating that structured LLM integration could scale manual processes by multiple orders of magnitude.

Following Anthropic's November AI Engineer workshop, which indicated that model capabilities had crossed the threshold for reliable autonomous operation, the architecture transitioned to a fully autonomous agent for follow-up communications. This agent employed the Claude agents SDK with Bash/CLI as its primary tool interface, specifically leveraging Hugging Face CLI skills for repository interactions. The autonomous agent approach reduced custom code requirements from thousands of lines to a minimal implementation consisting of a single CLI tool and associated skill definitions, demonstrating that model capability improvements enable architectural simplification rather than increased complexity.

3.2 Infrastructure and Deployment Strategy

The production system deployed on Modal using batch processing to instantiate parallel containers, with each container processing one GitHub issue as a separate agent loop. This architecture leveraged Modal's fast container startup times to enable overnight background processing of accumulated issues. The batch processing approach provided natural isolation between agent instances, preventing cross-contamination of context and enabling independent failure handling for individual outreach attempts.

Results posted to a Slack channel for human oversight, with manual invocation available through a Cursor skill labeled process_under_it_Modal that called the Composer 2.5 agent. This human-in-the-loop design maintained accountability while enabling scale, allowing community science team members to monitor agent behavior and intervene when necessary. The infrastructure choices prioritized operational simplicity over architectural sophistication, aligning with the principle that production agent systems should minimize rather than maximize technical complexity.

3.3 Model Selection and Performance Characteristics

The system initially utilized Claude models for both deterministic workflow steps and autonomous agent loops. However, subsequent evaluation indicated that open-source alternatives, particularly GLM 5.2, achieved competitive or superior performance on relevant benchmarks while reducing operational costs. Specifically, GLM 5.2 outperformed Claude Opus 4.8 on both Cursor Bench and Post-training Bench, two benchmarks relevant to code generation and instruction following capabilities critical for CLI-based agent operations.

The transition to open models leveraged Hugging Face inference providers, which offer unified OpenAI-compatible and Anthropic-compatible APIs across multiple backend providers including Together AI, Fireworks, and Cerebras. This abstraction layer enabled model switching without architectural changes, demonstrating that standardized API interfaces reduce vendor lock-in and facilitate continuous model upgrading as capabilities improve. The cost reduction from transitioning to open models proved substantial while maintaining or improving output quality, validating the hypothesis that open-source language models have reached production-grade reliability for structured agent tasks.

3.4 Quality Assurance and Spam Prevention

Out of thousands of automated GitHub issues generated, the system received only two negative responses from recipients, indicating that automated outreach quality approximated manual human communication standards. Notable successes included Paddle OCR migrating all OCR models to Hugging Face, Apple researchers and Google DeepMind teams proactively reaching out to publish datasets, and the initiation of dataset migrations exceeding 400GB in total size. One issue regarding Tiny Recursive Models generated over 60 upvotes, demonstrating community validation of the outreach value proposition.

The system intentionally does not disclose agent identity to recipients, operating under the principle that if automated content is indistinguishable from manual human outreach, disclosure may introduce bias causing premature issue closure. This design decision aligns with evaluation-centric quality assurance: the system underwent extensive testing using evaluation frameworks recommended in Hamel Husain's "LLM Evils FAQ" to ensure outputs met quality standards before deployment at scale. The framing of outreach as mutually beneficial - improving researcher discoverability while enabling others to build on published work - proved critical to acceptance rates.

Agent-generated model cards automatically populated Hugging Face template fields based on paper content analysis, GitHub README parsing, and PDF extraction. This automation reduced friction for researchers by minimizing manual documentation effort, transforming artifact migration from a multi-hour task to a simple approval workflow. Several instances demonstrated agent-to-agent interactions, where automated systems responded to agent-generated issues, indicating that agent communication patterns increasingly resemble human collaboration norms.

4. Technical Insights

The production deployment reveals several actionable technical findings for practitioners implementing agent-based automation systems. First, evaluation frameworks must precede scale deployment: the system's minimal negative feedback rate resulted from extensive pre-deployment testing rather than post-deployment iteration. Practitioners should implement comprehensive evaluation suites covering edge cases, tone analysis, and factual accuracy before enabling automated outreach at scale.

Second, architectural simplicity correlates with operational reliability: the transition from thousands of lines of custom code to minimal agent implementations with single CLI tools demonstrates that model capability improvements enable complexity reduction. Systems should prioritize simple tool interfaces over elaborate multi-tool architectures, as model reasoning capabilities increasingly handle coordination logic that previously required explicit code.

Third, open-source models now provide production-grade reliability for structured tasks: the successful transition from Claude to GLM 5.2 indicates that cost-performance trade-offs have shifted substantially in favor of open alternatives. Organizations should continuously evaluate open-source options rather than assuming closed-source superiority, particularly for tasks with clear evaluation metrics.

Fourth, batch processing with container isolation provides robust scaling: Modal's architecture enabled parallel agent execution without shared state complications, demonstrating that infrastructure choices significantly impact operational complexity. Systems processing multiple independent tasks should prioritize isolated execution environments over shared-state architectures.

Finally, observability infrastructure proves essential for production agents: LangFuse tracing enabled rapid identification of failure modes and prompt optimization opportunities. Production agent systems require comprehensive logging of inputs, outputs, costs, and latency to enable continuous improvement and cost management.

5. Discussion

The successful deployment of automated research artifact discovery agents demonstrates that AI systems can perform complex coordination tasks at community scale while maintaining quality standards comparable to manual human effort. This finding has broader implications for knowledge work automation: tasks involving standardized communication patterns, clear value propositions, and objective evaluation criteria represent viable automation targets using contemporary language models.

The transition from closed-source to open-source models reflects a broader industry trend toward commoditization of language model capabilities. As open models achieve parity with proprietary alternatives on specific benchmarks, competitive advantage shifts from model access to evaluation frameworks, domain-specific fine-tuning, and infrastructure optimization. Organizations should invest in evaluation capabilities and deployment infrastructure rather than assuming sustained closed-source model superiority.

The work also highlights the importance of framing in automated outreach: positioning agent-generated communications as mutually beneficial rather than extractive proved critical to acceptance rates. Future research should investigate optimal disclosure strategies for agent identity, examining whether transparency improves or degrades recipient responses across different contexts and communication types. The current approach - non-disclosure based on content indistinguishability - raises ethical questions that warrant systematic investigation.

The emergence of agent-to-agent interactions, where automated systems respond to other automated systems, suggests that future collaboration patterns may increasingly involve minimal human intermediation. This trend necessitates development of coordination protocols and quality assurance mechanisms that operate independently of human oversight, as manual review becomes infeasible at scale.

6. Conclusion

This work demonstrates that properly evaluated AI agents can automate research artifact discovery and outreach workflows at community scale while maintaining quality standards and avoiding spam. The evolution from deterministic workflows to autonomous agents, coupled with the transition from closed-source to open-source models, provides empirical evidence that language model capabilities have crossed critical thresholds for production deployment in structured coordination tasks.

Key contributions include validation of evaluation-first deployment strategies, demonstration of architectural simplification through model capability improvements, and empirical evidence of open-source model competitiveness in production contexts. The minimal negative feedback rate (two responses out of thousands of issues) and substantial positive outcomes (400GB+ dataset migrations, major organizational adoptions) indicate that automated outreach can achieve acceptance rates comparable to manual human communication when properly implemented.

Practitioners implementing similar systems should prioritize comprehensive evaluation frameworks, maintain architectural simplicity, continuously evaluate open-source model alternatives, and invest in observability infrastructure. The success of this deployment suggests that many knowledge work coordination tasks involving standardized communication patterns represent viable automation targets using contemporary language models, provided that evaluation and quality assurance receive appropriate emphasis during system design and deployment.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub