Agents That Self Improve While You Sleep with Fuad Ali, Arize AI

Arize's Signal agent closes the self-improvement loop for AI agents by continuously reading production traces, clustering failures, proposing evals and code

Agents That Self-Improve While You Sleep: Closing the Production Feedback Loop for AI Agents

Abstract

Production AI agents fail in ways that resist classical debugging: non-determinism and a combinatorially large action space make failure diagnosis roughly an order of magnitude harder than for conventional software. This synthesis examines Arize Signal, an always-on triage agent designed to close the self-improvement loop for production agents by reading traces on a schedule, clustering related failures into single tickets, correlating traces against source code, proposing evaluators and code fixes, and validating those fixes through replay experiments. Findings are drawn from a case study of an instrumented shopping assistant containing a planted category-matching defect that inflated latency to 14-17 seconds. The analysis presents a five-stage observability maturity model, distinguishes evaluator classes by cost and expressiveness, and identifies the specific judgment points - PR review, dataset curation, eval design, and annotation - where human attention retains the highest leverage. Practical implications center on reallocating human effort from reading raw traces to governing automated findings.

1. Introduction

Organizations now operate thousands of agents spanning internal tooling, coding assistance, marketing, and customer-facing surfaces. The resulting volume of execution data exceeds human review capacity, producing a reactive operational posture in which failures surface through alerting rather than systematic review. A widely cited illustration is a Canadian car dealership chatbot that agreed to sell a vehicle for five dollars, an obligation the dealership ultimately declined to honor. The incident demonstrates that agent failures carry commercial consequences, not merely technical ones, and that the gap between "PagerDuty at 2am" response and proactive self-diagnosis has real cost.

Two terms anchor this analysis. A trace is the complete tree of an agent's tool calls and model interactions for a single request. An eval is a scoring function that determines whether the agent did the right thing, as distinct from merely recording what it did. A living fixture is a dataset of captured failures that grows continuously, functioning as an evolving regression suite rather than a static, one-time test definition.

The central thesis is that the labor of reading production data should be delegated to an agent, while human judgment is reallocated to the highest-leverage decision points: reviewing proposed fixes, curating which failures enter the fixture dataset, designing evaluators, and annotating ambiguous outputs. Section 2 establishes the maturity model and evaluator taxonomy underlying this approach. Section 3 analyzes the self-improvement loop through a worked case study. Section 4 synthesizes technical insights and trade-offs. Section 5 discusses limitations and open questions, and Section 6 concludes.

2. Background and Related Work

A five-stage maturity model organizes the capability progression observed across organizations deploying agents in production: (1) tracing, capturing the full trace tree of tool and model calls; (2) evals, scoring correctness rather than merely logging behavior; (3) automated trace review, delegating review to an agent because no human can read millions of traces; (4) failure capture as tests, persisting every detected failure as a regression case; and (5) fix validation, proving experimentally that a proposed remedy resolves the issue without introducing regressions.

A live audience poll conducted during the presentation indicated that most practitioners occupy stages one and two, with few having reached automated remediation or regression capture. This gap positions the maturity model as descriptive of an aspiration rather than of prevailing practice, and frames the subsequent case study as a demonstration of what stages three through five look like in operation. Instrumentation throughout the case study is provided by the Arize AX CLI, an open-source tool offering guided setup and auto-instrumenters for major model providers and agent frameworks, producing per-tool-call spans with arguments, results, and prompt token counts grouped into sessions.

3. Core Analysis

3.1 Case Study: A Category-Matching Defect

The demonstration agent - a shopping assistant ("Wonder Toys") built on FastAPI, Next.js, ChromaDB, and agent SDKs, with traces exported to Arize AX - contained a planted defect. Product categories were persisted as lowercase, hyphenated slugs, while the model passed human-readable, title-case, space-separated category names. This mismatch caused filter lookups to fail silently, triggering a fallback that scanned the entire product catalog rather than a filtered subset. The defect manifested in traces as three sequential, redundant search and LLM calls, pushing end-to-end latency to 14-17 seconds. This case illustrates a broader pattern: agent defects often originate not in model reasoning but in brittle interfaces between model output and downstream data structures, and such defects are only legible by inspecting the full trace tree alongside source code.

3.2 The Signal Loop: Detection, Diagnosis, and Remediation

Signal operates as an always-on agent that reads traces on a default six-hour schedule or in response to alert triggers (e.g., a latency monitor crossing 10 seconds). Rather than filing one ticket per anomalous trace, it clusters related issues into a single ticket, mirroring how a human reviewer would consolidate recurring symptoms. Because Signal is connected to the agent's source repository via GitHub, it correlates trace-level symptoms against code to identify root causes rather than merely surfacing statistical anomalies. In the demonstration, Signal required six minutes to identify the slug-casing root cause and open a corrective pull request. For each flagged issue, the interface offers single-click actions: resolve, ignore, add to a dataset, create an eval, or open a GitHub issue or PR, which keeps the human interaction surface narrow despite the underlying analysis being extensive.

3.3 Evaluator Taxonomy and the Living Fixture

Four evaluator classes are distinguished by cost and expressiveness: code/deterministic checks, which apply exact rules cheaply; LLM-as-judge, which performs qualitative assessment at the cost of one additional model call; agent-as-judge, the most expensive, used for open-ended goal evaluation; and decision models, a class of fast, cheap classifiers (released roughly one month before the presentation) that return yes/no/choice/score outputs suited to high-volume, low-latency scoring. In the case study, a live-authored eval instructed an LLM judge to detect "redundant search product retries" by inspecting tool-call trajectories - illustrating how qualitative failure patterns can be codified into reusable scoring prompts.

Captured failures are added to a living fixture dataset, which functions as a continuously growing regression suite rather than a fixed test file, analogous to how a learned skill like bicycle riding becomes ingrained rather than explicitly reasoned through on each attempt. Evaluators are sampled rather than run against full traffic, and retired once an issue's eval consistently passes (e.g., at a 99.999% healthy rate) to control token spend - treating eval coverage itself as a resource to be allocated rather than a fixed cost.

3.4 Fix Validation Through Experiments

Proposed fixes are validated via experiments, which rerun the agent against the living fixture dataset with the candidate fix applied and score outputs against the relevant evals. In the demonstration, a fix experiment achieved an 88.9% pass rate, judged sufficient to ship. Experiments additionally guard against regressions on previously passing evals when, for example, a system-prompt change is introduced - making fix validation a joint test of root-cause resolution and non-regression. A noted open problem is how to safely reproduce external dependencies (databases, third-party tools) during experiment replay; simple agents like Wonder Toys can respin a database and rerun end-to-end, but more complex production systems with stateful external integrations present an unresolved engineering challenge.

4. Technical Insights

Several implementation details carry direct operational relevance. First, Signal's default six-hour polling interval is configurable and can be preempted by alert-driven triggers, meaning review cadence should match a system's actual failure-rate profile rather than a fixed default. Second, decision models' confidence scores enable automatic routing of ambiguous cases to human annotation queues; in a spam-detection benchmark cited as evidence, 4.6% of cases fell into a 0.3-0.7 confidence band and were routed to humans, preserving 99.5% accuracy on the remainder - demonstrating that confidence-based routing can concentrate human effort on a small, high-value fraction of cases. Third, Signal is extensible via custom skills (e.g., DataDog integration, permission-boundary checks, RBAC systems) wherever a callable API exists, and supports bring-your-own execution harnesses (Codex, Cursor, Claude Code, Open Code) and sandbox providers (Modal, Daytona, E2B), avoiding lock-in to a single execution environment. Fourth, because Signal's own execution is traced, the same review loop can be applied reflexively to Signal itself - a dogfooding pattern using proxy metrics such as PR merge rate, session handoff rate, and dataset/eval creation rate.

5. Discussion

The central design tension in this approach is between automation coverage and the risk of "vibing with the agent's output" - accepting automated findings without ground truth, which the source material explicitly flags as a failure mode of meta-evaluation taken too far. This suggests that self-improvement loops are not self-validating in the limit; without external verification, cost and effort can be spent increasing apparent activity (tickets closed, PRs opened) without corresponding gains in real system quality. The explicit retention of human review for PR merge decisions, dataset curation, and eval design functions as a check against this risk, embedding human taste into otherwise automated pipelines at defined checkpoints rather than removing it entirely.

A second open question concerns resource allocation: the source material notes there is no universal rule (such as "5% of agent operating budget") governing how much to spend on self-improvement infrastructure relative to the agent's own operating cost, though cost comparisons are exposed in-platform. This gap indicates that self-improvement budgeting remains an empirical, per-organization decision rather than a settled practice, and is a candidate area for future standardization as more organizations adopt comparable loops.

Finally, the shift from post-hoc evaluation (reviewing traces after an agent has already acted) toward execution-time evaluation using fast decision models represents a forward-looking direction. Routing and intervention decisions made in-flight, rather than after the fact, would shift the self-improvement loop from a purely corrective function toward a preventive one, though this capability is described as nascent rather than established practice.

6. Conclusion

This analysis has examined a concrete implementation of a closed self-improvement loop for production AI agents, structured around a five-stage maturity model progressing from tracing through fix validation. The Wonder Toys case study demonstrates that root-cause diagnosis, fix proposal, and experimental validation can be substantially automated, provided the agent has access to both production traces and source code. The evaluator taxonomy - spanning deterministic checks, LLM-as-judge, agent-as-judge, and decision models - offers a practical framework for matching evaluation cost to failure-detection needs at scale.

The principal practical takeaway is that human attention is best concentrated at a small set of high-leverage checkpoints - PR approval, dataset curation, eval design, and low-confidence annotation - rather than spread across raw trace review, which does not scale with agent deployment volume. Organizations building or operating production agents at scale should treat observability maturity as a staged investment, with automated trace review and living fixtures as the natural next steps beyond basic tracing and evaluation.

Sources

• Agents That Self Improve While You Sleep - Fuad Ali, Arize AI - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub