An Agent That Fixes Its Own Mistakes with Saloni Gandhi & Ziwei Guo, LinkedIn

Recursive self-improvement (RSI) at LinkedIn uses a closed loop of evaluation, root-cause reflection, and prompt optimization to continuously improve an AI a

An Agent That Fixes Its Own Mistakes: Recursive Self-Improvement in Production at LinkedIn

Abstract

This synthesis examines a production implementation of Recursive Self-Improvement (RSI) applied to a deployed user understanding agent, in which a closed loop of automated evaluation, root-cause reflection, and prompt optimization iteratively improves agent behavior. The approach decomposes improvement into seven stages - trace generation, judge evaluation, agentic reflection, edit proposal, validation, prompt evolution, and holdout confirmation - operating over the agent harness rather than underlying model weights. Analysis of approximately 180-200 human reviews revealed that judge-human alignment varied sharply by score band (55% at score 1 versus 80% at score 3), and that targeted rubric recalibration raised overall alignment from 57% to 64% while reducing mean error from 0.96 to 0.74. The central finding is that the trustworthiness of the evaluator and reflection layers, rather than the optimization mechanism, constitutes the binding constraint on RSI reliability. Practical implications include treating evaluation infrastructure as a prerequisite capability for scaling self-improving agentic systems.

1. Introduction

Agentic systems built on large language models are typically improved through manual iteration: engineers inspect failures, rewrite prompts, adjust tool interfaces, and redeploy. Recursive Self-Improvement (RSI) reframes this activity as an automatable closed loop in which production traces feed an evaluation engine, the engine's findings are diagnosed for root cause, and the resulting diagnosis drives candidate edits to the system under improvement.

Two components define the architecture under study. The target agent is the system being improved - in this case, a user understanding agent that infers member preferences from behavioral data. The improvement engine is the apparatus that drives change by evaluating behavior, identifying root causes, and proposing modifications. Critically, the stated objective is not merely to raise agent performance at a fixed point in time but to accelerate the rate of improvement: "The objective of the RSI system isn't simply to improve the agent's performance at a given point in time, it's to accelerate how quickly the agent gets better." This framing targets the first derivative of the performance curve rather than its instantaneous value.

The central thesis advanced here is that loop reliability is bounded by evaluator and reflection quality, not by the sophistication of the optimization mechanism. Because the evaluation stage identifies bottlenecks and determines which corrective actions are taken, a judge that is poorly aligned with human feedback propagates error through every downstream stage, regardless of how well the edit-proposal mechanism performs. This analysis establishes the scope of optimization (Section 2), describes the loop and its measured findings on evaluator and reflection quality (Section 3), extracts implementation-relevant technical insights (Section 4), and discusses implications for scaling RSI across an enterprise (Section 5).

2. Background and Related Work

Optimizing the base LLM itself is explicitly out of scope for this project. Attention is instead directed at the harness - the scaffolding surrounding the model - comprising four optimizable surfaces: planning and logic (orchestration and decision paths), context and memory (session state), tool interface (APIs and datasets), and artifact storage (outputs and logs). These surfaces admit intervention at escalating levels of complexity, forming an optimization ladder: L0 prompt updates represent the simplest intervention; above this sits structural context optimization; L1/L2 involves automated design of entire workflows (meta-level information design); and the highest rungs involve optimizing the harness code itself and ultimately the optimizer's own code.

The project under discussion currently operates at L0, with prompt optimization as the active focus and an explicit roadmap to ascend the ladder as confidence in the loop accumulates. This staging matters conceptually: each rung widens the action space available to the improvement engine, but also widens the space in which a flawed evaluator can cause damage - making evaluator trustworthiness a gating concern before the ladder is climbed further.

3. Core Analysis

3.1 The Seven-Step RSI Loop and Target Agent

The loop proceeds through seven stages: (1) running the agent to generate traces against real queries; (2) passing traces through an LLM-as-judge evaluation engine to produce scores; (3) agentic reflection to identify root cause among agent, model, prompt, tool-call, or judge/rubric failure; (4) generating edit proposals - currently restricted to prompt edits - via a method referred to as JIPA; (5) validating edits using the same evaluation engine and rubrics; (6) evolving the production prompt with validated edits; and (7) confirming gains on a holdout set before promotion to the new production baseline.

The target agent ingests a member ID to retrieve search history, profile data, location, and preference signals, and emits a structured output describing inferred attributes (e.g., preferred job titles), each paired with a confidence score, supporting citations, and explanatory reasoning. This output structure matters methodologically: because each field is independently verifiable against cited evidence, it creates the substrate that the judge can score and that human reviewers can audit for alignment.

3.2 Evaluation Quality: The L1 Judge

The judge scores agent outputs on a 1-3 scale (1 = not good, 2 = fair, 3 = very good) across 15 dimensions, with rationale notes per dimension. Analysis of roughly 180-200 human reviews found alignment with human judgment varied sharply by score: approximately 55% at score 1, 57% at score 2, and 80% at score 3 - indicating the judge was substantially better at recognizing strong outputs than at correctly identifying weak ones. Dimension-level alignment was similarly uneven, with "preferred job titles" the weakest dimension, and domain/expertise (~47%) and work type (~50%) also underperforming.

Root causes identified included the judge evaluating plausibility rather than factual completeness for job titles, unclear boundaries between domain, function, and skill categories, absent partial-credit rules, penalization of harmless formatting differences, and inconsistent assumptions about inferred preferences. Of 37 judgments flagged as overly harsh, 31 involved outputs scored as 1, pointing to a specific recalibration need at the low end of the scale. Following rubric fixes - ignoring harmless formatting differences, normalizing titles, and clarifying handling of concurrent employment, seniority, and geographic inference - human alignment rose from 57% to 64%, and mean error fell from 0.96 to 0.74.

3.3 Reflection Quality: The L2 Agent

Reflection operates in four sub-steps: clustering failing rows by dimension, applying an L2 judge to assign a root-cause tag per row, merging near-duplicate root causes into canonical issues, and ranking canonical issues by frequency. These canonical issues are cached to inform subsequent loop iterations and to generate scorecards for human review.

Root-cause attribution proved harder for reviewers to agree on than failure identification itself: human reviewers concurred more often when the L2 agent attributed a failure to the underlying model than when it attributed the failure to the rubric. A structural gap was identified in the original reflection taxonomy - it lacked a category for "the L1 judge itself was wrong," causing the reflection agent to default blame onto the rubric when the true issue was an incorrect first-pass judgment. Compounding this, the reflection agent sometimes reused the L1 judge's stated rationale without independently validating it against the agent's actual output, producing what the source material terms a "convincing but incorrect" explanation that risks modifying the wrong system component entirely.

4. Technical Insights

Several implementation-relevant patterns emerge from this work. First, score-band-specific alignment measurement (rather than a single aggregate alignment figure) exposes where a judge is unreliable; here it revealed the judge was systematically harsher and less trustworthy at the low end of the scale (55% vs. 80% alignment), a finding that would be masked by an averaged metric. Second, rubric recalibration - adding explicit partial-credit rules, normalizing superficial formatting variance, and clarifying ambiguous category boundaries - produced measurable gains (57%→64% alignment, 0.96→0.74 mean error) without touching the underlying judge model, suggesting specification quality is often a higher-leverage lever than model capability. Third, the reflection taxonomy must explicitly include "evaluator error" as a candidate root cause; omitting it creates a structural bias toward blaming the rubric or the target agent when the judge itself is the fault. Fourth, majority voting across repeated holdout judgments is used to dampen LLM non-determinism before a prompt update is promoted to production, a lightweight variance-control technique applicable to any LLM-judged validation gate. A limitation worth noting: validation and promotion remain human-in-the-loop by design, reflecting that automation of the optimization step has outpaced trust in the evaluation step - edit generation (JIPA) and prompt evolution are automated, but production promotion is not.

5. Discussion

The findings suggest a layered dependency structure for RSI maturity: a base agent must exist before it can be evaluated, evaluation must be trustworthy before reflection can meaningfully diagnose root cause, and reflection must be reliable before an automated optimization loop can be trusted to act without constant human oversight. This ordering implies that organizations attempting to build self-improving agents cannot skip directly to automated optimization; the evaluation and reflection layers are prerequisite infrastructure, not peripheral tooling.

The quote "many evaluation disagreements were not simply model failures, they were reflecting ambiguities in the rubric as well as the underlying evaluation policy" is significant because it reframes a large share of apparent judge errors as specification problems rather than capability problems. This has a direct practical corollary: investment in rubric clarity and taxonomy completeness (e.g., adding "judge error" as a root-cause category) can yield outsized returns relative to investment in judge model upgrades. This connects to a broader industry pattern in LLM-as-judge deployments, where alignment studies frequently surface rubric ambiguity as a dominant error source.

An open gap is how the five identified scaling requirements - standardized/calibrated judges, automated feedback loops, scalable candidate generation, validation and trust mechanisms, and democratized/reusable RSI components - will be sequenced as the project ascends the optimization ladder beyond L0 prompt edits into structural context and workflow-level optimization, where the action space (and the potential blast radius of an untrustworthy evaluator) grows substantially.

6. Conclusion

This work demonstrates that a closed-loop RSI system can measurably improve agent behavior through automated evaluation, reflection, and prompt optimization, but that the loop's reliability is gated by evaluator and reflection trustworthiness rather than by the sophistication of the optimization step itself. The practical takeaway for teams building similar systems is to prioritize evaluation infrastructure - explicit rubric design, complete root-cause taxonomies, and variance-control techniques like majority voting - before scaling automation or ascending to higher rungs of the optimization ladder. As organizations build analogous self-improving agents, the LinkedIn case suggests that measurable, score-band-aware alignment auditing of the judge layer should be a standard diagnostic practice before any optimization claims are trusted.

Sources

• An Agent That Fixes Its Own Mistakes - Saloni Gandhi & Ziwei Guo, LinkedIn - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub