'From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization

Automated agentic workflows can proactively identify and verify high-ROI performance optimization opportunities in production systems by combining runtime in...

By Sean Weldon

From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization

Abstract

Modern software teams face systematic uncertainty in performance optimization investigation phases, where discovery and analysis duration ranges unpredictably from hours to weeks. This research examines an automated agentic workflow system designed to proactively identify and verify high-return-on-investment performance optimization opportunities in production environments. The approach bridges production runtime intelligence with code-level reasoning through function-level instrumentation, utilizing Claude Code agents integrated with Model Context Protocol (MCP) and ClickHouse columnar databases. Key architectural innovations include a skills layer for consistent agent behavior, ROI-based scoring algorithms prioritizing hot path frequency and business impact, and autonomous verification mechanisms requiring 80-90% confidence thresholds. Findings demonstrate that context-driven automation can surface optimization opportunities invisible to reactive engineering workflows while maintaining human oversight through prioritized, verified recommendations. This work contributes methodological frameworks for production-to-code context bridging and establishes reliability thresholds necessary for autonomous agentic engineering.

1. Introduction

Performance optimization in production software systems has traditionally operated through reactive crisis management cycles. Engineering teams ignore performance degradation until system behavior crosses critical thresholds, apply emergency fixes under time pressure, then return to ignoring incremental performance issues - a pattern characterized as the leaky bucket pattern. This approach creates operational inefficiency, knowledge concentration in specific engineers who understand critical code paths, and systematic underinvestment in performance improvements that never reach crisis severity.

Recent empirical evidence compounds these challenges. Google's 2026 DORA (DevOps Research and Assessment) metrics reveal a paradox in AI adoption outcomes: while individual developer effectiveness increased measurably, organizations simultaneously experienced elevated software delivery instability and reduced team throughput. This finding suggests that acceleration of individual tasks without systemic workflow integration creates new bottlenecks and coordination failures at team scale.

The fundamental challenge lies in what this research terms investigation phase opacity. Engineering organizations can estimate implementation time for specific performance optimizations with reasonable accuracy, but cannot predict the duration required to discover, analyze, and validate optimization opportunities. As noted in the source material, "We would know how much time it takes to do a specific optimization, but we never know how long it's going to take to investigate it and to find it." This uncertainty ranges from one hour to multiple weeks, preventing rational resource allocation and causing high-value opportunities to remain systematically undiscovered.

This research addresses the central question: Can automated agentic workflows reliably identify, verify, and prioritize performance optimizations in production systems while achieving sufficient trust thresholds for autonomous operation? The analysis examines the technical architecture required for production-to-code context bridging, mechanisms for ensuring agent reliability and verification, and human-centered delivery approaches that respect reviewer time while building organizational trust in automated recommendations.

2. Background and Related Work

2.1 The Production Context Problem

Production monitoring systems exhibit a characteristic dual pathology: simultaneous information overload of low-signal data and critical gaps in actionable intelligence. Traditional observability metrics - CPU utilization, memory consumption, P90 latencies - operate at infrastructure granularity, fundamentally disconnected from the function-level code reasoning required for optimization decisions. This context gap prevents effective translation from observed system symptoms to specific code modifications.

The challenge extends beyond mere data volume. As the research notes, "Metrics don't connect to the way that an agent needs to reason about the code, which is at the function level." Agents operating on code require understanding of which specific functions consume resources, how they connect to user-facing endpoints or background processes, and what execution patterns correlate with performance degradation. Infrastructure metrics alone cannot provide this granularity.

2.2 Agent Reliability Thresholds and Agentic Engineering

Interactive agent usage and autonomous automation occupy fundamentally different reliability domains. When engineers employ agents interactively through IDE integrations or chat interfaces, accuracy thresholds of 60-80% prove acceptable because humans provide immediate correction and contextual judgment. However, agentic engineering - defined as autonomous workflows operating without real-time human supervision - requires 80-90% confidence thresholds before recommendations reach human reviewers.

This distinction necessitates architectural approaches emphasizing verification, grounding in actual system behavior, and deterministic output over raw model capability. The research emphasizes that "agentic engineering requires 80-90% trust threshold for autonomous automation - fundamentally different from interactive agent use." Achieving this reliability demands systematic solutions to agent failure modes including plausible but unverified suggestions, superficial fixes that address symptoms rather than root causes, and inconsistent reasoning across similar scenarios.

2.3 DORA Metrics and AI Adoption Patterns

The DORA framework measures software delivery performance through deployment frequency, lead time for changes, change failure rate, and time to restore service. Recent findings indicate that AI tool adoption increased individual task completion speed while simultaneously degrading team-level delivery stability and throughput. This pattern suggests that optimizing individual workflow components without addressing systemic coordination creates new failure modes, particularly when teams experience what the research describes as feeling "more effective individually, but as a team, our throughput is the same and our software breaks more often."

3. Core Analysis

3.1 Production-to-Code Context Bridging Methodology

The research introduces a prod-to-code approach that captures function-level performance overhead connected to specific endpoints, event consumers, and scheduled jobs. Rather than collecting exhaustive traces for all requests, the system implements threshold-based forensic evidence collection, triggering detailed instrumentation only when requests exceed P99 latency thresholds or custom-defined performance boundaries.

This selective instrumentation addresses the context problem by providing agents with precisely the granularity required for code-level reasoning while avoiding data volume overwhelm. The implementation utilizes Model Context Protocol (MCP) integration to capture runtime intelligence showing which functions consume resources during degraded performance scenarios. Critically, this data connects infrastructure-level symptoms (elevated latency) to code-level causes (specific function execution patterns), enabling agents to reason about optimization opportunities with production evidence.

The architecture employs a HUD (heads-up display) visualization layer that makes forensic data comprehensible to human engineers, ensuring that automated analysis remains interpretable and verifiable. This design choice reflects the principle that autonomous systems must produce outputs that humans can validate efficiently, respecting reviewer time while maintaining oversight.

3.2 Skills Layer and Agent Consistency

Agent reliability challenges emerged across three primary failure modes during system development. First, plausible unverified suggestions occurred when agents generated syntactically correct and logically coherent recommendations that nonetheless failed when implemented. Second, complex query patterns proved problematic when agents attempted to query ClickHouse columnar databases using traditional relational database patterns, producing incorrect results. Third, lazy fixes manifested when agents suggested exception catching or superficial changes rather than addressing root causes.

The solution implements a skills layer - reusable query patterns and analysis methodologies that encode domain expertise. Specific skills include error origin analysis, memory spike correlation to pod activity, missing index detection, artificial delay identification (timeouts, sleeps), and sequential assignment detection. This layer serves dual purposes: reducing variance in agent behavior across similar scenarios and embedding senior engineer methodology into automated workflows.

The research emphasizes that "it's not just the understanding of the data and how to connect it. It's also being quite thorough on what we want to do in that process and building the playbook of how a senior engineer would do that." This insight highlights that effective agentic systems require not only appropriate context but also structured reasoning frameworks that guide analysis toward high-quality outcomes.

3.3 Autonomous Verification and ROI-Based Prioritization

The system implements autonomous verification before human review, addressing the trust threshold requirements for agentic engineering. When agents identify optimization opportunities and generate fixes, the workflow automatically reruns affected tests and measures runtime impact on the specific flows being optimized. This verification ensures that "the human gets something after we already detected it," providing evidence of both correctness and measurable improvement.

Prioritization employs a scoring algorithm incorporating three dimensions: hot path metrics (invocation frequency), business impact classification (proximity to revenue-generating flows like payments and sign-ups), and change risk assessment (migration requirements, architectural complexity). The system explicitly prioritizes highest ROI opportunities rather than highest absolute impact, balancing business value against engineering effort required.

This prioritization philosophy reflects the principle that "not everything worth fixing is worth the engineering cost." Rather than generating dozens of automated pull requests, the system delivers one carefully prioritized recommendation at a time, building team appetite and habit while respecting cognitive load. The research notes this approach as critical for organizational adoption: "Start with one recommendation at a time to build team appetite and habit."

3.4 Vendor-Neutral Infrastructure Architecture

The technical implementation maintains vendor neutrality across compute providers, model choices, and execution harnesses. GitHub Actions orchestrates weekly scheduled workflows and webhook-based triggers (responding to SLO breaches or custom events). The Claude Code agent serves as the current reasoning engine, but the architecture supports model substitution as capabilities evolve.

Reports deliver to Slack channels with evidence packages including production traces, verified fixes, business impact scoring, and human-readable explanations. This delivery mechanism emphasizes maintainability and feedback loops, enabling workflow evolution as organizational expectations change. The architecture explicitly supports the principle that "autonomous automation must verify fixes in runtime and confirm business value before human review to respect reviewer time."

4. Technical Insights

4.1 Implementation Architecture

The system architecture comprises four layers: data collection, query language, skills, and automations. The data layer implements function-level instrumentation with P99 threshold-based forensic collection using Model Context Protocol integration. The query layer provides a custom HUD query language optimized for ClickHouse columnar database patterns, abstracting complexity from agent reasoning.

The skills layer encodes reusable analysis patterns including N+1 query detection, missing index identification, and artificial delay discovery. The automations layer combines these components to enable auto-fixing workflows, dead code detection, and performance improvement identification. This layered approach enables incremental capability expansion while maintaining system reliability.

4.2 Performance Optimization Targets

The system targets specific optimization categories with measurable runtime impact: artificial delays and timeouts, unnecessary sleep calls, N+1 database queries, missing indexes on frequently queried columns, and sequential assignments that could be parallelized. These categories were selected based on frequency in production systems and clarity of verification - each optimization type produces measurable latency reduction that can be automatically validated.

4.3 Trade-offs and Limitations

The approach requires instrumentation overhead for function-level performance capture, though threshold-based collection mitigates this impact by activating detailed tracing only during degraded performance scenarios. The system depends on comprehensive test coverage for autonomous verification, limiting applicability in codebases with sparse testing. Additionally, the 80-90% confidence threshold means some valid optimizations may be filtered out to maintain reliability, potentially missing opportunities that human engineers would recognize.

The weekly automation cadence balances discovery frequency against organizational change absorption capacity. More frequent runs could identify issues faster but risk overwhelming teams with recommendations, while less frequent runs might miss time-sensitive optimization opportunities.

5. Discussion

The research demonstrates that context over cleverness represents the critical success factor for agentic engineering systems. Raw model capability proves insufficient without appropriate production context, domain-specific skills, and verification mechanisms. The finding that "right context, skills, and domain knowledge make agents dramatically more useful than raw model capability" challenges approaches that focus primarily on model scale or architectural sophistication.

The work surfaces a broader insight about agent utility beyond acceleration: "Agents surface opportunities beyond acceleration: automate phases that don't happen in day-to-day engineering." This suggests that the highest-value applications of autonomous systems may not be speeding existing workflows but rather enabling analysis that teams systematically deprioritize due to investigation phase uncertainty. Performance optimization represents an ideal domain for this approach because opportunities remain invisible until investigated, yet investigation effort cannot be justified without knowing potential impact.

The requirement for 80-90% confidence thresholds in autonomous systems establishes a design constraint that shapes architectural choices throughout the system. This threshold necessitates verification loops, skills layers for consistency, and ROI-based filtering to ensure that recommendations reaching human reviewers merit attention. Organizations implementing similar systems must design for this reliability level from inception rather than attempting to retrofit verification onto systems designed for interactive use.

Future research directions include extending the methodology to security vulnerability detection, technical debt identification, and architectural improvement recommendations. Each domain presents similar characteristics: high-value opportunities that remain undiscovered due to investigation costs, measurable impact metrics for verification, and potential for automated analysis with human oversight. Additionally, investigation of organizational adoption patterns - how teams build trust in autonomous recommendations and integrate them into existing workflows - represents critical work for practical deployment.

6. Conclusion

This research presents a comprehensive methodology for automated agentic performance optimization that addresses the fundamental challenge of investigation phase uncertainty in software engineering. By bridging production runtime intelligence with code-level reasoning through function-level instrumentation, implementing skills layers for agent consistency, and maintaining autonomous verification with ROI-based prioritization, the approach achieves the 80-90% confidence thresholds necessary for agentic engineering deployment.

The key contributions include: (1) production-to-code context bridging methodology using threshold-based forensic collection, (2) skills layer architecture for consistent agent behavior encoding senior engineer methodology, (3) ROI-based scoring algorithms balancing business impact against implementation risk, and (4) autonomous verification mechanisms that respect reviewer time while maintaining human oversight. These components collectively enable proactive identification of optimization opportunities that would remain invisible in reactive engineering workflows.

Practical applications extend beyond performance optimization to any domain where investigation costs prevent discovery of high-value improvements. Organizations implementing autonomous agentic systems should prioritize context quality, verification mechanisms, and human-centered delivery over raw model capability. The finding that individual acceleration without systemic integration degrades team outcomes emphasizes the importance of designing autonomous systems that enhance rather than disrupt existing coordination patterns. As AI capabilities continue advancing, the principles established here - context over cleverness, verification before review, and ROI-based prioritization - provide foundational guidance for building trustworthy autonomous engineering systems.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub