Better AI Models Won't Fix Your Engineering Team - Nicholas Arcolano, Jellyfish

AI output gains are driven far more by how engineers and organizations use AI tools (skill use, consumption, team/company culture, individual ability) than b...

Abstract

This synthesis examines an empirical investigation into the determinants of software engineering output in the era of generative artificial intelligence. Drawing on telemetry from approximately 50,000 engineers across 447 companies using Cursor and Claude Code, together with roughly 500,000 developer-weeks of behavioral data, the analysis decomposes variance in merged pull requests (PRs) per engineer per week into five explanatory factors: tool/model choice, skill use, consumption, company and team environment, and individual engineer characteristics. The central finding is that model and tool selection explains less than 1% of observed variation, while individual and organizational factors jointly account for the overwhelming majority. A natural experiment arising from an exogenous shift in model usage corroborates this null result. The findings suggest that engineering leaders should redirect attention from procurement decisions toward culture, workflow design, and skill development.

1. Introduction

The rapid proliferation of AI-assisted development tools has produced a procurement-centric discourse within engineering organizations. Decisions about which frontier model to adopt - Opus versus Sonnet, ChatGPT versus alternatives - and which integrated development environment to standardize on (Cursor versus Claude Code) consume substantial leadership attention, often under the implicit assumption that better models translate directly into better output.

Several terms require definition. Tool choice refers to the specific model or client application an engineer employs. Skill use denotes structured, reusable AI capability invocations, measured as runs per engineer per week. Consumption refers to the volume of AI usage, operationalized through token counts. Environment captures company- and team-level attributes including culture, codebase characteristics, and code review process. The primary dependent variable throughout is merged PRs per engineer per week, selected as a proxy for delivered throughput.

The central thesis is that output gains derive far more from how engineers and organizations use AI than from which AI they use. As stated in the source material: "If you're not getting the outcomes that you want with AI, chances are it's actually not the model, the problem is you." Section 2 establishes the conceptual framework. Section 3 presents the core variance decomposition and supporting evidence. Section 4 extracts actionable technical findings. Sections 5 and 6 discuss implications and limitations.

2. Background and Related Work

The analysis adopts a golf analogy framework to partition sources of performance variation into four categories: clubs correspond to tools and models, swing technique corresponds to skill, the course corresponds to the organizational environment, and the player corresponds to the individual engineer. This framing is instructive because equipment effects in golf are real but bounded. Average PGA driving distance was 257 yards in 1980 and has risen approximately 20% since, with a discrete 15-yard jump around 2000 during the solid core ball era, attributable to materials technology rather than athleticism. Despite this, instructors continue to emphasize fundamentals over equipment, implying that marginal returns on technique dominate marginal returns on hardware.

Two complementary methods operationalize this framework empirically. The first is a statistical variance decomposition model attributing variation in PR throughput across five factors. The second is a natural experiment methodology, exploiting an exogenous shock in tool adoption to test whether model switching produces measurable output changes independent of confounding selection effects.

3. Core Analysis

3.1 The Grok Natural Experiment

SpaceX's acquisition of Cursor triggered an unusually rapid shift toward Grok usage, propagated through UI changes that promoted Grok and altered the behavior of Cursor's auto mode - rather than through organic engineer preference. This exogenous shock constitutes a natural experiment: because the shift in model share was driven by interface changes rather than engineer choice, it isolates the causal effect of model switching from selection bias. Comparing merged PRs per engineer against the magnitude of model-share change revealed a cloud of points centered near zero with no statistical correlation. As summarized in the source material: "The story here is a non-story... Output didn't change." This result anticipates the broader variance decomposition presented below.

3.2 Variance Decomposition of Output Drivers

Analysis of approximately 50,000 engineers at 447 companies, using Cursor and Claude Code telemetry, decomposed variation in PR throughput into five components. Tool choice explains less than 1% of variation. Skill use explains approximately 3%. Consumption (token volume) explains approximately 9%. Company and team environment explains approximately 12%. Individual engineer characteristics account for the remainder, approximately three-quarters of the explained variance. This ordering directly inverts the attention typically allocated to these factors in organizational decision-making, where tool and model selection dominates discourse despite contributing the least explanatory power.

Consistent with this decomposition, switching between Opus and Sonnet, between ChatGPT and alternatives, between Cursor and Claude Code, or between autocomplete and agentic modes produced no meaningful change in PR throughput - described in the source material as a "nothingburger" statistically.

3.3 Skill Use, Consumption, and Environment

Despite explaining only 3% of variance in isolation, skill use is notably underutilized and shows strong effects when activated. As of September, nearly half of engineering weeks recorded zero skill use, while only 12% reflected high-level use. Engineers reaching high skill use (30+ runs per week) demonstrated a 30% increase in PR throughput holding other factors constant, suggesting substantial unrealized upside from behavioral change rather than tool acquisition.

Consumption, based on roughly 500,000 developer-weeks of data, shows a positive but saturating relationship with throughput, likened to the diminishing returns of the rocket equation: more tokens yield more throughput, but at a decreasing rate. Top 10% token users produce roughly double the output of median users while consuming approximately 10 times the tokens - an efficiency trade-off with direct cost implications.

Company and team environment accounts for approximately 12% of variance, roughly one-third attributable to team-level factors and two-thirds to company-level factors such as culture, codebase quality, and review process. Holding consumption fixed, a 2x spread in output exists between the top and bottom 10% of companies - variation that, unlike model choice, falls within direct managerial control.

3.4 Individual Variability and Temporal Stability

The individual engineer factor exhibits a 3.5x spread between top and bottom decile performers, wider than the company-level 2x spread, confirming that individual characteristics remain the dominant source of variation even in an AI-augmented environment. A longitudinal study of 10,000 engineers found that approximately 44% cycled out of the top quartile every six months prior to widespread AI adoption, a rate that tightened only slightly to 41% post-AI. This stability suggests AI has not dramatically re-sorted which engineers perform best, reinforcing that individual craft and skill remain central determinants of output.

4. Technical Insights

Several implementation-relevant findings emerge from this analysis. First, the variance decomposition (Tool <1%, Skill ~3%, Consumption ~9%, Company/Team ~12%, Individual ~75%) indicates that procurement-driven interventions offer the lowest expected return among available levers. Second, the natural experiment design - leveraging an exogenous shock (the Cursor-Grok UI shift) rather than relying solely on observational correlation - strengthens causal confidence in the null tool-effect finding, addressing a common limitation of cross-sectional AI productivity studies. Third, the saturating consumption curve implies diminishing marginal returns on token spend, a consideration directly relevant to cost-conscious deployment strategies; organizations optimizing for efficiency rather than raw throughput may find diminishing ROI beyond moderate consumption levels. Fourth, the skill-use gap (nearly half of engineering weeks showing zero skill use) represents a low-cost, high-leverage intervention point, since the 30% throughput gain associated with high skill use requires behavioral and training investment rather than new technology procurement. Finally, the company-level 2x spread, holding consumption constant, demonstrates that environmental factors - unlike model choice - are both substantial in magnitude and actionable by leadership.

5. Discussion

These findings carry implications that extend beyond the immediate dataset. The persistent emphasis on model and tool selection in industry discourse appears misaligned with where measurable leverage actually resides. This mirrors the golf analogy's core insight: equipment matters, but instruction and environment matter more, and only the latter two are substantially within an organization's control after a baseline tool has been adopted.

A notable gap concerns causal mechanisms underlying the company-level 2x spread and the individual 3.5x spread. The analysis identifies these effects statistically but does not fully decompose which specific cultural, procedural, or individual-ability factors drive them. Future investigation might examine which company practices - code review norms, onboarding, codebase architecture - mediate the observed company-level variation, and whether targeted skill-use training programs can shift the nearly 50% of zero-skill-use weeks toward higher engagement.

The temporal stability finding - only a modest tightening from 44% to 41% quartile churn - also warrants further scrutiny as AI tooling matures. If future, more capable models produce larger absolute capability gaps between skilled and unskilled users, this churn rate could shift further; the current data represent an early snapshot rather than a stable equilibrium.

6. Conclusion

This analysis demonstrates that engineering output in AI-augmented environments is driven overwhelmingly by individual and organizational factors rather than by model or tool selection, which explains less than 1% of variance in a 50,000-engineer, 447-company dataset. Skill use, consumption, and environment collectively explain roughly a quarter of variance, with individual characteristics accounting for the remainder. The practical takeaway is that engineering leaders seeking throughput gains should prioritize culture, workflow design, and skill development over procurement cycles. Unlike a golf course, an organization's "course" - its environment - can be directly reshaped by its leadership, offering a tractable and evidence-based path toward improved engineering outcomes.

Sources

• Better AI Models Won't Fix Your Engineering Team - Nicholas Arcolano, Jellyfish - Original Creator (YouTube)
• Analysis and summary by Sean Weldon using AI-assisted research tools

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub

At 22Atlas, the studio I run, we build on Grok too: see the Grok bot we ship for clients.