Evals and Benchmarks
If you cannot measure it, you cannot ship it with confidence. These posts cover evaluating LLM apps and agents: designing evals, building benchmarks, using models as judges, and reading benchmark results with the right amount of suspicion.
- 'How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude - Laurie Voss, Arize AI'
- MCP Doesn't Suck. Your Agent Does. - Jan Čurn, Apify
- We Let Claude Code and Codex Race Human Researchers - Elie Bakouch, Prime Intellect
- Modality Misalignment and Originality Attribution in Short-Form Video - Aditya Gautam, Meta
- The Dark Arts of Skill Engineering - Paul Bakaus, Renaissance Geek (Impeccable)
- Are LLM Performance Benchmarks Reliable? - Ashok Chandrasekar & Jason Kramberger, Google
- '1 Trillion Phone Calls/yr, 10% Error rate: The Crisis in Voice AI - Sumanyu Sharma, Hamming AI'
- 'Build-Time vs. Run-Time: Why Dev Tools Fail in Production - Averi Kitsch & Prerna Kakkar, Google'
- Building Trust in Coding Agents: Verification, Evals, and Agent Fleets
- Can LLMs Write Fast Multi-GPU Kernels? - Simran Arora, Together AI
- The Agentic Commerce Stack - Ahnaf Prio, Best Buy
- Inside 847 Production Clinical AI Notes - Sebastian Fox, Composo
- 'Preferences Over Benchmarks: Model Routing - Archana Kamath & Tyler Gillam, DigitalOcean'
- 'Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards - Dan Bjornn, Lease End'
- When Will The Benchmaxxing Plague End? - Nick Heiner, Surge AI
- Reinforcement Learning without Verifiable Rewards - Will Brown, Prime Intellect
- Your Moat Is Your Data Model - Mike Phipps, Gates Foundation
- Don't Ship Skills Without Evals - Philipp Schmid, Google DeepMind
- Recursive Model Improvement - Lee Robinson, Cursor, SpaceXAI
- How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI
- What if the harness mattered more than the model? - Aditya Bhargava, Etsy
- Frontier results, on device - RL Nabors, Arize
- The Agentic AI Engineer - Benedikt Sanftl, Mutagent
- Your Agent Failed in Prod. Good Luck Reproducing It. - Tisha Chawla & Susheem Koul, Microsoft
- 'The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen'
- 'The Production AI Playbook: Deploying Agents at Enterprise Scale - Sandipan Bhaumik, Databricks'
- 'Sovereign Escape Velocity: Ownership w Open Models - Gus Martins, & Ian Ballantyne, Google DeepMind'
- Stop Making Models Bigger, Make Them Behave - Kobie Crawford, Snorkel
- Evals Are Broken, Use Them Anyway — Ara Khan, Cline
- LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize
- 'Beyond Transcription: Building Voice AI That Understands Conversations — Hervé Bredin, pyannoteAI'
- The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI
- 'SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius'
- How Lovable self-improves every hour — Benjamin Verbeek, Lovable
- Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel
- '20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna'
- Can LLMs generate Enterprise Quality Code? — Prasenjit Sarkar, Sonar
- Why (Senior) Engineers Struggle to Build AI Agents — Philipp Schmid, Google DeepMind
- How agent o11y differs from traditional o11y — Phil Hetzel, Braintrust
- The maturity phases of running evals — Phil Hetzel, Braintrust
- Agentic Evaluations at Scale, For Everybody — Nicholas Kang & Michael Aaron, Google DeepMind
- Does GenAI "belong" to data scientists? — Phil Hetzel, Braintrust
- 'Skill issue: Lessons from skilling up coding agents to use Langfuse - Marc Klingen, Clickhouse'
- Fighting AI with AI — Lawrence Jones, Incident
- AIE Singapore Day 2 ft. Google DeepMind, OpenClaw, Adaption, Arize, Cloudflare, Robot Company & more
- AIE Singapore Day 1 ft. Minister, NanoClaw, OpenAI, Google, Vercel, Cursor & more
- Combine Skills and MCP to Close the Context Gap — Pedro Rodrigues, Supabase
- Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize
- Building a Chess Coach — Anant Dole and Asbjorn Steinskog, Take Take Take
- 'Dark Factory: How OpenClaw Ships Faster Than You Can Read the Diff — Vincent Koc'
- Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha, Raindrop
- 'Playground in Prod: Optimising Agents in Production Environments — Samuel Colvin, Pydantic'
- 'Skill Issue: How We Used AI to Make Agents Actually Good at Supabase — Pedro Rodrigues, Supabase'
- Context Is the New Code — Patrick Debois, Tessl
- Prompt Enhancement Through Learning Loop
- Shipping AI That Works