AI Brief, November 15: SO-Arena reliability gap, DeepSeek V3 in JAX, Nvidia petition

SO-Arena finds fast models unreliable, DeepSeek V3 lands in 190 lines of JAX, and a petition to reinstate Jensen Huang tops 115,000 names.

By Sean Weldon

Benchmarks and tooling dominate today, led by SO-Arena, which reports that the strongest fast "System One" model ranks only seventh on decision reliability. Alongside it: a 190-line JAX reimplementation of DeepSeek V3, a WebKit browser built for coding agents, and continued fallout from Jensen Huang's exit at Nvidia.

SO-Arena: top "System One" model ranks seventh on decision reliability

SO-Arena is a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families, including interactive tasks where models execute actions in sandboxes. Beyond accuracy, it measures decision stability under controlled option-set changes and whether models recognize when no valid option is available. Across 21 models evaluated, the authors report that the benefits of scaling and additional reasoning vary substantially by domain, and that the best-performing System One model ranks only seventh in reliability.

Why it matters: If you route production traffic to fast, low-token models on the basis of aggregate accuracy, this suggests you also need to test decision stability and abstention separately.

Sources: Hugging Face Daily Papers

DeepSeek V3 architecture reimplemented in 190 lines of pure JAX

A Show HN project implements the architecture from the DeepSeek-V3 technical report in 190 non-blank, non-comment lines of pure JAX, with no Flax, Equinox or PyTorch: parameters are plain dicts of arrays and each model part is a readable function. The author pretrained it from scratch on a MacBook Air M5 with 24 GB on public-domain detective fiction, reaching the same validation loss at step 2000 with and without the MoE balancing bias (3.921 vs 3.923). The README is explicit about the gaps: dense MoE compute with no sparse dispatch, no weight absorption in MLA, no FP8 training or YaRN, and 36.2M parameters across 6 layers with a 256-token context.

Why it matters: It gives a readable reference for MoE routing and MLA internals that you can run locally, without reading production-scale training code.

Sources: github.com / Hacker News discussion

Agweb is a macOS WebKit browser built for coding agents to drive from the shell

Agweb is a macOS browser built on WebKit that coding agents control from the shell with a Playwright-CLI-style command. The author says it never steals focus, gives each agent its own workspace and tabs, shares logins through named profiles, suspends idle tabs, and uses native macOS events so hover menus and drag-and-drop work. Action replies report what changed on the page, which the author says avoids a snapshot after every step and uses far fewer tokens. Limits are stated: WebKit only, so there is no Chrome DevTools Protocol and Playwright cannot attach, and it requires macOS 26 or later.

Why it matters: For teams running several coding agents at once, this is a lighter alternative to headless Chrome stacks, at the cost of Chrome-only site compatibility.

Sources: github.com / Hacker News discussion

datasette-llm 0.1a11 adds OpenAI-compatible endpoints for local models

Simon Willison released datasette-llm 0.1a11, the intermediate plugin that configures LLMs for use by other Datasette plugins. The release adds support for OpenAI-compatible endpoints, so Datasette can use a model hosted anywhere that speaks the OpenAI protocol. Willison shows a uvx command that points the default model at a local LM Studio server to run a Datasette Agent instance, and notes that llama.cpp just hit v1.0.0.

Why it matters: Datasette's AI plugins can now run against local models served by LM Studio, Ollama or llama.cpp instead of a hosted API, which matters for data that cannot leave your machine.

Sources: Simon Willison

Petition to reinstate Jensen Huang passes 115,000 signatures

TechCrunch reports that a petition calling on Nvidia to reinstate ousted CEO Jensen Huang had more than 115,000 signatures as of Saturday afternoon, including, per the petition site, 5,965 current Nvidia employees, 1,108 former employees and 63,063 shareholders, with Elon Musk, Amjad Masad, Tim Sweeney and Garry Tan among signers. Huang resigned Thursday, four days after confessing to an affair with a chip executive whom U.S. intelligence suspects of being a Chinese spy, and has agreed to testify before Congress. CFO Colette Kress is interim CEO. TechCrunch also reports that Nvidia's longtime chief security officer and head of product security stepped down in August, and that Asian governments are seeking confirmation of chip supply commitments, with South Korea's Science Ministry saying Nvidia reaffirmed its commitments.

Why it matters: Nvidia's leadership vacuum is now touching supply-chain reassurance across Asia, which is worth watching if your roadmap depends on GPU availability.

Sources: TechCrunch AI / TechCrunch AI / TechCrunch AI

TOME-Math turns 2,118 textbooks into proof supervision, +16.7 points on math

TOME-Math converts 2,118 university-level mathematics textbooks into roughly 380K structured proof-supervision records, preserving chapter order, proposition-proof links, dependency references and source provenance rather than flattening everything into problem-solution pairs. The authors report that fine-tuning on the full corpus raises the unweighted mean score across held-out evaluations and established public benchmarks by 16.7 percentage points over the base model, and exceeds all three public instruction-data baselines in aggregate. They say the corpus will be released.

Why it matters: It is a concrete argument that how you organize training data, not just how much you have, drives reasoning gains, and a dataset to test that against once it ships.

Sources: Hugging Face Daily Papers


How this brief is made: each morning, stories are picked from AI lab announcements, analysts, Hacker News, Hugging Face Daily Papers and the tech press, then summarized with AI assistance from the linked sources. Every item links to where its facts came from.

Sean Weldon builds apps and custom software at 22Atlas.

Start your build

More about Sean