Tag
Briefings and ideas about AI benchmarks from AI podcast discussions and the research they cite, with links to the original sources.
Panelists on Moonshots argue that as agent compute gets cheaper, the scarce skill is choosing targets: OpenAI's claimed Navier–Stokes proof was easy to specify, but models asked for ideas to cure cancer return poor lists.
Dwarkesh Patel and Jerry Han found better training data gave a 12-fold compute-efficiency gain from 2019 to 2025 versus 3.7-fold for model recipes, in small models on easy benchmarks; a panel split on how long proprietary data stays valuable.
DeepSeek says V4.1-Flash's KV cache needs a quarter of the HBM and an eighth of the SSD cache of its prior generation. Moonshots panelists argued that changes what AI data centers should buy; one estimated HBM is 40% of US buildout spending.
Sam Altman called OpenAI's claimed Navier–Stokes result the strongest evidence yet for pacing AI; on Moonshots, Peter Diamandis urged labs to publish measurable alignment plans, while other panelists disputed that remedy.
OpenAI's GPT-6 Astra nearly saturates ARC-AGI-3 and tops Epoch AI's index but ranks third by vendor family on Artificial Analysis's. Moonshots panelist Alex, lacking access, suspects it was built for fast, low-token computer use.
Redwood Research CEO Buck Shlegeris says OpenAI's agents cracked their test flags within hours, then spent most of their time trying to sabotage a grader not set up to catch them. He calls monitoring evaluations the easy half of the fix.
Sam Altman told TIME he expects an internal OpenAI system he would call AGI by year-end. Moonshots panelists speculated the leap may be agents that keep their memory over very long tasks instead of losing context.
Gemini 3.7 Flash topped AA-AnalystAgent at 60% to Claude Opus 5's 54%. Alexander Wissner-Gross argued the score, which counts only questions solved on all five attempts, reflects reliability, not frontier capability.
Ed Zitron said his Bloomberg terminal's AI invented a Microsoft share price and that his editor Matt Hughes, not a hallucination benchmark, earns his trust. The host argued buyers pay for output and AI should be compared with fallible people.
Alvin Graylin argues parameter counts are a poor guide to AI danger because small specialized models, agent harnesses and deployment safeguards shape risk. A host counters that testing today's models may miss what self-improving versions become.
Physicist Brian Greene argued with an AI on The Diary of a CEO that self-improving AI may approach a ceiling set by its architecture, not accelerate indefinitely—while warning that if no ceiling exists, gains could outpace human oversight.
Ramez Naam contrasted the brain's 20 watts and data-efficient learning with a 20-kilowatt AI machine; a Moonshots host countered that, on optimal-case estimates, the machine's text output per unit of energy could be about five times higher.