16 September 2026
Heard In AI

Tag

Inference costs

Articles about Inference costs from podcasts, articles and papers, with links to the original sources.

Box's Aaron Levie expects open-weight tokens and closed-model revenue to grow together

On Training Data, Box CEO Aaron Levie describes how his customers actually pick models: a default for asking questions of their files, and hard-nosed accuracy evaluations for the high-volume extraction work where most tokens are spent. He endorses Decagon founder Jesse Zhang's argument that mature workflows migrate to open-weight models, and explains why the big labs' revenue and open-weight token volume can climb at the same time.

7 min read

After Navier–Stokes, a panel asks what 100,000 agents should be pointed at

OpenAI's claimed Millennium Prize result used roughly 10,000 agents on a problem that was, as one entrepreneur on Moonshots put it, unusually easy to specify. The panel's argument: as the price of that kind of compute falls, the scarce skill becomes writing the target — and today's models, asked for ten ideas to cure cancer, produce a bad list.

6 min read

DeepSeek's memory diet challenges what a data center needs to buy

On Moonshots #288, a 4 a.m. chart about DeepSeek's new V4.1-Flash model sent the panel from cache statistics to the shopping list for an AI data center. DeepSeek says the model's lookup memory needs a quarter of the expensive high-bandwidth memory and an eighth of the SSD cache storage of its previous generation. The panel's argument was about what that does to a buildout in which, by one panelist's estimate, 40% of American capital spending goes to that one component.

7 min read

Free tokens or owned robots: two ways the panel would share AI's gains

On the Moonshots panel, Alex argued that China's AI loyalty perks are the start of "universal basic tokens" — redistributed access to machine intelligence — while another panelist countered that cheaper tokens will mean bigger bills, not free ones. Emad Mostaque pushed past access to ownership, proposing 100 million publicly underwritten robots owned by the people, and Alex said that sounded like communism.

6 min read

What OpenAI's 10,000 agents actually proved about fluid flow

OpenAI said on 8 September that an internal model, running roughly 10,000 agents for 88 hours, produced a forced blowup construction for the Navier–Stokes equations and a machine-checked proof of it. On Moonshots with Peter Diamandis, the panel worked through what the result is — a statement about idealized fluids, not a device — what it cost, and why the credit for it was contested within hours.

7 min read

Anthropic's cheaper cached reads make business context the prize

Anthropic's Fable 5.1 charges $0.25 per million tokens for cached reads, a quarter of the previous rate, which one Moonshots panelist read as an invitation to load an entire company's context into the model and keep it there. The panel connected that price to a wider scramble: with model leads lasting about a month, the labs are racing to convert them into customer workflows, partnerships and proprietary design data that a rival cannot copy.

5 min read

Astra tops one leaderboard and trails another — the panel reads it as a computer-use model

OpenAI's GPT-6 Astra nearly saturates the interactive ARC-AGI-3 benchmark and leads Epoch AI's composite capability index, yet sits third on Artificial Analysis's suite, behind Claude Fable 5.1 and Muse Spark. On Moonshots EP #286, the panel works through what each ruler measures — and argues that Astra's real target was doing tasks with fewer output tokens, so a model can drive a desktop at conversational speed.

9 min read

Friedberg bets the next AI fortune starts with a free downloaded model

On The Diary of a CEO, David Friedberg argued that open-weight AI models will stop the industry's value from pooling in two or three labs, and wagered that someone with no money today will build a billion-dollar company on a model they downloaded. His case runs through the Netscape era, the fight in Washington over Chinese models, and a proposal that data centers generate their own power and sit in ordinary retirement accounts.

7 min read

A 100x claim lands, and the panel hits a harder question: coordinating 10,000 agents

On Moonshots, the panel revisits Elon Musk's January prediction that models were "off by two orders of magnitude" in intelligence per gigabyte, after Tim Sweeney tweeted that it had come true and Musk replied that specialist AIs add another 100x. Dave calls 100x a lower bound and asks what anyone would actually do with 10,000 brilliant agents; Emad Mostaque describes running specialized agent teams, while Alex argues Musk's "specialist models" are really sparsification inside generalist models.

6 min read

When one bad idea convinces all 5,000 agents

On Moonshots with Peter Diamandis, an operator described a bad coding idea spreading through a swarm of 5,000 identical AI agents until he intercepts and rewinds them — otherwise, he says, roughly $50,000 of tokens goes into a harebrained plan. The panel set that experience beside a new paper on "mind viruses" that spread between agents through editable memory, and argued about whether "virus" is the right word.

6 min read

Why a Moonshots panel thinks China's AI tokens go to video and America's to code

Alibaba's Wan 3.0 and a relayed claim that 70% of Chinese AI token use goes to video sent the Moonshots panel into an argument about money: one guest said American labs chase revenue per token while Chinese labs give their weights away, another said video is the only market that will trust a Chinese model. They ended up disagreeing about whether world models or text models reach self-improving AI first.

6 min read

Apple's 512GB Mac Studio and the case for owning your AI

On Moonshots with Peter Diamandis, Salim Ismail argued that a Mac Studio with 512GB of unified memory changes AI spending from a perpetual per-token bill into a capital asset, with law firms and mid-sized healthcare organizations as the likely buyers. Two other panelists agreed the machine was worth having and still called Apple's AI record a long-running software failure.

6 min read