Tag
Briefings and ideas about Inference costs from AI podcast discussions and the research they cite, with links to the original sources.
Tim Sweeney tweeted that Elon Musk's predicted 100x gain in intelligence per gigabyte has come true; Musk says specialist AIs add another 100x. Dave asks how 10,000 agents would work together; Alex says specialists are really sparsification.
Dave Blundin of Link Ventures says one bad coding idea can spread through all 5,000 of his identical AI agents, costing roughly $50,000 in tokens unless he rewinds them. Panelists linked it to a paper on "mind viruses" spreading between agents.
Moonshots panelists debated a relayed claim that 70% of China's AI token use goes to video: Alex Wissner-Gross said US labs chase revenue per token, and Dave Blundin that video sells where enterprises won't trust Chinese models.
Salim Ismail argued Apple's 512GB Mac Studio turns AI spending from a per-token bill into an owned capital asset for law firms and mid-sized healthcare groups. Fellow panelist Dave called it a tiny step by a company he says has no AI strategy.
Moonshots panelist Alex floated a hypothetical in which OpenAI rents Jalapeño inference-chip capacity to rivals, after the panel relayed OpenAI's own claimed gains over NVIDIA chips and argued inference is moving off NVIDIA.
AI critic Ed Zitron argues chatbots help with small tasks but aren't worth trillions, and that subscriptions hide what usage costs. He asks whether users like the host's fiancée would pay the real per-token price.
Parag Agrawal, founder of Parallel Web Systems, says agents already multiply web searches many times over. He forecasts that in a couple of years the web will tell agents when something worth acting on has changed.
Parag Agrawal says Parallel entered web search by selling a search agent that crawled after each request, replacing outsourced human research, and treating an index as a latency optimization to build later.
Alvin Graylin argued that learning from Claude's outputs cannot explain all of China's AI gains, citing a claimed KV-cache cut, though his defense of distillation doesn't address Anthropic's allegations of fraudulent access.
Alvin Graylin argues China is competing to spread useful AI through industry and open-weight models adopted abroad, not racing to reach AGI first. The contrast is one of emphasis: America's AI Action Plan also promotes adoption.
Alvin Graylin argues cheaper open models and local computing could erode the cloud revenues that AI data center debt relies on, likening NVIDIA's plan to mobilize over $500 billion in outside financing to subprime lending, as a warning, not proof.
Ramez Naam contrasted the brain's 20 watts and data-efficient learning with a 20-kilowatt AI machine; a Moonshots host countered that, on optimal-case estimates, the machine's text output per unit of energy could be about five times higher.