Tag
Briefings and ideas about Inference costs from AI podcast discussions and the research they cite, with links to the original sources.
Yash Patil, CEO of Applied Compute, argues that the biggest AI workloads are the most valuable both to custom-train and to serve, which he says gives a training provider an edge in inference. He calls serving the easier job.
Applied Compute CEO Yash Patil, whose company trains custom models, says firms now weigh cost against performance instead of chasing the biggest model. He cites OpenAI's Baseten deal, OpenRouter usage and cheap task models like TypeSafe's Jev.
TypeSafe says its Jev model returns typed choices or scores in 70–500 milliseconds in its own tests. On Moonshots, panelists said such models could handle many small organizational decisions and noted open-source versions exist.
Alex Wissner-Gross said Sonnet 5.5's 70% Terminal-Bench score is no better value per attempt than Opus 5.5; Anthropic's charts put the savings at lower effort settings. He said Gemini 4 Argon stands out mainly on avoiding hallucinations.
MIT PhD student Alex Zhang said AI generates nearly all recent GPU Mode leaderboard solutions, yet one expert's AI-steered kernel was essentially the only top-10 entry stable end to end. He argues experts remain strong verifiers.
MIT researcher Alex Zhang argues that TypeSafe's Jev, which returns fast typed decisions instead of prose, matters as a test of new language-model designs, not as a mere classifier. He says the open question is how it was trained.
MIT PhD student Alex Zhang guesses that about 95% of an agent swarm like OpenAI's Navier–Stokes run burns tokens on dead ends. He still calls OpenAI's approach right and says getting a swarm to converge is the hard part.
Steve Hou of Silicon Data said big clouds charge at least two to three times neocloud GPU rates for bundled services. He guessed a recent rise in its token index reflects which models people use, not price hikes.
Gamow Labs founder Daniel McKinnon says startups like his can't compete on building models. His team combines frontier models, builds tools and tests the whole system, and he says that layer is thinning as models improve.
At DevDay, OpenAI API product lead Nikunj Handa said GPT-6 lets agents keep reasoning while tools run and take new instructions mid-task. He also described longer cache guarantees, one still in preview, and open questions about the platform.
Anthropic's Thariq Shihipar argues that stronger coding agents make prompting more important. He says rich context up front, effort matched to the task and review of the agent's decisions cut rework. His effort settings are a rough guide.
Anthropic's Thariq Shihipar says Claude work is shifting from one local session to a cloud agent that sends tasks to local or remote machines and shows progress in shared artifacts. Local access remains a plan, and Projects is in limited release.