Tag
Briefings and ideas about Transformers from AI podcast discussions and the research they cite, with links to the original sources.
Alexander Wissner-Gross argued next-word prediction as a training task may last indefinitely, while the transformer architecture is already being replaced piece by piece. Another panelist guessed chips move to photonics in 18 months to two years.
John Schulman expects frontier AI models to keep growing as compute and GPUs expand. Charlie O’Neill expects little parameter growth for a few years, because long reinforcement-learning runs reward cheap inference.
DeepSeek says V4.1-Flash's KV cache needs a quarter of the HBM and an eighth of the SSD cache of its prior generation. Moonshots panelists argued that changes what AI data centers should buy; one estimated HBM is 40% of US buildout spending.
World Labs' Atlas generates video along a user-set camera path and rebuilds scenes from photos, inventing what was not photographed. On Moonshots, Alex inferred from its documentation that it trains on Gaussian splats.
Moonshots panelists dismissed OpenAI's reported kill switch for GPT-6 Astra, rated at its top cyber-risk tier, as marketing, saying it won't address possible reasoning hidden from readable text or model copies on other clouds.
OpenAI's GPT-6 Astra nearly saturates ARC-AGI-3 and tops Epoch AI's index but ranks third by vendor family on Artificial Analysis's. Moonshots panelist Alex, lacking access, suspects it was built for fast, low-token computer use.
Peter Diamandis, after meeting SK hynix and Solidigm leaders, argued memory, not GPUs, now limits AI. Panelists blamed new AI workloads and bust-wary suppliers, and floated etching model weights into chips as a possible fix.
BDH-CQ's authors report solving 118 of 400 public ARC-AGI-1 tasks, with up to two answers each, at an estimated $0.00070 inference cost per task. Emad Mostaque welcomed the experiment; Alex doubted it shows progress beyond one benchmark.