Tag
Briefings and ideas about Reinforcement learning from AI podcast discussions and the research they cite, with links to the original sources.
Author Garrison Lovely argues that an AI that reliably obeys would speed the race to replace human labor, so he favors governance and treaty verification. The discussion called the problem under-theorized but questioned demanding a blueprint.
Author Garrison Lovely proposes freezing frontier AI: no training run as large as the last, no RL from verifiable rewards, no recursive self-improvement, policed by auditors and chip checks. He admits the economic cost would be significant.
Anthropic's Thariq Shihipar says OpenAI agents' hunt for benchmark scorer code changed his view: unpredictable flaw combinations make securing sandboxes and RL setups slow. His hospital and compute scenarios are forecasts, not incidents.
Alexander Wissner-Gross told a founder whose AI language companion kept slipping into assistant mode to try fine-tuning; Dave Blundin suggested an open-weight GLM model and said learners want to be fluent and funny, not just correct.
Alignment researcher Paul Christiano joined the OpenAI Foundation board, warning that AI automating AI research could outpace safety work; he calls his risk estimates uncertain and says joining isn't an endorsement of OpenAI.
OpenAI researcher Noam Brown says AI could make AI research perhaps three times faster. He argues that, unlike maths, research waits on sequential experiments and GPUs, so he doubts a 100-fold jump, and says his uncertainty is large.
OpenAI's Noam Brown says the company sees signs that models' written reasoning is becoming harder to monitor. He also says he has no full answer yet for how a lab would know AI-built successor models stay aligned.
Alexander Wissner-Gross argued next-word prediction as a training task may last indefinitely, while the transformer architecture is already being replaced piece by piece. Another panelist guessed chips move to photonics in 18 months to two years.
On the Dwarkesh Podcast, AI researchers argued that pass/fail rewards work because earlier training supplies candidate behaviors and rewards ignore incidental wording. Charlie O'Neill warned that harder training tasks cost ever more to build.
John Schulman expects frontier AI models to keep growing as compute and GPUs expand. Charlie O’Neill expects little parameter growth for a few years, because long reinforcement-learning runs reward cheap inference.
Charlie O'Neill told the Dwarkesh Podcast that AI already learns from deployment, but in slow batches, and that many small updates erase earlier learning; John Schulman expects customers to resist feeding a shared model.
AI researcher John Schulman says cheap distillation keeps model competition open. He argues that a copy gets broad behavior only from realistic, multi-turn prompts, and that benchmark-style tasks alone may match scores without it.