Tag
Briefings and ideas about Synthetic data from AI podcast discussions and the research they cite, with links to the original sources.
On the Dwarkesh Podcast, AI researchers argued that pass/fail rewards work because earlier training supplies candidate behaviors and rewards ignore incidental wording. Charlie O'Neill warned that harder training tasks cost ever more to build.
John Schulman expects frontier AI models to keep growing as compute and GPUs expand. Charlie O’Neill expects little parameter growth for a few years, because long reinforcement-learning runs reward cheap inference.
AI researcher John Schulman says cheap distillation keeps model competition open. He argues that a copy gets broad behavior only from realistic, multi-turn prompts, and that benchmark-style tasks alone may match scores without it.
Dwarkesh Patel and Jerry Han found better training data gave a 12-fold compute-efficiency gain from 2019 to 2025 versus 3.7-fold for model recipes, in small models on easy benchmarks; a panel split on how long proprietary data stays valuable.