Tag
Briefings and ideas about In-context learning from AI podcast discussions and the research they cite, with links to the original sources.
Goodfire's Eric Bigelow argues in-context learning covers nearly everything a model does to adapt as it reads. He expects context, memory and harnesses, not per-user fine-tuning, to drive personalization, but hasn't researched continual learning.
Goodfire researcher Eric Bigelow resampled a GPT-3.5 model's reasoning at every token and saw answers collapse at single tokens, some arbitrary. He argues the "decision" happens in sampling; newer reasoning models often fork more smoothly.
Keerthana Gopalakrishnan of Google DeepMind says robot models are at a GPT-2 stage. In her view, a GPT-3 moment needs few-shot teaching to work across many tasks and one brain to run reliably on a new robot body, a test language models never faced.
In one DrivingBench session, OpenAI's GPT-6 Astra drove a real Corolla through a cone course on its second try. Alexander Wissner-Gross said specialist robotics models are unnecessary; Dave Blundin questioned reliability and compute cost.
Box CEO Aaron Levie says shifting access permissions, like a lawyer walled off from rival matters, complicate training company knowledge into model weights. He wants a rubric on what to bake in and expects answers to vary by domain.
Charlie O'Neill told the Dwarkesh Podcast that AI already learns from deployment, but in slow batches, and that many small updates erase earlier learning; John Schulman expects customers to resist feeding a shared model.
Rich Sutton and Khurram Javed's Oak Lab wants deployed AI to update its weights from individual experience, using per-weight learning rates and continual backpropagation—backed by earlier experiments, not yet a demonstrated general-purpose system.
BDH-CQ's authors report solving 118 of 400 public ARC-AGI-1 tasks, with up to two answers each, at an estimated $0.00070 inference cost per task. Emad Mostaque welcomed the experiment; Alex doubted it shows progress beyond one benchmark.