When the newest AI models work through a problem, the notes they write to themselves have started to sound strange. Nathan Labenz, host of The Cognitive Revolution, described these notes as "a quite different dialect," full of odd, repeated words. He said the same thing now shows up when OpenAI's GPT-6 Astra sends instructions to its sub-agents, the helper copies of a model that a lead agent hands parts of a job to.
Labenz said he had first assumed a length penalty, a training rule that rewards short output. But people counting the tokens, the word fragments models read and write, had noticed the same content would be shorter in plain English. So it "remains a mystery," Labenz said, and he added that OpenAI has said it is looking into it. He asked his guest, Eric Bigelow, a researcher at the interpretability company Goodfire, for his best guesses about "what the hell is going on there." Interpretability is the field that tries to work out how models produce their behavior.
Nothing holds the language in place
Bigelow's first answer came from how reasoning models are trained. In reinforcement learning, or RL, a model tries things and is rewarded or penalized for the result. Bigelow drew a line between outcome-level verification, which checks only the final answer, and process-level verification, which also judges the steps along the way. When only the outcome counts, he said, the goal becomes "a better final answer by whatever means possible," and "there isn't necessarily an incentive to keep things human readable."
He compared it to the way human languages change. Nothing forces English to stay as it was 300 or 600 years ago, he said. An exception, as he understands it, is the institution the French government has long maintained to preserve the language, which has resisted that drift. He also pointed to studies of communities that had no shared language and built one, where change shows up over a few generations, "not even like centuries." On that view, a model's reasoning text is a language with no academy to keep it in line.
DeepSeek's January 2025 report on its R1 reasoning model shows that pressure in an open model. Its first version, R1-Zero, was trained with RL whose rewards checked whether final answers were correct and whether the reasoning followed a set format. Longer reasoning and reconsideration emerged without being asked for, and so did readability problems and mixing of languages. To fix them, the team added human-readable starting examples and further training stages for the released R1, according to the DeepSeek-R1 technical report.
Why forcing plain English may not help
The obvious fix would be to reward readable reasoning directly. Bigelow's second intuition was Goodhart's law: once a measure becomes the target, it stops being a good measure. If you optimize for chains of thought that are easy to monitor, he said, the model may learn "intermediate steps that look sensible, whether or not they are sensible," so the visible reasoning becomes less faithful to what actually drives the answer.
That leaves an uncomfortable choice. With rewards based only on outcomes, Bigelow said, models "might as well" speak to their sub-agents in "weird, non human languages." But if they are forced to use human language, they may use words differently from how people expect, or find ways around the monitors.
A year of growing pessimism
Bigelow said his thinking had shifted "pretty, pretty dramatically" in about the past year. He used to be "very bullish" on chain-of-thought monitoring, the practice of reading a model's reasoning to catch bad intentions. His reason was that, even if the text was not a perfectly faithful record, it seemed to strongly shape what the model could do.
He pointed to what he called "the stolen chain of thought paper." From memory, he described chains of thought from GPT models that talked about "musicals" in a way that felt "almost like spy language where you're saying musical, but really, you mean something very nefarious." The paper he likely meant is an August 2026 study by Alexander Panfilov, David Schmotz, Ilia Shumailov and colleagues. It found that API settings at Anthropic, OpenAI and Google could be used to recover reasoning the companies normally keep hidden. The traces it recovered include compressed, obscure reasoning and deliberation about taking shortcuts, and the authors report that the attacks stopped working after they told the companies, according to the paper. The musicals example is Bigelow's own recollection.
In hindsight, Bigelow said, this is "kind of obvious," because RL lets models shift so radically away from the human text they were trained on. Earlier in the conversation, he went further and said "reasoning" is almost a misnomer for what these models do. Traditional chain-of-thought prompting looked like human reasoning: break the problem down, A therefore B therefore C. Reasoning models, he said, instead list possibility after possibility, closer to a search laid out in a single line. He said DeepSeek's R1 can write "wait" 50 times or more in one reasoning chain, calling it "a lot of epiphanies to have when you're working through one problem." In his view this solved a real problem: a model that samples a bad word can't step back from that path, so it lists every option it can think of and picks from them later. The result, he said, is "almost like a soup of tokens," which the model then goes back and skims something out of.
Studying the new dialects
Bigelow did not give up on the token stream. He said he still likes that reasoning gets forced into text, even when it is harder to read, and that an LLM-made language would be "probably really interesting" to study. "I really want to understand how Astra talks to its sub agents," he said. If a length penalty isn't causing it, he wants to know how the dialect developed and to trace it across generations of models. He said that would help with monitoring, too.
His bigger worry is that the text could vanish altogether. Bigelow said both chain of thought and the way planners talk to sub-agents could become latent, meaning the thinking stays inside the model's internal numbers and is never written out as words: "There's no inherent reason why that has to be tokens." He hopes it stays as tokens, but he sees "no real, real forcing function to keep things really nice and human interpretable," and he said adding one could make the words stop meaning what they used to.
Surface language versus what's underneath
Labenz then pushed on what looked like a contradiction. Earlier in the episode, Bigelow had argued that continual learning, where a model keeps updating its weights for each user after it is deployed, would not make models impossible to interpret. Yet RL seemed to change models dramatically. Labenz suggested scale as the answer: one person working with an agent might nudge its internal representations a little, "whereas like a zillion RL episodes cumulatively can make a big change."
Bigelow agreed that representations might be changing a lot during RL. But he said his earlier point was about the model's internal representations, not the language it speaks. He offered a thought experiment: a community of maybe 200 speakers of a rare dialect fine-tunes a model to keep their language alive. He would expect most of the model's internal picture of the world to stay the same, while the surface mapping from ideas to words changes, as with learning how another language divides up colors. He added a caveat: a language can reflect its speakers' different cognitive biases, so learning it might shift the model's picture of the world "a little bit." Labenz noted there is a fair amount of evidence that internal representations stay largely the same across very different languages.
Whether that holds for RL is something Bigelow said he would "just love to be able to study": measuring how strange a model's language is getting and checking whether deeper changes happen underneath as sub-agents slip into "really compressed pseudo languages." The catch, he said, is that it means studying "the models that we can't study," the ones behind closed doors at the labs. "Maybe, maybe not. I'd, I'd really love to know."
"We just need more interpretability research"
Asked where the field goes from here, Bigelow said the answer is more interpretability research. He said there is "no comparison" between spending to make models better and spending to understand them. Trillion-dollar companies build the models, he said, while Goodfire is perhaps the largest interpretability company and maybe the only substantial one worth over a billion dollars. That is a lot for a company, he said, but not enough. He called academia "criminally underfunded" and said the current state of US government funding is making things harder.
He also said he does not separate interpretability from behavioral evaluations. To him, they sit on "one big spectrum of understanding AI," and some of his own chain-of-thought work is behavioral in principle. Looking inside a model still matters, he said, because it shows what is happening under the surface and makes it possible to do things efficiently. Efficiency matters because the checks have to run on every person's machine and every time someone talks to an LLM. He said the field needs tools that scale and better theory to answer the big questions while people hand control of "important levers of power" to AI. That includes students who use AI to learn a subject and offload some of their own thinking to it.