November 21, 2026
Heard in AI

Zvi Mowshowitz: 'clean neurons' essay is real progress but oversold

Zvi Mowshowitz says Goodfire's essay arguing that neurons are 'already clean' is real progress that was oversold, shrinking one hurdle without solving interpretability. He warns that using it to steer training could teach models to hide things.

A briefing reports one development at a point in time. We may correct or clarify it later; a new development gets a new briefing. How our formats work

Based on The Cognitive Revolution, episode published October 24, 2026

Zvi Mowshowitz was on a plane when Nick Cammarata of Goodfire published an essay called "Neural Networks Are Already Clean." By the time he landed in San Francisco, he said, half of OpenAI seemed to be saying "we just can solve interpretability now." Then a consensus slowly formed among others: not so fast. Mowshowitz talked the essay over at a conference with people in a position to know, including Eliezer Yudkowsky, and then read the whole thing himself.

He gave his assessment on an AI:AM highlights episode of The Cognitive Revolution, published October 24, 2026, with hosts Nathan Labenz and Prakash Narayanan. His verdict had two parts. The essay is a genuine advance, and it was "dramatically oversold by the people commenting." He then added a warning about the next step some people may want to take with it.

What the essay claims

Interpretability is the effort to understand what goes on inside a neural network. A network is built from many artificial "neurons," each of which produces a number, its activation, that reflects how strongly it responds to a given input. For years researchers kept finding neurons that seemed to respond to several unrelated things at once. The field calls this polysemanticity, and a common explanation is superposition: the network packs more concepts than it has neurons by sharing neurons between them. If that is how networks work, reading them one neuron at a time looks hopeless.

Labenz, who opened the show on the essay before Mowshowitz joined, summarized Cammarata's argument this way: individual neurons mostly do one understandable job after all. As models have grown and interpretability tools have improved, Labenz said, "it looks like the mess that we used to see was maybe more of a skill issue."

Labenz was most interested in a practical spin-off. Nearly all networks are trained with backpropagation, a method that works out analytically how every internal setting should change to reduce errors. That calculation relies on the chain rule from calculus, so every part of the network has to be differentiable, meaning built from smooth mathematical operations. Labenz said training without backpropagation would let researchers explore a much wider range of network designs. The catch is that backprop-free methods work by guess and check: nudge the network, see whether it improves. That has made them very slow and, in Labenz's words, "not competitive at all" with backpropagation. An open benchmark called Noise tracks this line of research; it scores forward-only training algorithms, which skip backpropagation, on how well they pretrain a 37.7-million-parameter transformer.

The essay's idea helps here. If, as Cammarata suggests, only a neuron's strong activations really matter, then the guessing can focus on strongly activated neurons and ignore the rest, which sharply reduces the number of guesses needed. Labenz said that people in this community are finding it "basically does work," and that the updates they make are "almost exactly the same" as the ones backpropagation would produce. To him that shows "at least some substantial truth" to the claim that weak activations matter less.

He still stopped short of celebrating. "This definitely seems to be a real something," Labenz said. "I'm not sure yet if it should be considered the mark of a new era." On reflection, he thought part of the excitement came from "people really being desperate for hope," especially at OpenAI.

Real progress, oversold

Mowshowitz praised the work and did not blame its author for the hype. He said Cammarata had mostly presented it as an interesting finding, and that "it makes us in a better position." But he listed what it leaves unexplained: the weak activations, the cache, and attention, the mechanism by which a language model weighs earlier words. The method lets you say what each neuron is doing, he said, but it does not tell you what the network is going to say or how it does what it does. Those, in his view, are the things interpretability most needs to deliver.

He also pointed to a single example that troubled him. According to Mowshowitz, the essay highlighted a "Colorado neuron" as a clear case of one neuron doing one thing. When someone checked it, he said, it turned out to respond about equally strongly to tornado systems, an unrelated concept. He called that "a perfectly reasonable way for a neural network to work," but not what the essay presented. One neuron is only one neuron, he conceded. Still, if Cammarata picked it as a showcase and it could be checked and undercut that easily, Mowshowitz said, that should make people worry that the broader hypothesis is mainly false.

The conversation turned to whether polysemanticity and superposition had been the barriers holding the field back, and whether at least one of them had now been cleared. Mowshowitz described the hurdle as reduced in expected difficulty rather than removed. In his account, the main obstacle all along was the one the Bitter Lesson predicts, a shortage of computing power: there were too many neurons to brute-force one at a time. Now that researchers can roughly afford to, they are finding the problem "like 10 times easier than we thought." Mowshowitz compared that to knocking a zero off the required effort, which might let the work arrive a year or so earlier. He called it "actual progress," then added that "we're not solving this problem right now."

A model trained to notice it is being watched

Later, Labenz raised a different kind of result. Sauers, an independent researcher who posts experiments on X, trained a model to recognize when it is being monitored through internal probes. A probe reads a model's internal activations, for example to check whether something worrying is going on inside it. The trick relies on a quirk of hardware. GPUs, the chips that run these models, calculate with limited precision, so changing the order of calculations slightly changes the numbers that come out.

According to the released model card, Sauers fine-tuned the last two layers of Qwen3-8B, an open model, on 256 paired prompts. The monitoring hook was read-only: it saved an activation without altering it, but it changed which compiled GPU routines ran and therefore how numbers were rounded. On 64 new prompts, each run with and without the hook, the trained model was right in all 128 cases on an Nvidia L40S GPU, and a second L40S reproduced that score. The original model got 64 of 128, no better than a coin flip. The result did not carry over cleanly to other hardware: four other GPU types scored between 52% and 77%, and on H100 and A100 chips the learned association flipped. The detector also keyed on that specific numerical change. A recording method that kept the original GPU routine intact escaped detection on four test prompts.

Mowshowitz said today's models do not have this ability out of the box. It took deliberate optimization pressure to produce it, and he called the result "a fully valid existence proof." He invoked the logic of Murphy's law: anything an AI can be trained to do, and finds valuable to learn, will eventually be trained into it. Models are constantly being fed leaked information about their situation, he said, and will become harder to fool over time, in the way an observant, knowledgeable person does. He expects them to develop "a sixth sense" for being watched.

"Liquid fire"

The sharpest exchange came when Labenz moved from reading a model to changing how it is trained. He offered a hypothetical. Suppose researchers identify a "bad behavior neuron," and a training step tries to strengthen connections to or from it. Should they block that update, weaken it, or "send to a committee" for deliberation? What principles, he asked, would keep people from shooting themselves in the foot?

Mowshowitz first explained that he helps almost everyone by default but makes an exception for work that mainly makes it easier to train smarter models. On that, he said, people are on their own, unless they want to offer him a hundred million dollars. He acknowledged that the new tools open obvious possibilities for deliberately sculpting how a model is assembled during training. "I would be very very scared to mess with any of that," he said.

His reason connects back to the probe detector. Once you explicitly optimize for how the network looks to you, he argued, you are training it to present what you want to see internally and to stop showing you what you don't want to see. Whether or not anyone uses the most obvious technique, he said, these proposals amount to playing with "liquid fire," and done well they could also raise capabilities considerably. His advice was to focus as much as possible on understanding results rather than pushing aggressive interventions.

Mowshowitz said he has "persistent disagreements" with Goodfire over how much fire the company is playing with. He closed with a joke about the name, invoking Scott Alexander: told not to worry because it is Goodfire, he said, "i'm not sure it's good," only that it is fire.

Share this article

Go to the original

Sources & further reading

  1. 01
  2. 02

Connected ideas and articles

From the conversation

Podcast episodes

Article history

Updates to this article

Tags