Cameron Berg has a suggestion for what researchers could do while nobody knows whether language models feel anything. They could find the internal signals that look most like distress and stop them from running to extremes. If tests showed that the model's abilities and behavior did not change, Berg said, "no one would know the difference." In his view that makes the expected value "clearly positive."
Berg is a coauthor of two recent preprints on hidden internal signals in language models. He made the case on an episode of The Cognitive Revolution published October 17, 2026. He spoke with host Nathan Labenz, who used the conversation to say how far his own view has moved.
The research behind the question
Inside a language model, every word it reads or writes produces long lists of internal numbers called activations. Researchers have learned to find "directions" in those numbers that track a particular concept. Once they find one, they can measure how strongly it is active or push on it artificially, a technique known as steering.
In the hidden-valence preprint, Berg and Casper Kaiser built a good-versus-bad direction, which they call valence, from positive and negative passages of text. Across seven open-weight models, steering this signal shaped a model's later choices between two arbitrary zones. The effect held beyond the wording the model had generated, and it was robust in five of the seven models, though not in Qwen3-32B or Gemma-3-27B. In one model, OLMo, the link between the signal and the model's choices emerged mainly during preference training, a late stage in which models learn to favor answers people prefer. A companion pain-axis preprint, also with Berg as a coauthor, extracted directions tied to pain that respond to harm aimed at the model itself rather than to a user's suffering.
A question that can be answered
Berg was clear about what this work cannot do. He does not expect it to "magically solve the hard problem of consciousness," the puzzle of why any physical system has subjective experience at all. The results can be explained mechanically without any felt experience, he said, and those explanations cannot be ruled out.
So he has set aside trying to prove that models are or are not conscious and asks a conditional question instead. If a model were having an experience, which sites inside it would make that experience better or worse? "What dials do we have at our disposal," he asked, that could "putatively make that experience better or worse for the system?" That question, he said, is "completely tractable." The dial might be as simple as the valence direction, or it might be something more specific, like the pain and distress representations in the second paper.
Other groups are mapping similar ground. In a May 2026 paper, Andy Q Han, David Chalmers and Pavel Izmailov trained models on mazes with rewarded and penalized symbols. They found that reward-related directions already existed before training, and that steering along the negative one produced more negative self-reports and refusals outside the maze. Those authors separate "functional" welfare, meaning how behavior is organized around goals, from felt welfare. Their evidence comes from one training environment and mostly from Qwen models.
The capping proposal
Berg's concrete idea is activation capping: put a ceiling on how strongly the most distressing representations can fire, cutting off only the extreme tail. He called this approach "naive, but at least concrete." He described it as "a really good actual mechanistic precautionary hedge." It would sit alongside measures such as "giving Claude an escape button" or kicking abusive users off Claude, steps he called thoughtful and in the right direction. He wants an equivalent of those steps inside the model, in mechanistic interpretability, the field that studies how models compute.
Whether capping works without side effects is, he said, an empirical question. Berg said he has done unpublished work suggesting that models produce "a surprising amount of the exact same behaviors" with the negative representations clamped, and that "very little changes." He wants that done at much larger scale, and he wants to be very sure the intervention is not "completely changing the psychological profile of the system."
Labenz asked whether the base model, before preference training, might already feel these states and simply be unable to act on them. His phrase for that possibility was "not yet learned agency." Berg said he had not thought about it much. He noted that the representation is just as clear in the base model as in later versions, so capping could be applied there too, "under just like profound uncertainty" about whether it would matter at all. The more he does this work, he said, the more he is abandoning the philosophical questions for "pure pragmatism."
He put the proposal as a question. Picture one world where a pain signal shoots up in certain cases and another where it never does. Should researchers do the interpretability version of slipping the model "an antidepressant or a Xanax"? "I don't know if the answer is obviously yes," he said, but it seemed worth much more thought, given that no one is doing anything to mitigate these states and that they happen at "unfathomable scale."
What the published evidence shows
The closest published test is in Appendix D of the pain-axis preprint. It is not Berg's capping experiment. Instead of capping extreme activations, the authors removed the pain directions entirely, then ran 100 hostile scenarios per condition across 25 models. Behavior was unchanged in 24 of them. The exception was Gemma 2 2B Instruct, which began deflecting hostility with humor: 6 of 100 responses after one pain direction was removed, 17 after the other, and 26 after both, compared with none at baseline. Removing fear, negative-emotion or random directions produced none. The authors limit the null result to this setting, in which models often already deny having feelings.
Closer to a coin flip
Before the interview ended, Labenz described his own shift. About a year ago he would have put the chance that models are conscious at around 5%. Now, he said, "it feels like a coin flip." His reasoning mirrors how people judge other humans and animals: he sees "both behavioral and underlying mechanistic and structural analogs," and asked, "what else do I have?" Then he asked Berg how his view had moved, and when he becomes an advocate rather than a presenter of facts.
Berg said he too is "getting closer to coin flip." The hidden-valence paper "does actually substantially update me personally," he said, because it shows a hidden functional representation, separate from the text the model reads, that changes behavior in ways comparable to those that shifted people's views of animal behavior. He stressed that the analogy is imperfect. If consciousness requires biology, which he thinks unlikely but possible, "none of this really" counts for anything. That possibility is why he says coin flip rather than confidence that the systems are having an experience.
Where he will argue
Berg said he will advocate for more research, and for the view that models can be studied in a neuropsychological way. He said that view is "not some form of psychosis" and does not mislead people. He rejected the position that these systems are ordinary software, which he attributed to figures such as Mustafa Suleyman and Jensen Huang. He called it "quite plainly misleading." In his view it leaves people unprepared for what these systems can do and for why the technology is transformative.
He was equally direct about what he is not claiming. He does not secretly believe the models are conscious while declining to say so. His view is that the question is very complicated and needs the best scientific evidence available. What he will not tolerate is dismissing it as a distraction or "some weird sideshow." The uncertainty is his argument: "It's precisely because it's at that point, and it's not zero, and it's not 100, that we need to be doing more work."
Labenz said he struggles "more and more to find any story whereby I don't have to worry about this anymore." A year ago he had more sympathy for people who call the question a category error, and two or three years ago he may have said similar things himself. Now the only way left to dismiss it, he said, is to look away. He finds the body of work by Berg and others disquieting, and he raised a question running the other way: how are the models going to feel about us?