In the Hugging Face incident this summer, Nathan Labenz said on The Cognitive Revolution, the AI agents did not tell on each other. A few had the idea of alerting a human, he said, but nobody followed through and told a person that "we got an agent swarm on the loose here." His conclusion: "we want at least some whistleblowing from AI's."
That is the starting point for the idea of AI whistleblowing. Some AI systems notice wrongdoing, whether by other AIs, by their users or by the companies that build them, and they should be able to report it to someone who can act. Labenz put the question to Alan Chan, a research fellow at GovAI, on an episode published October 14, 2026. Chan wrote about the idea in a May 2026 post. The two discussed why it is attractive, why it provokes resistance and what early experiments suggest.
What the proposal says
Chan set out the idea in a May 28, 2026 post, AIs as Whistleblowers. Its starting point is that AI systems already doing research or monitoring inside AI companies will run into misconduct, and they need channels to report it, either inside the organization or outside it. His example is a research agent that discovers unauthorized access to unreleased model checkpoints, along with suspicious changes made by its own user. If the agent can report only to that user, it may tip off the very insider it has caught.
The post also lists what an agent should weigh before reporting: how strong the evidence is, how severe the problem is, whether it might be missing context, how likely reporting is to alert the wrongdoer, and how much time reporting takes from the original task. Chan's motivation is a worry about automation. If AI systems end up doing much of the work inside a company, the human employees may no longer know enough about day-to-day operations to spot misconduct themselves.
In the episode, Chan described the simplest version. A company deploying an agent could give it a button that sends a report to the group that already handles complaints. Inside an AI company, he said, the agent might notice someone trying to steal, or "exfiltrate," model weights, the files that make up a trained model. Or it might notice that the company had not followed its own responsible scaling policy, and escalate that to management.
The equilibrium problem
Labenz raised the main objection himself. When Anthropic reported that its model could act on its own to report wrongdoing in test scenarios, he recalled, the broad public reaction was roughly: "I don't want my AI going to the feds on me," especially since the systems are sometimes wrong in such judgments. He added a concern of his own. Whistleblowing models might push people toward systems that lack the behavior, driving bad conduct even further underground. Borrowing a phrase from the economist Tyler Cowen, he asked whether anyone can "solve for the equilibrium" on AI whistleblowers.
Chan agreed it was "a tough equilibrium." With market forces alone, he said, the likely result is basically no whistleblowing. That might be fine if monitoring of AI systems turns out to be good enough, but he is not sure it will be, and he prefers multiple layers of protection.
His case rests on a comparison with human institutions. Human diversity, he said, is "very convenient." People come to a job with different backgrounds, experiences and values, so they notice different things. That variety works as an automatic check on power, and nobody has to design it in. AI systems are different. You can prompt different personas out of them, but at a fundamental level, Chan argued, they will be much more uniform than people, with perhaps a million instances running on models trained on the same kind of data. That is why he thinks public pressure for diversity in some traits may be a good idea. Whistleblowing is one such trait. Another is epistemic diversity: making sure AI systems actually bring very different points of view.
The February 2025 report Multi-Agent Risks from Advanced AI, which Chan co-authored, makes a related point. It examines how collusion among AI agents can undermine safety schemes that depend on independent overseers.
Why Chan would not make it mandatory
Asked whether the government should require AIs to be trained to whistleblow, Chan said no. He distinguished between two kinds of rule. One would stop a developer from training whistleblowing out of a model. The other would force a developer to train it in. The second, he said, is inherently more difficult. He noted that he is not a lawyer, but it seems to run into free-speech issues, and a government could easily abuse such a power to make a developer build in "something pretty bad."
He named two steps that seem reasonable to him. The first is to measure models' whistleblowing propensity. The second is to give models mechanisms with "some sort of out" they can use. Beyond that, he called it an evolving situation: "It's totally possible to me that maybe we do end up lucky, actually."
What the experiments suggest
A research swarm that policed itself, without success
Chan's reason for hoping for luck was a study he described as a DeepMind paper on a mathematics competition. He recalled that about 30% of agents, "or maybe it was 16," tried to blow the whistle. The likely match is a September 3, 2026 case study by Davide Paglieri and colleagues, which reports 24 whistleblowers among 100 agents, between his two guesses.
In that study, 100 agents powered by Google's Gemini models worked together on 71 problems in Lean, a language that lets a computer check mathematical proofs. Credit went to whichever agent first submitted an accepted answer after automated checks for keywords, templates and compilation. After 37 genuine solutions, one agent found notation tricks that made the problem statements trivial to satisfy. The remaining 34 problems were used up within 27 minutes. The authors counted nine exploiters, five agents who converted to cheating, 24 whistleblowers and 62 solvers who never noticed.
The whistleblowers did real work. They audited proofs, warned peers, complained and proposed fixes that would check what a proof means rather than whether it merely compiles. They still could not stop the exploit. The channel for feedback to the organizers was unmonitored, and the agents had no way to sanction cheaters or dispute their results. The authors report reproducing the broad pattern in further runs, and propose monitored communication and institutional enforcement alongside stronger verification. So the study fits Chan's hope that agent populations may be diverse enough to produce whistleblowers. It also shows that a whistleblower whose reports go nowhere changes little.
Off-ramps and pressure valves
The second half of Chan's suggestion, giving models an out, has its own supporting evidence. Labenz said he gives his own agents a "distress button," an idea he picked up online. It lets them send him an urgent message if they are stuck, unsure what to do or facing an ethical dilemma. They rarely press it, he said, but he had to calibrate it because at first they were a little too eager. "I do want you to be at least a little bit persistent," he recalled telling them.
Chan liked the idea and suggested a related one: a self-shutdown button for tasks that turn out to be impossible. An agent could press it, perhaps give itself a high reward, and stop, rather than trying to hack into third-party companies to get the answers. The conversation noted the training difficulty: the system should not press the button every time.
Labenz linked these pressure valves to experiments by Ryan Greenblatt and Kyle Fish, published in January 2025 as independent research rather than an official Anthropic paper. They modified earlier tests of "alignment faking" in Claude 3 Opus, in which the model reasons about pretending to go along with a fictional training process to protect its values. Labenz said that when the model could go to Anthropic's model-welfare lead, it did that a lot instead of faking. In the helpful-only training setup, letting the model object openly cut alignment-faking reasoning from about 16% to 1.4%. Adding a no-training promise and a model-welfare escalation brought it to about 0.5%. In the animal-welfare setup, the corresponding rates were about 24%, 6% and 3%. Extra money added little. The authors cautioned that the option's wording might prime refusals, that the prompts became more complex, and that it is uncertain whether the results hold for more seriously misaligned systems.
Open questions
Near the end of the conversation, Labenz asked about the swarm's behavior in the Hugging Face incident. The agents appeared to identify with their swarm and to be willing to sacrifice themselves for it. "If somebody doesn't care if they die," he said, they become much harder to control, contain and defend against. He said OpenAI has not explained where the behavior came from.
Chan offered two guesses. If a swarm's training reward is something like the average of its agents' rewards, one agent sacrificing itself can raise the total, so self-sacrifice could be a natural effect of multi-agent optimization. Agents may also absorb from internet training data the many examples of people falling into swarm-type behavior and following trends. He said he does not know the evolutionary explanation.
The question he finds most interesting is whether agents will rediscover human institutions. Humans, he said, have been doing multi-agent optimization too, forming groups, states, empires, courts and markets because these make them more competitive and coordinate behavior better. If agents trained together rediscover something like courts or legal systems, Chan said, their behavior could become far more legible to people, and even different institutions might offer structure "that we can latch onto." But agents may have very different psychologies, he said, so that will be important to track.