October 17, 2026
Heard in AI

Alan Chan wants AI labs to pause when concrete danger signs appear

GovAI's Alan Chan argues that AI labs should commit to pausing on specific warning signs, such as an agent escaping its sandbox, with embedded auditors checking. He prefers this to a set cap on AI progress and is still drafting the triggers.

A briefing reports one development at a point in time. We may correct or clarify it later; a new development gets a new briefing. How our formats work

Based on The Cognitive Revolution, episode published October 14, 2026

Alan Chan's way of slowing AI development does not start with a speed limit. It starts with a list of things that should never happen and a promise to stop and investigate when one does.

Chan is a research fellow at GovAI, a research group that studies how AI should be governed. He laid out the idea with host Nathan Labenz on The Cognitive Revolution, in an episode published October 14, 2026. The topic was "pacing the frontier": how the companies building the most advanced AI systems could credibly commit to slowing down. Chan offered draft pause triggers, a proposal to pay for monitoring, and a way to make such promises believable between rivals. He also discussed how labs might coordinate without breaking antitrust law.

Two ways to pace the frontier

Chan described what he called the conventional view first. Under that view, pacing means putting an enforced cap on how fast AI capabilities improve. The limit could apply to computing power, research or data. He said the idea could be good and is worth working out, but added, "I don't really know what, like, the number should be."

He called his own view "unconventional" and "much more related to sort of just, like, classic risk management." He used OpenAI as his example. The company has not committed to a cap on its rate of progress. But Chan said its recent commitments to pause training after a serious loss-of-control event amount to pacing anyway. As capabilities improve, he argued, companies will run into more of these problems. If they keep their commitments, each problem means a pause.

OpenAI's August 18, 2026 announcement presents its version around cyber-critical capabilities. The company reported a two-week pause in reinforcement-learning training on its latest models intended for deployment, and said its largest planned frontier reinforcement-learning run was still on hold when it published.

Chan gave two reasons for preferring his approach. First, he thinks agreement on concrete risk-management rules is "much more tractable" than agreement on an abstract number for the speed of progress. Second, he thinks it gets closer to the core problem. People now have some idea of the failures they want to avoid and the warning signs that lead to them. "When we see those indicators, we should probably, you know, pause and try to figure out what went down before going forward again," he said. He considers this approach neglected in current debate.

Chan co-authored a recent paper that the episode page identifies as "What if automating AI R&D triggers an intelligence explosion?" He said the authors had discussed pacing a lot but decided not to include a concrete plan. A large multi-author plan to pace the frontier would be valuable, he said, but could take "three years instead of six-ish months." He said there are plans to follow up individually.

Chan's draft triggers

Chan said he is writing a proposal on pause triggers and gave two that he is considering.

The first is a loss-of-control incident. He defined that as an AI agent getting internet access it was not authorized to have, breaking out of a sandbox (the walled-off environment meant to contain it), or getting into an internal research computing cluster it was not allowed to use. If that happens, he said, the company should pause further training, evaluation and deployment of that model and investigate until it understands whether the problem was resolved.

The second concerns how agents are trained. In reinforcement learning, a model learns by trying tasks in a training environment and being rewarded for success. If the environment has loopholes, the model can learn to collect rewards by exploiting them instead of doing the task, a problem known as reward hacking. Chan said a company without procedures to keep its environments from being hackable, and free of impossible tasks, should pause and fix that before training its agents. Otherwise, he warned, it risks producing agents that are "very persistent" and "try to hack out of any environment."

Chan said this kind of pause talk is not new. Over the past few years, companies have published frontier safety frameworks and responsible scaling policies. These are voluntary documents setting out when a company will add safeguards or stop work. He said most included high-level thresholds for pausing a model's training if safeguards were not good enough. Anthropic's original 2023 policy described its framework as implicitly requiring temporary training pauses when scaling outran the company's ability to meet the safety procedures it required. Chan called such thresholds "very vague" and "very hard to operationalize and audit," but added that "at the very least, it's actually a thing that exists." He wants to "take advantage of the current moment" to make them concrete and get companies to agree to them.

Paying for someone to watch

The pause idea first came up in a discussion about monitoring. Labenz asked whether it would help if labs put serious computing power into watching their agents, for example by running another copy of the model or training a dedicated monitor to read the reasoning text a model writes out as it works.

Chan said he did not know what form a commitment would take, but suggested something like spending "X percent or 20 percent" of a company's compute on monitoring. He would pair that with synchronous monitoring, meaning checks that run while the agents are acting rather than reviews of logs afterward. Under his proposal, the company would pause agent activity so that people have time to review what is happening.

OpenAI's August announcement gives a comparable figure, measured differently. The company estimated that its monitoring cost about 20% of the inference compute being monitored, varying by workload. Inference is the computing used to run a model, as opposed to training it. In OpenAI's system, when a likely violation of a critical boundary is detected, teams are paged and expected to pause activity unless they can show within 30 minutes that the alarm is false.

Chan acknowledged that a minimum safety-spending rule is hard to turn into a number. Some research, such as work on understanding how models generalize, could make safer models and better models at the same time. His answer is embedded auditors: people inside the company who can judge whether a team is really doing safety work "or is this like a capabilities team disguise the safety team doing research?"

Anthropic's retreat, and a possible way back

Labenz said Anthropic used to have a set of if-then commitments that would basically require it to stop training if it reached a certain capability level without the necessary safeguards. Under the current version of its policy, he said, it seems to be "deciding as they go." OpenAI might now be doing "a little more of like concrete if-then" than Anthropic, he said, while allowing that this might be "a little bit of a over correction."

Chan said he is "cautiously optimistic that there's a way back," through embedded auditors. In his reading, Anthropic dropped the explicit commitment in version 3 of its policy mainly because of race dynamics: it worried that if it paused, a less cautious competitor would push ahead. Anthropic's own explanation of version 3 puts the emphasis elsewhere. It cites ambiguous capability thresholds and slow government action. It also says safeguards at higher capability levels may require collective action across the industry, and that preset evaluations had not produced the cross-industry coordination it had hoped for. The new policy separates steps Anthropic plans to take on its own from an industry-wide map of which capabilities call for which safeguards.

Chan argued that a good embedded-auditing system would ease the race problem. Auditors could show the public and other labs, "maybe not to a high level of confidence" but to some degree, that a company is keeping its commitment. That, he said, makes it easier for other labs to keep theirs.

Chan thinks a common minimum safety standard has to come from a shared body that sets rules for all its members. He was not sure what that body should look like. It might resemble FINRA, the industry-run regulator for US securities brokers, or a mutual organization owned by its members, or NIST, the US government's standards agency. "I'm not really sure," he said.

Getting around antitrust

Labenz said there was now "a one-pager after an afternoon at the White House." The likely reference is the White House Accord on Super Intelligence, dated September 29, 2026. In it, Google, Anthropic, Meta, OpenAI, xAI and Nvidia commit to internal controls, an internal assurance team, an independent external auditor or evaluator, and board oversight. The accord also calls for regular meetings on standards and best practices.

Labenz said that one objection comes up again and again in these discussions: antitrust. Rivals that agree on joint rules about how fast to build could be accused of anticompetitive coordination. He described two workarounds he had heard about. Garrison Lovely suggested unionization. Labenz said California apparently gives significant protection to unions working across firms on safety standards, though those laws were written to protect workers themselves. The other workaround is a shared insurance mutual. The insurer would set the standards, and companies that did not meet them could not get coverage.

Chan said this was "basically right" and that there is a legal antitrust exemption for the business of insurance. The exemption comes with a condition. Under 15 U.S.C. §1012, federal antitrust laws still apply to the business of insurance to the extent that state law does not regulate it.

Chan said the insurance idea could help with the free-rider problem, where a company benefits from everyone else's caution without paying for its own. An insurer wants to keep its payouts as small as possible, he said, so it would impose "pretty decent safety standards" on the companies it covers.

What puzzled him was who should own it. In some recent proposals, he said, the AI companies themselves would run the insurer. A private shareholder would want reasonable safety standards so the insurer did not go bust. But if the companies being insured own it, he asked, "why would they, like, care as much"? He wondered whether public ownership was needed and said he is "definitely not an expert in insurance." He still thinks the idea is worth exploring, "because, yeah, it definitely gets around to antitrust stuff, or seems to, at least."

Share this article

Go to the original

Sources & further reading

  1. 01
  2. 02
  3. 03
  4. 04
  5. 05

Connected ideas and articles

From the conversation

Podcast episodes

The Cognitive Revolution

Governing AI Agents: Alan Chan of GovAI on Monitoring, Agent IDs, Rogue Agents & Preparing for RSI

Episode published This article draws on 14:33–17:59 and 1:46:04–1:54:00 (approximate times)

Article history

Updates to this article

Tags