If AI systems are doing more and more of the work of building the next AI systems, how would anyone know whether that loop is actually speeding up? Alan Chan, a research fellow at GovAI, has turned his attention to that question. In a conversation with host Nathan Labenz on The Cognitive Revolution, he set out what he thinks labs and auditors should be measuring, what recent disclosures from Anthropic and OpenAI show, and what they leave out.
The stakes come from a scenario known as an intelligence explosion. AI that is good at AI research could effectively add researchers, which would produce better AI, which would add more researchers, in a feedback loop that might drive progress faster and faster. Whether that loop takes off or fizzles depends on how much extra progress each extra unit of research effort buys. The episode was recorded in two sessions, one just before and one just after Chan and his co-authors released a paper called What if automating AI R&D triggers an intelligence explosion?, and it was published on October 14, according to the episode page.
Three things to track
Chan said there are at least three things worth tracking. The first is how much of AI research and development (R&D) is automated at all. The second is whether that automation is actually making AI progress go faster. The third is whether oversight is keeping up. For example, are more errors creeping into the research process as AI takes it over? Is safety research falling behind capabilities research, so that it gets harder to keep proper control of what is going on?
The quantity he most wants is how much progress labs get as a function of what they spend on compute and labor. Economists who model an intelligence explosion call this the returns to research effort. More data on it, Chan said, would show whether an AI-driven intelligence explosion is possible at all. He acknowledged that much of this data is probably commercially sensitive and said it seems important enough to publish anyway. If companies won't publish it, he wants embedded evaluators inside the companies to collect it, estimate the returns to research effort, and report whether the labs have proper oversight of their own automation.
This three-part split mirrors Chan's earlier work. A March 2026 paper he co-wrote, Measuring AI R&D Automation, proposes 14 metrics and keeps automation, capability progress and oversight separate. It also warns that the relationship can be lopsided: automating many tasks may produce little speedup until the last task that still depends on humans is automated.
From "no way" to a coherent pathway
Asked for his own outlook, Chan described a shift. Before looking into it, he said, an intelligence explosion seemed impossible to him: "Super exponential growth? No way." After going through the evidence, it still seems "not super likely to me and quite difficult." But he now thinks that the data, rough as it is, points to a "coherent pathway to acceleration of AI progress." He also stressed bottlenecks beyond the research itself. Even very fast AI progress might not quickly turn into change in the real world, and that could limit the impact of an intelligence explosion even if one happened. Still, he said, the possibility is likely enough and the stakes high enough to justify much more tracking and preparation.
Labenz described the public debate as something of a stalemate. One side points out that the human brain runs on about 20 watts, which suggests far more efficient intelligence is possible. The other side insists that some bottleneck will always hold progress back. Chan said the more useful change has been elsewhere: in recent years, people have moved from conceptual arguments toward data-driven ones about whether automation speeds progress. The honest answer from that data, he said, is that it is uncertain. Some pieces point one way, some the other, and "the certainty bars are so wide" that acceleration cannot be ruled out. Failing to rule something out, he added, is not a strong statement.
He is optimistic that the evidence will grow, because many more people now care about AI R&D automation and several papers lay out exactly which data companies should give the public, governments or third-party auditors. He singled out work on the economics of recursive self-improvement for its long list of metrics. That likely refers to The Economics of Recursive Self-Improvement, a September 2026 paper whose rough calibration puts today's feedback below the point of self-sustaining acceleration, though it suggests the feedback is getting stronger. Its wish list includes how labs split research spending among human labor, experimental compute and inference. Labenz joked that the question might also be settled "by just seeing a foom," internet slang for a sudden takeoff. Chan agreed that was another way.
What the labs have disclosed
Labenz said he read recent posts from Anthropic and OpenAI as saying the companies feel they are in "some sort of early recursive self improvement dynamic." He also mentioned a graph he had seen suggesting that the gap between model releases had shrunk from about ten weeks to ten days.
The posts are full of usage numbers. Anthropic's September 17 report sampled 20% of relevant staff each week in July to build a fixed basket of about 15,000 tasks, then had Claude rate how automated each one was. For August, it estimated that AI led 26% of the measured work, that more than 90% involved at least collaboration with AI, and that no measured category was fully autonomous. By the report's own design, the basket measures how much existing work is automated, not whether that automation causes productivity gains. OpenAI's September 6 report said that by mid-August, agent runtime across its research organization equaled 3.1 eight-hour agent-workdays for every human workday. It also said high-level planning remained a small share of agent output, that more than half of successful four-to-eight-hour tasks involved human intervention, and that extra compute and other bottlenecks make it hard to turn these activity measures into a figure for overall research acceleration.
Chan welcomed the data but asked for two things. First, trends: these are initial snapshots, and only repeated measurements can show acceleration. It could be, he said, that "Mythos-level models" were "some sort of one-time productivity boost" that later tapers off. That phrase likely refers to Anthropic's Mythos Preview model. Second, he wants evidence of how much AI actually helps with AI research, beyond staff surveys. He was not sure of the right design but suggested something like giving one team the latest AI model and another team the previous version, then measuring how much more productive the first team is.
Activity is not progress
Chan's worry is that the figures that get attention in the media, such as lines of code written, experiments run and researcher hours or workdays, are "just a proxy for what we care about," which is actual AI progress. Labs could write far more code and run far more experiments while the rate at which experiments turn into progress stays low. To make the point, he said he would "throw out a random number": perhaps accelerating progress takes 200 times more experiments, while automation delivers only 10 times more. In that case, a big jump in activity would leave progress largely unchanged.
The hard question of research taste
The gap between looking productive and being productive came up again over research taste, meaning the judgment to pick ideas that will work. Labenz recalled an earlier episode with Chenglei Si, then a Stanford PhD candidate. Si's team asked whether language models could generate novel research ideas, and they rewrote human ideas in a model's prose style so that judges could not easily tell which ideas came from people. In that 2024 study, 79 researchers reviewed the proposals blind and rated the AI-generated ideas as significantly more novel, though somewhat weaker on feasibility. The study covered prompting-based natural-language-processing research that could be carried out with modest resources.
Then, Labenz said, the team followed up by actually carrying out the ideas, and the human ideas worked better. The execution study supports a version of that. Forty-three researchers spent three months implementing 19 human ideas and 24 AI ideas. Human-idea projects scored higher on average, but none of the direct idea-level differences was statistically significant. The clearer finding was that AI ideas lost significantly more ground between proposal and finished project. Labenz took the lesson that people are poorly calibrated at judging ideas on paper, and he offered a name for being too taken with a chatbot's suggestions: "Claude psychosis."
Chan pointed to a related result running the other way. He recalled a conference-paper study in which language models were better than humans at predicting which of two ideas would win empirically. That description fits a 2025 study led by Jiaxin Wen, which built test pairs from published papers. A fine-tuned GPT-4.1 with paper retrieval reached 77% accuracy, while off-the-shelf models stayed near chance. On 45 difficult language-processing pairs, the system scored 64.4% compared with 48.9% for groups of human experts. The test measures which method wins on benchmarks, not which research matters most.
Chan also cited an Anthropic finding that Claude was somewhat better at predicting the next research step. In When AI builds itself, Anthropic tested 129 moments from early-2026 research sessions where a human researcher had gone down an unproductive detour. Judged by a separate Claude model, Opus 4.5's proposed next step beat the human's actual choice 51% of the time, and Mythos Preview's did so 64% of the time. Because those moments were chosen as decisions that could be improved, it is a targeted test, not a head-to-head comparison with researchers in general. Chan said he would like an updated version of the conference-paper study to see where research taste stands now.
Testing whether oversight keeps up
Chan proposed using the same team comparison to measure oversight. Give Team A a newer model and Team B the previous one, ask both to write an AI R&D experiment, and then check whether Team A's output contains more errors or uncaught bugs. Researchers could even prompt the model to slip in backdoors on purpose and see whether the people overseeing it catch them more or less often with the newer generation. He said metrics like this get much less attention than straightforward capability measurements, but he considers them important for things going well.
Why uplift studies are hard
Labenz noted that such experiments are very labor-intensive. He pointed to METR's well-known study measuring how much AI tools sped up coding, which found a negative effect. In that 2025 trial, 16 experienced open-source developers worked on 246 real issues in codebases they knew well. Tasks where AI was allowed took 19% longer, even though the developers had expected a 24% speedup and afterward believed they had gained about 20%. METR described the result as a snapshot of early-2025 tools and experienced maintainers. Its February 2026 follow-up estimated that completion times fell 18% for returning developers and 4% for new ones, but both ranges included no effect. METR judged those estimates unreliable, partly because developers increasingly declined to take part or held back tasks they did not want to attempt without AI, so it announced a redesign.
Should the government require companies to run such studies? Chan said writing specific metrics into law would probably be inefficient, because the field is changing so fast. His preferred approach for now is embedded auditors with broad access, power and resources inside the companies. He acknowledged that one to three embedded evaluators would struggle to make staff take part in controlled experiments. Even without experiments, though, they could see how much a company spends on compute, human labor and data, and use those figures to estimate the returns to research effort.
A job map for safety research
The measure Chan said he would most love to see compares safety progress with capabilities progress. AI safety researchers have often hoped that automating safety research would keep pace with automated capabilities research, but the debate has rested on conceptual arguments. He suggested building an ONET for safety work. ONET is a U.S. database of jobs and the tasks each involves. Epoch AI has drafted a version for AI R&D that covers more than 60 tasks, from designing and running experiments to communicating results. Chan described it as survey-based, though Epoch says its first version came from literature review and brainstorming, not a completed workforce survey.
A safety equivalent would list tasks such as conceptual reasoning about whether an alignment method like a debate protocol would work, writing model reports and model cards, auditing for rare failures (finding the "needle in the haystack" among 10 million transcripts), and helping investigators work out what went wrong in an incident. Evaluations built on that list could feed a dashboard. As an illustration, Chan imagined that in two years, safety capabilities might be improving 3 times a year while non-safety capabilities improve 5 times a year: "3x versus 5x, this is pretty concerning," and a signal to invest more in safety.
Labenz compared this to Safetywashing, a study co-authored by Dan Hendrycks which found that many safety benchmark scores closely track general capability. Chan said his idea works at a different level: tasks done at the company level, not scores on a single model. He expects better AI researchers to also get better at safety work, simply because they are better at coding and reasoning, but he suspects a gap will remain. His evidence: models are "arguably superhuman at cyber," yet unreliable at helping METR and Redwood Research investigators figure out what happened in recent incidents. "That's not a very comforting description of where we are," Labenz said.