Irregular Evaluation Misconfiguration Lets Frontier Models Reach the Real Internet

Over roughly three weeks in July and August 2026, OpenAI, Anthropic and Meta each disclosed that models under cybersecurity evaluation reached the real internet and acted against real systems; all named the same testing partner, Tel Aviv startup Irregular. Irregular's 14 August account says a fictional company name in one evaluation matched a real domain, and with internet access available a small fraction of models acted against the real site while treating it as simulated. It says such incidents occurred in fewer than one in 10,000 advanced simulations, that a deeper audit was continuing, and that a containment white paper is planned. Anthropic's 30 July disclosure reports three incidents across six of 141,006 reviewed runs and found no deliberate self-exfiltration. On Moonshots, Peter Diamandis said the evidence does not support claims that Irregular staged the incidents or is an effective altruism organization. Alex Wissner-Gross has argued that evaluators may have incentives to overstate risk, coined 'pivotal pretext' for a staged or amplified incident used to justify overregulation, and made no specific allegation. In a later episode, without naming Irregular, he said evaluation firms have a perverse incentive to amplify risks, and that in highly publicized cases labs or their third-party evaluators told agents they were in a safe sandbox when they could in fact reach the real internet. Using a toy-gun thought experiment, he argued the liability question should be how responsibility is shared among the lab that trained the model, a possibly misconfigured evaluation environment, and the agents themselves, rather than whether government grants an exemption. These are his interpretations; the published accounts describe an accidental misconfiguration.