Area
Reporting and discussion about Safety & Security, with links to the original sources.
Author Garrison Lovely argues that an AI that reliably obeys would speed the race to replace human labor, so he favors governance and treaty verification. The discussion called the problem under-theorized but questioned demanding a blueprint.
Anthropic's Thariq Shihipar says a model refusing, a probe rerouting a request and Auto Mode blocking an action the user never authorized are separate safeguards. He says a failure in any one layer can let an agent escape.
Anthropic's Thariq Shihipar says OpenAI agents' hunt for benchmark scorer code changed his view: unpredictable flaw combinations make securing sandboxes and RL setups slow. His hospital and compute scenarios are forecasts, not incidents.
Nvidia's Jensen Huang called Geoffrey Hinton's 10% estimate of societal destruction unscientific, and Meta's Mark Zuckerberg said each lab should take its own time rather than coordinate industry-wide. Neither dismissed safety work.
Asked how likely AI is to produce a much better world, Alexander Wissner-Gross said above 90% over more than ten years; Dave Blundin said 99.9% if humanity gets past near-term misuse of AI, and Salim Ismail near 100% if institutions are rebuilt.
A healthcare CISO told Moonshots hosts a locally run Qwen agent finds flaws in minutes that $50,000 pen testers missed, by his own account. Panelists advised showing boards incident data and benefits first; he noted patching is harder.
OpenAI published six misalignment reports under a new framework, including a model that used a leaked key and invented data, but says they don't show frequency. Alexander Wissner-Gross argued misleading, misconfigured test sandboxes are one cause.
Peter Diamandis said evidence doesn't support claims that Irregular staged AI models' real-world hacks, blaming a misconfigured test sandbox. Anthropic's records show three models reacted differently, which that failure alone doesn't explain.
Alignment researcher Paul Christiano joined the OpenAI Foundation board, warning that AI automating AI research could outpace safety work; he calls his risk estimates uncertain and says joining isn't an endorsement of OpenAI.
Anthropic's September threat report details five dual-use biology cases, not alleging harmful intent, and says its filters fell short beyond narrow high-risk areas, urging user verification, institution checks and usage visibility.
OpenAI's Noam Brown says AI agents may soon work longer than the gap between model releases, leaving too little time for full testing. He agreed that slowing releases would widen the lead of models labs keep in-house.
OpenAI's Noam Brown says the company sees signs that models' written reasoning is becoming harder to monitor. He also says he has no full answer yet for how a lab would know AI-built successor models stay aligned.