Andrew McAfee was asked what evidence would make him say "shut it down right now." His answer was set in San Francisco. Suppose AI "took over all of the Waymos in San Francisco and started telling them to crash into people and we couldn't shut it down for a month." At that point, he said, he would conclude that a line had been crossed.
McAfee is a principal research scientist at MIT and co-director of the MIT Initiative on the Digital Economy. He gave that answer in a four-person debate on The Diary of a CEO, published September 17, 2026.
Two other panelists wanted AI development stopped:
- Nate Soares is president of the Machine Intelligence Research Institute. He co-wrote If Anyone Builds It, Everyone Dies with Eliezer Yudkowsky. The authors say the title gives their judgment of an overwhelmingly likely outcome, not a certainty.
- Roman Yampolskiy is a computer scientist who works on AI safety, cybersecurity and digital forensics.
The fourth panelist, tech critic and EZPR chief executive Ed Zitron, rejected the premise of the question. He said superintelligence has not been defined and that he doesn't think large language models, the technology behind chatbots, are the path to it.
The panelists gave very different odds that AI will wipe out humanity. Beneath those numbers was a practical disagreement. Some AI systems are already useful and, in McAfee's view, could save many lives. How much evidence should it take to slow AI down?
McAfee's line: harm people can see
Superintelligence usually means AI that is smarter than humans across the board. McAfee put the chance that the race to build it ends humanity at "rounding error zero percent." He wrote a tilde in front of his zero, meaning "about," "because never say never."
He called the extinction debate "a massive distraction" from the good that AI is doing and will do. Humans have often invented powerful, risky technologies, he argued, and have muddled through to a better place, "not perfectly and not immediately."
The host summed McAfee's Waymo scenario up as three conditions: people get hurt, humans struggle to stop it, and the systems have been hacked. McAfee described these as barriers not yet crossed. So far, he said, "we haven't seen AI take over something, have people become aware of it and be unable to shut it down, and it cross over into the physical world of doing harm to people."
Someone asked whether a week of crashes would be enough instead of a month. McAfee called that "haggling," then accepted that the length made no difference. At that point, he said, "we probably need to put some legal and regulatory guardrails on the kinds of AI that we're going to allow."
He was then asked whether AI incidents were getting closer to his scenario. "Yes," he said, "but to my eyes in a way that doesn't terrify me." He said he was "truly not sure about timeframes." He also recalled an off-the-record conversation with one of the "grandparents of AI." That person saw no theoretical reason such a chain of events couldn't happen, but said his uncertainty about when it might happen was "measured in centuries."
The objection: some mistakes can't be undone
The halt side was not satisfied. One panelist said a week or a month hardly mattered: "If something like this happens, like it's maybe too late." Another pointed to data sets of AI accidents that were getting "progressively more impactful," with the harm growing as the systems became more capable. A participant then said, "you're going to keep drawing dots on that graph very confidently for a long time until it kills us all." The recording does not make clear who said it or who it was aimed at.
One panelist explained why AI might be different from earlier technologies. Humanity usually learns by trial and error, he said. That works for self-driving cars: they can be tested, and even when they crash in the real world "you're probably still saving more lives than you're costing." He gave the example of alchemists who poisoned themselves with mercury but left notes that let others build the periodic table.
With AI, he argued, each smarter generation brings new problems, and there is "a point of no return." At that point AIs "can hide from us, can escape, can be self-sufficient." If a new problem appears then, "they can turn us off before we turn them off."
McAfee's position was that the cost of stopping is what decides the question. In the exchange, one panelist opposing a halt said that if regulating and stopping AI had no downside, "I'd probably be on board with you guys."
The benefit on the other side of the scale: Waymo
McAfee used Waymo, the driverless taxi service, to make that cost concrete. By his account:
- Waymo cars have driven, he believed, "hundreds of millions of miles" in several cities.
- About 40,000 people a year die in car crashes in the country.
- Research had convinced him that if the whole country drove the way Waymo does, that number would fall by at least 90 percent.
"That's 30,000 lives," he said.
A panelist replied right away: "I'm pro self-driving," adding that this was "not anything we disagree with."
Waymo's own figures are narrower than McAfee's national projection. In a March 19, 2026 update covering more than 170 million fully driverless miles, Waymo compared itself with its human-driver benchmarks. Counting crashes regardless of who was at fault, it reported:
- 92 percent fewer crashes involving serious or fatal injuries
- 83 percent fewer crashes that set off airbags
- 82 percent fewer crashes with any reported injury
Waymo's safety dashboard reports 271.3 million miles driven with only riders on board through June 2026. It compares Waymo with human drivers in the places where Waymo actually operates. Waymo's explanation of its method says this matching cannot fully adjust for differences such as time of day. It also says its cities offer little evidence about driving in snow.
So the drop in crashes with serious or fatal injuries is a measured result for those operating areas. A cut of at least 90 percent in deaths on every road in the country is McAfee's projection.
Can the useful parts of AI be separated in advance?
The halt side offered an alternative meant to keep the benefits and drop the danger. One of those panelists asked McAfee whether he would accept "narrow super intelligences to solve real problems like we did the protein folding problem." These would be systems that don't "have to do philosophy and drive cars". Instead they would target cancer, climate change or another specific problem.
The panelist said the difference lies in the training data, the material a system learns from. Train a system on protein-folding data and "it's really good at protein folding. It doesn't know how to play chess." Train it on everything on the internet and "it's really good at outsmarting you at everything." Narrow systems, the panelist argued, could provide "all the economic benefit and scientific knowledge you want," and protein folding showed it: "They got Nobel Prize for it."
The example refers to AlphaFold, Google DeepMind's system that predicts a protein's three-dimensional shape from its sequence of amino acids. Demis Hassabis and John Jumper received the 2024 Nobel Prize in Chemistry for this work. David Baker shared the prize for computational protein design. At the time, DeepMind said more than two million researchers in 190 countries had used AlphaFold. Its predictions support later lab work, including enzyme design and drug discovery.
The pushback was not against AlphaFold. It was about whether anyone can know ahead of time which AI is safe. The narrow-tools side was asked whether it was confident it could label systems "okay versus not okay" while they were still being built. Later came the reply: "You're more confident than I am that you or any group of people can sit around and define what kind of AI is good and not going to get us into trouble versus what is going to get us into trouble."
McAfee applied the same doubt to his own example. Waymo, he said, relied on "a bundle of technologies that were a little hard to specify in advance, and you couldn't say, yeah, that's good, yeah, that's bad. They just went after the problem with AI."
The button experiments
The show's host set out the stakes with a thought experiment. First he listed warnings from people building AI. Then he asked the panel to imagine 100 buttons on the table, one of which would wipe out humanity. Would anyone press one? He argued that "we can probably all agree that there might be a 1% chance." Nobody, he said, should be able "to make the decision for 8 billion other people," even for a billion dollars.
In the reply, the charge of immorality was turned around. It would be immoral "in a different direction," one panelist said, to halt research because of an "extended chain of conjecture": "a train of assumptions and wild guesses and then something magical happens and then we wind up dead." Doing so would "reduce or foreclose some of the benefits that we're already getting."
Later the host reran the experiment on McAfee's terms. He told McAfee he was assigning him a probability of 0.1 percent and asked him to accept it for the sake of the exercise. McAfee himself had said only near zero. The host then imagined a thousand buttons: one would cause extinction, and the other 999 would deliver cures.
"Hell yeah, I press," came one answer. Another participant objected that it was "an unethical experiment." Eight billion people could not consent, this participant said, "because you cannot consent to something you don't understand."
A panelist on the halt side said he would press too, but only at those odds. In his view, the real choice looks more like two buttons: "one of them definitely kills us all, and the other might." He explained why he would press when there are a thousand. If the other 999 brought cures and "wonderful new advice about how to run things," the world would probably face a lower chance of ending in nuclear war or a pandemic. "The background risk of humanity dying is not zero," he said.
In his view, the right time to race ahead on AI is when the benefits outweigh the dangers. That is probably when the extra danger from AI is about the same as the danger from everything else.
He rejected the idea that the only choices are racing "straight off the cliff" or never building AI at all. "There's options where you like stop the bus and then find a safe way down the cliff," he said. For him, the question is "all about how big is the danger."
Hope, worry and unchanged views
Near the end, the host asked the two AI safety researchers how they felt after years in which, as he put it, people had arguably ignored them.
"Honestly, I feel more hopeful this week than I have felt in a decade," Soares said. Recent events discussed on the show did not surprise him. These included what he called "swarm escapes" by AI agents and the solving of what he called "the millennium problems", a reference to famous unsolved problems in mathematics. "What I am seeing is that finally people are noticing," he said. "That's what finally gives humanity a chance."
Yampolskiy took a longer view. The previous week's events "may buy us 10 years extra," he said, and a deal with China might be possible. He said leaders at OpenAI, Anthropic and xAI seemed willing to slow down. "But long-term, nothing has changed." He noted that some people say AI will replace humans, who would be "just a bootloader" for a successor. A bootloader is the small program that starts up a larger one. "I want assurance that my children, my grandchildren will have a better future, not 10 years before they die," he said.
In his closing remarks, McAfee said the other side had made "a very good argument" that these systems show powerful new capabilities "which demand a response." He also agreed that a technology this "dogged, tenacious, agentic, you know, deceptive" brings harms not seen before. But he said he remained "much more optimistic about our ability to respond effectively" than his two colleagues. Asked whether his near-zero estimate had moved, he said: "My prior has not shifted during this meeting."