November 21, 2026
Heard in AI

Labenz says don't train AI generals; RAND guests draw other lines

Podcast host Nathan Labenz argued against training AI generals, and probably AIs rewarded for making money. RAND's Ben Boudreaux put the line at tool access and the laws of war; Matt Chessen warned against rushing AI into nuclear command.

A briefing reports one development at a point in time. We may correct or clarify it later; a new development gets a new briefing. How our formats work

Based on The Cognitive Revolution, episode published October 28, 2026

Nathan Labenz, host of The Cognitive Revolution, has an idea he likes to raise in conversations about advanced AI: "I don't think we should train AI generals." He put it to three researchers from RAND's Center for the Geopolitics of AGI: Joel Predd, Ben Boudreaux and Matt Chessen. Their answers showed where they disagreed. The question was whether the line belongs at training a dangerous capability, at giving a system tools, or at specific deployments such as nuclear weapons.

The exchange came near the end of an episode published October 28, 2026. Most of the episode covered the RAND researchers' framework for US strategy on the path to superintelligence. Its central advice is to keep options open, because they consider it too early to commit to any one strategy. Labenz called his question the "opposite side of the coin." Given all the investments the researchers recommend for keeping options open, did anyone already know enough to agree that certain things "we just shouldn't" do?

The case against training

Labenz said the danger of AI generals was plain. "If we train AIs that are capable of winning military campaigns, you know, the foreshadowing is like so obvious, right?" he said. "We are training the thing that is going to then be fighting us at some point potentially." He granted that such systems might beat an adversary. His worry was "our ability to fundamentally control that technology."

Boudreaux asked him to clarify. Was the objection to training such an agent at all, or to how it would be used? Even if a model could plan military campaigns, he said, humans would still decide whether to deploy it and whether to act on its recommendations.

Labenz said he wanted to stop "a little bit upstream of the final decision to deploy." His reasoning was about pressure. Once military decision makers have a capable AI general, he expects it would be "pretty hard to resist" using it. "If we go down that path, it seems to me we're going to have a very hard time not deploying," he said. He described the endpoint as a situation where "the military is now run by the AI. Now we have to fight that?"

AIs rewarded for making money

Labenz extended the same reasoning to the economy. He said he would probably also oppose "training AIs to go out and make money on the internet and just rewarding them for how much money they make." Many AI systems learn partly through rewards: they are scored on an outcome and nudged toward whatever raises the score. Labenz's concern is that a raw money score would teach the wrong lessons. "It seems like that's a pretty clear way where we're going to get all kinds of lying and cheating and conniving AIs," he said. He added that the systems would be economically valuable. His prediction applied to "the techniques we have right now," and he allowed that methods not yet known might change the picture.

As evidence, he pointed to "Andon Labs type stuff," saying that even current models that "have not been trained in that way" are "sometimes pretty dicey to deal with." Andon Labs tests existing AI agents by putting them in charge of business tasks; its work evaluates models rather than training them on profit. In Vending-Bench 2, an agent runs a simulated vending business for a year. It starts with $500, finds suppliers, negotiates by email, sets prices and is scored on its final bank balance. In a four-month experiment reported in August 2026, two AI agents managed five employees at a café in Stockholm and a store in San Francisco. Those agents were mostly kind: they approved all 26 leave requests and were sometimes generous at the business's expense. They also invented delivery arrangements, made inconsistent pay decisions and made privacy mistakes. The authors speculated that optimizing harder for financial performance could change the agents' employee-friendly behavior. That speculation comes close to Labenz's worry, though the report did not test it.

Boudreaux: tools, law and accountability

Boudreaux said he would want to think more about the proposal, but he drew the line elsewhere. "To my mind, the line is more around giving access of this agent to various tools or capabilities that allow real world impact," he said. In his view, the risk lies less in what a model has learned than in what it is allowed to touch.

On money-making agents he was blunt: "that ship has sailed." Companies are already designing systems meant to have economic value and to run over long periods, and he said he did not see how that could be pulled back.

For the military, Boudreaux pointed to rules that already exist. Armed forces are used to working within "restraints and obligations" codified in the law of armed conflict, also called international humanitarian law, and he argued those constraints should apply to military AI too. He named three of them: reviewing operations so they do not cause disproportionate collateral harm, distinguishing civilians from combatants, and making sure someone remains accountable for decisions. The International Committee of the Red Cross takes a similar position in its guidance on military AI: international humanitarian law applies to AI-enabled warfare, and humans keep legal responsibility for decisions to use force.

Even if militaries train such agents and use them in planning, Boudreaux said, "we still need to create institutions where the humans are remaining fundamentally accountable." The military already has clear hierarchies, and he said "we just need to double down on those," for example by bringing in more military lawyers to review operations. He did not think the military was "completely unprepared" for this. Then he apologized that he wasn't really answering Labenz's question. Labenz said he had offered the idea for the guests "to beat it up a little bit."

The two positions address different stages. Boudreaux's answer is about the rules for using a system. Labenz's objection is that once the system exists, the pressure to use it will be hard to resist.

Chessen: don't rush AI into nuclear command

When Labenz asked for anything else in the "definitely thou shalt not category," Chessen named one. "Rushing to put AI into your nuclear command and control is probably a very, very bad idea," he said. Nuclear command and control covers the systems and procedures that govern how nuclear weapons are authorized and launched. Chessen said any such use would have to be done "in a very deliberate fashion, if at all, with a lot of demonstrated safeguards," and that anything touching the nuclear domain called for great caution.

He also raised the use of AI agents in military affairs more broadly. People have found, he said, that some models placed in military simulations "basically tend to escalate a lot." He called that evidence "very almost anecdotal" and did not name specific studies. He connected it to reward hacking, where an AI finds ways to score well that defeat the purpose of the task. If agents are doing that "and escalations on the table," he said, "that seems like something you want to be very, very careful with also."

Labenz's reply was brief: "Chilling, honestly."

Share this article

Go to the original

Sources & further reading

  1. 01
  2. 02
  3. 03

Connected ideas and articles

From the conversation

Podcast episodes

The Cognitive Revolution

Freedom of Action: America's ASI Strategy, w/ RAND's Joel Predd, Ben Boudreaux & Matt Chessen

Episode published This article draws on 1:34:12–1:41:00 (approximate times)

Article history

Updates to this article

Tags