Grigore Rosu thinks the way to deploy an AI system nobody can fully check is to surround it with something that can be checked. His proposal is to give up on proving that an entire AI agent behaves correctly. Instead, write down the handful of things it must never do, and build a monitor that is mathematically proven to catch them.
Rosu, who runs the company Runtime Verification, laid out the idea on The Cognitive Revolution in an episode published October 17, 2026. Host Nathan Labenz stayed uneasy afterward. Once the guests had left, he wondered whether formal verification might end up playing the role RLHF did: a technique that came out of safety work and mostly made AI more useful.
A provable "police system"
Formal verification means using mathematics to prove that software meets a precise written specification, instead of testing it on examples and hoping the tests covered everything. Doing that for a modern AI model is out of reach. Rosu said it is impossible to prove a model's weights correct, meaning the vast sets of numbers it learns during training.
His alternative comes from runtime verification, which checks a system while it runs. Rosu said the field was invented "back in 2001 or so" when he was a research scientist at NASA. It works with safety specifications rather than a full description of the whole system: "This should not happen. And if it happens, then this should be the recovery." The aim is to steer the system to stay inside what he called a "stability envelope or safety envelope." With formal methods, he said, one can now generate monitors that are provably correct for those specifications.
The agent itself may depend on many pieces that are flawed or unproven. "But I can prove my monitor," Rosu said. He described the result as "a police system around to make sure that you don't do bad things," and said his team believes this is the future for safe AI.
A NASA technical memorandum on a tool called Copilot 3, published in 2020, shows what such monitors look like in practice, though in software for safety-critical embedded systems rather than AI. It describes version 3 of the Copilot language for writing monitor specifications. The framework translates those specifications into C code with static memory requirements, which can be compiled to run on embedded hardware.
Flexibility versus "this should never happen"
Labenz pressed on a tension. What he wants from an agent, he said, is much like what he wants from a human employee: the ability to "get over the hump and solve problems that were not anticipated." That makes it hard for him to declare in advance that something should never happen "in a way that I can trust," because the agent's ability to handle the unexpected is part of the point. He asked whether this was an inherent trade-off or whether there was "some way to square that circle."
Everett Hildenbrandt, CEO of Runtime Verification, who joined Rosu on the episode, answered with bridges. "You're not going to ask an agent like, hey, go build a bridge. We know how to build bridges," he said; there are already automated, deterministic processes for that. But someone might ask an agent to get them to the moon, and the agent might decide it first needs a bridge to an island where rockets will launch. The creativity, whether an agent's or a human's, should sit on top of "foolproof things that behave how the agent expects." His bottom line: "we need to give these things tools that act reliably, basically."
The RLHF comparison
That answer stuck with Labenz. If he understood it correctly, he said, the link between these methods and agent control was really about giving agents reliable tools. That made him wonder whether the whole formal verification wave might eventually be viewed the way RLHF is.
RLHF, or reinforcement learning from human feedback, trains a model on people's judgments of which outputs are better. Labenz credited the idea to Paul Christiano. He said it had come up that weekend on his timeline: David Sacks and others had said alignment researchers never came up with anything useful, and others replied that they came up with RLHF. Labenz agreed it turned out to be extremely useful. But he said safety-minded researchers often lament it, because it made models tremendously more useful and "didn't really solve the alignment problem." Alignment here means getting AI systems to pursue the goals people actually intend. Reward hacking, where a model maximizes its score in ways that defeat the real goal, still shows up "all over the place," he said. His question was whether RLHF advanced the safety profile of systems differentially, or just accelerated capability and the race to deploy.
The conversation then turned to what reward hacking really is. One framing explored in the discussion was that it is a misspecification problem: the model went after the reward exactly as it was specified, and the gap lies between what humans intended and what they wrote down. In that framing, the software did what it was supposed to do.
What reliability might not buy
Labenz then imagined verification succeeding. Tools become very reliable, sandboxes can no longer be broken out of, and "you can't use these like, you know, package managers as secret message boards anymore because all these loopholes have been closed." He called the upside obvious. The downside, which he said people should not "lull ourselves to sleep with," is that this makes it safer to deploy AIs "more or less as they are today," without having solved alignment, because they are blocked from hacking in various ways.
In practice, he expected more reliable systems, easier deployments and better results. That would make it easier to put the AI "into the space between us and all the actual levers of control over the world." The AIs might then have extremely reliable tools while their intentions, long-term goals and inner structure remain an unsolved question.
Labenz said he had raised this with friends who work on these methods, and that the conversation put his "intuitive unease" into starker relief. He described a possible spot where people are "still just deploying untrusted AIs with like much better guardrails and much more reliable tools." If people then hand over the day-to-day operation of the world, he said, all that reliability buys little should the AIs turn out to be misaligned. It might even lead there faster, and could make them "more effective at doing whatever sort of, you know, surprising thing they may do when the time comes."