Nathan Labenz, host of The Cognitive Revolution podcast, got to try Google's Gemini 4 before its wide release. He found it competitive. He is still not putting it into the system of AI agents he runs on his own computer.
"Why won't I swap the model immediately?" Labenz asked on the episode published October 17, 2026. "It's because there's not enough of a step change to make me take any additional risk, and I do perceive there to be some." His priorities have shifted, he said: "I'm not relentlessly performance maxing quite as much as I used to be. I'm now a little bit more trust and reliability maxing."
Letting one model test another
Labenz did the first round of testing by handing it to an agent. An AI agent is a model that carries out tasks on its own, using tools and files rather than only answering questions. He asked Anthropic's Claude to go through his recent work threads, pick a range of projects and tasks, and send the new Gemini model the same context the original model had received. Claude then produced side-by-side comparisons and ran its own evaluations, or "evals", the structured tests people use to score model output. Labenz called it a sign of "how high leverage the ability to instruct agents to do this is."
He came away with "the general sense that it is like a competitive model," in the same class as the latest models from OpenAI and Anthropic. Those rivals still won the head-to-head comparisons "a bit more," he said, but Gemini 4 "puts Google back in the game."
He listed the weaknesses of his own test. Claude judged the comparisons, "so obviously some bias there," since one company's model was grading a competitor. His own review of the essays was limited. And the tasks came from a setup whose prompts he had probably tuned over time for the models he already uses, so "it's not exactly an apples to apples comparison."
The question at a Chicago workshop
The testing connected to a question from a little workshop he ran in Chicago, where he showed his setup. That setup had been "very Claude centric," Labenz said, and he had recently promoted OpenAI's models to parity with Claude. Someone asked whether he would bring anyone else into the mix.
He had to think about why he hadn't. His answer, he said, was a mix of security, "as measured by things like Adam Gleave's" measures of how easy models are to jailbreak, and "familiarity and predictability" from his own experience. "It wasn't really benchmarks," he said.
A jailbreak is a prompt or trick that gets a model to ignore its safety rules. For Labenz, the worry is concrete. "I've got these autonomous agents," he said. "They're talking to my main agent on my computer that has all my stuff." He said he felt good that Gleave's team found the OpenAI and Anthropic models "quite hard to jailbreak." Gemini was "next on that list, but definitely a step down." As for xAI's Grok, he said it "may be similarly powerful, but why would I take that risk in a high context environment" like his personal laptop.
"To a remarkable degree, the models are competing for my use on trust and security already at this point," Labenz said.
What the jailbreak measurements show
The published work that likely matches Labenz's description is the AI Security Leaderboard report from FAR.AI, released on August 4, 2026. Its ordering lines up with his. The researchers combined 67 attack techniques into 1,500 candidate variants and tested them against 360 attacker goals in five severe-misuse domains. They then searched for "universal" jailbreaks, reusable attacks that succeeded on at least 75 percent of the goals in a domain for a given model.
Random search turned up 63 such jailbreaks against Grok 4.5 and 18 against Gemini 3.1 Pro. When experts guided the attack combinations, the counts rose to 385 and 231. The tested Claude and GPT models, Claude Fable 5 and GPT-5.6 Sol, resisted the attacks.
The figures have limits. They count reusable attacks the researchers found, not the share of ordinary requests that go wrong. The testing took place in June and July, so it covered Gemini 3.1 Pro rather than Gemini 4. It also focused on fixed attacks aimed at getting help with serious misuse, which is related to, but not the same as, the hijacking risk Labenz described for his agents.
That second risk is prompt injection: text planted on a website or in a document that tries to give an AI agent new instructions when it reads the page. Google has published its own number here. In its October 13 safety announcement for Gemini 4 Argon, the company reports a 0.7 percent attack success rate across 15 adaptive attempts on Gray Swan's indirect prompt-injection benchmark, and credits adversarial training and real-time security checks. That result comes from Google itself, and Labenz did not mention it.
Trust over small gains
Labenz acknowledged that his caution sits awkwardly with the advice he gave in the workshop. "My whole point of the talk was like, you need to own the context and you got to be ready to swap the model," he said. Yet he is "reluctant to swap them out."
The reason, he said, is how much his agents can now reach, combined with how good the current models already are. "I am giving so much access that, you know, you could be slightly better on the slide design," Labenz said, "but what I really care about is like that you're going to stay on the rails." When an agent does research and lands on a website carrying a prompt injection attack, he wants a model that won't fall for it. That, he said, "actually matters a lot more to me now than like often like imperceptible performance gains."