November 21, 2026
Heard in AI

Anthropic says Claude-coded Rust port cut API CPU use roughly 40-50%

Anthropic's platform team says its Messages API now runs on a Rust port that Claude coded while engineers designed and verified it, cutting CPU use by a ballpark 40 to 50%. Its next step is keeping conversation state on the server.

A briefing reports one development at a point in time. We may correct or clarify it later; a new development gets a new briefing. How our formats work

Based on Latent Space, episode published October 27, 2026

Anthropic now serves its Messages API, the main way developers send requests to Claude models, from code written in Rust. When the project started, the team doing it did not have a single Rust developer. Claude wrote the code. Engineers designed the system and checked that the new version gave the same answers as the old one. The reason for the project was a shortage of ordinary computer processors, or CPUs, rather than the specialized AI chips that usually dominate talk about computing power.

The account comes from an episode of the Latent Space podcast, published October 27, 2026, in which Anthropic's Katelyn Lesse and Angela Jiang talked about the engineering behind the Claude Platform. The Rust version had been fully rolled out the week before the interview was recorded, the Anthropic side said.

From a Python monolith to a CPU wall

The Messages API gives developers direct access to the model. An application sends a list of messages and some settings, and gets back Claude's reply along with a count of the tokens, the chunks of text the model processes. According to the account in the episode, a little over a year before the interview this service was a Python monolith, one large program written in Python and running in a single Google Cloud region, and it worked fine. Since then the request volume has grown enormously, and the service has had to expand to several regions.

The conversation named scale and safety as the two things the team spends most of its time on. Most discussion of AI computing concerns GPUs and other accelerators, the chips that run the models. But in the discussion, CPU capacity was described as just as much of a problem today as the accelerator layer.

Anthropic needs CPUs for many jobs. They run its product services, the API and its apps. They also run sandboxes, sealed-off environments where code can execute without touching anything else, and continuous integration, the automatic builds and tests that run every time someone changes code. Much more code is being written now, the discussion noted, which means more of those test runs and more CPU to handle them. Training came up as a separate consumer.

Inside the company, CPU was described as a zero-sum resource. Anthropic negotiates deals with cloud providers, then divides what it gets among everything from research sandboxes to product sandboxes. Companies used to be able to scale up CPU on demand as long as the budget allowed. Now capacity has to be planned in advance, a change the discussion portrayed as a painful new reality for the whole industry, not only Anthropic.

The conversation set out three responses. The first is simply to get more CPU. The second is to run sandboxes more efficiently without weakening tenant isolation, the separation that keeps one customer's work away from another's. The third is to make the applications and APIs themselves cheaper to run, and the Python program was described as very CPU-inefficient.

A team without Rust developers

Rust is a programming language valued for speed and careful use of hardware. Choosing it was described in the episode as a bold decision at the time, because the team had no Rust developer.

The work was split between people and Claude: Claude wrote the code, while the engineers designed the system, which the discussion described as a distributed system, software spread across many machines that must work together. Claude also helped the engineers talk through ideas and learn what they needed to design it. Because the implementation itself was Claude writing code, the two things that mattered most, according to the account, were the plan and the verification.

For verification, the team built a complete copy of the API in Rust. It then sent shadow traffic through it, meaning duplicates of real requests that both systems processed, and compared the outputs of the Python and Rust versions. Jiang summed up the approach as "feature flags are all you need." Feature flags are switches that let engineers turn new code on for a chosen share of traffic.

In the past, the discussion noted, no team would have signed up for a migration like this. It would have been carved into pieces and taken many years. This time the team ported the whole system at once and then turned it on gradually. Anthropic hired a couple of people with more Rust experience along the way.

The conversation then turned to whether the team had resisted the urge to add new features during the rewrite. The answer was that it had, and the project was described as a port rather than a rewrite. A small dedicated team did the work while the rest of the API team kept building features. Static analysis tools, programs that inspect code without running it, flagged any new feature built without matching Rust work. The port team handled those cases but did not add features of its own or change how the system worked.

The results

The numbers were offered as a ballpark: CPU savings of roughly 40 to 50%, and a 10 to 20% improvement in time to first token, the wait before the first piece of a reply comes back. Jiang said the gains are larger on longer-running workloads. "It'll get faster," she said.

Next: keeping the conversation on the server

The port team's next goal is redesign work, some of it more visible to customers, according to the discussion. Today, a developer having a multi-turn conversation through the Messages API sends the whole history again with every new request. Anthropic is starting to roll out an option that would let customers who accept storage of their conversation state for a period of time skip that step. For long conversations, where the context builds up, the expectation expressed in the episode is that this brings significant latency and efficiency gains.

This differs from prompt caching, which the conversation also raised. Prompt caching lets the API reuse its earlier processing of the opening part of a prompt it has recently seen. By default a cached entry lasts five minutes and is refreshed each time it is reused. But the developer still sends the full conversation each time. Server-side state would remove that resending.

Jiang said the customers who benefit are few in number, but measured by token volume, those long workloads are "going to be the most."

The discussion turned to what kind of infrastructure suits a messages API that has to pick up where a conversation left off. Cloudflare's Durable Objects were suggested as the right shape. They pair a small unit of computing with durable storage, and every client can reach the same object by name, so in principle each conversation could live in its own object. Rivet, an open-source, self-hostable platform built on a similar idea, was mentioned as the open-source counterpart. In response, the episode's account was that Anthropic is experimenting with how to shape its compute, but because it runs on all three major clouds it cannot rely on one kind of machine or service everywhere beyond the most basic versions.

Who gets to sell Anthropic CPU

This part of the conversation closed with a question on behalf of listeners who have CPU to sell. According to the answer, the Anthropic group that secures its accelerator compute now spends much more time on CPU, working out where and how to buy it. There is a quality bar and a scale bar, and Anthropic will not integrate and run on certain machines if it cannot get enough of them.

Share this article

Go to the original

Sources & further reading

  1. 01
  2. 02
  3. 03
  4. 04

Connected ideas and articles

From the conversation

Podcast episodes

Latent Space

How Anthropic ships the #1 LLM in the World safely: Separating the Brain from the Hands

Episode published This article draws on 17:36–19:08 and 53:28–1:03:19 (approximate times)

Article history

Updates to this article

Tags

Modal's CEO estimates agents help write 95% of its merged code

Erik Bernhardsson, Modal's CEO, estimates that agents write about 95% of the code merged there "to some extent." He says that, contrary to his expectations, very senior engineers rather than junior ones have embraced agents most, that humans still need to supervise them closely, and that demand for engineers will keep rising.

5 min read