November 21, 2026
Heard in AI

Anthropic says it curbs distillation for safety, not revenue

Anthropic's Angela Jiang says the company made its API harder to distill after a public paper showed hidden model reasoning could be extracted. She says safety, not lost revenue, is the main reason it fights distillation.

A briefing reports one development at a point in time. We may correct or clarify it later; a new development gets a new briefing. How our formats work

Based on Latent Space, episode published October 27, 2026

Some of the work behind Claude Opus 5.5 was, in Angela Jiang's words, "stuff that we haven't talked about in the blog post." "We did do a lot of work on trying to make our API harder to distill from," Jiang said. That work covered how a model's reasoning can be passed back into a conversation and replayed across different models.

Jiang and Katelyn Lesse, who work on Anthropic's Claude Platform, described the changes on Latent Space. The conversation was recorded in the studio the day after Opus 5.5 launched. Anthropic announced the model on September 22, 2026, and the episode was published on October 27.

What was being taken

Reasoning models work through a problem in text before they give an answer. Providers generally keep that reasoning from view. A developer instead receives it as a protected or encrypted block and sends it back on the next turn, so the model can pick up its own train of thought. Distillation means training another model on a stronger model's outputs, and reasoning is especially valuable material for that. On the show, the discussion noted that Anthropic has long known that extracting a model's chain of thought is one of the most powerful ways to distill it.

The gap became public in "Stealing Reasoning Traces from Proprietary LLM APIs," a paper by Panfilov and colleagues dated August 10, 2026. Testing API versions available in early July, the researchers found that a weaker compatible model could be induced to transcribe encrypted reasoning that a stronger model had generated. That opened the door to extracting reasoning, leaking private data and hiding prompt injections, where text slipped into a conversation tries to redirect the model.

The privacy findings came from 315,320 reasoning blocks that the authors decoded from 6,708 public conversation records. Automated classification flagged 1,028 blocks, about 0.3%, as leaking private information. In 328 sessions, or 4.9%, the authors found at least one real sensitive item, and genuine user sessions yielded 62 distinct API keys and 33 passwords. The authors could not check their transcriptions word for word, because the original plaintext was not available to them, and their search of public traces was not exhaustive. After the researchers disclosed the problem, provider fixes made the demonstrated attacks impossible to reproduce by August.

Asked whether this was the paper she meant, Jiang said, "Yes, that paper." She praised how the researchers handled it: "It's super responsible disclosure, and we're super thankful for them coming to us." When the conversation turned to whether the paper caused a freak out inside Anthropic, the answer given was that it probably did not, because the company already knew about the gap. Anthropic had been exploring for some time whether to require developers to keep reasoning blocks intact across turns.

Why Anthropic calls it a safety problem

The discussion traced the concern to Anthropic's mission. The company runs safeguard classifiers, automated filters that check what its models produce. A distilled copy would run without those filters, which, as it was put on the show, could lead to bad outcomes in the world.

Jiang added that she sees many people on X describe distillation as mainly a commercial problem, because a copied model can be run cheaply. "While true, it's actually not something we really care about very much," she said. Anthropic thinks about it mostly from "a safety point of view," she said, and argued that the lost revenue is small anyway: "Realistically, if you distill from us, will you actually pay us?"

Threading the needle

The hard part, as described in the conversation, was the trade-off between protection and changes that make the API more painful to use. Anthropic sorts that pain into two tiers. In the first, developers must change their integration, which is annoying, but they can still do everything they could do before. In the second, they lose something they used to be able to do. Anthropic tries very hard to avoid the second tier.

Jiang called the balancing act "threading the needle" between legitimate reasons to replay reasoning, such as a developer's own compaction strategy, and the same move made by a bad actor. She mentioned changes such as "prefix lock thinking" and limits on replaying thinking tokens across different models.

Anthropic's preserved-thinking documentation explains the mechanism. A reasoning block can be bound to everything that came before it in the conversation: the system prompt, the tools and the earlier messages. If that earlier part changes, an incompatible block can be rejected or removed, and further checks tie blocks to compatible models and accounts. A conversation that is only ever appended to keeps working. Switching models can make earlier reasoning temporarily unavailable without deleting the history. This targets the paper's method directly, since the attack depended on handing one model's encrypted reasoning to another. The paper's authors had proposed similar defenses, including cryptographically binding reasoning to users and conversation context, and noted the cost to model switching and history editing.

According to the discussion, Anthropic had made a few incremental changes over its last couple of models to enforce preserved reasoning across turns. The newer work deals with people who spread extraction across multiple accounts.

Moving customers without breaking them

Anthropic did not impose the new rules overnight. As the episode described it, the company contacted maintainers of open-source harnesses, the software that wraps a model in a loop of tool calls so it can carry out tasks, and told them which upgrades they would need. It also worked with large customers building their own harnesses on the API.

Compaction caused the clearest conflict. Long-running agents often compact a conversation by summarizing or trimming older turns so it fits in the model's context window. Some customers' compaction strategies would have broken under the requirement to preserve reasoning blocks, the discussion noted. That fits the documentation, since rewriting earlier turns changes the context a block is tied to. Anthropic's answer was a compaction feature built into the API to replace what customers had been doing manually, one that works with preserved thinking.

When preserved thinking first rolled out with the Fable 5.1 model, Anthropic grandfathered some existing customers and gave them a grace period to change their integrations. It tracked those customers through a "trust grant," a record of what each account has been allowed to do. Anthropic's October 12 post on distillation defenses says preserved thinking initially applied to accounts created on or after August 31, 2026.

What changed

Jiang said distillation has become "significantly harder" by the measures Anthropic can see, and that the negative effects it worries about appear to have fallen. She noted that the paper's results were published very publicly and trended on Twitter, "so many people could use it," and said closing the gap "was great."

The October 12 post, published after the recording, describes three layers: restricting reasoning replay, a two-stage classifier that detects extraction requests, and limits or bans for accounts that keep trying. Anthropic estimates that extracted reasoning tokens on its own platform fell ten- to a hundredfold after it introduced the classifier, and it reports false positives below 0.1%. The company also acknowledges the costs: developers had to change their integrations, some blocked requests are still charged, and it is not certain the defenses will hold against sophisticated attackers.

Share this article

Go to the original

Sources & further reading

  1. 01
  2. 02
  3. 03
  4. 04

Connected ideas and articles

Background

Stories behind this article

  • Anthropic releases Claude Opus 5.5 The article presents the distillation defenses as unannounced work behind the Opus 5.5 launch and dates the interview to the day after that launch. The release frames the story, but the article is not about the model's pricing, capabilities or adoption.

From the conversation

Podcast episodes

Latent Space

How Anthropic ships the #1 LLM in the World safely: Separating the Brain from the Hands

Episode published This article draws on 7:30–7:35, 7:43–14:25 and 28:40–30:07 (approximate times)

Article history

Updates to this article

Tags

Why Anthropic keeps its agents' credentials out of the sandbox

Anthropic split its managed agents' "brain" from their "hands" for speed and reliability. In a Latent Space episode with Anthropic's Katelyn Lesse and Angela Jiang, security was described as the bigger gain. The discussion also argued that many reported sandbox breakouts happen in training-style setups that lack production safeguards.

6 min read

How Tenex keeps its coding agents' context in the repository

Tenex, which builds AI systems for large companies, says it stores plans, agent decision logs and project context beside its code so agents can read them. In the episode, Tenex said Opus 5.5 is the main coding model for most of its team, and that pairing it with Codex as a reviewer still gives the best results.

7 min read