November 21, 2026
Heard in AI

Modal's CEO says serving-speed tricks give a less durable edge

Modal CEO Erik Bernhardsson says tricks for serving AI models faster spread to rivals and give a less durable edge. He argues the lasting advantage is Modal's own infrastructure plus a broad platform, so it open-sources most serving optimizations.

A briefing reports one development at a point in time. We may correct or clarify it later; a new development gets a new briefing. How our formats work

Based on Unsupervised Learning, episode published October 28, 2026

Erik Bernhardsson, co-founder and CEO of the cloud computing company Modal, expects much of the market for serving AI models to be "quite rough." Competitors can make a model answer faster and more cheaply, he said, but that kind of improvement "is a little bit of a race to the bottom." He sees a more durable edge elsewhere: in infrastructure Modal built itself, and in a platform of products that customers buy together.

Bernhardsson made the case on Unsupervised Learning, a podcast hosted by Jacob Effron and Patrick Chase, both AI investors at Redpoint Ventures. The episode was published on October 28, 2026, and was his second appearance on the show. According to the episode description, Modal is now past a $1 billion run rate. Redpoint is an investor in Modal, and Chase led that investment.

How Modal got into serving language models

Inference is the work of running a trained AI model to produce answers, as opposed to training it. Large language models produce text in tokens, small pieces of words, and many providers charge for inference by the token. Modal rents computing power, including graphics processors (GPUs), to developers who run their code in its cloud.

Bernhardsson said Modal stayed "a little bit on the sidelines" of language-model inference until what he called the DeepSeek moment in early 2025. He described DeepSeek's model as the first Chinese open-source model "that was like actually really good," and his conclusion was that Modal had "got to get in here." The release he meant is most likely R1: DeepSeek's R1 documentation describes a reasoning model with 671 billion parameters, of which 37 billion are active for each token, released alongside smaller distilled versions.

Modal then built a team focused on inference optimization. Bernhardsson said the company had long been good at hosting inference, but teams usually had to deploy models themselves or work with Modal on custom setups. "In fact, we didn't even have an inference product until very recently," he said. Language-model inference now accounts for about a third of Modal's revenue, according to Bernhardsson.

A few months before the episode, Modal started moving "up in the stack." Customers can now deploy single-tenant models, meaning a copy of a model running only for them, with one click or command line instruction. State-of-the-art Chinese models and a number of US ones are available this way. Shared endpoints are also available, letting customers call ready-hosted models with fully token-based pricing. Bernhardsson did not name the products. The dedicated offering most likely corresponds to Auto Endpoints, which Modal introduced on June 23, 2026, with serving code customers can inspect and engine-level metrics. In October, Modal added Endpoint Candidates, which tests alternative deployments against copies of real production requests before one is promoted. Bernhardsson called these products "the first out of like many more products to come."

Why serving models is still trial and error

When Chase asked what remains unsolved, Bernhardsson said inference is still "pretty early" and "one of those like very empirical sciences": people try things and see what works.

His first example was disaggregated prefill, which he called "a very non-obvious idea." A model answers in two phases. In prefill, it reads the whole prompt. In decode, it generates the reply token by token. Disaggregation runs the two phases on different machines and copies the intermediate data between them over a fast network link. The vLLM documentation describes its version of the feature as experimental and aimed at controlling latency rather than increasing throughput.

The data being copied is the KV cache, the model's stored working memory for text it has already processed, which spares it from recomputing that text. Bernhardsson said there is "a lot of complexity" around it. The open-source engines Modal works with each have their own approach. vLLM's PagedAttention stores the cache in small blocks so less GPU memory is wasted. SGLang's RadixAttention keeps caches around so repeated prompts and conversation histories can reuse earlier work. He also mentioned more efficient kernels, the low-level programs that run on the GPU, and quantization, which stores a model's numbers at lower precision.

Modal has 20 to 30 people focused on language-model performance, Bernhardsson said. They have to read papers and test the steady flow of new software to see what works. He expects patterns to emerge over time, but the models keep changing as well. Mixture-of-experts designs, which activate only part of a model for each token, are one source of added complexity. If someone replaced the transformer, the architecture behind today's language models, how KV caching works could change "overnight." "Maybe they will. Maybe they won't. I don't know," he said.

Why Modal gives most of its tricks away

Effron noted that many inference improvements of the past few years have been published as open source. He asked how Bernhardsson decides what his team should build itself. "We actually love open source because I think a lot of our alpha is in the proprietary side, in the infrastructure layer," Bernhardsson replied, borrowing finance slang for an edge. As an example he named autoscaling, which adds and removes machines as demand changes.

Modal works closely with the teams behind vLLM and SGLang, two widely used open-source inference engines, and contributes to both projects. Bernhardsson said that sets Modal apart from other inference providers. A few things stay internal, "but almost everything we push to open source." The engines were not state of the art a few years ago, he said, "but now they are." Modal's Auto Endpoints announcement likewise lists upstream contributions to SGLang as part of how it serves models.

He had described the infrastructure he considers hard to copy in an earlier answer: "a massive source of alpha in this market is just like not being afraid to do hard things." To load container images quickly (a container packages a program with everything it needs to run), Modal built its own file system, and later its own container runtime and its own scheduler. It does not use Kubernetes or Docker, the standard tools for that job, and relies on little off-the-shelf software beyond Google's gVisor for isolation and, recently, KVM. The payoff, he said, is very fast cold starts, the delay before a newly launched container can do work, which matters for applications that scale up and down quickly. Modal added CPU and GPU snapshots that load data straight into the GPU and start "even these like large models in a few seconds." Over the past six months it also built a system that copies data between machines using RDMA, a technique for moving data directly between computers' memory. Modal's own engineering write-up from May gives measured figures. For a small model, mean startup time with vLLM fell from about 96 seconds to about 14, including time spent waiting for a machine. At that point, GPU snapshots were limited to setups with a single GPU. "It was a longer route," Bernhardsson said, "but now I think we have some incredibly unique technology that's very hard to replicate."

Knife fights versus a platform

Effron asked what will separate the winners in inference. Bernhardsson said plain token serving is already being commoditized, with OpenRouter, a service that routes requests among many model providers, "basically commoditizing the underlying providers." Providers will still compete on efficiency, and he said Modal will still have to do "the knife fighting" of using the latest kernels and more efficient KV caching. There is so much money in it that this will continue for a long time. But "these tricks will sort of spread and people figure out how to copy them," he said, so the advantage they give is less durable.

OpenRouter does not rank providers on price alone. On the day the episode was published, it described a redesign of its routing that scores providers on response speed, errors and usable tool calls as well as effective price.

The more durable advantage, Bernhardsson argued, is offering "the entire platform of a lot of different things": sandboxes for running code safely, reinforcement learning, ordinary fine-tuning, other kinds of models and batch processing. "Customers, they don't want to buy like single trick ponies," he said. "They want to buy like a multi-product set of tools."

Chase noted that Modal's biggest customers use several products, and Bernhardsson gave examples. Cognition uses Modal at large scale for inference and for sandboxes that run reinforcement-learning rollouts. Ramp runs background agents on it, including its internal coding agent. Suno and Runway generate audio and video. Decagon handles voice and text customer support, and DoorDash runs internal applications, mostly built on language models. He also mentioned biotech companies doing drug-discovery work such as molecular simulations, and robotics companies including 1X that train robot models and batch-process video. Modal's Clusters announcement describes 1X pre-training a world model for its NEO robot on Modal.

Where the platform stops

Reinforcement learning as a service tests the platform's limits. In this approach, specialists help a company post-train a model on its own tasks after the model's main training. Effron raised the prediction that every company will have its own model. Bernhardsson split the question. Weighted by revenue, he expects most models to be post-trained. Counted by number of companies, he expects most will never need to post-train their own. The large ones will, he said, pointing to Cognition training its own code models, and those companies also serve a lot of tokens.

Parts of that market are attractive, he said, because better models can be served at lower cost and higher throughput. "Some of it looks like it's kind of like just a service business." He expects the tension over the next few years to be whether post-training can be turned into a product, and said Modal thinks a lot about building one "and not turn it into just like a service business."

Modal's recent material shows both versions. A May case study describes how Applied Compute, a customer, embeds its researchers with client companies to design rewards and training environments, hands-on work closer to the service model. On October 22, 2026, Modal introduced Spindle, an open-source training server, still in beta, that runs in a customer's own Modal workspace. Users write their own losses, rewards and training loop while Spindle handles distributed execution and bills for GPU time. Spindle fits the product version Bernhardsson described, though he did not name it.

Asked which kinds of AI infrastructure could remain large standalone businesses, Bernhardsson said, "we're not going to go into model training, I think." He expects many custom models from other companies and gave TypeSafe's Jev as an example. Modal's case study describes Jev as a model that makes structured decisions instead of generating free-form text, and says it served a trillion tokens on Modal within three days of launch. In his experience there are "always like more boxes on the market map than you expect," though he also thinks every company that succeeds eventually has to sell several products.

He rejected the idea that infrastructure companies must expand into everything. People sometimes ask why Anthropic or OpenAI don't "launch their own law firm," he said, when those companies reached scale by being very good at serving models. Modal, in his telling, will pick "a cluster of like very adjacent things." "We're not going to do everything," he said, "but we're going to do a very large cluster of adjacent things and do it exceptionally well."

Share this article

Go to the original

Sources & further reading

  1. 01
  2. 02
  3. 03
  4. 04
  5. 05
  6. 06
  7. 07
  8. 08
  9. 09
  10. 10
  11. 11
  12. 12

Connected ideas and articles

From the conversation

Podcast episodes

Unsupervised Learning

Ep 95: Modal CEO on the Future of Inference, the Chaos of the GPU Market & Origins of the Sandbox Bet

Episode published This article draws on 8:19–10:01, 10:23–14:44, 15:02–16:06, 21:21–24:55 and 48:15–51:40 (approximate times)

Article history

Updates to this article

Tags