November 21, 2026
Heard in AI

Modal's CEO: pick the cheapest GPU that fits the model

Erik Bernhardsson, Modal's CEO, said AI serving jobs are usually limited by memory, so the cheapest GPU that holds the model tends to win. He expects top models to keep growing into chips' memory, though he said it "might taper off."

A briefing reports one development at a point in time. We may correct or clarify it later; a new development gets a new briefing. How our formats work

Based on Unsupervised Learning, episode published October 28, 2026

Erik Bernhardsson, co-founder and chief executive of the AI infrastructure company Modal, has a short rule for choosing the hardware that runs an AI model: "you basically pick the cheapest GPU that fits the model." He said the rule works "in almost all cases."

Bernhardsson explained it on Unsupervised Learning, in an episode published October 28, 2026. The hosts, Jacob Effron and Patrick Chase, are AI investors at Redpoint Ventures, and Chase led Redpoint's investment in Modal. Modal runs customers' AI workloads on graphics processors (GPUs), the chips used to train and run AI models. Much of that work is inference: producing answers from a model that has already been trained. Deciding which chips to hold, and which jobs to put on which chips, is a large part of running the business.

NVIDIA still leads

Chase asked how much the "long tail" of non-NVIDIA chips matters to Modal's supply. "NVIDIA is still like the king here," Bernhardsson said. In his view it has the best chips, and there is "way more demand" for them than for anything else. Modal has run experiments with alternatives, and he said AMD's chips, Amazon's Trainium and Google's TPUs are "starting to be like pretty good." There are also plenty of older chips in the market, he added.

Not every model needs the newest chip

The challenge, he said, is that developers of the most capable language models keep pushing how many parameters, the numerical values a model learns in training, they can fit onto hardware. The latest of these models need a full node, meaning one server's worth of the newest chips. His example was eight of NVIDIA's B300s, "because that's how many you can fit on a node." In the future, he said, such models may need several nodes. NVIDIA's specification table lists the B300 with 288GB of memory, so eight of them hold about 2.3 terabytes.

Much of the work Modal sees falls outside that category, Bernhardsson said: older or smaller language models tuned for performance, voice models for audio transcription or speech synthesis, and pharmaceutical and biology models. Those often run more cost-efficiently on older hardware, such as NVIDIA's Hopper generation, the still older Ampere generation, or cards like the L40S and RTX Pro 6000.

Modal is ramping up "very, very aggressively" on NVIDIA's newest Blackwell generation, he said, but it also sees plenty of demand for older chips. "People forget about the hoppers," he said. Hopper H100s are still the most common GPU at Modal, and the company runs "some very like large production use cases" on them.

The memory rule

Effron asked whether the industry generally matches models to the most price-efficient chip. Bernhardsson said there is "a little bit of room to improve," but called it less of a problem than it might seem, because jobs are often limited by memory. A model has to fit in the chip's memory; once you know how much it needs, the smallest and cheapest GPU that holds it is usually the right pick. If a model takes 80 gigabytes, he said, it could run on a Hopper or on an older A100 with 80GB. In that case the choice depends a little on FLOPs, the chip's raw arithmetic speed.

NVIDIA's table shows why such ties come up: the A100 comes in 40GB and 80GB versions, and the H100 has 80GB configurations. Modal's own GPU guide adds two refinements to the rule. Memory has to hold a model's activations, the intermediate results produced while it computes, as well as its stored weights. And when a language model answers only a few requests at a time, the speed at which data moves through memory can limit performance more than arithmetic does, so a newer chip does not always deliver a proportionate gain.

Switching chips automatically

Bernhardsson said Modal wants to take the choice off customers' hands. Over the next six months, he said, the company is "quite interested" in letting people deploy a model while Modal automatically switches GPU types underneath, in a way that stays consistent with "your dollar per request or dollar per token." The more chip types a job can run on, he said, the better Modal can pool capacity, move work onto chips that are sitting underused and run its fleet at higher utilization. That might also let it give customers discounts.

For now, the GPU guide lets developers name a GPU type, request several GPUs per container and give an ordered list of fallbacks for code that works on more than one configuration. One example in the guide accepts either an H100 or two 40GB A100s.

Models will grow to fill memory

Effron then raised fleet planning: Modal has to guess what workloads it will serve a year or two out. He described a debate over whether most tokens will come from very large models, or whether models will have saturated most tasks and get smaller and smaller.

Bernhardsson said the trend over the last three or four years has been toward larger and larger models, and he expects it to continue. "If B300s had more memory, I'm sure, you know, a Kimi K3 would be bigger," he said, because developers train models to fit the chips available. Kimi K3 is a model from Moonshot, whose July 22, 2026 announcement describes it as having 2.8 trillion parameters. The link between K3's size and the B300's memory is Bernhardsson's own reading; Moonshot's announcement does not draw it.

His reasoning rests on how he thinks about language models. He described one as "kind of a crude, like type of world model" that tries to encode as much information as it can, which calls for something "almost like lookup tables of facts." That, he said, pushes memory requirements up. He allowed that "it might taper off, like who knows," but for now, he said, "people will make use of every possible memory." Give developers more of it, he said, and "they'll just train more parameters for sure."

Share this article

Go to the original

Sources & further reading

  1. 01
  2. 02
  3. 03

Connected ideas and articles

From the conversation

Podcast episodes

Article history

Updates to this article

Tags

Positron's CTO makes the case for a memory-first AI chip

Positron CTO Thomas Sohmers said AI agents are multiplying demand for memory. He claimed Positron's first-generation product sustained 93% of theoretical memory bandwidth, against an average of 30–40% for NVIDIA GPUs when decoding with transformer models. Its next chip, Asimov, is planned for 2027.

7 min read