A caller on the audience-question episode of Moonshots with Peter Diamandis opened with a shout-out to his wife, Melanie, who had pushed him over and over to start following the panel, and then asked something longer-term. Every technology so far, he said, follows an S-curve: slow at first, then steep, then flattening. What did the panel see on the horizon as a fundamentally new approach after large language models?
The question was handed by name to Alexander Wissner-Gross, a computer scientist and founder of Reified who goes by Alex on the show. He started by splitting it. "First, I would distinguish between LLMs and Transformers," he said. The two words often get used as if they named one thing, and in his view the answer depends entirely on which one you mean.
The job, the machine, and the chip
Three separate layers sit inside the phrase "large language model," and the panel's answer works through all three.
The first is the training objective: the job the system is given while it learns. For an LLM, Alex said, that job is the language modeling task, "which is essentially predicting next words." The model sees text with something missing and has to guess what belongs there. Nobody has to label the data, because the answer is already in the text — this is what makes the task self-supervised.
The second layer is the architecture: the arrangement of mathematical components that carries out the job. The transformer is one such arrangement, introduced in the 2017 paper Attention Is All You Need.
The third is the hardware the whole thing runs on — today, racks of electronic chips.
Alex's argument is that the first layer may never need replacing, while the second is already being replaced. The self-supervised task of "next word, next token, next bit prediction," he said, "has effectively infinite longevity to it. You can scale that task, the self-supervised task, from here to solving everything." He was not sure anything more was needed for AI than predicting the next bit, or, more generally, the missing bit. You could generalize beyond it, he said, but he doubted you had to.
Transformers he treated very differently. They are, as he has said on the show before, being "ship of Theseus style replaced bit by bit" — planks swapped one at a time until little of the original vessel is left. Look at the frontier models today, whether Chinese open-weight or American closed-weight releases, and in terms of their raw pieces they are mostly different from what Attention Is All You Need first came up with. On that layer, he said, "we're already up some sort of sigmoid curve" — the S-curve the caller had asked about.
Which planks are staying
Another panelist then went through the transformer part by part. The transcript does not say by name who this was, though the speaker refers back to what "Alex said", so it was not Alex. This panelist mostly expected the transformer's pieces to persist.
The MLP — the multilayer perceptron, also called the feed-forward part, a plain stack of numerical layers that transforms each position in the sequence — is, the panelist said, "identical to what existed 20 or 30 years ago." It is a pretty good bet, the panelist said, that it stays.
Next came the residual connection, described as "the residual superhighway": rather than each layer building a fresh representation from scratch, the layer nudges a running vector of numbers that flows straight through the network. "You're just modifying a vector instead of recreating it every layer." That, too, the panelist expects to survive.
The genuinely new ingredient is attention, the mechanism that lets each position in the sequence draw on information from other positions. Here the panelist offered an image. "The more you dig in to attention, it's more like a notebook keeping notes than it is like a brain function." As a concept it may get radically restructured, the panelist said, but something will remain in that role: a "knowledge accumulation notebook" sitting behind or beside the MLP at each layer.
The 2017 paper set out a narrower result than the forecasts now built on top of it. Vaswani and colleagues combined multi-head attention, position-wise feed-forward networks, residual connections and positional information, and tested the result on machine translation and English constituency parsing. Training used roughly 4.5 million English–German sentence pairs and 36 million English–French pairs; the larger model trained for three and a half days on eight P100 GPUs and reached 28.4 BLEU on the English–German newstest2014 set, a measure of overlap with reference translations rather than a percentage of correct answers. The authors found learned and sinusoidal ways of encoding position performed almost identically, and that piling on too many attention heads hurt quality. The parsing work also went well, though the model did not beat every comparator. Their conclusion was that attention-based translation trained faster, and they proposed extending the design beyond text.
The layer another panelist expects to move fastest
Having kept the architecture mostly intact, the same panelist put the change underneath it. The panelist said they were "100% sure" the compute substrate will move to photonics — computing carried out with light rather than electrons. After that, the panelist said, physics itself gets reinvented by the current AI, and the substrate may move quicker still to something beyond photonics. The structures above would be the same; they would simply be "running millions of times faster." At that point, echoing Alex, the panelist expected the systems to think of everything else after that and explain it to us.
His timeline for all of this: "in the next really 18 months to two years is my guess." The exchange closed on "It's just wicked exciting" and a remark that the panel could have talked about the topic for hours.
Why a prediction machine is not necessarily a parrot
Earlier in the same session, a participant had raised the objection that hangs over Alex's first claim: if these things are prediction machines, where does originality come from? The pitfall they fall into, the participant said, is seeing the LLMs as prediction machines, when what interests them is the emergent "ahas" off the beaten path that spark creativity. A panelist replied that an episode recorded that day, due out the next, argued that AIs are probably being inventive.
One of the hosts, not identified by name in the transcript, called the framing a misconception. Reasoning models, this host said, should not be characterized as prediction machines or stochastic parrots — the label for a system that recombines its training text without understanding it. They have been so thoroughly reinforcement fine-tuned on reasoning challenges that calling them prediction machines is, in that host's words, "tantamount to saying basically that they're incapable of novelty or insight, which is most definitively not true at this point."
Read next to Alex's answer to the S-curve question, that reply deals with the obvious worry about his first claim. In the panel's account, keeping next-bit prediction as the training objective does not cap what a system can do. What gets built around that objective keeps changing underneath it: the architecture, the reinforcement training and, if the second panelist's forecast holds, the chips.