Through 2026, a long-running research argument moved back to the centre of the AI conversation: whether continuing to scale text-based transformer language models is on its own enough to reach the next level of capability, or whether it needs to be joined by different architectures that learn about the physical world. Researchers at Google DeepMind have been among the more prominent voices making the case that text prediction alone leaves something important out, and that so-called world models are a complementary path worth taking seriously. This piece is a survey of that debate and of the main alternative architectures, not a verdict on it. Both readings, that transformers keep scaling and that they need company, remain genuinely open.
What the transformer does, and where the argument starts
Almost every well-known large language model today is a transformer, the architecture introduced in 2017. It is trained to predict the next token in a sequence, and at scale that objective produces systems that write, summarise, translate, reason through many problems, and generate code. The scaling hypothesis, which has held up remarkably well for several years, says that more data, more parameters, and more compute keep making these systems better across a broad range of tasks.
The argument this post surveys does not dispute that transformers are useful or that they have improved. It makes a narrower claim: that a system trained purely to predict text, however large, may be missing an entire mode of understanding, the kind that comes from interacting with an environment and seeing what happens. In this view, some capabilities do not appear simply by adding parameters to a text predictor, and reaching them will require architectures that learn the structure of the world in a different way.
The DeepMind framing: text prediction and the missing leap
One of the clearest recent statements of this position comes from a 2026 position paper by Tom Zahavy, a researcher at Google DeepMind, titled "LLMs Can't Jump." The paper argues that a language model can plausibly carry out the deductive steps of science, deriving conclusions from stated premises, but that it is structurally poorly suited to the creative step of inventing new premises in the first place. Zahavy draws on the philosopher Charles Sanders Peirce's distinction between kinds of inference and on the history of Albert Einstein's path to general relativity to argue that the hardest part of scientific discovery is a form of abductive reasoning, a leap to a new framework that is not already present in the training text.
The paper's proposed direction is notable for this survey: it suggests that physically consistent, action-controllable world models, of the kind DeepMind has been building, could provide the sensory grounding that a pure text model lacks, letting a system run experiments inside a simulated environment rather than only rearranging symbols it has read. It is a position paper, an argument rather than an experimental result, and it is one voice in a large field. But it captures the shape of the broader case well: not that transformers are wrong, but that grounding in an environment is a capability text alone may not supply.
The honest framing is not transformers versus world models, but which problems each is built to solve, and how far a hybrid can carry both.
The longer-standing technical critique
The intuition that transformers have specific limits is older than the 2026 coverage, and parts of it are backed by formal and empirical work worth citing precisely.
On the theoretical side, a 2024 paper by Binghui Peng, Srini Narayanan, and Christos Papadimitriou, "On Limitations of the Transformer Architecture," uses tools from communication complexity to argue that a single transformer layer cannot reliably compose functions once the domains involved are large enough. Their running example is identifying a person's grandparent in a genealogy, a two-step composition, and they argue that a class of compositional tasks is unlikely to be solvable by transformers at large scale, under widely held conjectures in computational complexity.
On the empirical side, the 2023 paper "Faith and Fate: Limits of Transformers on Compositionality," by Nouha Dziri and colleagues, tested transformer models on multi-digit multiplication, logic grid puzzles, and dynamic-programming problems. It reported that the models tend to solve such tasks by reducing multi-step reasoning to what the authors call linearized subgraph matching, and that accuracy can decay quickly as task complexity grows, rather than reflecting a systematic, general procedure.
Two caveats keep this honest. These results describe specific families of tasks, not a ceiling on everything transformers can do, and the field has partly responded to them, for example with chain-of-thought prompting, tool use, and separate reasoning phases at inference time that let models work through steps they cannot do in a single pass. The critiques are real and specific; they are not, on their own, proof that scaling has stopped paying off.
The alternatives, explained plainly
If the concern is grounding and generalisation, what are the candidate architectures being proposed alongside or instead of pure text transformers? Three come up most often. They are not interchangeable, and none is a drop-in replacement for a chat model.
Four architecture families, side by side
The transformer LLM is one design among several. World models, vision-language-action models, and state-space models differ in what they take in, what they learn, and where they are strong. Most researchers treat these as complementary rather than as a single winner.
Simplified for comparison. Real systems mix these ideas: world models and VLA models reuse transformer components, and several 2026 designs interleave attention with state-space layers.
World models learn the dynamics of an environment: given the current state and an action, they predict what the environment looks like next. DeepMind's Genie line is a concrete example. Genie 3, released as a research preview in August 2025, is described by DeepMind as a general-purpose world model that generates interactive environments a user can navigate in real time at 24 frames per second and 720p resolution, remaining largely consistent for several minutes, and supporting "promptable world events" such as changing the weather or adding objects with a text instruction. DeepMind frames world models as "a key stepping stone on the path to AGI," partly because they can generate an effectively unlimited supply of environments in which to train agents.
Vision-language-action (VLA) models are the robotics-facing branch of the same idea. Rather than only producing text, they map camera images and a language instruction directly to robot actions. DeepMind's RT-2, introduced in 2023, popularised the approach, and the Gemini Robotics family, launched in March 2025 and extended through 2026 with on-device and enhanced reasoning variants, brings a large multimodal model's reasoning into physical control. Other groups, including Physical Intelligence with its pi-series models, pursue the same paradigm. VLA models are the most direct attempt to ground language in the physical world, because the training signal is whether an action actually works.
State-space models (SSMs) are a different kind of alternative, aimed less at physical grounding and more at efficiency. Mamba, introduced by Albert Gu and Tri Dao in December 2023, replaces attention with a selective recurrent state and scales linearly with sequence length rather than quadratically. Its authors reported roughly five times higher inference throughput than a comparable transformer and found that a Mamba model could match transformers about twice its size on language modelling. SSMs are frequently discussed not as a replacement for transformers but as a component, and several 2026 designs interleave attention layers with state-space layers in a hybrid.
The other side of the debate
It would be a distortion to present the world-models case as settled. Many researchers hold that transformer-based systems are still far from a ceiling, and the evidence for that view is also concrete.
The clearest counterpoint is test-time compute, the reasoning-model approach in which a model is allowed to think for longer at inference, producing and checking intermediate steps before answering. That direction has delivered large gains on exactly the kind of multi-step and mathematical problems the compositionality critiques flagged, and it did so without abandoning the transformer. Leaders at frontier labs continue to argue that scale plus better training and inference will carry very far. Anthropic chief executive Dario Amodei, in his 2024 essay "Machines of Loving Grace," described the plausible near-term arrival of what he called "a country of geniuses in a datacenter," a view grounded in continued LLM-derived progress rather than a new architecture.
There is also a practical point. World models, VLA systems, and SSMs are promising but earlier in their trajectory than large language models, and several of their most cited results are demonstrations or research previews rather than broadly deployed products. A fair reading is that these are complementary research directions gathering momentum, not a proven successor that has displaced the transformer.
Who is arguing what
The debate does not split cleanly along company lines, but a few positions are well documented. Beyond the DeepMind work on world models, Meta chief AI scientist Yann LeCun has argued for years that autoregressive text models are not on their own a sufficient path to human-level intelligence, and has championed Joint-Embedding Predictive Architectures, or JEPA, which learn to predict in an abstract representation space rather than generate tokens. Fei-Fei Li's startup World Labs frames "spatial intelligence," a physics-aware understanding of three-dimensional environments, as the next frontier and has raised on the order of a billion dollars to pursue it. On the other side, much of the current product frontier, from OpenAI and others, continues to advance by scaling and refining transformer-based systems, including the reasoning-model line.
Google
Meta
OpenAI
AnthropicThe most common position among practitioners is not a binary. It is that these approaches will be combined: transformers for language and reasoning, world models and VLA systems for grounding and control, and efficiency techniques such as state-space layers woven in where they help. The interesting question is less which single architecture wins than how they compose.
What this means for people building on AI
For anyone whose work sits on top of these systems rather than inside the labs building them, the practical lesson of an unsettled architecture debate is about portability. If it is genuinely unclear whether the most capable system in two years is a larger transformer, a transformer paired with a world model, or a hybrid that includes state-space layers, then welding a workflow to the assumptions of one model family is a bet on an open research question. The lower-risk posture is to keep the workflow as the fixed point and treat the underlying model as something you can swap as the field moves. A model-agnostic workspace such as Metir AI reflects that principle, so that adopting a different or newer model for a task is a setting rather than a rebuild.
The same logic applies at the level of skills and expectations. Understanding what each architecture family is actually good at, language and deduction for transformers, dynamics and planning for world models, grounded control for VLA systems, efficiency for SSMs, is more durable than betting on any single one to subsume the rest.
The bigger picture
The 2026 resurgence of this argument is best read as a sign of a field taking its own open questions seriously, not as an obituary for the transformer. The case that pure text prediction leaves out physical grounding is coherent and, in specific technical forms, formally and empirically supported. The case that scaled transformers keep improving, especially with test-time reasoning, is also supported and visible in shipping systems. Those two things can be true at once. World models, vision-language-action models, and state-space models are real research directions with real early results, and the most likely near-term outcome is combination rather than replacement. The useful stance is to hold both readings, follow the evidence as it arrives, and avoid locking any workflow to the assumption that today's dominant architecture is the last one.
Sources:
- LLMs Can't Jump (position paper) | Tom Zahavy
- Publications | Tom Zahavy
- Language models can't spark scientific revolutions, but world models might | The Decoder
- Genie 3: A new frontier for world models | Google DeepMind
- On Limitations of the Transformer Architecture (arXiv:2402.08164) | Peng, Narayanan, Papadimitriou
- Faith and Fate: Limits of Transformers on Compositionality (arXiv:2305.18654) | Dziri et al.
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces (arXiv:2312.00752) | Gu, Dao
- RT-2: New model translates vision and language into action | Google DeepMind
- Gemini Robotics | Wikipedia
- Machines of Loving Grace | Dario Amodei
- Spatial intelligence is the next frontier of AI, says World Labs' Fei-Fei Li | Fast Company
Image credits
Header image: Demis Hassabis, chief executive of Google DeepMind, photographed during 2024 Nobel Week, by Jennifer 8. Lee via Wikimedia Commons, licensed under CC BY-SA 4.0. In-body photograph of a Boston Dynamics and DARPA Atlas robot during testing, by DARPA via Wikimedia Commons, public domain. Both images reviewed before use.