The Inference Speed War: Why Tokens Per Second Became AI's New Battleground

For most of the modern AI era, the headline number was intelligence. Which model scored highest, reasoned deepest, wrote the best code. In mid-2026, a second number has forced its way into the conversation: how fast a model actually runs. When Cerebras confirmed in July that it was serving OpenAI's frontier GPT-5.6 Sol at up to 750 tokens per second, roughly fifteen times a typical GPU deployment, it crystallised a shift that had been building all year. Speed is no longer a footnote. It is becoming a product.

This piece explains why that happened, how the competing hardware architectures actually differ, and what the numbers mean for the way AI gets built.

Why speed suddenly matters

For a single question and a single answer, inference speed is close to invisible. A model that streams at 60 tokens per second feels responsive enough when you are reading its reply in real time. Nobody reads faster than the text arrives.

Agents changed the math. An autonomous agent completing a task does not produce one answer. It produces a long internal chain: it plans, calls a tool, reads the result, reasons about it, calls another tool, corrects a mistake, and repeats, sometimes for dozens or hundreds of steps. Most of those tokens are never shown to a human. They are the model thinking and acting. And every one of them is billed against the clock.

Two consequences follow. First, latency compounds. A workflow with fifty sequential model calls feels completely different at 60 tokens per second than at 750, and the difference is the gap between a task that takes minutes and one that takes seconds. Second, throughput becomes an economic variable, because faster serving of the same hardware means more completed tasks per dollar of infrastructure. As agents move from demos into production, speed stops being a nicety and starts being a constraint on what is even feasible.

The tokens-per-second gap between architectures

Peak single-stream output throughput, July 2026. Specialised inference silicon versus a typical GPU cloud serving a frontier model. Higher is faster.

Running GPT-5.6 Sol at 750 tokens per second, Cerebras reports roughly 15 times the speed of a frontier model on batched GPU serving. Open models on the same wafer clear 3,000.

The three architectures competing for the fast lane

The reason a frontier model can run at 60 tokens per second on one setup and 750 on another comes down to a bottleneck that has little to do with raw compute. Generating text one token at a time is memory-bound: for every token, the hardware must move the model's weights from memory to the processing units. The winner is usually whoever moves those weights fastest, not whoever has the most raw math throughput. Three approaches attack that bottleneck differently.

Wafer-scale engines (Cerebras)

Cerebras builds a single chip the size of a dinner plate. Its WSE-3 packs around 900,000 cores and enough on-chip memory to hold large models without shuttling weights back and forth to external DRAM. Because the weights live on the chip, the memory-bandwidth wall that throttles conventional hardware largely disappears. That is how Cerebras reports roughly 3,000 tokens per second on open models like gpt-oss-120B, and 750 on a frontier model as large as GPT-5.6 Sol.

Language Processing Units (Groq)

Groq's LPU is a purpose-built inference chip that emphasises deterministic, low-latency execution. Rather than one enormous wafer, it uses a streaming architecture designed for consistent, predictable token delivery, which is well suited to interactive applications where steady responsiveness matters more than peak throughput. Groq reports figures in the range of 476 tokens per second on gpt-oss-120B and around 750 on Llama 3.3 70B.

Reconfigurable dataflow (SambaNova) and GPUs (NVIDIA)

SambaNova's reconfigurable dataflow units are tuned to serve many users at once, optimising aggregate throughput under heavy concurrent load rather than single-stream peak. GPUs, meanwhile, remain the general-purpose workhorse of AI. They are unmatched for training and flexible for everything else, but for pure single-stream token generation they carry the memory-movement overhead that the specialised chips are built to avoid. That is the gap the 750-versus-50 comparison is really measuring.

The trade-offs the raw numbers hide

A single "tokens per second" figure flatters whichever vendor quotes it, so it is worth naming what the number leaves out.

Peak versus concurrent. A headline speed is often single-stream. Real deployments serve thousands of users at once, and throughput per user falls under load. The right question is speed at the concurrency you actually run.
Model portability. Specialised inference silicon shines on the models it has been tuned for. Getting a brand-new frontier model running well on non-GPU hardware takes engineering effort, which is part of why serving GPT-5.6 Sol on wafer-scale silicon was itself news.
Cost per token, not just speed. Fast and cheap are not the same thing. The economically interesting metric is completed work per dollar, which folds together speed, hardware cost and utilisation.
Training versus inference. None of these specialised chips displaces GPUs for training frontier models. This is a race for the inference layer specifically, which is a large and growing slice of total AI spend but not the whole picture.

Where this leaves builders

The strategic takeaway is not that GPUs are finished. They remain the foundation of training and the default for most serving. The takeaway is that inference has fragmented into a genuine market with different winners for different jobs: wafer-scale for raw single-stream speed, LPUs for consistent interactive latency, dataflow units for high concurrency, and GPUs for flexibility and everything that is not yet ported.

For teams building on top of models, that fragmentation is mostly good news, because it pushes both speed up and cost down. It also adds a decision most builders were not making a year ago: not just which model, but on which infrastructure, at what speed, for what price. Increasingly the answer varies by workload, which is why abstraction layers that route a request to the right model and the right backend, rather than hard-wiring one, are becoming part of the standard toolkit. A platform such as Metir AI sits at that layer, letting a team send a latency-sensitive agent to fast infrastructure and a bulk job to whatever is cheapest without rewriting anything.

The bigger picture

The arrival of speed as a first-class metric is a sign of maturity. Early in a technology's life, the only question that matters is whether it works at all. As it becomes infrastructure, the questions multiply: how fast, how cheap, how reliable, at what scale. AI is now firmly in that second phase. Intelligence still leads the headlines, but in mid-2026 the quiet race over tokens per second may end up shaping which agentic products are actually practical to run, and which stay stuck in the demo.

Match every workload to the right model and speed

As inference splinters into fast, cheap and flexible lanes, the advantage goes to teams that can route each job to the best fit rather than betting on one setup. Metir AI brings the leading models together in one workspace so you can send latency-sensitive agents, high-volume batches and hard reasoning each to the option that suits them, without juggling providers. Try Metir AI free and build on the whole frontier, not one corner of it.

Sources: