On October 19, 2026, Meta Engineering published a detailed look inside Prometheus, its gigawatt-scale AI supercluster in New Albany, Ohio. The post is unusually specific about the unglamorous part of frontier training: how often a giant cluster breaks, and what it takes to keep it productive anyway. Meta says Phase 1 is complete and was used to train its Muse models, including Muse Wonder.
Meta
NVIDIA
Arista NetworksThe headline claims
According to Meta, Prometheus is a former web-traffic data center converted into a frontier-model training site. The target is roughly one million distributed GPUs operating in lockstep. The current build is described as hundreds of thousands of GPUs across traditional and AI-optimized buildings plus rapid-deployment structures, linked by "miles of fiber".
Two efficiency figures stand out. Goodput, the share of time the cluster does useful training work, was over 85 percent for pre-training and over 90 percent for reinforcement learning on the Muse Wonder runs. Mean time between interruptions was about 47,000 GPU hours during training, later improved to over 54,000 GPU hours through firmware fixes.
Three generations of racks
Meta describes three NVIDIA-based rack platforms, each a 72-GPU NVLink domain.
Peak FP4 compute per 72-GPU domain across Meta's three rack platforms
Peak FP4 petaflops per NVL72 domain as published by Meta. Clemente is stated as over 1 exaFLOP, so its bar is a lower bound.
Peak figures are vendor-rated maxima, not sustained training throughput. Meta reports over 85% goodput on pre-training runs at the cluster level.
| Platform | System | Memory per GPU | Rack power | Cooling |
|---|---|---|---|---|
| Catalina | GB200 NVL72 (Ariel variant) | 192 GB HBM3e | About 120 kW per two-rack pair | Not stated in Meta's post |
| Clemente | GB300 NVL72 | 288 GB HBM3e | About 135 kW per rack | Air-assisted or facility liquid cooling |
| San Miguel (upcoming) | Vera Rubin NVL72 | 288 GB HBM4 | Over 170 kW per rack | Fully liquid cooled |
Meta claims a 5 to 10 percent gain for Clemente over GB200 and 10 to 20 percent better performance per watt for San Miguel over GB300 NVL72. These are Meta's own figures. For background on the Rubin generation, see our coverage of Nvidia's Vera Rubin ramp and the OCP Global Summit announcements.
Reliability is the real constraint
The most instructive section concerns failures. Moving from 8-GPU domains (NVL8) to 72-GPU domains (NVL72) increased the fault blast radius by 9 times, because one bad component now affects a larger group. Meta's response combines spare capacity with software that routes around faults.
- In-rack spares. Each NVL72 rack runs a 64-GPU slice and keeps 8 GPUs as spares.
- Reserve racks. About 2 percent extra racks are held for whole-rack faults.
- Worked example. A 32,768 active-GPU run needs 523 NVL72 racks (37,656 GPUs). Meta puts this at about 13 percent above the active footprint, with about 11 points coming from the 64-of-72 slice design.
- PAFT. Parallelism-Aware Fault Tolerance isolates a faulty data-parallel replica while the rest of the cluster keeps training.
- Checkpointing. Asynchronous delta checkpoints with in-memory staging cover faults larger than one replica.
Meta reports that interruptions peaked near 1,000 per 8,000 GPUs per day in mid-2025, then fell 99 percent over a year. The reliability target is 2 interruptions per 8,000 GPUs per day, roughly 100,000 GPU hours between interruptions.
A sanity check on the numbers
Our own arithmetic gives a useful cross-check. Eight thousand GPUs running for a day is 192,000 GPU hours, so 2 interruptions per day corresponds to about 96,000 GPU hours between interruptions, consistent with Meta's "roughly 100K" target. A 54,000 GPU-hour interval would correspond to about 3.6 interruptions per 8,000 GPUs per day, and 47,000 GPU hours to about 4.1. A 99 percent fall from 1,000 would be about 10 per day. These figures come from different runs and measurement windows, so they should be read as order-of-magnitude consistency, not a single time series.
At this scale, uptime is an engineering product. The 99 percent drop in interruptions matters as much to training throughput as any new chip.
The main interruption sources and fixes Meta lists are NVLink signal integrity (over 600,000 NVLink connections, screened with diagnostic burns before production), HBM and LPDDR memory (firmware tuning, memory scrubbing and predictive retirement of degrading devices), NVLink switch health (millisecond-resolution monitoring and automated rerouting) and the backend network (hardware fault detection and rerouting around failed fibers or ports).
Networking: Ethernet, not InfiniBand
Meta uses a dedicated Ethernet back-end network for GPU-to-GPU traffic. Each building fabric connects up to 20,736 GPUs and began on a Disaggregated Scheduled Fabric before moving to a Non-Scheduled Fabric built on 51.2T shallow-buffer switches, running Meta's FBOSS software stack. Between buildings, a Backend Aggregation layer spans 2 km to over 100 km on Arista 7800-series chassis with Broadcom Jericho3 silicon, delivering multiple petabits per second per region pair. Optics are 2x400G FR4 LITE inside buildings and 800G-ZR coherent beyond about 10 km.
Power: the load swings problem
Synchronized training makes the whole cluster behave like one load. Meta says swings of 30 to 40 percent are normal, and starting or pausing a large job can move tens of megawatts within seconds. Its answer is a cluster-wide power manager called Dynamo Dimmer, which gradually lowers GPU power limits near the electrical ceiling instead of halting jobs, backed by rack-level battery units, capacitor trays for peak smoothing and facility battery storage. Each GPU draws 1,400 W.
Meta says it pays the full cost of energy in New Albany and matches energy use with clean and renewable sources. Its next site, Hyperion in Richland Parish, Louisiana, applies Prometheus lessons with closed-loop liquid cooling and full grid connectivity with Entergy. Our earlier coverage of Meta's infrastructure financing is in Meta and BlackRock's El Paso data center.
What this means beyond Meta
- Goodput is a first-class metric. A cluster with 85 percent goodput delivers noticeably more training per dollar than one with 70 percent at identical hardware cost, so reliability engineering is a direct cost lever.
- Bigger domains trade failure radius for bandwidth. The 9 times larger blast radius is the price of NVL72's communication gains, and spares and fault-tolerant software are how that price is paid.
- Power is now a software problem. Throttling gradually, rather than halting, is a design pattern that other large operators are likely to need.
- Open hardware designs travel. Meta says the IcePack liquid-cooled network rack design is being shared through OCP.
For application teams, the downstream effect is indirect but real: reliability and efficiency at the training layer influence how quickly new model generations ship and what they cost. That is one reason model-agnostic platforms like Metir let people move between providers as capabilities and pricing change, rather than tying work to a single lab's release cadence.
Sources:
- Inside Prometheus: Meta's gigawatt-scale AI supercluster (Meta Engineering)
- Everything That Happened in AI Today, Tuesday, October 20 (The Neuron)
Image credits
- Hero and in-body photo: "Facebook, Highway 126, Prineville" from the Digital Public Library of America collection, via Wikimedia Commons, licensed CC BY 4.0. Pulled 2026-10-21 and reviewed visually before use.