Epoch AI Tests Six Models on Its Own Job: Reliable on Defined Tasks, Weak on Judgment

On October 8, 2026, the research group Epoch AI published "Can AI automate Epoch?", by Kelly Hong and Greg Burnham. The question is concrete: can current frontier models complete Epoch-quality work end to end? Their answer: "We find that it cannot yet replace workers, at least not at Epoch." The study is small, but its failure patterns are more informative than most leaderboard rows.

OpenAI

Anthropic

xAI

Google

Moonshot AI

Qwen

Design of the study

Epoch ran each model once per task, at its highest reasoning setting, inside its own coding harness:

Rubrics were developed with Epoch staff and mix objective and subjective criteria. A single human grader reviewed each output, and the authors also drew on qualitative observations from the outputs and trajectories.

What the results show

• Leaders. Fable 5.1 and GPT-6 Astra had the highest aggregate scores and led most categories, mainly because of reliability on well-defined work such as coding and computational analysis. Epoch says: "Although Fable 5.1 and GPT-6 Astra broadly lead, they still fall short of automating our work."
• Open-weight models lag. Kimi K3 and Qwen 3.8 Max trailed further, including on well-defined tasks. Kimi K3 scores 158 on the Epoch Capabilities Index, roughly tied with Grok 4.6, yet struggled with basic tasks Grok 4.6 handled more reliably. One Kimi K3 data insight applied a filter to all arXiv papers instead of physics papers and never checked it.
• Accuracy of closed models. None of the closed-weight models made factual errors in the Data Insight tasks, and their graphics were factually accurate.
• Computer use improved. Models that failed earlier in 2026 at porting an article to Substack now succeed.
• One task is automated. Epoch removed a graphic design task after Fable 5.1 produced an Epoch-quality output, and says it now largely automates that step.

Epoch's text does not report per-model numeric scores, which appear only in a figure, and the authors say the qualitative findings are more informative than the numbers.

The failure patterns, and why they matter

Epoch's list of failures is the most useful part of the report, because each maps to something a benchmark score would hide:

1. Missing house standards. Models missed implicit conventions. Fable 5.1's performance chart was far more information-heavy than needed and used badges differently from Epoch's reference.
2. Audience fit. GPT-6 Astra chose a niche physics topic (high energy physics and machine learning bibliography trends) instead of a general AI trend suited to Epoch's audience.
3. Under-specified research design. Astra proposed a promising question, but left key elements undefined.
4. Flawed setups presented as findings. An Astra pilot used a 4,096-token budget that truncated 61 of 280 responses. After fixing the budget, it framed the result as "sensitivity to the acquisition budget," which the authors call misleading.
5. Verbosity. Fable 5.1 produced dense executive summaries and long captions, against Epoch's principle of minimal detail.
6. Convergence. Fable 5.1, Gemini 3.8 Flash and Grok 4.6 all chose the same polling topic, and Kimi K3 and Qwen 3.8 Max both proposed similar self-evaluation directions.

Analysis: what a verifiable benchmark cannot see

Epoch's framing is that typical benchmarks emphasize easily verifiable tasks, which capture only part of real work. Its own results illustrate the gap. Reliability on specified steps is largely solved for the leading models in this sample, while the weak points are taste, audience judgment and knowing when a result is an artifact of the setup. Item 4 is notable because the error was in how a result was interpreted, not in the code that produced it, so an automated grader checking the output would likely have passed it.

The convergence finding adds a second angle. When several models independently pick the same topic, outputs can be competent and still undifferentiated, which matters for any work whose value is originality.

Caveats

• Small sample. Eleven tasks, one run each, so scores are noisy.
• Single, subjective grader. Rubrics mix objective and subjective criteria and one person graded each output.
• Epoch-specific tasks. Results may not generalize to other organizations.
• Harness confound. Some gaps may reflect tooling, not models: Astra in Codex has a built-in browser extension, while Qwen 3.8 Max uses Qwen Code with Chrome DevTools MCP.

For a complementary cost view of agent setups, see our analysis of Vals AI's multi-agent study. For how fast these capabilities get cheaper, see Epoch's 13x per year cost decline.

What it means for teams

The report suggests a division of labor that many teams already practice: delegate well-specified production steps, and keep humans on audience, standards and interpretation. It also supports keeping a small personal test set of real tasks rather than relying on public rankings. Because the leading model varied by category in this study, a workspace that lets you run the same task across several models, such as Metir, is a practical way to build that test set.

Sources:

• Can AI automate Epoch? - Epoch AI
• Multi-agent Vibe Code Bench - Vals AI
• AI cost decline, Epoch 13x per year - Metir

Image credits

Header image: data center aisle of server racks, by PiDatacenters, Wikimedia Commons, CC BY-SA 4.0 (source); illustrative. In-body image: aisle of server racks, by Christopher Bowns, Wikimedia Commons, CC BY-SA 2.0 (source).