A benchmark published on arXiv in August 2026 asks a deceptively simple question: can a frontier AI model, given only the list of papers a researcher had read before a discovery, reconstruct the idea that discovery contained? The answer, across the strongest models tested, is that they can do it between roughly 3% and 15% of the time. The benchmark, called Reconstruction, is one of the more carefully designed attempts yet to measure whether large language models can generate genuinely novel scientific ideas, as opposed to recalling ones they have already seen. Its low scores are the point, and they are worth understanding in detail.
OpenAI
Anthropic
Google
DeepSeekThe result cuts against a common intuition. Models that can pass graduate exams, write working code and summarize dense papers might be expected to also connect the dots between existing work into a new hypothesis. Reconstruction suggests that last step, the leap that defines original research, remains largely out of reach.
What the Benchmark Actually Tests
The setup is elegant. For a given published paper, the authors take the "seed" idea (the paper's core contribution) and remove it, along with the paper's title, abstract and any literature published at or after its date. What remains is the bibliography: the set of earlier papers the researcher had access to before making the discovery. The model is handed that reference context and asked to propose the hypothesis. An independent language model then acts as a judge, comparing the model's proposal to the held-out ground-truth idea and scoring whether it matches.
A blind test, engineered against cheating
How Reconstruction forces a model to reason toward an idea instead of retrieving a known answer.
For each seed paper, build a reference set of work published strictly before its date.
No contemporaneous or later literature is allowed in, blocking hindsight leakage.
Citations are given anonymous IDs so the model cannot look up the seed by its sources.
The model never sees the seed paper’s title or abstract while generating hypotheses.
An independent LLM judge scores the proposed idea against the held-out ground-truth idea.
The care in the design is almost entirely about preventing one failure mode: the model simply recognizing the paper and repeating its known conclusion. A model trained on much of the internet has very likely seen the actual papers being tested. To make the task a genuine test of reasoning rather than recall, the authors freeze each bibliography to only pre-publication work, cut off citations by time, give references anonymous IDs so the model cannot identify the seed paper from its sources, and withhold the seed's title and abstract entirely. Only if those defenses hold does a high score mean what it appears to mean.
Why the Scores Are So Low
A 3% to 15% match rate is strikingly low for systems this capable, and the reason is instructive. Recovering a research idea from its antecedents is not a retrieval task and not a summarization task. It is a synthesis task under deep uncertainty: from dozens of prior results, most combinations of which lead nowhere, the model has to identify the specific gap and the specific novel move that a human researcher found worth pursuing. There are enormous numbers of plausible-sounding hypotheses one could propose from any given bibliography, and only a narrow set corresponds to what actually turned out to be the important idea.
How often frontier models recover the real idea
Match rate against the held-out ground-truth research idea, as reported by the Reconstruction benchmark. Axis runs 0 to 40%.
Even the strongest models cluster in a low band.
A 2.4x reported improvement over the single-model baseline, still far from reliable.
Bars illustrate the reported band and the 2.4x multiplier; they are not exact per-model point scores.
This is a different kind of difficulty from the ones most benchmarks capture. Exam-style tests have a defined correct answer the model can reason toward. Coding tasks have a specification and a test suite. Idea generation has neither: the target is one particular insight out of a vast space of possibilities, and the model gets no feedback signal to steer by. That the strongest models land in the low double digits at best, and often the low single digits, is a measure of how much harder open-ended discovery is than the closed-ended tasks the field has largely optimized for.
Passing an exam is finding the answer to a known question. Research is finding the question worth asking. Those are not the same skill.
On what Reconstruction measures
The One Result That Improved Things
The benchmark did find a lever that helps. Rather than relying on a single model, a pipeline that used cross-model review, generating candidate ideas and then having multiple models evaluate and select among them with a "Swiss" selection process over the top four models, reached a reported 2.4x improvement over the single-model baseline. That is a meaningful gain, and it points at something real: idea quality improves when diverse models critique and filter each other's proposals rather than any one model working alone.
But the improved number should be read in context. A 2.4x gain on a 3% to 15% base is still, in absolute terms, a system that fails to recover the real idea the large majority of the time. The uplift is genuine evidence that ensembling and critique help; it is not evidence that AI can reliably do original science. Both readings have to be held at once: the technique works, and the task remains largely unsolved. Overstating either side misrepresents what the benchmark shows.
Why This Matters Beyond the Number
The result matters because "AI will accelerate scientific discovery" has become one of the central promises of the field, and Reconstruction offers a rare, concrete way to check part of that claim. It suggests that the specific act of generating the novel hypothesis, the creative core of research, is currently a weak point, even as models grow stronger at the surrounding tasks: reviewing literature, running analyses, writing up results, checking proofs. That distinction is useful. It implies the near-term path for AI in science runs through augmenting researchers on the mechanical and analytical parts of the work rather than replacing the ideation at its center.
It also reframes how to think about the ensemble result. The finding that multiple models reviewing and selecting each other's work outperforms any single model is a small, empirical version of a broader pattern: for hard, open-ended problems, diversity of models plus critique beats a lone model running unaided. That has practical implications for how anyone builds serious AI systems, well beyond the narrow question of automated science.
The Careful Reading
There are limits to what any single benchmark establishes, and it is worth stating them. Reconstruction measures idea recovery on published papers across six domains, one machine-learning area plus five Nature-family fields; it is not a measurement of every kind of scientific creativity, and a model that struggles to reconstruct a paper's central idea from its bibliography might still contribute to discovery in ways the test does not capture, such as surfacing overlooked connections or generating many candidate directions for a human to filter. And as with any benchmark, the anti-leakage protections are only as good as their execution; the authors clearly took them seriously, but the general risk that a model has partially seen the target is the permanent hazard of evaluating systems trained on the open web.
What Reconstruction does establish, cleanly, is a lower bound on a specific and important capability, measured with unusual rigor. Read that way, the low scores are not a verdict that AI cannot do science. They are a precise statement of how far the current generation is from doing the hardest part of it alone.
For teams applying AI to hard analytical work today, the ensemble finding is the practical takeaway: no single model is the right answer for every difficult task, and having several models generate, critique and select tends to beat trusting one. That is the same logic behind a model-agnostic approach like Metir AI, which routes across models from different providers rather than depending on any one, so the strongest available model, or combination of models, can be brought to each problem.
Sources:
- Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies | arXiv
- Blind Benchmark Catches Frontier AI at Just Three Percent on Research Idea Recovery | Tech Times
- Reconstruction benchmark, full text | arXiv HTML
Image credits
Header image: Scientific journals on display in a university library, by Vmenkov via Wikimedia Commons, licensed under CC BY-SA 3.0. In-body photograph of a university library reading room by Szalax via Wikimedia Commons, released under CC0.