On October 19, 2026, OpenAI announced two new evaluations aimed at older programming languages and reported large gains for its GPT-6 family. On a COBOL migration test, GPT-6.1 Sol fully migrated 97.9% of programs at higher reasoning effort, against 5.4% for GPT-5.6 Sol, according to Dataconomy's October 20 summary. On a second test, MultiLangBench, GPT-6 Astra averaged 87.4% across 20 languages. OpenAI designed and ran both evaluations, and no outside group has verified the results.
OpenAIWhat the COBOL test measures
The task is to convert COBOL programs into C++, Java, Kotlin, Python or Rust without changing behaviour. Scoring uses hidden tests that require byte-for-byte matching output, and a migration counts only if every test passes. That is a strict bar. COBOL's fixed-point arithmetic, for example, drops extra decimal digits unless rounding is requested, so a rewrite that rounds can run without errors and still produce different financial results. A test that checks exact output catches that class of silent error.
OpenAI's COBOL migration test: share of programs fully migrated
A program counts only if every hidden test matches the original byte for byte.
Source: OpenAI evaluation announced October 19, 2026, as reported by Dataconomy. Run by OpenAI; not independently verified.
OpenAI says the 97.9% result brings the test close to saturation, meaning it can no longer separate strong models from each other. The practical reading is that this benchmark has done its job and a harder one is needed, not that migration is solved.
MultiLangBench: building a database in 20 languages
The second evaluation asks a model to build a SQLite-compatible database engine from scratch in each of 20 languages, checked against the reference SQLite implementation using tests adapted from SQLite's own suite. Reported averages:
| Model | MultiLangBench average |
|---|---|
| GPT-6 Astra (max effort) | 87.4% |
| GPT-6 Luna | 40.3% |
| GPT-5.6 Sol | 26.8% |
GPT-6 Astra scored above 83% in 19 of the 20 languages, and COBOL was the hardest language. On cost, the report puts GPT-6 Luna's estimated API cost at $0.77 per sample against $3.48 for GPT-5.6 Sol, about 78% lower, though Luna scored 40.3% against Sol's 26.8% so the comparison is of two weaker models. The Dataconomy summary does not state the reasoning effort for the GPT-5.6 Sol and Luna runs. Separately, the COBOL test's low-effort cost is described differently across outlets, so this analysis does not rely on it.
A migration that runs cleanly can still be wrong. The test counts only programs whose output matches byte for byte.
Analysis of OpenAI's COBOL evaluation design
Why low-resource languages matter
OpenAI groups COBOL, Fortran and Visual Basic as low-resource languages, with less training code and documentation than Python or JavaScript. It cites COBOL as supporting more than 40% of online banking systems and 95% of ATM transactions, and Fortran as used in weather forecasting and nuclear reactor safety analysis. Those are OpenAI's figures and are not independently sourced here. The language was created in 1959, and Dataconomy, citing ITHardware, reports it rose from 23rd to 16th on the TIOBE index between October 2025 and this month, a sign of renewed attention rather than new adoption.
Reading the numbers with care
Four caveats shape how much weight the headline deserves.
- Vendor-run evaluations. OpenAI built the tests and ran them. Vendor benchmarks are informative, but without an outside replication the numbers describe OpenAI's setup, not the field. OpenAI says it plans to publish the MultiLangBench setup, test suite and per-language results on GitHub, which would allow replication.
- Benchmark programs are not production systems. A test program is self-contained. Real mainframe estates carry decades of undocumented business rules, batch schedules, database and middleware dependencies, and data formats. OpenAI itself says production use still requires expert review and testing against the system's business rules and dependencies.
- A big jump from a low base. Going from 5.4% to 97.9% is a large relative change, but the earlier model's score was very low, so part of the story is that the task moved from near-impossible to routine in one generation. The comparison also does not state the older model's reasoning effort.
- Behavioural equivalence is the hard part. Byte-for-byte output checks are strong evidence, but they test the inputs in the suite. Edge cases outside the tests remain the residual risk.
What this means for modernisation projects
For organisations with COBOL estates, the result moves the bottleneck. If translation of individual programs is increasingly reliable, the scarce work becomes specifying behaviour, building regression suites from production data, running old and new systems in parallel and managing cutover. These are engineering and governance tasks, and they are where migration budgets are likely to concentrate. Teams evaluating this will probably want to run their own sample programs across several models and reasoning settings, since cost and accuracy trade off sharply with effort, which is an area where a model-agnostic workspace such as Metir is convenient. For the broader release context, see our coverage of the GPT-6.1 Sol release, benchmarks and pricing.
What to watch next
- Publication of the MultiLangBench repository and independent reruns.
- Whether banks and public agencies report pilot migrations with measured defect rates.
- A successor benchmark built on larger, multi-file systems with real dependencies.
Sources:
- Dataconomy: GPT-6.1 Sol scores 97.9% on OpenAI's COBOL test (October 20, 2026)
- AI Weekly: AI News Today, October 21, 2026
- Metir: GPT-6.1 Sol release, benchmarks and pricing
Image credits
Header and in-body image: punched card with COBOL text, via Wikimedia Commons, licensed under CC BY-SA 3.0.