OpenAI Claims Big Jumps on COBOL and Fortran, But the Benchmarks Do the Talking

OpenAI published results on October 19, 2026 claiming large improvements in how its newest models handle what it calls low-resource programming languages — COBOL, Fortran, Visual Basic, OCaml and others that have far less public code and documentation than Python or JavaScript. The company introduced two evaluations to measure the gains: MultiLangBench, which asks a model to build a SQLite-compatible database engine in 20 languages, and a COBOL migration test that checks whether rewritten programs behave identically to the originals. The headline figures are dramatic, and the structure of the tests deserves as much attention as the scores.

Why OpenAI is pointing at COBOL

OpenAI frames the work around infrastructure that is hard to replace. Citing external sources, it says COBOL underpins more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions, while Fortran appears in weather forecasting and nuclear reactor safety analysis. The company says enterprise customers and developers told it that coding models struggle with these languages.

The maintenance problem OpenAI describes will be familiar to anyone who has inherited an old system: documentation may be incomplete, the original authors may be gone, and the only authoritative statement of a business rule may be the code itself. That last point is the crux. When behavior is undocumented, a rewrite that "works" is not necessarily a rewrite that is correct.

MultiLangBench: build SQLite, 20 different ways

MultiLangBench asks a model to implement a SQLite-compatible database engine from scratch in a specified language. According to OpenAI, the model has to organize a codebase, manage state, handle errors and add features without breaking what already works. To compare results across languages, the evaluation uses database files as a common interface: the candidate program takes an initial database and a SQL query and emits a modified database, which is then checked against the reference SQLite implementation. OpenAI says it adapted tests from SQLite's existing test suite to this format, and reports both test pass rate and API cost.

At maximum reasoning effort, OpenAI says GPT‑6 Astra averages an 87.4% test pass rate across the 20 languages, up from 26.8% for GPT‑5.6 Sol, with gains in every language. The per-language numbers it highlights: COBOL 10.9% to 72.1%, OCaml 25.6% to 85.8%, Fortran 21.4% to 83.5%, and Visual Basic 27.5% to 89.4%. Astra clears 83% in 19 of 20 languages, with COBOL the laggard.

The company also positions GPT‑6 Luna as the cheaper option: a 40.3% average at maximum reasoning effort, also improving on GPT‑5.6 Sol in every language, at an estimated average API cost of $0.77 per sample against $3.48 — roughly 78% lower. OpenAI says it is releasing the MultiLangBench setup, test suite and per-language results on GitHub so developers can inspect the tests and check the languages they care about.

That release is the most useful part of the announcement. It also invites scrutiny, because the task is a greenfield build against a public reference implementation with a public test suite. Writing a new database engine in COBOL is a demanding exercise in code generation; it is not the same exercise as navigating a decades-old codebase whose structure no one chose on purpose. OpenAI does not claim otherwise, but buyers should not read an 87.4% average as a readiness score for legacy maintenance work.

The migration test is the more interesting one

The second evaluation asks models to migrate COBOL programs to C++, Java, Kotlin, Python or Rust while preserving behavior. OpenAI's framing here is sharper than the usual "can it code" pitch. It gives a concrete failure mode: COBOL's fixed-point arithmetic discards excess decimal digits unless rounding is explicitly requested, so a rewrite that rounds instead will run cleanly while producing different financial results. Scoring uses hidden tests requiring byte-for-byte parity on observable outputs, and a migration counts only if every test passes.

All-or-nothing grading against byte-exact output is a strict bar, and it is the right one for this problem domain. On that test, OpenAI says GPT‑6.1 Sol reaches 86.6% accuracy at low reasoning effort versus 5.4% for GPT‑5.6 Sol at the same setting, at 61% of the API cost. Pushing reasoning effort higher takes accuracy to 97.9%, which OpenAI itself describes as nearly saturating the evaluation.

Saturation cuts both ways. It is evidence of real progress on the specific task, and it means the benchmark has little headroom left to differentiate future models. OpenAI does not disclose how many programs the evaluation contains, how large they are, whether they were drawn from production systems or written for the test, or how the hidden tests were constructed. Without that, it is hard to judge how representative the 97.9% is. The company also notes, correctly, that applying this to production software still requires expert review and testing against the system's business rules and dependencies.

Reading the numbers carefully

A few things complicate direct comparison. The two evaluations feature different models — GPT‑6 Astra and GPT‑6 Luna on MultiLangBench, GPT‑6.1 Sol on the migration test — so there is no single model with a reported score on both. The reasoning-effort settings also differ: MultiLangBench results are quoted at maximum effort, while the migration results are given at low and higher effort without the cost figure for the higher setting being specified. Cost figures are described as estimated API cost, and OpenAI does not publish the underlying token counts or pricing in this material.

There is also no human baseline in either evaluation, no latency data, and no information about how many attempts a model gets per sample. For teams estimating whether an automated migration is cheaper than a contracted one, those are the numbers that decide the business case.

The strategic read is straightforward. Legacy modernization is a large, slow, expensive category of enterprise work, and it is one where the buyer's main objection to AI is not capability but risk. By grading on behavioral parity rather than plausibility, OpenAI is addressing that objection on its own terms. Whether the result holds outside a benchmark harness — against copybooks, batch jobs, database bindings and the rest of the environment a mainframe program actually lives in — is not something these two evaluations can settle.

What to watch

• Whether independent teams reproduce the MultiLangBench results using the released GitHub setup, particularly in COBOL, where OpenAI reports the lowest score.
• Whether OpenAI publishes the COBOL migration evaluation itself, or details on the provenance and size of the programs used. It was not released alongside MultiLangBench in this announcement.
• A successor benchmark for migration. At 97.9%, the current test is close to exhausted and will stop distinguishing models quickly.
• Reported outcomes from actual modernization projects, including how much expert review time a model-generated migration still requires — the figure that determines whether the cost math works.
• Pricing and availability clarity across the GPT‑6 Astra, Luna and 6.1 Sol variants, since the cost comparisons here rest on estimates rather than published rates.