A language that asks to be trusted with generated code should be careful about its own claims. This post covers the two benchmarks in the repository: the one we publish numbers for, and the one we deliberately do not.
Runtime: compiled Geno versus hand-written Python
The runtime suite has 77 problems, each written twice: once in Geno and once in idiomatic Python. We compile the Geno version and time both. Problems that finish in under 5 ms on both sides are skipped, because at that scale timer noise and interpreter warm-up dominate.
| Metric | Value |
|---|---|
| Problems measured | 53 of 77 |
| Median ratio | 1.60× |
| P90 ratio | 2.17× |
| Best / worst | 0.92× / 2.84× |
| At or under 2× | 45 of 53 (84.9%) |
The target is at most 2× for at least 80% of problems, and the committed run meets it. Across repeated runs the pass rate moves between about 85% and 91% and the median between 1.57× and 1.63×.
Where the overhead comes from
Compiled Geno keeps guards that Python does not have: integer bit lengths and collection sizes. The slowest problems are dominated by i * i products in tight loops, where each multiplication keeps a bit-length guard because multiplying adds the operands' bit lengths, which makes it a real integer-bomb path. The compiler moves every other check to where it is provably needed, and the results page documents each relaxation alongside the test that pins it.
Correctness: the benchmark we have not published
The question everyone asks is whether models write more correct programs in Geno than in Python. The repository contains a benchmark designed to answer it: matched problems, prompts for both languages, extraction, evaluation, pass@k and error categories.
We have not published frontier-model results, and Geno makes no correctness claim against Python. A credible answer needs at least three frontier models with exact snapshot IDs, a recorded run date and repository revision, pass@1 for every model and language, difficulty and error breakdowns, and the raw prompts and responses alongside the numbers. Until that checklist is complete, the honest statement is that we do not know yet.
The methodology and configuration are public. If you have the budget to run it before we do, we would like to hear what you find.