GENO

Blog/Design

No. 005

Examples are the specification

David IachOctober 14, 20262 min read

Abstract

Ordinary Geno functions must carry at least one example clause, and geno test runs them all. Why we made examples mandatory, and what that buys you when a model writes the code.

In Geno, a function without an example does not type-check. That rule surprises people more than anything else in the language, so here is the reasoning behind it.

func clamp(value: Int, lo: Int, hi: Int) -> Int
    example 5, 0, 10 -> 5
    example -1, 0, 10 -> 0
    example 15, 0, 10 -> 10
    if value < lo then return lo end if
    if value > hi then return hi end if
    return value
end func

Plausible is cheap

Language models are very good at producing code that looks right. Looking right is not the problem worth solving. The useful question is whether the code does what was asked, and a reviewer can only answer it if what was asked is written down somewhere checkable.

An example clause is that written-down intent, in the smallest form that still executes: these inputs, this output. It sits next to the signature, where a reader looks first and where a model writing the function is forced to commit to it.

What geno test does with them

geno test evaluates every example of every function through the interpreter and reports each mismatch with the value it actually got:

$ geno test -v score.geno
  FAIL  score.geno (3/4) 14ms
        x score_label: expected "passing", got "needs work"

3 passed, 1 failed, in 0.04s

That line is the whole point. When a model writes score >= 75 where the examples say 60 passes, the failure names the function, the expectation and the actual value. It is exactly the feedback a repair prompt needs, and nobody had to write a test file.

Contracts cover what examples cannot

Examples pin down specific points. requires and ensures state rules for every call:

func factorial(n: Int) -> Int
    requires n >= 0
    ensures result >= 1
    example 5 -> 120
    example 0 -> 1
    if n <= 1 then return 1 end if
    return n * factorial(n - 1)
end func

Both compile into the generated Python and JavaScript, so a precondition still holds when the program runs outside the interpreter.

Details that matter in practice

  • Floats compare with a relative tolerance of 1e-9. Copy the full value from a real run. A rounded 22.857 fails on purpose: a tolerance loose enough to accept it would also hide real errors.
  • Zero-argument functions use example () -> value.
  • Put examples on the boundaries. The examples behind that failure are 90, 89, 60 and 59, one on each side of each threshold, so they leave exactly one right answer. Examples at 95, 70 and 40 would also catch >= 75, but they pass for any passing bar from 41 to 70, so they could not tell a model that the bar is 60.
  • Exempt functions are main, async functions, app lifecycle hooks and functions marked @untested, whose results depend on the world outside the program. Helpers used only by test blocks are exempt too.
  • Escape hatch: @untested marks a function you cannot sensibly exemplify, and geno test --fail-on-untested makes CI refuse it.

The cost

Mandatory examples make Geno a little slower to write by hand. In exchange, every ordinary function documents its own behaviour, every file carries a regression suite, and every model-written program arrives with evidence attached. For the programs Geno is designed for, that trade is worth making.