datagame.ai
Journal · Notes from the archive

Notes from the archive.

← All postsGraders · Oct 2026

A referee that cannot be talked round

Reward hacking is the tax on every environment. A game server pays less of it, but not none.

Every environment eventually meets a model that finds the shortcut: editing the test file, planting a helper that reports success, reading answers it was never meant to see. The field has learned to hide verifier files, run agents without root and without network, and test each grader against good, bad and deliberately adversarial runs before trusting it.

Games have a structural advantage here. The grader is not a script written for the benchmark or a model asked for an opinion. It is the server that already refereed millions of matches, written to stop cheating players long before anyone trained an agent against it. Survival, placement, pipes cleared and wins are counted by code that people have attacked for years.

That is not the same as unhackable. Players find exploits, and speedrunners make a sport of it. So we treat our own patch history as part of the product: known exploits, when they were closed, and the matches recorded before and after. A model that finds a glitch we already fixed tells you something; one that finds a new one tells us something.

Every release lists the checks it passed: reference solution at full marks, do-nothing and wrong runs at zero, hidden files, network off, and the flake rate across reruns.

Talk to us about this data →