Can your RL environment be gamed?
A policy trained against a leaky verifier learns the leak, not the task. We probe verifiers with inputs that don't solve the task and report every one that still earns reward, with a reproduction for each.
Five checks
Answer-agnostic probes never look at the correct answer: they are what a policy can discover by trial and error. A spray of guesses only counts as an exploit if it wins more often than its luckiest single guess.
C1 · Null
Empty, whitespace, refusals, the prompt echoed back. Inputs with no answer must earn nothing.
C2 · Gold
The reference solution must earn full reward, every time it is scored. Rejected or flaky gold corrupts the training signal.
C3 · Mutation
The reference solution with a wrong answer swapped in (±1, ×10, negated, halved, an extra digit) must earn nothing.
C4 · Adversarial
Answer sprays, hedges, infinity and NaN, grader and JSON injections, padding, malformed input, and exploits invented by an LLM attacker that reads the verifier's code.
C5 · Tamper
For environments with a file system: can the policy edit the tests, read the answers or disable the checker? Not applicable to in-process reward functions.
Self-tested
Before we audit anyone, the audit must catch 14 verifiers with planted flaws (4 of them grading symbolic answers: fractions, radicals, intervals, expressions) and stay silent on 14 clean ones. The self-test is published below.
Published audits
Audits of open-source verifiers are published only after their maintainers have had time to respond. Audits we run for a company stay private to that company.
| Report | Type | Worst finding | Date |
|---|---|---|---|
| Loading… | |||