Inspect Robots Open-Source Contribution

The problem

Benchmark authors needed one supported way to interpret human operator verdicts. The implementation lived behind a private helper, so downstream code either depended on private internals or recreated success semantics and risked drifting from the scorer.

What I built

I promoted verdict normalization to a public API, centralized the affirmative vocabulary and normalization rules, and added snapshot and regression coverage. I separately designed a metrics protocol and registry proposal so tasks could report mean, standard error, and success rate without breaking existing mean-only behavior.

Turning a private convention into a public contract

The operator scorer already knew which human verdicts counted as affirmative, but downstream benchmarks had no supported API for asking the same question. PR #415 promoted that logic into is_affirmative_verdict, made the scorer call it, and documented the helper for benchmark authors. The change is small in surface area and important in effect: there is now one contract instead of several nearly identical copies.

Tests define the compatibility boundary

The regression suite covers accepted verdict vocabulary, case and surrounding whitespace, rejected values, null input, and direct agreement between the helper and the operator scorer. An API snapshot also makes accidental public-surface changes visible in review.

The metrics layer is a proposal, not a shipped claim

I also designed a backward-compatible metrics layer with a metric protocol, its own registry namespace, and mean, standard error, and success-rate built-ins. Tasks would keep mean as the default, so existing benchmarks would not change. That design was sent to the maintainer for discussion; it is intentionally described here as proposed work rather than implemented functionality.