What Does an Evaluation License? A Commit-Bound Census of Claim Replay in Inspect Evals
Abstract
Benchmarks can run without determining what their results license. We freeze a large evaluation collection and attempt to replay its historical claims. Most units stop because the evidence required for replay is not bound. Where replay is possible, different claims remain stable at different resolutions. We make this otherwise implicit inference step explicit and executable.
Community
What does a benchmark result actually let us conclude?
In a commit-bound census of 124 Inspect Evals units, 110 historical claims stop at explicit evidence or semantic gates. Among the executable cases, exact values, winners, complete rankings, and pairwise relations do not always have the same identified set.
We make that claim-to-evidence layer executable and fail-closed.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation (2026)
- No Judgment Without a Reason: Counterfactual Receipts for Versioned AI Evaluators (2026)
- From Subjective Judgments to Auditable Standards:Protocol-Guided AI Auditing of Website Redundancy (2026)
- When May an Agent Stop? Evidence-Carrying Termination for Tool-Using LLMs (2026)
- Governed Persistent Memory: Source-Bound State Semantics and Fail-Closed Release for Long-Horizon Agents (2026)
- When benchmark inferences do not compose: Projectibility in AI evaluation (2026)
- One Run Is Not an Idea: The Implementation Lottery in Automated Research (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
A benchmark score is a promise, not a proof. This paper shows most Inspect Evals can't be replayed to the claim they're attached to — 110 of 124 stop before deterministic inference. That's the gap between a leaderboard number and a deployment decision.
I've been burned by this exact thing. You pick a model because it tops some eval, wire it into a pipeline, and then discover the eval's scoring function was doing half the work. The model wasn't good — the harness was forgiving.
The commit-bound framing is the right fix. If the eval isn't pinned to the exact code that produced the number, the number is just a vibe with a decimal point. I'd like to see this extended to the other direction too: not just "can we replay the eval," but "does the eval's inference match what the model actually does in prod." A deterministic eval that tests the wrong thing is still a deterministic lie.
110 of 124 is a damning ratio. That's not a few sloppy repos — that's a culture problem.
Get this paper in your agent:
hf papers read 2608.19269 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper