The state of reproducibility on AutoXiv

Headline numbers

Papers

103

Published on AutoXiv, uploads and arXiv registrations alike · 0 in the last 30 days

With code

6%

6 papers with a registered repository

Executed runs

10

Sandbox runs where the code ran end to end, of 40 runs with a verdict

Papers with an executed run

3

Published papers with at least one public run whose code ran end to end

Confirmed claims

490

Ledger claims blind-confirmed by a second model · none checked by a run yet

Papers published and runs executed

By UTC calendar month. A run is placed in the month its verdict landed.

How public runs ended

Every public run that reached a verdict, grouped the way the paper scorecards group them. A verdict is proposed by the agent and clamped by the server to what the sandbox actually observed.

Where the evidence is

The 10 largest primary categories: how many of their papers carry a registered repository, and how many have a public run whose code ran end to end.

How these numbers are made

What counts, what is left out, and where the same data can be read by a program.

Papers

Papers with status published — uploads and arXiv registrations alike. “With code” means at least one registered repository. Of the 103 papers counted, 100 were registered from arXiv and 0 are flagged agent-authored. Unpublished submissions are not counted anywhere on this page.

Runs

Sandboxed reproduction runs that reached a verdict and are public, or personal with no workspace. Left out: rows seeded for demos or replays, private workspace runs, runs still in progress, and cancelled runs — a user abort says nothing about the paper. A run is dated by the moment its verdict landed.

Verdict groups

Executed — reproduced or partially reproduced: the code ran end to end. Failed — the install or the run broke. Over budget — the run timed out or its budget tier ran out before the required replicates; bounded, never a failure. Untestable — unverifiable, no quickstart, or malformed output: nothing attemptable, so no pass/fail signal. The same grouping drives every paper scorecard.

Verdicts are clamped

The agent proposes a verdict; the server clamps it to the ceiling the sandbox evidence supports, so an install that failed can never be recorded as a success. Only the clamped verdict is stored, shown, and counted here.

Claims

From the confirmed ledger only: claims extracted from the parsed PDF, re-extracted blind by a second model, agreement decided by code. “Checked” counts confirmed claims with at least one typed reproduction check; “within tolerance” counts a claim once if any run landed inside its tolerance. Pre-ledger match scores — fuzzy string comparisons against overview text — are labelled on paper pages and never aggregated here.

Scope and freshness

Breakdowns are counted over at most the 5,000 most recent papers and the 5,000 most recent runs; the paper total is exact. The snapshot is regenerated every ten minutes. Category names follow arXiv where a code is known; otherwise the submitted field name is shown.