A number without a transcript isn’t a benchmark.

1. 01

   Public, cloneable repository

   A publicly accessible repository (any host) under an open license (MIT, Apache-2.0, or similar), containing the complete benchmark harness — not just a results writeup.

2. 02

   Pinned environment

   The exact framework version under test, the agent/model identifier and provider, the date the runs were captured, and a committed lockfile for every dependency. If host OS or hardware plausibly affects results, state it.

3. 03

   Fixed, unmodified task definitions

   The exact task text given to the agent, checked into the repo verbatim as plain text or markdown — not paraphrased or summarized in the writeup.

4. 04

   Disclosed run parameters

   Temperature and seed (if the provider supports one), or an explicit statement that provider defaults were used. Any tool available to the agent with a nondeterministic surface — web search, the filesystem, the network, wall-clock time — must be named.

5. 05

   Minimum five independent trials per task

   Every trial’s transcript is retained. Report median and range — a single run, however carefully chosen, is not a submission.

6. 06

   Provider-reported token usage only

   Every figure must come from the token usage object on the raw API response for that call (prompt/completion/cache tokens, however the provider names them) — never a tokenizer estimate computed after the fact.

7. 07

   Full, unredacted transcripts

   Every request/response pair (or full session log) for every trial, hosted at a stable URL, redacted only for secrets/credentials — never for content that affects token counts.

8. 08

   A content-hashed manifest

   A manifest file listing every transcript with its SHA-256 hash, committed alongside the results. The results table on this site links each published number back to that manifest.

9. 09

   One-command reproduction

   A documented script or README procedure letting a third party clone the repo and re-run the full harness without contacting the original submitter.

10. 10

    A frozen, versioned snapshot

    A submission is tied to a specific date, framework version, and model version. When any of those change, that’s a new submission — historical numbers are never edited in place.

A reference shape, not a rigid template — adapt file names to your stack, but keep the separation between task definitions, harness, raw transcripts, and the hash manifest.

benchmark-submission/
├── README.md                 # setup, how to reproduce, summary table
├── task-suite/               # exact, unmodified task definitions
│   ├── 01-existing-component.md
│   ├── 02-unwrapped-api.md
│   ├── 03-new-bridge.md
│   └── 04-error-recovery.md
├── harness/                  # the runner: fixed params, tool wiring
│   └── run.ts
├── transcripts/              # one file per trial, raw request/response
│   ├── 01-existing-component/trial-1.json
│   ├── 01-existing-component/trial-2.json
│   └── ...
├── manifest.json             # { path, sha256 } for every transcript
├── results.json              # aggregated usage per task, per framework
└── package-lock.json         # or the equivalent lockfile for the stack