Input tokens
Everything sent to the model on a given turn: system prompt, conversation history, and tool results fed back in.
Methodology
What “rigorous” means here.
A token-usage benchmark is only as useful as the discipline behind it. This page defines exactly what we measure, why perfect determinism isn’t achievable for agentic tasks, and how a comparison isolates one variable at a time. The concrete checklist every submission must satisfy lives on the submission format page.
[01]What we measure
“Token usage” is not one number. Every submission reports each of the following, per task and in aggregate, taken directly from the model provider’s API usage field on the raw response:
Output tokens
Everything the model generated: reasoning, code, and tool calls.
Cache read tokens
Input tokens served from a provider’s prompt cache. Reported separately because cached tokens carry a different cost profile than a full read — folding them into “input” hides most of the effect a framework’s boilerplate size has on repeat calls.
Cache write tokens
Tokens written into a provider’s prompt cache for the first time on a turn. Distinguishing this from a cache read is what makes multi-turn sessions comparable across frameworks with very different amounts of static context.
[02]On determinism
Why we say “reproducible,” not “deterministic.”
Even at temperature 0 with a fixed seed, an agentic coding task is not perfectly deterministic end to end. Tool outputs the agent reads back — compiler errors, file listings, package registry responses, timestamps — can vary between runs, and an agent’s downstream token usage depends on what it reads. Any benchmark that reports a single run as “the” token count for a framework is reporting noise as if it were signal.
Instead of pretending this away, Agent Tokenomics requires every nondeterministic input to be disclosed, requires a minimum of five independent trials per task, and reports the median and range rather than a single figure. A framework comparison is a claim about a distribution, not a point estimate.
[03]Isolating the variable
For a comparison to say anything about the frameworks themselves rather than the setup, everything else is held constant: the exact model and version, sampling parameters, the tools available to the agent, the system instructions, and the number of trials. Each comparison page documents precisely what was held constant for that pairing. Where a task must name a framework-specific API or class, the task wording changes only in that unavoidable noun — never in structure, difficulty, or the amount of guidance given.
[04]Categories of token cost
Every task suite breaks token cost into categories so a framework can win on one axis and lose on another without that nuance getting flattened into a single headline number:
Setup tokens
The cost of establishing framework context (imports, type declarations, boilerplate) before any real work happens.
Per-call tokens
The marginal cost of one additional interop call once setup is done.
Error-recovery tokens
The cost incurred when a first attempt fails and the agent must read an error and retry.
Total task tokens
End-to-end cost for a fully working solution — the number most people actually care about.
Ready to run a benchmark?
The submission format page turns everything above into a concrete checklist and review process.
Read the submission format