Reproduce it

The harness is a small bash + Node toolchain in nstudio/token-economics ↗. Everything a trial produces is archived, so the published numbers can be audited without re-running anything — and re-running is one command per trial.

01Requirements

macOS with Xcode 26.5 (17F42) and an iOS Simulator
Node 22+ and the Claude Code CLI (Claude Code 2.1.220 was used)
A Claude subscription or API key — the runner handles headless auth automatically, including subscription auth via an isolated throwaway config
Frameworks under test: NativeScript 9.1-alpha · nativescript-vue 3 · Vite · LynxJS 3.6 · vue-lynx 0.5.1 · Sparkling 2.0.1 (brownfield host)

02Repository layout

token-economics/
  PLAN.md                 methods — the canonical experimental design
  harness/
    frameworks/<id>.json  one per framework: repos, build gates, docs MCP
    studies/<slug>.json   one per comparison: arms, spec version, baseline tag
    spec/v<X.Y>/          versioned SPEC.md + fixed assets
    prompts/              phase templates (never edited between trials)
    run-trial.sh          one trial: reset → 3 phased sessions → gates → archive
    run-main.sh           the full interleaved main run (resumable)
    analyze.mjs           transcripts → summary.csv / summary.json
    manifest.mjs          SHA-256 evidence manifest (write + verify)
    perf/                 release rebuilds + size/launch/memory + UI-test driver
  apps/
    <fw>-benchmark/       each framework’s tagged blank start state
    trials/               every trial’s final source tree, exported
  results/<study>/
    <trial-id>/           manifest, transcripts (JSONL), diffs, build logs
    perf/                 per-app runtime measurements + summaries
    MANIFEST.sha256       hash of every archived artifact
    REPORT.md             findings write-up

03Run the token benchmark

cd harness

# one trial = reset to baseline → 3 phased headless sessions → build gates
./run-trial.sh nativescript-vs-lynxjs nativescript my-trial-1
./run-trial.sh nativescript-vs-lynxjs lynxjs my-trial-2

# or the full main run: 5 interleaved rounds, sequential, resumable
./run-main.sh nativescript-vs-lynxjs 5

# recompute every number from the archived transcripts
node analyze.mjs nativescript-vs-lynxjs

# verify the archived evidence is byte-identical to what was published
node manifest.mjs nativescript-vs-lynxjs --check

Each phase runs in a fresh isolated session with exactly one docs MCP; the harness — not the agent — verifies the build gate, commits the phase, and archives the prompt, headless summary, full transcript, build log, and diff. Trials that fail are recorded, not retried.

04Run the performance suite

cd harness/perf

# rebuild every trial’s final tree in Release and archive the .apps
STUDY=nativescript-vs-lynxjs ./build-release.sh

# size + 5 cold launches + idle memory/CPU per app (quiet host!)
STUDY=nativescript-vs-lynxjs SIM_UDID=<simulator-udid> ./measure.sh
node analyze-perf.mjs nativescript-vs-lynxjs

# tap-to-result platform-API latency via the generic UI-test driver
STUDY=nativescript-vs-lynxjs SIM_UDID=<simulator-udid> ITERS=3 ./run-interactions.sh
node analyze-interactions.mjs nativescript-vs-lynxjs

The UI-test driver operates any app that follows the spec’s pinned labels — it never modifies the agent-built artifacts. Use a long-lived simulator for the interaction suite: pristine simulators lack the speech-recognition services entirely, and speech is a physical-device metric regardless.

05Vary it

Different model: MODEL=<model-id> ./run-main.sh nativescript-vs-lynxjs 5 — never compare trials across models; run full sets per model.
New framework versions: update a baseline, verify it builds green, re-tag benchmark-baseline, and run a fresh set — the harness is version-agnostic.
Different framework pair: add a harness/frameworks/<id>.json and a harness/studies/<slug>.json — the runners are registry-driven, so a new comparison is data, not a code change. See PLAN.md for the full design rationale, then submit your results.

← Back to results Open the repository