Methodology

One trial takes a framework’s repository from a tagged blank baseline (benchmark-baseline, a stock scaffold plus the byte-identical spec) to the complete app, in three sequential fresh agent sessions: phase 1 = UI shell, phase 2 = HealthKit, phase 3 = Speech. Each session is a new headless run with zero memory of the previous one, so token cost attributes exactly to its phase. After each session the harness — never the agent — runs the full build gate, commits phase-N, and archives the diff. A red build stops the trial and is recorded as a failure. The main run is 8 trials per framework, interleaved NS → Expo → NS → …, strictly sequential on one machine.

• Throwaway config per session. Every session runs with an isolated configuration directory created fresh and deleted afterward: no personal settings, no global instructions, no conversation memory, no permission history. Only the repo’s committed CLAUDE.md travels in.
• One docs source each, nothing else. Strict MCP configuration loads exactly one documentation server per framework — docs.nativescript.org/mcp (HTTP) for NativeScript, mcp.expo.dev/mcp (HTTP, robot-token auth) for Expo — and no other tools beyond the standard file/shell set. Web fetches remained available to both sides and were counted.
• MCP schema weight, measured not assumed. Tool schemas ride in context on every turn, so an asymmetric MCP could inflate one arm on its own. The turn-1 fixed prefix (system prompt + CLAUDE.md + every tool schema) was 34,574 tokens for NativeScript and 34,740 tokens for Expo — under 1% apart, so the measured gap is agent work, not context overhead.
• Identical prompts. Phase prompts are stored templates, identical across frameworks except the framework name, docs-server name, and build command. They were never edited between trials.

• Pinned everything. Same model (claude-sonnet-5), same harness (Claude Code 2.1.220 (headless)), same 160-turn per-phase cap, same Xcode (Xcode 26.5 (17F42)), for every measured session on both sides.
• Byte-identical spec. SPEC.md and the audio fixture are checksum-identical in both repos. Every user-visible label is pinned exactly — which is also what later allowed one generic UI-test driver to operate all 16 agent-built apps for performance measurement.
• Structurally identical CLAUDE.md. Same sections, same level of help — project layout, build commands, docs sources, one-line native-access pointer. No implementation hints on either side.
• Stock scaffolds, asymmetries included. Both sides start from their framework’s official template. Expo’s ships agent onboarding that NativeScript’s does not — an AGENTS.md, a CLAUDE.md, and an enabled official Claude Code plugin — so it was removed at baseline to keep the level of help equal, recorded verbatim and disclosed rather than quietly deleted.
• Failures are data. Every started trial is in the published record: 78/78 phases build-green across both studies, no hidden retries. Two Expo phase-1 sessions returned headless summaries that under-reported usage by more than 90% against their own transcripts; transcripts are the ground truth, so token counts stand and only the summary-derived cost is withheld for those phases.

• Tokens — four buckets, never collapsed. Input, output, cache-write, and cache-read are read from the session transcripts (the ground truth), deduplicated per API message. Output tokens are the primary metric — they track the agent’s actual work. Cache reads dominate raw counts in agentic sessions, so a single “total” would drown the signal.
• Cost — estimated list price. Costs are estimated list-price figures (subscription auth). Two Expo phase-1 sessions returned headless summaries under-reporting usage by >90% against their transcripts; cost exists only in the summary, so it is withheld for those phases rather than published as a plausible-looking wrong number. Token counts are transcript-derived and exact on both arms.
• Everything else per phase. Agent turns, wall time, tool-call mix (reads / edits / shell / docs-MCP / web), docs-lookup counts, and lines-of-code added split JS-side vs native-side from the per-phase git diffs.
• Provenance per trial. Model, harness version, Xcode, MCP config and versions, auth mode, caps, timestamps, and outcome are recorded in each trial’s manifest.json in the repository.

Build-green is a hard automated gate, but builds-green ≠ feature-perfect. The spec ends with an operator checklist (permission flows including deny paths, step logging and persistence, visible transcription streaming, ≥90% word-match against the reference transcript), run against the archived apps. A trial that fails acceptance gets exactly one standardized remediation session, whose tokens count into that trial — bounding assistance while measuring the realistic quantity: tokens to a working app.

• Training-data familiarity favors Expo. React Native has the largest training-data footprint of any mobile stack; NativeScript is far less represented. Both sides had official docs servers and docs usage was counted, but this cannot be removed — and it is part of real-world token economics today.
• React on one arm, Vue on the other. The one unavoidable asymmetry: each framework is measured with the UI layer its own ecosystem actually uses. Phase 1 is the direct read on what that is worth, and it is the cheapest phase on both arms — 13.0K output tokens median for NativeScript against 8.8K for Expo — so it cannot carry the native-feature result either way.
• The headline comes from the controlled study. The free-choice condition (n=8 per arm) lets agents decide how much to verify, and that choice dominates its numbers. The no-difference headline is drawn from the companion condition (n=5 per arm) where interactive verification was forbidden on both arms, with compliance checked in all 10 transcripts. Both conditions are published in full.
• Two sample sizes, both small. Medians with full ranges are published for every phase. The arms overlap heavily, so this study supports “no detectable difference at this n”, not “proven identical”.
• Single model, single machine, single point in time. Different models or future framework versions may shift the result — which is exactly what replication runs are for.