Methodology

One trial takes a framework’s repository from a tagged blank baseline (benchmark-baseline, a stock scaffold plus the byte-identical spec) to the complete app, in three sequential fresh agent sessions: phase 1 = UI shell, phase 2 = HealthKit, phase 3 = Speech. Each session is a new headless run with zero memory of the previous one, so token cost attributes exactly to its phase. After each session the harness — never the agent — runs the full build gate, commits phase-N, and archives the diff. A red build stops the trial and is recorded as a failure. The main run is 5 trials per framework, interleaved NS → Lynx → NS → …, strictly sequential on one machine.

• Throwaway config per session. Every session runs with an isolated configuration directory created fresh and deleted afterward: no personal settings, no global instructions, no conversation memory, no permission history. Only the repo’s committed CLAUDE.md travels in.
• One docs source each, nothing else. Strict MCP configuration loads exactly one documentation server per framework — docs.nativescript.org/mcp (HTTP) for NativeScript, @lynx-js/docs-mcp-server 0.2.4 (pinned, local) for LynxJS — and no other tools beyond the standard file/shell set. Web fetches remained available to both sides and were counted.
• MCP schema weight, measured not assumed. Tool schemas ride in context on every turn, so an asymmetric MCP could inflate one arm on its own. The turn-1 fixed prefix (system prompt + CLAUDE.md + every tool schema) was 34,489 tokens for NativeScript and 34,804 tokens for LynxJS — under 1% apart, so the measured gap is agent work, not context overhead.
• Identical prompts. Phase prompts are stored templates, identical across frameworks except the framework name, docs-server name, and build command. They were never edited between trials.

• Pinned everything. Same model (claude-sonnet-5), same harness (Claude Code 2.1.220 (headless)), same 160-turn per-phase cap, same Xcode (Xcode 26.5 (17F42)), for every measured session on both sides.
• Byte-identical spec. SPEC.md and the audio fixture are checksum-identical in both repos. Every user-visible label is pinned exactly — which is also what later allowed one generic UI-test driver to operate all 10 agent-built apps for performance measurement.
• Structurally identical CLAUDE.md. Same sections, same level of help — project layout, build commands, docs sources, one-line native-access pointer. No implementation hints on either side.
• Stock scaffolds, asymmetries included. Both sides start from their framework’s official template. Template differences (Lynx’s Sparkling host app ships more plumbing) are part of each framework’s real starting economics, not noise to remove.
• Failures are data. Every started trial is in the published record. The one infra event (an auth token rotation between trials, caught by preflight before any tokens were spent) is preserved and labeled infra-invalid, not silently retried.

• Tokens — four buckets, never collapsed. Input, output, cache-write, and cache-read are read from the session transcripts (the ground truth), deduplicated per API message. Output tokens are the primary metric — they track the agent’s actual work. Cache reads dominate raw counts in agentic sessions, so a single “total” would drown the signal.
• Cost — estimated list price. Costs are estimated list-price figures (runs used subscription auth); token counts are exact either way.
• Everything else per phase. Agent turns, wall time, tool-call mix (reads / edits / shell / docs-MCP / web), docs-lookup counts, and lines-of-code added split JS-side vs native-side from the per-phase git diffs.
• Provenance per trial. Model, harness version, Xcode, MCP config and versions, auth mode, caps, timestamps, and outcome are recorded in each trial’s manifest.json in the repository.

Build-green is a hard automated gate, but builds-green ≠ feature-perfect. The spec ends with an operator checklist (permission flows including deny paths, step logging and persistence, visible transcription streaming, ≥90% word-match against the reference transcript), run against the archived apps. A trial that fails acceptance gets exactly one standardized remediation session, whose tokens count into that trial — bounding assistance while measuring the realistic quantity: tokens to a working app.

• Training-data familiarity favors NativeScript. A 2014 framework is better represented in model training data than a 2025 one. Both sides had official docs servers, and docs usage was counted — but this cannot be fully removed, and it arguably is part of real-world token economics today.
• LynxJS is measured via Sparkling. TikTok’s productized app layer is the fair analog to NativeScript and the path a real Vue-Lynx team would take — but it is not raw Lynx embedding, and the report is titled accordingly.
• n=5 per side. Medians with full ranges are published. The non-overlapping per-trial ranges make the direction unambiguous; exact ratios are estimates, not constants.
• One capped phase. One Lynx HealthKit phase hit the 160-turn cap; capped numbers are floors, so if anything the measured gap is understated.
• Single model, single machine, single point in time. Different models or future framework versions may shift the ratios — which is exactly what replication runs are for.