Measuring Pyvorin Adoption — Real Metrics for 1.0.9

There is no metered-credit figure to chart in pyvorin-native: compilation is local, un-metered and un-credited, so any KPI built on credit burn measures a product you do not have. Adoption measurement has to be rebuilt from what the toolchain actually emits — per-function compile reports, benchmark JSON, cache state and exit codes. This page defines five metrics, every one derived from a verified command output.

The theme connecting them: measure coverage and trend, not volume. Telemetry event counts are a poor proxy — compile and execute events are sampled at 1% — and raw run counts reward activity rather than value.

Metric one: native-path coverage

The first question an adoption programme answers is "how much of our target code is on the native path". The product answers it per file:

python -m pyvorin support src/hotpaths.py
python -m pyvorin inspect src/hotpaths.py

inspect prints a summary line — Native eligible: 4/6, Fallback required: 1/6, Failed: 1/6, Risk level: LOW — which is the numerator and denominator of your coverage figure. Add --json to support and the same rows arrive as parseable records, which is how the coverage metric feeds a CI dashboard without screen-scraping. Define the target set deliberately: the functions your profiling says dominate runtime, not every function in the repository. A coverage number against "all Python files" is ungameable only in the sense that nobody can win it; a coverage number against the hot-path list from your last profiling pass is a metric a team can actually move.

Metric two: CI benchmark deltas

Coverage tells you what compiled; timing tells you what it bought. Run the benchmark in CI on a schedule and store the JSON:

python -m pyvorin bench src/hotpaths.py --function entrypoint --runs 5 --json > bench.json

The fields worth trending are timing.mean_ms, timing.median_ms, timing.p95_ms and the top-level correct flag. Track the CPython comparison too — run --compare reports a speedup alongside — because a Pyvorin-only improvement that CPython also gained is an environment shift, not an adoption win. One methodological caution from our own runs: timings on a noisy CI host scatter, so treat the rolling median across nightly runs as the signal and any single run as noise. A worked assertion, in the shape we run ourselves:

python -m pyvorin bench src/hotpaths.py --function entrypoint --runs 5 --json | \
  python -c 'import json,sys; d=json.load(sys.stdin); \
assert d["correct"] and not d["report"]["drift_detected"]; \
print(d["timing"]["median_ms"])' >> bench-history.log

The assertion fails the job on a correctness regression before the number is ever recorded — timing history contaminated by invalid runs is a metric you cannot trust later.

Metric three: compile cache behaviour

Compilation cost is paid once and then amortised through the disk cache, so cache behaviour is a real efficiency metric. 1.0.9 exposes no explicit hit-rate counter, but the signal is derivable: report.compile_time_ms in the bench JSON is ~200 ms on a cold compile of a small function and effectively zero when the cached artefact loads. Sampling that field across CI runs gives a cold-compile ratio, and python -m pyvorin cache status (file count and disk usage, verified: 158 files, 1.20 MB) bounds growth against the 500-entry LRU cap.

Watch the two failure shapes. A cold-compile ratio pinned near 1.0 means the cache is not persisting between runs — on ephemeral CI runners the fix is a cached job workspace, since the artefact cache lives inside the install tree. A cache that fills and thrashes suggests per-run cache keys churning, usually from compiling variant option sets; standardise the flags you benchmark with.

Metric four: fallback-rate trend

The fallback rate is the adoption metric most worth a dashboard, because it moves for interesting reasons: a dependency upgrade introduces constructs the frontend demotes, a refactor drags a helper into a tainted call graph, a new module arrives pre-migrated. All three are visible in the verified JSON fields:

"status": "COMPATIBILITY_EXECUTED",
"fallback_used": true,
"fallback_reason": "unsupported AST pattern; calls a compat-mode function",
"correct": true

Aggregate fallback_used and the per-function fallback_reason from nightly compile --report runs, and track the CI gate alongside: exit 2 under --fail-on-fallback on files already claimed as native is a regression alert, not a statistic. Two companion fields belong on the same dashboard — drift_detected and overflow_count — because a rising drift count is not a fallback, it is a correctness warning wearing similar clothes. The field definitions are documented under error handling and diagnostics.

Metric five: correctness and licence health

Two gate metrics complete the set, both cheap enough to run on every CI job. Correctness: assert correct: true and drift_detected: false in every bench JSON your pipeline produces — an oracle mismatch is an exit-1 compile and a hard stop, but only if you wire the assertion in. Licence: pyvorin check exits non-zero when the gate fails, and pyvorin status emits the expires_at timestamp for runway warnings. Neither number is glamorous. Both gate every other metric on this page, because coverage without a licence and speed without correctness are figures that cannot survive contact with production. A team that has adopted Pyvorin but let the licence lapse has adopted a fallback interpreter.

Metrics to retire

Three legacy KPIs deserve explicit retirement. Compile-credit burn: metered credits do not exist in the native product, and tracking them imports a hosted-service mental model the architecture does not have. Telemetry event volume: compile and execute events are 1%-sampled metadata, fine for product telemetry, useless as an adoption measure — a tenfold usage increase can sit inside sampling noise. And "functions compiled, cumulative": it never decreases, never distinguishes hot paths from dead code, and says nothing about whether the compiled functions are the ones your users feel. Every metric on this page can move in both directions. That is what makes it a metric.

A judgement call on targets

The temptation in any adoption programme is a single coverage target — "95% of functions native by Q4". Resist it, and set two targets instead: a coverage floor on the profiled hot-path list, and a ceiling on the fallback rate of files already claimed as native. The pair matters because they pull in opposite directions. Chasing raw coverage pushes teams to compile fallback-prone functions, which raises your fallback rate while adding nothing to runtime; guarding the fallback ceiling alone lets coverage stall at the easy files. We have watched a single "percent native" number produce both pathologies in the same quarter. Two dials, one for ambition and one for honesty, are harder to game and easier to explain to the people paying for the licences.

Where to go next

Benchmarks — the canonical workload results that calibrate what timing deltas are worth chasing.
Correctness validation — the oracle and drift machinery behind metrics four and five.
Supported workloads — which categories your coverage metric should concentrate on.
Pyvorin CLI reference — every command and JSON field the five metrics read from.

Last reviewed 22 July 2026 against pyvorin-native 1.0.9 installed at /root/pvfinal. All commands, JSON fields, exit codes and sample outputs cited here were verified in real runs on 13 September 2026; the 1% telemetry sampling figure and the absence of compile metering come from the installed binaries, not from documentation.