Benchmark Methodology: How We Measure Pyvorin

Every figure published on the Pyvorin benchmarks page — the 3.16x geometric mean, the 1.35x median, the 202.8x best case, the 0.05x worst — came out of one harness, on one host, under the conventions documented here. That specificity matters more than usual, because Pyvorin Native 1.0.9 contains two distinct compiler paths, and the path used to produce a number changes the number. If you intend to quote any of these results to a colleague, a procurement team or a finance director, you need to know which path produced it, what was excluded and why, and how to rerun the measurement yourself.

The short version: the canonical published figures come from the legacy CLI bench path — the direct native compiler that python -m pyvorin bench invokes. A route study run on 13 September 2026 re-ran the same suite through the newer morphic-first router and through an experimental autotune mode. Both results are reported below, with their exclusions stated, and neither replaces the canonical run.

Harness conventions

The suite holds 71 self-contained Python programs spread across 26 categories: numerical computing, ETL, compression, cryptography, parsing, simulation, image processing, graph algorithms, sorting, statistics, web request handling, string manipulation and others. Each workload is run twice on the same host — once under stock CPython, once under Pyvorin with native compilation active. No source code leaves the machine at any point; compilation and execution are local and in-process.

Timing conventions:

Warmup runs precede measured runs, so timings describe steady-state execution rather than first-call behaviour.
Compilation time is reported separately from execution time, so amortisation can be judged against your own call frequency.
Per-workload memory caps were fixed at 4096 MB in the benchmark database before the canonical run and were active throughout it. Earlier configurations used smaller caps that predated the compiler's own build footprint; the change is part of the methodology, not an afterthought.
CPU timeouts are active, and the host is quiet — no co-tenant load during the run.

One convention deserves emphasis: outputs are compared for correctness on every run, before any timing is accepted. A workload that returns the wrong answer is failed, not timed. This gate is not decorative. It is what caught the most serious defect the route study uncovered.

The correctness gate

Every workload's output is checked against the CPython oracle — the interpreter's own result for the same program. Only workloads whose compiled output matches are admitted to the timing table. This is the difference between “the compiler ran it” and “the compiler ran it correctly”, and it is the reason the published tables can be trusted at the row level.

The gate earned its keep on 13 September 2026. When the suite was re-run through the morphic-first router, serialization.json_serialize returned an incorrect result — a miscompile on the router path in the 1.0.9 build, now logged as a product bug at the highest priority. Separately, tree.bst_operations failed to compile through the router at all, hitting an error on the router's CPython-backend path that the legacy path does not exhibit. Both workloads were excluded from the router figures, with the reasons recorded.

That was a deliberate choice, and it is the kind of decision a methodology page should expose. We publish the router study at n=69 with two named exclusions rather than at n=71 with a silently wrong answer averaged into the geomean. An excluded workload with a stated reason beats an inflated sample size every time. The legacy path, incidentally, compiles both workloads correctly — json_serialize lands in its usual 1.9x–3.6x category and bst_operations runs at roughly 1.0x — which is precisely why path identity belongs in every published figure.

The two compiler paths

Pyvorin 1.0.9 ships two ways from Python source to native code, and conflating them produces meaningless comparisons.

The legacy CLI bench path. python -m pyvorin bench constructs the native compiler directly. This path includes the ETL-to-morphic lowering bridge, which is enabled by default via PYVORIN_ENABLE_ETL_MORPHIC_BRIDGE, but it does not include the morphic tensor planner. The benchmark harness adapter uses this path. Every canonical figure in the documentation — the 3.16x suite geomean, 54 faster and 17 slower, best 202.8x, worst 0.05x — was produced by it, across all 71 workloads.

The morphic-first router. The unified compiler router, PyvorinCompiler(backend='auto'), attempts the morphic tensor planner first, falls back to the native LLVM compiler second, and diverts to an honest CPython fallback with a recorded reason if neither can hold. The morphic suite worker uses this path. Its 13 September study produced a 3.25x geomean over 69 counted workloads, with 48 faster and 21 slower.

Two findings from that study are worth understanding before quoting the 3.25x. First, not one of the 69 workloads engaged a morphic-pattern backend: the router tried the planner first, the workload shapes pattern-missed, and native LLVM took over. That is the designed behaviour — the router is a routing layer, not a different compiler, and “ran through the router” does not mean “ran as morphic tensors”. Second, the 0.09x suite-level gap between 3.16x and 3.25x is small, but it is a property of the pipeline stage order, not noise; per-workload differences in both directions are larger and real.

Hence the documentation rule adopted after the study: any quoted benchmark figure must state which compiler path produced it. A figure without a path label is an unverifiable figure.

The autotune study, and why autotune is not the default

The third configuration in the study ran the router with autotune enabled — an experimental mode that explores alternative lowering plans per workload rather than accepting the default plan. The result: a 3.09x geomean, with 43 workloads faster and 26 slower.

Read the shape of that result. Autotune is not a free upgrade; it is a trade. The study recorded large wins on image kernels — a Sobel filter gained 16.8x over the default route — and real regressions elsewhere, including a 24.7x slowdown on a windowed aggregate ETL workload. When one mode wins big on some shapes and loses big on others, enabling it by default means shipping other people's regressions.

There is a second, harder problem: the study is single-sample. Each workload ran once per configuration. A mode that produces both +16.8x and −24.7x outcomes across shapes needs repeated measures before anyone can trust the aggregate to one decimal place. Our own recommendation coming out of the study is that autotune must never regress against the default plan — a keep-best guard should sit in front of it — before it is considered for default status. Until that guard exists and repeated-measures data exists to validate it, autotune stays off by default, and the documentation says so.

The best-route ceiling, and why we do not publish it

The study also computed a fourth figure: for each workload, the best result across all three configurations, with a geomean of 3.64x over the 69 shared workloads. We call this the best-route ceiling, and we are explicit that it is an engineering ceiling, not a publishable benchmark.

The reason is structural. The ceiling constructs a hypothetical runner that does not exist: for every workload, it takes the maximum of three different configurations. No single installation, no single command, no single customer deployment achieves 3.64x on this suite, because no single configuration was used to produce it. Quoting it as a headline would be cherry-picking by construction — the exact practice of quoting each workload's best run as if it were one run. The ceiling has real engineering value: it shows how much headroom exists if the router's two defects are fixed and autotune gains its keep-best guard. That is where it belongs, and that is where it stays.

Publication rules

The route study also settled how figures may leave the building. Four rules now govern every benchmark number we publish.

Name the compiler path. Legacy CLI path and morphic-first router are different pipelines, and their figures are not interchangeable. A number without a path label does not ship.

No wins-only subsets. The July 2026 artefact contains a subset figure — a 7.54x geomean across the fifty workloads that ran faster than CPython — that is arithmetically true and methodologically worthless, because the same parse shows sixteen of sixty-six completed workloads ran slower, down to 0.01x. Any subset defined by the outcome it keeps is marketing arithmetic, and we do not publish it as a suite result.

No unverifiable statistics. A “6.69x median” quoted from the July material could not be reproduced from any parse of the artefact — not the full set, not the wins-only subset, not any subset we tried. A figure that cannot be regenerated from the raw data is not a statistic; it is a rumour with decimals. It does not appear in our documentation.

Exclusions are named, never silently dropped. The router study runs at n=69 rather than n=71 because two workloads hit router defects; both are listed, both have bug reports, and the legacy-path results for the same workloads are published alongside so the reader can see what was lost. That is the standard every figure on the benchmarks page is held to.

Hardware, run counts and disclosure

The canonical run was executed on 13 September 2026 against Pyvorin Native 1.0.9, built 12 September 2026, running on CPython 3.12 on linux x86-64. One host. One run per configuration per workload. The suite statistics — geomean, median, counts — are descriptive summaries of that run, not confidence intervals; single-sample methodology cannot produce error bars, and pretending otherwise would be its own form of dishonesty.

The archived July 2026 run illustrates why absolutes travel badly between builds. That run completed 66 of 71 workloads (five then failed to compile; all five compile and run in 1.0.9), with a 3.33x geomean and a 2.45x median. The September build changed the compiler as well as the caps, so the two runs are not a controlled comparison; the geomean moved by 0.17x in the wrong direction from an upgrade that plainly improved the product, because the denominator changed. The distribution's shape — strong numeric winners, a real tail of string- and I/O-bound losers — is stable across both runs. The absolutes are not. Treat this page as a map of what to expect, then measure your own code.

Reproduce the numbers

Everything above can be rerun locally. The commands below were executed against the installed 1.0.9 build during the preparation of this page.

# Which functions Pyvorin can take natively, and at what tier
python -m pyvorin support your_script.py

# Benchmark an entrypoint function (zero-argument)
python -m pyvorin bench your_script.py --function your_entrypoint --runs 5

# Machine-readable output with full timing detail
python -m pyvorin bench your_script.py --function your_entrypoint --runs 3 --json

A real run, captured during verification, shows what honest output looks like — correctness first, timing second:

Benchmark: demo.py (entrypoint)
  warmup: 3
  runs:   3
  correct: True
  status: COMPILED_FULL
  compile_time: 261.208 ms
  min:    0.386 ms
  mean:   0.397 ms
  median: 0.390 ms

The --json variant exposes the full record — raw_times_ms, stddev_ms, p95_ms, compile_time_ms, correct, fallback_count, deopt_count — which is the minimum record any benchmark you run for a purchasing decision should keep. Store the raw runs, the script hash, the Pyvorin version and the date. Derived statistics without raw runs are marketing, not measurement.

You can also run a miniature version of the route study on your own code. The bench flags --no-vectorize, --no-parallel and --no-pgo isolate the contribution of each optimisation stage, so a workload where vectorisation is the entire win looks different from one where the gain came from parallel reduction. A run per configuration, recorded in the same JSON format, gives you per-workload evidence of which stage carries your speedup — the same discipline, at the scale of one function.

Where to go next

Benchmarks — the full canonical results table and the honest reading of it.
Benchmarking correctly — the measurement mistakes that produce misleading numbers, and how to avoid them.
How to benchmark a function — a step-by-step run of the bench command on your own code.
Quick start — install Pyvorin and get to a first measured result.

Last reviewed 6 March 2026 against the canonical benchmark artefact and route study dated 13 September 2026 (Pyvorin Native 1.0.9, built 12 September 2026, CPython 3.12, linux x86-64, 4096 MB memory caps active, quiet host). Commands quoted on this page were executed against the installed build during preparation.