Benchmarking Correctly

A benchmark is a claim about the future: run this code tomorrow and it will take roughly this long. Most Python benchmark numbers fail that test, not because the tools lie but because the measurement was built to produce an impressive figure rather than an accurate one. This page collects the mistakes we see most often, shows how Pyvorin's own benchmark command is designed to avoid them, and gives you the verified commands — run against the installed 1.0.9 build — to produce numbers you would stake a decision on.

How the benchmark command measures

The tool is one command:

python -m pyvorin bench your_script.py --function your_entrypoint --runs 5 --warmup 1

Its design choices map one-to-one onto the classic mistakes. Warmup runs are separate from measured runs, so first-call effects — compilation, cache population — never contaminate the timing. The reported figure is a median over the measured runs, with mean, minimum, maximum, p95 and standard deviation available in the JSON output, so a single outlier cannot impersonate a result. Compilation time is reported on its own line, never folded into execution time, so you can judge amortisation on your own schedule. And before any timing is trusted, the harness executes the function under CPython as ground truth and compares outputs: correct: True is a precondition for the numbers meaning anything at all. Representative output, captured from the installed build:

Benchmark: demo.py (entry)
  warmup: 1
  runs:   3
  correct: True
  status: COMPATIBILITY_EXECUTED
  compile_time: 90.6 ms
  min:    2.902 ms
  mean:   2.951 ms
  median: 2.911 ms

The mistakes that produce garbage numbers

Timing the cold compile. Compilation takes tens of milliseconds; a hot loop may take microseconds. Fold them together and a 100x workload can look slower than CPython. Compile once, measure steady state, report compile time separately — which is exactly what the output above does.

Benchmarking functions that are too fast. Sub-millisecond functions are dominated by measurement noise: timer granularity, cache warmth, scheduler whims. If your function returns in ten microseconds, enlarge the input until the run takes tens of milliseconds, or benchmark the loop that calls it a thousand times. The 0.05x workloads in our own suite are this failure in reverse — micro-workloads where any fixed overhead dominates.

Measuring the wrong thing. Network calls, disk I/O and time spent inside C extensions are not interpreter work, and no compiler removes them. Our measured suite makes the point honestly: web request handling geomean 0.9x, string manipulation 0.8x, parsing 0.9x. If your profile says the time goes to recv or read, the benchmark to run is not this one.

Reporting the best run. Minimums are flattering and meaningless; the scheduler handed you a quiet machine once. Median over multiple runs is the honest central figure, with spread reported alongside it. For CI, --json exposes p95_ms and stddev_ms — alongside correct, compile_time_ms and the run status — so your pipeline can fail on variance, not just on regression, and can reject any run where correctness did not hold.

Trusting a number without correctness. A speedup from a function that returns the wrong answer is a bug at higher throughput. The harness compares the compiled result against the interpreter's output; treat correct: False as a hard stop, and treat any benchmark tool without an equivalent check as suspect.

Isolating what actually helped

When a number looks surprisingly good, scepticism is the correct engineering response. Pyvorin's optimisation stages can be toggled independently, so you can attribute the speedup rather than admire it:

python -m pyvorin bench your_script.py --function entry --runs 5 --no-vectorize
python -m pyvorin bench your_script.py --function entry --runs 5 --no-parallel
python -m pyvorin bench your_script.py --function entry --runs 5 --no-pgo

If the speedup survives every disable, it came from baseline compilation; if it vanishes with --no-vectorize, it was SIMD. This is the difference between understanding your workload and superstition. --metrics adds OpenMetrics text after the report for scraping into your monitoring stack.

Controlling the environment

Two runs of the same command on the same machine can disagree by several percent because the machine changed, not the code. Before comparing runs — across days, branches or configurations — fix what you can: the same host, the same Python version, the same process count, and nothing else running that competes for the cores you are measuring. Cloud instance types with noisy neighbours are a special case; if your numbers must be portable across machines, collect them on several instance types and report the spread rather than the best. A benchmark run in CI should also record the environment alongside the result — Python version, CPU model, and the flags used — because a number without its environment is a number without a provenance.

Reading published numbers, including ours

Our own published suite — 71 workloads, 48 faster than CPython, 22 slower, one at parity, geomean 3.09x, median 1.29x — deserves the same scrutiny you would give a vendor you distrust. Three cautions apply. One host, one date: the shape of the distribution transfers better than any absolute figure. Suite geomean is not your speedup: the median of 1.29x is the more honest description of a typical workload in that suite, and your position depends on how much of your runtime is acceleratable pure-Python execution. Speedup is not savings: shorter execution converts to lower compute spend only where your costs scale with the accelerated workload. Any benchmark page that cannot show you its slower rows is telling you a fraction of the truth; the full table, losses included, is at the benchmarks page.

A short checklist

Before quoting any number — ours, a competitor's, or your own:

• Warmup separate from measurement, median over at least five runs, spread reported.
• Compilation time reported apart from steady-state time, amortisation judged for your call frequency.
• Correctness verified against the interpreter before timing is trusted.
• Workload shape confirmed CPU-bound and pure-Python before a compiler is blamed or credited.
• Stages toggled independently so the speedup has an attributable cause.

The habit underneath the checklist is simple: benchmark to make a decision, and match the measurement's care to the decision's cost. An afternoon of measuring before a platform commitment is cheap; a platform chosen on a cherry-picked figure is the most expensive measurement failure there is.

Where to go next

• How to benchmark a function — the step-by-step workflow for your first measurement.
• Benchmarks — the full 71-workload results, losses included.
• Benchmark methodology — how the suite is run and how correctness is checked.
• Unsupported code and the fallback path — why some functions measure as interpreter-speed, and when that is correct behaviour.

Last reviewed 6 July 2026 against Pyvorin Native 1.0.9: every command on this page was run against the installed build, and the sample output shown was captured from those runs. Suite figures are extracted from the benchmark artefact dated 13 September 2026.