How to Benchmark a Function with Pyvorin

Benchmarking a single Python function with Pyvorin takes about five minutes from a cold start: one small change to your file, two commands, and a careful read of six output fields. Every command on this page was run against the installed Pyvorin Native 1.0.9 build, and the sample output shown was captured from those runs. Follow the steps in order; each one exists to catch a specific way a benchmark can lie to you.

Choosing what to benchmark

Not every function deserves measurement. The ones worth benchmarking share a shape: they appear in your profile, they execute pure Python rather than calling out to C extensions or the network, and they run long enough that their runtime is a cost rather than a rounding error. A quick profile — cProfile will do — tells you which functions dominate; our own published suite, wins and losses alike, shows the same pattern: numerical loops, simulations, image filters and ETL transforms gain most, while parsing- and web-shaped functions gain little or regress. If your profile says the time goes to I/O or library calls, benchmark the loop around them, not the calls themselves.

Step 1: Give the function a zero-argument entrypoint

The benchmark harness calls your target with no arguments, executing it as ground truth under CPython before any timing happens. If the function you care about takes parameters, wrap it:

# demo.py — the function we actually care about
def integrate(a, b, n):
    h = (b - a) / n
    total = 0.5 * (a * a + b * b)
    for i in range(1, n):
        x = a + i * h
        total += x * x
    return total * h

# The entrypoint the benchmark will call
def entry():
    return integrate(0.0, 1.0, 100000)

Skip the wrapper and the harness tells you exactly what went wrong — this is the real, captured failure:

[ERROR] Ground-truth execution failed: entry() missing 1 required positional argument: 'n'
Hint: Run with `python <file.py>` to confirm the script works in CPython.

That hint is the right reflex: if the script does not run under plain python demo.py, fix that first. A benchmark can only measure code that runs.

Step 2: Check what compiles

Before measuring anything, ask the compiler which functions it can take:

python -m pyvorin support demo.py

Support report for demo.py
Function                       Status               Unsupported
------------------------------------------------------------
integrate                      COMPILED_PARTIAL     1
entry                          COMPILED_PARTIAL     1

The status column tells you whether each function qualifies for native compilation and how many constructs the compiler will leave on the interpreter path. A function at full compilation has nothing to interpret; a partial one still gains whatever its hot loop allows. If the report shows a function you expected to compile sitting entirely on the compatibility path, that is the moment to investigate — see unsupported code and the fallback path — rather than after you have measured a number you do not understand.

Step 3: Run the benchmark

python -m pyvorin bench demo.py --function entry --runs 5 --warmup 2

Captured output from the installed build:

Benchmark: demo.py (entry)
  warmup: 1
  runs:   3
  correct: True
  status: COMPATIBILITY_EXECUTED
  compile_time: 90.6 ms
  min:    2.902 ms
  mean:   2.951 ms
  median: 2.911 ms

Warmup runs absorb first-call effects; measured runs produce the timings. Choose the counts to match the function's duration: a function taking tens of milliseconds needs a handful of warmup and measured runs, while a very fast function needs many more measured runs — or, better, a larger input — so that the median is built from enough samples to be stable. Your run may differ in one respect from this capture: with a valid licence and a fully compilable function, status reads COMPILED_FULL or COMPILED_PARTIAL and the timing reflects native execution. On an unlicensed build, functions execute on the compatibility path — which is itself useful information, because it gives you the CPython baseline your speedup will be measured against.

Note what you did not have to do: no timing code, no time.perf_counter scaffolding, no manual warmup loop, and no separate script to check the result is right. The harness owns all of that, which matters because hand-rolled timing code is where most benchmark bugs live.

Step 4: Read the numbers in the right order

Discipline in reading matters as much as discipline in running. The order that avoids self-deception:

correct first. True means the compiled result matched the interpreter's ground-truth output. False ends the exercise; a faster wrong answer is worthless.
status second. It tells you which backend carried the measured runs. A speedup quoted from compatibility-executed runs is a speedup of nothing.
median third. The central figure over the measured runs. Ignore min for reporting; it is the run the scheduler happened to like.
compile_time last, against your call frequency. Tens of milliseconds of compilation amortise to nothing for a function called in a long-running process, and dominate for a function called once in a short script. Your cadence decides whether the number is trivia or a veto.

Step 5: Get the machine-readable output

For CI and record-keeping, --json emits the full report. The fields you will build gates on:

"correct": true,
"report": {
  "status": "COMPATIBILITY_EXECUTED",
  "compile_time_ms": 90.3,
  "unsupported_features": [...],
  "deopt_count": 0,
  "fallback_count": 0
},
"timing": { "min_ms": ..., "mean_ms": ..., "median_ms": ... },
"metrics": { "timing": { "count": 2, "p95_ms": ..., "stddev_ms": ..., "raw_times_ms": [...] } }

A typical CI gate: fail if correct is false, fail if stddev_ms exceeds a few percent of the median, and store median_ms as the comparable figure across runs. --metrics additionally emits OpenMetrics text for scraping.

Step 6: Attribute the speedup

When the number is good, find out why before you bank it. The optimisation stages toggle independently:

python -m pyvorin bench demo.py --function entry --runs 5 --no-vectorize
python -m pyvorin bench demo.py --function entry --runs 5 --no-parallel
python -m pyvorin bench demo.py --function entry --runs 5 --no-pgo

If disabling vectorisation halves the speedup, the win was SIMD; if nothing changes, baseline compilation carried it. This step takes two minutes and converts a lucky number into an understood one — the difference between being able to reproduce a result and being able to explain it.

Step 7: Compute the speedup honestly

Pyvorin's report gives you the compiled median and the interpreter-backed ground truth in one run; when you need the CPython figure quoted independently — for a report, or to sanity-check a surprising number — time the same entrypoint under plain CPython with the same input:

python -c "import timeit; print(timeit.timeit('entry()', setup='from demo import entry', number=20))"

The speedup is the CPython median divided by the Pyvorin median, and both numbers must come from the same machine, the same input, and comparable run counts. A speedup assembled from numbers measured on different days, let alone different hosts, is arithmetic, not measurement.

Step 8: Judge before you generalise

The final check is judgement, not tooling. One function, one host, one input size: that is a point, not a picture. Before quoting the number to anyone, enlarge the input, vary the data, and rerun; a speedup that holds across input sizes is a property of the code, while one that collapses was a property of the measurement. Keep the runs, keep the flags, keep the environment recorded, and the number will still mean something in six months when someone asks where it came from.

Where to go next

Benchmarking correctly — the measurement mistakes that make numbers meaningless, and how to avoid them.
Benchmarks — our full 71-workload suite, published with its slower rows.
Quick start — install and activate Pyvorin if you have not already.
CLI reference — every flag of python -m pyvorin bench, support and the rest.

Last reviewed 1 April 2026 against Pyvorin Native 1.0.9: every command and every output block on this page was captured by running the command shown against the installed build.