How to Run a Speed Proof

A speed proof is a before/after measurement of one function on one machine, documented well enough that a sceptical finance director could re-run it and get the same answer. Pyvorin gives you every ingredient — a correctness check, warmed timed runs, a median, and the compile cost reported separately — but the defensibility comes from discipline, not from the tool. This page is that discipline, applied end to end with commands we ran against the installed pyvorin-native 1.0.9 build and output captured from those runs. Follow it and the number you quote will survive an audit; skip the warmup and it will not survive the first re-run.

An older version of this page described a single proof command that produced a savings estimate. No such command exists in the product CLI. What exists is better: a small set of verified commands whose output you assemble into the proof yourself, which forces you to understand each number you report.

The workload

Every proof needs a fixed subject. Ours is a saved file, math_ops.py, containing a hot function and a zero-argument entrypoint, unchanged for the entire exercise:

def square_sum(n: int) -> int:
    total = 0
    for i in range(n):
        total += i * i
    return total

def entry():
    return square_sum(100000)

Two properties make it suitable, and both are worth copying. The work is CPU-bound pure Python with no I/O, so the measurement reflects the compiler rather than the network card. And the entrypoint takes no arguments, which the harness requires: it executes the target as ground truth under CPython before any timing happens. If your real function takes parameters, wrap it the same way — the wrapper is part of the measured subject and stays fixed for both sides of the comparison.

Step 1: Capture the baseline

The CPython figure is your "before", and it must come from the same machine, the same input and comparable run counts as the "after". The run --compare command measures both sides in one invocation with matched warmup and timed loops, which removes the most common way proofs go wrong — comparing numbers that were never measured the same way:

python -m pyvorin run math_ops.py --function entry --compare --runs 5 --warmup 2

Captured output:

File:     math_ops.py
Target:   entry
Warmup:   2
Runs:     5
CPython:  3.121 ms (result=333328333350000)
Pyvorin:  0.009 ms (result=333328333350000)
Speedup:  346.24x
Correct:  YES
Functions:
  square_sum                     NOT_EVALUATED
  entry                          COMPILED_FULL

Three fields carry the proof. Correct: YES — both backends returned the identical result, 333328333350000, so the comparison is between two correct programs, not between a correct one and a fast wrong one. entry COMPILED_FULL — the Pyvorin side actually ran native code; a compatibility-executed "after" would be proof of nothing. And the paired timings with matched warmup and run counts. The 346.24x headline is real captured output from our machine — and it is also exactly the kind of number you must not generalise from. It reflects one small integer loop on one host. Your proof measures your function.

Step 2: Build the distribution, not the anecdote

Warmup deserves a sentence of its own, because it is the step people skip and the step that most changes the answer. The first call into freshly compiled native code pays costs that steady-state calls do not — library initialisation, cache warming, first-touch page faults. Timing without warmup measures those costs and reports them as the function's speed. The harness runs your warmup count untimed before recording anything; use it. For a function in the millisecond range, two or three warmup runs suffice; for very fast functions, raise both counts so the median rests on enough samples to be stable.

Five runs is a sketch. For a defensible figure, take more samples and report the median, using bench with --json so every raw sample lands in the record:

python -m pyvorin bench math_ops.py --function entry --runs 7 --warmup 2 --json

Captured fields (abridged):

"timing": {
    "min_ms": 0.00795070081949234,
    "mean_ms": 0.009216567767517907,
    "median_ms": 0.008120667189359665
},
"metrics": {
    "timing": {
      "count": 7,
      "p95_ms": 0.012307288125157356,
      "stddev_ms": 0.0019773764137734013,
      "raw_times_ms": [ 0.01342780888080597,
        0.009692739695310593, 0.009212177246809006,
        0.008081085979938507, 0.00795070081949234,
        0.008120667189359665, 0.008030794560909271 ]
    },
    "fallback_count": 0,
    "deopt_count": 0
}

Why median rather than the minimum: the minimum is the run the scheduler happened to like — here, 0.00795 ms, nearly 40% below the median. The median is the central sample; it moves little if one run is interrupted. Report it, keep the raw times, and record the spread: a median with a large standard deviation relative to itself is a noisy environment, and a noisy environment is a caveat your write-up must carry, not hide. Also note fallback_count and deopt_count at zero — on a real proof, non-zero values there are a reason to stop and read how to handle compilation failure before trusting anything above them.

Step 3: Separate the compile cost

Every honest proof addresses the compile-time line. The harness reports it separately from execution for a reason:

compile_time: 238.903 ms

That is what compilation cost on a cold cache. Re-run the same command and the second report shows the effect of the disk compile cache — a repeat compile of the same function reported 0.000 ms in our runs, because the artefact in <site-packages>/.pyvorin_cache/disk_compile was keyed by an unchanged source hash. In a long-running process, compilation happens once and amortises across every call; in CI, a warm cache restores the artefact instead of recompiling it. Either way, the steady-state figure your proof should lead with is the execution median, with the compile cost stated alongside as a one-off, not silently folded in and not silently dropped.

Do the break-even arithmetic yourself, in public. Using our captured numbers: a compile cost of roughly 240 ms against a per-call saving of about 3.11 ms breaks even after roughly 80 calls. Below that cadence, the proof should say so. Above it, the compile cost is trivia and you say that instead. The formula is division, not judgement; the judgement is choosing to show it.

Step 4: Attribute before you bank

A speedup you cannot explain is a number you cannot defend. The optimisation stages toggle independently, so attribute the win while it is fresh:

python -m pyvorin bench math_ops.py --function entry --runs 7 --no-vectorize
python -m pyvorin bench math_ops.py --function entry --runs 7 --no-parallel
python -m pyvorin bench math_ops.py --function entry --runs 7 --no-pgo

If disabling vectorisation collapses the median, SIMD carried the result; if nothing moves, baseline compilation did. One of these runs is also a cheap replication check: the flagged and unflagged medians should sit close together, and a large gap between two medians of the same code is telling you about your machine, not your compiler. The deeper measurement pitfalls — warmup cheats, shared-host noise, input sizes chosen to flatter — are catalogued on the benchmarking correctly page, and the everyday mechanics of the harness are in how to benchmark a function.

Step 5: Write it up so it survives contact

The write-up is where most proofs quietly die. A record that will still mean something in six months contains, at minimum:

• The exact subject. File name, the function measured, the input size, and the fact that the file did not change between the baseline and the measured runs.
• The environment. Hardware, operating system, Python version, pyvorin-native version. Our captures came from CPython 3.12.3 with pyvorin-native 1.0.9; yours will differ, and the difference is data.
• The method. The verbatim commands, warmup and run counts, and which figure you are reporting — we recommend the median, with the minimum and standard deviation kept as corroboration.
• The correctness gate. The correct: True / Correct: YES line, quoted. A speed proof without it is a rumour.
• The compile cost and the amortisation verdict. One-off cost, break-even call count, and your function's actual cadence.
• The caveats. One host, one input size, one workload shape. Our own published suite — 71 workloads, geomean 3.16× on the test host — includes 17 workloads slower than CPython, which is precisely why a single-function proof must scope its claims to its own measurement.

Resist rounding the headline into marketing. If the median says 346×, report 346× on this machine with this input, and state both qualifiers in the same sentence. The moment the qualifiers detach from the number, the proof stops being evidence and becomes a claim — and claims get audited in a way measurements do not.

Where to go next

• How to benchmark a function — the harness mechanics behind every command used here.
• Benchmarking correctly — the measurement mistakes that make numbers meaningless.
• How to handle compilation failure — what to do when the proof's status column is not COMPILED_FULL.
• Caching compiled artifacts in CI — keep the compile cost out of every repeated build.

Last reviewed 9 April 2026 against pyvorin-native 1.0.9 installed at /root/pvfinal. The --compare output, the --json timing fields and the cold-cache versus warm-cache compile times shown are captured from real runs; the break-even figure is arithmetic on those captured numbers, stated as such.