Custom Benchmark Suites with pyvorin bench --json

A benchmark that is not repeatable is a rumour. If you want numbers you can defend — to a reviewer, a customer, or your own team in three months — you need a suite: fixed workloads, captured raw timings, stored baselines, and a machine-readable output format that CI can diff. pyvorin-native 1.0.9 gives you that format through python -m pyvorin bench --json, and this page builds the whole pipeline on top of it using only commands verified against the installed build.

Everything quoted below was run against pyvorin-native 1.0.9. The timing values are real output from a live run on the verification host; treat them as shape references, not promises, because your hardware will differ.

The one command at the centre

bench compiles a target function, verifies the result against a CPython ground-truth execution, then times it. The --json flag emits the full report as JSON. These are its options, captured from the installed CLI:

python -m pyvorin bench --help

usage: pyvorin bench [-h] [--json] [--vectorize | --no-vectorize]
                     [--parallel | --no-parallel] [--pgo | --no-pgo]
                     [--fast-math | --no-fast-math] [--function FUNCTION]
                     [--warmup WARMUP] [--runs RUNS] [--script-mode]
                     [--metrics]
                     file

The flags that matter for suite work: --function (or -f) targets a specific function in a multi-function file, --warmup and --runs fix the measurement loop, and --no-vectorize / --no-parallel / --no-pgo let you attribute a speedup to a specific optimisation stage. Bench needs no licence for measurement, which makes it safe on shared CI runners.

What the JSON report contains

Start with a minimal workload. Bench calls the target function with no arguments, so wrap parameters in defaults or constants:

# benchmarks/numeric_sum.py
def numeric_sum() -> int:
    total = 0
    for i in range(100000):
        total += i * i
    return total

if __name__ == "__main__":
    print(numeric_sum())

Run it and keep the JSON:

python -m pyvorin bench benchmarks/numeric_sum.py --json --runs 10 --warmup 3 > results/numeric_sum.json

The fields you build a suite on, from a real run on the verification host:

"timing": {
  "count": 10,
  "min_ms": 0.007650349289178848,
  "mean_ms": 0.00830749049782753,
  "median_ms": 0.008031493052840233,
  "p95_ms": 0.00975483562797308,
  "max_ms": 0.010223593562841415,
  "stddev_ms": 0.0007926956347514717,
  "raw_times_ms": [ 0.010223593562841415, ... ]
}

Surrounding that block, the report carries correct (a boolean from the ground-truth comparison), ground_truth, compile_time_ms, fallback_count, deopt_count, the function status (COMPILED_FULL on this run), and a run_time_ms headline figure. Store the entire document. The headline number is the least useful part of it; raw_times_ms, p95_ms and stddev_ms are what let you reason about noise, and fallback_count is your early-warning that a "result" was actually the interpreter in a trench coat. If correct is false, the timing is worthless — gate on it first.

Wrapping multiple functions

A suite with one function is a demo. Organise workloads as one file per function, or several functions per file with explicit targeting:

# benchmarks/kernels.py
def numeric_sum() -> int:
    total = 0
    for i in range(50000):
        total += i
    return total

def string_build() -> str:
    parts = []
    for i in range(1000):
        parts.append(str(i * 7))
    return ",".join(parts)

if __name__ == "__main__":
    print(numeric_sum(), len(string_build()))

When a file contains more than one compilable function, auto-detection picks one; pin the target explicitly so the suite does not silently drift to a different function as files evolve:

python -m pyvorin bench benchmarks/kernels.py --function string_build --json --runs 10 --warmup 3 \
  > results/string_build.json

The engineering judgement here is about workload sizing. A function that finishes in eight microseconds, like the numeric loop above, is dominated by timing noise on any shared host; functions that run for tens of milliseconds give you a stable median at the cost of slower CI. Our recommended pattern: pick input sizes that land the compiled run in the 1–50 ms band where practical, keep the same sizes forever once recorded as a baseline, and resist "tuning" them after a regression — changing the workload to make the numbers behave is how benchmark suites go stale without anyone noticing.

Aggregating results across the suite

A small aggregator turns a directory of JSON reports into a summary table. Everything it reads is a field verified above:

# aggregate.py
import json, pathlib, statistics

rows = []
for path in sorted(pathlib.Path("results").glob("*.json")):
    report = json.loads(path.read_text())
    t = report["metrics"]["timing"]
    rows.append({
        "workload": report["target"],
        "correct": report["correct"],
        "median_ms": t["median_ms"],
        "p95_ms": t["p95_ms"],
        "stddev_ms": t["stddev_ms"],
        "cv": t["stddev_ms"] / t["mean_ms"],  # coefficient of variation
        "fallbacks": report["metrics"]["fallback_count"],
    })

for r in rows:
    print(f"{r['workload']:16s} correct={r['correct']} "
          f"median={r['median_ms']:.3f}ms p95={r['p95_ms']:.3f}ms "
          f"cv={r['cv']:.3f} fallbacks={r['fallbacks']}")

The coefficient of variation — standard deviation over mean — is the single best health metric in the table. A workload with a CV above about 0.1 on a quiet host is telling you the measurement environment is noisy or the workload is too short to time; fix that before comparing anything against a baseline. A rising CV is also an early signal that a machine has changed (noisy neighbour, thermal throttle) even when the medians still look plausible.

Storing baselines and gating CI

Commit a baseline per stable machine class, generated on a quiet host with pinned --runs and --warmup:

python -m pyvorin bench benchmarks/numeric_sum.py --json --runs 30 --warmup 5 > baseline/numeric_sum.json

Then gate regressions in CI with a tolerance band — medians move, and a brittle gate trains people to ignore it:

# gate.py — fail CI if any workload regresses more than 15%
import json, pathlib, sys

baseline = {p.stem: json.loads(p.read_text())
            for p in pathlib.Path("baseline").glob("*.json")}
failures = []

for p in pathlib.Path("results").glob("*.json"):
    name = p.stem
    if name not in baseline:
        continue
    b = baseline[name]["metrics"]["timing"]["median_ms"]
    c = json.loads(p.read_text())
    if not c["correct"]:
        failures.append(f"{name}: incorrect result")
        continue
    m = c["metrics"]["timing"]["median_ms"]
    if m > b * 1.15:
        failures.append(f"{name}: median {m:.3f}ms vs baseline {b:.3f}ms")

if failures:
    print("\n".join(failures))
    sys.exit(1)
print("benchmark gate: OK")

Two disciplines make this trustworthy. Pin the runner environment as far as your platform allows — same machine class, quiet host, fixed CPU count — because you are comparing medians, not physics; and version the baseline files alongside the code, so a deliberate performance change updates the baseline in the same pull request that causes it. A baseline that only ever changes when someone notices a failure is not a baseline, it is a mirror.

For the CPython comparison side — the speedup number — use python -m pyvorin run file.py --compare, which times the same loop under CPython and reports the ratio, verified in a live run:

CPython:  3.041 ms (result=333328333350000)
Pyvorin:  0.008 ms (result=333328333350000)
Speedup:  370.88x
Correct:  YES

Treat single-run speedups as indicative. The honest figure for publication is a repeated-measures median of speedups, and the benchmarking correctly page covers why, along with warm-up and caching effects that can dominate short runs.

Failure modes to expect

The suite will meet real failure modes, and handling them explicitly is what makes it repeatable. A ground-truth failure ([ERROR] Ground-truth execution failed) usually means the target takes arguments — bench invokes it with none — so wrap parameters in constants. An unexpected fallback_count above zero means that workload ran through the CPython fallback rather than native code; its timing belongs in the fallback report, not the native baseline. A correct: false result must fail the gate unconditionally, whatever the timing says. And a compile status like COMPATIBILITY_EXECUTED instead of COMPILED_FULL tells you the function used constructs the native path handled through a compatibility tier — record it and decide whether that workload belongs in the native suite or the compatibility suite. The unsupported and fallback page describes those tiers in full.

If you need a pure-CPython reference measurement for a given function — for an honest A/B of "compiled versus interpreter" on identical hardware — the mechanisms are on the forcing the CPython fallback page.

Where to go next

Benchmark methodology — how the published figures are produced, and which compiler path they come from.
Benchmarking correctly — warm-up, caching, and repeated-measures discipline for numbers you can defend.
How to force the CPython fallback — verified mechanisms for pure-interpreter reference runs.
Unsupported code and the fallback path — reading fallback_count and the compatibility tiers.

Last reviewed 5 August 2026 against pyvorin-native 1.0.9 installed at /root/pvfinal. The bench --json fields, the multi-function targeting behaviour, and the run --compare output were all captured from live runs on the verification host; timings are shape references and will differ on other hardware.