Correctness Validation

pyvorin-native treats CPython as the oracle for your program's semantics. Every benchmark compares compiled output against a ground-truth interpreter run before reporting a number, every compile that can run a function without arguments validates it against the same oracle, and any disagreement discards the compiled claim rather than shipping the doubt. This page documents that machinery as it exists in 1.0.9 — the flows, the JSON fields, and the failure behaviour — all verified against the installed build.

The bench ground-truth flow

When you run the benchmark, the first thing the tooling does is not timing. It executes your entrypoint under plain CPython and captures the result:

$ python -m pyvorin bench workload.py --json
{
  "file": "workload.py",
  "target": "main",
  "correct": true,
  "ground_truth": 74992500.0,
  ...
}

The compiled function is then run on the same inputs and its result compared against ground_truth; the top-level correct flag records the outcome. In our verification run the compiled result matched exactly and correct was true. If the entrypoint cannot run without arguments, bench stops before timing with a precise error — Ground-truth execution failed: add() missing 2 required positional arguments — because a benchmark without a correctness anchor is a number without meaning. Point --function at a zero-argument callable to give the oracle something to hold.

The compile-time oracle

Compilation applies the same discipline per function, whenever the function can be invoked without arguments. A successful validation shows up in explain as:

main
  Status:  COMPILED_FULL
  Oracle:  passed
  Public claim allowed: True

A disagreement is terminal for the claim. In our run, a generator function produced:

gen     FAILED   122.934 ms | Oracle mismatch: compiled result differs from CPython

Nothing about this is ambiguous. The meaning is exact: native code ran, the interpreter ran, the results differed. The function is marked FAILED, public_claim_allowed is false, and the compile exits 1. There is no configuration in 1.0.9 that promotes an oracle-mismatched function to compiled, and no retry setting that softens the verdict. The compiled claim is only made when it can be proven, and abandoned when it cannot — which is also why a FAILED row in a nightly CI report should be treated as a release blocker, not a flaky test to retry.

When arguments are required, the oracle cannot self-invoke, and the tooling says so rather than pretending: Runtime validation not performed: function requires arguments, with cpython_oracle_status set to not_run. Such a compile is real native code but an unverified one — which is why --fail-on-fallback exits 2 on it with the same suspicion it applies to an actual fallback. Zero-argument entrypoints earn you the strongest claim the product can make.

The correctness fields in every report

The JSON report from bench --json or compile --report carries a structured correctness block per function. Fields we captured from a real run:

The counters were zero in our verification runs — which is itself the point: these fields exist to be asserted in CI, so a regression turns a silent drift into a failing build. Assert correct and drift_detected on every report your pipeline produces; the definitions behind the related failure modes are on the error handling and diagnostics page. Watch overflow_count too when your kernels do widened arithmetic on large values — it exists precisely because silent numeric wraparound is the class of bug an oracle comparison on typical inputs might not catch on its own.

Correctness after validation: the guard contract

Validation covers the code as compiled. At runtime, native execution rests on assumptions recorded at compile time — argument counts, dtypes, memory layout, aliasing between arrays the kernel assumed were independent. Guards check those assumptions at each call boundary, and the contract on failure preserves correctness by construction: when the original Python is available, the call diverts to a lazily compiled fallback and returns its result; when no fallback exists, the wrapper raises GuardRuntimeError with a structured record of which guard failed and why, instead of guessing. A partially applied mutation is never replayed. Correctness is not traded for speed at any point in this pipeline. The guard mechanics are documented on the error handling and diagnostics page.

What validation is not

Three limits, stated plainly. First, the oracle compares outputs on the inputs you run — it is dynamic validation, not a proof for all inputs. Property-based test suites pair naturally here: a run of generated cases through bench extends the oracle's coverage far beyond the hand-written example. Second, a function that was never validated (argument-taking, oracle not run) is not certified correct merely because compilation succeeded; the tooling is explicit about which claim has been earned. Third, the compatibility path — where unsupported constructs demote a function to interpreted execution — is correct by construction, because the fallback is your original program running under CPython; the validation record still says so (fallback_used: true with correct: true), keeping "failed to compile" and "failed to run" as separate, individually recorded events.

What does not happen is equally worth stating: a mismatch is never uploaded, billed, or silently logged server-side as a data point to ignore. The record is local and structured — FAILED, the reason, the flags above — and the compiled path is simply not used. If you believe you have found a genuine miscompile, the material to send us is your reproducing function; the tooling has already done the diagnosis.

A judgement call on claiming results

The public_claim_allowed flag encodes a policy we recommend adopting internally as well as externally. In our own measurement practice we apply a three-tier claim rule: results from functions with Oracle: passed can be quoted anywhere; results with the oracle unrun can be used for engineering decisions but marked as unvalidated; anything FAILED or drift_detected is never quoted at all, in either direction — not as a speedup and not as a regression, because an unverified number is not evidence of anything. It is tempting, especially after a long tuning session, to lead a report with the best figure on screen. The flag exists to make that a deliberate decision rather than an accident of which function happened to be selected when the benchmark ran.

Where to go next

• Benchmarks — the canonical results, all produced under this validation discipline.
• Error handling and diagnostics — the failure taxonomy around oracle mismatches and guard failures.
• Supported workloads — which functions reach the validated states most often.
• Benchmark methodology — how timing and correctness checks are run together.

Last reviewed 14 July 2026 against pyvorin-native 1.0.9 installed at /root/pvfinal. The ground-truth flow, oracle messages, field names and values above were captured from real runs on 13 September 2026; the claim that mismatches are handled locally and never uploaded reflects the verified outbound surface, which contains no error-reporting endpoint.