Prefect Integration

A Prefect flow calls a Pyvorin-compiled step the same way it calls any other external process: through a shell command with a meaningful exit code. Pyvorin Native 1.0.9 compiles your Python function to a shared library ahead of time, caches it on disk keyed by a hash of the source and options, and runs it in-process with the result checked against CPython. None of that requires a Prefect SDK integration, which is why the whole recipe fits in four verified commands.

This page walks the recipe in order: classify your functions with support, compile with compile, execute inside a flow task via run, and pin the evidence with bench --json. The same pattern applies to the other orchestrators in this series — see Apache Airflow integration and Dagster integration — with the flow-specific judgement calls noted below.

Why there is no plugin

Pyvorin's compile path is local and in-process; the package contains no network code in the compilation pipeline, so building and running a native artefact never calls out to a service. The only integration surface an orchestrator needs is the ability to launch a process and read its exit status. Prefect tasks can do that directly, which keeps the dependency footprint of your flow unchanged and the upgrade path independent of any middleware. When something fails, you debug a command line, not a plugin's internals. There is also a local-development benefit worth noting: because the runner is just a command, you can execute the exact task command on your laptop — with --compare attached — and get the same compiled behaviour the flow will see in production, before any deployment machinery is involved.

Step one: classify the functions

Point support at the exact module the flow will execute, in the flow's runtime environment:

python -m pyvorin support flows/transform.py

Captured on the build host against an illustrative module containing two numeric functions:

Support report for /tmp/loopdemo/whileloop.py
Function                       Status               Unsupported
------------------------------------------------------------
collatz_steps                  COMPILED_FULL        0
guarded_sum                    COMPILED_FULL        0

The statuses are the compiler's own gate printed early, not a prediction. Anything below COMPILED_FULL still runs correctly — partially supported islands execute through the recorded CPython fallback described in Unsupported code and the fallback path — but you want to know which regime you are in before the flow's schedule does. Run this in CI on every change to the module; it takes seconds.

Step two: compile ahead of the run

python -m pyvorin compile flows/transform.py --fail-on-fallback
python -m pyvorin compile flows/transform.py --json --report build/compile.json

Compilation of a single function takes roughly a quarter of a second on the build host used for this page, including LLVM lowering and linking against the runtime libraries. The artefact lands in the on-disk compile cache — inside the installation tree, keyed by a hash of source, function name, options and runtime timestamps, bounded to 500 entries with per-entry checksums. If your flow workers are ephemeral containers, warm the cache when you build the image rather than at task time; the mechanics are the same as any cached build input and are covered in How to cache artifacts in CI. The --fail-on-fallback flag makes a declined function a non-zero exit at build time, which is the cheapest possible place to catch it.

Step three: run the step inside a task

Keep the compute in a plain Python module. The flow task launches it through the Pyvorin runner:

python -m pyvorin run flows/transform.py --script-mode -- \
    --date 2026-06-03 --partition 14

# Staging only: attach a CPython comparison and timing
python -m pyvorin run flows/transform.py --script-mode --compare

--script-mode executes the file as __main__ and passes sys.argv through, so an existing task script changes only in how it is launched. An illustrative task wrapper — adjust the decorator and call style to the Prefect version you run, since we cannot verify a specific release on this host — is:

# Illustrative only — check your Prefect version's task API.
import subprocess
from prefect import flow, task

@task
def transform(date: str) -> None:
    subprocess.run(
        ["python", "-m", "pyvorin", "run",
         "flows/transform.py", "--script-mode", "--", "--date", date],
        check=True,
    )

@flow
def nightly(date: str):
    transform(date)

The subprocess boundary is a feature, not a compromise. It isolates the compiled process from the flow runner's memory space, makes the exit code the single contract the flow needs to honour, and keeps retries clean: Prefect retries the task, the command reruns, the cached artefact makes the rerun cheap. Passing data between tasks still goes through your normal mechanism — files, object storage, task results — because the runner reads and writes plain Python values. None of your task code needs to know whether the native path fired or a fallback carried it; the correctness check inside run and bench covers both regimes against the CPython ground truth, so a green task exit means a correct answer, not merely a finished one.

Step four: record the measurement

python -m pyvorin bench flows/transform.py --function transform --runs 5 --json \
    > build/bench.json

The JSON report carries the compilation status, a correctness verdict against the CPython ground truth, per-run timings, and fallback and deopt counts. Commit it as a build artefact and diff it between deploys; a falling speedup across releases tells you a hot loop changed shape before your stakeholders notice the bill. Here is the engineering judgement that flow-based orchestration makes sharper than most: if you map a task over many partitions, compile once and map the runs — never let each mapped invocation pay compilation. On this host a cold compile costs around 250 ms, trivial for a nightly flow and worth real money at ten thousand mapped tasks a day. A warmed cache absorbs a mapped rerun entirely, but only if the compile step ran first; make it a deployment dependency of the flow, not a side effect inside the mapped task.

What number to expect

Measured, not promised. The canonical suite of 71 workloads reports a 3.16x geometric mean, a 1.35x median, 54 workloads faster than CPython and 17 slower — published in full, losses included. Every figure in that suite was produced by the same bench command this page asks you to run, on a quiet host with memory caps fixed, so the distribution is a reproducible artefact rather than a marketing summary. Workloads shaped like typical flow steps sit at both ends of that distribution: etl.windowed_aggregate measured 134.62x and financial.moving_average 22.53x, while compression- and string-handling workloads measured at or below 1x. A step that mostly waits on a database or serialises JSON has little interpreted bytecode to remove; Pyvorin will say so, and the honest response is to leave that step alone rather than force a number.

When things go wrong

Failure arrives as a normal process failure, which is what flow retry logic is built around. A compile-time decline is a non-zero exit with the recorded reason, catchable at build time via --fail-on-fallback. A runtime guard failure diverts the individual call to a lazily compiled copy of the original Python — correct semantics, recorded in the fallback log and visible in the run and bench reports as fallback and deopt counts. A licensing problem fails at the licence gate before your code runs. Nothing fails halfway through a native call with state you cannot reconstruct, because the wrapper that bridges Python and native code validates its assumptions on entry and declines rather than guessing.

Where to go next

• Apache Airflow integration — the same recipe for DAG-based scheduling.
• Dagster integration — compiled steps as software-defined assets.
• How to cache artifacts in CI — warming the compile cache for ephemeral workers.
• Benchmarks — the full measured distribution behind the expectations above.

Last reviewed 3 June 2026 against Pyvorin Native 1.0.9 (installed package at /root/pvfinal). All commands were run locally and their output captured; compile timings are host-specific and will differ on other machines.