Apache Airflow Integration — Compile DAG Tasks with Pyvorin

Pyvorin Native 1.0.9 compiles a Python function to machine code ahead of time and leaves the compiled artefact on disk; running it later is a local operation with no service to deploy and no source leaving the machine. That makes the Airflow integration question a simple one: where in a DAG do you put the compile step, and how does a task invoke the compiled code. This page answers both, using only mechanics verified against the installed package — the support, compile, run and bench commands run through python -m pyvorin.

The pattern below is deliberately orchestrator-neutral at its core. Pyvorin ships no Airflow plugin, and it does not need one: every stage of the workflow is an ordinary shell command with a meaningful exit code, which means any Airflow operator that can launch a process — a BashOperator, a PythonOperator wrapping a subprocess call — is a sufficient integration point. The orchestrator schedules; the compiler works the same way it does on your laptop. That neutrality cuts both ways: upgrading Airflow never breaks your compilation step, and upgrading Pyvorin never touches your DAG definitions, because the contract between them is a command line and a cache directory.

One assumption runs through everything below: the workers that run your tasks have the same Pyvorin build and the same compiled cache available at run time as the environment where you verified it. The steps for making that true in CI come after the basic pattern.

The shape of the integration

Three commands carry the whole integration. python -m pyvorin support tells you which functions in a file will compile natively. python -m pyvorin compile performs the compilation and leaves the result in the on-disk compile cache. python -m pyvorin run executes a script with the native path active, checking the result against CPython. A fourth, python -m pyvorin bench --json, produces the machine-readable measurement you keep as evidence. Nothing here imports Airflow, and nothing here requires network access at task time.

Compilation happens locally and in-process. The compile path contains no network code, so the only outbound calls the package can make are licence validation and telemetry, both documented in the architecture report — neither touches your source or your data. For teams with strict egress rules, that property is worth stating in the same change request as the speedup.

Step one: verify support before you schedule anything

Run the support report against the exact file the DAG will execute, in the same environment the DAG runs in:

python -m pyvorin support dags/transform.py

On this host, against a small illustrative module with two numeric functions, the report reads:

Support report for /tmp/loopdemo/forloop.py
Function                       Status               Unsupported
------------------------------------------------------------
trapezoid                      COMPILED_FULL        0
small_unroll                   COMPILED_FULL        0

The statuses come from the same compatibility gate the compiler itself applies, so this report is not an estimate — it is the decision, printed early. Functions marked COMPILED_PARTIAL or COMPATIBILITY_EXECUTED still run correctly; they just run through the CPython fallback for the unsupported parts, which is the honest behaviour described in Unsupported code and the fallback path. The point of running support in your pipeline is to find out which case you are in before a scheduled run finds out for you.

Step two: compile as a build step, not a mystery

Compilation is an explicit, fast operation — around a quarter of a second per function on the build host used for this page, including linking. Treat it as part of your deploy or image-build step:

# Fail the build if anything silently falls back
python -m pyvorin compile dags/transform.py --fail-on-fallback

# Save a machine-readable report alongside the artefact
python -m pyvorin compile dags/transform.py --json --report build/compile_report.json

The compiled shared library lands in the on-disk compile cache, keyed by a hash of the source, the function name, the compile options and the runtime and compiler timestamps, with per-entry checksums and a 500-entry LRU bound. Unchanged code on a rebuilt worker is a cache hit, not a recompilation. If your DAG runs on ephemeral workers, warm that cache during image build or in an init step — the mechanics of carrying the cache across CI runners are covered in How to cache artifacts in CI. The --fail-on-fallback flag exits non-zero if any function declines native compilation, which turns a silent degradation into a build failure you can route to an on-call rota like any other.

Step three: execute the task through the Pyvorin runner

The task script stays ordinary Python; the runner applies the compiled path and verifies the answer:

python -m pyvorin run dags/transform.py --script-mode -- \
    --input s3://bucket/raw/2026-06-01.parquet

# With a measurement against CPython attached
python -m pyvorin run dags/transform.py --script-mode --compare

--script-mode runs the file as __main__ with sys.argv passed through, so existing task scripts need no changes beyond how they are launched. --compare additionally times the same entrypoint under CPython and prints a speedup figure with a correctness check — useful in a staging DAG where the number is still interesting. An illustrative Airflow task wiring this as a shell step looks like this:

# Illustrative only — adjust operator arguments to your Airflow version.
from airflow import DAG
from airflow.operators.bash import BashOperator

with DAG("pyvorin_etl") as dag:
    transform = BashOperator(
        task_id="transform",
        bash_command="python -m pyvorin run /opt/dags/transform.py --script-mode",
    )

We name the operator rather than reproduce a full DAG definition because the Airflow API surface shifts between releases and we cannot verify any specific version on the build host. The contract that matters is the command line, and that is fully verified.

Step four: keep the number

A speedup claim that lives only in a Slack message decays. Put the measurement in CI, next to the compile report:

python -m pyvorin bench dags/transform.py --function transform --runs 5 --json \
    > build/bench.json

The JSON report carries the status (COMPILED_FULL on success), a correctness verdict against the CPython ground truth, per-run timings and the fallback and deopt counts. Store it as a build artefact and diff it between releases; a speedup that drifts downward across deploys is an early warning that someone changed a hot loop in a way the compiler no longer recognises. Judgement call: record the bench JSON on every merge, but gate the build only on correctness and compilation status. Timing gates belong on dedicated benchmark runners, not on shared CI agents where a noisy neighbour can fail an otherwise good change — and where a failure teaches the team to ignore the gate, which is worse than having none.

Set expectations from measured data

What should the number be. The canonical 71-workload benchmark suite reports a 3.16x geometric mean and a 1.35x median, with 54 workloads faster than CPython and 17 slower — the full distribution, published with the losses included. Within the suite, the workloads that resemble typical DAG transforms did well: etl.windowed_aggregate at 134.62x and financial.moving_average at 22.53x, while string- and compression-bound workloads sat at or below 1x. If your task spends its time inside database drivers, HTTP clients or C library calls, there is little interpreted bytecode left to remove and Pyvorin will report a number near 1x — that is the compiler being honest, and it is your signal to spend the effort elsewhere.

Failure modes a DAG will actually see

Three failure shapes are worth rehearsing. A compile-time decline produces a recorded fallback reason and, with --fail-on-fallback, a non-zero exit — the task fails loudly at build time, never silently at 3 a.m. A runtime guard failure diverts that call to a lazily compiled copy of the original Python and continues with correct semantics, recorded by the fallback logger; run and bench surface the fallback and deopt counts in their reports so you can see it happened. And an unlicensed or expired environment fails at the licence gate, not mid-task. In every case the failure is a normal process exit with a message, which is exactly what an Airflow task is already built to handle — retries, alerts and all.

Where to go next

Prefect integration — the same three-command pattern adapted to flow-based orchestration.
Dagster integration — running compiled steps as software-defined assets.
How to cache artifacts in CI — carrying the compile cache across ephemeral workers.
Benchmarks — the measured distribution behind the expectations above.

Last reviewed 1 June 2026 against Pyvorin Native 1.0.9 (installed package at /root/pvfinal). Every command on this page was run locally and its output captured; compile timings are from the build host and will vary on other hardware.