Dagster Integration — Assets Backed by Native Code

Dagster's asset model asks a natural question of a compiler like Pyvorin: can the computation behind a software-defined asset be native code, materialised on the normal cadence, with the lineage and freshness guarantees intact. With Pyvorin Native 1.0.9 the answer is yes, and the integration is four shell commands long — support, compile, run and bench --json — because the compiled artefact lives on disk and needs no service beside it.

This page gives the asset-oriented version of the pattern used across this series: Apache Airflow integration schedules the same commands as DAG tasks, and Prefect integration launches them from flow tasks. Here the judgement calls are about materialisation — when to recompile relative to when an asset recomputes.

No plugin required

Pyvorin compiles locally and in-process; the compilation pipeline contains no network code, and the resulting shared library is loaded by the same CPython process that runs your asset's code. Dagster needs nothing from Pyvorin and Pyvorin needs nothing from Dagster beyond the ability to launch a process and check its exit code. That keeps your asset definitions free of a vendor coupling: the compute module is ordinary Python, the asset body is a launcher, and either side can be replaced without touching the other. Lineage and freshness — the reasons teams pick an asset model in the first place — stay entirely in Dagster's hands, because from the orchestrator's point of view the launcher is just another materialisation of the same asset.

Step one: classify the compute module

python -m pyvorin support assets/transform.py

Against an illustrative module with two numeric functions, captured on the build host:

Support report for /tmp/loopdemo/forloop.py
Function                       Status               Unsupported
------------------------------------------------------------
trapezoid                      COMPILED_FULL        0
small_unroll                   COMPILED_FULL        0

The report is the compiler's own compatibility gate, run early. Functions below COMPILED_FULL are not broken — partially supported functions execute their unsupported islands through the recorded CPython fallback, as detailed in Unsupported code and the fallback path — but an asset you believe is native while it is mostly fallback is an operational lie, and this command exists to prevent it. Run it in CI on every change to the compute module.

Step two: compile when the code ships

python -m pyvorin compile assets/transform.py --fail-on-fallback
python -m pyvorin compile assets/transform.py --json --report build/compile.json

A single-function compile takes about a quarter of a second on the build host, LLVM lowering plus linking against the runtime libraries included. The artefact is written to the on-disk compile cache — inside the installation tree, keyed by a hash of source, function name, options and runtime timestamps, with per-entry checksums and a 500-entry LRU bound. The natural place for this step is the same pipeline stage that publishes your code package: code and artefact then advance together, and an asset that materialises on Tuesday is running the compiler's output from Monday's deploy, not recompiling ad hoc. For ephemeral executors, warm the cache in the image build; the cross-runner mechanics are covered in How to cache artifacts in CI. --fail-on-fallback converts a declined function into a build failure, which is where you want to learn about it.

Step three: materialise through the runner

The asset body launches the compute module through the Pyvorin CLI. Dagster's designed integration point for external processes is its Pipes protocol; the exact launcher API shifts between releases and we cannot verify a specific one on this host, so the snippet below uses a plain subprocess and names Pipes as the place to attach if you want structured metadata back:

# Illustrative only — check your Dagster version's asset and Pipes APIs.
import subprocess
from dagster import asset

@asset
def enriched_orders() -> None:
    subprocess.run(
        ["python", "-m", "pyvorin", "run",
         "assets/transform.py", "--script-mode"],
        check=True,
    )

Before wiring the asset into a schedule, run the same command by hand. The runner applies the compiled path, checks the output against CPython, and prints the status — so the first materialisation you see in the UI is one you already saw succeed locally. The command line is the verified part:

python -m pyvorin run assets/transform.py --script-mode
# Staging: attach a CPython comparison and timing
python -m pyvorin run assets/transform.py --script-mode --compare

--script-mode runs the module as __main__ with sys.argv passed through, so the compute script keeps its existing interface. Data still flows through your normal IO layer — warehouse tables, object storage, files — which keeps Dagster's dependency graph honest: the asset's upstream dependencies remain the tables it reads, not the compiler. That honesty has a practical payoff: an asset that slows down after a code change shows up in the freshness and runtime metrics you already watch, and the bench artefact from step four tells you whether the change or the data is responsible. Retry behaviour is the standard kind: a failed materialisation reruns the command, and a warm compile cache makes the rerun as cheap as a plain Python one.

Step four: keep evidence with the deploy

python -m pyvorin bench assets/transform.py --function transform --runs 5 --json \
    > build/bench.json

The JSON report records compilation status, a correctness verdict against the CPython ground truth, per-run timings and the fallback and deopt counts. Treat it like a test artefact: stored per deploy, diffed between deploys. The asset-specific judgement is about partitioning. If an asset is partitioned by day and recomputed on backfill, recompilation belongs to code changes, not to partitions — a hundred-partition backfill should hit the cache a hundred times and compile zero times, because the cache key covers the source hash and the source has not changed. If you find an executor recompiling per partition, the cache is not being carried into the execution environment, and that is a packaging fix, not a reason to abandon the compile step.

Expectations, measured

The canonical 71-workload suite reports a 3.16x geometric mean and a 1.35x median, with 54 workloads faster than CPython and 17 slower, published with the slow ones included. As with the other orchestrators in this series, the right response to a near-1x measurement is to leave the asset on plain Python and direct the optimisation budget where the compiler has bytecode to remove; forcing a number out of an IO-bound asset wastes both compute and credibility.

Asset-shaped workloads appear at both ends of the distribution: etl.windowed_aggregate measured 134.62x and financial.moving_average 22.53x, while compression- and string-bound workloads measured at or below 1x. An asset dominated by warehouse round-trips lands near 1x because there is little interpreted bytecode to remove; spend the effort where the measured distribution says the wins live.

Failure modes

All failures surface as ordinary process exits, which asset sensors and retries already understand. There is no new failure class to page on: the launcher either exits zero with a correct result or exits non-zero with a message, and an asset that raises simply fails to materialise, on schedule, like any other. A compile-time decline exits non-zero with the recorded reason when --fail-on-fallback is set. A runtime guard failure diverts the affected call to a lazily compiled copy of the original Python, preserving semantics and incrementing the fallback and deopt counts that run and bench report. Licensing problems stop at the licence gate before your compute starts. And because the call bridge validates its assumptions on every entry, a native artefact never returns a wrong answer silently — it declines, records, and continues as Python.

Where to go next

Apache Airflow integration — the same commands behind DAG tasks.
Prefect integration — the same commands behind flow tasks.
How to cache artifacts in CI — keeping the compile cache warm across executors.
Benchmarks — the full measured distribution.

Last reviewed 5 June 2026 against Pyvorin Native 1.0.9 (installed package at /root/pvfinal). Every command was executed locally and its output captured; compile timings are build-host figures and will vary elsewhere.