Making Python ETL Pipelines Faster, Measured

An ETL pipeline is not one workload. It is a chain of transforms with different physics — row-window mathematics at one station, string parsing at the next, a merge or join after that — and no single number describes all of them. Measured per transform on our 71-workload suite, Pyvorin Native 1.0.9 accelerated a windowed aggregation by 134.62x while a merge-join workload ran at 0.64x, slower than the interpreter. Both results are real, both are reproducible, and the difference between them tells you exactly where compilation belongs in a pipeline and where it does not. This page walks the measured ETL results transform by transform, shows a worked before-and-after, and makes the pipeline-boundary judgement explicit.

What the ETL category measured

Five workloads in the suite model common transform shapes. Each is a self-contained pure-Python program run under stock CPython and under Pyvorin on the same host, with the compiled output verified against the interpreter's result before any timing was trusted.

The category geomean across these five is 4.62x. Read the spread, not the average: the same compiler, on the same day, on the same host, produced its best result in the entire suite and a forty-per-cent regression within one category. That spread is the page.

Why the windowed aggregate wins

The 134.62x result is the best in the ETL category and among the strongest in the suite, and its shape explains it. A windowed aggregation — a rolling mean, a moving maximum, a cumulative figure recomputed over a fixed span of ordered rows — is a pure arithmetic loop over contiguous data. Every row does the same float operations; no strings are parsed, no objects are built, no library is called. It pays the interpreter's full tax rate: dispatch on every operation, boxing on every number, reference counting on every intermediate. And it is exactly the code a compiler can prove things about — fixed-width float windows, stable loop bounds, local accumulators — so escape analysis keeps values off the heap, bounds-check elimination removes per-index guards, and the loop vectorises. The workload that CPython ran in 470.61 ms completes in 3.50 ms compiled. The physics are identical to the suite's best result overall, 202.83x on a trapezoidal integral, and to the numerical category's 10.91x geomean: tight, typed, pure-Python loops are where compilation takes its ground.

The function behind the winning row has this shape — a rolling mean over a numeric series, nothing but a loop and two accumulators:

def rolling_mean(values, window):
    out = []
    running = 0.0
    n = len(values)
    w = min(window, n)
    for i in range(n):
        running += values[i]
        if i >= w:
            running -= values[i - w]
        out.append(running / w)
    return out

Every property the compiler exploits is visible in those ten lines: float arithmetic on a contiguous list, a single induction variable, an accumulator that never escapes, and no calls. Type inference proves the numeric types; escape analysis keeps the accumulators in registers; the loop vectorises. Ten lines of ordinary Python, no annotations, no foreign syntax — that is the whole integration surface for a transform of this kind.

Transport, serialisation and the pipeline edges

Real pipelines have edges as well as transforms, and the measured edges sit exactly where the cost model says they should. Serialisation — turning records into JSON or back at the boundary — measured a 1.94x category geomean, respectable but modest, because half its time is library calls. Web-shaped request handling at the ingestion edge measured 0.97x, a wash, and generic data-processing glue measured a 1.0x geomean, in effect a wash. The pattern holds at every seam of the pipeline: the closer the code runs to I/O and library calls, the less interpreted loop exists to reclaim, and the closer the measured result drifts towards parity from below. This is why the pipeline-boundary judgement below treats edges and transforms as different species, and why the support report — which marks each function's compile eligibility before you benchmark — is the first artefact to read rather than the last.

Why the merge join loses

The 0.64x result is just as instructive. A merge join spends its time comparing record keys — strings, typically — and shuttling rows between data structures. String comparison in CPython already runs at C speed; the remaining Python-level costs are the ones compilation handles worst: allocating and discarding small objects, updating reference counts, and dispatching through generic comparison machinery that the compiled path must still respect. There is little interpreted-loop work to reclaim and real guard-and-boxing overhead to pay, so the compiled path finishes in 12.49 ms against CPython's 7.96 ms. The suite's neighbouring categories tell the same story from both sides: string manipulation at a 0.85x geomean, parsing at 0.86x. Joins, parses and format conversions are string-and-object work; windowed mathematics is loop work. The compiler's advantage is a property of the transform, not the pipeline.

The middle rows are the honest middle. Distinct counting at 2.86x and top-N selection at 7.81x gain because their inner loops are comparably typed; group-by at 1.09x is a wash because its time goes to dict machinery that both paths share. Nobody should promise you a single ETL speedup. Anyone who does is averaging these rows into a lie.

The group-by row deserves one more look, because it is where most real pipelines live. At 1.09x it is functionally unchanged by compilation — the dict that drives the grouping dominates both paths equally, and shared data-structure costs are the least acceleratable code in the suite. The practical reading: transforms built around Python dicts and their hashing behave like group-by, whatever their domain name, and transforms built around float arithmetic over ordered data behave like the windowed aggregate. Naming the transform by its data structure is more predictive than naming it by its business purpose.

A worked before-and-after

Take the shape behind the best row and put it in a pipeline. A nightly job ingests a day of sensor readings and runs a rolling thirty-minute mean over the day's series — the windowed-aggregation shape, pure Python over numeric arrays — and suppose, as our labelled assumption, that this stage runs ten thousand times a night across the partitioned dataset. On the measured figures, each invocation occupies 470.61 ms under CPython and 3.50 ms compiled: the stage that ran for roughly 78 minutes a night occupies roughly 35 seconds — 134.62x on the stage, on the same host, with output verified identical. The ten-thousand-invocation count is the assumption to replace with your own; the per-invocation timings are measured.

Now the judgement that actually matters. The stages upstream — reading the files, splitting lines, decoding timestamps — are parsing-shaped, and the measured parsing category says such code sits at 0.86x, slightly slower compiled. So the compiled pipeline does not compile the ingestion stage. It keeps the parser on the interpreter, hands parallel numeric arrays to the compiled rolling mean, and takes the results back. The seam between stages is where the engineering lives: the benchmark command below tells you per function which side of the seam each one lands on, and the suite numbers tell you what to expect from each side before you run it.

Measure your own transforms

Every figure above came from the standard benchmark command, run locally. The same command works on your pipeline's stages:

# See which functions in a stage compile, and which fall back
python -m pyvorin support transform.py

# Benchmark a stage's entrypoint on production-like data
python -m pyvorin bench transform.py --function entry --runs 5 --warmup 2

# Machine-readable record for CI
python -m pyvorin bench transform.py --function entry --json

Run it per stage, not per pipeline. A pipeline-level measurement averages a 134.62x stage with a 0.64x stage and produces a number that describes neither — the same mistake as averaging the suite. Stage-level numbers, on your data and your hardware, are the only inputs worth feeding a design decision. The measurement discipline, including warm-up, run counts and the correctness check, is covered in How to benchmark a function.

The pipeline-boundary judgement

Splitting at the seam is a decision an engineer makes with data, not a setting. The workflow: profile the pipeline, identify the stages whose time is pure-Python arithmetic loops — the windowed aggregates, the per-row scoring, the statistical passes — and benchmark each candidate stage alone. Expect parsing, serialisation and join stages to measure at or below parity, and leave them on the interpreter without regret; a 0.64x stage included in the compiled scope makes the pipeline slower, and there is no configuration that fixes a wrong boundary. The fallback path keeps the mixed program semantically identical to CPython throughout, so the seam can move stage by stage as measurements land. Teams that get the most from compilation are not the ones that compile the most code; they are the ones that measured first and compiled the right half.

Where to go next

• Benchmarks — the full 71-workload table, including every category cited above.
• How to benchmark a function — the measurement discipline for your own stages.
• Compiler pipeline overview — why window loops vectorise and join loops do not.
• Why Python is slow — and what actually helps — the cost model behind the spread on this page.

Last reviewed 7 September 2026 against Pyvorin Native 1.0.9 and the canonical benchmark artefact dated 13 September 2026. Every timing and speedup on this page is extracted from the artefact's ETL, string and parsing rows, not typed by hand; the worked example is arithmetic on those measured figures, with its nightly-invocation count stated inline as a labelled assumption.