Example Workload: ETL Pipeline

ETL pipelines are the most common shape of Python compute in business: read files, clean rows, transform, aggregate, write. They are also the workload where the gap between "compiled" and "faster" is widest in both directions — the same suite that measures a windowed aggregate at 134.6x measures a merge join at 0.6x. An ETL pipeline handled honestly is therefore a segmentation exercise before it is anything else. This page works through one illustrative pipeline, names the measured anchor for every stage, and shows where the real decision sits.

Labels, as ever: this is an example workload, not a customer story. Every speedup below names a workload in the benchmark artefact dated 13 September 2026, extracted from that table rather than typed from memory, and any arithmetic that combines them is labelled an illustrative model with its assumptions stated in full.

The workload shape

Take a morning pipeline typical of operations teams. It reads a directory of export files, parses each row into a typed record, validates fields against rules, joins records against a slowly-changing reference table, applies per-customer transforms, computes windowed aggregates per product line, and writes two output files: a clean dataset and a summary table. Five layers: I/O, parsing, joining, transforming, aggregating.

Map those layers onto what a compiler can see. I/O is fixed cost — the disk and the network do not care about your bytecode. Parsing is string work. Joining against a reference table is dictionary lookup plus, if implemented naively, a nested loop that walks the reference table per record. Transforming and aggregating are the pure-Python numeric loops. Only the last two layers are candidates, and the join sits in a grey zone that the measurements below resolve.

Measured anchors per stage

The ETL category as a whole spans 0.6x to 134.6x with a geomean of 4.6x — the widest range of any category in the suite. That spread is not noise. It is the category telling you that "ETL" is not one workload. Windowed aggregates run long numeric loops over materialised rows; merge joins chase pointers through sorted structures and allocate constantly. One shape is a compiler's home ground; the other is a memory-system benchmark. The honest way to quote this category is per workload, exactly as the table does, and any page that quotes "ETL is 4.6x" without the range is quoting a geomean as if it were a result.

The join row deserves a second look, because joins are where pipeline authors lose the most time to false expectations. A hash join backed by dictionary lookups spends its time inside CPython's C-implemented dict code — never slow. A compiled nested-loop join can be slower than either. If your reference table fits in memory, the fix for join performance is algorithmic (hash it once, not the compiler), and the measurements back that instinct.

An illustrative model

Assume profiling shows 15% of runtime in parsing, 25% in the join and lookups, 50% in transforms and aggregates, 10% in I/O and serialisation. Assume — labelled, not measured — 25x on the transform and aggregate share, between the measured 7.8x and 134.6x anchors. The model: 0.15 + 0.25 + 0.50/25 + 0.10 = 0.52 of the old runtime, about 1.9x end to end. Modest. Deliberately so — that is what honest middle-of-range assumptions produce, and it is still the difference between a pipeline that blocks the morning and one that finishes before the first stand-up.

The model's sensitivity to the join share is the real lesson. Push the join to 45% and the same assumptions yield 1.4x. If your pipeline is join-dominated, no compiler on this evidence is the lever; a better join algorithm or a real database is. A compiler does not fix a data-structure problem, and pretending it does wastes the quarter.

What a pilot run looks like

Before committing the pipeline to a compiled module, run a contained pilot. Take one representative input file, extract the single heaviest aggregate function into a standalone module, and run the full proof sequence on it. A pilot of that shape takes an afternoon including the write-up, and it answers the only question that matters: whether your transform loop, on your data shapes, measures anywhere near the published anchors. The honest outcomes are three. The number lands near the anchors and the pilot proceeds. The number lands near 1x, in which case your transform is probably hash- or library-bound and the pipeline needs a different fix. Or the function is declined, in which case the fallback path has already told you why, and the answer was free.

Measure the pilot against the production input distribution, not a toy file. Windowed aggregates over a thousand rows behave nothing like the same code over fifty million; the suite's own micro-workloads, where compilation overhead dominates and results fall to 0.05x, exist precisely to warn against measuring small. If a full production file is unwieldy for iteration, use a fixed random sample and record its size in the write-up, so the proof subject is reproducible.

Where the engineering judgement sits

The decision that determines the outcome here is where to place the seams. Parse once into tuples, drop the parser. Pre-hash the reference table before the loop, drop the join from the compiled module. Keep the compiled surface to pure transforms and aggregates over plain in-memory structures, with no string methods and no file handles inside. An engineer who points the compiler at the whole 600-line pipeline will measure a blended number that tells them nothing; the one who draws the seams first gets a number they can act on — and a compiled module small enough that the fallback path, should any construct be declined, is isolated and visible rather than smeared across the run.

There is also a cadence judgement. Compilation costs time once, then amortises. The speed-proof page puts the captured cold-compile cost at roughly 240 ms for a small function, with a disk cache — keyed by source hash — making repeat compiles report 0.000 ms. A pipeline that runs once a day on a cold process pays that cost once per day, trivially. A pipeline invoked as dozens of short-lived processes, each compiling the same functions, should keep the cache warm or the compile cost will eat the win on small inputs. Match the deployment shape to the amortisation, not the other way round.

Proving it on your own pipeline

Extract the windowed aggregate from your pipeline into its own module with a zero-argument entrypoint, then measure both backends in one invocation:

python -m pyvorin support your_transforms.py
python -m pyvorin run your_transforms.py --function entry --compare --runs 5 --warmup 2
python -m pyvorin bench your_transforms.py --function entry --runs 7 --warmup 2 --json

Trust only output showing Correct: YES, COMPILED_FULL on the target, and matched warmup and run counts on both sides. Report the median of the raw samples, keep the compile-time line visible, and do the break-even division in the write-up. The full method, including how to make the result survive an audit, is how to run a speed proof; harness details are in how to benchmark a function. Neither takes an afternoon, and both apply unchanged to a pipeline stage — the cost of skipping them only appears later, quoted in a meeting as if it were a measurement.

Limits, stated plainly

One host, one date, one suite: the anchors are a map, your measurement is the territory. Parsing and joins are real regressions in the table, not edge cases — compiling them slows the pipeline, and the fallback path exists precisely so a declined or guard-failed function keeps correct CPython behaviour, as covered in unsupported code and the fallback path. And if the pipeline is I/O- or database-bound, every figure on this page is beside the point: fix the data movement first.

Where to go next

• How to run a speed proof — the measurement discipline behind every number quoted here.
• Benchmarks — the full 71-workload table with all seventeen slower-than-CPython rows intact.
• Example workload: log analytics — the same segmentation logic on a sibling pipeline shape.
• Unsupported code and the fallback path — what "declined" means in practice and why it is safe.

Last reviewed 12 May 2026 against the benchmark artefact dated 13 September 2026 (71 workloads, 54 faster / 17 slower than CPython, ETL category range 0.6x–134.6x). Anchor figures are extracted from that artefact; end-to-end arithmetic is a labelled illustrative model. This page describes an example workload; it is not a customer case study.