Example Workload: Log Analytics

A log-analytics pipeline is a compilation problem wearing an operations costume. The pipeline reads a day's worth of application logs, parses each line, filters, enriches and aggregates, and writes summary tables. Some of those stages are exactly the shape a native-code compiler removes; others are string work that CPython already executes inside tight C loops. Telling the two apart before you spend a week on integration is the whole exercise.

This page is an example workload, not a customer story. No customer is quoted and no deployment is claimed. The speedup figures cited below are real measurements from Pyvorin's benchmark artefact dated 13 September 2026 — 71 workloads run on one host, each named so you can check it against the published table — and the pipeline arithmetic is labelled as an illustrative model wherever a number is modelled rather than measured.

The workload shape

Consider a representative pipeline: four gigabytes of combined access and application logs land in a directory each night. The pipeline must extract a timestamp, severity, service name and latency field from every line, drop lines that match a set of known-noise patterns, bucket the survivors into five-minute windows per service, and compute per-window counts, latency percentiles and error ratios. Output is a set of aggregate tables a dashboard reads the next morning.

Split that mentally into three layers. The I/O layer reads files from disk and decompresses them; its cost is fixed by your storage and nothing on this page changes it. The parsing layer turns raw bytes into structured fields; it is dominated by string scanning and regex. The aggregation layer takes structured records and computes windows, counts and statistics over them; it is dominated by numeric loops over in-memory data. A compiler can only attack the third layer, and it attacks it only when the loop is written in acceleratable pure Python.

That last qualifier matters more than it sounds. A percentile computed with the standard library's statistics module calls C code; CPython was never slow there. A percentile computed in a hand-rolled Python loop over a window of samples is interpreted bytecode dispatch — precisely what disappears under compilation.

What accelerates, and what does not

The benchmark table contains direct evidence for both halves of this pipeline. Start with the parsing layer, because it is where expectations most often go wrong. The suite's three parsing workloads ran at 0.7x to 1.0x — slower than stock CPython. The closest to our pipeline is parsing.log_line_parse, which measured 0.9x. String work tells the same story: string.log_parser measured 0.7x, and the string category's geometric mean is 0.85x. The suite's own analysis of these categories is that the time goes to allocation, reference counting and C library routines, leaving little interpreted-loop work for a compiler to remove. Expect no win on log parsing. Expect a small loss if you compile it, and do not waste effort trying to fix that with flags.

The aggregation layer is the opposite. statistics.correlation — a pure-Python loop over two numeric series — measured 52.5x. etl.windowed_aggregate, the closest published analogue to per-service windowed counting, measured 134.6x. etl.top_n_selection, relevant to "slowest endpoints" tables, measured 7.8x. Even the honest middle of the distribution is useful rather than magical: the median across all 71 workloads is 1.35x, and the suite geomean is 3.16x. Those numbers describe the map, not your result; your pipeline's position on it depends on what fraction of its runtime is acceleratable loop work.

Read the two slow rows as carefully as the fast ones. They are not defects; they are the product boundary. A compiler that promises to speed up string scanning is promising something the measurements contradict, and any article that hides those rows is selling you a regression.

An illustrative model of the end-to-end effect

Suppose profiling your nightly run shows the split typical of this shape: 25% of wall-clock time in parsing, 65% in aggregation loops, 10% in I/O and serialisation. Assume — and this is an assumption, labelled as such — that compilation gives you 20x on the aggregation share. That is not a measured figure; it is a deliberately moderate point inside the measured range of 7.8x to 134.6x for the aggregation-like workloads above. The model then says: new runtime is 0.25 + 0.65/20 + 0.10 = 0.3825 of the old, or about 2.6x end to end.

Notice what the model refuses to do. It does not quote 134.6x as "the speedup of log analytics". It applies a measured range to one stage, with the stage's share measured by you, on your machine. The single most valuable hour in this whole exercise is the hour spent with a profiler finding out what your actual percentages are. A pipeline that spends 80% of its time in gzip decompression and regex will model out near 1.0x no matter how good the compiler is on loops.

Where the engineering judgement sits

The decision that actually determines the outcome is not technical. It is the decision to restructure the pipeline so that parsing and aggregation are separable. That means parsing lines into tuples once, handing a structured record list to a pure-Python aggregation module with no I/O and no string manipulation inside it, and compiling that module alone. The engineer who skips this separation and points the compiler at a 400-line file where parsing and aggregation are interleaved will measure a muddled 1.1x and conclude the product does not work. The engineer who separates the stages first will measure something interpretable, and the number will mean something.

This separation has a second virtue: the fallback path. Where Pyvorin declines to compile a function — an unsupported construct, a guard failure at runtime — the router runs honest CPython fallback for that function and records the event, as covered in unsupported code and the fallback path. Keeping the compiled surface small and pure makes fallback events rare and diagnosable rather than silent.

Proving it on your own pipeline

Do not take the model or the table as your number. Take a real aggregate function from your pipeline — the windowed counter, say — wrap it in a zero-argument entrypoint the way the speed-proof method requires, and measure both backends in one invocation:

python -m pyvorin support your_aggregation.py
python -m pyvorin run your_aggregation.py --function entry --compare --runs 5 --warmup 2
python -m pyvorin bench your_aggregation.py --function entry --runs 7 --warmup 2 --json

The --compare run gives you the CPython baseline, the Pyvorin time, a correctness check that both backends returned the identical result, and the compilation status of the function — you want COMPILED_FULL, because a compatibility-executed "after" number is proof of nothing. The --json bench run gives you the raw samples to report a median instead of an anecdote. The full discipline — warmup, median-of-N, compile-cost amortisation, the write-up that survives an audit — is in how to run a speed proof, and the everyday harness mechanics are in how to benchmark a function.

Limits, stated plainly

Three caveats belong on this page in print rather than in a footnote. First, every figure in the anchor table comes from one host on one date; the shape of the distribution transfers better than any absolute number, which is why your own measurement outranks ours. Second, the parsing slowdown is real and worth real money if you ignore it: compiling a stage that measures 0.7x slows your pipeline, and the fix is to not compile that stage. Third, throughput is not latency. This pipeline is batch; if any part of your log handling is an inline request path where microseconds matter, compiled batch Python is not a low-latency substitute for a dedicated compiled service, whatever the throughput figure says.

None of this is an argument against the approach. It is the argument for applying it where the measurements point and refusing it where they do not.

Where to go next

• How to run a speed proof — the measurement discipline for turning a claim into an auditable number.
• Benchmarks — the full 71-workload table these anchors are drawn from, including every slower-than-CPython row.
• Example workload: ETL pipeline — the same analysis applied to a batch transformation pipeline.
• Unsupported code and the fallback path — what happens when the compiler declines, and why it is safe.

Last reviewed 6 May 2026 against the benchmark artefact dated 13 September 2026 (71 workloads, 54 faster / 17 slower than CPython, suite geomean 3.16x). Figures cited are extracted from that artefact; the end-to-end arithmetic is a labelled illustrative model, not a measurement. This page describes an example workload; it is not a customer case study.