High-Frequency Data Feeds

A high-frequency feed pipeline has two different kinds of work in it, and conflating them is the most common way this evaluation goes wrong. The first kind moves bytes: receiving packets, decoding frames, sequencing messages and publishing them onward. The second kind computes over values: normalising fields, aggregating intervals, reconciling symbols, deriving features. Pyvorin Native 1.0.9 accelerates the second kind — the Python that loops over feed data once it has arrived — and this page is about placing that acceleration correctly. The hot wire path stays in the feed handler's native stack. The batch and normalisation transforms behind it are where the measured wins live.

The boundary that decides everything

Compiled Python is not a low-latency wire path, and no benchmark justifies pretending otherwise. Byte handling on a receive loop — socket reads, framing, incremental decoding — is either bound by the network and kernel or by C library routines that CPython already calls directly; there is no interpreted loop for a compiler to remove. Pyvorin's own suite confirms the shape: web request handling, the closest published analogue to per-message wire work, averaged 0.97x across five workloads with a worst case of 0.38x. If your requirement is defined in fractions of a millisecond on the receive path, the answer is a native handler written in its own ecosystem, and Pyvorin is not a candidate for that component.

Behind the handler, the picture changes. Once messages are buffered into batches, intervals or windows, the work becomes exactly what a native-code compiler is built for: interpreted Python loops over numeric data. That is the ground to take, and the suite has direct measurements of it. The rest of this page is about which transforms qualify, which do not, and how to keep the two kinds of work from silently trading places under schedule pressure.

What compiles well on feed data

The etl category averages 4.62x, but the spread inside it is the real guidance. The windowed aggregate — computing interval statistics over a stream of records — is the canonical feed-transform win at 134.62x. Top-N selection, the shape of a most-active-symbols or best-quote computation, lands at 7.81x. Then the category's own regression: a merge-join at 0.64x, where the cost sits in allocation and comparison logic the compiler cannot improve. Feed transformation is not automatically acceleratable; the loop has to dominate, and the support and bench commands below exist precisely to answer that per function.

What does not compile well

The honest list. String-shaped transforms — symbol normalisation implemented as string surgery, message formatting, textual field scrubbing — map to the suite's weakest categories: string manipulation averaged 0.85x and parsing 0.86x, with individual workloads as low as 0.71x. Per-message handlers map to the web category's worst rows, down to 0.38x, and the suite's absolute floor of 0.05x belongs to micro-workloads where the compiled path's fixed overheads own the runtime — the per-message scale, in other words. If the transform touches every message individually and does little per message, batch first or leave it out of scope. Pyvorin accelerates computation over accumulated data, and it is worth the licence money only where that computation is substantial enough to measure.

A reference arrangement

The deployment shape that survives contact with production separates the pipeline at the buffer:

feed handler (native stack)  ->  ring buffer / batch interval
    -> normalise.py  (Pyvorin-compiled: field mapping, unit conversion)
    -> aggregate.py  (Pyvorin-compiled: interval stats, top-N, distinct)
    -> store / publish

The handler — in whatever language it is already written — owns sequencing, gap detection and onward publication, and keeps its latency characteristics untouched. At each flush interval it hands the batch to compiled Python functions that normalise and aggregate. Those functions were compiled at build time; their artefacts ship in the service image, and nothing compiles on the data path. A function the compiler cannot take runs as its original Python and is recorded, so a regression in coverage degrades a transform to interpreter speed rather than failing the batch — the fallback semantics documented on the unsupported code and fallback page.

The reference arrangement also answers the deployment question people ask next: where the Python stage runs. The compiled transforms are ordinary functions in an ordinary Python service, deployed like any other — behind the buffer, scaled against batch volume rather than message rate. That is the shape covered in the microservices implementation page: compile at build time, ship the warm cache in the image, gate the build on --fail-on-fallback. What you should not do is reintroduce the transport into the Python stage one convenience at a time — a socket read here, a decode there — because each one drags the stage back towards the categories that lose.

Measure the transforms, not the feed

The evaluation unit is the individual transform, benchmarked on production-shaped batches:

# Which functions compile fully, which fall back
python -m pyvorin support transforms/aggregate.py

# Release gate: fail if anything falls back silently
python -m pyvorin compile transforms/aggregate.py --fail-on-fallback

# Measured ratio on batches that match production size
python -m pyvorin run transforms/aggregate.py --function interval_stats --compare --runs 5 --json

Two measurement disciplines matter here. Batch size first: a transform benchmarked on 100,000-record batches will flatter itself relative to the 2,000-record batches production actually hands it, and the speed proof should use the real number. Correctness second: --compare checks native output against CPython on the same inputs, which is the property that matters when the downstream consumer is a pricing or risk system. The full procedure, including the CI gate for the compiled cache, is on the how to run a speed proof page and the CI artefact caching page. For the deployment shape around the transforms, see implementing Pyvorin in microservices.

One trap in this measurement: the transform you benchmark is not always the transform that runs. If the production code path applies the transform per message and your benchmark batches the inputs, you have measured an architecture you have not built. Match the call pattern as well as the data shape, and if the per-message pattern benchmarks poorly, that is a finding about the architecture — batch first, then compile — not a failure of the measurement.

The judgement call: where to draw the line

In practice the line gets drawn under schedule pressure, and it drifts. Our rule, applied and re-applied: a transform belongs on the compiled side only if it loops over a batch of arrived records and its profile shows interpreted Python dominating; everything that touches per-message bytes, strings or library calls stays in the handler or upstream. When a transform straddles the line — parsing a text field, then aggregating the result — split it at the parse boundary, put the parsing where the parsing categories say it belongs, and compile only the arithmetic half. Teams that draw the line once, at design time, end up re-drawing it after the first honest benchmark; teams that draw it per function, with measurements, rarely move it. The suite's own spread — 134.62x against 0.64x inside a single category — is the argument for deciding function by function rather than pipeline by pipeline.

Where to go next

• How to run a speed proof — measure each transform on production-shaped batches.
• Benchmarks — the full canonical table, winning and losing rows.
• Unsupported code and the fallback path — runtime behaviour when a transform declines to compile.
• Implementing Pyvorin in microservices — build-time compilation and artefact shipping.

Last reviewed 22 June 2026 against pyvorin-native 1.0.9 installed at /root/pvfinal (verified build of 12 September 2026). All workload figures are from the canonical 2026-09-13 benchmark artefact with memory caps active. No per-message throughput, wire-path or per-event latency claims are made on this page; the wire path is explicitly out of scope.