Skip to content

TurbineTurbine

High throughput, low drama

Inference that is simpler and uses 2× to 8× less CPU than Flink, RisingWave, FastStream or Quix.

Turbine runs your Python code on streams of events — payments, clicks, sensor readings, logs — as they happen, and does everything around it in Rust. It reads them from Kafka, remembers what it has seen, and publishes results continuously, on a cluster of machines that carries on when one of them fails.

What Turbine does

A stream has no end: events keep arriving, so a stream processor is a program that never finishes. It reads each event as it is produced, does something with it, and publishes the result a moment later. The diagram follows one such pipeline — flagging suspicious card payments — through the five things Turbine does for you. The last two are the point: what Turbine computes is used where it is computed, scored by your model and read by your application, with no export step in between.

inside node 1 Kafka topic “payments” 0 1 2 3 partitions; node 1 reads partition 0 your Python function process(batch, state) called on each batch read · write state, on local disk spend per card · 10 min ML model fraud score · ONNX scores every result Kafka topic “fraud-scores” for downstream systems your app, a dashboard “card 42: spend so far?” HTTP API · FastAPI · console asked live, no database in between Turbine cluster node 1 partition 0 node 2 partitions 1 + 2 node 3 down moves S3 bucket snapshots of the state snapshot snapshot restore data, as it flows an API call, on demand snapshots, failover 1 2 3 4 5
A fraud-detection pipeline, as an example. Top: node 1, enlarged — it reads its share of the payments, partition 0, in batches; your function keeps a running spend per card, a model scores each result on its way out, and an app reads one card's spend from the state over the API, live, and charts it. Bottom: the cluster around it — the other nodes read the other partitions, and every node snapshots its state; when node 3 goes down, node 2 takes over its partition and restores the state from the snapshot. Click a box to jump to its section, anywhere else to enlarge.

1. Read streams, even at high volume

inside node 1 Kafka topic “payments” 0 1 2 3 partitions; node 1 reads partition 0 read · decode into batches in Rust a batch your Python function called on each batch results encode · publish back to Kafka in Rust Kafka topic “fraud-scores” for downstream systems exactly once, if you turn it on: each batch is one Kafka transaction its results and its read position commit together, or not at all — a crash never makes a result appear twice 1
Step 1, enlarged. Rust reads the events, decodes them into batches, then encodes and publishes what your function returns: your code sees batches, never Kafka. With exactly-once on, each batch is one Kafka transaction. Click to enlarge.

The events come from a Kafka topic (Kafka or a compatible broker such as Redpanda), split into partitions that can be read in parallel. Turbine reads them continuously and calls your function on batches of events, never one at a time: reading, decoding and publishing run in Rust, and your code stays plain Python. A single partition typically sustains hundreds of thousands of events per second, and adding partitions adds throughput — how much depends on what your function does. Turn on exactly-once for a subscription, and each batch becomes one Kafka transaction: its results and its read position commit together, so a crash never makes a result appear twice (see Delivery guarantees). See the Quick Start and Kafka & Partitioning.

2. Remember, across restarts

inside node 1 Kafka topic “payments” partition 0 your Python function called on each batch read · write state, on local disk next to your code: no network round trip key spend, last 10 min (a window) card 42 $1,240 $1,285 $1,262 card 17 $85 $112 $97 card 93 $430 $418 $455 2 snapshot, as the stream advances restore, after a lost disk or on the node taking over S3 bucket each snapshot: the state + its read position after a restore, reading resumes at the saved position
Step 2, enlarged. Each batch updates the state, on the node's own disk; snapshots carry it to S3 with the read position. A restore brings both back, and the events since the snapshot are read again. Click to enlarge.

Most useful questions about a stream need memory: how much has this card spent in the last ten minutes? Is this the first payment from that country? Your function gets a key-value state, kept on the node's local disk (RocksDB) right next to the code, so reading it costs no network round trip; windows compute aggregates over time (counts, sums, percentiles…) on top of it. The state is snapshotted to object storage (S3 or an S3-compatible store) as the stream advances: a node that takes over a partition, or that lost its disk, restores the state from there and resumes reading from the position saved with it. See State & Timers and Windowing; what happens to results published since the last snapshot is set by the delivery guarantee you choose.

3. Run as a cluster that survives a failure

Kafka topic “payments” 0 1 2 moves Turbine cluster the nodes agree (Raft) no coordinator to deploy node 1 partition 0 node 2 partition 1 partitions 1 + 2 node 3 partition 2 node 3 back node 3 down 3 partition 2 waits grace period, 60 s by default snapshots restore restore S3 bucket snapshots of the state
Step 3, enlarged. Every node runs the same app on its own partitions. When node 3 stays down past the grace period, node 2 takes partition 2 over, restoring its state from the latest snapshot; when node 3 is back, it gets partition 2 back the same way. Click to enlarge.

Start the same app on several machines, each given the same list of peers, and they form a cluster: the partitions are shared among the nodes, and each node runs the same code on its own share. When a node goes down and does not come back within a grace period (60 seconds by default), its partitions move to the surviving nodes, which restore their state from the latest snapshot and carry on. The nodes agree among themselves on who processes what (Raft): there is no coordinator to deploy. See Deployment and Kubernetes.

4. Score with a model, inside the pipeline

trained in PyTorch, scikit-learn, XGBoost or LightGBM exported to ONNX loaded at start inside node 1 your Python function computes the features, from windows results ML model fraud score · ONNX in the same process: no model server results + score card spend 10 min score 42 $1,240 $1,285 $1,262 0.88 0.93 0.91 17 $85 $112 $97 0.02 0.04 0.03 93 $430 $418 $455 0.07 0.06 0.09 Kafka topic “fraud-scores” for downstream systems 4
Step 4, enlarged. A model trained elsewhere and exported to ONNX runs inside the node: each batch of results is scored on its way out, and the score travels with it to the output topic. Click to enlarge.

Declare an ONNX model on a subscription and every result it publishes is scored on its way out, in the same process, with no call to a model server. Train in PyTorch, scikit-learn, XGBoost or LightGBM, then export to ONNX. The features can come from a window over the stream itself — the spend over the last ten minutes, the error rate over the last hour — so the features and the model live side by side, and a score exists as soon as its result does. See Inference.

The score travels with the result to the output topic; it does not come back into the state. To keep it there too — the last score per card, say — call the model from your function instead, and write the score like any other value: see Calling a model yourself.

5. Ask the state questions, live

Turbine cluster asks any node live, no database in between routed to node 1, which holds card 42 node 3 holds card 23 node 1 partition 0 state card 42 $1,300 $1,260 $1,420 $1,500 $1,440 $1,580 $1,380 $1,200 card 17 … card 93 … API HTTP · FastAPI node 2 partition 1 state card 8 … card 51 … card 64 … API HTTP · FastAPI node 3 partition 2 state card 23 $240 card 76 … card 30 … API HTTP · FastAPI your app, a dashboard “card 42: spend so far?” HTTP API · FastAPI · console 5 an LLM agent calls the API as a tool: “card 23: spend so far?”
Step 5, enlarged. Any node answers: node 2 routes the dashboard's question to node 1, which holds card 42, and node 3 answers the LLM agent itself. Both read the state live, as the stream updates it. Click to enlarge.

What Turbine remembers can be read while it runs. An HTTP API answers "what has card 42 spent?" from whichever node holds that card, the built-in web console browses the state and the open windows, and a FastAPI app can run in the same process and serve it from its own routes. The state does not need copying into a database before it can be served. See Console & API.

A first example

A Turbine handler receives a RecordBatch (Apache Arrow) and an optional TurbineState. The example below counts incoming messages per partition, persisting the running total in RocksDB:

from turbine import KafkaBroker, RecordBatch, Turbine, TurbineState

kafka = KafkaBroker(bootstrap="localhost:9092")
app = Turbine(kafka)

@app.subscribe(kafka.topic("events"), output=kafka.topic("counts"))
def aggregate(batch: RecordBatch, state: TurbineState) -> RecordBatch:
    total = state.get("total") or 0
    total += batch.num_rows
    state.put("total", total)
    return batch

app.run()

TurbineState exposes get(key), put(key, value), delete(key), get_bytes(key), put_bytes(key, value). State is checkpointed automatically and survives restarts.

Install

uv add --prerelease=allow turbine-fast

The distribution is turbine-fast, the import name is turbine, and it requires free-threaded CPython 3.14 (3.14t) on Linux x86_64 — that constraint is not incidental, it is what the design is built on. Everything is in the wheel: the engine, the state store, the REST API and the web console. See Installation for the interpreter, a local broker, and a smoke test; the project is in alpha, so pin the version you deploy.

Why Turbine

⚡ Performance — Rust handles the hot path (broker I/O, serde, state). Python never sees individual messages — only columnar batches via Apache Arrow zero-copy. Throughput scales with partitions: a single partition typically sustains several hundred thousand messages per second on a stateful pipeline with S3 checkpoints, and aggregate throughput grows roughly linearly as you add partitions. Real-world numbers depend on the cost of the user callback and the workload (decoding, aggregation, sink); the gap between a no-op pipeline and a complex one can be wide.

🐍 Free-threaded Python (3.14t) — Turbine targets the free-threaded build of CPython 3.14, so multiple partitions execute in true parallel within a single process — no GIL serialization between worker threads. PyArrow compute kernels and your user callbacks actually run on multiple cores, which is what makes the per-partition throughput compose as you add partitions instead of flattening on the GIL. Set TURBINE_ALLOW_GIL=1 to start with the GIL enabled and accept reduced Python parallelism. See Installation.

📦 Batch-level acknowledgment — Offsets are committed only after the entire batch has been processed, state flushed, and output produced. A crash mid-batch means the batch is replayed from the last committed offset, giving you at-least-once delivery with minimal duplication.

When to use Turbine

Turbine is not the right tool for every streaming workload. If you need request/reply over a broker, or brokers other than Kafka, a framework like FastStream is more appropriate.

Legend: ✅ best choice — 🟡 not optimal — ❌ bad choice.

Use case FastStream Turbine
High-volume aggregation / analytics ❌ Slow — Python serde per message ✅ Better — Arrow batches in Python, Rust on the hot path
Stateful processing (counters, windows, joins) ❌ No built-in state store; adding one through an external database is possible but at a significant performance cost ✅ Better — RocksDB per partition + checkpoints
Exactly-once delivery (no output duplicates across crashes) ❌ Not a framework primitive — handlers must be idempotent, or you wire aiokafka transactions by hand ✅ Better — opt-in per subscribe via processing_guarantee="exactly_once", configurable crash-recovery policy
Automatic high availability 🟡 Kafka consumer groups only ✅ Better — Raft consensus, automatic failover even for stateful processing
Controlled rolling upgrades 🟡 Depends on broker ✅ Better — drain/rejoin workflow
Columnar workloads (group-by, filter, compute) ❌ Slow — per-message Python serde ✅ Better — PyArrow columnar, zero-copy
Request/reply, RPC over broker ✅ Better — built-in pattern ❌ Not supported
Multi-broker (RabbitMQ, Redis, NATS + Kafka) ✅ Better — backends included ❌ Kafka only (NATS JetStream planned)

Rule of thumb: request/reply, or several brokers → FastStream. High-volume aggregation, stateful processing, or cluster-level HA → Turbine.

Why Turbine is not async

User code is plain def process(batch, state) — no async/await anywhere on the user surface. The reasoning:

  • Parallelism is already covered. On free-threaded Python (3.14t), each partition runs on its own thread without GIL contention. If one partition blocks on I/O, the others keep going. Async would duplicate that story rather than extend it — and an event loop is single-threaded by default, so it would compete with free-threading instead of complementing it.
  • The hot-path I/O isn't in Python. Kafka producer/consumer is in Rust, the state store is local RocksDB. The user never awaits broker traffic; there is no per-message network round-trip to overlap.
  • Async has a fixed per-await cost. Coroutine scheduling adds measurable overhead at the rate Turbine calls your handler — a cost paid in every workload, for a benefit that only matters in a narrow one.

The narrow case is per-batch external lookups (Redis enrichment, HTTP enrichment, secondary DB). Today this is solved without async by batching the lookup itself (redis.mget(keys_of_batch), httpx-with-thread-pool, etc.) — the batch model makes those naturally vectorisable. If a real workload ever shows async would actually win, we'd add it as opt-in (async def process()) rather than retrofit the whole API.