Installation

Prerequisites

Free-threaded Python is required

Turbine's per-partition parallelism relies on the free-threaded (no-GIL) CPython build, and the published wheels target it exclusively — cp314t (Python 3.14t). On a standard GIL interpreter there is no compatible wheel and no source fallback on PyPI, so the install fails outright. The t suffix denotes the free-threaded build; uv requests and pins it as 3.14t. Python 3.13's free-threaded build was experimental and is not supported (free-threading became official in 3.14, PEP 779).

• Python 3.14t (free-threaded) — see the warning above.
• Linux x86_64. The wheels are manylinux_2_28_x86_64; macOS, Windows and arm aren't published yet — build from source there (see From source).
• A Kafka-compatible broker, reachable before you run an app — Redpanda or Kafka, local or remote (Redpanda is what Turbine is tested against). If you don't have one, A local broker to develop against below starts one in a container.
• (Optional) an object store (S3, MinIO, GCS) for durable checkpoints. The local-filesystem default is fine for a first run, and for any single-node deployment; a cluster needs a shared one — see Checkpoint storage.

What the broker connection supports

• Compression: all four Kafka codecs — gzip, snappy, lz4 and zstd. Turbine reads topics whatever codec their producers chose, and nothing needs configuring.
• Security protocol: PLAINTEXT only. TLS (SSL, SASL_SSL) and SASL authentication (PLAIN, SCRAM, OAUTHBEARER, Kerberos) are not supported yet. Most managed services require them (Confluent Cloud, Amazon MSK with IAM or SCRAM, Aiven), so Turbine cannot reach those today. A broker on a private network that accepts unauthenticated plaintext connections works.

These instructions use uv — install it first if you don't have it.

Install

The PyPI package is turbine-fast; the import name is turbine.

In a uv project (recommended)

uv init my-pipeline && cd my-pipeline
uv python pin 3.14t          # free-threaded is mandatory
uv add --prerelease=allow turbine-fast

uv python pin writes a .python-version and downloads the free-threaded interpreter if you don't already have it; uv add then resolves the matching cp314t wheel and pulls the runtime deps (pyarrow, python-dotenv). --prerelease=allow is required while Turbine is published only as a pre-release.

Into a standalone virtualenv

uv venv --python 3.14t       # free-threaded interpreter
uv pip install --prerelease=allow turbine-fast

Check what you installed

Check which version was installed and whether the GIL is off:

uv run python -c "from importlib.metadata import version; print(version('turbine-fast'))"
uv run python -c "import sys; print(sys._is_gil_enabled())"   # must print: False

If the second command prints True, Turbine refuses to start by default. This also applies when an imported extension re-enables the GIL on a free-threaded build. To run deliberately with reduced Python parallelism, set TURBINE_ALLOW_GIL=1 when launching the app. Turbine emits a warning and starts. The setting applies to startup and requires a Turbine build compatible with your interpreter. PyPI wheels require free-threaded Python 3.14.

Third-party libraries can turn the GIL back on

The interpreter alone is not enough. A C extension that has not declared itself safe without the GIL silently re-enables it for the whole process the moment it is imported — Python only emits a RuntimeWarning — and every partition is back to sharing one core. Check any compiled dependency before adding it:

uv run python -c "import sys; import <lib>; print(sys._is_gil_enabled())"   # must print: False

Running with -W error::RuntimeWarning turns that warning into an import error, which is a cheap way to catch it in CI. Forcing the GIL off anyway (PYTHON_GIL=0) is possible but at your own risk: the extension did not claim to be thread-safe without it.

Kafka clients

Turbine handles its own broker I/O, but you may want a plain Kafka client next to it — an admin script, a route in a FastAPI app, a test producer. Not every client works on 3.14t (checked October 2026; re-run the command above after an upgrade):

Client	On Python 3.14t
kafka-python	Works — pure Python, the GIL stays off. Its optional crc32c accelerator keeps the GIL off too.
aiokafka	Its compiled record codec re-enables the GIL. Set AIOKAFKA_NO_EXTENSIONS=1 in the environment (before the import) to use its pure-Python fallback: the GIL stays off, at the cost of slower record encoding and decoding.
confluent-kafka	Through 2.15.x its bindings do not declare GIL safety: importing it re-enables the GIL. Versions 2.12.1–2.13.0 publish free-threaded wheels, but the import still turns the GIL back on; the latest stable (2.15.1) has no free-threaded wheel and builds from source (it needs the librdkafka headers). The 2.16.0 release candidates keep the GIL off; until 2.16.0 is final, not a practical option.

Don't produce from inside a handler with your own client

A client of your own sits outside Turbine's delivery guarantees. Records it sends from a handler are not part of the batch's commit: on a restart the batch is replayed and they are sent again, and under exactly_once they escape the transaction entirely. To write from a handler, use the subscription's output= or side outputs, which follow the subscription's guarantee. Keep a separate client for work outside the pipeline.

Type-checking note: pyarrow ships no type stubs. If you import it in your handlers and run a type checker (ty, mypy, pyright…), add stubs to silence "missing library stubs or py.typed marker" warnings:

uv add --dev pyarrow-stubs

Pinning a version

Turbine is published as alphas (0.2.0aN). Between them the Python API can still change, and so can on-disk state and internal topic formats — an upgrade may require dropping state and replaying. Pin an exact version in anything you deploy, and upgrade deliberately:

uv add --prerelease=allow "turbine-fast==0.2.0a2"

State-format break after 0.2.0a2

The alpha following 0.2.0a2 introduced a new persisted state format (early-lifecycle-v2, bumped again from v1 by the gated close-time early-fire cleanup). A worker that finds a state directory or a checkpoint written by an earlier alpha halts the affected partition — /health reports degraded and turbine_halted_workers names it — rather than reinterpreting data it cannot trust. This applies to every stateful app, including ones that never used early-fire windows. To upgrade: point the app at a fresh state directory and a fresh checkpoint location, then replay the input needed to rebuild state. There is no migration path.

From source (development, or other platforms)

The published wheel is feature-complete: stateful pipelines with durable checkpoints, the HTTP surface (health, metrics, assignments — used by Prometheus, and on a single node too), multi-node clustering, and the web console, which every node then serves at /console. All of it works straight from pip/uv install. Build from a clone when you're developing Turbine itself, or when you need a platform with no published wheel (macOS, Windows, arm) — this needs the Rust toolchain and maturin:

uv venv --python 3.14t
maturin develop --features python,state-rocksdb,checkpoint,cluster,cluster-raft,web-console

This compiles the Rust core (including RocksDB), so the first build takes several minutes. Inside the repo, prefer the project's just recipes (just py-build) over raw cargo.

A source build serves the console only once you have built the bundle — it is compiled into the extension, so just console-build (which needs pnpm) must run before just py-build. Until then /console answers with the command to run rather than the UI. Published wheels ship it already built.

A local broker to develop against

Any reachable Kafka or Redpanda works. If you don't have one, save this as docker-compose.yml next to your project and run docker compose up -d — a single-node Redpanda on localhost:9092, plus a web UI on localhost:8080 to watch topics and messages:

services:
  redpanda:
    image: docker.redpanda.com/redpandadata/redpanda:v26.2.2
    container_name: redpanda
    command:
      - redpanda start
      - --smp 1
      - --overprovisioned
      - --node-id 0
      - --kafka-addr PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      - --advertise-kafka-addr PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
    ports:
      - "9092:9092"   # Kafka API — this is what KafkaBroker(bootstrap=...) points at
      - "9644:9644"   # Redpanda admin API
      - "8081:8081"   # Schema Registry

  console:
    image: docker.redpanda.com/redpandadata/console:latest
    depends_on: [redpanda]
    ports:
      - "8080:8080"
    environment:
      KAFKA_BROKERS: redpanda:29092

Pin the broker image rather than tracking latest: a dev broker that changes under you breaks on the worst possible day.

Turbine does not create your topics

Turbine creates the internal topics it needs on its own (the hops behind repartition_by= and combine_by=). It never creates your input, output=, dead-letter or side-output topics — a subscription whose output topic doesn't exist produces nothing, which reads like a broken pipeline rather than missing setup. Create them explicitly (recommended — it is also how you control their partition count, see Partitioning):

docker compose exec redpanda rpk topic create my-input my-output

Or, for local convenience only, let the broker create them on first use:

docker compose exec redpanda rpk cluster config set auto_create_topics_enabled true

For durable checkpoints on an object store, add a MinIO service and point checkpoint_url at it — the settings are the standard AWS_* variables, covered in Checkpoint storage.

Smoke Test

A 10-line app that reads from a topic and prints batch sizes — enough to confirm Turbine is wired up and the broker is reachable.

1. Create a test topic and push a few messages (rpk shown against the compose stack above; kafka-topics.sh / kafka-console-producer.sh do the same on Kafka):

   docker compose exec redpanda rpk topic create smoke-test
   docker compose exec redpanda \
     bash -c 'for i in 1 2 3; do echo "{\"i\":$i}"; done | rpk topic produce smoke-test'
   

2. Save the snippet below as smoke.py and run it:

   from turbine import KafkaBroker, RecordBatch, Turbine
   
   kafka = KafkaBroker(bootstrap="localhost:9092")
   app = Turbine(kafka, from_earliest=True)
   
   @app.subscribe(kafka.topic("smoke-test"))
   def hello(batch: RecordBatch) -> None:
       print(f"Got {batch.num_rows} messages")
   
   app.run()
   

   uv run python smoke.py
   

You should see Got 3 messages after a moment. While it runs, the node also serves its API and the web console on http://localhost:8400/console — see Console & API. Stop it with Ctrl-C.

If you do, Turbine is installed correctly and the broker is reachable. Continue to Quick Start.