Installation

Prerequisites

Free-threaded Python is required

Turbine's per-partition parallelism relies on the free-threaded (no-GIL) CPython build, and the published wheels target it exclusively — cp314t (Python 3.14t). On a standard GIL interpreter there is no compatible wheel and no source fallback on PyPI, so the install fails outright. The t suffix denotes the free-threaded build; uv requests and pins it as 3.14t. Python 3.13's free-threaded build was experimental and is not supported (free-threading became official in 3.14, PEP 779).

• Python 3.14t (free-threaded) — see the warning above.
• Linux x86_64. The wheels are manylinux_2_28_x86_64; macOS, Windows and arm aren't published yet — build from source there (see From source).
• A Kafka-compatible broker, reachable before you run an app — a local Redpanda or Kafka (Redpanda is officially tested; the repo ships docker/dev/docker-compose.yml), or any reachable broker.
• (Optional) an object store (S3, MinIO, GCS) for durable checkpoints. The local-filesystem default is fine for a first run.

These instructions use uv — install it first if you don't have it.

TODO

Document a turn-key local stack: docker compose up for Redpanda and MinIO, plus the matching KafkaBroker(...) + Turbine(broker, checkpoint_url=…) boilerplate. Reference compose file: docker/dev/docker-compose.yml.

Install

The PyPI package is turbine-stream; the import name is turbine.

In a uv project (recommended)

uv init my-pipeline && cd my-pipeline
uv python pin 3.14t          # free-threaded is mandatory
uv add --prerelease=allow turbine-stream

uv python pin writes a .python-version and downloads the free-threaded interpreter if you don't already have it; uv add then resolves the matching cp314t wheel and pulls the runtime deps (pyarrow, pydantic, python-dotenv). --prerelease=allow is required while Turbine is published only as a pre-release.

Into a standalone virtualenv

uv venv --python 3.14t       # free-threaded interpreter
uv pip install --prerelease=allow turbine-stream

Either way, confirm the GIL is really off before going further — if this prints True, you're on a standard build and Turbine won't scale:

uv run python -c "import sys; print(sys._is_gil_enabled())"   # must print: False

Type-checking note: pyarrow ships no type stubs. If you import it in your handlers and run a type checker (ty, mypy, pyright…), add stubs to silence "missing library stubs or py.typed marker" warnings:

uv add --dev pyarrow-stubs

From source (development, or other platforms)

The published wheel is feature-complete: stateful pipelines with durable checkpoints, the HTTP surface (health, metrics, assignments — used by Prometheus, and on a single node too), multi-node clustering, and the web console, which every node then serves at /console. All of it works straight from pip/uv install. Build from a clone when you're developing Turbine itself, or when you need a platform with no published wheel (macOS, Windows, arm) — this needs the Rust toolchain and maturin:

uv venv --python 3.14t
maturin develop --features python,state-rocksdb,checkpoint,cluster,cluster-raft,web-console

This compiles the Rust core (including RocksDB), so the first build takes several minutes. Inside the repo, prefer the project's just recipes (just py-build) over raw cargo.

A source build serves the console only once you have built the bundle — it is compiled into the extension, so just console-build (which needs pnpm) must run before just py-build. Until then /console answers with the command to run rather than the UI. Published wheels ship it already built.

Smoke Test

A 10-line app that reads from a topic and prints batch sizes — enough to confirm Turbine is wired up and the broker is reachable.

1. Create a test topic and push a few messages (Redpanda CLI shown):

   docker compose -f docker/dev/docker-compose.yml exec redpanda \
     rpk topic create smoke-test
   docker compose -f docker/dev/docker-compose.yml exec redpanda \
     bash -c 'for i in 1 2 3; do echo "{\"i\":$i}"; done | rpk topic produce smoke-test'
   

2. Save the snippet below as smoke.py and run it:

   from turbine import KafkaBroker, RecordBatch, Turbine
   
   kafka = KafkaBroker(bootstrap="localhost:9092")
   app = Turbine(kafka, from_earliest=True)
   
   @app.subscribe(kafka.topic("smoke-test"))
   def hello(batch: RecordBatch) -> None:
       print(f"Got {batch.num_rows} messages")
   
   app.run()
   

   uv run python smoke.py
   

You should see Got 3 messages after a moment. If you do, Turbine is installed correctly and the broker is reachable. Continue to Quick Start.