Skip to content

Kubernetes

Run a Turbine cluster as a StatefulSet. It gives each pod what a Turbine node needs: a stable name its peers can reach (turbine-0, turbine-1, …), a volume of its own that follows it when it is rescheduled, and an ordinal that serves as its node id.

This page has everything a production deployment needs: the manifests and what each setting is for, how to size the shutdown and the memory, how to open the console and scrape the metrics, the day-to-day operations, and what the usual failures look like. How a cluster behaves — forming, a node going down, a node leaving — is described in Deployment; this page shows how it maps onto Kubernetes.

namespace Browser your team Ingress TLS · its own host Service turbine-console lists ready pods only PDB maxUnavailable: 1 one eviction at a time StatefulSet turbine turbine-0 node 0 /ready 200 volume state · Raft log turbine-1 node 1 /ready 200 volume state · Raft log turbine-2 node 2 /ready 503 catching up volume state · Raft log Raft Raft Service turbine (headless) lists every pod, ready or not pods reach each other by name Secret turbine token · S3 keys envFrom PodMonitor pods to scrape Prometheus scrapes every pod /metrics Kafka topics in and out consume produce S3 bucket checkpoints upload restore data, as it flows an API call, on demand management, configuration
The whole deployment. Two Services list the same pods differently: turbine-console only the ready ones — turbine-2, still catching up after a restart, gets no client traffic — and the headless turbine every one of them, since the pods must reach each other before any of them is ready. Click a box to jump to its manifest, anywhere else to enlarge.

What you need first:

The app

The app reads its settings from the environment. Raft needs no code at all — the TURBINE_RAFT_* variables of the StatefulSet below are enough:

# app.py
import os

from turbine import KafkaBroker, Turbine

kafka = KafkaBroker(bootstrap=os.environ["KAFKA_BOOTSTRAP"])
app = Turbine(
    kafka,
    app_id="my-pipeline",
    state_dir="/var/lib/turbine",                  # the pod's volume
    checkpoint_url=os.environ["CHECKPOINT_URL"],   # shared storage, e.g. s3://bucket/turbine
)

The manifests

Four files, applied together with kubectl apply -f .. Two optional ones come further down: a Service and an Ingress for the console and a PodMonitor for the metrics.

The names are linked to each other: the StatefulSet is called turbine, its pods carry the label app: turbine, the headless Service is called turbine too, and the peer template spells out turbine-{ordinal}.turbine. To rename, change all of them together.

Secret

secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: turbine
type: Opaque
stringData:
  TURBINE_API_TOKEN: "<output of: openssl rand -hex 32>"   # the same token on every pod
  AWS_ACCESS_KEY_ID: "<access key id>"
  AWS_SECRET_ACCESS_KEY: "<secret access key>"

Every pod reads it whole. The token protects the API and is how the pods authenticate to each other, so it must be the same on every pod; see Securing the API for what it covers. Create the Secret from your secret store rather than committing this file. On EKS, IAM roles for service accounts replace the two AWS keys — see the AWS_WEB_IDENTITY_TOKEN_FILE row in S3 and S3-compatible endpoints.

Headless Service

service.yaml
apiVersion: v1
kind: Service
metadata:
  name: turbine
spec:
  clusterIP: None                  # headless: one DNS name per pod
  publishNotReadyAddresses: true   # peers must resolve each other before they are ready
  selector:
    app: turbine
  ports:
    - name: api
      port: 8400

This Service is how the pods find each other:

  • clusterIP: None gives each pod a stable DNS name, turbine-0.turbine, turbine-1.turbine, …
  • publishNotReadyAddresses: true is required. A pod is not ready until it is part of a cluster, and the cluster cannot form until the pods resolve each other: without it, DNS lists only ready pods and nothing ever becomes ready.

It is for the pods, not for clients: because it also lists pods that are not ready, do not put an Ingress or an application in front of it — see Opening the console.

StatefulSet

statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: turbine
spec:
  serviceName: turbine
  replicas: 3
  podManagementPolicy: Parallel    # start every pod at once: the cluster forms when all answer
  selector:
    matchLabels:
      app: turbine
  template:
    metadata:
      labels:
        app: turbine
    spec:
      enableServiceLinks: false    # no Docker-link variables: see below
      terminationGracePeriodSeconds: 60
      affinity:
        podAntiAffinity:           # one pod per host, so a host failure costs one node
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:
                    app: turbine
      containers:
        - name: app
          image: registry.example.com/my-pipeline:1.0.0
          terminationMessagePolicy: FallbackToLogsOnError
          ports:
            - name: api
              containerPort: 8400
          env:
            - name: TURBINE_RAFT_PEERS
              value: "http://turbine-{ordinal}.turbine:8400"
            - name: TURBINE_RAFT_INITIAL_SIZE
              value: "3"
            - name: TURBINE_RAFT_REBALANCE_DELAY_MS
              value: "60000"       # longer than a pod takes to come back, see below
            - name: KAFKA_BOOTSTRAP
              value: "kafka:9092"
            - name: CHECKPOINT_URL
              value: "s3://my-bucket/turbine"
            - name: AWS_REGION
              value: "eu-west-1"
          envFrom:
            - secretRef:
                name: turbine      # TURBINE_API_TOKEN, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
          resources:               # an example: size them as described below
            requests:
              cpu: "2"
              memory: 4Gi
            limits:
              memory: 8Gi
          volumeMounts:
            - name: state
              mountPath: /var/lib/turbine
          startupProbe:
            httpGet:
              path: /health
              port: api
            periodSeconds: 2
            failureThreshold: 60   # up to 2 minutes to start serving
          livenessProbe:
            httpGet:
              path: /health
              port: api
            periodSeconds: 10
            failureThreshold: 6
          readinessProbe:
            httpGet:
              path: /ready
              port: api
            periodSeconds: 5
            failureThreshold: 1   # fast withdrawal; also reacts to transient probe failures
  volumeClaimTemplates:
    - metadata:
        name: state
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi

What each setting is for:

  • TURBINE_RAFT_PEERS as a template. {ordinal} stands for a pod's ordinal. Each pod takes the ordinal at the end of its host name as its node id (turbine-2 is node 2), listens at the template filled with it, and lists as peers every ordinal from 0 to TURBINE_RAFT_INITIAL_SIZE - 1 — plus its own, for a pod added later. The template must name pods through the headless Service: turbine-2.turbine is the pod's stable DNS name.
  • TURBINE_RAFT_INITIAL_SIZE is the number of pods of the first deployment: the cluster is formed once all of them answer. Set it to replicas the first time, then leave it alone — it plays no part once the cluster exists.
  • podManagementPolicy: Parallel is required. The default (OrderedReady) creates turbine-1 only once turbine-0 is ready, and turbine-0 waits for turbine-1 to answer before forming the cluster.
  • enableServiceLinks: false. By default Kubernetes injects into every pod a set of variables for each Service of the namespace, named after the Service — among them <NAME>_PORT=tcp://<address>:<port>. A Service named turbine-api would therefore set TURBINE_API_PORT to an address, and every pod would refuse to start (TURBINE_API_PORT must be an int, got 'tcp://…'). Turbine finds its peers through DNS and uses none of these variables.
  • The volume holds the node's partitions and its cluster membership (see Keep each node's state directory). A pod rescheduled elsewhere keeps its volume and resumes as the member it was. Size it for the state of the partitions a pod may hold, with room for the ones it takes over when another pod is down.
  • checkpoint_url on shared storage is what lets a partition move to another pod with its state. A cluster with stateful subscriptions refuses to start without one; see Checkpoint storage.
  • The token comes from the Secret, the same value on every pod. The probes need none: /health and /ready answer without it.
  • terminationMessagePolicy: FallbackToLogsOnError makes the end of a failed container's logs available in kubectl describe pod when its termination-message file is empty. It helps diagnose startup refusals even before the pod can contact the cluster. Kubernetes keeps only a short log tail there; use kubectl logs POD --previous for the full previous attempt. See Kubernetes termination messages.
  • Probes: /health for liveness and startup, /ready for readiness — never /ready for liveness, see Health probes. The HTTP server starts before the pod joins the cluster or restores any state, so the startup probe only covers the start of the process; a pod restoring a large partition keeps answering /health and is not restarted. A pod is ready once it is a voter of a cluster with a leader and has caught up with it; until then it runs no partition.
  • terminationGracePeriodSeconds must cover the pod's shutdown — see Sizing the shutdown.
  • TURBINE_RAFT_REBALANCE_DELAY_MS is how long a stopped pod keeps its partitions (the default, one minute, is written out so it sits next to the other numbers). It starts once the pod has finished stopping, after its final checkpoint, so the grace period does not count against it. What must fit in it is the rest of a restart: the leadership hand-over (up to ten seconds), the pod's re-creation, the image pull, the start and the catch-up with the cluster. Past it, the partitions move to the other pods, and the pod gets a share again when it returns.
  • resources — see CPU and memory.
  • Anti-affinity is preferred, so a cluster with fewer hosts than pods still schedules. Make it required… if you would rather have a pod pending than two Turbine nodes sharing a host.

PodDisruptionBudget

pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: turbine
spec:
  maxUnavailable: 1                # drains take one pod at a time
  selector:
    matchLabels:
      app: turbine

A node drain, a managed node upgrade or an eviction then takes one Turbine pod at a time, and only once the previous one is ready again — back in the cluster and caught up. Three voters tolerate one down; letting a drain take two would stop the cluster.

Withdrawing client traffic before shutdown

As soon as Turbine receives a stop request, /ready returns 503 with status: "unavailable" and reason: "stopping"; /health stays 200 throughout drainage. The FastAPI integration also withdraws readiness when the host receives SIGTERM or SIGINT, before its lifespan exits.

That alone does not close the routing window. Kubernetes sets a terminating endpoint's ready condition to false, but its serving condition continues to follow the readiness probe. A load balancer that uses terminating endpoints for drainage can therefore keep routing to the pod; sticky-session cookies can keep reconnections on it too. Meanwhile, uvicorn closes its listening socket as soon as SIGTERM arrives. See Kubernetes' endpoint conditions.

For a client-facing service, add this lifecycle block to the app container. It first withdraws readiness, then gives probes and the load balancer time to notice, while both listeners remain open. Kubernetes sends SIGTERM after the hook returns; workers and Raft stay running during the hook. This is a local traffic withdrawal, not a cluster leave.

lifecycle:
  preStop:
    exec:
      command:
        - /app/.venv/bin/python
        - -c
        - |
          import os
          import time
          import urllib.request

          port = os.environ.get("TURBINE_API_PORT", "8400")
          headers = {}
          token = os.environ.get("TURBINE_API_TOKEN")
          if token:
              headers["Authorization"] = f"Bearer {token}"
          request = urllib.request.Request(
              f"http://127.0.0.1:{port}/ready/stop", headers=headers, method="POST"
          )
          try:
              with urllib.request.urlopen(request, timeout=2) as response:
                  response.read()
          finally:
              time.sleep(10)

The interpreter path is the one created by the packaging example; adapt it if your image installs Python elsewhere.

Ten seconds is an example: allow at least one readiness probe period plus the load balancer's measured synchronization time. failureThreshold: 1 in the StatefulSet above withdraws the pod at the first failed probe, including a transient timeout during a CPU spike or an election. To tolerate those failures, keep the default threshold of three and lengthen the hook to cover all three probe periods plus load-balancer synchronization. The hook waits even if its HTTP request fails, but that delay alone cannot guarantee readiness withdrawal: fix authentication or connectivity errors reported by the hook. Check that the client-facing Service does not set publishNotReadyAddresses: true, and verify that your ingress stops sending new requests with an existing sticky cookie once serving is false. A single remaining pod still means an interruption when it stops.

The termination grace period starts before preStop, so budget for the hook, the web server's connection-drain timeout, Turbine's final checkpoints and leadership hand-over, plus a margin. A sleep without the readiness withdrawal can leave a terminating endpoint serving throughout the sleep. A direct SIGTERM without this hook still withdraws readiness, but cannot wait for routing updates before the web listener closes. See Kubernetes' pod termination flow.

Sizing the shutdown

A restart, from the stop to the pod's return, crosses two settings in turn:

the pod probes settings its partitions SIGTERM rolling update, drain, eviction… stop announced exit ready the rolling update moves on finishes the batch in flight final checkpoints, uploaded together hands over the leadership ≤ 10 s if it leads pod re-created, image pulled process starts catches up with the cluster runs its partitions again startup probe: /health 200 /ready 503 catching_up /ready 200 terminationGracePeriodSeconds past it: SIGKILL, a crash TURBINE_RAFT_REBALANCE_DELAY_MS counts from the stop: the whole return must fit in it processed waiting: nobody consumes them, their lag grows resumed from its own volume not back when the delay ends? its partitions move to the other pods
One pod's restart, not to scale. The grace period covers the stop; the rebalance delay starts when the stop is announced, so it must hold the leadership hand-over and the whole return. Nobody consumes the pod's partitions in between. /ready turns 200 only once the pod has caught up — that is what the rolling update and the disruption budget wait for before stopping the next pod. Click to enlarge.

On SIGTERM a pod:

  1. finishes the batch in flight on each of its partitions;
  2. writes a final checkpoint of every partition that changed since its last one, and uploads them — all at the same time;
  3. announces its stop to the cluster and, if it is the leader, hands the leadership over — usually well under a second; each of the two waits at most ten seconds when the cluster does not answer — and exits.

terminationGracePeriodSeconds must cover the three. Past it, Kubernetes kills the pod and the stop becomes a crash, so its partitions take the crash-recovery path instead of a clean hand-over: a pod back on its volume within the rebalance delay resumes from its local state, while a partition that moves to another pod starts from its last periodic checkpoint instead of the final one — see Crash Recovery.

Measure it: every stop logs its own duration. The last line a pod writes as it stops says how long the stop took, and where the time went:

node stopped in 20.4 s (workers and their final checkpoints 18.1 s, stop announcement and leadership hand-over 2.3 s)

The two parts follow each other, so they add up to the total; the same durations travel as fields of the line (total_ms, workers_ms, cluster_ms). Read it after a rolling update and keep the grace period comfortably above the longest one: the grace period counts from the start of termination (including any preStop hook), the line from when Turbine noticed the stop request — usually within a second, later if it was busy at that moment, restarting a worker for instance — and the process still has to exit after it.

Before the first stop, estimate it from the uploads. Checkpoint uploads are incremental: a checkpoint sends only the storage files written since the previous one, so the final checkpoint carries what changed since the last periodic one — not the whole state, which goes up only when the bucket holds none of it yet (a partition's very first checkpoint). The periodic checkpoints upload the same kind of increment, so turbine_last_checkpoint_upload_seconds — how long each partition's last upload took — tells you what its final one costs. A pod's partitions upload their final checkpoints at the same time and share its bandwidth: add their last upload durations for a conservative estimate of the upload step, then add one batch, up to twenty seconds for the cluster step when the cluster does not answer, and a margin.

CPU and memory

Memory. Each partition a pod consumes reads ahead of processing: the Kafka client prefetches up to about half a million records, or 2 GB, per partition — whichever comes first. A partition that keeps up holds far less; one with a backlog to catch up (after a restart, a rebalance, a slow stretch) fills it. So size the memory limit from what a pod uses while catching up, with the most partitions it can be given, not from what it uses at steady state. A pod killed for memory is a crash, and it would be killed again at the same point of the next catch-up.

State. A stateful pod also keeps part of its state in memory: a block cache shared by its partitions, which takes a quarter of the memory limit by default (of the node's memory if you set no limit). State write buffers share that budget and trigger background flushes against that capacity. This is a soft target, so allow headroom for bursts and pending flushes. Caches of recent values remain separate per partition and thread: size for the most partitions the pod can be given. Memory describes these costs and settings.

CPU. Turbine sizes its thread pools to the CPUs the container may use: a CPU limit caps them, a request does not. Leave the limit out to let a pod use a whole node's spare CPU during a catch-up; set one to bound it, knowing that it also bounds the threads.

Opening the console

The pods' addresses are internal to the cluster: your browser cannot reach them, so listing a few nodes in the console's settings does not work from outside. Two ways in.

For one person: a port-forward

kubectl port-forward pod/turbine-0 8400:8400
# then open http://localhost:8400/console

Any pod will do: each one shows the whole cluster. Nothing is exposed and no object is created; it lasts as long as the command, and does not follow the pod if it restarts.

For a team: a Service and an Ingress

console.yaml
apiVersion: v1
kind: Service
metadata:
  name: turbine-console            # not "turbine-api": see enableServiceLinks above
spec:
  selector:
    app: turbine                   # no publishNotReadyAddresses: ready pods only
  ports:
    - name: api
      port: 8400
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: turbine-console
spec:
  ingressClassName: nginx          # your Ingress controller's class
  tls:
    - hosts:
        - turbine.example.com
      secretName: turbine-console-tls
  rules:
    - host: turbine.example.com    # a host of its own, served from its root
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: turbine-console
                port:
                  name: api

The console is then at https://turbine.example.com/console. Three things this layout gets right that the obvious one does not:

  • A second Service, not the headless one. The turbine Service lists pods that are not ready — it has to, for the cluster to form. A pod that has just restarted serves HTTP before it has caught up, and answers with its own, older view of the cluster: behind that Service the console would flip between the current picture and an outdated one from one refresh to the next, during every rolling update. This one keeps Kubernetes' default and routes to ready pods only.
  • A host of its own, at the root path. The console loads its files from /console/… and calls the API at the root of the address it was served from (/cluster/nodes, /windows, …). Under a sub-path such as https://ops.example.com/turbine/, neither resolves.
  • Not named turbine-api. That name sets TURBINE_API_PORT in every pod of the namespace unless the StatefulSet sets enableServiceLinks: false — the manifest above does both.

This exposes the whole API, not only the pages

Everything the API does is reachable at that address: reading your state, window contents and dead-lettered records (often production payloads), and changing cluster membership (the console's node actions, /cluster/leave, /cluster/force_release…). The API token grants everything; the scrape token only allows /metrics. Filtering HTTP methods at the Ingress is no way out either: the console reads state with POST.

So: never without TURBINE_API_TOKEN; terminate TLS at the Ingress, or the token travels in clear; and restrict who can reach the address with what your Ingress controller offers (an IP allow-list, an authenticating proxy).

Metrics

Each pod serves its Prometheus metrics at /metrics on the api port. Scrape every pod, not a Service: each one reports its own partitions. Configure a separate TURBINE_METRICS_TOKEN on the Turbine pods so Prometheus can scrape without admin rights (see Scraping the metrics). Add it to the turbine Secret before applying the PodMonitor, keeping TURBINE_API_TOKEN for administration and node-to-node calls:

stringData:
  TURBINE_METRICS_TOKEN: "<output of a separate: openssl rand -hex 32>"

Merge this key into the existing Secret, then restart the Turbine pods to load it. It must differ from TURBINE_API_TOKEN. Prometheus needs only the scrape credential; if you copy credentials to a separate monitoring Secret, copy only this key and adapt the PodMonitor's Secret name.

With the Prometheus Operator, download turbine-podmonitor.yaml and apply it in the StatefulSet's namespace, alongside the turbine Secret:

curl -fsSLO https://docs.turbine.fast/grafana/turbine-podmonitor.yaml
kubectl apply -n default -f turbine-podmonitor.yaml
turbine-podmonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: turbine
  labels:
    app: turbine
spec:
  jobLabel: app
  selector:
    matchLabels:
      app: turbine
  podMetricsEndpoints:
    - port: api
      path: /metrics
      interval: 15s
      authorization:
        type: Bearer
        credentials:
          name: turbine
          key: TURBINE_METRICS_TOKEN
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_name]
          targetLabel: instance

The Secret is read from the PodMonitor's namespace, not Prometheus's namespace. With no namespaceSelector, this monitor discovers pods in its own namespace only. The pod label app: turbine selects the pods; their named container port api selects port 8400. jobLabel: app takes the job value from the selected pod's app label, giving job="turbine" for the dashboard and alert rules. The relabeling uses the pod name as instance, so each node stays visible separately.

Prometheus must select both the namespace and the PodMonitor's metadata labels. For example, merge these fields into your existing Prometheus resource, adapting default to the app's namespace and preserving any selections you already need:

spec:
  podMonitorNamespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: default
  podMonitorSelector:
    matchLabels:
      app: turbine

Charts can add a release-label selector: add the matching label to the PodMonitor or adapt the chart's selector. Prometheus needs RBAC to discover pods in the app's namespace; the operator needs access to the monitor and its authorization Secret. An accepted PodMonitor alone does not prove that Prometheus selected it.

In Prometheus's Status → Target health, expect one target per Turbine pod, job="turbine", the pod name as instance, and state UP. A missing target usually means a namespace/label/port selection or RBAC problem; a target returning 401 means the Secret or its key differs from the app's scrape token. /metrics stays protected. For a second deployment, use a distinct job value and adapt the alert rules' job selector; the node names alone need not be globally unique.

Alert rules can be loaded through a PrometheusRule selected by the same operator.

Without the operator, a scrape job that finds the pods by itself. Prometheus needs the scrape token in a file: mount the Secret's TURBINE_METRICS_TOKEN key into its pod, or a copy of it if Prometheus runs in another namespace.

prometheus.yml
scrape_configs:
  - job_name: turbine
    scrape_interval: 15s
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [default]         # the StatefulSet's namespace
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/turbine/TURBINE_METRICS_TOKEN
    relabel_configs:
      # The Turbine pods' `api` port only.
      - source_labels: [__meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
        regex: turbine;api
        action: keep
      # The pod's name as `instance`: the node's name in the dashboard.
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: instance

Monitoring has the Grafana dashboard and the signals to alert on.

Day-to-day operations

  • First deployment. Every pod starts at once; turbine-0 forms the cluster when all of them answer, and the others join it. Until then no pod is ready. If one pod can never start (a volume that will not bind, a node pool that is full), see How a node finds its cluster for forming it without that pod.
  • Rolling update (a new image, a changed setting). The StatefulSet restarts the pods one at a time and waits for each to be ready — back in the cluster and caught up — before stopping the next. A restarted pod keeps its partitions, provided it is back within TURBINE_RAFT_REBALANCE_DELAY_MS (what the delay must cover is listed above). Check runtime compatibility first; see Updating the application.
  • A pod or a host lost. The pod's partitions wait for the delay, then move to the others. When it comes back with its volume, it resumes or gets a share again.
  • Draining a node. The disruption budget lets Kubernetes evict one Turbine pod at a time, and only once the previous one is ready again.
  • Scaling up. kubectl scale statefulset turbine --replicas=4. The new pod lists itself and the existing ones, finds the cluster and joins it; the existing pods are not restarted.

  • Scaling down. Removing a pod stops it, and a stopped node is still a member. Have the pods being removed leave first — the highest ordinals, which the StatefulSet removes — then scale. The leave goes to the pod that leaves:

    kubectl port-forward pod/turbine-3 8400:8400 &
curl -fsS -X POST -H "Authorization: Bearer $TURBINE_API_TOKEN" \
  http://localhost:8400/cluster/leave
kill %1
kubectl scale statefulset turbine --replicas=3

    Skipping the leave keeps the pod in the quorum: the cluster then tolerates one failure fewer. The removed pod's volume stays behind (Kubernetes keeps it); a later scale-up brings the pod back on it, and it joins the cluster again. - A lost volume (deleted, or a pod rescheduled onto an empty one) needs nothing either: the pod sees that the cluster still counts it while it remembers nothing, has itself removed and joins again.

Updating the application

Separate an application change from an upgrade of Turbine itself. The first upgrade to the subscription registry, and the first upgrade to startup-refusal diagnostics, require every node to stop before the new runtime starts. Follow the runtime upgrade procedure before using rolling application updates. An older runtime may not be able to read the cluster data written by the newer one; reverting its image is then not a supported rollback.

For a coordinated runtime upgrade, first back up the state directories and check the release's migration requirements in that procedure. The StatefulSet above keeps its PVCs on scale-down (persistentVolumeClaimRetentionPolicy.whenScaled defaults to Retain). If you changed that policy, verify it retains the PVCs before stopping the cluster. Keep checkpoint storage too.

Stop all pods and wait for their deletion:

kubectl scale statefulset/turbine --replicas=0 &&
kubectl wait --for=delete pod -l app=turbine --timeout=5m

If either command fails, stop here: do not start the new runtime until every old pod has stopped. Perform any required migration while the cluster is stopped, then set the new image and restore the original replica count (three here):

kubectl set image statefulset/turbine app=registry.example.com/my-pipeline:2.0.0 &&
kubectl scale statefulset/turbine --replicas=3 &&
kubectl rollout status statefulset/turbine --timeout=5m

Replace the example image with the intended runtime upgrade and update the source manifest to match. For the first subscription-registry upgrade, keep the original subscription declaration for this restart. Preserve the existing app_id, PVCs and checkpoints; a runtime upgrade is separate from an application declaration change.

Once every node runs a compatible registry-aware runtime, keep the same app_id and the existing subscription names when changing the application:

  • Adding, removing or reordering subscriptions supports a rolling update. During the rollout, a subscription runs only on nodes whose application declares it.
  • Removing a subscription keeps its state and checkpoints. Reintroducing the same name on the same source topic reuses the retained history, subject to the broker's input-topic and committed-offset retention.
  • Renaming a subscription creates a new consumer group and fresh worker state; it does not migrate the old subscription's data. Changing an existing subscription's persisted state or input layout still needs its own compatibility check; the registry does not make every application change safe.

A stateless subscription's only resume position is its consumer group's committed offset. The broker can expire those offsets while the subscription is removed (7 days from the last commit by default, configurable). If they expire, reintroducing it uses the fresh-group starting policy: the topic head by default, skipping records produced during its absence, unless you explicitly choose another starting position. An exactly-once stateful subscription with committed timer history instead stops with a recovery error when that broker history is missing. No keepalive runs for a removed subscription; set broker retention longer than the planned absence, with margin. See Broker commit-history retention. The input topic must also retain the unconsumed records for any restart to read them.

Preserve the pods' volumes and checkpoint storage, including the subscription registry, throughout the update and any rollback. Do not change app_id, delete volumes or clear cluster data to get past a refusal. See Subscription identities for the identity rules.

Apply the new manifest, then wait for the rollout with a bound:

kubectl apply -f statefulset.yaml
kubectl rollout status statefulset/turbine --timeout=5m

Choose the timeout to cover the whole rollout, including image pulls and recovery on each pod. A timeout ends the command; it does not stop the rollout or roll it back. If startup is refused, use the recovery procedure below.

Recovering a refused rollout

A pod whose startup configuration is refused exits and normally enters CrashLoopBackOff. The rolling update waits for that pod to become ready before replacing the next one. Pods not yet replaced keep the previous application; pods already updated keep the new one. Turbine reports the refusal but does not revert the StatefulSet's template.

  1. Find the failed pod and read its reason. For example, if turbine-2 failed:

    kubectl get pods -l app=turbine
kubectl describe pod turbine-2
kubectl logs turbine-2 --previous

    A surviving pod's console may also retain the refused node's reason after cluster contact. Publication is best effort; use the pod's diagnostics if no reason appears. A failure before cluster contact, such as a refused legacy store or a port already in use, is available only through those diagnostics. Before the first container restart, omit --previous to read the current attempt's logs.

  2. Correct the manifest, or select a known working application revision with a compatible Turbine runtime. Inspect the history first; the example restores revision 3, which you must replace with your chosen revision:

    kubectl rollout history statefulset/turbine
kubectl rollout history statefulset/turbine --revision=3
kubectl rollout undo statefulset/turbine --to-revision=3

    If a runtime upgrade changed the cluster's persisted format, keep the compatible runtime and correct the application/configuration instead of downgrading it. A rollout revision restores only the pod template: restore any independently changed Secrets or ConfigMaps too. Their referenced contents are not part of that revision. Keep the corrected source manifest in sync with the deployed template so the next kubectl apply does not restore the refused configuration.

  3. After restoring the template, check the pod. If it still runs the refused revision, delete only that failed pod, then wait again:

    kubectl delete pod turbine-2
kubectl rollout status statefulset/turbine --timeout=5m

    This preserves its volume; the StatefulSet recreates it from the restored template. Do not delete the healthy pods or their PVCs. The StatefulSet forced-rollback caveat can also affect this page's Parallel policy: parallel initial creation does not make a failed rolling update repair itself when the template is reverted.

Finally, verify that all pods are ready and the surviving console shows the expected subscriptions and their processing health. Readiness confirms cluster admission, not that every partition worker is healthy. An admitted restart clears that node's retained startup refusal.

When something goes wrong

Symptom Likely cause What to do
After the first deployment, no pod ever becomes ready podManagementPolicy left at OrderedReady, publishNotReadyAddresses missing, or one pod cannot start Fix the two settings. For a pod that will never start, form the cluster without it: How a node finds its cluster
Every pod crashes at start: TURBINE_API_PORT must be an int, got 'tcp://…' A Service named turbine-api in the namespace, with service links on enableServiceLinks: false on the StatefulSet, or rename the Service
A pod stays not ready, /ready says no_raft_leader No quorum: more than one pod down, or the pods cannot reach each other on the API port (a NetworkPolicy, a different token — listed as unauthorized in the console's Nodes view) Bring pods back; check that each pod reaches turbine-N.turbine:8400 and that every pod has the same token
A pod exits because startup was refused Incompatible startup configuration or an admission error Read the reason in kubectl describe pod, kubectl logs POD --previous or the surviving console, then follow Recovering a refused rollout
A pod stays not ready, /ready says catching_up It cannot get the cluster's current state from the leader Same checks as above, from that pod
Partitions stop being consumed after a pod went away Expected for TURBINE_RAFT_REBALANCE_DELAY_MS; they move afterwards Nothing, or bring the pod back; turbine_partitions_awaiting_rebalance counts them
With exactly-once, a partition that moved never starts Its previous owner died hard and could not hand it over Recovering a stalled partition handoff
A pod is OOMKilled while catching up, again and again Memory limit below what a catch-up prefetches Raise the limit: CPU and memory
A stopping pod is killed before it finishes (no node stopped line at the end of its logs) terminationGracePeriodSeconds shorter than the stop — the stop becomes a crash Raise it: Sizing the shutdown
The console flips between two pictures of the cluster The Ingress points at the headless Service A second Service: For a team