Monitoring¶
Every node serves its Prometheus metrics at /metrics on the API port. This page
covers how to scrape them, the Grafana dashboard made for them, the signals to alert
on, and what to know before writing your own queries. The full list of metrics closes
the page.
Scraping the metrics¶
Scrape every node, not an address that load-balances between them: each node reports the partitions it runs, plus a few cluster-wide values as that node sees them.
When the API has a token, /metrics requires a bearer token. Give Prometheus a
separate scrape token with TURBINE_METRICS_TOKEN (or
Turbine(metrics_token=...)) on every node it scrapes:
This token grants access only to /metrics: it cannot read application state or
change cluster membership or readiness. It requires TURBINE_API_TOKEN (or
api_token=...) to protect the rest of the API; startup refuses a missing admin
token, identical tokens, or an empty or invalid token. Surrounding whitespace is
trimmed. Constructor arguments take precedence over the environment.
The admin token still works for existing scrapers. Without a scrape token,
/metrics keeps requiring the admin token; with neither token configured, the API
and metrics remain open. Probes and the static console page remain public
(see Securing the API).
Copy only the scrape token into Prometheus's credentials file. Changing it requires restarting each affected Turbine node and updating Prometheus's credential. Scrape tokens are local to each node and are never used for node-to-node calls; using one shared value across scrape targets lets them share one Prometheus scrape job.
scrape_configs:
- job_name: turbine
scrape_interval: 15s
authorization:
type: Bearer
credentials_file: /etc/prometheus/turbine-token # the scrape token, alone in the file
static_configs:
- targets: [node-1:8400, node-2:8400, node-3:8400]
On Kubernetes, Metrics gives a PodMonitor for the Prometheus Operator, and a scrape job that finds the pods by itself.
The output is the classic Prometheus text format, with # HELP descriptions for
every built-in metric family that appears in the scrape. A 15-second
interval suits it: the gauges that Turbine computes itself (lag, watermarks, rates)
refresh every few seconds to every 10 seconds.
The Grafana dashboard¶
turbine-dashboard.json is a dashboard for these
metrics. It was built on Grafana 13.2 and Prometheus 3.15.
- Import it: Dashboards → New → Import, upload the file, then pick your Prometheus data source.
- Or provision it: put the file in a directory read by a dashboard provider. On Kubernetes with the Grafana Helm chart, a ConfigMap carrying the file and the label the chart's dashboard sidecar watches does the same.
In the data source's settings, set Scrape interval to the interval of your scrape
job. The rates are computed over $__rate_interval, which Grafana derives from it, and
a rate needs at least two samples.
The variables at the top select the data source, the scrape job, the nodes and the
subscriptions. Select the scrape job carrying your Turbine targets. Job and node
choices come from Prometheus scrape targets, so process panels remain selectable
before any worker starts. A last one, Container, names the Turbine container for the
Kubernetes row (turbine by default).
| Row | Answers |
|---|---|
| Health | Is everything running? Nodes down, Raft leader, workers running, halted workers, partitions awaiting a rebalance, checkpoint store, age of the oldest checkpoint, DLQ over the last hour. Green everywhere means nothing needs a look. |
| Throughput & lag | Does the cluster keep up? Messages per second, consumer lag and event-time watermark lag, per subscription. |
| Where the time goes | Seconds spent per second in each stage of a batch (waiting for input, handler, produce, commit, timers), batch duration, inference, repartition transit. |
| State, windows & timers | State operations per second, timers registered, the close backlog, keys expired by state_ttl. |
| Joins & late data | Rows matched, unmatched and ignored by each interval join, late and overflowing rows, the skew between the join's inputs, rows that reached a window after it closed. |
| Checkpoints & cluster | Checkpoint size and upload time, checkpoint failures, and the events of the time range: worker restarts, fatal halts, exactly-once replays, the longest handoff wait, nodes excluded, unauthorized API calls. |
| Per worker (collapsed) | The main signals for each worker, to find the partition behind an aggregate. |
| Process resources (collapsed) | CPU cores and resident memory of each Turbine process, directly from /metrics on Linux. Includes its Python host and all threads; follows node/job selection, independently of the subscription. |
| Pod resources (collapsed) | CPU and memory of each pod's Turbine container, from the kubelets' cAdvisor. Kubernetes only, and only if Prometheus scrapes cAdvisor. |
The Subscription selector lists subscription names, including internal repartition
and combine stages. Curves show the name and its stable index, for example
score_users (sub 00). Two subscriptions on different source topics may have the
same name: selecting that name includes both, and their indices keep their curves
separate. Worker-based panels follow this selection; node-wide health and the panels
organized by join or window name retain their own scope.
Dashboard version 3 requires the subscription label on turbine_worker_kind
from every selected node. Older nodes do not contribute to the named subscription
panels. Upgrade all selected nodes before switching dashboards; version 2 remains
usable with the existing metric names and worker labels.
The batch-duration and repartition-transit p99 combines observations from all selected
running workers and nodes, per subscription, over $__rate_interval. Each batch
counts once, so a worker processing more batches contributes more observations. The
value is estimated from fixed histogram buckets; it is not a per-record latency.
An idle selection has no p99 until it has observations in that interval.
Dashboard version 2 uses the …_distribution_seconds_bucket and
…_distribution_ms_bucket series. Upgrade every selected node before using these p99
panels: during a rolling runtime upgrade, older nodes contribute no buckets and the
result covers only upgraded nodes. Existing summary queries and mean queries continue
working. Earlier versions of the dashboard showed the worst worker's recent p99.
What to alert on¶
At least:
| Signal | Means | See |
|---|---|---|
/ready failing for more than a few minutes |
the node is not in a working cluster | Health probes |
turbine_halted_workers > 0 |
a partition stopped on data it cannot process | Error handling |
turbine_processing_errors_consecutive climbing |
a handler fails batch after batch | Knowing when a partition has stopped producing results |
turbine_checkpoint_store_writable == 0 |
state is no longer being saved | Checked at boot, watched afterwards |
turbine_partitions_awaiting_rebalance > 0 for longer than the rebalance delay |
partitions of a node that went away are not consumed | When a node goes down |
turbine_handoff_barrier_waiting == 1 for minutes |
an exactly-once partition waits for an owner that died | Recovering a stalled partition handoff |
turbine_consumer_lag growing |
the cluster does not keep up | Performance tuning |
The dashboard's Health row shows the same signals.
Installing the alert rules¶
Download turbine-alerts.yaml, alongside the dashboard.
It contains starting rules for targets that cannot be scraped, halted workers,
repeated handler errors, unwritable checkpoint storage, blocked handoffs, delayed
rebalances and growing consumer lag. All rules select job="turbine"; change that
selector for another deployment and scrape every node under that job.
For standalone Prometheus, mount the downloaded file and add it to the configuration:
Check it with promtool check rules /etc/prometheus/rules/turbine-alerts.yaml before
reloading Prometheus. Configure Prometheus's alerting.alertmanagers and an
Alertmanager receiver to deliver notifications: loading rules alone only shows
pending and firing alerts in Prometheus.
With Prometheus Operator, use the same rules as the spec of a PrometheusRule.
Save the complete example below and apply it in the app's namespace:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: turbine
labels:
app: turbine
spec:
# Starting thresholds for one deployment scraped as job="turbine".
# Lag growth uses the worker-computed policy; tune other holds to the workload.
groups:
- name: turbine
rules:
- alert: TurbineTargetDown
expr: up{job="turbine"} == 0
for: 2m
labels:
severity: critical
annotations:
summary: Turbine metrics endpoint is unreachable or refuses the scrape
runbook_url: https://docs.turbine.fast/monitoring/#what-to-alert-on
- alert: TurbineHaltedWorkers
expr: turbine_halted_workers{job="turbine"} > 0
for: 1m
labels:
severity: critical
annotations:
summary: Turbine has halted partitions
runbook_url: https://docs.turbine.fast/monitoring/#what-to-alert-on
- alert: TurbineProcessingErrors
expr: |
(turbine_processing_errors_consecutive{job="turbine"} >= 3)
and on (job, namespace, instance, worker) (turbine_worker_running{job="turbine"} == 1)
for: 2m
labels:
severity: warning
annotations:
summary: Turbine repeatedly skips or dead-letters failing batches
runbook_url: https://docs.turbine.fast/monitoring/#what-to-alert-on
- alert: TurbineCheckpointStoreUnwritable
expr: turbine_checkpoint_consecutive_failures{job="turbine"} >= 3
for: 2m
labels:
severity: critical
annotations:
summary: Turbine cannot save checkpoints
runbook_url: https://docs.turbine.fast/monitoring/#what-to-alert-on
- alert: TurbineHandoffBlocked
expr: turbine_handoff_barrier_waiting{job="turbine"} == 1
for: 5m
labels:
severity: warning
annotations:
summary: Turbine partition handoff is waiting for the former owner
runbook_url: https://docs.turbine.fast/monitoring/#what-to-alert-on
- alert: TurbineRebalanceDelayed
expr: turbine_partitions_awaiting_rebalance{job="turbine"} > 0
for: 10m
labels:
severity: warning
annotations:
summary: Turbine partitions are still awaiting reassignment
runbook_url: https://docs.turbine.fast/monitoring/#what-to-alert-on
- alert: TurbineConsumerLagGrowing
expr: |
(turbine_consumer_lag_growth_state{job="turbine"} == 2)
and on (job, namespace, instance, worker) (
turbine_consumer_lag_growth_valid_until_seconds{job="turbine"}
> timestamp(turbine_consumer_lag_growth_valid_until_seconds{job="turbine"})
)
and on (job, namespace, instance, worker) (turbine_worker_running{job="turbine"} == 1)
labels:
severity: warning
annotations:
summary: Turbine consumer lag is growing above the configured threshold
runbook_url: https://docs.turbine.fast/monitoring/#what-to-alert-on
Merge the following selections into your existing Prometheus resource, retaining any other monitors/rules it must select. If your chart requires a release label, add that label to the PrometheusRule too:
spec:
ruleNamespaceSelector:
matchLabels:
kubernetes.io/metadata.name: default # the PrometheusRule's namespace
ruleSelector:
matchLabels:
app: turbine
The PodMonitor example explains the corresponding scrape selection, Secret placement and target checks. Check Alerts in Prometheus to confirm that these rules loaded, then verify delivery through your Alertmanager.
Choosing thresholds and responding¶
| Alert | Starting condition and hold | First action |
|---|---|---|
TurbineTargetDown |
Scrape fails for 2 minutes | Check the pod, network and authorization Secret. A 401 is a failed scrape too. |
TurbineHaltedWorkers |
At least one halted partition for 1 minute | Inspect the failing input and error policy. |
TurbineProcessingErrors |
At least 3 consecutive failing batches for 2 minutes on a running worker | Inspect skipped or dead-lettered data; a successful batch clears the condition. |
TurbineCheckpointStoreUnwritable |
At least 3 consecutive checkpoint-store failures, held for 2 minutes | Restore storage access; processing can continue while recovery protection falls behind. The three-failure threshold matches /health; a successful store operation clears it. |
TurbineHandoffBlocked |
An exactly-once handoff waits for 5 minutes | Check the former owner before following handoff recovery. |
TurbineRebalanceDelayed |
Partitions await reassignment for 10 minutes | Check node availability and the configured rebalance delay. Set this hold longer than that delay plus normal recovery time. A leader change restarts the hold on the new leader's series. |
TurbineConsumerLagGrowing |
A running worker reports fresh sustained lag growth (the same red state as the console) | Compare input rate and processing capacity; inspect normal bursts and backfills. |
The worker computes one lag-growth state for its logs, the console and the lag alert. It samples lag approximately every 10 seconds, including on an idle stream. The first successful observation establishes a baseline. A consecutive increase counts only when both observations exceed 1,000 records and the increase exceeds 10 records. Three increases produce a yellow warning; five produce a red growing state and make the alert eligible at the next evaluation, without another hold. A successful unchanged, decreasing or too-small increase, or a return to 1,000 or below, resets the run immediately. A large backlog that is draining stays green. For named inputs, the state uses their combined lag.
Failed observations clear the run to unknown. Each successful observation publishes its own validity deadline, 30 seconds later: without a new observation the state becomes unknown, including while a slow callback blocks the next sample. Successful observations retain their growth run across slow turns, so a worker taking more than 30 seconds per batch can still report a growing backlog. The console can show unknown between those observations. Stopping or restarting a worker clears its state. These observations are diagnostic and are not restored from checkpoints; a console reload preserves the live worker's state. Duplicate scrapes or page refreshes do not change the run. Node and subscription indicators take the most severe live-worker state; an unknown worker prevents an all-green result when no worker reports a warning or growth.
Older nodes without these metrics show unknown lag growth and do not participate
in this alert; upgrade all monitored nodes for full coverage. The rule matches
state, validity and liveness by job, namespace, instance and worker. Keep clocks
synchronized across Turbine nodes and Prometheus so validity deadlines agree.
The lag alert tests validity at the sample's scrape timestamp, so 30-second or
one-minute scrape intervals do not suppress a fresh state merely because the rule
evaluates later. It clears on the next scrape reporting draining, unknown,
retirement or an already-expired observation; unlike REST, it uses Prometheus's
last scraped view. The alert identifies workers through job, namespace,
instance and worker; it no longer carries topic or partition labels.
These are starting thresholds, not service-level guarantees. Worker alerts ignore
retired owners; an idle or missing metric does not mean a fault or a healthy zero.
TurbineTargetDown covers discovered targets only: deleting a target from discovery
removes its up series. Monitor desired pod replicas separately to detect missing
pods. These rules do not alert on /ready failures. A successful /metrics
scrape does not prove readiness. Keep Kubernetes readiness probes, and use an
HTTP probe exporter for /ready if readiness must also page through Prometheus;
these rules do not assume one is installed.
Reading the metrics¶
One label, worker. Per-partition metrics are labeled
worker="work-{sub_index}-p{partition}", where sub_index is the subscription's
stable registry index. A worker belongs to exactly one subscription, so when
several subscriptions read one topic
they have distinct worker labels: group by worker, or by the index taken out of it,
rather than by topic. The turbine_worker_kind info gauge also carries the declared
subscription name. Join it to a worker metric to add that name without adding it
to every metric family:
(turbine_consumer_lag and on (instance, worker) turbine_worker_running == 1)
* on (instance, worker) group_left(subscription)
max by (instance, worker, subscription) (turbine_worker_kind == 1)
Restrict all operands to the same scrape job. The max removes duplicate kind
labels if a worker changes nature during the process lifetime; keep the stable
index from worker when grouping subscriptions that share a name.
No metric names the node: that is the scrape target's
instance. Some metrics carry other labels instead of worker, such as the join
counters (join, the join's name) and the checkpoint store gauges (none).
Current workers only. When a partition moves to another node, the node it left
keeps exporting that worker's last values: a counter cannot be removed from a scrape.
turbine_worker_running tells them apart. Keep only the current ones with:
Timing distributions have two forms. The original metric, such as
turbine_batch_duration_seconds, remains a summary with quantile lines plus _sum
and _count. Its quantiles cover a sliding window of about a minute, per worker,
and feed the console. Never sum or average those quantiles across workers.
Batch duration and repartition transit also have cumulative histograms: insert
_distribution before _seconds or _ms. For example,
turbine_batch_duration_distribution_seconds
exports _bucket{le="…"}, _sum and _count, with the same labels as the summary.
To estimate a p99 across running workers, sum bucket rates before taking the
quantile:
histogram_quantile(0.99,
sum by (le) (
rate(turbine_batch_duration_distribution_seconds_bucket[5m])
and on (instance, worker) turbine_worker_running == 1
)
)
Add your job and node selectors to both operands. Keep le in the aggregation,
and add labels such as the subscription when separate curves are needed. Rates
handle counter resets; no observations produce NaN, not zero latency. Buckets
add 24 series per label set for each of these two metrics: 22 bucket series plus
_sum and _count. They span 0.1 ms to 300 s
for durations and 0.1 ms to 24 h for hop transit, with a final +Inf bucket.
Estimates interpolate within buckets; observations above the largest finite bound
cannot yield an accurate tail beyond that bound.
A mean over any set of workers is sum(rate(…_sum[5m])) / sum(rate(…_count[5m])),
with the same filters on both sides. Use either the summary or its distribution,
never sum the two forms together: they describe the same observations.
Gauges only the leader measures. turbine_liveness_excluded_nodes,
turbine_partitions_awaiting_rebalance and turbine_cluster_refused_nodes are 0 on every node but the Raft leader: read
them with max().
Rates are better computed by Prometheus. The _10s and _60s gauges are rates
that Turbine computes for the console. In queries, prefer rate() over the matching
_total counter, which survives a missed scrape.
Event counters start at zero. Prometheus sees no increase in a counter whose
series first appears at 1, so the counters of rare events exist at 0 from the start
of what they count: worker restarts and fatal halts, exactly-once replays and
checkpoint failures from the worker's start, late and overflowing rows from the join's
or the window's, refused API calls from the API's. Their first event shows in
increase() and rate(). Other series appear with their first value, such as the DLQ
metrics: to show 0 rather than no data, end the query with or vector(0).
Metrics reference¶
Units follow the suffix of the name (_seconds, _ms, _bytes); anything else is a count. The histogram rows export rolling summaries; batch duration and repartition transit also export cumulative _distribution histograms as described in Reading the metrics.
Process CPU and memory¶
On Linux, each /metrics scrape samples the serving process, including Python,
Rust, native libraries and all its threads. These series have no worker or
subscription labels; Prometheus adds job and instance. They survive application
restarts within the same host process. CPU resets when the process restarts.
| Metric | Type | Description |
|---|---|---|
process_cpu_seconds_total |
counter | Cumulative user and system CPU seconds, excluding child processes. rate(process_cpu_seconds_total[5m]) gives CPU cores used: 1 means one fully busy core; it can exceed 1 |
process_resident_memory_bytes |
gauge | Current resident set size (RSS), including shared resident pages; Linux reports an approximate value |
process_virtual_memory_bytes |
gauge | Virtual address-space size, including reserved and mapped regions; not physical memory consumed |
Dashboard version 4 adds Process resources without requiring cAdvisor. Older nodes contribute no process series. A memory limit and the container working set remain separate cAdvisor measurements in Pod resources: RSS is neither a container total nor a quota, and CPU cores are not normalized to the CPU quota. Processes outside Linux, or Linux scrapes where process measurements cannot be read, omit these series rather than reporting zero or retaining an old sample. Other Turbine metrics remain available.
Throughput & batches¶
| Metric | Type | Description |
|---|---|---|
turbine_messages_total |
counter | Total messages processed |
turbine_batches_total |
counter | Total batches processed |
turbine_throughput_10s |
gauge | Messages/second over the last 10s |
turbine_throughput_60s |
gauge | Messages/second over the last 60s |
turbine_repartition_records_total |
counter | Records re-emitted by a repartition hop, labeled by destination partition — compare partitions to spot key skew |
turbine_repartition_null_keys_total |
counter | Repartitioned rows whose grouping column was null (routed to the shared empty-key group) |
turbine_side_output_records_total |
counter | Records written to each side-output destination (labels: worker, destination — the side_outputs key or the Branch name), for branch= arms and ctx.emit alike. See Side outputs |
Latency¶
| Metric | Type | Description |
|---|---|---|
turbine_poll_seconds |
histogram | Time waiting for messages from the broker (front stage — pipelined, it overlaps the previous batch's processing) |
turbine_decode_seconds |
histogram | JSON → Arrow deserialization time (front stage) |
turbine_process_seconds |
histogram | User business logic time |
turbine_produce_seconds |
histogram | Encoding + sending to the output topic |
turbine_commit_seconds |
histogram | Writing state + offset to RocksDB |
turbine_batch_duration_seconds |
histogram | Back-stage elapsed time (process → produce → commit for a stateful worker; key extraction → produce → commit for a repartition hop). Excludes input polling, decoding and empty polls. Pipelined, poll+decode of the next batch overlap this — the un-hidden remainder is turbine_front_wait_seconds |
turbine_front_wait_seconds |
histogram | Time the worker waited for its next batch, including poll and decode. With a pipelined front, only the wait not hidden by the previous batch is counted: ≈ 0 when processing dominates, approaching poll + decode when the front is the bottleneck. On a serial loop, including replay, subscriptions with several inputs and repartition hops, counts the whole poll + decode. Not recorded on empty polls (an idle partition waits batch_timeout by design) |
turbine_pipeline_front_active |
gauge | The pipeline gate's current decision: 1 = pipelined front armed, 0 = serial loop. Moves when the adaptive gate (TURBINE_PIPELINE, default adaptive) switches modes; constant under a forced 0/1. The gate's decision, strictly speaking — a subscription with several inputs, or a consumer without a FrontPoller, runs serial whatever the gauge says |
turbine_repartition_transit_ms |
histogram | Repartition hop transit: re-emit wall time minus the freshest source timestamp in each batch. One observation per batch with a timestamp. Near zero when the hop keeps up; grows when it falls behind |
Lag & watermarks¶
| Metric | Type | Description |
|---|---|---|
turbine_consumer_lag |
gauge | Consumer lag (messages behind high watermark) |
turbine_consumer_lag_growth_state |
gauge | Shared lag-growth state: -1 unknown, 0 stable/below threshold/draining, 1 warning, 2 growing (label: worker). Use only before the published validity deadline (at read/scrape time) and for running workers |
turbine_consumer_lag_growth_valid_until_seconds |
gauge | Unix time in seconds when the last lag-growth observation expires (label: worker); 0 on failed observations or worker stop |
turbine_event_time_watermark_ms |
gauge | Event-time watermark per partition, epoch ms (event-time subscriptions only) |
turbine_event_time_watermark_lag_ms |
gauge | Wall clock − watermark, refreshed ~10 s — keeps growing on a silent stream |
turbine_event_time_null_total |
counter | Rows with missing/unparsable event time, dropped from event-time bucketing |
turbine_input_consumer_lag |
gauge | Messages behind the high watermark for each named input of a multi-input subscription (labels: worker, input, topic), refreshed ~10 s. turbine_consumer_lag carries their sum; this one says which input is behind. Named inputs only |
turbine_lag_skipped_messages_total |
counter | Messages deliberately skipped because a worker started further behind than max_lag_seconds (label: worker). Checked when a worker starts or restarts, not while it runs; every increment is data that was never processed. See Configuration |
State¶
| Metric | Type | Description |
|---|---|---|
turbine_state_ops_total |
counter | Total state operations committed (user puts/deletes + offset) |
turbine_state_ops_10s |
gauge | State operations/second over the last 10s |
turbine_state_ops_60s |
gauge | State operations/second over the last 60s |
turbine_state_keys_expired_total |
gauge | Expired stored entries removed by compaction since the state store opened (label: worker). Read back as a running total from the storage engine's background threads; resets when the store reopens. An old stored version can be counted after the active sweep has already deleted its logical key |
turbine_state_keys_expired_by_sweep_total |
counter | State keys removed by the optional active TTL sweep (labels: worker, scope; (ref) for reference values without an explicit scope). Only with TURBINE_TTL_SWEEP_MS > 0. The dashboard sums the compaction and sweep rates |
turbine_state_expiry_frontier_ms |
gauge | How far state expiry has advanced, epoch ms (state_ttl subscriptions only) |
turbine_state_keys_expired_idle_total |
counter | Of the keys the active state_ttl sweep removed, those that expired only because the partition went quiet and its clock coasted past them — by the data's own timestamps they were still alive (label: worker). A subset, not an extra outcome; 0 on a busy partition. Only with TURBINE_TTL_SWEEP_MS > 0. See State lifetime |
turbine_state_sweep_lag_ms |
gauge | How long the active state_ttl sweep took to walk the partition's whole state once, ms (label: worker) — roughly how long an expired key can keep its disk space before the sweep reaches it (it is invisible to reads either way). Set at the end of each complete pass; only with TURBINE_TTL_SWEEP_MS > 0, see Environment variable summary |
turbine_state_ttl_keys |
gauge | Keys under a state_ttl prefix that the last complete sweep pass found still alive (label: worker) — the population to read the expiry counters against. Only with TURBINE_TTL_SWEEP_MS > 0 |
turbine_state_cache_entries |
gauge | Entries held by the worker's caches of recently read and written state values (label: worker), summed over its threads. Bounded by TURBINE_STATE_CACHE_SIZE per thread; refreshed every 10 s |
turbine_state_cache_bytes |
gauge | Bytes of keys and values those caches hold (label: worker), without the memory the cache's own bookkeeping takes. The figure to size TURBINE_STATE_CACHE_SIZE from, once the state has grown: a cache that is full stays full |
turbine_block_cache_usage_bytes |
gauge | Cached state blocks plus reservations for state write buffers. One cache per process (no worker label), shared by all partitions. Reservations may shrink later than actual buffer usage; do not add turbine_memtable_usage_bytes to this value. Pinned blocks and in-flight work can exceed the soft capacity |
turbine_block_cache_capacity_bytes |
gauge | Shared soft budget for cached blocks and write buffers, TURBINE_BLOCK_CACHE_MB or by default a quarter of the process's memory limit (no worker label). See Memory |
turbine_memtable_usage_bytes |
gauge | State write-buffer arena allocations across all partitions (no worker label), already accounted for through cache reservations |
turbine_memtable_budget_bytes |
gauge | Shared write-buffer flush target, half the cache capacity (no worker label). A writer reaching this target completes a local flush before returning; concurrent batches can overshoot it |
The dashboard sums the compaction and sweep rates as TTL reclamation activity. Several removal events can refer to the same logical key: a sweep deletion can leave an older stored version for compaction to remove later. Read this rate as storage cleanup activity, not as a count of distinct logical expirations.
Resident fixed-window state¶
Resident mode charges open state, staged batches and complete-image capture to the same resident allowance. Leave capture headroom when sizing the application; a state that fits during ingestion can still refuse its checkpoint. See Resident storage.
| Metric | Type | Description |
|---|---|---|
turbine_resident_budget_bytes |
gauge | Process-wide resident allowance, one quarter of the resolved storage budget; no labels |
turbine_resident_reserved_bytes |
gauge | Process-wide reservations for resident state, pending batches and capture; no labels, not RSS |
turbine_resident_memory_bytes |
gauge | Accounted memory for one operator, including staged changes (labels: worker, window) |
turbine_resident_groups |
gauge | Committed group entries across retained panes, so a group present in two panes counts twice (labels: worker, window) |
turbine_resident_panes |
gauge | Committed temporal panes retained by the operator (labels: worker, window) |
turbine_resident_open_windows |
gauge | Committed owning windows awaiting retirement (labels: worker, window); sliding windows can share panes |
turbine_resident_snapshot_bytes |
gauge | Payload size of the last complete IPC image, excluding its binding header (label: worker) |
turbine_resident_snapshot_capture_seconds |
histogram | Synchronous capture of the coherent image and RocksDB checkpoint, in seconds (label: worker); excludes background upload |
Windows & timers¶
| Metric | Type | Description |
|---|---|---|
turbine_active_windows |
gauge | Persistent window panes awaiting close (label: worker, 10 s rolling max). Counts close timers only; excludes early emissions, join expiries and user timers. Includes future closes, so a high value alone does not mean closing is behind |
turbine_due_timers |
gauge | Timers due right now, both clocks — the close backlog. 0 between boundary bursts on a worker that keeps up; persistent due work reveals close debt that input lag alone can miss |
turbine_oldest_due_timer_age_seconds |
gauge | Oldest due age: processing-clock overdue age or event-time watermark lateness, not real waiting. /health reports close_backlog when continuously observed real debt reaches TURBINE_CLOSE_BACKLOG_THRESHOLD_MS (60 s default). A fresh backfill jump alone does not degrade health |
turbine_timer_service_seconds |
histogram | Wall time of a timer-service turn (the cooperative 5–250 ms slice that fires due timers between input batches). Recorded only for turns that fired at least one committed chunk. Priority-drain turns ignore the slice, so they show up here as long turns |
turbine_timer_service_exhausted_total |
counter | Timer-service turns that returned with due work left because their time slice was spent. A burst at a window boundary is normal; inspect persistent due work and input lag over complete window cycles to identify sustained overload |
turbine_timer_service_priority_total |
counter | Turns that prioritized draining before input ingestion, after one real second of continuously observed due work or at the independent 60 s timer-age safety gate. This can rise during healthy boundary bursts; it is not a degradation counter |
turbine_close_pop_seconds |
histogram | Wheel pops (both clock domains) for one fired close chunk. Non-empty chunks only — an idle worker would bury the boundary turns under near-zero samples |
turbine_close_dispatch_seconds |
histogram | Processor side of a close chunk: the on_timer batch dispatch — pane reads/deletes/decode, merge+finalize, on_close, encode. The slice the input-lane histograms never see (with turbine_close_pop/produce/commit, the chunk decomposition; the turn total is turbine_timer_service_seconds) |
turbine_close_produce_seconds |
histogram | Produce side of a close chunk: sending the closed panes, the producer flush, the combine partials/markers drain |
turbine_close_commit_seconds |
histogram | State commit of a close chunk: window/timer deletes + offset write (commit_timer_batch under exactly-once) |
turbine_timer_sync_seconds |
histogram | Worker timer-index synchronization after a successful state commit (label: worker); separate from turbine_commit_seconds |
turbine_timer_sync_ops_total |
counter | Worker timer-lane operations synchronized after successful commits, including puts, deletes and dormant/emit-flag records (label: worker); repeated arms and puts whose identical write was omitted still count |
turbine_timer_writes_elided_total |
counter | Identical active event-time timer writes omitted from successful state commits (label: worker); registrations still synchronize the scheduler after commit. A failed commit disables this optimization until timer recovery |
turbine_active_windows_now |
gauge | Current persistent window close timers (label: worker), the raw sample behind turbine_active_windows. Excludes early emissions, join expiries and user timers |
turbine_active_timers |
gauge | Current persistent timers (labels: worker, kind, whose values are window, join, user). Window timers include close and early emission deadlines; join timers count expiry buckets, each of which may hold many rows. Sum across kinds for the total |
turbine_timer_fired_total |
counter | Timers fired by the timer service, both clocks — window closes, early fires, your own timers, interval-join expiries — counted once their state commit succeeds (label: worker). Its rate is the close throughput to set against turbine_due_timers |
turbine_recovered_due_timers_total |
counter | Timers already due when a worker restarted and rebuilt its timers from state (label: worker). Read it by delivery mode: under at-least-once those timers replay and may re-emit output, so a spike is worth an alert; under exactly-once they fire without re-emitting, so the same spike is normal recovery. See Timer output after a hard crash |
Joins¶
| Metric | Type | Description |
|---|---|---|
turbine_join_buffered_rows |
gauge | Committed retained rows per side (labels: worker, join, side="left" or "right"), including matched rows waiting for expiry. Restored from state after restart or reassignment. Failed transactions do not alter the count; retiring rows decreases it after commit. Buffer size and expiry-timer count measure different things |
turbine_join_matches_total |
counter | Joined rows an interval join emitted on a match (label: join, the subscription name; no worker label — one series for all the node's partitions). With emit="first", at most one per left row. Like every join counter, it counts processing attempts: a batch that fails can count before its transaction commits |
turbine_join_unmatched_rows_total |
counter | Left rows a how="left" join emitted unmatched at expiry, with null right columns (label: join; no worker label). Stays at 0 for an inner join, which discards its unmatched rows |
turbine_join_ignored_matches_total |
counter | Further matches an emit="first" join counted and ignored because their left row had already been emitted (label: join; no worker label) |
turbine_join_duplicate_rows_total |
counter | Repeated left rows suppressed by dedupe_left=True while their original identity remains buffered (labels: join, side="left"; no worker label). Identity uses all on columns and the canonical event time; the first accepted payload wins. Counts processing attempts, so failed transactions can count again on retry. See Interval joins |
turbine_join_overflow_total |
counter | Rows refused because their equality key already buffered max_rows_per_key rows on that side (labels: join, side; no worker label). Each increment halts the partition: raise the limit and restart to resume |
turbine_join_late_rows_total |
counter | Rows behind the join's tolerance (allowed_lateness_ms) that could neither match a retained row nor be buffered, so were dropped — or sent to late_data under on_late="route" (labels: worker, join, side). In a left join, a late right row that still resolves a retained prediction is not counted |
turbine_join_input_skew_ms |
gauge | Event-time gap between the most and the least advanced input's data watermark on this partition, ms (label: worker), refreshed ~10 s. Reported for every subscription with named inputs, join or not. A large value means one input runs behind and holds the common frontier back — see Progress and silence |
Late data¶
| Metric | Type | Description |
|---|---|---|
turbine_late_rows_dropped_total |
counter | Rows dropped because their event-time window had already closed (past window_end + allowed_lateness_ms), under on_late="drop", the default (labels: worker, window — the window's name=). Alert on it: a growing value means allowed_lateness_ms is smaller than your real out-of-orderness. See Time model |
turbine_late_rows_routed_total |
counter | The same late rows, published to late_data under on_late="route" instead of dropped (labels: worker, window). Switching policy reads as one series going quiet and the other waking up. See Keeping late rows instead of dropping them |
turbine_late_rows_idle_closed_total |
counter | Of the late rows above, dropped or routed, those rejected only because the partition went quiet and window closing coasted past them on the wall clock (labels: worker, window). A subset, not a third outcome: non-zero means part of what you lose depends on your traffic pattern rather than on the data's timestamps |
Checkpoints & snapshots¶
| Metric | Type | Description |
|---|---|---|
turbine_last_checkpoint_timestamp_ms |
gauge | Wall-clock time of the last uploaded checkpoint |
turbine_last_checkpoint_offset |
gauge | Consumer offset in the last uploaded checkpoint. In cluster mode, read the Raft pointer for the latest approved offset |
turbine_last_checkpoint_size_bytes |
gauge | On-disk size of that checkpoint |
turbine_last_checkpoint_upload_seconds |
gauge | How long that checkpoint's upload took. Uploads are incremental (only the files written since the previous checkpoint), so this is also about what a partition's final checkpoint costs when its pod stops — the number to size a termination grace period from, see Sizing the shutdown |
turbine_checkpoint_store_shared |
gauge | 0 when the checkpoint destination is local to this node. Alert on it together with cluster mode: a stateful cluster on a node-local store loses state on every partition handoff |
turbine_checkpoint_store_writable |
gauge | 0 when the store's last answer was a failure. Kept current by real uploads, and by a small probe when uploads go quiet (an idle partition never checkpoints) |
turbine_checkpoint_consecutive_failures |
gauge | Store failures since the last success. Past three, /health reports degraded |
turbine_checkpoint_failures_total |
counter | Cumulative checkpoint failures by step (operation: checkpoint, snapshot, upload, prune, download, coherence; probe for the store probe, with worker="-") |
turbine_snapshot_pointer_hit_total |
counter | Worker starts that found the cluster's record of their partition's latest approved checkpoint, so recovery can fetch that checkpoint without listing the store (label: worker). Counted on every stateful start, whether or not local state made a download necessary. Cluster mode only |
turbine_snapshot_pointer_miss_total |
counter | Starts where the record named a checkpoint unavailable in the store (label: worker). Ordinary cluster recovery tries the other Raft-approved checkpoints after enrollment. A brand-new partition without a pointer counts neither a hit nor a miss. Cluster mode only |
turbine_snapshot_promote_failures_total |
counter | Checkpoint uploads whose promotion was refused or could not be confirmed by Raft (label: worker). The turbine_last_checkpoint_* gauges describe uploads. The cluster pointer identifies the latest approved checkpoint. Cluster mode only |
turbine_snapshot_pointer_age_seconds |
gauge | Time since each partition's latest checkpoint was recorded in the cluster (labels: sub_index, topic, partition; no worker label), refreshed every 10 s. Every node reports every partition, so aggregate with max by (topic, partition). The older it is, the more a crash has to replay; it also grows on a partition with nothing to commit, which takes no new checkpoint. Cluster mode only |
Cluster & workers¶
| Metric | Type | Description |
|---|---|---|
turbine_worker_kind |
gauge (info) | 1 while the worker runs, 0 after exit, carrying the worker's nature (kind="stateful", "stateless" or "repartition") and declared subscription name. Join the name onto worker-labeled series with group_left(subscription); the stable subscription index remains in worker |
turbine_worker_running |
gauge | 1 while the node runs this worker, 0 once it stops (partition revoked, or halted by a fatal error). Per-worker series are never removed from a scrape — a counter cannot be — so this is what separates a worker's last values from its current ones |
turbine_worker_stopped_at_ms |
gauge | Wall-clock time the worker stopped, 0 while it runs. Answers "since when": a worker that stopped seconds ago is a rebalance in flight, one that stopped an hour ago is history |
turbine_worker_restarts_total |
counter | Supervisor restarts of a worker. A steadily growing value is the crash-loop signature — alert on it: throughput at zero alone cannot distinguish a crash-loop from an idle topic |
turbine_raft_has_leader |
gauge | 1 when this node sees a Raft leader, 0 when it does not (no quorum → nothing can be assigned or rebalanced). Cluster mode only |
turbine_liveness_excluded_nodes |
gauge | Cluster nodes currently left out of partition assignment because they stopped answering. Only the leader can measure this; every other node reports 0, so read it as max() across nodes. Cluster mode only |
turbine_cluster_refused_nodes |
gauge | Current cluster members whose last startup refusal is retained. A node stops contributing after successful admission or removal from membership; removal preserves its refusal history. Followers report 0; use max() across nodes and alert on > 0. Cluster mode only |
turbine_handoff_barrier_waiting |
gauge | 1 while a partition (labels: sub_index, topic, partition) waits for its previous owner to hand over. Cluster + exactly-once only |
turbine_handoff_barrier_waiting_seconds |
gauge | How long that wait has lasted. Seconds is normal during a rebalance; minutes means it is stuck and needs manual recovery |
turbine_partitions_awaiting_rebalance |
gauge | Partitions still assigned to cluster nodes that stopped answering: nobody consumes them until the node comes back or raft_rebalance_delay_ms expires. Alert on > 0 for longer than the rebalance delay. Only the leader can measure this; every other node reports 0, so read it as max() across nodes. Cluster mode only. See When a node goes down |
Exactly-once¶
| Metric | Type | Description |
|---|---|---|
turbine_eos_state_gap_events |
counter | Size of the state gap found when a worker starts — events the broker has committed but local state never saw — added whichever on_crash_recovery policy handles it (label: worker; a multi-input subscription adds input). Exactly-once only |
turbine_eos_state_gap_skipped_events |
counter | Events of that gap whose state updates were skipped, and lost for good, under on_crash_recovery="accept" (label: worker). Alert on any increase. Exactly-once, single-input subscriptions only. See Accept the gap |
turbine_eos_state_gap_halt_total |
counter | Worker starts refused because the gap exceeded halt_if_gap_exceeds; the partition halts with a fatal error (label: worker). Exactly-once, single-input subscriptions only. See Circuit-breaker |
turbine_eos_replay_started_total |
counter | Silent replays started when a worker found a gap under on_crash_recovery="replay", the default (label: worker). The partition produces nothing until the matching turbine_eos_replay_completed_total ticks. Exactly-once, single-input subscriptions only. See Default: replay the gap |
turbine_eos_replay_completed_total |
counter | Replays that caught up with the broker's committed position, at which point outputs are enabled again (label: worker). Started minus completed is the number of partitions still replaying |
turbine_eos_residual_window_detected_total |
counter | Deprecated alias: +1 per worker start that found a state gap, whatever its size (label: worker). Use turbine_eos_state_gap_events instead |
turbine_eos_receipt_keepalive_total |
counter | Empty transactions that re-commit the partition's sink receipt from an idle poll, so the broker cannot expire its commit history (label: worker): on the first idle poll after the worker starts, then at most once per TURBINE_EOS_RECEIPT_KEEPALIVE_SECS (24 h default). Exactly-once only. See Timer output after a hard crash |
turbine_eos_receipt_reenrolled_total |
counter | Partitions that adopted the broker receipt under TURBINE_EOS_RECEIPT_REENROLL, accepting that timers in the gap may fire again and duplicate output (labels: topic, partition; no worker label). Each adoption is also a loud WARN; unset the variable once the app runs again |
Errors & DLQ¶
| Metric | Type | Description |
|---|---|---|
turbine_dlq_messages_total |
counter | Dead-lettered / dropped records per worker, with the actual input topic (labels: worker, topic, partition, phase, action) |
turbine_dlq_messages_60m |
gauge | Dead-lettered / dropped records over the last 60 min, counted once per worker across all inputs (labels: worker, topic, partition). The topic label is the subscription's anchor topic, including when a different input was rejected |
turbine_processing_errors_consecutive |
gauge | Batches this worker's handler has failed in a row while skip / dlq kept it running; back to 0 on the first one that goes through. Alert on it: nothing else says the partition has stopped producing results — it is not halted and its offsets keep advancing. Past three, /health reports degraded |
turbine_halted_workers |
gauge | Partitions currently stopped by a fatal halt. Alert on > 0: throughput on the remaining partitions hides the hole otherwise |
turbine_worker_fatal_total |
counter | Fatal worker halts, cumulative per worker. The halt is contained to that partition — the app keeps running |
turbine_compute_errors_total |
counter | Rows an expression kernel nulled under on_compute_error="null", the default (label: op, the kernel that failed — divide, cast, parse_ts, …; no worker label: one series per kernel for the whole node). A spike means bad upstream data. See Compute errors |
turbine_branch_rows_dropped_total |
counter | Rows matching no branch= arm, discarded because the subscription declared on_unmatched="drop" (label: worker). The series only exists when you asked for the loss, so it waking up always means data is leaving the pipeline. See Saying what happens to the rest |
Inference¶
| Metric | Type | Description |
|---|---|---|
turbine_inference_rows_total |
counter | Rows scored by the subscription's infer= model (label: worker). See Inference |
turbine_inference_seconds |
histogram | Time spent scoring one output batch — building the feature matrix and running the model (label: worker). Scoring happens as results are produced, so this time is also inside turbine_produce_seconds (turbine_close_produce_seconds for window closes) |
turbine_inference_session_loads_total |
counter | Model sessions opened, one per partition on its first scored batch (label: worker). A number that keeps climbing means partitions keep moving or workers keep restarting |
combine_by¶
| Metric | Type | Description |
|---|---|---|
turbine_combine_partial_messages_total |
counter | Partial aggregate states the ingesting side of a combine_by subscription sent to the internal topic, one per key per window at each close (label: worker). Compare with turbine_messages_total to see what the combine saves on the wire |
turbine_combine_marker_messages_total |
counter | Progress confirmations the ingesting side sent to the internal topic, telling the merge stage that this partition has shipped everything up to its watermark (label: worker). Sent at window-boundary cadence, including from idle partitions |
turbine_combine_late_partials_total |
counter | Partials the merge stage received for a window it had already closed, dropped rather than merged into a duplicate pane (label: worker, the merge-stage worker). At-least-once redeliveries; stays 0 under exactly-once |
turbine_combine_min_watermark_lag_ms |
gauge | Wall clock minus the lowest watermark confirmed across all source partitions, as seen by the merge stage, ms (label: worker) — how late merged results are. A frozen or steadily growing value names a source partition that stopped confirming, e.g. one whose records are all dropped by your error policy. Appears once every source partition has reported; updated on each merge-stage batch |
Other¶
| Metric | Type | Description |
|---|---|---|
turbine_partition_key_placement |
gauge | What the records of this partition say about the producer's placement, sampled from every batch: -1 a record sits where murmur2 would not have put it (durable — stops the startup cleanup and turbine-cli rescale, declared or not), 1 enough distinct keys sit where murmur2 puts them (a diagnostic, authorises nothing), 0 nothing to say yet (new app, few distinct keys, or the 10-minute window that follows a partition-count change). Keyed stateful subscriptions on more than one partition only (label: worker). See Producer placement |
turbine_api_unauthorized_total |
counter | API requests refused with 401 because they carried no token or the wrong one (no labels, not even worker). Only moves when a token is set; a steady climb usually means a node or a scraper still holds an old token. See Securing the API |