Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Metrics And Observability

Nervix exposes runtime graph metrics in two forms:

  • raw Prometheus metrics at the observability server’s /metrics endpoint
  • owner-routed summarized metrics in DESCRIBE command output

The two surfaces use the same semantic labels where they overlap, but they use separate storage. Prometheus receives raw counters and histogram buckets through a Prometheus registry for external aggregation. DESCRIBE reads Nervix runtime state and includes derived values such as rates and recent percentiles. Relay descriptions are routed to the scheduled relay owner; nonowners do not maintain relay metrics.

Errors And Diagnostics defines the error information that may be reported and its sensitivity and log-level rules. This chapter defines metric surfaces and labels.

Observability Server

The observability listener exposes:

  • /livez: process liveness
  • /readyz: readiness once a leader is known
  • /metrics: Prometheus text output for graph, interconnection, and allocator metrics

Use the node’s observability address, not the data-plane HTTP listener:

curl http://127.0.0.1:<observability-port>/metrics

Prometheus metrics are intentionally branch-aggregated. They aggregate across concrete relay branches and do not include a branch key label. This keeps Prometheus cardinality bounded when a relay is branched by high-cardinality values such as tenant, user, account, or device id. Branch lifecycle metrics identify the declared branch, not the concrete key.

Metric Labels

Graph metric series use these labels:

  • domain: owning domain
  • target_kind: runtime target kind, such as RELAY, INGESTOR, JUNCTION, DEDUPLICATOR, REINGESTOR, WINDOW_PROCESSOR, EMITTER, or LOOKUP
  • target: relay or node name
  • physical_node_id: Nervix cluster node where the metric was observed
  • direction: received or sent
  • stream: logical relay associated with the observation, or - when no relay applies
  • peer_kind and peer: relay peer labels for node-to-relay observations, or - when no peer applies
  • branch: named branch declaration on branch lifecycle metrics
  • reason: branch eviction reason, either lru or ttl
  • le: Prometheus histogram bucket boundary

For a single-input emitter, sent metrics retain that input relay as stream. A multi-input emitter’s received metrics identify the actual source relay, while its sent metrics aggregate the shared sink pipeline with stream="-" because one flush may contain work from several sources.

An emitter’s sent metrics count the records its destination delivered, each once, however many attempts it took: a flush that completes after retries counts its delivered records once, and a record the destination refused follows ON MESSAGE ERROR and is not sent. When a flush fails for good, the records its sink delivered before the failure are counted as its other records are routed to their error policy. A sink that publishes on its own commit, such as Iceberg, counts what each commit publishes. Sent bytes measure a delivered record’s payload as the emitter’s buffer measures it; an HTTP request’s method, target and headers are not payload, and a request without a body carries none. See HTTP inspection and metrics.

DESCRIBE output uses the same concepts but renders physical_node_id as physical_node for readability.

Example Prometheus series:

nervix_messages_total{domain="prod",target_kind="RELAY",target="notifications",physical_node_id="node-1",direction="received",stream="notifications",peer_kind="-",peer="-"} 42

Example DESCRIBE INGESTOR edge metric section:

metrics:
  outgoing_edges:
    messages_total sent relay=notifications physical_node=node-1 total=42 wall_rate_per_sec=12.5 domain_rate_per_sec=10 wall_rate_ema_1m_per_sec=11.2 wall_rate_ema_15m_per_sec=8.7 domain_rate_ema_1m_per_sec=9.8 domain_rate_ema_15m_per_sec=7.4

Raw Metrics

Nervix records these raw metric families:

  • nervix_messages_total: total messages received or sent
  • nervix_batches_total: total batches received or sent
  • nervix_bytes_total: total bytes received or sent
  • nervix_messages_per_batch: histogram of message count per batch. Its finite Prometheus bucket boundaries are 1, 2, 5, 10, 50, 100, 500, 1000, 1024, 2048, 4096, 8192, 16384, 32768, and 65536, followed by +Inf. The rolling HDR histograms used by DESCRIBE track batch sizes through 65,536 messages.
  • nervix_delivery_latency_seconds: histogram of delivery latency between graph nodes
  • nervix_relay_buffer_len: histogram of runtime relay buffer occupancy in queued batches
  • nervix_branch_instances: current concrete branch keys with at least one runtime instance on the physical node
  • nervix_branch_evictions_total: total concrete branches evicted on the physical node, split by reason="lru" or reason="ttl"
  • nervix_ingestor_quiesce_buffered_records: raw payloads currently retained in an ingestor’s per-instance quiesce buffers, including a payload that is draining until its messages enter their ingest group
  • nervix_ingestor_quiesce_buffered_bytes: raw payload bytes currently retained in those buffers
  • nervix_ingestor_quiesce_dropped_total: payloads deliberately discarded by DROP, buffer overflow, memory-pressure zero-capacity behavior, or an interrupting termination
  • nervix_ingestor_quiesce_rejected_total: endpoint requests or connections refused while an ingestor cannot accept them
  • nervix_ingestor_unfolding_refused_total: payloads a source handed over without an acknowledgement that an ingestor did not deliver because the node’s extension workers had no room to unfold them, for a source whose transport would not survive a held loop; see When The Extension Workers Are Full
  • nervix_ingestor_transport_redeliveries_total: pending acknowledgements and rejections an ingestor discarded after its RabbitMQ, SQS or MQTT source reconnected. The broker redelivers what the lost connection did not acknowledge, so each counted position is ingested again
  • nervix_session_subscriptions: open session subscriptions the node delivers from a relay. The node advertises interest in the relay to the cluster exactly while this is above zero, and the series stays at 0 once the relay’s last subscription on the node closes
  • nervix_session_subscription_dropped_rows_total: rows DROPPING session subscriptions on the node discarded because their session could not take them in time
  • nervix_client_ingestor_producers: producers attached to a client ingestor the node executes
  • nervix_client_ingestor_forwarded_producers: those of them whose sessions another node serves
  • nervix_client_ingestor_outstanding_batches: batches those producers submitted that have no outcome yet
  • nervix_client_ingestor_outstanding_bytes: the Arrow IPC bytes of those batches
  • nervix_client_ingestor_admitted_batches: batches holding a slot of the ingestor’s acknowledgement window
  • nervix_client_ingestor_submissions_total: batches the ingestor answered, by outcome and cause
  • nervix_jemalloc_active_bytes: bytes in active allocator pages
  • nervix_jemalloc_allocated_bytes: bytes allocated by the process
  • nervix_jemalloc_mapped_bytes: bytes mapped by active allocator extents
  • nervix_jemalloc_metadata_bytes: bytes dedicated to allocator metadata
  • nervix_jemalloc_resident_bytes: resident data-page bytes mapped by the allocator
  • nervix_jemalloc_retained_bytes: retained virtual-memory mapping bytes

Histograms follow Prometheus conventions and include _bucket, _sum, and _count series. Current bucket boundaries are:

  • messages per batch: 1, 2, 5, 10, 50, 100, 500, 1000, +Inf
  • delivery latency seconds: 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 30, +Inf
  • relay buffer length: 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, +Inf

Prometheus receives raw values only. The /metrics endpoint is encoded by the Prometheus client registry, not by Nervix internal summary state. Prometheus should compute external queries, alerts, and dashboards with normal PromQL aggregation.

The four ingestor-quiesce families, nervix_ingestor_unfolding_refused_total and nervix_ingestor_transport_redeliveries_total use domain, ingestor, and physical_node_id labels. Buffer families are gauges; dropped, rejected, refused and redelivered families are monotonic counters. They are process-local: quiesce buffers do not migrate during termination or failover.

Quiesce, refusal and redelivery children are registered when the ingestor starts, and DESCRIBE INGESTOR registers the redelivery child at zero on the node that runs the ingestor. Client-ingestor outcome children are registered when its endpoint starts, and subscription loss counters when delivery starts. These bounded series may therefore appear at zero before their first observation. Their labels and counting contracts are unchanged.

The two session-subscription families use domain and relay labels and describe the node that exports them: the subscriptions its sessions hold and the rows those sessions lost. A client is told of its own losses directly, as described in Sessions. The server exports no other session metrics; Client Session Protocol describes what clients and operators observe instead.

Client Ingestors

The six client-ingestor families describe the node that executes the ingestor and use domain and ingestor labels. The first five are gauges, rewritten whenever the ingestor’s endpoint changes them, and they are 0 once the endpoint ends. admitted_batches never exceeds the window the ingestor declares, however many producers are attached.

nervix_client_ingestor_submissions_total is a counter with two more labels drawn from fixed sets. outcome is not_admitted, completed, processing_failed, or outcome_unknown. cause is the refusal of a batch that was not admitted (invalid_batch, suspended, busy, draining, producer_ended, credit_exceeded), the failure of one that failed (ack_timeout, rejected), the uncertainty of one whose outcome is unknown (interrupted, owner_lost), or none for a completed batch. No label carries a payload value, a producer, or an attachment identity. A batch the serving session refused before it reached the node that executes the ingestor, such as one beyond its producer’s credit, is answered by the session and not counted here. So is every batch the node that forwards a producer answers itself once it lost the node that executes the ingestor: a batch it never cleared for admission as not_admitted with producer_ended, and a cleared one as outcome_unknown with owner_lost. Ingestors describes the same counts in SHOW INGESTORS and DESCRIBE INGESTOR.

Client Emitters

Client emitter metrics use domain and emitter labels on the executing node. The gauges nervix_client_emitter_consumers, nervix_client_emitter_forwarded_consumers, nervix_client_emitter_forwarded_credit_bytes, nervix_client_emitter_retained_batches, and nervix_client_emitter_retained_bytes describe current in-memory attachments and prepared Arrow IPC awaiting an application decision. nervix_client_emitter_forwarded_retained_batches and nervix_client_emitter_forwarded_retained_bytes count the subset assigned to consumers served on another node. nervix_client_emitter_incomplete_batches counts assigned batches whose application decision is still pending. These gauges return to zero when that endpoint ends. The counters nervix_client_emitter_retries_total, nervix_client_emitter_acks_total, and nervix_client_emitter_rejections_total retain results across endpoint restarts on the same node. Retries include explicit application retry, ACK timeout, and consumer loss; a repeated confirmed ACK does not increment the ACK counter. DESCRIBE EMITTER reports the same values for the scheduled owner.

Restoring a client producer or consumer creates a new server attachment. The attachment gauges fall when the prior session detaches and rise only after the replacement open succeeds; retained batch and incomplete-attempt gauges remain bounded by the existing endpoint budgets. None uses a session, producer, consumer, delivery or attempt identity as a metric label. Applications can sample the Rust handles’ connection() states to distinguish an active, interrupted, restoring or reopen-required handle. An uncertain producer result and an uncertain consumer settlement remain typed call outcomes; these gauges do not imply replay or a confirmed ACK.

Delivery Latency

Every processor, reingestor, and emitter input records the delivery latency of each batch it accepts. A row’s latency runs from its ingestion high watermark to the instant the input accepts the batch. Processors and reingestors read that instant from the domain clock, so in a paced domain their latency is measured in domain time; emitters read the wall clock. A row whose high watermark is later than that instant has no latency and adds nothing to the histograms, although it still counts as a received message.

A batch is recorded as a whole. Each row’s ingestion watermarks travel with the batch in Arrow buffers of Unix-nanosecond timestamps, and one pass over the high watermarks finds the batch’s latest watermark and folds every row’s latency into buckets. Each latency series then takes the whole batch in one update: the rolling histograms behind DESCRIBE merge the buckets under one lock with one wall-clock reading, and the Prometheus histogram takes the batch in one flush.

  • The batch’s latest high watermark is its domain timestamp. It stamps the batch’s traffic counters and places its latencies in the domain-clock windows. A batch older than a window’s current step lands in its own step only while that step is still retained; an older step is never reopened.
  • The rolling histograms record each latency to the nearest millisecond, halves rounding up, and record anything above 30 seconds as 30 seconds, so a slow tail is never dropped from the percentiles.
  • The Prometheus histogram receives every latency in seconds and counts it in the boundaries listed under Raw Metrics; a latency above 30 seconds counts only in +Inf.

Interconnection Metrics

Every node also exposes its own interconnection: the authenticated HTTP/2 pools it carries internal traffic over, the execution admission every variable-size operation passes through, its consensus retention, and the reactor delay they share. These series describe the exporting node only, so they carry no node label, and they are aggregated by dimensions whose value sets are fixed: class, direction, operation, reason, and outcome. None of them carries a branch key, a peer identity, a domain, or an operation identifier.

class is the traffic class on transport series — management, commands, replication, relay, or bulk — and the execution class on admission series — management, commands, relay, bulk, credentials for memory, and control_cpu, credentials_cpu, data_cpu, extension_cpu, bulk_cpu, consensus_storage, filesystem_storage for workers. operation is the reserved request subquota: shared, append, resource, snapshot, discovery, liveness, progress, admission, cancellation, or terminal.

Transport pools:

  • nervix_interconnect_connections: physical connections held, by class and direction
  • nervix_interconnect_streams: HTTP/2 stream slots leased on outbound connections, by class
  • nervix_interconnect_pending_operations: requests admitted and not yet resolved, by direction and operation
  • nervix_interconnect_connections_established_total: connections established, by class
  • nervix_interconnect_connection_failures_total: connections that failed to establish or ended, by class and reason, where reason is setup, handshake, capacity, closed, or resolution for an attempt whose peer’s advertised host did not resolve
  • nervix_interconnect_stream_resets_total: streams that ended without a result, by class and reason, where reason is deadline, capacity, peer, shutdown, or malformed
  • nervix_interconnect_quota_failures_total: requests refused because their reserved subquota was full, by direction and operation
  • nervix_interconnect_requests_total and nervix_interconnect_request_seconds_total: typed requests completed and the round trip they took, by operation. Divide the seconds by the count for average latency; operation="liveness" is the peer health probe, so that ratio is health RTT, and operation="progress" is replaceable fenced domain-clock progress
  • nervix_interconnect_bulk_bytes_total: bytes carried by streamed bodies, by class and direction. This rises while a transfer is running, so it reads as bulk progress

Relay admission:

  • nervix_interconnect_relay_channels, nervix_interconnect_relay_attempts, nervix_interconnect_relay_grants: logical channels with unresolved work, attempts whose outcome has not been retired, and transfer grants issued and not yet spent. These aggregate bounded authenticated peer owners. Ending a peer releases its protocol capacity immediately, including when the application still borrows an admitted intake; that intake’s retained metadata stays charged to relay memory until its last borrow ends. A delivery the receiver holds back because its relay memory has no room for the delivery’s acknowledgement watches stays an unresolved attempt here, with its relay memory near nervix_execution_memory_capacity_bytes, and counts no rejection until it is admitted
  • nervix_interconnect_relay_admissions_total and nervix_interconnect_relay_admission_wait_seconds_total: attempts resolved, by outcome (admitted, rejected, cancelled), and the time they spent between an accepted reservation and their resolution
  • nervix_interconnect_unresolved_outcome_age_seconds: how long the oldest unacknowledged relay outcome has been waiting. A rising value means acknowledgements are not coming back

Execution admission:

  • nervix_execution_memory_capacity_bytes and nervix_execution_memory_reserved_bytes: the transient memory reserved for one class, and what it is currently holding
  • nervix_execution_memory_reservations_total and nervix_execution_memory_rejections_total: charges granted and charges refused outright
  • nervix_execution_workers, nervix_execution_jobs_running, nervix_execution_jobs_pending: how many jobs one class may have on the blocking pool at once, how many it holds, and how many wait
  • nervix_execution_jobs_total, nervix_execution_jobs_completed_total, nervix_execution_job_rejections_total: jobs admitted, jobs that left a worker, and jobs refused because the class already held its whole wait queue
  • nervix_execution_job_queue_seconds_total and nervix_execution_job_work_seconds_total: time admitted jobs spent waiting for a worker, and time completed jobs spent holding one

Node scheduling:

  • nervix_node_scheduler_delay_seconds_total, nervix_node_scheduler_delay_peak_seconds, and nervix_node_scheduler_samples_total: how late the node’s reactor has been polling work that was already runnable. A timer asks to be woken every 250 milliseconds; the excess it is woken by is the delay every other task on that worker paid

Consensus retention:

  • nervix_consensus_log_last_index, nervix_consensus_log_snapshot_index, nervix_consensus_log_purged_index: the highest Raft index the log holds, the highest the current snapshot covers, and the highest already removed. A position that does not exist yet reports -1
  • nervix_consensus_log_retained_bytes: encoded bytes held by the node’s current Raft log entries
  • nervix_consensus_snapshot_pinned_generations, nervix_consensus_snapshot_pinned_readers, nervix_consensus_snapshot_unreferenced_generations: snapshot generations an outgoing transfer holds against deletion, the readers those pins are held for, and generations waiting to be deleted

These series answer two operational questions directly. A class that is saturated shows a reserved byte count at its capacity, or running jobs at its worker count with pending jobs behind them, while the other classes stay free — that is the isolation the pools exist for. A control answer arriving late while bulk work runs shows up as reactor delay and as request seconds rising for the shared or liveness operation, not as a connection failure.

DESCRIBE Metrics

DESCRIBE commands include metric summaries for the described target when metrics exist. For a scheduled target, the request is answered from its owner. Node traffic is grouped under incoming_edges and outgoing_edges; relay descriptions keep owner-buffer utilization under relay_buffers rather than mixing relay traffic into the relay node:

DESCRIBE RELAY notifications WHERE (user_id = 42);
DESCRIBE INGESTOR kafka_notifications;
DESCRIBE JUNCTION route_notifications;
DESCRIBE DEDUPLICATOR dedup_txns;
DESCRIBE REINGESTOR repartition_notifications;
DESCRIBE WINDOW PROCESSOR latency_window;
DESCRIBE EMITTER kafka_notifications_out;
DESCRIBE HASH MAP user_profiles;
DESCRIBE DOMAIN;

DESCRIBE DOMAIN summarizes active-domain traffic from per-node metric state. Its input_output section aggregates ingestor and emitter metrics. Its processed section aggregates metrics for all runtime nodes in the domain, including processing nodes.

DESCRIBE EMITTER reports a status: of OK, WAITING, or ERROR, with a detail: line explaining anything other than ordinary operation. WAITING means the emitter is healthy but holds no connection because it has asked a shared client’s pool for one and has not been given it yet; see Database Client Connection Pools. A full pool is a waiting state, not a failure, so an acquisition timeout or a lost connection is reported as ERROR instead.

DESCRIBE INGESTOR also reports the declared quiesce: policy, current quiesce state:, all four quiesce values and the node’s nervix_ingestor_transport_redeliveries_total, using their Prometheus family names. A connected ingestor in any active mode is reported as status: quiesced, not stopped. Per-payload activity remains at debug or trace log levels and payload values are never logged.

Counter summaries include:

  • total: accumulated raw counter value
  • wall_rate_per_sec: total divided by wall-clock elapsed time since the local series started
  • domain_rate_per_sec: total divided by the observed domain-time span, when records carry domain timestamps
  • wall_rate_ema_1m_per_sec and wall_rate_ema_15m_per_sec: exponentially decayed wall-clock rates
  • domain_rate_ema_1m_per_sec and domain_rate_ema_15m_per_sec: exponentially decayed domain-clock rates

Histogram summaries include:

  • p50_1m, p90_1m, p99_1m: one-minute wall-clock decayed percentiles
  • p50_15m, p90_15m, p99_15m: fifteen-minute wall-clock decayed percentiles
  • domain_p50_1m, domain_p90_1m, domain_p99_1m: one-minute domain-clock decayed percentiles
  • domain_p50_15m, domain_p90_15m, domain_p99_15m: fifteen-minute domain-clock decayed percentiles

Histogram DESCRIBE lines do not include raw count / sum values or rates. For messages_per_batch, the raw observation count is the batch count and the raw sum is the message count; those are already reported clearly by batches_total and messages_total. The histogram answers distribution questions such as typical and tail batch size. Percentiles are rendered as decimal estimates interpolated within the configured histogram bucket range rather than as raw bucket boundary labels. The same rule applies to delivery latency: messages_total and batches_total answer throughput questions, while delivery_latency_seconds answers latency distribution questions.

Relay buffer summaries use relay_buffer_len. The percentile values are queued batch slots observed in the single buffer on the relay owner, and capacity=<n> shows that buffer’s declared bound. A branched relay can hold interleaved concrete branches in the same owner buffer, so DESCRIBE RELAY reports observed lengths as percentiles instead of rendering a separate current depth per branch. The fixed one-batch producer and consumer-node dispatch slots are not additional relay-buffer metric series.

A - value means the derived value is not available. This is common for domain-clock values when no domain timestamp has been observed or when the observed domain-time span is zero.

Branch Lifecycle Signals

Prometheus exposes branch lifecycle state without a concrete branch-key label:

  • nervix_branch_instances{domain,branch,physical_node_id} is a gauge of live concrete branch keys on that physical node. A key is counted once even when multiple local graph nodes hold runtime state for it.
  • nervix_branch_evictions_total{domain,branch,physical_node_id,reason} counts each concrete key once when branch eviction begins on that physical node, even if multiple local graph nodes hold the key. reason="lru" is a MAX INSTANCES ... EVICT LRU capacity eviction; reason="ttl" is idle expiration.

Normal shutdown, schedule replacement, and runtime detachment reduce the live gauge but do not increment the eviction counter. Relay branch presence and its TTL or LRU decisions exist only on the relay owner, so a relay-owner failure loses those live values. A schedule replacement reduces state only for runtime nodes whose assignment changed; a drain, failover, or colocation consolidation leaves unaffected owners continuous. Lifecycle metrics are live process-local Prometheus state and are not persisted or replicated.

Concrete branch-local inspection remains available through DESCRIBE RELAY <relay> WHERE (...). DESCRIBE does not provide a common branch inventory or eviction history. Prometheus deliberately has no branch-key label. See Capacity Planning For Branched Graphs for the cost model.

Wall Clock And Domain Clock

Nervix reports two time bases because they answer different questions:

  • wall-clock rates describe actual handled load per second of real process time
  • domain-clock rates describe records per second of event/domain time

For unpaced domains or records without usable timestamps, domain-clock values may be unavailable. For paced domains, domain-clock values follow the event timestamps and domain pace rather than the speed of test execution or wall-clock ingestion.

The moving rates are online exponential summaries. The percentile windows keep one bounded HDR histogram per step, 10 seconds for the one-minute windows and one minute for the fifteen-minute windows, rather than every observation. Both keep memory bounded and let metric state be snapshotted and replicated without retaining the observations themselves.

Replication And Drain Behavior

Nervix maintains internal metric state for DESCRIBE, edge statistics, and runtime recovery. Stateful processor summaries may use their node-owned snapshot path. Relay metrics are different: the relay owner is their sole authority, and buffer, traffic, concrete-presence, and branch-local relay summaries are neither persisted nor replicated.

DESCRIBE RELAY and DESCRIBE RELAY ... WHERE (...) are routed to that authority. A planned move fences the moved subgraph, drains admitted work, and activates the committed revision before the gate opens. Relay metrics start fresh on the new owner because they do not travel with materialized state. When a state-replica slot exists, a live former primary becomes the first replica candidate after the move.

Only moved runtime nodes cross that boundary. Internal DESCRIBE and edge metrics for unaffected nodes and concrete branches remain continuous during drain, graceful-shutdown drain, and placement consolidation. There is no shutdown-specific metric family: a draining node is identified by its phase log records and by terminating: true in its status, as described in Shutdown And Recovery. A timed-out precommit handoff leaves the schedule and metric authority unchanged. Unexpected owner loss continues through failover immediately and starts fresh relay metrics on the new owner.

Prometheus export is a separate, live process-local registry. Traffic metrics ignore branch identity, while lifecycle metrics retain only the bounded declared branch name. Prometheus registry values are not snapshotted into Nervix internal metric state and do not migrate during a planned handoff or failover.