Consensus Storage And Replication

Nervix uses Raft to replicate the control-plane state described in the Control Plane. Each node keeps its own durable log, applied state, votes, membership, and snapshots. This chapter defines when those records are durable, how a leader paces replication, how storage and memory stay bounded during catch-up, and what recovery may observe.

Consensus traffic uses the authenticated pools and admission classes described in Cluster Interconnect. Consensus storage is node-local: a quorum makes a command committed, while every node independently persists and applies the committed log.

The replicated transaction report and its content-addressed topology use this durable path; Transaction Quiescence And Impact Inspection defines which planned and actual facts that report retains. This chapter owns log reading, snapshot transfer, and the storage bounds that carry large reports.

Consensus request and append-stream responses carry the fixed ConsensusRequestError wire outcome, classified at its source owner for the compiler failure-contract gate. Local storage and Raft failures retain their contextual reports before this projection. Opening the node database also preserves the typed storage cause below startup context; a consensus keyspace in that database returns the separate layout context. Errors And Diagnostics owns the compiler and propagation rules.

Durable Write Path

Every consensus mutation uses a full synchronization barrier that asks the filesystem to persist both data and filesystem metadata. The device and filesystem must honor that request for the host-power-loss guarantees in this chapter to hold.

The path from a proposal to its reply has four distinct boundaries:

client        leader log             follower logs          leader state machine
  |               |                       |                         |
  |-- propose --->|                       |                         |
  |               |-- SyncAll append ---->|-- SyncAll append       |
  |               |<----- quorum has the committed entry ----------|
  |               |------------------------------------------------>|
  |               |                      SyncAll applied range       |
  |<------------------------- applied response ---------------------|

Appended Batches

Entries already ready for storage are encoded into the fewest reservation-bounded atomic writes. One write performs one synchronization. A larger submission is split only when the next complete entry would exceed the write’s admitted memory; an entry is never split between writes.

The append call returns to Raft after the storage job has been admitted to the single ordered consensus worker. Its I/O-completion signal is delivered later, after the write and synchronization finish. Raft can keep its core responsive while storage works, but it cannot count the append as durable before that signal. A vote, truncation, purge, or later append queued behind it observes the same worker order.

A storage failure marks that node’s consensus store failed. Later writes return consensus storage stopped after a failed write; restart the node to recover. The error can arrive after the device completed the write, so a failed or disconnected request is an uncertain outcome, not proof that the command was absent. Recovery reads the durable state to decide.

Consensus startup, proposal, and membership operations carry contextual error reports locally. Database open failures retain their I/O cause; Raft configuration, startup, and handler registration failures retain the failure beneath the startup context. A rejected proposal keeps its Raft cause while classifying a known redirect as leadership loss, a fatal storage failure as storage, and other Raft write failures separately. A state-machine refusal remains a typed conflict, distinct from a storage or leadership failure. These classifications determine whether the control plane redirects, retries, or reports an uncertain failure; it does not parse the error message to decide.

Transaction mutation checks also create local reports. Raft’s applied response retains its existing serialized TransactionMutationError outcome, so report frames are local to the state machine evaluation. The proposer creates a new report from the exact typed outcome received over Raft and adds transaction context when the control plane takes ownership. The response encoding, client acknowledgement boundary, and recovery rules in this chapter are unchanged.

Applied Ranges And Client Replies

Committed entries are applied in order. Entries already available together share an atomic write until adding the next entry’s changes would exceed the reservation. That write contains the final semantic records and metadata for the whole range. After synchronization succeeds, the node publishes the resulting coherent revision and then answers every entry in the range.

One command may expand when its archive becomes independently keyed transaction report, commit-plan, and topology records. An applied command therefore reserves 16 MiB from the Commands class: up to 8 MiB for the normalized records and keys, with the other half retained for Fjall’s journal encoding and moving batch descriptors. The reservation is released after synchronization. This keeps a multi-mebibyte report atomic without making storage memory proportional to retained history.

A crash after a log write and before application does not lose an acknowledged command. If the entry was committed, recovery streams it from the log and applies it. If it was not committed, a later leader may discard it. In either case, the client had not yet received an applied response. A crash or lost connection after durable application but before delivery of the response can leave the client uncertain even though the command took effect; persistent administrative requests use their stable execution reference to join or retrieve that result.

Transaction application completion persists the actual quiescence history and actual effects in the same applied state-machine boundary that changes the step from applying to applied or failed. Recovery therefore sees ordered requested, confirmed, failed or uncertain, and released outcomes without inferring them from a client timeout. These records contain control-plane scope, attribution, and non-sensitive diagnostics only. Runtime payloads, acknowledgement maps, connector buffers, and handoff bytes remain volatile data-plane state.

Restore Checkpoint Reclamation

Restore checkpoint reclamation borrows the applied state through with_restore_state_reclamation and retains its read guard across node-local bounded storage deletion. Its caller takes the checkpoint installation barrier after this guard, in the same order as staging and publication. The retention view keeps the current generation of every applying restore and any generation ahead of the applied log; without an applied log it keeps all generations. Leadership and lease expiry alone do not end an applying execution’s retry lifetime. Terminal, expired, missing or superseded executions no longer retain an applied unpublished generation. The runtime store protects its selected durable publication independently. Within the selected initial or restored namespace, the store retains only chunk sets referenced by the same view’s segmented header and exact checkpoint revision; segmented or inline replacement, header purge and incomplete unpublished revisions permit bounded chunk deletion. Snapshot retention preserves preexisting readers, while the installation barrier serializes ordinary, replica and ownership writes with cleanup. This adds no consensus record or log mutation and never removes a replicated incomplete-installation gate. See Backup And Restore for the quota, metrics and node-local cleanup boundary.

Coordinated WASM Reset Publications

An NSPL reset first records a typed ordered transaction step with its captured planning inputs, statement identity, and reset scope. That record changes no processor Model or schedule by itself; the coordinator uses its retained execution reference for the Publishing and Ready mutations. If the leader stops after recording the step, transaction recovery resumes that same reference. An expired outer command reference is refused at admission rather than starting a new step.

A coordinated WASM state reset stores its command execution reference, exact branch scope, phase, and advanced guest-state generation in the domain schedule. The first applied schedule phase, Publishing, is the irreversible authority boundary: after that applied response, restart and leader failover continue the new generation even when its fresh initial checkpoint is not usable yet. Repeating the same request rejoins that publication rather than appending another generation.

The initial guest checkpoint is runtime state and follows its separate local and replica synchronization path. Live replicas install the published generation before the owner starts its initial checkpoint. Reset preparation never publishes the temporary absence of a selected branch through the branch-lifecycle checkpoint, and reinserting the branch advances that checkpoint’s revision before the new guest checkpoint is offered. Once the guest checkpoint is durable on the owner and every assigned replica, a second consensus mutation changes the same reset to Ready. Success is returned only after every live node applies that phase while the selected relay scope remains fenced. A follower or offline replica that skips the intermediate runtime activation still catches up through the ordered schedule records and cannot accept a checkpoint from the replaced generation. Thus a pre-publication failure leaves the old schedule authoritative, while a post-publication failure is recovered as committed but not yet usable rather than rolled back or acknowledged early. See Coordinated Reset.

Replication Pacing

The leader keeps one ordered append stream to each follower. That stream uses the replication pool; heartbeats and leadership probes use a separate management connection so a slow or flow-controlled log transfer cannot consume the path that maintains the leader lease.

Before opening an append stream, the network adapter consumes the caller’s first request. OpenRaft applies its retry delay and cancellation while producing that input; even an immediate local admission rejection therefore passes through the caller’s gate. An ended input opens no transport. Leading empty requests, including finite catch-up probes, use the management heartbeat operation and its deadline. Each answer confirms that request’s previous-log boundary; a conflict, higher vote or incomplete acceptance ends that input before any data stream opens. The first request that carries entries remains charged while stream setup runs and joins the remaining requests exactly once after setup succeeds. Subsequent empty commit updates stay on that ordered stream alongside data batches. Setup retains its five-second deadline; waiting for the caller’s next request belongs to the caller’s lifecycle, rather than to connection setup.

One replication batch targets 1 MiB and stops at 1,024 entries. Commands remain whole, so the first command may take a batch past the target; the encoded replication operation is bounded at 2 MiB. Submissions and answers stay in log order. A conflict, higher vote, partial acceptance, stream failure, or liveness failure ends that stream generation after its first result. Raft resumes from the last progress the follower confirmed.

The leader permits at most 16 unacknowledged batches for one follower. It also waits whenever its unacknowledged bytes plus the next 1 MiB target would exceed 16 MiB. Because one complete command may cross the target, measured outstanding bytes may exceed that target budget by the excess of one encoded batch; the 2 MiB operation limit remains the hard bound on that batch. The node’s Commands memory budget separately bounds the aggregate encoded and decoded copies across all peers.

Progress-Based Liveness

Opening an append stream has a five-second deadline. Once open, the stream has no age limit. With a batch outstanding, it is considered stalled only after five continuous seconds in which both of these are true:

• no answer arrived;
• the follower accepted none of the leader’s bytes.

An answer or transport flow-control progress restarts the idle window. A stream with no outstanding batch can remain idle indefinitely. The time required for a follower to synchronize one batch is therefore not treated as a failure while the transfer is still moving. The interconnect frame writer uses the same progress contract: every nonzero flow-control grant restarts its write idle bound, including grants within one large append frame. Catch-up at a limited byte rate can therefore preserve its append stream across many five-second intervals. The native writer checks both a moving and a stalled receiver. The public rate-limited follower scenario sends opaque TLS bytes through a bounded test proxy at 32768 bytes per second and checks the applied domain backlog within its assertion bound and the bytes carried by the paced path. Append stream openings observe actual data-stream setup; initial empty probes do not open those streams. Native transport regressions cover finite probes, the transition to data, ordered empty commit updates and rejected probes. The public catch-up scenarios retain their stream-count and time assertions and change no product deadline.

Heartbeats keep the shorter deadline derived from the configured heartbeat interval, capped at five seconds. With the default 250 ms heartbeat, the current soft deadline is 187.5 ms. Replication answers also prove leader contact, so sustained writes suppress redundant heartbeats for an automatically derived window. The window is no longer than the heartbeat interval or half the gap between that interval and the minimum election timeout. This keeps the heartbeat interval plus the suppression window below the minimum election timeout.

The independent paths matter when a follower has slow storage: its append answer may take much longer than a heartbeat deadline, while management traffic still prevents an election. Changing the heartbeat interval does not change the append stream’s five-second progress bound.

A Quorum Over A Lossy Link

When only a bare quorum of voters is reachable, such as two of three while the third is down, every commit needs all of them and the leader’s link to each. When such a link loses packets but stays up, TCP recovers them by retransmission under the host kernel’s retransmission timeout, which grows exponentially while the losses continue; Nervix does not tune it. An exchange whose deadline passes before its retransmission arrives fails and is retried: a heartbeat at its soft deadline, and an append stream at its five-second setup or progress bound. Commits can therefore stall, leadership can move between the reachable voters, and a control command, which needs several commits in sequence, can time out.

Nervix sets no bound on such a stall. It lasts until the loss ends or another voter becomes reachable, and it loses no committed entry.

Follower Memory And Backpressure

A follower takes a Commands-budget reservation before it decodes an append batch. It retains that reservation until Raft has durably appended the batch and produced its answer. The default resident window is four decoded batches per live stream. Reaching the window stops frame reads before another batch is decoded, closes the HTTP/2 flow-control window, and makes the leader wait.

The default 24 MiB Commands budget is validated to hold the complete resident window beside one maximum-size 2 MiB batch being encoded. Other command and replication work shares that aggregate budget, so concurrent or superseded streams cannot grow memory without admission. Ending a stream releases its queued batches and reservations; batches from a generation the leader abandoned do not remain uncharged in the Raft core.

Follower Catch-Up And Learner Promotion

A follower whose required log suffix is still retained receives bounded append batches over its stream. If the leader has already purged part of that suffix, the follower receives the current snapshot and then the remaining log. Client proposals and management heartbeats continue while the catch-up runs.

A discovered node joins as a learner. Each membership reconciliation gives a blocking learner catch-up attempt ten seconds. Promotion to voter is proposed only after that attempt reports the learner caught up; the membership change has its own ten-second bound. A timeout leaves the node a learner, records raft membership reconciliation failed, and the one-second reconciliation loop tries again while ordinary replication continues.

When discovery reports a different interconnect endpoint for an existing member, reconciliation replaces that member’s recorded Raft address. A returning voter keeps its voter role; it is not re-added as a learner. Once the committed membership contains the advertised address, subsequent reconciliation passes propose no further address change. The stored address supplies recovery contact hints; Raft traffic resolves the authenticated node identity through the interconnect.

The consensus event stream and info log record raft add learner, raft wait for learner, raft update address, raft promote voters, and raft membership updated transitions. SHOW CLUSTER STATUS reports the local raft.last_log_index, raft.last_applied, and each member’s learner or voter role and recorded address.

Bounded Log Reading

Committed-range application, startup replay, and leader-bounded replication streams read the log in successive admitted chunks. A chunk targets 1 MiB or 1,024 entries, whichever is reached first, and one oversized first entry travels alone. The reader releases the ordered storage worker between chunks and retains the chunk’s memory charge until its entries have been handed to the consumer.

There is no range-size failure at 2 MiB or at any other aggregate backlog size. A committed range and a startup backlog may span any number of chunks. The state-machine worker applies the entries already read before requesting another chunk, so a large backlog does not have to exist as one allocation. A replication read also revalidates the leader’s durable vote at every chunk boundary; it stops rather than reading through a truncation performed under another leader.

Log Retention And Admission

retained_bytes is the sum of the encoded values for the log entries the node currently holds. It includes entries still in Fjall’s memtable and excludes purged entries, tombstones, stale segment space, snapshots, and other keyspaces. Append adds encoded entry bytes; purge and truncation subtract the removed entries in the same ordered storage operation. Opening the database rebuilds the counter by scanning the current log, so a process failure between a durable mutation and its in-memory counter update does not skew the recovered value.

A node starts one snapshot after either of these thresholds is crossed:

• 10,000 committed entries since the completed snapshot;
• 64 MiB of appended entry bytes since the completed snapshot.

Only a completed snapshot permits covered log entries to be purged. By default the node retains the newest 1,000 covered entries or 64 MiB of covered suffix, whichever bound is reached first. Entries newer than the snapshot remain regardless of those settings. A failed or unfinished build therefore does not remove the log needed to recover.

When retained_bytes is already above the 1 GiB admission cap, a new control-plane mutation waits up to 30 seconds for snapshotting and purge to reclaim space. If it remains above the cap, the write fails with:

the retained raft log holds <retained> bytes against a <cap>-byte cap and did not reclaim within 30s

The check occurs before admitting the next mutation, so the mutation that crosses the cap may complete. Reads, health checks, replication, snapshotting, and recovery remain available while new mutations wait.

Snapshot Lifecycle

Sealing And Publication

A snapshot generation is a sequence of sections containing keyed state-machine records plus a manifest that names the generation, applied log position, membership, section count, and total bytes. The builder opens one consistent database view and seals it one section at a time. The default section limit is 8 MiB. Each section acquires the Bulk budget for one consensus-storage turn, is encoded and synchronized, and releases that reservation before the next section starts. Concurrent state-machine writes can proceed between sections without changing the pinned view. The section builder charges each key and value together with its archived record envelope, root, and alignment before accepting the record. The encoded section therefore stays within 8 MiB even when it contains many small keyed report records; the complete generation grows by adding sections.

The manifest is synchronized only after every section is durable. Publishing that one record makes the generation active atomically with its applied position and membership. A build interrupted before manifest publication leaves the previous generation active.

Generations And Pins

The newest published generation is always retained. An outgoing transfer pins the generation it is reading. Multiple readers can share one pin, and at most one older generation may remain pinned beside the active generation. If publication would leave a second older generation pinned, the oldest transfer is made obsolete; its next read fails and Raft restarts it from the newest snapshot.

A superseded generation with no reader becomes unreferenced. Its sections are removed as part of a later durable consensus batch. Startup discovers sections that no published manifest names and marks those generations for the same deletion path. This keeps deletion ordered and durable without treating an unfinished build or abandoned transfer as active state.

Transfer And Installation

One complete snapshot transfer has a 30-second deadline. The sender reads one section at a time and sends it in 64 KiB chunks through the Bulk pool. The receiver holds at most one section in memory, validates its declared size and offsets, and synchronizes each completed section into a newly claimed generation. A restarted transfer abandons the earlier staged generation.

The receiver stages at most one transfer from each sending node, and a new transfer from that node supersedes it. Every transfer carries an identity its sender allocates, and the receiver refuses a chunk or a finish whose identity is not that of the transfer it is staging, so a late chunk of a superseded transfer never lands in its successor. Because the receiver only compares identities from one sender, an identity is unique among the transfers one node sends: every Raft connection of the node draws from the node’s own sequence, which starts again when the node starts.

After every declared section arrives, Raft revalidates the vote and whether the snapshot still applies. Installation then synchronizes the new manifest together with an installation marker. That write is the recovery boundary: before it, the old snapshot and state machine remain authoritative; after it, startup must finish installing the marked generation.

The installer clears the prior state-machine records, writes each staged section in order, and finally removes the marker while publishing the new generation. Every step is idempotent. A process that stops after the marker was published redoes the complete replacement on its next start, regardless of how many clear or section writes had finished. It exposes no recovered state until the replacement and current-shape validation succeed.

Storage Layout And Compatibility

The databases have independent journals, memtables, rotation, and synchronization. A consensus SyncAll neither flushes runtime journal bytes nor waits behind a runtime journal write. Both use the bounded storage executor, but consensus has its own single ordered worker.

The dedicated database owns exactly four current keyspaces: raft_count_logs, raft_count_meta, raft_count_state, and raft_count_snapshots. Opening it validates the complete keyspace namespace before reading any records. Its state metadata requires the current RestoredLifecycle encoding, including the durable restore installation gate. Native counts in commands, queued transactions, progress, outcomes, and plan, report and topology headers are archived as fixed-width 64-bit values with checked native decoding. Log replay, state recovery and snapshot installation therefore retain their complete magnitudes. See Archived Counts.

This layout has one current shape. A consensus database containing an unknown keyspace, a malformed current archive, or incomplete or inconsistent current state fails with invalid consensus record storage; recreate the node's stored state. Recreate the node’s stored state and let it rejoin from the cluster.

Archive shape and nesting are validated before a durable batch or state record is read back. Vocabulary values inside a valid shape are checked as each value is read. If one is refused after earlier list or fixed-array elements or a boxed or shared value have been read, the archive reader drops those values and releases their allocation. The whole record fails; no partial record is installed or retained during replay, catch-up, or snapshot installation.

Recovery accepts exactly the records the state writer stores. Each state record sits under the canonical encoding of its key, so a key spelled another way, or one with bytes after its encoding, is refused instead of being read as a neighbouring record. A record whose value names its own identity must name the key it is stored under: a domain, a schedule’s domain and each of its nodes, a user, a resource version, node status or upload, a transaction, its commit plan header or impact report header, and a command execution. The state keyspace holds those records and the metadata and nothing else, so a record no family owns is refused as well. Recovered resource records hold at most one upload for each resource version, and an expired command execution’s reference carries the issue time its expiry is ordered by. A transaction is in a state its transitions reach: an open or committing transaction queues exactly the statements and source bytes it accepted, a finished one queues none, a commit’s results apply consecutive statements from the first and its progress stays inside the queue, and a failure names a statement the transaction accepted. Every log read checks that a log record is stored under its own entry’s index.

Shutdown And Forced Endings

Consensus remains available throughout stop admission and graph drain. During terminal teardown, the node stops Raft, which ends its append stream generations, then submits an idle barrier to the ordered storage worker. Reaching that barrier proves every earlier submitted durable write returned and released the consensus store before its database handle is dropped. Dropping it closes the dedicated database, which finishes even when one of its background workers failed during the close; see Shutdown And Recovery. The interconnect is stopped after consensus, so peers first observe the append protocol ending and later the transport closing.

A repeated signal, an expired shutdown deadline, or SIGKILL skips that barrier. Recovery then sees the last complete atomic storage boundaries: an append batch is present or absent; an applied range contains its records and final metadata together or not at all; and a marked snapshot installation is redone in full. See Shutdown And Recovery for the exact SIGTERM, SIGKILL, stream-teardown, and restart behavior.

The recovered applied state includes the last stored membership and its peer endpoints. Startup discovery reads that membership before Raft has published its live topology metrics, so a former bootstrap leader can use surviving peers as authenticated gossip seeds. The stored endpoints are contact hints; current peer identity, incarnation, and endpoint still come from discovery. See Cluster Interconnect.

Observability

Normal membership changes appear at info. Append generation endings and invalid or foreign batches appear at debug; a stalled generation reports that the stream neither answered nor accepted bytes for five seconds. Snapshot requests caused by the byte threshold appear at info, and byte-retention purge requests appear at debug.

Tuning

The server exposes heartbeat, election, snapshot, and retention policy as command-line arguments and matching environment variables:

Replication batch and memory limits are fixed by the current server execution policy; the server does not expose flags for them:

Use the timing settings as one policy. Lowering the election minimum without lowering the heartbeat interval can make the configuration invalid at startup. Raising the heartbeat interval does not allow slower append storage because append liveness is progress-based and fixed at five seconds.

Snapshot and covered-log settings trade storage and catch-up method. Smaller thresholds build and transfer snapshots more often; smaller covered suffixes make a lagging follower switch to a snapshot sooner. Larger thresholds and suffixes retain more log and require a retained-log cap large enough for the working set. The cap should leave room for entries newer than the latest completed snapshot, because those entries cannot be purged regardless of the covered-log settings.

Restored Domain Installation Gate

The consensus state machine persists each restored domain’s pending or installing state alongside its restore execution and mutation lease. Creating the domain establishes a pending start gate; admitting state installation records leader tenure, execution, lease revision and the committed log index as its generation. Recording the model step removes the gate only for that exact authority. The record is included in snapshots and restored after restart, independently of lease release or terminal execution failure. Runtime state bytes remain in the node-owned state store. A local storage mutation holds the applied-state read guard through its authority validation, checkpoint mutation and handle clearing; a newer applied authority or start-gate release cannot cross that boundary. The current state encoding requires these records and rejects a database whose encoding does not match, requiring recreation under the alpha persistence contract. The current encoding also requires a restore’s explicit lifecycle policy and its installed-state effect’s clock. RESUME changes the domain to running and removes its complete-set installation gate in that single applied effect; it preserves the archived generation and mapping.