Client Session Protocol

The client session protocol is the public boundary between a client and a Nervix node. It carries NSPL commands and their outcomes, transaction attachment and inspection, completion and choice lookups, domain selection and the observations that follow it, Row subscriptions, domain clock attachments, producers that submit batches to client ingestors, resource uploads, backup downloads, and restore streams. The CLI, the web console, the Rust client, every host of the shared Rust binding, and independent implementations in other languages all speak it.

The protocol owns framing, verification, the session limits, request correlation, cancellation, typed dispositions, the redirect to the leader, the lifecycle of a session, and delivery of the unsolicited frames a session receives. The operation behind a request still owns what that request means: the control plane decides what a command does and when it is complete, the transaction planner decides what a transaction affects, the resource catalog decides when a version exists, and the runtime decides what a relay delivers and when a submitted batch completes. A reply is therefore never more than the statement it makes. Four facts stay distinct at this boundary, and a client that confuses them either repeats an effect or reports one that did not happen:

• Bytes transferred. A frame reached the other side. Nothing about a request follows from it.
• Work admitted. The cluster durably recorded a command under its execution identity. The work continues whatever happens to the session that sent it.
• Effects completed. The command reached its completion boundary, and its terminal outcome is retained under that identity.
• Waiter cancelled. A client stopped waiting. Before admission nothing began; after admission the effect continues and can be recovered by its identity.

This chapter explains why the protocol behaves as it does. The Client Implementation Manual states, as rules, what a client implementation must do. Each links to the other instead of repeating it. Command Completion owns the completion boundary and the retention of execution identities, Control Plane owns the transaction lifecycle, Transaction Quiescence And Impact Inspection owns the impact report, Resource Versions And Bindings owns the version lifecycle behind an upload, Backup And Restore owns what a backup archive holds, Domain Clock owns the clock a session can follow, Errors And Diagnostics owns how typed failures become diagnostics, and Shutdown And Recovery owns the phases of a node stop.

A Boundary Separate From The Interconnect

Clients never speak the Cluster Interconnect, and nodes never use the session protocol to talk to each other. The two boundaries answer different questions and are built differently:

A follower does not forward a client’s leader operation over the interconnect on the client’s behalf. It answers with a redirect that names the leader, and the client reconnects there, so every leader operation arrives at the leader on a session the client itself holds. Subscriptions are the one path that crosses both boundaries: the relay owner fans a batch out to the subscriber’s node over the interconnect, and that node encodes Row frames for its own sessions.

Ownership By Layer

The server is the composition root: it is the only crate that names the wire crate, the command pipeline, and the runtime together. The wire crate names neither Arrow nor the parser, so an implementation in another language needs only the schema. The Rust client additionally names the language edge, which it uses to split a request into statements and to recognize the statements it serves itself, such as USE; everything it knows about the cluster arrives over the protocol.

The Rust client and CLI declare their edge ownership in gated compiler source contracts. Their session maps remain client-owned synchronization outside the server record-path acquisition rule. Server relay subscription fan-out remains recurring data-plane execution with its own source contracts and exact-operation repair expectations. The annotations change no session framing, recovery or delivery guarantee. Data-Plane Concurrency owns this compiler boundary. Execution Primitives owns the shared backend selection, portable reference types and declared unmodeled exceptions used by the Rust client, binding and browser; those build choices change no wire contract.

Wire Format

One Schema, Eight Frame Roots

crates/client-wire/schema/session.fbs declares every shape of the public boundary, once. Eight root files name the table a frame is rooted at and the four-byte file identifier it carries:

A frame is one finished FlatBuffer that carries its root’s identifier and no size prefix. The transport delimits frames, so a frame never has to describe its own length and a receiver never parses a length that a peer chose.

A ClientMessage holds one request and the request identity that correlates its replies. A ServerMessage holds either a Reply to one request or one unsolicited body: a server notice, leadership, the domain list, the selected domain’s snapshot and cluster summary, subscription rows and the notices about them, a domain clock frame, a producer’s admission change or end, or the ending of the session. An unsolicited body carries no request identity and never completes a request.

Verification Before Reading

A receiver verifies a frame once, when it takes ownership of the bytes and before it reads any field. The frame must fit the session frame limit, hold at least a root offset and identifier, carry the expected identifier, and pass FlatBuffers verification. Verification bounds every offset and length, requires every (required) field, validates every string as UTF-8 with its terminator, and limits nesting depth, the number of tables, and the apparent size of the traversal. The table and traversal budgets scale with the frame being verified rather than with the limit: a frame of n bytes may hold at most n / 4 tables and may expand to at most eight times its own size, so a small frame whose offsets alias one another cannot make a receiver walk far more than it sent.

Reading then applies the rules the verifier does not know. Every union is required, so the NONE discriminant is never valid. A union discriminant or enum value the schema does not declare is rejected, and so is an absent optional scalar that a comment requires, a zero where a comment requires a non-zero value, an empty collection that must not be empty, a vector or string longer than the session limits, a set that is not in canonical order, and a value the receiver’s platform cannot represent. A receiver never substitutes a default for a rejected value.

Scalars declared = null are optional: absent means not provided, and a present zero is a value. The distinction carries meaning in several places. An append that expects position zero is not an append that expects nothing, and a float cell holds its value as an optional scalar because a defaulted field would lose the sign of a negative zero. Other scalars read as their declared default, zero or false, when absent.

Verified bytes are immutable, and every value the wire crate hands out borrows from them, so no view reads unverified bytes and no read verifies again. A verified frame pins the buffer it was received in for as long as it, or any value that keeps it, is alive. The gRPC codec therefore copies a frame smaller than 64 KiB out of the transport’s read buffer, which grows to the largest message the connection has carried, and takes a larger frame without a copy.

Session Limits

One set of limits governs a session. The transports configure their message limits from it, so an oversized message is refused before it is buffered, and every encoder checks each string and vector against it before the FlatBuffers builder grows, so no Nervix encoder produces a frame that a receiver holding the same limits rejects.

Every Nervix server and client uses these defaults. The wire crate refuses a configuration whose frame limit is outside 1 KiB to 256 MiB, whose transfer limit is below the frame limit, whose nesting limit is outside 16 to 128, or whose string limit exceeds the transfer limit. The upper bounds keep the verifier’s budgets representable in the 32-bit address space of the browser build, and the nesting bound caps the stack a hostile frame can make the verifier and the decoders use.

Replies Larger Than One Frame

A reply may be larger than one frame; a rendered transaction report is the usual case. The server encodes the complete reply once, from one read of its state, and cuts that single frame into TransferPart replies. The parts of one reply share its request identity and its total size, arrive in order with contiguous offsets starting at zero, and each carries a non-empty chunk. The client joins them and verifies the result as a frame of its own, under the transfer limit, which must hold a Reply to the same request whose body is not itself a transfer part.

A transferred reply is therefore exactly as trusted as one that fit a frame. It is never delivered truncated, and it can never mix two reads of the server’s state. Parts of one reply stay in order, but other replies and unsolicited frames may arrive between them, which is why a client keeps one reassembly per request identity. A reply above the transfer limit is not sent at all: the request receives the typed rejection ReplyTooLarge instead. A reply that carries an impact report is encoded on the bulk workers under the bulk memory class; when neither memory nor a worker is available for it, the request receives ServerBusy, which is safe to repeat because such a reply belongs to a request that changes nothing.

One Current Form

The protocol has exactly one current form. Frames carry no version, a session negotiates nothing, and neither side keeps a compatibility reader for an earlier schema. A changed shape is a coordinated change of the server, the Rust client, the binding, the console, the conformance clients, and the corpus in one commit. A peer built against another schema meets the ordinary verification and reading rules: an undeclared request kind is rejected as UnsupportedRequest, an undeclared enum value as UnsupportedValue, and a client rejects a server discriminant it does not know rather than skipping it.

The conformance corpus in crates/client-wire/conformance pins the bytes the Rust encoder writes for a set of requests, replies, and events chosen to reach what live traffic does not, such as extreme values, nested lists, and every refusal of a domain clock attachment, together with corpus.report, the reading of those bytes that every other reader must reproduce. just test-client-wire checks that the checked-in bytes are exactly what the encoder writes, and just update-client-wire-corpus regenerates them for review.

The registered client representation properties also compare complete generated current values through the production codecs. Separate targets verify malformed bytes under fixed limits; Arrow properties compare selected logical values and validity with decoded Row cells, using float bits and explicit redaction, while binding properties check column copies and retained frame lifetime. The client representation coverage map records their schema coverage and ownership boundaries. Both ordinary corpus replay and sanitizer-backed fuzzing run the same assertions through the property target inventory.

Transports, Endpoints, And Authentication

Every live node runs both client listeners. Neither depends on leadership or placement, and both close only when the node begins to stop.

Native gRPC

Native clients reach the gRPC service nervix.session.Session, which serves four methods and answers every other path with UNIMPLEMENTED:

Each gRPC message holds exactly one frame, delimited by gRPC’s own length prefix. The messages are not protocol buffers, and no generated gRPC service code is involved: the codec hands the frame bytes to the transport unchanged and verifies what it receives. The server speaks HTTP/2 only, accepts no message compression, and sets gRPC’s message limits to the frame limit, so a message above it is refused before it is buffered.

The listener serves plaintext HTTP/2 on --addr (NERVIX_ADDR, default 127.0.0.1:47391) and advertises --grpc-advertise-addr (NERVIX_GRPC_ADVERTISE_ADDR), or the listen address when that is unset. --grpc-mode https (NERVIX_GRPC_MODE) serves the listener over TLS on --grpc-https-listen-addr instead and advertises --grpc-https-advertise-addr. That mode presents the development certificate that scripts/generate_dev_tls.sh writes under tls/dev in the source tree the server was built from; no option selects another certificate, and the Kubernetes deployment runs the listener in plaintext mode.

Console WebSocket

The web console reaches the same session engine over a WebSocket, GET /console/ws on the console listener: --web-console-listen-addr (NERVIX_WEB_CONSOLE_LISTEN_ADDR, default 0.0.0.0:47420), and optionally a TLS listener, --web-console-https-listen-addr with --web-console-tls-cert and --web-console-tls-key. Each binary message holds exactly one frame in either direction and nothing else: no size prefix, never two frames, never part of one. Ping, pong, and close messages belong to the WebSocket connection and never carry a frame.

The console session is served only by the leader. A console session on any other node is ended at once with a SessionEnding whose reason is a LeaderRedirect naming the leader, and the console reconnects to the leader’s advertised console endpoint. A console session also ends this way when its node stops leading, which the node checks every 250 milliseconds.

The console listener serves three WebSockets, each carrying one kind of call with exactly the frames its gRPC method carries:

A frame of another call’s root fails verification and closes the connection with 1007. A download connection takes exactly one request: another data message after it closes the connection with 1008, and a close or a failed connection stops the download before it sends another frame, which releases only its own hold on the archive. Any node serves a download and a restore, and each answers from its own role exactly as its gRPC method does: a node that does not retain the archive and is not the leader answers a download with a redirect, and a restore sent to a follower is answered with a redirect outcome.

A WebSocket client cannot half-close a stream the way a gRPC client does: once it sends a close, it reads nothing more. A restore over the console WebSocket therefore ends with the chunk that completes the size its start declares, and the connection stays open for the reply. A close or a failed connection before that chunk ends the attempt in transport, which changes nothing and releases what the leader staged for it. A frame the restore cannot read, such as a chunk before the start or a second start, ends the stream at that frame, and the restore refuses it without reading further. The server closes with 1011 the rare answer it cannot fit into a frame.

The console has no upload stream: it uploads a resource through its own HTTP path, /console/resources/upload, where the leader builds the archive from the files it receives, as Resource Versions And Bindings describes.

Authentication

Both transports authenticate a registry user with Basic credentials before any frame is read. A gRPC call presents authorization: Basic <base64(user:password)> metadata, on every call, and a call that fails authentication ends with UNAUTHENTICATED. The console upgrade presents the same credentials in an Authorization: Basic header or, because a browser cannot set headers on a WebSocket, as the auth query parameter; a failure answers 401 with a Basic challenge and never upgrades. GET /console/auth checks credentials without opening a session. Passwords are verified against Argon2 hashes held in consensus, on the node’s credentials worker under a charge of the working memory the stored hash’s parameters make Argon2 allocate. That class serves only password work, so saturated data, extension or bulk work never delays a login, and concurrent attempts wait for it in turn. A node whose credentials class cannot take the verification now answers UNAVAILABLE, or 503 without an upgrade on the console: the credentials were not judged, so the attempt neither counts as a failure nor paces the user, and the same credentials may be presented again. After a failed attempt for a user, that user’s later attempts are paced to ten per second until one succeeds.

Authentication establishes who owns what a session does: its transaction binding, its execution references, its subscriptions, and its uploads. It is the only access control. Every authenticated user may run every command in every domain; ownership decides only who may attach and inspect a transaction, and scopes execution references and upload identities to their user.

Transport Failures

A failure of the framing ends the connection, because nothing after it can be trusted. Everything a well-formed frame can get wrong is answered inside the session, as a typed reply to the request it names.

Requests, Lanes, And Cancellation

Request Identity Versus Execution Identity

Every ClientMessage carries a request identity: a non-zero 64-bit value the client chooses and never reuses within the session. Every reply to that request is a Reply naming the same identity, and a request receives exactly one terminal reply, delivered whole or as transfer parts. The identity exists only to pair replies with the request waiting for them. It is scoped to one session, which is one gRPC Exchange call or one WebSocket connection, and a new session starts over.

A command’s execution reference is a different identity with a different purpose. It names the command’s effects durably, across retries, redirects, reconnects, leader changes, and full restarts, and it outlives every session that carries it. A retry after a lost reply is a new request, with a new request identity, that repeats the same execution reference. Confusing the two breaks both guarantees: repeating a request identity cannot recover an effect, and minting a new execution reference for a retry can apply an effect twice.

Two Lanes

A session serves each request on one of two lanes:

• The ordered lane serves the requests that change the session or may change the cluster: commands, transaction attach, subscribe and unsubscribe, domain clock attach and detach, and producer and consumer opens. They run one at a time in the order the client wrote them. A mutating command never overtakes the command written before it, and a subscription never opens before the command that created its relay has completed. Every command runs on this lane, including a SHOW or DESCRIBE, so within one session a read command waits behind a command that is waiting for its completion boundary.
• The concurrent lane serves the requests that only read the session: completion suggestions, choice lookups, listing and selecting domains, and transaction inspection. Each runs as its own task beside the ordered lane, from the view of the session that the ordered lane last published, so a long command never delays a completion, an inspection, or a domain request. Their replies can arrive in any order. Selecting a domain changes only which domain’s observations the session receives.

A cancellation is on neither lane: it is answered as soon as it arrives. Neither is a submitted batch or a producer close: the receive loop hands both to the producer’s own task and reads the next request at once; see Producers. A session admits at most 64 requests in flight, counting both lanes and producer closes but not submitted batches, which their producer’s credit bounds instead. A request beyond that is refused with TooManyRequestsInFlight rather than queued, which also bounds the requests waiting for the ordered lane; nothing about the refused request was admitted. The web console never sends a request that keeps its order past that limit: it holds the request, in the order it was issued, until an earlier reply frees a place. Its completions, choice lookups, and domain selections go out at once and report the refusal if they find the session full. A request identity that is already in flight is refused with DuplicateRequestId, and that refusal necessarily names the same identity as the request still in flight.

Choice lookups on the concurrent lane return typed values for structured client controls. A domain dependency selects internal schemas, branches, relays, codecs, VHOSTs, signaling protocols, JSON/CBOR/AVRO wire schemas, or resource catalogs; a domain and relay reference select relay fields; a domain and codec reference select that codec’s output schema fields; a domain and resource reference select completed resource versions. Wire-schema targets have separate discriminants, so a codec cannot mistake a JSON wire schema for a CBOR or AVRO schema with the same name. A visual ingestor uses a distinct source-reference target for each of the 14 connector and entity pairs that read an external system; a client producer ingestor selects its schema through the schema target instead. Each source-reference target requires a domain and offers clients of its pair’s connector, except the HTTP endpoint and WebSocket endpoint targets, which offer endpoints of that TYPE. Its decoding-codec target offers codecs that can decode. An unbranched-relay target needs a domain; a branched-relay target also needs the selected branch and offers relays with exactly that branch. Ingestor error relays use the unbranched target. A branch-field target takes a domain and branch reference. Ingestor decoded and output fields use the existing codec-field and relay-field targets. A junction or reingestor asks for another input relay with a domain and the first relay reference; candidates have that relay’s exact named schema and branch. The same dependencies with the output-relay target select the input branch, and with the materialized-relay target select only relays with materialized state in that branch. Route construction of an unbranched or named branch uses the ingestor relay targets, while branch keys and record fields use the branch-field and relay-field targets. These processor targets resolve against the attached transaction prefix, and their page digests include the selected input definition. The typed target and dependencies remain part of the cursor identity. A completed-version value is either an explicit number or LATEST, not a label to parse. Resource catalogs include resources staged earlier in the attached transaction, while version choices include completed uploads only. The cursor binds the selected candidate set and its definitions; a changed context returns StaleContext instead of continuing an earlier page. Sessions owns the complete lookup contract and the server’s transaction-aware resolution.

flowchart LR
    Client -- ClientMessage --> Decode[Verify and decode]
    Decode -- commands, attach, subscribe, unsubscribe, clock attach and detach, producer and consumer opens --> Ordered[Ordered lane: one at a time, in written order]
    Decode -- suggest, choices, domains, inspection --> Concurrent[Concurrent tasks]
    Decode -- cancel --> Cancel[Answered at once]
    Decode -- submit batch, close producer --> Pumps[Producer tasks]
    Pumps --> Control
    Ordered --> Control[Control lane: 16 frames]
    Concurrent --> Control
    Cancel --> Control
    Clocks[Domain clock deliveries] --> Control
    Deliveries[Subscription deliveries] --> Rows[Subscription lane: 16 frames]
    Control -- always taken first --> Transport
    Rows -- taken while the control lane is empty --> Transport

What A Session Sends First

A session queues its frames on two bounded outbound lanes. Replies, transfer parts, session events, and domain clock frames wait on the control lane, which holds 16 frames. The session ending never waits: the transport writes it after the control frames already queued, and nothing after it. Subscription rows and the notices about them wait on the subscription lane, which holds 16 frames shared by every subscription of the session. The transport always takes a queued control frame first and takes subscription frames only while the control lane is empty. A reply, including the reply to the unsubscribe that stops a subscription, therefore never waits behind rows the client has not read. The lanes never reorder what the transport already took: a frame handed to the transport stays ahead of every frame queued after it. An attached clock’s tick holds one replaceable control slot per domain. Its delivery task can overwrite the slot while the lane is full or after queueing it, until the transport takes it; a changed clock state withdraws a superseded tick still in that slot.

Because every frame is at most the frame limit, the two lanes bound what a session holds for a client that reads slowly. Backpressure reaches the client’s requests too: a rejection or a cancellation outcome waits for room on the control lane before the session reads the next request, so a client that reads nothing eventually stops being read. The server has no idle or slow-client timeout; such a session ends only with its transport.

Cancellation

CancelRequest names the request identity it targets. The server answers with CancelOutcome: Requested when the target is in flight, or NotInFlight when it is not. The target still receives its own terminal reply: RequestCancelled, or its ordinary reply if it finished before the cancellation took hold. The two replies can arrive in either order.

Whether a cancellation stops an effect depends on one atomic decision per request. A request admits itself immediately before its first effect, and a cancellation that arrives first decides the request cancelled for good. RequestCancelled reports which came first:

• BeforeAdmission. The request was not admitted and never will be. It has no effect.
• AfterAdmission. The request was admitted. Cancelling ends only the wait for it. A command’s effect continues and is recovered by its execution reference, exactly as after a lost reply; what an admitted cancellation leaves of the other ordered requests is described below.

A submitted batch cannot be cancelled: a CancelRequest naming it is refused with InvalidRequest, and the batch still receives its outcome, because a batch the ingestor may already have admitted cannot be withdrawn.

A command admits itself just before its durable admission or first effect. Transaction attach, subscribe, unsubscribe, clock requests, and producer and consumer opens admit themselves when the ordered lane starts serving them, and a concurrent request never admits itself, so cancelling one always stops it. A command cancelled after admission keeps the ordered lane until its work finishes, and an attach cancelled after admission still binds the transaction. A domain clock attach cancelled after admission also keeps the ordered lane until it is served, but then attaches nothing, because the server delivers only an attachment whose reply it queued; a domain clock detach cancelled after admission has still detached, and sending it again reports the session’s state. A cancellation never rolls back admitted work: it is how a client stops waiting, not how it undoes a command, and undoing is a new command.

Rejections

A request that is not served receives RequestRejected with a typed reason, the offending field when there is one, and a message for display. The session keeps serving after any rejection.

The first five refuse a request before anything about it runs. ReplyTooLarge and ServerBusy replace a reply that could not be prepared. For a read, repeating the request reads again. For a command, the missing reply says nothing about the command’s effect, which is recovered by its execution reference like any other lost reply.

Command Dispositions

A command’s reply is a CommandOutcome: the execution reference it answers, whether the outcome was produced now or recovered from the durable record of an earlier attempt, one typed disposition, a message for display, diagnostics with optional source spans, and, for a request of several statements, the outcome of each statement in written order. It also carries the transaction this session is bound to while the command was served, the operation admission when the command accepted one operation into the transaction, the typed inspection, WASM state, or resource description a describing statement produced, and the summary of a backup’s archive or the report of a restore.

The disposition is the one field a client decides from; the message is for a person. Each disposition belongs to one phase of the command and makes one statement about its effect:

Consensus proposal failures reach the session with typed leadership, conflict, storage, and transaction-mutation causes in local reports. The session selects its disposition from those causes and the command’s admission phase, while the display message retains the underlying Raft or I/O reason. Report carriage does not change request correlation, the durable acknowledgement boundary, or the stored transaction response.

OutcomeOrigin distinguishes an outcome produced by this request, Executed, from one read back from the durable record of an earlier attempt with the same reference, Recovered. A recovered failure is the same failure: its message and diagnostics are the ones retained when it happened.

A request of several statements reports each statement’s own disposition, which is one of CommandCompleted, RequestFailed, or LeaderRedirect, in written order, and the command’s disposition is that of the statement that ended it. Several statements in one request always belong to a transaction: the request begins one and appends to it, possibly committing it too, or appends several statements to the transaction the session holds. Outside a transaction a request carries one statement. DESCRIBE TRANSACTION and SHOW TRANSACTIONS read beside the transaction and are always sent alone.

The leader’s reference check and replicated admission return the same typed dispositions. A reference the history no longer holds returns ExecutionReferenceExpired; a conflicting request detected during replicated admission returns ExecutionReferenceConflict with the conflict kind. Neither refusal admits the request. Exact Recovery explains the retry fence.

Four Boundaries

The dispositions exist because a client can lose sight of a command at four different points, and each point means something different:

sequenceDiagram
    participant C as Client
    participant S as Session on the leader
    participant R as Replicated execution record
    participant E as Effect owners
    C->>S: CommandRequest(request 7, reference X)
    Note over C,S: Bytes transferred: nothing about X follows yet
    S->>R: admit X with its semantic command
    Note over S,R: Work admitted: X continues without this session
    R->>E: apply, activate, release
    E-->>R: completion boundary reached, outcome retained under X
    Note over R,E: Effects completed
    S--xC: Reply(request 7) lost with the session
    C->>S: CommandRequest(request 1 of a new session, reference X)
    S->>R: look up X
    R-->>S: retained outcome
    S-->>C: Reply(request 1): CommandCompleted, origin Recovered

A cancelled waiter is the fourth point: a CancelRequest that wins before admission leaves no record of the reference at all, and one that arrives after admission ends only the wait. In every case the same reference, sent again, finds the truth: nothing recorded admits the command now, an applying record joins it, and a terminal record returns it.

Exact Recovery

A client recovers a command it lost sight of by sending the same command again under the same execution reference. The protocol makes that safe by keeping one durable record per reference and answering every repetition from it. Command Completion owns the record’s lifecycle, its retention, and the retry fence; this section describes what a client can rely on.

What Is Recorded

Every persistent command is recorded under its reference before its first effect: model creation, alteration, and removal, REBIND RESOURCE, CREATE DOMAIN, ALTER DOMAIN, CREATE USER, CREATE RESOURCE, START, STOP, DROP NODE, CORDON NODE, UNCORDON NODE, DRAIN NODE, RELOCATE, RESET WASM PROCESSOR ... STATE, and BACKUP. So is every transaction request: a request that begins, commits, or reverts a transaction, one sent while the session is bound to a transaction, and one that carries an expected transaction position. An ordinary request carries at most one persistent statement; several statements in one request always form a transaction request.

Reads record nothing. SHOW, DESCRIBE, LOOKUP, DESCRIBE TRANSACTION, and SHOW TRANSACTIONS accept any well-formed reference, and repeating one reads again. Uploads have an identity of their own and are covered in Resource Uploads.

The record binds the reference to one semantic request: the authenticated owner, the request’s domain, the expected transaction position, and a digest of what the request asks for. For an ordinary command the digest covers the parsed statement, so a retry that differs only in whitespace or keyword case is the same request. For a transaction request it covers the exact request text. A new reference is admitted only when it is a UUIDv7 whose creation time lies after the cluster’s retry fence and no more than five minutes ahead of the leader’s clock; a reference that fails either check is refused before any effect.

How A Repetition Is Answered

The leader serializes the requests that carry one reference, then consults its record:

• No record. The request is admitted as new, subject to the checks above.
• Applying. The request joins the admitted execution and waits for it. After a leader change, the new leader resumes every applying record on its own in its next reconciliation pass, which runs every 250 milliseconds, so the work completes whether or not anyone repeats it.
• Finished. The request returns the retained outcome once that outcome is authoritative on each node in the leader’s current completion participant set. The leader can retire an unreachable node from that set; the node must catch up before its local reads reflect the completed effect. If the wait fails, the reply is OutcomeUnknown with cause NotYetAuthoritative, and a later repetition returns the outcome.

The first reply is itself rebuilt from the durable record, so the reply to the original attempt and the reply to a repetition are identical except for their origin. The record keeps the disposition, message, diagnostics, per-statement outcomes, transaction status, operation admission, and the archive summary of a completed backup. It does not keep the typed inspection, WASM state, or resource description that a describing statement returns, because those statements are reads and record nothing.

An execution whose attempt returns OutcomeUnknown, a leader redirect, or TransactionDetached stays applying. The finalization boundary returns that disposition unchanged and leaves recovery under the original reference to determine the terminal outcome. Reconciliation follows the same rule when an attempt is interrupted after admission.

The same rule covers every transaction control:

• BEGIN. The new transaction’s identity is fixed when the request is admitted. Repeating a BEGIN whose reply was lost returns the same transaction and binds the repeating session to it, instead of opening a second one.
• Appends. Each statement of a request is recorded under an identity derived from the request’s reference and the statement’s position in it. Repeating an append with the same reference, position, and text returns the admission recorded for it, without preflighting it again or renewing the transaction’s activity; the transaction is not appended to twice.
• COMMIT. Repeating a commit joins the committing execution or returns its recorded outcome.
• REVERT. Repeating a revert of a transaction its owner already reverted succeeds again.
• Multi-statement requests. A request of several statements is one execution with one outcome per statement, and a repetition after a leader change or a full restart returns those outcomes without repeating any effect.

Conflicting Reuse

A reference names one request forever. Repeating it with a different owner, domain, expected position, or content, checked in that order, is refused as ExecutionReferenceConflict naming what differed, and nothing about the new request is admitted. A session bound to a transaction other than the one the reference was recorded against is refused as a position conflict. When two leaders race and the replicated state machine detects the conflict after the leader’s local check, the refusal carries the same typed ExecutionReferenceConflict and kind.

Bounded History And Expired Identities

The history is bounded by count, 65,536 records by default, counting applying records, finished records, and tombstones together. When it is full a new reference is refused with an explicit capacity failure, and admitted work is never evicted to make room. A finished record is retained for the retry validity after the command finished, 15 minutes by default, and then reclaimed; an applying record is never reclaimed. Reclamation advances a durable, monotonic retry fence first, so a reclaimed reference, repeated at any later time, fails the admission check and never starts its effect again. The refusal is ExecutionReferenceExpired whether the history still holds a tombstone or only the retry fence. The outcome of the original attempt can no longer be recovered.

Why Aggregate State Proves Nothing

A client that lost the reply to an append, or to a commit, might be tempted to look at the transaction instead: if it is COMMITTED and its operation count moved, the append must have been in it. That inference is false. Another session can attach the transaction, commit it without the outstanding append, and leave the same counts. Finishing a transaction clears its queued statements, so an append that was never admitted cannot match anything afterwards, and the only witness of what happened to a particular request is the record of its own reference. A client therefore reports an append or a commit only from the outcome recorded for that exact request.

sequenceDiagram
    participant A as Session A
    participant L as Leader
    participant B as Session B
    A->>L: append S (reference R, expected position 0)
    Note over A,L: R is not admitted yet
    B->>L: attach transaction T, taking it over
    B->>L: COMMIT
    L-->>B: CommandCompleted: T is COMMITTED
    L-->>A: R refused: S was never admitted, and T holds no S
    Note over A: T being COMMITTED says nothing about S

Domain Mutation Ownership And Plan Fencing

Exact recovery answers “did this command happen”; ownership and fencing answer “could anything else have changed the domain under it”. Both are control-plane contracts, and a client observes them only through the outcomes they produce.

One Owner Per Domain

A command that changes a domain acquires the domain’s mutation lease when it is admitted, and a transaction acquires it when its commit is admitted. The lease is replicated state. It names its owner, a command’s execution reference or a transaction’s identity, and carries a recovery fence: the log position of the entry that acquired it. Every write to the domain’s replicated state, its definition, schedule, and lifecycle, must present exactly the lease currently held. A write that presents none while a lease is held, a different one, or one from a superseded acquisition is refused, and so are automatic schedule changes and clock-authority updates while any lease is held. The command or transaction releases the lease when it finishes, whatever its outcome.

Because the lease lives in replicated state rather than in a leader’s memory, it survives leadership changes and restarts. A new leader resumes the applying record, or the committing transaction, that holds the lease and continues under it; a deposed leader can no longer propose, and a stale acquisition fails on its fence. A second mutation of the same domain is therefore refused at once, with a failure that names the owner, rather than queued behind the first, while mutations of other domains proceed in parallel. A client that receives such a refusal decides whether to send the command again once the owner has finished.

Plans Apply Only To What They Planned

A command or commit is planned from one captured set of inputs: the domain’s state, its resources and completed versions, its schedule, and cluster membership, voters, and cordons. The replicated state machine applies the plan only if those inputs are still equal to what was captured, and refuses it otherwise, naming the input that changed. Eligibility that lives outside consensus, liveness and incarnations, is checked again by the leader immediately before it publishes. A plan can therefore never publish a candidate derived from one schedule over a newer one: a RELOCATE planned before a schedule revision fails rather than overwrite that revision.

What happens after such a refusal depends on who owns the plan. An ordinary command applies through a frozen internal attempt; when its captured inputs change before its effect is recorded, it keeps the failed attempt and starts a new attempt derived from the current revision under the same execution reference, so its caller sees one outcome. An explicit transaction freezes its plan when COMMIT is admitted: stale inputs at admission leave the transaction OPEN, and a step whose inputs changed after admission fails the transaction with the applied prefix preserved, rather than being replanned against state the client never reviewed. Transaction Quiescence And Impact Inspection defines the frozen plan and the preview it is checked against.

Transactions Over The Protocol

A transaction is replicated control-plane state; its binding to a session is not. Control Plane owns the lifecycle, the eligible statements, and the limits, and Transaction Quiescence And Impact Inspection owns planning, previews, and reports. This section describes how a transaction travels over the protocol.

Opening. BEGIN is an ordinary CommandRequest for the selected domain, which must already exist, and every later request of the transaction names that domain. The outcome’s TransactionStatus carries the transaction’s identity, domain, state, and accepted and applied operation counts; the identity is the durable handle a client keeps. One request may also begin the transaction, append to it, and commit it at once.

Appending. Each append carries the position it expects: the number of operations the transaction had accepted, as the client last learned it. A statement accepted into the transaction returns TransactionOperationAdmission with its one-based operation number and the whole-transaction preview identity after it. An append that is refused leaves the transaction and its position unchanged, so the client corrects it and appends again. Other sessions keep seeing committed configuration until the commit.

Binding, attach, and takeover. The session that begins a transaction is bound to it. The binding is soft state in the leader’s memory: it routes commands, and it is gone after a leader change or when the session ends. A session binds an existing transaction with AttachTransactionRequest, which the leader grants only to the transaction’s owner; the reply is TransactionAttached with the current status, TransactionAlreadyFinished with the final status and aggregate outcome while its tombstone is retained, or a failure for an identity that is unknown or retained no longer. A leader that holds no binding for the session’s transaction answers the session’s next transaction request with TransactionDetached, and the client attaches again and repeats the request under the same reference. A finished or failed attach does not resolve commands whose replies were lost: the client repeats each under its own execution reference and expected position. The leader returns a recorded outcome before planning a new request; a command it never admitted receives a failure. Attaching from a second session takes the binding over, and the displaced session’s next transaction request receives TransactionTakenOver before anything is admitted. While a session holds a transaction, the server refuses the requests that belong to the session rather than the transaction, subscribe and unsubscribe and domain clock attach and detach, and the Rust client and the console refuse USE and LIST DOMAINS themselves.

Committing. COMMIT may name the preview identity the client reviewed: the transaction identity, its accepted position, and the fingerprint of its planning basis, taken from the last append’s admission or from inspecting the attached transaction. If that preview no longer describes the transaction, the commit is refused with PreviewStale naming the expected and the current identity, nothing applies, and the transaction stays open and bound. A preview fences only a commit of a transaction that the same request did not itself begin or append to, because such a request has already moved the transaction past what the client reviewed. A commit that is admitted runs its frozen plan step by step; each step is atomic, and a failed step leaves the applied prefix in place rather than rolling back the transaction. A commit’s outcome arrives only when the final step is usable on every live node, and a client waits through COMMITTING for the outcome recorded under the commit’s own reference.

Inactivity and retention. An OPEN transaction expires once it has been inactive for its idle timeout, 15 minutes by default, whether or not a session is bound to it. Attaching it, appending to it, and a commit’s admission or failure renew that deadline; an open connection, a bound session, and reads of the transaction do not. The deadline is a durable UTC instant, so time the cluster spends stopped counts toward it, and an overdue transaction expires once the cluster runs again. A COMMITTING transaction never expires. A finished transaction remains as a tombstone, 15 minutes by default, during which attach and inspection report its outcome; afterwards its identity is unknown.

Ending a session. A session that ends uncleanly, through a transport failure, a node stop, or the server ending it, only releases its binding; the transaction stays open for attach until it expires. A session that the client closes cleanly, with no request in flight, reverts the open transaction it is bound to on the leader, which is how an interactive client that exits discards its unfinished work. Neither ending affects admitted work: an append already admitted stays admitted, and a COMMITTING transaction continues without its client.

Read consistency. A read, such as SHOW, DESCRIBE, or LOOKUP, is served by the node the session is on from its locally applied replicated state; it is not a linearizable read. Once a command completes, its effect is applied on every node in the leader’s completion participant set, so a later read through one of those nodes observes it. A node the leader has retired can still serve an older local state until it catches up. Completion for a session bound to a transaction resolves against the committed configuration with the session’s own queued statements applied, while every other session sees committed configuration alone. Inspecting a transaction reads it without changing its binding, domain, activity, or queue position.

Leader Discovery, Redirect, And Reconnect

What A Session Is Told

Every session receives LeadershipObserved first and DomainsObserved second, and each again whenever it changes; the serving node checks leadership every 250 milliseconds. Leadership is one of three states: the serving node leads, a remote node leads, with whichever of its endpoints discovery has established, or no leader is known. Selecting a domain with SelectDomainRequest opts the session into that domain’s observations as well: a ClusterObserved summary and a DomainSnapshotObserved carrying the domain’s live graph as a JSON document and its declared entities, at once and then every 500 milliseconds. A session that selects no domain receives neither. Server notices report runtime and control-plane errors and cluster events as text for display. A session that falls behind the node’s notice bus skips the notices it missed; the node logs how many. The Rust client holds at most 128 notices and 1 MiB its caller has not read. A notice that does not fit drops the ones held, and the caller’s next read reports that gap before the notices that follow it. Notices end with the session that delivered them, and the client’s notice stream continues with the next session it opens.

There is no separate discovery request. A client learns where the leader is from these observations and from the redirects in its replies. Endpoints come from the cluster’s discovery, which carries each node’s advertised gRPC and console URLs. A redirect never guesses: while no leader is known it names none, and while discovery has not established a leader’s endpoint the redirect names the leader without it.

Which Requests Need The Leader

The node a session is on serves completion and choice lookups, listing and selecting domains, subscriptions, domain clock attachments, producers, and the reads its locally applied state can answer: SHOW CLUSTER STATUS, SHOW TRANSACTIONS, SHOW CREATE, SHOW UDFS, SHOW INGESTORS, SHOW PLACEMENTS, SHOW RELAY MATERIALIZED STATE, LOOKUP, and the DESCRIBE statements of domains, resources, endpoints, lookups, placements, UDFs, relocations, relays, ingestors, junctions, deduplicators, reingestors, correlators, reorderers, window processors, WASM processors, and emitters. Every other statement needs the leader, and so do every transaction request, transaction attach and inspection, and uploads. A follower answers them with LeaderRedirect before anything is admitted, and the client reconnects to the leader and repeats the request. Because persistent commands and transactions only ever run on the leader, a follower holds no execution state a client could lose.

The console session is the exception: the web console is served only by the leader. A console session on any other node, and a console session whose node stops leading, is ended with a SessionEnding that names the leader, and the console reconnects to the leader’s console endpoint.

Deadlines And Cancellation

The server imposes no deadline on a request’s wait and no idle timeout on a session, and neither transport sends keepalives; a connection is judged dead by its transport or by the client’s own deadline. A client deadline bounds only the client’s wait. It never cancels admitted work, and Command Completion owns what the work does meanwhile. A client that gives up on an uncertain command keeps its execution reference and can recover the outcome later, within the retry validity.

The Rust client bounds each connection attempt by connect_timeout (10 seconds by default), each request by request_timeout (120 seconds), and each public call, including all of its retries and redirects, by retry_timeout (120 seconds). Without keepalives, a request that outlives request_timeout is the client’s only sign that its session may be dead, so it ends every request still waiting on that session, and the next request opens a new one. While an election converges it retries with a delay that starts at 100 milliseconds and doubles up to one second. When a call’s deadline passes before a command’s outcome is known, and the command may have been admitted, the call fails as uncertain and names the execution reference. The Rust client, the CLI, and the console never send CancelRequest: a caller that stops waiting only drops its waiter, and the reply, when it comes, is discarded.

nervix-cli backup --timeout extends its request and retry budgets by the named quiesce timeout, leaving the ordinary request budget for command admission, capture and the reply.

For native Rust sessions, the client loads a Hickory resolver during setup or uses one its owner provided. It resolves the hostname of each selected server, seed, and redirect on a new connection attempt, preserving the advertised URI’s authority and TLS name. Tonic’s connection timeout spans DNS, address attempts, and TLS. A lookup takes at most 30 seconds when no shorter connection deadline cancels it. Failed resolution is a connection failure under the same bounded reconnect policy. Server-internal sessions to a peer’s session service reuse the node’s loaded resolver. Browser WebSocket and fetch resolution remains owned by the browser. Name Resolution describes the resolver, its bounds and its failures.

Reconnecting A Session

A lost session takes everything session-scoped with it. As soon as its frames stop, the Rust client reports an interruption for every subscription and clock it held, and it rebuilds the session in a fixed order. Domain clock attachments and subscriptions come before the transaction, because a session that holds a transaction refuses both, and the transaction comes before any command that belongs to it:

1. It opens a new session against the servers it knows: its configured seeds, then at most 32 endpoints it learned from redirects and leadership observations, oldest first, then the server it was using, with the same backoff as an election.
2. It attaches every domain clock it followed.
3. It publishes the new session to its callers and ends the old one. When the old session was still open, as after a leader redirect, ending it reports an interruption for every subscription and clock it still held. Its frames stop before the interruption, so nothing it delivers afterwards can make an attachment look held.
4. It attaches every clock the old session began to follow while the new one was opening, opens again every subscription the server had acknowledged, each as a new generation, and opens its native producers and consumers again.
5. It attaches its transaction before it repeats any command that belongs to it.

The requests of steps 2 and 4 are sent before step 5 begins, not merely started, so the transaction can never overtake them on the ordered lane. Only then does the client repeat outstanding commands, each under its original execution reference and, for an append, its original expected position.

The new session can still refuse a restoration: a relay may no longer exist or not be running yet, the session may already hold as many requests as it admits, or a transaction a caller attached meanwhile may have made it refuse session-local requests. The client then reports a restoration failure for that subscription or clock, carrying the server’s message and the wait before the next attempt, and sends the request again on the same session. A restoration the session leaves unanswered past the request timeout ends that session instead, as any request that outlives the timeout does, and the next session restores it again. The wait starts at one second and doubles after each refusal up to thirty seconds. It keeps trying for as long as the session stays open and the client still wants the subscription or follows the clock; the next session starts again from one second. An attach answered DomainClockAlreadyAttached means the session already holds the attachment, so the client follows the clock on that session and reads its frames from then on.

The client’s event streams outlive the session. A caller reading subscription events, clock events, or notices keeps reading across the replacement: the interruptions mark the gap, and events of the new session follow them. Reading subscription or clock events opens a new session while one waits to be restored; reading notices never does, and waits for the session something else opens. A failed attempt to open one is returned to the reader, and its next read tries again.

The web console follows the same order on every connection: it selects its domain, opens again the subscription of every tab the server had acknowledged, and attaches its transaction before any ordered request it holds goes out. A restoration belongs to the connection that sent it; when that connection ends before the reply, the next connection restores the tab again. A transaction that has finished, or can no longer be attached, ends only the commands issued in it: restorations, subscription changes, and commands issued outside the transaction keep their place.

stateDiagram-v2
    [*] --> Connecting
    Connecting --> Open: a server accepted the session
    Connecting --> Failed: retry deadline passed
    Open --> Open: reply routed by request identity
    Open --> Redirecting: LeaderRedirect names the leader's endpoint
    Open --> AwaitingLeader: LeaderRedirect names no endpoint, or OutcomeUnknown
    AwaitingLeader --> Open: back off, then repeat under the same reference
    Redirecting --> Restoring: new session on the leader
    Open --> Lost: transport failure, session ending, or protocol violation
    Lost --> Restoring: a known server accepted a new session
    Lost --> Failed: retry deadline passed
    Restoring --> Open: clocks, subscriptions, and transaction attached again
    Failed --> [*]

A command moves through its own states within that session state machine:

stateDiagram-v2
    [*] --> Prepared: one execution reference, domain, expected position, and preview captured
    Prepared --> Sent: new request identity
    Sent --> Completed: CommandCompleted
    Sent --> Failed: RequestFailed, ExecutionReferenceConflict, ExecutionReferenceExpired, or TransactionTakenOver
    Sent --> Stale: PreviewStale
    Sent --> Sent: LeaderRedirect or OutcomeUnknown, repeated under the same reference
    Sent --> Reattaching: TransactionDetached
    Reattaching --> Sent: transaction attached again
    Sent --> Uncertain: session lost after sending
    Uncertain --> Sent: new session, same reference
    Uncertain --> GaveUp: caller deadline passed
    GaveUp --> Sent: caller repeats the same prepared execution
    Completed --> [*]
    Failed --> [*]
    Stale --> [*]

Row Subscriptions

A subscription is a read-only view of one relay, owned by the session that opened it. Its rows travel from the relay owner to the subscriber’s node as Arrow batches, and from there to the client as typed Row frames. Sessions owns the NSPL form and what a user sees; this section explains the protocol and data path behind it.

Opening A Subscription

SubscribeRequest names the domain, carries exactly one CREATE SUBSCRIPTION statement, and selects the subscription type. The statement carries the subscription’s name, relay, BLOCKING or DROPPING delivery, sample rate, and WHERE predicate; the server parses it once and compiles the predicate Model directly. The type is required, and Row, the only type, must be named: a request that omits it is refused rather than defaulted, and a value the server does not support is refused as UnsupportedValue rather than replaced by another encoding. A subscription’s type never changes.

Opening runs on the session’s ordered lane:

sequenceDiagram
    participant C as Client
    participant S as Session on node N
    participant G as Every other live node
    participant O as Relay fan-out on N
    C->>S: SubscribeRequest(domain, CREATE SUBSCRIPTION, Row)
    S->>S: validate against the scheduled relay definition, compile the predicate
    S->>S: take the next generation, wait for N to apply the current runtime revision
    S->>S: take N's interest lease on the relay and advertise it
    S->>G: wait until each publishes N's interest at the leased version
    S->>O: attach under an equal relay definition
    S-->>C: SubscriptionOpened(name, generation, Row, schema)
    Note over S,O: rows flow only after this reply is queued
    O-->>S: Arrow batches
    S-->>C: SubscriptionRows(name, generation, frames)

A refusal at any step releases everything taken before it, and the name stays free. The generation is a per-session counter that starts at 1, so a handle, the name together with its generation, is unique within its session only; a generation taken by a request that then failed is simply skipped. The visibility wait is what makes the reply meaningful: by the time a client reads SubscriptionOpened, every live node already routes the relay’s batches to the subscriber’s node, and an advertisement left over from before an earlier withdrawal cannot satisfy it.

The reply carries the schema and precedes every row. Delivery is held back until the reply is queued on the control lane, which the transport always serves first. When the reply cannot be queued, because the request was cancelled, the session ended, or the reply did not fit the session limits, the generation is abandoned before it delivers anything, so a client never receives rows it has no schema for. Batches the relay published during the opening are not delivered.

Encoding From Arrow Columns

The subscriber’s node encodes Row frames directly from the Arrow batch it received. It validates the batch against the announced definition once, then writes each selected cell from its typed column into the FlatBuffers builder; no JSON object, map of fields, or intermediate row is built. Every NSPL type has exactly one cell type, DATETIME travels as signed nanoseconds, floats keep their exact bits, and BYTES travels as raw octets. A sensitive field is written as RedactedCell in every row, whether or not it held a value, so the frame reveals neither the value nor its nullness. A null in a nullable field is NullCell; a list element is never null or redacted.

A frame holds rows of one concrete branch. Its branch key is written once per frame, with sensitive key fields redacted, and is absent exactly when the relay is unbranched. The encoder groups the selected rows of a relay batch into runs of adjacent rows with the same branch key, so interleaved branches produce one frame per run.

A frame holds at most 256 rows and at most the frame limit in bytes. When a row does not fit the frame being built, that frame is finished and the row starts the next one. A row that does not fit an empty frame cannot be delivered at all; the encoder then reports every selected row of that relay batch as skipped with cause EncodingFailed, and the subscription stays open. Each subscription encodes its own frames, so two subscriptions to one relay never share a frame.

Selection, Sampling, And Delivery

Each row passes the subscription’s predicate, then its sampling, then its delivery mode:

The delivery owner supplies its node’s bounded executor to predicate evaluation together with the batch’s domain-time snapshot. The selector borrows that executor for the call.

• Predicate. The WHERE expression is an ordinary BOOL expression over the relay record. A row is selected only where it is true, so a null result does not select it and is not reported. When a predicate is present, each relay batch takes one domain-time snapshot on the subscriber’s node, so every row and every volatile function of the batch sees the same instant. Rows whose evaluation fails are counted and reported once per batch as skipped with cause FilterFailed, and a batch whose domain time cannot be read is skipped whole with cause DomainTimeUnavailable.
• Sampling. BATCH SAMPLE RATE is applied to each selected row after the predicate, with a pseudo-random draw shared by every subscription on the node. A rate of 1 passes every row and a rate of 0 passes none.
• BLOCKING. A frame waits for room on the session’s subscription lane. While it waits, the subscription stops taking batches, and the relay’s fan-out waits for it: on the relay owner that holds back the relay itself, including its runtime consumers, and on another node it holds back that node’s ordered delivery from the owner, which waits within its relay admission bounds.
• DROPPING. A frame that finds the lane full is discarded, and its rows are counted. Before the subscription’s next rows, the session sends SubscriptionDeliveryLost with the count, if that report itself finds room; otherwise the next rows are discarded too and counted with the rest.

The subscription lane holds 16 frames shared by every subscription of the session. One slow BLOCKING subscription therefore also holds back the session’s other BLOCKING subscriptions and makes its DROPPING ones drop, while replies and session events keep flowing on the control lane.

Fan-Out Across Nodes

The relay owner is the only fan-out source. It delivers each admitted batch to its local subscriptions, to every node that advertises interest in the relay, and to its runtime consumers. It encodes one Arrow body for the whole subscription fan-out and shares it across nodes; a node that also hosts a runtime consumer of the relay receives the batch once, for that consumer, and its subscriptions piggyback on that delivery. Each batch reaches a subscription at most once. Delivery to a node is ordered per branch; batches of different branches may arrive in a different order than the owner published them. The subscription copy carries no acknowledgements, so a subscriber never holds back a source’s acknowledgement beyond the time its BLOCKING backpressure takes.

Generations, Endings, And Cleanup

Every frame about a subscription carries its handle. A name reused after deletion, or after the server ended its generation, opens a new generation, and the protocol never lets a frame about one generation be taken for another. That fence matters: the SubscriptionEnded of an earlier generation can still be queued when the same name is opened again, and because replies travel ahead of queued subscription frames, it can reach the client after the new generation’s opening reply.

The server ends a generation with SubscriptionEnded, its last frame, in two cases. RelayChanged means the relay was redefined, so the schema the subscription announced no longer describes its rows: a payload field’s name, position, type, nullability, or sensitivity changed, the relay became branched or unbranched, or its branch or branch key schema changed. RelayRemoved means the relay, or its domain, no longer exists. The relay fan-out ends every subscriber before a batch of another definition can reach it, so no row of a redefined relay is ever read against the earlier schema. Stopping and starting the domain, a rebuild that keeps the definition, a capacity or materialized-state change, and an ownership move keep the subscription delivering.

Deleting a subscription, or ending its session, withdraws it at once, even while its client reads nothing: the frames it still has queued are discarded, it stops waiting on the lane, it releases its relay receiver and its interest lease, and only then is the reply to the deletion queued. Nothing about the deleted generation follows that reply. A node advertises interest in a relay exactly while at least one of its subscriptions holds a lease on it, and every lease is released exactly once.

Gaps

A subscription is a live view. It keeps no offsets, is not persisted, and replays nothing. The protocol reports the losses the server knows of and states which ones it cannot report:

Restoration And Bounded Consumers

Nothing on the server restores a subscription. A client that wants one to outlive its session opens it again on the next session, which is a new generation with a new schema announcement, and must treat the time between as a gap. The Rust client does this for every subscription it holds that the server acknowledged and did not end, and fences its restoration by generation: a late reply for an attempt the caller has since cancelled is followed by a deletion before the name can be reused, and rows of a generation the client no longer holds are ignored. A restoration the new session refuses is reported as a restoration failure and sent again on that session after a growing wait, as Reconnecting A Session describes. Each interruption and refused attempt reaches the caller’s next read as soon as the client records it, without waiting for another change to the subscriptions it holds.

A generation the server ended is never restored, because a new session would not change why it ended: its relay was redefined, so the announced schema no longer describes its rows, or the relay no longer exists. The Rust client applies SubscriptionEnded as the frame arrives, before it queues the end for its caller, so the subscription reads Ended from then on, even to a caller that has not read the end yet. It stays Ended across later sessions until its caller subscribes again under the same name, which opens a new generation that announces the relay’s current definition, or deletes it. The end reaches the caller exactly once, after the generation’s other events: when the session ends before the caller reads it, the client discards that session’s unread events as it does for every subscription, reports no interruption, and reports the end itself. The one exception is a subscription whose events overflowed the client’s queue, for which the consumer overflow stays the last event of its generation.

A deletion asks the server only while an open session may hold the subscription. A subscription whose session ended, including one whose delivery had already failed, is deleted without a request, and so is an interrupted subscription the current session refused to open again: no session holds either, so the client simply stops wanting it and releases the name. A subscription the server ended is deleted without a request too: the server keeps an ended name only until it is reused or its session ends, and nothing of the ended generation remains to release. A deletion whose session ends before it is answered is complete, because the subscription ended with that session. A deletion that waited for an opening or a restoration still in flight asks the server only if that request opened the subscription. Only a name the client never held is always asked about, on a new session if the current one ended, and a refusal leaves that name free. Each subscription moves through Creating, Active, Interrupted, Restoring, DeliveryFailed, Ended, Closing, and DeletionFailed:

stateDiagram-v2
    [*] --> Creating: subscribe
    Creating --> Active: opening reply
    Creating --> [*]: refused
    Creating --> Interrupted: session lost after the server opened it
    Active --> Interrupted: session lost
    Interrupted --> Restoring: next session
    Restoring --> Active: opening reply for a new generation
    Restoring --> Interrupted: refused; reported, then retried on the same session
    Restoring --> Interrupted: session lost; retried on the next session
    Active --> DeliveryFailed: the client's event queue overflowed
    Active --> Ended: SubscriptionEnded
    DeliveryFailed --> Ended: SubscriptionEnded
    Ended --> Creating: subscribe under the same name
    Active --> Closing: unsubscribe
    Creating --> Closing: unsubscribe
    Restoring --> Closing: unsubscribe
    DeliveryFailed --> Closing: unsubscribe while its session is open
    Interrupted --> [*]: unsubscribe, which no session is left to answer
    DeliveryFailed --> [*]: unsubscribe after its session ended
    Ended --> [*]: unsubscribe, which needs no request
    Closing --> [*]: deleted, the session ended, or the opening it waited for was refused
    Closing --> DeletionFailed: deletion refused
    DeletionFailed --> Closing: unsubscribe again
    DeletionFailed --> [*]: the session ended

A consumer must stay bounded however fast rows arrive, and must never let an unread event hold back a reply. The Rust client charges each subscription event for the whole frame its rows keep alive and for the event itself. One subscription holds at most 32 events and one frame of the frame limit, 4 MiB, with the event that carries it; all subscriptions together hold four times as much, 128 events and 16 MiB of frames with their events. An event that does not fit drops that subscription’s queued events, reports a consumer overflow, and marks the subscription DeliveryFailed; the reader that routes replies never waits. An event stops counting once the application reads it, so a subscription whose events are read before the next one arrives stays within its own allowance however wide the server’s frames are, and overflows only when other subscriptions have filled the total. The gRPC receiver shares its receive buffer with a frame of 64 KiB or more rather than copying it, so a retained frame costs about its own size: four subscriptions each holding one 4.17 MB frame kept 16.78 MB live for 16.69 MB of frames, and the receiver keeps the allocation of the last large frame it read as its receive buffer after that frame is released. The CLI bounds its terminal output to 128 lines and 1 MiB, cuts a line above 8 KiB, and reports how many lines it omitted. It keeps reading every stream across reconnects, and when the client cannot reopen a session to restore subscriptions or clocks it prints why while the client keeps trying. The web console keeps at most 256 lines and 256 KiB per REPL and per subscription tab, and marks where it omitted earlier lines. It keeps the latest 256 commands and 256 KiB of its command history and the snapshot of the one domain it observes. It holds at most 64 requests of its controls waiting for a connection, carrying at most 4 MiB of text, and at most 256 held or awaiting their reply, carrying at most 16 MiB, and sends the ordered ones only while the server has room for them in flight; a request past either bound is not sent, and the control that issued it reports why. It drops the parts of a reply nobody awaits, such as a superseded completion.

A tab whose generation the server ended with SubscriptionEnded turns ended in the web console. It keeps its rows and the reason, and it is not restored on a later connection, which would not change why the server ended it. The operator resubscribes it under the same name, which opens a new generation announcing the relay’s current schema, or closes it. Closing an ended tab sends no deletion: the server keeps an ended name only until it is reused or the session ends, and nothing of the ended generation remains to release.

Domain Clock Attachment

A session can follow the clock of a domain. ATTACH DOMAIN CLOCK; and DETACH DOMAIN CLOCK; reach the server as AttachDomainClockRequest and DetachDomainClockRequest naming the active domain, and the reply to an attach carries the START generation and the clock as the serving node has it installed. Sessions owns the public contract and Domain Clock owns how an installation and accepted progress become observations; this section places the attachment in the protocol.

An attachment is not a subscription, and the protocol keeps the two apart end to end. It reads no relay, takes no interest lease, and has no name or generation of its own: it is keyed by its domain, so a session follows each domain clock at most once, and every frame about it names the domain. Its frames travel on the control lane rather than the subscription lane. Attach and detach run on the ordered lane, in order with the session’s commands. The attach reply is queued before delivery starts, so its state precedes every frame. If the installation has not changed since the reply and the node already holds a tick of that generation, its newest tick is the first frame after the reply. Each changed installation arrives as DomainClockObserved before any tick of its generation. A committed unassigned authority produces uninstalled and then the same mapping on reassignment; a direct authority move retains the mapping and produces no state frame.

DomainClockTicked carries the domain, generation, nonzero tick id, logical boundary, the authority’s UTC observation, and the serving node’s logical reading from a clock snapshot taken when the frame is built. The three timestamps use signed Unix nanoseconds. The serving reading lets a client anchor itself even when its local UTC differs from the cluster’s. The runtime publishes only accepted progress, fenced by generation and authority; a late attachment reads the newest accepted tick without waiting for another one. Tick ids can skip when the authority coalesces missed periods or a slow client’s pending tick is replaced.

A state frame waits for room on the control lane, and changes published meanwhile collapse into the newest installation. A tick occupies one replaceable slot per attached domain, including while the control lane is full, so a slow client never accumulates a tick backlog. Delivery withdraws a pending tick when state changes and rechecks installation before selecting another. Detach stops delivery before its reply is queued, so nothing about the domain follows that reply. When the serving node no longer has the domain, DomainClockAttachmentEnded with reason DomainRemoved is the last frame about the attachment.

Every node serves attachments from its own installation, which it derives from the same committed revision as every other node, and from progress it already accepted, so an attachment adds no interconnect traffic and survives nothing: it ends silently with its session. The Rust client reports the gap as an interruption when the session ends, clears its previous tick, and attaches every clock it followed again on its next session. An attach that session refuses is reported as a restoration failure and sent again on the same session after a growing wait; an attach it leaves unanswered past the request timeout ends that session instead, and the next session attaches the clock again, which a caller already waiting for clock events opens at once. A reply that the session already follows the clock moves the attachment to that session: the caller keeps the clock the previous session last reported, without a tick, until that session’s later state or tick frames arrive. When the new session’s attach reply arrives before the old session ends, as after a leader redirect, the attachment moves without an interruption: the client reports the clock only if it changed, keeps the newest tick of a clock still paced in the same generation, and reports a tick of the new session only when it is newer. A detach the client sends while a clock waits to be restored reaches the current session: a closed session is opened again first, which attaches the clock before the detach releases it, and an open session that has not restored it answers DomainClockNotAttached, after which the client no longer follows the clock. The client sends no restoration attach of a clock behind a detach of it: a detach waits until every restoration attach already decided for the clock is sent, so each reaches its session first, and a restoration that would attach the clock while the detach is in flight waits for the detach to end and attaches it only if the client still follows it. After DomainClockAttachmentEnded, a detach is answered DomainClockNotAttached and an attach is served as a new attachment. The shared binding exposes this event stream through nx_session_next_clock_event and a dedicated retained nx_clock_event handle, and the clock the client holds for each followed domain, starting with the one the attach reply carried, through nx_session_domain_clock and a retained nx_domain_clock handle. The web console attaches the selected domain clock once per session, detaches it on selection changes, and restores it on reconnect; its REPL sends the same typed requests for explicit attach and detach statements. These requests enter the console’s bounded session hand-off; a local refusal is shown to the operator and sends no request. Both requests are refused while the session holds a transaction, like every other session-local request.

A node answers an attach only once it has installed the cluster’s committed domains since it started. Before that it holds no domain, so it cannot tell one the cluster lacks from one it has not installed yet, and an attach that reaches a restarting node waits on the ordered lane rather than hearing that a live domain was not found. The server bounds that wait only by the session, as it bounds every wait, so a client that needs an answer sooner applies its own deadline. Ordered requests the session sends after the attach wait behind it. DomainNotFound therefore always means that the committed state the node installed holds no such domain, and the Rust client ends an attachment it was restoring when a new session answers it so, as it does after DomainClockAttachmentEnded.

The CLI’s domain-clock subcommand uses the Rust client’s typed attach reply and clock event stream. It prints the reply and then the same state, tick, interruption, and end lines as its REPL, including the fresh state after the client restores an attachment. Ctrl-C sends a detach request before the process exits; an attach refusal exits nonzero with its typed reason.

Producers

A session can open producers that submit typed batches to a client ingestor. Ingestors owns what an ingestor admits and how an outcome is decided, and Sessions owns the public contract of a producer; this section places producers in the protocol.

Opening, Submitting, And Closing

OpenIngestorRequest names a domain explicitly, the ingestor, the fields the producer expects, and the credit it asks for: how many batches and how many bytes it may have outstanding. The open runs on the ordered lane. The serving node checks the session’s transaction binding, the requested limits, the committed domain state and schedule, and the ingestor’s kind, reserves room in the session’s budget of 32 producers and 32 MiB and in its own node budget of 128 MiB, and attaches the producer through the node that executes the ingestor, which checks the expected fields against the execution it runs. The reply is ProducerOpened or a typed ClientProducerRefusal with a message; a refused open leaves nothing attached and reserves nothing. ProducerOpened carries the domain, the ingestor, and a ClientProducerDescription: the attachment identity, the input schema, the domain’s START generation, the endpoint contract digest, the ingestor’s window, ACK timeout and retry backoff, the granted credit with the largest batch one submission may carry, and whether admission is open.

The request identity of the open names the producer: every later request and frame about it carries that identity as its ProducerId, so a client needs no second identity space. The producer accepts batches from the moment its reply is queued. It belongs to the session: USE does not retarget it, and it cannot be opened while the session holds a transaction.

SubmitBatchRequest carries the producer and one canonical Arrow IPC stream as opaque bytes. The receive loop checks the batch against the producer’s credit, takes its share, and hands it to the producer’s task without waiting for anything; the batch’s reply, a SubmissionOutcome, follows whenever its outcome is decided. The session never decodes a batch: the node that executes the ingestor validates and admits it. A batch beyond the credit is answered at once as not admitted with CreditExceeded, and the producer is closed and later ended as ProtocolViolated; the batches it already submitted still receive their outcomes.

CloseIngestorRequest stops admission for the producer: its queued batches are refused, its admitted batches are answered as they resolve, and the close is answered Closed once every batch has its outcome, or NotOpen when the session holds no producer by that identity.

Outcomes, Admission, And Ends

SubmissionOutcome carries one terminal ClientSubmissionOutcome and a bounded, non-sensitive message. The four outcomes keep the boundary’s four facts apart:

• NotAdmitted. No row entered the graph. The cause is an InvalidBatch defect, Suspended, Busy, Draining, ProducerEnded, or CreditExceeded. Only Suspended and Busy are temporary.
• Completed. The batch’s source acknowledgement root resolved successfully.
• ProcessingFailed. The batch was admitted and its acknowledgement failed, with AckTimedOut or Rejected. Some effects may have happened.
• OutcomeUnknown. The batch may have been admitted, and no terminal result can be established: Interrupted when the ingestor’s execution stopped or the producer ended with the batch unresolved, OwnerLost when the node that executes it, or the connection to that node, was lost.

A transport receipt is never an outcome, and a reply is never sent before the outcome is decided. The session never replays a batch: the client decides, and only a temporary refusal is safe to send again unchanged.

ProducerAdmissionChanged tells the client that admission is Suspended or Open again. Changes are coalesced, so a client that reads late receives the newest state. ProducerEnded carries a ClientProducerEndReason and is the last frame about the producer; every batch it held has its outcome before it: EndpointChanged, EndpointRemoved, DomainStopped, Relocated, ShuttingDown, OwnerLost, or ProtocolViolated. Ingestors says which change causes each. A producer closed by its client ends with the close reply and no ProducerEnded.

During an ingestor alteration, admitted buffered batches keep their original outcome and credit while admission is suspended. The server drains them before publishing the replacement contract, including partial branch batches that must first pass through a shared relay. An intake hold and the full entity hold overlap, so no intermediate Open admits work between those phases. A failed drain keeps the committed contract and producer attachments; it does not replay an unresolved submission. A successful contract change delivers the admitted outcomes before EndpointChanged.

Ordering And Backpressure

Each producer has one task on the serving node that is the only writer of its replies and frames, so a batch’s outcome, the producer’s admission changes, and its end reach the client in the order the ingestor produced them. Outcomes, admission changes, and ends travel on the control lane. A batch keeps its share of the credit until its reply is queued, so a client that sends a batch only once an earlier reply freed room never exceeds its credit, and a producer whose client stops reading outcomes stops receiving room for new batches. The credit therefore bounds what a producer holds on the serving node, whatever the graph does, while commands, subscriptions, and clock frames keep moving beside it.

The ingestor admits a producer’s batches in the order they reach it. A batch refused as Busy or Suspended reaches it again only when the client sends it again after the backoff, behind every batch the client sent in the meantime, so batches outstanding at the same time can be admitted, and their output delivered, in another order than they were first sent. A client that needs submission order keeps one batch outstanding.

Forwarding To The Executing Node

A client may open a producer through any live node. When the serving node does not execute the ingestor, it opens a producer link to the node that does over the interconnect, reserves the producer’s bytes in its own budget, and relays the batches, outcomes, admission changes, and end; the executing node reserves the same bytes again for the batches it retains. Cluster Interconnect owns the link. The client sees the same protocol either way. The executing node admits a forwarded batch only after the serving node cleared it, so when the link is lost, whether the executing node crashed or stopped answering, the serving node still knows which batches may have entered the graph. Every batch it never cleared is answered NotAdmitted with ProducerEnded, which the client may submit again on a new producer without duplicating its effects; every batch it cleared has an OutcomeUnknown with cause OwnerLost; and the producer then ends as OwnerLost. When the serving node itself is lost, its sessions end with it, as When The Session Ends describes.

When The Session Ends

When the session ends, including because the node that serves it crashed, its producers detach: the executing node stops accepting their batches, drops the ones it held queued or awaiting their clearance without admitting them, answers nobody, and lets admitted batches finish in the graph. No server attachment survives the session. The Rust client retains the application’s desired producer and its submission ledger. It reports every batch sent without an outcome as OutcomeUnknown(SessionLost) and never resends it. On a new exchange it sends a fresh OpenIngestorRequest, compares the returned START generation, endpoint contract, schema, policy and grant with the first open, then publishes the new attachment to the same producer handle. A different generation or contract, a removed or stopped endpoint, or an incompatible schema makes the handle require an explicit new open. Temporary owner and capacity refusals are retried with bounded physical backoff while the new exchange lives. Closing or dropping the handle fences a late open and releases any attachment it created. The shared binding opens producers through these same Rust client handles, so its hosts restore them the same way; the web console does not open producers.

A serving node whose runtime has not yet passed its process-start linearizable catch-up barrier refuses producer and consumer opens as EndpointUnavailable. Its local domain snapshot may still show the state before a committed START, so it does not classify a missing or stopped endpoint as terminal until that barrier passes. The client retries this temporary refusal with its bounded restoration backoff.

Emitter Consumers

OpenEmitterRequest names the domain and TO CLIENT emitter, an exact field list, and requested outstanding batch and byte limits. The serving node validates the committed schedule and schema, reserves the requested bytes against the session’s 32 MiB and node’s separate 128 MiB consumer budgets, and attaches to the executing node before sending OpenEmitterOutcome.Opened. It refuses an open that cannot hold the emitter’s maximum IPC batch. At most 32 consumer handles belong to one session. The request identity of the open is its ConsumerId. The protocol and its FlatBuffers shape are identical over native gRPC and the console WebSocket. An opened reply also carries the domain START generation and the emitter endpoint contract fingerprint. The fingerprint is BLAKE3 of canonical emitter NSPL with flush cadence normalized to immediate, a 0xff separator, and the ordered output fields serialized as JSON. A flush-only change therefore keeps the live endpoint contract while changes to construction, publishing or ACK behavior do not.

ReadEmitterBatchRequest is a concurrent, potentially long read. It occupies one of the 64 ordinary in-flight request places but does not hold the ordered lane or receive loop. Its reply is a ReadEmitterBatchOutcome: Batch carries one canonical Arrow IPC stream, a stable delivery identity, a fresh attempt reference, source relay, opaque branch fingerprint, member count and execution-time snapshot; Ended says the attachment is gone. It does not distinguish relocation from removal or a changed contract. A client retaining the desired consumer reports the interruption, then checks a fresh open on the current exchange against its pinned generation and contract. A large reply uses the normal bounded transfer parts. Reading the reply never acknowledges it. The Rust client decodes a batch only when its stream declares exactly the Arrow schema of the fields the emitter declared at the open and holds one record batch of the member count, and treats any other stream as an unexpected reply. The registered client-emitter-deliveries property holds that decoder to the stream a client emitter’s sink writes: the declared schema, every delivered row in order, and every value and null.

SettleEmitterBatchRequest carries the consumer, attempt reference, and Ack, Retry, or Reject with a bounded non-sensitive reason. It runs beside the ordered lane and answers with Confirmed, StaleReference, WrongConsumer, InvalidReason, or ConsumerEnded. An ACK waits for that confirmation. ACK is idempotent while its bounded result remains; a reference revoked by retry, timeout, detach, or reassignment cannot settle a newer attempt. The emitter’s physical backoff controls retry. CloseEmitterRequest detaches and answers Closed or NotOpen.

A read the server answers with a batch assigns that attempt to its consumer, so a reply nobody waits for would strand the attempt until its ACK timeout. A caller of the Rust client that stops waiting for a read therefore leaves the read with its consumer, and the next read of the same attachment receives its reply instead of sending a read of its own; a read of an attachment that has since ended is dropped with it. Closing a consumer, like closing a producer, sends the close from a task of its own, so a caller that stops waiting still releases the attachment, and a second close, or a close while the handle waits to be restored, finds nothing attached and returns at once. The close also ends every read left with the consumer, including one left while the close ran, so no read outlives the close of its consumer.

Consumer and producer operations can share one session. A read awaiting output runs beside commands, producer submissions and their outcomes, clock observations, and consumer settlement. Each server attachment is bound to its session exchange. The Rust client keeps a desired consumer across reconnects and emitter relocation, reports one interruption before the next delivery for each exchange it lost, whether the loss or a reply racing it is observed first, and opens a new attachment only while the generation, contract, schema and granted behavior still match its original open. Removed or changed endpoints require an explicit new consumer. A delivery reference remains bound to the exchange that delivered it; the client never sends its ACK on the replacement exchange, and a settlement whose answer was lost is uncertain. The serving node forwards remote consumers on an authenticated relay-class duplex stream and reserves their granted bytes on both the serving and executing nodes. Loss of the stream or either endpoint revokes outstanding attempts. There is no durable cursor or consumer result history across an owner loss; upstream replay may be needed. Emitters owns the source acknowledgement guarantee and Cluster Interconnect owns the forwarding form.

The executing node exports active consumer and forwarding credit gauges, retained IPC work gauges, and retry, application ACK, and rejection counters. DESCRIBE EMITTER reads those values from the scheduled owner, including when the command enters through another node.

Resource Uploads

A resource archive travels on its own gRPC call, UploadResource, beside the session rather than inside it. The client streams one UploadStart frame, then the archive as UploadChunk frames in order, and half-closes the stream; the server answers with exactly one UploadReply. The start names the domain, the declared resource, the upload identity, and the exact size of the archive, and each chunk is non-empty and fits one frame. There is no manifest, digest, or finish frame: the leader hashes the archive as it stages it and verifies everything else when it installs it. The console WebSocket carries no upload stream.

The Rust client builds one deterministic archive in a temporary file before opening an upload call. Archive construction uses bounded 1 MiB read and write buffers. Only a successfully terminated and flushed archive becomes an upload body. An I/O failure abandons the partial file and returns BuildUploadArchive above its originating I/O error; cancellation abandons it without finalizing in a destructor or spawning cleanup work. The command deadline may cancel construction and retains its typed deadline outcome. Once construction succeeds, every transport attempt reads the same complete archive and keeps the same upload identity.

An upload is identified by its upload identity, not by an execution reference. The identity is chosen by the client, at most 128 bytes of ASCII letters, digits, -, _, and ., and is durable under the key of the authenticated user, the domain, and the resource. The first complete body admitted under a key binds it to that archive’s digest and takes the next version number of the resource. Repeating the same identity with the same archive joins the installation or reports its recorded outcome, with the same version, and the reply’s origin says whether this call installed it (Executed) or found it recorded (Recovered). The same identity with another archive fails as InstallationFailed with a message naming the conflict and, when the leader finds the identity recorded, the version it already holds. Upload identities never expire. Because the digest is computed from the body, recovering an uncertain upload means sending the whole archive again; the protocol has no request that asks for an upload’s outcome without it.

Uploads are leader operations. A follower reads the start frame, answers LeaderRedirect, and reads no chunk. The leader refuses what a stream gets wrong with a typed failure:

A body that ends early or overruns its declaration admits nothing and leaves its identity unused, so it can be sent again. Once the complete body is admitted, installation belongs to the cluster: it continues when the uploading connection closes, a new leader resumes it, and ResourceInstalled arrives only after every live node incarnation has verified and installed the same digest. If leadership moves while the upload installs, the reply is a LeaderRedirect, and repeating the same identity and archive on the new leader recovers the recorded outcome. Resource Versions And Bindings owns what admission, installation, publication, and completion mean, and Command Completion owns the completion boundary.

Authentication gives an upload its owner and nothing more: any authenticated user may upload a version of any resource declared in a domain, and the user scopes only the identity, so two users who choose the same identity string upload two versions.

Backup Downloads

A backup is a persistent command that leaves an artifact behind: its archive, which can be far larger than any reply. BACKUP therefore travels as an ordinary CommandRequest and is recorded under its execution reference like every other persistent command, while its archive travels on a call of its own, DownloadBackup, keyed by that reference. Backup And Restore owns what an archive holds and how it is laid out.

The leader runs the backup. Each domain is captured at its own applied revision and records whether its cut was quiesced, live, stopped, or configuration-only. The leader assembles the archive in its staging area, and only then completes the command. The CommandCompleted outcome carries the archive’s summary: its size and BLAKE3 digest, the capture time, the instant the node stops retaining the archive, whether resource bytes are included, the number of users, and each domain’s revision, cut kind and quiesce counters, section count, and bytes. The summary is part of the recorded outcome, so repeating the command under its reference returns the same summary and never assembles a second archive.

A download call carries exactly one BackupDownloadRequest, which names the backup’s execution reference, and the node authenticates the call from its metadata before it reads the request. A node that retains the archive answers with a BackupArchiveStart naming the archive’s size and digest, then the archive in order as non-empty BackupArchiveChunk frames of at most 256 KiB, then a BackupArchiveComplete. Every other answer is one frame:

An archive is retained on the node that assembled it until the first download whose completion frame was queued collects it, or until the retry validity of the backup’s reference ends. Any node reads that end from the reference’s UUIDv7 time, so an expired reference is refused the same way before and after its execution record is reclaimed. A node that does not retain the archive sends the client to the leader, and the leader, which then does not retain it either, answers from the recorded outcome. If leadership moves while a backup runs, the new leader resumes the applying command and assembles an archive of its own, and the archive the former leader assembled is never offered again.

Each running domain has its own quiesce timeout; the serial capture does not imply a total command timeout. Native clients bound the overall backup command wait with ConnectOptions::backup_wait_timeout (ten minutes by default, one millisecond through 24 hours), including every reply, redirect and reconnect. An exhausted wait returns UncertainCommand with the original reference and leaves admitted work recoverable. A new wait attempt retains that reference and its original selected domain, scope, resource inclusion and capture options; the client’s local output path is excluded from the server’s semantic request binding. A changed capture request conflicts, and an expired reference cannot admit a replacement execution. Waiting again does not extend server retry validity or the archive’s retention. Downloading begins only after the summary arrives and has separate per-frame stall bounds.

The node reads the archive only as fast as the client takes frames, at most four frames ahead of it, so a slow client holds back reads instead of growing what the node buffers for it. The call is not bounded by the request deadline, because a transfer can take far longer than a command; the Rust client bounds the wait for each frame instead. A download that ends early releases only its own hold on the archive, which stays retained for the client to download again. The protocol has no resume offset: a client starts a failed download again from the first byte, and accepts the archive only when the completion frame has arrived and every byte matches the summary’s size and digest.

The web console sends BACKUP on its session and downloads the archive over a console WebSocket of its own, from the console of the node whose session reported the backup. It verifies the archive exactly as the Rust client does, and hands it to the browser as a download only once it is complete and matches the summary. The browser tab records the backup’s execution reference, query and domain until the archive is downloaded, so a reloaded page repeats the command under the same reference, recovers the summary, and downloads the archive again while it is retained. DESCRIBE BACKUP never reaches a node at all: a client reads the archive file on its own machine, and a node answers the statement sent as a command with RequestFailed.

Restore Streams

A restore reads an archive that exists only on the client’s machine, so the archive has to reach the leader before the restore can even be planned. RESTORE therefore never travels as a CommandRequest: the client sends the statement together with the archive on a call of its own, RestoreBackup, and the call’s one reply carries the restore’s outcome. A node answers RESTORE sent as a command with RequestFailed. Backup And Restore owns what a restore recreates and the order of its steps.

The client streams one RestoreStart, then the archive in order as non-empty RestoreChunk frames of at most 256 KiB, and half-closes the stream. The start carries the request identity of the reply, the restore’s execution reference, the statement as canonical NSPL, and the archive’s exact size and BLAKE3 digest. The reference is a persistent command’s execution reference, and the request it is bound to is the statement together with the archive’s size and digest: a restore is admitted once under it, repeating the same statement and archive under it joins that restore, and another statement or archive under it is answered ExecutionReferenceConflict, like any persistent command.

The reply is a RestoreReply holding either a CommandOutcome, exactly as a command’s reply would hold it, or a RestoreUploadFailed that refuses the stream itself and leaves nothing changed:

A refusal of the stream is not recorded; sending the stream again stages the archive again. Every other answer is a CommandOutcome under the start’s execution reference:

• A follower reads the start, answers LeaderRedirect, and reads no chunk.
• A leader that finds the restore finished answers with its recorded outcome, whose origin is Recovered, and reads no chunk.
• A leader that holds the archive of the restore while it applies answers OutcomeUnknown with the StillApplying cause, and reads no chunk.
• Otherwise the leader stages the whole archive, verifies it, and plans the restore. A restore it refuses there is answered RequestFailed and is not admitted. A restore it admits, or finds admitted without its archive, runs to its outcome, which carries the typed RestoreReport: the mode, the archive’s size and digest, the capture time, what the users step did, each restored domain with its counts, required lifecycle status and required archived start generation and, for a dry run, its model run’s transaction impact report, and each step with its outcome.

A restore that failed at a step is RequestFailed, and its report names the step as Failed, the steps before it as Applied, and those after it as NotAttempted. A dry run reports every step as Planned, and is never admitted or recorded.

A domain whose restore has not published its complete runtime-state generation refuses START with a definitive command failure naming the incomplete installation. Planning rejects it before admitting a lifecycle step, and consensus checks the same gate at activation. Failure of the restore, release of its mutation lease, and node restart do not release this gate. Only completion of the whole installation does. With RESUME, complete publication, gate release and the transition to Running at the archived generation and clock mapping precede the successful outcome. A client can immediately inspect that lifecycle or attach its clock. A dry run reports the planned lifecycle without activation. DomainInfo in DomainList and DomainsChanged carries a required start_version beside its status and pace. Wire verification rejects missing required lifecycle and generation fields; the decoder never invents a default generation.

The call is not bounded by the request deadline, because an archive can take far longer to send than a command takes to run; the Rust client bounds each frame by it, and then the wait for the reply once the last frame was sent. Once the leader has received the whole archive, the restore is no longer tied to the call: a client that goes away does not stop it, and repeating the stream under the same reference returns its outcome. A call that ends before the whole archive arrived changes nothing and releases what the leader staged for it.

Only the leader the archive was streamed to holds it, until the restore finishes or the retry validity of its reference ends. If leadership moves, or that leader restarts, while the restore applies, the reply is a LeaderRedirect or the call fails, and repeating the stream on the new leader hands it the archive, from which it resumes the restore at its first step not recorded. The protocol has no resume offset: every repetition sends the archive from its first byte.

The web console streams a restore over a console WebSocket of its own, to the console of the leader its session follows. It reads the archive file the operator chose once to measure and digest it before the start, and again as it streams. The stream ends with the chunk that completes the declared size instead of a half-close, and the console watches for a reply that arrives before every chunk was sent. It repeats a redirected, interrupted, or still-applying restore under the same reference, statement and archive, and reports the outcome as unknown, naming the reference, when ten minutes of repetitions leave it unknown.

Node Stop And Restart As A Client Observes Them

Shutdown And Recovery owns the phases of a node stop. This section describes what a client sees at each ending and what it recovers.

Graceful stop. When a node begins to stop, it closes admission on its session gRPC and console listeners and ends the sessions they carry. Every request a session had not yet admitted is decided cancelled before admission and never begins an effect. Work that was admitted keeps running and no reply follows for it: a transaction commit continues, and the node waits for it until its shutdown deadline. Subscriptions and domain clock attachments stop with their session, and an upload stream still waiting for its client ends at once. A console session receives SessionEnding with reason ServerShuttingDown after the control frames it had already queued, then a normal close. A backup download or a restore stream on a console WebSocket of its own ends at once with close code 1001, and a restore admitted with its whole archive goes on without its call. A gRPC connection is cut when admission closes, so a native client sees its stream fail, as after any transport loss, rather than the ending frame. Either way the client reconnects to another node and repeats each outstanding command under its execution reference, which joins the admitted work or returns its retained outcome.

Forced ending. A repeated signal, an expired shutdown deadline, SIGKILL, or a crash ends the connection without warning. A command whose reply was not received may or may not have been admitted, and the client cannot tell which from the connection. It repeats the command under the same execution reference on whichever node now leads. If the command was admitted, the new leader resumes it from its durable progress, and the repetition joins it or returns its retained outcome with origin Recovered; if it was not, the repetition admits it for the first time.

Leadership change. A gRPC session on the former leader receives LeadershipObserved naming the new leader, or no leader during an election, and its next leader operation is redirected. A command that leadership moved under returns OutcomeUnknown with cause LeadershipLost, and repeating it on the new leader recovers its outcome. A console session on the former leader ends with a SessionEnding that names the new leader, and the console reconnects there.

Full restart. When every node stops and restarts from its own storage, the cluster reopens its committed state, including every retained execution record, transaction, and upload record. Repeating an execution reference admitted before the restart returns its retained outcome without repeating its effect, a COMMITTING transaction resumes from its recorded step, and an open transaction whose inactivity deadline passed while the cluster was down expires once it runs again. Nothing a session held survives.

A subscription therefore always has a gap across the loss of its session. The server cannot report that gap, because the session that would carry the report is the one that was lost. The Rust client reports it instead, as an interruption of every subscription and clock the lost session held, before it restores them.

Bindings For Other Hosts

A host language reaches the protocol in one of two ways. It loads the shared Rust binding, nervix-client-ffi, whose C ABI is crates/client-ffi/include/nervix_client.h, or it implements the protocol itself from the schema. The binding is the supported path for C, C++, Python, JVM, and Ruby hosts. It drives the Rust client’s own state machine, so request correlation, leader redirects, reconnection, execution identity across retries, and desired-subscription restoration are decided once, in the library, and a host never retries, reroutes, or reinterprets an outcome on its own. A separate SDK per language would have to reimplement each of those decisions and be qualified again.

The binding’s lifecycle follows from that choice:

• Handles. A host opens a session with nx_session_connect and releases it with nx_session_free; outcomes, events and batches read from a session stay valid after it is freed, and its producers, consumers and deliveries keep it running until they are released too. Every object a function hands out is owned by the caller and has exactly one release function, and a borrowed pointer stays valid only until the object it was read from is released.
• Execution identity. nx_session_prepare captures one command and its durable execution reference before anything is sent. nx_session_execute runs it, and running the same prepared execution again after an uncertain, cancelled, or timed-out call recovers the command’s outcome instead of running it twice. Every failure of nx_session_execute names that reference.
• Threads. Each session runs on a runtime of its own, with two threads the library owns. The library never calls host code, so no host function ever runs on a library thread. Every blocking call runs on the calling thread until it completes, fails, or is cancelled, and a session may be used from several threads at once; commands of one session still run one at a time. A call made from a thread that is itself driving a Tokio runtime cannot block on the library’s runtime and aborts the process, so a Rust host calls the binding from a blocking thread.
• Cancellation and deadlines. Every blocking call takes an optional nx_cancel. A token triggered from any thread makes the call return NX_ERROR_CANCELLED, and a token created with a deadline makes it return NX_ERROR_DEADLINE once the deadline passes. Cancelling a wait never rolls back admitted work.
• Bulk access. A Row event keeps the verified frame it was decoded from, after checking the batch against its subscription’s schema. nx_event_frame borrows that frame without copying it, a column accessor copies one whole column into caller-provided memory in one call, and nx_event_cell_varlen borrows one string or bytes value. Events are reference counted: nx_event_retain adds a reference, and the event and its frame are freed with the last nx_event_release, from any thread.
• Clock events. nx_session_next_clock_event uses the same cancellation and deadline rules as the subscription wait, but returns a separate nx_clock_event reference. Its kind, domain, generation, state, paced mapping, tick and end-reason accessors expose the fields each event carries. An accessor for a field its event lacks returns NX_ERROR_TYPE. Retain and release preserve the event and its borrowed domain name across threads until the last reference ends.
• Attached clocks. nx_session_domain_clock reads, without blocking, the clock the Rust client holds for a followed domain, or NULL for a domain it does not follow. After an attach completes it is the clock the attach reply carried, so a host reads the generation and committed mapping of a running clock before its first tick, and every later observation and tick replaces it in the order the session received them, so it is never older than an event the host has taken. Its accessors expose the domain, generation, state, paced mapping and newest tick, and its projections answer logical time, physical waits and admission with the Rust client’s arithmetic. A read never changes; retain and release share it across threads.
• Producers and consumers. nx_session_open_ingestor and nx_session_subscribe_emitter open the Rust client’s producer and consumer with the host’s exact fields, built with nx_fields, and the credit it asks for; a refusal carries its typed nx_open_refusal. A producer submits a batch and waits for credit, and its outcome is taken with nx_producer_rejoin, listed with nx_producer_pending and let go with nx_producer_release, each with the Rust producer’s semantics: a cancelled wait leaves the submission and its outcome with the producer. A consumer hands out reference-counted nx_delivery handles; releasing one settles nothing, and nx_delivery_ack, nx_delivery_retry and nx_delivery_reject return the server’s typed settlement. Both keep the Rust client’s restoration: a consumer reports NX_ERROR_INTERRUPTED once per gap, and a handle whose endpoint changed reports NX_ERROR_REOPEN_REQUIRED with its typed reason. The binding keeps no acknowledgement, credit or reconnect state of its own.
• Batches. A producer’s batch is built with nx_batch_builder from the host’s columns, one level of a column’s type at a time, and every buffer is copied before the call returns. A delivered batch is decoded the first time it is read, held to the consumer’s schema and member count, and read the same way. Every batch is its canonical Arrow IPC stream, which nx_batch_ipc and nx_delivery_ipc borrow and nx_producer_submit_ipc accepts from a host’s own Arrow tooling, so the binding encodes nothing a second way and the observation API stays free of Arrow.
• Typed failures. A failing call returns an nx_error whose kind separates an invalid argument, a failed connection, a failed session, an uncertain outcome that carries the execution reference or leaves a settlement unconfirmed, a server refusal, a deadline, a cancellation, a protocol violation, a type mismatch, a session or handle that ended, an interrupted consumer, and an endpoint that has to be opened again. The binding classifies the Rust client’s error_stack::Report<ClientError> by its current context, and a failed backup download by the BackupDownloadError beneath that context, never by text, and nx_error_message returns the report’s whole chain.

The binding connects with the Rust client’s default options. It exposes no seeds, timeouts, or certificate authority, so it reaches a node over plaintext and connects to it directly. It exposes commands, completion, subscriptions and their events, domain clock events and attached clocks, bulk row access, and producers and consumers of client endpoints with their batches, but not the typed transaction status, inspection, choice lookups, notices, or leadership. A host executes ATTACH DOMAIN CLOCK;, reads the clock the reply carried with nx_session_domain_clock, and reads later observations and ticks through the clock-event wait; the Rust client restores the attachment after reconnect. An attach reports a refusal as a failed disposition carrying the server’s reason, not as a typed refusal. Its Row column accessors cover scalar, string, and bytes fields; a list field of a subscription reports its levels, and its values are read from the borrowed frame with generated code, while an endpoint batch’s lists are read one level at a time.

Independent implementations exist as qualification clients rather than supported SDKs. The Go client speaks native gRPC with flatc --go output and google.golang.org/grpc, and the TypeScript client speaks the console WebSocket with flatc --ts output, unchanged under Node.js and Bun. They correlate replies by request identity, follow a leader redirect with the same execution reference, and fail rather than report success when an outcome is unknown. Neither restores subscriptions or domain clock attachments after reconnection; that recovery belongs to the Rust client and, through it, to every binding host. The FlatBuffers runtimes for Go and TypeScript have no verifier, so those clients check each frame’s identifier and every union discriminant and required value they read, and the Rust verifier remains the only full structural verification. Conformance Evidence lists the probes, the runtimes they cover, and how to run them.

Measured Costs

The protocol’s costs were measured against the accepted FlatBuffers Row form and recorded, with their raw results and reproduction commands, in tests/client-wire-16-performance.md and tests/evidence/client-wire-16. The figures below summarize that record. They were taken on a shared host with a release server and a debug measurement harness, so they show where the cost of each stage lies and bound regressions. They are not throughput guarantees, and the record does not attribute a difference between two of its runs to the protocol unless it says so.

Construction and verification. The server writes each Arrow column value directly into FlatBuffers tables, with no record-shaped JSON object, map of fields, or intermediate row. A frame is verified once, and every later read borrows from it:

Verification and borrowed reads allocate nothing; reading one cell per frame takes about 0.2 µs. Owning every cell of 1,024 narrow rows allocates about 172 KB, and rendering them as the JSON text the CLI and console display allocates about 724 KB, with escaped field names computed once per batch rather than once per row.

Copies and retained buffers. Variable-length values are copied once, into the builder, and the finished buffer becomes the queued frame without another copy. The gRPC sender copies a frame into its output buffer, about 77 µs for the 4.67 MB large batch. The gRPC receiver copies a frame below 64 KiB into an allocation of its own and shares the receive buffer of a larger one. A WebSocket message must be owned, so the console transport copies a frame once more, about 79 µs for the same batch. The 256-row and frame-byte limits keep a frame from growing into an unbounded buffer: the large batch held 8.38 MB of live allocation for 4.67 MB of frames, the byte-limited batch 4.28 MB for 4.22 MB, and live allocation returned to its baseline once the frames were released.

End to end. The record repeats the workload of the original Protobuf baseline. In its last run, on a clean commit, the release server answered a native command in 2.9 ms at p50 and 3.9 ms at p99, delivered a typed HTTP input to a Row subscriber in 23.6 ms and 28.9 ms, served a live graph snapshot on the console WebSocket in 176 µs and 397 µs, and completed a one-file upload in 15.0 ms and 15.6 ms. Of the command time, about 2.1 ms at p50 is preparation in the debug client, dominated by its statement parser, and about 1.0 ms is the prepared request and its reply. Control commands sent while a subscriber was paused or draining slowly completed in 4.5 ms and 4.9 ms at p50, because replies take the control lane ahead of queued rows. The median command request and response were 176 and 1,640 bytes, and a one-row subscription frame 1,320 bytes.

Transport and bindings. Opening a session on one node took about 2 to 4 ms over plaintext and 4 to 5 ms over TLS; the per-command difference between the two did not stay stable across runs. Through the C ABI, borrowing a verified frame takes about 60 to 70 ns, retaining and releasing an event 80 to 90 ns, and copying the states, fixed-width values, or variable-length values of a 100-row column into caller buffers 64 to 122 µs. Through CPython and ctypes, each call costs about 0.6 to 6 µs, which is why a host reads a batch column by column rather than value by value.

The record also sets regression alarms derived from the Protobuf baseline: a p99 of 6.5 ms and a 1,900-byte response for a command, 90 ms and 1,500 bytes for a typed Row delivery, 3.6 ms and 4,600 bytes for a graph snapshot, and 75 ms and 200 bytes for an upload acknowledgement. They are qualification thresholds, not promises.

Guarantees And Non-Guarantees

The protocol guarantees:

• Correlation. A reply completes exactly the request whose identity it names, in whatever order replies arrive, and never a request of another session.
• Bounded, verified frames. Every frame fits the session limits and is verified before it is read. A reply larger than a frame arrives whole or not at all.
• One outcome per identity. Repeating a command under the execution reference that admitted it joins the admitted execution or returns its retained outcome. It never applies the effect a second time, and a reference whose record was reclaimed is refused rather than run again.
• Exact proof. The outcome of an append or a commit comes only from the record of that exact request, never from the state of the transaction as a whole.
• Written order. The requests that change a session run in the order the client wrote them.
• Responsive control. Replies and session events are never queued behind subscription rows, and a command waiting for its completion boundary delays no concurrent request.
• Typed rows. A Row frame carries every value with its declared type and nullability, withholds every sensitive value, and names its concrete branch. Its schema precedes it, and no row of a redefined relay reaches a subscription announced under the earlier definition.
• Named losses. A subscription reports the rows the server skipped, the rows a DROPPING subscription discarded before its next rows, and its end. The Rust client and its bindings also report the loss of the session a subscription lived on, and every attempt to restore it that a new session refused.
• One outcome per batch. Every submitted batch receives exactly one terminal outcome, never completed before its acknowledgement root resolved, and a producer’s end follows every outcome it owed.
• Verifiable archives. A backup’s archive arrives with the size and digest its recorded outcome names, and a download that ends early leaves it retained for another attempt.

It does not provide:

• Exactly-once submission. A batch whose outcome is unknown or failed may have had effects, and nothing replays or deduplicates it. The application keeps its source data and decides.
• Exactly-once delivery. A subscription is a live view with at-most-once delivery per admitted batch. Rows in transit can be lost without a report while relay ownership moves or a node-to-node delivery fails, and a source may redeliver a record that the graph then passes again.
• Automatic rollback. A failed transaction keeps the applied prefix of its execution steps, and a cancelled waiter keeps its admitted effect. Undoing either is a new command.
• Durable subscription replay. Subscriptions keep no offsets and are not persisted. A restored subscription starts with the rows that arrive after it opens again.
• Guaranteed restoration. The Rust client opens a subscription or endpoint again, or attaches a clock again, only when the serving exchange accepts it. A subscription whose relay no longer exists stays interrupted until its caller deletes it. A producer or consumer whose endpoint contract or domain generation changed requires a fresh application open. There is no durable consumer cursor.
• A required Arrow client library. Rows arrive as typed FlatBuffers cells; a client needs only code generated from the schema.
• Columnar subscriptions. SubscriptionType has one value, Row. A server refuses a type it does not declare as UnsupportedValue and never substitutes another encoding.
• A durable session. A session, its request identities, its selected domain, its subscriptions, its domain clock attachments, its producers, and its transaction binding end with the connection. Only execution records, transactions, and upload records outlive it.
• Liveness detection. Neither transport sends keepalives, and the server has no idle timeout. A client that needs to detect a half-open connection bounds its own waits.
• A durable archive. A backup’s archive lives in the staging area of the node that assembled it and is lost when that node stops, or once its reference’s retry validity ends; the recorded outcome outlives it. Only the first complete download receives it.
• Authorization. Every authenticated user may run every command in every domain.

Observability

The protocol makes a client’s view of its own work explicit rather than inferred:

• Correlation. Every reply names the request identity it answers, and every command outcome names its execution reference. A client that logs both can pair a reply with the request it sent and with every later attempt of the same command. The CLI prints the execution reference of a command whose outcome is unknown, so an operator can see what a later retry will recover.
• Dispositions and origin. A command outcome states its disposition, and whether it was executed now or recovered from the durable record. OutcomeUnknown names its cause, a conflict names what differed, and a stale preview names both identities.
• Transactions. SHOW TRANSACTIONS lists live transactions and retained tombstones with their owner, domain, state, pending count, progress, age, and idle time, from any node. DESCRIBE TRANSACTION reads one transaction’s full impact report; see Transaction Quiescence And Impact Inspection.
• Subscriptions. A client learns of its own losses from SubscriptionDeliveryLost, SubscriptionRowsSkipped, and SubscriptionEnded, and the Rust client reports a lost session as an interruption of each clock and of each subscription the server had not ended, and each refused attempt to restore one as a restoration failure with the server’s message. Client::subscription_lifecycle reads each subscription’s state, including Ended for one the server ended. The web console shows each tab’s state on the tab: pending, active, interrupted, restoring, ended, resubscribing, or closing. Each node exports nervix_session_subscriptions, the number of subscription leases it holds per relay, and nervix_session_subscription_dropped_rows_total, the rows its DROPPING subscriptions discarded, both labeled by domain and relay; see Metrics And Observability. Skipped rows are reported only to the client.
• Leadership. LeadershipObserved tells every session which node leads and where to reach it, and SHOW CLUSTER STATUS shows each node’s availability.
• Producers. A client learns every batch’s outcome and every admission change and end of its producers. SHOW INGESTORS and DESCRIBE INGESTOR show each client ingestor’s attached and forwarded producers, its outstanding batches and bytes, and the batches holding its window, and the node that executes it exports them as metrics with the answered batches by outcome and cause; see Metrics And Observability.
• Backups. A completed backup’s outcome carries its archive’s summary. The node that assembled the archive logs its assembly and its collection by a complete download at info, naming the execution reference, and every other end of a download at debug.

The server keeps its own records at the levels the logging contract assigns. Listener startup is logged at info. A reply replaced by a rejection, a reply the session ended before it could queue, a subscription notice or loss report that does not fit a frame, domain clock attach, detach, and end, and producer opens, refusals, closes, and ends are logged at debug. A session that falls behind the notice bus, an event or snapshot that does not fit a frame, a failed revert at a clean close, and a failed remote subscription dispatch are logged at warn. Session opening and closing, individual requests, and rows are not logged, and no log line carries a payload value. The server exports no metric for sessions, requests, rejections, or authentication attempts; a client that needs them measures them itself. Producer counts are exported by the node that executes the client ingestor, not per session.

Retained Subscription And Producer Dependencies

A subscription generation retains its domain lifecycle when delivery starts. Filtered batches read the currently installed clock through that capability, including a domain started after the subscription opened. Unavailable-clock skipped-row outcomes remain unchanged. It also retains the node’s executor and passes it to predicate evaluation for bounded execution. Its dropped-row counter is resolved with the generation. A native ingestor endpoint resolves every public batch-outcome metric child at startup and retains its execution’s acknowledgement trackers. These internal ownership rules change neither session framing nor client recovery behavior.