Domains And Time

Every runtime graph in Nervix runs inside a domain.

This chapter describes the public language and runtime behavior. See Domain Clock for the internal clock architecture.

Nervix currently supports:

• CREATE PACED DOMAIN <id> WITH PERIOD <duration> SKEW <duration> [PLACEMENT <policy>];
• CREATE UNPACED DOMAIN <id> [PLACEMENT <policy>];
• CREATE DOMAIN <id> [PLACEMENT <policy>];

CREATE DOMAIN <id> is the short spelling for CREATE UNPACED DOMAIN <id>.

Domain creation is never transaction content. A transaction is bound to one already-existing domain, so CREATE DOMAIN runs on its own before BEGIN; queueing it is rejected. See Replicated NSPL Transactions.

Paced Domains

Paced domains maintain a domain clock.

PERIOD must be positive and no larger than 18446744073709551615ns. SKEW may be zero but must fit in that same 64-bit nanosecond duration range. Invalid durations are rejected before the domain is stored. Duration text anywhere in NSPL whose spans could add up to 18446744073709551615 seconds, however they are written, is rejected: a duration literal when the statement is parsed, and a duration string argument, such as the delay of PERCENTILE_LINEAR_HISTOGRAM, when the statement is validated.

While the domain is running:

• Nervix produces domain ticks
• paced ingestors admit records whose effective timestamp falls within SKEW of an eligible logical tick center

Centers are origin + n × PERIOD, where origin is the committed logical start and n is nonnegative. The current domain mapping determines the reached frontier. Only the newest 256 positions through that frontier are eligible; position zero is eligible immediately on start. SKEW is an inclusive logical-time distance: an event exactly at either edge is accepted. Overlapping windows admit their union, while gaps between windows reject events.

Admission reconstructs these positions from the installed mapping. Delayed, missing, or coalesced tick notifications do not change eligibility. A hypothetical future center is never eligible: an event can extend past the last reached center only by its declared SKEW. An event before the origin can be admitted within the origin’s tolerance while that position remains eligible. TIME RATE changes when the frontier advances in real time; it does not scale source timestamps, PERIOD, or SKEW.

Paced time is also important for expiration:

• branch TTL uses domain logical time in paced domains, including the relay owner’s cluster-wide branch-presence decision
• materialized-state cleanup follows the same logical-time rule
• deduplication expiry, reorder and correlation retention, window completion, and guest-requested WASM timeouts are logical deadlines on the same clock
• branch activity is the domain time at which a branch’s last input was accepted, so an idle relay owner or processor never spends part of a branch’s TTL waiting for that input

The maintenance scan that releases expired branches and the snapshot wakeups that persist branch-local state run on the physical clock, and so does the WASM executor’s own safety timeout. Their predicates still compare logical deadlines, so a scan never expires a branch early: its interval is release latency measured after the logical boundary, reported separately from the TTL itself. See TTL.

Deterministic Roto UDFs preserve reproducibility when paced input is replayed at an accelerated time rate. See the deterministic-by-default and VOLATILE contract, including common-subexpression reuse and the rule that user code is never constant-folded.

Unpaced Domains

Unpaced domains do not produce ticks.

Ingestors in an unpaced domain admit records as they arrive, and branch TTL uses wall clock time.

Placement Default

Every domain has a fallback placement policy. It applies to directly connected runtime-node pairs that no named placement rule claims. Because relays are scheduled runtime nodes, a producer-to-relay edge and a relay-to-consumer edge each receive the default independently. The clause is optional on every domain-creation form, and omission means NEUTRAL so ordinary scheduler heuristics remain active:

CREATE PACED DOMAIN production
  WITH PERIOD 1s SKEW 100ms
  PLACEMENT REQUIRE COLOCATION;

CREATE UNPACED DOMAIN archive
  PLACEMENT SUGGEST SEPARATION;

Change the default for the session’s active domain with the nameless domain alteration:

ALTER DOMAIN SET PLACEMENT PREFER COLOCATION;

A newly effective REQUIRE COLOCATION default can relocate runtime nodes during graph activation. Soft policies influence only future placement decisions and do not migrate existing assignments. See Placement Policies for named overlays, path-gating, rank resolution, and enforcement.

Start And Stop

Domain lifecycle commands apply to the active domain:

• START;
• START AT NOW [TIME RATE <float>];
• START AT <rfc3339_timestamp> [TIME RATE <float>];
• STOP;

Important runtime consequences:

• START; resumes from persisted domain-owned runtime state when a source supports it
• START AT NOW reinitializes paced time and domain-owned source offsets from current wall clock
• STOP preserves persisted runtime state
• START clears materialized relay state for the active domain before new execution proceeds

Successful lifecycle commands are completion boundaries. START returns after every current live node has prepared the same revision and every configured listener and assigned source has completed startup. STOP returns after intake and runtime work have stopped remotely, including when it is the last running domain. A dependent connection, observation, or publish can follow immediately through any live node.

The lifecycle state, active paced-clock anchor, and one clock authority are replicated. The authority identifies a concrete incarnation of a named cluster node and carries a revision that advances whenever ownership changes or is revoked. After leader failover, reconciliation uses that committed state; a completed transactional START therefore remains effective without re-executing its commit step, and leadership transfer alone does not establish a new anchor or authority.

Every live node installs that committed mapping before it builds executable work for the domain. The installed capability is bound to the domain name and START generation. A joining node uses the existing logical origin, physical UTC anchor, and rate; it does not establish a new anchor from its join time. An unpaced domain receives actual UTC through the same domain-bound capability. A missing domain, stopped clock, uninstalled paced mapping, or task bound to an earlier generation is a lifecycle error and never selects wall time as a paced fallback. Reads on one node do not move backward within a generation.

Nervix assumes every cluster host maintains UTC closely enough for the deployment’s event-time contract. Operators must run and monitor a clock-synchronization service on every node. The committed mapping prevents authority changes and joins from inventing a new logical origin, and each node clamps its own reads so they do not move backward within a generation. It does not make simultaneous reads on different hosts identical or establish a distributed total order. Host UTC offset affects a newly established START AT NOW, unpaced observations, and each node’s projection of a paced mapping; TIME RATE multiplies that projection error. SKEW is event-admission tolerance around logical tick centers, not an allowance for unsynchronized hosts.

The authority begins producing once its own node installs the mapping and fence of the consensus revision that assigns it. Each live peer receives progress once it has installed that fence. A node installs a reassigned fence ahead of its graph installation, so neither the authority nor a peer waits for its own or another node’s graph installation or activation. Join, restart, owner loss, and membership change can select a new authority without changing the clock mapping. When the authority’s node becomes unavailable, the consensus leader commits a replacement and retries a proposal its new leadership could not yet accept; if that node also led consensus, the remaining voters elect a leader first. A progress report is accepted only when its START generation, authority revision, node incarnation, and authenticated peer match the committed authority. Duplicate, delayed, reordered, or superseded progress is ignored and cannot create a domain. STOP revokes the authority in the same replicated lifecycle transition; a later START commits a new generation and mapping. Automatic ALTER quiescing keeps the authority and clock running.

Progress crosses the interconnection as a typed HTTP/2 management request in the reserved progress subquota. For each domain and each remote node that installed its clock, the authority retains one replaceable pending report and permits one delivery attempt at a time. A newer frontier replaces an older pending frontier while connection capacity or a response is delayed. The interconnection bounds all in-flight progress requests independently of the stream and memory reservations for health, cancellation, admission, and terminal outcomes. A joining or reconnected node receives the newest retained frontier once it advertises that it installed the required domain clocks.

Each progress attempt has a two-second physical deadline. A failed attempt retries the newest frontier after a 200-millisecond physical backoff, and authority shutdown cancels both the request and that backoff. The response only confirms that the authenticated receiver evaluated the report; the committed generation and authority fence decide whether it is current. Progress never replaces the committed mapping or moves a node’s logical clock. Its wire value contains typed logical and UTC timestamps and never contains a process-local monotonic instant.

An explicit START AT timestamp must fit exactly in signed Unix nanoseconds. The inclusive range is 1677-09-21T00:12:43.145224192Z through 2262-04-11T23:47:16.854775807Z; valid RFC 3339 values immediately outside those endpoints are rejected by the command. Nervix converts accepted text to a timestamp at the language boundary and carries that timestamp through persistence and runtime state without reparsing it. An offset is converted to UTC, fractional digits past the ninth are truncated, and a leap second, which RFC 3339 writes as :60, reads as the instant one second past the second before it, because Unix nanoseconds have no leap seconds.

TIME RATE accepts every positive finite f64, including scientific notation such as 5e-324 and 1.7976931348623157e308. Zero, negative values, infinities, and NaN are rejected. The committed start mapping remains the sole clock anchor for that run. Tick delivery records progress but never re-anchors logical time, so a delayed tick cannot move time backwards.

Clock projection rounds fractional logical nanoseconds down. Converting a logical target to a physical wait rounds fractional physical nanoseconds up, which prevents a deadline from firing early. Timestamp, duration, boundary, and tick-id overflow are errors. If tick production wakes after several periods have passed, it emits one tick at the latest due logical boundary and next targets the first future boundary.

Domain cadence and data-lifecycle policy create logical deadlines from a bound domain clock. Connection, retry, cancellation, drain, and other operational policy create physical deadlines on the process monotonic clock. The two deadline kinds cannot be interchanged. A logical wait revalidates its domain and generation after every wake and returns both the logical instant that became due and a fresh execution-time snapshot. Cancellation is a separate typed outcome.

HTTP EVERY polling and generator EACH scheduling have an immediate first occurrence; Prometheus EVERY polling first becomes due after one interval. Later occurrences remain anchored to that initial logical schedule. If work takes long enough to miss several occurrences, Nervix runs once for the newest due instant and advances directly to the first future boundary. It never bursts through the missed intervals. Prometheus sends the due instant as its query evaluation time, then takes a fresh domain execution snapshot when the response is decoded and evaluated. Skipped polls and generators during quiescing do not stop or re-anchor the cadence.

COLLECT FOR, FLUSH EACH, and the Iceberg COMMIT EACH are domain-logical durations, in emitters exactly as in every other flush-based node. A paced domain’s TIME RATE therefore accelerates or slows them, and a stopped clock cannot silently turn any of them into a wall-clock wait. Each source relay and concrete branch owns its collection deadline; each concrete output branch owns its route deadline. A deadline starts only when data enters an empty buffer and is cleared when that buffer is released by time, size, force flush, shutdown, or error handling. For COMMIT EACH that buffer is the Iceberg emitter’s staged files: its deadline starts when a flush stages rows while none are staged, not at the domain time those rows’ batch was accepted at, and a retry or drain commits them without reading the clock.

FLUSH IMMEDIATE instead uses a physical 100 µs minimum on the process monotonic clock, regardless of domain rate. The ingestor source-group bound of 1,024 messages and 5 ms idle close are also physical and run before route-local flushing. Connector retry and backoff timers remain separate physical controls; none of these operational waits is projected through the domain clock.

An emitter’s publish cadence and its publish attempts are separate state. Retry backoff, the acknowledgement keepalive that holds an upstream MODE ACK source open while a sink is unavailable, the sink acknowledgement timeout, the stop and drain deadlines, and an HTTP Retry-After a server returns are all physical: a paced domain never shortens or lengthens the real wait an external system asked for. A failed attempt keeps the emitter’s pending batches, their acknowledgements, and its unchanged cadence deadline; while a retry is scheduled the emitter waits for the retry instead of the cadence, and a forced flush or drain still releases the buffer. The maximum batch size releases a buffer that has reached it as soon as no retry is pending.

Following A Domain Clock

A client session can follow the active domain’s clock instead of inferring it from the domain list:

ATTACH DOMAIN CLOCK;
DETACH DOMAIN CLOCK;

The attach reply, and a frame after later changes of the installed clock, carry the START generation and the clock as the serving node has it installed: stopped, uninstalled, unpaced, or paced with its PERIOD, SKEW, logical origin, UTC anchor, and TIME RATE. Changes made before a frame is sent arrive together as the newest clock. A paced clock is the committed mapping this chapter describes, so a client projects the domain’s logical time for its own UTC observation with the same arithmetic every node uses, including rounding down, and reconstructs the admission window from the origin, PERIOD, and SKEW. A driver of a paced simulation can therefore choose TIMESTAMP AT values the ingestor admits, such as the newest reached center, and compute how long to wait until a logical instant; the Paced Simulation Drivers are two runnable such drivers. The projection uses the client host’s UTC, so the synchronization requirement above extends to such a client: its offset from the cluster’s hosts, multiplied by TIME RATE, shifts the logical instants and admission centers it projects, and the offset itself moves the end of every physical wait it computes. The session also receives each newest accepted tick as a replaceable frame with its id, logical boundary, authority UTC observation, and serving node’s logical reading. A client can pace from those accepted boundaries even if its own UTC is offset; the serving node’s reading gives it a cluster-side anchor.

Both statements are session-local and are refused while a transaction is open, and a session follows each domain clock at most once. See Domain Clock Attachment for the delivery contract and Rust Client Library for the helper that performs this arithmetic, which the shared C binding exposes to its hosts through nx_session_domain_clock.

Execution-Time Snapshots

Each accepted unit of domain work reads its domain clock once and uses that execution-time snapshot for every expression in the unit. This applies to ingestion, processor input and output programs, branching, deduplication, ordering, correlation and window expressions, inferencer mappings, emitter filters, construction, VALUES and SQS FIFO-group expressions, and subscription filters. Buffered emitter batches and retries retain the snapshot assigned when the batch was accepted. Processor flushes, scheduled callbacks, and other later executions receive a new snapshot when that work begins. A message released from REQUIRED WAIT also begins again with a fresh snapshot.

A message error records the snapshot of the operation that failed, and its error-route SET program uses that same instant. Generator output and WASM rows without a source token receive that instant as both ingestion watermarks. WASM rows with a source token retain the source metadata. Guest initialization, batch processing, timeouts, flushing, and state lifecycle calls each receive the snapshot of their owning execution. Roto and WASM hosts have no context-free time entry point: their now() or domain-time import returns that supplied snapshot and never reads UTC on its own.

Generated timestamps at external boundaries follow their declared clock class. An omitted Sentry event timestamp uses the emitter batch’s domain execution time, while an explicit timestamp is preserved. OTEL VALUES use domain execution time, while observed_time_unix_nano records actual UTC at export. HTTP-date Retry-After values are interpreted against actual UTC and converted to a physical monotonic wait.

Automatic Model-Alteration Quiescing

Nervix derives a quiesce level from the complete validated model diff. ALTER SCHEMA, ALTER WIRE ... SCHEMA, relay schema changes, and relay branching changes use an internal paused lifecycle state when their domain is running. Dynamic changes such as relay capacity do not pause the domain. There is no NSPL PAUSE or RESUME statement.

Before changing the live graph, the leader validates the complete candidate graph without writing it while holding the domain’s exclusive ALTER lock. For an entity-pause alteration, Nervix gates the affected relays or stops the affected ingestor instances on every live node, then waits only for their rings and target-node work to drain. Unrelated graph paths continue to run. For a domain-pause alteration, Nervix instead stops domain ingestion and generators on every node, keeps the processing graph and domain clock alive, force-flushes ingestor route, processor, and emitter output, and waits for ingestors, generators, ACK roots, and emitter buffers to drain. Both waits are condition-based and bounded to 60 seconds by default.

After a successful drain, Nervix atomically installs the model batch, replaces the schedule while ingestion is still withheld, and resumes the domain on the new graph. A timeout or cutover failure leaves the mutation unapplied, restores the old graph, and automatically resumes it with a clear outstanding-work error. An ALTER on a stopped domain only validates and persists the new schedule. Pure CREATE and DROP batches keep the immediate schedule-rebuild behavior. An all-no-op batch writes and publishes nothing and never pauses.

For an HTTP emitter, this means a pending request must finish at the origin selected when it was admitted before a new method, path, body mode, or client takes effect. A failed drain keeps that emitter definition active. An operator facing a persistently unavailable origin may restore it, or STOP, change the configuration while stopped, and START. Stopping leaves no prepared HTTP request or retry state to carry over; an acknowledged source may redeliver unresolved attached work, including work the destination applied before its successful response was lost.

Pause is not a restart: domain clock state, start version, broker offsets, branch identity, and eligible handoff residue are preserved across the quiesce cycle.

Ingestion Timestamps

RESTORE ... RESUME retains the archived start generation, latest start point and paced clock mapping after installing all compatible runtime state. That mapping projects the downtime, so TIMESTAMP AT admission uses the restored clock’s reached windows. Unpaced domains resume without a mapping. A default restore stays stopped. A later normal START advances the generation and clears materialized relay rows while preserving compatible WASM state, source offsets and branch lifecycle. To start from the archived frontier with a new mapping, use a default restore and explicit START AT '<frontier>' TIME RATE <rate>. DESCRIBE DOMAIN exposes the generation, start point and mapping; LIST DOMAINS exposes pace, status and generation. See Backup And Restore.

Every ingested record receives internal ingestion metadata, including mandatory low and high watermarks with nanosecond precision.

Timestamp sources:

• TIMESTAMP NOW selects the installed domain’s logical time when the ingest group is delivered. Buffered intake released after automatic quiescing receives that delivery time. In an unpaced domain, the same clock capability supplies actual UTC. In a paced domain that delivery time is tested against the admission window like any event time: it lies less than one PERIOD after the newest reached tick center, so a SKEW smaller than PERIOD rejects the records delivered between two windows, and a SKEW of at least PERIOD admits every delivery.
• TIMESTAMP AT <field> preserves the decoded DATETIME field as the event time. Connector timestamps selected when this clause is omitted in an unpaced domain, and broker metadata, retain their external values. None of these values is multiplied by TIME RATE.

In paced domains, ingestors must declare a timestamp source explicitly. In unpaced domains, timestamp metadata is still recorded, but it is not used to gate admission.

Both ingestion watermarks initially equal the selected event time and travel with the record through relays and across nodes. Timestamp selection and admission share one delivery snapshot. An unknown, stopped, or uninstalled domain clock produces a lifecycle error; a paused domain withholds delivery until its ingestion path resumes.

Window processors also use this metadata. Duration windows evaluate input event time from the record low watermark. Emitted aggregate records receive a low watermark equal to the minimum input low watermark in the emitted window and a high watermark equal to the current domain time at emission.