Deterministic Interconnect Simulation

The interconnect has a seeded network simulation built on Turmoil. Each Nervix transport runs on a simulated host with its own Tokio runtime, virtual clock, TCP stack, and DNS name. The hosts talk over simulated links that the test can delay, partition, hold, and repair. The code above the socket is the production code: TLS 1.3 mutual authentication, HTTP/2 pools, typed rkyv envelopes, Arrow IPC payloads, relay admission and reconciliation, connection lifecycle, and quota accounting all run unchanged. The same seed and inputs make the same decisions in the same simulated order, so a network failure that once exposed a defect can be run again exactly.

This chapter owns that simulation: where its code lives, how it builds, what a simulated host controls, how a run is supervised, the fault model, how runs are compared and replayed, the scenarios and what each one proves, the commands, the CI budget, the qualification evidence, and the limits of every claim. Three rules hold throughout.

• The simulation tests interconnect contracts. It never claims that a whole Nervix node, a consensus group, gossip, the domain clock, a store, or an external system was simulated.
• A replay compares decisions, not bytes. A trace records identities, admission decisions, outcomes, and the simulated time at which they happened. It never records key material, certificate bytes, ciphertext, or payload values.
• A simulated host crash is not a process crash. It tears a runtime down with ordinary destructors, so durability and SIGKILL behavior keep their real-process evidence.

The harness is test code outside the layer order: it may name any layer, and product code never names it. Product contracts stay with the chapters that own them: node-to-node transport with Cluster Interconnect, in-process ordering under Shuttle with Data-Plane Concurrency, node stops and crash recovery with Shutdown And Recovery, domain time with Domain Clock, and the Cucumber cluster harness with Integration Test Lifecycle.

What Each Suite Establishes

Four suites exercise the interconnect. Each establishes something the others cannot, and none of them substitutes for another.

Code Ownership

Every product seam has exactly one production behavior, and the production value is the default a node uses. A seam exists so that a controlled environment can supply a value; it is not a switch between two production behaviors. The harness never reaches into product state: it constructs a transport through its public API, supplies a clock and an entropy source, and observes the result through requests, snapshots, and outcomes.

Build Modes

Execution Primitives owns backend and target selection, permission contracts and the difference between compile-time probes and actual execution evidence. This chapter owns what the Turmoil hosts simulate, the faults and the completed scenario evidence.

The simulation is its own build. Its scheduler replaces the real network, so it can never share a compilation with the production or Shuttle builds.

Only the Turmoil recipes pass --cfg tokio_unstable, and just validate-primitive-boundary rejects the flag in any other recipe, in Cargo configuration, in a workflow and in a build script. It rejects --cfg turmoil everywhere, and a bare turmoil in a cfg predicate: the mode is the turmoil feature, never a cfg every crate of the build would read. With it, Turmoil seeds each host runtime’s scheduling from the simulation seed and configures an unhandled task panic to shut that host’s runtime down, which fails the run. Without it, scheduling would not follow the seed and a panicking task would only print, so the runner refuses to start. The workspace declares the cfg in its check-cfg list so ordinary builds accept the attribute without enabling it. Turmoil 0.7.2, locked in Cargo.lock, is used without its unstable filesystem feature.

Validation keeps the modes apart:

• just validate-execution-mode-dependencies fails when the normal dependency graph of the workspace, or of any package built on its own, with or without default features, contains Turmoil, Loom, Shuttle, a Shuttle wrapper or Deloxide, or enables an execution mode on the primitive boundary.
• just validate-primitive-boundary rejects every path to Tokio’s or Turmoil’s sockets, Tokio’s timers, or the standard library’s monotonic clock and sockets outside nervix-primitives, and a turmoil dependency outside the boundary unless it is optional behind the package’s own turmoil feature, as the interconnect’s harness takes it. The simulated path therefore cannot name an operating-system socket or clock that would escape its host.
• just validate-execution-mode-conflicts builds nervix-primitives with every pair of the loom, shuttle, turmoil and deloxide modes and with all four, and nervix-interconnect with Shuttle while its execution dependency selects Turmoil, and requires the diagnostic naming the modes in each. It builds each mode for the browser’s target and requires the boundary’s own diagnostic first, and builds the NSPL formatter, a product binary, with each mode and requires the diagnostic that a modeled build is a test artifact.
• just lint runs Clippy over every target of both packages with the turmoil feature, beside the production and Shuttle lints.
• just test runs the workspace with all features while excluding every package that offers a scheduler mode, runs those packages with default features, and then runs just test-turmoil.

The Simulated Host

A simulated host is a named Turmoil host running one production transport. Everything the transport owns is constructed inside the host: its credential bundle, executor, peer resolver, transport, and peer targets. Live sockets and transports are never shared between hosts. Hosts share only bounded readiness and completion signals with each other and with the observing client.

Sockets And DNS

In the Turmoil build the primitive boundary’s nervix_primitives::net selects Turmoil’s TcpListener, TcpStream and UdpSocket, and every transport is built with PeerResolver::simulated, which answers from the simulated host’s DNS table through nervix_primitives::net::lookup_host, a lookup the boundary offers in that mode alone. The listener, every outbound dial, and peer resolution therefore use simulated TCP and DNS. A peer endpoint such as server:7443 resolves to the address Turmoil assigned to the host named server, in IPv4 or IPv6 by configuration, and a name the table does not hold fails as a name that does not exist. The scenarios register peers at their advertised endpoints, so, as in production, every connection attempt resolves the peer’s name again inside its setup deadline and dials the answer. Nothing in the simulated path opens an operating-system socket, constructs the production Hickory resolver, reads a resolver configuration or hosts file, or sends a DNS packet, so a fault scenario cannot escape to the real network and no resolver state is shared between hosts. The layers above the socket, tokio-rustls, h2, the envelope codec, and Arrow IPC, are the production code. These scenarios exercise the transport over simulated names; Hickory’s DNS protocol, cache, and failure handling are checked by the resolver crate’s own tests against local DNS authorities outside the simulation, as Name Resolution describes.

Bounded CPU Execution

The transport encodes and decodes through nervix-execution, which submits every admitted CPU job through the boundary’s task::spawn_cpu. In the Turmoil build that runs the job as a Tokio task on the host’s scheduler instead of on the blocking pool, so the job body is one scheduling step of the simulation. The executor is the mechanism’s only caller: the boundary check rejects it anywhere else, so no caller runs CPU work on the scheduler without admission. Queue slots, per-class worker reservations, memory charges, and cooperative cancellation follow the same policy as production. A queued caller that leaves releases its slot and charge; a running job keeps its worker and charge until it exits, even when its caller leaves.

Turmoil can vary the order of tasks around a job, but not instructions inside it. Races inside a job need Shuttle or real threads. Storage jobs still run on a real blocking thread and are outside the simulated path. Only bounded executor probes and interconnect codec jobs are approved for the simulation; storage jobs, external drivers, and unbounded CPU work are not.

Time

A simulated host has three distinct notions of time, and the simulation controls each one differently.

Outside a simulated host the simulated UTC clock has no reading, and a certificate check fails instead of falling back to the host wall clock. The certificate fixtures’ epoch, 2027-01-15T00:00:00Z, lies after the wall clock of any machine the suite runs on today, so a check that consulted the host clock by mistake would find every fixture certificate not yet valid. The transport clock is physical infrastructure time only: it never defines domain time and never stamps connector arrivals.

Identities And Entropy

Node names are fixed per host. Each transport’s process epoch and relay grant identifiers come from a seeded source, SimulatedEntropy, whose stream is named after the host. One seed therefore allocates the same identities on every host in every process, while two hosts sharing the seed draw different values. Fixtures choose their own delivery identities and acknowledgement registrations, such as the channel incarnation and ACK identifier of each relay case and the run of the node that registers it, so a trace can name them.

Certificates are generated for every run with fixed validity windows: 2027-01-01 to 2028-01-01 for the transport fixtures, and windows placed relative to the epoch for the certificate-clock checks. Validity decisions therefore replay while key bytes differ between runs. Rustls and AWS-LC keep their real cryptographic randomness; it changes ciphertext and never a decision the trace records.

The transport’s concurrent maps use per-process hash seeds. Every walk over one of them that causes effects runs in semantic key order, as Cluster Interconnect lists, so map order cannot reach a decision.

The Runner

SimulationConfig::run builds one Turmoil simulation from one configuration, lets a setup closure add hosts and clients, and steps it on a dedicated scheduler thread until every client has finished or a bound expires.

Configuration

Every run is built from exactly these inputs, and a failure record stores all of them.

The runner validates the configuration before it starts a thread. A zero duration, an epoch before the Unix epoch, a build without tokio_unstable, or a real-time bound too large for the machine’s clock each fail with their own error, naming the scenario and seed.

Host Lifecycle And Supervision

A host is registered with its software, a closure that Turmoil calls when the host starts and again each time the host is bounced. Each call builds the host’s transport afresh. A client is a future that runs once; the run completes when every client has returned. Scenarios therefore add an observer client that waits until each host has reported that it finished, so hosts, which never complete a run themselves, still bound it.

• HostSupervisor::run spawns a host’s body as a task on the host’s runtime and returns the body’s error or panic as the host’s result. A failed host fails the next step.
• A panic in any other task on a host shuts that host’s runtime down, which fails the run. The runner reports the first panic raised on the scheduler thread, with its message and location, rather than Tokio’s generic shutdown message. A panic that something caught, such as a supervisor or Tokio while dropping a task, is reported the same way.
• A host shuts down every transport it bound before its body returns, and checks the transport’s connection counts after shutdown.
• SimulationConfig::run_with_control, used through ScenarioRun::simulate_with_control, calls a control closure after every step that did not end the run. A fixture uses it to crash, release, repair, and bounce hosts at a protocol milestone a host published, never at a guessed step count.
• When the steps complete, the simulation and every host runtime are dropped in a cleanup phase of their own, inside the same real-time deadline. A panic while dropping fails a run whose steps all succeeded.

How A Run Ends

The real-time deadline is fixed before the scheduler thread starts and is measured outside it, so it expires even when a host blocks that thread and simulated time cannot advance. The blocked thread is then abandoned and ends with the process that runs the attempt. A cleanup failure that follows a failed run is written to standard error beside the run’s own failure. The real thread and wall clock decide only when a test fails, never how a simulated request proceeds.

Fault Model

A fault plan is written with Turmoil’s link and host controls. Each control is an abstraction of a network or host event, and a scenario’s claim is limited to the abstraction it used.

The model has consequences that every scenario respects:

• Simulated TCP does not retransmit. A message dropped by a partition is lost for good, so after a repair an established HTTP/2 session can be unusable. The established-exchange scenarios require both peers’ protocol pings to retire unusable sessions and allow reconnection without restarting either host or manually replacing a pool. No scenario infers kernel retransmission timing or packet-loss recovery.
• Holding and one-way partitions cannot be combined on one link, so scenarios use them separately.
• Turmoil 0.7.2 has no disconnect control. TCP closure is exercised by shutting a listener down or crashing a host.
• A request whose deadline expires while its messages are held stays expired when they are released.
• Turmoil’s own socket count can keep failed connection attempts after a partition. Scenarios measure session cleanup with the transport’s connection counts instead.
• Random link failures are disabled. A scenario’s faults are its written plan, applied at named protocol points.

Semantic Traces And Replay Rules

A run records a semantic trace: an ordered list of events, each with the simulated time it happened at, the host that recorded it, and what was decided. Fixtures record events at decisions: a connection became ready, a setup failed, a deadline expired, a relay was admitted or fenced, an epoch changed, a resource was released. The same scenario, seed, and inputs must record the same trace.

• An event names identities, admission decisions, and outcomes. It never contains key material, certificate bytes, ciphertext, or payload values, which differ between runs without changing a decision or do not belong in a record. Payload assertions compare values without printing them.
• An event never contains a value that describes the process rather than the simulation, such as a process identifier, a real time, a memory address, or an iteration over a hashed collection. process_dependent_trace is a deliberately divergent scenario that records its process identifier, and a regression test requires it to be recorded as diverged.
• Every committed seed runs twice, each time in a fresh process, and the two traces and outcomes must be identical. The first differing event, or the first event only one run has, is a divergence and fails the seed. A divergence that can affect behavior is a defect to fix, never a seed to drop.
• A replay compares its outcome and its trace with the record, event by event.

Two nondeterminism sources remain outside the trace by design. Rustls and AWS-LC randomness changes ciphertext and key bytes but no decision. Tokio numbers tasks across the whole process, and a runtime tears its tasks down in an order derived from those numbers, so when a simulated host crashes, the order in which its connections close depends on every task the process created before. Each attempt and each replay therefore runs in a fresh process, where that count starts from the same state. Tokio’s per-runtime scheduling seed is not a residual source: Turmoil derives it from the simulation seed because the build sets tokio_unstable.

Scenarios, Seeds, And Attempts

A scenario is one case of a test: a name unique within the test, a written fault plan, and its committed regression seeds. Scenario::check takes a function that builds a seed’s configuration and a function that exercises it, and runs them in the mode the environment selects.

The driver sets a fifth variable, NERVIX_TURMOIL_ATTEMPT, on the process it starts for one attempt, and clears the other four there.

Seed Selection

Committed seeds are small and deliberate: one or a few per case, distinct across cases, and listed beside the fault plan they exercise. A case’s committed set grows only by a seed that exposed a defect, committed with its fix, so the regression set records what the simulation has found. The sweep explores beyond them: just test-turmoil-sweep runs sixty-four consecutive seeds from 1000 through every case by default. A seed that fails in a sweep leaves an ordinary record, and once the defect it found is fixed, it joins the committed seeds of its case. Seed 1036 of the receiver restart with a delayed relay response is the first such seed.

Fresh-Process Attempts

For each seed, the driver writes a record marked running, then starts the test binary again with the exact test name, --exact --include-ignored --test-threads=1 --nocapture, and the attempt’s request. That process runs the case once and writes back its outcome and trace. The driver waits for the run’s real-time bound plus 30 seconds, then kills the process. A process that ends without writing its result yields an unreported outcome holding its ending and the last 40 lines of its output. A process that passes without running the requested case is a driver error, because the test and case names no longer match. After both attempts pass and agree, the record is removed.

Failure Records

A failed, panicked, diverged, or unreported seed leaves a JSON record at <failures>/<test>/<case>-seed-<seed>.json, where the test and case are reduced to lowercase letters and digits joined by dashes. The test then fails with a summary of the trace and the command that replays the record.

A record holds identities, inputs, outcomes, and semantic events only. It never holds certificates, key material, ciphertext, or payload values, so it is safe to attach to an issue or a CI artifact.

Commands

Every recipe builds with the repository’s kache compiler wrapper and passes --cfg tokio_unstable itself.

The replay prints a verdict for every run:

The record and replay path is checked itself. injected_failure_is_recorded_and_replayed_in_a_fresh_process injects a failure into a scenario in a fresh process, requires exactly one record, replays it in a second fresh process, and requires the recorded outcome and trace. diverged_and_panicked_runs_leave_records starts two deliberately failing scenarios, one whose trace names the process that ran it and one whose check after the simulation fails, and requires a diverged and a panicked record. just test-turmoil-replay-check does the same through the documented commands. The suite still contains no failing test: the injection exists only in the environment of the processes those checks start, and the failing scenarios are ignored unless a check starts them.

An Example Investigation

The failure below was produced on purpose with the recorded injection, which exercises the same path a real failure takes. The output is quoted from that run, with paths shortened to the repository root.

NERVIX_TURMOIL_INJECT_FAILURE=5s just test-turmoil-simulation \
    transport::network_disruption_respects_deadlines_and_repairs_authenticated_service --exact

1. Read the failure. The test fails with the case, seed, run ordinal, outcome, trace, and the replay command:

   simulation transport::network_disruption_respects_deadlines_and_repairs_authenticated_service / "partition before connect" seed 61 (first run) failed: simulation partition before connect seed 61: host or client failed: injected harness failure at 5s simulated time
     0.001s client PartitionBeforeConnect installed; peer registered
   failure record: target/turmoil-failures/transport-network-disruption-respects-deadlines-and-repairs-authenticated-service/partition-before-connect-seed-61.json
   replay it in a fresh process: just test-turmoil-replay target/turmoil-failures/transport-network-disruption-respects-deadlines-and-repairs-authenticated-service/partition-before-connect-seed-61.json
   

   In CI the same record is in the job’s turmoil-failures artifact. Its run field says whether the first attempt failed or only the repeat did. A repeat failure or a diverged outcome points at nondeterminism before it points at the product.

2. Compare the build. The record’s build names the commit, lockfile hash, and toolchain. Check out that commit when the current branch has moved; the replay reports every remaining difference.

3. Replay it.

   just test-turmoil-replay target/turmoil-failures/transport-network-disruption-respects-deadlines-and-repairs-authenticated-service/partition-before-connect-seed-61.json
   

   recorded outcome (first run): failed: simulation partition before connect seed 61: host or client failed: injected harness failure at 5s simulated time
   replayed outcome: failed: simulation partition before connect seed 61: host or client failed: injected harness failure at 5s simulated time
   trace: all 1 recorded events replayed identically
     0.001s client PartitionBeforeConnect installed; peer registered
   turmoil replay: reproduced the recorded outcome and trace
   

   The command exits with a failure status because the failure reproduced. The trace ends at the last decision before the failure: the client registered its peer 1 ms into the run, and the injected client failed at 5 seconds, before the twelve-second partition ended.

4. Narrow it. Read the case’s fault plan and the fixture’s events around the last recorded one. To see what happened between two events, add a temporary tracing subscriber that stamps lines with turmoil::sim_elapsed() and prints the lengths of byte arrays rather than their contents, then rerun the replay. Compare two runs of a divergence line by line from the first differing event.

5. Fix it at its owner. A product defect gets its fix and regression at the owning boundary, with a Cucumber scenario when the behavior is public. A harness defect gets its fix in the runner or driver. Replay the record again until it prints turmoil replay: the recorded failure did not reproduce, commit the seed to its case’s committed seeds, and run just test-turmoil.

Seed 1036 is the recorded history of this procedure. The first sweep found that its two runs of the receiver restart with a delayed relay response diverged after the simulated crash while they shared one process. Tracing both runs with simulated timestamps showed that the crashed receiver’s connections closed in a different order, and that the order followed Tokio’s process-wide task numbering. The driver moved every attempt and replay into a fresh process, and seed 1036 became a committed seed of that case.

CI And Budgets

CI runs the simulation as a job of its own, turmoil, only for a pull request labeled turmoil, beside the default tests and the shuttle and loom jobs. Its build enables the turmoil feature and tokio_unstable, so it shares no compilation with them, and a failed case’s record is its own artifact. The job runs just coverage-native-extras test-turmoil, which executes the canonical gating suite once under LLVM source instrumentation, then runs just test-turmoil-replay-check independently. Replay qualification contributes no counters. The instrumented build lives under target/native-coverage-build-turmoil; its failure records and suite logs, together with the ordinary replay’s target/turmoil-failures, are uploaded as the turmoil-failures artifact when the job fails.

The coverage-turmoil artifact retains the product-source report, completion.json, executions.jsonl and export.log from the collection’s own attempt. Only a report beside a complete record establishes successful coverage; a failed or interrupted attempt retains its failure evidence. Each fresh simulation process inherits LLVM_PROFILE_FILE with %p-%m, so its counters are merged with the parent and the other attempts rather than overwritten. The exporter matches every raw profile to a retained executable by build ID and includes Turmoil-selected product paths in primitives, execution and interconnect. It excludes dependencies, generated files and harness files below tests, examples or benches directories.

After the suite and replay qualification pass, the job publishes an advisory Turmoil patch coverage comment through just coverage-patch --scope turmoil. Its marker is independent of the ordinary coverage comment, and publication checks that the PR is open at the tested head. A fast simulation run cannot replace the ordinary report, and a failed collection never supplies a successful report. Coverage percentages and comment failures remain advisory; the simulation, replay and collection verdicts remain mandatory. The ordinary coverage/CRAP job uses ordinary reports only. just coverage-turmoil <output> uses this same collector for local inspection.

The job uses the shared native CI linker, so adding --cfg tokio_unstable to RUSTFLAGS preserves Wild linking. It installs flatc, because the interconnect’s test build takes its generated vocabulary values from nervix-arbitrary, whose WASM protocol bindings are generated when that crate is built.

Every bound nests inside the next, so the innermost expired bound names the stuck run, and the job still has time to upload what it found. This follows the convention of The Suite Watchdog.

The sweep’s 25 minutes is three times the 8m11s its default 64 seeds took after the build, one attempt at a time, on the 32-thread workstation where it was introduced.

Measured wall times for the qualified revision:

Scenario Matrix

Every case below is a Scenario in the simulation target. An ordinary run executes its committed seeds twice each, in fresh processes; the sweep replaces the seeds without weakening any assertion.

The simulation target also holds 26 runner and driver checks, such as the supervision, bound, clock, entropy, and trace checks and the fresh-process record and replay checks, and two ignored scenarios that only those checks start. The library-level checks use fixed seeds outside the driver and are rerun by name: the execution strategy’s queue, admission-order, and charge checks; the seeded rkyv round trips of seeds 1–12, alternating IPv4 and IPv6; and twelve certificate-clock checks, one of which compares the traces of two fresh processes.

Authenticated Exchange

The exchange cases prove that the simulated path is the production path. A server binds, shuts down, and rebinds its listener in the same host, so listener cleanup is observable. The client then sends a typed request carrying an Arrow batch and the same batch as a relay payload, and each side decodes it with the production codec and checks the values without printing them. The three-host case adds a second peer and a rejected dial whose DNS name the certificate does not carry, so authentication failure is proven over the same connection path that succeeds. The typed Arrow reply archives its native row count with the vocabulary’s fixed-width 64-bit adapter, matching the product’s count representation. The rejection assertions inspect the TransportError at the report’s current context; relay timeout, cancellation, and indeterminate-delivery assertions do the same. This keeps the simulated failure class independent of the report’s diagnostic text while production transport and wire causes remain available underneath it.

The certificate-clock checks, beside the transport, prove that time-dependent authentication follows the simulated clock on both peers: not-yet-valid and expired boundaries of the local and the peer certificate, rejection by the client’s and by the server’s Rustls verifier under skew, the drain of an accepted session at the earlier expiry on both sides, and the setup deadline of a stalled handshake.

Link Faults

The two setup partitions stay in place for twelve seconds of simulated time, so bounded reconnect behavior is observable: setup failures are counted, liveness times out, and no connection appears. The established-link faults first complete an authenticated exchange, then expire a one-second liveness probe and a two-second typed request, and require at least three seconds of simulated time to have passed. Transport readiness and authenticated liveness are checked before each disruption; after each repair, the client reconnects, passes liveness, and completes another exchange. The symmetric and one-way partitions keep both processes and their listeners running; protocol ping deadlines close the unusable sessions on both ends so inbound slot limits do not prevent recovery. Snapshots bound open connections, pending requests, reconnect failures, and simulated sockets throughout, and the transport’s connection counts are checked after shutdown.

Relay Reconciliation And Cancellation

The relay cases drop the receiver-to-sender reply once an Arrow batch has entered the receiver’s queue, then reconnect to the same receiver process. The retained attempt reconciles to its admitted outcome and semantic ACK, and the application sees one enqueue. Cancellation competes with admission through the attempt’s one fence: before the grant, while the reply is lost, and while reconnecting, cancellation wins; after admission, it returns the admitted outcome. The checks hold within the receiver’s retention contract in Ordering, Retry, And Reconciliation. They say nothing about a receiver restart, which the next group covers.

Receiver Restart

The restart cases lose the reply at one of two milestones, after the Arrow body entered the receiver’s queue and after runtime admission, or hold an admitted attempt’s reply. At the milestone the receiver host publishes, the control closure crashes the receiver, repairs the one-way partition or releases the held reply, and bounces the host. The bounced host binds a new transport under the same node name with a distinct process epoch from its seeded entropy. The sender’s retained delivery resolves as indeterminate against that epoch and never enters the restarted receiver’s queue; a released pre-crash reply can confirm only that the old process received the body. A new delivery identity then crosses the rebound listener and is admitted. The fixture checks the Arrow batch and both process identities at the protocol boundary, and that the receiver crashed and restarted exactly once.

Backup Section Transfer

The backup cases send the production typed capture, inventory, and bulk fetch messages. A request that claims another process epoch is rejected before the capture or stream handler runs. The partition case bounds a stalled fetch by its request deadline, leaves the staged section available when the link is partitioned, repairs the link, and reads the verified section. The restart case ends an owner process after its first stream chunk; the incomplete stream fails, and the rebound owner has no stage from that process. These are transport and in-memory staging boundary checks; the Cucumber cluster scenarios verify the server’s archive and restore behavior.

Stalled-Peer Isolation

A hub and two peers run on three hosts. The stalled peer accepts shared management operations and never answers, leaves resource streams unread, and sends a relay batch the hub holds without admitting. The healthy peer keeps exchanging shared management work, liveness, typed Arrow commands, whole resource streams, and admitted relay batches with the hub throughout. Every observation of a host checks its pool connections per peer, leased streams per connection, admissions per direction and subquota, worker queues, and memory budgets against their configured bounds.

Key transitions assert exact values. The stalled peer holds all 32 shared management streams of its connection while liveness, cancellation, and discovery still use the reserved streams, and one more shared operation waits until its own deadline. A burst fills the one bulk worker and its eight-job queue, the two submissions beyond them are refused without a charge, and control work is admitted in the same pass. Unread streams stop at one HTTP/2 window each and release their admission and memory at the five-second progress deadline. The held relay reserves its exact body, largest decoded batch, and scratch, plus its retained grant metadata, until cancellation and release of the borrowed intake.

The fixture then holds the stalled link. The hub’s own liveness deadline ends its probe to the stalled peer while the healthy peer keeps exchanging every traffic class. Removing the stalled peer from membership ends the hub’s operations to it and closes the hub’s pools to it at once, without that peer’s cooperation. Tearing the stalled host down while the link is still held closes the connections the hub opened to it at once, together with the handlers still waiting to answer, and releasing the link closes the hub’s inbound connections from the torn-down peer. The Traffic And Resource Isolation contract is what these values check.

Simulated Restart And Real Process Crash

Sim::crash cancels the crashed host’s Tokio tasks and drops its runtime. Destructors run, and the host’s connections close in the order its tasks are torn down, which Tokio’s task numbering sets. Sim::bounce calls the host’s software again, which rebinds the listener and allocates a new process epoch. The simulated hosts have no filesystem and no background thread of their own.

A simulated restart therefore establishes the transport’s volatile contract: a new process epoch fences in-memory delivery state, an unresolved delivery becomes indeterminate, and the transport never replays it into the new process. It establishes nothing about fsync, power loss, SIGKILL, partial writes, consensus or WASM checkpoint durability, or full-node recovery.

Those claims keep their real-process evidence. tests/features/cluster/process_crash_recovery.feature starts a server process and kills it with SIGKILL before checking recovery, and the Shutdown And Recovery, Consensus Storage And Replication, and WASM State And Recovery chapters own the durable contracts it checks. A Turmoil host bounce is never cited as durability evidence.

Adding A Bounded Scenario

A new case follows the same contract as the existing ones.

1. Name the claim. Write the interconnect contract the case checks and the fault that challenges it. A publicly observable outcome also needs its Cucumber scenario; the simulation adds fault timing, not public coverage.
2. Place it with its owner. Add it to the fixture module whose contract it checks, or to a new module declared from transport.rs with its own ownership header.
3. Declare the scenario. Give it a name unique within its test, a fault plan that says which controls it applies and when, and committed seeds that no other case uses.
4. Bound it. Start from the fixture’s configuration and set the smallest simulated duration, step limit, and real-time bound that complete the plan with room to spare. Keep the network parameters lossless, so the written plan is the only fault.
5. Build each host inside its host. Bind a transport per host with SimulatedEntropy named after the host and a SimulatedUtc clock. Share only readiness and completion signals between hosts, run each host body under HostSupervisor::run, and have the observer client wait for every host to finish.
6. Stay inside the simulated boundary. Use nervix_primitives::time for every wait and instant and nervix_primitives::net for every socket, and call nervix_primitives::task::consume_budget().await at the top of every loop. Never sleep a real thread, read the wall clock, open an operating-system socket, touch the filesystem, or submit storage or unbounded CPU work.
7. Inject faults at protocol points. Apply link controls from inside a host when the host reaches a decision, or apply host controls from the control closure when a host publishes a milestone.
8. Record decisions. Record an event at every decision the case asserts, with identities and outcomes only, never payloads, key material, process identifiers, or map-ordered listings.
9. Prove release. Shut every transport down before its host exits, assert its connection counts afterwards, and check every pool, queue, and reservation the plan touches against its bound.
10. Register it. Add the test to its invocation in tests/turmoil-inventory.toml, which just test-turmoil holds the suite to: a registered test that does not run, or runs ignored, fails the suite, and so does a simulation test that runs without being registered.
11. Run it. Iterate with just test-turmoil-simulation <test> --exact, run just test-turmoil-sweep to look for divergence across seeds, then just test-turmoil and just validate.
12. Keep this chapter current in the same change: the scenario matrix, its bounds, and any new limit.

Findings And Retained Regressions

The simulation changed product code only to remove nondeterminism sources that reached a decision. Its other findings are recorded here and in their owning chapters as current behavior. The qualification run found no further product defect and no behavior-affecting trace difference.

Guarantees And Limits

The simulation guarantees, for the audited path:

• The production transport runs over Turmoil TCP and DNS only, and no scenario reaches the host network.
• Transport deadlines follow the simulated clock, certificate validity follows the supplied UTC clock, and neither falls back to the host clock.
• One scenario, seed, and set of inputs makes the same recorded decisions, at the same simulated times, in every fresh process of the same build on the same host configuration.
• Every failure leaves a record that replays the same inputs in a fresh process and reports whether the failure reproduced.
• Every run, attempt, suite, and sweep ends inside a real-time bound, including one whose host blocks the scheduler thread.

The audit behind those guarantees covered the boundary’s selection of sockets, timers, and CPU jobs and the simulated resolver, the transport identities and certificate clocks, request and relay deadlines, the map walks with side effects, the bounded CPU executor, and the runner. Its limits:

• The runner’s real thread and wall clock decide only when a test fails, never how a request proceeds.
• Execution storage jobs use a real blocking thread and are outside every case.
• Default worker counts follow the host’s CPU count, so a replay is established for the tested host configuration, not independently of host capacity.
• Rustls and AWS-LC keep their randomness, so traces compare authenticated decisions, never ciphertext.
• The TCP model neither retransmits nor models kernel timing, and Turmoil 0.7.2 has no disconnect control.
• A simulated host crash is not evidence of durability, as described above.

Prerequisites For Whole-Cluster Simulation

Running whole Nervix nodes under Turmoil needs more than substituting TCP. Each of these is an open prerequisite, not part of this simulation’s claim:

• Raft and gossip entropy. OpenRaft election timing and Chitchat gossip draw their own randomness. Both would need a seeded owner without changing production entropy.
• Storage and crash semantics. Fjall runs background threads and writes real files. A simulated store would need defined filesystem, fsync, and crash behavior, kept apart from the real-process durability evidence.
• Host attribution. Work a node spawns, including blocking tasks, timers, DNS queries, and sockets, must run on the host that owns it; code that escapes to another runtime or thread escapes the simulation.
• Domain time. Domain-clock authority, generations, and deadlines would need controlled time that preserves the Domain Clock contracts.
• Connectors. Every connector’s I/O, acknowledgement, and commit boundary would need to be classified as a simulated fixture or a real-process test.
• Host-independent capacities. Execution worker counts would need fixed values, so admission traces stop depending on the host’s CPU count.

Combining Shuttle’s scheduler with Turmoil’s network is a separate question. Shuttle owns the interleaving of in-process tasks and Turmoil owns a runtime per host; the two scheduler modes are mutually exclusive by build. Driving Shuttle over a simulated network would need a network substrate integrated with Shuttle’s own scheduler, such as turmoil-net, and is not part of either suite.

Qualification Evidence

The simulation was qualified on 25 September 2026 at revision 7032bca2, which also contains the final Shuttle qualification. The results below are from that run, except the CI row, which is from the change that recorded it.

No product defect and no behavior-affecting trace difference was found in the qualification. Seed 1036 remains a committed regression, and no failing seed was dropped.

The commands in this chapter were run again on d2cec5f2, the revision it documents, on a 32-thread workstation. just test-turmoil passed in 1m01s with a warm build cache, just test-turmoil-replay-check passed in 2 seconds, and just test-turmoil-sweep passed all 16 cases over its 64 seeds in 7m05s, leaving no failure record, with a largest process of 130,764 KiB. The example investigation above was recorded and replayed on the same revision.

Routing the simulation’s sockets, timers and CPU jobs through the primitive boundary was requalified on 29 September 2026 on a 24-thread workstation that other builds shared. just test-turmoil passed, including the host isolation case and the boundary’s own simulated-host checks, and just test-turmoil-replay-check reproduced its injected record. Every case of just test-turmoil-sweep passed its default 64 seeds twice in fresh processes with identical traces, leaving no failure record. Under that shared load the sweep ran past its 25-minute budget during its last test, the receiver restart group, which then passed its 64 seeds alone in 2m30s.