Integration Test Lifecycle

Paced Interconnect Fixture

The rate-limited follower catch-up scenario starts a TCP byte pacer before restarting a stopped node. Its listener takes the existing advertised endpoint and forwards opaque bytes to a private interconnect listener on the restarted node. The advertised endpoint stays fixed so retained peer routes also reach the pacer. Its port remains reserved for the test run, as for other retired test endpoints. The host, TLS identity, resolver policy and product deadlines stay fixed. Each accepted connection uses a 1024-byte inbound buffer, delays each chunk by its configured byte rate before forwarding, and forwards responses promptly. A scenario-owned task set retains and reaps connections. Cancellation or fixture drop closes the listener and aborts all its connections; the fixture cannot survive its node specification.

The scenario starts the follower once without the harness’s catch-up retries, records append stream opening baselines before that start, and waits for the applied NSPL domain backlog within its explicit assertion bound. The opening count is a diagnostic observation because normal initial catch-up probes can also open streams. An atomic maximum of bytes forwarded on one connection proves that the paced path carried more than one production progress interval. This observation does not inspect TLS or frame bytes and establishes no ordering the product relies on.

The Cucumber scenario suite runs Nervix the way an operator does. It starts clusters, drives them through the session API, brokers, and HTTP endpoints, and asserts on what they do. Many scenarios run at once in one test process, each with nodes of its own, so the harness around them is a concurrent system with failure modes of its own: a node that never becomes ready, a status request that never gets a reply, a node that never stops, a step that never returns.

This chapter defines the harness’s lifecycle contract: how a node, a scenario, and the whole run start, how the harness observes them, how it stops them, and which deadline bounds each of those steps. Three rules hold throughout.

• Every lifecycle operation has one owner and one absolute deadline, fixed when its phase starts. A retry, a poll, or a cleanup inside the phase receives only what is left of it.
• Every ending has a typed outcome. A node task is running or ended in one of four ways, a readiness probe reports what it observed, and a scenario publishes the phase it is in. No harness path reports an unknown reason.
• The run ends itself before the CI job limit and leaves the job time to upload the diagnostics it produced.

The harness is test code outside the layer order: it may name any layer, and no product code names it. It observes product behavior and never redefines it. Product deadlines stay with the chapters that own them: node shutdown with Shutdown And Recovery, node-to-node communication with Cluster Interconnect, client requests, redirects, reconnects, and exact recovery with Client Session Protocol, and command identity and completion with Command Completion.

What Runs Where

Before starting ordinary CI producers, Check verifies its exact tested merge and every changed path. A verified documentation-only selection excludes scenarios with the other ordinary suites; validation and title checks remain enabled. This exclusion records a source decision rather than scenario execution, and cancellation of a required run never supplies successful evidence. Explicit mode labels retain their own selections. Documentation-only Selection owns the allowed paths, immutable event snapshot and retained CI selection record.

The suite is the scenarios test target, tests/scenarios.rs, running the features under tests/features. Its lifecycle owners live in tests/common.

A scenario binary built for the deloxide mode, which just test-deloxide, just test-deloxide-order, just test-deloxide-stress or just test-deloxide-stress-restore builds, starts its deadlock diagnostics in main, right after it configures the lifecycle of its test dependencies and before it builds its runtime, so its one detector is installed before any tracked lock or runtime worker exists and every in-process node shares it; the helper process that holds a dependency container open starts its own. It records its evidence in NERVIX_DEADLOCK_EVIDENCE when that is set. A binary whose diagnostics cannot start exits with status 4 before any scenario runs, and the first active deadlock any of its tracked locks reports ends the whole run with status 3, its description on standard error. A server child started with the fixture’s deadlock evidence option records its own evidence under its root, and is a diagnostic node only when the binary beside the scenario binary was built for the mode. The ordinary suite leaves the @deadlock_diagnostics scenarios out by default: their steps assert a detector an ordinary build does not have.

The diagnostic lane also selects the ordinary @restore_installation, @client_ingestor_alter_drain, @memory_pressure_pause, @client_io_03_consumer_restore, @client_io_03_generation, @udf_column_builder, @vhost_tls_rebinding and @inferencer_branch_batches scenarios and the local @deadlock_reports inspection/export/triage workflow. They exercise the blocking applied-state guard through interrupted checkpoint staging, complete publication and runtime handle clearing, including a delayed coordinator after leadership transfer and a successor’s START, the memory-pressure pause of starting and running ingestors, Rust client consumers restored after a cluster restart and closed by a domain stop, a Roto function’s column builder, the HTTPS listener’s TLS configuration as a VHOST’s bundle is rebound under HTTPS ingestion, and a batched inferencer’s ONNX session across interleaved branches. The report tool is an ordinary local executable, even when the scenario process selects order analysis. Potential findings remain recorded without ending the process; any unreviewed workload finding prevents diagnostic qualification. Real server children and the scenario process record the same compile-time/runtime selection. The ordinary CLI built by the test dependencies drives these diagnostic nodes through its public protocol; the lane supplies its path explicitly because the diagnostic binary has a separate build directory. The feature input is a quoted file glob, and the lane requires every tagged scenario run to execute and pass without retries. The recorded evidence covers tracked blocking locks reached by those workloads; async waits, capture atomics, dependency locks and cross-node waits retain their other concurrency checks.

The @restarted_voter_observation scenario runs in both selections, reaching whole-cluster teardown, reopening each node’s store, and automatic scheduling after the first relayed voter heartbeat. Its watch-channel barrier and concurrent fault map are untracked; the diagnostic evidence covers blocking locks reached by restart and scheduling. The native metadata restore scenario also runs in both selections after its Kafka replication stream and acknowledgement boundaries are repaired, retaining complete primary and replica metadata comparisons and distributed placement checks.

The measured restore and backup steps sample each node’s public bulk and restore-metadata executor gauges every 20 milliseconds while the ordinary CLI runs. They also record the harness process’s jemalloc allocated and resident bytes. The executor measurements separate the fixed checkpoint I/O and publication grant from retained archive descriptions, typed native values and planning allocations. Process measurements include every in-process node, dependency fixture and harness allocation; isolated one-node and three-node runs provide attributable comparisons. Sampled peaks are observations, not continuous maxima, and allocator resident bytes exclude mappings outside jemalloc. The scenarios also compare bulk refusal counters before and after each restore, and a measured backup’s check compares them and every node’s sampled bulk peak with that node’s published bulk budget. The large-native-metadata oracle stops the cluster after public restore and exact replay, then opens each node’s database through the production snapshot reader. It compares every lifecycle key, timestamp, incarnation, revision and Kafka topic/partition/offset on primary and replica copies. Those complete-value inspection allocations occur outside the restore measurement. The runtime’s testing-only inspection API returns decoded current metadata; the harness owns all archive comparisons and assertions. The scenario takes the CLI’s execution reference and replays it through the public session before and after restart, comparing the complete typed restore report. It saves the restored work’s public owners and replicas before restarting, requires distributed placement in the three-node case, and compares every saved placement after START and after receiving isolated output for every tenant. A failover that drops replicas or recreates guest state cannot make this oracle pass by moving all execution onto one node.

just test-native-metadata-restore-repeat runs that scenario’s three-node example ten consecutive times at concurrency four with zero retries, preserving its normal harness deadlines. It stops at the first failure or a summary that does not contain exactly one passing scenario. Every attempt keeps its source revision and patch, each process output and the available node log under target/native-metadata-restore; its completion record is written only after all ten pass.

The complete-generation fixture allows 120 seconds for every post-restore input to reach its raw relay. Its forty one-mebibyte guest saves exercise sequential source acknowledgements and durable replica checkpoints beside the other diagnostic workloads. The full active diagnostic selection reached 77 of 80 inputs at the former 30-second bound, while the exact same binary reached all inputs and preserved every branch in the focused four-case replay. This condition ends as soon as the exact expected count is observed; it retains the complete output, state and branch assertions.

The large materialized restore fixture prepares its rows through an acknowledged client producer. One round has a 120-second budget and one in-flight batch, including explicit replays of reported processing failures. Each replay replaces the same materialized keys with the same complete values, so partial effects retain the fixture’s value oracle. Every attempt and terminal result is logged; an unknown outcome, final admission refusal or expired round fails preparation. The producer closes after every tenant’s batch has completed, before the scenario checks all generator rows and takes the backup cut. Source admission, input delivery and the final archive and restore assertions remain separate observable steps.

The lane then selects @paced_simulation_reopen from the public paced-driver feature. It builds the Rust driver in the lane’s selection and supplies its path to the same scenario fixture, so the driver installs its own detector before entering its runtime. Both drivers exercise endpoint replacement, pending submission ownership, explicit replay and replacement refusal against diagnostic nodes on one and three nodes, without retries. Python uses the ordinary shared binding; its locks and condition variables are outside this detector.

The buffered-route replacement scenario accounts for every reading in the first run’s outcomes and explicitly replays unresolved ledger entries. Ending a producer can refuse a pending or already planned submission even when its replacement opens in the same START generation. The scenario requires every planned reading to have exactly one effect after replay; each invocation’s summary counts only that invocation, including an empty replay when no reading was refused.

The lane’s diagnostic owner tests each start their detector in a fresh process before any store, publication or executor lock: the state store’s staging, snapshot views, queued writers, interrupted publication and bounded cleanup, and the materialized publication lifetimes. Each must pass its libtest accounting and record exactly one qualifying process observation in both diagnostic selections. The tracked locks’ conformance checks of nervix-primitives run the same way, each in a fresh process in both selections. They record no evidence file, because the primitive crate sits below the crate that owns evidence: a finding reaches a sink that aborts the check, and the lane fails on that ending.

Test dependencies start through one suite-owned environment. If Docker creates a named container but cannot bind its randomly selected host port, that owner removes the failed container and tries up to three more times with a new port selection. It also removes the failed container when those attempts are exhausted, so a scenario retry does not collide with its name. Other startup failures reach the scenario directly.

The OpenTelemetry Collector dependency exposes its stdout and stderr to scenario assertions through one follower that the suite-owned environment starts with the container. The follower streams both outputs from the container’s start into an in-memory transcript of at most 1 GiB per output, and every scenario observes that transcript without the environment’s lock or another read of the container’s logs. A Docker logs read answers with the output as it stood when the read began, and the Collector’s output grows for the whole run, so no assertion re-reads it. The environment stops the follower before it tears the container down. A batching scenario reads the Collector’s debug exporter output to check the number and order of records in each received export request, using unique test markers to distinguish simultaneous scenarios sharing that container.

The debug exporter prints each export as two log entries: a summary that counts the export’s records, and then the export’s body. The Collector prints each entry whole, but an entry of a concurrent export can print between a summary and its body, so a summary does not identify the body it counts. The batching scenario therefore reads each export from its body alone. The body’s first line names its signal, and each log record, span or data point in it starts a line of its own, which is what the scenario counts. The logger prints the entry’s fields after the body’s last line, so a body is read only once its fields have arrived. The suite’s Collector configuration turns off the sampling of the Collector’s own logger. By default that logger prints only the first ten entries that share a message in each ten-second interval, and every hundredth after that, so it drops the summaries of exports that arrive close together.

The raw session fixture records domain-clock replies and frames in the order it reads them. A wait for a detach reply can queue ticks that arrived before the reply; the post-detach assertion checks the recorded wire order, clears those already queued frames, and waits for any new frame. An unread pre-reply tick therefore does not masquerade as delivery after detach.

The generator cadence assertion groups equal logical timestamps across concrete branches and checks increasing timestamps within each branch. It allows records from separate branch tasks to arrive in either order, as their delivery has no shared ordering contract.

tests-deps builds the CLI, the NSPL formatter, the paced simulation’s Rust driver and the shared C binding its Python driver loads in the normal target directory. The full and focused client coverage recipes build a standalone instrumented CLI beside their instrumented server binary and place the normal NSPL formatter there. The scenario runner selects the covered CLI through NERVIX_TEST_CLI_PATH, so its one-shot completion and command paths contribute to the same LCOV report as the CLI’s binary unit tests and the server’s public scenarios. The general coverage-scenarios and coverage-scenarios-append recipes provision the same instrumented CLI and formatter, so a selection containing CLI scenarios needs no separate binary setup. The focused CLI process coverage recipe exercises transaction inspection and the clock-following process scenarios.

test-scenarios-coverage, coverage-scenarios and coverage-scenarios-append also build the paced simulation’s Rust driver and the shared binding with coverage flags, and the scenario runner selects them through NERVIX_PACED_SIMULATION_PATH and NERVIX_CLIENT_LIBRARY. A driver process inherits the profile location of the scenario binary that starts it, so the Rust driver and the binding loaded by the Python driver write their profiles into the same report when they exit.

The CI scenarios job uses the shared native CI linker for the server, CLI and scenario harness, including their instrumented builds. Coverage flags do not replace Wild linker selection.

The suite has one pool of run slots. Its size is the number of CPUs times the concurrency factor, set by NERVIX_TEST_CONCURRENCY_FACTOR or --concurrency-factor and 1 by default; --concurrency sets an absolute slot count. The CI scenarios job uses factor 3, so its 16-CPU runner has 48 slots. The instrumented inventory consumed 78,433 slot-seconds with 115 attempts still active, almost the entire 78,720 slot-seconds that 32 slots could supply in the unchanged 41-minute suite budget. The additional slots use the observed CPU and memory headroom; their complete-suite qualification retains runner load and the actual margin rather than assuming that work stays constant as concurrency changes. Cucumber may take up all parsed scenarios without charging a slot. Admission grants feature capacity and a run slot in one decision, after both are available. A scenario holds both through the end of teardown. The active-scenario diagnostic names the feature limit or run slot a waiting scenario needs. No scenario runs alone or prevents unrelated scenarios from starting.

The coordinated WASM state-reset feature runs one scenario at a time. The web console REPL, execution graph, transaction inspector, and domain clock features share a limit of two. Both limits affect only their named features. The parser takes these limited features up before the bulk of the suite, and available run slots favor a queued limited successor over queued ordinary work. The web console group grants its two slots to the feature with the fewest prior grants, so one feature cannot keep the other three waiting behind its whole queue. Their chains can therefore make progress throughout the suite rather than after it. Cucumber retries a failed scenario twice; --retry 0 turns retries off for a focused run. The before hook reads the attempt kind from its active-scenario registration. Each feature queue keeps retries ahead of first attempts, and admission selects an eligible retry before any first attempt, including a limited feature’s first attempt. A retry still needs its feature capacity and a run slot together: one waiting for a full feature group leaves free slots available to other work. Ordinary and WASM-reset queues retain FIFO order within an attempt kind; the web-console group retains its feature fairness. The finite retry count bounds this priority; first attempts cannot be kept waiting by an unlimited retry stream. Cancelled waiters return both capacities before dispatch continues.

The CI jobs divide the work at the scenario boundary:

The checks, tests, and scenarios jobs use Go 1.26 and check out the source before installing their pinned Go formatters. This makes the installer and the later cache publication and report scripts available when a tool setup step fails. The installer retains every attempt in a go-tool-bootstrap-<job> artifact and retries only a checksum database transport error within its eight-minute budget. The publication and report steps check for their scripts before calling them, so a failed checkout does not add a second failure that obscures the original one. The jobs’ existing deadlines and test budgets are unchanged.

The tests and scenarios jobs also sample runner CPU utilization and steal time every five seconds. Every kache-backed job uses kache 1.2.1, records doctor output without making it a test failure, publishes a cache report, and diagnoses its five most expensive misses with why-miss. The shared S3 cache keeps executable and test-binary outputs, with stores sized for the full builds: every kache-backed job and Docker image build uses a 1 TiB store ceiling. This is an upper bound, not a disk reservation. The runner’s actual disk capacity remains the practical limit.

The shared remote is Cloudflare R2 through its S3 API. Kache reads the base artifacts prefix first. Pull-request jobs also read and publish entries, manifests, and shards under the shared artifacts-pr prefix, configured in the daemon’s TOML file. This lets successive PR runs reuse their builds. Kache lets only a protected-branch push publish to artifacts, and no workflow runs on one; scheduled and manually dispatched native jobs retain kache’s read-only policy. Docker image builds run kache inside BuildKit, which does not see the GitHub Actions environment, so their builder publishes to artifacts. Fork PRs still need credentials to access R2. Kache’s doctor displays the generic read-only CI policy even when the PR prefix is enabled; the job’s daemon reports its actual write mode, which the start step prints and publication records.

CI configures the cache bucket through the required repository variables KACHE_BUCKET (bucket name), KACHE_BUCKET_REGION (S3 region), and KACHE_BUCKET_ENDPOINT (S3 service URL), and reads its credentials from the repository secrets KACHE_BUCKET_ACCESS_KEY_ID and KACHE_BUCKET_SECRET_ACCESS_KEY. Each kache-backed job passes them, as kache’s KACHE_S3_* environment variables, to python3 scripts/kache_remote.py start right after the setup action. Docker image builds pass the same settings to their builder, whose build command runs scripts/kache_remote.py configure before it starts the daemon. Either way the service URL is written into the daemon’s TOML configuration before startup.

For Cloudflare R2, set KACHE_BUCKET_REGION to auto and KACHE_BUCKET_ENDPOINT to https://<account-id>.r2.cloudflarestorage.com, without the bucket name in the URL. The bucket name belongs in KACHE_BUCKET. scripts/kache_remote.py rejects URLs with a bucket path; supplying the full bucket URL can make the remote cache appear empty.

Cache Publication

scripts/kache_remote.py owns the remote of every kache-backed job: the check.yaml, Bolero, benchmark and build-book jobs, and the builder stage of every Docker image build.

kunobi-ninja/kache-action installs kache, sets RUSTC_WRAPPER, the store and a job-scoped runtime directory, and stops the job’s daemon in its post step. It receives no S3 inputs, so its post step neither saves a manifest nor pushes. The Start kache remote step that follows it:

1. writes the daemon’s configuration, $RUNNER_TEMP/kache-remote.toml, with the bucket, region, service URL, base prefix and pull-request prefix. The credentials stay in the environment.
2. exports KACHE_CONFIG, the credentials, and the job’s KACHE_MANIFEST_KEY and KACHE_NAMESPACE to the job’s later steps. A matrix job’s key includes its matrix value, so every build variant keeps its own manifest and shards.
3. starts the daemon, and fails the job unless kache --json daemon status reports it running with that file and kache --json stats reports that the file names s3://<bucket>/artifacts. A daemon that has not answered yet gets three polls, two seconds apart. The step prints whether the daemon publishes uploads or, under kache’s read-only policy, skips them.

During the build the daemon publishes each entry the build stores: kache records a durable upload intent, uploads the entry in the background, and retries an upload that failed for a retryable reason every five seconds.

Publish kache cache runs scripts/kache_remote.py publish whatever the job’s result: just before the kache report where the job publishes one, and as the job’s last step otherwise. For a job whose work succeeded, it saves the build manifest with kache save-manifest, then reads the daemon’s upload counters from kache telemetry write every two seconds until no upload is queued or in flight. It waits at most 120 seconds, its kache commands included. The step’s 4-minute timeout-minutes only backstops that bound. Every kache answer it reads is a versioned kache document, and an answer outside its schema is reported rather than read. The outcome is one of:

Publication never changes the job’s verdict. Every outcome exits 0 except cancelled, whose step ends with the signal’s status, as the runner asked. incomplete, unavailable and cancelled also add a workflow warning. A job that failed before its checkout has no script to run, so its step adds only a notice. The record, publication.json and publication.md, lands in $RUNNER_TEMP/kache-report, which the kache report artifact uploads where the job has one, and in the job summary. An image build runs publish --job-status success as the last command of its build stage, which reaches it only when every artifact has been built.

No job runs kache sync --push. Kache’s push-only sync lists every key under both prefixes before it decides what to push. On 2026-10-14 and 2026-10-15 that listing grew from 650,859 to 690,780 keys, took six to eight and a half minutes in every kache-backed job, and found nothing the daemon had not already published.

Product Deadlines And Harness Deadlines

Two kinds of deadline meet in every scenario.

A product deadline is configuration a node enforces on itself: its shutdown and drain timeouts, its command retry validity and transaction idle timeout, its node-unavailability and election timeouts. Test clusters start with test values for some of them. A scenario about one of them configures it, drives the node to it, and asserts the outcome the owning chapter defines. No harness deadline shortens a product deadline while the scenario can still observe it.

A harness deadline bounds how long the harness waits: for a status reply, for a node to become ready, for a cluster to stop, for the whole run. Its expiry changes nothing inside a node. It ends a wait with a typed failure, takes apart a receiver or a child process the harness owns, or ends the run. It never ends a node. A node’s task is only the future the node was started as: the node spawns the tasks that hold its databases, its listeners and its peers, and they would outlive an abort of that future with nothing left to join them. A node that has not stopped when a deadline passes is therefore left to end by its own shutdown, keeps what it holds until it has, and is recorded as still stopping, not as something the harness ended.

The cluster health scenario can fail application-health responses for one ordered probing-node and responding-node pair while other probes remain healthy. It can also hold one node immediately before that node reports a runtime revision prepared, wait until the hold is reached, and release it after asserting command completion. These are per-scenario injected conditions; their waits have harness bounds and do not shorten the product’s completion deadline.

An injected hold is the opposite case: a product deadline races it. The scenario holds a controlled operation at a gate for a workload duration, counted from the moment it observes the gate, and the node keeps its own deadline running while the operation waits. The backup wait scenarios hold one domain’s quiesced cut for 12 seconds and the next one’s for 14 seconds, each inside its own 20-second cut budget. The scenario’s margin is the part of the budget the hold leaves, and a runner task that another scenario’s synchronous step stalls for longer than that margin would lengthen a hold timed on the runner task until the product deadline expires. tests/common/injected_hold.rs therefore waits for each gate, holds it for its duration and releases it on a task of its own, and the step that injects the holds only awaits that task. A stalled runner delays when the step learns of a release, never the release itself. The step still watches the controlled operation, and a command that ends before its gates were held fails the step with the command’s own outcome.

The boundary between them is kept in four places.

• Ordinary commands. The NSPL commands a scenario runs go through the production Rust client, with its execution identity, redirects, and reconnects. This includes the transaction qualification graph’s setup commands, which retain each execution reference if leadership changes while setup is applying. When a command document has server statements after a subscription, the harness sends those statements through the client and opens a raw session for the subscription on the node requested by the step. A standalone subscription stays on that requested node, including after failover. The large materialized backup workload submits each graph statement through the production client so a leadership change during its many relay definitions recovers that statement by reference; its Row subscription opens on a raw session after the graph is ready. A leader setup step whose other commands need one session, because they create a subscription or open a transaction, runs them over a raw wire session; when leadership moves before one of them reached the leader, or after the leader admitted it and before its outcome was known, the step sends that command again with the same execution reference to the current leader, as a client must, and goes on there, within 60 seconds per command and only while the session holds no subscription or transaction yet. The paced simulation’s published graph is loaded through one native client so its BEGIN/COMMIT state and command references survive an interrupted finalization. Raw wire probes retain the protocol’s received dispositions. A paced-driver file fixture writes only the scenario’s ledger or effect file before the driver starts. The published driver reads that file, and the scenario checks the process exit and the occurrence count of its terminal cause. No harness deadline shortens native commands. Only the status path described below is harness-owned, and it is a separate test-only boundary: it opens its own session and never redirects, reconnects, or retries by itself, so it adds no second client policy.
• Cleanup. Scenario cleanup runs after the scenario’s assertions, so it spends the harness cleanup budget rather than waiting for a product shutdown deadline. A scenario whose subject is shutdown, drain, or deadline expiry stops its nodes itself, inside its body, under the product deadlines it configured.
• How nodes stop. The harness stops an in-process node through its shutdown coordinator, with the same stop request a first SIGTERM makes to a server process, and the node runs its shutdown phases under its configured deadline. Registering termination signals, the forced-exit supervisor, and exit statuses belong to the process boundary, which only the server-process fixture crosses. See Shutdown And Recovery. A real-process cluster can deliver SIGKILL to all three voters before waiting for any child to exit. On restart it launches all existing stores before awaiting readiness, so no one voter is required to answer without the persisted quorum. Each node must then report the same leader and all three voters through its public status endpoint. It can also fault one voter while the other two keep the quorum: SIGKILL, after which a step waits for that child to exit and checks the signal, or SIGSTOP, which freezes the process with its connections open, so its peers hear nothing from it and see no reset until SIGCONT lets it run again. A killed voter restarts from its own database and ports, and the step then waits for the same leader and three voters as a whole restart does. While a voter is killed or frozen, the fixture asks only the running ones which node leads. The restarted-voter observation fixture stops every in-process node before arming its fault. It blocks direct gossip between the first leader and one voter, then holds that voter’s first relayed heartbeat while allowing the other voters’ heartbeats to advance. The leader’s reconciliation starts after that sample arrives, so slow harness startup cannot consume the product’s observation grace. A one-shot barrier captures a scheduling pass inside that grace with the voter marked unavailable; its elapsed-time decision is captured before the barrier waits. The scenario checks the failure-detector verdict and final ownership through public cluster status. Releasing the barrier waits for that pass to finish before restoring heartbeats and application health. While the heartbeat is held, peer startup waits for listeners; Raft catch-up is checked after release, so the fixture never waits for progress its own fault prevents. This is an in-process scheduling regression; immutable-image external Chaos runs retain the real-process crash and restart evidence. The durable follower catch-up fixture creates client writes while the restarted follower applies its backlog and samples commands memory throughout that interval. Once the backlog is applied, stopping the writer finishes its admitted command before counting completed writes: the response can arrive after all voters have already applied that command. The fixture still checks the storage-derived catch-up bound, append-stream limit and commands-memory budget. Its initial burst of domain creation also uses one production client, retaining each command’s execution identity across a leadership change during finalization. A raw session’s recoverable disposition cannot end the fixture before follower catch-up starts. A listener bind failure retries only that listener family: the gRPC pair, HTTP, HTTPS, observability, web console, or interconnect. A readiness timeout may still move all listeners within the same startup budget. A changed interconnect address remains reserved while a peer may dial it. Metrics sampling follows the actual observability endpoint before the next launch. The durable catch-up scenario deliberately occupies the stopped follower’s observability port until cleanup of its first startup attempt, so a metrics bind collision cannot silently turn the measured catch-up into an interconnect address and membership change. The planned Kafka handoff scenario installs its subscription and emitter stall, publishes and flushes both branch records, and then starts intake. Both records are therefore available to its first parallel batch. The source host waits for that batch’s attached ACKs before polling the next batch; separate publications after intake starts do not establish one batch. just coverage-harness-liveness <filter> measures the ordinary startup and other harness owner regressions with counters of their own, as the diagnostic lane section describes. A patch that also changes product code passes the retained lcov.info to just coverage-patch as another --report, beside the workspace report.
• Test defaults. An in-process node’s shutdown timeout defaults to four minutes rather than the product’s 50s, which leaves the bounded shutdown phases scenarios configure by default room to finish, so only a scenario about the deadline reaches it. A server process runs with the product defaults unless its scenario passes an option such as --shutdown-timeout.

Deadline Ownership

Every lifecycle operation of the harness is bounded by exactly one of these deadlines. Each budget is a constant in the module that owns it, under tests/common unless named otherwise; where a second module runs the operation, it is named after the owner.

Everything else a scenario’s body does, including its NSPL commands, broker and metric waits, and assertions, keeps whatever bound its step declares. Some of those steps check their bound only between requests, so a request that never returns outlasts it; the suite budget bounds them in every case. It is also the only bound on the time a scenario spends queued for its concurrency permits and on closing the browser during cleanup, neither of which has a budget of its own. Browser cleanup explicitly awaits the Playwright driver’s shutdown before dropping its handle. The driver’s own five-second exit grace is asynchronous, so cleanup keeps the shared scenario runner and its watchdog available rather than waiting in Playwright’s blocking destructor.

Absolute Phase Deadlines

A phase deadline is one instant on the monotonic clock: the moment the phase started plus its budget. Operations inside the phase receive that deadline rather than a fresh timeout, so the phase ends when its budget says it does, however many requests, retries, or polls it made.

• Nesting. An operation inside a phase that may take at most a budget of its own gets whichever comes first: that budget from now, or the phase’s deadline. A status request inside a status wait, a startup attempt inside a node startup, and a node startup inside a cluster construction are all nested this way.
• Bounding. Running an operation within a deadline ends it when the deadline passes. An operation that begins after the deadline has passed does not run at all, so a stream of immediately ready work cannot keep a phase alive.
• Polling. A poll repeats an attempt until one output is accepted or the deadline passes. Every attempt receives the same deadline, the pause between attempts never sleeps past it, and on expiry the poll returns the last output it rejected and the last failure it saw.
• Owned work. A wait for something its owner must keep, such as the join handle of a node task, borrows it for the length of the wait and never takes it. Whether the wait finishes, runs out of time, or is cancelled from outside, the owner still holds the handle, so no path leaves a node’s task running with nothing able to join it.

The deadline keeps its start and its budget rather than their sum, as the product’s shutdown deadline does, so a budget of any length cannot overflow the clock.

Deriving A Budget

A harness budget is derived from named policy inputs rather than written as an independent literal. Each input is a measured duration, the slowest healthy value observed while the whole suite ran at the CI concurrency factor, or a headroom factor that multiplies one. Every derived budget is checked against the budgets around it by assertions evaluated at compile time, so a change that breaks an ordering fails to build rather than producing a harness that outwaits itself.

The assertions keep these orderings, among others:

• A status request is materially shorter than the waits and startup attempts that repeat it, and one attempt outlasts three readiness requests that never reply.
• Status snapshots are shorter than the waits whose failures they explain.
• A startup attempt outlasts the slowest healthy startup, a cleanup slice is short beside the attempt it ends, and the node startup budget stays under its ceiling.
• The cluster cleanup budget is shorter than the watchdog of a node a scenario stops itself, and the test shutdown timeout lies between the default shutdown phases and that watchdog.
• The suite watchdog’s cleanup window fits inside the reserve, and so do the dependency stop and the runtime shutdown together.

The 22-minute observed suite ceiling plus 15 minutes of slack fits the 41-minute budget with four minutes to spare. The second split-job run passed in 20m26s with seven retries and 94.3% run-slot utilization. Increasing either measured input eventually fails the compile-time assertion. Measure both again whenever the suite, its concurrency, or the runner changes.

Status Requests And Status Waits

The harness reads cluster state with SHOW CLUSTER STATUS;. A status request opens a new connection to the node’s session endpoint, in plaintext or over TLS trusting the test certificate authority when the node serves its session API over HTTPS. It opens a session authenticated as the default test user, sends the command with a request identity, no domain, and a fresh execution reference, and reads the session’s frames until the reply naming that request identity arrives, keeping the session’s request stream open so the node never sees the session end while it answers.

The request has one deadline: 10 seconds, or what is left of the phase that sent it. Its four operations run in order, each within whatever that deadline has left.

A node that never completes an operation holds the request in it until the deadline: a listener that never accepts leaves a TLS handshake waiting, a node can accept the connection and never open the session, and a node can take the command and never answer it, or answer only with unrelated events.

A request ends in one typed failure: the deadline passing, naming the operation still pending and the budget; an endpoint or TLS authority that could not be configured; a command frame that could not be encoded; a connection, session, or response-stream error carrying the transport’s status; a session that ended before the result; a frame from the node that does not decode, or a reply to the command that is not a command result; or, when the caller needs the status text, an unsuccessful command result with its disposition, message, and diagnostics. Client diagnostic scenarios inspect the failed disposition and source span from that result, so the assertion uses the delivered diagnostic rather than parsing the displayed message.

A status wait polls one node every 200 milliseconds until its parsed status satisfies a condition, within 40 seconds from the start of the wait. The waits the harness performs are a current leader, a specific leader or a leader other than a given node, a Raft state, the expected voter set, an interconnect status to a peer, an applied index at least the leader’s, and a fragment of the status text. Two waits read every node at once: a leader the running nodes agree on, and a consistent leader, which requires every node to name the same leader, that leader to report itself Leader, and three consecutive polls to agree. Every request of a wait receives the wait’s deadline, so a wait whose last request never replies still ends at its original deadline rather than one request timeout later. A wait that expires fails its step with the elapsed time, the last status text it rejected, and the last typed failure it saw. The meaning of the connected interconnect status these waits read belongs to Cluster Interconnect.

The stopped-voter DROP NODE scenario waits for the public status report to show that the voter is no longer visible in gossip before removing it, then requires the exact remaining voter set. Application health can report an unavailable voter while gossip still sees it live, so an unavailable verdict does not establish membership-removal readiness. Scenarios that exercise application-health failure classification assert that source under their own controlled health fault.

The correlator match-policy scenario observes each left input’s received counter on the correlator owner before posting the next independent input. An HTTP acceptance alone does not order execution across ingestors. Its two-minute correlation window accommodates those bounded readiness waits without expiring the records whose match policy it asserts. The Kafka handoff scenario that pauses after entity-gate engagement uses a test-configured two-minute gate lease so its broker-delivery assertions and pause release fit inside the handoff budget under concurrent suite load.

Collection and route-flush timing assertions measure observed row arrival from the HTTP publication instant. Concurrent branch posts retain one publication origin, and every expected row must match its tenant and sequence and arrive after the declared lower bound, inside the unchanged delivery budget. Starting a new silence window after an HTTP response would reject a valid row whose collection or flush deadline elapsed while the harness awaited that response. The production owner’s timing checks retain the independent branch-deadline assertions.

The MQTT changed-address restart scenario waits for the post-restart marker within its bounded delivery window. Its persistent QoS 1 source may replay the pre-restart marker before that row; broker receipt does not establish that its source acknowledgement reached the broker before the node stopped. The OTEL metric scenario observes one successful export from each concrete emitter owner before checking Collector output. HTTP intake acceptance does not establish sink export, and the per-owner counters retain exact progress assertions for both HTTP and gRPC.

The OTEL Collector content wait keeps its ten-second bound, from the step’s start to the text’s appearance in the followed transcript, counting output printed before the step began. Each look searches only what arrived since the previous look, plus the bytes an occurrence could span across it, and every append wakes the wait, so the text is seen as soon as the follower reads it. No look re-reads the whole output, so a record delivered within the bound cannot be missed by a long read that began before the delivery and ended after the bound. If the bound expires, or the follower stops first, the failure reports whether the text arrived after the wait ended, how far the follower has read each output and how it stopped, the latest 200 Collector lines that name the scenario’s test identifier, and the last command output. A collector timeout alone does not establish that the emitter lost a record or failed to export it; retained diagnostics and a new attempt remain separate evidence.

A scenario step can also read status inside a window of its own, passing that window as the phase deadline. Where the window bounds how long something is watched rather than how long one read may take, every read keeps a full request timeout instead.

The harness session client keeps domain-clock replies and frames in arrival order. When a detach reply ends an attachment, it retires that domain’s unread frames from the pending observation queue while preserving them in the ordered log. Assertions about events after detachment therefore start at the acknowledged boundary, including when ticks arrived while the client awaited the reply.

Node Tasks And Readiness

An in-process node is the server application started as one Tokio task. That task is the node’s root, not the whole node: the application spawns the tasks that run its domains, its consensus, its gossip and its interconnect, and those tasks hold its databases and listeners until the application’s own shutdown stops them. The harness holds the root task through a single owner, which is in one of four states:

• Not started. No task has been spawned.
• Running. The task is live, and the owner holds its join handle.
• Left stopping. The task outlived the wait its owner could give it and was handed to a keeper, which waits for the node to end.
• Terminal. The task ended, and the owner holds how: a clean application exit, an application error carrying the node’s own report, a panic, or a cancellation, which is how a task ends when the runtime that ran it is dropped.

Running and ended are read from the owner’s state and never inferred from whether an error message exists. Inspecting a task that has finished joins it and keeps its terminal outcome, so a diagnostic never takes the handle a later stop needs. A wait receives its deadline and reports that the task was never started, had already ended, ended within the deadline, was still running when the deadline passed, or had already been left stopping.

The owner never aborts the task. An abort would end only the root future and leave the tasks the node spawned running with its databases, its listeners and its connections to its peers, owned by nothing and joined by nothing: a second launch into the same directory would find the database locked, and the orphaned node would keep working beside every scenario that followed it. A node ends by its own shutdown instead. One that is still running when its owner stops waiting is left stopping: the owner hands the task to a keeper that waits for it to end, and the keeper holds what the node still uses, the cluster’s storage, the node’s seven ports, its consensus fault registration and the cluster’s place in the live-cluster registry, and gives them back only then. A node whose handle is dropped while it runs is asked to stop and left stopping the same way. A keeper prints the end it waited for:

node '<node>' ended <elapsed> after it was left stopping: <terminal outcome>

A node that never ends keeps what it holds until the process drops the runtime.

A readiness probe is one status request, and it reports one of three outcomes:

• Ready. The node returned a successful command result, or one saying it is not the leader. Either way it answered an authenticated command.
• Request failed. The status request ended with its typed failure, including the operation a passed deadline interrupted.
• Non-ready response. The node answered with an unsuccessful result, kept with its kind, message, and diagnostics.

The readiness wait probes every 200 milliseconds within the attempt’s deadline and remembers the most recent outcome. The owner inspects the task before each probe and after it. The wait succeeds when a probe is ready and the task still runs. It fails with node task terminated before readiness when the task has ended, and with startup deadline expired when the deadline passed while the task was running. Either failure names everything needed to tell resource pressure from a deterministic failure:

node '<node>' startup attempt <n> failed after <elapsed>: <failure>; task state: <state>; last readiness outcome: <outcome>

The task state is not started, running, or terminal with the terminal outcome, and the last readiness outcome is the most recent probe or no readiness probe completed.

Node Startup

Starting one node has one advertised worst case, 84 seconds, whatever happens inside it. The budget starts before the first launch, and every readiness probe, every cleanup after a failed attempt, and every pause between attempts receives only what is left of it.

           one startup budget: 84 s, fixed before the first launch
 |<------------------------------------------------------------------->|
 | readiness | cleanup | pause | readiness | cleanup | pause | launch 3  |
 |  <= 36 s  |  <= 5 s |  1 s  |  <= 36 s  |  <= 5 s |  1 s  | remainder |
 |<-------- attempt 1 -------->|<-------- attempt 2 -------->|

The budget pays for two attempts at their full length. A third launch is allowed because a launch that fails at once, such as one whose listen address another process took first, leaves most of the budget unspent; that launch receives whatever the earlier ones left.

After an attempt fails, the harness classifies the failure, cleans up, and records the attempt.

Cleanup requests the node’s stop and waits for its task within a 5-second slice of what the budget has left. A node that never became ready has no drain to finish, so the product’s shutdown deadline plays no part here. A node whose task ended inside the slice gives back its consensus fault registration, and after a transient failure it moves the selected listeners to freshly allocated ports. A bind failure preserves unrelated addresses; a readiness timeout selects all seven ports. The harness pauses one second before launching again.

A node still stopping when the slice passes is never launched again. It still holds its database and its listeners, and it will until its own shutdown has finished, so a fresh launch into the same directory could only fail on the lock its predecessor holds. The node keeps its task, its shutdown coordinator and its fault registration, the startup ends, and whatever stops the cluster afterwards waits for the node or leaves it stopping.

The startup ends in one of five ways, each reported with the node, its budget, the time spent, and the record of every attempt:

• the last attempt failed for a terminal reason;
• every launch the attempt limit allows was spent;
• the budget ran out before another launch could start;
• the node the last attempt launched had not stopped when its cleanup slice ended;
• fresh ports for the next launch could not be allocated.

Each attempt’s record names its number out of three, its classification, how much of the budget had been spent once it was cleaned up, its typed failure, and its cleanup: nothing was launched, the task stopped with its terminal outcome, or the node was still stopping at the cleanup deadline. A cleanup that the spent budget left no time for records the node as still stopping too, and that startup ends as the spent budget or the spent attempts rather than as a node that would not stop.

The harness prints every transition to standard error as it happens, so a run heading for exhaustion is legible while it is still going:

node '<node>' startup: ready after <elapsed> on attempt <n>/3; <remaining> of its 84s budget left
node '<node>' startup attempt <n>/3 [<transient or terminal>] failed <elapsed> into the budget: <failure>; cleanup: <cleanup>; <remaining> of its 84s budget left

The budget bounds what the harness waits for, not what the operating system schedules. A runner that stops running the harness’s own tasks for longer than an attempt delays the check that ends it, and the measured startups above include such a runner.

Cluster Construction

A cluster is built one node at a time, bootstrap node first. The construction has one deadline, 84 seconds times the node count, and each node’s startup budget is nested inside it, so no node can outlive the construction. Once a node is ready, the harness asks the other nodes for the current leader and that leader’s applied index within one 40-second lookup, and then waits up to 40 seconds for the new node to apply as far.

Once every node has started, construction waits for a current leader on the bootstrap node. A cluster of more than one node must also converge, and each check is its own 40-second status wait: a current leader on every node, the full voter set, a consistent leader on all nodes, and connected interconnect status from every node to every peer, which is six waits in a three-node cluster. A construction therefore waits up to 84 seconds per node for startups and then up to 40 seconds for each status wait that follows.

The start step logs cluster start requested: with the node count, domain, and test id. When construction fails, the harness stops the nodes it had started the way scenario cleanup does, all of them together under the one 60-second cleanup budget, because a construction that failed has no product shutdown deadline left to observe. It logs cluster start failed: with the error, followed by the cleanup record and every node that was still stopping or panicked when there was one, and fails the step.

The other ways a scenario starts nodes reuse the same budgets. Restarting a whole cluster stops its nodes and builds it again with a fresh construction deadline. Adding a node, or starting one a scenario stopped, spends one 84-second node startup and the catch-up that follows it; adding a node also repeats the convergence checks. A node started while its consensus connectivity is blocked spends the node startup without the catch-up. A scenario whose subject is a node that cannot apply what the leader committed starts it with one attempt’s 36-second readiness and no retry, because a retry would restart a node that had already consumed the failure the scenario armed.

Scenario Phases

A scenario publishes the phase it is entering, so a reader sees the work in flight rather than the last work that finished.

queued -> started -> body complete -> teardown started -> teardown diagnostics -> stopping -> finished

finished is published only once cleanup has completed. A scenario holding any other phase has not finished, and a scenario whose log ends at a phase marker is still inside that phase.

Browser assertions that wait for an acknowledged subscription tab to disappear poll for up to 10 seconds. This bounds the server reply and browser update together; on expiry, the step reports the tab text that remains visible.

The active-scenario registry holds every scenario that has started and not yet ended. A scenario registers in its before hook, before it acquires its permits, so a scenario that never gets them is visible while it waits. Its entry holds:

• its identity: the feature, the scenario name, and the line where it begins, which tells apart the example rows of an outline that share one name;
• its attempt, counted from one, which is exact because Cucumber takes up a retry only after the run before it has ended;
• its phase, when it entered that phase, and when it started.

The entry is keyed by a registration the scenario’s world owns rather than by anything the feature file supplies, and it leaves the registry when that world is dropped, whether the scenario passed, failed, or panicked. The suite’s report prints scenarios in order and can keep a finished scenario’s world until earlier scenarios have reported, so a finished entry can still be present when the registry is read.

Entering a phase also appends a marker to the scenario log:

scenario <phase>: feature="<feature>" scenario="<scenario>" line=<line> age=<age> <detail>

body complete carries result= with the body’s result, which is passed, skipped, before hook failed, or step failed with its error. finished carries body= with the same result and teardown= with the cleanup record. Cleanup reports itself separately: a node left stopping and a node that panicked while stopping are recorded beside the scenario’s result and never turn a passing body into a failure.

Scenario Teardown

The after hook runs every scenario’s cleanup through the phases above.

body complete
     |
teardown started      release paused health responses, domain-clock progress pauses, and
     |                command pauses; stop durable catch-up writers and clear consensus
     |                commit delays
teardown diagnostics  every node's status at once within 10 s; the scenario's context
     |
stopping              abort CLI output readers and kill their child processes;
     |                drop HTTP load, held uploads, server processes and process clusters,
     |                and observers;
     |                stop HTTP and gRPC receivers within 6 s; close the browser and the
     |                session;
     |                stop the cluster within 60 s;
     |                release proxies, silent peers, permits, and fixture ports
finished

Teardown Diagnostics

The harness asks every node for its status at once, under one 10-second budget for the whole cluster, so a node that never replies delays cleanup by at most that long and cannot hide another node’s status. Each node’s outcome goes to the scenario log on its own line: its status text, the operation its request was still in when the budget passed, or its typed failure.

scenario teardown <identity>: node=<node> status=<status text>
scenario teardown <identity>: node=<node> status_timeout=<operation> pending after 10s
scenario teardown <identity>: node=<node> status_error=<failure>

A scenario context: line follows with the scenario’s domain and test id, its last command error and output, its last server error, and its last subscription and broker payloads. Diagnostics are best effort: whatever they observe, cleanup continues into stopping. Steps that explain their own failures can collect the same snapshots.

Cluster Teardown

A cluster’s nodes stop within one deadline of 60 seconds for all of them together.

1. Every node is asked to stop before any node is awaited.
2. Every node’s task is awaited concurrently under the one deadline. Each wait borrows its task from the node’s owner, so a node still stopping when the deadline passes is still owned.
3. Once the waits are over, each node whose task ended, whatever ended it, gives back what it holds: its seven port leases and its consensus fault registration.
4. Each node still stopping is left to end itself. Its task goes to a keeper together with what the node still uses, and the keeper gives that back once the task has ended.

A cluster of three wedged nodes therefore costs one budget rather than three, and no path ends a node or takes anything from under one that still runs. The cleanup record reads stopped <n> of <n> node(s) in <elapsed> of a 60s budget, <n> still stopping. Dropping the cluster afterwards removes the temporary storage its nodes wrote as soon as no node left stopping still shares it. Two more records appear only when they apply:

scenario cleanup unfinished: node "<node>" was still stopping at the cleanup deadline, in shutdown phase <phase>, <the node's own deadline>, and keeps its storage and ports until it ends
scenario live during unfinished cleanup: <identity> attempt=<n> phase=<phase> phase_age=<age> age=<age>
scenario teardown failed: <node record ending in a panic>

A node left stopping is read through its shutdown coordinator before it goes to its keeper, as Scenario-Driven Stops describes, so the record says which of the node’s own shutdown phases the cleanup deadline found it in.

An unfinished cleanup is usually cleanup that ran beside work heavy enough to starve it, so the harness names every scenario live at that moment, and the keeper’s own line later says when the node ended. A node whose task panicked while stopping has no other witness than its record. In the qualification runs that characterized them, cleanups that ran out their budget were nodes of WASM scenarios whose setup had already failed with not-a-leader, and those scenarios passed when retried.

A step that replaces a scenario’s cluster with a fresh one stops the old cluster under the same budget and fails when a node of it was left stopping: that node still runs beside the cluster that would replace it, so the replacement has not happened.

The consensus fault registration keeps an observer of the node’s consensus state, as does every observer a step took from it, so the node’s consensus database stays open after the node’s task has ended until the last of those observers is dropped. That drop closes the database synchronously, on the task that makes it: the close synchronizes the journal and waits until none of the database’s background workers is left. Cleanup and scenario-driven stops give the registration back on the runner task, so every scenario and the suite watchdog wait for that close, while a keeper gives it back on a runtime worker. The close never waits for a worker whose flush or compaction failed; see Terminal Teardown.

What Cleanup Releases

Harness state goes back only after the tasks that used it have ended, so the next scenario never finds a port, a fault, or a proxy taken. What a node left stopping still uses stays with its keeper until that node has ended.

• CLI output readers, the terminal task of an interactive CLI among them, are aborted and their kill_on_drop child processes are dropped before node teardown, which also removes the REPL’s working directory. Background HTTP load, held uploads, server processes and real-process clusters are also dropped first. Dropping any server child kills it and returns its ports.
• Broker and syslog observers, HTTP and gRPC receivers, the browser, and the session are closed before the cluster stops.
• The TCP proxies and silent interconnect peers a scenario placed in front of its nodes are released once those nodes have ended.
• The ZeroMQ and syslog ports drawn for the scenario return to the pool after their nodes and observers have ended. The scenario publishes finished and then releases its feature permit and run slot, so admission includes all teardown work.

Scenario-Driven Stops

A step that stops a node, stops every node, or restarts the cluster is part of the scenario’s subject, so it runs under the product’s deadlines with a harness watchdog around them rather than under the cleanup budget. The harness requests the stop and waits for the node’s task for the longest of five minutes, the node’s configured shutdown timeout, and its configured drain phases: the application drain when graceful drain is enabled plus the runtime’s branch stop timeout for the configured domain drain. A scenario that configures a longer product deadline raises the watchdog with it, so the harness never cuts a product deadline short. Node-specific stop steps resolve a saved scenario placeholder before selecting the node, including the graceful stop used to exercise ownership handoff during drain.

Once the node’s task has ended, the stop gives back the node’s consensus fault registration, which closes its consensus database as Cluster Teardown describes, and a stop that ended cleanly then checks that the node released its node and consensus database locks. The step fails when the node’s task panicked, when the watchdog passed with the node still stopping, or when a lock outlived the stop. A node still stopping keeps its task, its ports and its fault registration, and the scenario’s cleanup waits for it again or leaves it stopping. Stopping every node requests every stop first and then waits for each node in turn. A scenario can also begin stopping a node without waiting, to observe how the cluster reacts while the node is on its way down; its cleanup waits for it later.

The harness prints every such stop to standard error when its wait ends, as it prints a startup:

node '<node>' stop: ended <elapsed> after its stop request, inside its <watchdog> watchdog: <terminal outcome>; shutdown phases: stop admission <outcome>, drain support <outcome>, terminal teardown <outcome>
node '<node>' stop: still stopping <elapsed> after its stop request, at the end of its <watchdog> watchdog, in shutdown phase <phase>, <the node's own deadline>

The elapsed time runs from the first stop request the harness made of that launch of the node to the moment its task ended. The task records its own ending, so the stops of a whole cluster, which are requested together and awaited in turn, each report how long that node took rather than how long the harness took to reach it. A stop is printed by the wait that observed the ending, and only by that one.

The second line is also the failure of the step. A node that outlived its watchdog is read through its shutdown coordinator, which changes nothing in the node: the phase is the product’s StopRequested, DrainSupport or TerminalTeardown, and the node’s own deadline reads <remaining> before its own shutdown deadline or past its own shutdown deadline. An in-process node has no deadline supervision to end it, so a node read in TerminalTeardown past its own deadline is running the final stops the product lets run to completion, while one read in DrainSupport is still inside a drain its scenario configured. A node that had already been left stopping under a keeper has handed its coordinator over, and its record says so instead.

Ports

Every port the harness binds is drawn from one pool shared by all scenarios in the process. A draw binds port zero on the loopback interface, reads the port the operating system chose, closes it, and reserves it in the pool. The operating system would let a second scenario bind a closed port, so the reservation is the only thing that keeps two scenarios apart. It cannot see sibling worktrees running the same suite, which is why a node startup retries a lost bind on fresh ports.

A port returns only once nothing can still dial it. Every scenario names its nodes node-1, node-2, and node-3, so a node of an unrelated scenario that binds a port a peer still dials passes the peer-identity check and is caught only when its introduction fails to verify, which takes long enough to fail a scenario. Keeping every port reserved is not an option either: the ports a whole run draws exceed the range the operating system draws from, because Linux hands a port-zero bind an odd port from the lower half of its ephemeral range, some seven thousand ports.

A draw is synchronous and runs on the runner task. A draw that looped until the operating system offered a port the pool did not hold would stall every scenario and the suite watchdog with it, so a draw that lands on reserved ports 65,536 times in a row fails with the pool exhausted, naming how many ports the process holds, and gives back the ports it had already reserved. A pool with one free port among seven thousand still yields it within that many draws almost every time, and a pool with none ends the draw in about a second.

Server Processes

Scenarios about signals, exit statuses, process startup, command-line values, and open-file limits run nervix-server as a real child process, because an in-process node cannot show whether the process boundary delivers a signal to its shutdown coordinator. The fixture gives each process its own ports, database directory, and interconnect credentials, forms a single-node cluster, and captures its standard output and error in one log. It removes every NERVIX_* variable and RUST_LOG from the child’s environment, so the runner’s configuration cannot silently reconfigure the server. The options and environment variables a scenario gives the process are applied after that removal, written exactly as the scenario states them, so a scenario can hand the node a value it must refuse; such a scenario starts the process without waiting for readiness and asserts how it exits. A claim about a node process dying while its peers keep running, such as what a client producer is told when the node that executes its ingestor or serves its session is killed or frozen, runs the three-process cluster and faults one member, because only a real process death closes, or stops answering on, every connection the node held at once without running any of its shutdown.

Readiness uses the same probe outcomes as an in-process node, probing every 100 milliseconds within 120 seconds, and fails at once with the exit status when the process exits first. Waiting for an exit is bounded by 120 seconds, and waiting for a log line or an HTTP admission by 60 seconds. Those failures quote the last 80 lines of the process log. Dropping the fixture kills the process, so cleanup never waits for a graceful stop and a failed scenario never leaves a server running. What the process does between a signal and its exit is product behavior; see Shutdown And Recovery.

Test Dependencies

The external Chaos directed-rate probe times the sender’s connection and the receiver’s complete byte count through helper-local kernel uptime. A successful observation requires the same kernel boot identity and boot-time offset at both helpers. Namespace identifiers are diagnostic values; they do not identify the clock coordinate. The ten-millisecond clock precision produces a duration upper bound. Its lower bound subtracts two ticks from that upper bound because both endpoint readings can round down. Rate-effect qualification requires the fault’s lower bound to reach twice the baseline’s upper bound. Docker control time remains in separate helper-lifetime evidence and the unchanged ninety-second total probe budget. Missing, malformed or mismatched clocks fail the observation with raw helper logs retained. Direct Docker controls exercise launch delay, actual connection and receiver stalls, byte-count failures, failed helper operations and the real rate effect; they qualify the harness measurement and establish no product recovery outcome.

Brokers, databases, and the other external systems that scenarios use run as containers, started the first time a scenario needs one and shared by every scenario of the run. The run removes them when it ends; NERVIX_TESTCONTAINERS_MODE=reusable, which just test-scenarios-reuse sets, keeps them for the next run instead. Scenarios still provision the topics, queues, tables, and other entities they use explicitly.

The attached- and detached-window Kafka acknowledgement scenarios create their input topic before START and observe the consumer group’s membership and the ingestor’s readiness before injecting the output failure. Their bounded subscription assertions therefore exercise replay and acknowledgement isolation after source initialization, without spending those assertions’ budgets on automatic topic creation and consumer discovery.

The attached-window fixture publishes both rows through one producer and allows a thirty-second partial-batch wait. Collection ends as soon as its two-message ACK batch is full; the window can then produce the output whose failure rejects both records. Separate publication steps can exceed a short collection window, leaving one retained window row waiting for input while the source waits for that partial batch’s ACK. The fixture retains its five-second ACK timeout and exact four-record replay assertion within eight seconds.

The Pulsar broker announces a maxMessageSize of 1 MiB rather than Pulsar’s 5 MiB default, the same limit the MQTT and NATS brokers keep, so one scenario message can exceed each broker’s limit. Its admin API serves the topic-level maxMessageSize policy a scenario sets on its own topic; the step waits up to 60 seconds for the broker to read the policy back, which shows that the policy applies to the next message.

Harness Redis connections use an explicit ten-second budget for connection setup and each command response to tolerate scheduling delay under parallel load. The driver’s one-second connection and 500ms response defaults are too short under concurrent scenario load: the three-node JAQ transformation scenario reached a running ingestor but its publishing client timed out. The budget belongs to the test client used for individual publishes, bursts, and subscriber-count observations. A request failure still fails the step; a publish with an ambiguous result is never retried. The existing bounded subscriber wait only republishes when Redis confirms that the publish reached zero subscribers. Focused harness regressions delay setup and publish replies by two seconds and verify that a silent broker still reaches the connection deadline.

After the run, the suite stops its dependencies within 2 minutes. A stop that does not finish is abandoned and its containers are left to the runner, because waiting without a bound is how a run that already has its result loses it to the job’s own timeout. Dropping the runtime then waits at most 60 seconds for blocking tasks a scenario left parked in a driver.

A container that fails to stop or to be removed does not keep the others from being torn down. Its failure keeps a ContainerTeardownError report with the Docker or Testcontainers cause beneath it, and so does a Ryuk cleanup container the suite fails to remove. The teardown stays typed until the process prints it, on one line: suite dependency teardown failed: and the whole chain of every failure, the failures separated by semicolons. A teardown that failed is not clean, so it fails a run whose scenarios all passed; after a suite timeout it is printed beside the timeout instead.

HTTP Receivers

A scenario about a node sending HTTP requests to an external endpoint starts an in-process HTTP/1.1 receiver in its place, because only a receiver the harness controls can capture exactly what arrived and choose exactly how to answer: a status sequence, a delayed, held, or lost response, a stalled body, or malformed framing. A receiver can serve TLS with a certificate for chosen names and can require the client certificate it issued, whose files a node mounts as a resource. Its response script can generate exact and over-limit header counts and value sizes separately for an interim or final block. It timestamps a request when its complete body arrives, before its scripted response delay begins, so a scenario can assert the measured gap between two requests without racing a window in which nothing should happen.

Three controls keep the remaining HTTP assertions free of timing races too. A Retry-After date is computed when its response head is written, the scripted delay after the current UTC and rounded up to a whole second, so the date always asks for at least that delay however late the request arrives. A response held until released stays unanswered, and so the attempt that sent it unresolved, until a scenario step releases it with a response of its own choice; only the client’s own timeout or the receiver’s stop ends it sooner, so a scenario can observe state that must hold while a request is unresolved without betting on how long a scripted delay lasts. A request for a target the scenario answered by target takes that answer instead of the next scripted response, which gives each request of independent branches or source relays its own outcome whatever order they arrive in. Answering the same target again replaces its answer, so an endpoint that kept failing one request can recover while the scenario watches the retries.

The receiver also records two facts about how its clients treat their answers, each read once the requests it concerns have arrived. It keeps the most requests that were ever awaiting a response at once: a request awaits from its capture until the receiver begins writing its final head, or until its connection ends without one, so a client that sends each request only after reading the previous final head never has two. And it counts the responses its clients abandoned: a held response, a stalled body, or a body still being written whose connection the client closed first. A response body can be generated at a scripted size and is written chunk by chunk, so a body many times the socket buffers of a loopback connection proves that a client which stopped at the final head never read it. A stop of the receiver itself abandons nothing, and an abandoned response is not a fault.

Everything a receiver holds is bounded, and exceeding a bound is recorded as a fault, not captured.

Every await a receiver connection makes also waits for the receiver’s stop, so a held response or a stalled body ends as soon as cleanup begins. The receivers of a scenario stop together in the stopping phase, before the cluster, under one 6-second budget: 5 seconds for every connection to end on its own, after which the rest are aborted and joined, and 1 second to join the accept loop, after which it is aborted and joined too. The derivation asserts that this budget is shorter than the cluster’s cleanup budget. Each stop is recorded in the scenario log:

HTTP receiver cleanup: <name>: stopped <n> connection(s) in <elapsed> of a 6s budget, <n> forced, <n> panicked; captured <n> request(s), recorded <n> fault(s)
scenario cleanup forced: HTTP receiver <name>: <the same record>

The second line appears only when a connection or the accept loop had to be aborted, or panicked. A receiver’s port is drawn with the scenario’s fixture ports and goes back with them at the end of cleanup, once the nodes that dialed it have ended.

gRPC Receivers

The node trace-export scenarios start one or three real server processes with --otel-enabled, separate service names and the scenario’s DNS configuration. A local gRPC collector captures their OTLP trace requests; the steps decode the current trace schema and require spans from every node and a query for the collector’s fixture name. This uses the process cluster’s existing readiness and failure diagnostics. Processes are dropped before the DNS authority, resolver files and collector, including when a scenario fails. The fixtures use no external collector container.

A scenario about a node calling an external gRPC service, such as an OTLP/gRPC collector, starts an in-process receiver that serves HTTP/2 without TLS in its place. It reads each unary call to the end of its request and captures the method, the header fields, the compressed flag, and the one request message exactly as it arrived. It then answers the call with the next scripted answer: an empty response message with grpc-status: 0, a trailers-only status, no answer at all while the receiver closes the whole connection, or no answer until the client resets the call. Once the script is empty, calls are accepted. The receiver never interprets a message, so a scenario step decodes what it captured.

An OTLP receiver is the protocol-neutral name a scenario uses for either transport: Given OTLP receiver "<name>" is running for "<protocol>" starts a gRPC receiver for grpc and an HTTP receiver for http/protobuf, and one script vocabulary — accept, reject, unavailable, lose response, hold response — maps to the answer each transport gives for it.

A gRPC receiver has the HTTP receiver’s bounds on its message, its captured calls, and its kept faults. A call whose length prefix already declares more than 16 MiB is refused as a fault before its message is read, and a request that is not exactly one gRPC message is a fault rather than a capture. Every await a connection or call makes also waits for the receiver’s stop. The gRPC receivers of a scenario stop together with its HTTP receivers under the same 6-second budget, and each stop is recorded in the scenario log:

gRPC receiver cleanup: <name>: stopped <n> connection(s) in <elapsed> of a 6s budget, <n> forced, <n> panicked; captured <n> call(s), recorded <n> fault(s)
scenario cleanup forced: gRPC receiver <name>: <the same record>

A call that panics takes its connection with it, so the stop counts it among the panicked connections.

Broker Observers

A scenario about a node that publishes to a broker observes the destination through a client of its own: a Kafka consumer, a subscriber of MQTT, RabbitMQ, Redis, Pulsar or SQS, a ZeroMQ socket, or a core NATS subscription. Its steps then assert on what the broker delivered to that client, in the order the broker delivered it.

Those steps run on the runner task, so how an observer hands a message to its step decides how many polls of that task a burst of messages costs. Most observers run one task on the binary’s runtime that reads the broker’s client and hands each message to the step over a bounded channel, and the scenario aborts that task when it drops the observer. A core NATS observer has no task of its own: the NATS client already reads what the server delivers into the subscription’s own buffer on a task of its own, and the step takes each message straight from that buffer. A step that is behind takes as many buffered messages in one poll of the runner task as the poll’s cooperative budget allows.

Under the suite’s concurrency the runner task comes back to a step only after it has polled every other woken scenario. While the NATS observer handed each message on through a one-message channel, every message waited for a poll of its own, and on a loaded runner the NATS emission scenarios drained their 32,768 messages too slowly for their 90-second bound although the observing client had already received every one of them.

The NATS client buffers at most 65,536 messages for one subscription and, as a slow consumer, drops what arrives beyond them, so a scenario keeps the messages its step has not yet taken below that bound.

A step that waits for a sequence of messages and times out names how many messages the observing client had received from its broker, for a client that counts them, so a message that never reached the observer reads differently from one the step had not yet taken.

DNS Authorities

A scenario about how nodes find one another by name selects its cluster’s peer addressing before the cluster starts, with Given cluster peers are addressed by "<addressing>". Every other cluster uses literal IPv4 loopback endpoints, starts no DNS fixture, and lets its nodes load the host’s own resolver configuration, as a production node does.

A cluster addressed by names starts one in-process DNS authority on a loopback UDP port drawn from the port pool. The harness writes a resolver configuration with the search domain nervix.test, ndots:1, and one-second queries with one retry, and a hosts file, into the cluster’s directory, and starts every node with that configuration and the authority as its only name server. Every answer carries a one-second TTL, positive or negative, so a zone change reaches a node within seconds of the cached answer expiring. A scenario can move a stopped node to 127.0.3.<n> behind the same name, and answer a node’s name as a name that does not exist, a name with no address, or not at all. A name outside the zone does not exist, with a negative TTL of zero. Every test certificate names localhost, 127.0.0.1, ::1, node-<n>, and node-<n>.nervix.test, so each addressing presents the name its peers expect.

A scenario can also publish a dependency it started under a name in the zone, such as rabbitmq.nervix.test, answered with the dependency’s literal address or with a list of loopback addresses in order, and later answer that name as a name that does not exist, a name with no address, or not at all. A dependency’s published port listens on every local address, so an answer that cannot connect needs TCP forwarders: listeners at chosen loopback addresses such as 127.0.5.<n>, all on one port drawn from the pool, that forward to the dependency and count the connections each accepted. Stopping a forwarder closes its listener and every connection it carried, so a client that still holds a cached answer naming it cannot reconnect there. A scenario can restart that listener on its still-reserved port to let the same client reconnect while its target node stays up; the CLI clock process scenario uses this to observe reattachment after a transport loss without changing the node’s clock installation. An address the scenario names without a forwarder refuses connections on that port. The TLS certificate the harness gives its containerized dependencies names localhost, 127.0.0.1 and *.nervix.test, so a dependency reached through a zone name presents a certificate for the name its client dialled, and a name outside the zone does not match it.

The authority answers from memory and holds nothing else: it counts questions for at most 1,024 names, and a scenario can assert that its nodes asked nothing for their names. It answers until the cluster is dropped at the end of cleanup, after every node has stopped, so no node’s lookup outlives it; dropping it aborts its answer loop, and its port goes back to the pool. How long a node waits for an answer is product behavior, bounded by the budget of the path that asked and described in Bounds, Deadlines And Cancellation; the scenario’s own waits are status waits.

The resolver crate’s focused checks use the same authority, started on an unused port of their own, to observe DNS messages directly: TTL expiry, negative caching, refusals, silence, the lookup budget, the concurrency bound, and runtime teardown. just test-dns runs them. Evidence states what each form of DNS evidence establishes.

Client Probes

The cross-language conformance scenarios in client_conformance.feature run a small client of each runtime against a scenario’s cluster, or against the checked-in conformance corpus. The in-process probe drives the shared Rust binding’s C ABI from a blocking task of the binary’s runtime, because every call of the binding blocks its caller; every other probe is a child process that the fixture starts with its target in NERVIX_PROBE_* variables and its standard input closed. A probe prints one report line per observation on standard output, and the fixture keeps every line it read.

A probe’s waits are bounded twice. The step that starts it waits at most 180 seconds for the line that says its subscription is open or its clock attach completed. A later step can wait, for the duration it names, for one more line the probe prints, so a scenario acts between the probe’s observations: the clock probe prints the first tick of a generation before the scenario stops and starts the domain, and the interruption before the scenario restarts the TCP forwarder the probe entered through. Every line read on the way is kept for the report. The step that reads the report waits the duration the step names for the probe to end, 180 seconds against a cluster and 60 against the corpus. Each probe also ends itself: a binding probe gives up on its rows, or on each stage of the clock it follows, after 120 seconds, and the Go and TypeScript probes give up on their whole run after 170. A failure quotes every report line read so far, the exit status, and the probe’s standard error. Dropping the fixture kills a child process, so a failed scenario never leaves a probe running; the in-process probe ends when its session fails against the stopped cluster, or at its own deadline.

Every example of a runtime other than the in-process probe is tagged @client_conformance_toolchain and one @client_probe_<runtime> tag, and the suite excludes the first tag unless a run selects its own tags, because those examples need toolchains the suite’s job does not install. just test-client-conformance builds every probe artifact and runs them; its first argument is the tag expression that selects runtimes. The client-conformance CI job, which runs only for a pull request labeled client-conformance, runs it on its own runner with a 60-minute limit and a 15-minute suite budget, which leaves the builds before the scenarios up to 40 minutes of the limit and keeps the same 5-minute reserve.

The Suite Watchdog

The scenario run has one budget, 41 minutes from the moment it starts, which --suite-budget or NERVIX_TEST_SUITE_BUDGET replaces with a duration such as 4m, read by the same guarded duration parser as a node’s options. The budget is a clock rather than a count of failures. Cucumber’s fail-fast stops scheduling scenarios and leaves those already running where they are, so it cannot end a run whose step, diagnostic, or node stop never returns; the clock ends such a run at the same instant as one whose work returned at once. Until the budget expires, the suite keeps running its intended failure and retry coverage.

The watchdog reads two registries: the active-scenario registry, and the live-cluster registry. A cluster registers when a scenario builds it. Each node registers when its task is spawned, through a registration the task itself owns, so the node leaves the registry when its task ends, whether it returned, failed, or panicked, and the watchdog can tell a cluster that stopped from one still running without owning either. A cluster stays registered for as long as its scenario holds it or one of its nodes was left stopping, so a node that outlived its scenario’s cleanup is still named. The registry holds each node’s stop, its shutdown coordinator, rather than the node, so the watchdog never reaches into a scenario that is still using it.

When the budget expires:

1. The watchdog reads both registries before changing anything, so the diagnostic is the state the run expired in, and prints it to standard error at once. If the job’s own limit kills the process during what follows, it has still said what it was doing. The scenario log receives the same diagnostic, with the cleanup outcome, once the cleanup window has ended.
2. It asks every live node to stop, then waits one 60-second window for all of them together, checking every 50 milliseconds.
3. It drops the run. That ends every scenario future still held, which kills the server processes the scenarios started and leaves the nodes they still hold stopping.
4. The suite stops its dependencies within 2 minutes and drops the runtime within 60 seconds, which ends every node task that is still running.
5. The process prints the dependency teardown and the cleanup outcome, flushes its output, and exits with status 124.

suite timeout: the <budget> suite budget expired with <n> scenario(s) active
  suite timeout active: <identity> attempt=<n> phase=<phase> phase_age=<age> age=<age> nodes=[<nodes>]
  suite timeout unclaimed cluster: <identity> nodes=[<nodes>]; its scenario had already left the registry
suite timeout cleanup: asked <n> node(s) to stop and waited <elapsed> of a 60s window; every node ended itself

When a node is still running at the end of the window, the cleanup line names its cluster instead of reporting that every node ended itself, and the scenario log adds suite timeout cleanup forced: the run was dropped with nodes still running. A live cluster whose scenario has already left the registry is a leak, and the unclaimed line is its only record.

The watchdog does not run after hooks. Faults a scenario injected stay in place while its nodes are asked to stop, so a node held by one can still be running at the end of the window, and it ends when the process drops the runtime. The induced timeout during qualification had one such node, held by a consensus commit delay that only its scenario’s cleanup releases.

The CI Reserve

The suite runs inside just test-scenarios-coverage in the CI scenarios job, after that job’s instrumented server, CLI and paced simulation driver builds. The tests job runs at the same time on another runner. The scenarios job’s timeout-minutes is 60, and the suite budget is derived from it so that the job ends on its own.

For focused local coverage, just coverage-scenarios <lcov-path> <scenario-options> starts a fresh measurement. just coverage-scenarios-append <lcov-path> <scenario-options> retains the current profiles and reuses unchanged instrumented artifacts for another scenario selection. When collecting a different source revision, first run just coverage-clean-workspace so instrumented binaries and line mappings from earlier sources cannot enter the new report. After collecting unit and scenario profiles, just coverage-report-workspace exports all workspace packages, including executed test files, as lcov-workspace.info with the checkout path beside it, and just coverage-merge maps that report onto the repository.

Both CI collectors export lcov-workspace.info from every collected profile, with no package selection, and with --no-default-ignore-filename-regex, and write the absolute checkout path they ran in to coverage-source-root.txt beside it. Test lines stay in these reports, so merging native extra reports cannot make executed library properties appear uncovered because their files were filtered. Executing a crate’s tests does not include that crate in a report whose package selection names only the root package. Filtering happens in the merge, not in the exports: the coverage job runs just coverage-merge over the coverage-tests, coverage-scenarios and coverage-native-extras artifacts, which maps every record onto the files tracked at the tested revision, drops dependency, toolchain and generated records by class, leaves out the native collections of other modes, and fails rather than publish a report that misses an input or a workspace member. It passes that one report to just coverage-patch, so library properties contribute to patch coverage alongside server and client tests. Patch coverage, CRAP scores and comment publication are advisory and never change the build verdict; the CRAP step fails only when its analysis cannot describe the merged report. The complexity verdict is the just ratchet count of functions above cyclomatic complexity 30, which the checks job enforces and which reads no coverage. The PR comment and workflow summary identify the exact comparison and the coverage-merged artifact retains the report, its merge record target/coverage-merge/record.json, target/patch-coverage.md and the CRAP reports target/crap-report/crap.json and target/crap-report/crap.md. The scenario collector also starts coverage in each published Python paced driver using the shared scripts/paced_simulation_coverage startup hook and configuration. It combines those completed process reports into target/paced-simulation/python.lcov. CI merges this scenario report with the deterministic Python open-policy report from the extra checks, alongside ordinary Rust coverage. The STOP/START generation-following outline also carries @paced_simulation_reopen, so both full Deloxide selections reach its Rust and Python drivers on one-node and three-node clusters. The merge reads each producer’s checkout path explicitly when it maps LCOV sources from Blacksmith onto files on its GitHub-hosted runner.

A healthy suite finishes inside its budget and exits 0 or reports its failures. A wedged one exits 124 with its diagnostic and leaves the upload its reserve. Either way, a step that runs whatever the job’s result uploads the whole tests/logs directory as the scenario-logs artifact.

A suite that passes ends before its budget, so the reserve then holds the coverage export, the uploads and cache publication, which waits at most 120 seconds. Saving the manifest took 5 to 14 seconds in the scenario jobs measured on 2026-10-14, and the daemon had already published every entry those jobs built. A suite that fails or runs out of its budget leaves publication not-run, so the reserve stays with the cleanup.

The first split-job PR run completed its suite in 21m49s with 88.3% run-slot utilization and four retries. One Raft snapshot scenario failed all three attempts because its setup accepted a purge that preceded the backlog under test. The corrected scenario waits for a purge beyond the measured backlog peak. The next run passed in 20m26s with 94.3% utilization and seven retries, so the 22-minute ceiling and 15-minute slack fit both measured runs.

The job’s limit remains the emergency guard outside the budget rather than the mechanism that ends a wedged run. A job the limit cancels is killed wherever its scenarios are: the logs it uploads end mid-scenario, with no summary and no record of what each scenario was doing. The limit and the harness’s copy of it change together.

A same-commit rerun of the scenarios job replaces that job’s four report artifacts: scenario logs, runner load, kache report, and scenario coverage. This lets a rerun publish its own diagnostics under GitHub Actions’ immutable artifact names. When the coverage job is rerun, its advisory reporter updates its marked PR comment for the same PR head instead of adding a comment.

Runner Stalls

The watchdog is polled on the runner task, so it cannot see, let alone end, a step that blocks that task. The run is therefore polled beside a tick on the same task, due every 250 milliseconds. A step that blocks inside its poll delays the tick until it returns, so how late the tick is polled is how long the task was held. A stall of one second or more goes to the scenario log with every scenario live during it:

runner task stalled for <duration>
scenario live during runner stall: <identity> attempt=<n> phase=<phase> phase_age=<age> age=<age>

The log line is a diagnostic, not a failure: a stall delays every wait and bound the other scenarios measure, and the line says when and for how long. Backup steps hand their archive reads, rewrites, digests and fixture generation, and the inspection of stopped nodes’ databases, to blocking tasks; the native metadata restore scenario, whose archive holds lifecycle and offset records above 32 MiB each, held the runner task for about 30 seconds while it inspected three restored nodes before they did.

Diagnostic Lane

just test-deloxide, just test-deloxide-order, just test-deloxide-stress and just test-deloxide-stress-restore run the Deloxide diagnostic lane: every workload tests/deloxide-inventory.toml registers, in the build of the diagnostic feature its selection compiles, supervised by scripts/deloxide_lane.py. Data-Plane Concurrency owns what the lane proves, its failure classes and its applicability records; this section owns how it runs and how long it may take.

The lane starts after its prerequisites, just tests-deps, and builds each invocation with Cargo before it runs it. The prerequisites provision what the workloads load, as they do for the ordinary suite: the WASM guests, the generated ONNX models and the ONNX runtime library, whose path the lane passes to every invocation. The probes, the tracked locks’ conformance checks and the owner tests are libtest executables, run directly so that a signal reaches the lane as a signal rather than as Cargo’s status. The scenario binary runs the lane’s tagged scenarios without retries. The active-only and order selections run them in two inventory invocations: the restore, client, diagnostic, remote ACK, Roto function, TLS rebinding and inferencer scenarios, then the paced-driver scenarios with the diagnostic Rust driver; the stress selections run the same scenarios in three invocations of their own, described below. Active mode, with or without stress, runs each invocation as one process. Order mode starts a fresh process for each tagged feature in the first invocation; the materialized restore feature runs each scenario separately and its six large examples each have their own process, as do the four complete-generation restore examples, selected by the inventory’s order_tags. The lane checks every chunk’s Cucumber summary against its registered examples, retains its logs and evidence separately, and stops on the first failed chunk. Each process of the active-only and order selections runs the inventory’s fixed number of scenarios at once, four, rather than one per CPU: a diagnostic build pays for its tracked acquisitions on every lock, and a fixed count puts the same load on its nodes locally and on CI’s 16-vCPU runner.

The lane resolves every feature pattern against its repository root before passing it to Cucumber. A relative pattern would make Cucumber traverse the whole working directory, including generated build artifacts. An exact feature uses an absolute pattern with two identical literal alternatives, so the walker starts at that feature’s parent directory and still selects only that file. This bounds discovery without changing the registered examples or their deadlines.

Under the native coverage collector the lane instruments only workspace crates, the scope the ordinary coverage build has. When every crate was instrumented, wasmtime’s compiler included, CI’s WASM restore scenarios ran five times as long as an uninstrumented local run, 338 seconds against 64 for one of them, and its restored runtime revision missed the product’s readiness deadline: the test cluster’s 10-second node unavailability timeout plus the 30-second propagation bound. It missed it at four scenarios at once as at eight, because the time went to preparing the restored WASM processor, not to waiting for a run slot. The product deadline stays as it is; the lane removes the instrumentation the product’s own work never needed. Each invocation records its evidence in its own directory of the attempt, and the binary’s tests/logs are copied into the attempt after it ends, because the next invocation truncates them.

Each scenario invocation passes the binary NERVIX_TEST_SUITE_BUDGET, its bound less the inventory’s teardown reserve, so a suite that runs out of time ends itself with the suite timeout diagnostic and its own cleanup before the lane would stop it. The reserve is six minutes: the four the suite’s own cleanup can take after its budget, with two to spare. A bound that expires anyway ends the binary with SIGTERM and, after the stop grace period, SIGKILL for its whole group; Ryuk removes the containers the binary started when its connection closes. A stall of the runner task itself ends only this way, because the suite watchdog runs on that task, and the binary then prints no Cucumber summary. Unless a tracked lock takes part in the stall, the lane records it as timed-out with no finding: a wait inside a dependency, such as the storage engine’s own worker channel while a database closes, is visible only in a stack of the binary’s threads taken before the bound expires.

In CI the deloxide job, which runs only for a pull request labeled deloxide, runs each selection on its own 16-vCPU runner through just coverage-native-extras test-deloxide, test-deloxide-order, test-deloxide-stress or test-deloxide-stress-restore, then just test-deloxide-qualification for the same selection, and uploads every attempt whatever the verdict.

The table’s measurements come from the CI run of 2026-10-07 in which both selections passed with every owner of tracked locks reached: 110 workloads in deloxide and 118 in deloxide-order, with the lane instrumenting workspace crates only and running 55 scenario examples at four at once. The first run in which both selections passed, on 2026-10-06 with 97 and 105 workloads and 49 scenario runs, took 30m27s and 23m18s in the lane’s step. The six scenario runs and seven conformance checks added between those runs fit inside their variation. A deloxide-order run that included the one-node restore of a deduplicator keyspace and a window above the bulk budget, together with its three-node example, took 21m47s in the lane’s step with 107 workloads and 51 scenario runs before those six scenario runs and seven conformance checks were added. The table predates the four remote ACK examples and the order-mode per-example process split, so its durations are historical measurements rather than a timing claim for the current combined inventory; the inventory and job bounds remain enforced.

The two stress selections compile one build, the active-only build with the bounded disturbance of StressConfiguration::LANE, and between them run every scenario of the active-only selection but the report tool’s triage, one at a time: a stress build enters Deloxide’s process-wide detector twice on every mutex and write acquisition, and when the selections were introduced, with four scenarios in one process, their nodes contended on it and missed startup and request deadlines that a plain diagnostic build of the same scenarios met. Two Stressed Scenarios In One Process records what made them miss. deloxide-stress runs the active-only selection’s probes, conformance checks and owner tests with the disturbance, the probe that proves it, the stress-scenarios invocation of 35 lifecycle scenario runs tagged @deloxide_stress, bounded at 30 minutes, and the stress-paced-simulation invocation of the 36 paced-driver runs with the stressed Rust driver, bounded at 40. deloxide-stress-restore runs the build’s probes and the stress-restore-scenarios invocation of 43 capture, restore and checkpoint-transfer runs tagged @deloxide_stress_restore, bounded at 70 minutes. Each selection has the inventory’s 80-minute budget and its own CI runner.

The split follows from what the disturbance costs. It barely lengthens a scenario, because the lifecycle scenarios wait on product cadences, timers and deadlines rather than on the CPU: in the CI run of 2026-10-09 the 33 runs the stress selection then held took 854 seconds one at a time, against 885 seconds of run-slot time for the same runs at four at once in the active-only lane. One at a time, a stressed lane therefore costs the sum of its scenarios’ durations, and the 44 scenario runs and 36 paced-driver runs not yet stressed took 1,587 and 658 seconds in the same run. One lane holding all of them one at a time would have needed about 52 minutes of scenarios after 11 minutes of diagnostic builds, about 63 of its 80 budgeted minutes, before the variation between CI runs, before the large-state WASM handoff added since, and before any scenario added later. Chunking that lane into a fresh process per feature keeps the sum and adds a process start-up per chunk. Running more stressed scenarios at once in one process, or fewer nodes in each of several concurrent scenarios, is the configuration whose nodes missed product deadlines then. The restore scenarios are the longest family, so they became their own selection: a budget and a runner of their own, and the same build, disturbance and probes. In the CI run of 2026-10-10 that introduced the split, deloxide-stress took 28m41s in the lane’s step and 27 minutes inside its budget: about 9.5 minutes of diagnostic builds, 6m19s for the 35 stressed lifecycle runs and 10m51s for the 36 paced-driver runs, with 30 qualifying evidence files and no finding. deloxide-stress-restore took 39m40s in its step and 37 minutes inside its budget: 5.5 minutes of builds and 31m30s for the 43 restore runs, with no finding. One lane holding both would have needed 58.5 of its 80 minutes, before any variation between runs. The disturbance delays one in twenty nested mutex and write acquisitions by at most 200µs, and the product’s physical and logical deadlines are not widened for it. A stressed workload that misses a product deadline is a finding to reproduce and fix in its owner, not a bound to raise.

The budget is measured on CI’s runner, whose diagnostic builds kache serves warm, and a local run pays its cold builds out of the same budget. On 2026-10-10 a shared 40-CPU host already at a load of 36 to 50 spent 33 minutes building the stressed scenario binary and ran the heaviest restore scenarios two to three times slower than CI, so deloxide-stress-restore expired its budget after 17 of its 43 runs, every one of which passed. Run the stress lanes where the CPUs are not already saturated, or again once their builds are warm; an expired budget ends the lane with status 124, never as a clean result.

Two Stressed Scenarios In One Process

At two stressed scenarios in one process, the restore scenarios failed in two ways the one-at-a-time selections never did: the cluster is restarted found a restored node still stopping at the end of its 300-second watchdog, and node startups spent their whole 84-second budget. Neither was a deadlock among tracked locks. Every evidence file of those runs qualified with no finding, and both failures are a node that could not get through terminal teardown, or through startup, beside the other nodes of its process.

The retained run of tests/features/cluster/backup_materialized.feature at 6c0d23de holds the signature. Its 28 stops by the cluster is restarted took 130.5 seconds at the median and 297.7 at the slowest, 2.3 seconds inside the watchdog, where the plain diagnostic build of the same revision took 0.9 and 39.6. Of its 129 node stops, 69 never reported the end of terminal teardown. Three things made a stop that slow, and CPU contention is what turned them into a failure: the run shared a 40-CPU host whose one-minute load was 47 at the median and between 28 and 75.

• Every node swept its whole acknowledgement registry every second, and a sweep that overran the second could not be cancelled. The remote acknowledgement silence sweep took a guard for each of a node’s 16,384 correlation positions whether or not the node had forwarded anything, and a stress build enters Deloxide’s one detector twice for each of them. On that host one sweep took longer than the second between sweeps. Its task then found the next tick already due and went from one sweep into the next without waiting, and the cancellation that runtime teardown joins the task with is observed only when the task waits. The run’s first cleanups show it: four nodes that had served for under ten seconds logged nothing through 59 seconds of terminal teardown while the process kept three CPUs busy. The sweep now visits only the positions a node’s deliveries and admissions have claimed, which keeps it far inside its second, and its task returns to its executor after every sweep, so teardown’s cancellation ends it after the sweep in progress even when that sweep overruns; see Terminal Teardown.
• Nodes the harness gave up on kept running. A cleanup that reached its deadline aborted the node’s root task, which ended that one future and left every task the node had spawned, its sweep among them, running for the rest of the process. A node aborted this way never reported the end of its teardown. The harness never aborts a node now, as Node Tasks And Readiness describes.
• A stopping domain waited for its generators one at a time. When every node of a cluster stops at once, a generator that is reading its source relay from another node, or publishing to one, waits on a peer that no longer answers, and runtime teardown gave each such generator its own two-second grace in turn. This one is independent of the detector: it is in the plain build’s 39.6 seconds too. The generators of a domain now share one grace; see Terminal Teardown.

The measurements below repeat that run’s configuration on a 32-CPU workstation shared with other builds: the whole feature in the deloxide-stress build, two scenarios in one process, sixteen runtime workers on sixteen CPUs, no retries, and the row, startup, stop and cleanup deadlines unchanged.

Without competing load, 6c0d23de itself is as fast here as the current revision: in the feature’s first four examples its restarted clusters stopped in 0.06 seconds at the median and 0.48 at the slowest. The sweep’s cost shows only once the process has to compete for its CPUs. A build of c39c5388 with nothing changed but the full sweep restored was run beside an unchanged one, the process confined to four CPUs that twelve busy loops competed for. The unchanged build passed 18 of 18 in 443 seconds. The build with the full sweep took 1,301 seconds and 1.7 times the CPU time, failed both three-node examples with forty relays when deliveries to a peer timed out during their input rounds, and stopped its restarted clusters in 5.3 seconds at the median against 0.36. 6c0d23de itself, run the same way, matched that build and not the unchanged one: 1,345 seconds, the same two examples failed, one of them with the missing rows of the retained run, and its restarted clusters stopped in 4.5 seconds at the median.

What a three-node stop still costs after this is bounded by the deadlines of the calls its nodes had in flight when the stop began, and each wait is named in the node’s log. A relay owner that is delivering to a peer which has already stopped waits out the five seconds that delivery’s admission is given and the five its cancellation is given. A checkpoint announcement to a replica that has stopped is waited for until its five-second dispatch deadline has passed. A replica’s placement task that is fetching or acknowledging a checkpoint when its owner stops does not observe the stop, and such tasks are aborted one after another, two seconds each. The node’s generators share their two seconds. In Every node stopping at once ends its generators together, the slowest node of the three-node cluster took between 2.0 and 12.0 seconds to stop in ten runs of the ordinary build. Before the generators shared their grace it took between 22.0 and 30.1 seconds in five of six runs, and 8.1 in the sixth, where two of the nodes had no generator waiting on a peer.

The stress selections keep running one scenario at a time: their budgets were measured that way, and two per process remains the comparison below rather than a lane.

For a focused comparison, just test-diagnostic-scenarios <selection> <feature> --name <scenario> --concurrency 2 --retry 0 builds and runs the public scenario in deloxide, deloxide-order or deloxide-stress, with the lane’s prerequisites and ordinary CLI. Its builds share the diagnostic target directory. The recipe resolves its feature file to an absolute path: Cucumber’s relative input glob walks the entire working directory, including build artifacts, before starting any scenario. The absolute input glob matches the exact file in its containing directory. Argument boundaries, including a scenario name with spaces, are preserved. Each attempt retains its output under target/deloxide/focused/<selection>/run.* and rejects an empty scenario selection. It records detector evidence there unless NERVIX_DEADLOCK_EVIDENCE names another directory. Use a fresh evidence directory for each run and retain the standard output, scenario log and suite summary together. The complete lane remains the entry point for inventory accounting and diagnostic evidence qualification.

In this comparison, --concurrency counts scenarios in one process. Their in-process nodes share one runtime and one detector; the runtime still has one worker per available CPU. Record the available CPU count, affinity and host load, and keep them the same for the plain and stressed selections. A failure’s phase matters: missing generator rows immediately after restore and missing rows after a subsequent restart exercise different installation boundaries. Preserve the first failure, the missing relay and tenant identities, and any stop or startup deadline outcome before running a control. A passing control or replay describes that run only.

The oversized materialized-generation assertion preserves its missing relay and tenant pairs, expected round and value length, domain, and subscription node when its row deadline expires. Before teardown it queries SHOW RELAY ... MATERIALIZED STATE on at most three nodes for the first four missing relays. The observations share a ten-second budget, with at most two seconds per query. They keep the placement, branch, round, value byte length and watermarks, without printing the fixture’s large values. Query failure or timeout stays a diagnostic observation; the original exact-row assertion still fails. This observation budget starts after the row deadline and grants no additional time for a missing row to satisfy the assertion.

just test-diagnostic-harness <selection> <filter> runs focused harness regressions in the chosen diagnostic build. just coverage-harness-liveness <filter> measures ordinary harness regressions in a separate target directory and retains its log and lcov.info, including the harness source, in a fresh target/harness-coverage-reports/run.* directory. Both commands reject a selection that completed no tests or left selected tests ignored. These counters measure the harness itself and do not join product coverage or qualify a diagnostic lane. After replaying an instrumented ordinary scenario binary from that target directory without a source edit, just coverage-harness-liveness-report <output> exports the retained profiles without another test invocation.

Public materialized-state inspection can read installed or checkpoint state and can forward an empty non-owner observation to the primary. Record the addressed node and returned placement; rows returned by a replica’s endpoint alone do not prove that replica installed the generation. Correlate the observations with the restore execution report, runtime log and lifecycle snapshot before deciding which publication or delivery owner needs a regression.

A debugger attached to this process suspends every in-process node while it collects thread stacks. Record that suspension separately and retain the attempt as a diagnostic run; its elapsed time and deadline outcomes cannot qualify an uninterrupted throughput comparison. Preserve an unprofiled failure before collecting stacks on a diagnostic replay. Thread stacks can identify workers blocked in a tracked lock or the detector itself, but a suspended async task is not a worker’s call stack. Retain the node’s lifecycle and placement observations alongside the dump when deciding whether the failure belongs to drain, state publication or scheduling contention.

When a step’s timeout expires it sends SIGTERM to the step’s process group. The collector records its attempt as interrupted and forwards the signal to the lane, which ends the process it is running with that process’s whole group, records interrupted in lane.json and exits; the upload steps still run. A failed or interrupted attempt is kept as it is: a later run, or a warm rerun, creates a new attempt and never qualifies the one that failed.

How Failure Reaches CI Output

The job log carries the scenario binary’s standard output and standard error, and the scenario-logs artifact carries tests/logs.

Server processes write their logs into their own temporary directories, and a failure quotes the tail it needs.

Exit Status

124 is the status timeout(1) reports. It differs from 0, from 1, and from the 101 a panic ends with, so a wedged suite is told apart from a passing or a failing one by its exit status alone.

Adding A Harness Operation

The append-setup regression observes each live source’s test-only setup counter before issuing a burst of public domain creations with one voter stopped. It checks that both survivors apply the burst, that attempts to the departed voter fit OpenRaft’s existing 200 ms retry series plus two boundary attempts per source, and that the returning voter catches up. Its count bound uses the whole observation’s elapsed time, so parallel-suite delay does not tighten the assertion. A zero attempt observation fails rather than claiming that pacing was exercised. The counter shares the probe’s existing map with successful opens and is absent from the ordinary production recorder.

A new operation that starts, observes, or stops something on the harness’s behalf follows the same contract.

1. Find its phase. An operation inside a phase, such as a startup attempt, a status wait, or a cleanup, takes that phase’s deadline as a parameter and nests its own budget in it. Only a new phase starts a deadline of its own, and never inside a loop that retries within a phase: a retry that restarts its budget turns a bounded wait into an unbounded one.
2. Derive its budget. Name its policy inputs, usually the slowest healthy duration measured with the whole suite at the CI concurrency factor and the headroom that multiplies it. Derive the budget from them with constant arithmetic, and assert its order against the budgets around it: shorter than the phase that repeats it, and inside the cleanup budget or the reserve when it runs during cleanup. Record the measurement beside the input.
3. Bound it by what it owns. An operation that owns nothing runs within the deadline, or is polled until it, keeping its last output and last failure. A wait for something its owner must keep borrows it and leaves it with that owner when the deadline passes. A task that spawned nothing of its own may be aborted and then joined; a node is never aborted, because the abort would end only its root task. A loop yields on every iteration and pauses only until the deadline.
4. Keep its failure typed. Return a report of a semantic error enum the module owns, name the operation a passed deadline interrupted, and read state from typed variants. Never reduce an outcome to a boolean, infer a state from whether a message exists, or report an unknown reason. Failures carry no credentials and no payload values.
5. Never block the runner task. Every scenario and the suite watchdog share it. A synchronous loop needs a bound of its own, as the port draw has, and blocking work belongs on a blocking or spawned task: a step hands its file, encoding, digest or database work to off_the_runner in tests/common/runner_offload.rs and awaits it, and the runner-lag observer logs a step that still holds the task. A duration a scenario injects for a product deadline to race is timed on a task of its own, as injected holds are, because another scenario can still stall the runner task.
6. Make cancellation release it. Whatever a scenario starts belongs to the scenario’s world or its cluster, so dropping the run releases it: spawned work behind handles that abort on drop, child processes killed on drop, and node tasks through the single task owner, which asks a node that still runs to stop and leaves it stopping under a keeper. A new kind of node publishes its stop to the live-cluster registry through a registration its task owns, and its cleanup belongs to the stopping phase. Ports, faults, proxies, and storage go back only after the tasks that used them have ended.
7. Report it. Transitions a reader needs while a run is still going go to standard error or to the scenario log with the scenario’s identity. A failure fails its step with the typed error’s message, and diagnostics gathered on a failure path stay within the diagnostic budget.
8. Test it. Add a focused regression to tests/harness_liveness.rs that drives a stand-in endpoint or task, on a paused clock where the budget is long, so the bound is proven without production-scale waits, and run just test-harness-liveness. Run the affected features with just test-scenarios --input <feature> at the CI concurrency factor, NERVIX_TEST_CONCURRENCY_FACTOR=2.
9. Keep this chapter current in the same change.

Guarantees And Limits

The harness guarantees:

• Every status request, status wait, readiness wait, node startup, teardown diagnostic, cluster cleanup, and post-run teardown has one absolute deadline, and no retry or poll restarts it.
• No diagnostic, timeout, or cleanup path aborts a node task or leaves one unowned. A node that outlasts a wait stays with its owner or with the keeper it was handed to, and keeps its storage, its ports and its registrations until its task has ended.
• No node is launched over one that is still stopping, so a launch never meets a database lock its own predecessor holds.
• A cluster’s cleanup costs one budget whatever its size, and releases what a node held only after that node’s task has ended.
• finished is published only after cleanup has completed.
• A gate a step holds is released its injected duration after the hold observed it, however long the runner task is stalled.
• The run ends within its budget, the cleanup window, and the bounded teardown, with a status that tells a timeout from a failure.

Its limits:

• Budgets bound what the harness waits for, not what the operating system schedules. A starved runner delays the checks that end a wait.
• The watchdog is polled on the runner task, so it cannot end a synchronous stall of that task. The job’s limit is the guard that remains.
• Provisioning a Kafka topic’s partitions and waiting for a consumer group’s members query the broker synchronously on the runner task, for up to 5 seconds per query, and every scenario of the run waits while one of those queries runs.
• Cleanup and stops give back a stopped node’s consensus fault registration on the runner task, so when no step still holds an observer of the node’s consensus state, the node’s consensus database closes there, and every scenario of the run waits until the close has synchronized the journal and the database’s last background worker has left.
• Kafka committed-offset assertions query on a blocking worker with a one-second request timeout. A query error retries within the assertion’s original deadline; persistent errors fail rather than count as an offset observation. An offset that crosses a forbidden threshold fails immediately, and a below-threshold assertion completes only after a successful observation at the end of its window.
• The watchdog runs no after hooks, so a node held by a fault its scenario injected can outlast the cleanup window and ends with the runtime.
• Nothing in the process can end a node that does not end itself: in-process nodes share one runtime with the harness. A node left stopping that never finishes its shutdown keeps running, with its storage and ports, until the process drops the runtime.
• The time a scenario spends queued for permits and closing the browser have no budget of their own, and some ordinary step waits check their bound only between requests. The suite budget bounds all of them.
• A construction’s convergence checks are separate 40-second waits after its startup deadline.
• The suite budget’s derivation holds with no margin.

Qualification Evidence

just test-harness-liveness runs the focused regressions that hold this contract. They drive stand-in session services and a stand-in NATS server on real loopback sockets, and stand-in node tasks, most of them on a paused clock. The tests job runs them beside the separate scenario job.

The high-parallelism qualification was recorded on 23 September 2026 for the change that landed as cec5f764, at the CI concurrency factor of two scenarios per CPU with Cucumber’s two retries unless a run says otherwise. The local runs used a 24-CPU workstation shared with other builds, whose load average stayed between 7 and 40, so 48 scenarios ran at once; CI used its 16-CPU runner, so 32 did.

After every run, no nervix-server process, scenario binary, or dependency container remained. The harness liveness qualification ledger records each run in full, with its artifacts, the executable evidence for every item of the qualification matrix, and the comparison with the run the job limit cancelled.