Data-Plane Concurrency

Nervix keeps the paths that carry records, batches, remote relay frames, and acknowledgements free from shared coordination that is unrelated to the record being processed. This is the contentionless data-plane rule. It applies after a runtime task has started and its graph, routing, state, metric, and connector handles have been resolved.

Execution Plans describes the revision that installs and publishes those handles before record and batch work begins.

SIMD Kernels defines the pure buffer boundary that folds batch latency and computes admission, arithmetic, and byte masks. Its cached CPU level is process-wide detection under an unmodeled permission; it carries no protocol state. Batch-local reductions preserve this chapter’s ownership and bounded synchronization rules, including one accumulator lock and one wall-clock read per delivery-latency series per batch.

The hot paths are:

• accepting one record from an ingestor
• admitting and fanning out one relay batch
• executing one processor batch for a concrete branch
• receiving and admitting one remote relay frame
• creating, sharing, and resolving one acknowledgement

These paths may apply bounded backpressure and preserve required ordering. Their steady-state work must not acquire a Mutex or RwLock, including a read lock, merely to discover configuration, ownership, topology, a service, or a metric series. A read lock still changes shared lock state, competes with writers, and can join an unrelated reader to a writer’s delay.

A shared concurrent map synchronizes on every access, not only on entry(). DashMap::get, contains_key, len and iteration take a shard’s read side, and get_mut, entry, insert, remove, retain and alter take its write side, whether or not the key is present; a borrowed Ref holds its shard until it is dropped. A steady hot path therefore does not look an established value up in a shared map at all, whether by borrowed lookup or by entry(). It retains the handle it resolved when its task, branch, channel, or attempt was created. A shared registry serves the cold paths around that handle: registration and first installation, where entry() gives racing installers a single winner, replacement, teardown, and observers.

Remote ACK And Admission Owners

just bench-remote-owners measures registration, admission, Alive, ordered parking/resume and terminal resolution with actual ACK roots at one and 64 rows per delivery, reports delivery tail latency and relay-memory reclamation, and measures an empty sweep of the positions its deliveries claimed beside the fixed position count. Its authenticated frame probe sends three current Arrow rows, admits them, returns the exact terminal reply and checks retired attempt counts. A separate 5,000-frame peer-owner probe completes while another peer’s protocol guard is continuously held and checks returned permits and memory. These native probes also run through the existing benchmark smoke producer; local debug timings describe their measured build and host load, rather than a production throughput guarantee.

Remote record correlation uses an immutable array of delivery and admission routing positions. Each delivery generation owns its mutable rows under one short synchronous guard; a row’s outcome is independent of the other rows. Each side claims unused positions on demand and returns retired positions to its own bounded free queue, preserving admission room when delivery positions fill. Shutdown seals first claims before closing the queues and visits only positions that were claimed. The silence sweep, which runs once a second on every node, has the same bound: it reads each side’s first-claim count and takes a guard only for the positions below it, so a node that forwards nothing takes none and a sweep’s cost follows the positions its deliveries have used, not the 16,384 positions a node is built with. The counts bound the scan and order nothing: a position’s guard publishes its correlation, and nothing relies on a sweep having observed a claim. A share whose first claim races a sweep is counted from that sweep or from the first one after its registration, so it still fails between the silence timeout and one sweep interval later. Runtime shutdown cancels the sweep task and waits for it. The task returns to its executor after every sweep, because a sweep that overran its interval finds the next tick already due and that tick does not suspend the task, so it observes the cancellation between sweeps and a sweep’s guard count also bounds how long a stopping node waits for it. A diagnostic build pays for every one of those guards: the acquisition-order selection enters Deloxide’s process-wide detector on each tracked acquisition and release, and the nodes of every scenario in one process share that detector, which is why a recurring pass takes guards in proportion to its live work and never to a fixed capacity. Record allocation and receiver batch watcher tasks reserve relay memory before creating their retained rows. A batch task multiplexes row progress and keepalive polls every 100 milliseconds; each row and the single task have fixed charges. The cadence remains shorter than a one-second source ACK timeout while task cardinality stays bounded by admitted batches. A receiver whose relay budget has no room for a batch’s watches holds the unadmitted batch on its own channel lane and reads the budget again every 10 milliseconds; it registers for no release notification and never queues in the budget’s semaphore, whose released permits would otherwise go to the held charge ahead of the relay work that frees it. The batch task returns each watch’s share as that watch ends, through its exclusively owned reservation, so the hold involves no shared state beyond the budget’s own atomic permit count. The registered wire number selects the position, exact generation and row without a concurrent map lookup. The process identity is checked before routing. Generation exhaustion seals the position, and cancellation, timeout, terminal resolution and shutdown compete to take each row once. The ACK tree resolves after the routing guard is released.

Relay services retain the domain drain tracker resolved during construction. Received rows create ACK roots through that handle; they never register or discover the tracker per row. Watchers read the membership writer’s immutable incarnation publication, so registrar retirement ends their charged task lifetime without a gossip lock.

One authenticated peer transport owns its ordinary grant, attempt, channel, watermark and outbound correlation collections. Connections and operation leases retain that owner. Its guard serializes identity and ordering decisions for this peer; there is no node-wide actor or map guard in the recurring protocol. The peer guard precedes the exact record guard, neither crosses an await, and permit release remains possible while runtime borrows an admitted record. Ordered attempt and grant retirement keeps cancellation effects deterministic. Cold binding and membership changes publish routing through CAS. Cluster Interconnect owns the capacities, grace periods and reconciliation deadlines.

Production-owner Shuttle checks cover delayed events against a reused position, registration against shutdown, admission waiter drop against a reply, final silence sweeps against terminal and Alive reports, a first claim against a sweep, a sweep that completes while an unclaimed position’s guard is held, body claim against cancellation, admission against peer ending, and another peer’s progress while one peer holds its guard. The production admission choice has a Loom model for one irreversible verdict; replacing its compare-and-exchange with an overwriting swap must fail and replay. ArcSwap, queues and external semaphore internals remain outside that atomic claim. Registered bounded Bolero sequences check complete verdicts, capacity and generation outcomes. The remote-correlation-lifetimes sequence uses an independent sequential model to predict registration identities and FIFO reuse before calling the production owner. It varies both pools’ capacity, receiver identities, attached and detached batch rows, admission receiver closure, single and bulk admission, Alive and sequenced parking/resume, terminal replies, clear, relay-memory refusal and shutdown. Every operation and every individual silence sweep checks exact root outcomes and progress, drain and handoff counts, admission updates, per-receiver expiry counts, retained memory and the positions ever first-claimed. Stale generations and non-record admission row numbers are also checked through the public routing operations. This is sequential representation evidence; the registered Shuttle checks above own races, and the runtime scenarios own the timed sweep loop and its public delivery behavior. Turmoil covers transport loss, retry, cancellation, reconnect and process restart within its documented scope; public one/three-node branch delivery and producer-restart scenarios cover host ACKs. External immutable-image faults remain separate evidence.

Deloxide applies to the new delivery, peer and record blocking guards and their acquisition order. Both diagnostic selections run the remote ACK scenarios alongside their existing workloads. It does not model publication, async channels, network waits or the atomic verdict; a clean run proves only the tracked acquisitions the diagnostic processes executed. Generated measurements and fault evidence belong on the task, and a failed required attempt remains recorded.

Leading empty consensus append probes use management capacity. One task owns their replies and the transition to an ordered data stream; subsequent empty commit updates stay on that stream. Rejected probes end the input before data admission. This transition adds no shared map, lock or memory-ordering claim. Native real-transport and public catch-up checks cover it; Deloxide does not track these async network waits.

Consensus append setup consumes its caller’s first request before opening the transport. The caller owns the async retry and cancellation gate; the adapter retains and submits that request once, without adding a lock or retry timer. Native admission and ended-input regressions exercise this boundary, and the public three-node scenario checks paced failed setup while the surviving quorum commits and the returning voter catches up. The test probe counts attempts and successful opens in its existing blocking map. Deloxide does not track the request-stream or network waits; the applicable Turmoil suite and real-process Chaos faults retain their separate claims.

Relay, Transport And Source Lifetimes

A concrete producer retains its branch channel handle. The relay publishes branch membership in a persistent immutable table; concurrent first binding selects one allocation with CAS. Each branch publishes an owner/consumer routing generation, whose cancellation parent fences ingress and outbound slots, including candidates created during replacement. Eviction cancels the exact branch before withdrawing its allocation. An independently expiring receiver branch permits a still-live producer to bind a fresh channel after invalidation. The producer’s own ending moves it to Ended and prevents rebinding. Domain terminal teardown cancels the relay publication itself.

Channel gates serialize encode, sequence assignment and dispatch for that channel only. Gate waits select cancellation, and transport admission retains its existing physical deadline. They do not serialize unrelated branches or peers. A subscription snapshot selects only live advertised peers, fenced by incarnation and interest version. Equal advertisements retain channels; changed selection cancels the preceding subscription generation. Branch and destination registration copy only a persistent changed-key path. Tables are never rebuilt for an established batch.

Transport targets publish immutable fixed pool arrays. A slot atomically claims one reconnect worker for its lifetime and publishes its authenticated connection through ArcSwapOption. Leasing reads these publications, scans the fixed class slots and retains the selected connection. The connection registry serves registration, exact retirement and observers. Endpoint, peer or TLS replacement cancels slots before publishing replacement handles; a changed DNS answer affects only the next dial. Cluster Interconnect owns quotas, deadlines and delivery rules.

A fresh intake selects its domain publisher from an immutable persistent table. Executing tasks retain the selected publisher and observe complete snapshots. Domain removal withdraws its table entry; terminal shutdown clears the table after ending executions. Buffered error routes similarly send through the worker retained by the bound plan, with no failure-path discovery acquisition.

Source polling writes a retained instance readiness scalar. Starting and ready may alternate; retired is final, so a late poll or drop cannot change a replacement instance’s readiness. These scalar transitions and worker claims use relaxed atomic operations and publish no other data. Shuttle checks production binding, refresh, retirement and polling races. ArcSwap internals remain opaque; this delivery adds no cross-location memory-ordering claim requiring a new Loom weakening. Existing memory-ordering claims remain with their owning models. Deloxide tracks the existing sequence and route-task blocking locks; it does not track ArcSwap, scalar atomics or async channel ordering gates.

just bench-retained-channels measures published versus retained branch selection, established stream leasing and bounded branch churn. Its ordinary native bodies also run in bench-smoke, whose shared native coverage producer prepares browser, guest and runtime dependencies before instrumentation. The measured bodies retain the usual allocation assertions; their timing output is workload evidence, not a portable latency threshold.

The resulting design uses four forms of ownership:

• Reconfigurable read-mostly state is built privately and published by atomically replacing one immutable reference.
• A task resolves stable services, state authorities, slots, and metric series when it starts and retains those handles.
• Scalar progress and counts use atomics with an ordering chosen for the contract they enforce.
• State that changes for every row or batch belongs exclusively to one task, branch, delivery channel, or source attempt. That owner mutates ordinary collections directly: a task-local HashMap, BTreeMap, IndexMap or JSON map and its entry API are not synchronization, and exclusive ownership does not pin the task to an operating-system thread. When other tasks need to observe such state, the owner publishes an immutable value of it; observers never reach into the owner’s collection.

A shared concurrent map is justified only for state that several owners register into or that observers read across owners, and only while its accesses stay on those cold paths. Hot-path synchronization that remains is an ordering fence or bounded protocol this chapter names, with the exact key that scopes it and the capacity or deadline that bounds its wait. The concurrent map inventory records every concurrent map on the data plane and its neighbours, how often each is reached, and the disposition of every map that is still reached on a record, batch, remote-frame, acknowledgement or steady-poll path. Until its named repair lands, such a site is debt, not accepted design.

The Data Plane chapter defines payload, persistence, branch, and ACK semantics. This chapter owns the concurrency contract for executing them. The Cluster Interconnect, Domain Clock, and Shutdown And Recovery chapters remain authoritative for their respective distributed and lifecycle protocols.

Publication Boundaries

An atomic publication is a complete immutable value. A writer derives the next value without changing the published one, then replaces the reference once. A reader keeps the reference it loaded for as long as it needs it and therefore observes either the complete preceding value or the complete replacement. It never observes fields being changed in place.

Publication is a node-local visibility mechanism, not a durability boundary. The control plane, runtime-state store, or gossip protocol still owns the source of truth. Runtime publication makes one already-decided revision available to the data plane without putting its readers behind the writer.

| Restore state publication | The control plane stages checkpoints under a replicated authority, then admits one fixed-size storage job, validates a complete generation namespace and durably publishes its active pointer. It holds the consensus applied-state read guard through authority validation, synchronous storage mutation and runtime-handle clearing. The state store reuses its existing latest-snapshot installation mutex for staging, pointer publication and bounded cleanup. Checkpoint jobs retain their selected namespace and validate it under that barrier; entity replacement and purge also select the active namespace while holding the barrier. | A newer applied generation and the release of the start gate require the state-machine write guard. Every node therefore replaces a whole state set before START is available, and stale coordinators cannot mutate a running restored domain. The store retains authority and inventory for exact retry and monotonic rejection. Database readers select pointer, header and chunk values from one snapshot, which retains its complete set across bounded deletion of obsolete namespaces and unreferenced active chunk sets. Publication runs on stopped-domain lifecycle paths. The admitted maintenance loop also follows running checkpoint replacement and purge under the installation barrier; record delivery adds no lock. |

Restore storage qualification includes the store’s cancellation, corruption, snapshot, queued writer, active chunk liveness and many-key cleanup regressions in just test-deloxide, just test-deloxide-order and just test-deloxide-stress, plus the public one-node and three-node @restore_installation workloads, which just test-deloxide-stress-restore also runs under the bounded scheduling disturbance, for large saves, many checkpoints, native lifecycle and Kafka records above the default 32 MiB bulk budget, active chunk replacement and purge, durable publication failure, restart and delayed coordinators. The many-checkpoint three-node case includes one replica per state; its one-MiB saves fit the replication operation’s limit. Deloxide tracks the installation barrier and other boundary blocking locks; Fjall’s internal locks, async authority ordering and memory ordering keep their separate ordinary, Shuttle, Loom and Turmoil checks. Measurements and diagnostic artifacts are delivered to the owning task. The registered large-WASM-handoff diagnostic scenario reaches replica installation and planned handoff with two branch saves above that message limit. It exercises the state store’s tracked installation locks in both Deloxide selections; the bulk stream’s async waits and transport faults remain outside the detector and have their own Turmoil and Cucumber evidence.

Materialized capture shares row carriers under the existing assignment barrier, together with the revision, ownership fence and branch generation. Membership insertion and eviction use that same barrier; ordinary updates of existing branches keep their existing admission protocol. Fresh backup capture admits its bounded row-view metadata before allocation. Archive conversion stages one identity/Arrow group at a time. Native loading independently bounds typed-key and row-view metadata to 8 MiB, charges retained decoded columns to relay memory through installation, and uses bulk memory for one section’s decoding scratch. A cumulative relay admission refusal ends loading instead of waiting while retaining earlier groups. A pinned database reader retains one snapshot and one stored chunk through all section reads. Replica installation checks monotonic revision under the existing exclusive assignment barrier before replacing any entry.

RESUME changes the domain to running at its archived lifecycle in the consensus effect that records complete publication and releases the installation gate. No atomic field or memory ordering in the capture epoch, assignment protocol or relay revision changed. Existing Loom models continue to qualify their ordering; new Shuttle checks qualify branch membership capture, replica revision races and publication before activation. Paused generators release their admitted-work guard after flushing and reacquire it before producing more output.

Native restore conversion retains the immutable verified-description owner and its preparation reservation while an admitted storage job streams the current checkpoint encoding into a quota-owned file. The job owns the disk permit until its file closes or becomes a completed artifact, including cancellation. Entry serialization and 64 KiB writes check cancellation between bounded units. Each bounded artifact read also retains its file and quota in the storage job until it exits. Local and remote installation retain their exact file source and use the existing authority and installation barriers; conversion adds no lock to record paths.

Stored branch capture also retains one immutable database selection, opens only the scheduled checkpoint, and streams it in cancellable 64 KiB units into a quota-owned file. Its admitted read-only mapping and size index remain owned through each bounded conversion job; cancellation does not release their memory or disk permits before that job exits. Keyspace and window codecs convert only the selected archive group. This adds no lock or publication protocol. Existing capture and restore lifecycle locks remain tracked by Deloxide: the stored-branch scenario runs in active and order selections and in the restore stress selection. Deloxide does not observe the mapping, CPU jobs or async admission; ordinary cancellation/refusal owner tests and complete Bolero byte comparisons cover those boundaries. The capture fence’s Shuttle and Loom models retain ownership of its unchanged publication and ordering invariants.

The backup coordinator waits for each node’s admitted-work counters to reach zero, then orders one confirming force-flush generation across the cluster. The generation’s obligations use the same atomic counters as shutdown; a parked REQUIRED WAIT batch stays visible separately and cannot hold the cut open. Rows a window retains park their shares too, and the cut captures them as window state. | Metric series handles | Each relay, node, ingestor, emitter, or concrete branch resolves its label set, internal series, and Prometheus child when its owning task or branch is created. | Recording uses the retained series directly. Counters update atomically; a histogram records through its already-resolved per-series accumulator without a registry lookup, key construction, or map guard. A node input folds a whole batch’s delivery latencies in one kernel pass over the batch’s high-watermark column before recording, so each latency series locks its accumulator and reads the wall clock once per batch rather than once per row, and its Prometheus child takes the batch in one flush. Whether a series was ever observed is an atomic flag set under that lock and read without it. Registration and removal stay on lifecycle paths. | | MQTT broker packet limit | Each MQTT sink’s event-loop task stores the Maximum Packet Size of every CONNACK its client receives, or the broker’s silence about one, as one atomic scalar. A reconnect replaces it. | A publish loads the scalar once per record to reject a packet the broker would refuse before handing it to the client. The value stands alone, so relaxed ordering is its whole contract, and a record published before the first CONNACK is measured only against the protocol’s largest packet. |

An atomic replacement gives consistency for the value it publishes. A transition involving several owners still needs a protocol. Graph and schedule changes use quiescence and relay gates; clock and state operations carry generations; subscription creation uses a visibility handshake. Publication removes read-side contention without weakening those transition contracts.

The control plane classifies each committed schedule change and builds a complete typed execution revision before handing it to the runtime. The revision keeps placement, state identity, handoff fingerprint, and all node plans together. Runtime application reads that revision for incremental swaps, relocation, remote consumer edges, and full or passive rebuilds. A failed application keeps the last successfully applied schedule as the predecessor for its retry; a stale revision cannot replace that predecessor. Each planned domain also carries that predecessor’s source identity. If an earlier attempt left a domain partly or wholly installed at another revision, the runtime rebuilds it instead of applying a delta against the wrong predecessor.

Schedule application prepares every changed local processor plan before publication. A preparation failure leaves the preceding routing revision observable. Publication swaps the complete typed routing snapshot with its installed execution revision. Unchanged processor specifications, schema fingerprints and resolved branch contracts keep their exact plan allocation, including prepared programs and compiled WASM modules.

Retained Task Dependencies

A source or sink retains one failure publication containing both its safe error and optional retry. Success reads that slot and clears it only when it holds a failure. Identical failure reports keep the same immutable observation. Reporting an error without a new retry keeps the active retry, which remains visible as publishing work to drains even when the emitter buffer is empty. DESCRIBE loads error, backoff and remaining wait from one observation. Emitter confirmation guards retain one counter resolved at startup, and task teardown removes its registry interest.

Every pooled Redis or SQL sink registers a wait slot once. The connector serializes its connection borrows. A ready borrow writes nothing; the first Pending poll publishes its wait and owns a guard that clears it on success, error or cancellation. Sink destruction removes only its own registry slot, so a replacement sink’s slot survives an earlier sink’s teardown.

Generators retain the domain acknowledgement tracker. Ingest executions, groups and endpoint bindings retain the same domain/ingestor tracker pair. Task and branch metric handles retain the exact placement’s dirty mark. Client batch outcome counters, quiesced payload counters/gauges and subscription drop counters retain their Prometheus children, including every declared outcome label combination, so recording resolves no label set.

Subscription generations also retain the node’s executor and pass it directly to predicate evaluation. Executor admission continues to own worker and memory bounds for extension calls. The admission benchmark also constructs the current emitter context, resolving its routing, metrics mark, status publication and confirmation counter before the timed encoding runs.

Each handoff entity has one immutable coordination-owner set and notification owner. Tasks retain that entity slot across repeated freezes. A watch registers before reading the publication; engagement and release replace it before notifying. Concurrent owners derive their replacement with the primitive publication boundary’s reference update. Domain removal drops registry interest.

Domain-clock publication contains lifecycle pause, generation and start-point observations alongside its installation. Ingestion reads admission and time from that one publication; Kafka offset polls read its generation; generators, processors, materialized relay tasks and subscriptions use the clock or lifecycle handle bound at startup. Each WASM state retains its entity assignment slot, which publishes state identity and checkpoint execution/replica owners together. A generation replacement or removal therefore fences a callback without a runtime registry lookup.

Clock authority reconciliation binds a producer to its locally installed mapping, generation and authority fence before starting it. Each node’s domain clock owner installs a committed authority-only reassignment ahead of graph installation. Each installed domain records the committed revision of its lifecycle, which keeps an older graph installation from restoring a superseded fence. A graph installation held on a departed node therefore stalls neither the replacement authority nor a receiver. Each remote delivery loop starts after its receiver advertises installation of that revision’s domain clocks. An obsolete producer still retires before its successor starts, and a leader proposes an authority that did not commit again on a fixed interval instead of waiting for the next state change. The takeover Cucumber scenarios in domain_clock_contract.feature exercise this ownership path in the active, order and lifecycle stress Deloxide selections; their async installation, watch and transport waits remain outside tracked-lock detection.

Force-flush participants retain a private idle, available or closed scalar. Idle and already-claimed polls acquire no coordinator mutex. An available indication enters the mutex, which remains the sole authority for generations and claims. The coordinator visits participants in participant-ID order when publishing readiness, so deterministic replay sees the same sequence of scheduling points. Publication of a request precedes the watch wakeup; claim release makes the obligation available again. The relaxed hint publishes no data from another location. Watch synchronization and ArcSwap ordering belong to opaque primitive dependencies; this change introduces no Nervix cross-location memory-ordering claim. Production Shuttle checks explore generation publication, claims, release and close, and the idle regression counts actual coordinator acquisitions.

The status and freeze transition contracts have registered Bolero sequence properties. Status observers and freeze registration/release have production-owner Shuttle checks. just bench-task-handles records raw same-host timing and allocated-byte samples for the repaired paths. The retained task handle measurements record the measured operations, host, samples and limits. The concurrent map inventory records their lifecycle registration, observation and teardown sites.

Server endpoint intake

EndpointIntakeRoutes owns one immutable host-and-path table containing each domain’s configured endpoint definitions and the source lifetimes bound to them. Domain install or teardown and source bind or unbind derive a replacement from the current publication with ArcSwap::rcu. Concurrent writers retry against the latest complete table, preserving unrelated domains and sources. A passive installation publishes no endpoint definitions. The listening socket remains present on every node independently of this table.

An HTTP request resolves one route using borrowed host and path keys; only uppercase hosts need a normalization allocation. A WebSocket retains the resolved route and its signaling protocol from upgrade through all later frames, including signaling data frames. Readers share the route allocation and borrow its prepared intakes; they do not clone route vectors, programs, codecs, or sender maps and do not return to a concurrent registry for each frame.

Each source lifetime publishes its optional intake through ArcSwapOption. Close, drop, domain replacement, domain teardown, and runtime clear publish absence before withdrawing that lifetime. Retained routes therefore refuse later admission even after a replacement source starts. A request that already loaded a present intake keeps its lease until dispatch finishes. Unbind removes the exact source allocation, so a late close cannot remove a replacement with the same typed identity. Configuration withdrawal removes the route; source withdrawal alone leaves configured metadata available to report unavailable intake. These publications remain volatile data-plane state.

Unit regressions and the registered Bolero operation sequence exercise the production owner and retained lifetimes. Shuttle races whole-domain publication with resolution, unbind with admitted requests and replacement, and domain teardown with retained routes. These checks treat publication operations as opaque scheduling points. Both publications reuse the primitive boundary’s existing implementation: no new memory-ordering protocol is introduced, and Loom does not model their internals. Listener ownership and network primitives retain their existing contracts; Turmoil is outside this owner’s scope. Public scenarios cover one and three nodes; the endpoint Chaos workload belongs to Typed Ratchet 10. The inventory ledger records the routing measurement and verification evidence.

Mutable Execution State

Per-row mutation is kept close to the lane that orders it. A global concurrent map is an index into those lanes, not the owner of their inner state. Once a task has found its lane, later mutation does not repeatedly acquire the registry guard.

Ownership handoff drain before freeze

The runtime closes intake and the affected boundary gates, then requests a force flush and confirms the admission hold. The coordinator can observe pending work and its timeout diagnostics while the installed ingestor routes and processors finish that flush. Each participant’s exact operation drain-status query publishes its ownership freeze only when the local affected scope is drained. Capture requires both the held operation and its own coordination identity’s freeze. The status query retains the operation’s registry guard through publication, so release cannot withdraw its hold between observing drain and publishing capture readiness.

This is lifecycle work: the polls read existing quiesce and acknowledgement registries, and hot paths continue to retain their counters and freeze publications. The existing force-flush and quiesce checks cover completion visibility; freeze checks cover publication and wakeup order. The ownership drain reads force-flush obligations before admitted-work counts. Acquiring a completed flush therefore makes a parked message that the flush resumed visible to that drain. loom_an_ownership_handoff_drain_sees_work_resumed_before_flush_completion exercises the production counter method; its qualification weakens that method’s existing acquire. The handoff publication check races the production freeze decision against the actual participant’s flush completion. The public shutdown regression holds an admitted Kafka route until handoff publishes its flush request, then verifies completed drain and exactly one acknowledged output. The diagnostic lane registers that path in all three selections; async wakeups, transport and deadlines remain outside tracked blocking-lock detection.

Emitter payload assembly

One emitter task owns the released carriers and their flush order. Its batch packer borrows each carrier’s source relay, concrete branch key and ordered metadata while it prepares selected Arrow rows. It clones the Arc-backed Arrow batch once per carrier passed to the packer; no row acquires a lock or increments an Arc reference count for source identity. The open candidate holds at most BATCH MAX MESSAGES prepared members and is sealed on metadata or source changes, a failed member, or the message-count limit. Each encoding attempt uses the codec’s bounded writer under MAX SIZE. The emitter retains original batch and row positions for acknowledgements and errors.

A payload offered to the sink stays in the same task’s buffer until the sink answers for it. The task marks each member row prepared, and the one answer for the payload resolves every member: a row resolves once, so neither a repeated answer nor a later attempt resolves its acknowledgement share again. An HTTP emitter’s prepared request is such a payload with exactly one member, retained by the same owner with its request fields and body bytes, so its answers follow the same rules, and so is an OTEL emitter’s Export request, retained with the protobuf bytes its connector prepared and the rows they carry. No lock guards this state; only the emitter task touches its buffer, and resolving a member is one operation on the lock-free ACK tree. A confirmed payload’s members are acknowledged in the same step that releases the payload, while a rejected payload’s members return to pending until their message errors are delivered, so an attempt its stop deadline cuts short leaves every member either resolved or still owned by the buffer.

Terminal teardown races the whole emitter task against its domain shutdown signal. It can cancel an in-flight connector await after the graph drain budget expires, dropping the task’s volatile payloads and ACK guards together. That cancellation cannot resolve a member as delivered; an acknowledged source retains its own redelivery boundary. Entity-pause swaps do not use this terminal cancellation: the old task must complete its deadline-bounded stop before the new task starts. The Shuttle check over the emitter’s prepared payload and ACK owner races a connector await with the terminal shutdown signal and asserts that dropping an unanswered request leaves its attached source unacknowledged.

Branch processor state

Each concrete branch has one dispatch lane and one mutable processor runtime. The lane admits one batch for that branch and queues later batches, which preserves processor order. Different branches have different runtimes and execute independently. Expiry and eviction first detach a branch from the instance registry and then take ownership of its runtime, so registry access is not nested over branch execution.

Deduplicator keys

A deduplicator keyspace belongs to the task for one concrete branch. That task prunes, reserves, and releases keys directly. When the keyspace changes, the task marks it dirty and periodically publishes an immutable generation whose key allocations are shared with the live keyspace.

Snapshot persistence, replication, ownership handoff, and a replacement branch task read the last published generation. They never read or lock the live keyspace. Publication occurs during the replication cadence, before a handoff checkpoint, and when the branch stops, so snapshot encoding can proceed while the current task continues processing rows.

Window and WASM state

A window and its aggregate accumulators belong to one branch task. The task mutates the live window directly and publishes a complete immutable window generation for persistence, replication, handoff, and restoration. Snapshot encoding reads that generation instead of holding the branch runtime while it serializes. The publication shares the retained input and aggregate-argument Arrow columns through row views. The snapshot task seals those views as bounded Arrow sections on the bulk executor, while a separate bounded typed section carries each group of histogram delayed removals. Encoding never materializes a scalar-field copy of the retained payload. A window whose rows exceed one snapshot section is sealed group by group into quota-owned pieces and published as a segmented checkpoint under the state store’s installation barrier, as a materialized relay’s periodic checkpoint is; the snapshot task holds no lock while it seals, and the barrier is held only by the storage job that writes the segments and the header. Restore opens the sections on the bulk executor and reuses their columns to rebuild exact accumulators and sketch panes.

Evicting a concrete branch resets its retained window rows and aggregate structures before the branch task’s final publication. The published generation is empty, so a later appearance of the same branch key cannot inherit the evicted window or its sketch panes. After the final checkpoint, the owner releases the evicted branch’s in-memory publication. A branch that appears without a restored lifecycle entry also publishes an empty initial window, even if a previous lifetime of its key left a checkpoint behind. A processor task therefore installs every branch its lifecycle names before it dequeues any input, and a restore the bulk executor refuses installs none of them and is attempted again; see Restoring Processor Branches. Stopping a branch for an ownership handoff follows the normal finalization path and publishes its retained window instead. The branch lifecycle and window checkpoint carry the same incarnation, assigned when the concrete branch appears. A restore whose incarnations differ begins with empty window state and marks it for publication, so a delayed checkpoint from the preceding lifetime cannot restore its panes.

A WASM guest instance likewise belongs to one branch task. At the end of every guest callback the task asks the guest to save its computation state and checkpoints the returned buffer: it writes it to stable storage, waits for the replicas the checkpoint names, and only then publishes it as the committed checkpoint a recreated instance restores. The branch runs no further callback until that checkpoint completes or fails. The publications retain the saved buffer rather than copying it for every persistence or replication reader. Input buffered by the guest host and ACK tokens remain execution state and are never included in a guest save. See WASM State And Recovery for the checkpoint’s stages.

Each branch publishes checkpoint revision, boundary, and stage together through one immutable observation. An inspection read samples that publication and replica progress without taking the branch task’s execution lane or advancing durability. A failure before guest state was captured is an explicit observation without a new revision; any earlier committed checkpoint remains the restore source. The published observation contains no guest bytes.

Every branch save is addressed by the guest-state generation in the committed schedule. Forced recovery publishes a new generation with the replacement schedule, so a late save, replica installation, or recovered checkpoint from the generation it replaced cannot address current state. The state store performs durable guest-state writes on its storage workers; the async worker that owns the branch never performs that storage operation synchronously.

A coordinated reset enters the selected branch task through its existing supervisor command lane. That lane is the serialization point with input callbacks, timeout callbacks, branch eviction, and task replacement. Preparation takes ownership of the selected task, reaches its stop boundary, and keeps the handoff only until either pre-publication abort restores it or generation publication makes it obsolete. Sibling branch tasks keep their own lanes and continue running. After publication, the supervisor creates fresh tasks and waits for their initial checkpoints before it drops the retained old tasks and accepts completion.

Checkpoint replication

Every replicated runtime state owns its replication: the domain offsets of a Kafka ingestor, the state of a deduplicator, window or WASM processor branch, a materialized relay, an entity’s branch-aggregated metrics, and the branch lifecycle of a branch-keyed entity. On the node that owns a placement, the replication records the highest revision each replica reported holding on its stable storage, and offers the placement’s newest checkpoint to the replicas that do not hold it yet. On a replica, it carries the owner’s announcements to the task that keeps the replica’s copy current. It lives and ends with the state it replicates, so a WASM checkpoint or a Kafka offset commit offers its revision through the state it already holds, and no checkpoint, commit, offer enters a node-wide announcement or progress map. State installation publishes a resolved route containing the actual state handle and the entity’s existing assignment slot. Frames select that route from one immutable publication, never from the state, execution or identity registries. An announcer retains its selected route for its whole lifetime and reads the current primary and replica set from the assignment slot.

Branch-aggregate byte capture keeps the narrow lifecycle_call expectation for scanning the metrics inventory at an explicit snapshot boundary. Each placement retains one encoded snapshot under its own blocking mutex. Capture reads the dirty revision first and chooses that revision’s bytes once; subsequent descriptors, streams and persistence reuse them, including elapsed metric fields. Initial, replica and persisted recovery install the received bytes with their revision. The guard serializes capture and installation, covers only that placement’s snapshot work and never crosses an await. Metric recording advances the dirty revision without taking this guard; the next capture includes those measurements. The guard belongs to cold checkpoint work, not to record or batch execution. The two interleaved-tenant window drain and relocation scenarios run in both diagnostic selections and the stressed restore selection. State-route selection uses the installed state handle and retained assignment slot. Checkpoint synchronization sends a bounded revision, length and digest description through that selected route. The replica fetches the exact revision through the bulk pool, admits the whole allocation and verifies every byte before installation. A handoff destination follows the same sequence from the source’s captured checkpoint while its entity gate remains held. Neither path adds a shared hot-path lock or a new wait-and-notify protocol; Deloxide sees the existing state-store installation lock but cannot see stream scheduling or async memory admission.

A replica’s progress only rises. Acknowledgements travel independently, and a replica acknowledges what it holds again whenever it is offered a checkpoint, so an acknowledgement of an older revision can arrive after a newer one; it leaves the newer one in place. Each wait for replicas registers for the next report before it reads the progress: a WASM checkpoint waiting for the replicas its boundary names, a Kafka offset commit waiting for its replica quorum, and a WASM state reset waiting for the branch lifecycle that authorizes its first checkpoint. The reset wait retains its route, follows the current replica assignment and rejects loss of identity or primary ownership. A checkpoint promised to replicas never confirms with fewer replicas, using the same typed boundary as a guest checkpoint. A 100-millisecond recheck observes assignment changes even without a replica report. A report that lands between the read and the wait therefore wakes the wait instead of leaving it to its deadline.

One announcer at a time offers a placement’s revision. An offer raises the revision the running announcer offers or, with none running, starts one. The announcer sends the revision to the replicas that lag behind it, again every 100 milliseconds, and ends in the same step that finds every replica the committed schedule assigns holding it, so a revision offered while it ends is either sent by that step or starts the next announcer. It also ends when this node stops being the placement’s primary, when the replicated state goes away, and when the runtime stops. An announcer whose task ends before it finished hands its announcement back to the next offer.

The runtime’s checkpoint announcement task owner cancels both dispatch and retry waits when it closes after domain drain. It joins every cancelled task before withdrawing replication routes, so shutdown releases the task’s exact retained state without waiting for a remote dispatch timeout.

The snapshot task of a deduplicator or window branch has the runtime as its owner too, because it outlives its branch task: once that task is gone, stopped or ended at its stop grace, the snapshot task still writes what the branch published last. A stopping runtime closes this owner after domain drain, which tells every task still waiting for its branch to make that write, and joins every one of them before it closes the announcement owner. Nothing is cancelled: the join is what keeps the node’s storage from being released under a write, so it lasts as long as the slowest of those writes. The owner is a task tracker and a cancellation token, with no lock.

The announcement, the progress and the announcer’s handover change together under a short lock that belongs to the one placement and is never held across an await. The placement’s originator, its one announcer and its replicas’ acknowledgements are the only participants; unrelated placements never share it. An acknowledgement or an announcement names its placement, and the node that receives it uses the handle resolved when that state was installed. The assignment and the placement must still name the same schema and guest-state generation. Cold replacement or teardown ends the route before withdrawing it, so a retained route never attaches to its successor, including when the successor has the same placement key. A frame already admitted through a borrowed state may finish on that exact state. On a replica, announcements of a lifecycle or one of its branches use the published entity lifecycle, which the entity’s replica task retains. An announcement of an entity a node holds no state for wakes nothing and creates no state.

A replica installs a branch checkpoint only while the branch lifecycle it holds names the branch. Each lifecycle checkpoint a node holds is decoded once into the set of branches it names, so installing a branch checkpoint looks its branch up in that set, through the lifecycle handle the replica task retains, instead of copying and decoding the lifecycle each time. Holding a replica’s copy of a branch checkpoint compares revisions and moves the newer payload into an immutable publication owned by the entity lifecycle. Lifecycle pruning visits only that entity’s passive copies without acquiring a registry shard. Catch-up steps and installation read the assignment slot the lifecycle retains. Cold assignment cleanup also discards superseded passive generations, while an already borrowed checkpoint can finish. Cold promotion consumes the passive copy; non-branch handoff snapshots remain in a cold recovery registry.

Replica catch-up

A replica keeps each branch-keyed entity it replicates current through one replica task: the entity’s branch lifecycle and, for a deduplicator, window or WASM processor, the state of every branch. The task retains the entity’s lifecycle handle and alone owns what it learned: the owner’s catalog as far as it read it, the revision this node holds of each branch it looked at, and the branches it still has to fetch or acknowledge. Nothing else reads or changes that record, so it takes no lock.

The owner keeps, with each entity’s lifecycle, a catalog of the newest replicable revision of every branch state it owns. A branch state registers when it is created, records every revision it offers its replicas, each generation a deduplicator or window branch publishes and each WASM checkpoint on the owner’s stable storage, and leaves the catalog when it goes away. A branch state that another one of the same branch replaced neither changes nor removes its successor’s entry. The catalog is one immutable value that each change replaces by a read-copy-update, so recording a revision never waits for a reader and a reader never waits for a branch. Every change is numbered, and a replica asks for the changes after the cursor its previous read returned. A cursor of another catalog, a cursor older than the oldest removal the catalog kept, or no cursor at all restarts the listing from the catalog’s beginning. A listing is paged, 256 changes at a time.

A round runs when the task starts, when the owner announces a checkpoint, and once every replication poll interval. It synchronizes the lifecycle, reads the catalog’s changes, and looks only at the branches that changed, that the lifecycle names for the first time, that the owner announced, or whose earlier step failed. The task reads what this node holds of a branch once, from its own copy, and keeps that current itself. A round in which no branch changed therefore sends two requests to the owner, reaches no shared map for any branch on either node, and does no work per branch, however many branches the entity has. The branches that changed are fetched and installed at most 16 at a time.

The owner’s announcements of the entity’s lifecycle and branch checkpoints are left with the entity’s lifecycle handle on the replica, the newest one of each, under a short lock that belongs to that one entity and is never held across an await, and each wakes the task. The task takes them all at the start of its next round, so an announcement that lands while a round runs is taken by the next one, and one of a revision the replica already holds is acknowledged again. An announcement that is lost delays a checkpoint by at most one poll interval.

Relay branch presence

A relay’s owner task owns its concrete branch instances: their activity order, the TTL scan that expires them, and the LRU eviction that bounds them. It keeps them in an ordinary indexed map beside a persistent set of their keys, and admits, evicts and expires through one owner type that changes both in the same step. A step that changes the membership publishes the set, sharing its structure with the membership published before it, so creating or releasing one branch copies only the path to that branch, never every branch the owner holds. A batch that creates one branch and evicts another publishes once, so an observer never sees the relay above its capacity. A batch for an established branch refreshes that branch’s activity in the owner’s map and publishes nothing; it takes no lock and writes no shared timestamp.

Each owner lifetime claims the presence when its task is spawned. The claim publishes an empty membership that replaces whatever an earlier owner left behind, including the branches of an owner aborted before its teardown ran, and from then on only the claiming owner changes what the presence publishes: a replaced owner that is still ending compares its lifetime with the published one and publishes nothing. Dropping the owner state, whether the task finishes its drain or is aborted, releases the presence by publishing an empty membership unless a successor has claimed it. Publication and claims are one ArcSwap compare-and-swap or read-copy-update each; the protocol adds no atomic of its own.

The owner publishes a new branch before it fans out the batch that created it and an eviction before it invalidates the evicted branch’s delivery slots, so a consumer that has processed a batch reads a membership that holds its branch.

Materialized relay entries

One assigned materialized-relay originator owns updates. The state contains at most one latest record for each concrete branch. Its origination capability moves into the branch runtime or the scheduled relay-state task and cannot be cloned. That task owns an ordinary map of mutable branch records. An established branch retains its row publication: timestamp comparison uses the owner’s record, then publishes one immutable Arrow-backed row view. It acquires no shared-map shard and does not republish or scan branch membership. A larger high watermark wins, then a larger low watermark; equal watermarks retain the first record and advance no revision.

Replacing an existing branch’s record is admitted under the originator’s assignment generation. Adding the first record for a branch or evicting a branch also changes the branch lifecycle, so that narrower operation uses the assignment barrier and advances a branch generation. The persistent membership publication changes only at those boundaries or a snapshot installation. Eviction first clears the exact row publication, then withdraws that member. Recreation allocates a new publication; a retained ended member never attaches to it. Rebinding waits for operations admitted under the preceding assignment before rebuilding the new task’s mutable selections from the published views. Previous writer tokens refuse further mutations.

Domain routing retains one materialized publication per relay and its relay-state epoch. The installed state-replication routes supply the relay’s immutable branch index. Dependency reads and generator ticks load that retained relay index and the installed row views, rather than looking up or iterating node-wide materialized-state, presence or epoch registries. Presence comes from the retained relay services; placement identity comes from the shared immutable assignment publication. Live state owns absence: an empty concrete branch does not fall back to the relay snapshot, and a missing live row does not fall back to a pre-eviction stored snapshot. Cold installation and public observation retain their lifecycle registries.

A snapshot capture holds the assignment boundary only long enough to take immutable row views, their revision, their ownership fence, and their branch generation. The Arrow columns stay shared, and sealing and encoding happen after the boundary is released. A snapshot from an earlier assignment or branch generation cannot restore state that a newer owner or eviction superseded. Installation also refuses a lower revision within the same fence and branch generation. A cached sealed snapshot answers a current read only when its revision and assignment fence still match. One sealed generation is cached and one build is admitted per placement; outstanding readers and captures keep only their selected row views and shared Arrow carriers. Publication never copies a row payload or a whole relay for an established update. Snapshot archives and storage ownership retain the current container and backup contract.

A stopped backup selects materialized checkpoint readers alongside its other state from one immutable database view. Those readers keep the selected namespace, metadata and chunks through bounded group conversion, even if a later publication replaces or reclaims the generation. They retain one stored chunk and one charged group at a time and publish no runtime state. Running and paused backup capture continues to borrow current row views under the assignment barrier.

The required-wait observation creates its notification before reading dependencies. The primitive boundary registers a notify_waiters observer at future creation, so an update after the read and before its first poll still wakes it. Broadcast publication occurs once after each changed batch and after a deletion. Periodic rechecks also cover remote changes; they do not establish the local wake invariant.

The registered materialized-publication-lifetimes property exercises bounded updates, captures, deletes and rebinding, then restores every captured schema, logical value, watermark and branch. Production-owner Shuttle checks cover update/capture, eviction/recreation, assignment replacement and required-wait registration without a polling timer. The Loom invariant server.state-assignment.drained-publication proves the synchronous admission drain observes preceding writes; its relaxed completion weakening must fail. ArcSwap internals are opaque to these models. Supported Turmoil transfer checks establish network framing and replacement scope, not disk durability or whole-cluster recovery. Typed Ratchet 10 owns mandatory immutable-image materialized smoke/soak qualification. The same-host just bench-materialized-state measures row and batch cost for 1, 256 and 4096 branches, allocation and captured-carrier retention; instrumented benchmark smoke execution supplies coverage, not performance evidence. Assignment/capture barriers and sealed-cache locks are tracked by Deloxide; async notifications and opaque publication internals remain outside its detector. just test-deloxide, just test-deloxide-order and just test-deloxide-stress include the materialized-publication diagnostic process, which starts the detector and exercises timestamp replacement, capture, sealing, installation lifetimes, rebinding, lazy relay installation, branch eviction and bounded complete snapshot properties. The latter also checks potential acquisition-order cycles among tracked lock instances. Each selection retains its own evidence alongside the other diagnostic processes under target/deloxide/test-deloxide; these instrumented executions supply correctness evidence, while the ordinary native benchmark supplies the row-publication cost measurements.

Sealing retains bounded identity and Arrow pieces on quota-owned staging disk and concatenates through one 64 KiB read buffer. Row projection and its IPC result share one bulk admission and one CPU job; projected columns never wait for a second grant while holding the first. Typed identity conversion is charged before it allocates. The sealed cache holds the immutable file and descriptor, so its memory does not grow with container size. Periodic writers select their namespace before executor admission, write one 64 KiB database chunk per batch, synchronize the chunks, then synchronize the replacement checkpoint header. A queued writer whose namespace was replaced is inert, and a pinned reader continues through the same database snapshot. Replica installation rejects a lower revision before clearing its current entries. An incoming materialized snapshot releases its response stream at EOF, before local verification or Arrow decoding. Those local operations retain their own staging and memory admission; they do not keep Snapshot request or connection-stream permits while waiting for executor work.

Kafka offset catch-up checks the revision before retaining or encoding a checkpoint. Capture holds the assignment barrier only to select a conservative revision and immutable partition topology; encoding retains that topology and reads shared offset slots after the barrier returns. A commit included ahead of the captured revision is offered again by the following revision. The native encoder writes into quota-owned staging disk with a fixed 64 KiB I/O grant and separately admitted serializer scratch. Bulk transfer carries bounded chunks rather than a whole resident checkpoint. At EOF the receiver releases its Snapshot stream, verifies the staged file, and admits native decoding separately. It builds a replacement table directly from archived entries, checks cancellation between entries, and publishes only under the replica’s existing assignment token. A failed conversion changes no installed offsets; rebinding rejects a delayed installer. No new concurrent map, per-record lock, or atomic ordering is introduced by this transfer path. An unchanged poll repeats the held revision through the existing replica assignment admission. This recovers a lost acknowledgement without encoding another checkpoint, while promotion or replacement prevents a retained poll from reporting progress through its former token. The runtime passes the native bulk operation’s thirty-second budget into the quorum wait for both committed and reset Kafka offsets. The offset-state owner supplies the register-before-read wait mechanism; it no longer chooses a separate response budget that can expire while an admitted native checkpoint is still progressing. A missing replica acknowledgement still fails at this bounded deadline.

Acknowledgement state

Each source attempt owns one in-memory acknowledgement tree. Fan-out reserves shares in an atomic pending count, and each completion resolves one share with a compare-and-swap. A second atomic word encodes either a typed tracking state with its active-share count or the completed state; its reserved raw representation stays inside the encoding’s owner. That state records whether shares still count against ownership handoff; a message parked on materialized REQUIRED WAIT does not keep a handoff blocked, and neither does a row a window retains until the window steps past it. Both owners hold one park guard per retained message and drop it before they acknowledge, so the acknowledgement removes the share the holder owns. Remote ACK progress publishes a monotonic sequence with the root’s parked handoff state. An upstream remote share follows that state, and its root publishes the resulting transition farther upstream. A delayed progress event cannot reactivate a newer parked state. Terminal ACK resolution still owns the pending share and releases the parked guard with it.

A handle owns an active share only when its attachment reserved a pending share. An attachment that finds the root complete may observe that root, but cannot park or remove a share owned by another handle. Resolution removes the owned pending share before changing active-share tracking, so a terminal transition cannot make an unsuccessful attachment look like active work.

Root trackers count attempts rather than individual shares. Their transition ordering may briefly overcount but cannot let handoff miss active work. Exactly one terminal transition takes the one-shot sender from its slot and releases that guard before notifying the waiter. ACK trees, shares, and trackers are volatile hot-path state and are never persisted.

A WASM branch holds back the success of the inputs a guest callback decided with one more attached share per input. The branch task owns those shares for the length of the callback’s checkpoint and resolves them itself: it releases them when the checkpoint completes and negatively acknowledges them when it fails. Holding them needs no lock and no shared registry.

A share a node forwarded to another node with a relay delivery waits in the node’s registry of forwarded acknowledgements under its registration number, beside two atomics: whether the delivery was admitted, and how many sweeps found no report from the receiver since its last one. A report, the admission, and the once-a-second sweep change them through a borrowed lookup, under the shard’s shared lock. The sweep fails a share only through an exclusive removal whose predicate rechecks the count, so a report that took the shared lock first keeps the share, and a report after the removal finds no entry. A terminal outcome, the delivery’s own failure, and the sweep each take the share out of the registry with one removal, so exactly one of them resolves it. The atomics establish no ordering across locations; the shard lock orders the recheck against every earlier report.

Client ingestor endpoints

Each client ingestor a node executes has one endpoint task that owns every producer attached to it: the batches each queued, the round-robin order among them, the batches handed to the execution’s admission worker or admitted and awaiting acknowledgement, and whether admission is open. Producers, the admission worker, the quiesce watch, and lifecycle changes reach that state only as commands on the task’s channel, applied in the order they were sent, so nothing locks it. An execution is installed before its admission worker and quiesce watch start, so the watch’s first report, which opens admission, is applied after the installation.

The admission worker takes one batch at a time over a channel of one. A worker that stops refuses the batch it was handed and never took, and closes the channel so that the endpoint refuses any later batch itself. The worker’s task is joined before the endpoint learns that the execution stopped, so a batch the endpoint still holds then was taken by a worker that was aborted when its stop outlasted the grace period, possibly while dispatching it: its outcome is unknown, never refused. The endpoint task awaits each admitted batch’s acknowledgement itself, among its commands and before further ones once it resolves, so an ending endpoint leaves no task behind, and a resolution for an attachment that already ended finds nothing to answer.

A producer’s turn gives its next queued batch a slot of the window whether or not the worker is free, so a busy worker never passes a producer over. A local producer’s batch is then ready for the worker. A batch that another node forwards first waits among the producer’s batches awaiting clearance while the endpoint asks the serving node over the producer’s own clearance channel. The clearance returns as a command, and clearances arrive in the order they were requested, so the one that arrives names the oldest batch awaiting it, or a batch the endpoint has already refused, which it ignores; the cleared batch is then ready for the worker too. A free worker takes a ready batch, which already holds its slot. A refusal, a close, an end, or a detach answers or drops the batches still awaiting clearance or the worker, which never reached it, and releases their slots in the same command, so a lost serving node never leaks a slot of the window.

After each command the task publishes its producer, outstanding, and window counts into plain atomics that DESCRIBE reads, and into metric gauges it resolved once. Each count is exact when written, and a reader may see one count of a change before another.

The node’s producer byte budget is one atomic counter. A reservation adds its bytes with a compare-and-swap that refuses to pass the budget, and dropping the reservation subtracts exactly those bytes, so concurrent opens through different sessions and links never overcommit it.

A session’s producer credit is held under a short mutex, taken by the receive loop when a batch arrives and returned by the producer’s task before it queues that batch’s reply, so a client that sends only after reading a reply always finds room. The mutex never crosses an await.

Client emitter attempts

Each running native client emitter has one actor for its volatile output. Its channel serializes consumer attach/detach, prepared-payload publication, assignment, settlement, timeout and cancellation. An attempt has exactly one owner and ACK reference. Revocation returns its worker’s batch and byte credit and marks that reference stale before the payload can be assigned again. Sequential dispatch checks every earlier unresolved delivery of the same source relay and concrete branch; parallel dispatch counts assigned attempts against that stream’s shared window. The actor keeps only a bounded queue of completed references for duplicate ACK recognition.

The node’s consumer grant and retained-output byte counters are separate atomics. A session’s consumer count and credit use one short mutex; neither that mutex nor a runtime map guard crosses an await. A retained payload reserves actual bytes until application settlement or cancellation, and its prepared Arrow bytes and source members remain owned by the emitter host until the result is applied. A consumer read uses an asynchronous receiver lock only for that consumer; it cannot hold the session receive loop or the ordered command lane. Settlement is a concurrent request, so a quiescing command cannot prevent the application ACK it waits for. The shuttle_competing_consumer_grants_never_exceed_the_node_budget check explores competing session grants against the production atomic budget. The shuttle_consumer_loss_and_ack_race_release_one_retained_delivery check runs the production delivery owner with deterministic attempt references and no timer wakeups to explore ACK versus consumer detachment. The shuttle_publish_cancellation_and_ack_race_release_one_reservation check also races publisher cancellation against application ACK and verifies retained byte credit returns once. Timeout revocation is covered by the ordinary owner and public scenarios.

Rust client attachment recovery

The client retains desired producer and consumer handles independently of their wire attachments. Their weak registries are reached for attachment registration, admission changes, termination and reconnect snapshots, not for discovering an attachment on each submission or delivery. Producer wire entries include the exchange generation beside the attachment ID. A snapshot upgrades live weak handles and releases its map guard before any handle transition or network await. Close/drop removes desired entries, exchange termination removes its producer wire entries, and snapshots prune expired weak handles. The concurrent map inventory records each map’s access frequency and disposal.

Each desired handle has one lifecycle owner, with a short selected mutex over its typed attachment phase and a watch notification for changes. Beginning or installing a restore checks that phase and its exchange generation under the same mutex as close. Close is terminal: a restore reply that loses that transition cannot install its attachment and releases the grant it obtained. Another exchange loss can interrupt only the restore or active attachment of that exchange. No lifecycle guard crosses an await. Submissions and deliveries keep their originating attachment, so replacing it cannot retarget an uncertain submission or an application ACK.

The four production-owner Shuttle checks listed below race close with starting restoration and with another loss during restoration. They establish that the handle stays closed and cannot begin a subsequent restore. Public scenarios cover exchange I/O, late open replies, reservation cleanup and delivery interruption; those network effects are outside these in-process checks. Client Session Protocol and its implementation manual own the restoration and application-outcome contract.

Node quiesce accounting

Every entity on a node keeps one set of quiesce counts. A drain reads them to decide whether the node still holds work, and the hot paths that take a message in, collect it, park it on required materialized state, buffer its output, or finish it adjust them.

Each total a drain reads is a counter of its own: everything the node holds, and the admitted subset of it that a local drain waits for. A total summed across several counters at read time can miss an item that is between two of them, because the reader loads the count the item has already left and then the count it has not yet joined. A drain that misses the last item reports a node holding no work while it still holds one, and an entity gate that believes that alters the entity under the message. Messages parked on REQUIRED WAIT and outstanding force-flush obligations are counted on their own for the drain’s report, and neither exchanges items with the admitted count.

An adjustment that moves work between the counts raises every count that rises before it lowers any count that falls, so a drain reading during the move sees at least the work the node holds. A processor publishes its collected inputs, parked messages, and buffered outputs together for the same reason: a batch that left an input collector for an output buffer was admitted work throughout the move. Overcounting only delays a drain, while undercounting lets one conclude early, so the ordering is chosen in that direction. The counts are hot-path scalars and are never persisted.

A local drain reads these counts together with the domain’s relays, ingestors, generators, acknowledgement roots and emitters, and work keeps moving between them while it reads. No single order of reads follows every move. A node that takes a batch from a relay counts it before the relay lets it go, so the relay has to be read first, while a relay counts a batch a node publishes before the node lets it go, so the node has to be read first. Each relay therefore keeps two totals on each node that only rise: the batches it admitted, from its owner’s publishers or from deliveries another node routed to its consumers here, and the batches it handed on. It holds their difference, and the admitted total is also its admission sequence. A drain observation reads every relay’s sequence before and after everything else, and a relay whose sequence changed in between counts as work still moving. A batch the observation missed reached the place where it rests only after the read that would have counted it there, so it entered some relay between that relay’s two readings. The observation reads force-flush obligations first: a participant completes its obligation after everything it did for the flush, including resuming a message parked on REQUIRED WAIT, so an observation that sees the obligation complete sees that work as well. The totals cost a relay’s owner path the same two read-modify-writes per batch as the pending count they replace, a routed delivery two more, and a drain reads them only on its own poll.

A broker source’s batch keeps its acknowledgement roots counted beyond their resolution. The source host creates confirmed roots, and a confirmed root that resolves hands its domain and ingestor counts to the completion that observes it instead of releasing them. The batch acknowledgement drops the completion of every root that resolves while another is still unresolved and keeps the completion of the last, which the source loop drops once acknowledge or reject has returned. The hand-over is part of the root’s one terminal transition, so a drain that reads the counts by read-modify-write never finds them at zero between a batch’s last resolution and its transport’s confirmation. An unresolved root keeps holding them itself, unless it waits only on REQUIRED WAIT state, which a drain does not wait for, and a completion that dropped before its root resolved has the counts released as the root resolves. See Confirmed acknowledgements.

An ingestor route’s force-flush participant releases its partial batches into branch execution; the admitted client submission’s tracked ACK root stays outstanding until its downstream work settles. For a model alteration with a shared relay gate, the control plane suspends the affected ingestors on every node and drains those roots before closing the relay gate. Otherwise a flushed batch could wait behind the gate whose drain waits for that same root. Intake and full entity holds overlap on the same quiesce control: releasing one EntityHold removes only its share, so the remaining hold keeps admission suspended. The client admission Shuttle check exercises that overlap with racing batch admission; the public buffered alteration scenarios exercise delivery through local and remote relay owners.

A task that must not publish while an ownership handoff has frozen its entity observes that freeze through one watch, which registers for the next freeze change before it reads the freeze. A release landing between a read and a registration wakes nothing, because the waiter does not exist yet, and a task that missed one keeps a stale freeze that disables exactly the force-flush and deadline arms that would have woken it again. The release lifts the freeze before it wakes anyone, so a waiter that rereads it sees the entity thawed rather than parking on a change that has already happened. The ingestor route task, ingestor branch maintenance loop, and processor branch task all take this observation before deciding which force-flush and deadline arms to enable. Their wake contract is therefore the watch owner’s registration order, checked by shuttle_an_ownership_handoff_freeze_observation_registers_before_its_read against the production release path. Other data-plane waits on watch subscribe before their deciding read, or use an existing receiver whose borrow leaves a subsequent changed able to see an intervening send.

Ordering Fences

Three fences remain because they define externally observable order or ownership. Each is scoped to the smallest identity that carries the guarantee. They do not provide general-purpose mutual exclusion for unrelated batches.

Per-slot delivery gate

A remote delivery slot is scoped to one destination, relay, payload role, and concrete branch. Its asynchronous gate permits one encode-and-send operation at a time for that logical channel. The gate preserves FIFO order from serialization through admission and ensures the channel sequence is allocated in the same order. Other destinations, relays, roles, and branches use different slots and continue independently.

Established slots are found with a borrowed lookup. entry() is used only when concurrent first deliveries must agree on a single slot; allowing two racing installations would create two sequence owners and could interleave the channel. The small sequence state is updated only while the slot gate is held, and its synchronous guard never crosses an await.

Dispatch-gate engagement

The relay dispatch gate separates ordinary publication from a model change, ownership handoff, or shutdown operation that must know every earlier dispatch has left. On the open path, a publisher raises an atomic in-flight count with a read-modify-write and then checks an atomic closed flag. An engagement closes the gate and then reads the count with a read-modify-write as well. Read-modify-writes of one count are totally ordered, so either the publisher’s comes first and the engagement counts it, or the engagement’s comes first and the publisher’s acquires it, observes the gate closed, rolls its count back, and waits. A plain load of the count would not do: it may return a count from before an increment that preceded it, while that publisher’s own load of the flag still finds the gate open. Leaving releases what a dispatch did under its permit to the engagement’s next read, and the last dispatch to leave a closed gate wakes the engagements waiting for it.

The engagement state is consulted only while the gate is closed. Once all earlier permits leave, the engagement owns a lease and the protected mutation can proceed. The acquisition deadline bounds waiting for that fence; a successfully acquired lease remains engaged until its owner releases or drops it.

An owner buffer admits a batch before its fan-out begins. Fan-out therefore takes a nonwaiting dispatch permit before it reads any local or remote runtime-consumer route and holds the permit until fan-out has attached the consumer ACK shares and resolved the owner’s share. A schedule swap cannot replace the routes while that permit is live. If the gate has closed first, fan-out fails the record ACKs and returns the batch for source retry. Waiting for the gate here would deadlock the swap, because the swap’s drain counts the buffered batch. The same gate protocol orders the nonwaiting attempt and the swap engagement.

The three-node attached-emitter move scenario injects a stale zero buffered-batch drain report while an owner batch is paused. That puts fan-out beside the local swap even when the coordinator’s usual drain would wait for the batch, and verifies that the local gate independently prevents an ACK for a batch the moved consumer missed.

A WASM state reset adds a narrower gate without putting a lock on relay dispatch. Under a short whole-relay publication fence, the relay atomically replaces an immutable list of scoped reset gates. Dispatch loads that list once, compares the batch’s branch fingerprint with each typed scope, and waits only on matching gates. Removing a lease atomically republishes the list and releases its gate. This gives publication and dispatch one total order while unrelated branches neither acquire shared locks nor wait for the reset.

Client batch admission fence

A client batch is validated before it is dispatched, and a quiesce can engage in between. The admission worker therefore tracks the batch’s ACK root with the ingestor’s drain accounting first, and only then reads the quiesce publication. Either the read comes before the engagement, so the drain that follows counts the root and waits for it, or the read observes the engagement and the batch is refused with its root resolved before anything was dispatched under it. No batch can be dispatched after a drain concluded that the ingestor held no admitted work. The admission tracks the root with a read-modify-write, and the drain reads each root count with a read-modify-write after the engagement published, so one side observes the other in the C11 memory model, not only on processors whose read-modify-writes are full barriers.

Assignment generations

A runtime-state assignment packs its generation and capability into one atomic word. A hot operation increments the admission counter for that generation before comparing its token with the published binding. A rebind serializes with other rebinds, publishes the successor binding, and waits only for operations admitted under the generation it superseded. It reads that generation’s admission count with a read-modify-write, so an operation whose admission came first is waited for, and one whose admission came later acquires the rebind’s read and observes the successor binding. An operation that finishes releases what it did to the rebind’s read. Even and odd generation counters let work admitted under the successor proceed without extending that wait.

This fence prevents a former owner from mutating or describing state after reassignment. Snapshot capture holds the assignment barrier so its observed binding stays current for the whole capture. Whole-state installation is authorized exclusively under its exact binding: it cannot overlap a capture or an admitted origination. Ordinary record updates use atomic admission, continue while a capture merely holds the barrier, and never take that barrier.

Bounded Synchronization Outside The Open Path

Some synchronization remains after a hot operation has selected a state that explicitly requires it:

• a quiesced ingestor, and one draining what it retained, locks only the bounded retained-payload buffer selected by its declared MAX SIZE policy: to retain a payload, to take the oldest out for delivery, and to end that delivery. No guard crosses an await.
• a retained payload whose unfolding finds the extension class’s wait queue full waits for a place in that queue. The executor’s semaphore hands a freed place to waiting jobs in the order they asked, ahead of any job asking afterwards. The wait is scoped to the one class, holds no lock, and ends when a place frees or when the ingestor’s shutdown or a new quiesce interrupts it
• a live payload a paced source or a source whose loop may be held handed over without an acknowledgement waits for a place the same way. The source host registers for the next quiesce publication before it compares it with the decision the payload was taken in under, so a new decision always ends the wait, and the host then decides on the payload again; the ingestor’s stop ends it as well. The race holds no lock and no guard across an await
• one concrete processor branch serializes its own batches, while other branches remain independent
• one already-resolved histogram series serializes its bounded accumulator update, once per recording: a batch’s delivery latencies arrive as one folded recording, so the lock is taken once per batch; counter series are atomic
• one inbound relay attempt serializes its small receipt, admission, rejection, and cancellation transition within the interconnect’s quotas and deadline
• snapshot construction and state installation serialize per placement, off the row path
• a WASM branch waits for its own checkpoint after each guest callback, bounded by the checkpoint deadline. The state store’s durability barrier lets one writer at a time run a storage synchronization for every writer waiting, through one atomic runner slot and a ticket watermark; it holds no lock across that wait, and the synchronization runs on the storage workers. The round of a synchronization belongs to the storage job that flushes: it frees the runner slot and records its outcome when that job ends, so a writer that stops waiting neither frees the slot while the flush still runs nor loses a failed flush. Replica progress reaches the waiting branch through the replication of its own state.
• one placement’s replication changes its announcement, its replicas’ reported progress and the handover between its announcers under a lock scoped to that placement, taken by its originator, its one announcer and its replicas’ acknowledgements, and never held across an await; see Checkpoint replication
• a replica’s copy of one branch checkpoint is held under the registry’s guard for one revision comparison and a move, never a copy of its payload
• the owner’s announcements pending for one branch-keyed entity on a replica are added to under a lock scoped to that entity, one newest announcement per branch, and taken all at once by the entity’s one replica task, never across an await; see Replica catch-up
• an ACK root locks only its single terminal sender transition

These sites are accepted for the contract and bound named above. A lock that merely makes shared state convenient, protects immutable configuration, or repeats registry discovery on every batch does not belong in this category.

The simulated relay fault checks drive the production receipt, admission, and cancellation APIs through authenticated connections. They synchronize on the received Arrow batch and verify the same attempt cannot be admitted twice after a lost reply. Cancellation before grant and while receipt or reconnection is unresolved must win the attempt’s existing admission fence before the runtime can admit it. A cancellation after admission returns the admitted outcome.

Relay fan-out itself uses one bounded queue per consumer. Publishers share no fan-out lock; capacity and receiver counts are atomic, and a publisher registers for notification only when a consumer queue is full. A waiting publisher raises a waiting count before it reads a consumer’s admission count, and a consumer lowers its admission count before it reads the waiting count. Both reach the admission count by read-modify-write, so either the publisher reads the room the consumer freed or the consumer reads the waiting count raised and wakes the publisher. Removing one consumer does not stop the others, and changing capacity does not discard batches already queued.

A publication counts the consumers it delivers to from the same snapshot of the registered consumers that it delivers through. A consumer that registers meanwhile receives later publications only, so the acknowledgement shares a publication reserves for its attached consumers always match its deliveries, and a routed delivery, which holds no dispatch permit, cannot resolve a share it never reserved.

Ratchet And Review

just ratchet, just validate and just validate-ci require the pinned compiler’s Nervix source diagnostics to pass across the complete declared matrix. Finite API recognition lives in tools/nervix-lint/report/src/rules.rs. Architectural contracts live with their source owners; generated inventories under target/ are evidence, never approval inputs. Other structural ratchets retain their own checked-in counts.

The compiler recognizes resolved Mutex, RwLock and DashMap acquisitions through aliases, re-exports, dereference adjustments, UFCS and selected trait implementations. Borrowed reads, mutating and try operations, and iteration remain visible. Ordinary collection entries, held-guard entries and I/O do not acquire a recognized lock. Borrowed DashMap iteration acquires lazily; consuming iteration owns its storage. Authored arguments remain checked through external macros, including operation expectation macros.

Source contracts

Gate annotations with cfg_attr(nervix_lint, ...). Only the isolated analysis driver registers the nervix tool and enables that cfg; ordinary stable and browser source needs no unstable feature. Contracts use the narrowest meaningful function, trait, type, impl, module or crate boundary:

#[cfg_attr(nervix_lint, nervix::context(recurring, reason = "one callback per admitted batch"))]
fn process_batch() { /* retained execution state */ }

#[cfg_attr(nervix_lint, nervix::context(bounded,
    reason = "the retained grant serializes its terminal transition",
    key = "one admitted relay grant", bound = "one synchronous transition; no guard across await"))]
struct GrantState { /* retained bounded state */ }

recurring identifies record, batch, frame, acknowledgement and steady poll execution. lifecycle identifies installation, replacement, snapshot cadence and teardown. observer identifies observation outside execution, and outside names an edge or harness outside this rule’s data plane. bounded remains recurring but permits the explicitly described retained protocol; its reason, identity key and capacity or deadline are required. These are reviewed architectural assertions, not proofs of exclusive ownership, bounded waiting or runtime cadence.

An explicit callable contract wins over defaults. An implementation inherits its trait method’s contract before its impl/type or lexical defaults; a recurring trait method’s override remains recurring or names a bounded protocol. A type contract applies to its implementations. Duplicate, conflicting, malformed and misplaced annotations fail. A binding whose initializer owns exactly one anonymous body can supply that body’s context: use it when a constructor installs a recurring task. Nested task bodies still require their actual entry contract.

The compiler re-evaluates supported local call edges and closure/callback bodies to a fixed point. A recurring caller makes an unannotated local helper recurring, including one in a lifecycle module. An explicit lifecycle callable is an installation boundary: entering it from recurring execution emits nervix::lifecycle_call. Callee and trait contracts survive cross-crate metadata and renamed imports. During recurring execution, unresolved generic, dynamic trait and function-pointer application calls require a callable/trait contract or nervix::dispatch(reason = "...") on their owning callable; a cold module default is insufficient. A dispatch contract states the external driver or callback boundary; it does not exempt compiler-visible local callback bodies or acquisitions.

Local iteration and polling implementations remain call edges. Generated await polling is the Rust scheduling protocol, rather than a separate application callback diagnostic. Synchronization inside external libraries, opaque external future implementations and unsupported whole-program relationships remain outside the recognizer’s claim. The checker supplies no universal ownership or effect proof.

Resolved architecture boundaries

nervix::layer(kind, reason = "...") states the exact documented crate/module, type, trait, impl or callable owner: primitives, vocabulary, language, engine, decisions, data_plane, control_plane, edge or harness. The narrowest contract wins; associated implementations inherit explicit trait metadata and impl-self contracts. Contracts survive compiler metadata and move with their source owner. An unrelated trait module default cannot replace an implementation body’s lexical owner. An engine that receives validated plans also declares nervix::input(plans, ...); general infrastructure is not presumed to execute graph plans.

nervix::value(model|name|cluster_access, ...) classifies exact semantic ADTs or traits. The current configuration Model union, shared name vocabulary and cluster observation capability carry those roles. nervix::model_in_data_plane rejects the configuration union in data-plane and plan-consuming interfaces or bodies, including normalized generic containers, aliases, constructors and patterns. An ordinary user type called Model has its own meaning. These checks do not treat every vocabulary field or schema descriptor as a configuration Model.

nervix::parser_dependency diagnoses resolved language definitions outside the language, edges and harnesses, including imports, type/trait references, qualified calls and methods. A separate source/manifest validator retains the owning crate’s import path before cfg selection, including renamed dependencies, inactive module imports and authored macro tokens. It derives language packages and consumer ownership from source contracts and Cargo/module declarations. Approved plain primitive and vocabulary re-exports keep their actual semantic owner; importing from a language crate still requires an allowed provenance owner even if resolution reaches a plain value.

The complete architecture rule matrix separates these resolved contracts from compiler-independent source properties. Integer/Duration clamping and Option/Result control-flow diagnostics use the same source-local exception machinery and legal configuration coverage; they do not classify a custom method by its name. Their serial analysis does not supply concurrency execution evidence. Explicit Result branches preserve the same reports and delivery outcomes when a caller expands an adapter sequence.

nervix::cluster_await diagnoses future-returning access on the classified cluster capability or trait when awaited from recurring data-plane execution. It follows the existing supported local helper/callback graph, selected concrete methods, UFCS, async closures, awaited expression wrappers and simple local future-binding chains. A local I/O await has no cluster role. A lifecycle cluster read remains lifecycle work. Dynamic/generic application dispatch needs its existing source effect contract; opaque external future implementations, arbitrary stored-future relationships and whole-program effects are not inferred. Execution graphs and recurring reachability are conservative within the declared local graph; annotations are reviewed architectural assertions.

nervix::node_identity, nervix::installed_state and nervix::state_contract implement the exact identity and representation contracts in Typed States. All six architecture diagnostics use the same normal lint levels, exact-operation expectations and complete current configuration evidence as synchronization and failure rules. No count, product owner catalog or source-review hash grants permission.

Diagnostics and reviewed exceptions

nervix::sync_acquisition diagnoses an acquisition on ordinary recurring execution. nervix::unknown_effect diagnoses an acquisition without a context or an unresolved application call reached from recurring execution. nervix::invalid_contract rejects contract and exception errors. The lints use ordinary Rust warn/deny/expect levels; the required gate also rejects unresolved Nervix warnings.

The driver also emits the resolved failure-contract, outcome-discard and panic diagnostics described in Errors And Diagnostics. Their exact error/return classifications use nervix::error_boundary; execution-context inheritance does not classify an error or exempt a failure. All diagnostics share the same one-operation expectation checks and complete configuration evidence.

Retained repair debt names its owning task at the exact operation:

let state = nervix_primitives::expect_lint!(nervix::sync_acquisition,
    "Typed Ratchet 15 (86bca1web): retain the replica entity before frame handling",
    states.get(&entity));

expect_lint! puts a normal reason-bearing Rust expectation on one binding, evaluates the operation once and returns its value. It supplies stable syntax and introduces no runtime policy, allocation or lock. A direct gated expectation on one operation binding works too. A lifecycle call admitted only during first installation or terminal teardown explains that phase and its concrete lifetime in the same narrow form. Blanket allow, undocumented or broad expectations, lint caps and unfulfilled expectations fail. The checker counts distinct HIR operations against Rust’s effective expectation identity, including macro expansion; widening a binding to cover a second operation fails even when Rust would regard the expectation as fulfilled.

Task handles remove recurring status, freeze, metric and checkpoint lookups. Source readiness, relay channels, buffered error delivery, domain selection and transport leasing use retained handles or immutable publications. Materialized writers own their mutable branch records and readers retain per-relay publications. Remote acknowledgement rows and admission waiters use fixed routing positions with exact generations; authenticated connections retain bounded peer protocol owners. Replica catch-up retains the entity’s lifecycle handle, assignment slot and its own record of each branch. Replication frames, synchronization and listing requests select published state handles; announcers retain their route and assignment slot. Their runtime registries handle installation, replacement and teardown. A session’s clock delivery retains the observer its attach resolved and reads only that observer’s lifecycle publication and progress watch. Accepted progress reaches the runtime’s domain registry once per report and then compares and replaces inside the watch, and an attach waits on the committed-domain installation count rather than returning to the registry. Bounded retained placement progress remains explicitly documented. Testing fault selectors name their exact emitter, ingestor, domain/branch, checkpoint window or acknowledgement link and finite read/removal steps; they release map guards before a scenario-controlled pause. These contracts describe test selection, not a product wait deadline. The executor-saturation lookup belongs to testing fault control. The concurrent map inventory records the runtime ownership review; source contracts and expectations are the executable policy.

Complete analysis and qualification

The isolated tooling workspace uses nightly-2026-09-17; product builds use stable. The matrix analyzes ordinary workspace libraries/binaries, the browser’s wasm32 library and binary, the server’s testing capability, server/interconnect/primitives under Shuttle, server/consensus/primitives under Loom, interconnect/primitives under Turmoil, and the server’s library and binary, the deadlock diagnostics and primitives under Deloxide, where every blocking acquisition resolves to a tracked adapter that the rules recognize as the lock it wraps. Turmoil’s runtime cfg comes only from its just recipe, which preserves the parent’s diagnostic mode. Compilation supplies no concurrency execution evidence. Test targets and undeclared feature combinations are not claimed. Mandatory source/manifest checks retain import provenance, inactive cfg and authored macro checks.

Cargo artifacts, complete side reports and declared roots must agree, including when there are zero sites. Reports retain every owner, configuration and expansion instance; generated target and external source sites are exclusions. Completion identities cover current source and annotations, Rust rules, compiler, driver, validator, locks, dependency declarations, Cargo configuration, diagnostic mode and relevant flags. Source changes during a run fail. The workspace wrapper nests beneath configured kache. Missing reports establish a new supported cache namespace and rebuild only isolated authored artifacts; the gate preserves RUSTC_WRAPPER.

just test-typed-ratchet qualifies APIs and source diagnostics in the legal modes, ordinary Bolero properties, stable gated syntax and paired Rust API doctests. just qualify-typed-ratchet-cache qualifies fresh/Cargo-fresh runs, an actual kache dependency hit, changed inputs, missing and interrupted output, and two worktrees. Native coverage uses the matching LLVM tools and records the driver executions that ran. Sanitizer campaigns follow the shared Bolero label policy. Calibration, measured cost and complete generated evidence belong on Typed Ratchet 02A.

Use the generated listing during review:

just typed-ratchet --show
just typed-ratchet --configuration shuttle --inventory

Partial selections and warning-mode inventories identify their diagnostic/matrix scope explicitly; they cannot satisfy the required full diagnostic gate.

For every new or moved site and every shared-map access, the reviewer establishes all of the following:

1. Frequency. Trace its callers and decide whether it can run per record, row, batch, remote frame, ACK share, or steady poll iteration. A site on one of those paths is rejected unless it is an ordering fence or bounded protocol described by this chapter. Registration, first installation, replacement, teardown and observer reads are cold and are classified as such, separately from the recurring path.
2. Owner. Identify the one component that changes the state. Immutable reconfiguration uses a whole-value publication, a scalar uses an atomic, a stable dependency is bound when the task starts, and branch-local state belongs directly to the branch task. An ordinary collection that one owner holds, and its entry API, are not synchronization and need no justification.
3. Lookup behavior. A steady path does not look an established value up in a shared map, with entry() or with a borrowed get, contains_key or iteration, all of which take a shard lock. It retains the handle resolved when its task, branch, channel or attempt was created. entry() belongs to a cold or racing installation whose single-winner property is part of the contract.
4. Fence contract. An allowed ordering fence or bounded protocol names the order it preserves, the exact key that limits its scope, the capacity or deadline that bounds waiting, and whether any guard crosses an await. Unrelated branches, relays, destinations, or assignments must still progress.
5. Lifecycle separation. Startup, registration, schedule application, snapshot sealing, and teardown may synchronize with their peers, but their guards must not leak into a hot callback or be held while awaiting data-plane work.
6. Mechanical result. Run just ratchet; inspect the site list when the count changes; update the baseline only when the count fell. A lower aggregate count does not make a newly introduced hot-path lock acceptable, and renaming a method or moving a file is not a repair.
7. Inventory. Record a new concurrent map, or a changed frequency or disposition of an existing one, in the concurrent map inventory in the same change.

Installation-only fields declare their lifetime at the owner. The resolved installed-state rule rejects replaceable read-write locks in that representation, while an optional name/shared handle with an unstated lifetime requires explicit review. The current publication or one-time primitive must express the actual owner; type shape alone cannot establish a write-once lifetime.

Primitive Selection

Execution Primitives owns the shared primitive surface, its complete per-mode matrix, feature and target selection, authored-source scope, and unmodeled, blocking-work and thread-creation permissions. Every synchronization owner in this chapter obtains its primitives there. The boundary selects mechanisms; the source contracts and production-owner checks below establish what a caller may do with them.

Diagnostic Deadlock Detection

A diagnostic node selects the deloxide mode. The deloxide-order feature adds historical order instrumentation to that same mode, and the deloxide-stress feature adds Deloxide’s bounded scheduling disturbance to its active-only selection; neither is another execution mode, and the two cannot be enabled together. Ordinary builds keep their primitive reexports and contain no detector. Build the diagnostic node with just build-diagnostic-server [deloxide|deloxide-order|deloxide-stress]; its target directory is separate from the ordinary server.

Runtime disabling does not recover an ordinary or active-only fast path. The pinned Deloxide 1.1.0 graph build enters its global detector even on uncontended mutex and write acquisitions. Reads enter that detector in either diagnostic build. The order build also tracks actual guard lifetimes locally, including when runtime checking is disabled, so active self waits remain observable. No timing from these builds is ordinary product performance evidence.

Scheduling stress. A deloxide-stress build compiles Deloxide’s random preemption into its tracked locks. A thread that already holds a tracked lock and asks for another sleeps first with a fixed probability, for a delay drawn between two bounds, and a configuration may also yield a thread after every release. An active cycle needs two threads that each hold what the other wants at the same moment; widening the window between one nested acquisition and the next makes the lifecycle paths that take nested locks meet that moment far more often than an idle schedule does. Disturbance never decides a finding: the detector reports the same cycles it would report without it.

StressConfiguration is the typed configuration, recorded with the selection in every process’s evidence. It is bounded by construction: a probability in millionths of at most one, delays of whole microseconds between 1µs and 2ms, the shortest no longer than the longest. The lane’s configuration, StressConfiguration::LANE, preempts one in twenty nested acquisitions for 20µs to 200µs and does not yield after releases. The longest delay holds one runtime worker for a small fraction of the shortest physical product deadline, and the product’s deadlines are not widened for it. A first configuration, a quarter of nested acquisitions for 50µs to 1ms with a yield after every release, made the stressed nodes miss their startup and request deadlines; so did one in twenty without the yield while four scenarios shared one process, which a plain diagnostic build of the same scenarios met. The cost is not the delays alone: a stress build enters Deloxide’s process-wide detector twice on every mutex and write acquisition, once to decide the delay and once to acquire, so every node of every scenario in one process contends on that detector. The stressed scenarios therefore run one at a time. Deloxide draws its preemptions and delays from entropy it seeds itself, so a configuration reproduces the distribution of the disturbance, not one schedule; a replay repeats the workload and the configuration.

Deloxide applies the disturbance only inside its blocking acquisitions, and its stress build disables its own optimistic mutex and writer fast paths. The tracked adapters therefore skip their own immediate try in this build and enter Deloxide’s blocking acquisition directly, registering the waiting site first. Deloxide keeps its reader fast path, so a shared acquisition that finds no writer is not delayed. Because every mutex and write acquisition then reaches Deloxide’s global detector, its held set is populated as in an order build, and the adapters keep the caller’s live guards so a self wait is still reported. Deloxide’s component-based strategy is not offered: it appends every nested acquisition to a list it never trims and scans that list on each one, so its memory and its cost per acquisition grow with the life of the process. The random strategy keeps one preemption counter per lock it delayed, which grows with the locks a process constructs and is bounded by the process’s life within its invocation bound.

The stress selections run every active-only probe, conformance check and owner test again, and between them every scenario of the active-only selection but the report tool’s triage, which drives no lifecycle path. deloxide-stress runs the lifecycle scenarios tagged @deloxide_stress and the paced-driver scenarios with the stressed Rust driver; deloxide-stress-restore compiles the same build under its own name and runs its probes and the capture, restore and checkpoint-transfer scenarios tagged @deloxide_stress_restore, in a lane of its own because the stressed scenarios of both, one at a time, would take most of one lane’s budget. The probe.stress-disturbance probe proves the disturbance is applied in each: twenty thousand nested acquisitions must wait at least the shortest delay for half the preemptions the configuration expects, and record the lane’s configuration. Its first run found that the adapters’ immediate try bypassed the disturbance entirely, and that a self wait went unreported once Deloxide’s held set was populated; both are fixed in the adapters, and the probe and the stress selection’s self-wait probes are their regressions.

The tracked surface. Every sync::blocking mutex, read-write lock and condition variable is an adapter over Deloxide. The adapters retain new, lock, try_lock, read, write, try_read, try_write, get_mut, into_inner, Default, From and guard dereferencing. Locks are not poisoned; Debug tries the lock and never waits. Timed, upgradable, mapped, fair and reentrant acquisitions are unavailable because this backend cannot track their contract. Constructors and successful guards retain caller locations; an acquisition that waits separately registers its waiting site and requested access. The compiler synchronization gate classifies these as the acquisitions they wrap.

A condition variable keeps the Shuttle adapter’s wait and notify_all interface and an exact count of the waiters in each notification generation. Waiting removes the caller guard’s held lease while the lock is released, and reinstalls it on reacquisition. Its internal state guard’s lease also leaves while the vendor condition variable releases that lock. A notified waiter reacquires the caller’s lock through its tracked adapter, so it can participate in an active cycle.

Installation and active findings. The process installs once, after its signal registration and before any tracked lock or runtime worker, through nervix-deadlock::DiagnosticRun. The server’s main, the diagnostic scenario binary and the Rust paced driver own that placement. A tracked constructor before installation panics with a configuration error; a second installation fails. Installation keeps the vendor banner off standard output by temporarily redirecting that descriptor to /dev/null.

A WaitForGraph finding describes threads currently waiting in a cycle. A self wait is also an active cycle: the mutex adapter checks a failed acquisition against its calling thread’s actual live exclusive guard in an order-instrumented or stressed build and sends that one-thread cycle through the same bounded handoff. This establishes a real incompatible owner/wait pair; it does not infer an active cycle from historical order. Deloxide 1.1.0’s common-held-lock filter otherwise discards this case once instrumentation or stress populates its held set. Recursive mutex acquisition and an upgrade from one’s own read guard remain disposable-process regressions in every selection.

Historical order and source evidence. A LockOrderViolation is a PotentialCycle, never an active outage. For example, a thread can successfully take A then B, release both, and successfully take B then A. The run retains the potential cycle and continues the workload. The pinned upstream graph adds requested mutex and write acquisitions; it does not add requested read edges. Opposite read-only orders complete without an order finding. A read guard held before a requested mutex or write still contributes a historical edge, with its shared mode and multiplicity retained. This limit also excludes some dangerous mixed orders: absence of a finding is not coverage of every reader/writer combination.

The boundary retains directed edges between run-local lock instances, not between construction lines. Each edge carries construction sites, Live, Ended or Unrecorded lock lifetime at correlation, and source witnesses with run-local thread identity, optional name, held and requested sites and modes, attempt count, and the number of identical live guards at the held site. Thread names and construction sites copied into history survive registry removal. Lifetime observations are not an atomic snapshot of all locks. Missing context is explicit. Source attempts include unsuccessful nonblocking calls; only upstream successful acquisitions decide that a historical cycle exists. Attempts are not claimed as successful-acquisition counts.

A complete cycle has no repeated closing node and rotates to its smallest lock identity. Witness sets sort by source context and reject duplicates. Deduplication keys on the directed instance cycle within one process, merges distinct source/mode witnesses, takes the maximum cumulative attempt count for each context, and counts delivered callbacks separately. It preserves direction, lock instances, known construction, ended lifetimes and guard multiplicity. New source or lifetime context invalidates an existing review. It never equates instances merely because a constructor line was reused.

Bounds and handoff. The live registry owns the live lock, waiting attempt and thread records; none of its shard guards cross a tracked acquisition. Order instrumentation adds a run-bounded history of at most 16,384 directed instance edges, 1,024 source witnesses per edge, and 64 guard leases per thread. Ending a lock does not reclaim the historical edge budget. Refused history or count overflow becomes an explicit overload finding. The witness budget retains distinct storage worker identities across checkpoint staging. Its process regression retains all 512 worker contexts in one historical edge; a separate 1,025-context workload must record order-history overload and fail qualification. The independent edge and held-guard limits remain fatal too.

The vendor dispatcher has an unbounded channel and swallows callback panics. The boundary callback therefore only retains bounded identity vectors, submits to its lock-free ReportHandoff of 64 reports, counts refusals, and unparks the findings thread. Each queued active report retains at most 64 threads/waits; an order report retains at most 65 identities to describe a 64-edge prefix. The findings thread correlates source context and calls the sink serially. Missing order payloads are loss, not a clean run. Sink panic aborts. Installation failure closes the handoff. A periodic wake also delivers history overload that produced no upstream cycle.

The history, handoff, output and artifact limits are Nervix retention limits. They do not bound the vendor graph, its cache, its pre-callback allocation or its dispatcher backlog. No exact upstream flush barrier is available: evidence is an observation at the time it is read, and a process can end with upstream callbacks pending. Deloxide 07 owns dependency and upstream graph/dispatcher hardening; full diagnostic workloads retain the actual observed bounds and gaps.

Recording and exit policy. --deadlock-evidence (NERVIX_DEADLOCK_EVIDENCE) selects an existing local directory; without it the run uses standard error. The run first records empty evidence and atomically replaces its own file after each finding. The current header is NVXDLEVD, evidence kind 1, format version 3; unsupported versions fail clearly. The artifact holds at most 16 records, with one slot reserved for retention loss after at most 15 distinct potential cycles. Each cycle retains at most 64 edges and each text at most 512 UTF-8 bytes, remembering truncation. An artifact is at most 128 MiB. Descriptions have a 128 MiB per-process budget; repetitions with unchanged source context update the artifact without repeating the description. Changed context stays visible. Exceeding a bound retains loss and cannot qualify.

An active cycle ends the process with status 3 after recording. Potential order continues with unreviewed evidence. Handoff/history/retention overload, failed recording or output, or a recording operation exceeding its ten-second budget ends with status 4. The deadline is cancelled after a successful potential recording; no detached deadline later kills a continuing workload. On expiry it exits directly: a diagnostic write to a blocked descriptor must not delay the deadline itself. Exit bypasses handlers, as for a forced exit. Ordinary logging, visualization and browser startup are unnecessary. Nothing uploads evidence, logs protected values, or uses source/lock identities as metric labels.

Local inspection and triage. The ordinary nervix-deadlock-report tool needs no detector or running node. Select and export complete current values, inspect the original indexes, and write a reviewed copy of the original process evidence:

just deadlock-report inspect /var/tmp/nervix-deadlocks/deadlock-PID-TIME.rkyv --source potential --export /var/tmp/potential.rkyv
just deadlock-report triage /var/tmp/nervix-deadlocks/deadlock-PID-TIME.rkyv --finding 0 --basis non-overlap --reason 'The owning operation serializes both paths before either lock is acquired.' --regression 'owner::tests::serialized_operations; retained run artifact' --output /var/tmp/reviewed.rkyv
just deadlock-report qualify /var/tmp/reviewed.rkyv

The placeholder artifact and regression names must be replaced with the observed run and a real retained regression. Review uses correction, non-overlap, shared-readers or lifecycle and requires a nonempty, untruncated reason and regression reference. A reader proof checks every recorded mode is shared; other proofs are explicit engineering assertions, not machine verification of a referenced test. A correction cites the fixed owner and the subsequent regression run. A non-overlap proof names the enforced serialization; a lifecycle proof establishes disjoint instance/operation lifetimes. A source line alone proves none of them. Triage keeps the complete original cycle. Missing/truncated context or active findings cannot receive a potential proof.

Qualification of whole-process evidence fails on any active, lost, incomplete or unreviewed finding. A selected export retains its selection scope and cannot qualify its source process, even when the filter yielded no findings. Loss always remains visible through a selection. The qualify command exits 5 for nonqualifying evidence and 4 for an invalid artifact or operation. It qualifies recorded evidence; the engineer also verifies workload completion and successful checks, the run selection, pending-callback limit, revision and declared coverage. Keep reviewed copies separate from files a live recorder replaces. The operator’s workload is not ended merely because an unreviewed historical cycle exists.

The diagnostic lane. just test-deloxide, just test-deloxide-order, just test-deloxide-stress and just test-deloxide-stress-restore run the lane of each selection, and CI’s deloxide job runs all four side by side for every pull request labeled deloxide, the label each change the Deloxide rule applies to carries. A selection is named after the diagnostic build feature it compiles, or extends that name and declares the feature when it runs workloads of the same build in a lane, budget and CI runner of their own; a build records one diagnostic selection in its evidence whichever selection runs it. An invocation runs in every selection unless the inventory names the selections that run it. The stress selection runs every probe, conformance check and owner test of the active-only selection, its stress-scenarios invocation of the one- and three-node lifecycle scenarios tagged @deloxide_stress, which drive client contract transactions and their rejected rollbacks, the drain of buffered branches before an alteration, interleaved branches with remote acknowledgement owners, a lost acknowledgement and a producer restart, session restarts that resume a producer and consumer, a domain restart that replaces its consumer, a memory-pressure pause of scheduled ingestors, a Roto function’s column builder, TLS rebinding of every HTTPS listener, batched inference over interleaved branches, a restarted voter’s ownership, and graceful shutdown with a slow cordon release and diagnostic server processes that stop gracefully, and its stress-paced-simulation invocation of the paced-driver scenarios with the stressed Rust driver. The restore selection runs the build’s probes and its stress-restore-scenarios invocation of the scenarios tagged @deloxide_stress_restore: backup capture and restore of WASM branches, Kafka offsets, deduplicator keys and windows, materialized relays and native metadata, complete restore generations, staging quota and reclamation, delayed restore coordinators, bounded multi-domain backup waits and their recovery, and the drain of a large-state WASM owner through bulk checkpoint streams. Every stressed scenario runs one at a time under its own stress. identity, with the scenario, examples and invariant of its active-only counterpart and the owners that counterpart reaches. tests/deloxide-inventory.toml is the lane’s bounded inventory. It registers every workload a selection runs, each with a stable identity, the invariant it owns, the selections that must run it and the coverage it declares: the disposable-process probes of nervix-deadlock, the tracked locks’ conformance checks of nervix-primitives, the diagnostic owner tests of the server library, and the tagged scenarios with the number of example runs each must make. The three @remote_ack_owners scenarios add four example runs across interleaved branches, a lost acknowledgement and producer restart; every selection exercises the tracked correlation and peer admission owner locks. The @shutdown_kafka_handoff scenario runs in every selection and reaches the ingestor quiesce control’s tracked locks while a stopping node closes intake, waits for its attached emitter’s handoff, and finishes admitted Kafka acknowledgements. The async gate wait, remote delivery and scheduling order remain outside the detector’s observation and have the Cucumber and chaos checks in Shutdown And Recovery. The @shutdown_leader_change scenario also runs in every selection and reaches the tracked entity-gate state while a stopping follower repeats the remaining handoff through a new leader. Its leadership observation, transport waits and shared shutdown deadline remain outside the detector’s observation. Both scenarios carry @deloxide_stress, with separate stress workload registrations that reach their owners under bounded scheduling disturbance. The inventory also bounds the lane: one real-time budget for a selection’s diagnostic compilation and execution after its prerequisites, a bound for every invocation, a stop grace period, the reserve the scenario binary keeps for its own teardown inside its invocation’s bound, and how many scenarios run at once, which is the same on every machine.

The fault harness shares its observation and release barriers through src/fault_injection/pause.rs. The barrier owns watch-backed reached, released and delivered state and the one-shot atomic claim; fault selection remains in the harness’s owning modules. These async observations introduce no tracked blocking-lock owner. Deloxide continues to measure the registered product locks, while the public scenarios and model checks exercise pause ordering at their applicable boundaries.

The registered @restarted_voter_observation workload reaches whole-cluster restart and automatic scheduling when the leader first hears a voter through a relayed heartbeat. Its test-only watch channels and concurrent fault map are untracked. The registered native metadata restore workload exercises distributed Kafka checkpoint replication, acknowledgement, restart and exact restoration above the resident replication message budget; its stream waits and assignment atomics retain their complementary checks.

For each invocation the lane builds the selection’s executable, lists what the build holds or reads the tagged scenarios from the feature files, and refuses a registered workload that the build no longer holds or holds ignored, a discovered probe, conformance check, owner test or tagged scenario that is not registered, a scenario whose example runs changed, and an invocation that selects nothing. It reports how many workloads it discovered, selected, executed and saw complete. Global detector configurations never share a process: each probe workload is a disposable child, each conformance check and each owner test installs its detector in a fresh process, and each scenario binary process installs once in main. Active detection runs its tagged scenarios together; order analysis starts a fresh process per feature and gives the six large materialized restore and four complete-generation restore examples their own processes. These restores each start several nodes and carry large state payloads, so per-example order histories stay within the detector and host budgets. Their order_tags in the inventory select and account for every example. Every diagnostic server and paced driver process installs its own detector. Each selection compiles one diagnostic build, deloxide-stress-restore the stress selection’s under its own name, and the diagnostic builds stay apart from ordinary, fuzz, Loom, Shuttle and Turmoil builds.

The conformance checks are the tracked adapters’ own: which locks a diagnostic build selects, that Debug formatting tries a lock and never waits for it, that a value moves in and out of a lock, the exact count of waiters a notification wakes, the refusal of a second installation, the registry’s record of a thread across its exit, and a tracked wait that proceeds once its lock is released. nervix-primitives sits below nervix-deadlock, which owns evidence files, so a conformance check records none. Its detector’s sink panics on any finding and a panicking sink aborts the process, so the lane ends such a check as signaled, with the finding in its retained output. just test-primitives-deloxide runs the same checks in one process for each selection, beside the other modes’ conformance.

Stress widens timing windows; it does not make untracked waits visible. A Tokio mutex, channel or Notify wait, a DashMap shard, dependency locks, an atomic publication and a wait on another node are invisible to the detector with or without disturbance, and a workload that stalls on one of them ends as timed-out, not as an active cycle. Those owners keep their Shuttle, Loom, Turmoil and Chaos evidence. A dependency’s own channels are invisible in the same way: closing a database waits for the storage engine’s background workers through the engine’s worker channel and active-worker count, with no tracked lock involved, so that close is held by the pinned engine’s own worker-stop tests and the store-close regression beside ApplicationStartup::close_stores, not by the lane. A stressed workload that fails is reproduced first as a focused failing test against its owner before the owner changes.

Every process the lane starts runs in a session of its own. A process that outlives its bound receives SIGTERM and, after the stop grace period, SIGKILL with its whole group. The lane is a child subreaper, so a process that left its group is found too, and anything still running after an invocation ends is killed and fails the lane. The lane names each ending:

A failed invocation’s evidence is qualified before the lane reports it, so a node that deadlocked inside a scenario is reported as the active cycle it recorded rather than as the step it failed. After each invocation that records evidence, every process observation must be present, completely written and qualify. nervix-deadlock-report qualify prints one evidence summary: line of counts per file, and the lane reports repeated deliveries of a retained potential cycle apart from findings lost to overload. The first failure ends the lane.

A potential cycle fails the lane. The lane qualifies only the current run’s whole-process evidence, and it keeps no ledger that approves a recurring cycle: lock identities are run-local, and a source line never stands in for a lock instance. A correction removes the order from the exercised workload, and the lane is its regression. A reviewed infeasibility proof qualifies the retained evidence of the run it reviews, for the owner’s investigation and record; it never qualifies a later run.

Each run keeps a fresh attempt directory below <build target>/test-deloxide/<selection>/, which no later run reuses or removes: every log, the probes’ artifacts, every process’s evidence, the description of every finding with its source sites, and lane.json. The record holds the revision and whether the tree was modified, the toolchain, Cargo and Deloxide’s locked version and source, the inventory’s digest, the selection, the features of every build and the bounds, and for every launch its exact command, working directory, environment additions, bound, ending, duration, discovery and accounting, with the qualification of every evidence file. just test-deloxide-replay <lane.json> <launch> runs one recorded launch again, exactly, in a fresh attempt. OS schedules and stress timing are not recorded, so a replay can take another interleaving; deterministic reproduction belongs to Shuttle and Loom.

just test-deloxide-qualification proves the supervision on the probe binary’s real processes, through the lane’s own launch, evidence and classification code. A two-thread deadlock, an evidence directory that cannot be written, an untracked wait that never ends, an aborted process and, in the order selection, an unreviewed serial inversion and retention overload must each fail the lane with its own class and status, keep their output and the evidence they could write, and leave no process behind. A consistent-order control must pass, and the recorded deadlock must replay to the same class.

Under just coverage-native-extras test-deloxide, test-deloxide-order, test-deloxide-stress or test-deloxide-stress-restore, the lane builds in the collector’s instrumented target of the feature its selection compiles, with instrumentation on workspace crates only, so dependencies such as wasmtime’s compiler keep the speed the product’s deadlines assume. It takes the report tool, CLI and shared client binding from the target its prerequisites built, starts every test executable through Cargo’s configured runner, and hands lane.json to the collector, which exports nothing unless the lane completed. CI’s deloxide job runs each selection this way, then the qualification, and uploads the completion records, the diagnostic LCOV reports and every attempt directory whatever the verdict. Diagnostic coverage stays out of ordinary coverage and its CRAP report. The Integration Test Lifecycle chapter owns the job’s bounds and reserves.

Applicability. [[owner]] records in the same inventory cover every source file in which the compiler finds a tracked blocking acquisition in a diagnostic configuration. Each names the registered workloads whose processes reach its tracked locks, the path the lane does not reach and why, or both. just validate-deloxide-applicability, part of just validate and CI’s checks job, reads the acquisition catalog just ratchet writes and fails on an owner without a record and on a record whose file no longer acquires a tracked lock. It reports how many owners registered workloads reach and how many record a path the lane does not reach. The catalog is the compiler’s resolution of the Deloxide adapters’ operations, not a search for lock names. Every diagnostic library and binary configuration enables testing, as the lane’s server builds do, so acquisitions in testing-only fault controls require owner records too. This includes the server’s scheduling, drain and executor-occupancy controls and the consensus storage fault controls. Ordinary and modeled configurations retain their own feature selections. A record is a reviewed declaration: the lane’s coverage report is evidence about it, not a proof of ownership. Format equality, malformed and out-of-bounds rejection, normalized-cycle deduplication and the summary’s counts are registered Bolero properties with the same assertion in ordinary and sanitizer execution.

The native backup-wait workloads reach the client’s pending-reply locks during multi-domain capture, bounded wait expiry and recovery under the same execution reference. They run on one- and three-node clusters in both selections and name the client request and exchange owners in the inventory. The detector observes their blocking reply-registration and cancellation locks. Async command and reconnect mutexes, physical deadlines and network scheduling remain outside its observation; the public scenarios and client-session regressions assert their wait, expiry, leader-change and recovery outcomes directly. The bounded native-client case runs inside the diagnostic harness and reaches expiry and recovery with tracked client locks. The CLI scenarios exercise diagnostic nodes around ordinary subprocesses, whose own locks remain untracked.

The lane’s state-store owner test starts its detector in a fresh process before constructing store or executor locks and runs the current generation regressions: cancellation before and after pointer publication, snapshot readers, queued checkpoint writers, corrupt and incomplete chunks, exact retry after reopen, stale authority, bounded cleanup across 600 checkpoints and a 40 MiB guest save admitted with a 2 MiB reservation. Its evidence is retained beside the probe and scenario evidence. The lane also exercises quota refusal and exact staging retry, partial chunks without receipts across reopen, reclamation with applying and future generations retained, published and initial state preservation, snapshot readers across deletion, and interrupted bounded reclamation resumed after restart. Active-generation owner checks also exercise ordinary and same-revision inline writes, replica installation, handoff, forced recovery, entity and stale-identity purge, queued writers, exact publication retries, referenced-revision cursors and cancelled active deletion across restart. Current-shape storage fixtures also qualify segmented replacement and incomplete unpublished revisions in both initial and restored namespaces, with retained readers and reopen. Retained readers keep complete deleted payloads; SST observations distinguish logical deletion from snapshot release and explicit compaction. Header copies stay bounded, while Fjall’s value size queries and database caches retain their separate allocation owner. The node-local sweeper takes the applied consensus read guard before the checkpoint installation barrier, matching staging and publication; both remain held through bounded deletion. The public quota scenarios retain active staging through completed sweeps and prove failed targets remain gated after reclamation and restart on one and three nodes. The selected scenarios run without retries in a binary built for the mode: in-process diagnostic nodes running a workload, real diagnostic server processes that stop gracefully with evidence that records a running detector and no findings, each on one and three nodes, the buffered client alterations with overlapping holds and failed drains or full-hold engagement, and the restore installation workloads. The latter exercise the blocking applied-state authority guard through staging failure, complete publication, runtime handle clearing and a delayed coordinator across leadership transfer and a successor’s START. The large-generation scenarios cover two 20 MiB saves and forty 1 MiB saves on one and three nodes, failure after durable publication, cluster restart, branch isolation, saved source offsets and positive reclamation after resumed checkpoints and the state purge from module rebinding. The three-node many-save case retains one replica per state and observes reclamation of both owner and replica chunks. The large branch-state scenario restores a 40 MiB deduplicator keyspace and a 40 MiB window beside 24 small branches on one node, through a failure after durable publication, a stopped restore and a resumed one. Once the resumed domain runs, the window persists as a segmented checkpoint under the installation barrier and reopens from it after a cluster restart. Its three-node example runs in the ordinary suite only. Restore workload setup sends durable mutations and activation through the Rust client, which recovers the same execution reference when leadership moves; the raw subscription session keeps its own row stream. Large-generation and staging archive captures use a two-minute command budget so positive capture waits accommodate concurrent diagnostic workloads. The three-node copy restore cordons its coordinator before planning, so every guest save uses the remote chunk transfer and the receiver’s quota-owned completed file. Those restore scenarios also run in the ordinary public suite. The memory-pressure scenarios take the node’s pause lock as the pause collects every registered quiesce control and as each starting ingestor reads the pause, on one and three nodes. The client consumer scenarios take the Rust client consumer’s state and parked-read locks in the scenario process as a consumer is restored after a session restart and closed by a domain restart. The Roto scenario takes the bool column builder’s lock in every push, push_null and finish of an admitted function call, for a selected, a dropped and an absent value. The TLS rebinding scenario replaces the HTTPS listener’s TLS configuration on every node as a TLS VHOST is created and rebound, while each connection the background HTTPS posts open reads it. The inferencer scenario takes a branch’s ONNX session lock for each batch of two interleaved branches. Each of the three runs on one and three nodes. On three nodes with a replica, the deduplicator, window and guest-save restore scenarios leave the owner’s lifecycle and branch checkpoint announcements with the replica’s lifecycle handle, where the replica task takes them under the same lock.

The ONNX inspection admission scenario saturates the leader’s bulk workers and wait queue, observes the typed CREATE INFERENCER refusal, releases the occupancy and creates the same inferencer on one and three nodes. Its paired stress registration reaches the occupancy holders mutex under bounded scheduling disturbance. Metadata inspection keeps ONNX’s operator concurrency at one, so loading and optimization run on the bulk worker the executor admitted. Debug phase events distinguish worker queue delay, session initialization and release while the thirty-second product bound stays in place. The dependency’s own locks and threads remain outside Deloxide; the isolated native owner regression measures the live temporary session’s process thread count, and the public branch-batch scenario retains its exact interleaved-branch output oracle.

Shuttle checks the production handoff’s delivery/refusal and close races, including counter drain, through the registered replay-capable harness, in the Shuttle build where no detector is installed. The queue internals and OS wakeups are opaque; the relaxed refusal counter publishes no other location. OnceLock publishes the installation port; no new independent cross-location atomic protocol is introduced. Existing primitive Loom conformance remains separate. The lane’s supervisor runs outside every diagnostic process: it adds no synchronization to a process it observes and reads only the processes’ exits and the evidence files they wrote. Deloxide models no async or network scheduling. Tokio locks, channels, Notify, DashMap shards, dependency locks, atomic protocols, missing notifications and cross-node waits remain outside its detection claims and retain Shuttle/Loom/Turmoil/Chaos checks. No diagnostic check proves safety for an untracked path or an order the workload did not take.

Diagnostic images under external chaos. The lane runs diagnostic nodes in a scenario binary or as local processes; the external chaos suite runs them as packaged containers under real process, pause and network faults. The diagnostic target of Dockerfile.debian builds the server of one selection, NERVIX_DIAGNOSTIC_SELECTION, with the ordinary CLI and formatter and the nervix-deadlock-report tool, on the release image’s CPU runtime. It declares the selection in its io.nervix.diagnostic.selection label, and every image, release or diagnostic, declares the revision it was built from in org.opencontainers.image.revision. The release targets copy their binaries from the builder’s release artifacts and the diagnostic target from its diagnostic ones, so a build of one kind cannot produce an image of the other, and the diagnostic image never enters a published manifest. just docker-build-diagnostic builds it; CI’s build-diagnostic job builds the deloxide-order image beside the release images whenever a chaos suite runs, and the suites run against both. The chaos runner reads the label: every node records its evidence in a directory of the run, the run freezes the nodes once its Docker event recording is closed, and the image’s own report tool qualifies every file before cleanup. A run fails unless each node holds one complete, qualifying evidence file of the declared selection for every process start the recording shows, and no node ended with status 3 or 4; an unreviewed potential cycle fails it as in the lane. The suite refuses an image whose labels name another kind or revision than its caller asks for. scripts/chaos/README.md owns the runner’s contract and artifacts. The evidence observes each process up to its kill, pause, stop or the freeze, without an upstream flush barrier, and only its tracked thread-blocking locks; recovery, ledger and timing verdicts stay the scenarios’ own, with the product’s deadlines and the suite’s budgets unchanged.

Deterministic Concurrency Verification

The ordering contracts above are checked against the production owners under Shuttle. A check runs several tasks or threads through one scheduler, which chooses an interleaving at visible synchronization points. It can expose a lost notification, an early drain, a double completion, or a stale generation without depending on which OS thread happened to run first. A deadlock is a failed check. The model is the surrounding schedule and test data; the protocol under test is the same type used by the data plane, not a copied implementation of it.

Shuttle does not model elapsed time. Under Shuttle the boundary’s time family is Shuttle’s: a sleep yields once, and a timeout does not measure its deadline and fires only when a check triggers it by task label through nervix_primitives::time::trigger_timeouts. Instant::now still reads the operating system’s monotonic clock, while paused-time controls do not advance a simulated clock. Checks of deadline ordering therefore use an already-passed or far-future instant and explicitly choose whether the timeout wins. Ordinary tests on a paused Tokio clock retain responsibility for actual timer behavior. No concurrency check uses a wall-clock bound, sleep poll, or recv_timeout to establish progress.

Only scheduler-visible operations create interleavings. In a Shuttle build the primitive boundary selects every family from Shuttle or from its own adapters, as the backend matrix shows, and every package that owns the feature forwards it to the boundary, so the synchronization inside vocabulary types such as AtomicTimestamp, inside dependencies such as the execution crate’s cancellation, and inside every connector is Shuttle’s too, and so are the timers of every crate in the graph. The feature changes the test execution environment, not the public protocol. Edge I/O, sockets, filesystem access, and signals remain real and are outside a Shuttle schedule. A Shuttle execution has no Tokio reactor, so a socket created inside a check panics and fails it rather than reaching the network unobserved. A check’s own records use unmodeled atomics on purpose, under the unmodeled permissions, so that recording an operation adds no scheduling point to it.

Some primitives are opaque to Shuttle: arc-swap, concurrent-queue and the futures crate’s atomic waker have no modeled implementation. The boundary runs each ArcSwap and ArcSwapOption load and store, each ArcSwap compare-and-swap and read-copy-update, each operation of a lock-free queue, each registration, wake and take of an atomic waker, and each read and write of a OnceLock between two scheduling points, and calls the primitive directly in ordinary execution. Shared ownership through triomphe or the standard library takes no scheduling point: a reference count is real in every mode. The boundary’s thread module supplies the scheduler-visible synchronous yield that admission spin waits use. A check may claim an ordering around an opaque primitive only when its relevant calls have visible scheduling points. Shuttle cannot interrupt an arbitrary instruction inside it, and a check never claims the primitive’s own memory safety.

A lost wakeup lives between a waiter’s read of some state and its registration for the notification a publisher sends after changing that state. Shuttle’s own Notify keeps its waiter registration behind a real lock and its watch channel reads its version without a scheduling point, so no schedule could place a publication in that window, and a read followed by a registration looked safe in a check even though a release between the two is lost in production. The boundary’s Notify and watch channel keep Tokio’s semantics and take a scheduling point immediately before each registration, each notification and each read of the version that registers or decides, and the atomic waker’s adapter takes one before and after each registration and wake. Tokio’s Notify registers a future for notify_waiters when the future is created, and for notify_one only when it is first polled or enabled; notify_one without a registered waiter stores one permit; a waiter a single notification chose passes it on when it is dropped before it observed it. A watch receiver marks the current version as seen when it subscribes. The conformance checks of crates/primitives/src/tests run the same scripts against Tokio and against the adapters, and the Shuttle checks there show that a publication between a read and the registration is reached: each broken order deadlocks in some explored schedule and the correct one never does. The ownership-handoff checks separately exercise release-before-wake and register-before-read on the production freeze owner. Reversing the latter order deadlocks under both random and PCT schedules, and the failing schedule replays.

Shuttle explores sequentially consistent schedules. It cannot prove that a chosen Relaxed, Acquire, or Release ordering is sufficient on weak-memory hardware; a claim that depends on one is a Loom model, described under Memory-ordering models. Cucumber remains the public behavior test: when a scheduling defect affects an NSPL operation, runtime output, or process outcome, its Shuttle regression is paired with a scenario through that interface. A scenario cannot exhaust the interleavings of an in-process protocol, and neither a Shuttle check nor a Loom model can verify the whole cluster, socket, disk, or browser path.

Check contract and runner

Each check names the invariant in its test name, drives competing operations on the production owner, and asserts the state at meaningful transitions or after all participants have joined. Use an explicit handshake or scheduler-visible yield to position a race. When both the decision read and waiter registration are scheduler-visible, a missed notification leaves the waiter pending and Shuttle reports the resulting deadlock. Name a bounded number of participants so the search remains reviewable; use bounded depth-first search for small races and random plus probabilistic concurrency testing (PCT) for larger ones.

Every check runs through the one Shuttle runner, nervix_model_harness::shuttle: 100 random and 100 PCT schedules of depth three, 1,000 of each with a depth-first search bounded at 1,000 schedules, or one scheduler with the check’s own bounds. Every schedule may take 10,000 steps; one that needs more fails its check, because a step cap is an exploration bound, not a product timeout. A random or PCT search that ran fewer schedules than it declares fails too, and a search that finishes prints a record naming what it explored. SHUTTLE_REPORT_STEPS=1 reports the highest observed step count when tuning a check.

The primitive boundary’s own conformance and race checks run in just test-primitives, which builds the boundary once per mode; its Shuttle checks explore their small models exhaustively with depth-first search, and each deliberately broken order must deadlock in some schedule.

crates/model-harness/shuttle-inventory.toml registers every check of nervix-execution, nervix-interconnect, nervix-client-core, nervix-connector, nervix-consensus, nervix-server and nervix-deadlock by its test’s full name. just test-shuttle lists the library tests whose full names contain shuttle_ in each registered package, built with its shuttle feature, and a run over the whole inventory fails when a registered check is missing or ignored, or when a discovered check is unregistered. Each selected check runs twice, each time in its own process: under the exploration it declares, then under Shuttle’s uncontrolled-nondeterminism detector, which runs 100 random schedules twice each and fails when the second run diverges. A run counts only when its test passed and the runner printed its record, and the command reports how many checks it discovered, selected, executed and saw complete. just test-shuttle <filter> selects the checks whose full name contains the filter in every package; a package with no match is fine, and a filter that selects nothing at all fails. The recipe uses the repository’s kache-backed build and prepares the server’s test dependencies; just test continues to run the ordinary suite. just cargo-clippy-shuttle, whose package checks also run in just lint, lints every Shuttle build with warnings denied, including every inventory package in test mode, where its checks are compiled, so a check that compiles with a warning fails validation.

A failed check leaves target/shuttle-failures/<package>/<fully-qualified-test-name>/: the schedule Shuttle persisted for the failing execution, the run’s output, and metadata naming the check, the run, the revision, the toolchain, Shuttle’s version and the replay command. just test-shuttle-replay <schedule> replays exactly that check in a fresh process with the schedule, which ends where its execution failed and so replays that failure. just test-shuttle-replay-check proves the path end to end: it fails one check deliberately after its invariant held, requires one persisted schedule, and requires it to reproduce the failure in a fresh process. CI’s dedicated shuttle job runs both, only for a pull request labeled shuttle, and uploads the failure directory in a tar archive as the shuttle-failures artifact when a check fails; fully qualified test names contain colons, which the artifact uploader rejects as paths. See the command recipes in Developing Nervix.

Protocols and their checks

The checks below hold the scheduler-visible parts of each protocol to their invariants. Test names are given relative to their owning module; the shuttle feature selects the modeled build. A family of names means each member runs independently through the recipe.

The checks of WASM checkpoint holds and the durability barrier use the same runner and replay contract. Among them, shuttle_an_abandoned_writers_synchronization_keeps_the_barrier_until_it_ends and shuttle_an_abandoned_writers_failed_synchronization_still_refuses_what_it_covered hold a synchronization round to the storage job that runs it: a writer that stops waiting frees neither the barrier nor the round’s failure. Their state semantics live in the WASM state documentation; they do not turn Shuttle into a disk or replica simulator. Deterministic interconnect simulation and Cucumber cover the network and process behavior outside this in-process scheduling boundary.

shuttle_a_vocabulary_atomic_is_a_scheduling_point_in_the_server_build (src/shuttle_selection_tests.rs) holds the selection itself to account: in the server’s Shuttle build, an operation on the vocabulary crate’s AtomicTimestamp must advance Shuttle’s scheduling-point count, which it can only do when the feature reached that dependency’s atomics.

Memory-ordering models

A claim that one thread’s writes are visible to another because of an ordering, rather than because both threads were scheduled in some order, is checked under Loom. Loom runs a model once for every schedule and every reordering the C11 memory model allows that it can distinguish, so a Release weakened to Relaxed shows up as an execution in which a reader sees the flag but not the write it was meant to publish. A model drives the actual synchronous production owner from Loom threads with Loom’s atomics, both selected through the primitive boundary by the package’s loom feature. When a protocol is embedded in asynchronous orchestration, the synchronous protocol is made independently testable and the runtime uses that same owner; a copied algorithm, a witness that a join or an extra lock publishes, or a real atomic does not make a model.

The execution, interconnect, consensus and server crates own a loom feature, and each forwards it to the primitive crate and to every dependency that owns one, so the whole library graph of each builds with Loom’s primitives. just cargo-clippy-loom, whose package checks also run in just lint, lints every Loom build: the models and their harness, the primitive boundary, and the server and consensus libraries both as they ship and in test mode, where models of their owners are compiled.

Each model names its invariant with an InvariantId and runs through nervix_model_harness::loom::explore, which explores it to exhaustion: no preemption bound, no permutation or time budget, a branch limit of 1,000 thread switches per execution that fails the model rather than ending the search, and Loom’s full thread count. A Loom setting in the environment that would change that search is refused. A completed search prints a record naming its invariant, its execution count and its bounds, and just test-loom accepts nothing else as a completed model.

Discovery, model execution, qualification and checkpoint replay use the workspace’s loom build profile. It inherits dev, retains debug assertions, overflow checks, debug information and the server’s allocation instrumentation, and compiles with optimization level one to reduce the allocation frames on Loom’s fixed coordinator stack. Qualification cleans only this profile’s package artifacts before and after each weakening, so a weakened artifact cannot become an ordinary build input or satisfy another qualification.

The coordinator starts and joins the model body without touching protocol state. The body and participants started through nervix_model_harness::loom::spawn each request a 1 MiB coroutine stack for instrumented allocators and debug frames. The coordinator retains Loom’s fixed default stack; its initial allocation and spawn path must fit before the larger body stack can run. Loom forwards that request to its coroutine implementation, whose allocation units need not be bytes. Its five thread slots include the coordinator, body and at most three other participants. The harness stack model executes a 64 KiB frame on both the body and a participant; reducing the stack request must reproduce the overflow and replay it from its checkpoint. Scheduling and memory-ordering checks still explore to exhaustion.

crates/model-harness/loom-inventory.toml registers every model by invariant. just test-loom lists the loom_* library tests of every registered package built with its loom feature, and a run over the whole inventory fails when a registered invariant’s test is missing, ignored or did not complete, or when a discovered model is unregistered. Each model runs in its own process, and the command reports how many models it discovered, selected, executed and saw complete; a filter that selects none fails. A failed model leaves target/loom-failures/<package>/<invariant>/: the Loom checkpoint of the failed execution, the run’s output, and metadata naming the invariant, revision, toolchain, Loom version and exploration bounds. just test-loom-replay resumes Loom from that checkpoint with location tracking and tracing, so the failed execution runs first. The artifacts hold model output only, never payloads or secrets. For a pull request labeled loom, CI runs the models in its dedicated loom job, which uploads the failure directory, and their qualification in the loom-qualification jobs. Invariant IDs use ASCII letters, digits, dots and hyphens, so these paths are valid artifact names; metadata retains the exact Rust test name used for replay.

Every model also registers a weakening that must make it fail. just test-loom-qualification applies each to a copy of the working tree, requires the model to fail with the registered message, and requires the checkpoint of that failure to replay it. This is what shows a model depends on the ordering it claims, rather than passing because something else synchronized its threads. A weakening whose original text no longer appears exactly once fails as well, so changing an owner’s ordering means revisiting its qualification.

Successful qualifications retain target/loom-qualification/<qualification-id>/ as well: the counterexample checkpoint, the checkpoint advanced during replay, both outputs and metadata with the revision, toolchain, exploration bounds and complete registered weakening. CI uploads these records whether the qualification job passes or fails. To replay a retained counterexample, restore its recorded revision, apply its recorded weakening in an isolated worktree and run just test-loom-replay <directory>; replay against the unweakened owner is a different check. The isolated build copy is still removed after qualification, and an unsuccessful control retains its output and any checkpoint it produced.

Loom models a SeqCst access as an acquire or a release, and models only fence(SeqCst) exactly. A protocol in which each side writes one location and then reads the other, such as a dispatch that raises the in-flight count before it reads the closed flag while a fence closes the gate before it reads the count, is therefore written so that its correctness follows from acquire and release alone. The side off the hot path reads the other side’s count with a read-modify-write, which reads the newest value in that count’s modification order: either the hot side’s read-modify-write came first and the cold side’s read sees it, or the cold side’s came first and the hot side’s read-modify-write acquires it, and with it the write the cold side made before. That is exact in the C11 model and checkable by Loom, and it costs the hot path nothing. The qualification of such a model weakens the cold side’s read-modify-write to a load, which is the store-buffering fault the protocol rules out.

Loom’s own limits bound every claim. It does not model every relaxed behavior the C11 model permits, and an operation inside a third-party dependency, such as a triomphe reference count or an arc-swap publication, is invisible to it and excluded from the claim rather than given a fictional model. The client batch admission fence reads its quiesce decision from such a publication, so its model checks the root counts the admission and the drain reach by read-modify-write, and the publication’s own ordering stays outside the claim. A relay’s subscription definition is another: an attachment and a redefinition each put a sequentially consistent fence between their own write and the read of the other’s, but the attachment’s write is its registration in the fan-out’s consumer list, an arc-swap publication, so the protocol has no Loom model and its Shuttle check orders the two. The state store’s durability barrier orders each writer’s ticket after the write it covers and each synchronization’s start before its flush, but those writes and the flush are the database’s, inside its own synchronization, so it has no Loom model either; its Shuttle checks hold which tickets a synchronization covers. A standalone counter carries no cross-location claim, whatever its ordering. Relay branch presence is such a case: its owner lifetimes and publications are arc-swap compare-and-swap and read-copy-update operations with no Nervix-owned atomic beside them, so it has no Loom model; its Shuttle checks order its publications against observers and successors. Checkpoint replication is another: its announcement and progress change under one placement’s lock, and its wake-ups are the boundary’s Notify, with no Nervix-owned atomic beside them, so it has no Loom model either; its Shuttle checks order an announcer’s end against offers, and a wait’s registration against reports and announcements.

The cancellation protocol these models check is the bounded executor’s. Cancellation::armed creates both ends of one job’s cancellation: the executor keeps the obligation while its caller awaits the job and moves the signal into the job, which checks it between its bounded units. Dropping the awaiting caller drops the obligation and cancels the job, which keeps its memory charge until it returns; observing the job’s value disarms the obligation first, so an ordinary completion never reports itself as cancelled. The executor keeps admission, charges and cancellation policy; the protocol it runs is the one the models explore.