SIMD Kernels

Nervix applies explicit SIMD to contiguous typed buffers where generated-instruction inspection and measurement justify it. The optimization includes the caller: decoding into Arrow builders, resolving a column once, retaining bitmaps, or folding a whole run removes the per-row structural work that would otherwise hide a faster arithmetic loop.

This chapter owns the kernel boundary, dispatch, result guarantees, caller catalog, qualification, and measured evidence. Data Plane owns batches, branches, persistence, and ACKs; VM Functions owns expression execution and window structures; Domain Clock owns time and admission; Data-Plane Concurrency owns synchronization and execution modes. Schemas And Codecs, Syslog, and Connector Crates And The Connector Contract retain their wire contracts.

Ownership And The Admission Rule

nervix-simd-kernels, in crates/simd-kernels, belongs to the primitives layer, below the vocabulary. It owns portable operations on slices, offsets, validity bytes, and bitmask words. It depends on fearless_simd, simdutf8, pointer-width conversions, and self-contained error values. It must not know Arrow, Models, schemas, domain clocks, codecs, metric recorders, connectors, branches, or execution graphs. A nanosecond buffer is an integer buffer here; the caller gives those integers their meaning.

A new explicit kernel needs all three:

1. Inspection of the current caller’s optimized code shows scalar or lane-extracting work. A source loop, a batch API, or an operation’s name is insufficient evidence.
2. Every supported level and the forced fallback reproduce the complete scalar result, including failure bits, null masking at the caller, and unused tail bits. Integer, bitmap, and byte operations are exact. Window floating reductions follow the run contract below.
3. The caller supplies a typed contiguous slice or bitmap. Per-row boxing, field lookup, virtual type selection, or reconstruction from values already held in columns must be resolved first.

Kernel ownership does not grant clock, scheduling, synchronization, or allocation policy. The caller retains its bounded-executor admission, cancellation between bounded units, memory charges, and branch-local state. A kernel neither reads a clock nor starts a task. Its output is transient values and masks, never a new persisted execution representation.

The repository’s cast denial, checked arithmetic, typed errors, ownership headers, and debt ratchets apply here. Source-owned compiler diagnostics and narrow reason-bearing expectations govern architecture rules; a generated report or an approval count grants no exception.

Dispatch And Build Targets

The crate caches one fearless_simd::Level::new() result in a process-wide OnceLock from nervix_primitives::unmodeled::sync. The exact-file permission explains why CPU capability detection may outlive individual model executions. It carries no protocol state or ordering that a concurrency check relies on. Later calls reuse that level; short byte searches can return before resolving it at all.

dispatch! enters the selected target-feature context. Small #[inline(always)] functions generic over S: Simd perform the buffer loop there, using simd.vectorize where required. The dispatch expression returns values or masks. Fallible validation and contextual errors stay outside it; a dispatch body does not use ? or return from its enclosing caller. Callers name operations and types, and do not choose an instruction set or use std::arch themselves.

The workspace enables force_support_fallback so tests can explicitly request Level::fallback() even on SIMD hosts. This makes the fallback implementation reachable; it does not make an AVX2 binary runnable on a CPU below that binary’s baseline.

The effective compiler flags matter. A native build’s nominal SSE or AVX2 test token does not prove that its lower backend executed. The production-target recipes build separate x86-64-v3 artifacts, allowing an AVX-512 host to exercise AVX2 as well as its higher runtime arm. SSE and non-x86 qualification require builds and hosts that expose those backends. Current instruction and timing evidence below is x86 evidence; it establishes no AArch64 or WASM timing.

Results, Failures, And Bounds

Checked arithmetic returns the wrapped lane value together with its failure word, matching overflowing_*. The VM intersects failures with operand validity, zeroes failed values, makes them null, and constructs errors only for set bits. Null lanes cannot produce a message error, including when their underlying value buffers contain overflowing operands. Scalar-null operands produce an all-null result. Sliced arrays retain their value span and bitmap offset.

Integer sums, counts, comparisons, byte classes, and masks require exact scalar equality. Floating window admission uses eight compensated accumulation lanes, two-pass centered moments, and one Chan merge per run. This is a typed run algorithm, not a claim that every float loop has an explicit fearless_simd backend. Different batch/run partitions can round differently under the public window contract; changing VM float association is outside this boundary. Retraction refolds survivors rather than subtracting floating statistics. Window ownership, snapshot restore, and branch isolation remain defined by Window Aggregates And Sketches.

When a sum of finite operands overflows, the mean is computed from compensated operands divided by their largest contributing magnitude, then rescaled. Clamping the normalized mean to [-1, 1] keeps it finite at the representable limit. Constant finite runs retain their exact mean and zero centered moments in this path; ordinary finite sums keep the existing compensated division.

Output storage scales with the input run, or with a validated fixed histogram layout. Local padding, lane reductions, and block masks have bounded sizes. Tails are masked before publication; a padding value is never an extra admitted row or failure. Byte scanning uses overlapping loads inside the existing buffer rather than allocating padded tails. Unsafe conversion is confined to XmlChars::into_string: the spans accepted by simdutf8 partition the complete buffer, so joining them preserves UTF-8 and permits String::from_utf8_unchecked. Character rejection remains a typed result, and invalid UTF-8 takes precedence over an excluded XML character.

Kernel And Caller Catalog

The catalog includes library-dispatched operations and typed run algorithms so their contribution is not mistaken for an explicit vector kernel. The callers retain schema validation, sensitivity, route errors, ACK attribution, and branch ownership.

JSON Boundaries

Decode copies a borrowed payload into reusable writable scratch because simd-json mutates its input. It builds no intermediate serde JSON tree or materialized row map. Schema compilation resolves field keys and Arrow types; datetime text is parsed once and base64 chunks append to the binary builder. Exact integer shape/range, strict unknown-key rejection, nullability, invalid UTF-8, and document limits remain codec decisions. Workspace serde_json::preserve_order remains for JAQ’s public object-member-order contract, independently of schemaful decoding.

Encode classifies quotes, backslashes, and controls below 0x20 over each string values buffer in 64-byte blocks, then summarizes offset pairs into row masks. Non-ASCII UTF-8 remains verbatim. Clean strings copy directly; marked rows enter escaping. Numbers use itoa/ryu; datetime and base64 text write without another escape scan. Sensitivity redaction, nested values, ClickHouse’s widened F32 rendering, bounded-write refusal, and frozen retry bytes remain the owning encoder’s contract. The shared byte block classifier also implements the JSON escape class.

Latency And Domain Admission

Relay metadata carries low/high ingestion watermarks as Arrow-compatible nanosecond buffers. The delivery fold computes the latest watermark, excludes timestamps after the acceptance instant, rounds elapsed time to milliseconds with halves up, clamps at 30 seconds, and counts HDR buckets. The histogram layout validates the bounds under which floating lane division reproduces the integer rounding exactly. Scalar bucket scatter is intentional. Each retained series merges the fold into its four rolling windows under one lock with one wall-clock read, then flushes its local Prometheus histogram once. The domain acceptance instant and wall-clock window maintenance remain separate clock responsibilities.

Admission uses the first and last reached centers with inclusive skew. Ordered signed timestamp differences become exact unsigned distances, including across the signed extrema. Interior rows test their distance to the nearest period boundary. A window with no gaps needs no remainder; powers of two use masks; other periods prepare a floor reciprocal and correct its estimate once. The single-timestamp vocabulary check has the same semantics. A rejected or missing timestamp affects its row alone, with the existing validation/admit error and ACK attribution. The kernel does not establish which centers are eligible; Admission Windows defines that clock contract.

Window Runs And Bitmap Work

Validity readers carry a bitmap and its bit offset and count present rows by popcount. Boolean counts combine value and validity words. Signed/unsigned 64-bit sums accumulate low and high 32-bit halves in vector lanes, periodically reducing them to exact i128 before lane overflow. Smaller integer sums and generic min/max use typed scalar folds. Non-finite F32/F64 inputs produce column bitmaps. Bucket-index calculation and subsequent histogram scatter remain typed scalar work. Sketch admission reuses a run key buffer; BLAKE3, t-digest insertion, and candidate updates are not turned into lane-wise kernels.

Failure packing compares failure bytes with zero and extracts masks per register. Valid-lane-only VM operations read validity by 64-lane words: empty words do no work, full words use the ordinary loop, and mixed words evaluate only present lanes. One packing call covers the block, with clear bits past its end. Explicit checked arithmetic bypasses the intermediate failure bytes entirely. Selections use bitmap operations and Arrow filtering; required output columns with no nulls take a fast path. These operations remove repeated per-row mask reconstruction, without changing which row is selected or reported as failed.

Checked Arithmetic And Constant Division

Sums and differences wrap at their own width. Signed operand/result signs determine overflow; unsigned carry and borrow comparisons determine it. Products of 8-, 16-, and 32-bit operands widen to twice the width, compare against the original bounds, and narrow by truncation. A 64-bit checked product remains scalar because no wider exact lane is provided.

For a nonzero non-power-of-two divisor d, preparation computes m = floor(2^64 / d) once. Discarding the low 64-w bits yields floor(2^w / d) for width w. The high product gives a quotient at most one below the exact quotient, and one remainder comparison corrects it. Powers of two use a shift and mask. Signed division operates on unsigned magnitudes, including MIN, before sign restoration. Truncating numeric division and Euclidean datetime division keep their distinct rounding. Zero fails valid lanes; MIN / -1 fails, while the numeric remainder contract keeps MIN % -1 equal to zero.

Binning normalizes the value and origin, corrects their phase difference by at most one stride, and checks subtraction of the distance to the bin start. Calendar and zoned operations retain their calendar rules. Divisor preparation may execute wide division once; selected reciprocal lane bodies contain no integer division instruction. The exact wide date_diff fallback still divides in i128.

Byte Classes And Text Codecs

A ByteClass supplies an ordinary inclusive byte range, excluded bytes outside it, and listed bytes inside it. A byte is a member when it lies outside the range without being excluded, or is listed. Constants become splatted vector comparisons. Each excluded/listed list is bounded by MAX_NAMED_BYTES at compile time; supported examples and a paired compile_fail doctest exercise that current API restriction.

first_in stops at the first block holding a member. ByteScanner retains the current block’s mask for successive forward positions, classifying each block at most once. A final partial block uses the last 64 bytes of a sufficiently long buffer and shifts away the overlap. Buffers shorter than a block use 16-byte registers and an overlapping final load. first_in below 16 bytes uses the scalar loop before resolving the level. No padded-tail copy is needed.

XmlChars checks the XML 1.0 Char ranges: tab, LF, CR, U+0020–U+D7FF, U+E000–U+FFFD, and U+10000–U+10FFFF. The owned-byte path validates UTF-8 and classifies bounded 4 KiB spans, carrying sequences across classification boundaries and adjusting validation boundaries to character starts. Once a character is excluded it still validates the remaining bytes, because invalid UTF-8 outranks character rejection. An already valid &str needs only character classification. Syslog framing’s memchr, UTF-8 validation’s simdutf8, and OTEL’s faster-hex choose their own SIMD; the host adds no competing dispatch layer for their internals.

Qualification

Tests compare every level available to their build/host, the baseline, and the forced fallback with a complete scalar oracle. They cover empty runs, short inputs, every tail length, word/block boundaries, sliced validity, nulls, shared operands, signed extrema, and unused bits. Byte arithmetic and division include exhaustive operand pairs. Float-window tests check the documented run algorithm and numerical bounds; they do not assert bit equality across different partitions.

Use the repository recipes and retain its kache compiler wrapper:

just test-simd-kernels
just test-vm
just test-bolero simd-
just build-simd-kernels-x86-64-v3
just build-vm-bench-x86-64-v3

Registered Bolero targets use the same production path and complete assertion in ordinary randomized/corpus runs and sanitizer-backed libFuzzer. Current targets include simd-checked-lanes, simd-constant-division, simd-byte-classes, simd-xml-chars, simd-json-escapes, simd-paced-admission, simd-elapsed-histograms, simd-flag-packing, and simd-window-runs and simd-window-moments; syslog-stream-framing, syslog-structured-data, and columnar-json-rows cover the connector/codec consumers. Elapsed buckets are compared with HdrHistogram’s lowest equivalent values, and float window sums with an exact sum within the compensated-summation bound.

The moment reference uses exact addition partials and fused-product residuals for runs of at most 128 finite values with exponents from −200 through 200. Counts and histogram placement are exact. For ε = f64::EPSILON, n = max(count, 1), input absolute sum S, and reference mean μ, its mean error bound is B = 4ε|μ| + [2(n + 16)ε]² S/n. The M2 bound is 16(n + 16)ε M2 + 4n B²; the cross-moment bound is 32(n + 16)ε Σ|centered products| + 4n B₁B₂. Independent sliced null masks select paired contributions. Separate constant runs at 1e308, -1e308, and f64::MAX check that overflow of an intermediate sum preserves the exact constant mean and zero centered moments. Histogram cases check each edge, both clamps and null positions. Every applicable property must remain registered in tests/bolero-targets.toml. Ordinary runs remain required on PRs; sanitizer CI runs only for PRs labeled fuzz. Scheduled, workflow_dispatch, and unlabeled PR runs expect a sanitizer skip. A skip is neither execution nor coverage evidence. Explicit qualification uses just fuzz <target> <seconds> or just fuzz-all <seconds> and records the actual inventory executed. See Property Testing And Fuzzing.

Current Rust API restrictions use source-adjacent compile_fail doctests paired with compiling examples of supported imports and signatures, run in normal tests and CI. Retain compiler-resolved lint fixtures, feature/dependency/target checks, public scenarios, and model tests under their owners. Use LLVM tools matching the producing compiler, as declared by its toolchain.

Public behavior retains Cucumber evidence on one and three nodes: exact JSON ingestion/emission, batch latency metrics, reached admission-window edges, interleaved branch window aggregates, sparse/dense/empty filters with row errors, checked numeric and datetime errors, split Syslog TCP/TLS frames, SQS noncharacters, and OTEL ingestion. Kernel equivalence does not establish ACK, branch, transport, or recovery semantics. Synchronization changes additionally need the evidence required by Data-Plane Concurrency; a pure buffer kernel introduces no claim about concurrent ordering.

Measurement And Recorded Evidence

Inspect optimized instructions for the actual payload target, using nm -C and objdump -d -C on the binaries the production-target recipes build. Check every operand shape and width, including tails and fallback. A source abstraction’s scalar implementation can be folded back into vector instructions by LLVM; conversely, an early-exit loop can remain scalar even when the target supports AVX2. Instruction evidence must name the revision, toolchain, flags, and host.

Criterion compares a kernel with the loop it replaces on identical operands in one process. Include scalar/shared operands, null/error density, short inputs, and divisor preparation and allocation where the caller pays them. Pass reference operations as inlinable closures: a function pointer per lane measures indirect-call overhead. A byte scan’s reference must preserve its early exit; removing that exit can let LLVM vectorize a different algorithm.

just bench-json-encode
just bench-relay-interaction
just bench-window-admission
just bench-checked-lanes-x86-64-v3
just bench-constant-division-x86-64-v3
just bench-byte-classes-x86-64-v3
just bench-syslog-framing
just bench-vm numeric_kernels
just bench-vm datetime_kernels
just benchmark-ab <baseline-ref> 3 <workload>

An application-throughput claim requires serialized same-host just benchmark-ab, exact output parity, and equivalent configuration. Pin timing runs to one core on mixed-core hosts and interleave baseline/candidate rounds. Report intervals and interference; a bounded backlog rate does not establish maximum sustainable throughput. Store new raw logs, benchmark outputs, disassembly, and validation reports on the corresponding ClickUp task, rather than committing transient artifacts. The summaries below are architectural evidence, not fresh measurements.

The arithmetic inspection corrected the initial assumption about I64: every signed sum and difference was a scalar flag-per-lane loop in the inspected x86-64-v3 VM, while unsigned sums and differences were already vectorized. Own-width overflow checks also measured faster than widening sums (native i8: 93 versus 161 ns; i32: 171 versus 247 ns per 1,024 lanes). Unsigned operations now share the checked-arithmetic owner, with parity or a measured improvement; their presence is not evidence that LLVM previously left them scalar.

Constant-division production-target experiments measured I64 quotient at 1,302 ns in vector lanes versus 1,745 ns with scalar reciprocal, but U64 quotient at 856 versus 788 ns and U64 remainder at 877 versus 797 ns. A repeated I64 remainder comparison measured 1,685 versus 1,534 ns. These 1,024-lane comparisons explain the retained scalar operations. Separately, VM truncation/binning measured 20.005 → 14.348 µs, fixed elapsed arithmetic 9.546 → 7.329 µs, and numeric constant programs about 26–68% less time. Combined to_unix/from_unix remained unchanged; the latter is multiplication. Unchanged controls also moved under shared-host load, so these are kernel/VM observations, not application-throughput guarantees.

Text-scanning qualification found that a padded-tail copy compiled into memset/memcpy and made an 8-byte header 4.5× slower. The shipped overlapping-load and below-16-byte scalar paths remove that cost. This is why every byte-kernel benchmark includes short inputs. Framer A/B used one MiB of frames read eight KiB at a time, with interleaved pinned rounds against the prior decoder; it remains connector-level evidence rather than a Kafka pipeline measurement.

Adding A Kernel And Scope Limits

1. Resolve the typed caller and ownership first. Reuse Arrow or an existing dispatched library when it supplies the exact semantics. Inspect the current optimized loop before adding code.
2. Write the full scalar contract, including bounds, signed arithmetic, failures, bitmap offsets, empty input, and tails. Establish a failing focused check; observable behavior also needs its public Cucumber scenario before product changes.
3. Put the operation at the primitives boundary. Validate fallible inputs before dispatch, resolve the cached level, and return values/masks from the generic buffer body. Keep allocation proportional and local padding bounded; keep irregular scatter in its owning caller.
4. Qualify every available backend and forced fallback, with appropriate baseline builds. Register differential properties in both Bolero modes and pair API-rejection doctests with supported examples. Record the limits of host/target coverage.
5. Measure the complete caller and inspect generated instructions. Keep a scalar operation where it wins. Any end-to-end claim additionally needs same-host A/B. Publish raw evidence on the task and update this chapter and the caller’s authoritative chapter in the same change.
6. Run focused checks and just validate; lower ratchet baselines that fall. Keep public contracts and NSPL skill routing current when their surfaces change, then run just book dev.

Do not add explicit kernels for operations LLVM already vectorizes without a measured caller benefit, or duplicate Arrow and library dispatch. Division by a column has no constant reciprocal and stays scalar. Checked 64-bit multiplication lacks a wider exact lane. VM floating arithmetic and platform-libm transcendental calls keep their current contracts. Branch keys, deduplicator and reorderer keys, lookups, correlator keys, and subscription sampling still contain irregular string/map/hash work: typed keys and a demonstrated buffer algorithm must precede SIMD there. No kernel may combine state across concrete branches, weaken schema/sensitivity checks, change clock authority, hide an error, or persist hot-path payloads.