Property Testing And Fuzzing

Bolero is Nervix’s default property-test engine for representation correctness. Each property is an ordinary Rust test beside its production owner and a registered custom libFuzzer target. The generator, production path and assertion are the same in both modes. tests/bolero-targets.toml is the single inventory. Its IDs remain stable while a generator-domain version advances when the current representable shape changes.

Property Contract

A lossless encode/decode or representation conversion asserts equality with the complete original value over bounded, valid current inputs. Equality includes every significant field and order, branch and domain identity, nullability, type and sensitivity. Where PartialEq is not the contract, the property uses an explicit complete oracle. Float bit patterns and Arrow schema, logical values and validity are examples; physical buffer layout need not match when it is not public semantics. Canonicalizing, lossy and one-way transformations state and test their actual contract. Malformed input has separate rejection and boundedness targets, so a round-trip generator cannot pass by producing mostly invalid values. Reference models, operation sequences, scalar/SIMD differential checks, encoded-size bounds and segmentation invariance are other useful Bolero properties.

Generators stay in test code, construct bounded valid values, cover supported variants and deliberate boundaries, and never reintroduce historical shapes. They may not add test-only product APIs, row carriers, persistence for volatile state, or inner-layer parser dependencies. A Bolero property is independent of live services. Public behavior, concurrency and recovery retain their own Cucumber, Shuttle, Loom, Turmoil and external Chaos evidence.

Target And Representation Register

Each target declares its Cargo manifest. Product targets use the root workspace; the synchronization properties use tools/nervix-lint/Cargo.toml. The shared runner discovers declared tooling workspaces from root metadata and uses the selected manifest in ordinary, sanitizer, replay and qualification commands. Compiler executions are fixture checks, not work done in each fuzz iteration.

A target names the test executable that holds its property: its package’s library unit tests, lib, the unit tests of one of its binaries, bin:<name>, or one integration test, test:<name>. The web console’s backup dialog belongs to the console’s browser binary, whose unit tests compile natively, so its properties are bin:nervix-web-console targets; Bolero and the generators stay development dependencies outside the browser’s graph. Before instrumenting a target, the runner reads Cargo metadata to resolve its package, target kind, name and source root. It selects the emitted executable by that identity, including integration tests in folders, a binary’s unit tests beside a library of the same name, and explicitly configured library or test paths. A missing or ambiguous Cargo target or executable fails the run before the campaign starts.

The CI sanitizer job budgets cold ordinary and instrumented builds separately from each target’s bounded campaign. Its overall limit is three hours; per-build, per-case and per-campaign deadlines remain enforced by the shared runner. Validation uses an 8-CPU runner and a two-hour limit to complete Clippy and the declared compiler synchronization matrix from a cold cache. Bolero inventory and compiled discovery run through the dedicated Bolero commands and workflow. The native extra-checks job retains its 8-CPU allocation and the same two-hour limit for ordinary instrumentation and per-mode primitive conformance; the Shuttle and Loom suites run in jobs of their own.

The inventory also records exact full test names, required features, corpus paths, case timeouts and each invariant. Its corpus path is Bolero’s source-adjacent __fuzz__/<test-name>/corpus directory. Ordinary tests replay those files before randomized cases; checked-in corpus inputs are regression cases. Each target starts with current-domain boundary seeds. Generated fuzz corpus and crash files live in retained artifacts under target/bolero/runs. Promote a verified minimized failure to the checked-in corpus when it remains a meaningful regression for the current domain. Breaking shape changes replace obsolete seeds.

The storage representation coverage map ties the consensus, registry, runtime-state and resource store targets to the stored shapes they cover, their complete oracles, the projections their keys make and their corruption boundaries. A filter selects every target whose ID or test name contains it, so just test-bolero runtime-state runs the runtime-state store targets and just test-bolero storage:: the consensus and registry storage targets.

The client representation coverage map ties the client targets to current schema families, complete oracles, ownership and malformed-input boundaries. Request, reply and event coverage checks compare generated families with the schema’s union declarations, so adding a family requires extending its generator.

The Arrow, codec and interconnect representation coverage map ties the relay body, producer batch, client emitter delivery, WASM generated pool, schemaful codec, JAQ-native and protobuf codec number, routed relay payload, relay wire message, client producer link frame and client consumer stream frame targets to their oracles, each codec’s value domain and the contract boundaries they record. Its server targets share one generator of schemas and batches over every current field type and one logical oracle that compares batches cell by cell with floats by their bits. just test-bolero codec- runs the four codec targets, just test-bolero jaq-native the numbers a JAQ program and its native formats carry, and just test-bolero client-producer-link and just test-bolero client-consumer-stream the frames a forwarded client producer and consumer travel in between nodes.

Generated Domains

The WASM representation coverage map registers protocol messages, native host conversions, SDK Arrow and snapshot conversions, archive state descriptors and stored checkpoints. just test-bolero wasm selects their seventeen targets; just test-wasm also builds and exercises the supported Rust and Go reference guests. The malformed targets separately cover header lengths, offsets, vector counts, tags and invalid descriptor fields. Shared guest-message generators live only in the nervix-arbitrary test harness; guests and product libraries acquire no generator dependency. These properties preserve values and retain the existing owners of callback, ACK, checkpoint and recovery semantics.

nervix-arbitrary supplies bounded vocabulary values and entropy; properties compose these into their owning frame or Arrow representations in test code. It reads a property’s bytes as a sequence of bounded choices, so the same bytes always build the same value and a saved failure replays exactly. It is a harness outside the layer order: it depends only on the vocabulary, and states what NSPL can spell as a rule over values instead of calling the parser, so the vocabulary’s own properties use it without a language dependency. No product code names it.

For native client producers and consumers it draws the window a client asks for and is granted, with counts at one, at the bound the vocabulary declares and at the largest count their type holds, a producer’s whole description, every outcome of a submitted batch with each cause it names, and physical durations that land on zero, one nanosecond and the longest duration as often as anywhere else. It also lists one field of every declared type, each scalar, a list of it and a fixed-length array of such lists, across the four combinations of optional and sensitive, for a property that must carry every type on every run.

A generator draws from one of two domains:

• NSPL holds the values canonical NSPL spells and reads back as themselves. The language properties draw from it.
• Vocabulary holds every value the vocabulary types hold, including states NSPL has no spelling for: negative, signed-zero and non-finite numeric literals, empty arrays, a CASE without a WHEN, casts to collection types, and the statement and Model states listed under Uncovered Boundaries. Stored and archived forms carry these states, so the archive properties draw from it.

Both domains reach every Model family, every statement form, and every expression form: integer, float, boolean, string and null literals with their range ends, arrays, field references in every scope, every unary and binary operator nested to a bounded depth, casts and TRY_CAST, IF, CASE with and without an operand, IN and BETWEEN with their negations, the JSON value and existence forms, and calls to built-in functions and UDFs. Every form is written in every region a statement embeds an expression, including a call to a built-in named like a keyword that begins the next clause, such as max, right or replace, and a field in the output or right scope. Routes are generated in every shape a node family allows: transforming and set-only construction, INHERIT, ordered SET, FLUSH EACH and FLUSH IMMEDIATE, and branches declared per route or node-wide. A name of any kind may spell any NSPL keyword, statement and expression keywords alike, and a field, UDF or function name inside an expression or a route construction may also hold a -, ~ or ., or begin with a digit, which canonical NSPL writes between backticks. Names reach both length limits, and counts reach the largest value their field holds. The language properties also sweep sixteen deterministic byte sequences through every Model family, every emitter sink and every statement form on every ordinary run, so none is left to the random cases.

The NSPL Postgres generator keeps DO UPDATE targets nonempty and leaves at least one VALUES column outside the conflict target, as the grammar requires. The vocabulary domain also generates updates whose target includes every mapped column, so archive equality still covers those values.

Relay capacities in both domains reach u64::MAX on native 64-bit targets. Archived native counts use the vocabulary’s fixed-width count representation. Dedicated count properties assert complete current-record equality at zero where permitted, u32::MAX + 1, u64::MAX, and generated values. The public scenario Relay capacities keep every bit through replication and a full cluster restart checks creation and alteration at both positive boundaries in one-node and three-node clusters, including a follower read and full restart.

Rejection targets start from valid canonical text and edit up to three characters, inserting, deleting or replacing delimiters, quotes, comment markers, line endings, digits and non-ASCII characters. The result is mostly no longer NSPL and sometimes still is, so both the rejection and the acceptance paths are exercised.

The expression-text target edits a canonical expression up to four times, deleting a character or inserting a piece or putting one in a character’s place. The pieces are whitespace and line comments, a . with and without spaces beside it, exponent letters, digits and a minus, statement punctuation an expression never reads, and keywords only an expression reserves beside ones it reads as names. It asserts that every entry point agrees on the result: a subscription’s WHERE, the standalone expression reader, a one-element expression list and a route construction’s WHERE read it as the same expression, or none of them reads it as one.

Uncovered Boundaries

The generated domains exclude states whose representation is not lossless. Each exclusion is a recorded boundary, not a claim:

• Canonical NSPL writes a negative literal, -0.0 included, as - applied to its magnitude, and reads it back that way. NaN and the infinities have no spelling, and rendering them fails with a typed error. The NSPL domain therefore holds only non-negative finite literals.
• Outside an expression or a route construction, a generated name is one lower-case ASCII identifier, because canonical NSPL writes a name holding a -, ~ or ., or beginning with a digit, between backticks only inside one. The name properties cover the whole name rule on each name type directly. In the NSPL domain a relay is never named message or branch, and a placement member is never named like an ALTER operation keyword, because NSPL reads those words there as its own.
• DROP has no form for branches, generators, hash maps, signaling protocols, WASM processors or window processors, an HTTP emitter has no BATCH clause, a correlator’s filter has no spelling, and a Postgres ON CONFLICT DO UPDATE whose target names every mapped column is rejected, because it leaves no column to update. Only the vocabulary domain generates those states, for the archive properties.
• A time rate’s text, JSON and archived forms keep every bit. The workspace builds serde_json with float_roundtrip, which reads a decimal as the float nearest to it; its default reader is best-effort and lands one unit in the last place away for a tenth to a quarter of the decimals that computed floats print as.

Behavioral Properties

Registered properties also hold behavioral contracts against independent references:

• Operation sequences. expiry-map-operations drives the expiry map with weighted operation sequences against an IndexMap model and a ledger that counts every stored value’s drops. Miri runs the same property through just test-expiry-map-miri, which lifts Miri’s isolation so Bolero reads its checked-in corpus, with four random cases of at most 512 input bytes; every step still runs and checks its own answer, and the complete state is compared every sixteenth step and at the end. Mutation testing runs the property in every mutant through just test-expiry-map-mutants. connector-confirmation-window publishes generated hand-overs, refusals, pauses and scripted broker answers through the brokered sink confirmation window on a paused clock, and compares the instant the publish ends, the answers the window holds after each hand-over, the records it settles and the failure that ends it with a reference model of the documented window rule computed in whole milliseconds.
• Numeric references. The decimal rounding kernels are compared with the exact decimal expansion of each value and with a 128-bit integer reference. That a value holding no more fractional digits than the count is its own rounding is checked apart from the rounded results. approx-into-exact holds the lossless round trips, and approx-into-rounding compares every rounding and truncating conversion with a bit-level IEEE 754 reference: round to nearest, ties to even, and an infinity past the largest finite value, so u128::MAX narrows to an F32 infinity.
• Kernel differentials. SIMD targets compare every level the host supports, the baseline and the forced scalar fallback with a complete scalar oracle, tails, offsets and unused bits included. A compensated float sum is checked against an exact sum of Shewchuk partials within the bound of compensated summation, because the window contract promises no bit equality across run partitions.
• JSON rendering. columnar-json-rows renders generated logical values on its own: strings through serde_json, bytes through a reference base64 encoder or octet escaping, and instants as RFC 3339. A float must be a JSON number of at most its type’s significant digits that reads back to its exact bits; its exponent’s spelling is the writer’s, which writes 1e172 where serde_json writes 1e+172.
• Encoded sizes. The bounded writer, codec row and container encodings, emitter subdivision and packing, row request division, row preparations, retained retry payloads, SQS records and batch requests, MQTT PUBLISH packets and Postgres inserts are checked against the final bytes, framing, headers and escaping included where the size contract includes them. Generated measures grow and shrink with a candidate, so no property assumes that a larger candidate encodes larger.

Lifecycle, acknowledgement and cancellation races keep their production-owner Shuttle checks. Kafka, NATS and Pulsar enforce their size limits inside their client libraries, and the MySQL, MongoDB, ClickHouse and OTEL measures share the division rule while their own measurements keep example tests.

Commands And Enforcement

Server properties embed the real web console assets. The Bolero recipes build them before discovery, requiring Trunk and the repository Rust toolchain’s wasm32-unknown-unknown target. Both CI modes install those prerequisites. Asset builds stage Trunk’s complete output and preserve byte-identical published files, including their timestamps, so preparing assets again keeps the native server’s compiled artifacts reusable.

On a shared host, a cold sanitizer build can exceed the runner’s build deadline. just prepare-bolero <id> prepares one exact registered target through the same instrumented build path that runs its campaigns. CARGO_BUILD_JOBS=1 just prepare-archive-counts-fuzz uses that command for the declared server library shared by the registry and window count properties. Then run the registered campaigns with just fuzz. Preparation records its own build log and metadata with a 7,200-second compilation deadline; it does not count as a completed fuzz campaign. An expected sanitizer skip contributes no execution or coverage evidence. The Check workflow’s validation job and the Bolero randomized job retain their separate scopes and limits. Check calls the reusable Bolero workflow with its exact tested SHA, event and label snapshot. Scheduled and manual Bolero runs execute randomized checks and record a deliberate sanitizer skip.

Documentation-only Selection

Before starting ordinary suites, Check’s test-selection job proves which source it selects for. Its checkout must be the commit the event tests: a merge whose first parent is its base and whose second parent is the event’s exact head. The event and the tested merge need not name the same base. GitHub records a pull request’s base as its base branch’s tip when the pull request opens, and moves it forward only to the newest base commit that a later head merges, while it builds the tested merge on the base branch’s tip whenever the head changes. A push made after the base branch advanced, whose head does not merge the newest base, therefore tests a merge whose base is newer than the one its event names, and label events that follow name the same recorded base.

Selection accepts the tested base when it is the event’s base, or when the event’s base is one of its ancestors. The checkout holds only the tested merge and its parents, so to decide the second case selection fetches the tested base’s complete history as commits, without trees or file contents, and then answers from that local history alone. When the tested base is the event’s own base, it fetches nothing. A checkout that is not the tested merge, a tested commit that does not merge the event’s exact head into one base, an event base that the tested base does not descend from, such as a newer base or one from another branch, and a failed history fetch all fail selection, so none of them can authorize an exclusion.

Selection then reads every changed path between the tested base and the tested merge: what the merge changes on the base it was tested against, never the base branch’s own advance. Rename detection is disabled so deleting or moving product code into a prose location still requires execution. Only AGENTS.md, README.md, Markdown under docs/, and Markdown under .agents/skills/ qualify for the documentation-only exclusion. An empty diff, code, tests, build configuration, workflow definitions, scripts and unknown paths require suites; no-docker supplies no evidence about this choice.

A verified documentation-only selection excludes the ordinary tests, scenarios, extra tests, diagnostic evidence and their coverage aggregation. It excludes the entire reusable Bolero call unless the event carries fuzz. Validation and the conventional PR title check remain independent and mandatory. Label-selected mode checks remain selected by their labels; fuzz still requires both ordinary properties and the complete sanitizer inventory even for documentation. Label addition and removal compute a fresh selection from that event’s label snapshot.

The ci-test-selection artifact records run, attempt, job, event and action, the tested merge, the event’s base, the tested base and the head, every changed path, labels and each ordinary/Bolero requirement. Two different bases record an advance that selection proved. Its selected verdict records a source decision, never successful property execution or coverage. Actual job conclusions remain in the workflow run. The reusable workflow’s bolero-selection record is produced only after all of its required execution succeeds; failed, cancelled, skipped or missing required producers still fail its gate.

A rerun of test-selection reads the original event and tested merge and proves the same ancestry, so it reaches the same selection after the base branch advances again. A validation-only rerun uses the original event and tested commit. Its ordinary producer selection remains in the caller, outside the reusable workflow’s always() gate, so deliberate non-code exclusion never becomes a cancelled property campaign to excuse. Historical attempts retain their actual cancellations; this policy does not turn an interrupted required campaign into success. just test-ci-selection exercises source identity, advanced, stale and tampered bases, stale heads, other merges, shallow checkouts, mixed changes, renames, label events, reruns and strict gates; just coverage-ci-selection measures that Python contract without launching product campaigns. Selection does not claim a model-checker or public Cucumber result.

Sanitizer Shards

A single job running every sanitizer campaign outgrew its limit as the inventory grew. With all 159 targets it took 209 to 258 minutes: discovery took 27 of them, the two server sanitizer builds with one Cargo build job 42, and the campaigns 132. CI therefore divides the inventory into shards that run in parallel, each in a job of its own:

• A harness, one package’s test target with one feature set, is never split, so a shard builds only the harnesses it runs. The inventory’s prepared_targets names one target of each harness too large for a campaign’s build deadline: today the server library with its empty and testing feature sets.
• The plan deals the harnesses to the shards, the longest estimate first, each to the shard that holds the least estimated time. A harness’s estimate is its build, 30 minutes for a prepared harness and 3 for any other, plus the inventory’s PR duration and 15 seconds for each of its targets. The plan depends only on the inventory and the shard count, so every runner computes the same one, and just fuzz-plan <count> prints it.
• Each shard job discovers every registered target, prepares its own declared harnesses with one Cargo build job and the 7,200-second preparation deadline, and runs just coverage-bolero 30 <number>/<count>: the runner tests, its share of the campaigns and the complete failure qualification. Only the server preparation is held to one build job; discovery and library builds use every core. A shard samples CPU use, steal time and used memory every five seconds and adds the summary to its job summary.
• bolero-fuzz-combined downloads every shard’s completion evidence, without the raw profiles, and runs just combine-bolero-fuzz. It accepts only shards 1 to N of one plan, from one workflow run and attempt at the tested revision and inventory. Each must be complete, must have run exactly its planned targets and must hold a complete record for each of them and a report. Together they must select every registered target once. The job then writes the combined record and the merged Rust source report.

Each shard job allows 180 minutes. bolero-gate requires the randomized job, every shard and the combined proof when the PR carries fuzz, and requires all of the sanitizer jobs to be skipped otherwise. tests/release-evidence.toml holds a qualifying revision to the same checks, every shard of the matrix included.

The fuzz profile uses one codegen unit, optimization level two and debug level one. The full server package uses optimization level one and no debug output to keep its instrumented build within the compilation budget. Both profiles retain debug assertions, overflow checks, AddressSanitizer and libFuzzer coverage feedback. Live campaigns also use Rust source instrumentation; ordinary and release builds retain their own settings.

An instrumented build has a 1,800-second deadline. On a loaded development host, BOLERO_BUILD_TIMEOUT_SECONDS=7200 just fuzz <id> 30 grants compilation more time. The override must be a positive integer and affects only compilation; the inventory’s case limit and the requested campaign duration remain enforced. CI uses the default deadline. The runner prints the effective compilation deadline alongside the build command and retains timeout evidence.

The campaign process has a deadline of its requested duration plus the greater of 30 seconds and three case timeouts. The runner passes -print_funcs=0 to libFuzzer for preparation, campaigns, minimization and failure qualification. This disables the NEW_FUNC progress messages that otherwise start expensive symbolization on large sanitizer binaries. Crash reports and exact replay retain symbolization where a symbolizer is available; sanitizer CI exposes its installed LLVM symbolizer under the executable name libFuzzer discovers.

Ordinary execution prepares each selected package, harness and exact feature set once with cargo test --no-run, under a separate 1,800-second compilation deadline. Discovery can require the union of a package’s features; its binary does not prepare a target with a different declared feature set. Each property still has a 240-second execution bound and must complete its exact randomized iteration count and replay every checked-in corpus input. A failed preparation ends the run before that harness executes any property. The ordinary CI job reserves 60 minutes for discovery, the declared feature builds and every registered target.

Install the dated sanitizer nightly named in the inventory and the pinned CLI. Server properties also require the native server build dependencies, including protoc for the Pulsar driver; both ordinary and sanitizer CI jobs install the Protocol Buffers compiler, Trunk and the browser target. Every compiled Bolero recipe builds the embedded web console before discovering the server properties, using the same configured compiler wrapper as the product build.

rustup toolchain install nightly-2026-09-17 --profile minimal
just install-cargo-bolero
just help
just fuzz-list
just test-bolero
just test-bolero nspl-model
just fuzz nspl-model 30
just fuzz-all 30
just fuzz-plan 3
just fuzz-all 30 2/3
just prepare-bolero-shard 1/3
just coverage-bolero 30
just qualify-bolero
just fuzz-replay nspl-model <saved-input>
just fuzz-replay-sanitizer nspl-model <saved-input>
just fuzz-reduce nspl-model <saved-input>
just combine-bolero-fuzz <shard-evidence> <output>

just validate-bolero compares the inventory with all workspace packages declaring Bolero, scans their property macros within each owning Cargo package, and queries compiled targets through filtered library and integration-test discovery with the inventory’s declared features. Source scanning stops at nested Cargo manifests and excludes Cargo build output; an authored module named target remains part of its package. It rejects missing, unregistered, duplicate, ignored and zero-selected targets, plus corpus paths different from Bolero’s actual work directory. Discovery executes only bolero_ tests of the registered library and integration-test targets under Bolero’s selection mode; it cannot start the server’s scenario harness. It reports discovered, selected, executed and completed counts. just validate and just validate-ci do not run Bolero checks. Bolero commands perform discovery before their selected action, and the dedicated ordinary Bolero CI job owns this check. The inventory includes server-owned Arrow properties, so Bolero recipes prepare the web console assets the server library embeds through just bolero-deps. This compiles assets only; it starts no live service. Both Bolero CI jobs install Trunk, the browser Rust target and native server build dependencies before compiled discovery. Console builds stage the complete real Trunk output and publish only changed assets. Identical assets retain their file identity and timestamps, so repeated preparation preserves an already instrumented server build; failed builds leave the published assets intact.

just test-bolero requires the configured randomized-case count and checked-in corpus replay for every selected property, checking Bolero’s reported input counts. Fuzz runs use real libFuzzer coverage with AddressSanitizer, bounded input length and per-case timeout. The fuzz profile enables optimization, debug information, debug assertions and overflow checks. Builds use the pinned nightly and keep the configured kache wrapper. Modeled execution features are excluded; Loom, Shuttle and Turmoil remain independent build invocations. The server package uses the current profile’s lighter optimization and omits debug output for its single codegen unit; sanitizer instrumentation, coverage, debug assertions and overflow checks remain enabled. The pinned cargo-bolero CLI does not expose a Cargo test-harness selector. Each instrumented build therefore records a run-local Rustup adapter that forwards the declared --lib, --bin or --test selector to its Cargo test calls and forwards toolchain/version queries unchanged. It preserves the toolchain, features, sanitizer flags and compiler wrapper. Library properties compile and execute only their registered harness; they do not compile or invoke the server’s Cucumber executable. The adapter selects the pinned compiler through scripts/bolero_rustc.py, still beneath the configured kache wrapper. For proc-macro2 with span-locations, that compiler adapter removes only its generic cfg(fuzzing): the dependency otherwise deliberately replaces source locations with zero, violating the source parser’s contract and collapsing distinct findings. Sanitizer, coverage and libFuzzer flags remain intact, and every other crate keeps its configuration. The adapter’s content digest separates build directories. The source-location property and saved regression corpus assert the same exact locations in ordinary and sanitizer execution. On a shared host, a cold server build can exceed the campaign runner’s 30-minute compilation limit. just prepare-bolero <id> prepares that same pinned, scoped sanitizer build with a two-hour limit and retains its log and preparation metadata. It does not run a campaign or record fuzz completion. just prepare-archive-counts-fuzz uses this shared path for the declared registry library harness. just prepare-bolero-shard <number>/<count> prepares the declared harnesses one shard runs, as each CI shard does before its campaigns. The target must subsequently pass just fuzz <id> with the ordinary build and case limits; CI campaigns continue to use that bounded runner.

PR CI runs a required ordinary randomized/corpus job through Check’s reusable Bolero call. The sanitizer libFuzzer shards run only when the PR has the fuzz label, with 30 seconds of engine time per target. Every job uses the native CI linker, including sanitizer builds that supply their own compiler flags. Adding or removing the label reevaluates the jobs, and a push to a labeled PR runs the shards again. Scheduled and manual workflow runs execute the ordinary properties. Job limits reserve additional time for compilation, artifacts and cleanup. A cache may seed a campaign but cannot skip a target or replace checked-in regressions. An empty selection, timeout, engine failure, sanitizer finding or property failure fails the job.

Each fuzz run retains its corpus, crashes, log and metadata with revision, target, domain version, toolchain, features, flags, result and exact-input reproduction commands. Retain failure inputs before cleanup. fuzz-replay stages saved bytes in the target’s actual crash directory and runs the ordinary assertion with zero randomized cases. A sanitizer finding passes there, because only the instrumented build detects it: fuzz-replay-sanitizer builds the target’s instrumented harness and runs the exact saved bytes once, and a run’s metadata records both commands. fuzz-reduce uses libFuzzer crash minimization, verifies that the minimized bytes still fail the instrumented target and records whether the ordinary assertion fails on them as well. A saved input inside a run directory replays only while it matches the digest its run’s metadata holds. A missing, unreadable or foreign record, or changed bytes, is a corrupt replay record and is refused; a copy outside the run directories carries no record and replays as given. A random seed reproduces one generated case, not an entire entropy-driven campaign.

just qualify-bolero proves that every way a run must fail does fail, and keeps a qualification.json naming each case, its requirement, what it observed and its evidence:

• A property mismatch fails the campaign and saves its input, which then minimizes and replays.
• An AddressSanitizer heap-buffer-overflow fails the campaign with its report, and the instrumented replay of the saved input reproduces it.
• A case past its timeout, an engine that cannot start and a campaign naming no compiled target fail.
• Compiled discovery of generated packages accepts an exact registration and fails on an ignored, an unregistered and a missing property.
• A filter, a shard plan or a campaign that would run nothing fails.
• An input that differs from its run record, and one whose record cannot be read, are refused.

The deliberately failing targets live in the standalone tests/bolero-qualification package, outside the workspace and the inventory. Every CI shard runs the qualification, so each shard’s runner proves its own failure paths.

Live Rust Source Coverage

just fuzz <id> <seconds> and just fuzz-all <seconds> collect Rust source coverage from the inputs the live libFuzzer process executes, including candidates it discards from its retained corpus. just coverage-bolero <seconds> runs the command/artifact tests, all live campaigns and failure qualification while collecting the separate Python runner report. It does not rerun the randomized producer that CI already requires independently.

The shared build path adds -C instrument-coverage to cargo-bolero’s sanitizer flags, retaining the pinned nightly, assertions, optimization settings, case/input limits and kache wrapper. Encoded Cargo flags are rejected because they would override cargo-bolero’s sanitizer flags. Preparation and the zero-run build probe discard their counters. Ordinary discovery, exact replay and deliberate failure qualification also discard counters; they cannot enter a product campaign report or mix stable profiles with nightly profiles.

Each invocation allocates a fresh target/native-coverage/bolero-fuzz/fuzz/<nightly>/<attempt>/ directory; a shard’s attempt name ends in -shard-<number>-of-<count>, and its record names the shard and the digest of its plan. Each selected target has its own targets/<id>/ directory there, with profiles/%p-%m.profraw, executions.jsonl, lcov.info, export.log and completion.json. The runner records the exact executable returned by the hashed fuzz build and its Cargo fingerprint, effective compiler flags and profile settings. It holds the fuzz build directory through execution and export, so another collector cannot replace those objects. The shared native exporter matches profile binary IDs to retained executables and any required child objects, using the producing nightly’s own llvm-profdata and llvm-cov. Missing objects, absent counters and unreadable or mixed-format profiles fail collection. LCOV paths are repository-relative; dependency, generated and harness files follow the native collector’s exclusion policy. Tests inline in an included source file retain that file’s attribution.

A successful target has a complete record only after a successful engine exit with nonzero completed inputs and a successful source export. The aggregate report beside the invocation’s completion record is complete only after all selected registered targets complete. The record retains the inventory digest and exact entries, selection and discovered/executed/completed counts, revision and whether it was modified, compiler/LLVM identity, workflow run and attempt, event and labels, campaign limits, input throughput and source attribution. Each target’s record also identifies the checked-in corpus the campaign started from, by its input count and a digest of every seed’s name and content. A focused invocation or a shard states its selection and never claims the rest of the inventory ran; only the combined record of every shard does.

A crash, abort, deadline or interrupt stays failed or interrupted even if it left some counters. The runner retains its original corpus, crash input, logs and incomplete evidence; it does not export those counters as a successful campaign. Failure minimization and exact replay remain independent qualification, with their original failure retained. No corpus replay substitutes for the live measurement.

Each CI shard publishes coverage-bolero-fuzz-<number> with its live campaign reports, records and raw profiles, bolero-fuzz-evidence-<number> with its records and aggregate report but no profiles or per-target reports, bolero-runs-<number> for corpus/crash and qualification artifacts with the Python runner report as python.lcov, and runner-load-bolero-fuzz-<number> with its CPU and memory samples. coverage-bolero-fuzz holds the combined record and the merged report, which sums the shards’ line, function and branch counters because they ran disjoint campaigns, and bolero-selection the gate’s exact producer verdicts, including deliberate skips. Fuzz reports remain separate from ordinary workspace coverage and its CRAP report. A sanitizer skip publishes no fuzz coverage artifact.

Adding A Target

Put the Bolero test beside its production owner, name it bolero_..., bound iterations and input length, and assert the complete current-value contract. Register the package, full test name, source, domain version, features, corpus, budgets and invariant in the inventory. Check in current-domain seeds, run just validate-bolero, just test-bolero <id> and just fuzz <id> 30, then run just validate. Both CI jobs include the target in the same change; the shard plan places it with its harness, or deals a new harness to a shard, so a target needs no shard assignment. A new harness whose sanitizer build exceeds a campaign’s build deadline joins prepared_targets. A public product bug found this way first gets a failing focused reproducer and affected public Cucumber scenario before its fix.