Schemas And Codecs

Nervix separates internal runtime schema from wire schema.

Internal Schemas

An internal schema describes the typed runtime record:

CREATE IF NOT EXISTS SCHEMA notification (
  user_id U32,
  created_at DATETIME,
  payload STRING OPTIONAL,
  cpu_last_64 ARRAY<F32, 64>,
  image ARRAY<F32, 3, 224, 224>,
  detections VEC<ARRAY<F32, 6>> OPTIONAL
);

Schemas must declare at least one field.

A schema may name a field with any word, keywords included. An expression writes a field named like a reserved word between backticks where it stands bare, and as it is after a scope such as input..

These types are the values Nervix stores in runtime records and uses for subscription matching and processor logic. BYTES may be a deduplication or reordering key, ordered lexicographically by octet, but branch key schemas cannot contain BYTES, including nested ARRAY or VEC elements.

BYTES holds arbitrary octets in an Arrow Binary column. It is distinct from STRING. Use bytes_from_utf8 and bytes_to_utf8 for explicit text conversion, or hex_decode and base64_decode for encoded text. A CAST never converts between BYTES and text. Lookup schemas, materialized-state dependencies, generator source relays, and window aggregate arguments cannot contain BYTES; their declarations fail validation. LOOKUP_HASH_MAP key expressions also cannot produce BYTES.

ARRAY<T, D1, ..., Dn> is a fixed rectangular array. Each dimension maps to one nested Arrow FixedSizeList level, so ARRAY<F32, 2, 3> maps to FixedSizeList<FixedSizeList<Float32, 3>, 2> and remains a 2-by-3 value. VEC<T> is a variable-length sequence and maps to Arrow List<T>. The element type is recursive, so fixed and variable axes can be mixed, for example VEC<ARRAY<F32, 6>> and ARRAY<VEC<STRING>, 4>.

ARRAY and VEC are distinct and are never implicitly converted. Every fixed axis must have a positive length no greater than 2147483647, the largest length an Arrow FixedSizeList carries, fixed arrays must contain exactly that many elements at runtime, and dense multidimensional values must retain their nested shape. JSON and CBOR represent both with nested JSON-style arrays. AVRO uses nested array schemas with item types inferred recursively from the internal schema.

Append OPTIONAL to either an internal schema field or a wire schema field when the value may be absent. Optional fields are omitted from runtime records and emitted JSON payloads when no value is present.

Wire Schemas

Wire schemas describe the serialized format on the transport side.

Declared wire schemas are either STRICT or LOOSE. Strict wire schemas reject payload fields that are not declared by the wire schema. Loose wire schemas accept extra payload fields and drop them before decoding into the internal schema.

JSON wire schema:

CREATE IF NOT EXISTS WIRE JSON SCHEMA notification_wire MODE STRICT (
  user_id integer,
  created_at string,
  payload string OPTIONAL
);

CBOR wire schema:

CREATE IF NOT EXISTS WIRE CBOR SCHEMA notification_wire MODE LOOSE (
  user_id integer,
  created_at string,
  payload string OPTIONAL
);

AVRO wire schema:

CREATE IF NOT EXISTS WIRE AVRO SCHEMA notification_wire MODE STRICT (
  user_id LONG,
  created_at STRING,
  payload STRING OPTIONAL
);

JSON, CBOR, and AVRO wire schemas must declare at least one field. JSON and CBOR also accept exact numeric and datetime wire types (U8 through I64, F32, F64, and DATETIME) alongside the generic JSON types. The exact variants retain their type in the canonical definition and in completion suggestions. A codec binds an exact wire type only to the internal type of the same name: a U8 wire field to a U8 field, and a DATETIME wire field, whose value is RFC 3339 text, to a DATETIME field without an ENCODE ... AS RFC3339 rule.

The web console’s Create menu provides structured editors for internal schemas and each of these three declared wire formats. Internal collection controls build nested ARRAY and VEC types from an exact scalar type. Field order, OPTIONAL, SENSITIVE, wire mode, and exact format are preserved in the completed Model and its canonical command. See Web Console.

SYSLOG is different: it is a predefined singleton wire schema whose shape comes from the syslog protocol. It has no user-defined name, CREATE WIRE declaration, mode, field list, ALTER, or DROP lifecycle. A codec references it directly:

CREATE CODEC syslog_codec FROM SYSLOG TO SCHEMA syslog_event;

Its fixed field contract is documented in Syslog.

Altering Schemas

Internal and declared wire schemas can be changed without dropping their whole dependent graph first. Operations in one statement run from left to right, and each operation sees the result of the previous one.

Internal schema operations are:

• ADD FIELD <field> <type> [OPTIONAL] [SENSITIVE]
• DROP FIELD <field>
• RENAME FIELD <field> TO <field>
• ALTER FIELD <field> SET TYPE <type>
• ALTER FIELD <field> SET OPTIONAL and ALTER FIELD <field> DROP OPTIONAL
• ALTER FIELD <field> SET SENSITIVE and ALTER FIELD <field> DROP SENSITIVE

For example:

ALTER SCHEMA notification
  ADD FIELD note STRING OPTIONAL,
  RENAME FIELD created_at TO received_at,
  ALTER FIELD payload SET SENSITIVE;

DROP SENSITIVE is the explicit way to downgrade a field. Nervix still rebuilds and validates the whole candidate graph, including every downstream leakage rule. A schema must retain at least one field, added and renamed names must be unique, and the target of every drop, rename, or field alter must exist.

Wire schema operations are:

• ADD FIELD <field> <wire_type> [OPTIONAL]
• DROP FIELD <field>
• RENAME FIELD <field> TO <field>
• ALTER FIELD <field> SET TYPE <wire_type>
• ALTER FIELD <field> SET OPTIONAL and ALTER FIELD <field> DROP OPTIONAL

JSON, CBOR, and AVRO wire schemas are separate entity kinds. They may use the same name without colliding, and the exact format is required for every ALTER:

ALTER WIRE JSON SCHEMA notification_wire
  ADD FIELD note string OPTIONAL;

Mode changes use that same exact entity kind:

ALTER WIRE JSON SCHEMA notification_wire MODE LOOSE;

Use one explicit transaction when a type or shape change requires coordinated updates. Model mutations for one domain—CREATE, schema ALTER, relay ALTER, and DROP—are validated against one candidate graph and committed atomically. The following replacement changes the wire and internal types together and recreates their codec without exposing an intermediate invalid graph:

BEGIN;
ALTER WIRE JSON SCHEMA notification_wire
  ALTER FIELD user_id SET TYPE number;
ALTER SCHEMA notification
  ALTER FIELD user_id SET TYPE F64;
DROP CODEC notification_codec;
CREATE CODEC notification_codec
  FROM WIRE JSON SCHEMA notification_wire
  TO SCHEMA notification;
COMMIT;

If any operation or dependent model fails validation, none of the mutations are persisted. SHOW CREATE SCHEMA and exact wire forms such as SHOW CREATE WIRE JSON SCHEMA notification_wire render the resulting canonical definitions. Dropping a wire schema is exact-format too, for example DROP WIRE JSON SCHEMA notification_wire.

On a running domain, a schema ALTER is applied through an automatic quiesce cycle: Nervix validates first, stops new ingestion and generators, force-flushes buffered output, drains in-flight work, installs the new graph, and resumes. This internal state is not a user-facing lifecycle command. A drain timeout rejects the batch and resumes the old graph. On a stopped domain, Nervix validates and persists the change without a quiesce cycle.

Runtime state whose record layout derives from an altered schema is recreated. Independent stateful nodes retain their state, and so does state that depends on no schema at all: domain-owned Kafka offsets and the counters behind a node’s metrics survive a schema change of their own node. Persisted schema-bound state is keyed by the fingerprint of the schemas it was written under, so a stale layout is never restored as the new type. Relay subscriptions closed by the rebuild report that they must be recreated against the current schema.

Codecs

A codec maps one transport payload format to one internal schema.

CREATE IF NOT EXISTS CODEC notification_codec
  FROM WIRE JSON SCHEMA notification_wire
  TO SCHEMA notification;

Schemaful codecs are type-strict. A JSON string wire field does not implicitly decode into an internal DATETIME field. Declare the wire conversion explicitly:

CREATE IF NOT EXISTS CODEC notification_codec
  FROM WIRE JSON SCHEMA notification_wire
  TO SCHEMA notification
  ENCODE created_at AS RFC3339;

created_at is the internal schema field name. The matching wire field must be a string, and the internal field must be DATETIME.

For an internal BYTES field, declare BYTES in the JSON or CBOR wire schema. Its wire value is a canonical RFC 4648 standard base64 string with padding; malformed or unpadded text is a decode error. AVRO BYTES fields use Avro’s native byte sequence. A wire STRING field cannot bind an internal BYTES field. Nested ARRAY and VEC byte elements use the same representation.

Live WIRE JSON ingestion copies each connector-owned payload into mutable scratch retained by the open ingest group, then parses it with simd-json using parser buffers retained beside that scratch. The decoder walks simd-json’s borrowed values directly into the batch’s typed Arrow builders. It resolves schema field hashes and Arrow data types when the codec and batch are built, parses each datetime once, and streams base64 output into the binary builder. It does not construct a serde JSON tree or a row map. Strict and loose field behavior, optional nulls, exact integer ranges, nested sequence shapes, invalid UTF-8 and malformed escape rejection remain the public wire-schema contract.

On emission, a compiled WIRE JSON codec writes each object in schema field order directly from its typed Arrow columns. Field names are escaped when the codec is compiled. A runtime-selected SIMD classifier marks quotes, backslashes and control bytes in each string column once per batch; clean UTF-8 strings are copied without an escape pass, while marked strings use JSON escaping. Optional null fields are omitted, required null fields fail encoding, datetimes use their declared wire rule, and bytes use padded base64. Nested arrays retain their declared shape. The encoder writes into the caller’s output buffer, including a bounded buffer when MAX SIZE applies.

JAQ-native codecs parse a transport payload in a jaq-supported format and run explicitly directed JAQ transformations. An ingestion transformation turns every value the payload holds into zero or more JSON objects, and each object is decoded into the internal schema as one message (Unfolding Payloads):

CREATE IF NOT EXISTS CODEC notification_cbor
  FROM CBOR
  TO SCHEMA notification
  WITH JAQ TRANSFORMATIONS ON INGESTION '.';

CREATE IF NOT EXISTS CODEC notification_xml
  FROM XML
  TO SCHEMA notification
  WITH JAQ TRANSFORMATIONS
    ON INGESTION '{user_id: (.c[] | select(.t == "user_id").c[0] | tonumber)}';

Protobuf codecs compile .proto files from an uploaded resource, decode or encode the selected message with prost-reflect, and use JAQ to translate between the protobuf JSON view and the internal schema:

CREATE IF NOT EXISTS CODEC notification_proto
  FROM PROTOBUF
  USING RESOURCE proto_bundle VERSION 1
  CONFIG {'file' = 'notification.proto', 'include' = '.'}
  MESSAGE 'nervix.test.Notification'
  TO SCHEMA notification
  WITH JAQ TRANSFORMATIONS ON INGESTION '{user_id: .user_id, payload: .payload}';

The resource contains the .proto files. USING RESOURCE requires VERSION <n> or VERSION LATEST, and the codec compiles its descriptors from the one version it stores. CONFIG declares compile parameters; file/files select source files and include/includes select import roots, all relative to the resource root. If no file is listed, all .proto files in the resource are compiled.

A protobuf codec that a batching emitter encodes through names the message a batch is published as, after MESSAGE:

CREATE CODEC notification_proto
  FROM PROTOBUF
  USING RESOURCE proto_bundle VERSION 1
  CONFIG {'file' = 'notification.proto', 'include' = '.'}
  MESSAGE 'nervix.test.Notification'
  BATCH MESSAGE 'nervix.test.NotificationBatch'
  TO SCHEMA notification
  WITH JAQ TRANSFORMATIONS ON EMITTING '{user_id: .user_id, action: .action}';

BATCH MESSAGE names a message in the same compiled descriptors, and the domain build fails when it does not exist. Without an ON EMITTING BATCH transformation it must declare exactly one field, repeated <MESSAGE>, which holds the members. With one, it may have any shape the transformation’s output is a valid instance of.

Current schemaful codec wire formats are:

• JSON, with an explicit JSON wire schema
• CBOR, with an explicit CBOR wire schema
• AVRO, with an explicit AVRO wire schema

Current predefined fixed-contract wire format:

• SYSLOG, a singleton wire schema referenced directly by a codec

Current JAQ-native codec formats are:

• JSON
• YAML
• TOML
• XML
• CBOR

Current protobuf codec format:

• PROTOBUF, with resource-backed .proto files, inline compile config, and message name

Value Fidelity

A schemaful codec decodes every row it encodes back to the same values: every integer width to its extremes, booleans, any Unicode text, bytes, datetimes to the nanosecond, nested ARRAY and VEC values, and nulls of optional fields. The formats themselves set the bounds of that promise:

• JSON has no number for a NaN or an infinity. WIRE JSON writes a non-finite float as null, which reads back as a null of an optional field and refuses a required field or a list element. WIRE CBOR writes the float itself but reads every value through the same JSON model, so it reads a non-finite float as a null.
• A JSON or CBOR number read into an F32 field becomes the nearest F32, and the shortest decimal a writer prints for an F32 reads back as that same F32, also where the number lies exactly halfway between two of them once read as a 64-bit float. A number beyond the type’s range rounds to an infinity. Every finite float the codec writes reads back to its exact bits.
• WIRE AVRO keeps every float bit pattern. It writes an unsigned element of an ARRAY or VEC as an Avro long, so an element above 9223372036854775807 fails to encode.
• A datetime travels as RFC 3339 text in UTC; text written with another offset reads as the same instant.
• A WIRE AVRO payload is one raw Avro datum, without an object container or a schema registry header, and a WIRE CBOR payload is one CBOR data item.

The JAQ-native and protobuf codecs are transformations rather than encodings of a row: a program decides the shape in each direction. A number a program passes on unchanged keeps its value: a 64-bit integer exactly, and a finite float to its bits through JSON, YAML, TOML, CBOR and the double and float fields of a protobuf message, because every decimal is read as the float nearest to it. The formats set two bounds of their own:

• TOML holds signed 64-bit integers. A larger unsigned value is written as it stands and refused when it is read.
• A protobuf message does not carry a field that holds its default value, so its JSON view omits such a field: a zero of either sign, false, empty text or an empty list. An ingestion program that needs one restores it, as (.level // 0) does; a negative zero comes back positive.

SYSLOG writes its timestamp to the microsecond in UTC; see Syslog.

JAQ Transformations

JAQ-backed codecs must declare a JAQ transform. The concise JAQ Reference links the full upstream manual for the exact jaq-core release Nervix embeds and summarizes the Nervix-specific boundary:

CREATE IF NOT EXISTS CODEC notification_codec
  FROM JSON
  TO SCHEMA notification
  WITH JAQ TRANSFORMATIONS
    ON INGESTION '.payload'
    ON EMITTING '{payload: .}';

Semantics:

• no-wire codecs must use FROM JSON|YAML|TOML|XML|CBOR ... WITH JAQ ...
• protobuf codecs must use FROM PROTOBUF USING RESOURCE ... VERSION <n> | LATEST CONFIG {...} MESSAGE ... [BATCH MESSAGE ...] WITH JAQ ...
• codecs using declared wire schemas must use FROM WIRE JSON|CBOR|AVRO SCHEMA ... and do not carry JAQ transforms
• codecs using the predefined SYSLOG wire schema must use FROM SYSLOG TO SCHEMA ... and do not carry JAQ transformations or field encoding rules
• WITH JAQ TRANSFORMATIONS requires ON INGESTION, ON EMITTING, or both in that order, and ON EMITTING may be followed by ON EMITTING BATCH (Batch Transformations)
• ON INGESTION runs on every value the parsed native or protobuf payload holds and may yield zero or more JSON objects, each of which becomes one message compatible with the internal schema (Unfolding Payloads)
• ON EMITTING runs after the runtime record has been converted into JSON and must yield exactly one native-format or protobuf-message value

JAQ-backed encode/decode runs on the node’s extension workers, so an expensive or non-terminating transform never stalls an async ingestor or emitter task and never holds the workers that relay bodies are encoded and decoded on. A node whose extension workers cannot take a payload now refuses it without judging it: an ingested payload fails its dispatch rather than its decode, and an emitter keeps the rows and retries them. A payload no sender can present again is not refused when its source can wait: a payload an ingestor’s quiesce buffer retained waits in the buffer, and a paced poll, or a payload a source that survives a held loop handed over without an acknowledgement, waits until the extension workers take it (When The Extension Workers Are Full). JAQ JSON output intentionally preserves object member declaration order. The serde JSON preserve_order feature exists for that public JAQ behavior; schemaful WIRE JSON ingestion does not depend on serde JSON object storage.

Batch Transformations

A JAQ-backed codec — JAQ-native or protobuf — may declare a third transformation for emitters that declare a batching clause:

CREATE IF NOT EXISTS CODEC notification_envelope
  FROM JSON
  TO SCHEMA notification
  WITH JAQ TRANSFORMATIONS
    ON EMITTING '{id: .user_id, action: .action}'
    ON EMITTING BATCH '{schema_version: 2, count: length, records: .}';

ON EMITTING BATCH is written only after ON EMITTING, because its input is built from that transformation’s outputs: one array holding the member values of a batch, in publication order. It must yield exactly one value, valid in the codec’s format; for a protobuf codec, a valid instance of the declared BATCH MESSAGE. The program compiles with the codec, so a program that cannot compile fails the domain build. A batching Sentry emitter requires its codec to declare this transformation. ON EMITTING keeps its meaning exactly: it runs per record and yields one value per record.

The transformation replaces the format’s batch container, so an array mapping, an envelope and a larger value are all ordinary uses:

ON EMITTING BATCH '.'
ON EMITTING BATCH 'map({id: .user_id})'
ON EMITTING BATCH '{count: length, records: .}'

Its output is written by the codec’s format writer and measured against the emitter’s MAX SIZE like any container. Whatever it reports as a count, Nervix still counts, acknowledges and limits the source records that went in. It sees only the member values, never the source row, materialized state or headers, so it cannot reach a sensitive value that construction did not already leak.

Batch Containers

An emitter that declares a batching clause publishes several records as one payload. Each record contributes one member value, exactly the value it would have been published as alone: the wire object for WIRE JSON, WIRE CBOR and WIRE AVRO codecs, the ON EMITTING output for JAQ-native and protobuf codecs, and the complete RFC 5424 message for SYSLOG. Without an ON EMITTING BATCH transformation the members are written in the format’s own container, by the same writer, with the same whitespace and number formatting, that writes one record:

TOML and XML have no top-level sequence, so their containers use the single key and the single root element; an XML member must therefore be an element. A SYSLOG frame can say its header once only for members that share it, so records whose PRI, HOSTNAME, APP-NAME, PROCID, MSGID or STRUCTURED-DATA differ are published in separate frames, while each member keeps its own timestamp inside the array:

<134>1 2026-09-24T10:15:00Z app-01 checkout - - - ["<134>1 2026-09-24T10:15:00Z app-01 checkout - - - order accepted","<134>1 2026-09-24T10:15:01Z app-01 checkout - - - order shipped"]

A consumer reads a container back with the same codec: a JAQ-native codec whose ON INGESTION unfolds the array, such as '.[]', yields the members as separate messages within the unfolding limit, and a protobuf codec whose MESSAGE is the batch message can unfold its repeated field, such as '.events[]'.

Unfolding Payloads

A JAQ-backed codec decodes a payload as a stream. The payload is parsed into the values its format holds, ON INGESTION runs on each value in payload order, and every object the program yields becomes one message, in the order the program yields it:

An empty JSON, CBOR, YAML, or XML payload holds no values and decodes into no messages. jq stream semantics apply unchanged: .[] unfolds an array into one message per element, a comma yields several messages, and select or empty yields no message for the values it rejects.

CREATE IF NOT EXISTS CODEC order_lines_codec
  FROM JSON
  TO SCHEMA order_line
  WITH JAQ TRANSFORMATIONS ON INGESTION '.lines[] | select(.quantity > 0)';

Every output must be a JSON object that fits the internal schema: surplus keys are dropped, an absent or null value is accepted only for an OPTIONAL field, and every value must match its field’s type exactly. Messages carry no position of their own; a program that needs one computes it, for example with range(length) as $i | .[$i] + {index: $i}.

A payload is decoded as a whole or rejected as a whole. A malformed value, a program evaluation error, an output that is not an object or does not fit the schema, or more than 65,536 messages rejects the entire payload, including the messages produced before the failure. The rejection is a decode failure of the payload, not a route message error: it is reported as a runtime error event and logged, MODE NO ACK continues with the next payload, and acknowledged modes retry or redeliver it as they do any payload that fails to decode. The diagnostic names the codec, the cause, and the zero-based position of the input value, plus the output’s position when an output is at fault. It never quotes payload values, so a mismatched field is described by its JSON kind and a program evaluation error omits the evaluator’s message, which is logged at trace level instead.

The messages of one payload join one source ingest group together, so they share its domain execution snapshot, and every one of them sees the payload’s source metadata and headers. Each message is filtered, constructed, branched, routed, and error-handled on its own, and ingestor metrics count messages. Source acknowledgement stays per payload: a payload is acknowledged once every message it unfolded into has been acknowledged, a negative acknowledgement of any of them negatively acknowledges the whole payload so that its already acknowledged messages are delivered again with it, and a payload that unfolds into no messages is acknowledged as soon as it decodes. ACK PARALLEL MAX <n> admits n payloads however many messages each unfolds into.

Why The Split Matters

The schema split lets Nervix:

• keep runtime typing independent from transport shape
• support multiple wire formats
• normalize awkward inbound JSON without changing the internal data model
• reshape outbound payloads during emission without changing the internal record layout