Skip to content

sizes: per-format byte ranges from the specification's size rules - #242

Draft
Kinflou wants to merge 2 commits into
masterfrom
sizes/per-format
Draft

Kinflou wants to merge 2 commits into
masterfrom
sizes/per-format

Conversation

@Kinflou

@Kinflou Kinflou commented Nov 11, 2026

Copy link
Copy Markdown
Member

Step 3 of per-format sizes (docs plan: real data-format sizes). Replaces the codec-independent wire-size estimate with a per-format byte range, driven by the specification's size rules (comline-catalog bumped to specification@9a7142c, ComlineProject/specification#4).

What it does

schema::ir::size works on the frozen IR and returns Range { min, max: Option<u64>, upper_bound } for a TypeRef, a format's Sizes and a Lookup for the names in it. None means it can't be sized: a union, a non-unit enum or an enum with no variants, a format without sizes, an unresolved name, a type that has to contain itself.

  • Headers (varint_leb128, msgpack_str, msgpack_array, delimited) are applied to the byte or item count at each end: the least uses the header for 0 or the minimum, the most the header for the maximum (char[max N] is header(4N) + 4N in UTF-8, 2 + 6N in JSON).
  • @max_depth expands a recursive struct to its depth (an option at level D+1 is absent, an array empty); each level is sized once, so a tree of 8 children 20 deep is instant. Without a limit the maximum is None. Each struct in a cycle counts its own nesting.
  • Where the formats stop: a count or length above 2^32 - 1 is cut to what a header can say, and a fixed one above it can't be written (no size). A maximum above u64::MAX is None; a minimum above it isn't sized. Documented in the module header, tested in where_the_formats_stop.
  • witnesses() builds a least and a most value in the spec's typed notation ({"u8": 0}, ...) from the same code as the range, for the runtime's differential test (step 4). Bounded by a budget; a value too large to write down is left out.
  • FrozenLookup resolves names across schemas (own declarations, uses with aliases and globs, qualified paths, then any schema's), the way compat finds declarations.

Replaced

schema::idl::size: WireSize, size_of_type, size_of_struct and size_of_enum are gone. The editor's entry is now range_of_type, range_of_field, range_of_struct and range_of_enum, which freeze the grammar through TypeRef::from_grammar_sized and call the same sizer. SizeLookup and SizeTarget are unchanged. A constant a size names that the caller can't resolve is "no size", not 0.

The language server uses all of it in handlers/hover.rs; its next core bump has to change that (see below).

Doc model

Additive (DOC_MODEL_VERSION stays 1; schemas/doc-model.json re-blessed): SizeRange { format, min, max?, upper_bound? } as sizes on struct, enum and type alias units, sizes on each function argument and returns_sizes on the function, for each official format the package's protocols use (the package default when it has none). Absent where it can't be told, for a dependency's units, and in model_frozen (a built version keeps no protocol to say which formats).

Tests

  • Every case of the spec's size vectors (176 across the three formats): the computed range equals the case's [min, max] and upper_bound; the min/max witnesses conform to the case's type; for each primitive, core's witnesses are values the case itself measures as the least and most.
  • Hand-written: bounded types finite, unbounded None, msgpack tier boundaries (strings, arrays, struct field counts), JSON escaping worst case, recursion with and without @max_depth (mutual, deep, too deep), union / non-unit enum / no variants / unknown format None, limits, aliases across schemas, witnesses.
  • Doc model: ranges per format for a package on packed@1 and on json@1.

How the vectors reach the tests

The git dependency's checkout isn't reachable from a test without asking cargo, and comline-catalog doesn't expose a path, so the three sizes-*.json files (about 1 MB) are vendored in core/tests/spec/vectors with a REV. core/tests/spec/vendor.sh --check (a new CI job) compares them with the spec at that rev, and a test fails when REV and the comline-catalog rev in Cargo.toml differ, so a bump can't leave stale vectors.

Language server

handlers/hover.rs imports WireSize, size_of_type, size_of_struct and size_of_enum. On its next bump it replaces them with size::range_of_type / range_of_field / range_of_struct / range_of_enum (taking size::rules("packed@1"), or the package's format, and a constant resolver, &|_| None to keep today's behaviour) and renders Option<Range> instead of WireSize::{Fixed, Variable, Unknown}. field_size's optional special case goes: range_of_field handles it. HoverSizeLookup stays as is.

Local work

The checkout's uncommitted edits to schema/idl/size.rs and the size2.rs stub are superseded by this; rebase will conflict on size.rs.

Kinflou added 2 commits November 11, 2026 08:40
Replace the codec-independent wire-size estimate (WireSize, size_of_type)
with a sizer over the frozen IR that computes, for a type, the least and
most bytes it takes in a format, from the format's `sizes` in the
catalog (comline-catalog bumped to the spec's 9a7142c).

- schema::ir::size: Range, Lookup/FrozenLookup, range(), witnesses().
  Headers are applied to the byte count at each end, a count above
  2^32-1 is cut or unwritable, @max_depth expands a recursive type (each
  level sized once), a union, a non-unit enum or an unresolved name has
  no size. One code path builds the range and the min/max witness
  values in the spec's typed notation.
- schema::idl::size: the editor's entry over the grammar, through
  TypeRef::from_grammar_sized: one sizer. WireSize, size_of_type,
  size_of_struct and size_of_enum are gone; SizeLookup/SizeTarget stay.
- doc model: per-format `sizes` on structs, enums, aliases and each
  function's arguments and reply (returns_sizes); schema re-blessed.
- tests: every case of the spec's size vectors (vendored under
  tests/spec with REV and vendor.sh --check, checked in CI against the
  catalog pin), plus tier boundaries, JSON escaping, recursion, limits.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant