Skip to content

REQ and KLL: reject images with an inconsistent N; cap REQ sections at 48 - #810

Open
leerho wants to merge 1 commit into
mainfrom
deserialize-n-sections-checks
Open

leerho wants to merge 1 commit into
mainfrom
deserialize-n-sections-checks

Conversation

@leerho

@leerho leerho commented Nov 20, 2026

Copy link
Copy Markdown
Member

Java counterparts of checks added in ds-cpp #587. Independent of #808 and #809 (no conflicts).

REQ

The section count could overflow. The number of sections in a compactor starts at 3 and doubles when the compactor's state reaches 2^(numSections − 1). Going from 48 to 96 needs a state of 2^47, which only a crafted image provides. In Java the section count is a byte, so doubling 96 wraps it to −64, and the compactor is then unusable and can't be serialized and read back.

  • ensureEnoughSections() now stops at MAX_NUM_SECTIONS = 48. C++ Mostly spelling errors and documentation corrections. #587 also stops at 48. Real sketches never reach 96, so normal results don't change.
  • heapify rejects a compactor with more than 48 sections. Before, up to 127 were accepted as long as sectionSize × numSections ≤ k².

N wasn't checked. In an estimation-mode image, N is now checked against the total weight of the retained items, where each item at level h weighs 2^h. Compaction keeps that total equal to N, so any difference means the image is corrupt. The check guards against overflow when it adds up the weights.

KLL

For full compact and updatable images, KllMemorySegmentValidate now rejects an n that is negative, or less than the number of retained items, since each retained item weighs at least one. This covers both heapify and wrap, for all four types. Empty and single-item images don't store n.

Testing

  • ReqCorruptImageTest:
    • checkTooManySections: 49 sections are rejected and 48 are accepted.
    • checkSectionDoublingStopsAtMax: a compactor with state 2^50 and 48 sections stays at 48 after a merge.
    • checkNNotEqualToWeight: N + 1, a negative N and Long.MAX_VALUE are all rejected, without large allocations.
  • KllCorruptNTest: for doubles, floats, longs and items, with heapify and wrap, and both compact and updatable images:
    • an n equal to the retained count is accepted;
    • retained − 1, −1 and Long.MIN_VALUE are rejected.
      All three tests fail on main.
  • The full suite passes (2,454 tests), including the CL files. javac -Xlint:all -Xdoclint:all,-missing reports nothing in the changed files.

🤖 Generated with Claude Code

…at 48

REQ
- The number of sections of a compactor stops doubling at 48
  (MAX_NUM_SECTIONS), as in C++. Doubling past it needs a state of 2^47,
  reachable only from a crafted image, and overflowed the byte that holds
  the number of sections, so the sketch could not be read back.
- heapify rejects a compactor with more than 48 sections.
- heapify rejects an estimation-mode image whose N is not the total
  weight of its retained items.

KLL
- Validation of full compact and updatable images rejects an n that is
  negative or less than the number of retained items.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant