Repository navigation
Conversation
- compress() no longer loses items when growing the next level's buffer fails: ReqCompactor.compact() now allocates that space first, then merges the promoted items in, and only then trims its own buffer. - ensureEnoughSections() allocates before it changes the section fields. - If compression is interrupted, the retained-item count and nominal size are recomputed from the compactors. - update() stores min and max only after the item has been appended. - ReqSketch javadoc says what an exception during update leaves behind. Results, serialized images and buffer capacities are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leerho
requested review from
proost and
tisonkun
and
a balanced review from Copilot
November 19, 2026 17:12
There was a problem hiding this comment.
🟡 Changes recommended
A throwing compression hook can leave cached queries stale after an update, and iterator allocation failures during recovery can leave compactor counters inconsistent.
2 open issues
What changed in this PR
This PR extends update-failure handling to REQ sketches, following the classic quantiles and KLL changes. It aims to preserve sketch consistency when allocation or compression fails.
Changes:
- Reserves promotion capacity before moving items and delays section-field changes until allocation succeeds.
- Updates min/max after appending, repairs counters after interrupted compression, and documents failure behavior.
- Adds failure-path tests for compaction and continued sketch use.
| File | Description |
|---|---|
src/test/java/org/apache/datasketches/req/ReqUpdateFailureTest.java |
Adds compaction-failure and sketch-consistency tests. |
src/main/java/org/apache/datasketches/req/ReqSketch.java |
Reorders update bookkeeping, adds compression recovery, and documents exceptions. |
src/main/java/org/apache/datasketches/req/ReqCompactor.java |
Reserves promotion storage and allocates before changing section fields. |
src/main/java/org/apache/datasketches/req/FloatBuffer.java |
Makes ensureSpace() package-private for compactor use. |
Review effort: Balanced
| if (Float.isNaN(item)) { return; } | ||
| ensureCapacity(); //first use allocates, before anything changes | ||
| final FloatBuffer fbuf = compactors.get(0).getBuffer(); | ||
| fbuf.append(item); //may grow the buffer; nothing else has changed yet |
Comment on lines
+585
to
+586
| retItems = computeTotalRetainedItems(); | ||
| maxNomSize = computeMaxNomSize(); |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Follows #805 (classic quantiles) and #806 (KLL): the same kind of failure, now in
ReqSketch.Problem
In
ReqSketch.compress(),ReqCompactor.compact()trimmed the compacted items from its own buffer and returned the promoted half. The caller then merged that half into the next level, which may have to grow its buffer. If that allocation failed, the promoted items were gone, andretItemsandmaxNomSizeno longer matched the compactors.update()also set min and max before appending the item, so a failed append left a min or max for an item that was never added.Changes
ReqCompactor.compact()now takes the next level's buffer. It reserves space there first, merges the promoted items in, and only then trims its own buffer and advances its state and coin.ensureEnoughSections()allocates before it changes the section fields.compress(), the catch block recomputesretItemsandmaxNomSizefrom the compactors and rethrows. The sketch stays valid and compresses again on the next update.update()stores min and max after the append.FloatBuffer.ensureSpace()is now package-private.ReqSketchclass javadoc: a new paragraph, "Exceptions during update".A merge can still leave part of the other sketch applied if it throws. This PR leaves that as it is, as in #806.
Testing
ReqUpdateFailureTest:compactorUnchangedWhenPromotionFails: the merge into the next level fails (its buffer isn't sorted), and the compactor's image, coin and the next buffer stay unchanged. Covers both HRA and LRA.sketchConsistentWhenCompressionFails: aReqDebugthrows while a level is being added mid-compression. After that, N, min, retained count, nominal size and iterator weight still agree, and the sketch keeps working and round-trips. This one guards the catch path. It also passes on main, because no hook can inject anOutOfMemoryErrorat the allocation itself.javac -Xlint:all -Xdoclint:all,-missingreports nothing in the changed files.🤖 Generated with Claude Code