Repository navigation
Conversation
Compactor: - The copy and converting constructors release their storage and the copies made when a copy throws, or when the converting constructor rejects the order of the items. - merge() works out the sections and the capacity first and allocates once, then copies the other compactor's items and destroys the copies if one throws. Only then does it change the state and the sections. It used to change them first, and leaked the copies made before a copy that threw. It compares with the sketch's comparator, not C(). - compact() allocates the space in the next level, and the growth that more sections need, before it flips or draws the coin and changes anything. A failed compaction used to leave the coin flipped (or a random bit drawn), and could change the sections and then fail to grow. Sketch: - update() compares the item with min and max and copies it first, and sets min and max only after the item is in level zero. - merge() copies the new min and max before it sets either. The same gaps are addressed in datasketches-java by apache/datasketches-java#808, except the item-loss window in compress(), which C++ does not have: compact() reserved the next level's space before moving any item. Results are byte-identical to before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coverage Report for CI Build 44649722539Warning Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes. Coverage increased (+0.04%) to 96.924%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
A compactor that has never held an item has no storage, so serialize() passed a null pointer with a count of 0, and the default serde passed it on to memcpy(). UBSan on Linux reports that as undefined behavior. Serialize the items only when there are some; the bytes are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| if (sorted_ && !std::is_sorted(begin(), end(), comparator_)) { | ||
| destroy_items(); // the destructor doesn't run if a constructor throws | ||
| throw std::logic_error("items must be sorted"); | ||
| } |
There was a problem hiding this comment.
If comparator_ itself throws inside std::is_sorted, destroy_items() is not reached and the allocated buffer and copied items leak. Wrapping the post-copy_items check in try { ... } catch (...) { destroy_items(); throw; } avoids that.
| sort(); | ||
| auto offset = hra_ ? capacity_ - num_items_ : num_items_; | ||
| auto from = hra_ ? begin() - other.get_num_items() : end(); | ||
| auto to = from + other.get_num_items(); | ||
| auto other_it = other.begin(); | ||
| for (auto it = from; it != to; ++it, ++other_it) new (it) T(conditional_forward<FwdC>(*other_it)); | ||
| if (!other.sorted_) std::sort(from, to, comparator_); | ||
| if (num_items_ > 0) std::inplace_merge(hra_ ? from : begin(), items_ + offset, hra_ ? end() : to, C()); | ||
| num_items_ += other.get_num_items(); | ||
| if (!other.sorted_) std::sort(from, from + num, comparator_); |
There was a problem hiding this comment.
Since other's items are now copied into [from, from + num) before sort(), self-merging (sketch.merge(sketch)) when level 0 is unsorted sets this->sorted_ = true (aliasing other.sorted_), skips std::sort(from, from + num, comparator_), and passes an unsorted range to std::inplace_merge (sketch.merge(std::move(sketch)) also moves out of *this before merging):
req_sketch<int> s(12, true);
s.update(1);
s.update(5);
s.merge(s); // leaves [1, 5, 1, 5] with sorted_ == true| const uint32_t ne = nearest_even(ssr); | ||
| if (state_ >= static_cast<uint64_t>(1ULL << (num_sections_ - 1)) | ||
| && section_size_ > req_constants::MIN_K | ||
| if (state >= static_cast<uint64_t>(1ULL << (s.num - 1)) |
There was a problem hiding this comment.
For k = 1024, 4 doublings reach s.num = 48 (s.size = 256), which passes check_header (num_sections <= 64). If state >= (1ULL << 47) (for example on a deserialized sketch), next_sections doubles s.num to 96 (ne = 182 >= MIN_K), and the next iteration of while (next_sections(...)) in merge() evaluates 1ULL << 95 (UB, and serialize() writes num_sections = 96 which check_header rejects). Stopping once s.num > 32 keeps s.num <= 48 consistent with check_header.
| write(os, padding); | ||
| write(os, num_items_); | ||
| serde.serialize(os, begin(), num_items_); | ||
| if (num_items_ > 0) serde.serialize(os, begin(), num_items_); // an empty compactor may have no storage |
There was a problem hiding this comment.
Symmetric case on the read side: deserialize_items (L518, L575) still calls alloc.allocate(0) and serde.deserialize(..., 0) when num == 0, leaving items_ != nullptr while capacity_ == 0 (and passing (nullptr, 0) to memcpy if an allocator returns nullptr for zero-sized allocations).
| if (is_new_min) new_min = optional<T>(static_cast<const T&>(item)); | ||
| if (is_new_max) new_max = optional<T>(static_cast<const T&>(item)); | ||
| reset_sorted_view(); | ||
| compactors_[0].append(std::forward<FwdT>(item)); |
There was a problem hiding this comment.
If item aliases an element in this and compactors_[0] is at num_items_ == capacity_ (such as right after deserialize(), or after a failed compression filled the buffer), append() calls grow() and frees the backing array before constructing from item (heap-use-after-free under ASan, same pattern as #573):
req_sketch<std::string> s0(12);
s0.update(std::string(64, 'a'));
auto b = s0.serialize();
auto s = req_sketch<std::string>::deserialize(b.data(), b.size());
s.update(s.get_quantile(0.5)); // heap-use-after-free in append()Materializing T before grow() also avoids reallocating compactors_[0] when T's copy constructor throws.
| if (is_new_min) min_item_ = std::move(new_min); | ||
| if (is_new_max) max_item_ = std::move(new_max); | ||
| ++num_retained_; | ||
| ++n_; |
There was a problem hiding this comment.
Should update() and merge() also cap n_ at 2^63 - 1 as in #585? Without an overflow check, ~60 repeated merges wrap n_ to 0 (is_empty() becomes true while items are retained) and reach 1ULL << 64 UB at level 64.
| if (is_new_min) min_item_ = std::move(new_min); | ||
| if (is_new_max) max_item_ = std::move(new_max); |
There was a problem hiding this comment.
Committing min_item_ and max_item_ before merging any compactor leaves them modified even when compactors_[0].merge() throws on allocation or on the first item copy: an empty target stays is_empty() == true while holding live min_item_ / max_item_, and a 1-item target {10} failing to merge {1, 999} ends up with n == 1, sole item 10, min == 1, max == 999. In addition, if num_levels == 1 after remove_empty_levels_above(num_levels) in catch, exact-mode serialize() omits min_item_ and max_item_, so get_min_item() / get_max_item() disagree with a serialization round-trip.
| // the converting constructor checks that the sorted levels are sorted by the comparator | ||
| const req_sketch<int, ordered_by> copy(sketch, ordered_by(true)); | ||
| REQUIRE(copy.get_n() == 2000); | ||
| REQUIRE(sketch.get_rank(1000) == Approx(0.5).margin(0.05)); |
There was a problem hiding this comment.
sketch.get_CDF(...) / get_PMF(...) still fail to compile with ordered_by (and use the wrong order for stateful default-constructible comparators) because quantiles_sorted_view::check_split_points (common/include/quantiles_sorted_view.hpp:204, 213) is static and constructs Comparator() instead of using comparator_.
This fixes the REQ gaps found while reviewing #582, and the C++ side of the gaps that apache/datasketches-java#808 addresses in Java.
Compactor:
merge()first works out the sections and capacity it will need, then allocates once. Next it copies the other compactor's items, destroying the copies if one throws. Only after that does it change the state and the sections. Before, it changed the state and sections first, and it leaked the copies made before a copy that threw. It also now compares with the sketch's comparator instead ofC().compact()now allocates the space in the next level, and any growth that more sections need, before it flips or draws the coin or changes anything. Before, a failed compaction left the coin flipped or a random bit drawn, and it could change the sections and then fail to grow. A failed compaction now leaves both compactors as they were and draws no random bit.Sketch:
update()compares the item with min and max and copies it first. It sets min and max only after the item is in level zero, so a failed append no longer leaves a min or max for an item that isn't retained.merge()copies the new min and max before it sets either.The docs of
update()andmerge()now say what a failure leaves behind. A comparator that throws while items are sorted or merged leaves the sketch undefined, as decided for Java in apache/datasketches-java#805.One difference from Java: C++ never had the window in which #808's
compress()could lose promoted items, becausecompact()already reserved space in the next level before moving any item.Tests:
const&merges: the target must stay consistent and usable;The five allocation tests fail on master and pass here. The comparator test needs this change to compile. Serialized results are byte-identical to master; I checked 600 seeded combinations of HRA and LRA, k = 4, 12 and 50,
const&and rvalue merges, and double and string sketches. All tests pass under ASan and UBSan.🤖 Generated with Claude Code