Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion kll/include/kll_helper_impl.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,9 @@ uint8_t kll_helper::floor_of_log2_of_fraction(uint64_t numer, uint64_t denom) {
if (denom > numer) { return 0; }
uint8_t count = 0;
while (true) {
// denom * 2 > numer, without overflowing denom
if (denom > (numer >> 1)) { return count; }
denom <<= 1;
if (denom > numer) { return count; }
count++;
}
}
Expand Down
19 changes: 19 additions & 0 deletions kll/include/kll_sketch.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ namespace kll_constants {
const uint16_t MIN_K = DEFAULT_M;
/// max value of parameter K
const uint16_t MAX_K = (1 << 16) - 1;
/// maximum weight of a weighted update, the same as in datasketches-java
const uint64_t MAX_WEIGHT = (1ULL << 61) - 1;
}

/**
Expand Down Expand Up @@ -246,6 +248,20 @@ class kll_sketch {
template<typename FwdT>
void update(FwdT&& item);

/**
* Updates this sketch with the given data item repeated the given number of times.
* The result is equivalent to calling update(item) weight times, at a cost that grows
* with the logarithm of the weight rather than with the weight itself.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: when weight < levels_[0] (up to 65,534 for MAX_K), update(item, weight) runs an O(weight) copy loop into level zero rather than O(log(weight)). Describing the two paths (direct level-zero insertion when weight fits in the remaining free space, and merging a binary-level sketch otherwise) would avoid overstating the complexity bound for large k.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, fixed in #585. The doc now describes both paths.

* If cross-language portability is required, callers should ensure that
* the input string uses a compatible encoding (valid UTF-8).
* The first update allocates storage for items, unless reserve() or a merge already did.
* @param item from a stream of items
* @param weight number of times the item is repeated, from 1 to kll_constants::MAX_WEIGHT (2^61 - 1)
* @throw std::invalid_argument if the weight is 0 or greater than kll_constants::MAX_WEIGHT
*/
template<typename FwdT>
void update(FwdT&& item, uint64_t weight);

/**
* Merges another sketch into this one.
* If sketches contain strings, callers are responsible for ensuring that
Expand Down Expand Up @@ -592,6 +608,9 @@ class kll_sketch {
std::unique_ptr<T, items_deleter> items, uint32_t items_size, optional<T>&& min_item,
optional<T>&& max_item, bool is_level_zero_sorted, const C& comparator);

// for weighted update
kll_sketch(uint16_t k, const T& item, uint64_t weight, const C& comparator, const A& allocator);

// common update code
inline void update_min_max(const T& item);
// items_ is allocated lazily, and these are the only places that deal with it being nullptr:
Expand Down
74 changes: 72 additions & 2 deletions kll/include/kll_sketch_impl.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,70 @@ void kll_sketch<T, C, A>::update(FwdT&& item) {
reset_sorted_view();
}

template<typename T, typename C, typename A>
template<typename FwdT>
void kll_sketch<T, C, A>::update(FwdT&& item, uint64_t weight) {
if (!check_update_item(item)) { return; }
if (weight == 0) throw std::invalid_argument("weight must be at least 1");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Throwing on weight == 0 matches datasketches-java's KllSketch, though two points may be worth discussing for cross-sketch / cross-language alignment:

  1. In datasketches-rust (feat(kll): add weighted update datasketches-rust#293), we made KllSketch::update_with_weight(item, 0) a no-op to match frequent_items_sketch::update(item, 0) (which is also a no-op in both Java and C++) and CountMinSketch, since repeating update(item) zero times is naturally an identity operation when weight is unsigned (e.g., pre-aggregated bucket counts that can be 0). Happy to discuss whether we want to keep KLL aligned with Java's exception here or align zero-weight handling across sketches; we documented the difference explicitly on the Rust side for now.
  2. Because check_update_item(item) runs on line 199 before the weight validation, sketch.update(NaN, 0) (and sketch.update(NaN, MAX_WEIGHT + 1)) returns silently while sketch.update(1.0f, 0) throws std::invalid_argument.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both follow datasketches-java. Its update(item, weight) returns on NaN first, then throws on a weight below 1, and C++ does the same in the same order. Whether a weight of 0 should be a no-op in every language is a good question, but it's a cross-language change, so I'd rather take it to dev@ than decide it here. For now C++ matches Java.

if (weight > kll_constants::MAX_WEIGHT) {
throw std::invalid_argument("weight must be less than 2^61: " + std::to_string(weight));
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAX_WEIGHT (2^61 - 1) bounds a single call, but merge (const uint64_t final_n = n_ + other.n_; at line 319) and the small-weight path do not check for uint64_t overflow on n_:

  • 5 calls of update(x, MAX_WEIGHT) already push n_ >= 2^63 (producing images beyond Java's signed long range).
  • 9 calls of update(x, MAX_WEIGHT) wrap n_ modulo 2^64 to 2^61 - 9. Because kll_helper::sum_the_sample_weights wraps modulo 2^64 in the same way, assert_correct_total_weight() still passes on k = 200, silently leaving a corrupted get_n() (and on k = 8, accumulating 68+ items at level 60 eventually throws "depth > 60" mid-merge after mutating min_item_ and level zero).

Checking weight > std::numeric_limits<uint64_t>::max() - n_ (and other.n_ > std::numeric_limits<uint64_t>::max() - n_ in merge, analogous to Java's Math.addExact) before mutating the sketch would prevent silent wraparound.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. #585 caps n at 2^63 - 1, Java's Long.MAX_VALUE, since Java's merge uses Math.addExact on a signed long. update(), the weighted update and merge() now throw std::overflow_error before changing anything. With n below 2^63, a sketch can't reach the 62 levels that make level_capacity throw "depth > 60", so that failure partway through a merge goes away too.

if (weight == 1) {
update(std::forward<FwdT>(item));
return;
}
allocate_items(); // first, so that a failure leaves the sketch unchanged
reset_sorted_view(); // before any change, so that a failed copy can't leave a stale view
if (weight < levels_[0]) { // fits into level zero without compaction

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

internal_update() only compacts when levels_[0] == 0, so weight == levels_[0] also fits into level zero without compaction. With <, hitting the exact capacity boundary (for example, k = 200 with 150 updates followed by update(x, 50)) takes the merge path and transitions into estimation mode (num_levels_ = 6, 153 retained items) at n == k, whereas 50 plain update(x) calls keep all 200 items in exact mode. Using <= keeps exact mode up to n == k unless < is intentionally kept to mirror Java's guard.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. Java uses the same guard (weight < levelsArr[0]). With <=, C++ would build a different sketch from Java for the same input at that boundary, and we keep the two byte-identical.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense! One related detail just as an FYI (no action needed here): for k > 200 (DEFAULT_K), #569 initializes a new C++ sketch with items_size_ = 200 and levels_[0] = 200 (until reserve() or lazy growth), whereas Java heap KLL initializes levelsArr = {k, k}. As a result, on a fresh k = 400 sketch, update(x, 250) has 250 < levels_[0] false in C++ (taking the merge path into estimation mode) while 250 < levelsArr[0] is true in Java (staying in level zero).

update_min_max(static_cast<const T&>(item));
// construct each copy before counting it, so that a failed copy leaves a valid sketch
for (uint64_t i = 1; i < weight; ++i) {
new (&items_[levels_[0] - 1]) T(static_cast<const T&>(item));
internal_update();
}
Comment thread
proost marked this conversation as resolved.
Resolved
new (&items_[levels_[0] - 1]) T(std::forward<FwdT>(item));
internal_update();
} else {
merge(kll_sketch(k_, static_cast<const T&>(item), weight, comparator_, allocator_));
}
}

// An item at level h carries weight 2^h, so one copy of the item at each level
// whose bit is set in the weight is an exact sketch of that many copies.
// The weight is less than 2^61, so this needs at most 61 levels.
template<typename T, typename C, typename A>
kll_sketch<T, C, A>::kll_sketch(uint16_t k, const T& item, uint64_t weight, const C& comparator, const A& allocator):
comparator_(comparator),
allocator_(allocator),
k_(k),
m_(kll_constants::DEFAULT_M),
min_k_(k),
num_levels_(static_cast<uint8_t>(64 - count_leading_zeros_in_u64(weight))),
is_level_zero_sorted_(true),
n_(weight),
levels_(num_levels_ + 1, 0, allocator),
items_(nullptr),
items_size_(0),
min_item_(item),
max_item_(item),
sorted_view_(nullptr)
{
for (uint8_t level = 0; level < num_levels_; ++level) {
levels_[level + 1] = levels_[level] + static_cast<uint32_t>((weight >> level) & 1);
}
items_size_ = levels_[num_levels_];
items_ = allocator_.allocate(items_size_);
// the destructor doesn't run if a constructor throws, so release the items here
uint32_t num_constructed = 0;
try {
for (; num_constructed < items_size_; ++num_constructed) new (&items_[num_constructed]) T(item);
} catch (...) {
for (uint32_t i = 0; i < num_constructed; ++i) items_[i].~T();
allocator_.deallocate(items_, items_size_);
throw;
}
}

template<typename T, typename C, typename A>
void kll_sketch<T, C, A>::update_min_max(const T& item) {
if (is_empty()) {
Expand Down Expand Up @@ -257,7 +321,7 @@ void kll_sketch<T, C, A>::merge(FwdSk&& other) {
const uint32_t index = internal_update();
new (&items_[index]) T(conditional_forward<FwdSk>(other.items_[i]));
}
if (other.num_levels_ >= 2) { merge_higher_levels(other, final_n); }
if (other.num_levels_ >= 2) { merge_higher_levels(std::forward<FwdSk>(other), final_n); }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forwarding other into merge_higher_levels only avoids copies on the (self_pop == 0) && (other_pop > 0) branch in populate_work_arrays (kll_sketch_impl.hpp:963). Once this is in estimation mode and has items at the same higher level as other ((self_pop > 0) && (other_pop > 0), line 968), populate_work_arrays calls the 8-argument kll_helper::merge_sorted_arrays(const T* buf_a, ..., const T* buf_b, ...) (kll_helper_impl.hpp:164), which has two issues when an item copy (or comparator) throws:

  1. other.items_ is passed as const T* without forwarding, so elements from other are still copy-constructed (new (&buf_c[c]) T(buf_b[b++])). In addition, buf_a (this->items_) is also const T*, so new (&buf_c[c]) T(std::move(buf_a[a])) (kll_helper_impl.hpp:177, 180) produces const T&& and invokes T's copy constructor rather than its move constructor right before calling buf_a[a++].~T().
  2. If any copy or comparison throws inside populate_work_arrays, workbuf's deleter (kll_sketch_impl.hpp:891) deallocates raw memory without destroying the objects already constructed in workbuf (leaking them), while this->items_ has already destroyed elements in levels_[0]..levels_[lvl] and [levels_[lvl], a) without updating levels_. Subsequent destruction in ~kll_sketch() (kll_sketch_impl.hpp:146) then calls ~T() a second time on those destroyed slots (double-free / UB).

The new fragile_item test (kll_sketch_allocation_test.cpp:319) does not reach this branch because k = 200 with 5 updates stays at num_levels_ == 1 (self_pop == 0 for all lvl >= 1). Changing that fixture to k = 8 with 9 initial updates and weight = 18 reproduces a leak at fail_at = 5 and an ASan double-free in ~kll_sketch() at fail_at = 6.

Taking non-const T* buf_a in merge_sorted_arrays (and forwarding from buf_b when other is an rvalue), along with tracking constructed elements in workbuf and destroyed slots in this->items_ on unwind, would close this gap.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, confirmed. Your fixture SEGVs under ASan. I was wrong in my earlier reply: the weighted update's merge copies no items only while this sketch has nothing at the levels it merges into. In estimation mode it still copied, after this sketch's items had already been moved out. The same was true of merge(const&) before this PR.

Fixed in #585. The merge now copies other's items into the work buffer first. If a copy throws, it destroys them, and this sketch is unchanged. After that, this sketch's items are merged in front of them using moves only, so nothing is copied from a const T*. Your fixture is one of the tests. The helpers now use the sketch's comparator instead of C().

n_ = final_n;
if (other.is_estimation_mode()) { min_k_ = std::min(min_k_, other.min_k_); }
assert_correct_total_weight();
Expand Down Expand Up @@ -1097,7 +1161,13 @@ void kll_sketch<T, C, A>::reset_sorted_view() {
template<typename T, typename C, typename A>
kll_sketch<T, C, A>::const_iterator::const_iterator(const T* items, const uint32_t* levels, const uint8_t num_levels):
items(items), levels(levels), num_levels(num_levels), index(items == nullptr ? levels[num_levels] : levels[0]), level(items == nullptr ? num_levels : 0), weight(1)
{}
{
// start at the first non-empty level: level zero can be empty after a merge
while (level < num_levels && levels[level] == levels[level + 1]) {
++level;
weight *= 2;
}
}

template<typename T, typename C, typename A>
typename kll_sketch<T, C, A>::const_iterator& kll_sketch<T, C, A>::const_iterator::operator++() {
Expand Down
113 changes: 113 additions & 0 deletions kll/test/kll_sketch_allocation_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,9 @@

#include <catch2/catch.hpp>
#include <new>
#include <stdexcept>
#include <string>
#include <utility>

#include <kll_sketch.hpp>
#include <test_allocator.hpp>
Expand Down Expand Up @@ -227,4 +229,115 @@ TEST_CASE("kll sketch allocation: failure at each allocation of the updates", "[
REQUIRE(test_allocator_net_allocations == 0);
}

TEST_CASE("kll sketch allocation: failed first weighted update leaves the sketch unchanged", "[kll_sketch]") {
test_allocator_total_bytes = 0;
test_allocator_net_allocations = 0;
{
allocation_control control;
const failing_allocator<std::string> alloc(control);
const kll_string_failing_sketch fresh(200, std::less<std::string>(), alloc);
for (const uint64_t weight: {uint64_t(10), uint64_t(1000)}) { // copies into level zero, and a merge
kll_string_failing_sketch sketch(200, std::less<std::string>(), alloc);
control.arm(0);
REQUIRE_THROWS_AS(sketch.update(item(1), weight), std::bad_alloc);
check_empty(sketch, fresh);
control.disarm();
sketch.update(item(2), weight);
REQUIRE(sketch.get_n() == weight);
REQUIRE(sketch.get_min_item() == item(2));
}
}
REQUIRE(test_allocator_total_bytes == 0);
REQUIRE(test_allocator_net_allocations == 0);
}

// Fail each allocation made by a stream of weighted updates in turn: the sketch must stay valid and usable.
TEST_CASE("kll sketch allocation: failure at each allocation of the weighted updates", "[kll_sketch]") {
test_allocator_total_bytes = 0;
test_allocator_net_allocations = 0;
{
allocation_control control;
const failing_allocator<std::string> alloc(control);
const kll_string_failing_sketch fresh(8, std::less<std::string>(), alloc);
bool completed = false;
for (long fail_at = 0; !completed; ++fail_at) {
kll_string_failing_sketch sketch(8, std::less<std::string>(), alloc);
control.arm(fail_at);
try {
for (int i = 0; i < 300; ++i) sketch.update(item(i), static_cast<uint64_t>(i % 7) * 100 + 1);
completed = true;
REQUIRE(fail_at > 0);
} catch (const std::bad_alloc&) {
if (fail_at == 0) check_empty(sketch, fresh);
check_usable(sketch, control);
}
control.disarm();
}
}
REQUIRE(test_allocator_total_bytes == 0);
REQUIRE(test_allocator_net_allocations == 0);
}

namespace {

long copies_allowed = -1; // copies left before a copy throws; negative means no limit

// An item whose copy constructor throws once copies_allowed runs out.
// Each item owns memory from test_allocator, so an item that is never destroyed shows up as a leak.
struct fragile_item {
int* value;
explicit fragile_item(int v): value(test_allocator<int>(0).allocate(1)) { *value = v; }
fragile_item(const fragile_item& other): value(nullptr) {
if (copies_allowed == 0) throw std::runtime_error("copy failed");
if (copies_allowed > 0) --copies_allowed;
value = test_allocator<int>(0).allocate(1);
*value = *other.value;
}
fragile_item(fragile_item&& other) noexcept: value(other.value) { other.value = nullptr; }
fragile_item& operator=(fragile_item other) noexcept { std::swap(value, other.value); return *this; }
~fragile_item() { if (value != nullptr) test_allocator<int>(0).deallocate(value, 1); }
};

struct fragile_less {
bool operator()(const fragile_item& a, const fragile_item& b) const { return *a.value < *b.value; }
};

using kll_fragile_sketch = kll_sketch<fragile_item, fragile_less, test_allocator<fragile_item>>;

} // namespace

// Fail each item copy made by a weighted update in turn: nothing may leak,
// and the sketch must stay usable, with no stale sorted view.
TEST_CASE("kll sketch allocation: failure at each item copy of a weighted update", "[kll_sketch]") {
test_allocator_total_bytes = 0;
test_allocator_net_allocations = 0;
for (const uint64_t weight: {uint64_t(10), uint64_t(1000)}) { // copies into level zero, and a merge
CAPTURE(weight);
bool completed = false;
for (long fail_at = 0; !completed; ++fail_at) {
CAPTURE(fail_at);
kll_fragile_sketch sketch(200, fragile_less(), test_allocator<fragile_item>(0));
for (int i = 0; i < 5; ++i) sketch.update(fragile_item(i));
const fragile_item probe(2);
const double rank_before = sketch.get_rank(probe); // caches a sorted view
REQUIRE(rank_before > 0);
copies_allowed = fail_at;
try {
sketch.update(fragile_item(100), weight);
completed = true;
} catch (const std::runtime_error&) {}
copies_allowed = -1;
REQUIRE(sketch.get_n() >= 5);
REQUIRE(sketch.get_n() <= 5 + weight);
// a copy of the sketch builds its sorted view from the current items
const kll_fragile_sketch copy(sketch);
REQUIRE(sketch.get_rank(probe) == copy.get_rank(probe));
sketch.update(fragile_item(-1));
REQUIRE(*sketch.get_min_item().value == -1);
}
}
REQUIRE(test_allocator_total_bytes == 0);
REQUIRE(test_allocator_net_allocations == 0);
}

} /* namespace datasketches */
Loading
Loading