Skip to content

fix: align merging snapshot updates and manifest managers with Java - #1115

Draft
wgtmac wants to merge 1 commit into
apache:mainfrom
wgtmac:codex/java-snapshot-parity
Draft

wgtmac wants to merge 1 commit into
apache:mainfrom
wgtmac:codex/java-snapshot-parity

Conversation

@wgtmac

@wgtmac wgtmac commented Nov 12, 2026

Copy link
Copy Markdown
Member

Problem

The merging snapshot pipeline differs from Java in manifest pruning, retry state, delete validation, and execution order. For example, combining delete manifests from multiple snapshots into one index cannot represent successive deletion vectors for the same data file. Repeated application of copied append manifests can also lose or duplicate summary counts.

Solution

Align MergingSnapshotUpdate, ManifestFilterManager, and ManifestMergeManager with Java Iceberg at 0cb5041b9c7a. Keep method order, branch order, and terminology close to the Java implementation to make comparison easier.

  • Track inherited manifest locations and use trusted references and partition summaries to prune filtering. Preserve cumulative delete identities, cache metrics evaluators by partition, and count duplicate deletes per manifest.
  • Match manifest grouping and packEnd ordering, first-manifest equality, retry caching, replaced-manifest accounting, and writer closure on failure.
  • Build delete indexes per snapshot, match DV conflict-validation shortcuts, and preserve copied-append summaries across repeated application and retry cleanup.
  • Match DV merge sequence inheritance and validate sequence/spec/partition compatibility. Use the configured executor for filtering, bin merging, and DV reads while preserving result order and stopping work after failure.
  • Add regression coverage for these behaviors, including null/NaN partition identity, partial failures, executor errors, and retry cleanup.

Retain the existing C++ Result/Status, explicit executor, materialized reader, and transaction staging ownership contracts. This change does not introduce Java's encryption integration or event bus.

Validation

Rebased onto apache/iceberg-cpp main at 19573767858c16cb64a19e5c20afb303b8c6ada6 and rebuilt in Debug mode on macOS arm64, with tests and the bundled data libraries enabled (ICEBERG_BUILD_REST=OFF).

  • Full schema_test, manifest_test, and table_update_test suites passed: 1,876 tests passed and 48 were conditionally skipped (3/3 CTest targets).
  • The three changed public headers passed a C++20 compatibility-mode consumer syntax check.
  • pre-commit for all changed files and git diff --check passed.

Match Java manifest pruning, filtering, bin merging, and per-snapshot
delete conflict validation. Preserve retry summaries and sequence
inheritance, and use the configured executor while retaining result
order and cleanup semantics.
Copilot AI balanced review requested due to automatic review settings November 12, 2026 14:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants