Repository navigation
[branch-56] fix: carry input requirements across FFI_ExecutionPlan (backport #27428) - #27541
Merged
xudong963 merged 2 commits intoNov 18, 2026
Merged
Conversation
## Which issue does this PR close? - Closes apache#27427. ## Rationale for this change A `ForeignExecutionPlan` reports the `ExecutionPlan` defaults for its input requirements (`UnspecifiedDistribution`, no required ordering, `maintains_input_order = false`), whatever the wrapped plan requires. When a physical optimizer rule runs on the other side of the FFI boundary from the plan it optimizes, it can't see those requirements. That happens with a foreign `QueryPlanner` whose session rules belong to the host. `EnsureRequirements` then puts only a round-robin repartition under a `FinalPartitioned` aggregate, which can split one group key across several partitions. For FFI query planner users this is a regression from 55 to 56: a query that works on 55.0.0 fails at planning time on 56. Until apache#26461, `SanityCheckPlan` ran as a host-side rule and was just as blind, so the plan got through (correct by luck in the affected test, unsafe in general). Since apache#26461 the check runs inside `DefaultPhysicalPlanner` and rejects it. This is the only failure in the datafusion-python upgrade to 56 (apache/datafusion-python#1826). The issue has the full error, the plan and the 55 → 56 history. ## What changes are included in this PR? - `FFI_ExecutionPlan` gets four callbacks: `input_distribution_requirements`, `required_input_ordering`, `maintains_input_order` and `benefits_from_input_partitioning`. `ForeignExecutionPlan` overrides the matching methods, and also the deprecated `required_input_distribution` so it agrees. - `ForeignExecutionPlan` fetches the requirements once in `TryFrom<FFI_ExecutionPlan>`, the way it already does `properties`, instead of crossing FFI on every optimizer call. A malformed ordering requirement fails the conversion rather than being dropped. - `FFI_PhysicalExpr` gets an `eq` callback, so two `ForeignPhysicalExpr`s wrapping equal expressions from the same library compare equal. They used to compare by identity. Once requirements cross FFI, that could make planning fail `SanityCheckPlan` when a foreign plan's foreign child already met its partitioning or ordering requirement. Expressions from different libraries always compare unequal, since one library can't safely read another's private data. - New stable types: - `FFI_Distribution` and `FFI_InputDistributionRequirements` in `physical_expr/partitioning.rs`. Co-partitioning crosses as a bool, because `InputDistributionRequirements` can only be built with all children co-partitioned or none. - `FFI_PhysicalSortRequirement` and `FFI_OrderingRequirements` (hard/soft alternatives) in `physical_expr/sort.rs`. - The deprecated `Distribution::HashPartitioned` crosses as `KeyPartitioned`. The optimizer treats the two the same, and this keeps the deprecated variant out of the ABI. - A note in the 56.0.0 upgrade guide covering both layout changes. It is in the 56 guide because this is going into 56.0.0 (see the last section). ## What is the testing strategy for this PR? - `test_ensure_requirements_sees_foreign_plan_requirements` reproduces the bug. It plans `SELECT a % 2, count(*) ... GROUP BY 1` with `target_partitions = 4`, removes the hash repartition, sends the `FinalPartitioned` aggregate across a mock FFI boundary, and runs `EnsureRequirements`. It asserts that a hash repartition is inserted and that `check_plan_tree_sanity` passes. Without the fix it fails: the plan gets `RoundRobinBatch(4)`. - `test_ffi_execution_plan_input_requirements_round_trip`: default and non-default requirements, including co-partitioning, soft ordering and the deprecated per-child accessor, come back unchanged from a foreign plan. - `foreign_requirements_met_by_foreign_child` in the FFI integration tests crosses a real cdylib. The foreign plan requires `KeyPartitioned([a])` over a `Hash([a], 4)` child, or a hard ordering on `a` over a `SortExec` on `a`. `SELECT * FROM t` must plan without adding a repartition or sort. With identity equality it fails `SanityCheckPlan`. - `test_ffi_physical_expr_eq_across_library_images` loads two copies of the test library with `get_module_copy`. The same `Column` from each copy compares unequal, and two from the same copy compare equal. It fails if the library check in `eq` is removed. - Unit round-trip tests for the new FFI types in `partitioning.rs` and `sort.rs`. - End to end: I cherry-picked this onto the `branch-56` commit apache/datafusion-python#1826 builds against. The previously failing `test_spawning_plan_across_three_libraries` now passes, along with the rest of that repo's FFI example suites. That run predates the `eq` callback and the requirement caching. ## Are there any user-facing changes? The `FFI_ExecutionPlan` and `FFI_PhysicalExpr` layouts change, so FFI providers and consumers need to be rebuilt against the same release. Foreign plans now report their real input requirements to optimizer rules. I'll backport this to `branch-56` before 56.0.0 is final (only `56.0.0-rc1` is tagged so far). The datafusion-python upgrade to 56 needs it, and backporting before the release means 56.0.0 and later 56.0.x releases share one layout. The change applies to `branch-56` without conflicts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 701e556)
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more. @@ Coverage Diff @@
## branch-56 #27541 +/- ##
===========================================
Coverage 83.29% 83.30%
===========================================
Files 1170 1170
Lines 475858 476209 +351
Branches 475858 476209 +351
===========================================
+ Hits 396361 396695 +334
- Misses 55479 55484 +5
- Partials 24018 24030 +12
... and 9 files with indirect coverage changes 🚀 New features to boost your workflow:
|
xudong963
added a commit
that referenced
this pull request
Nov 18, 2026
…t (backport #27407) (#27542) ## Which issue does this PR close? Backports #27407 to `branch-56` for the 56.0.0 release tracked in #24461. The same fix was already backported to `branch-55` (#27408) and `branch-54` (#27409). Related to #27431 and #27388. ## Rationale for this change `cargo audit` fails on `branch-56` with 3 vulnerabilities in the locked `quinn-proto` and `quinn-udp` versions: - `quinn-proto` 0.11.18: [RUSTSEC-2026-0523](https://rustsec.org/advisories/RUSTSEC-2026-0523), [RUSTSEC-2026-0524](https://rustsec.org/advisories/RUSTSEC-2026-0524) - `quinn-udp` 0.5.14: [RUSTSEC-2026-0538](https://rustsec.org/advisories/RUSTSEC-2026-0538) These versions are in the 56.0.0 RC1 lockfile. They also fail the `security_audit` check on #27541. RC2 is being cut anyway, so this backport keeps the vulnerable versions out of the released lockfile. ## What changes are included in this PR? Clean cherry-pick of 73e0b03. Only `Cargo.lock` changes: | Crate | Before | After | | ------------- | ------- | ------- | | `quinn-proto` | 0.11.18 | 0.11.19 | | `quinn-udp` | 0.5.14 | 0.5.16 | ## What is the testing strategy for this PR? Covered by the `security_audit` CI job. Locally, `cargo audit` (0.22.2) reports the 3 vulnerabilities above on the current `branch-56` lockfile. After this change it exits with 0 vulnerabilities, leaving only the allowed warnings. ## Are there any user-facing changes? No. These are semver-compatible dependency updates in `Cargo.lock` only. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
xudong963
added a commit
that referenced
this pull request
Nov 18, 2026
## Which issue does this PR close? - Part of #24461 ## Rationale for this change #27541 and #27542 were backported to `branch-56` for RC2 after the changelog was regenerated in #27382, so they are missing from `dev/changelog/56.0.0.md`. ## What changes are included in this PR? Regenerates the changelog with `dev/release/generate-changelog.py 55.0.0 69d40d1 56.0.0`, where 69d40d1 is the current `branch-56` head, then formats it with `prettier`. This adds entries for: - #27541 (listed under breaking changes because of its `api change` label: the `FFI_ExecutionPlan` and `FFI_PhysicalExpr` layouts change) - #27542 - #27382 It also updates the commit count (1206 → 1209) and the credits. The author shown for #25337 changes from `dd-annarose` to `anna-lscr`. The script reads the current GitHub login for the commit author, so this reflects a renamed account. ## What is the testing strategy for this PR? N/A (changelog only). `ci/scripts/doc_prettier_check.sh` passes. ## Are there any user-facing changes? No 🤖 Generated with [Claude Code](https://claude.com/claude-code)
andygrove
added a commit
to nuno-faria/datafusion-python
that referenced
this pull request
Nov 18, 2026
The rc2 tag includes the backported FFI_ExecutionPlan input-requirements fix (apache/datafusion#27541), which resolves the _test_three_library_query_planner SanityCheckPlan failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Backports #27428 (fix for #27427) to
branch-56for the 56.0.0 release tracked in #24461.Rationale for this change
The datafusion-python upgrade exposed a planning regression in 56.0.0 RC1. When the query planner and optimizer run on opposite sides of the FFI boundary, the optimizer can miss an aggregate's hash partitioning requirement and produce a plan that fails
SanityCheckPlan. This blocks apache/datafusion-python#1826.What changes are included in this PR?
Clean cherry-pick of 701e556. Foreign execution plans expose their input requirements, and physical expressions originating in the same library can be compared for equality. The 56.0.0 upgrade guide describes both ABI changes.
What is the testing strategy for this PR?
Verified on the backport branch:
cargo clippy --all-targets --all-features -- -D warningscargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests, including theEnsureRequirementsregression and the cross-library plan-requirements and expression-equality testscargo fmt --alland the repository's Markdown formatting checkLocal
cargo auditreports RUSTSEC-2026-0523, RUSTSEC-2026-0524 and RUSTSEC-2026-0538 for the existingquinn-protoandquinn-udpversions. The unmodified RC1 lockfile reproduces all three findings. This backport preserves the dependency versions.Are there any user-facing changes?
Yes. Foreign plans preserve their input requirements. The layouts of
FFI_ExecutionPlanandFFI_PhysicalExprchange. FFI providers and consumers built against RC1 must be rebuilt together for RC2. Including the fix before 56.0.0 is final keeps one ABI for the published 56.x series.