Skip to content

[branch-56] fix: carry input requirements across FFI_ExecutionPlan (backport #27428) - #27541

Merged
xudong963 merged 2 commits into
apache:branch-56from
xudong963:xudong963/backport-27428-branch-56
Nov 18, 2026
Merged

xudong963 merged 2 commits into
apache:branch-56from
xudong963:xudong963/backport-27428-branch-56

Conversation

@xudong963

Copy link
Copy Markdown
Member

Which issue does this PR close?

Backports #27428 (fix for #27427) to branch-56 for the 56.0.0 release tracked in #24461.

Rationale for this change

The datafusion-python upgrade exposed a planning regression in 56.0.0 RC1. When the query planner and optimizer run on opposite sides of the FFI boundary, the optimizer can miss an aggregate's hash partitioning requirement and produce a plan that fails SanityCheckPlan. This blocks apache/datafusion-python#1826.

What changes are included in this PR?

Clean cherry-pick of 701e556. Foreign execution plans expose their input requirements, and physical expressions originating in the same library can be compared for equality. The 56.0.0 upgrade guide describes both ABI changes.

What is the testing strategy for this PR?

Verified on the backport branch:

  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests, including the EnsureRequirements regression and the cross-library plan-requirements and expression-equality tests
  • cargo fmt --all and the repository's Markdown formatting check

Local cargo audit reports RUSTSEC-2026-0523, RUSTSEC-2026-0524 and RUSTSEC-2026-0538 for the existing quinn-proto and quinn-udp versions. The unmodified RC1 lockfile reproduces all three findings. This backport preserves the dependency versions.

Are there any user-facing changes?

Yes. Foreign plans preserve their input requirements. The layouts of FFI_ExecutionPlan and FFI_PhysicalExpr change. FFI providers and consumers built against RC1 must be rebuilt together for RC2. Including the fix before 56.0.0 is final keeps one ABI for the published 56.x series.

## Which issue does this PR close?

- Closes apache#27427.

## Rationale for this change

A `ForeignExecutionPlan` reports the `ExecutionPlan` defaults for its
input requirements (`UnspecifiedDistribution`, no required ordering,
`maintains_input_order = false`), whatever the wrapped plan requires.
When a physical optimizer rule runs on the other side of the FFI
boundary from the plan it optimizes, it can't see those requirements.
That happens with a foreign `QueryPlanner` whose session rules belong to
the host. `EnsureRequirements` then puts only a round-robin repartition
under a `FinalPartitioned` aggregate, which can split one group key
across several partitions.

For FFI query planner users this is a regression from 55 to 56: a query
that works on 55.0.0 fails at planning time on 56. Until apache#26461,
`SanityCheckPlan` ran as a host-side rule and was just as blind, so the
plan got through (correct by luck in the affected test, unsafe in
general). Since apache#26461 the check runs inside `DefaultPhysicalPlanner`
and rejects it. This is the only failure in the datafusion-python
upgrade to 56 (apache/datafusion-python#1826). The issue has the full
error, the plan and the 55 → 56 history.

## What changes are included in this PR?

- `FFI_ExecutionPlan` gets four callbacks:
`input_distribution_requirements`, `required_input_ordering`,
`maintains_input_order` and `benefits_from_input_partitioning`.
`ForeignExecutionPlan` overrides the matching methods, and also the
deprecated `required_input_distribution` so it agrees.
- `ForeignExecutionPlan` fetches the requirements once in
`TryFrom<FFI_ExecutionPlan>`, the way it already does `properties`,
instead of crossing FFI on every optimizer call. A malformed ordering
requirement fails the conversion rather than being dropped.
- `FFI_PhysicalExpr` gets an `eq` callback, so two
`ForeignPhysicalExpr`s wrapping equal expressions from the same library
compare equal. They used to compare by identity. Once requirements cross
FFI, that could make planning fail `SanityCheckPlan` when a foreign
plan's foreign child already met its partitioning or ordering
requirement. Expressions from different libraries always compare
unequal, since one library can't safely read another's private data.
- New stable types:
- `FFI_Distribution` and `FFI_InputDistributionRequirements` in
`physical_expr/partitioning.rs`. Co-partitioning crosses as a bool,
because `InputDistributionRequirements` can only be built with all
children co-partitioned or none.
- `FFI_PhysicalSortRequirement` and `FFI_OrderingRequirements`
(hard/soft alternatives) in `physical_expr/sort.rs`.
- The deprecated `Distribution::HashPartitioned` crosses as
`KeyPartitioned`. The optimizer treats the two the same, and this keeps
the deprecated variant out of the ABI.
- A note in the 56.0.0 upgrade guide covering both layout changes. It is
in the 56 guide because this is going into 56.0.0 (see the last
section).

## What is the testing strategy for this PR?

- `test_ensure_requirements_sees_foreign_plan_requirements` reproduces
the bug. It plans `SELECT a % 2, count(*) ... GROUP BY 1` with
`target_partitions = 4`, removes the hash repartition, sends the
`FinalPartitioned` aggregate across a mock FFI boundary, and runs
`EnsureRequirements`. It asserts that a hash repartition is inserted and
that `check_plan_tree_sanity` passes. Without the fix it fails: the plan
gets `RoundRobinBatch(4)`.
- `test_ffi_execution_plan_input_requirements_round_trip`: default and
non-default requirements, including co-partitioning, soft ordering and
the deprecated per-child accessor, come back unchanged from a foreign
plan.
- `foreign_requirements_met_by_foreign_child` in the FFI integration
tests crosses a real cdylib. The foreign plan requires
`KeyPartitioned([a])` over a `Hash([a], 4)` child, or a hard ordering on
`a` over a `SortExec` on `a`. `SELECT * FROM t` must plan without adding
a repartition or sort. With identity equality it fails
`SanityCheckPlan`.
- `test_ffi_physical_expr_eq_across_library_images` loads two copies of
the test library with `get_module_copy`. The same `Column` from each
copy compares unequal, and two from the same copy compare equal. It
fails if the library check in `eq` is removed.
- Unit round-trip tests for the new FFI types in `partitioning.rs` and
`sort.rs`.
- End to end: I cherry-picked this onto the `branch-56` commit
apache/datafusion-python#1826 builds against. The previously failing
`test_spawning_plan_across_three_libraries` now passes, along with the
rest of that repo's FFI example suites. That run predates the `eq`
callback and the requirement caching.

## Are there any user-facing changes?

The `FFI_ExecutionPlan` and `FFI_PhysicalExpr` layouts change, so FFI
providers and consumers need to be rebuilt against the same release.
Foreign plans now report their real input requirements to optimizer
rules.

I'll backport this to `branch-56` before 56.0.0 is final (only
`56.0.0-rc1` is tagged so far). The datafusion-python upgrade to 56
needs it, and backporting before the release means 56.0.0 and later
56.0.x releases share one layout. The change applies to `branch-56`
without conflicts.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

(cherry picked from commit 701e556)
@xudong963 xudong963 added the api change Changes the API exposed to users of the crate label Nov 18, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @xudong963

@codecov-commenter

codecov-commenter commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.63456% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.30%. Comparing base (7f00716) to head (76acb4e).

Files with missing lines Patch % Lines
datafusion/ffi/src/physical_expr/sort.rs 81.33% 10 Missing and 4 partials ⚠️
datafusion/ffi/src/execution_plan.rs 94.05% 2 Missing and 10 partials ⚠️
Flag Coverage Δ
all-tests 83.30% <92.63%> (+<0.01%) ⬆️
sqllogictests 60.85% <ø> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@             Coverage Diff             @@
##           branch-56   #27541    +/-   ##
===========================================
  Coverage      83.29%   83.30%            
===========================================
  Files           1170     1170            
  Lines         475858   476209   +351     
  Branches      475858   476209   +351     
===========================================
+ Hits          396361   396695   +334     
- Misses         55479    55484     +5     
- Partials       24018    24030    +12     
Files with missing lines Coverage Δ
datafusion/ffi/src/physical_expr/mod.rs 87.79% <100.00%> (+0.17%) ⬆️
datafusion/ffi/src/physical_expr/partitioning.rs 94.36% <100.00%> (+2.47%) ⬆️
datafusion/ffi/src/execution_plan.rs 90.11% <94.05%> (+1.51%) ⬆️
datafusion/ffi/src/physical_expr/sort.rs 84.78% <81.33%> (-15.22%) ⬇️

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

xudong963 added a commit that referenced this pull request Nov 18, 2026
…t (backport #27407) (#27542)

## Which issue does this PR close?

Backports #27407 to `branch-56` for the 56.0.0 release tracked in
#24461. The same fix was already backported to `branch-55` (#27408) and
`branch-54` (#27409).

Related to #27431 and #27388.

## Rationale for this change

`cargo audit` fails on `branch-56` with 3 vulnerabilities in the locked
`quinn-proto` and `quinn-udp` versions:

- `quinn-proto` 0.11.18:
[RUSTSEC-2026-0523](https://rustsec.org/advisories/RUSTSEC-2026-0523),
[RUSTSEC-2026-0524](https://rustsec.org/advisories/RUSTSEC-2026-0524)
- `quinn-udp` 0.5.14:
[RUSTSEC-2026-0538](https://rustsec.org/advisories/RUSTSEC-2026-0538)

These versions are in the 56.0.0 RC1 lockfile. They also fail the
`security_audit` check on #27541. RC2 is being cut anyway, so this
backport keeps the vulnerable versions out of the released lockfile.

## What changes are included in this PR?

Clean cherry-pick of 73e0b03. Only
`Cargo.lock` changes:

| Crate         | Before  | After   |
| ------------- | ------- | ------- |
| `quinn-proto` | 0.11.18 | 0.11.19 |
| `quinn-udp`   | 0.5.14  | 0.5.16  |

## What is the testing strategy for this PR?

Covered by the `security_audit` CI job. Locally, `cargo audit` (0.22.2)
reports the 3 vulnerabilities above on the current `branch-56` lockfile.
After this change it exits with 0 vulnerabilities, leaving only the
allowed warnings.

## Are there any user-facing changes?

No. These are semver-compatible dependency updates in `Cargo.lock` only.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
@xudong963
xudong963 merged commit 69d40d1 into apache:branch-56 Nov 18, 2026
40 checks passed
xudong963 added a commit that referenced this pull request Nov 18, 2026
## Which issue does this PR close?

- Part of #24461

## Rationale for this change

#27541 and #27542 were backported to `branch-56` for RC2 after the
changelog was regenerated in #27382, so they are missing from
`dev/changelog/56.0.0.md`.

## What changes are included in this PR?

Regenerates the changelog with `dev/release/generate-changelog.py 55.0.0
69d40d1 56.0.0`, where 69d40d1 is the
current `branch-56` head, then formats it with `prettier`. This adds
entries for:

- #27541 (listed under breaking changes because of its `api change`
label: the `FFI_ExecutionPlan` and `FFI_PhysicalExpr` layouts change)
- #27542
- #27382

It also updates the commit count (1206 → 1209) and the credits.

The author shown for #25337 changes from `dd-annarose` to `anna-lscr`.
The script reads the current GitHub login for the commit author, so this
reflects a renamed account.

## What is the testing strategy for this PR?

N/A (changelog only). `ci/scripts/doc_prettier_check.sh` passes.

## Are there any user-facing changes?

No

🤖 Generated with [Claude Code](https://claude.com/claude-code)
andygrove added a commit to nuno-faria/datafusion-python that referenced this pull request Nov 18, 2026
The rc2 tag includes the backported FFI_ExecutionPlan input-requirements
fix (apache/datafusion#27541), which resolves the
_test_three_library_query_planner SanityCheckPlan failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api change Changes the API exposed to users of the crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants