Skip to content

fix: carry input requirements across FFI_ExecutionPlan - #27428

Merged
andygrove merged 3 commits into
apache:mainfrom
andygrove:ffi-plan-input-requirements
Nov 17, 2026
Merged

andygrove merged 3 commits into
apache:mainfrom
andygrove:ffi-plan-input-requirements

Conversation

@andygrove

@andygrove andygrove commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

A ForeignExecutionPlan reports the ExecutionPlan defaults for its input requirements (UnspecifiedDistribution, no required ordering, maintains_input_order = false), whatever the wrapped plan requires. When a physical optimizer rule runs on the other side of the FFI boundary from the plan it optimizes, it can't see those requirements. That happens with a foreign QueryPlanner whose session rules belong to the host. EnsureRequirements then puts only a round-robin repartition under a FinalPartitioned aggregate, which can split one group key across several partitions.

For FFI query planner users this is a regression from 55 to 56: a query that works on 55.0.0 fails at planning time on 56. Until #26461, SanityCheckPlan ran as a host-side rule and was just as blind, so the plan got through (correct by luck in the affected test, unsafe in general). Since #26461 the check runs inside DefaultPhysicalPlanner and rejects it. This is the only failure in the datafusion-python upgrade to 56 (apache/datafusion-python#1826). The issue has the full error, the plan and the 55 → 56 history.

What changes are included in this PR?

  • FFI_ExecutionPlan gets four callbacks: input_distribution_requirements, required_input_ordering, maintains_input_order and benefits_from_input_partitioning. ForeignExecutionPlan overrides the matching methods, and also the deprecated required_input_distribution so it agrees.
  • ForeignExecutionPlan fetches the requirements once in TryFrom<FFI_ExecutionPlan>, the way it already does properties, instead of crossing FFI on every optimizer call. A malformed ordering requirement fails the conversion rather than being dropped.
  • FFI_PhysicalExpr gets an eq callback, so two ForeignPhysicalExprs wrapping equal expressions from the same library compare equal. They used to compare by identity. Once requirements cross FFI, that could make planning fail SanityCheckPlan when a foreign plan's foreign child already met its partitioning or ordering requirement. Expressions from different libraries always compare unequal, since one library can't safely read another's private data.
  • New stable types:
    • FFI_Distribution and FFI_InputDistributionRequirements in physical_expr/partitioning.rs. Co-partitioning crosses as a bool, because InputDistributionRequirements can only be built with all children co-partitioned or none.
    • FFI_PhysicalSortRequirement and FFI_OrderingRequirements (hard/soft alternatives) in physical_expr/sort.rs.
  • The deprecated Distribution::HashPartitioned crosses as KeyPartitioned. The optimizer treats the two the same, and this keeps the deprecated variant out of the ABI.
  • A note in the 56.0.0 upgrade guide covering both layout changes. It is in the 56 guide because this is going into 56.0.0 (see the last section).

What is the testing strategy for this PR?

  • test_ensure_requirements_sees_foreign_plan_requirements reproduces the bug. It plans SELECT a % 2, count(*) ... GROUP BY 1 with target_partitions = 4, removes the hash repartition, sends the FinalPartitioned aggregate across a mock FFI boundary, and runs EnsureRequirements. It asserts that a hash repartition is inserted and that check_plan_tree_sanity passes. Without the fix it fails: the plan gets RoundRobinBatch(4).
  • test_ffi_execution_plan_input_requirements_round_trip: default and non-default requirements, including co-partitioning, soft ordering and the deprecated per-child accessor, come back unchanged from a foreign plan.
  • foreign_requirements_met_by_foreign_child in the FFI integration tests crosses a real cdylib. The foreign plan requires KeyPartitioned([a]) over a Hash([a], 4) child, or a hard ordering on a over a SortExec on a. SELECT * FROM t must plan without adding a repartition or sort. With identity equality it fails SanityCheckPlan.
  • test_ffi_physical_expr_eq_across_library_images loads two copies of the test library with get_module_copy. The same Column from each copy compares unequal, and two from the same copy compare equal. It fails if the library check in eq is removed.
  • Unit round-trip tests for the new FFI types in partitioning.rs and sort.rs.
  • End to end: I cherry-picked this onto the branch-56 commit Upgrade DataFusion to version 56 datafusion-python#1826 builds against. The previously failing test_spawning_plan_across_three_libraries now passes, along with the rest of that repo's FFI example suites. That run predates the eq callback and the requirement caching.

Are there any user-facing changes?

The FFI_ExecutionPlan and FFI_PhysicalExpr layouts change, so FFI providers and consumers need to be rebuilt against the same release. Foreign plans now report their real input requirements to optimizer rules.

I'll backport this to branch-56 before 56.0.0 is final (only 56.0.0-rc1 is tagged so far). The datafusion-python upgrade to 56 needs it, and backporting before the release means 56.0.0 and later 56.0.x releases share one layout. The change applies to branch-56 without conflicts.

🤖 Generated with Claude Code

@github-actions github-actions Bot added documentation Improvements or additions to documentation ffi Changes to the ffi crate labels Nov 14, 2026

@viirya viirya left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down. The four callbacks are the right set: every other method ForeignExecutionPlan doesn't forward has a conservative default.

Two things before merging:

  1. Backport target. version() reports only the major version, and the new fields shift clone/release/version. Shipping this in a 56.0.x patch would leave 56.0.0 and 56.0.x libraries with different layouts and no way to detect the mismatch. Since 56 is still at rc1, can this target 56.0.0 instead, with the upgrade note in the 56 guide?
  2. Test coverage of foreign expressions. See the inline comment on the EnsureRequirements test. The cross-library path compares expressions by identity, and the test doesn't reach that path.

ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
)?;
let mut ffi = FFI_ExecutionPlan::new(stripped, None);
ffi.library_marker_id = crate::mock_foreign_marker_id;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the plan gets the mock marker. The expressions inside the requirements are created with the real marker, so they come back as native Columns. With a real library boundary they become ForeignPhysicalExpr, whose eq is ptr::eq, and every call to these callbacks builds new wrappers. On the host, a foreign requirement then never matches an existing partitioning or ordering. The result is still correct, but it can add repartitions or sorts that native planning wouldn't. Does the datafusion-python EXPLAIN show any? For example, an aggregate over a join on the same key, or a window over sorted input. If it does, could you open a follow-up issue, or add a test in ffi_integration that crosses a real cdylib?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with @viirya, and it's worse than extra repartitions: a correct foreign subtree now fails host planning. I returned a plan from the integration-test cdylib from a TableProvider::scan: its top node requires KeyPartitioned([a]) and its child is already Hash([a], 4) (same with a hard sort on a over SortExec(a)). SELECT * FROM t plans on the base of this PR, but here it fails with SanityCheckPlan: the plan produced by physical optimizer rule 'EnsureRequirements' does not satisfy the order and distribution requirements, because each requirement call builds new ForeignPhysicalExpr wrappers, which only equal themselves.

A structural eq callback on FFI_PhysicalExpr fixes both shapes, with no extra repartition or sort, and the FFI unit and integration suites pass with it. Since this PR already changes the ABI, could it go in here, with the test below and a line about FFI_PhysicalExpr in the 57 upgrade note?

--- a/datafusion/ffi/src/physical_expr/mod.rs
+++ b/datafusion/ffi/src/physical_expr/mod.rs
@@ pub struct FFI_PhysicalExpr {
     pub hash: unsafe extern "C" fn(&Self) -> u64,
 
+    /// Structural equality with `other`. Returns `false` when `other` was
+    /// created by a different library.
+    pub eq: unsafe extern "C" fn(&Self, other: &Self) -> bool,
+
@@
+unsafe extern "C" fn eq_fn_wrapper(
+    expr: &FFI_PhysicalExpr,
+    other: &FFI_PhysicalExpr,
+) -> bool {
+    // `other.inner()` is only valid when this library created `other`.
+    (other.library_marker_id)() == crate::get_library_marker_id()
+        && expr.inner() == other.inner()
+}
@@ (both constructors: `clone_fn_wrapper` and `From<Arc<dyn PhysicalExpr>>`)
             hash: hash_fn_wrapper,
+            eq: eq_fn_wrapper,
@@ impl PartialEq for ForeignPhysicalExpr {
-        // FFI_PhysicalExpr cannot be compared, so identity equality is the best we can do.
-        std::ptr::eq(self, other)
+        std::ptr::eq(self, other) || unsafe { (self.expr.eq)(&self.expr, &other.expr) }
@@ fn ffi_physical_expr_hash() {
-        assert_ne!(&foreign_1, &foreign_2);
+        assert_eq!(&foreign_1, &foreign_2);
Cross-library test: fails on this PR with SanityCheckPlan, passes with the fix

datafusion/ffi/src/tests/mod.rs (producer, exported as its own symbol like datafusion_ffi_test_create_exec_with_byte_metrics):

/// A plan whose top node requires its input hash-partitioned (or sorted) on
/// `a`, and whose child already provides it.
#[unsafe(no_mangle)]
pub extern "C" fn datafusion_ffi_test_create_exec_with_requirements(
    hard_ordering: bool,
) -> FFI_ExecutionPlan {
    use crate::execution_plan::tests::TestInputRequirements;
    use datafusion_physical_expr::Distribution;
    use datafusion_physical_expr_common::sort_expr::{
        LexRequirement, OrderingRequirements, PhysicalSortRequirement,
    };
    use datafusion_physical_plan::InputDistributionRequirements;
    use datafusion_physical_plan::repartition::RepartitionExec;
    use datafusion_physical_plan::sorts::sort::SortExec;

    let schema = Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, false)]));
    let a = Arc::new(Column::new("a", 0)) as Arc<dyn PhysicalExpr>;
    let leaf = Arc::new(EmptyExec::new(Arc::clone(&schema))) as Arc<dyn ExecutionPlan>;
    let (child, reqs): (Arc<dyn ExecutionPlan>, _) = if hard_ordering {
        let ord =
            LexOrdering::new([PhysicalSortExpr::new_default(Arc::clone(&a))]).unwrap();
        let req = OrderingRequirements::new(
            LexRequirement::new([PhysicalSortRequirement::new(Arc::clone(&a), None)])
                .unwrap(),
        );
        (
            Arc::new(SortExec::new(ord, leaf).with_preserve_partitioning(true)),
            TestInputRequirements {
                distribution: InputDistributionRequirements::new(vec![
                    Distribution::UnspecifiedDistribution,
                ]),
                ordering: vec![Some(req)],
                maintains_input_order: vec![true],
                benefits_from_input_partitioning: vec![false],
            },
        )
    } else {
        (
            Arc::new(
                RepartitionExec::try_new(
                    leaf,
                    Partitioning::Hash(vec![Arc::clone(&a)], 4),
                )
                .unwrap(),
            ),
            TestInputRequirements {
                distribution: InputDistributionRequirements::new(vec![
                    Distribution::KeyPartitioned(vec![Arc::clone(&a)]),
                ]),
                ordering: vec![None],
                maintains_input_order: vec![false],
                benefits_from_input_partitioning: vec![true],
            },
        )
    };
    let plan = EmptyExec::new(schema)
        .with_children(vec![child])
        .with_input_requirements(reqs);
    FFI_ExecutionPlan::new(Arc::new(plan), None)
}

datafusion/ffi/src/tests/utils.rs (loader):

pub fn get_requirements_exec(
    hard_ordering: bool,
) -> Result<crate::execution_plan::FFI_ExecutionPlan> {
    let lib_path = find_library()?;
    let lib = unsafe {
        libloading::Library::new(&lib_path)
            .map_err(|e| DataFusionError::External(Box::new(e)))?
    };
    let create_exec: libloading::Symbol<
        extern "C" fn(bool) -> crate::execution_plan::FFI_ExecutionPlan,
    > = unsafe {
        lib.get(b"datafusion_ffi_test_create_exec_with_requirements")
            .map_err(|e| DataFusionError::External(Box::new(e)))?
    };
    let plan = create_exec(hard_ordering);
    #[expect(clippy::mem_forget)]
    std::mem::forget(lib);
    Ok(plan)
}

datafusion/ffi/integration-tests/tests/cases/execution_plan.rs (consumer; needs async-trait in the crate's dev-dependencies):

mod foreign_requirements {
    use std::sync::Arc;

    use async_trait::async_trait;
    use datafusion::catalog::{Session, TableProvider};
    use datafusion::logical_expr::{Expr, TableType};
    use datafusion::prelude::SessionContext;
    use datafusion_common::Result;
    use datafusion_ffi::tests::utils::get_requirements_exec;
    use datafusion_physical_plan::{ExecutionPlan, displayable};

    /// A host-side table whose scan returns a plan built by another library.
    #[derive(Debug)]
    struct ForeignScanTable(Arc<dyn ExecutionPlan>);

    #[async_trait]
    impl TableProvider for ForeignScanTable {
        fn schema(&self) -> arrow_schema::SchemaRef {
            self.0.schema()
        }
        fn table_type(&self) -> TableType {
            TableType::Base
        }
        async fn scan(
            &self,
            _: &dyn Session,
            _: Option<&[usize]>,
            _: &[Expr],
            _: Option<usize>,
        ) -> Result<Arc<dyn ExecutionPlan>> {
            Ok(Arc::clone(&self.0))
        }
    }

    /// A correct foreign subtree (its top node's requirement is met by its own
    /// child) must plan on the host without errors or extra repartitions/sorts.
    #[tokio::test]
    async fn foreign_requirements_met_by_foreign_child() -> Result<()> {
        for hard_ordering in [false, true] {
            let ffi = get_requirements_exec(hard_ordering)?;
            let foreign: Arc<dyn ExecutionPlan> = (&ffi).try_into()?;
            let ctx = SessionContext::new();
            ctx.register_table("t", Arc::new(ForeignScanTable(foreign)))?;
            let plan = ctx
                .sql("SELECT * FROM t")
                .await?
                .create_physical_plan()
                .await?;
            let display = displayable(plan.as_ref()).indent(false).to_string();
            assert!(
                !display.contains("\n  RepartitionExec: partitioning=Hash")
                    && !display.contains("\n  SortExec"),
                "redundant repartition or sort added:\n{display}"
            );
        }
        Ok(())
    }
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied your fix and test, thanks. I confirmed the test fails without the eq callback. The 57 upgrade note now covers FFI_PhysicalExpr too.

self.input_distribution_requirements().into_per_child()
}

fn input_distribution_requirements(&self) -> InputDistributionRequirements {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are called many times per node during EnsureRequirements/EnforceSorting, and each call crosses FFI and rebuilds the expression wrappers. Requirements are fixed for a plan instance. Could they be converted once in TryFrom<FFI_ExecutionPlan>, the way properties is?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirements are now converted once in TryFrom<FFI_ExecutionPlan> (5a4be3e).

Comment thread datafusion/ffi/src/execution_plan.rs Outdated
FFI_Option::Some(ordering) => OrderingRequirements::try_from(ordering)
.inspect_err(|e| {
log::error!(
"Invalid required input ordering from foreign plan {}: {e}",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning None here drops a hard ordering requirement. As far as I can tell this can't happen with a well-formed provider, because LexRequirement and OrderingRequirements are non-empty by construction. Could a comment say so, so the fallback doesn't read as a safe default?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since conversion now happens in TryFrom<FFI_ExecutionPlan>, a malformed ordering fails construction with an error instead of falling back to None. A comment explains why a well-formed provider can't hit it.

/// One distribution per child.
distributions: SVec<FFI_Distribution>,
/// Whether all children must be co-partitioned.
co_partitioned: bool,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: InputDistributionRequirements stores co-partitioned child indexes and is #[non_exhaustive]. A short note that the bool relies on all-or-none construction would flag this spot if subsets become possible.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note added.

Comment thread datafusion/ffi/src/execution_plan.rs Outdated
unimplemented!()
}

fn required_input_distribution(&self) -> Vec<Distribution> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: The test never reaches this override, because ForeignExecutionPlan::required_input_distribution calls its own input_distribution_requirements. Can it be removed?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andygrove thanks for tracking this down. This changes the FFI_ExecutionPlan layout, so the PR needs the api change label (the upgrade-guide entry is already there). The FFI review checklist in .ai/skills/datafusion-ffi blocks merge until it's added.

@jayzhan211 jayzhan211 added the api change Changes the API exposed to users of the crate label Nov 15, 2026
@andygrove

Copy link
Copy Markdown
Member Author

@viirya Thanks! Addressed in 5a4be3e.

  1. Backport target: we open PRs against main and then backport to the release branch, so I'll backport this to branch-56 before 56.0.0 is final, with the upgrade note in the 56 guide. That way 56.0.0 and later 56.0.x releases share one layout.
  2. Foreign-expression coverage: I added foreign_requirements_met_by_foreign_child in ffi_integration, which uses a real cdylib. With identity equality it fails SanityCheckPlan, and it passes with the new FFI_PhysicalExpr::eq.

andygrove and others added 2 commits November 16, 2026 07:37
ForeignExecutionPlan fell back to the ExecutionPlan defaults for
input_distribution_requirements, required_input_ordering,
maintains_input_order and benefits_from_input_partitioning. Physical
optimizer rules running on the other side of the boundary, such as a
foreign QueryPlanner's session rules, could not see what the plan
required: EnsureRequirements added only a round-robin repartition under
a FinalPartitioned aggregate.

Add FFI callbacks and stable representations for Distribution,
InputDistributionRequirements, PhysicalSortRequirement and
OrderingRequirements, and forward them from ForeignExecutionPlan.

Closes apache#27427

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rements

Address review feedback:

- Add an `eq` callback to `FFI_PhysicalExpr` so `ForeignPhysicalExpr`s
  wrapping equal expressions from the same library compare equal. Identity
  equality meant a foreign plan's input requirements never matched its
  foreign children's partitioning or ordering, so host planning failed the
  sanity check (or added redundant repartitions/sorts).
- Fetch input requirements once when building `ForeignExecutionPlan`
  instead of crossing FFI on every optimizer call, and reject a plan with
  a malformed ordering requirement rather than silently dropping it.
- Document that the co-partitioned flag relies on all-or-none construction.
- Remove an unused `required_input_distribution` override in the test plan.
- Add a cross-library integration test and mention the `FFI_PhysicalExpr`
  layout change in the 57 upgrade guide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andygrove
andygrove force-pushed the ffi-plan-input-requirements branch from 5a4be3e to f7f5a51 Compare November 16, 2026 14:38
@andygrove

Copy link
Copy Markdown
Member Author

@andygrove thanks for tracking this down. This changes the FFI_ExecutionPlan layout, so the PR needs the api change label (the upgrade-guide entry is already there). The FFI review checklist in .ai/skills/datafusion-ffi blocks merge until it's added.

Added. Thanks @jayzhan211

@andygrove

Copy link
Copy Markdown
Member Author

@andygrove thanks for tracking this down. This changes the FFI_ExecutionPlan layout, so the PR needs the api change label (the upgrade-guide entry is already there). The FFI review checklist in .ai/skills/datafusion-ffi blocks merge until it's added.

Added. Thanks @jayzhan211

Note that when this is backported to branch-56 it may stop being a breaking API change in main/57

@viirya viirya left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround. Two small follow-ups inline: a test for the cross-library branch of the new eq, and where the upgrade note should live given the 56.0.0 target.

hasher.finish()
}

unsafe extern "C" fn eq_fn_wrapper(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The marker check is what keeps this from reading another library's private_data, but no test reaches the false branch. ffi_physical_expr_hash only exercises the same-library path, because ForeignPhysicalExpr stores ffi_expr.clone() and clone_fn_wrapper resets the marker. Could a test compare expressions from two images loaded with get_module_copy and assert they're unequal?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added test_ffi_physical_expr_eq_across_library_images in 0150c0f. It loads two copies with get_module_copy and asserts that the same Column from each copy compares unequal, while two from the same copy compare equal. I confirmed it fails with the marker check removed.

COPY (SELECT * FROM t ORDER BY c1) TO 'ordered.parquet';
```

### `FFI_ExecutionPlan` carries input requirements

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is going into 56.0.0, should this section move to 56.0.0.md here too, so main and branch-56 agree?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to 56.0.0.md in 0150c0f, so main and branch-56 agree.

… note to 56

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@timsaucer timsaucer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neither of my comments is a blocker to merge. I think this is very good and follows all the existing patterns in the repo. Thank you for fixing this!

Comment on lines +315 to +327
`benefits_from_input_partitioning`, so a `ForeignExecutionPlan` now reports the
same input requirements as the plan it wraps. Previously these fell back to the
`ExecutionPlan` defaults, so physical optimizer rules running across the FFI
boundary (for example with a foreign `QueryPlanner`) could produce plans that
did not satisfy them.

`FFI_PhysicalExpr` also has a new `eq` callback, so two `ForeignPhysicalExpr`s
wrapping equal expressions from the same library now compare equal. Previously
they compared by identity, so a foreign plan's requirements never matched its
children's partitioning or ordering on the host.

Both structs have a new layout; rebuild FFI providers and consumers against the
same release.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've seen my agents creating these kinds of notes in the upgrade guide, except I don't think they actually help anyone during upgrading. These are more like release note changes rather than instructions for how a user might need to upgrade.

Comment on lines +1269 to +1272
use datafusion::arrow::array::Int32Array;
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use datafusion::arrow::record_batch::RecordBatch;
use datafusion::common::tree_node::{Transformed, TreeNode};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should all the use statements within these test methods be up at the mod level instead?

@viirya viirya left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing everything. LGTM.

@andygrove

Copy link
Copy Markdown
Member Author

@Xuanwo I think this is ready to merge now, to unblock creating rc2

@andygrove

Copy link
Copy Markdown
Member Author

@jayzhan211 the api change label is on, and your FFI_PhysicalExpr::eq fix and test are in f7f5a51. Could you take another look when you get a chance? GitHub still shows your earlier changes-requested review, and it's the only thing blocking merge.

@andygrove
andygrove enabled auto-merge November 17, 2026 19:54
@andygrove
andygrove added this pull request to the merge queue Nov 17, 2026
Merged via the queue into apache:main with commit 701e556 Nov 17, 2026
44 checks passed
@andygrove
andygrove deleted the ffi-plan-input-requirements branch November 17, 2026 23:26
@alamb

alamb commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Thank you @jayzhan211 and @andygrove

xudong963 added a commit that referenced this pull request Nov 18, 2026
…ackport #27428) (#27541)

## Which issue does this PR close?

Backports #27428 (fix for #27427) to `branch-56` for the 56.0.0 release
tracked in #24461.

## Rationale for this change

The datafusion-python upgrade exposed a planning regression in 56.0.0
RC1. When the query planner and optimizer run on opposite sides of the
FFI boundary, the optimizer can miss an aggregate's hash partitioning
requirement and produce a plan that fails `SanityCheckPlan`. This blocks
apache/datafusion-python#1826.

## What changes are included in this PR?

Clean cherry-pick of 701e556. Foreign
execution plans expose their input requirements, and physical
expressions originating in the same library can be compared for
equality. The 56.0.0 upgrade guide describes both ABI changes.

## What is the testing strategy for this PR?

Verified on the backport branch:

- `cargo clippy --all-targets --all-features -- -D warnings`
- `cargo test -p datafusion-ffi -p datafusion-ffi-integration-tests
--features integration-tests`, including the `EnsureRequirements`
regression and the cross-library plan-requirements and
expression-equality tests
- `cargo fmt --all` and the repository's Markdown formatting check

Local `cargo audit` reports RUSTSEC-2026-0523, RUSTSEC-2026-0524 and
RUSTSEC-2026-0538 for the existing `quinn-proto` and `quinn-udp`
versions. The unmodified RC1 lockfile reproduces all three findings.
This backport preserves the dependency versions.

## Are there any user-facing changes?

Yes. Foreign plans preserve their input requirements. The layouts of
`FFI_ExecutionPlan` and `FFI_PhysicalExpr` change. FFI providers and
consumers built against RC1 must be rebuilt together for RC2. Including
the fix before 56.0.0 is final keeps one ABI for the published 56.x
series.

Co-authored-by: Andy Grove <agrove@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api change Changes the API exposed to users of the crate documentation Improvements or additions to documentation ffi Changes to the ffi crate v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FFI_ExecutionPlan loses input distribution and ordering requirements, so optimizer rules across FFI produce invalid plans

5 participants