Skip to content

Upgrade DataFusion to version 56 - #1826

Open
nuno-faria wants to merge 5 commits into
apache:mainfrom
nuno-faria:upgrade_datafusion_56
Open

nuno-faria wants to merge 5 commits into
apache:mainfrom
nuno-faria:upgrade_datafusion_56

Conversation

@nuno-faria

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

N/A.

Rationale for this change

A new DataFusion version is being released: apache/datafusion#24461

What changes are included in this PR?

There are new types like Show (which replaced DescribeTable), AsOfJoin, and MaterializedCte, as well as some other minor changes. I will add them as comments.

Are there any user-facing changes?

New DF version. Other than that the main breaking change is that DescribeTable has been removed. I added a note to the upgrade guide.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}

/// Retrieves the right input `LogicalPlan` to this `Join` node
/// Retrieves the equality keys of this `Join` node

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noticed when creating the ASOF join that the prev comment was wrong.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

&self,
_ctx: &dyn Session,
projection: Option<&Vec<usize>>,
projection: Option<&[usize]>,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines 5352 to +5388
@@ -5385,7 +5385,7 @@ def map_entries(map: Expr) -> Expr:
def element_at(map: Expr, key: Expr) -> Expr:
"""Returns the value for a given key in the map.

Returns ``[None]`` if the key is absent.
Returns ``[]`` if the key is absent.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


def test_sql_with_options_no_statements(ctx):
sql = "SET time zone = 1;"
sql = "SET time zone = 'UTC';"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timezone is now validated.

Comment on lines 1792 to +1814
@@ -1811,7 +1811,7 @@ def test_read_csv_with_options(tmp_path, as_read, global_ctx):
[
pa.array(["Alice", "Bob", "Phil", "Karin"]),
pa.array([30.0, 25.0, 75.0, 50.0]),
pa.array(["New York; NY", None, "Detroit' MI", "Stockholm\nSweden"]),
pa.array(["New York; NY", "", "Detroit' MI", "Stockholm\nSweden"]),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now appears as an empty string since it does not match the null_regex. From apache/datafusion#25254

Comment on lines +2722 to +2724
with pytest.raises(
Exception, match=r"Invalid compression level \d+ for \w+\. Expected a level"
):

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nuno-faria

Copy link
Copy Markdown
Contributor Author

@timsaucer I think the problem with the _test_three_library_query_planner comes from the FFI crate upstream:

Exception: DataFusion error: FFI error: SanityCheckPlan: the plan produced by physical optimizer
rule 'EnsureRequirements' does not satisfy the order and distribution requirements

caused by

Error during planning:

Plan:
  AggregateExec: mode=FinalPartitioned, gby=[numbers.A % Int64(2)@0 as numbers.A % Int64(2)], aggr=[count(Int64(1))]
    FFI_ExecutionPlan: RepartitionExec, number_of_children=1
      AggregateExec: mode=Partial, gby=[CAST(A@0 AS Int64) % 2 as numbers.A % Int64(2)], aggr=[count(Int64(1))]
        FFI_ExecutionPlan: DataSourceExec, number_of_children=0

does not satisfy distribution requirements: KeyPartitioned[[numbers.A % Int64(2)@0]])

Child-0 output partitioning: RoundRobinBatch(4)

It appears that the problem is now shown by apache/datafusion#26461.

@andygrove

Copy link
Copy Markdown
Member

@nuno-faria you were right that the _test_three_library_query_planner failure comes from upstream. I filed apache/datafusion#27427 and opened a fix in apache/datafusion#27428.

FFI_ExecutionPlan never carried a plan's input requirements: input distribution, required ordering, maintains_input_order. When the host's optimizer rules run across FFI on the planner library's plan, EnsureRequirements can't see that FinalPartitioned needs hash-partitioned input, so it adds only a round-robin repartition. On 55, SanityCheckPlan also ran as a host-side rule and was just as blind, so the test passed. apache/datafusion#26461 moved the check into DefaultPhysicalPlanner, so 56 rejects the plan.

I cherry-picked the fix onto the branch-56 commit this PR pins (0f3b5a08) and built this branch against it. The query-planner example suite passes (51/51, including test_spawning_plan_across_three_libraries), and so does the FFI example suite (59/59). Once the fix lands on branch-56 and the pin here is bumped, this test should pass without changes on our side.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an AI generated review. Feel free to ignore any points that you do not agree with.

Blocking

  1. test_spawning_plan_across_three_libraries fails because of an upstream FFI bug. FFI_ExecutionPlan doesn't carry input requirements across the boundary. Tracked in apache/datafusion#27427, with a fix in apache/datafusion#27428. With the fix applied to branch-56, this PR passes all FFI example tests (51/51 and 59/59).
  2. CI's TPC-H step will fail next. tpchgen-cli 4.0 removed --format. Here the step was skipped only because the FFI step failed first. #1832 fixes it, and this PR will need to rebase onto it.
  3. Dependency source. [patch.crates-io] points at a moving git branch-56, and the comment above it says to remove it before a release candidate. For now it could bump to the current branch-56 tip (cce6374e6, which includes the 56.0.0 version bump). I ran the full suite against that tip with the FFI fix applied, and it is clean: main suite with doctests, both FFI example suites, TPC-H 22/22 and clippy. Once 56.0.0 is on crates.io it should switch to that.

Should fix

  1. Show.schema() returns None for DESCRIBE <table> (crates/core/src/expr/show.rs). Previously DescribeTable.schema() returned the table's schema in this case. Part of this comes from upstream: ShowKind::Columns now holds only table_ref and detail, with no schema. But the binding exposes no table_ref and none of the other kinds' payloads (variable name, function filter, etc.), so users can't get the table back. The upgrade guide shows kind() == "columns" but doesn't mention that schema() is now None. It could suggest ctx.table(name).schema() instead.
  2. Some user-visible behavior changes are missing from upgrade-guides.md. The tests and docstrings were updated for these, but the guide wasn't:
    • map_extract / element_at with a missing key now returns [] instead of [None].
    • CSV null_regex now also applies to the data, not only to schema inference (apache/datafusion#25254). Fields that don't match the regex, including the empty fields of truncated rows, come back as "" instead of null. This looks like intended upstream behavior and the test update is correct, but it is worth a note.
    • Time zone values in SET are now validated.
  3. The new bindings aren't covered from Python. test_asof_join and test_materialized_cte only compare display_indent() output, which is upstream's Rust Display. No test reaches AsOfJoin, MaterializedCte, MaterializedCteScan, CreateExternalCatalog or DropCatalog through to_variant() or their accessors (match_condition(), on(), etc.). A wrong match arm or a broken accessor in PyLogicalPlan::to_variant would still pass.
  4. crates/core/src/expr/describe_table.rs is orphaned. Its mod line was removed, but the file is still tracked and still references the removed upstream DescribeTable. It can be deleted.
  5. MSRV. The workspace has rust-version = "1.88", but DataFusion 56 declares rust-version = "1.95.0" (and rust-toolchain.toml is now 1.99.0). It should be raised to at least 1.95.

Minor

  1. The Python pin also applies to Windows. In .github/workflows/build.yml, uv sync --python 3.14 is in the job whose matrix is os: [macos-latest, windows-latest], so the Windows abi3 build is pinned too, while the commit message says macOS only. The pin is also repeated in two jobs.
  2. Show(schema, output_schema) can only build ShowKind::Describe, and kind() returns bare strings. That's fine for now, but a docstring note listing the possible kind() values would help users who match on them.

philippemnoel pushed a commit to paradedb/datafusion that referenced this pull request Nov 18, 2026
## Which issue does this PR close?

- Closes apache#27427.

## Rationale for this change

A `ForeignExecutionPlan` reports the `ExecutionPlan` defaults for its
input requirements (`UnspecifiedDistribution`, no required ordering,
`maintains_input_order = false`), whatever the wrapped plan requires.
When a physical optimizer rule runs on the other side of the FFI
boundary from the plan it optimizes, it can't see those requirements.
That happens with a foreign `QueryPlanner` whose session rules belong to
the host. `EnsureRequirements` then puts only a round-robin repartition
under a `FinalPartitioned` aggregate, which can split one group key
across several partitions.

For FFI query planner users this is a regression from 55 to 56: a query
that works on 55.0.0 fails at planning time on 56. Until apache#26461,
`SanityCheckPlan` ran as a host-side rule and was just as blind, so the
plan got through (correct by luck in the affected test, unsafe in
general). Since apache#26461 the check runs inside `DefaultPhysicalPlanner`
and rejects it. This is the only failure in the datafusion-python
upgrade to 56 (apache/datafusion-python#1826). The issue has the full
error, the plan and the 55 → 56 history.

## What changes are included in this PR?

- `FFI_ExecutionPlan` gets four callbacks:
`input_distribution_requirements`, `required_input_ordering`,
`maintains_input_order` and `benefits_from_input_partitioning`.
`ForeignExecutionPlan` overrides the matching methods, and also the
deprecated `required_input_distribution` so it agrees.
- `ForeignExecutionPlan` fetches the requirements once in
`TryFrom<FFI_ExecutionPlan>`, the way it already does `properties`,
instead of crossing FFI on every optimizer call. A malformed ordering
requirement fails the conversion rather than being dropped.
- `FFI_PhysicalExpr` gets an `eq` callback, so two
`ForeignPhysicalExpr`s wrapping equal expressions from the same library
compare equal. They used to compare by identity. Once requirements cross
FFI, that could make planning fail `SanityCheckPlan` when a foreign
plan's foreign child already met its partitioning or ordering
requirement. Expressions from different libraries always compare
unequal, since one library can't safely read another's private data.
- New stable types:
- `FFI_Distribution` and `FFI_InputDistributionRequirements` in
`physical_expr/partitioning.rs`. Co-partitioning crosses as a bool,
because `InputDistributionRequirements` can only be built with all
children co-partitioned or none.
- `FFI_PhysicalSortRequirement` and `FFI_OrderingRequirements`
(hard/soft alternatives) in `physical_expr/sort.rs`.
- The deprecated `Distribution::HashPartitioned` crosses as
`KeyPartitioned`. The optimizer treats the two the same, and this keeps
the deprecated variant out of the ABI.
- A note in the 56.0.0 upgrade guide covering both layout changes. It is
in the 56 guide because this is going into 56.0.0 (see the last
section).

## What is the testing strategy for this PR?

- `test_ensure_requirements_sees_foreign_plan_requirements` reproduces
the bug. It plans `SELECT a % 2, count(*) ... GROUP BY 1` with
`target_partitions = 4`, removes the hash repartition, sends the
`FinalPartitioned` aggregate across a mock FFI boundary, and runs
`EnsureRequirements`. It asserts that a hash repartition is inserted and
that `check_plan_tree_sanity` passes. Without the fix it fails: the plan
gets `RoundRobinBatch(4)`.
- `test_ffi_execution_plan_input_requirements_round_trip`: default and
non-default requirements, including co-partitioning, soft ordering and
the deprecated per-child accessor, come back unchanged from a foreign
plan.
- `foreign_requirements_met_by_foreign_child` in the FFI integration
tests crosses a real cdylib. The foreign plan requires
`KeyPartitioned([a])` over a `Hash([a], 4)` child, or a hard ordering on
`a` over a `SortExec` on `a`. `SELECT * FROM t` must plan without adding
a repartition or sort. With identity equality it fails
`SanityCheckPlan`.
- `test_ffi_physical_expr_eq_across_library_images` loads two copies of
the test library with `get_module_copy`. The same `Column` from each
copy compares unequal, and two from the same copy compare equal. It
fails if the library check in `eq` is removed.
- Unit round-trip tests for the new FFI types in `partitioning.rs` and
`sort.rs`.
- End to end: I cherry-picked this onto the `branch-56` commit
apache/datafusion-python#1826 builds against. The previously failing
`test_spawning_plan_across_three_libraries` now passes, along with the
rest of that repo's FFI example suites. That run predates the `eq`
callback and the requirement caching.

## Are there any user-facing changes?

The `FFI_ExecutionPlan` and `FFI_PhysicalExpr` layouts change, so FFI
providers and consumers need to be rebuilt against the same release.
Foreign plans now report their real input requirements to optimizer
rules.

I'll backport this to `branch-56` before 56.0.0 is final (only
`56.0.0-rc1` is tagged so far). The datafusion-python upgrade to 56
needs it, and backporting before the release means 56.0.0 and later
56.0.x releases share one layout. The change applies to `branch-56`
without conflicts.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
xudong963 added a commit to apache/datafusion that referenced this pull request Nov 18, 2026
…ackport #27428) (#27541)

## Which issue does this PR close?

Backports #27428 (fix for #27427) to `branch-56` for the 56.0.0 release
tracked in #24461.

## Rationale for this change

The datafusion-python upgrade exposed a planning regression in 56.0.0
RC1. When the query planner and optimizer run on opposite sides of the
FFI boundary, the optimizer can miss an aggregate's hash partitioning
requirement and produce a plan that fails `SanityCheckPlan`. This blocks
apache/datafusion-python#1826.

## What changes are included in this PR?

Clean cherry-pick of 701e556. Foreign
execution plans expose their input requirements, and physical
expressions originating in the same library can be compared for
equality. The 56.0.0 upgrade guide describes both ABI changes.

## What is the testing strategy for this PR?

Verified on the backport branch:

- `cargo clippy --all-targets --all-features -- -D warnings`
- `cargo test -p datafusion-ffi -p datafusion-ffi-integration-tests
--features integration-tests`, including the `EnsureRequirements`
regression and the cross-library plan-requirements and
expression-equality tests
- `cargo fmt --all` and the repository's Markdown formatting check

Local `cargo audit` reports RUSTSEC-2026-0523, RUSTSEC-2026-0524 and
RUSTSEC-2026-0538 for the existing `quinn-proto` and `quinn-udp`
versions. The unmodified RC1 lockfile reproduces all three findings.
This backport preserves the dependency versions.

## Are there any user-facing changes?

Yes. Foreign plans preserve their input requirements. The layouts of
`FFI_ExecutionPlan` and `FFI_PhysicalExpr` change. FFI providers and
consumers built against RC1 must be rebuilt together for RC2. Including
the fix before 56.0.0 is final keeps one ABI for the published 56.x
series.

Co-authored-by: Andy Grove <agrove@apache.org>
The rc2 tag includes the backported FFI_ExecutionPlan input-requirements
fix (apache/datafusion#27541), which resolves the
_test_three_library_query_planner SanityCheckPlan failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andygrove

Copy link
Copy Markdown
Member

@timsaucer @nuno-faria I pushed a commit to bump the dependency to 56.0.0-rc2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants