Skip to content

Move the engine pin to afdc4212a5 and follow its settings, expression context and bind only describe changes - #773

Merged
evertlammerts merged 3 commits into
duckdb:mainfrom
evertlammerts:fix-main-pin-progress-bar
Nov 13, 2026
Merged

evertlammerts merged 3 commits into
duckdb:mainfrom
evertlammerts:fix-main-pin-progress-bar

Conversation

@el-marvin

@el-marvin el-marvin commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

The nightly Release workflow on main builds the sdist against the engine's main tip, and since duckdb/duckdb#28571 the client no longer compiles there: run 43314281155 stops on ClientConfig::enable_progress_bar, and the tip also breaks five sites in src/python_udf.cpp through duckdb/duckdb#28594. Main's own CI builds the pinned engine and stays green.

The progress bar is now a generic setting, so SetDefaultConfigArguments enables it for an interactive session with Settings::Set<EnableProgressBarSetting> at session scope, the value a user's SET also writes; a Jupyter kernel without ipywidgets sets the disable reason and leaves the setting off, as before. Every ExpressionExecutor now carries a Context, and the UDF wrappers take the client through HasClientContext and GetClientContext where they took GetContext. Two new cases read and set the progress bar from an interactive child, a script child and a kernel without ipywidgets. The macOS wheel jobs failed on the new engine's types.hpp, whose aligned delete needs macOS 10.13, because cmake/duckdb_loader.cmake compiled the ten binding objects with -mmacosx-version-min=10.7 while the engine's objects took the build's target (11.0 on the arm64 seed, 10.15 on the x86_64 one); the flag goes, so every object takes the deployment target the build sets instead of a second number kept in a second place. The Windows jobs failed on MSVC's C3493 in the engine's types.cpp (duckdb/duckdb#28622), which the engine fixed in duckdb/duckdb#28741, inside the new range. The fork's CI builds no wheels, so a pin move's first wheel build is the upstream one.

Four engine changes in the range reach the tests. duckdb/duckdb#28597 makes the whitespace and comments in front of a statement part of its text; the client follows the engine's text and adds no stripping of its own, as on the 2.0 line, so four expectations change and one case checks that a comment between two statements stays with the second. duckdb/duckdb#28611 drops invertible DECIMAL and widening integer casts from equalities, so n IN (1, 3.5) now reaches an Arrow source as n IN (1) and dec = 1.505 over a DECIMAL(10, 2) folds to false and never reads the source: two predicate cases and one LazyFrame case follow. duckdb/duckdb#28701 reverts the MERGE multi-match check as a trade the engine means to undo in 2.1, so the appender test that expects the refusal stays as it was under a strict xfail, which reports the day the engine restores the check. duckdb/duckdb#28690 makes BindStatement bind only, so the errors of the later planning passes (the plan depth limit, a positional join in a correlated subquery) move from con.sql to fetchall; the client's own bind path for a statement with parameter values, which builds its own Planner, now plans in PlanningMode::BIND_ONLY too, so a relation describes and fails at the same place with and without values, and one test class pins that place for both, and for a query appender, which is accepted at creation and refused at the flush. One consequence for a user: such a statement inside an explicit transaction now aborts it at fetchall, where before the error at con.sql left the transaction usable.

The pin moves from 23221edfd577 to afdc4212a58c, 718 commits. The fast suite passes on the box (16805 passed, 83 skipped, 224 xfailed). The roundtrip benchmark against the old pin, wall time at one thread, leaves fetchall over DOUBLE and DECIMAL within its spread and shows LIST 7 to 9 percent slower (71.3 against 77.8 ms for 100,000 rows of three BIGINT) with identical client code on that path: every LIST value makes its type with make_uniq<ListTypeInfo>, and engine commit 0170852ae31 put alignas(128) on a member of LogicalTypeInfo, so that allocation grew from 64 bytes to 256. The cost is on engine main and is for the engine to take up. The nightly benchmark gate still compares against a baseline at engine 71f47d410f and enforces nothing until it is regenerated.

@evertlammerts evertlammerts added the submodule-bump Intentional DuckDB submodule pin move label Nov 13, 2026
@el-marvin el-marvin changed the title Move the engine pin to 5c8008b4cf and follow its settings and expression context changes Move the engine pin to afdc4212a5 and follow its settings and expression context changes Nov 13, 2026

@el-marvin el-marvin left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested.

I reviewed f3ad6d9, two commits on merge base 6f03883: cd9c19f moves the engine to 5c8008b4cf86 and follows it, and f3ad6d9 moves it on to afdc4212a58c and drops the macOS version flag of the binding objects. I built the head and the merge base the same way (Release, the repository's two step uv install), ran the changed test files on both builds and the head's versions of them on the merge base build, ran the full fast suite on both, built weaker versions of the progress bar follow and one fix of the client's bind path in a clone of the head, and measured the fetch, conversion and appender paths at one thread with wall time and instruction counts.

The premise holds. At engine main the client does not compile: the nightly Release run 43314281155 stops on ClientConfig::enable_progress_bar, and the merge base's src/python_udf.cpp gives five errors against the new headers. The pointer is exactly afdc4212a58c3850bcb23db0dc46d205b07da048, the tip of engine main, 718 commits after 23221edfd577. Both compile follows sit at the right layer and match the engine: the progress bar is switched on with Settings::Set<EnableProgressBarSetting>(context, SetScope::SESSION, Value::BOOLEAN(true)), the call the engine's own ADBC driver makes after duckdb/duckdb#28571 (src/common/adbc/adbc.cpp), and ExpressionState::GetClientContext() throws the binder error the old GetContext() threw when there is no client, at all six call sites. Every forced expectation fails on the merge base build for the reason the description gives and passes on the head. The macOS change is right: it removes a second deployment target kept in a second place.

Four things must change before merging: the new progress bar tests pass with two wrong follows; the client's own bind path for statements with parameter values did not follow duckdb/duckdb#28690, so con.sql now raises a planning error at different times with and without values; the MERGE test now pins a result the engine calls a correctness regression; and the benchmark sentence of the description does not match what I measure, while the one regression I can reproduce, fetchall over LIST, is left unnamed.

Findings

1. The progress bar tests pass with two wrong follows: the setting written at global scope, and a return right after the Jupyter display function is set (tests/fast/api/test_query_progress.py:115, inline).

Severity: should fix. Category: missing coverage. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.

Ask: in the interactive child, also print a cursor's value and the value after RESET enable_progress_bar; in the widget child, print current_setting('enable_progress_bar') before the SET progress_bar_time = 0.

2. The bind path for statements with parameter values still plans in full, so con.sql describes a statement the planner refuses when it has no values and raises at once when it has them (src/engine/connection.cpp:419, outside the diff).

Severity: should fix. Category: design. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.

Session::Bind without values calls context->BindStatement, which plans in PlanningMode::BIND_ONLY at the new pin. Its sibling with values builds its own planner and keeps the default, full mode:

			    // The planner types each parameter from its value, as a rebind for the values run with does
			    Planner planner(*context);
			    planner.parameter_data = values;
			    planner.CreatePlan(std::move(expanded.statements[0]));

With q = SELECT * FROM t WHERE EXISTS (SELECT * FROM u POSITIONAL JOIN (SELECT t.a) s), the statement of the engine's own test for duckdb/duckdb#28690:

call merge base head
con.sql(q).columns raises at the call ['a', 'b'], raises at fetch
con.sql(q + " AND a = ?", params=[1]).columns raises at the call raises at the call
con.sql(q + " AND a = $x", params={"x": 1}).columns raises at the call raises at the call
30 nested filters under max_expression_depth=50, no values raises at the call describes, raises at fetch
the same with one value raises at the call raises at the call
con.query_appender(...) over a DML with the same subquery raises at creation accepted, raises at flush

So the description's sentence that the errors of the later planning passes move from con.sql to fetchall and that "the client does not change for it" holds only without values, and a parameterized relation keeps paying for the passes the engine change removed from a describe. With Planner planner(*context, PlanningMode::BIND_ONLY); in a clone of this head, every con.sql row above describes and raises at fetch, as the query appender already does, and tests/fast/api with tests/fast/relational_api pass (3268 passed and 35 skipped on this head, the one failure the timing test test_query_progress noted below; 3269 passed on the first commit).

Ask: plan in PlanningMode::BIND_ONLY there too, and add one test that pins where a planning refusal surfaces: con.sql with and without values (columns described, the engine's error at fetch) and query_appender (accepted at creation, the error at flush).

3. The MERGE test now pins a result the engine calls a correctness trade it means to undo (tests/fast/api/test_appender.py:1410, inline).

Severity: should fix. Category: correctness. Evidence: ran. Fix: containment. Origin: engine. Base: introduced.

Ask: keep test_a_key_repeated_within_one_merge_raises with its old expectation under pytest.mark.xfail(strict=True), its reason naming duckdb/duckdb#28701, so the strict marker reports an unexpected pass the day the engine restores the check and dropping the marker is the only edit then.

4. The benchmark sentence of the description does not hold, and the regression I can reproduce, fetchall over LIST, is not named (description, and the engine's src/include/duckdb/common/types.hpp:295).

Severity: should fix. Category: performance. Evidence: ran. Fix: containment. Origin: engine. Base: introduced.

The description says fetchall over DOUBLE, DECIMAL and LIST is 5 to 7 percent slower. Release builds of this head and the merge base made the same way, one thread, best of five in two alternating runs, user space instructions of the calling thread from a perf_event_open counter:

case merge base head wall instructions
fetchall DOUBLE, 1M rows 133.7 ms 135.4 ms +1.2% 1281.5M to 1283.2M, +0.1%
fetchall DECIMAL(18,2), 1M rows 1096.8 ms 1068.6 ms -2.6% 9529.2M to 9108.3M, -4.4%
fetchall LIST of 3 BIGINT, 250k rows 189.3 ms 204.8 ms +8.2% 2007.4M to 2132.1M, +6.2%
fetchall BIGINT, 1M rows 131.2 ms 135.1 ms +3.0% +0.1%
fetchall VARCHAR, 1M rows 199.7 ms 200.4 ms +0.4% +0.1%
fetchnumpy DOUBLE and DECIMAL, df DECIMAL and LIST within 5% within 2%
to_arrow_table DECIMAL and LIST 2.6 ms, 5.4 ms 2.8 ms, 5.6 ms noise at this size +0.1% (rerun), +0.5%
appender, 200k floats 24.5 ms 25.6 ms +4.2% +0.4%
appender, 200k Decimal 270.0 ms 247.0 ms -8.5% -7.7%

The first commit's pin gave the same picture (LIST +6.5% instructions, DECIMAL -4.5%, DOUBLE +0.0%). DOUBLE does not move, DECIMAL gets faster, LIST gets slower. The LIST cost is per value and comes from the engine: the client builds each row through Vector::GetValue (src/pyresult.cpp:184), every LIST value makes its type with make_uniq<ListTypeInfo>(child) (engine src/common/types.cpp:1648), and engine commit 0170852ae31 (in duckdb/duckdb#28622) put alignas(128) on a member of LogicalTypeInfo, so that allocation went from 64 bytes aligned to 8 to 256 bytes aligned to 128 (sizeof and alignof printed from both pins' headers). The same commit gives DECIMAL immortal shared type infos, which is the speedup, and the same alignment is what the macOS seed jobs of the first commit rejected.

Ask: replace the sentence with measured figures for the cases above (or the roundtrip benchmark's own cases, named one by one, before and after), name 0170852ae31 as the source of the LIST cost, and link an engine issue for it. I would not hold the pin for it: the cost is already on engine main, so the next engine release brings it regardless.

5. A cursor gets neither the interactive default nor the ipywidgets disable reason (src/pyconnection.cpp:1375, outside the diff).

Severity: minor. Category: correctness. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: pre-existing.

Not blocking. In a kernel without ipywidgets, con.execute("SET enable_progress_bar = true") is refused while con.cursor().execute(...) is accepted and reads True, on the merge base and on the head; in an interactive session a cursor reads False while its connection reads True. DuckDBPyConnection::Cursor() opens its Connection without SetDefaultConfigArguments, so the new kernel test's claim holds for connect() only. Ask, for a later change: apply SetDefaultConfigArguments to a cursor's context too.

6. A comment after the last semicolon keeps the semicolon in the text the relation embeds (src/pyconnection.cpp:1037, outside the diff).

Severity: minor. Category: correctness. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: pre-existing.

Not blocking. con.sql("SELECT 2 AS a; /* x */").filter("true") raises Parser Error: syntax error at or near ";" on both builds, as does SELECT 1; SELECT 2 AS a; followed by a line comment. PrepareScript trims only trailing space and semicolons, so the text keeps its ; inside SELECT * FROM (...). Ask, for a later change: trim trailing comments before the semicolons.

Since the first commit

  • cmake/duckdb_loader.cmake no longer adds -mmacosx-version-min=10.7 to the binding objects. In run 43448104579 the binding objects of src/CMakeFiles/python_src.dir took 10.7 while the engine objects took 11.0 on the arm64 seed and 10.15 on the x86_64 seed (the MACOSX_DEPLOYMENT_TARGET = "10.15" of [tool.cibuildwheel.macos], raised to 11.0 for arm64), and every binding object ninja reached (four on arm64, five on x86_64) failed on types.hpp:333, "aligned deallocation function of type 'void (void *, std::align_val_t) noexcept' is only available on macOS 10.13 or newer". Without the flag they take the build's target like the engine objects, both above 10.13, and the wheels that delocate repairs keep the tags the engine objects already set: run 43457043339 builds every object at 11.0 on arm64 and 10.15 on x86_64 and ships macosx_11_0_arm64 and macosx_10_15_x86_64. Nothing else in the repository sets a macOS minimum (the CMake files, pyproject.toml and the workflows), and a local macOS build without the variable now gives every object the compiler's default.
  • The engine moves 23 more commits (six merges) to engine main's tip: duckdb/duckdb#28741 makes the decimal type count static constexpr, the MSVC C3493 of the Windows seeds; #28787 makes profiling_renderer_settings a generic MAP setting, which removes ClientConfig::profiling_renderer_settings (unused by the client) and extends settings.hpp (included by src/pyconnection.cpp since this change) with map support, both additive for the client; #28681 normalizes VARIANT in function metadata, #28695 changes the ALP size estimate, #28720 adds includes, #28801 removes a workflow. Of the changed headers, client_config.hpp and settings.hpp are included under src/.

Behaviour that changes with the pin (for the maintainer to know; not blocking)

  • Statement text now carries the whitespace and comments in front of it (duckdb/duckdb#28597): Statement.query and extract_statements (a comment in front of the first statement stays on it, a comment between two statements now goes with the second, each statement keeps its own ;), DuckDBPyRelation.sql_query() (con.sql("SELECT 1; /* c */ WITH ...").sql_query() now starts with /* c */), and error context lines (the second statement of "SELECT 1;\n SELECT nonexistent_col FROM t" reports LINE 2:; the caret under it is misplaced on both builds, which the engine notes as a follow up). Relations built from such text behave the same: filter, aggregate, query, join, union, order and limit chains, captured Python variables, DESCRIBE, parameters and writes give the merge base's rows on every case I tried.
  • Errors of the planning passes after the bind (the plan depth limit, decorrelation refusals) surface at fetch for a relation and at flush for a query appender (duckdb/duckdb#28690); finding 2 is about making that hold with values too.
  • A MERGE that matches one target row twice applies the rows instead of raising (duckdb/duckdb#28701); finding 3.
  • n IN (1, 3.5) over an INTEGER reaches an Arrow source as n IN (1), and an equality no DECIMAL(10,2) value can meet folds and skips the source (duckdb/duckdb#28611). Nineteen IN, NOT IN, NULL and out of range shapes over a dataset and a LazyFrame give the native table's rows on both builds.
  • Settings: duckdb.connect(config=...) now accepts checkpoint_on_shutdown, trim_free_blocks and profiling_coverage (the merge base refused them), current_setting('debug_order_verification') reads NONE instead of none, duckdb_settings() has 173 rows instead of 170, allow_extension_repositories names its levels differently (the default undecided became hosts), and SET GLOBAL enable_progress_bar raises InvalidInputException ("can only be set per session") instead of CatalogException.
  • duckdb.tokenize("SELECT t.a") returns the dot as an operator instead of a numeric constant (duckdb/duckdb#27453), and a UDF declared from types lists its parameters by type in a "no function matches" candidate list (duckdb/duckdb#28533).

Smaller things

  • The description says the engine's objects "took its own 11.0 target"; that is the arm64 seed, the x86_64 seed's engine objects took 10.15.
  • n IN (1, 2.5) now translates and sits in REFUSALS under "other operators and operands"; it belongs in SHAPES beside widening-cast-in-dropped-with-its-out-of-range-member. The case it replaced was the only IN whose column side is a cast; ColumnOf refuses that for comparisons too (cast-to-a-wider-integer), so the guard is still covered, but an IN over a cast that survives (a TIMESTAMP_MS column against TIMESTAMP constants) would keep the IN branch pinned.
  • TestQueryProgress::test_query_progress (unchanged here) fails on this 192 core machine with "query finished too quick": 4 of 20 single runs on the first commit, 1 of 20 on the merge base, and once in the merge base's full suite. It runs the query on default threads; on a CI runner it is unlikely to trip.

Outside findings not taken

  • The vectorized UDF body calls state.GetClientContext() four times and the native body once per row: one pointer test next to a Python call, no measurable cost; style.
  • The money = 1.505 case asserts the source is never read instead of comparing against now: the result [] is pinned, and the folding is the behaviour the description claims; an engine that stops folding should trip it.
  • A native UDF without a client context calls the Python function once before failing: the merge base does the same, and the one path I found, an index over the UDF, is refused at commit on both builds.
  • The new tests do not check that a terminal bar is drawn: drawing is engine code, and the widget test covers the client's display function.
  • The description's 5 to 7 percent figure, which the code-review skill repeated: taken as finding 4 with my own figures, which differ.

Prerequisites in the engine

  • Finding 3: engine src/execution/operator/persistent/physical_merge_into.cpp lost the multi-match check in duckdb/duckdb#28701, whose message says it "trades correctness for performance, and a proper fix will be made for 2.1"; engine main has not restored it. Ship with the strict xfail; it reports an unexpected pass when the engine restores the check.
  • Finding 4: alignas(128) on LogicalTypeInfo (engine src/include/duckdb/common/types.hpp:295, commit 0170852ae31) makes every per value LIST type an over aligned 256 byte allocation. Engine main still has it. Ship, and report it upstream with the LIST figure; a client side containment (converting LIST columns without a Value per row) is a change of its own.

The engine range, by what the bindings had to follow

  • Removed or changed members the bindings use: duckdb/duckdb#28571 (the progress bar fields of ClientConfig) and duckdb/duckdb#28594 (the ExpressionState accessors) are the two compile follows. The other removals (the DBConfigOptions fields of #28700, ClientConfig::profiling_renderer_settings of #28787, DatabaseInstance::Services, AddDefaultCastToType, LogicalExecute, RowIdHandling, the filter hint functions) are not referenced by the bindings, which compile. Of the changed public headers, 24 are included directly under src/ (two of them, client_config.hpp and settings.hpp, change again in the second range); their changes are additive or in members the bindings do not use, apart from the two follows.
  • Statement text (#28597): followed in four expectations and one new case; the relation layer copes (see the behaviour list).
  • Binder and planner (#28690, #28761, #28704, #28624, #28648, #28533): #28690 needed a follow in Session::Bind with values (finding 2).
  • Optimizer and pushdown (#28611, #28717, #28506, #28641, #28573, #28716, #28715, #28649, #28579, #28768, #28718, #27620): #28611 followed in three predicate cases; the rest change plans, not results the client asserts.
  • DML (#28701, #28739): #28701 followed in one test (finding 3).
  • Settings, extensions and security (#28571, #28700, #28714, #28719, #28663, #28578, #28637, #28635, #28709, #28684, #26085, #28787): Python visible through settings only (behaviour list); connect() config handling and the instance cache behave as before (a second connect with another custom_user_agent is still refused).
  • Types and casts (0170852ae31 in #28622, #28741, #28681, #28673, #28692, #28672, #28667, #28061, #28563): DECIMAL faster and LIST slower per value (finding 4); casts through GetCastFunctionInput::Default() cost the same.
  • Execution, memory and storage (#28781, #28752, #28687, #28646, #28748, #28707, #28674, #27136, #28508, #28682, #28315, #28465, #27979, #28475, #28650, #27334, #28696, #26814, #25570, #28751, #28735, #28669, #28695): no client follow; the streaming consumer change does not change when the client holds the GIL.
  • Arrow scan and export, the appender and the table scan operator: untouched in the range. C API: duckdb.h unchanged; duckdb_v2.h changed and is unused by the bindings.
  • Serialization of older plans and CI only changes: nothing to follow.

What I verified

  • Head f3ad6d9, merge base 6f03883, ten files; engine 23221edfd577 to afdc4212a58c, 718 commits, the tip of engine main. The first commit, cd9c19f at engine 5c8008b4cf86, was reviewed the same way first.
  • Builds: head and merge base, Release, the repository's two step uv install, Python 3.13; a clone of the head for weaker progress bar follows (no Settings::Set, set before the Jupyter check, both on the first commit; SetScope::GLOBAL and return after the display function on both commits) and for PlanningMode::BIND_ONLY in Session::Bind with values. The stable ABI syntax check (Py_LIMITED_API=0x030C0000) passes for src/pyconnection.cpp and src/python_udf.cpp on head and merge base.
  • The six changed test files on the head: 1138 passed, 1 failed (test_query_progress, the timing test above). On the merge base build with its own files: 1134 passed, 1 failed (the same test). The head's files on the merge base build: 11 failed, each a forced expectation: the MERGE test (the base raises "MERGE INTO command cannot affect the same target row more than once"), four _prepare_script cases and test_pystatement (the base texts lack the leading whitespace and keep a trailing space after select $1;), n IN (1, 3.5) on a dataset and a LazyFrame (nothing pushed), the DECIMAL fold (the base reads the source), and the two predicate translation cases (the base keeps the casts). The two new progress bar tests pass on the merge base build, as a behaviour preserving follow should.
  • Weaker follows: no Settings::Set fails test_an_interactive_session_enables_the_progress_bar; set before the Jupyter check fails test_a_jupyter_kernel_without_ipywidgets_refuses_the_progress_bar; SetScope::GLOBAL and the early return pass every test, on both commits (finding 1).
  • Full fast suite, eight workers, 600 second timeout per test: head 16800 passed, 83 skipped, 223 xfailed in 562.72s; first commit the same figures in 698.75s; merge base 16795 passed, 1 failed (test_query_progress), 83 skipped, 223 xfailed in 705.91s. The difference of four is the two new progress bar tests, the new _prepare_script case and the new predicate case. The generated function namespace, aggregate and keyword checks (tests/fast/expression/test_expression_generated.py) pass on the head.
  • Probes: leading statement text through relation chains, planning error timing with and without values, UDFs in SELECT, WHERE, CHECK, generated columns, defaults, UPDATE, prepared statements, macros and an index (identical on both builds), IN shapes against Arrow sources, settings and connect() config, a cursor in a kernel without ipywidgets, comments after the last semicolon.
  • Benchmarks as in finding 4.
  • CI: On cd9c19f (run 43448104579): Code-quality checks, the full API build with assertions, the sdist and both manylinux seed wheels passed; the four non Linux seed wheels failed and were caused by the pin, macOS arm64 and x86_64 on types.hpp:333 in the binding objects compiled with -mmacosx-version-min=10.7, Windows amd64 and ARM64 on types.cpp(1580): error C3493: 'TYPE_COUNT' cannot be implicitly captured, both from engine 0170852ae31, with the abi3 test jobs of those platforms failing behind them. On f3ad6d9 (run 43457043339): Code-quality checks and Submodule sanity pass (the submodule-bump label is on), the sdist passes, and all six seed wheels pass: macOS arm64 and x86_64 with every object, the binding objects included, at 11.0 and 10.15 (macosx_11_0_arm64, macosx_10_15_x86_64), Windows amd64 and ARM64 with no MSVC error. The full API build with assertions passes too. The two cp311 sanity jobs and the sixteen abi3 test jobs across interpreters were still running at 16:26 UTC; nothing on this head is red.
  • Outside reviews: Codex at xhigh effort (two findings, both confirmed by the variant builds and merged into finding 1) and the code-review skill at xhigh effort over cd9c19f's range in the head tree (fourteen findings: findings 2, 3, 5 and 6 come from it after my own probes, the rest are triaged above or folded into findings 1, 2 and 4).

What held

  • The progress bar in every environment the client distinguishes: a script (off), an interactive session (on; a user's SET turns it off, RESET gives off, SET progress_bar_time then RESET progress_bar_time gives off, as on the merge base), a kernel without ipywidgets (off; a SET refused with the ipywidgets message), a kernel with ipywidgets (the widget is shown).
  • GetClientContext() at all six UDF call sites; native and Arrow UDFs in every statement position above give the merge base's results and the same refusal for an index.
  • No other client code reaches a removed engine member or a moved setting; the instance cache and custom_user_agent handling are unchanged.
  • The relation layer with leading comments and whitespace (block and line comments, WITH, captured variables, parameters, writes, DESCRIBE).
  • Arrow pushdown results for the IN and DECIMAL shapes the engine now rewrites.
  • Arrow export and the appender: the engine code they run is untouched in the range, and their instruction counts match within half a percent except the faster Decimal appender.


con = duckdb.connect()
print("enabled", con.execute("SELECT current_setting('enable_progress_bar')").fetchone()[0])
con.execute("SET enable_progress_bar = false")

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The progress bar tests pass with two wrong follows: the setting written at global scope, and a return right after the Jupyter display function is set.

Severity: should fix. Category: missing coverage. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.

I built weaker versions of SetDefaultConfigArguments in a clone of the head and ran tests/fast/api/test_query_progress.py and a -c child that reads a cursor and RESET (the global and early return rows on both commits, the other two on the first commit; this function is the same in both):

variant new tests widget test cursor in an interactive session after RESET enable_progress_bar
this head pass pass False False
no Settings::Set interactive test fails pass False False
Settings::Set before the Jupyter check kernel test fails pass False False
SetScope::GLOBAL pass pass True True
return; after config.display_create_func = JupyterProgressBarDisplay::Create; pass pass False False

The global write passes because the child reads its own connection and a session SET overrides the global value; the merge base gives False for a cursor and after RESET, so that variant changes behaviour a user sees. The early return passes because the widget child runs SET progress_bar_time = 0, and the engine's ProgressBarTimeSetting::OnSet switches the bar on by itself (Settings::Set<EnableProgressBarSetting>(context, SetScope::SESSION, Value::BOOLEAN(true)) in src/main/settings/custom_settings.cpp), so no test sees a notebook start with the bar off.

Ask: in _READ_AND_SET, also print con.cursor().execute("SELECT current_setting('enable_progress_bar')").fetchone()[0] and the value after RESET enable_progress_bar (both False); in _JUPYTER_KERNEL, print current_setting('enable_progress_bar') before the SET progress_bar_time = 0 and assert True.

Comment thread tests/fast/api/test_appender.py Outdated
assert table_rows(con) in ([(1, "first")], [(1, "second")])

def test_a_key_repeated_within_one_merge_raises(self, con):
def test_a_key_repeated_within_one_merge_applies_one_of_its_rows(self, con):

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MERGE test now pins a result the engine calls a correctness trade it means to undo.

Severity: should fix. Category: correctness. Evidence: ran. Fix: containment. Origin: engine. Base: introduced.

On the merge base build this case raises Invalid Input Error: MERGE INTO command cannot affect the same target row more than once. A target row matched more than one source row; ensure the source rows are deduplicated or the ON condition is unique.; on the head it applies a row. The engine change behind it, duckdb/duckdb#28701, says of itself: "A target matched by several source rows is again updated, counted and returned once per match. ... this trades correctness for performance, and a proper fix will be made for 2.1", and it deletes test/sql/merge/merge_multi_match_error.test rather than changing its expectation. So the new name and assertion record a known wrong result as the client's behaviour, and "applies one of its rows" is not what happens: on the head, the same MERGE over VALUES (1, 'first'), (1, 'second') against a one row table reports [(2,)] changed rows and RETURNING merge_action, * gives [('UPDATE', 1, 'first'), ('UPDATE', 1, 'second')], while the final state is [(1, 'first')], which is all the test looks at.

Ask: keep test_a_key_repeated_within_one_merge_raises with its pytest.raises(duckdb.InvalidInputException, match="more than once") expectation under @pytest.mark.xfail(strict=True, reason=...), the reason naming duckdb/duckdb#28701 and the planned restore, so the strict marker reports an unexpected pass when the engine restores the check and dropping the marker is the only edit then.

@el-marvin el-marvin left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is ready to merge.

I reviewed c96ad1e, one commit on top of round 1's head f3ad6d9: Session::Bind with parameter values now plans in PlanningMode::BIND_ONLY, a new test class pins where a planning refusal surfaces, the MERGE test is back to its refusal under a strict xfail, the progress bar children print more readings, and one predicate case moved lists. The engine pointer is unchanged at afdc4212a58c. I rebuilt the head, ran the new and changed tests on the head and the head's test files on the merge base build, built the two wrong progress bar follows and the old full planning line in a clone of this head and ran the tests against each, re-ran round 1's probes, the fetch benchmark against the merge base build, and the full fast suite. Every round 1 request is met, and the three requests on code and tests are each pinned by a test that fails without them. The premise and the layer stand as in round 1: the client does not compile at engine main without the two follows, and the bind follow now matches the engine's own BindStatement. The maintainer still has to do three things outside the code: wait for the abi3 (stable ABI) test jobs (still running at 17:20 UTC with nothing red; the macOS x86_64 abi3 jobs have not completed on any head of this pull request yet), file the engine issue for the LIST cost the description names and link it here, and know about the transaction behaviour listed below. Nothing here blocks merging. The items below are nits the author may take or leave.

Since the last review

  • 1, the progress bar tests passed with two wrong follows: addressed. Built in a clone of this head and run against tests/fast/api/test_query_progress.py: with SetScope::GLOBAL, test_an_interactive_session_enables_the_progress_bar fails ('cursor True' != 'cursor False', and after reset True); with return; after config.display_create_func = JupyterProgressBarDisplay::Create;, test_a_jupyter_kernel_shows_the_progress_bar_as_a_widget fails (the child prints enabled False before the SET progress_bar_time = 0). The head passes the three child process tests.
  • 2, a describe with parameter values planned in full: addressed. src/engine/connection.cpp:420 now reads Planner planner(*context, PlanningMode::BIND_ONLY);, the mode ClientContext::BindStatement uses. Round 1's probe on this head, with q = SELECT * FROM t WHERE EXISTS (SELECT * FROM u POSITIONAL JOIN (SELECT t.a) s): con.sql(q + " AND a = ?", params=[1]), the same with $x and a dict, and 30 nested filters under max_expression_depth = 50 with one value all describe ['a', 'b'] at con.sql and raise at fetchall, as the same statements without values do; execute with values still raises at the call, and a writing statement with values still runs and raises at con.sql. TestErrorsOfThePlanningAfterTheBind tells the change apart: with the old Planner planner(*context); built back into the clone, the qmark and named decorrelation cases and the qmark depth case fail at con.sql while the three cases without values pass; on the merge base build all six fail at the call (con.sql or query_appender raises), because the engine there plans in full on both paths. The engine sets the names and types before the passes that bind only skips, and in builds with assertions checks that those passes leave the statement properties and the parameter types as they were (BindSignatureState in the engine's src/planner/planner.cpp); the stable ABI syntax check passes for src/engine/connection.cpp.
  • 3, the MERGE test pinned the reverted check's result: addressed. test_a_key_repeated_within_one_merge_raises has its merge base body under @pytest.mark.xfail(strict=True, ...): XFAIL on this head, and on the merge base build, which still refuses the repeated match, it fails as [XPASS(strict)], which is what the day the engine restores the check will look like. The MERGE probe of round 1 still gives two changed rows and two RETURNING rows on this head. The reason states the engine's behaviour in words rather than a pull request number, which reads better in the file than what I asked for.
  • 4, the description's fetch figures: addressed. The description now gives LIST 7 to 9 percent slower and names engine commit 0170852ae31 and the 64 to 256 byte ListTypeInfo allocation. My rerun on this head against the merge base build, made as in round 1: fetchall over 250,000 LIST of three BIGINT 186.4 to 200.3 ms (+7.5% wall, +6.2% user space instructions), DOUBLE at a million rows 129.3 to 129.8 ms (+0.4%, instructions +0.1%), DECIMAL(18,2) 1082.8 to 1058.9 ms (-2.2%, instructions -4.4%), BIGINT and VARCHAR within 2% wall and +0.1% instructions; fetchnumpy, df and to_arrow_table within 4% wall and 2% instructions, the Decimal appender 7.8% faster. These match round 1 within a percent and agree with the description: LIST in its 7 to 9 percent, DOUBLE and DECIMAL within a wall time spread, DECIMAL if anything a little faster by instruction count. The engine issue is not linked yet; that is the maintainer's to file.
  • 5, a cursor gets no SetDefaultConfigArguments: still open, as agreed for a later change. Not blocking.
  • 6, a comment after the last semicolon breaks a chained relation: still open, as agreed for a later change. Not blocking.
  • Smaller things from round 1. The macOS sentence now gives both seeds' targets, and n IN (1, 2.5) moved to the shapes under decimal-cast-in-dropped-with-its-fractional-member with the same expectation. The case for an IN over a cast that survives was declined because "the engine rewrites it into an OR of two cast equalities"; at this pin it does not (below).

Smaller things

  • The interactive test's cursor False line pins the behaviour of round 1 finding 5, which I asked for; a later change that gives a cursor the interactive default has to flip it. The after reset False line alone already fails the global scope follow, so the cursor line is optional. The test comment says what it pins, which is enough if you keep it.
  • An IN whose column side is a cast that survives the optimizer still has no case. On this head, _duckdb._translate_predicate with the file's COLUMNS offers tms IN (TIMESTAMP '2024-01-01 00:00:00', TIMESTAMP '2024-01-02 00:00:00') (and the same with three members) as (CAST(tms AS TIMESTAMP) IN ('2024-01-01 00:00:00'::TIMESTAMP, '2024-01-02 00:00:00'::TIMESTAMP)), refused, with no OR. One REFUSALS entry with that text would keep the ColumnOf guard of the walker's COMPARE_IN branch (src/arrow/predicate_translation.cpp:235) pinned, the role in-casting-the-column had before the engine change took it.
  • The title on GitHub, "follow its settings and expression context changes", does not name the planning follow the description now describes.

Behaviour that changes with the pin (for the maintainer to know; not blocking)

Round 1's list stands. One addition that comes with the bind follow: a statement the later planning refuses no longer raises at con.sql, so inside an explicit transaction the refusal now arrives at fetchall as a run error and aborts the transaction, where the merge base raised at con.sql and left it usable. With BEGIN and then con.sql(q) or con.sql(q + " AND a = ?", params=[1]), with q as above: the merge base raises at the call and a following SELECT 1 gives [(1,)]; this head returns a relation, fetchall raises BinderException, and SELECT 1 raises TransactionContext Error: Current transaction is aborted (please ROLLBACK), as con.execute(q) does on both builds. Without values this came with the engine change and was already on round 1's head; with values it comes with this commit, which is the consistent choice.

Outside findings not taken

Codex at xhigh effort over this commit reported no findings. Its one remark, that the values overload leaves StatementSignature::parameters empty, holds and predates this pull request: no caller reads parameters from a bind with values (_reading_source and _bind_with_stand_ins in duckdb/_relation.py read names, types, captured tables, return type, the deferred flag and the write flag).

What I verified

  • Head c96ad1e, merge base 6f03883; the round's delta is f3ad6d9..c96ad1e, five files, 76 insertions and 10 deletions; the engine at afdc4212a58c in both.
  • Builds: the head (Release, the repository's two step uv install, Python 3.13), the merge base build from round 1, and a clone of this head for three variants: SetScope::GLOBAL, a return after the Jupyter display function, and Planner planner(*context); on the values path.
  • The seven changed test files on the head: 1249 passed, 1 xfailed (the MERGE test), 1 failed (TestQueryProgress::test_query_progress, the timing test that flakes on this 192 core machine on head and base, noted in round 1). The head's versions on the merge base build: 18 failed, 1233 passed: the eleven forced expectations of round 1 (the MERGE test now as [XPASS(strict)], the predicate case under its new id), the six cases of the new class, and the timing test.
  • Variants, tests/fast/api/test_query_progress.py and the new class, as in the ledger above.
  • Full fast suite on the head, eight workers, 600 second timeout per test: 16804 passed, 1 failed (TestQueryProgress::test_query_progress, the timing test), 83 skipped, 224 xfailed in 529.10s. Round 1's head gave 16800 passed, 83 skipped, 223 xfailed: six new cases, and the MERGE test moved from passed to xfailed. The description's 16805 passed is the same count with the timing test passing.
  • Probes: bind with and without values (con.sql, fetchall, execute, writes, query_appender, a decorrelated subquery with values, parameter typing from values, an open transaction), on the head, the merge base build and the full planning variant; IN over a surviving cast; the stable ABI syntax check (Py_LIMITED_API=0x030C0000) for src/engine/connection.cpp.
  • Benchmark as in the ledger: round 1's harness, Release builds of the head and the merge base made the same way, one thread, best of five in two alternating rounds, wall time and user space instructions of the calling thread.
  • The description, sentence by sentence: the nightly failure and the two compile follows, the session scope write, the macOS and Windows sentences, the four engine changes and what follows each in the tests, the bind follow and the test class, the suite figure and the benchmark sentence all hold at this head (the baseline at engine 71f47d410f is benchmarks/baseline.json).
  • CI on c96ad1e: run 43466503542, read at 17:20 UTC. Passed: Code-quality checks, Submodule sanity (the submodule-bump label is on), the sdist, the full API build with assertions, and the seed wheels for macOS arm64, manylinux aarch64 and x86_64, and Windows amd64 and ARM64. Still running: the macOS x86_64 seed and the two cp311 manylinux sanity jobs; the abi3 test jobs across interpreters start after the seeds and had not started. Nothing is red. On f3ad6d9 (run 43457043339) every job that finished passed; the two macOS x86_64 abi3 test jobs were cancelled by this push after 31 minutes, so those two have not yet completed on any head of this pull request.
  • Outside reviews: Codex at xhigh effort over f3ad6d9..c96ad1e (no findings). I did not run the code-review skill again: it ran at xhigh over the full range in round 1, and this round's C++ delta is the one mode argument above, read here with its callers.

What held

  • The progress bar follow against every wrong version tried in both rounds: no Settings::Set, set before the Jupyter check, global scope, an early return; each fails one of the progress bar tests.
  • A describe with values gives the names, types, return type and write flag of the full planning it replaced: the decorrelated subquery with values describes [VARCHAR, DOUBLE] and fetches [('y', 2.5)] on all three builds, SELECT ? AS x with a string and with None and $v * 2 with a float type as before, and a writing statement with values still runs at con.sql.
  • execute with values plans in full and raises at the call as before (probed), and a prepared statement still goes through the engine's own Prepare (Session::Prepare, read); the query appender binds through the overload without values and is accepted at creation and refused at flush.

@el-marvin el-marvin changed the title Move the engine pin to afdc4212a5 and follow its settings and expression context changes Move the engine pin to afdc4212a5 and follow its settings, expression context and bind only describe changes Nov 13, 2026
@evertlammerts
evertlammerts added this pull request to the merge queue Nov 13, 2026
Merged via the queue into duckdb:main with commit b418cbc Nov 13, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready to merge submodule-bump Intentional DuckDB submodule pin move

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants