Repository navigation
Move the engine pin to cb5673685153 and follow its statement text - #772
evertlammerts merged 1 commit into
Conversation
…ich keeps the comments in front of a statement
el-marvin
left a comment
There was a problem hiding this comment.
Changes requested.
I reviewed 93e7020 against the merge base 007b3bc (round 1, so there is no earlier delta). The change is the external/duckdb pointer, 76e6bf79426 to cb5673685153, four statement text expectations and one new case in tests/fast/api/test_duckdb_connection.py and tests/fast/api/test_seam.py; no bindings source changes. I built the head and the merge base as Release builds the same way, and also used the released 2.0.0 and 1.5.6. The premise holds: Release run 43366106286 on v2.0-cyanoptera built the sdist from 007b3bc against the engine branch tip cb5673685153 and failed exactly the four tests this change edits (4 failed, 18152 passed, 129 skipped, 164 xfailed), each on the text of a second statement. Following the engine's text is the right layer: the slicing belongs to the engine (Parser::ParseQuery and ParseIterator, a15352e5994, the backport of duckdb/duckdb#28597), the bindings pass it through, and nothing else in the 81 engine commits needs a bindings change. The expectations, the new case, the pointer and the fast suite all hold.
What keeps this from merging is the record the pin move leaves, the same kind of record the previous pin move on this branch carries for its HUGEINT cast. Engine commit 01f4768bdeb makes the DOUBLE and FLOAT to DECIMAL cast nearly twice as slow on values with more decimals than the target scale, and con.append of a float DataFrame into a DECIMAL column 59 to 63 percent slower, and the description does not say so (finding 1). That is a paragraph of description; whether to land the pin with the cost on record or wait for the engine stays the maintainer's call. Findings 2 and 3 are nits. The text a user sees changes (finding 3 lists it); it is the engine's, it matches 1.5.6 for the leading text, and the maintainer can object to it there. The submodule-bump label is on and Submodule sanity passes; nothing else is left for the maintainer outside the code but that decision.
Findings
1. The pin makes the DOUBLE and FLOAT to DECIMAL cast up to 1.95 times slower, through engine commit 01f4768bdeb, and con.append of a float DataFrame into a DECIMAL column 59 to 63 percent slower; the description does not record it.
Severity: should fix. Category: performance. Evidence: ran. Fix: containment. Origin: engine. Base: introduced.
Ask: put the figures below in the description, with the commit and what it fixes, and leave to the maintainer whether to land the pin with the cost on record or wait for a faster helper in the engine; no bindings change is wanted.
01f4768bdeb (the backport of duckdb/duckdb#28307) replaces the C runtime's round in DoubleToDecimalCast (external/duckdb/src/common/operator/cast_operators.cpp:2918) with a new helper that branches on the fraction and on the sign (external/duckdb/src/include/duckdb/common/numeric_utils.hpp:39-50):
const T truncated = std::trunc(value);
// the fraction is exact, so the comparison with a half is exact
if (std::fabs(value - truncated) < static_cast<T>(0.5)) {
return truncated;
}
return value < 0 ? truncated - static_cast<T>(1) : truncated + static_cast<T>(1);It fixes a misrounding of the mingw-w64 C runtime; on Linux, macOS and MSVC the values are the same before and after (the commit says so, and round(0.5), round(-2.5), 2.5::DECIMAL(3,0) and round(0.49999999999999994) agree on both builds here). When the value has more decimals than the target scale, the first branch goes either way at random and the processor mispredicts up to about once per value. Release builds of the head and the merge base made the same way, arm64, one thread, ten million values, best of five calls in two alternating runs:
SELECT sum(x::DECIMAL(18,3)) over a NumPy float64 column, merge base against head, with the branch misses of each:
- 4 decimals, both signs: 105.9 to 107.3 ms against 195.1 to 200.7 ms; 0.09 to 0.10 M against 8.78 to 8.83 M branch misses.
- 4 decimals, positive: 103.8 to 103.9 ms against 172.9 to 180.7 ms; 0.08 to 0.09 M against 5.07 M.
- 3 decimals, positive, which fits the scale: 104.2 to 108.3 ms against 105.3 to 111.9 ms; 0.08 to 0.09 M against 0.12 M.
- Whole numbers, positive: 99.2 to 101.8 ms against 105.8 to 106.0 ms; 0.09 M against 0.08 to 0.09 M.
The instruction count is 6 to 7 percent higher in every case (1443 M against 1533 to 1546 M); the time is in the mispredictions. A separate run of the first row gave 96.7 to 99.2 ms against 189.0 to 191.0 ms, 1.93 and 1.95 times. SELECT sum(x) over the same column, the control, is 15.4 to 16.7 ms on both builds with 117.4 M instructions on both. A cast to DECIMAL(38,6) of the same values costs 1.5 to 2.4 percent more time and 4 percent more instructions.
The bindings reach it on their own ingestion paths:
con.append("t", df)(src/pyconnection.cpp:957) of five million float64 values with four decimals into a DECIMAL(18,3) column: 78.5 and 78.6 ms on the merge base, 125.1 and 127.7 ms on the head. Into a DOUBLE column: 41.5 to 43.2 ms on both.- The appender's per-value
CastToColumn(src/include/duckdb_python/staged_rows.hpp:28): 200,000 Python floats into DECIMAL(18,3) take 1.8 to 2.9 percent more time (113.5 and 115.1 ms against 116.8 and 117.2 ms), since converting the Python object dominates; into DOUBLE it is flat. - The NumPy scan's casts of staged object values (
src/numpy/numpy_scan.cpp:451) and its per-value fit check (:884) run the same cast; I did not time them separately.
duckdb-python main already carries the helper: its pin 23221edfd577, set by d3a03d3 on 2026-11-10, contains upstream cd30952063d. Upstream main has the same helper unchanged. I measured arm64 only; on x86-64 the C runtime's round is a library call, so the ratio there may differ (see Questions). Probes: round-1/probes/bench_decimal_cast.py, bench_decimal_branches.py, bench_append_decimal.py.
2. The new case pins a comment between two statements, but no test pins a comment before the first statement, which this pin now keeps and hands to the relation layer. tests/fast/api/test_duckdb_connection.py:143
Severity: minor. Category: missing coverage. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.
Ask: add an exact Statement.query assertion for a leading comment and one relation built on such a text; the evidence is in the inline comment.
3. The description says the client "strips nothing" and names only Statement.query, but the client trims the end of the last statement, and the change also shows in sql_query().
Severity: minor. Category: docs. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.
Ask: say instead that the client keeps the engine's leading text and trims only the end of the last statement, and add the sql_query() example below, so the maintainer objects, if at all, to what a user sees and not only to Statement.query.
_prepare_script drops trailing whitespace and semicolons (src/pyconnection.cpp:1094-1097), which is why the expectation in test_last_statement_is_not_run is " INSERT INTO u VALUES (1)" for an input ending in (1);, and sql() and DuckDBPyRelation.query() strip whitespace at both ends through _strip_semicolons (duckdb/_relation.py:318-322). So whitespace never reaches a relation; a comment does. What a user sees change on this pin, from my probes on the two builds:
Statement.query:extract_statements("select $1; select 21")gives['select $1;', ' select 21'], where the merge base and 2.0.0 give['select $1; ', 'select 21']and 1.5.6 gives['select $1', ' select 21']. A comment before the first statement is kept (finding 2).DuckDBPyRelation.sql_query():con.sql("SELECT 1; /* c */ SELECT 2 AS b").sql_query()is'/* c */ SELECT 2 AS b', and a line comment in front of a single statement stays in itssql_query()the same way; the merge base, 2.0.0 and 1.5.6 give'SELECT 2 AS b'for both.current_query()in a second statement returns' /* c */ SELECT current_query()', as 1.5.6 did; the merge base and 2.0.0 return'SELECT current_query()'.- The
LINE 1:echo of an error in a second statement now starts with the space (LINE 1: SELECT nope FROM range(1)). Its caret sits at column 25 on both engines, while the word it points at starts at column 15 on the merge base and at column 16 on the head: it is off by the length of the text before the statement either way, which the engine commit leaves for a follow-up.
The 1.5.6 sentence holds for the leading space. 1.5.6 also left the semicolon off the first statement, which 2.0.0 and this pin keep; the description does not claim otherwise.
Smaller things
- A comment after the terminator of the last statement breaks every relation built on it, on the merge base and on the head alike:
con.sql("SELECT 1 AS a; /* tail */").filter("a = 1").fetchall()raisesParser Error: syntax error at or near ";", and so does a line comment in the same place;fetchall()on the relation itself works._strip_semicolons(duckdb/_relation.py:318-322) and_prepare_script(src/pyconnection.cpp:1094-1097) drop a;only when it ends the text, so it stays inside the parentheses_wrapadds. This pin does not change it (the last statement ran to the end of the input before as well); it is worth its own issue, not a change here. - The description says the roundtrip benchmark moves no gate over its threshold. No gate casts a DOUBLE to a DECIMAL (
benchmarks/test_types_roundtrip_perf.py:32-33build their DECIMAL columns from integers), so the sentence can hold and still miss finding 1. The figures in finding 1 belong beside it.
Questions
- Whether x86-64, the platform of most wheels, shows the same ratio. There the C runtime's
roundis a library call, andstd::truncmay be one too in a build without SSE4.1, so the code before and after may differ from arm64's; the branch on the fraction is new on both. I measured arm64 only. A run ofround-1/probes/bench_decimal_branches.pyon an x86-64 build of each engine would settle it.
Outside findings not taken
- Codex rated the leading comment test "should fix". I took it as a nit (finding 2): the bindings slice every statement the same way and every relation path I ran composes such a text, so the test protects; it fixes nothing.
- Codex asked for a parameter-looking string inside a leading comment.
_markersskips comment tokens (duckdb/_relation.py:481-485), andcon.sql("/* why? */ SELECT ? AS i", params=[7])gives[(7,)]on both builds through fetch, filter, union, aggregate, query, create_view, df and limit; nothing to ask. - Codex's trailing comment after the terminator: confirmed on both builds, pre-existing; under Smaller things.
- Codex's note on extension signature hashing in
TaskExecutor: no file undersrc/parallelchanged in this range.
Prerequisites in the engine
Finding 1 is the engine's. RoundToNearestAwayFromZero (external/duckdb/src/include/duckdb/common/numeric_utils.hpp:39-50) backs DoubleToDecimalCast and, per the commit, the unary round, round with a precision, interval literal fractions, the microseconds of make_time and make_timestamp, and equi_width_bins. Upstream main has the same helper, unchanged since cd30952063d on 2026-11-03, and duckdb-python main already pins it. My recommendation: ship the pin with the cost on record, since it also turns this branch's Release run green, brings about fifteen planner, binder and optimizer fixes, and only brings the 2.0 line to where main already is; and take the cost to the engine, where the C runtime's round everywhere but MinGW, or a select in place of the branch, would remove it. I did not try either variant.
The 81 engine commits
git log 76e6bf79426..cb5673685153 in the engine mirror: 81 commits, 10 of them merges, 8 on the first-parent line. I read them by what the bindings can see.
- Headers the bindings include. 42 headers under
src/includechanged; six of them are included fromsrc/:common/enum_util.hpp(a newTransformChildKindenum),common/numeric_utils.hpp(addsRoundToNearestAwayFromZero),common/string_util.hpp(addsGetURIAuthority),function/function.hpp(addsFunctionParameter::UnnamedandIsNamed),main/settings.hpp(addsDebugForceFlushAppendFailureSetting) andplanner/binder.hpp(addsSetBindingIndexKey). Each is an addition; no declaration the bindings use changed, and the head builds. No binding source changed, so I did not repeat the limited API compile. - Engine functions the bindings call.
Parser::ParseQueryandClientContext::IterateStatements(throughParseIterator) slice statement text differently, a15352e5994; the bindings read it inStatement.query(src/pystatement.cpp:32),_prepare_script(src/pyconnection.cpp:1089-1097) and the error position ofStatementGroup::Fail(src/engine/connection.cpp:566).Parser::Tokenize, whichduckdb.tokenizeandtext_tokenswrap, now emits the dot oft.aandt.*as an operator instead of a numeric constant, 04e1fd57fa0; the binding consumers look only at comments,;,?,$and the first word, and the suite passes.DoubleToDecimalCastrounds throughRoundToNearestAwayFromZeroinstead ofround, 01f4768bdeb; the appender'sCastToColumn(src/include/duckdb_python/staged_rows.hpp:28) and the NumPy scan's casts (src/numpy/numpy_scan.cpp:451,:884) reach it once per value (finding 1). No other changed engine function is called fromsrc/. - Python-visible texts. Statement text, relation text and
current_query()(finding 3). A Python UDF's candidate signature in a binder error readsplus_one(BIGINT) -> BIGINTinstead ofplus_one(col0 BIGINT, /) -> BIGINT, feee9285c75.duckdb.tokenize("select t.a")returns an operator for the dot. The extension load error gains a hint for an extension installed from a user repository, 01b74d68a5d; the plain miss reads the same. - Types, casts, result schemas.
duckdb_types()andduckdb_keywords()are identical on both builds.TIMESTAMPTZ_NS = TIMESTAMPTZnow binds without a cast, 368eae0edf1.roundand the DOUBLE to DECIMAL cast give the same values on Linux (round(0.5),round(-2.5),2.5::DECIMAL(3,0)andround(0.49999999999999994)agree on both builds); the commit fixes MinGW, which no wheel uses. No result schema changed. - Settings and functions.
duckdb_settings()gainsparquet_use_page_index(on); the newdebug_force_flush_append_failureis a debug setting and is not listed.duckdb_functions()gains three overloads of theparquet_page_indextable function and nothing else. The generated keyword, aggregate and function family checks pass on the head (tests/fast/expression/test_expression_generated.py, 3 passed). Parquet writes no page index unlessWRITE_PAGE_INDEXis set, so files written with default settings are unchanged. - Extension loading. Autoload entries for
st_asgridand the iceberg and glue functions, and version bumps of out-of-tree extensions (iceberg, glue, spatial, vortex, delta, unity_catalog, azure), which the wheel does not build. - The C API.
capi_function_signature.hppand the v2 tokenizer changed; the bindings include no C API header, and no Arrow, ADBC orduckdb.hfile changed. - Engine behaviour behind existing calls. Refusals under
enable_external_access = false(CREATE SECRET, ATTACH ... EXTERNAL RESOURCE), secret scope boundaries and remote path traversal; a failed COMMIT that cannot undo its partial flush now invalidates the database; index binding before a checkpoint, buffer leaks, a dangling segment link on out of memory; about fifteen planner, binder and optimizer fixes, one of them a new self-join to group-by rewrite (32c8a25387a); Parquet validation and the page index; CSV, ICU collation and plan deserialization fixes.
The bindings must follow the statement text and nothing else.
What I verified
- Head 93e7020, one commit; range 007b3bc..93e7020, the same three files as the diff on GitHub.
- The pointer: cb5673685153 is the tip of duckdb/duckdb v2.0-cyanoptera (the engine mirror and the GitHub branch read agree) and lies on its first-parent line; 76e6bf79426 is 8 first-parent commits back; the range holds 81 commits, 10 of them merges.
- The premise: the failed job of run 43366106286 ("Build sdist", workflow_dispatch of Release on v2.0-cyanoptera at 007b3bc,
duckdb-sha: cb5673685153). Its four failures aretest_pystatementat line 137 (statements[1].query.startswith("select 21")) and the threeTestPrepareScriptcases, which got'\n FROM u',' INSERT INTO u VALUES (1)'and' SELECT ?': the new expectations exactly. - Builds: the head, and the merge base in its own clone with its engine at 76e6bf79426, each with the repository's two step uv install as the pull request's own build ran it, Release. The released 2.0.0, and 1.5.6 from PyPI in a scratch environment.
- The changed tests and the generated module checks (
test_pystatement,TestPrepareScript,tests/fast/expression/test_expression_generated.py): head14 passed. The head's two test files on the merge base engine:4 failed, 10 passed, the four changed expectations (assert 'select $1; ' == 'select $1;',('FROM u', 'select'),('INSERT INTO...1)', 'insert'),('SELECT ?', 'select')). The new case comes after the first failing line oftest_pystatement; on the merge base the first statement of that text is'select 1; /* between */ 'and the second starts atselect 2, so it fails there too. The merge base's own files on the merge base:14 passed. - The fast suite, all of
tests/fastwith eight workers and a 600 second timeout per test: head18167 passed, 113 skipped, 164 xfailed in 694.67s, the description's figures. The merge base, the same way:18167 passed, 113 skipped, 164 xfailed in 656.23s, the same figures; the nightly sdist job, which skips more tests on its platform, had4 failed, 18152 passed, 129 skipped, 164 xfailedon the merge base with the new engine. - Catalogs, head against merge base:
duckdb_types()andduckdb_keywords()identical;duckdb_settings()gainsparquet_use_page_index;duckdb_functions()gains threeparquet_page_indexoverloads. The keyword, aggregate and function family modules match their generators on the head. - Probes in
round-1/probes/:statement_text.py(statement text,_prepare_script,sql_query(),current_query(), relations over a leading comment),released_text.py(2.0.0 and 1.5.6),visible.py(tokenizer, UDF signatures, error echo and caret, settings, extension load error, TIMESTAMPTZ_NS, rounding values),catalog.py,relation_comments.py(ten relation calls over six texts, and the trailing comment case), and the three benchmarks of finding 1. Benchmarks ran with no other test or build of mine running. - Outside opinions. Codex in read only mode at xhigh effort, briefed on the statement text consumers, the other engine commits and the tests (
round-1/codex-report.md); every point is triaged above, and the coverage point and the per-value reach of the rounding change are taken with my own evidence. I did not run the code-review skill: the diff it reads is a pointer and eleven test lines, while this change's risk sits in the 81 engine commits behind the pointer, which the skill does not read and Codex was briefed on. - CI on 93e7020: 12 checks passed at 15:38 UTC: Submodule sanity (run 43453538791, after the
submodule-bumplabel was added; the first run, 43448097014, failed only on the missing label, and its reachability check passed), the code-quality checks, the sdist build, the six Seed wheel builds, Sanity cp311 manylinux aarch64, Test pandas 2 py3.12 manylinux x86_64 and the full-API build with assertions. Sanity cp311 manylinux x86_64 and 16 Test abi3 jobs had not finished. Nothing has failed.
What held
- The four expectations and the new case are the engine's text byte for byte, and each fails on the merge base engine, so they pin the change rather than any text.
- "No header the bindings include needs a change": six of the 42 changed headers are included from
src/, and all six changes are additions (TransformChildKind,RoundToNearestAwayFromZero,GetURIAuthority,FunctionParameter::Unnamed,DebugForceFlushAppendFailureSetting,SetBindingIndexKey). - "81 commits, mostly extension bumps and backports of merged fixes": 18 of the 71 commits that are not merges bump, patch or register out-of-tree extensions, and nearly all the rest are backports.
- "1.5.6 carried the same leading space":
[' select 21']for the second statement on 1.5.6 and on the head. - Leading comments in the relation layer: texts that begin with a comment compose through filter, union, join, aggregate, query, create_view, df and limit with the merge base's results, including a WITH over a pandas frame and a second statement after
SELECT 1;. - The tokenizer change (04e1fd57fa0) reaches
duckdb.tokenizeandtext_tokens; their consumers induckdb/_relation.pylook only at comments,;,?,$and the first word, and the suite passes. - The caret of an error in a second statement sits at the same column on both engines; only the echoed line gains the leading space.
- Rounding values:
round(0.5),round(-2.5),2.5::DECIMAL(3,0)andround(0.49999999999999994)are the same on both builds; finding 1 is about time only.
| assert statements[1].type == duckdb.StatementType.SELECT | ||
| assert statements[1].named_parameters == set() | ||
|
|
||
| commented = duckdb.extract_statements("select 1; /* between */ select 2 -- after") |
There was a problem hiding this comment.
The new case pins a comment between two statements, but no test pins a comment before the first statement, which this pin now keeps and hands to the relation layer. tests/fast/api/test_duckdb_connection.py:143
Severity: minor. Category: missing coverage. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.
Ask: add an exact Statement.query assertion for a leading comment and one relation built on such a text, for example duckdb.extract_statements("/* lead */ select 1")[0].query == "/* lead */ select 1" and con.sql("/* lead */ WITH x AS (SELECT 1 AS a) SELECT * FROM x").filter("a = 1").fetchall() == [(1,)].
The first statement now starts at offset 0 (Parser::ParseQuery, stmt_start = 0), so a comment in front of it is kept where the merge base dropped it:
merge base 76e6bf7942: extract_statements("/* lead */ select 1") -> ['select 1']
head cb56736851: extract_statements("/* lead */ select 1") -> ['/* lead */ select 1']
released 2.0.0: ['select 1'] released 1.5.6: ['/* lead */ select 1']
That is the case duckdb/duckdb#28597 is about, and it is new input for the relation layer: on the merge base no relation text ever began with a comment. I ran texts that begin with a comment (a WITH over a table, a WITH over a pandas frame, a bare FROM, a ? parameter behind /* why? */, a DESCRIBE, and a second statement after SELECT 1;) through fetchall, filter, union, join, aggregate, query, create_view, df and limit on both builds. Every result is the same on both; only sql_query() differs, by the comment. The one prefix check in the relation layer that does not skip comments is _assemble:
if fold_name(body.lstrip()[:4]) == "with":
body = _wrap(body, "_q")(duckdb/_relation.py:334). Its callers wrap raw text before they reach it, so no path I tried fails, and _with_clause, _query_statement and _markers skip comments through the tokenizer. The two lines keep it that way. Nothing in the bindings treats the first statement differently from the others (src/pystatement.cpp:32 and src/pyconnection.cpp:1089-1093 slice every statement the same way), which is why this is a nit and not a request.
el-marvin
left a comment
There was a problem hiding this comment.
This is ready to merge.
I reviewed 93e7020 again, round 2. The head has not moved since round 1: the author answered by rewriting the description and by a reply, so the delta is text only, and the binary, the tests and the engine pointer are the ones I reviewed in round 1. On the same Release builds of the head and the merge base 007b3bc I re-ran the round 1 reproducers: the statement text probe prints the same output on both builds, the changed tests and the generated module checks pass on the head (14 passed), and the head's two test files fail on the merge base engine exactly as before (4 failed, 10 passed). I read the new description sentence by sentence against the tree, the engine sources and a benchmark in the author's own shape. The premise and the layer stand as in round 1: the engine slices the statement text and the bindings pass it through.
What the maintainer still decides outside the code: whether to land the pin with the DOUBLE to DECIMAL cost the description now records, or wait for a faster rounding helper in the engine. My recommendation from round 1 stands: land it, since main has carried the same helper since d3a03d3. Seven Test abi3 jobs on 93e7020 were still running at 15:54 UTC; nothing has failed. Nothing here blocks merging. The items below are nits the author may take or leave.
Since the last review
-
1. The DOUBLE to DECIMAL cast cost of engine commit 01f4768bdeb (should fix): addressed. The description now says that the backport of duckdb/duckdb#28307 rounds without the C runtime's
round, thatsum(x::DECIMAL(18,3))over 5 M float64 values takes 84 ms against 54 ms andcon.appendof the same rows into a DECIMAL(18,3) column 108 ms against 76 ms, that main has carried the change since d3a03d3, and it leaves land or wait to the maintainer. I re-measured in the author's shape, 5 M values uniform in [0, 1000) and unrounded, one thread, best of five, two alternating runs per build, on the same head and merge base builds:sum(x::DECIMAL(18,3)): 50.3 and 55.2 ms on the merge base, 87.0 and 90.7 ms on the head; 0.05 M against 2.54 M branch misses, 723 M against 773 M instructions.con.appendinto DECIMAL(18,3): 77.4 and 82.8 ms against 114.2 and 115.5 ms.- Controls, flat:
sum(x)8.3 to 8.6 ms andcon.appendinto DOUBLE 37.7 to 42.0 ms on both.
The author's figures agree with these within the run to run spread. The author's baseline, engine a7e6fc10d2f, is 71 commits before the old pin 76e6bf79426 and does not contain 01f4768bdeb either; my merge base figures show that "before" holds against the old pin too. My round 1 figures were larger (1.9 times for the cast, 1.6 times for the append) because those values had both signs, where the helper's sign branch also mispredicts (8.8 M misses per 10 M values against 5.1 M for positive values); positive values are the milder case, not a contradiction. The main sentence holds: d3a03d3 moved main's pin from 71f47d410f6, which lacks upstream cd30952063d, to 23221edfd577, which contains it, and main's pin has not moved since.
-
2. No test pins a comment in front of the first statement (minor): addressed by decision. The author points at the engine's own test, and the reason holds for the text.
external/duckdb/test/api/test_parse_statement_iterator.cpp:201-219at the new pin feeds/* lead */ SELECT 1; ...toParseIteratorand toParser::ParseQueryand assertsstmts[0]->query == "/* lead */ SELECT 1;"and that both slicers agree (line 217). The client reads that same text:StatementGroupiterates throughClientContext::IterateStatements(src/engine/connection.cpp:468), which wraps aParseIterator, andStatement.queryreturnsstatement->query.substr(loc.offset, loc.length)(src/pystatement.cpp:32). A client case would pin the engine a second time. The relation half of my ask, a relation built on such a text, is client code the engine test does not reach; I do not hold the pull request for it (see Smaller things), since this change does not touch the relation layer and every relation path I ran composes such a text on both builds. -
3. The description said the client "strips nothing" (minor): addressed. It now says the client adds no stripping of its own, so
Statement.query,sql_query()andcurrent_query()show the leading text, and that_prepare_scriptkeeps trimming the end of the last statement (src/pyconnection.cpp:1094-1097). The probe shows a leading comment in all three. A correction of my own: the wording I proposed in round 1, "trims only the end of the last statement", was loose.sql()also strips whitespace at the front, through_strip_semicolons(duckdb/_relation.py:318-319,text = text.strip()), so inside a relation a leading comment shows and a leading space does not:con.sql("SELECT 1; /* c */ SELECT 2 AS b").sql_query()is'/* c */ SELECT 2 AS b', whilecon.execute("SELECT 1; /* c */ SELECT current_query()")returns' /* c */ SELECT current_query()'. The new sentence inherits that looseness from my wording; it is right about what a user notices, the comment, and I do not raise it again.
Findings
None new.
Smaller things
- The description's figures are for positive values with a random fraction. Values of both signs cost more, about 1.9 times for the cast and 1.6 times for the append (round 1). "Positive" in the sentence, and a clause saying #28307 fixes a misrounding of the MinGW C runtime only, which no wheel is built with, would give the maintainer the whole trade.
- A relation built on a text that begins with a comment, for example
con.sql("/* lead */ WITH x AS (SELECT 1 AS a) SELECT * FROM x").filter("a = 1").fetchall() == [(1,)], has no client test. Its natural home is the fix for the trailing comment defect below, which changes the same stripping code (duckdb/_relation.py:318-322,src/pyconnection.cpp:1094-1097). con.sql("SELECT 1 AS a; /* tail */").filter("a = 1")still raisesParser Error: syntax error at or near ";"on both builds, as does a line comment in the same place. It is older than this change, and the author takes it to an issue of its own; agreed.
Questions
- The x86-64 ratio is still unmeasured, by me and by the author. It would change the size of the recorded cost, not the decision the maintainer is asked to make. A run of
round-1/probes/bench_decimal_branches.pyon an x86-64 build of each engine would settle it.
Outside findings not taken
None this round, because no outside review ran. The code is byte for byte the tree Codex reviewed at xhigh effort in round 1, whose findings were all triaged then. The delta is the description and a reply, and I checked each sentence here with a probe or a read of the tree and the engine sources. The code-review skill would read the same pointer and eleven test lines as in round 1, where I judged it adds nothing.
Prerequisites in the engine
Finding 1 is the engine's, as in round 1. RoundToNearestAwayFromZero (external/duckdb/src/include/duckdb/common/numeric_utils.hpp:39-50) backs DoubleToDecimalCast. On GitHub the header's last change is still cd30952063d on duckdb/duckdb main and 01f4768bdeb on v2.0-cyanoptera, so the engine has not moved. Recommendation unchanged: ship the pin with the cost on record, and take the cost to the engine, where the C runtime's round on every platform but MinGW, or a select in place of the branch, would remove it.
What I verified
- Head 93e7020, still one commit, the same as round 1 (the pull request's head on GitHub and the tree's HEAD agree); range 007b3bc..93e7020. The tree and the merge base clone are clean.
- Builds: the round 1 Release builds of the head and the merge base, reused; each first import reported no work to do.
round-1/probes/statement_text.pyon both builds: output identical to round 1.- The changed tests and the generated module checks (
test_pystatement,TestPrepareScript,tests/fast/expression/test_expression_generated.py): head14 passed in 0.85s. The head's two test files on the merge base engine:4 failed, 10 passed in 1.20s, the same four assertions as round 1 ('select $1; ' == 'select $1;',('FROM u', 'select'),('INSERT INTO...1)', 'insert'),('SELECT ?', 'select')); the merge base's files restored afterwards. The merge base's own files:14 passed in 1.12s. round-1/probes/relation_comments.pyon both builds: the same results as round 1; only generated relation and source names differ.- New probes in
round-2/probes/:bench_author_setup.py(the figures under 1) andleading_text.py(whatextract_statements,sql_query()andcurrent_query()underexecute()andsql()show for leading spaces and comments, on both builds). - Description sentences: "
Statement.querywas never promised trimmed" (its docstring,src/pystatement.cpp:9, is "Get the query equivalent to this statement."); the main pin note (the duckdb-python mirror:git ls-treeof d3a03d3, its parent and main; the engine mirror: an ancestry check of cd30952063d against both pins); the figures (above). The other sentences are unchanged from round 1 and verified there; I did not rerun the full fast suite, since nothing in the tree changed (round 1:18167 passed, 113 skipped, 164 xfailedon both builds). - CI on 93e7020 at 15:54 UTC: 22 passed, among them Submodule sanity (run 43453538791,
pull_requeston 93e7020, success, after thesubmodule-bumplabel), the code quality checks, the sdist build, both Sanity jobs, the six Seed wheel builds, Test pandas 2 and the full-API build with assertions. Seven Test abi3 jobs were pending (py3.12 macOS x86_64 and Windows amd64, py3.14 manylinux aarch64, py3.15 macOS x86_64, manylinux aarch64 and x86_64, Windows amd64). One job, the wheel matrix whose name is not expanded, was skipped by its own condition. None failed. - Outside opinions: none run, for the reason given above.
What held
- The four new expectations and the new case are the engine's text byte for byte and fail on the merge base engine.
- The description's figures hold against the old pin itself, not only against the author's older baseline.
- "main has carried that change since d3a03d3": d3a03d3 is the commit that brought the helper into main's pin.
- A leading comment reaches
Statement.query,sql_query()andcurrent_query(), as the description now says, and every relation built on such a text gives the merge base's results.
b3098a6
The 2.0 Release workflow builds the sdist against the tip of the engine's v2.0-cyanoptera branch, and run 43366106286 failed four tests in tests/fast/api against engine cb5673685153: test_pystatement and the three TestPrepareScript cases compare the text of a second statement, which now begins right after the previous semicolon.
The engine's backport of duckdb/duckdb#28597 makes the whitespace and comments in front of a statement part of that statement's text, where the parser used to start at the first token. The client follows the engine's text and adds no stripping of its own, so
Statement.query,sql_query()andcurrent_query()show the leading text;_prepare_scriptkeeps trimming the end of the last statement, where a semicolon would end the embedding.Statement.querywas never promised trimmed, and 1.5.6 carried the same leading space. The four tests expect the new text, and one new case checks that a comment between two statements stays with the second.The pin moves from 76e6bf79426 to cb5673685153, 81 commits, mostly extension bumps and backports of merged fixes; no header the bindings include needs a change. One of them costs time: the backport of duckdb/duckdb#28307 rounds a DOUBLE to DECIMAL without the C runtime's round, which fixes a misrounding seen only on MinGW, and on the box at one thread the sum of 5 M positive float64 values cast to DECIMAL(18,3) takes 84 ms against 54 ms before, con.append of the same rows into a DECIMAL(18,3) column 108 ms against 76 ms; main has carried that change since d3a03d3. The fast suite passes on the box (18167 passed, 113 skipped, 164 xfailed), and the roundtrip benchmark against the old pin moves no gate over its threshold.