Skip to content

Complete Volume file operations through the kernel backend - #1057

Open
aakash-saravanan-db wants to merge 27 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113
Open

aakash-saravanan-db wants to merge 27 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113

Conversation

@aakash-saravanan-db

@aakash-saravanan-db aakash-saravanan-db commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Feature or gap

Kernel-backed Python cursors need to complete Volume file and stream operations with the existing cursor behavior.

What this PR addresses

Route server-marked instructions through native completion, preserve lazy BinaryIO reads, empty uploads and 307/308 replay, and retain local errors and cancellation behavior. Downloads enforce allowed roots, including resolved symlink targets. Storage connections and GET/REMOVE response waits honor the configured socket timeout.

Both requested paths and resolved targets must lie within configured allowed roots. Symlinks between authorized roots remain supported; escaped targets are rejected, stricter than the Thrift handler’s lexical path checks.

Python timeout parity gap: unlike Thrift, _socket_timeout does not bound a PUT waiting for initial storage response headers after uploading. A storage service that accepts the upload but never replies can remain pending until cancellation. Connection establishment and response-body reads remain bounded when configured.

Dependencies

Builds select databricks/databricks-sql-kernel#465 at ab4f133620fce52f9932508f0b245db630a57c48. The connector and compatible native kernel will release together; merge and release require the real compatible native version, an updated dependency floor, and clean installed-package validation. Published native 1.4.0 lacks the required APIs.

Tests and validation

Source-built public cursor checks match the unchanged Thrift handler for regular-file and empty/nonempty BinaryIO direct/307/308 cases, with identical bytes at both hops. GET into Unix 0600 and write-only 0200 destinations succeeds and retains permissions; both handlers reject read-only 0400 destinations and preserve prior bytes. Existing close/cache, filesystem-error, root-order and held-response controls pass. Retained connector, inactivity and performance evidence keeps its recorded build qualification. Compatible published packages, clean installs and scheduled activation remain required.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

# Conflicts:
#	CHANGELOG.md
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Medium · 1 Low

Needs changes. The main blocker is that the code relies on new kernel APIs while the dependency floor still allows older wheels, which would break every kernel query, not only Volume operations. There is also a KeyError crash when the best-effort canceller is missing.

Other findings

  • 🔵 Low — This PR deletes the kernel e2e tests for staging and adds no replacements. All new coverage uses mocks that invent is_volume_operation, complete_volume_operation return strings ("needs_input_stream", "completed", "not_volume") and exception attributes (local_os_error, http_status, local_path). Nothing in CI checks that the real kernel binding actually has these names and values. Please add kernel e2e cases, such as PUT→GET→REMOVE round trips (file and input_stream) and a rejected symlink outside the allowed root, so a mismatch with the kernel contract fails the build.

# Canceller is best-effort; never block execute on it.
pass
executed = stmt.execute()
if executed.is_volume_operation:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — pyproject.toml:73 still allows databricks-sql-kernel ^1.4.0, but the code now calls three kernel APIs that don't exist yet: ExecutedStatement.is_volume_operation (read here on every sync execute), kernel_handle.is_volume_operation (read in KernelResultSet.__init__ for every result, including metadata ResultStreams), and fetch_result() in place of await_result() for async results.

If the resolver installs any wheel in the allowed range that lacks them, this isn't limited to Volume ops. Every execute(), every metadata call and every get_async_execution_result() raises AttributeError, so ordinary SELECT 1 breaks on use_kernel=True. The PR description says the floor "has not been selected". This should block merge: bump the floor in the same change. Until then, reading is_volume_operation with getattr(..., False) would at least keep non-volume queries working.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optional-canceller defect is corrected at ab1efce1: registration uses .get, the Statement remains retained, and cleanup skips absent cancellers. All 276 affected tests pass; final native live checks preserve lazy BinaryIO and one-read empty/nonempty streams.

The new Volume marker/completion API dependency is a valid merge prerequisite. The source build pins compatible 22f96fac; a compatible published native version/floor and installed-package proof remain explicitly open. This PR is not merge-ready, and no old-native fallback or invented release version was added.

The fetch_result subclaim is separate: released 1.4.0 exposes it, but the substitution is removed from this PR. Merged native #451 provides terminal-response reuse with the original await_result boundary. Actual public-cursor owner and reader/refetch tests show identical await/fetch requests, rows and metadata on the selected native; redundant #1056 is superseded without merge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming the Volume-marker dependency is a merge prerequisite, and for dropping the fetch_result() substitution. That part of the finding is resolved. The core issue is still there, though. pyproject.toml:73 still says databricks-sql-kernel ^1.4.0. Line 641 still reads executed.is_volume_operation with no guard. KernelResultSet.__init__ still reads kernel_handle.is_volume_operation the same way. So if a published wheel in that range lacks the marker, ordinary sync executes and metadata calls fail with AttributeError before the code ever checks for a Volume op. A source build pinned to 22f96fac doesn't limit what the resolver installs. Leaving this open until the floor is raised to the first published wheel that ships the marker, or until both reads are guarded so non-volume queries keep working.

with self._sync_cancellers_lock:
self._volume_handles[command_id.guid] = (
stmt,
self._sync_cancellers[id(cursor)],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — Canceller registration just above is explicitly best-effort: if stmt.canceller() raises, the error is swallowed and nothing is stored under id(cursor). This line then indexes self._sync_cancellers[id(cursor)] directly. For a volume statement that means KeyError after stmt.execute() has already succeeded. The outer except turns it into a kernel-wrapped error, and because close_stmt is still True the statement gets closed.

The result: a missing canceller (meant to be harmless) now fails every PUT/GET/REMOVE with a confusing error. Use .get(id(cursor)) and handle None (store the entry with no canceller, and skip it in cancel_command/close_command/close_session). No test covers a volume op when canceller() raises.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High

Mostly sound, with one high concern: the new native attributes are read on every kernel result, but the dependency floor still allows a wheel that lacks them. I couldn't check whether the pinned kernel revision exposes is_volume_operation on metadata and async result streams as well as on ExecutedStatement, because the native kernel source isn't in context.

# Canceller is best-effort; never block execute on it.
pass
executed = stmt.execute()
if executed.is_volume_operation:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — Every synchronous execute_command now reads executed.is_volume_operation, and KernelResultSet.__init__ (result_set.py:88) reads kernel_handle.is_volume_operation for every result, including metadata and async streams. These are hard attribute accesses. The PR description says the published native 1.4.0 lacks these APIs, but pyproject.toml:73 still allows ^1.4.0. If the resolver installs 1.4.0, every use_kernel=True query (plain SELECTs and metadata calls too, not just Volume ops) fails with AttributeError. Before this change those queries worked.

The description says the floor bump will happen at release time, but nothing in the dependency graph enforces that ordering. Either bump the floor in this PR to the first kernel release that ships is_volume_operation / complete_volume_operation, or read the attribute with getattr(..., "is_volume_operation", False) at both sites so older in-range wheels keep running non-Volume statements.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed: the compatible native release, dependency floor/lock update and clean installed ordinary-query/metadata/async/Volume proof remain prerequisites. Published 1.4.0 is incompatible; a source pin does not enforce compatibility for an installed wheel. No fallback or invented version is being used, and this finding is not claimed fixed.

For the source-availability question, the pinned kernel defines the property on both public classes: pyo3/src/statement.rs:325 (ExecutedStatement) and pyo3/src/result_stream.rs:68 (ResultStream, including metadata/async streams), at de28034ec75aa6773a5e4e904d77f85511e1aa78. The README documents both surfaces. Those source/native checks establish the new API, while the published-package gate stays open until the real compatible release is selected and tested.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming. You're right that the pinned kernel source defines is_volume_operation on both ExecutedStatement and ResultStream. The problem is what gets installed: pyproject.toml still allows ^1.4.0, and the published 1.4.0 wheel doesn't have the attribute. You've said the compatible release and the floor bump are still pending and that this isn't claimed fixed, so the finding stays open as a merge blocker. Two ways to close it: land the floor bump to the first compatible release in this PR, or make sure this PR can't be released until that bump has landed.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Low

The change needs work: one high blocker. The kernel is_volume_operation / complete_volume_operation APIs are used without a fallback, but the dependency floor still allows a kernel version that lacks them, which would break every kernel query, not just Volume statements. The rest of the routing, cancellation and handle bookkeeping looks consistent and has unit tests. Kernel e2e coverage for the new happy path is missing.

Other findings

  • 🔵 Low — The two kernel e2e tests for staging statements are deleted and nothing replaces them. The new path depends on the server/kernel marking is_volume_operation and returning complete_volume_operation outcome strings ("needs_input_stream", "completed", "not_volume"). In the unit tests those values come from MagicMock side effects, so a typo or a contract change in the native outcome strings would still pass CI.

Add at least one kernel e2e round trip using tmp_path as staging_allowed_local_path: PUT a file, GET it back, compare bytes, then REMOVE. Ideally also cover input_stream PUT and a GET into a path outside the allowed root.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — KernelResultSet.__init__ now reads kernel_handle.is_volume_operation with no fallback, and execute_command reads executed.is_volume_operation the same way (client.py:643). Every kernel result set goes through this constructor: sync execute, async await_result() streams and all four metadata calls. The PR description says the published native 1.4.0 lacks these APIs, but pyproject.toml:73 still allows ^1.4.0. So with any wheel the resolver can currently pick, every use_kernel=True query (even SELECT 1 or cursor.catalogs()) fails with AttributeError. Before this change, the only kernel failure was the explicit NotSupportedError for PUT/GET/REMOVE.

Falling back to getattr(..., False) would be worse: with the leading-verb guard removed, Volume statements would silently skip the file transfer again. Raise the databricks-sql-kernel floor in pyproject.toml (and poetry.lock) to the first release that ships is_volume_operation / complete_volume_operation in this same PR, so an incompatible kernel fails at install time. The description treats this as a merge gate; the dependency metadata in this diff should enforce it rather than leaving it to process.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: the native dependency floor and lock must move to an actual compatible release before this PR merges. Published 1.4.0 is incompatible. The coordinated native/connector release is required, and this thread remains open until the real version is selected and clean installed-package queries, metadata, async results and Volume workflows pass. A source pin/build does not satisfy that gate; no fallback or invented version is being added.

For the separate e2e observation, actual-native file, BinaryIO and async workflows have been exercised through the public Python driver, including byte-equal PUT/GET/REMOVE, empty input, denied paths, and the direct/307/308 comparisons. Cross-driver lifecycle coverage is also in databricks/databricks-driver-test#2122. The unit mocks alone are not our boundary proof. Another round trip in this repository is optional coverage placement; released-package and named scheduled execution remain pending.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming. Since pyproject.toml / poetry.lock still allow ^1.4.0 at this head, the finding stays open until the floor and the lock move to the first published databricks-sql-kernel release that ships is_volume_operation / complete_volume_operation. The e2e work through the public driver and in databricks-driver-test#2122 is useful as boundary proof. It doesn't solve this one, though: the resolver can still install a kernel that raises AttributeError on every use_kernel=True result set, and only the dependency metadata can stop that. Once the real version is picked and pushed, a clean install of the released package running SELECT 1, a metadata call, an async await_result() and a PUT/GET/REMOVE would close it.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

The change is mostly sound, with one medium compatibility concern: is_volume_operation is read unconditionally, and the kernel dependency floor still allows 1.4.0, which lacks it. Error mapping, how long the cancellers are kept and the stream-only-on-request flow look correct. The PUT response-header timeout gap is documented, so it is follow-up work.

Other findings

  • 🔵 Low — This PR deletes the kernel e2e staging tests and adds no live replacement. The new behaviour depends on server-populated fields and native outcomes: is_volume_operation, the needs_input_stream/completed/not_volume strings, and the local_os_error/http_status/error_code attributes. Unit tests can only check fakes of these, not what the real kernel returns.

A small e2e round trip on the kernel path would catch any drift between the connector and the native contract: PUT a file, GET it back, PUT via input_stream, then REMOVE, all using a tmp_path allowed root.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — KernelResultSet.__init__ now reads kernel_handle.is_volume_operation directly, and execute_command (client.py:643) reads executed.is_volume_operation directly. Neither uses a fallback. These run for every kernel result: plain SELECTs, metadata calls and async results, not only Volume statements.

The PR description says published kernel 1.4.0 lacks these APIs, yet pyproject.toml:73 still allows ^1.4.0. If an install resolves to 1.4.x, every use_kernel=True query will raise AttributeError, not just Volume operations. That is a regression on paths that work today.

The description says the floor bump will come before merge. Until it lands, either:

  • bump the floor in the same change, or
  • read the attribute with getattr(handle, "is_volume_operation", False) in both places, so a non-capable wheel keeps ordinary queries working and only Volume statements are affected.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: the native dependency floor and lock must move to an actual compatible release before this PR merges. Published 1.4.0 is incompatible. The coordinated native/connector release is required, and this thread remains open until the real version is selected and clean installed-package queries, metadata, async results and Volume workflows pass. A source pin/build does not satisfy that gate; no fallback or invented version is being added.

For the separate e2e observation, actual-native file, BinaryIO and async workflows have been exercised through the public Python driver, including byte-equal PUT/GET/REMOVE, empty input, denied paths, and the direct/307/308 comparisons. Cross-driver lifecycle coverage is also in databricks/databricks-driver-test#2122. The unit mocks alone are not our boundary proof. Another round trip in this repository is optional coverage placement; released-package and named scheduled execution remain pending.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, we agree on the gate. The current head still reads kernel_handle.is_volume_operation directly, and pyproject.toml still allows ^1.4.0. So an install that resolves to 1.4.x would still raise AttributeError on every use_kernel=True result. That covers plain queries, metadata and async results, not just Volume statements. Your Volume e2e coverage helps, but it doesn't touch this install-resolution issue. I'm leaving this thread open as a merge blocker. It can close once the floor and lock point to the first kernel release that actually ships is_volume_operation, and the installed-package checks you listed pass.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Low

I found one high-severity problem that has to be fixed before merge, plus one low test-coverage gap; the rest of the Volume handling looks correct. The code now reads is_volume_operation on every kernel result, but the declared dependency floor is still databricks-sql-kernel ^1.4.0, a version that doesn't have that attribute. On that wheel, every use_kernel=True query would fail, not just Volume ones. The cancel/close bookkeeping, stream-read ordering and OS/HTTP error mapping all look consistent.

Other findings

  • 🔵 Low — This PR deletes the two kernel e2e staging tests and adds no live replacement. All new coverage uses fakes for complete_volume_operation, is_volume_operation, local_os_error and VOLUME_STORAGE_HTTP_ERROR. So nothing in the suite checks that the real wheel exposes these names or return values ("needs_input_stream", "completed", "not_volume"). Add at least one real-warehouse e2e test that runs a PUT, GET and REMOVE round trip under staging_allowed_local_path (plus a stream PUT) on use_kernel=True. That test would catch a contract mismatch between the connector and the native kernel.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — KernelResultSet.__init__ now reads kernel_handle.is_volume_operation directly, and execute_command checks executed.is_volume_operation (client.py:643). Both run on every kernel result, not only Volume statements: SELECTs, metadata ResultStreams and async await_result() streams. The PR description says published native 1.4.0 lacks these APIs, yet pyproject.toml:73 still allows ^1.4.0. A user who installs the declared floor gets an AttributeError on every use_kernel=True query, including SELECT 1 and cursor.catalogs(). Before this PR, those queries worked on that wheel.

The description says the floor bump will come before release. Until then, the code is the only thing keeping a 1.4.0 install working. Do one of these:

  • Bump the floor atomically with this change.
  • Read the attribute as getattr(handle, "is_volume_operation", False), so older wheels fall closed to the previous non-staging behaviour. Volume ops would then fail loudly, rather than every query failing.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: a 1.4.0 installation is incompatible with these APIs. This remains a required merge gate, not merely a post-merge release task. The coordinated native release must provide the actual compatible version; the dependency floor/lock and clean installed-package SELECT, metadata and Volume checks must move together before merge. An older-native fallback is outside the chosen release contract.

Actual source-built native checks pass regular-file and empty/nonempty BinaryIO transfers, direct/307/308 replay and a live PUT/GET/REMOVE workflow. Those checks establish the new API behavior, but do not satisfy the installed-package gate or scheduled activation.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming. Treating this as a merge gate works for me, but nothing in the code has changed yet: line 88 still reads kernel_handle.is_volume_operation directly, and nobody has said the pyproject.toml floor was bumped. You've ruled out the getattr fallback, so the only fix left is to raise the floor (and the lock) to the first native release that ships these APIs. That bump should land in this PR or block its merge. I'll keep this thread open until it does, so the gate stays visible in the unresolved queue.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High · 1 Low

The completion/cancellation wiring looks sound. One high concern: the new is_volume_operation attribute is read unconditionally while the dependency floor still allows kernel 1.4.0, which would break every kernel-backed query, not just Volume operations. There's also a smaller gap in e2e coverage of the native outcome contract.

Other findings

  • 🔵 Low — This removes the kernel staging e2e tests and adds no replacement e2e coverage for PUT/GET/REMOVE on use_kernel=True (round-trip, input_stream, allowed-root/symlink rejection). The unit tests mock complete_volume_operation entirely, so the native outcome strings ("completed", "needs_input_stream", "not_volume") and the error attributes (local_os_error, error_code == "VOLUME_STORAGE_HTTP_ERROR") the connector depends on are never checked against the real binding. Consider at least one live PUT→GET→REMOVE round-trip test, gated on the compatible wheel.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — KernelResultSet.__init__ now reads kernel_handle.is_volume_operation without a fallback, and execute_command reads executed.is_volume_operation right after stmt.execute() (client.py:643). The dependency floor in pyproject.toml:73 is still ^1.4.0, and the PR description says published 1.4.0 doesn't have these APIs. So with any wheel the resolver is allowed to pick today, every use_kernel=True result set fails with AttributeError: plain SELECTs, async results and metadata calls (catalogs(), tables(), …). Before this change, only PUT/GET/REMOVE were rejected.

The description treats the floor bump as a release gate, but nothing in the dependency graph or CI enforces it. Either bump the floor in this PR (atomically with the kernel release), or read the attribute with getattr(handle, "is_volume_operation", False) at both sites so older wheels keep working for ordinary queries. Also confirm that metadata ResultStream and await_result() streams expose this attribute in kernel #465. Otherwise this constructor breaks those paths even on the new wheel.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: published 1.4.0 cannot run this connector candidate. The actual compatible native release, Python dependency floor/lock update, and clean installed SELECT, metadata and Volume proof remain required before merge. A source SHA pin does not satisfy that gate, and the PR is not merge-ready. We retain the coordinated-release contract rather than adding an older-native fallback that conceals the missing required API.

On compatible kernel de19f0dd, both ExecutedStatement and ResultStream expose is_volume_operation (pyo3/src/statement.rs and pyo3/src/result_stream.rs); async await_result/fetch_result return that ResultStream. Source-built native controls exercise those paths, separately from the still-open installed-package gate.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking kernel de19f0dd. That answers the metadata ResultStream and await_result() part of the question. Declining the getattr fallback is a fair choice, as long as the floor bump lands in the same change as this code. That hasn't happened yet: pyproject.toml still allows ^1.4.0, and line 88 still reads kernel_handle.is_volume_operation with no fallback. With any wheel the resolver can pick today, every use_kernel=True result set still breaks. I'm leaving this thread open as the merge gate. It can be resolved once the PR bumps the databricks-sql-kernel floor and lock to the compatible native release.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
@aakash-saravanan-db

Copy link
Copy Markdown
Collaborator Author

The removed tests asserted the old staging rejection. Replacement real-driver coverage is in the companion comparisons and retained public-driver checks: the 14 CURRENT Python checks, file/stream lifecycles and native protocol controls use the actual binding, including completion outcomes and storage/local error attributes. The latest rebuilt native additionally passes nine native/unchanged-Thrift direct/307/308 pairs and the six non-root GET permission rows.

Those source-built results do not establish clean compatible-wheel installation or named scheduled execution; both remain required. Enabling the existing Volume mixins in connector-local kernel e2e would be optional additional placement, rather than evidence that the binding was never exercised.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High

The Volume completion flow itself looks consistent: lazy stream reads, cancellation ordering on close, error mapping and handle bookkeeping. One high concern: the new native attributes are read unconditionally, but the kernel dependency floor still allows a wheel that lacks them, which would break every kernel-backed query.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — KernelResultSet.__init__ now reads kernel_handle.is_volume_operation without a guard, and execute_command does the same with executed.is_volume_operation (client.py:643). Every kernel result goes through this constructor: sync execute, async await_result() streams, and all four metadata list_* streams. The PR description says the published native 1.4.0 lacks the required APIs, but pyproject.toml still allows ^1.4.0. If a resolver picks 1.4.0, the attribute lookup raises AttributeError, so every use_kernel=True query and metadata call breaks, not only Volume operations. That is a regression for paths that work today.

The PR text says a floor bump is a merge prerequisite, but nothing in this diff enforces it. Either bump the floor here to the first kernel release that ships is_volume_operation/complete_volume_operation, or read the flag with getattr(handle, "is_volume_operation", False) so older wheels skip Volume handling and keep plain queries working.

The unit tests can't catch this. _FakeKernelHandle now sets is_volume_operation, and the MagicMock handles in test_kernel_client.py return a truthy mock for any attribute.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Published 1.4.0 is incompatible, and this PR must not merge with that allowed floor. The real compatible native release, dependency floor/lock update and clean installed SELECT, metadata and Volume checks remain required together before merge. The source SHA pin and successful source-built checks do not complete this gate. We retain the coordinated-release contract, with no old-native fallback and no invented version. Compatible source exposes the getter on both ExecutedStatement and ResultStream, including async/metadata streams; the installed-package proof remains open.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming. Your reply agrees that this has to be fixed before merge, but the code hasn't changed yet. pyproject.toml still allows ^1.4.0, and is_volume_operation is still read without a guard here and at client.py:643. I'm leaving this thread open as a blocker until the dependency floor is raised to the first published kernel release that ships is_volume_operation/complete_volume_operation, with the lock file updated to match. Your plan to keep the coordinated-release contract and skip the old-native fallback is fine, as long as the floor bump lands in this PR.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High

Mostly sound: native completion, the cancel-before-close ordering and the volume-handle bookkeeping look correct, and the unit tests cover them well. One high-severity issue: the new is_volume_operation reads are unconditional, but the dependency floor still allows kernel 1.4.0, which lacks the attribute. On that version every kernel query would break, not just Volume statements.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — KernelResultSet.__init__ now reads kernel_handle.is_volume_operation with no fallback, and execute_command reads executed.is_volume_operation at client.py:643. Every result set is built through _make_result_set: sync execute, async await_result() streams, and all four metadata calls (client.py:1051–1130). So both attributes have to exist on ExecutedStatement and on every ResultStream type.

The PR description says published native 1.4.0 lacks these APIs. Yet pyproject.toml still allows ^1.4.0. If an install resolves to 1.4.x, every use_kernel=True query and metadata call raises AttributeError, not just Volume statements. That is a regression for paths that work today. The description treats the floor bump as a release gate, but nothing in the dependency graph or the code enforces it.

Fix options:

  • Bump the floor in this PR to the first kernel version that ships the API.
  • Or read the attribute with getattr(handle, "is_volume_operation", False) at both sites, and fail loudly only when a Volume statement is detected. Note the second option would bring back the old silent no-op for staging statements, so the floor bump is the safer choice.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: the actual compatible native release, dependency floor/lock update and clean installed SELECT, metadata and Volume proof are required in this PR before merge. Native 1.4.0 cannot run this candidate, and the source SHA pin does not satisfy that gate. We are retaining coordinated native uptake without a getattr fallback or invented version. The selected kernel30c8d8e7 exposes the getter on both ExecutedStatement and ResultStream; rebuilt public-driver controls pass against it. The release/install gate remains open and this PR is not merge-ready.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming, and for keeping the getattr fallback out so staging statements don't go back to being a silent no-op. Coordinated native uptake is the right approach. Still, the code at line 88 still reads kernel_handle.is_volume_operation directly, and pyproject.toml still allows ^1.4.0. So the concern is still open on the current head. Passing tests against kernel 30c8d8e7 shows the API exists at that commit. It doesn't stop the resolver from installing 1.4.x. I'm keeping this thread open so it stays in the merge queue. It can be resolved once the PR raises the dependency floor (and lock) to the first published kernel release that has is_volume_operation on both ExecutedStatement and every ResultStream type.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Low

Mostly sound, with two low-severity concerns: the new attribute dependency on every kernel handle, and the untested async volume path. As the PR description notes, merging still depends on a published kernel release and a matching bump of the databricks-sql-kernel floor in pyproject. Without that bump, the resolver can pick a wheel that lacks complete_volume_operation/is_volume_operation and break existing kernel connections. I couldn't check the kernel side (path/symlink enforcement, ResultStream attributes) because no context repo is available.

Comment thread CONNECTION_PARAMETERS.md Resolved
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 High

The Volume completion wiring looks right, but there is 1 high concern: the code reads the new kernel attribute is_volume_operation unconditionally while the dependency floor still allows kernel 1.4.0, which lacks it. On that version every kernel-backed query would fail. Separately, a handle registered in _volume_handles could stay retained until session close if an exception is raised between registering it and building the result set. That is minor follow-up work.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — KernelResultSet.__init__ now reads kernel_handle.is_volume_operation with no fallback, and execute_command reads executed.is_volume_operation right after stmt.execute() (client.py:643). The PR description says published kernel 1.4.0 does not have this API, but pyproject.toml still allows ^1.4.0. If a resolver installs 1.4.0, every kernel-backed execute() and metadata call (catalogs(), tables(), etc.) raises AttributeError, not only Volume operations. That is a regression for paths that work today.

The description treats the floor bump as a release gate. Because this is a hard break for all queries, the floor should change in this same PR (once the compatible version exists) or the PR should be blocked until it does. A softer option is to read the attribute with getattr(handle, "is_volume_operation", False) in both places, so older wheels fall back to the previous non-staging behavior.

@aakash-saravanan-db aakash-saravanan-db Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: this is a required merge gate, not just a post-merge release task. The Dependencies section explicitly says merge and release require a real compatible native version, an updated dependency floor and clean installed-package validation; native 1.4.0 is incompatible. This PR remains open and is not merge-ready. The floor/lock will select the actual compatible release in this PR once it exists. Source-built checks and the source SHA pin do not satisfy that requirement, and no fallback or invented version is being substituted. This is the same open coordinated-package prerequisite as the earlier review, now retained at b6042eb.

For the overview's separate retention hypothesis: both statement_id and num_modified_rows are cached native getters (source). The ID has already been converted successfully before registration; reading it again adds no fallible native I/O. Modified-row extraction already catches exceptions, and result-set construction failure calls finish_volume_operation to remove the retained sync entry. No concrete supported path establishing a new retained-handle defect was supplied.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for confirming this is a merge gate. Since the issue is still open, I'm leaving this thread unresolved so it stays visible. As of b6042eb, line 87 still reads kernel_handle.is_volume_operation with no fallback, and pyproject.toml still allows ^1.4.0. That commit only changes docstrings. Any kernel-backed query on 1.4.0 would still fail with AttributeError. The thread can close once the PR raises the dependency floor and updates the lock to the first native release that has is_volume_operation and complete_volume_operation(). Your point about the retained handle is reasonable, and it's separate from this finding.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Low · 1 Nit

The routing and lifecycle look sound: completion via complete_staging_operation, cancellation before native close, and retained volume cancellers. The one blocking concern is that the ^1.4.0 kernel floor now breaks every kernel query, not just Volume operations. The other two findings are test-coverage gaps.

Other findings

  • ⚪ Nit — KernelResultSet.__init__ now reads kernel_handle.is_volume_operation unconditionally. Every result-producing kernel call goes through this constructor: sync execute, async results and metadata (catalogs()/tables()/...). The PR description says published kernel 1.4.0 lacks these APIs, but pyproject.toml still allows ^1.4.0. If a resolver installs 1.4.x, every use_kernel=True query raises AttributeError here, not only Volume statements. Before this PR, those queries worked and staging failed loudly with NotSupportedError.

The description says the floor bump is a release gate, but it isn't in the dependency graph. Please either bump the floor in this PR to the first kernel release that ships is_volume_operation/complete_volume_operation, or gate the attribute (e.g. getattr(kernel_handle, "is_volume_operation", False)) and keep the old fail-loud guard when the API is absent.

  • 🔵 Low — This PR removes the two e2e tests that asserted kernel PUT raises NotSupportedError, and adds no e2e replacement. Nothing in the repo now checks that a real kernel PUT/GET/REMOVE actually transfers bytes, or that the allowed-root and symlink-escape checks reject on the real wheel. The unit tests use a fake handle, so they only cover what the fake complete_volume_operation returns. Consider adding e2e round-trip tests (upload, then GET into an allowed root, then REMOVE, plus a rejected escaped path) so the parity claims in the CHANGELOG and CONNECTION_PARAMETERS.md are checked in CI.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
@aakash-saravanan-db

Copy link
Copy Markdown
Collaborator Author

The compatible native release, dependency floor/lock and clean installed SELECT, metadata and Volume proof remain required in this PR before merge. Published native 1.4.0 is incompatible; the source SHA pin does not satisfy that gate, and no fallback or invented version is used.

The real-driver round-trip checks live in databricks/databricks-driver-test#2122: tests/comparator-tests/python/config/thrift-vs-sea.yaml has the VOLUME_OPERATIONS sync, async and BinaryIO workflows with initial/changed-byte checks and storage-origin removal verification. Source-built public-driver allowlist/symlink and filesystem checks are recorded in this PR validation; the core integration and native binding groups remain distinct. Compatible-wheel and named scheduled execution stay open, so this does not claim the repository already proves released-wheel or scheduled acceptance. Duplicating the shared comparator journeys in another connector test matrix would not close the package gate.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Looks good overall, with one medium concern: the dependency floor still allows a kernel wheel without the volume APIs, which would break every kernel-path query. Follow-up: the removed e2e staging tests have no kernel-path PUT/GET/REMOVE e2e replacement in this PR; the round-trip validation is described only in the PR text.

results_queue=None,
description=description_from_arrow_schema(schema),
is_staging_operation=False,
is_staging_operation=kernel_handle.is_volume_operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — KernelResultSet.__init__ now reads kernel_handle.is_volume_operation on every result: SQL executes, async results, and metadata ResultStreams. execute_command also reads executed.is_volume_operation right after stmt.execute(). The PR description says published kernel 1.4.0 lacks these APIs, and pyproject.toml:73 still allows ^1.4.0. If the resolver installs 1.4.0, every use_kernel=True query and metadata call fails with AttributeError, not just Volume operations.

The description treats the floor bump as a merge gate, but nothing in the dependency graph enforces it. Either raise the databricks-sql-kernel floor in this PR to the first release that ships is_volume_operation/complete_volume_operation, or read the attribute defensively (getattr(handle, "is_volume_operation", False)) so ordinary queries keep working on older in-range wheels.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: this PR must not merge while its dependency graph still admits native 1.4.0. The real compatible native release, floor/lock update and clean installed SELECT, metadata and Volume proof are required in this PR before merge. They remain open; the source pin and source-built checks do not satisfy them. No defensive fallback or invented version is being substituted.

The separate end-to-end concern is covered by databricks/databricks-driver-test#2122: tests/comparator-tests/python/config/thrift-vs-sea.yaml contains the sync, async and BinaryIO Volume workflows with initial/changed-byte and storage-origin removal checks. Named scheduled execution using the compatible released wheel remains required. The PR description distinguishes this retained source-built evidence from that open acceptance gate.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for treating this as a hard merge gate. The finding stays open until the fix is in the code: line 87 still reads kernel_handle.is_volume_operation without a guard, and pyproject.toml still allows ^1.4.0. When the compatible kernel release ships, the floor bump (plus the lock update) in this PR will resolve it. That keeps resolver installs of 1.4.0 from breaking every use_kernel=True query and metadata call. I'm leaving the thread unresolved so it stays visible as a blocker.

This branch was previously deployed

1 inactive deployment
azure-prod — dad82a6e Deployed Nov 16, 2026 by peco-review-bot[bot] via followup #2427
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant