Repository navigation
Complete configured Volume transfers through the native backend #613
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
d59a8f3
018997c
0c45168
2ed8d04
28e4169
15708ae
ab895d5
f910e8e
9038b5f
9e6ef83
fee3d8f
495ebe8
1b4aaaf
216fb20
df53f88
37c3d4a
dacfebb
6b113b7
f36e1f7
1ea845f
3aef74a
30877d5
3c9c3e6
e45bf99
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 0b5d0edfed32c0745d5abc2468fc3070e2eb2a17 | ||
| ab4f133620fce52f9932508f0b245db630a57c48 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -46,12 +46,14 @@ import { LogLevel } from '../contracts/IDBSQLLogger'; | |
| import { RowMode } from '../contracts/IDBSQLSession'; | ||
| import Status from '../dto/Status'; | ||
| import HiveDriverError from '../errors/HiveDriverError'; | ||
| import StagingError from '../errors/StagingError'; | ||
| import ParameterError from '../errors/ParameterError'; | ||
| import OperationStateError, { OperationStateErrorCode } from '../errors/OperationStateError'; | ||
| import ArrowResultConverter from '../result/ArrowResultConverter'; | ||
| import ResultSlicer from '../result/ResultSlicer'; | ||
| import KernelResultsProvider from './KernelResultsProvider'; | ||
| import { arrowSchemaToThriftSchema, decodeIpcSchema, patchIpcBytes } from './KernelArrowIpc'; | ||
| import { decodeNapiKernelError } from './KernelErrorMapping'; | ||
| import { decodeNapiKernelError, ErrorWithSqlState } from './KernelErrorMapping'; | ||
| import { | ||
| KernelStatement, | ||
| KernelNativeAsyncStatement, | ||
|
|
@@ -84,7 +86,10 @@ export type KernelOperationStatement = KernelStatementHandle & Partial<KernelSta | |
| * (`KernelResultsProvider` → `ArrowResultConverter` → `ResultSlicer`) consumes | ||
| * either interchangeably. | ||
| */ | ||
| type KernelFetchHandle = Pick<KernelStatement, 'fetchNextBatch' | 'schema'>; | ||
| type KernelFetchHandle = Pick< | ||
| KernelStatement, | ||
| 'fetchNextBatch' | 'schema' | 'isVolumeOperation' | 'completeVolumeOperation' | ||
| >; | ||
|
|
||
| /** | ||
| * The rich operation-status surface the kernel exposes on a terminal sync | ||
|
|
@@ -220,6 +225,8 @@ export default class KernelOperationBackend implements IOperationBackend { | |
| // already-terminal statement. Drives both fetch and result-metadata. | ||
| private fetchHandlePromise?: Promise<KernelFetchHandle>; | ||
|
|
||
| private volumeCompleted = false; | ||
|
|
||
| // Async path only: set once the kernel reported the statement Succeeded (a | ||
| // status poll, or `awaitResult()` resolving). The kernel exposes no | ||
| // synchronous terminal-state accessor, so the observed success is retained. | ||
|
|
@@ -247,16 +254,15 @@ export default class KernelOperationBackend implements IOperationBackend { | |
| this.cancellableExecution = cancellableExecution; | ||
| this.blockingStatement = statement; | ||
| // Lifecycle surface. The async/metadata handles expose both cancel/close. | ||
| // The sync-execute path uses a composite: `cancel()` always routes to the | ||
| // cancellable execution (lock-free, interrupts a running `result()` | ||
| // mid-compute and is a no-op once terminal); `close()` closes the resolved | ||
| // The sync-execute path uses a composite so SQL execution and subsequent | ||
| // transfer completion remain cancellable. `close()` closes the resolved | ||
| // terminal statement once `result()` produced it, OR — if `result()` is | ||
| // still in flight — proactively cancels the running execution so the server | ||
| // stops computing immediately rather than running on until the kernel's | ||
| // drop-guard fires whenever this handle is eventually GC'd. | ||
| this.lifecycleHandle = cancellableExecution | ||
| ? { | ||
| cancel: () => cancellableExecution.cancel(), | ||
| cancel: () => (this.blockingStatement ? this.blockingStatement.cancel() : cancellableExecution.cancel()), | ||
| close: () => (this.blockingStatement ? this.blockingStatement.close() : cancellableExecution.cancel()), | ||
| } | ||
| : ((asyncStatement ?? statement) as KernelStatementHandle); | ||
|
|
@@ -304,6 +310,28 @@ export default class KernelOperationBackend implements IOperationBackend { | |
| // Fetch / metadata (owned by the kernel-results pipeline). | ||
| // --------------------------------------------------------------------------- | ||
|
|
||
| public async completeStagingOperation(allowedPaths: string[]): Promise<void> { | ||
| failIfNotActive(this.lifecycle); | ||
| const handle = await this.getFetchHandle(); | ||
| try { | ||
| const outcome = await handle.completeVolumeOperation(allowedPaths, true); | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — Nothing checks that the loaded native binding actually has If the binding can't be pinned to a compatible release when this merges, add a feature-detect: when
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed: compatible published native packages are a merge prerequisite, including ordinary statements with stagingAllowedLocalPath. Source KERNEL_REV does not update installed platform dependencies. The matching release must supply completion, all platform pins/locks must advance, and clean installed queries plus Volume workflows must pass before merge. The accessible proxies currently expose 1.4.0, whose API is incompatible. Keeping the gate open; no old-native fallback or unavailable version is being introduced. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for confirming. I agree that pinning a compatible published native release is the right fix, and that it has to land before merge. Until then, the code at this line still calls
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The release plan is coordinated native/connector version updates. Published 1.4.0 remains unsupported for these calls: before connector release, compatible platform pins and locks plus clean installed ordinary-query and Volume validation are required. Source validation selects kernel 855847bd with completeVolumeOperation. We are continuing review of that source pair while keeping package integration as a release prerequisite; no fallback, invented version or completed package proof is claimed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for laying out the release plan. Your reply doesn't challenge the finding. The code at line 318 still calls There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — If the pins can't be bumped in this PR, check
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed; this remains a merge prerequisite and also affects ordinary queries when the staging option is set. Published native 1.4.0 does not supply this completion API. The coordinated release must provide the compatible API, then all platform pins/locks must select it and clean installed-package ordinary-query/Volume workflows must pass. Source-built validation does not close that gate. We are requiring the new native API rather than silently returning raw rows with an older binary or selecting an invented version. The PR description keeps this dependency unresolved; this comment is acknowledged, not claimed fixed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for confirming this. Requiring the new native API and treating it as a merge gate makes sense, and so does not quietly falling back to raw rows. The current code still calls There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — The installed native packages are pinned to 1.4.0, which has no The merge plan relies on the pins being bumped later, but nothing in the code enforces it. A small guard would avoid the regression. For example, if
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed that published 1.4.0 is incompatible. This remains an explicit merge/release gate: select the real compatible native release, update all platform pins and locks, and validate a clean installed package on ordinary queries and Volume workflows before merging. The source pin and source-built tests do not complete that gate. We require the new native APIs and a coordinated release; this change will not silently fall back to ignored staging or raw instructions. No fallback or invented version is being used to close the finding. It remains open until the compatible packages and installation proof exist. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for confirming the 1.4.0 incompatibility. Requiring a coordinated native release instead of a silent fallback is a fair design choice. Right now, though, the gate exists only in the merge plan. At head, line 318 still calls There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The PR description says published native 1.4.0 has no completion API. With that binding (or any stale cached
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Published native 1.4.0 is incompatible; the compatible native versions, all eight platform pins/locks and clean installed-package ordinary-query/Volume checks remain required before merge. This is a coordinated release dependency, with no older-native fallback in the chosen contract. The source pin selects the current kernel, and source-built checks do not complete the package gate. Checking an old binding at load time could improve its diagnostic, but cannot make it compatible or replace that gate. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed that the version pins are the real compatibility gate, and that a shape check can't make an old binding work. My concern is narrower: what happens when the pin gate is bypassed by a stale cached There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — Before this PR the option was logged and ignored, so callers that set it globally and run ordinary queries break. The facade check
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Published native 1.4.0 is incompatible. The actual compatible versions, all eight platform dependency pins/locks, and clean installed ordinary-query/metadata/Volume proof remain required before merge; a source SHA pin does not satisfy those requirements. This is a coordinated-release dependency, with no older-native fallback in the selected contract. A stale-binding method check could improve the load/runtime diagnostic, but cannot make the missing API compatible or replace the package gate. The PR remains blocked on that gate. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed, a method check on the handle can't make native 1.4.0 compatible, and bumping the pins is the real fix. That's still outstanding, though: line 318 calls There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low —
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The incompatible-package concern is valid and remains a required merge gate: compatible native platform versions, all dependency pins/locks, and clean installed SELECT/metadata/Volume proof. Native 1.4.0 lacks these APIs. The source build selects kernel30c8d8e7; both the rebuilt sync/async lifecycle comparisons and ordinary-query controls pass. Coordinated native uptake is required, so we are retaining direct calls rather than adding an older-native compatibility path. This PR is not ready to merge until that release gate passes. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for confirming. Requiring a coordinated native release instead of adding a compatibility path is a reasonable choice. The code at this line still calls There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The sentinel instruction can reach this branch; it is not unreachable. Node has no input-stream upload API, so needs_input_stream produces the explicit StagingError. decodeNapiKernelError returns that ordinary Error unchanged, and it cannot satisfy either remapping condition. No incorrect branch behavior is established. An additional test of this direct rejection remains optional; core handshake/owned-byte coverage and distinct Node storage/instruction/error/lifecycle coverage remain. Unknown future outcomes retain the earlier optional disposition; supported native inputs require coordinated platform pins/locks. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You're right that this branch is reachable, and your explanation of why There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — The PR does say merge depends on updated pins. Until that happens, though, every environment that installs from npm gets this regression. Either guard with
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed: compatible native packages, all eight platform pins/locks and clean installed SELECT, metadata and Volume proof are required before this PR can merge. The declared There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks, blocking the merge on the native package bump is one of the resolutions I suggested, and choosing it over a silent no-op makes sense. Line 317 still calls |
||
| if (outcome === 'needs_input_stream') { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — No test covers the
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a useful optional branch test, but source inspection does not establish a classification defect: the needs_input_stream branch creates a StagingError, and decodeNapiKernelError returns a nonsentinel Error unchanged (KernelErrorMapping.ts:270–274). The class therefore survives this catch. Keeping the existing focused tests and recording the uncovered branch; no behavior change is needed for this finding. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You're right that today's code keeps the class:
peco-review-bot[bot] marked this conversation as resolved.
Resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — No test covers the
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This branch is unchanged in the source-pin update. The same branch was source-reviewed and adjudicated as optional test hardening in the preceding thread (#discussion_r4576690182); no runtime defect or required missing workflow was established. That disposition still applies to the unchanged code. Retaining the existing focused completion/error-provenance tests without another code/test push for this optional case. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree with your reading of the code: There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — No unit test covers the
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The sentinel branch is reachable and intentionally rejects with StagingError('Volume PUT requires a local file path'). decodeNapiKernelError returns an ordinary non-envelope Error unchanged, and this StagingError satisfies neither remapping condition, so its class/message survive. This repeats the existing branch-test disposition. The extra mocked assertion remains optional; the distinct storage/instruction/error/lifecycle tests remain. No behavior defect is demonstrated. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree with your reasoning about the runtime behaviour. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — No test covers the There is also no
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The sentinel branch is reachable, and the plain StagingError is passed through by decodeNapiKernelError; it matches neither remapping condition in this catch. The additional mocked assertion remains optional, with no incorrect runtime outcome established. Distinct native owned-input, adapter error/cancellation/lifecycle and real-driver transfer groups remain. The shared end-to-end workflows are in databricks/databricks-driver-test#2122, tests/comparator-tests/node/config/thrift-vs-sea.yaml. The retained source-built sync/async DBSQL comparisons at kernel029c59cc have two matches, 20 successful positive steps and zero differences, including initial/changed bytes and storage-origin removal. Compatible installed-native and named scheduled proof remain open; duplicating that shared journey in another connector matrix would not close either gate. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Pointing to the shared comparator workflows in databricks-driver-test#2122 is a fair answer to the e2e half of this, and I'm not claiming the current code behaves incorrectly. The unit-test gap is still open, though. The |
||
| throw new StagingError('Volume PUT requires a local file path'); | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed that this rejection branch has no dedicated test. Node exposes file PUT rather than a BinaryIO API; reaching it requires a server instruction for the stream sentinel. The branch throws an existing StagingError, and decodeNapiKernelError returns that non-envelope error unchanged. No runtime defect is established. A third row in the existing test table remains optional branch hardening; the required file lifecycle, completion/raw modes and error-origin paths retain unit and actual-driver coverage. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for confirming. I agree the branch is hard to reach from Node's file-PUT API and that |
||
| } | ||
| this.volumeCompleted = outcome === 'completed'; | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — The native binding returns
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The compatible pinned NAPI producer maps the closed The review-body nits remain acknowledged: preserving cause/extra metadata could improve StagingError diagnostics, while the current shape follows the audited Thrift error; the changelog blank line is cosmetic. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed that the pinned NAPI producer emits only
peco-review-bot[bot] marked this conversation as resolved.
Resolved
|
||
| } catch (err) { | ||
| const mapped = decodeNapiKernelError(err); | ||
| const metadata = (mapped as ErrorWithSqlState).kernelMetadata; | ||
| if ( | ||
|
peco-review-bot[bot] marked this conversation as resolved.
Resolved
|
||
| (mapped instanceof ParameterError && !metadata?.httpStatus) || | ||
| metadata?.errorCode === 'VOLUME_STORAGE_HTTP_ERROR' | ||
| ) { | ||
| throw new StagingError(mapped.message); | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Medium — When a transfer fails with a
Possible fix: for HTTP failures, build the message from
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Checked against Preserving additional diagnostics on legacy The portable local-error branch now has four cases in the existing test: raw errno, absent/zero errno with PermissionDenied→EACCES, and unknown kind→EIO. They check negative errno, path, metadata/cause and absent SQLSTATE. The four cases, broader affected tests and typecheck pass. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for checking the native source at There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — When a transfer fails with That means a caller can't tell a storage 403 from a 404 or other failure except by parsing the message. It also can't correlate the failure with a query id. The local-filesystem branch keeps that metadata, so transfer failures end up less diagnosable than the other error classes this PR separates out. Fix: copy the metadata onto the wrapper, e.g.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 8f800ba narrows conversion to the tagged VOLUME_STORAGE_HTTP_ERROR case and local ParameterError without HTTP metadata. Deferred instruction-fetch 401 and 400 now retain their mapped error identity, SQLSTATE and metadata; the existing backend fixture asserts those controls. Extra structured metadata on legacy storage StagingError remains an optional API enhancement: its existing Thrift surface is the staging class/message, which this adapter preserves. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks. Narrowing the conversion in 8f800ba fixes part of this: instruction-fetch 400 and 401 errors now keep their class, SQLSTATE and metadata. But line 330 still throws There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — When a policy or storage-HTTP failure is re-thrown as
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Additional metadata/cause on StagingError would improve diagnostics. The current mapping deliberately retains the established Thrift staging error class/shape, while SQL instruction errors keep their decoded metadata. Native storage HTTP failures retain their specific The review-body constructor comment is also a valid optional wording improvement: cancellation uses the cancellable execution while pending and the resolved statement afterward. Existing cancellation/lifetime behavior is preserved. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Understood that keeping the established There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — When a transfer failure is re-wrapped as
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The diagnostic loss is real and remains the previously acknowledged optional enhancement. Keeping kernelMetadata/cause would improve storage-error correlation without changing the StagingError class. The current class/message follows the audited Thrift staging contract; deferred SQL instruction failures retain their decoded metadata. The specific storage diagnostic is retained, and the strict deletion oracle rejects generic statement/instruction 404s without relying on dropped metadata. This finding remains visible; no required baseline behavior or newly demonstrated failure calls for another adapter change. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for confirming the diagnostic loss is real. Keeping the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — When a transfer failure is reclassified, A contained fix is to copy
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is the acknowledged optional storage-diagnostic enhancement. Storage failures retain the audited StagingError class and the specific storage diagnostic; deferred SQL instruction HTTP failures retain decoded metadata as stated in the description. The comparator’s REMOVE verification in databricks/databricks-driver-test#2122 requires storage-origin evidence and rejects generic instruction/statement 404s independently of these dropped properties. Copying kernelMetadata/cause would improve correlation, but no required Thrift behavior or newly demonstrated incorrect transfer/error outcome requires an adapter change. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for clarifying. I accept that the PR description's metadata promise covers deferred SQL instruction HTTP failures rather than storage failures, and that the comparator in databricks-driver-test#2122 checks for storage-origin evidence without needing these properties. The code at line 329 still rebuilds the error from There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — Converting a tagged storage HTTP error or a local-policy After the rewrap, a caller cannot tell a 403 from a 404 except by parsing the message, and cannot link the failure to its statement. The deferred instruction errors next to this path keep their metadata intact (the tests check this), so these two error kinds now behave differently. Consider copying
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is the acknowledged optional storage-diagnostic enhancement. Storage failures retain the audited StagingError class and the specific storage diagnostic; deferred SQL instruction HTTP failures retain decoded metadata as stated in the description. The comparator’s REMOVE verification in databricks/databricks-driver-test#2122 requires storage-origin evidence and rejects generic instruction/statement 404s independently of these dropped properties. Copying kernelMetadata/cause would improve correlation, but no required Thrift behavior or newly demonstrated incorrect transfer/error outcome requires an adapter change. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Understood, and I agree this is Low severity and doesn't change Thrift behavior or transfer outcomes. Your reply also confirms the diagnostics are lost: line 329 still builds a bare |
||
| } | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Low — When a transfer error is reclassified as
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The metadata/cause loss is real and remains the previously acknowledged optional diagnostic enhancement. The StagingError class and message follow the audited Thrift staging contract; SQL/instruction-side failures keep their decoded metadata. Genuine storage failures retain the specific storage diagnostic, and deletion verification rejects generic instruction/statement 404s without depending on discarded metadata. This repeats the earlier disposition; no new required behavior failure is established. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks. We agree on the facts: the loss of |
||
| throw mapped; | ||
| } | ||
| } | ||
|
|
||
| public async fetchChunk({ | ||
| limit, | ||
| disableBuffering, | ||
|
|
@@ -316,6 +344,7 @@ export default class KernelOperationBackend implements IOperationBackend { | |
| // Cancel-mid-fetch propagation: if cancel() has flipped the | ||
| // lifecycle flag, fail locally without a wire round-trip. | ||
| failIfNotActive(this.lifecycle); | ||
| if (this.volumeCompleted) return []; | ||
| // Cooperative cancel (parity with ThriftOperationBackend): the facade | ||
| // supplies `isClosed` and re-checks `failIfClosed()` after we return, so | ||
| // bailing with `[]` at a yield point is the contract-correct way for a | ||
|
|
@@ -336,6 +365,7 @@ export default class KernelOperationBackend implements IOperationBackend { | |
|
|
||
| public async hasMore(): Promise<boolean> { | ||
| failIfNotActive(this.lifecycle); | ||
| if (this.volumeCompleted) return false; | ||
| try { | ||
| // `hasMore()` prefetches the next batch, so it fails like `fetchChunk()`. | ||
| const slicer = await this.getResultSlicer(); | ||
|
|
@@ -379,7 +409,7 @@ export default class KernelOperationBackend implements IOperationBackend { | |
| // `arrowSchema` doesn't hit apache-arrow@13's `Unrecognized type | ||
| // "Duration" (18)`. Matches what the per-batch fetch path already does. | ||
| arrowSchema: patchIpcBytes(arrowSchemaIpc.ipcBytes), | ||
| isStagingOperation: false, | ||
| isStagingOperation: Boolean(handle.isVolumeOperation), | ||
| }; | ||
| this.metadata = meta; | ||
| return meta; | ||
|
|
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Uh oh!
There was an error while loading. Please reload this page.