Repository navigation
Evaluate intersects_extent@1 with DuckDB core && semantics - #14
Merged
Merged
Conversation
…tics FilterApplier treated the extension call as a constant TRUE: rows were never filtered by a pushed `geom && const` (an EXACT filter DuckDB removes from its plan), and under NOT the predicate dropped every row. The example worker then re-applied it through the embedded engine's spatial st_intersects_extent, which treats NaN coordinates as never intersecting and so dropped rows DuckDB core keeps (297 of 1849 corpus pairs). Add WkbExtent, an ISO WKB (LE/BE, Z/M/ZM, nested collections) extent reader matching core: empty geometries and all-NaN POINTs contribute no vertex, a NaN axis on any other vertex is unbounded, bounds inclusive, exact doubles. Malformed/unsupported values (including EWKB flags) raise FilterV2Exception: an ADVISORY predicate then keeps the batch's rows, a REQUIRED one fails the scan. SpatialFilterExampleFunction now relies on FilterApplier alone. Tests run the DuckDB 2.0 conformance corpus (43 geometries, 1849 pairs) through FilterApplier and through the example worker's filter step for all four LE/BE combinations, plus advisory-failure and NULL cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DuckDB 2.0 core reads EWKB, as do the TypeScript and C# SDK evaluators, so rejecting it failed EXACT-mode scans DuckDB would answer. WkbExtent now accepts the Z (0x80000000), M (0x40000000) and SRID (0x20000000) high-bit flags per (sub-)geometry in either byte order: Z/M select the vertex width like the ISO 1000/2000/3000 offsets, and the 4-byte SRID is skipped. A type carrying both conventions, or a truncated SRID, is still rejected. Tests: the whole corpus re-encoded as EWKB (SRID on the outer geometry only, and on every part) must match DuckDB core for all LE/BE combos, plus hand-written EWKB POINT+SRID and POLYGON Z+SRID in both byte orders must match their ISO equivalents. The corpus is regenerated to 46 geometries / 2116 cases (adds POINT Z (nan nan 5), POINT Z (nan nan nan), MULTIPOINT Z ((nan nan 5), (9 9 9))). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The VGI DuckDB extension (v2.0,
67c7710) now derivescolumn && <constant>from exact spatial predicates (ST_Intersects,ST_Within, and others) and pushes it to every worker that declaresduckdb.spatial/intersects_extent@1. This SDK's&&evaluator is therefore in the correctness path for ordinary spatial queries. The semantics are now specified in Query-farm/vgi-python#105 (spec section 5.11.1), with a shared conformance corpus of 46 geometries and 2,116 ordered pairs taken from DuckDB 2.0 core&&. This branch copies that corpus and runs every case in all four byte-order combinations.What was wrong
PushdownFiltersreturned TRUE for every row. A query's owngeom && <constant>, which DuckDB removes from its plan as an exact filter, therefore never filtered anything. UnderNOT, every row was dropped.st_intersects_extent. 297 cases per encoding returned false where DuckDB returns true.Fix
WkbExtentreads ISO WKB and EWKB in either byte order, nested collections included, and follows DuckDB core semantics.PushdownFiltersuses it.FilterApplier.Tests
IntersectsExtentConformanceTestcovers the corpus, an EWKB re-encoding of it, hand-written EWKB cases and failure cases.SpatialFilterConformanceTestin the example worker../gradlew :vgi:test :vgi-example-worker:test: all new tests pass. Two existing tests fail only because their Python conformance peer cannot start in a local environment where a sibling checkout is missing.🤖 Generated with Claude Code