Skip to content

Drop the duplicate SQLite cache path index - #22346

Merged
JukkaL merged 1 commit into
python:masterfrom
hauntsaninja:sqlit
Nov 2, 2026
Merged

JukkaL merged 1 commit into
python:masterfrom
hauntsaninja:sqlit

Conversation

@hauntsaninja

Copy link
Copy Markdown
Collaborator

Authored by Codex, here's its description below.

The UNIQUE constraint on files2.path creates an index on that column. The explicit path_idx indexed the same keys, so every cold-cache insert updated two indexes. Existing databases keep their index and old mypy versions can read the new layout.

This reduced cold torch 2.1.0 import wall time by 1.22% (3.263 s to 3.226 s, 95% interval 1.00% to 1.44%). Self-check fell by 0.43% (6.198 s to 6.171 s, interval 0.20% to 0.68%). These were 16 alternating pairs per input against the parent commit, using compiled mypy on CPython 3.11.8 with a fresh cache and no workers. All diagnostic streams matched. The torch improvement also replicated against the unmodified fork at 1.25% across 14 pairs.

Eight pairs per primer input showed 1.62% on pandas, 1.33% on colour, and 0.88% on jax. The first zulip series was noisy. An independent 16-pair series measured 0.78% (interval 0.50% to 1.02%).

The 16 torch shard files occupied 15,847,424 bytes, down from 16,080,896. Both caches contained 3,061 rows. All 1,020 module data blobs were byte-identical. Each version reused the other's cache and SQLite's integrity checks passed in both directions. The cache diff and fine-grained cache suites passed (552 tests).

The UNIQUE constraint on files2.path creates an index on that column.
The explicit path_idx indexed the same keys, so every cold-cache insert
updated two indexes. Existing databases keep their index and old mypy
versions can read the new layout.

This reduced cold torch 2.1.0 import wall time by 1.22%
(3.263 s to 3.226 s, 95% interval 1.00% to 1.44%). Self-check fell
by 0.43% (6.198 s to 6.171 s, interval 0.20% to 0.68%). These were
16 alternating pairs per input against the parent commit, using compiled
mypy on CPython 3.11.8 with a fresh cache and no workers. All diagnostic
streams matched. The torch improvement also replicated against the
unmodified fork at 1.25% across 14 pairs.

Eight pairs per primer input showed 1.62% on pandas, 1.33% on colour,
and 0.88% on jax. The first zulip series was noisy. An independent
16-pair series measured 0.78% (interval 0.50% to 1.02%).

The 16 torch shard files occupied 15,847,424 bytes, down from
16,080,896. Both caches contained 3,061 rows. All 1,020 module data
blobs were byte-identical. Each version reused the other's cache and
SQLite's integrity checks passed in both directions. The cache diff and
fine-grained cache suites passed (552 tests).
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

@JukkaL JukkaL left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

@JukkaL
JukkaL merged commit 98c1f3e into python:master Nov 2, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants