Repository navigation
feat(index): do not index .json data by default - #154
Merged
Merged
Conversation
On the engraphis corpus .json was 62.3% of the index (31 123 of 49 920 chunks),
and every one of those chunks came from the fallback chunker: no JSON grammar is
installed, .json is not in PARSE_EXTENSIONS, and the optional language-pack (53
languages) does not ship JSON either - all three checked. Capped or not, those
chunks carry no symbol and no address, so the question was whether they are
needed at all. Measured: they are not.
Full 2 134-file corpus (hardlink mirror, identical content):
mode chunks db time hit@1
json indexed 49 920 287 MB 6.8 min 8/8
json capped at 20 25 209 172 MB 6.5 min 7/8
json not indexed 18 747 143 MB 6.0 min 7/8
Code is untouched: .py 16 380 and .js 1 397 chunks in both the uncapped and the
JSON-free index; exactly 375 .json files disappear (1 592 -> 1 217 files). The
symbol layer stays at 36 589. The only lost hit is the query aimed at the dataset
itself ("coding_memory_v1"); grep_code and read_live_file read from disk and
still find every .json.
FileGuard now skips .json unless MSCODEBASE_INDEX_DATA=1. .ipynb stays indexed:
it is JSON, but its chunks come from parsed code cells.
Turning the flag on later does not backfill an existing index: file hashes do not
change, so delete <project>/.mscodebase/chunks.db* and reindex.
Guards: tests/test_data_files_not_indexed.py (5 tests). Mutations M60-M62 (guard
removed, flag default flipped, notebooks treated as data) each turn a test red.
Docs: CONFIGURATION.md EN/RU/ZH. KNOWN_ISSUES entry with the three-way measurement.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
This was referenced Nov 17, 2026
ManSio
added a commit
that referenced
this pull request
Nov 17, 2026
) * feat(index): keep agent-facing json - manifests and tool contracts The blanket .json skip from #154 also dropped the one class an agent actually uses: manifests (how the project is wired) and tool contracts (what the server exposes, with annotations and input schemas). Inventory of the corpus first: 375 .json / 24.5 MB, of which docs/benchmark- evidence is 284 files / 22.1 MB of frozen artefacts (command, result, source_sha256), eval/datasets is 56 files with Python source embedded as strings (their .py oracles are indexed as code anyway), and only 18 files are contracts or manifests. 40 files carry code-like strings; the real code is already in the index. FileGuard now indexes .json when it is a known manifest name (package.json, package-lock.json, npm-shrinkwrap.json, tsconfig.json, plugin.json, marketplace.json, *.plugin.json, composer.json, deno.json, renovate.json) or when its head carries contract keys (inputSchema / surfaces / annotations, file up to 512 KB). MSCODEBASE_INDEX_DATA=1 still indexes everything. Measured on the full corpus: 8 of 445 .json pass the allowlist, the index grows from 18 747 to 18 814 chunks (+67, +0.4%) and the database stays at 143 MB. The value is real: the query "destructiveHint" now returns docs/MCP_CONTRACT.json as top-1; before the allowlist the contract was not in the index at all. Guards: tests/test_data_files_not_indexed.py (8 tests). Mutations M63-M66 (allowlist ignored, allowlist too wide, contract keys not recognised, size limit removed) each turn a test red. Docs: CONFIGURATION.md EN/RU/ZH list both exceptions. KNOWN_ISSUES entry extended with the inventory and the +67 chunk cost; the two oldest entries moved to docs/archive/KNOWN_ISSUES_2026_10.md to stay under the 300-line gate. * docs(diary): note the json allowlist for manifests and contracts --------- Co-authored-by: ManSio <mscodebase@intelligence.local>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(index): do not index .json data by default
On the engraphis corpus .json was 62.3% of the index (31 123 of 49 920 chunks),
and every one of those chunks came from the fallback chunker: no JSON grammar is
installed, .json is not in PARSE_EXTENSIONS, and the optional language-pack (53
languages) does not ship JSON either - all three checked. Capped or not, those
chunks carry no symbol and no address, so the question was whether they are
needed at all. Measured: they are not.
Full 2 134-file corpus (hardlink mirror, identical content):
Code is untouched: .py 16 380 and .js 1 397 chunks in both the uncapped and the
JSON-free index; exactly 375 .json files disappear (1 592 -> 1 217 files). The
symbol layer stays at 36 589. The only lost hit is the query aimed at the dataset
itself ("coding_memory_v1"); grep_code and read_live_file read from disk and
still find every .json.
FileGuard now skips .json unless MSCODEBASE_INDEX_DATA=1. .ipynb stays indexed:
it is JSON, but its chunks come from parsed code cells.
Turning the flag on later does not backfill an existing index: file hashes do not
change, so delete /.mscodebase/chunks.db* and reindex.
Guards: tests/test_data_files_not_indexed.py (5 tests). Mutations M60-M62 (guard
removed, flag default flipped, notebooks treated as data) each turn a test red.
Docs: CONFIGURATION.md EN/RU/ZH. KNOWN_ISSUES entry with the three-way measurement.