Skip to content

feat(index): do not index .json data by default - #154

Merged
ManSio merged 2 commits into
mainfrom
perf/skip-json-data
Nov 17, 2026
Merged

ManSio merged 2 commits into
mainfrom
perf/skip-json-data

Conversation

@ManSio

@ManSio ManSio commented Nov 17, 2026

Copy link
Copy Markdown
Owner

feat(index): do not index .json data by default

On the engraphis corpus .json was 62.3% of the index (31 123 of 49 920 chunks),
and every one of those chunks came from the fallback chunker: no JSON grammar is
installed, .json is not in PARSE_EXTENSIONS, and the optional language-pack (53
languages) does not ship JSON either - all three checked. Capped or not, those
chunks carry no symbol and no address, so the question was whether they are
needed at all. Measured: they are not.

Full 2 134-file corpus (hardlink mirror, identical content):

mode                  chunks     db      time     hit@1
json indexed          49 920   287 MB   6.8 min    8/8
json capped at 20     25 209   172 MB   6.5 min    7/8
json not indexed      18 747   143 MB   6.0 min    7/8

Code is untouched: .py 16 380 and .js 1 397 chunks in both the uncapped and the
JSON-free index; exactly 375 .json files disappear (1 592 -> 1 217 files). The
symbol layer stays at 36 589. The only lost hit is the query aimed at the dataset
itself ("coding_memory_v1"); grep_code and read_live_file read from disk and
still find every .json.

FileGuard now skips .json unless MSCODEBASE_INDEX_DATA=1. .ipynb stays indexed:
it is JSON, but its chunks come from parsed code cells.

Turning the flag on later does not backfill an existing index: file hashes do not
change, so delete /.mscodebase/chunks.db* and reindex.

Guards: tests/test_data_files_not_indexed.py (5 tests). Mutations M60-M62 (guard
removed, flag default flipped, notebooks treated as data) each turn a test red.

Docs: CONFIGURATION.md EN/RU/ZH. KNOWN_ISSUES entry with the three-way measurement.

On the engraphis corpus .json was 62.3% of the index (31 123 of 49 920 chunks),
and every one of those chunks came from the fallback chunker: no JSON grammar is
installed, .json is not in PARSE_EXTENSIONS, and the optional language-pack (53
languages) does not ship JSON either - all three checked. Capped or not, those
chunks carry no symbol and no address, so the question was whether they are
needed at all. Measured: they are not.

Full 2 134-file corpus (hardlink mirror, identical content):

    mode                  chunks     db      time     hit@1
    json indexed          49 920   287 MB   6.8 min    8/8
    json capped at 20     25 209   172 MB   6.5 min    7/8
    json not indexed      18 747   143 MB   6.0 min    7/8

Code is untouched: .py 16 380 and .js 1 397 chunks in both the uncapped and the
JSON-free index; exactly 375 .json files disappear (1 592 -> 1 217 files). The
symbol layer stays at 36 589. The only lost hit is the query aimed at the dataset
itself ("coding_memory_v1"); grep_code and read_live_file read from disk and
still find every .json.

FileGuard now skips .json unless MSCODEBASE_INDEX_DATA=1. .ipynb stays indexed:
it is JSON, but its chunks come from parsed code cells.

Turning the flag on later does not backfill an existing index: file hashes do not
change, so delete <project>/.mscodebase/chunks.db* and reindex.

Guards: tests/test_data_files_not_indexed.py (5 tests). Mutations M60-M62 (guard
removed, flag default flipped, notebooks treated as data) each turn a test red.

Docs: CONFIGURATION.md EN/RU/ZH. KNOWN_ISSUES entry with the three-way measurement.
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0ba12123-51ef-408c-9174-c1d60fa950ad

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@ManSio
ManSio merged commit 9631a75 into main Nov 17, 2026
15 checks passed
@ManSio
ManSio deleted the perf/skip-json-data branch November 17, 2026 17:11
ManSio added a commit that referenced this pull request Nov 17, 2026
)

* feat(index): keep agent-facing json - manifests and tool contracts

The blanket .json skip from #154 also dropped the one class an agent actually
uses: manifests (how the project is wired) and tool contracts (what the server
exposes, with annotations and input schemas).

Inventory of the corpus first: 375 .json / 24.5 MB, of which docs/benchmark-
evidence is 284 files / 22.1 MB of frozen artefacts (command, result,
source_sha256), eval/datasets is 56 files with Python source embedded as
strings (their .py oracles are indexed as code anyway), and only 18 files are
contracts or manifests. 40 files carry code-like strings; the real code is
already in the index.

FileGuard now indexes .json when it is a known manifest name (package.json,
package-lock.json, npm-shrinkwrap.json, tsconfig.json, plugin.json,
marketplace.json, *.plugin.json, composer.json, deno.json, renovate.json) or
when its head carries contract keys (inputSchema / surfaces / annotations,
file up to 512 KB). MSCODEBASE_INDEX_DATA=1 still indexes everything.

Measured on the full corpus: 8 of 445 .json pass the allowlist, the index grows
from 18 747 to 18 814 chunks (+67, +0.4%) and the database stays at 143 MB.
The value is real: the query "destructiveHint" now returns
docs/MCP_CONTRACT.json as top-1; before the allowlist the contract was not in
the index at all.

Guards: tests/test_data_files_not_indexed.py (8 tests). Mutations M63-M66
(allowlist ignored, allowlist too wide, contract keys not recognised, size limit
removed) each turn a test red.

Docs: CONFIGURATION.md EN/RU/ZH list both exceptions.
KNOWN_ISSUES entry extended with the inventory and the +67 chunk cost; the two
oldest entries moved to docs/archive/KNOWN_ISSUES_2026_10.md to stay under the
300-line gate.

* docs(diary): note the json allowlist for manifests and contracts

---------

Co-authored-by: ManSio <mscodebase@intelligence.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant