Repository navigation
fix(tools): make symbol answers and investigate evidence honest - #157
Merged
Merged
Conversation
Audit on a foreign corpus (engraphis, 19 179 chunks): five questions through
MCP, and every answer was checked against the sources on disk instead of the
index.
1. get_symbol_info reported one definition for a name that has several:
"intent_recall - 1 defs" while read_only_api.py:300, service.py:2634 and
routes/v2_api.py:2330 all declare it. build_call_graph now returns every
same-name node in "definition" (primary first, so defs[0] keeps its meaning)
and the answer lists them as AMBIGUOUS.
2. Caller counts were min(total, 15) with depth-2 mixed in and scripts/ silently
dropped: "3 callers" hid a fourth call from scripts/graph_cli.py, "15" hid
233 for recall. Direct and transitive are counted separately now, the list
says "showing 5 of N", and hidden script callers are named.
3. A module constant answered "0 callers", which reads as "unused"
(_DESTRUCTIVE_PATTERNS is used at mcp_server.py:2543). Non-callable kinds now
say the call graph does not apply and point to grep_code.
4. search_symbols matches substrings ("%reweight%"), so a symbol that does not
exist answered "10 definitions". Exact/qualified matches are separated from
"similar names", and the latter are labelled as NOT definitions.
5. investigate counted chunks whose symbol name appears only in the E26
FILE/SYMBOLS header as "name match" (54% of a 2 000-chunk sample; 37% in
core/recall.py). The name channel now compares against the body and prefers
the chunk that IS the definition (symbol_name == variant), so "where is the
recall engine implemented" answers engraphis/engines/recall.py:77 instead of
four method chunks of the class.
6. graph_query(action="related") accepted a symbol and returned
"Completed ... related_files: []". It now rejects a non-path target with
VALIDATION_FAILED and names the expected input.
Also: the impact channel crashed with IndexError on one symbol and the tool only
said "degraded: impact:IndexError". It now isolates per symbol and logs the
traceback with the symbol name; the module had no logger at all, and adding one
exposed that a NameError there would break the whole answer.
Guards: tests/test_symbol_info_honesty.py (6 tests) and
tests/test_evidence_body_match.py (8 tests). Mutations M67-M73 (M69 is a
deliberate no-op control) each turn a test red.
Measured on engraphis after the change: intent_recall -> 3 defs and "1 hidden";
_DESTRUCTIVE_PATTERNS -> "call graph does not apply"; reweight -> "no exact
symbol, similar names"; investigate -> verdict answer with engines/recall.py:77;
related -> VALIDATION_FAILED. Symbol calls stay at 50 ms.
Open (KNOWN_ISSUES, yellow): one impact IndexError at graph.py:1080 is not
reproducible after the change; the channel now logs the symbol if it returns.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
added 2 commits
November 18, 2026 00:29
Independent review of #157 found five majors; all are fixed here. 1. The kind whitelist was incomplete (function_item, impl_item, interface_declaration, ...), so the answer said "the call graph does not apply" and printed "Called from" three lines below. It is a deny-list now: only variables/constants/properties/fields are declared non-callable, and an unknown kind keeps its callers. 2. Exact-name checks were case-sensitive while the graph lookup uses LIKE (case-insensitive): "recallengine" stopped finding RecallEngine and the answer claimed "no exact symbol, similar names only". Both comparisons use casefold. 3. Direct caller/callee counters were read from new result fields that only the graph adapter sets; the legacy SymbolIndex and _hybrid_build_call_graph do not, so those answers printed "0 callers" next to a non-empty caller list. Counts are derived from the depth already present in every row, and the derived fields are gone from the adapter. 4. related validated a dot in the name instead of a real file: Makefile, Dockerfile and .env were rejected as "symbols", while Class.method still slipped through. It now resolves the path and requires it to exist. 5. AMBIGUOUS listed 10 of N without saying so; it now adds "showing 10 of N". Nits taken: hidden script callers are counted at depth 1 only (they are printed next to direct counts), the "similar names" page no longer claims a total, and name_rows compares symbol_name case-insensitively without relying on dict identity. Docs: CHANGELOG [Unreleased] entry. Diary cites PR #157 instead of a commit hash that amend had rewritten. Guards: test_symbol_info_honesty.py 10 tests (adds case-insensitive lookup, unknown-kind callers, legacy shape without derived fields, dotted-tail miss), test_evidence_body_match.py 9 (adds extension-less file). Mutation M74 restores the dotted-tail fallback and turns the test red.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(tools): make symbol answers and investigate evidence honest
Audit on a foreign corpus (engraphis, 19 179 chunks): five questions through
MCP, and every answer was checked against the sources on disk instead of the
index.
get_symbol_info reported one definition for a name that has several:
"intent_recall - 1 defs" while read_only_api.py:300, service.py:2634 and
routes/v2_api.py:2330 all declare it. build_call_graph now returns every
same-name node in "definition" (primary first, so defs[0] keeps its meaning)
and the answer lists them as AMBIGUOUS.
Caller counts were min(total, 15) with depth-2 mixed in and scripts/ silently
dropped: "3 callers" hid a fourth call from scripts/graph_cli.py, "15" hid
233 for recall. Direct and transitive are counted separately now, the list
says "showing 5 of N", and hidden script callers are named.
A module constant answered "0 callers", which reads as "unused"
(_DESTRUCTIVE_PATTERNS is used at mcp_server.py:2543). Non-callable kinds now
say the call graph does not apply and point to grep_code.
search_symbols matches substrings ("%reweight%"), so a symbol that does not
exist answered "10 definitions". Exact/qualified matches are separated from
"similar names", and the latter are labelled as NOT definitions.
investigate counted chunks whose symbol name appears only in the E26
FILE/SYMBOLS header as "name match" (54% of a 2 000-chunk sample; 37% in
core/recall.py). The name channel now compares against the body and prefers
the chunk that IS the definition (symbol_name == variant), so "where is the
recall engine implemented" answers engraphis/engines/recall.py:77 instead of
four method chunks of the class.
graph_query(action="related") accepted a symbol and returned
"Completed ... related_files: []". It now rejects a non-path target with
VALIDATION_FAILED and names the expected input.
Also: the impact channel crashed with IndexError on one symbol and the tool only
said "degraded: impact:IndexError". It now isolates per symbol and logs the
traceback with the symbol name; the module had no logger at all, and adding one
exposed that a NameError there would break the whole answer.
Guards: tests/test_symbol_info_honesty.py (6 tests) and
tests/test_evidence_body_match.py (8 tests). Mutations M67-M73 (M69 is a
deliberate no-op control) each turn a test red.
Measured on engraphis after the change: intent_recall -> 3 defs and "1 hidden";
_DESTRUCTIVE_PATTERNS -> "call graph does not apply"; reweight -> "no exact
symbol, similar names"; investigate -> verdict answer with engines/recall.py:77;
related -> VALIDATION_FAILED. Symbol calls stay at 50 ms.
Open (KNOWN_ISSUES, yellow): one impact IndexError at graph.py:1080 is not
reproducible after the change; the channel now logs the symbol if it returns.