Skip to content

fix(tools): make symbol answers and investigate evidence honest - #157

Merged
ManSio merged 3 commits into
mainfrom
fix/symbol-answer-honesty
Nov 17, 2026
Merged

ManSio merged 3 commits into
mainfrom
fix/symbol-answer-honesty

Conversation

@ManSio

@ManSio ManSio commented Nov 17, 2026

Copy link
Copy Markdown
Owner

fix(tools): make symbol answers and investigate evidence honest

Audit on a foreign corpus (engraphis, 19 179 chunks): five questions through
MCP, and every answer was checked against the sources on disk instead of the
index.

  1. get_symbol_info reported one definition for a name that has several:
    "intent_recall - 1 defs" while read_only_api.py:300, service.py:2634 and
    routes/v2_api.py:2330 all declare it. build_call_graph now returns every
    same-name node in "definition" (primary first, so defs[0] keeps its meaning)
    and the answer lists them as AMBIGUOUS.

  2. Caller counts were min(total, 15) with depth-2 mixed in and scripts/ silently
    dropped: "3 callers" hid a fourth call from scripts/graph_cli.py, "15" hid
    233 for recall. Direct and transitive are counted separately now, the list
    says "showing 5 of N", and hidden script callers are named.

  3. A module constant answered "0 callers", which reads as "unused"
    (_DESTRUCTIVE_PATTERNS is used at mcp_server.py:2543). Non-callable kinds now
    say the call graph does not apply and point to grep_code.

  4. search_symbols matches substrings ("%reweight%"), so a symbol that does not
    exist answered "10 definitions". Exact/qualified matches are separated from
    "similar names", and the latter are labelled as NOT definitions.

  5. investigate counted chunks whose symbol name appears only in the E26
    FILE/SYMBOLS header as "name match" (54% of a 2 000-chunk sample; 37% in
    core/recall.py). The name channel now compares against the body and prefers
    the chunk that IS the definition (symbol_name == variant), so "where is the
    recall engine implemented" answers engraphis/engines/recall.py:77 instead of
    four method chunks of the class.

  6. graph_query(action="related") accepted a symbol and returned
    "Completed ... related_files: []". It now rejects a non-path target with
    VALIDATION_FAILED and names the expected input.

Also: the impact channel crashed with IndexError on one symbol and the tool only
said "degraded: impact:IndexError". It now isolates per symbol and logs the
traceback with the symbol name; the module had no logger at all, and adding one
exposed that a NameError there would break the whole answer.

Guards: tests/test_symbol_info_honesty.py (6 tests) and
tests/test_evidence_body_match.py (8 tests). Mutations M67-M73 (M69 is a
deliberate no-op control) each turn a test red.

Measured on engraphis after the change: intent_recall -> 3 defs and "1 hidden";
_DESTRUCTIVE_PATTERNS -> "call graph does not apply"; reweight -> "no exact
symbol, similar names"; investigate -> verdict answer with engines/recall.py:77;
related -> VALIDATION_FAILED. Symbol calls stay at 50 ms.

Open (KNOWN_ISSUES, yellow): one impact IndexError at graph.py:1080 is not
reproducible after the change; the channel now logs the symbol if it returns.

Audit on a foreign corpus (engraphis, 19 179 chunks): five questions through
MCP, and every answer was checked against the sources on disk instead of the
index.

1. get_symbol_info reported one definition for a name that has several:
   "intent_recall - 1 defs" while read_only_api.py:300, service.py:2634 and
   routes/v2_api.py:2330 all declare it. build_call_graph now returns every
   same-name node in "definition" (primary first, so defs[0] keeps its meaning)
   and the answer lists them as AMBIGUOUS.

2. Caller counts were min(total, 15) with depth-2 mixed in and scripts/ silently
   dropped: "3 callers" hid a fourth call from scripts/graph_cli.py, "15" hid
   233 for recall. Direct and transitive are counted separately now, the list
   says "showing 5 of N", and hidden script callers are named.

3. A module constant answered "0 callers", which reads as "unused"
   (_DESTRUCTIVE_PATTERNS is used at mcp_server.py:2543). Non-callable kinds now
   say the call graph does not apply and point to grep_code.

4. search_symbols matches substrings ("%reweight%"), so a symbol that does not
   exist answered "10 definitions". Exact/qualified matches are separated from
   "similar names", and the latter are labelled as NOT definitions.

5. investigate counted chunks whose symbol name appears only in the E26
   FILE/SYMBOLS header as "name match" (54% of a 2 000-chunk sample; 37% in
   core/recall.py). The name channel now compares against the body and prefers
   the chunk that IS the definition (symbol_name == variant), so "where is the
   recall engine implemented" answers engraphis/engines/recall.py:77 instead of
   four method chunks of the class.

6. graph_query(action="related") accepted a symbol and returned
   "Completed ... related_files: []". It now rejects a non-path target with
   VALIDATION_FAILED and names the expected input.

Also: the impact channel crashed with IndexError on one symbol and the tool only
said "degraded: impact:IndexError". It now isolates per symbol and logs the
traceback with the symbol name; the module had no logger at all, and adding one
exposed that a NameError there would break the whole answer.

Guards: tests/test_symbol_info_honesty.py (6 tests) and
tests/test_evidence_body_match.py (8 tests). Mutations M67-M73 (M69 is a
deliberate no-op control) each turn a test red.

Measured on engraphis after the change: intent_recall -> 3 defs and "1 hidden";
_DESTRUCTIVE_PATTERNS -> "call graph does not apply"; reweight -> "no exact
symbol, similar names"; investigate -> verdict answer with engines/recall.py:77;
related -> VALIDATION_FAILED. Symbol calls stay at 50 ms.

Open (KNOWN_ISSUES, yellow): one impact IndexError at graph.py:1080 is not
reproducible after the change; the channel now logs the symbol if it returns.
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 42ea06e8-ff52-4146-be5b-819b4f4c2602

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

ManSio added 2 commits November 18, 2026 00:29
Independent review of #157 found five majors; all are fixed here.

1. The kind whitelist was incomplete (function_item, impl_item,
   interface_declaration, ...), so the answer said "the call graph does not
   apply" and printed "Called from" three lines below. It is a deny-list now:
   only variables/constants/properties/fields are declared non-callable, and an
   unknown kind keeps its callers.

2. Exact-name checks were case-sensitive while the graph lookup uses LIKE
   (case-insensitive): "recallengine" stopped finding RecallEngine and the answer
   claimed "no exact symbol, similar names only". Both comparisons use casefold.

3. Direct caller/callee counters were read from new result fields that only the
   graph adapter sets; the legacy SymbolIndex and _hybrid_build_call_graph do
   not, so those answers printed "0 callers" next to a non-empty caller list.
   Counts are derived from the depth already present in every row, and the
   derived fields are gone from the adapter.

4. related validated a dot in the name instead of a real file: Makefile,
   Dockerfile and .env were rejected as "symbols", while Class.method still
   slipped through. It now resolves the path and requires it to exist.

5. AMBIGUOUS listed 10 of N without saying so; it now adds "showing 10 of N".

Nits taken: hidden script callers are counted at depth 1 only (they are printed
next to direct counts), the "similar names" page no longer claims a total, and
name_rows compares symbol_name case-insensitively without relying on dict
identity.

Docs: CHANGELOG [Unreleased] entry. Diary cites PR #157 instead of a commit hash
that amend had rewritten.

Guards: test_symbol_info_honesty.py 10 tests (adds case-insensitive lookup,
unknown-kind callers, legacy shape without derived fields, dotted-tail miss),
test_evidence_body_match.py 9 (adds extension-less file). Mutation M74 restores
the dotted-tail fallback and turns the test red.
@ManSio
ManSio merged commit 9a76d93 into main Nov 17, 2026
15 checks passed
@ManSio
ManSio deleted the fix/symbol-answer-honesty branch November 17, 2026 22:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant