Skip to content

Add org F: decoy fields that only the values can settle - #3

Merged
TAM-DS merged 8 commits into
mainfrom
org-f-decoys
Nov 13, 2026
Merged

TAM-DS merged 8 commits into
mainfrom
org-f-decoys

Conversation

@TAM-DS

@TAM-DS TAM-DS commented Nov 13, 2026

Copy link
Copy Markdown
Owner

What this adds

Test org F, "Decoys". Orgs A to D put Claude's source accuracy at 100%, so they cannot tell one prompt or model from another. In org F, five target fields each have two plausible sources, and the field whose name fits best is the wrong one:

Target Decoy Correct source What settles it
accounts.industry Account.Industry Account.Vertical__c Industry is empty on every record
accounts.tier Account.Tier__c Account.Plan__c Tier__c holds support levels; Plan__c holds Free/Standard/Enterprise
contacts.phone Contact.Phone Contact.MobilePhone Phone holds a four-digit extension
contacts.role Contact.Role__c Contact.Buying_Role__c Role__c holds job functions
deals.stage Opportunity.Stage__c Opportunity.StageName Stage__c holds forecast categories

The name-matching baseline scores 65.5% on org F, with five wrong sources.

Not yet done

Claude has not been run on org F. To run it:

git fetch && git checkout org-f-decoys
python scripts/run_eval.py --orgs org_f --runs 3

Nothing in the prompt was changed for this org.

174 tests pass.


Generated by Claude Code

claude and others added 8 commits November 12, 2026 22:02
Orgs A to D put source accuracy at the ceiling, so they cannot tell one
prompt or model from another. Org F gives five target fields two
plausible sources each, where the field whose name fits best is wrong:
an empty standard Industry beside a filled Vertical__c, a Tier__c that
holds a support level beside a Plan__c that holds the tier, a Phone
that holds only an extension, a Role__c that holds a job function, and
a Stage__c that holds a forecast category.

The name-matching baseline scores 65.5% on org F with five wrong
sources. Claude has not been run on it yet; the README says so.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
run_eval.py used to write results/<label>.json from scratch, so running
one org erased the saved results for the others. It now starts from the
existing file and replaces only the orgs run, and each org records its
own date and prompt hash.

The README gains the org F result for Sonnet (87/87) and a comparison
with Haiku on the same orgs (430/435 at roughly a twelfth of the cost).
It states that the Haiku and org F raw results are not yet in the repo
and that the individual errors have not been read.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
…elled file

- results/claude-claude-sonnet-5-5-org-f.json was a byte-for-byte copy of
  the A to D results, not org F: the org F file had already been
  overwritten by the bug fixed in c0ff35c. It is removed, and the README
  says the Sonnet org F figures come from terminal output and need a rerun.
- Haiku's five misses, read from the raw results: three are the right
  field and the right question with the source left empty; two are the
  phone decoy on org F. In one of those, Haiku decided a 50-of-50
  rule_test failure was an effect of masking. rule_test's description and
  result now state that the rule runs on real values and only the
  display is masked.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
Three runs, 87 of 87 fields, one allowed question and no extra ones.
The note that the raw file was lost now describes the earlier pass and
says the two passes are not a like-for-like repeat.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
@TAM-DS
TAM-DS merged commit e9d05f7 into main Nov 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants