Repository navigation
Add org F: decoy fields that only the values can settle - #3
Merged
Merged
Conversation
Orgs A to D put source accuracy at the ceiling, so they cannot tell one prompt or model from another. Org F gives five target fields two plausible sources each, where the field whose name fits best is wrong: an empty standard Industry beside a filled Vertical__c, a Tier__c that holds a support level beside a Plan__c that holds the tier, a Phone that holds only an extension, a Role__c that holds a job function, and a Stage__c that holds a forecast category. The name-matching baseline scores 65.5% on org F with five wrong sources. Claude has not been run on it yet; the README says so. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
run_eval.py used to write results/<label>.json from scratch, so running one org erased the saved results for the others. It now starts from the existing file and replaces only the orgs run, and each org records its own date and prompt hash. The README gains the org F result for Sonnet (87/87) and a comparison with Haiku on the same orgs (430/435 at roughly a twelfth of the cost). It states that the Haiku and org F raw results are not yet in the repo and that the individual errors have not been read. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
…elled file - results/claude-claude-sonnet-5-5-org-f.json was a byte-for-byte copy of the A to D results, not org F: the org F file had already been overwritten by the bug fixed in c0ff35c. It is removed, and the README says the Sonnet org F figures come from terminal output and need a rerun. - Haiku's five misses, read from the raw results: three are the right field and the right question with the source left empty; two are the phone decoy on org F. In one of those, Haiku decided a 50-of-50 rule_test failure was an effect of masking. rule_test's description and result now state that the rule runs on real values and only the display is masked. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
Three runs, 87 of 87 fields, one allowed question and no extra ones. The note that the raw file was lost now describes the earlier pass and says the two passes are not a like-for-like repeat. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Test org F, "Decoys". Orgs A to D put Claude's source accuracy at 100%, so they cannot tell one prompt or model from another. In org F, five target fields each have two plausible sources, and the field whose name fits best is the wrong one:
accounts.industryAccount.IndustryAccount.Vertical__caccounts.tierAccount.Tier__cAccount.Plan__ccontacts.phoneContact.PhoneContact.MobilePhonecontacts.roleContact.Role__cContact.Buying_Role__cdeals.stageOpportunity.Stage__cOpportunity.StageNameThe name-matching baseline scores 65.5% on org F, with five wrong sources.
Not yet done
Claude has not been run on org F. To run it:
git fetch && git checkout org-f-decoys python scripts/run_eval.py --orgs org_f --runs 3Nothing in the prompt was changed for this org.
174 tests pass.
Generated by Claude Code