Repository navigation
Week 6: the honest number, observability, and the front door - #7
Merged
Merged
Conversation
Every tool call has been recorded since week 1. trace.py reads that log back and summarizes it per run_id: the stages the session moved through, per-tool counts, failures with their messages, and the time window. Answers the question an FDE is always asked - what did the agent actually do, and did any of it fail? New CLI: `firstsync trace` (list runs) and `firstsync trace --run ID` (one run's detail), with --connection to filter. Read-only, and deliberately not an MCP tool: a person reads the agent's record, the agent does not narrate its own. 214 tests (5 new), mypy clean, no network. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
Adds the held-out evaluation artifact (org E, Sonnet 5.5), run once on a locked-down org that was never looked at while tuning the prompt or the answer keys. The repo's own scorer rescores it to 100%: 29/29 correct, 9 questions asked for exactly the 9 the key required, 0 extra. The proposals show judgment, not pattern-matching: it mapped a mislabeled field (Segment__c holds industries) from its values; asked for field-level access on required-but-hidden fields (Contact.Email, Opportunity.Amount) instead of inventing a source; asked once for the root cause of six blocked activity fields (Task unreadable) rather than six times; and declined to over-ask where a field was genuinely just absent. Front door: README now opens with an architecture diagram (Claude reads and proposes; a human is the only authorizing step; everything between is deterministic), documents `firstsync trace`, carries a held-out results section, and updates the tested table and status to week 6, feature-complete. 214 tests, mypy clean, no network. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature-complete. This closes the six-week build.
1. The held-out org (E) — the one number tuned against nothing
Org E was generated at the start, locked down (two objects and several fields the connected user cannot read), and never looked at while the prompt or answer keys were tuned. Run once, at the end — and rescored here by the repo's own scorer, not self-reported:
Every other 100% in the repo was measured on an org the prompt had seen. This one wasn't. And the proposals show judgment, which is the point of a locked-down org:
accounts.industryfrom a field calledSegment__cby looking at its sampled values (Retail, Manufacturing…), no question needed.contacts.emailanddeals.amountare required but hidden by field-level security; Claude named the missing field and asked, rather than guessing a source.Taskobject is unreadable (six blockedactivities.*fields); Claude asked for Read onTaskonce and pointed the rest at that one request, not six separate asks. Left optionalactivities.notesunmapped, no question.contacts.rolehad a tempting source (Title), but Title holds job titles, which the target says isn't the role. Left unmapped, no question. 9 asked = 9 needed, 0 extra.n=1 (the tuned orgs were 3 runs each);
--held-out --runs 3would settle parity later.2. Observability —
firstsync traceEvery tool call has been logged since week 1.
trace.pyreads that log back and summarizes each session: the stages it moved through, per-tool counts, and any failures. Read-only, and deliberately not an MCP tool — a person reads the agent's record; the agent doesn't narrate its own.3. The front door
README now opens with an architecture diagram (Claude reads and proposes; a human is the only authorizing step; everything between is deterministic code), documents
trace, carries the held-out results section, and updates the status to week 6, feature-complete.Tests
214 passing (5 new for trace),
mypy srcclean, no network, no API key.Generated by Claude Code