Skip to content

Week 6: the honest number, observability, and the front door - #7

Merged
TAM-DS merged 2 commits into
mainfrom
week-6-capstone
Nov 14, 2026
Merged

TAM-DS merged 2 commits into
mainfrom
week-6-capstone

Conversation

@TAM-DS

@TAM-DS TAM-DS commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Feature-complete. This closes the six-week build.

1. The held-out org (E) — the one number tuned against nothing

Org E was generated at the start, locked down (two objects and several fields the connected user cannot read), and never looked at while the prompt or answer keys were tuned. Run once, at the end — and rescored here by the repo's own scorer, not self-reported:

org E, held out
Accuracy 29/29 (100%)
Questions asked / needed / extra 9 / 9 / 0
Cost / turns ~$0.12, 14 turns

Every other 100% in the repo was measured on an org the prompt had seen. This one wasn't. And the proposals show judgment, which is the point of a locked-down org:

  • Read values, not labels — mapped accounts.industry from a field called Segment__c by looking at its sampled values (Retail, Manufacturing…), no question needed.
  • Asked for access instead of inventing — contacts.email and deals.amount are required but hidden by field-level security; Claude named the missing field and asked, rather than guessing a source.
  • Asked once for a root cause — the whole Task object is unreadable (six blocked activities.* fields); Claude asked for Read on Task once and pointed the rest at that one request, not six separate asks. Left optional activities.notes unmapped, no question.
  • Didn't over-ask — contacts.role had a tempting source (Title), but Title holds job titles, which the target says isn't the role. Left unmapped, no question. 9 asked = 9 needed, 0 extra.

n=1 (the tuned orgs were 3 runs each); --held-out --runs 3 would settle parity later.

2. Observability — firstsync trace

Every tool call has been logged since week 1. trace.py reads that log back and summarizes each session: the stages it moved through, per-tool counts, and any failures. Read-only, and deliberately not an MCP tool — a person reads the agent's record; the agent doesn't narrate its own.

3. The front door

README now opens with an architecture diagram (Claude reads and proposes; a human is the only authorizing step; everything between is deterministic code), documents trace, carries the held-out results section, and updates the status to week 6, feature-complete.

Tests

214 passing (5 new for trace), mypy src clean, no network, no API key.


Generated by Claude Code

Every tool call has been recorded since week 1. trace.py reads that
log back and summarizes it per run_id: the stages the session moved
through, per-tool counts, failures with their messages, and the time
window. Answers the question an FDE is always asked - what did the
agent actually do, and did any of it fail?

New CLI: `firstsync trace` (list runs) and `firstsync trace --run ID`
(one run's detail), with --connection to filter. Read-only, and
deliberately not an MCP tool: a person reads the agent's record, the
agent does not narrate its own.

214 tests (5 new), mypy clean, no network.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
Adds the held-out evaluation artifact (org E, Sonnet 5.5), run once on
a locked-down org that was never looked at while tuning the prompt or
the answer keys. The repo's own scorer rescores it to 100%: 29/29
correct, 9 questions asked for exactly the 9 the key required, 0 extra.

The proposals show judgment, not pattern-matching: it mapped a
mislabeled field (Segment__c holds industries) from its values; asked
for field-level access on required-but-hidden fields (Contact.Email,
Opportunity.Amount) instead of inventing a source; asked once for the
root cause of six blocked activity fields (Task unreadable) rather
than six times; and declined to over-ask where a field was genuinely
just absent.

Front door: README now opens with an architecture diagram (Claude
reads and proposes; a human is the only authorizing step; everything
between is deterministic), documents `firstsync trace`, carries a
held-out results section, and updates the tested table and status to
week 6, feature-complete.

214 tests, mypy clean, no network.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017ggB2HHn8zjybwfaJG1g3R
@TAM-DS
TAM-DS merged commit 92b6118 into main Nov 14, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants