Repository navigation
feat: add the read-only settings check and Atrinik's baseline - #108
Conversation
Add check_settings.py, one standard-library Python script that compares an organization's live GitHub settings with a baseline file. It only runs `gh api -X GET` as whoever runs it and writes nothing. It prints nothing on a match, otherwise one line per difference with the gh api command that would fix it, and keeps "could not verify" lines separate. Exit 0 match, 1 differences, 2 only unverifiable, 3 could not run. Policy is checked for organization settings, the Actions policy, organization rulesets (with a template for the per-repository required-check ruleset), each repository's effective default-branch and tag rules, security configurations and their attachment, and merge settings. Teams and grants, installed Apps and their selection mode, environment names and webhook hosts are compared by name only and reported as changed since last recorded. A repository the baseline does not classify fails the check. Without admin:org the per-repository checks still run and organization-level reads become unverifiable. baselines/atrinik.json is written from the live organization as approved on 2026-11-14, with two profiles, one exception (classic's advanced CodeQL configuration) and a reminders table of credentials and Apps as plain notes. The Policy validation job now also runs the offline tests (invented fixtures) and validates the baseline. README, AGENTS.md and CONTRIBUTING.md describe how to run it, the change procedure and the limits. Co-Authored-By: Claude <noreply@anthropic.com>
Dependency Review✅ No vulnerabilities or license issues found.Scanned FilesNone |
…rganization ruleset - gh not logged in, a bad token, a network failure, an unreadable repository list or an unexpected API response now exit 3 with a "could not run" line instead of exit 2 or a traceback. - An unexpected rule from an organization ruleset is left to the organization ruleset comparison; without admin:org it points to that ruleset's PUT fix, never a DELETE. - Every default-branch rule of a type must match the profile, so a differing repository-level rule is a difference; a tag ruleset with ref excludes no longer counts. README says what each check catches. Co-Authored-By: Claude <noreply@anthropic.com>
…opping the run - List every pull_request parameter in the standard profile (and the example fixture), so a repository ruleset that adds code-owner review or required reviewers is a difference. - Only a logged-out or rejected login stops the run; a timeout or connection error on one read is "could not verify". README exit 3 text now says the run stopped and should be rerun. - An unreadable tag ruleset body gives one "could not verify" line instead of reporting the tag rules as missing. - --validate checks the shape of profiles, overrides and ruleset rules, so a malformed baseline fails Policy validation instead of the live run. - Without admin:org, name the organization ruleset from the repository's ruleset listing, and say "record it" when the baseline lacks it. Co-Authored-By: Claude <noreply@anthropic.com>
…y baseline section
- A differing rule from a repository ruleset now names repos/O/R/rulesets/ID
as its fix instead of rewriting the organization ruleset that matches.
- --validate checks the shape of every section the check reads
(organization, organization_settings, actions keys, rulesets, the
required-check template, security_configurations, recorded, reminders),
so a typo fails Policy validation instead of the live run.
- A baseline field missing from a response the login sees in full is a
difference ("is not in the API response"), not a "needs admin:org?" line;
a hidden repository field names admin on the repository.
- Make room under the line limit by removing duplication: one pass over
profile shapes, the attachment check reads through Checker.read, the
security configurations reuse compare_fields, and the organization-level
prefetch is gone (about two seconds slower). No line is over 120 columns.
Co-Authored-By: Claude <noreply@anthropic.com>
Self-review receipt for #108 at 2861934What was reviewed. Fresh-context reviewers deep-reviewed the whole diff and the code it touches, with the intent brief in the PR body, in two rounds:
Round 0 findings (all fixed in
Round 1 findings (fixed in
The handoff I received for this receipt cut off after these two round-1 findings. Not reviewed in a clean round. No fresh-context review of Local validation at this head (from the commit and PR record): 30 offline tests pass; |
…lidate the repositories section A pull_request (or other) rule from an organization ruleset that differs from the profile now gets a fix that names that ruleset: the PUT of its baseline body when the baseline has it, otherwise "record ruleset NAME in the baseline, or change the rule in orgs/O/rulesets/ID". It no longer points at a provider ruleset that already matches. The unexpected-rule path uses the same helper. --validate now reports a repositories section that is not an object, a profile that is not a name and overrides that are not an object, with exit 3, instead of raising a traceback. Co-Authored-By: Claude <noreply@anthropic.com>
…baseline and keep --validate from crashing A rule from a recorded organization ruleset whose baseline body itself disagrees with the profile (different parameters, or a rule type the profile does not name) now gets the fix "make the profile and ruleset NAME agree in the baseline" instead of a PUT that would resend the live body. The verdict no longer depends on whether the login has admin:org: such a rule type is a difference with and without it. --validate now reports a security_configurations section that is a number, null or boolean with exit 3 instead of raising a traceback. Co-Authored-By: Claude <noreply@anthropic.com>
… non-object rule parameters An unexpected rule from a recorded organization ruleset was skipped per repository whenever the ruleset was listed, even when its body could not be read, so the drift showed only as "could not verify". Skip it only when check_rulesets actually read that body. --validate now requires each ruleset rule's parameters, when present, to be an object, so a bad required_check_ruleset fails validation instead of crashing the run and --ruleset. Co-Authored-By: Claude <noreply@anthropic.com>
…s command Three review rounds each found another case where the command printed for a repository's effective rule was a no-op, or where skipping a line hid a difference, and another baseline shape that crashed with exit 1. Simplify instead of patching: - A line about an effective rule (missing, different, unexpected, or the required checks) names the ruleset it comes from and always prints. It carries a command only for a repository ruleset. For an organization ruleset the command is on that ruleset's own line, which exists when the ruleset is missing, unrecorded or different from the baseline. - One guard in main(): exit 1 means differences, so any unexpected error, such as a baseline shape --validate does not know, ends in exit 3 with its traceback on standard error. Co-Authored-By: Claude <noreply@anthropic.com>
…le sources no command Review of 963a488 found: - A profile that names required_status_checks parameters no longer had them compared (a regression from the loop rewrite). They are compared again, with a test. - A rule whose source is not a repository ruleset, including one with no source at all, was labelled a repository ruleset and got a DELETE suggestion. Only a repository source gets a command now. - The pointer text and the README said a ruleset line always exists; when the ruleset could not be read it is a could-not-verify line. - The exit-3 message said nothing was compared; nothing was printed. - A compiled __pycache__ file had been committed. It is removed and ignored. Co-Authored-By: Claude <noreply@anthropic.com>
|
Self-review of What was reviewed
Findings on
Declined, with reason
State at 35 offline tests pass; the repository check and |
Summary
Adds the read-only settings check and Atrinik's baseline file, step 6 of the settings plan the owner approved on 2026-11-14.
check_settings.py: one Python 3 script, standard library only, 541 lines. It runs onlygh api -X GETas whoever runs it and writes nothing.baselines/atrinik.json: Atrinik's live state as approved and applied that day, with a reminders table of credentials and Apps as plain notes.Policy validationjob now runs them and validates the baseline.Intent brief
required_check_ruleset) plus each repository'schecks.--validatecatches the usual mistakes in a baseline; it does not have to know every wrong shape, because a shape it misses still ends in exit 3.admin:org; a repository it cannot classify fails it.atrinik/agent-integrationsfollow-up that names this check, running it in CI against the live organization (that would need a stored token), and hardening against hand-made baseline shapes beyond "never exit 1".Implementation / behavior
refs/tags/v[0-9]*, for presence; security configurations, the default for new repositories and each repository's attachment; merge settings.allorselected, environment names, webhook hosts.standard(22 repositories) andarchived(nawerhals). One exception:classicuses "Atrinik advanced CodeQL baseline" because it runs CodeQL itself.admin:org: organization-level reads become "could not verify" lines and the per-repository checks keep running.Validation
python3 -m unittest discover -s tests: 35 tests pass, offline.python3 check_settings.py --validate baselines/atrinik.json: valid.python3 .github/scripts/check_repository.py: no problems.git diff --check: clean.0e1ddac, as the owner's everydayghlogin (admin:org, noadmin:org_hook), exit 2:That line is expected: reading organization webhooks needs
admin:org_hook, which the everyday login does not have. The baseline records none, and a reminder asks the owner to look once at the Webhooks page.Limitations / follow-up
selectedApp reaches with this login, nor repository secrets, deploy keys, Pages, immutable releases, code-scanning languages or who may bypass a repository-level ruleset.atrinik/agent-integrationsskills follow-up that names this check.🤖 Generated with Claude Code