Skip to content

feat: add the read-only settings check and Atrinik's baseline - #108

Merged
zoeyrose merged 9 commits into
mainfrom
feat/read-only-settings-check
Nov 14, 2026
Merged

zoeyrose merged 9 commits into
mainfrom
feat/read-only-settings-check

Conversation

@zoeyrose

@zoeyrose zoeyrose commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds the read-only settings check and Atrinik's baseline file, step 6 of the settings plan the owner approved on 2026-11-14.

  • check_settings.py: one Python 3 script, standard library only, 541 lines. It runs only gh api -X GET as whoever runs it and writes nothing.
  • baselines/atrinik.json: Atrinik's live state as approved and applied that day, with a reminders table of credentials and Apps as plain notes.
  • Offline tests with invented fixtures. The required Policy validation job now runs them and validates the baseline.
  • README, AGENTS.md and CONTRIBUTING.md: how to run it, how a change is made, and the limits.

Intent brief

  • Goal: one small read-only check and one baseline file per organization, so drift shows up the next time anyone runs it, with no stored organization token and no schedule.
  • Decisions:
    • The baseline is JSON and is compared as a subset; lists match in any order. A field GitHub adds later is not a difference.
    • The per-repository required-check rulesets come from one template (required_check_ruleset) plus each repository's checks.
    • A repository's rules are read from GitHub's own view of its default branch. That also catches repository-level rulesets and an organization ruleset that stopped targeting the repository. Every rule of a type must match the profile, and a rule type the profile does not name is a difference.
    • A command is printed only where the check compared the object the command writes: an organization ruleset's body, a setting, a repository field, a repository ruleset. A line about an effective rule names the ruleset the rule comes from and points to that ruleset's own line. Those lines always print; nothing is skipped as "already reported". Earlier versions tried to work out a command for each such line and to drop repeated lines, and three review rounds each found another case where the command did nothing or a dropped line hid a difference.
    • Exit 1 always means differences. One guard turns any unexpected error into exit 3 with its traceback on standard error. --validate catches the usual mistakes in a baseline; it does not have to know every wrong shape, because a shape it misses still ends in exit 3.
    • A single read that fails is a "could not verify" line and the run goes on. Exit 2 means only such lines are left.
    • Teams, Apps, environments and webhooks are compared by name only and reported as "changed since last recorded".
  • Constraints: nothing that writes; no dates that fail it; at most 3 profiles and 10 exceptions; about 500 lines (a test stops it at 550); standard library only; it works without admin:org; a repository it cannot classify fails it.
  • Out of scope: the other organization's baseline (kept in a private repository and passed by path), the shared agent-guidance section, the atrinik/agent-integrations follow-up that names this check, running it in CI against the live organization (that would need a stored token), and hardening against hand-made baseline shapes beyond "never exit 1".

Implementation / behavior

  • Policy, field by field: organization settings; Actions permissions, allowed actions and workflow permissions; organization rulesets (missing, different and unrecorded ones are reported, each with its command); each repository's default-branch rules and required checks, including that GitHub Actions reports them; tag rules on refs/tags/v[0-9]*, for presence; security configurations, the default for new repositories and each repository's attachment; merge settings.
  • Names only: team names and per-repository grants, App slugs with all or selected, environment names, webhook hosts.
  • Classification: every repository must be listed with a profile. Atrinik uses two: standard (22 repositories) and archived (nawerhals). One exception: classic uses "Atrinik advanced CodeQL baseline" because it runs CodeQL itself.
  • Without admin:org: organization-level reads become "could not verify" lines and the per-repository checks keep running.
  • Cost: about 190 GETs for Atrinik, about 20 seconds.

Validation

  • python3 -m unittest discover -s tests: 35 tests pass, offline.
  • python3 check_settings.py --validate baselines/atrinik.json: valid. python3 .github/scripts/check_repository.py: no problems. git diff --check: clean.
  • An earlier head was also run live against a deliberately wrong copy of the baseline; it reported each planted difference and exited 1.
  • Live run against atrinik at 0e1ddac, as the owner's everyday gh login (admin:org, no admin:org_hook), exit 2:
could not verify: orgs/atrinik webhooks: Not Found (HTTP 404) (the login lacks the admin:org_hook scope)

That line is expected: reading organization webhooks needs admin:org_hook, which the everyday login does not have. The baseline records none, and a reminder asks the owner to look once at the Webhooks page.

Limitations / follow-up

  • One changed organization ruleset prints its own line plus one line for each repository it reaches. That is deliberate: the first line has the command.
  • The check does not read organization webhooks or which repositories a selected App reaches with this login, nor repository secrets, deploy keys, Pages, immutable releases, code-scanning languages or who may bypass a repository-level ruleset.
  • Fix commands for paired settings (the squash title and message) print one per line; set both in one call.
  • Follow-ups outside this PR: the other organization's baseline, the shared agent-guidance section, and the atrinik/agent-integrations skills follow-up that names this check.

🤖 Generated with Claude Code

Add check_settings.py, one standard-library Python script that compares
an organization's live GitHub settings with a baseline file. It only
runs `gh api -X GET` as whoever runs it and writes nothing. It prints
nothing on a match, otherwise one line per difference with the gh api
command that would fix it, and keeps "could not verify" lines separate.
Exit 0 match, 1 differences, 2 only unverifiable, 3 could not run.

Policy is checked for organization settings, the Actions policy,
organization rulesets (with a template for the per-repository
required-check ruleset), each repository's effective default-branch and
tag rules, security configurations and their attachment, and merge
settings. Teams and grants, installed Apps and their selection mode,
environment names and webhook hosts are compared by name only and
reported as changed since last recorded. A repository the baseline does
not classify fails the check. Without admin:org the per-repository
checks still run and organization-level reads become unverifiable.

baselines/atrinik.json is written from the live organization as
approved on 2026-11-14, with two profiles, one exception (classic's
advanced CodeQL configuration) and a reminders table of credentials and
Apps as plain notes. The Policy validation job now also runs the offline
tests (invented fixtures) and validates the baseline. README, AGENTS.md
and CONTRIBUTING.md describe how to run it, the change procedure and
the limits.

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

None

zoeyrose and others added 3 commits November 14, 2026 13:55
…rganization ruleset

- gh not logged in, a bad token, a network failure, an unreadable
  repository list or an unexpected API response now exit 3 with a
  "could not run" line instead of exit 2 or a traceback.
- An unexpected rule from an organization ruleset is left to the
  organization ruleset comparison; without admin:org it points to that
  ruleset's PUT fix, never a DELETE.
- Every default-branch rule of a type must match the profile, so a
  differing repository-level rule is a difference; a tag ruleset with
  ref excludes no longer counts. README says what each check catches.

Co-Authored-By: Claude <noreply@anthropic.com>
…opping the run

- List every pull_request parameter in the standard profile (and the
  example fixture), so a repository ruleset that adds code-owner review or
  required reviewers is a difference.
- Only a logged-out or rejected login stops the run; a timeout or
  connection error on one read is "could not verify". README exit 3 text
  now says the run stopped and should be rerun.
- An unreadable tag ruleset body gives one "could not verify" line instead
  of reporting the tag rules as missing.
- --validate checks the shape of profiles, overrides and ruleset rules, so
  a malformed baseline fails Policy validation instead of the live run.
- Without admin:org, name the organization ruleset from the repository's
  ruleset listing, and say "record it" when the baseline lacks it.

Co-Authored-By: Claude <noreply@anthropic.com>
…y baseline section

- A differing rule from a repository ruleset now names repos/O/R/rulesets/ID
  as its fix instead of rewriting the organization ruleset that matches.
- --validate checks the shape of every section the check reads
  (organization, organization_settings, actions keys, rulesets, the
  required-check template, security_configurations, recorded, reminders),
  so a typo fails Policy validation instead of the live run.
- A baseline field missing from a response the login sees in full is a
  difference ("is not in the API response"), not a "needs admin:org?" line;
  a hidden repository field names admin on the repository.
- Make room under the line limit by removing duplication: one pass over
  profile shapes, the attachment check reads through Checker.read, the
  security configurations reuse compare_fields, and the organization-level
  prefetch is gone (about two seconds slower). No line is over 120 columns.

Co-Authored-By: Claude <noreply@anthropic.com>
@zoeyrose

Copy link
Copy Markdown
Member Author

Self-review receipt for #108 at 2861934

What was reviewed. Fresh-context reviewers deep-reviewed the whole diff and the code it touches, with the intent brief in the PR body, in two rounds:

  • Round 0 at 1d70bb0 (whole diff).
  • Round 1 at a599451 (the round-0 fix commit and the code around it).

Round 0 findings (all fixed in a599451):

  1. P2 check_settings.py: a run that verified nothing exited 2, the same as Atrinik's normal result. Fixed: CannotRun; logged-out, HTTP 401, connection failures and an unreadable repository list exit 3 with could not run: ....
  2. P2 check_settings.py: an extra rule in an organization ruleset suggested a DELETE of that whole ruleset, once per repository. Fixed: organization-source rules are reported once at the organization ruleset with its PUT fix; without admin:org per repository with the PUT fix; DELETE only for repository rulesets.
  3. P3 check_settings.py: per-repository rule checks ignored ref excludes and repository-level rules of the same type. Fixed: every rule of a type must match; a tag ruleset with ref excludes does not count; README states what each check catches.
  4. P3 check_settings.py: an unexpected API response crashed with a traceback and exit 1. Fixed: exit 3 with could not run: unexpected API response (...).
  5. P3 PR body: said 19 required-check rulesets; there are 20. Fixed in the PR body.

Round 1 findings (fixed in fec93db):

  1. P3 check_settings.py: a repository-level pull_request rule adding code-owner review or required reviewers passed silently. Fixed: the standard profile lists every pull_request parameter (copied from ruleset 02), with a test.
  2. P3 check_settings.py: one transient timeout aborted the whole run and hid differences already found. Fixed: only a logged-out or rejected login stops the run; one failed read is a could not verify line; README exit-3 text says the run stopped and should be rerun.

The handoff I received for this receipt cut off after these two round-1 findings. fec93db and 2861934 also address further points (unreadable tag ruleset body, --validate shape checks, naming the organization ruleset without admin:org, pointing repository-ruleset fixes at repos/O/R/rulesets/ID, missing-field handling). I cannot list any further round-1 findings or their dispositions from the record I have.

Not reviewed in a clean round. No fresh-context review of fec93db or 2861934 (the current head) is on record for this receipt. The last change had no clean review round. A delta review of a599451..2861934 is still needed before this PR leaves draft.

Local validation at this head (from the commit and PR record): 30 offline tests pass; --validate baselines/atrinik.json is valid; the live Atrinik run exits 2 with only the expected organization-webhooks line.

zoeyrose and others added 5 commits November 14, 2026 14:51
…lidate the repositories section

A pull_request (or other) rule from an organization ruleset that differs
from the profile now gets a fix that names that ruleset: the PUT of its
baseline body when the baseline has it, otherwise "record ruleset NAME in
the baseline, or change the rule in orgs/O/rulesets/ID". It no longer
points at a provider ruleset that already matches. The unexpected-rule
path uses the same helper.

--validate now reports a repositories section that is not an object, a
profile that is not a name and overrides that are not an object, with
exit 3, instead of raising a traceback.

Co-Authored-By: Claude <noreply@anthropic.com>
…baseline and keep --validate from crashing

A rule from a recorded organization ruleset whose baseline body itself
disagrees with the profile (different parameters, or a rule type the
profile does not name) now gets the fix "make the profile and ruleset
NAME agree in the baseline" instead of a PUT that would resend the live
body. The verdict no longer depends on whether the login has admin:org:
such a rule type is a difference with and without it.

--validate now reports a security_configurations section that is a
number, null or boolean with exit 3 instead of raising a traceback.

Co-Authored-By: Claude <noreply@anthropic.com>
… non-object rule parameters

An unexpected rule from a recorded organization ruleset was skipped per
repository whenever the ruleset was listed, even when its body could not be
read, so the drift showed only as "could not verify". Skip it only when
check_rulesets actually read that body.

--validate now requires each ruleset rule's parameters, when present, to be
an object, so a bad required_check_ruleset fails validation instead of
crashing the run and --ruleset.

Co-Authored-By: Claude <noreply@anthropic.com>
…s command

Three review rounds each found another case where the command printed for
a repository's effective rule was a no-op, or where skipping a line hid a
difference, and another baseline shape that crashed with exit 1. Simplify
instead of patching:

- A line about an effective rule (missing, different, unexpected, or the
  required checks) names the ruleset it comes from and always prints. It
  carries a command only for a repository ruleset. For an organization
  ruleset the command is on that ruleset's own line, which exists when the
  ruleset is missing, unrecorded or different from the baseline.
- One guard in main(): exit 1 means differences, so any unexpected error,
  such as a baseline shape --validate does not know, ends in exit 3 with
  its traceback on standard error.

Co-Authored-By: Claude <noreply@anthropic.com>
…le sources no command

Review of 963a488 found:

- A profile that names required_status_checks parameters no longer had
  them compared (a regression from the loop rewrite). They are compared
  again, with a test.
- A rule whose source is not a repository ruleset, including one with no
  source at all, was labelled a repository ruleset and got a DELETE
  suggestion. Only a repository source gets a command now.
- The pointer text and the README said a ruleset line always exists; when
  the ruleset could not be read it is a could-not-verify line.
- The exit-3 message said nothing was compared; nothing was printed.
- A compiled __pycache__ file had been committed. It is removed and
  ignored.

Co-Authored-By: Claude <noreply@anthropic.com>
@zoeyrose

Copy link
Copy Markdown
Member Author

Self-review of 0e1ddac486936e918abe29c94da83fdd0c62be87. This covers everything since the last receipt at 2861934.

What was reviewed

  • Three delta rounds by fresh reviewers on 2861934, ad05ddf and 99bde27. Each found two or three more P3 cases in the same two places: the command worked out for a line about a repository's effective rules (a PUT that changed nothing, a skipped line that hid a difference), and baseline shapes that crashed with exit 1. They were patched in ad05ddf, 99bde27 and 93f5d82.
  • Because the fixes kept producing new cases, the design was simplified in 963a488 instead of patched again: effective-rule lines name their ruleset, always print, and carry a command only for a repository ruleset; one guard turns any unexpected error into exit 3. The per-case logic from the three patch commits is gone.
  • A fresh-context reviewer then reviewed the net delta 2861934..963a488 and all of check_settings.py, ran the tests, the repository check, --validate, the live read-only run, and compared old and new output on modified fixtures.
  • The same reviewer reviewed the fix commit 963a488..0e1ddac and found nothing.

Findings on 963a488, all fixed in 0e1ddac

  1. P2: a profile that names required_status_checks parameters no longer had them compared, so a wrong repository ruleset could exit 0. The loop compares them again; new test.
  2. P2: a compiled __pycache__ file was committed. Removed, and __pycache__/ is ignored.
  3. P3: the pointer text said a ruleset always has its own line; when the ruleset could not be read it is a "could not verify" line. Text and README corrected.
  4. P3: a rule with no source, or a source other than a repository, was labelled a repository ruleset and got a DELETE suggestion. Only a repository source gets a command now; new test.
  5. P3: the exit-3 message said nothing was compared. It now says nothing was printed.

Declined, with reason

  • A string check_integration_id such as "15368" passes --validate and gives "(or from another app)" lines. That is hardening of hand-made baseline shapes, which is out of scope; the result is visible differences, not a silent pass.
  • One changed organization ruleset prints its own line and one line per repository it reaches. Deliberate: nothing is skipped, and the first line has the command.

State at 0e1ddac

35 offline tests pass; the repository check and --validate baselines/atrinik.json pass; check_settings.py is 541 lines. Live read-only run against atrinik: exit 2 with the one expected line (organization webhooks need admin:org_hook).

@zoeyrose
zoeyrose marked this pull request as ready for review November 14, 2026 21:23
@zoeyrose zoeyrose self-assigned this Nov 14, 2026
@zoeyrose
zoeyrose requested a review from a team November 14, 2026 21:23
@zoeyrose
zoeyrose merged commit f710575 into main Nov 14, 2026
6 checks passed
@zoeyrose
zoeyrose deleted the feat/read-only-settings-check branch November 14, 2026 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant