[TakGN]

Codifies building indexer boxes on Debian 13 with k3s, starting with indexers-chic1 (Cherry Servers, Chicago, 84.32.71.3). Over the next three months it is the test bed for a new indexer setup, which sf and sf2 are meant to adopt later.

The box is built. site.yml -l indexers-chic1 ran from a controller on 2026-11-05 and passed. Its host key is pinned in inventory/known_hosts and matches the key file on the box. Merging is now safe for CI's site.yml runs.

What's in it

• Inventory: a new indexers group, and a managed_boxes parent for it and ipfs_nodes. Shared settings moved to group_vars/managed_boxes/. The IPFS boxes' resolved variables are unchanged, checked with ansible-inventory --host.
• site.yml and k3s-upgrade.yml: now cover indexers. Root login by key is tried before the password fallback. Boxes with undecided placeholders are refused.
• common role: Debian packages, unattended-upgrades also covering point releases on indexers, and reporting when a reboot is needed (needrestart reports only, plus a node-exporter metric).
• hardening role:
  • Removes cloud-image accounts.
  • Checks the effective sshd config with sshd -T. That check also runs on the IPFS boxes.
  • On indexers only (hardening_fix_cloud_init_sshd), deletes cloud-init's 50-cloud-init.conf and sets ssh_pwauth: false. Cherry's Debian image ships that file with PasswordAuthentication yes, which overrode 99-hardening.conf; the sshd -T check caught it on the first build. The IPFS boxes are untouched. None of them has the file, and all three already report passwordauthentication no, so the sshd -T check passes there.
• k3s: the indexers run v1.37.2+k3s1, the stable channel's latest on 2026-11-05. The IPFS boxes stay on v1.36.4.
• k3s-upgrade.yml: compares container IDs before and after each upgrade step, and fails if any container outside kube-system restarted.
• storage role: a new mode, md_xfs, beside the IPFS boxes' ext4_blank_nvme.
  • It creates a partition in the free space on each disk, an md array over them, and XFS at /ipni.
  • It never recreates anything or uses force flags, refuses non-blank members (dry runs included), and won't create an array from CI.
  • It accepts the data label only on the configured array with the configured members.
  • k3s waits until the data volume is mounted.
• indexers-chic1's layout:
  • The OS on a 300 GB md RAID1 with ext4, installed by Cherry.
  • Data on md RAID5 over the rest of each disk, with XFS (about 27 TiB).
  • The drives were switched to 4096-byte sectors before the install.
• platform.yml: now targets managed_boxes, so Argo CD on lith-1 manages the indexers too.
  • Each indexer opens 6443 to lith-1 only and gets the argocd-manager account.
  • The hub registers it as indexerschic1.
  • lith-1's monitoring ingest allowlist (8443) covers every managed box.
  • The Argo CD and monitoring settings moved from group_vars/ipfs_nodes to group_vars/managed_boxes. Their values are unchanged.
• Docs: README ("Indexer boxes", package updates and reboots, the build steps, Cherry's iKVM console) and AGENTS.md.

Tested

• yamllint and ansible-lint pass (production profile). --syntax-check passes on every playbook.
• site.yml on indexers-chic1, for real:
  • common, storage (array and XFS created, mounted at /ipni), hardening and k3s.
  • The node is Ready on v1.37.2, and sshd -T shows permitrootlogin no and passwordauthentication no.
  • The box was rebooted into the patched kernel 6.12.115+deb13.1 and came back with the array and k3s.
• platform.yml --check --diff from a controller:
  • indexers-chic1: 2 changes, both expected (6443 from 46.166.169.131, the argocd-manager account).
  • sing-1 and chic-1: no changes.
  • lith-1: stopped at the vault_argocd_age_key check. The key exists only as CI's ARGOCD_AGE_KEY secret. So the hub's part (registering indexerschic1, the new 8443 rule) is checked only by CI's dry run. Please read lith-1's diff on the run summary before approving the apply.

Not tested

• The container-ID check during a real k3s upgrade.
• sshd -T on the IPFS boxes. It runs on their next site.yml, and fails loudly if a drop-in overrides the hardening there.
• The md_xfs review fixes have not run through site.yml yet; CI's dry run is their first real run.

Follow-ups

• ipni/infra: deploy the monitoring stack to cluster indexerschic1, and alert on node_reboot_required.
• Remove the CVE-2026-98432 nft filter from boxes running a fixed kernel #33: remove the CVE-2026-98432 nft filter on boxes with a fixed kernel (indexers-chic1 already has one).
• Post-quantum SSH key exchange (mlkem768x25519-sha256) in KexAlgorithms.
• Still to decide: the indexer's host-network ports, managing the network bond from the repo, removing exim4, and needrestart reporting only on the IPFS boxes too.

🤖 Generated with Claude Code

[TakGN]

Two additional findings in md_xfs.yml (reviewed at 2d37e8318674eb5751f60574892f02f45f649ec8):

1. [P2] A filesystem label can bypass the missing-array refusal. At lines 55–64, finding storage_md_label anywhere skips the entire creation/refusal block, even when the configured array is absent. The later mount accepts that label without checking its device or RAID members. An unrelated XFS volume with that label could therefore be mounted at /ipni and accepted by k3s’s mountpoint check. Validate the filesystem’s backing array and member identity before accepting it.

2. [P2] Dry runs skip the nonblank-member refusal. At lines 137–154, both wipefs --no-act and its assertion are disabled in check mode. With existing partitions containing an unassembled array, an operator’s prescribed dry run passes this gate, while apply refuses. Probe partitions that already exist during dry runs; skip only those whose creation is being simulated.

These are static control-flow findings; I haven’t verified that either condition exists on the deployed box.

[TakGN]

Thanks. Both are fixed in 32e06ea.

1. Label bypassing the missing-array refusal. Before anything is mounted, a new check requires all three: the array exists, blkid -L for storage_md_label resolves to storage_md_dev, and the array's members (mdadm --detail --export) are exactly the configured ones, compared by kernel name. Otherwise the run stops. The create block is now gated on the array being missing alone.
2. Dry runs skipping the non-blank check. wipefs --no-act now runs in check mode too, on every member partition that already exists. It skips only partitions whose creation is simulated, and the refusal is no longer turned off in check mode.

Checked: I evaluated the guard against indexers-chic1's live values, read with read-only blkid, readlink and mdadm --detail --export commands. The live box passes, as does a fresh box. The label on another device, the label with no array, and an array with a different member are all refused. yamllint and ansible-lint pass.

Not checked: the tasks haven't run through site.yml on a box yet. CI's dry run is their first real run.

[TakGN]

Re-reviewed head 32e06ea2f04179b7c0daabe4ac30d0cdb60ad9a3.

Both previously posted findings are addressed: the role now checks array/member identity before accepting a label, and dry runs probe existing member partitions. Local Ansible fixtures confirmed that the identity guard accepts valid layouts and rejects a label without an array, a wrong label device, and wrong members.

Two earlier findings remain:

1. [P1] CI can still format an existing array. Filesystem creation sits outside the site_require_hardened refusal gate. If the configured array assembles but its filesystem signature is no longer recognized, CI can reach mkfs.xfs. Refusing recognizable filesystems does not guarantee preservation after signature damage. Require explicit provisioning authorization before formatting, including on an existing array. The new identity guard also accepts the assembled-array/no-filesystem case with site_require_hardened: true in the local fixtures.

2. [P2] SSH validation still misses Match overrides. sshd -T runs without connection parameters. A surviving Match User ipni block enabling password authentication can pass the assertion while permitting password login. Previously reproduced locally: plain -T reports password authentication disabled, while adding -C user=ipni,host=example,addr=192.0.2.10 reports it enabled. Validate applicable connection contexts and account for address-specific rules; see OpenSSH's documentation.

No additional findings in this pass. No live-host or destructive storage tests were run.