Skip to content

Per-stack morca.toml overlays, kept with the stack (#332) - #399

Merged
winding-lines merged 10 commits into
mainfrom
stack-overlay
Nov 12, 2026
Merged

winding-lines merged 10 commits into
mainfrom
stack-overlay

Conversation

@winding-lines

@winding-lines winding-lines commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Closes #332. Part of census#140.

Targets main, with #396 (the rename) merged in, and uses the new names. #400 (#331, apps declare their env and secrets) is stacked on this PR.

What it does

A project ships one morca.toml. Whatever one stack runs differently goes in that stack's overlay, a file kept beside its hetzner.env in ops, not in the project.

# ops-secrets dev/morca.overlay.toml
[pipelines.gharchive]
fills = "since_registration"

[pipelines.ballast]
enabled = true
  • Merging (TomlDoc.merge): morca loads morca.toml, then lays the overlay on it.
    • A key in the overlay replaces the file's value at that path, including a table or an array.
    • A table is merged key by key, so env = { A = "1" } keeps the file's other names.
    • An overlay can't remove a key. To get the default back, set the default value.
    • [defaults] in an overlay doesn't beat a pipeline's own key in the file, since the overlay merges into each table separately. To change a value a pipeline sets itself, set it in that pipeline's table.
  • Pipelines: an overlay table that names a pipeline the file doesn't have is refused, with overlay:line. A new key, enabled = false, makes morca register (with no binaries) skip a pipeline. So a pipeline that only one stack runs lives in morca.toml, and that stack's overlay enables it. The issue didn't ask for enabled, but without it orca.lab.toml can't become an overlay: the lab registers only ballast, and an overlay can't add or remove pipelines. A pipeline that's already registered stays registered, as it does today when a pipeline leaves the file.
  • Checking and showing:
    • morca config check [--overlay F] checks the merged result; problems point at the overlay's file:line.
    • The new morca config show --file [<file>] [--overlay F] prints the merged file without a database. Each value the overlay set is marked # overlay F:line.
    • A registration records overlay and overlay_hash. morca config show names the overlay and gives each value it set as [pipelines.x], overlay F:line.
    • The overlay comes from --overlay or MORCA_CONFIG_OVERLAY. --no-config and morca aws register don't use one.
  • Hetzner delivery:
    • CONFIG_OVERLAY_FILE in hetzner.env is read relative to that file's folder, like the other *_FILE settings, and may be SOPS-encrypted.
    • sync_settings sends it as /etc/orca/overlay/morca.overlay.toml, and settings.env gets MORCA_CONFIG_OVERLAY=/etc/orca/overlay/morca.overlay.toml. When it's taken out of hetzner.env, morca-stack sync --remove deletes the file and the key goes away.
    • /etc/orca/overlay is mounted, as a directory so a replaced file shows through, in the scheduler only. That's where morca register runs; the UI doesn't read the config.
    • morca-stack plan and apply have the new image's morca run config check with the overlay before anything is paused or replaced. A bad overlay stops the deploy at the plan. So does an image whose morca doesn't print with <overlay>: a morca from before this PR would quietly register without the overlay.
    • The overlay is one of the scheduler's spec files. Changing only the overlay replaces the scheduler (settings: its env file or the overlay changed), so deploy.sh quiesces and registers again. A stack without an overlay gets no new mount and no new spec, so this PR replaces nothing on it.

Design notes

  • I put the hetzner.env setting under CONFIG_OVERLAY_FILE, not MORCA_CONFIG_OVERLAY. It's a path on the laptop, like the other *_FILE settings, and the server-side MORCA_CONFIG_OVERLAY is derived from it.
  • src/morca/config.mojo grows to about 1,040 lines, which is over AGENTS.md's 1,000 but under the 1,500 split point. The TOML merge and the display code went into toml.mojo, and the rest of the change needs ProjectConfig's internals.

Census side (its own PR, after morca-mojo 0.25.0)

  • Ship one morca.toml. Add [pipelines.ballast] with memory_gb = "auto", cpus = "auto" and enabled = false, then delete orca.dev.toml and orca.lab.toml. The Dockerfile copies only morca.toml.
  • CI: drop the "orca.dev.toml matches orca.toml" step and the config check of the dev and lab files. scripts/changelog.sh's check that a run step's pipeline is in both files becomes morca.toml only.

ops-secrets side (in the release window)

  • dev/morca.overlay.toml:
    [pipelines.gharchive]
    fills = "since_registration"
    [pipelines.ballast]
    enabled = true
    fill_order, fill_max_runs and fill_lag stay but do nothing without fills.
  • lab/morca.overlay.toml:
    [defaults]
    enabled = false
    [pipelines.ballast]
    enabled = true
  • In dev/hetzner.env and lab/hetzner.env, add CONFIG_OVERLAY_FILE=morca.overlay.toml, and take ORCA_CONFIG=/opt/orca/orca.<stack>.toml out of extra.env.
  • The first dev and lab deploy with an overlay replaces the scheduler once, because settings.env gains a key. Pipelines the lab overlay disables stay registered if they ever were: the lab only had ballast.

Tests

Run locally on macOS:

  • test-toml: a new merge case.
  • test-config: the unit tests (overlay, enabled, locations, refusals) and tests/config.sh. It now covers check, show --file and register with --overlay and MORCA_CONFIG_OVERLAY, the recorded overlay, and register skipping a disabled pipeline.
  • tests/hetzner.sh: the overlay is sent, checked in the image, mounted only in the scheduler, and a change to it replaces the scheduler and re-registers. A bad overlay and an image without overlay support each stop at the plan, before any pause. Removing the setting removes the overlay.
  • tests/hetzner-stacks.sh: the relative path and the dry run.
  • tests/no-orca.sh and mojo format.

CI runs the rest.

🤖 Generated with Claude Code

https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6

winding-lines and others added 4 commits November 12, 2026 03:56
A project ships one morca.toml; what one stack runs differently goes in
an overlay kept with the stack. morca lays it on the file before anything
reads it (TomlDoc.merge: a key replaces, a table merges key by key), and
refuses an overlay table naming a pipeline the file hasn't.

- morca register, config check: --overlay F or MORCA_CONFIG_OVERLAY;
  config show --file prints the merged file with the overlay's values
  marked; a registration records the overlay and its hash.
- enabled = false: register without binaries skips the pipeline, so a
  pipeline only one stack runs lives in morca.toml too.
- Hetzner: CONFIG_OVERLAY_FILE in hetzner.env is sent as
  /etc/orca/overlay/morca.overlay.toml, mounted in the scheduler only,
  checked by the image's morca in plan and apply before anything changes,
  and is one of the scheduler's files, so a change re-registers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
morca-stack wrote one fixed app.env for every app: the lake, the S3 keys
and the UI's PostHog key as POSTHOG_KEY. Now an app's [apps.<name>] says
what it gets (port, check, env, lake, secrets, memory_gb), read from the
app's own image with the new `morca config apps` (the overlay laid on
it), and each app has its own env file, app-<name>.env.

- Secrets: a secret's value comes from /etc/orca/secrets/<id>, never
  printed; SECRETS_FILE may hold any secrets/orca/<id> key, and sync
  takes them. A missing one stops the deploy at the plan.
- APPS=console names the app; a port there is for an app with none in
  morca.toml, and one that differs is refused.
- An undeclared app, or an image without `config apps`, keeps the old
  fixed env, so today's census images deploy as before.
- MORCA_POSTHOG_HOST (POSTHOG_HOST in hetzner.env): where the UI's
  PostHog snippet goes, a proxy like /ph; PostHog's own host without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
winding-lines and others added 2 commits November 12, 2026 04:13
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
Base automatically changed from morca-rename to main November 12, 2026 13:09
@winding-lines
winding-lines marked this pull request as ready for review November 12, 2026 17:59
@winding-lines
winding-lines merged commit bbe2a94 into main Nov 12, 2026
32 checks passed
@winding-lines
winding-lines deleted the stack-overlay branch November 12, 2026 19:35
winding-lines added a commit that referenced this pull request Nov 12, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Per-stack orca.toml overlays, kept with the stack, not the project

1 participant