[winding-lines]

Closes #390

What changes

• OVERLAP is a stack setting in hetzner.env, synced by deploy.sh/up.sh (sync_settings) like BACKUP_KEEP. It is on unless OVERLAP=0, and a stack.conf without OVERLAP (a server set up before this) overlaps too. can-overlap says no: OVERLAP=0 in stack.conf when it's off.
• DRAIN_SECONDS is a new required setting in hetzner.env, with no default: how long, in seconds, an old scheduler may keep draining its runs after an overlapping deploy before it is stopped. The containers get it as MORCA_DRAIN_SECONDS.
  • deploy.sh and up.sh (need_settings → need_drain_limit) refuse a missing, empty, 0, negative or non-numeric value first, before morca-stack is sent or anything else changes on the server. The message names the setting and says what to set.
  • morca-stack apply refuses without it before it starts anything (a manual apply on the server), and configure takes only a whole number above 0 for it, and only 0 or 1 for OVERLAP.
  • morca scheduler with MORCA_OVERLAP=on refuses to start without MORCA_DRAIN_SECONDS (unset, 0, negative or not a number), before it migrates or touches the database (morca.instances.drain_limit).
• Since Outdated runs, and deploys that don't wait for them (#397, #390) #406 there was no limit when the setting was unset. That is gone: every instance now has a limit. The non-blocking deploy stays as it was: drain-old returns, and the old scheduler drains in the background until its runs end or the limit runs out.

Where the limit is enforced (design call)

• Primarily in the draining instance (Scheduler._lead, as before with MORCA_DRAIN_SECONDS). It knows when its drain began, and at the limit it kills what's left (control runs' process groups too), ends its row and exits. Because the row is ended, the leader requeues the killed runs as soon as their heartbeats are stale, without waiting for the row to go stale.
• The stack's reaper is a backstop. reap-old reads DRAIN_SECONDS from stack.conf and docker stops an old scheduler that still runs MORCA_STACK_DRAIN_GRACE (60 s) past the limit. The limit is counted from drain-old, which comes after the instance began draining, so a scheduler that knows its limit always goes first. This covers old schedulers started without the limit: on a stack that already had OVERLAP=1, the scheduler replaced by the first deploy after upgrading. drain-old --wait without --timeout uses the same limit plus grace. Nothing is killed before the limit.
• The binary's own MORCA_OVERLAP default stays off. Only the stack turns overlap on by default. So a scheduler run by hand or in a test is one at a time and doesn't need a drain limit, and only an instance requires one.

Upgrading (also in the changelog fragment and in docs/hetzner.md, "The drain limit")
Add DRAIN_SECONDS=<seconds> to hetzner.env before the first deploy with these scripts. Until then, deploy.sh stops with DRAIN_SECONDS isn't set: set DRAIN_SECONDS in <hetzner.env> ... and changes nothing. Overlap is then on unless you add OVERLAP=0:

• A stack that had overlap off pauses and waits once more, on the deploy that turns it on.
• A stack that already had OVERLAP=1 overlaps. The old scheduler it replaces was started without the limit, so its reaper stops it once the limit has run out.

Tests

• tests/hetzner.sh:
  • apply and configure refuse without a valid limit.
  • deploy.sh refuses a missing, empty, 0, -5 or 1h limit, with morca-stack not sent, stack.conf unchanged and the tags unchanged.
  • The default: a hetzner.env without OVERLAP sets OVERLAP=1, and MORCA_OVERLAP=on and MORCA_DRAIN_SECONDS reach morca.env.
  • A stack.conf without OVERLAP asks the image.
  • With OVERLAP=0 from hetzner.env, can-overlap says no without asking the image, MORCA_OVERLAP is dropped and the deploy pauses. Taking it out again turns overlap back on.
  • drain-old --wait and reap-old stop an old scheduler at the limit.
• tests/hetzner-stacks.sh: the synced keys.
• tests/overlap.sh: the scheduler refuses to start without the limit (unset, empty, 0, -5, soon), before the database is migrated.
• tests/outdated.sh: runs with an hour's limit instead of none.
• Locally: test-hetzner, test-overlap, test-outdated and format-check all pass.

Kept to the overlap and drain parts of these scripts, so that it merges cleanly with #410 (cgroups on by default). This PR merges after it.

🤖 Generated with Claude Code

https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6