Roll deployment restarts through a shared fleet-wide recovery limit

This commit is contained in:
2026-09-17 11:25:06 -04:00
parent 3f0d6dd87d
commit 8e04a190b3
12 changed files with 579 additions and 51 deletions
+19 -6
View File
@@ -31,13 +31,26 @@ audit diffs.
## How restarts stay safe
Every push (or beta open) creates a `deployment` row and marks each affected
server `pending`. A background sweep runs every 15s and:
File changes are staged sequentially. Their activation uses rolling restarts:
by default only **one server across the fleet** can restart or recover at a
time. Promotions, rollbacks, drift reconciliation, and beta restarts share this
limit. Set `DEPLOYMENT_RESTART_CONCURRENCY` to a positive integer to raise it.
1. Asks each server over RCON: how many players are on right now?
2. If 0 → tell Komodo to restart the server → row goes `done`.
3. If >0 → hold the row `pending` with the player count; try again next tick.
4. If RCON unreachable → hold `pending` with `error='rcon unreachable'`.
A background sweep runs every 15s. It checks recovering servers first, then
admits queued deployments in oldest-first order when there is capacity. Busy
or unreachable servers stay queued without taking a slot, so other empty
servers can proceed. A successful Komodo restart request retains its slot
until the container is healthy and RCON answers. Health checks start no sooner
than 15 seconds after the request returns, avoiding an immediate check of the
old process. An unhealthy server keeps its slot rather than letting failures
spread across the fleet. A timed-out restart also keeps its slot because Komodo
may have accepted the request.
Slots persist in SQLite across app restarts, and overlapping timer/API triggers
share the active run. Beta options to skip health or match gates still work;
skipping a beta gate does not release the fleet's recovery slot early. New-server
creation is separate from deployment restarts and still starts its container
directly.
No token, no session, no auth freeze. If Komodo or RCON aren't configured,
the sweep marks pending rows `manual` so nothing hangs forever.