Roll deployment restarts through a shared fleet-wide recovery limit
This commit is contained in:
@@ -31,13 +31,26 @@ audit diffs.
|
||||
|
||||
## How restarts stay safe
|
||||
|
||||
Every push (or beta open) creates a `deployment` row and marks each affected
|
||||
server `pending`. A background sweep runs every 15s and:
|
||||
File changes are staged sequentially. Their activation uses rolling restarts:
|
||||
by default only **one server across the fleet** can restart or recover at a
|
||||
time. Promotions, rollbacks, drift reconciliation, and beta restarts share this
|
||||
limit. Set `DEPLOYMENT_RESTART_CONCURRENCY` to a positive integer to raise it.
|
||||
|
||||
1. Asks each server over RCON: how many players are on right now?
|
||||
2. If 0 → tell Komodo to restart the server → row goes `done`.
|
||||
3. If >0 → hold the row `pending` with the player count; try again next tick.
|
||||
4. If RCON unreachable → hold `pending` with `error='rcon unreachable'`.
|
||||
A background sweep runs every 15s. It checks recovering servers first, then
|
||||
admits queued deployments in oldest-first order when there is capacity. Busy
|
||||
or unreachable servers stay queued without taking a slot, so other empty
|
||||
servers can proceed. A successful Komodo restart request retains its slot
|
||||
until the container is healthy and RCON answers. Health checks start no sooner
|
||||
than 15 seconds after the request returns, avoiding an immediate check of the
|
||||
old process. An unhealthy server keeps its slot rather than letting failures
|
||||
spread across the fleet. A timed-out restart also keeps its slot because Komodo
|
||||
may have accepted the request.
|
||||
|
||||
Slots persist in SQLite across app restarts, and overlapping timer/API triggers
|
||||
share the active run. Beta options to skip health or match gates still work;
|
||||
skipping a beta gate does not release the fleet's recovery slot early. New-server
|
||||
creation is separate from deployment restarts and still starts its container
|
||||
directly.
|
||||
|
||||
No token, no session, no auth freeze. If Komodo or RCON aren't configured,
|
||||
the sweep marks pending rows `manual` so nothing hangs forever.
|
||||
|
||||
Reference in New Issue
Block a user