Add safe manual instance rebalancing

This commit is contained in:
2026-10-08 12:37:56 -04:00
parent 26de43639f
commit 012cd51a66
45 changed files with 2006 additions and 154 deletions
+5
View File
@@ -29,6 +29,11 @@ A live policy owns the lobby counts for its selected maps. Overlapping live game
Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and game key waits.
[Capacity rebalancing](native-hosts.md#rebalance-instances) preserves lobby identities
and counts. Each move uses a Provisioning rebuild, which holds scaling for the
affected map until the move settles. The temporary outage does not request a
replacement lobby.
Current positive occupancy from either telemetry or RCON vetoes an empty claim. Empty intervals reset when players arrive, evidence becomes unknown, the instance or lifecycle changes, or observation continuity breaks. Controller downtime does not count toward the empty duration. Failed or uncertain automatic operations pause further actions until an operator inspects their Provisioning recovery and reviews the policy again.
## Pause and cancellation
+64
View File
@@ -442,6 +442,70 @@ restart and create complete only after post-operation health evidence; creation
and rebuild also verify the selected map and running Plutonium revision. The
base-game library and runtime revision are independent records.
## Rebalance instances
Open **Fleet → Capacity → Rebalance** to spread existing instances across active
hosts. Hosts also provides a shortcut. Operators and administrators can run a
rebalance; viewers can inspect its history.
Choose at least two participating hosts, a game selection, and a maximum of
1–20 moves (initially five), then choose **Preview rebalance**. The preview shows
current and projected instance counts, CPU and memory utilization, proposed
source/destination pairs, and exclusions. A larger host can carry more instances:
the planner balances resource utilization relative to each host's capacity.
The selected game limits which instances can move; the estimates include all
workloads on the participating hosts.
The planner uses current measurements and conservative five-minute host history,
current-generation instance resource samples, destination running/startup
allowances, and existing reservations. Each move must reduce the source/destination
pair's peak CPU-or-memory utilization by at least five percentage points. A
successfully moved instance is excluded for 30 minutes to limit repeated moves.
Unplaced queued launches do not reserve physical resources.
Only healthy instances with fresh, confirmed empty occupancy and recoverable
deployed files qualify. Occupied, stopped, unknown, stale, and incompatible
instances are excluded, as are beta, campaign, maintenance, and unresolved
recovery work. Positive current occupancy from either RCON or telemetry vetoes
an empty claim. A partially balanced result is normal when eligible instances
cannot improve the distribution enough.
Choose **Start rebalance** within five minutes of the preview. Submission
rechecks the proposed pairs; a changed plan requires a new preview. One durable
run executes at a time, and it never adds moves beyond the approved list. Before
each move it rechecks occupancy, identity, capacity, and competing work. A move
that no longer qualifies is skipped. After each healthy completion the run waits
for fresh reports from both affected hosts before considering its next move.
Provisioning reserves the destination before removing the source, then recreates
and health-verifies the lobby. The source host stays active. This is an empty-lobby
rebuild with a temporary outage; its address or port may change. The logical
identity, map, saved key, runtime credentials, runtime image, game-file library,
and recorded deployed configuration/scripts/plugins are retained. Untracked
local mutable files are not copied. If deployed bytes or credentials are no longer
available, the instance is excluded instead of silently upgrading to promoted files.
**Stop remaining moves** withdraws work that has not committed source removal.
Once source removal has been queued, that move continues through destination
creation and health verification. Stopping does not undo completed moves.
Unknown agent results retain their original operation and reservations until
reconciliation; they never authorize a duplicate instance. Failures stop the
remaining moves and link to Provisioning recovery. Recovering a move there does
not resume the rest of its run. Runs and operation links survive controller
restarts; completed/stopped runs are retained for 31 days.
Autoscaling retains ownership of lobby counts. A move is represented as a rebuild,
so the affected map's scaling decisions wait for it to settle instead of treating
the temporary outage as missing capacity.
`GET /api/rebalancing` returns host choices and the 20 most recent runs, with active
runs first. `POST /api/rebalancing` accepts `preview` with `settings` (`hostIds`,
`game: all|t5|t6`, `maxMoves`), `start` with `previewId`, or `stop` with the run `id`.
Writes use the existing operator authorization and same-origin checks. Repeating
Start with the same preview returns its original run. The controller planner and
run state live in `src/lib/server/rebalancing/`; all host mutations remain in
Provisioning. Rebalancing is manual; there is no periodic automatic policy.
## Drain and enable hosts
**Drain host** asks for confirmation, blocks new deployments, and migrates