Add safe manual instance rebalancing

2026-10-08 12:37:56 -04:00
parent 26de43639f
commit 012cd51a66
45 changed files with 2006 additions and 154 deletions
@@ -442,6 +442,70 @@ restart and create complete only after post-operation health evidence; creation
 and rebuild also verify the selected map and running Plutonium revision. The
 base-game library and runtime revision are independent records.

+## Rebalance instances
+
+Open **Fleet → Capacity → Rebalance** to spread existing instances across active
+hosts. Hosts also provides a shortcut. Operators and administrators can run a
+rebalance; viewers can inspect its history.
+
+Choose at least two participating hosts, a game selection, and a maximum of
+1–20 moves (initially five), then choose **Preview rebalance**. The preview shows
+current and projected instance counts, CPU and memory utilization, proposed
+source/destination pairs, and exclusions. A larger host can carry more instances:
+the planner balances resource utilization relative to each host's capacity.
+The selected game limits which instances can move; the estimates include all
+workloads on the participating hosts.
+
+The planner uses current measurements and conservative five-minute host history,
+current-generation instance resource samples, destination running/startup
+allowances, and existing reservations. Each move must reduce the source/destination
+pair's peak CPU-or-memory utilization by at least five percentage points. A
+successfully moved instance is excluded for 30 minutes to limit repeated moves.
+Unplaced queued launches do not reserve physical resources.
+
+Only healthy instances with fresh, confirmed empty occupancy and recoverable
+deployed files qualify. Occupied, stopped, unknown, stale, and incompatible
+instances are excluded, as are beta, campaign, maintenance, and unresolved
+recovery work. Positive current occupancy from either RCON or telemetry vetoes
+an empty claim. A partially balanced result is normal when eligible instances
+cannot improve the distribution enough.
+
+Choose **Start rebalance** within five minutes of the preview. Submission
+rechecks the proposed pairs; a changed plan requires a new preview. One durable
+run executes at a time, and it never adds moves beyond the approved list. Before
+each move it rechecks occupancy, identity, capacity, and competing work. A move
+that no longer qualifies is skipped. After each healthy completion the run waits
+for fresh reports from both affected hosts before considering its next move.
+
+Provisioning reserves the destination before removing the source, then recreates
+and health-verifies the lobby. The source host stays active. This is an empty-lobby
+rebuild with a temporary outage; its address or port may change. The logical
+identity, map, saved key, runtime credentials, runtime image, game-file library,
+and recorded deployed configuration/scripts/plugins are retained. Untracked
+local mutable files are not copied. If deployed bytes or credentials are no longer
+available, the instance is excluded instead of silently upgrading to promoted files.
+
+**Stop remaining moves** withdraws work that has not committed source removal.
+Once source removal has been queued, that move continues through destination
+creation and health verification. Stopping does not undo completed moves.
+Unknown agent results retain their original operation and reservations until
+reconciliation; they never authorize a duplicate instance. Failures stop the
+remaining moves and link to Provisioning recovery. Recovering a move there does
+not resume the rest of its run. Runs and operation links survive controller
+restarts; completed/stopped runs are retained for 31 days.
+
+Autoscaling retains ownership of lobby counts. A move is represented as a rebuild,
+so the affected map's scaling decisions wait for it to settle instead of treating
+the temporary outage as missing capacity.
+
+`GET /api/rebalancing` returns host choices and the 20 most recent runs, with active
+runs first. `POST /api/rebalancing` accepts `preview` with `settings` (`hostIds`,
+`game: all|t5|t6`, `maxMoves`), `start` with `previewId`, or `stop` with the run `id`.
+Writes use the existing operator authorization and same-origin checks. Repeating
+Start with the same preview returns its original run. The controller planner and
+run state live in `src/lib/server/rebalancing/`; all host mutations remain in
+Provisioning. Rebalancing is manual; there is no periodic automatic policy.
+
 ## Drain and enable hosts

 **Drain host** asks for confirmation, blocks new deployments, and migrates