Add safe manual instance rebalancing
This commit is contained in:
@@ -17,6 +17,7 @@ and launch your first server.
|
||||
- [Provisioning](guides/provisioning.md) — lifecycle actions, safety checks, queues, and diagnostics.
|
||||
- [Collections](guides/collections.md) — reusable map lineups and promoted launch files.
|
||||
- [Lobby autoscaling](guides/autoscaling.md) — policies, ownership, pause controls, and the pilot procedure.
|
||||
- [Rebalancing](guides/native-hosts.md#rebalance-instances) — preview and safely spread empty lobbies across active hosts.
|
||||
- [Maintenance](guides/maintenance.md) — health recovery, drift repair, updates, and scheduled restarts.
|
||||
|
||||
### Releases and analytics
|
||||
|
||||
@@ -29,6 +29,11 @@ A live policy owns the lobby counts for its selected maps. Overlapping live game
|
||||
|
||||
Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and game key waits.
|
||||
|
||||
[Capacity rebalancing](native-hosts.md#rebalance-instances) preserves lobby identities
|
||||
and counts. Each move uses a Provisioning rebuild, which holds scaling for the
|
||||
affected map until the move settles. The temporary outage does not request a
|
||||
replacement lobby.
|
||||
|
||||
Current positive occupancy from either telemetry or RCON vetoes an empty claim. Empty intervals reset when players arrive, evidence becomes unknown, the instance or lifecycle changes, or observation continuity breaks. Controller downtime does not count toward the empty duration. Failed or uncertain automatic operations pause further actions until an operator inspects their Provisioning recovery and reviews the policy again.
|
||||
|
||||
## Pause and cancellation
|
||||
|
||||
@@ -442,6 +442,70 @@ restart and create complete only after post-operation health evidence; creation
|
||||
and rebuild also verify the selected map and running Plutonium revision. The
|
||||
base-game library and runtime revision are independent records.
|
||||
|
||||
## Rebalance instances
|
||||
|
||||
Open **Fleet → Capacity → Rebalance** to spread existing instances across active
|
||||
hosts. Hosts also provides a shortcut. Operators and administrators can run a
|
||||
rebalance; viewers can inspect its history.
|
||||
|
||||
Choose at least two participating hosts, a game selection, and a maximum of
|
||||
1–20 moves (initially five), then choose **Preview rebalance**. The preview shows
|
||||
current and projected instance counts, CPU and memory utilization, proposed
|
||||
source/destination pairs, and exclusions. A larger host can carry more instances:
|
||||
the planner balances resource utilization relative to each host's capacity.
|
||||
The selected game limits which instances can move; the estimates include all
|
||||
workloads on the participating hosts.
|
||||
|
||||
The planner uses current measurements and conservative five-minute host history,
|
||||
current-generation instance resource samples, destination running/startup
|
||||
allowances, and existing reservations. Each move must reduce the source/destination
|
||||
pair's peak CPU-or-memory utilization by at least five percentage points. A
|
||||
successfully moved instance is excluded for 30 minutes to limit repeated moves.
|
||||
Unplaced queued launches do not reserve physical resources.
|
||||
|
||||
Only healthy instances with fresh, confirmed empty occupancy and recoverable
|
||||
deployed files qualify. Occupied, stopped, unknown, stale, and incompatible
|
||||
instances are excluded, as are beta, campaign, maintenance, and unresolved
|
||||
recovery work. Positive current occupancy from either RCON or telemetry vetoes
|
||||
an empty claim. A partially balanced result is normal when eligible instances
|
||||
cannot improve the distribution enough.
|
||||
|
||||
Choose **Start rebalance** within five minutes of the preview. Submission
|
||||
rechecks the proposed pairs; a changed plan requires a new preview. One durable
|
||||
run executes at a time, and it never adds moves beyond the approved list. Before
|
||||
each move it rechecks occupancy, identity, capacity, and competing work. A move
|
||||
that no longer qualifies is skipped. After each healthy completion the run waits
|
||||
for fresh reports from both affected hosts before considering its next move.
|
||||
|
||||
Provisioning reserves the destination before removing the source, then recreates
|
||||
and health-verifies the lobby. The source host stays active. This is an empty-lobby
|
||||
rebuild with a temporary outage; its address or port may change. The logical
|
||||
identity, map, saved key, runtime credentials, runtime image, game-file library,
|
||||
and recorded deployed configuration/scripts/plugins are retained. Untracked
|
||||
local mutable files are not copied. If deployed bytes or credentials are no longer
|
||||
available, the instance is excluded instead of silently upgrading to promoted files.
|
||||
|
||||
**Stop remaining moves** withdraws work that has not committed source removal.
|
||||
Once source removal has been queued, that move continues through destination
|
||||
creation and health verification. Stopping does not undo completed moves.
|
||||
Unknown agent results retain their original operation and reservations until
|
||||
reconciliation; they never authorize a duplicate instance. Failures stop the
|
||||
remaining moves and link to Provisioning recovery. Recovering a move there does
|
||||
not resume the rest of its run. Runs and operation links survive controller
|
||||
restarts; completed/stopped runs are retained for 31 days.
|
||||
|
||||
Autoscaling retains ownership of lobby counts. A move is represented as a rebuild,
|
||||
so the affected map's scaling decisions wait for it to settle instead of treating
|
||||
the temporary outage as missing capacity.
|
||||
|
||||
`GET /api/rebalancing` returns host choices and the 20 most recent runs, with active
|
||||
runs first. `POST /api/rebalancing` accepts `preview` with `settings` (`hostIds`,
|
||||
`game: all|t5|t6`, `maxMoves`), `start` with `previewId`, or `stop` with the run `id`.
|
||||
Writes use the existing operator authorization and same-origin checks. Repeating
|
||||
Start with the same preview returns its original run. The controller planner and
|
||||
run state live in `src/lib/server/rebalancing/`; all host mutations remain in
|
||||
Provisioning. Rebalancing is manual; there is no periodic automatic policy.
|
||||
|
||||
## Drain and enable hosts
|
||||
|
||||
**Drain host** asks for confirmation, blocks new deployments, and migrates
|
||||
|
||||
Reference in New Issue
Block a user