electric-cherry/docs/guides/autoscaling.md

Lobby autoscaling

Open Fleet → Autoscaling to keep empty lobbies available for new groups. Policies apply to one map variant or every map in a game. A game-wide policy applies the same minimum, maximum, and spare targets independently to each map, including all variants. A partly occupied Zombies lobby is not an empty spare, even when it still has player slots.

Autoscaling is opt-in. Installing the feature creates no policies and changes no lobby counts. Choose On and save a policy to enable autoscaling.

T5 game-wide policies cover the ten standard Zombies maps; T6 policies cover nine co-op map selections. Dead Ops Arcade, Grief, and Turned are not supported.

Configure a policy

Choose Add policy, select the game and scope, and enter an explicit maximum. The initial editor selects Off, with a minimum of one running lobby, one empty spare, and a 15-minute empty interval. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum.

Choose Save policy to apply changes directly. Saving checks the current fleet, policy revision, ownership conflicts, and host readiness. If saving is blocked, the form keeps your settings and shows the reason. Saved policies show lobby counts, key availability, and the current status. An active job appears once with its latest stage and a View job link. Use Lobbies for per-map spare coverage and occupancy evidence, Activity for decision changes grouped by job, and Metrics for observed capacity and provisioning measurements.

Autoscaling	Behavior
Off	No new actions. Unsubmitted work is withdrawn and delivered operations remain tracked until settled.
On	Adds lobbies and removes surplus lobbies after continuous empty evidence and all removal checks pass.

Plutonium keys belong to a game pool shared by every map, manual launch, collection, beta, and autoscaling policy for that game. An eligible creation uses the lowest available lobby number from 1–20 and reserves any free game key atomically. Key labels do not select a map or lobby. The policy shows saved and available game keys; policy maxima can exceed pool size, and a policy can be saved before any keys are added. An empty or exhausted pool holds new growth with a clear reason; it does not prevent surplus removal. Queued launches reserve their key, stopped instances retain it, and confirmed cleanup returns it to the pool. Failed or uncertain launches keep their reservation until safely reconciled. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent 1.7.0 or later advertising instance.autoscaling-remove on each affected host.

The desired count is min(maximum, max(minimum, occupied lobbies + empty spare target)). The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies allow one outstanding operation across their maps, prioritizing additions before removals. Once an operation settles, the next evaluation can act immediately on another eligible lobby. Surplus lobbies must still satisfy the configured continuous empty duration before removal. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins.

When trimming a map, autoscaling keeps occupied lobbies and the lowest-numbered empty lobbies needed for its minimum and spare targets. It selects the highest-numbered eligible surplus lobby for removal. The required empty lobbies are protected before checking idle timers, so Lobby 1 stays available while a higher surplus lobby completes its empty interval. Existing numbering gaps remain until normal growth needs another lobby; autoscaling does not replace healthy lobbies just to renumber them.

Existing policies saved in the removed Dry run or Create only modes switch to Off on upgrade. Review and turn autoscaling On to resume; pending work remains tracked until settled.

Ownership and blockers

A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected. Game-wide rules include every catalog map and apply each map’s targets independently while sharing the game’s key pool. Pool capacity is a game-wide limit; targets across maps are not guaranteed when all keys are in use. While autoscaling is active, competing collection launches and manual Create, Start, Stop, or Destroy requests are rejected with a policy link. Pausing globally or pausing the policy allows these manual actions through their usual provisioning checks. Restart and Rebuild remain available through the usual provisioning checks. Their pending work and unresolved recovery hold scaling decisions for the affected map, so temporary downtime does not trigger replacement lobbies. Exact collection matching also checks managed maps absent from the collection.

Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and game key waits.

Capacity rebalancing preserves lobby identities and counts. Each move uses a Provisioning rebuild, which holds scaling for the affected map until the move settles. The temporary outage does not request a replacement lobby.

Current positive occupancy from either telemetry or RCON vetoes an empty claim. Empty intervals reset when players arrive, evidence becomes unknown, the instance or lifecycle changes, or observation continuity breaks. Controller downtime does not count toward the empty duration. Failed or uncertain automatic operations pause further actions until an operator inspects their Provisioning recovery and reviews the policy again.

Pause and cancellation

Use Pause all to stop new decisions globally, or Pause on one policy. Use Resume to enable it again; the usual activation checks still apply. A global pause retains each live policy's ownership to prevent overlapping policies, but allows manual fleet, server, and collection actions. Policies paused after a failure also allow manual actions. Work already in progress still uses the normal per-server provisioning checks. Setting a policy Off releases ownership once its outstanding operation settles. Resuming the global switch preserves each policy's On or Off setting and restores the manual-action guard for active policies.

Unsubmitted creates and queued removals are withdrawn where possible. Cancelling an autoscaling operation from Provisioning also turns its policy Off so it cannot immediately request a replacement.

An automatic removal requires fresh controller eligibility, a short-lived authorization, and a host-local occupancy check. The agent claims authorization once, repeats its occupancy check, and must begin mutation within five seconds of requesting the claim. Occupied, unknown, expired, interrupted-before-mutation, or revoked attempts end without waiting to destroy the lobby later. A new attempt needs a new decision and another complete empty interval. Automatic removals cannot use the administrator safety override.

The authorization claim is the cancellation boundary. A pause after that point cannot guarantee withdrawal; the page retains the outstanding operation until its receipt is reconciled. Do not treat cancellation as proof that a delivered host command had no effect. Uncertain commands retain their identity and resource reservations.

Turning a policy Off does not recreate lobbies it removed or remove lobbies it created. Once operations settle, use Collections to restore a fixed lineup through the normal capacity and empty-server gates.

Delete a policy

Open the policy’s More actions menu, choose Delete policy, and confirm to remove its settings, decisions, and measurements. Existing lobbies, saved keys, and Provisioning history are kept. Deleting a live policy releases ownership without changing the lineup, and a new policy can reuse the same scope.

Deletion requires the current policy revision and no unsettled autoscaling operation. If work is outstanding, turn the policy Off and wait for its operation to settle before deleting. Delivered or uncertain host commands retain their policy and admission evidence until reconciliation finishes.

Pilot

The production observation period and live pilot have not been run as part of implementation. Begin them on the operator-selected maps after deploying the controller and compatible agents.

1. Select one or two representative maps and record the baseline lineup, keys, versions, host CPU and memory, and player-hours.
2. Select On after resolving blockers and checking agent capability. Set the minimum, maximum, and spare target, then choose Save policy.
3. Monitor request-to-ready time from Provisioning, browser visibility, successful external joins, empty lobby-hours, spare deficits, and removals. Increase the spare target if creation latency leaves new groups without a suitable lobby.
4. Expand one map at a time when spare availability and gameplay meet your requirements. Pause if they regress.

The page retains up to 31 days of decision evidence and observations, with the 30 most recent decision changes shown. Repeated unchanged evaluations update the decision's last-seen time instead of adding a row. Unresolved intent survives retention. Provisioning history retains the operation and policy backlink separately.

Measurement totals exclude collection gaps. Request-to-verified-readiness summaries include queue time and only completed creates; they do not establish browser visibility or successful joins. Metrics shows average and peak lobby counts, empty spare hours, and map time with no reported empty lobby. Its autoscaling signals compare historical counts with current policy targets. Use host CPU and memory measurements alongside player-hours and spare availability when assessing savings.

API and implementation

GET /api/autoscaling returns policies, current evidence, recent decisions, measurements, and global pause state. Viewers may read it; mutations require an operator or administrator, using the existing authentication and origin checks.

POST /api/autoscaling accepts save, delete, or global-pause. Delete requires id and the current revision. Save requests contain settings, plus id and revision when editing. Use mapSlug: '*' for every map in the selected game; existing map-specific requests are unchanged. Scope cannot change when editing. Save validates the current fleet atomically with the policy update; no preview or token is required. Global pause uses a boolean paused field. Agent removal claims use a separate host-authenticated endpoint and are bound to the exact command, host, instance generation, policy revision, and expiry.

Shared contracts live in src/lib/autoscaling.ts; the evaluator, snapshots, persistence, ownership, and authorization live in src/lib/server/autoscaling/. Provisioning remains the only lifecycle executor. See the approved implementation plan for the design and rollout criteria.