162 lines
27 KiB
Markdown
162 lines
27 KiB
Markdown
# Lobby autoscaling implementation plan
|
||
|
||
Status: Approved and implemented October 6, 2026. Controller support and agent 1.7.0 are complete; production pilot measurements and live rollout remain pending. See [usage and the pilot procedure](../guides/autoscaling.md).
|
||
|
||
The design below preserves the approved scope and subsequent implementation
|
||
record. Use the linked guide for current policy behavior, including game-wide
|
||
policies and shared game key pools added after the initial design.
|
||
|
||
Introduce an opt-in policy for each game and map variant that keeps a small number of empty, ready lobbies available. When the last spare fills, request another lobby through Provisioning. Remove surplus lobbies only after sustained, confirmed inactivity. This reduces idle hosting while preserving room for new groups, including when existing Zombies sessions have free player slots but are already well into a game.
|
||
|
||
Start with Create/Destroy on existing native hosts. Run the evaluator in dry run for two weeks, then enable creation for a small pilot before enabling removal. Host acquisition, VM shutdown, Stop/Start pooling, scheduled demand prediction, and waking servers from incoming packets are outside the first release.
|
||
|
||
Update October 7, 2026: Dry run and Create only have been removed. Current policies support Off and On, defaulting to Off. Existing policies in removed modes switch to Off on upgrade. Action cooldowns have also been removed; policies retain the continuous empty duration before removal and one outstanding operation at a time. Policies now save directly with server-side validation; no preview or confirmation token is required. The original implementation stages below record the initial design; follow the current [usage and pilot procedure](../guides/autoscaling.md) for operation.
|
||
|
||
## Existing foundations and required extensions
|
||
|
||
| Foundation | Reuse and required work |
|
||
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| Background worker | Add a requester to [pipeline/worker.ts](../../src/lib/server/pipeline/worker.ts), which already reconciles background work every 15 seconds independently of Provisioning. Keep host execution inside Provisioning. |
|
||
| Occupancy evidence | Reuse [status evidence](../../src/lib/server/fleet/status-evidence.ts) and shared freshness rules. Monitoring runs on a 30-second cadence; accepted evidence currently expires after 90 seconds. Positive occupancy vetoes an empty claim. Unknown is never zero. |
|
||
| Safe creation | Use the durable creation intent and placement path in [provisioning/native.ts](../../src/lib/server/provisioning/native.ts). Preserve CPU, memory, port, host eligibility, library, and startup concurrency checks. A blocked create remains queued. |
|
||
| Safe removal | Reuse the controller occupancy checks, instance generation checks, host-local check, and cleanup checkpoints. Add autoscaling-specific admission and cancellation behavior before enabling automatic destruction. |
|
||
| Requester eligibility | [provisioning/worker.ts](../../src/lib/server/provisioning/worker.ts) currently gives Create and Destroy separate paths that bypass the runtime requester eligibility check. Autoscaling ownership and policy checks must cover both paths, including delayed execution. A source label alone is insufficient. |
|
||
| Saved keys | [server-keys.ts](../../src/lib/server/fleet/server-keys.ts) stores keys in game pools. Each live or pending launch reserves a key until cleanup confirms it is safe to reuse. Allocate lobby identities numbered 1–20 independently of the key label. |
|
||
| Collections | [collections.ts](../../src/lib/server/fleet/collections.ts) already counts queued creations and launches promoted recipes. Exact matching can remove lobbies across the fleet, including maps omitted from the collection. Add ownership checks to preview and submission. |
|
||
| Hosting insights | [metrics-insights.ts](../../src/lib/metrics-insights.ts) provides historical suggestions for hosting trials. Use these to help choose policies; the live evaluator needs current empty-lobby evidence and must not execute historical suggestions directly. |
|
||
|
||
Create already uses the promoted launch recipe and capacity-aware placement. Start retains existing files and has recovery concurrency controls, but would need equivalent CPU and memory admission before becoming an autoscaling strategy. The proposed scope follows option 4 in [the idle CPU research](../research/t5-t6-idle-cpu-research.md).
|
||
|
||
## Policy and ownership
|
||
|
||
Use one policy per `(game, mapSlug)`, preserving configured variants such as T6 start locations. Do not group ambiguous legacy records by engine map alone. Resolve or exclude ambiguous identities before activation.
|
||
|
||
| Setting | Proposed initial behavior |
|
||
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| Mode | Off, Dry run, Create only, or Create and destroy. Default Off. |
|
||
| Minimum running lobbies | Default 1; a floor for desired ready capacity, subject to available keys and host capacity. |
|
||
| Maximum lobbies | Required explicit limit, at most 20. Count occupied lobbies, registered identities, reservations, and unresolved creations toward admission. |
|
||
| Empty spare target | Default 1. Count empty ready lobbies, not unfilled player slots. |
|
||
| Empty duration before removal | Default 15 minutes; adjustable after the pilot. |
|
||
| Action cooldown | Default 5 minutes between new policy actions, measured from durable admission. Cancellation and safety checks do not wait for cooldown. |
|
||
| Launch settings | Reuse normal map recipes and automatic placement on eligible active hosts. Use current promoted files when provisioning begins. |
|
||
|
||
Validate integer counts with `1 <= minimum <= maximum <= 20` and `1 <= spare target <= maximum`. Show the game pool’s saved and available counts independently of each map’s maximum. Policies can be saved with fewer keys than the maximum, or with no keys yet. Exhaustion holds growth across every map sharing that game, while eligible surplus removal can still return capacity. Game-wide targets are per map, but game keys are shared across all maps and launch paths.
|
||
|
||
Live activation explicitly hands management of the game's map variant to the policy and adopts its eligible existing native lobbies. Show those lobbies, exclusions, current keys, and the first proposed action in the activation preview. Do not activate across unresolved provisioning, collection launches, or ambiguous identity ownership. Dry run observes without acquiring ownership or reserving names, ports, or capacity.
|
||
|
||
While a policy is live, reject collection launches that would create or remove its lobbies. In exact mode, check the entire removal set, including managed maps absent from the collection. Apply the same check to manual Create, Start, Stop, Rebuild, and Destroy requests that compete with the policy. Explain which policy must be paused, with a link. Existing operator cancellation remains available; cancelling an autoscaling job pauses the policy so it cannot immediately recreate that intent.
|
||
|
||
Pause the entire policy's decisions while a member is in an active beta, campaign, maintenance operation, host drain, or unresolved recovery. This conservative first-release rule avoids treating temporarily unavailable lobbies as new demand. Those workflows can proceed through their existing gates; autoscaling resumes only after they settle and fresh observations are available. Repeat these checks during Provisioning admission because a conflict can begin after evaluation.
|
||
|
||
## Counting and decisions
|
||
|
||
Build a bounded snapshot per map from current identities, evidence, and operations. Deduplicate a registered lobby and its creation job by identity. Classify each member as occupied and ready, empty and ready, creating, removing, stopped, unknown or unhealthy, or protected by other work. Queued and starting lobbies reserve capacity but are not usable spares.
|
||
|
||
The first release holds new scaling actions whenever an existing member has unknown occupancy, is stopped or unhealthy, or has unresolved lifecycle work. Record the blocker and leave recovery to existing maintenance and operator flows. A confirmed empty map with no registered members can bootstrap from its minimum; it does not need occupancy observations for nonexistent servers.
|
||
|
||
When the snapshot is complete and eligible, calculate:
|
||
|
||
```text
|
||
desired running lobbies = min(maximum, max(minimum, occupied ready lobbies + spare target))
|
||
```
|
||
|
||
This is a bounded target, not a promise that capacity is immediately available. If occupied lobbies consume the maximum, keep them running and report that the spare target cannot be met. Reducing the maximum below the existing count never authorizes removing an occupied lobby.
|
||
|
||
Evaluate in this order:
|
||
|
||
1. Reconcile existing autoscaling intent and any unsettled host commands. Cancel an eligible queued removal if players arrive, evidence becomes unknown, the policy changes, a conflict appears, or removal would breach the minimum or spare target.
|
||
2. Apply mode, ownership, evidence, conflict, failure, and cooldown gates. Never submit a second policy action while a prior one remains unresolved. Count external reservations as well as policy jobs against the maximum.
|
||
3. If the ready count is below the desired count, propose one create using the lowest available lobby identity and any free key from that game. Recheck identity availability, game key availability, and the policy revision atomically when persisting the request. Admission rechecks recognize the pending launch’s own held key. Capacity waiting must reuse this request rather than add another each tick.
|
||
4. If the ready count exceeds the desired count, select one continuously empty eligible lobby whose removal preserves both the minimum and spare target. Prefer the longest-confirmed empty lobby, with lobby number as a stable tie-breaker. Create only mode records this opportunity without executing it.
|
||
5. Otherwise record a stable no-action reason. Deduplicate unchanged decisions so the 15-second timer does not flood history.
|
||
|
||
For example, with minimum 1, maximum 4, and one spare, two occupied lobbies plus one empty lobby need no action. If that empty lobby fills, request a fourth. When only one lobby remains occupied and three are empty, the target becomes two; remove surplus empty lobbies one at a time after their empty duration and action cooldown permit it.
|
||
|
||
Track continuous emptiness against the exact instance generation, using fresh observation timestamps. Re-reading the same sample does not extend the observation history. A positive count, unknown interval, lifecycle transition, or identity replacement resets eligibility. After a controller outage that breaks observation continuity, require a new full empty interval; downtime must not age a lobby into removal.
|
||
|
||
## Removal and cancellation contract
|
||
|
||
Automatic removals always use normal safety gates and never set a safety override. Revalidate policy mode and revision, ownership, membership, remaining spare capacity, conflicts, and occupancy immediately before claiming and dispatching removal. Preserve the agent's fresh local occupancy check immediately before mutation.
|
||
|
||
The existing [cancelRemoval](../../src/lib/server/provisioning/history.ts) transaction competes with the worker while a removal is waiting. It cannot undo removal after the worker has claimed it. General cancellation also cannot withdraw an already delivered host command. The agent can hold an occupied command and retry later, so cancelling only the controller row would leave stale destructive intent.
|
||
|
||
Before enabling Create and destroy, extend the controller/agent contract for autoscaling removals: an occupied or unknown host-local check must invalidate that autoscaling removal attempt rather than retain it for eventual execution. A later attempt requires a new controller evaluation and the full empty interval. Carry policy revision and a bounded authorization expiry, and require renewed controller authorization before an agent retry after delay or restart. Pausing or superseding a policy must invalidate authorization that has not begun mutation. Test these races with the real agent journal behavior; do not assume a controller cancellation implies host acknowledgement.
|
||
|
||
If mutation has already begun, show that cancellation was too late and let the existing operation reconcile its result. Do not release its identity, key slot, or port until removal and cleanup are confirmed. A cancelled, failed, or ambiguous job must not cause a replacement command to overtake an unsettled delivered command.
|
||
|
||
Pause on failed or ambiguous automatic operations and link to Provisioning recovery. Capacity waits are normal waiting states, not failures. After explicit recovery, resume from a new snapshot without replaying the old command. This bounds retries and avoids repeatedly creating failing lobbies with the same key or recipe.
|
||
|
||
## Storage and integration
|
||
|
||
Add shared policy and decision types plus a server-side evaluator under `src/lib/server/autoscaling/`. Separate a deterministic planner from database reads and Provisioning submission so dry run and live modes evaluate the same rules.
|
||
|
||
Persist policies with their revision, mode, settings, and timestamps; adopted instance membership; observation continuity; next-action time; and decisions linked to exact Provisioning operation IDs. Store the relevant evidence source and age, counts, intended action, blocker, and policy revision. Do not include keys or full player rosters in decision history.
|
||
|
||
Use database transactions and uniqueness constraints to enforce one policy per game/map, one unresolved autoscaling intent per policy, and unique lobby allocation across manual launches, collections, and autoscaling. Persist the decision and its operation link together. Process-local in-flight guards prevent overlapping passes, while durable constraints prevent duplicate work after a crash. Add indexes for policy membership, unresolved decisions, and recent history; prune decision history on the existing retention schedule with a proposed 31-day window.
|
||
|
||
Extend Provisioning source types, admission, history, and backlinks with an autoscaling source containing policy and decision identity. Keep creation recipe selection, placement, operation execution, and cleanup inside the existing services. Add no separate host command executor or per-map RCON polling loop.
|
||
|
||
Expose authenticated read and update routes using the existing fleet-management authorization rules. Use a revision check for edits and activation so stale forms cannot overwrite newer policy settings. Off and Dry run submit no host mutations. Moving out of a live mode withdraws cancellable policy work, reports unsettled commands, and releases exclusive ownership only when competing mutations can safely proceed.
|
||
|
||
## Operator experience and measurements
|
||
|
||
Add an Autoscaling view linked from Fleet, Collections, and Metrics. Show each map's mode, occupied and empty ready counts, pending work, minimum, maximum, spare target, game key availability, last evaluation, next eligible action, and current blocker. Detail includes the policy editor, affected lobbies, recent decisions, Provisioning links, and pause/resume controls. Dry run actions must be visibly hypothetical.
|
||
|
||
Activation and edits preview the resulting target and any proposed removals. Pause stops new decisions immediately and shows which queued actions were withdrawn and which delivered operations remain unresolved. Do not claim the fleet was restored when a policy is disabled.
|
||
|
||
Record empty-spare deficit minutes, unknown-status minutes, empty lobby-hours, action frequency, capacity/key blocks, cancellations, failures, and request-to-ready durations. Use Provisioning milestones for queue time and readiness, but measure browser visibility and actual successful joins separately in the pilot: a healthy container or RCON response alone does not establish player joinability.
|
||
|
||
Compare equivalent weekly periods and record game/runtime versions, player-hours, host CPU and memory, and provisioning latency. “Time with players” will naturally rise when idle lobbies leave its denominator; it does not by itself prove increased demand or CPU savings. Dry run estimates are conditional on the observed fleet, not a simulation of how players would redistribute after removal. Use measured creation and join latency to tune spare targets before broad rollout.
|
||
|
||
## Delivery sequence and acceptance
|
||
|
||
Complete and validate each phase in a separate commit before starting the next.
|
||
|
||
1. **Policy model and planner.** Add additive schema changes, identity-aware snapshots, policy validation, decision types, and the pure evaluator. Keep all modes non-mutating until admission integration exists. Acceptance: deterministic fixtures cover the target formula, variant separation, incomplete evidence, game pool exhaustion and reuse, bounds, and continuous empty timing.
|
||
2. **Dry run and visibility.** Register the requester, persist bounded decision history, and add the policy UI/API with dry run enabled. Acceptance: repeated ticks and controller restarts produce understandable decisions without creating Provisioning jobs or reserving identities, ports, or hosts. Begin the two-week observation period on representative maps.
|
||
3. **Ownership and live creation.** Add transactional intent, Provisioning source eligibility for Create/Destroy, collection/manual conflict checks, and Create only mode. Acceptance: no duplicate creation under concurrent ticks or submissions; no growth beyond the maximum; no capacity bypass; blocked creates remain single durable jobs. Pilot one or two maps after reviewing dry run evidence.
|
||
4. **Safe removal.** Implement continuous-empty selection, admission rechecks, queued cancellation, and the agent authorization contract above. Acceptance: positive/unknown occupancy blocks destruction; a held command cannot remove a lobby later without renewed eligibility; policy pause, revision changes, and new protected work invalidate pending intent. Enable Create and destroy only after these checks pass.
|
||
5. **Pilot review and rollout.** Compare CPU and empty lobby-hours alongside spare availability, successful joins, failures, and creation latency. Expand one map at a time only if savings are measurable, ordinary gameplay is preserved, and loss of spare availability stays within the pilot's agreed tolerance. Document the tuned settings and operating procedure.
|
||
|
||
Required behavioral scenarios across these phases:
|
||
|
||
| Scenario | Expected result |
|
||
| ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| Last spare fills, including a high-round partially filled session | One create when eligible; free player slots do not count as spares. |
|
||
| Create is queued, starting, or awaiting verification | Reservation counts once; it is not advertised as ready and no duplicate is queued. |
|
||
| Every eligible host is full or unavailable | Existing job waits; other policies and worker tasks continue. |
|
||
| Game key pool is empty or exhausted | Hold growth with a clear reason; other games remain independent and eligible surplus removal can proceed. |
|
||
| A launch is queued, stopped, failed, or uncertain | Its key stays reserved until confirmed cleanup; no other launch can use that key concurrently. |
|
||
| Occupancy evidence conflicts, expires, or arrives from an old lifecycle | Positive current occupancy wins; invalid evidence cannot establish emptiness. |
|
||
| Players join before claim, after dispatch, or while the agent holds removal | Queued work is withdrawn where possible; agent safety prevents occupied destruction; held autoscaling intent cannot execute later without a fresh decision. |
|
||
| Process restarts before or after intent submission | Existing operation is recovered by durable identity without replay or duplicate admission. |
|
||
| Collection exact match omits a managed map | Preview and submission report the ownership conflict, including stale-preview races. |
|
||
| Beta, campaign, maintenance, host drain, or recovery begins after evaluation | Admission rechecks block scaling and existing work retains its lifecycle ownership. |
|
||
| Operator pauses, changes limits, or cancels a policy job | Unstarted intent is invalidated; unsettled commands stay visible and keep reservations until reconciled. |
|
||
| A name is reused after confirmed deletion | Old observations, cooldown decisions, and commands cannot apply to the replacement generation. |
|
||
|
||
Run targeted Vitest coverage for planner, collections, provisioning, cancellation, telemetry, and routes, plus Python agent tests for the removal contract. Before each implementation phase is committed, run the relevant checks; before live release, run `npm run check`, `npm test`, `npm run test:agent`, and `npm run build`. Verify the UI with keyboard navigation and narrow layouts, including stale evidence, blocked capacity, and partial cancellation states.
|
||
|
||
## Rollback
|
||
|
||
Provide a global pause as well as per-policy pause. It stops new decisions and withdraws unstarted work using the same cancellation contract. Keep the controller and agent running to settle delivered operations and cleanup; do not delete policies or reservations to bypass recovery.
|
||
|
||
After outstanding operations settle, release map ownership. Operators can restore a fixed lineup through Collections using normal capacity and empty-server gates. Turning autoscaling off does not recreate removed lobbies or remove lobbies it created. Retain decision and Provisioning history for review, and keep the additive schema compatible with all policies Off.
|
||
|
||
## Implementation and validation record
|
||
|
||
The implementation adds the pure evaluator and persistent policy model, dry-run requester and UI, exclusive map ownership with capacity-aware creation, and revocable removal authorization. It creates no production policies. The authorization claim is the final cancellation boundary; agents invalidate an occupied or expired attempt and require a new decision rather than retrying it later. The local claim window is five seconds, measured from the start of the authorization request, with another occupancy check before mutation.
|
||
|
||
Validation includes planner and snapshot tests, duplicate-admission and stale-preview tests, collection exact-match conflicts, operator cancellation, authorization expiry and host identity, controller/agent occupancy races, and agent restart recovery. Agent packaging publishes version 1.7.0 with the new capability.
|
||
|
||
The full controller suite reported 1,934 passing tests and two existing failures in `server-detail-api.test.ts` (plugin classification) and `watchdog.test.ts` (an occupied-telemetry RCON expectation). Both failures reproduce independently at the pre-implementation commit `a22461a`. They are not autoscaling regressions. Targeted controller tests, all 228 Python agent tests, type checking, and the production build passed.
|
||
|
||
An isolated Firefox review exercised policy creation, preview, save, keyboard focus, and 1440, 390, and 320 pixel layouts. A narrow-layout heading issue found during review was corrected. The preview used a temporary database and no production host credentials. Production CPU savings, external joinability, spare targets, and acceptable deficit durations still require the two-week observation period and operator-run pilot.
|
||
|
||
Game-wide policy follow-up: selecting every map applies the targets independently to each catalog map and variant. Game-wide policies serialize actions across maps and cannot overlap live map-specific ownership. Existing map policies remain compatible.
|
||
|
||
Game key pool follow-up: keys are shared by all launch paths for a game. Lobby names and key labels are independent. Policy maxima no longer require dedicated key coverage on every map, and policies can be saved before adding keys. Snapshots account for reservations on other maps, planner creation waits when the pool is exhausted, and admission accepts its own reserved key. Keys are released only after confirmed cleanup.
|
||
|
||
Follow-up validation: 40 autoscaling tests and 118 related provisioning, collection, and cancellation tests passed, along with type checking and the production build. The earlier isolated Firefox review covered keyboard navigation and 1440, 390, and 320 pixel layouts. Its per-map key coverage checks described the previous behavior, superseded by game key pools.
|