Document lobby autoscaling implementation plan

docs/lobby-autoscaling-plan.md

@@ -0,0 +1,138 @@
+# Lobby autoscaling implementation plan
+
+Status: Proposed. October 6, 2026. No autoscaling behavior is implemented by this plan.
+
+Introduce an opt-in policy for each game and map variant that keeps a small number of empty, ready lobbies available. When the last spare fills, request another lobby through Provisioning. Remove surplus lobbies only after sustained, confirmed inactivity. This reduces idle hosting while preserving room for new groups, including when existing Zombies sessions have free player slots but are already well into a game.
+
+Start with Create/Destroy on existing native hosts. Run the evaluator in dry run for two weeks, then enable creation for a small pilot before enabling removal. Host acquisition, VM shutdown, Stop/Start pooling, scheduled demand prediction, and waking servers from incoming packets are outside the first release.
+
+## Existing foundations and required extensions
+
+| Foundation | Reuse and required work |
+| --- | --- |
+| Background worker | Add a requester to [pipeline/worker.ts](../src/lib/server/pipeline/worker.ts), which already reconciles background work every 15 seconds independently of Provisioning. Keep host execution inside Provisioning. |
+| Occupancy evidence | Reuse [status evidence](../src/lib/server/fleet/status-evidence.ts) and shared freshness rules. Monitoring runs on a 30-second cadence; accepted evidence currently expires after 90 seconds. Positive occupancy vetoes an empty claim. Unknown is never zero. |
+| Safe creation | Use the durable creation intent and placement path in [provisioning/native.ts](../src/lib/server/provisioning/native.ts). Preserve CPU, memory, port, host eligibility, library, and startup concurrency checks. A blocked create remains queued. |
+| Safe removal | Reuse the controller occupancy checks, instance generation checks, host-local check, and cleanup checkpoints. Add autoscaling-specific admission and cancellation behavior before enabling automatic destruction. |
+| Requester eligibility | [provisioning/worker.ts](../src/lib/server/provisioning/worker.ts) currently gives Create and Destroy separate paths that bypass the runtime requester eligibility check. Autoscaling ownership and policy checks must cover both paths, including delayed execution. A source label alone is insufficient. |
+| Saved keys | [server-keys.ts](../src/lib/server/fleet/server-keys.ts) stores keys by exact lobby name and accepts lobby numbers 1–20. Allocate only unused identities with saved keys; never copy a key to a different identity. |
+| Collections | [collections.ts](../src/lib/server/fleet/collections.ts) already counts queued creations and launches promoted recipes. Exact matching can remove lobbies across the fleet, including maps omitted from the collection. Add ownership checks to preview and submission. |
+| Hosting insights | [metrics-insights.ts](../src/lib/metrics-insights.ts) provides historical suggestions for hosting trials. Use these to help choose policies; the live evaluator needs current empty-lobby evidence and must not execute historical suggestions directly. |
+
+Create already uses the promoted launch recipe and capacity-aware placement. Start retains existing files and has recovery concurrency controls, but would need equivalent CPU and memory admission before becoming an autoscaling strategy. The proposed scope follows option 4 in [the idle CPU research](t5-t6-idle-cpu-research.md).
+
+## Policy and ownership
+
+Use one policy per `(game, mapSlug)`, preserving configured variants such as T6 start locations. Do not group ambiguous legacy records by engine map alone. Resolve or exclude ambiguous identities before activation.
+
+| Setting | Proposed initial behavior |
+| --- | --- |
+| Mode | Off, Dry run, Create only, or Create and destroy. Default Off. |
+| Minimum running lobbies | Default 1; a floor for desired ready capacity, subject to available keys and host capacity. |
+| Maximum lobbies | Required explicit limit, at most 20. Count occupied lobbies, registered identities, reservations, and unresolved creations toward admission. |
+| Empty spare target | Default 1. Count empty ready lobbies, not unfilled player slots. |
+| Empty duration before removal | Default 20 minutes; adjustable after the pilot. |
+| Action cooldown | Default 5 minutes between new policy actions, measured from durable admission. Cancellation and safety checks do not wait for cooldown. |
+| Launch settings | Reuse normal map recipes and automatic placement on eligible active hosts. Use current promoted files when provisioning begins. |
+
+Validate integer counts with `1 <= minimum <= maximum <= 20` and `1 <= spare target <= maximum`. Show when saved-key coverage limits the configured maximum; block live activation until enough usable identities have keys. Subsequent key loss blocks affected growth and does not trigger removal. Keys need not occupy consecutive lobby numbers.
+
+Live activation explicitly hands management of the game's map variant to the policy and adopts its eligible existing native lobbies. Show those lobbies, exclusions, current keys, and the first proposed action in the activation preview. Do not activate across unresolved provisioning, collection launches, or ambiguous identity ownership. Dry run observes without acquiring ownership or reserving names, ports, or capacity.
+
+While a policy is live, reject collection launches that would create or remove its lobbies. In exact mode, check the entire removal set, including managed maps absent from the collection. Apply the same check to manual Create, Start, Stop, Rebuild, and Destroy requests that compete with the policy. Explain which policy must be paused, with a link. Existing operator cancellation remains available; cancelling an autoscaling job pauses the policy so it cannot immediately recreate that intent.
+
+Pause the entire policy's decisions while a member is in an active beta, campaign, maintenance operation, host drain, or unresolved recovery. This conservative first-release rule avoids treating temporarily unavailable lobbies as new demand. Those workflows can proceed through their existing gates; autoscaling resumes only after they settle and fresh observations are available. Repeat these checks during Provisioning admission because a conflict can begin after evaluation.
+
+## Counting and decisions
+
+Build a bounded snapshot per map from current identities, evidence, and operations. Deduplicate a registered lobby and its creation job by identity. Classify each member as occupied and ready, empty and ready, creating, removing, stopped, unknown or unhealthy, or protected by other work. Queued and starting lobbies reserve capacity but are not usable spares.
+
+The first release holds new scaling actions whenever an existing member has unknown occupancy, is stopped or unhealthy, or has unresolved lifecycle work. Record the blocker and leave recovery to existing maintenance and operator flows. A confirmed empty map with no registered members can bootstrap from its minimum; it does not need occupancy observations for nonexistent servers.
+
+When the snapshot is complete and eligible, calculate:
+
+```text
+desired running lobbies = min(maximum, max(minimum, occupied ready lobbies + spare target))
+```
+
+This is a bounded target, not a promise that capacity is immediately available. If occupied lobbies consume the maximum, keep them running and report that the spare target cannot be met. Reducing the maximum below the existing count never authorizes removing an occupied lobby.
+
+Evaluate in this order:
+
+1. Reconcile existing autoscaling intent and any unsettled host commands. Cancel an eligible queued removal if players arrive, evidence becomes unknown, the policy changes, a conflict appears, or removal would breach the minimum or spare target.
+2. Apply mode, ownership, evidence, conflict, failure, and cooldown gates. Never submit a second policy action while a prior one remains unresolved. Count external reservations as well as policy jobs against the maximum.
+3. If the ready count is below the desired count, propose one create using the lowest available keyed lobby identity. Recheck identity availability and the policy revision atomically when persisting the request. Capacity waiting must reuse this request rather than add another each tick.
+4. If the ready count exceeds the desired count, select one continuously empty eligible lobby whose removal preserves both the minimum and spare target. Prefer the longest-confirmed empty lobby, with lobby number as a stable tie-breaker. Create only mode records this opportunity without executing it.
+5. Otherwise record a stable no-action reason. Deduplicate unchanged decisions so the 15-second timer does not flood history.
+
+For example, with minimum 1, maximum 4, and one spare, two occupied lobbies plus one empty lobby need no action. If that empty lobby fills, request a fourth. When only one lobby remains occupied and three are empty, the target becomes two; remove surplus empty lobbies one at a time after their empty duration and action cooldown permit it.
+
+Track continuous emptiness against the exact instance generation, using fresh observation timestamps. Re-reading the same sample does not extend the observation history. A positive count, unknown interval, lifecycle transition, or identity replacement resets eligibility. After a controller outage that breaks observation continuity, require a new full empty interval; downtime must not age a lobby into removal.
+
+## Removal and cancellation contract
+
+Automatic removals always use normal safety gates and never set a safety override. Revalidate policy mode and revision, ownership, membership, remaining spare capacity, conflicts, and occupancy immediately before claiming and dispatching removal. Preserve the agent's fresh local occupancy check immediately before mutation.
+
+The existing [cancelRemoval](../src/lib/server/provisioning/history.ts) transaction competes with the worker while a removal is waiting. It cannot undo removal after the worker has claimed it. General cancellation also cannot withdraw an already delivered host command. The agent can hold an occupied command and retry later, so cancelling only the controller row would leave stale destructive intent.
+
+Before enabling Create and destroy, extend the controller/agent contract for autoscaling removals: an occupied or unknown host-local check must invalidate that autoscaling removal attempt rather than retain it for eventual execution. A later attempt requires a new controller evaluation and the full empty interval. Carry policy revision and a bounded authorization expiry, and require renewed controller authorization before an agent retry after delay or restart. Pausing or superseding a policy must invalidate authorization that has not begun mutation. Test these races with the real agent journal behavior; do not assume a controller cancellation implies host acknowledgement.
+
+If mutation has already begun, show that cancellation was too late and let the existing operation reconcile its result. Do not release its identity, key slot, or port until removal and cleanup are confirmed. A cancelled, failed, or ambiguous job must not cause a replacement command to overtake an unsettled delivered command.
+
+Pause on failed or ambiguous automatic operations and link to Provisioning recovery. Capacity waits are normal waiting states, not failures. After explicit recovery, resume from a new snapshot without replaying the old command. This bounds retries and avoids repeatedly creating failing lobbies with the same key or recipe.
+
+## Storage and integration
+
+Add shared policy and decision types plus a server-side evaluator under `src/lib/server/autoscaling/`. Separate a deterministic planner from database reads and Provisioning submission so dry run and live modes evaluate the same rules.
+
+Persist policies with their revision, mode, settings, and timestamps; adopted instance membership; observation continuity; next-action time; and decisions linked to exact Provisioning operation IDs. Store the relevant evidence source and age, counts, intended action, blocker, and policy revision. Do not include keys or full player rosters in decision history.
+
+Use database transactions and uniqueness constraints to enforce one policy per game/map, one unresolved autoscaling intent per policy, and unique lobby allocation across manual launches, collections, and autoscaling. Persist the decision and its operation link together. Process-local in-flight guards prevent overlapping passes, while durable constraints prevent duplicate work after a crash. Add indexes for policy membership, unresolved decisions, and recent history; prune decision history on the existing retention schedule with a proposed 31-day window.
+
+Extend Provisioning source types, admission, history, and backlinks with an autoscaling source containing policy and decision identity. Keep creation recipe selection, placement, operation execution, and cleanup inside the existing services. Add no separate host command executor or per-map RCON polling loop.
+
+Expose authenticated read and update routes using the existing fleet-management authorization rules. Use a revision check for edits and activation so stale forms cannot overwrite newer policy settings. Off and Dry run submit no host mutations. Moving out of a live mode withdraws cancellable policy work, reports unsettled commands, and releases exclusive ownership only when competing mutations can safely proceed.
+
+## Operator experience and measurements
+
+Add an Autoscaling view linked from Fleet, Collections, and Metrics. Show each map's mode, occupied and empty ready counts, pending work, minimum, maximum, spare target, key coverage, last evaluation, next eligible action, and current blocker. Detail includes the policy editor, affected lobbies, recent decisions, Provisioning links, and pause/resume controls. Dry run actions must be visibly hypothetical.
+
+Activation and edits preview the resulting target and any proposed removals. Pause stops new decisions immediately and shows which queued actions were withdrawn and which delivered operations remain unresolved. Do not claim the fleet was restored when a policy is disabled.
+
+Record empty-spare deficit minutes, unknown-status minutes, empty lobby-hours, action frequency, capacity/key blocks, cancellations, failures, and request-to-ready durations. Use Provisioning milestones for queue time and readiness, but measure browser visibility and actual successful joins separately in the pilot: a healthy container or RCON response alone does not establish player joinability.
+
+Compare equivalent weekly periods and record game/runtime versions, player-hours, host CPU and memory, and provisioning latency. “Time with players” will naturally rise when idle lobbies leave its denominator; it does not by itself prove increased demand or CPU savings. Dry run estimates are conditional on the observed fleet, not a simulation of how players would redistribute after removal. Use measured creation and join latency to tune spare targets before broad rollout.
+
+## Delivery sequence and acceptance
+
+Complete and validate each phase in a separate commit before starting the next.
+
+1. **Policy model and planner.** Add additive schema changes, identity-aware snapshots, policy validation, decision types, and the pure evaluator. Keep all modes non-mutating until admission integration exists. Acceptance: deterministic fixtures cover the target formula, variant separation, incomplete evidence, saved-key gaps, bounds, and continuous empty timing.
+2. **Dry run and visibility.** Register the requester, persist bounded decision history, and add the policy UI/API with dry run enabled. Acceptance: repeated ticks and controller restarts produce understandable decisions without creating Provisioning jobs or reserving identities, ports, or hosts. Begin the two-week observation period on representative maps.
+3. **Ownership and live creation.** Add transactional intent, Provisioning source eligibility for Create/Destroy, collection/manual conflict checks, and Create only mode. Acceptance: no duplicate creation under concurrent ticks or submissions; no growth beyond the maximum; no capacity bypass; blocked creates remain single durable jobs. Pilot one or two maps after reviewing dry run evidence.
+4. **Safe removal.** Implement continuous-empty selection, admission rechecks, queued cancellation, and the agent authorization contract above. Acceptance: positive/unknown occupancy blocks destruction; a held command cannot remove a lobby later without renewed eligibility; policy pause, revision changes, and new protected work invalidate pending intent. Enable Create and destroy only after these checks pass.
+5. **Pilot review and rollout.** Compare CPU and empty lobby-hours alongside spare availability, successful joins, failures, and creation latency. Expand one map at a time only if savings are measurable, ordinary gameplay is preserved, and loss of spare availability stays within the pilot's agreed tolerance. Document the tuned settings and operating procedure.
+
+Required behavioral scenarios across these phases:
+
+| Scenario | Expected result |
+| --- | --- |
+| Last spare fills, including a high-round partially filled session | One create when eligible; free player slots do not count as spares. |
+| Create is queued, starting, or awaiting verification | Reservation counts once; it is not advertised as ready and no duplicate is queued. |
+| Every eligible host is full or unavailable | Existing job waits; other policies and worker tasks continue. |
+| Key is missing, removed, or rejected | Clear blocker or paused failure; no alternate identity receives the same key automatically. |
+| Occupancy evidence conflicts, expires, or arrives from an old lifecycle | Positive current occupancy wins; invalid evidence cannot establish emptiness. |
+| Players join before claim, after dispatch, or while the agent holds removal | Queued work is withdrawn where possible; agent safety prevents occupied destruction; held autoscaling intent cannot execute later without a fresh decision. |
+| Process restarts before or after intent submission | Existing operation is recovered by durable identity without replay or duplicate admission. |
+| Collection exact match omits a managed map | Preview and submission report the ownership conflict, including stale-preview races. |
+| Beta, campaign, maintenance, host drain, or recovery begins after evaluation | Admission rechecks block scaling and existing work retains its lifecycle ownership. |
+| Operator pauses, changes limits, or cancels a policy job | Unstarted intent is invalidated; unsettled commands stay visible and keep reservations until reconciled. |
+| A name is reused after confirmed deletion | Old observations, cooldown decisions, and commands cannot apply to the replacement generation. |
+
+Run targeted Vitest coverage for planner, collections, provisioning, cancellation, telemetry, and routes, plus Python agent tests for the removal contract. Before each implementation phase is committed, run the relevant checks; before live release, run `npm run check`, `npm test`, `npm run test:agent`, and `npm run build`. Verify the UI with keyboard navigation and narrow layouts, including stale evidence, blocked capacity, and partial cancellation states.
+
+## Rollback
+
+Provide a global pause as well as per-policy pause. It stops new decisions and withdraws unstarted work using the same cancellation contract. Keep the controller and agent running to settle delivered operations and cleanup; do not delete policies or reservations to bypass recovery.
+
+After outstanding operations settle, release map ownership. Operators can restore a fixed lineup through Collections using normal capacity and empty-server gates. Turning autoscaling off does not recreate removed lobbies or remove lobbies it created. Retain decision and Provisioning history for review, and keep the additive schema compatible with all policies Off.