Pool Plutonium keys by game for launches and autoscaling

docs/plans/lobby-autoscaling-plan.md

@@ -4,7 +4,7 @@ Status: Approved and implemented October 6, 2026. Controller support and agent 1
 
 The design below preserves the approved scope and subsequent implementation
 record. Use the linked guide for current policy behavior, including game-wide
-policies and saved-key requirements added after the initial design.
+policies and shared game key pools added after the initial design.
 
 Introduce an opt-in policy for each game and map variant that keeps a small number of empty, ready lobbies available. When the last spare fills, request another lobby through Provisioning. Remove surplus lobbies only after sustained, confirmed inactivity. This reduces idle hosting while preserving room for new groups, including when existing Zombies sessions have free player slots but are already well into a game.
 
@@ -19,7 +19,7 @@ Start with Create/Destroy on existing native hosts. Run the evaluator in dry run
 | Safe creation | Use the durable creation intent and placement path in [provisioning/native.ts](../../src/lib/server/provisioning/native.ts). Preserve CPU, memory, port, host eligibility, library, and startup concurrency checks. A blocked create remains queued. |
 | Safe removal | Reuse the controller occupancy checks, instance generation checks, host-local check, and cleanup checkpoints. Add autoscaling-specific admission and cancellation behavior before enabling automatic destruction. |
 | Requester eligibility | [provisioning/worker.ts](../../src/lib/server/provisioning/worker.ts) currently gives Create and Destroy separate paths that bypass the runtime requester eligibility check. Autoscaling ownership and policy checks must cover both paths, including delayed execution. A source label alone is insufficient. |
-| Saved keys | [server-keys.ts](../../src/lib/server/fleet/server-keys.ts) stores keys by exact lobby name and accepts lobby numbers 1–20. Allocate only unused identities with saved keys; never copy a key to a different identity. |
+| Saved keys | [server-keys.ts](../../src/lib/server/fleet/server-keys.ts) stores keys in game pools. Each live or pending launch reserves a key until cleanup confirms it is safe to reuse. Allocate lobby identities numbered 1–20 independently of the key label. |
 | Collections | [collections.ts](../../src/lib/server/fleet/collections.ts) already counts queued creations and launches promoted recipes. Exact matching can remove lobbies across the fleet, including maps omitted from the collection. Add ownership checks to preview and submission. |
 | Hosting insights | [metrics-insights.ts](../../src/lib/metrics-insights.ts) provides historical suggestions for hosting trials. Use these to help choose policies; the live evaluator needs current empty-lobby evidence and must not execute historical suggestions directly. |
 
@@ -39,7 +39,7 @@ Use one policy per `(game, mapSlug)`, preserving configured variants such as T6
 | Action cooldown | Default 5 minutes between new policy actions, measured from durable admission. Cancellation and safety checks do not wait for cooldown. |
 | Launch settings | Reuse normal map recipes and automatic placement on eligible active hosts. Use current promoted files when provisioning begins. |
 
-Validate integer counts with `1 <= minimum <= maximum <= 20` and `1 <= spare target <= maximum`. Show when saved-key coverage limits the configured maximum; block live activation until enough usable identities have keys. Subsequent key loss blocks affected growth and does not trigger removal. Keys need not occupy consecutive lobby numbers.
+Validate integer counts with `1 <= minimum <= maximum <= 20` and `1 <= spare target <= maximum`. Show the game pool’s saved and available counts independently of each map’s maximum. Policies can be saved with fewer keys than the maximum, or with no keys yet. Exhaustion holds growth across every map sharing that game, while eligible surplus removal can still return capacity. Game-wide targets are per map, but game keys are shared across all maps and launch paths.
 
 Live activation explicitly hands management of the game's map variant to the policy and adopts its eligible existing native lobbies. Show those lobbies, exclusions, current keys, and the first proposed action in the activation preview. Do not activate across unresolved provisioning, collection launches, or ambiguous identity ownership. Dry run observes without acquiring ownership or reserving names, ports, or capacity.
 
@@ -65,7 +65,7 @@ Evaluate in this order:
 
 1. Reconcile existing autoscaling intent and any unsettled host commands. Cancel an eligible queued removal if players arrive, evidence becomes unknown, the policy changes, a conflict appears, or removal would breach the minimum or spare target.
 2. Apply mode, ownership, evidence, conflict, failure, and cooldown gates. Never submit a second policy action while a prior one remains unresolved. Count external reservations as well as policy jobs against the maximum.
-3. If the ready count is below the desired count, propose one create using the lowest available keyed lobby identity. Recheck identity availability and the policy revision atomically when persisting the request. Capacity waiting must reuse this request rather than add another each tick.
+3. If the ready count is below the desired count, propose one create using the lowest available lobby identity and any free key from that game. Recheck identity availability, game key availability, and the policy revision atomically when persisting the request. Admission rechecks recognize the pending launch’s own held key. Capacity waiting must reuse this request rather than add another each tick.
 4. If the ready count exceeds the desired count, select one continuously empty eligible lobby whose removal preserves both the minimum and spare target. Prefer the longest-confirmed empty lobby, with lobby number as a stable tie-breaker. Create only mode records this opportunity without executing it.
 5. Otherwise record a stable no-action reason. Deduplicate unchanged decisions so the 15-second timer does not flood history.
 
@@ -99,7 +99,7 @@ Expose authenticated read and update routes using the existing fleet-management
 
 ## Operator experience and measurements
 
-Add an Autoscaling view linked from Fleet, Collections, and Metrics. Show each map's mode, occupied and empty ready counts, pending work, minimum, maximum, spare target, key coverage, last evaluation, next eligible action, and current blocker. Detail includes the policy editor, affected lobbies, recent decisions, Provisioning links, and pause/resume controls. Dry run actions must be visibly hypothetical.
+Add an Autoscaling view linked from Fleet, Collections, and Metrics. Show each map's mode, occupied and empty ready counts, pending work, minimum, maximum, spare target, game key availability, last evaluation, next eligible action, and current blocker. Detail includes the policy editor, affected lobbies, recent decisions, Provisioning links, and pause/resume controls. Dry run actions must be visibly hypothetical.
 
 Activation and edits preview the resulting target and any proposed removals. Pause stops new decisions immediately and shows which queued actions were withdrawn and which delivered operations remain unresolved. Do not claim the fleet was restored when a policy is disabled.
 
@@ -111,7 +111,7 @@ Compare equivalent weekly periods and record game/runtime versions, player-hours
 
 Complete and validate each phase in a separate commit before starting the next.
 
-1. **Policy model and planner.** Add additive schema changes, identity-aware snapshots, policy validation, decision types, and the pure evaluator. Keep all modes non-mutating until admission integration exists. Acceptance: deterministic fixtures cover the target formula, variant separation, incomplete evidence, saved-key gaps, bounds, and continuous empty timing.
+1. **Policy model and planner.** Add additive schema changes, identity-aware snapshots, policy validation, decision types, and the pure evaluator. Keep all modes non-mutating until admission integration exists. Acceptance: deterministic fixtures cover the target formula, variant separation, incomplete evidence, game pool exhaustion and reuse, bounds, and continuous empty timing.
 2. **Dry run and visibility.** Register the requester, persist bounded decision history, and add the policy UI/API with dry run enabled. Acceptance: repeated ticks and controller restarts produce understandable decisions without creating Provisioning jobs or reserving identities, ports, or hosts. Begin the two-week observation period on representative maps.
 3. **Ownership and live creation.** Add transactional intent, Provisioning source eligibility for Create/Destroy, collection/manual conflict checks, and Create only mode. Acceptance: no duplicate creation under concurrent ticks or submissions; no growth beyond the maximum; no capacity bypass; blocked creates remain single durable jobs. Pilot one or two maps after reviewing dry run evidence.
 4. **Safe removal.** Implement continuous-empty selection, admission rechecks, queued cancellation, and the agent authorization contract above. Acceptance: positive/unknown occupancy blocks destruction; a held command cannot remove a lobby later without renewed eligibility; policy pause, revision changes, and new protected work invalidate pending intent. Enable Create and destroy only after these checks pass.
@@ -124,7 +124,8 @@ Required behavioral scenarios across these phases:
 | Last spare fills, including a high-round partially filled session | One create when eligible; free player slots do not count as spares. |
 | Create is queued, starting, or awaiting verification | Reservation counts once; it is not advertised as ready and no duplicate is queued. |
 | Every eligible host is full or unavailable | Existing job waits; other policies and worker tasks continue. |
-| Key is missing, removed, or rejected | Clear blocker or paused failure; no alternate identity receives the same key automatically. |
+| Game key pool is empty or exhausted | Hold growth with a clear reason; other games remain independent and eligible surplus removal can proceed. |
+| A launch is queued, stopped, failed, or uncertain | Its key stays reserved until confirmed cleanup; no other launch can use that key concurrently. |
 | Occupancy evidence conflicts, expires, or arrives from an old lifecycle | Positive current occupancy wins; invalid evidence cannot establish emptiness. |
 | Players join before claim, after dispatch, or while the agent holds removal | Queued work is withdrawn where possible; agent safety prevents occupied destruction; held autoscaling intent cannot execute later without a fresh decision. |
 | Process restarts before or after intent submission | Existing operation is recovered by durable identity without replay or duplicate admission. |
@@ -153,6 +154,6 @@ An isolated Firefox review exercised policy creation, preview, save, keyboard fo
 
 Game-wide policy follow-up: selecting every map applies the targets independently to each catalog map and variant. Game-wide policies serialize actions across maps and cannot overlap live map-specific ownership. Existing map policies remain compatible.
 
-Key coverage follow-up: creation in every mode requires sufficient saved keys for the maximum on each selected map. Coverage is rechecked on save, including edits to active or dry-run policies. Turning an existing policy Off remains available after key loss.
+Game key pool follow-up: keys are shared by all launch paths for a game. Lobby names and key labels are independent. Policy maxima no longer require dedicated key coverage on every map, and policies can be saved before adding keys. Snapshots account for reservations on other maps, planner creation waits when the pool is exhausted, and admission accepts its own reserved key. Keys are released only after confirmed cleanup.
 
-Follow-up validation: 40 autoscaling tests and 118 related provisioning, collection, and cancellation tests passed, along with type checking and the production build. An isolated Firefox review verified that missing per-map keys disable Save, complete coverage permits game-wide dry-run creation, and keyboard navigation and 1440, 390, and 320 pixel layouts work.
+Follow-up validation: 40 autoscaling tests and 118 related provisioning, collection, and cancellation tests passed, along with type checking and the production build. The earlier isolated Firefox review covered keyboard navigation and 1440, 390, and 320 pixel layouts. Its per-map key coverage checks described the previous behavior, superseded by game key pools.