Pool Plutonium keys by game for launches and autoscaling

docs/guides/authentication.md

@@ -42,7 +42,7 @@ guessable token. An unset setup token holds setup closed.
 | Create users, change roles, disable accounts and reset passwords | No | No | Yes |
 | Own password and sign out | Yes | Yes | Yes |
 
-Operators can launch using server keys already saved by an administrator. They
+Operators can launch using available game-pool keys already saved by an administrator. They
 cannot save keys indirectly through a launch request. An operator's content
 permissions include executable scripts and plugins; assign that role only to
 people trusted to operate the game fleet.

docs/guides/autoscaling.md

@@ -8,7 +8,7 @@ Autoscaling is opt-in. Installing the feature creates no policies and changes no
 
 Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Dry run, with a minimum of one running lobby, one empty spare, a 20-minute empty interval, and a five-minute action cooldown. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum.
 
-Choose **Preview changes** to inspect the existing lobbies, saved key coverage, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving.
+Choose **Preview changes** to inspect the existing lobbies, game key availability, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving.
 
 | Mode | Behavior |
 | --- | --- |
@@ -17,15 +17,15 @@ Choose **Preview changes** to inspect the existing lobbies, saved key coverage,
 | Create only | Adds a lobby when eligible. Surplus removal remains advisory. |
 | Create and destroy | Adds lobbies and removes surplus lobbies after continuous empty evidence and all removal checks pass. |
 
-Creating a policy in any mode, including Dry run or Off, requires saved Plutonium keys for enough exact lobby identities to support the configured maximum on every selected map. Preview lists each shortage, and Save rechecks coverage transactionally. Non-Off edits also require coverage; an existing policy can always be turned Off if keys have been removed. Only supported lobby identities numbered 1–20 with nonblank saved keys count. Keys may be nonconsecutive: if only lobby 3 and lobby 8 are keyed, those identities can be used. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host.
+Plutonium keys belong to a game pool shared by every map, manual launch, collection, beta, and autoscaling policy for that game. An eligible creation uses the lowest available lobby number from 1–20 and reserves any free game key atomically. Key labels do not select a map or lobby. The preview shows saved and available game keys; policy maxima can exceed pool size, and a policy can be saved before any keys are added. An empty or exhausted pool holds new growth with a clear reason; it does not prevent surplus removal. Queued launches reserve their key, stopped instances retain it, and confirmed cleanup returns it to the pool. Failed or uncertain launches keep their reservation until safely reconciled. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host.
 
 The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies share one cooldown and one outstanding operation across their maps, prioritizing additions before removals. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins.
 
 ## Ownership and blockers
 
-A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected; dry-run policies may overlap. Game-wide rules include every catalog map and require key coverage for the maximum on each map. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish.
+A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected; dry-run policies may overlap. Game-wide rules include every catalog map and apply each map’s targets independently while sharing the game’s key pool. Pool capacity is a game-wide limit; targets across maps are not guaranteed when all keys are in use. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish.
 
-Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and missing-key waits.
+Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and game key waits.
 
 Current positive occupancy from either telemetry or RCON vetoes an empty claim. Empty intervals reset when players arrive, evidence becomes unknown, the instance or lifecycle changes, or observation continuity breaks. Controller downtime does not count toward the empty duration. Failed or uncertain automatic operations pause further actions until an operator inspects their Provisioning recovery and reviews the policy again.
 

docs/guides/collections.md

@@ -32,8 +32,9 @@ It fills deficits without changing existing servers' files.
 3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection).
    Choose **Fill missing servers** or **Match collection exactly**, then review
    the server IDs and estimated capacity. Native host ports are assigned later.
-4. Lobbies with a saved key show **Saved key ready** and reuse it automatically.
-   Enter a Plutonium key only for lobbies missing one. Update saved keys in **Keys**.
+4. Launches covered by unused keys show **Game key available**. Each game’s
+   available keys are counted once across the full lineup. Add an unused Plutonium
+   key for each remaining launch, or manage the game pools in **Keys**.
 5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact
    mode) and follow the collection progress page.
    If every target is already met, choose **Done**; no operation is created.
@@ -48,7 +49,8 @@ servers keep their containers, files, ports, and configuration without a restart
 The option is chosen per launch; opening another preview defaults to fill mode.
 
 Review the removal list and check its confirmation before applying. Removed
-servers lose their containers, files, and registrations; saved keys remain.
+servers lose their containers, files, and registrations; their keys return to the
+game pool only after confirmed removal.
 Unstarted creations are cancelled without booting a container. Running servers
 use the normal removal queue and require a fresh zero-player RCON response.
 Occupied or unreachable servers wait. Already pending removals are reused.
@@ -59,6 +61,11 @@ map, port, or pending-removal change since an exact preview requires a new revie
 Servers added after submission are not included. Applying the same preview again
 returns the same collection without repeating operations.
 
+Keys assigned to pending removals stay reserved until cleanup is confirmed. An
+exact collection that replaces running servers still needs enough currently
+available game keys for every new launch. If the pool is full, complete the
+removals first, then refresh the preview to use the released keys.
+
 Native launches can be queued even when the current port range is full. Removed
 servers retain their ports until cleanup completes; provisioning assigns newly
 available ports as each creation starts. A failed or cancelled removal can leave
@@ -94,11 +101,11 @@ is a warning: provisioning can wait for it. The preview does not reserve resourc
 Apply validates the reviewed configuration, lobby IDs, keys, and fleet changes, saves all jobs in
 one database transaction, and returns the progress page ID. Native jobs initially
 have no host or port; each job performs its capacity checks asynchronously during
-provisioning. If a server lacks a valid key or its reviewed identity changed,
+provisioning. If a server lacks an available game key or its reviewed identity changed,
 no jobs or key changes from that submission are committed. Legacy submissions also
 revalidate their reviewed ports. Refreshing
-the preview retains typed keys only for unchanged server IDs still missing a saved
-key, and resets removal confirmation.
+the preview retains typed keys for launches in the same game that still need
+keys, and resets removal confirmation.
 
 Submitting the same preview again returns the same collection, including after
 a lost response or an application restart. Reusing the collection with a new
@@ -138,13 +145,15 @@ remain visible in Provisioning activity and history.
 2. Choose the current promoted setup and select a game/map, or select a collection.
 3. Choose **Preview launch**. Review the image reference, configuration sources,
    managed files and versions, selected map, lobby, port, and prerequisite checks.
-4. Enter the Plutonium key for the displayed server ID, or reuse its saved key.
+4. Use an available key from the game pool, or enter an unused key for that game.
 5. Choose **Launch server** and follow the operation to verified health.
 
 A preview lasts 30 minutes. It neither reserves capacity nor saves a key. If
 another launch takes the lobby, submission stops before saving the key;
-refresh the preview and review the newly assigned ID. Refreshing a preview clears
-the typed key so it cannot accidentally be saved against a different lobby.
+refresh the preview and review the newly assigned ID. A typed key can follow
+the new lobby within the same game. Pool availability is checked again at
+submission, and keys are reserved atomically so concurrent launches cannot share
+a credential.
 
 Host selection, game-file versions, runtime configuration and lobby availability
 are checked before queueing. Provisioning assigns host capacity and physical
@@ -205,7 +214,8 @@ and drift reconciliation using the server's game/map scopes.
 Container tags can resolve to newer images; use a digest for a fixed image.
 Without a promoted baseline, Plutainer seeds its own defaults on first boot,
 which are not captured. Host, network, game-file volumes, destination directories,
-and RCON/Plutonium credentials use the environment and key store at launch time.
+and RCON/Plutonium credentials use the environment and assigned game-pool key at
+launch time.
 The current main config filename and launch settings apply to new deployments.
 
 Recipes do not include the separately stored Plutonium or integration credentials.
@@ -249,7 +259,9 @@ there while preserving query parameters.
   where mode is `fill` (default) or `exact`. Returns `mode`, `retainedServers`,
   `removals` (name/reason), creation `servers`, and configured `launchConcurrency`.
 - `POST /api/deployment-collections`: atomically queue `{ previewId, keys? }`,
-  where `keys` maps exact preview server IDs to optional replacement keys.
+  where `keys` maps preview server IDs to optional unused keys added to their
+  game's pool and assigned during launch. Omit entries covered by available
+  pool keys; these IDs do not permanently bind a key to a map or lobby.
   A satisfied preview returns `{ ok: true, collectionId: null }` with HTTP 200,
   without creating a collection, saving keys, or kicking the worker.
 - `GET /api/deployment-collections`: list collection progress summaries with

docs/guides/getting-started.md

@@ -10,7 +10,7 @@ HTTPS. Running game containers continue through controller or agent outages.
 - A controller machine with Docker Compose and an HTTPS reverse proxy.
 - Debian 12 or 13 amd64 game hosts with storage for game files and instances.
 - Your complete T5 or T6 game folders, including the DLC you intend to host.
-- Plutonium server keys for the exact lobby identities you will launch.
+- Plutonium server keys for each game, with one unused key per new concurrent lobby.
 
 The controller does not need access to a Docker socket, game-host mounts, or
 inbound RCON. Host agents handle local game commands and file operations.
@@ -67,13 +67,18 @@ base game files; each instance has its own writable data. See
 
 ## Launch your first server
 
-1. Save the exact Plutonium server keys under **Administration → Keys**.
+1. Save Plutonium server keys in the T5 or T6 pool under **Administration → Keys**.
 2. Create a server, or build a reusable map lineup in **Fleet → Collections**.
 3. Review the launch and apply it. Provisioning queues the work, then assigns a
    host and port when capacity is available.
 4. Follow the operation until readiness checks pass, then verify that players
    can find and join the server.
 
+Keys are shared across maps within their game. A launch assigns one available key
+until the server is removed; stopped servers keep their keys. Optional key labels
+can record the hostname chosen in Plutonium without binding the key to a map or
+lobby. Electric Cherry does not change the hostname registered with Plutonium.
+
 New launches use the latest verified game files and promoted configuration,
 scripts, and plugins. Uploading newer game files does not restart existing
 servers. Disruptive operations normally wait for a fresh empty lobby or a

docs/guides/native-hosts.md

@@ -33,7 +33,8 @@ the agent owns local execution and its durable operation journal.
    before launching.
    Apply saves the requested lobby IDs. Provisioning selects a host and atomically
    reserves UDP/TCP ports when capacity is available.
-   Saved Plutonium server keys remain associated with their exact server names.
+   An available Plutonium key from the game pool is reserved for each launch.
+   Its assignment survives stops and rebuilds and ends after confirmed removal.
 
 A fresh destination host downloads centrally stored game files without manual
 copying or per-container setup. EC does not supply the game content itself.

docs/guides/provisioning.md

@@ -30,9 +30,10 @@ or rebuild on its selected servers.
 
 Rebuild requires confirmation because **no instance data is preserved**. Its fresh
 files come from the latest promoted setup when creation begins, including changes
-promoted while waiting. The independently saved key in **Keys** must exist before
-teardown. Blueprint definitions, promoted artifacts, saved keys, and historical
-activity belong to Electric Cherry, and are not instance files.
+promoted while waiting. Its assigned key in **Keys** must exist before teardown
+and stays reserved throughout the rebuild. Blueprint definitions, promoted
+artifacts, saved keys, and historical activity belong to Electric Cherry, and are
+not instance files.
 
 Rebuild reserves the original ID and port throughout teardown and creation, even
 when a lower lobby is available. It shares the existing creation queue and startup
@@ -58,6 +59,22 @@ maintenance. A successful agent receipt is followed by actual container and game
 slot and are not automatically replayed. Stop requires an explicit stopped container state; an
 unreachable host agent is not evidence that a stop succeeded.
 
+### Game key pools
+
+Each create reserves one unused Plutonium key for its game before the launch is
+queued. T5 and T6 have independent pools; any map or lobby within that game can
+use a free key. Stopping a server keeps the reservation, and a rebuild keeps the
+same key. Confirmed removal releases it for another launch. Failed or ambiguous
+creation keeps the assignment while cleanup may still be required.
+
+An unstarted creation cancelled before dispatch releases its key immediately.
+Administrators can add and remove available keys in **Administration → Keys**;
+assigned keys cannot be removed. Existing keys are migrated into their game
+pool, retaining assignments for existing servers and pending work. If a legacy
+rebuild has an uncertain result and used a different key, both possible keys
+remain reserved until that server is destroyed. Labels are for identification
+only and do not change the hostname registered with Plutonium.
+
 ### Player checks and administrator overrides
 
 Disruptive operations require a fresh zero-player result from RCON or accepted