Pool Plutonium keys by game for launches and autoscaling

This commit is contained in:
2026-10-07 11:24:53 -04:00
parent 56e5a2d06d
commit ec45371478
47 changed files with 956 additions and 305 deletions
+1 -1
View File
@@ -42,7 +42,7 @@ guessable token. An unset setup token holds setup closed.
| Create users, change roles, disable accounts and reset passwords | No | No | Yes |
| Own password and sign out | Yes | Yes | Yes |
Operators can launch using server keys already saved by an administrator. They
Operators can launch using available game-pool keys already saved by an administrator. They
cannot save keys indirectly through a launch request. An operator's content
permissions include executable scripts and plugins; assign that role only to
people trusted to operate the game fleet.
+4 -4
View File
@@ -8,7 +8,7 @@ Autoscaling is opt-in. Installing the feature creates no policies and changes no
Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Dry run, with a minimum of one running lobby, one empty spare, a 20-minute empty interval, and a five-minute action cooldown. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum.
Choose **Preview changes** to inspect the existing lobbies, saved key coverage, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving.
Choose **Preview changes** to inspect the existing lobbies, game key availability, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving.
| Mode | Behavior |
| --- | --- |
@@ -17,15 +17,15 @@ Choose **Preview changes** to inspect the existing lobbies, saved key coverage,
| Create only | Adds a lobby when eligible. Surplus removal remains advisory. |
| Create and destroy | Adds lobbies and removes surplus lobbies after continuous empty evidence and all removal checks pass. |
Creating a policy in any mode, including Dry run or Off, requires saved Plutonium keys for enough exact lobby identities to support the configured maximum on every selected map. Preview lists each shortage, and Save rechecks coverage transactionally. Non-Off edits also require coverage; an existing policy can always be turned Off if keys have been removed. Only supported lobby identities numbered 1–20 with nonblank saved keys count. Keys may be nonconsecutive: if only lobby 3 and lobby 8 are keyed, those identities can be used. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host.
Plutonium keys belong to a game pool shared by every map, manual launch, collection, beta, and autoscaling policy for that game. An eligible creation uses the lowest available lobby number from 1–20 and reserves any free game key atomically. Key labels do not select a map or lobby. The preview shows saved and available game keys; policy maxima can exceed pool size, and a policy can be saved before any keys are added. An empty or exhausted pool holds new growth with a clear reason; it does not prevent surplus removal. Queued launches reserve their key, stopped instances retain it, and confirmed cleanup returns it to the pool. Failed or uncertain launches keep their reservation until safely reconciled. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host.
The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies share one cooldown and one outstanding operation across their maps, prioritizing additions before removals. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins.
## Ownership and blockers
A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected; dry-run policies may overlap. Game-wide rules include every catalog map and require key coverage for the maximum on each map. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish.
A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected; dry-run policies may overlap. Game-wide rules include every catalog map and apply each map’s targets independently while sharing the game’s key pool. Pool capacity is a game-wide limit; targets across maps are not guaranteed when all keys are in use. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish.
Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and missing-key waits.
Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and game key waits.
Current positive occupancy from either telemetry or RCON vetoes an empty claim. Empty intervals reset when players arrive, evidence becomes unknown, the instance or lifecycle changes, or observation continuity breaks. Controller downtime does not count toward the empty duration. Failed or uncertain automatic operations pause further actions until an operator inspects their Provisioning recovery and reviews the policy again.
+23 -11
View File
@@ -32,8 +32,9 @@ It fills deficits without changing existing servers' files.
3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection).
Choose **Fill missing servers** or **Match collection exactly**, then review
the server IDs and estimated capacity. Native host ports are assigned later.
4. Lobbies with a saved key show **Saved key ready** and reuse it automatically.
Enter a Plutonium key only for lobbies missing one. Update saved keys in **Keys**.
4. Launches covered by unused keys show **Game key available**. Each game’s
available keys are counted once across the full lineup. Add an unused Plutonium
key for each remaining launch, or manage the game pools in **Keys**.
5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact
mode) and follow the collection progress page.
If every target is already met, choose **Done**; no operation is created.
@@ -48,7 +49,8 @@ servers keep their containers, files, ports, and configuration without a restart
The option is chosen per launch; opening another preview defaults to fill mode.
Review the removal list and check its confirmation before applying. Removed
servers lose their containers, files, and registrations; saved keys remain.
servers lose their containers, files, and registrations; their keys return to the
game pool only after confirmed removal.
Unstarted creations are cancelled without booting a container. Running servers
use the normal removal queue and require a fresh zero-player RCON response.
Occupied or unreachable servers wait. Already pending removals are reused.
@@ -59,6 +61,11 @@ map, port, or pending-removal change since an exact preview requires a new revie
Servers added after submission are not included. Applying the same preview again
returns the same collection without repeating operations.
Keys assigned to pending removals stay reserved until cleanup is confirmed. An
exact collection that replaces running servers still needs enough currently
available game keys for every new launch. If the pool is full, complete the
removals first, then refresh the preview to use the released keys.
Native launches can be queued even when the current port range is full. Removed
servers retain their ports until cleanup completes; provisioning assigns newly
available ports as each creation starts. A failed or cancelled removal can leave
@@ -94,11 +101,11 @@ is a warning: provisioning can wait for it. The preview does not reserve resourc
Apply validates the reviewed configuration, lobby IDs, keys, and fleet changes, saves all jobs in
one database transaction, and returns the progress page ID. Native jobs initially
have no host or port; each job performs its capacity checks asynchronously during
provisioning. If a server lacks a valid key or its reviewed identity changed,
provisioning. If a server lacks an available game key or its reviewed identity changed,
no jobs or key changes from that submission are committed. Legacy submissions also
revalidate their reviewed ports. Refreshing
the preview retains typed keys only for unchanged server IDs still missing a saved
key, and resets removal confirmation.
the preview retains typed keys for launches in the same game that still need
keys, and resets removal confirmation.
Submitting the same preview again returns the same collection, including after
a lost response or an application restart. Reusing the collection with a new
@@ -138,13 +145,15 @@ remain visible in Provisioning activity and history.
2. Choose the current promoted setup and select a game/map, or select a collection.
3. Choose **Preview launch**. Review the image reference, configuration sources,
managed files and versions, selected map, lobby, port, and prerequisite checks.
4. Enter the Plutonium key for the displayed server ID, or reuse its saved key.
4. Use an available key from the game pool, or enter an unused key for that game.
5. Choose **Launch server** and follow the operation to verified health.
A preview lasts 30 minutes. It neither reserves capacity nor saves a key. If
another launch takes the lobby, submission stops before saving the key;
refresh the preview and review the newly assigned ID. Refreshing a preview clears
the typed key so it cannot accidentally be saved against a different lobby.
refresh the preview and review the newly assigned ID. A typed key can follow
the new lobby within the same game. Pool availability is checked again at
submission, and keys are reserved atomically so concurrent launches cannot share
a credential.
Host selection, game-file versions, runtime configuration and lobby availability
are checked before queueing. Provisioning assigns host capacity and physical
@@ -205,7 +214,8 @@ and drift reconciliation using the server's game/map scopes.
Container tags can resolve to newer images; use a digest for a fixed image.
Without a promoted baseline, Plutainer seeds its own defaults on first boot,
which are not captured. Host, network, game-file volumes, destination directories,
and RCON/Plutonium credentials use the environment and key store at launch time.
and RCON/Plutonium credentials use the environment and assigned game-pool key at
launch time.
The current main config filename and launch settings apply to new deployments.
Recipes do not include the separately stored Plutonium or integration credentials.
@@ -249,7 +259,9 @@ there while preserving query parameters.
where mode is `fill` (default) or `exact`. Returns `mode`, `retainedServers`,
`removals` (name/reason), creation `servers`, and configured `launchConcurrency`.
- `POST /api/deployment-collections`: atomically queue `{ previewId, keys? }`,
where `keys` maps exact preview server IDs to optional replacement keys.
where `keys` maps preview server IDs to optional unused keys added to their
game's pool and assigned during launch. Omit entries covered by available
pool keys; these IDs do not permanently bind a key to a map or lobby.
A satisfied preview returns `{ ok: true, collectionId: null }` with HTTP 200,
without creating a collection, saving keys, or kicking the worker.
- `GET /api/deployment-collections`: list collection progress summaries with
+7 -2
View File
@@ -10,7 +10,7 @@ HTTPS. Running game containers continue through controller or agent outages.
- A controller machine with Docker Compose and an HTTPS reverse proxy.
- Debian 12 or 13 amd64 game hosts with storage for game files and instances.
- Your complete T5 or T6 game folders, including the DLC you intend to host.
- Plutonium server keys for the exact lobby identities you will launch.
- Plutonium server keys for each game, with one unused key per new concurrent lobby.
The controller does not need access to a Docker socket, game-host mounts, or
inbound RCON. Host agents handle local game commands and file operations.
@@ -67,13 +67,18 @@ base game files; each instance has its own writable data. See
## Launch your first server
1. Save the exact Plutonium server keys under **Administration → Keys**.
1. Save Plutonium server keys in the T5 or T6 pool under **Administration → Keys**.
2. Create a server, or build a reusable map lineup in **Fleet → Collections**.
3. Review the launch and apply it. Provisioning queues the work, then assigns a
host and port when capacity is available.
4. Follow the operation until readiness checks pass, then verify that players
can find and join the server.
Keys are shared across maps within their game. A launch assigns one available key
until the server is removed; stopped servers keep their keys. Optional key labels
can record the hostname chosen in Plutonium without binding the key to a map or
lobby. Electric Cherry does not change the hostname registered with Plutonium.
New launches use the latest verified game files and promoted configuration,
scripts, and plugins. Uploading newer game files does not restart existing
servers. Disruptive operations normally wait for a fresh empty lobby or a
+2 -1
View File
@@ -33,7 +33,8 @@ the agent owns local execution and its durable operation journal.
before launching.
Apply saves the requested lobby IDs. Provisioning selects a host and atomically
reserves UDP/TCP ports when capacity is available.
Saved Plutonium server keys remain associated with their exact server names.
An available Plutonium key from the game pool is reserved for each launch.
Its assignment survives stops and rebuilds and ends after confirmed removal.
A fresh destination host downloads centrally stored game files without manual
copying or per-container setup. EC does not supply the game content itself.
+20 -3
View File
@@ -30,9 +30,10 @@ or rebuild on its selected servers.
Rebuild requires confirmation because **no instance data is preserved**. Its fresh
files come from the latest promoted setup when creation begins, including changes
promoted while waiting. The independently saved key in **Keys** must exist before
teardown. Blueprint definitions, promoted artifacts, saved keys, and historical
activity belong to Electric Cherry, and are not instance files.
promoted while waiting. Its assigned key in **Keys** must exist before teardown
and stays reserved throughout the rebuild. Blueprint definitions, promoted
artifacts, saved keys, and historical activity belong to Electric Cherry, and are
not instance files.
Rebuild reserves the original ID and port throughout teardown and creation, even
when a lower lobby is available. It shares the existing creation queue and startup
@@ -58,6 +59,22 @@ maintenance. A successful agent receipt is followed by actual container and game
slot and are not automatically replayed. Stop requires an explicit stopped container state; an
unreachable host agent is not evidence that a stop succeeded.
### Game key pools
Each create reserves one unused Plutonium key for its game before the launch is
queued. T5 and T6 have independent pools; any map or lobby within that game can
use a free key. Stopping a server keeps the reservation, and a rebuild keeps the
same key. Confirmed removal releases it for another launch. Failed or ambiguous
creation keeps the assignment while cleanup may still be required.
An unstarted creation cancelled before dispatch releases its key immediately.
Administrators can add and remove available keys in **Administration → Keys**;
assigned keys cannot be removed. Existing keys are migrated into their game
pool, retaining assignments for existing servers and pending work. If a legacy
rebuild has an uncertain result and used a different key, both possible keys
remain reserved until that server is destroyed. Labels are for identification
only and do not change the hostname registered with Plutonium.
### Player checks and administrator overrides
Disruptive operations require a fresh zero-player result from RCON or accepted
+10 -9
View File
@@ -4,7 +4,7 @@ Status: Approved and implemented October 6, 2026. Controller support and agent 1
The design below preserves the approved scope and subsequent implementation
record. Use the linked guide for current policy behavior, including game-wide
policies and saved-key requirements added after the initial design.
policies and shared game key pools added after the initial design.
Introduce an opt-in policy for each game and map variant that keeps a small number of empty, ready lobbies available. When the last spare fills, request another lobby through Provisioning. Remove surplus lobbies only after sustained, confirmed inactivity. This reduces idle hosting while preserving room for new groups, including when existing Zombies sessions have free player slots but are already well into a game.
@@ -19,7 +19,7 @@ Start with Create/Destroy on existing native hosts. Run the evaluator in dry run
| Safe creation | Use the durable creation intent and placement path in [provisioning/native.ts](../../src/lib/server/provisioning/native.ts). Preserve CPU, memory, port, host eligibility, library, and startup concurrency checks. A blocked create remains queued. |
| Safe removal | Reuse the controller occupancy checks, instance generation checks, host-local check, and cleanup checkpoints. Add autoscaling-specific admission and cancellation behavior before enabling automatic destruction. |
| Requester eligibility | [provisioning/worker.ts](../../src/lib/server/provisioning/worker.ts) currently gives Create and Destroy separate paths that bypass the runtime requester eligibility check. Autoscaling ownership and policy checks must cover both paths, including delayed execution. A source label alone is insufficient. |
| Saved keys | [server-keys.ts](../../src/lib/server/fleet/server-keys.ts) stores keys by exact lobby name and accepts lobby numbers 1–20. Allocate only unused identities with saved keys; never copy a key to a different identity. |
| Saved keys | [server-keys.ts](../../src/lib/server/fleet/server-keys.ts) stores keys in game pools. Each live or pending launch reserves a key until cleanup confirms it is safe to reuse. Allocate lobby identities numbered 1–20 independently of the key label. |
| Collections | [collections.ts](../../src/lib/server/fleet/collections.ts) already counts queued creations and launches promoted recipes. Exact matching can remove lobbies across the fleet, including maps omitted from the collection. Add ownership checks to preview and submission. |
| Hosting insights | [metrics-insights.ts](../../src/lib/metrics-insights.ts) provides historical suggestions for hosting trials. Use these to help choose policies; the live evaluator needs current empty-lobby evidence and must not execute historical suggestions directly. |
@@ -39,7 +39,7 @@ Use one policy per `(game, mapSlug)`, preserving configured variants such as T6
| Action cooldown | Default 5 minutes between new policy actions, measured from durable admission. Cancellation and safety checks do not wait for cooldown. |
| Launch settings | Reuse normal map recipes and automatic placement on eligible active hosts. Use current promoted files when provisioning begins. |
Validate integer counts with `1 <= minimum <= maximum <= 20` and `1 <= spare target <= maximum`. Show when saved-key coverage limits the configured maximum; block live activation until enough usable identities have keys. Subsequent key loss blocks affected growth and does not trigger removal. Keys need not occupy consecutive lobby numbers.
Validate integer counts with `1 <= minimum <= maximum <= 20` and `1 <= spare target <= maximum`. Show the game pool’s saved and available counts independently of each map’s maximum. Policies can be saved with fewer keys than the maximum, or with no keys yet. Exhaustion holds growth across every map sharing that game, while eligible surplus removal can still return capacity. Game-wide targets are per map, but game keys are shared across all maps and launch paths.
Live activation explicitly hands management of the game's map variant to the policy and adopts its eligible existing native lobbies. Show those lobbies, exclusions, current keys, and the first proposed action in the activation preview. Do not activate across unresolved provisioning, collection launches, or ambiguous identity ownership. Dry run observes without acquiring ownership or reserving names, ports, or capacity.
@@ -65,7 +65,7 @@ Evaluate in this order:
1. Reconcile existing autoscaling intent and any unsettled host commands. Cancel an eligible queued removal if players arrive, evidence becomes unknown, the policy changes, a conflict appears, or removal would breach the minimum or spare target.
2. Apply mode, ownership, evidence, conflict, failure, and cooldown gates. Never submit a second policy action while a prior one remains unresolved. Count external reservations as well as policy jobs against the maximum.
3. If the ready count is below the desired count, propose one create using the lowest available keyed lobby identity. Recheck identity availability and the policy revision atomically when persisting the request. Capacity waiting must reuse this request rather than add another each tick.
3. If the ready count is below the desired count, propose one create using the lowest available lobby identity and any free key from that game. Recheck identity availability, game key availability, and the policy revision atomically when persisting the request. Admission rechecks recognize the pending launch’s own held key. Capacity waiting must reuse this request rather than add another each tick.
4. If the ready count exceeds the desired count, select one continuously empty eligible lobby whose removal preserves both the minimum and spare target. Prefer the longest-confirmed empty lobby, with lobby number as a stable tie-breaker. Create only mode records this opportunity without executing it.
5. Otherwise record a stable no-action reason. Deduplicate unchanged decisions so the 15-second timer does not flood history.
@@ -99,7 +99,7 @@ Expose authenticated read and update routes using the existing fleet-management
## Operator experience and measurements
Add an Autoscaling view linked from Fleet, Collections, and Metrics. Show each map's mode, occupied and empty ready counts, pending work, minimum, maximum, spare target, key coverage, last evaluation, next eligible action, and current blocker. Detail includes the policy editor, affected lobbies, recent decisions, Provisioning links, and pause/resume controls. Dry run actions must be visibly hypothetical.
Add an Autoscaling view linked from Fleet, Collections, and Metrics. Show each map's mode, occupied and empty ready counts, pending work, minimum, maximum, spare target, game key availability, last evaluation, next eligible action, and current blocker. Detail includes the policy editor, affected lobbies, recent decisions, Provisioning links, and pause/resume controls. Dry run actions must be visibly hypothetical.
Activation and edits preview the resulting target and any proposed removals. Pause stops new decisions immediately and shows which queued actions were withdrawn and which delivered operations remain unresolved. Do not claim the fleet was restored when a policy is disabled.
@@ -111,7 +111,7 @@ Compare equivalent weekly periods and record game/runtime versions, player-hours
Complete and validate each phase in a separate commit before starting the next.
1. **Policy model and planner.** Add additive schema changes, identity-aware snapshots, policy validation, decision types, and the pure evaluator. Keep all modes non-mutating until admission integration exists. Acceptance: deterministic fixtures cover the target formula, variant separation, incomplete evidence, saved-key gaps, bounds, and continuous empty timing.
1. **Policy model and planner.** Add additive schema changes, identity-aware snapshots, policy validation, decision types, and the pure evaluator. Keep all modes non-mutating until admission integration exists. Acceptance: deterministic fixtures cover the target formula, variant separation, incomplete evidence, game pool exhaustion and reuse, bounds, and continuous empty timing.
2. **Dry run and visibility.** Register the requester, persist bounded decision history, and add the policy UI/API with dry run enabled. Acceptance: repeated ticks and controller restarts produce understandable decisions without creating Provisioning jobs or reserving identities, ports, or hosts. Begin the two-week observation period on representative maps.
3. **Ownership and live creation.** Add transactional intent, Provisioning source eligibility for Create/Destroy, collection/manual conflict checks, and Create only mode. Acceptance: no duplicate creation under concurrent ticks or submissions; no growth beyond the maximum; no capacity bypass; blocked creates remain single durable jobs. Pilot one or two maps after reviewing dry run evidence.
4. **Safe removal.** Implement continuous-empty selection, admission rechecks, queued cancellation, and the agent authorization contract above. Acceptance: positive/unknown occupancy blocks destruction; a held command cannot remove a lobby later without renewed eligibility; policy pause, revision changes, and new protected work invalidate pending intent. Enable Create and destroy only after these checks pass.
@@ -124,7 +124,8 @@ Required behavioral scenarios across these phases:
| Last spare fills, including a high-round partially filled session | One create when eligible; free player slots do not count as spares. |
| Create is queued, starting, or awaiting verification | Reservation counts once; it is not advertised as ready and no duplicate is queued. |
| Every eligible host is full or unavailable | Existing job waits; other policies and worker tasks continue. |
| Key is missing, removed, or rejected | Clear blocker or paused failure; no alternate identity receives the same key automatically. |
| Game key pool is empty or exhausted | Hold growth with a clear reason; other games remain independent and eligible surplus removal can proceed. |
| A launch is queued, stopped, failed, or uncertain | Its key stays reserved until confirmed cleanup; no other launch can use that key concurrently. |
| Occupancy evidence conflicts, expires, or arrives from an old lifecycle | Positive current occupancy wins; invalid evidence cannot establish emptiness. |
| Players join before claim, after dispatch, or while the agent holds removal | Queued work is withdrawn where possible; agent safety prevents occupied destruction; held autoscaling intent cannot execute later without a fresh decision. |
| Process restarts before or after intent submission | Existing operation is recovered by durable identity without replay or duplicate admission. |
@@ -153,6 +154,6 @@ An isolated Firefox review exercised policy creation, preview, save, keyboard fo
Game-wide policy follow-up: selecting every map applies the targets independently to each catalog map and variant. Game-wide policies serialize actions across maps and cannot overlap live map-specific ownership. Existing map policies remain compatible.
Key coverage follow-up: creation in every mode requires sufficient saved keys for the maximum on each selected map. Coverage is rechecked on save, including edits to active or dry-run policies. Turning an existing policy Off remains available after key loss.
Game key pool follow-up: keys are shared by all launch paths for a game. Lobby names and key labels are independent. Policy maxima no longer require dedicated key coverage on every map, and policies can be saved before adding keys. Snapshots account for reservations on other maps, planner creation waits when the pool is exhausted, and admission accepts its own reserved key. Keys are released only after confirmed cleanup.
Follow-up validation: 40 autoscaling tests and 118 related provisioning, collection, and cancellation tests passed, along with type checking and the production build. An isolated Firefox review verified that missing per-map keys disable Save, complete coverage permits game-wide dry-run creation, and keyboard navigation and 1440, 390, and 320 pixel layouts work.
Follow-up validation: 40 autoscaling tests and 118 related provisioning, collection, and cancellation tests passed, along with type checking and the production build. The earlier isolated Firefox review covered keyboard navigation and 1440, 390, and 320 pixel layouts. Its per-map key coverage checks described the previous behavior, superseded by game key pools.