Pool Plutonium keys by game for launches and autoscaling

This commit is contained in:
2026-10-07 11:24:53 -04:00
parent 56e5a2d06d
commit ec45371478
47 changed files with 956 additions and 305 deletions
+1 -1
View File
@@ -42,7 +42,7 @@ guessable token. An unset setup token holds setup closed.
| Create users, change roles, disable accounts and reset passwords | No | No | Yes | | Create users, change roles, disable accounts and reset passwords | No | No | Yes |
| Own password and sign out | Yes | Yes | Yes | | Own password and sign out | Yes | Yes | Yes |
Operators can launch using server keys already saved by an administrator. They Operators can launch using available game-pool keys already saved by an administrator. They
cannot save keys indirectly through a launch request. An operator's content cannot save keys indirectly through a launch request. An operator's content
permissions include executable scripts and plugins; assign that role only to permissions include executable scripts and plugins; assign that role only to
people trusted to operate the game fleet. people trusted to operate the game fleet.
+4 -4
View File
@@ -8,7 +8,7 @@ Autoscaling is opt-in. Installing the feature creates no policies and changes no
Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Dry run, with a minimum of one running lobby, one empty spare, a 20-minute empty interval, and a five-minute action cooldown. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum. Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Dry run, with a minimum of one running lobby, one empty spare, a 20-minute empty interval, and a five-minute action cooldown. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum.
Choose **Preview changes** to inspect the existing lobbies, saved key coverage, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving. Choose **Preview changes** to inspect the existing lobbies, game key availability, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving.
| Mode | Behavior | | Mode | Behavior |
| --- | --- | | --- | --- |
@@ -17,15 +17,15 @@ Choose **Preview changes** to inspect the existing lobbies, saved key coverage,
| Create only | Adds a lobby when eligible. Surplus removal remains advisory. | | Create only | Adds a lobby when eligible. Surplus removal remains advisory. |
| Create and destroy | Adds lobbies and removes surplus lobbies after continuous empty evidence and all removal checks pass. | | Create and destroy | Adds lobbies and removes surplus lobbies after continuous empty evidence and all removal checks pass. |
Creating a policy in any mode, including Dry run or Off, requires saved Plutonium keys for enough exact lobby identities to support the configured maximum on every selected map. Preview lists each shortage, and Save rechecks coverage transactionally. Non-Off edits also require coverage; an existing policy can always be turned Off if keys have been removed. Only supported lobby identities numbered 1–20 with nonblank saved keys count. Keys may be nonconsecutive: if only lobby 3 and lobby 8 are keyed, those identities can be used. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host. Plutonium keys belong to a game pool shared by every map, manual launch, collection, beta, and autoscaling policy for that game. An eligible creation uses the lowest available lobby number from 1–20 and reserves any free game key atomically. Key labels do not select a map or lobby. The preview shows saved and available game keys; policy maxima can exceed pool size, and a policy can be saved before any keys are added. An empty or exhausted pool holds new growth with a clear reason; it does not prevent surplus removal. Queued launches reserve their key, stopped instances retain it, and confirmed cleanup returns it to the pool. Failed or uncertain launches keep their reservation until safely reconciled. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host.
The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies share one cooldown and one outstanding operation across their maps, prioritizing additions before removals. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins. The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies share one cooldown and one outstanding operation across their maps, prioritizing additions before removals. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins.
## Ownership and blockers ## Ownership and blockers
A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected; dry-run policies may overlap. Game-wide rules include every catalog map and require key coverage for the maximum on each map. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish. A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected; dry-run policies may overlap. Game-wide rules include every catalog map and apply each map’s targets independently while sharing the game’s key pool. Pool capacity is a game-wide limit; targets across maps are not guaranteed when all keys are in use. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish.
Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and missing-key waits. Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and game key waits.
Current positive occupancy from either telemetry or RCON vetoes an empty claim. Empty intervals reset when players arrive, evidence becomes unknown, the instance or lifecycle changes, or observation continuity breaks. Controller downtime does not count toward the empty duration. Failed or uncertain automatic operations pause further actions until an operator inspects their Provisioning recovery and reviews the policy again. Current positive occupancy from either telemetry or RCON vetoes an empty claim. Empty intervals reset when players arrive, evidence becomes unknown, the instance or lifecycle changes, or observation continuity breaks. Controller downtime does not count toward the empty duration. Failed or uncertain automatic operations pause further actions until an operator inspects their Provisioning recovery and reviews the policy again.
+23 -11
View File
@@ -32,8 +32,9 @@ It fills deficits without changing existing servers' files.
3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection). 3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection).
Choose **Fill missing servers** or **Match collection exactly**, then review Choose **Fill missing servers** or **Match collection exactly**, then review
the server IDs and estimated capacity. Native host ports are assigned later. the server IDs and estimated capacity. Native host ports are assigned later.
4. Lobbies with a saved key show **Saved key ready** and reuse it automatically. 4. Launches covered by unused keys show **Game key available**. Each game’s
Enter a Plutonium key only for lobbies missing one. Update saved keys in **Keys**. available keys are counted once across the full lineup. Add an unused Plutonium
key for each remaining launch, or manage the game pools in **Keys**.
5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact 5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact
mode) and follow the collection progress page. mode) and follow the collection progress page.
If every target is already met, choose **Done**; no operation is created. If every target is already met, choose **Done**; no operation is created.
@@ -48,7 +49,8 @@ servers keep their containers, files, ports, and configuration without a restart
The option is chosen per launch; opening another preview defaults to fill mode. The option is chosen per launch; opening another preview defaults to fill mode.
Review the removal list and check its confirmation before applying. Removed Review the removal list and check its confirmation before applying. Removed
servers lose their containers, files, and registrations; saved keys remain. servers lose their containers, files, and registrations; their keys return to the
game pool only after confirmed removal.
Unstarted creations are cancelled without booting a container. Running servers Unstarted creations are cancelled without booting a container. Running servers
use the normal removal queue and require a fresh zero-player RCON response. use the normal removal queue and require a fresh zero-player RCON response.
Occupied or unreachable servers wait. Already pending removals are reused. Occupied or unreachable servers wait. Already pending removals are reused.
@@ -59,6 +61,11 @@ map, port, or pending-removal change since an exact preview requires a new revie
Servers added after submission are not included. Applying the same preview again Servers added after submission are not included. Applying the same preview again
returns the same collection without repeating operations. returns the same collection without repeating operations.
Keys assigned to pending removals stay reserved until cleanup is confirmed. An
exact collection that replaces running servers still needs enough currently
available game keys for every new launch. If the pool is full, complete the
removals first, then refresh the preview to use the released keys.
Native launches can be queued even when the current port range is full. Removed Native launches can be queued even when the current port range is full. Removed
servers retain their ports until cleanup completes; provisioning assigns newly servers retain their ports until cleanup completes; provisioning assigns newly
available ports as each creation starts. A failed or cancelled removal can leave available ports as each creation starts. A failed or cancelled removal can leave
@@ -94,11 +101,11 @@ is a warning: provisioning can wait for it. The preview does not reserve resourc
Apply validates the reviewed configuration, lobby IDs, keys, and fleet changes, saves all jobs in Apply validates the reviewed configuration, lobby IDs, keys, and fleet changes, saves all jobs in
one database transaction, and returns the progress page ID. Native jobs initially one database transaction, and returns the progress page ID. Native jobs initially
have no host or port; each job performs its capacity checks asynchronously during have no host or port; each job performs its capacity checks asynchronously during
provisioning. If a server lacks a valid key or its reviewed identity changed, provisioning. If a server lacks an available game key or its reviewed identity changed,
no jobs or key changes from that submission are committed. Legacy submissions also no jobs or key changes from that submission are committed. Legacy submissions also
revalidate their reviewed ports. Refreshing revalidate their reviewed ports. Refreshing
the preview retains typed keys only for unchanged server IDs still missing a saved the preview retains typed keys for launches in the same game that still need
key, and resets removal confirmation. keys, and resets removal confirmation.
Submitting the same preview again returns the same collection, including after Submitting the same preview again returns the same collection, including after
a lost response or an application restart. Reusing the collection with a new a lost response or an application restart. Reusing the collection with a new
@@ -138,13 +145,15 @@ remain visible in Provisioning activity and history.
2. Choose the current promoted setup and select a game/map, or select a collection. 2. Choose the current promoted setup and select a game/map, or select a collection.
3. Choose **Preview launch**. Review the image reference, configuration sources, 3. Choose **Preview launch**. Review the image reference, configuration sources,
managed files and versions, selected map, lobby, port, and prerequisite checks. managed files and versions, selected map, lobby, port, and prerequisite checks.
4. Enter the Plutonium key for the displayed server ID, or reuse its saved key. 4. Use an available key from the game pool, or enter an unused key for that game.
5. Choose **Launch server** and follow the operation to verified health. 5. Choose **Launch server** and follow the operation to verified health.
A preview lasts 30 minutes. It neither reserves capacity nor saves a key. If A preview lasts 30 minutes. It neither reserves capacity nor saves a key. If
another launch takes the lobby, submission stops before saving the key; another launch takes the lobby, submission stops before saving the key;
refresh the preview and review the newly assigned ID. Refreshing a preview clears refresh the preview and review the newly assigned ID. A typed key can follow
the typed key so it cannot accidentally be saved against a different lobby. the new lobby within the same game. Pool availability is checked again at
submission, and keys are reserved atomically so concurrent launches cannot share
a credential.
Host selection, game-file versions, runtime configuration and lobby availability Host selection, game-file versions, runtime configuration and lobby availability
are checked before queueing. Provisioning assigns host capacity and physical are checked before queueing. Provisioning assigns host capacity and physical
@@ -205,7 +214,8 @@ and drift reconciliation using the server's game/map scopes.
Container tags can resolve to newer images; use a digest for a fixed image. Container tags can resolve to newer images; use a digest for a fixed image.
Without a promoted baseline, Plutainer seeds its own defaults on first boot, Without a promoted baseline, Plutainer seeds its own defaults on first boot,
which are not captured. Host, network, game-file volumes, destination directories, which are not captured. Host, network, game-file volumes, destination directories,
and RCON/Plutonium credentials use the environment and key store at launch time. and RCON/Plutonium credentials use the environment and assigned game-pool key at
launch time.
The current main config filename and launch settings apply to new deployments. The current main config filename and launch settings apply to new deployments.
Recipes do not include the separately stored Plutonium or integration credentials. Recipes do not include the separately stored Plutonium or integration credentials.
@@ -249,7 +259,9 @@ there while preserving query parameters.
where mode is `fill` (default) or `exact`. Returns `mode`, `retainedServers`, where mode is `fill` (default) or `exact`. Returns `mode`, `retainedServers`,
`removals` (name/reason), creation `servers`, and configured `launchConcurrency`. `removals` (name/reason), creation `servers`, and configured `launchConcurrency`.
- `POST /api/deployment-collections`: atomically queue `{ previewId, keys? }`, - `POST /api/deployment-collections`: atomically queue `{ previewId, keys? }`,
where `keys` maps exact preview server IDs to optional replacement keys. where `keys` maps preview server IDs to optional unused keys added to their
game's pool and assigned during launch. Omit entries covered by available
pool keys; these IDs do not permanently bind a key to a map or lobby.
A satisfied preview returns `{ ok: true, collectionId: null }` with HTTP 200, A satisfied preview returns `{ ok: true, collectionId: null }` with HTTP 200,
without creating a collection, saving keys, or kicking the worker. without creating a collection, saving keys, or kicking the worker.
- `GET /api/deployment-collections`: list collection progress summaries with - `GET /api/deployment-collections`: list collection progress summaries with
+7 -2
View File
@@ -10,7 +10,7 @@ HTTPS. Running game containers continue through controller or agent outages.
- A controller machine with Docker Compose and an HTTPS reverse proxy. - A controller machine with Docker Compose and an HTTPS reverse proxy.
- Debian 12 or 13 amd64 game hosts with storage for game files and instances. - Debian 12 or 13 amd64 game hosts with storage for game files and instances.
- Your complete T5 or T6 game folders, including the DLC you intend to host. - Your complete T5 or T6 game folders, including the DLC you intend to host.
- Plutonium server keys for the exact lobby identities you will launch. - Plutonium server keys for each game, with one unused key per new concurrent lobby.
The controller does not need access to a Docker socket, game-host mounts, or The controller does not need access to a Docker socket, game-host mounts, or
inbound RCON. Host agents handle local game commands and file operations. inbound RCON. Host agents handle local game commands and file operations.
@@ -67,13 +67,18 @@ base game files; each instance has its own writable data. See
## Launch your first server ## Launch your first server
1. Save the exact Plutonium server keys under **Administration → Keys**. 1. Save Plutonium server keys in the T5 or T6 pool under **Administration → Keys**.
2. Create a server, or build a reusable map lineup in **Fleet → Collections**. 2. Create a server, or build a reusable map lineup in **Fleet → Collections**.
3. Review the launch and apply it. Provisioning queues the work, then assigns a 3. Review the launch and apply it. Provisioning queues the work, then assigns a
host and port when capacity is available. host and port when capacity is available.
4. Follow the operation until readiness checks pass, then verify that players 4. Follow the operation until readiness checks pass, then verify that players
can find and join the server. can find and join the server.
Keys are shared across maps within their game. A launch assigns one available key
until the server is removed; stopped servers keep their keys. Optional key labels
can record the hostname chosen in Plutonium without binding the key to a map or
lobby. Electric Cherry does not change the hostname registered with Plutonium.
New launches use the latest verified game files and promoted configuration, New launches use the latest verified game files and promoted configuration,
scripts, and plugins. Uploading newer game files does not restart existing scripts, and plugins. Uploading newer game files does not restart existing
servers. Disruptive operations normally wait for a fresh empty lobby or a servers. Disruptive operations normally wait for a fresh empty lobby or a
+2 -1
View File
@@ -33,7 +33,8 @@ the agent owns local execution and its durable operation journal.
before launching. before launching.
Apply saves the requested lobby IDs. Provisioning selects a host and atomically Apply saves the requested lobby IDs. Provisioning selects a host and atomically
reserves UDP/TCP ports when capacity is available. reserves UDP/TCP ports when capacity is available.
Saved Plutonium server keys remain associated with their exact server names. An available Plutonium key from the game pool is reserved for each launch.
Its assignment survives stops and rebuilds and ends after confirmed removal.
A fresh destination host downloads centrally stored game files without manual A fresh destination host downloads centrally stored game files without manual
copying or per-container setup. EC does not supply the game content itself. copying or per-container setup. EC does not supply the game content itself.
+20 -3
View File
@@ -30,9 +30,10 @@ or rebuild on its selected servers.
Rebuild requires confirmation because **no instance data is preserved**. Its fresh Rebuild requires confirmation because **no instance data is preserved**. Its fresh
files come from the latest promoted setup when creation begins, including changes files come from the latest promoted setup when creation begins, including changes
promoted while waiting. The independently saved key in **Keys** must exist before promoted while waiting. Its assigned key in **Keys** must exist before teardown
teardown. Blueprint definitions, promoted artifacts, saved keys, and historical and stays reserved throughout the rebuild. Blueprint definitions, promoted
activity belong to Electric Cherry, and are not instance files. artifacts, saved keys, and historical activity belong to Electric Cherry, and are
not instance files.
Rebuild reserves the original ID and port throughout teardown and creation, even Rebuild reserves the original ID and port throughout teardown and creation, even
when a lower lobby is available. It shares the existing creation queue and startup when a lower lobby is available. It shares the existing creation queue and startup
@@ -58,6 +59,22 @@ maintenance. A successful agent receipt is followed by actual container and game
slot and are not automatically replayed. Stop requires an explicit stopped container state; an slot and are not automatically replayed. Stop requires an explicit stopped container state; an
unreachable host agent is not evidence that a stop succeeded. unreachable host agent is not evidence that a stop succeeded.
### Game key pools
Each create reserves one unused Plutonium key for its game before the launch is
queued. T5 and T6 have independent pools; any map or lobby within that game can
use a free key. Stopping a server keeps the reservation, and a rebuild keeps the
same key. Confirmed removal releases it for another launch. Failed or ambiguous
creation keeps the assignment while cleanup may still be required.
An unstarted creation cancelled before dispatch releases its key immediately.
Administrators can add and remove available keys in **Administration → Keys**;
assigned keys cannot be removed. Existing keys are migrated into their game
pool, retaining assignments for existing servers and pending work. If a legacy
rebuild has an uncertain result and used a different key, both possible keys
remain reserved until that server is destroyed. Labels are for identification
only and do not change the hostname registered with Plutonium.
### Player checks and administrator overrides ### Player checks and administrator overrides
Disruptive operations require a fresh zero-player result from RCON or accepted Disruptive operations require a fresh zero-player result from RCON or accepted
+10 -9
View File
@@ -4,7 +4,7 @@ Status: Approved and implemented October 6, 2026. Controller support and agent 1
The design below preserves the approved scope and subsequent implementation The design below preserves the approved scope and subsequent implementation
record. Use the linked guide for current policy behavior, including game-wide record. Use the linked guide for current policy behavior, including game-wide
policies and saved-key requirements added after the initial design. policies and shared game key pools added after the initial design.
Introduce an opt-in policy for each game and map variant that keeps a small number of empty, ready lobbies available. When the last spare fills, request another lobby through Provisioning. Remove surplus lobbies only after sustained, confirmed inactivity. This reduces idle hosting while preserving room for new groups, including when existing Zombies sessions have free player slots but are already well into a game. Introduce an opt-in policy for each game and map variant that keeps a small number of empty, ready lobbies available. When the last spare fills, request another lobby through Provisioning. Remove surplus lobbies only after sustained, confirmed inactivity. This reduces idle hosting while preserving room for new groups, including when existing Zombies sessions have free player slots but are already well into a game.
@@ -19,7 +19,7 @@ Start with Create/Destroy on existing native hosts. Run the evaluator in dry run
| Safe creation | Use the durable creation intent and placement path in [provisioning/native.ts](../../src/lib/server/provisioning/native.ts). Preserve CPU, memory, port, host eligibility, library, and startup concurrency checks. A blocked create remains queued. | | Safe creation | Use the durable creation intent and placement path in [provisioning/native.ts](../../src/lib/server/provisioning/native.ts). Preserve CPU, memory, port, host eligibility, library, and startup concurrency checks. A blocked create remains queued. |
| Safe removal | Reuse the controller occupancy checks, instance generation checks, host-local check, and cleanup checkpoints. Add autoscaling-specific admission and cancellation behavior before enabling automatic destruction. | | Safe removal | Reuse the controller occupancy checks, instance generation checks, host-local check, and cleanup checkpoints. Add autoscaling-specific admission and cancellation behavior before enabling automatic destruction. |
| Requester eligibility | [provisioning/worker.ts](../../src/lib/server/provisioning/worker.ts) currently gives Create and Destroy separate paths that bypass the runtime requester eligibility check. Autoscaling ownership and policy checks must cover both paths, including delayed execution. A source label alone is insufficient. | | Requester eligibility | [provisioning/worker.ts](../../src/lib/server/provisioning/worker.ts) currently gives Create and Destroy separate paths that bypass the runtime requester eligibility check. Autoscaling ownership and policy checks must cover both paths, including delayed execution. A source label alone is insufficient. |
| Saved keys | [server-keys.ts](../../src/lib/server/fleet/server-keys.ts) stores keys by exact lobby name and accepts lobby numbers 1–20. Allocate only unused identities with saved keys; never copy a key to a different identity. | | Saved keys | [server-keys.ts](../../src/lib/server/fleet/server-keys.ts) stores keys in game pools. Each live or pending launch reserves a key until cleanup confirms it is safe to reuse. Allocate lobby identities numbered 1–20 independently of the key label. |
| Collections | [collections.ts](../../src/lib/server/fleet/collections.ts) already counts queued creations and launches promoted recipes. Exact matching can remove lobbies across the fleet, including maps omitted from the collection. Add ownership checks to preview and submission. | | Collections | [collections.ts](../../src/lib/server/fleet/collections.ts) already counts queued creations and launches promoted recipes. Exact matching can remove lobbies across the fleet, including maps omitted from the collection. Add ownership checks to preview and submission. |
| Hosting insights | [metrics-insights.ts](../../src/lib/metrics-insights.ts) provides historical suggestions for hosting trials. Use these to help choose policies; the live evaluator needs current empty-lobby evidence and must not execute historical suggestions directly. | | Hosting insights | [metrics-insights.ts](../../src/lib/metrics-insights.ts) provides historical suggestions for hosting trials. Use these to help choose policies; the live evaluator needs current empty-lobby evidence and must not execute historical suggestions directly. |
@@ -39,7 +39,7 @@ Use one policy per `(game, mapSlug)`, preserving configured variants such as T6
| Action cooldown | Default 5 minutes between new policy actions, measured from durable admission. Cancellation and safety checks do not wait for cooldown. | | Action cooldown | Default 5 minutes between new policy actions, measured from durable admission. Cancellation and safety checks do not wait for cooldown. |
| Launch settings | Reuse normal map recipes and automatic placement on eligible active hosts. Use current promoted files when provisioning begins. | | Launch settings | Reuse normal map recipes and automatic placement on eligible active hosts. Use current promoted files when provisioning begins. |
Validate integer counts with `1 <= minimum <= maximum <= 20` and `1 <= spare target <= maximum`. Show when saved-key coverage limits the configured maximum; block live activation until enough usable identities have keys. Subsequent key loss blocks affected growth and does not trigger removal. Keys need not occupy consecutive lobby numbers. Validate integer counts with `1 <= minimum <= maximum <= 20` and `1 <= spare target <= maximum`. Show the game pool’s saved and available counts independently of each map’s maximum. Policies can be saved with fewer keys than the maximum, or with no keys yet. Exhaustion holds growth across every map sharing that game, while eligible surplus removal can still return capacity. Game-wide targets are per map, but game keys are shared across all maps and launch paths.
Live activation explicitly hands management of the game's map variant to the policy and adopts its eligible existing native lobbies. Show those lobbies, exclusions, current keys, and the first proposed action in the activation preview. Do not activate across unresolved provisioning, collection launches, or ambiguous identity ownership. Dry run observes without acquiring ownership or reserving names, ports, or capacity. Live activation explicitly hands management of the game's map variant to the policy and adopts its eligible existing native lobbies. Show those lobbies, exclusions, current keys, and the first proposed action in the activation preview. Do not activate across unresolved provisioning, collection launches, or ambiguous identity ownership. Dry run observes without acquiring ownership or reserving names, ports, or capacity.
@@ -65,7 +65,7 @@ Evaluate in this order:
1. Reconcile existing autoscaling intent and any unsettled host commands. Cancel an eligible queued removal if players arrive, evidence becomes unknown, the policy changes, a conflict appears, or removal would breach the minimum or spare target. 1. Reconcile existing autoscaling intent and any unsettled host commands. Cancel an eligible queued removal if players arrive, evidence becomes unknown, the policy changes, a conflict appears, or removal would breach the minimum or spare target.
2. Apply mode, ownership, evidence, conflict, failure, and cooldown gates. Never submit a second policy action while a prior one remains unresolved. Count external reservations as well as policy jobs against the maximum. 2. Apply mode, ownership, evidence, conflict, failure, and cooldown gates. Never submit a second policy action while a prior one remains unresolved. Count external reservations as well as policy jobs against the maximum.
3. If the ready count is below the desired count, propose one create using the lowest available keyed lobby identity. Recheck identity availability and the policy revision atomically when persisting the request. Capacity waiting must reuse this request rather than add another each tick. 3. If the ready count is below the desired count, propose one create using the lowest available lobby identity and any free key from that game. Recheck identity availability, game key availability, and the policy revision atomically when persisting the request. Admission rechecks recognize the pending launch’s own held key. Capacity waiting must reuse this request rather than add another each tick.
4. If the ready count exceeds the desired count, select one continuously empty eligible lobby whose removal preserves both the minimum and spare target. Prefer the longest-confirmed empty lobby, with lobby number as a stable tie-breaker. Create only mode records this opportunity without executing it. 4. If the ready count exceeds the desired count, select one continuously empty eligible lobby whose removal preserves both the minimum and spare target. Prefer the longest-confirmed empty lobby, with lobby number as a stable tie-breaker. Create only mode records this opportunity without executing it.
5. Otherwise record a stable no-action reason. Deduplicate unchanged decisions so the 15-second timer does not flood history. 5. Otherwise record a stable no-action reason. Deduplicate unchanged decisions so the 15-second timer does not flood history.
@@ -99,7 +99,7 @@ Expose authenticated read and update routes using the existing fleet-management
## Operator experience and measurements ## Operator experience and measurements
Add an Autoscaling view linked from Fleet, Collections, and Metrics. Show each map's mode, occupied and empty ready counts, pending work, minimum, maximum, spare target, key coverage, last evaluation, next eligible action, and current blocker. Detail includes the policy editor, affected lobbies, recent decisions, Provisioning links, and pause/resume controls. Dry run actions must be visibly hypothetical. Add an Autoscaling view linked from Fleet, Collections, and Metrics. Show each map's mode, occupied and empty ready counts, pending work, minimum, maximum, spare target, game key availability, last evaluation, next eligible action, and current blocker. Detail includes the policy editor, affected lobbies, recent decisions, Provisioning links, and pause/resume controls. Dry run actions must be visibly hypothetical.
Activation and edits preview the resulting target and any proposed removals. Pause stops new decisions immediately and shows which queued actions were withdrawn and which delivered operations remain unresolved. Do not claim the fleet was restored when a policy is disabled. Activation and edits preview the resulting target and any proposed removals. Pause stops new decisions immediately and shows which queued actions were withdrawn and which delivered operations remain unresolved. Do not claim the fleet was restored when a policy is disabled.
@@ -111,7 +111,7 @@ Compare equivalent weekly periods and record game/runtime versions, player-hours
Complete and validate each phase in a separate commit before starting the next. Complete and validate each phase in a separate commit before starting the next.
1. **Policy model and planner.** Add additive schema changes, identity-aware snapshots, policy validation, decision types, and the pure evaluator. Keep all modes non-mutating until admission integration exists. Acceptance: deterministic fixtures cover the target formula, variant separation, incomplete evidence, saved-key gaps, bounds, and continuous empty timing. 1. **Policy model and planner.** Add additive schema changes, identity-aware snapshots, policy validation, decision types, and the pure evaluator. Keep all modes non-mutating until admission integration exists. Acceptance: deterministic fixtures cover the target formula, variant separation, incomplete evidence, game pool exhaustion and reuse, bounds, and continuous empty timing.
2. **Dry run and visibility.** Register the requester, persist bounded decision history, and add the policy UI/API with dry run enabled. Acceptance: repeated ticks and controller restarts produce understandable decisions without creating Provisioning jobs or reserving identities, ports, or hosts. Begin the two-week observation period on representative maps. 2. **Dry run and visibility.** Register the requester, persist bounded decision history, and add the policy UI/API with dry run enabled. Acceptance: repeated ticks and controller restarts produce understandable decisions without creating Provisioning jobs or reserving identities, ports, or hosts. Begin the two-week observation period on representative maps.
3. **Ownership and live creation.** Add transactional intent, Provisioning source eligibility for Create/Destroy, collection/manual conflict checks, and Create only mode. Acceptance: no duplicate creation under concurrent ticks or submissions; no growth beyond the maximum; no capacity bypass; blocked creates remain single durable jobs. Pilot one or two maps after reviewing dry run evidence. 3. **Ownership and live creation.** Add transactional intent, Provisioning source eligibility for Create/Destroy, collection/manual conflict checks, and Create only mode. Acceptance: no duplicate creation under concurrent ticks or submissions; no growth beyond the maximum; no capacity bypass; blocked creates remain single durable jobs. Pilot one or two maps after reviewing dry run evidence.
4. **Safe removal.** Implement continuous-empty selection, admission rechecks, queued cancellation, and the agent authorization contract above. Acceptance: positive/unknown occupancy blocks destruction; a held command cannot remove a lobby later without renewed eligibility; policy pause, revision changes, and new protected work invalidate pending intent. Enable Create and destroy only after these checks pass. 4. **Safe removal.** Implement continuous-empty selection, admission rechecks, queued cancellation, and the agent authorization contract above. Acceptance: positive/unknown occupancy blocks destruction; a held command cannot remove a lobby later without renewed eligibility; policy pause, revision changes, and new protected work invalidate pending intent. Enable Create and destroy only after these checks pass.
@@ -124,7 +124,8 @@ Required behavioral scenarios across these phases:
| Last spare fills, including a high-round partially filled session | One create when eligible; free player slots do not count as spares. | | Last spare fills, including a high-round partially filled session | One create when eligible; free player slots do not count as spares. |
| Create is queued, starting, or awaiting verification | Reservation counts once; it is not advertised as ready and no duplicate is queued. | | Create is queued, starting, or awaiting verification | Reservation counts once; it is not advertised as ready and no duplicate is queued. |
| Every eligible host is full or unavailable | Existing job waits; other policies and worker tasks continue. | | Every eligible host is full or unavailable | Existing job waits; other policies and worker tasks continue. |
| Key is missing, removed, or rejected | Clear blocker or paused failure; no alternate identity receives the same key automatically. | | Game key pool is empty or exhausted | Hold growth with a clear reason; other games remain independent and eligible surplus removal can proceed. |
| A launch is queued, stopped, failed, or uncertain | Its key stays reserved until confirmed cleanup; no other launch can use that key concurrently. |
| Occupancy evidence conflicts, expires, or arrives from an old lifecycle | Positive current occupancy wins; invalid evidence cannot establish emptiness. | | Occupancy evidence conflicts, expires, or arrives from an old lifecycle | Positive current occupancy wins; invalid evidence cannot establish emptiness. |
| Players join before claim, after dispatch, or while the agent holds removal | Queued work is withdrawn where possible; agent safety prevents occupied destruction; held autoscaling intent cannot execute later without a fresh decision. | | Players join before claim, after dispatch, or while the agent holds removal | Queued work is withdrawn where possible; agent safety prevents occupied destruction; held autoscaling intent cannot execute later without a fresh decision. |
| Process restarts before or after intent submission | Existing operation is recovered by durable identity without replay or duplicate admission. | | Process restarts before or after intent submission | Existing operation is recovered by durable identity without replay or duplicate admission. |
@@ -153,6 +154,6 @@ An isolated Firefox review exercised policy creation, preview, save, keyboard fo
Game-wide policy follow-up: selecting every map applies the targets independently to each catalog map and variant. Game-wide policies serialize actions across maps and cannot overlap live map-specific ownership. Existing map policies remain compatible. Game-wide policy follow-up: selecting every map applies the targets independently to each catalog map and variant. Game-wide policies serialize actions across maps and cannot overlap live map-specific ownership. Existing map policies remain compatible.
Key coverage follow-up: creation in every mode requires sufficient saved keys for the maximum on each selected map. Coverage is rechecked on save, including edits to active or dry-run policies. Turning an existing policy Off remains available after key loss. Game key pool follow-up: keys are shared by all launch paths for a game. Lobby names and key labels are independent. Policy maxima no longer require dedicated key coverage on every map, and policies can be saved before adding keys. Snapshots account for reservations on other maps, planner creation waits when the pool is exhausted, and admission accepts its own reserved key. Keys are released only after confirmed cleanup.
Follow-up validation: 40 autoscaling tests and 118 related provisioning, collection, and cancellation tests passed, along with type checking and the production build. An isolated Firefox review verified that missing per-map keys disable Save, complete coverage permits game-wide dry-run creation, and keyboard navigation and 1440, 390, and 320 pixel layouts work. Follow-up validation: 40 autoscaling tests and 118 related provisioning, collection, and cancellation tests passed, along with type checking and the production build. The earlier isolated Firefox review covered keyboard navigation and 1440, 390, and 320 pixel layouts. Its per-map key coverage checks described the previous behavior, superseded by game key pools.
+1 -1
View File
@@ -19,7 +19,7 @@ export interface ScalingMember {
emptyStartedAt?: number | null; emptyStartedAt?: number | null;
} }
export interface ScalingSnapshot { export interface ScalingSnapshot {
members: ScalingMember[]; reservedNames: string[]; keyedNames: string[]; blockers: string[]; members: ScalingMember[]; reservedNames: string[]; keyPool: { total: number; available: number }; blockers: string[];
} }
export interface ScalingDecision { export interface ScalingDecision {
action: 'create' | 'destroy' | 'wait'; reason: string; server: string | null; action: 'create' | 'destroy' | 'wait'; reason: string; server: string | null;
+3 -10
View File
@@ -2,7 +2,7 @@ import { createHash, randomUUID } from 'node:crypto';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { liveScaling, scalingSettings, type ScalingPolicy, type ScalingPreview, type ScalingReport, type ScalingView } from '../../autoscaling.js'; import { liveScaling, scalingSettings, type ScalingPolicy, type ScalingPreview, type ScalingReport, type ScalingView } from '../../autoscaling.js';
import { emptyObservation, planScaling } from './planner.js'; import { emptyObservation, planScaling } from './planner.js';
import { zombieMaps, parseServerName } from '../../map-aliases.js'; import { parseServerName } from '../../map-aliases.js';
import { scalingSnapshot } from './snapshot.js'; import { scalingSnapshot } from './snapshot.js';
import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisioning/native.js'; import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisioning/native.js';
import { cancelCreation } from '../provisioning/cancel-creation.js'; import { cancelCreation } from '../provisioning/cancel-creation.js';
@@ -30,13 +30,6 @@ export function previewScaling(input: unknown, id?: string, revision?: number):
nextActionAt: existing?.nextActionAt ?? 0, pausedReason: null, updatedAt: existing?.updatedAt ?? '' }; nextActionAt: existing?.nextActionAt ?? 0, pausedReason: null, updatedAt: existing?.updatedAt ?? '' };
const snapshot = scalingSnapshot(policy); const snapshot = scalingSnapshot(policy);
const blockers: string[] = []; const blockers: string[] = [];
// Creating any policy requires full coverage; existing policies must always be pausable.
if (!existing || policy.mode !== 'off') {
for (const map of zombieMaps(policy.game).filter(m => policy.mapSlug === '*' || m.slug === policy.mapSlug)) {
const count = snapshot.keyedNames.filter(name => parseServerName(name)?.mapSlug === map.slug).length;
if (count < policy.maximum) blockers.push(`Save keys for at least ${policy.maximum} lobby identities for ${map.friendly} before saving this policy (${count} saved; ${policy.maximum - count} more required).`);
}
}
if (liveScaling(policy.mode)) { if (liveScaling(policy.mode)) {
const owner = scalingOwner(policy.game, policy.mapSlug); const owner = scalingOwner(policy.game, policy.mapSlug);
if (owner && owner !== policy.id) blockers.push(`An overlapping policy owns this lineup. Pause it and settle pending work first: /autoscaling#${owner}`); if (owner && owner !== policy.id) blockers.push(`An overlapping policy owns this lineup. Pause it and settle pending work first: /autoscaling#${owner}`);
@@ -51,7 +44,7 @@ export function previewScaling(input: unknown, id?: string, revision?: number):
} }
const decision = planScaling(policy, snapshot, Date.now()); const decision = planScaling(policy, snapshot, Date.now());
const token = createHash('sha256').update(JSON.stringify({ settings, revision: policy.revision, globalPaused: scalingGloballyPaused(), const token = createHash('sha256').update(JSON.stringify({ settings, revision: policy.revision, globalPaused: scalingGloballyPaused(),
members: snapshot.members.map(m => [m.name, m.identity, m.state]), keys: snapshot.keyedNames.sort(), reservations: snapshot.reservedNames.sort(), blockers: snapshot.blockers, decision })).digest('hex'); members: snapshot.members.map(m => [m.name, m.identity, m.state]), keyPool: snapshot.keyPool, reservations: snapshot.reservedNames.sort(), blockers: snapshot.blockers, decision })).digest('hex');
return { policy, snapshot, decision, token, blockers }; return { policy, snapshot, decision, token, blockers };
} }
@@ -154,7 +147,7 @@ export async function runAutoscaling(): Promise<boolean> {
ON CONFLICT(policy_id,server) DO UPDATE SET identity=excluded.identity,observation=excluded.observation`).run(policy.id, member.name, member.identity, JSON.stringify(observation)); ON CONFLICT(policy_id,server) DO UPDATE SET identity=excluded.identity,observation=excluded.observation`).run(policy.id, member.name, member.identity, JSON.stringify(observation));
} }
// Evidence includes provenance and timestamps but never keys or player rosters. // Evidence includes provenance and timestamps but never keys or player rosters.
const evidence = JSON.stringify({ members: snapshot.members, reservations: snapshot.reservedNames, keyCount: snapshot.keyedNames.length, blockers: snapshot.blockers }); const evidence = JSON.stringify({ members: snapshot.members, reservations: snapshot.reservedNames, keyPool: snapshot.keyPool, blockers: snapshot.blockers });
const fingerprint = createHash('sha256').update(JSON.stringify([policy.revision, decision, snapshot.members.map(m => [m.name, m.identity, m.state])])).digest('hex'); const fingerprint = createHash('sha256').update(JSON.stringify([policy.revision, decision, snapshot.members.map(m => [m.name, m.identity, m.state])])).digest('hex');
const last = db().prepare('SELECT id,fingerprint FROM autoscaling_decisions WHERE policy_id=? ORDER BY evaluated_at DESC,rowid DESC LIMIT 1').get(policy.id) as { id: string; fingerprint: string } | undefined; const last = db().prepare('SELECT id,fingerprint FROM autoscaling_decisions WHERE policy_id=? ORDER BY evaluated_at DESC,rowid DESC LIMIT 1').get(policy.id) as { id: string; fingerprint: string } | undefined;
const execute = liveScaling(policy.mode) && (decision.action === 'create' || policy.mode === 'create-destroy' && decision.action === 'destroy'); const execute = liveScaling(policy.mode) && (decision.action === 'create' || policy.mode === 'create-destroy' && decision.action === 'destroy');
+7 -3
View File
@@ -21,7 +21,7 @@ export function planScaling(policy: ScalingPolicy, snapshot: ScalingSnapshot, no
const matches = (name: string) => (snapshot.members.find(m => m.name === name)?.mapSlug ?? parseServerName(name)?.mapSlug) === map.slug; const matches = (name: string) => (snapshot.members.find(m => m.name === name)?.mapSlug ?? parseServerName(name)?.mapSlug) === map.slug;
return planScaling({ ...policy, mapSlug: map.slug }, { return planScaling({ ...policy, mapSlug: map.slug }, {
...snapshot, members: snapshot.members.filter(m => matches(m.name)), ...snapshot, members: snapshot.members.filter(m => matches(m.name)),
reservedNames: snapshot.reservedNames.filter(matches), keyedNames: snapshot.keyedNames.filter(matches) reservedNames: snapshot.reservedNames.filter(matches)
}, now); }, now);
}); });
// Serialize game-wide work and prefer replenishing any map before trimming another. // Serialize game-wide work and prefer replenishing any map before trimming another.
@@ -50,8 +50,12 @@ export function planScaling(policy: ScalingPolicy, snapshot: ScalingSnapshot, no
if (policy.nextActionAt > now) return result('wait', 'Action cooldown is active.'); if (policy.nextActionAt > now) return result('wait', 'Action cooldown is active.');
if (occupied + empty < desired) { if (occupied + empty < desired) {
if (snapshot.reservedNames.length >= policy.maximum) return result('wait', 'Maximum lobby reservations reached.'); if (snapshot.reservedNames.length >= policy.maximum) return result('wait', 'Maximum lobby reservations reached.');
const available = snapshot.keyedNames.filter(name => !snapshot.reservedNames.includes(name)).sort((a, b) => a.localeCompare(b, 'en', { numeric: true })); if (!snapshot.keyPool.available) return result('wait', snapshot.keyPool.total
return available.length ? result('create', 'Replenish empty lobby capacity.', available[0]) : result('wait', 'No unused lobby identity has a saved key.'); ? `All ${policy.game.toUpperCase()} Plutonium keys are in use. Waiting for a free key.`
: `No Plutonium keys are saved for ${policy.game.toUpperCase()}.`);
const available = Array.from({ length: 20 }, (_, index) => `${policy.game}_zombie_${policy.mapSlug}_${index + 1}`)
.find(name => !snapshot.reservedNames.includes(name));
return available ? result('create', 'Replenish empty lobby capacity.', available) : result('wait', 'No unused lobby identity is available.');
} }
if (occupied + empty > desired && empty > policy.spares && occupied + empty > policy.minimum) { if (occupied + empty > desired && empty > policy.spares && occupied + empty > policy.minimum) {
const candidates = members.filter(m => m.state === 'empty' && m.emptySince !== null && m.observedAt! - m.emptySince >= policy.emptyMinutes * 60_000) const candidates = members.filter(m => m.state === 'empty' && m.emptySince !== null && m.observedAt! - m.emptySince >= policy.emptyMinutes * 60_000)
+5 -3
View File
@@ -1,5 +1,5 @@
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { validServerId } from '../fleet/server-keys.js'; import { serverKeyPool } from '../fleet/server-keys.js';
import { listServers, type ServerRow } from '../fleet/servers.js'; import { listServers, type ServerRow } from '../fleet/servers.js';
import { observationReader } from '../fleet/status.js'; import { observationReader } from '../fleet/status.js';
import { parseServerName, zombieMaps } from '../../map-aliases.js'; import { parseServerName, zombieMaps } from '../../map-aliases.js';
@@ -73,6 +73,8 @@ export function scalingSnapshot(policy: ScalingPolicy, now = Date.now(), exceptO
reservedNames.add(row.name); reservedNames.add(row.name);
if (!members.some(m => m.name === row.name)) blockers.push(`Unresolved native instance ${row.name}.`); if (!members.some(m => m.name === row.name)) blockers.push(`Unresolved native instance ${row.name}.`);
} }
const keyedNames = (handle.prepare("SELECT server_id AS name FROM server_keys WHERE length(trim(pluto_key))>0").all() as { name: string }[]).filter(row => validServerId(row.name) && matches(row)).map(r => r.name); for (const row of handle.prepare('SELECT server_id AS name FROM server_key_leases').all() as { name: string }[]) if (matches(row)) reservedNames.add(row.name);
return { members, reservedNames: [...reservedNames], keyedNames, blockers }; const ownServer = exceptOperation ? (handle.prepare('SELECT server FROM server_deployments WHERE id=?').get(exceptOperation) as { server: string } | undefined)?.server : undefined;
const { total, available } = serverKeyPool(policy.game, ownServer);
return { members, reservedNames: [...reservedNames], keyPool: { total, available }, blockers };
} }
+2
View File
@@ -10,6 +10,7 @@ import { backfillLeaderboard } from '../fleet/leaderboard-ingest.js';
import { initializeFleetSettings } from './fleet-settings.js'; import { initializeFleetSettings } from './fleet-settings.js';
import { migrateServerHistory } from './server-history.js'; import { migrateServerHistory } from './server-history.js';
import { migrateNativeInstances } from '../backends/schema.js'; import { migrateNativeInstances } from '../backends/schema.js';
import { migrateServerKeys } from './server-keys.js';
import { ensureAutoscalingSchema } from '../autoscaling/schema.js'; import { ensureAutoscalingSchema } from '../autoscaling/schema.js';
import { installPerformanceIndexes } from './performance-indexes.js'; import { installPerformanceIndexes } from './performance-indexes.js';
import { migrateFileBlobs } from './file-blobs.js'; import { migrateFileBlobs } from './file-blobs.js';
@@ -397,6 +398,7 @@ export function db(): Database.Database {
backfillLeaderboard(handle); backfillLeaderboard(handle);
initializeFleetSettings(handle); initializeFleetSettings(handle);
migrationStep('native instance schema and file plans', () => migrateNativeInstances(handle)); migrationStep('native instance schema and file plans', () => migrateNativeInstances(handle));
migrationStep('per-game server key pools', () => migrateServerKeys(handle));
migrationStep('autoscaling schema', () => ensureAutoscalingSchema(handle)); migrationStep('autoscaling schema', () => ensureAutoscalingSchema(handle));
// File-blob migration removes the legacy content column altogether. // File-blob migration removes the legacy content column altogether.
// Reordering it first would unnecessarily copy every artifact twice. // Reordering it first would unnecessarily copy every artifact twice.
+7 -3
View File
@@ -433,11 +433,15 @@ CREATE TABLE IF NOT EXISTS deployment_scripts (
); );
CREATE INDEX IF NOT EXISTS idx_deployment_scripts_dep ON deployment_scripts(deployment_id); CREATE INDEX IF NOT EXISTS idx_deployment_scripts_dep ON deployment_scripts(deployment_id);
-- Deliberately independent of servers: credentials survive server destruction. -- Credentials belong to a game and survive server destruction. Reservations
-- are installed by the key-pool migration after native instance schema exists.
CREATE TABLE IF NOT EXISTS server_keys ( CREATE TABLE IF NOT EXISTS server_keys (
server_id TEXT PRIMARY KEY, id TEXT PRIMARY KEY,
game TEXT NOT NULL CHECK (game IN ('t5','t6')),
label TEXT NOT NULL DEFAULT '',
pluto_key TEXT NOT NULL, pluto_key TEXT NOT NULL,
updated_at TEXT NOT NULL DEFAULT (datetime('now')) updated_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE(game, pluto_key)
); );
CREATE TABLE IF NOT EXISTS settings ( CREATE TABLE IF NOT EXISTS settings (
+100
View File
@@ -0,0 +1,100 @@
import { randomUUID } from 'node:crypto';
import type Database from 'better-sqlite3';
const keySchema = `CREATE TABLE IF NOT EXISTS server_keys (
id TEXT PRIMARY KEY,
game TEXT NOT NULL CHECK (game IN ('t5','t6')),
label TEXT NOT NULL DEFAULT '',
pluto_key TEXT NOT NULL,
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE(game, pluto_key)
);`;
const leaseSchema = `CREATE TABLE IF NOT EXISTS server_key_leases (
server_id TEXT NOT NULL,
key_id TEXT NOT NULL REFERENCES server_keys(id) ON DELETE RESTRICT,
is_current INTEGER NOT NULL DEFAULT 1 CHECK (is_current IN (0,1)),
reserved_at TEXT NOT NULL DEFAULT (datetime('now')),
PRIMARY KEY(server_id,key_id)
);
CREATE INDEX IF NOT EXISTS idx_server_key_leases_key ON server_key_leases(key_id);
CREATE UNIQUE INDEX IF NOT EXISTS idx_server_key_leases_current ON server_key_leases(server_id) WHERE is_current=1;`;
/** Preserve every occupied legacy assignment, even duplicated credentials. */
export function migrateServerKeys(handle: Database.Database): void {
handle.transaction(() => {
const legacy = (handle.prepare('PRAGMA table_info(server_keys)').all() as { name: string }[])
.some(column => column.name === 'server_id');
if (legacy) handle.exec('ALTER TABLE server_keys RENAME TO legacy_server_keys');
handle.exec(keySchema + leaseSchema);
if (!legacy) return;
const rows = handle.prepare(`SELECT server_id, pluto_key, updated_at FROM legacy_server_keys ORDER BY server_id`).all() as
{ server_id: string; pluto_key: string; updated_at: string }[];
const occupied = handle.prepare(`SELECT 1 WHERE
EXISTS (SELECT 1 FROM servers WHERE name=@server AND destroyed_at IS NULL)
OR EXISTS (SELECT 1 FROM native_instances WHERE server=@server AND retired_at IS NULL)
OR EXISTS (SELECT 1 FROM server_deployments WHERE server=@server AND server_deleted_at IS NULL
AND action IN ('create','rebuild') AND state NOT IN ('done','warning','cancelled'))`);
const find = handle.prepare('SELECT id FROM server_keys WHERE game=? AND pluto_key=?');
const serverGame = handle.prepare("SELECT game FROM servers WHERE name=? AND game IN ('t5','t6')");
const insert = handle.prepare('INSERT INTO server_keys(id,game,label,pluto_key,updated_at) VALUES(?,?,?,?,?)');
const lease = handle.prepare('INSERT INTO server_key_leases(server_id,key_id,is_current) VALUES(?,?,?)');
const assignments = new Map<string, string[]>();
function save(game: string, credential: string, label: string, updatedAt: string): string {
const existing = (find.get(game, credential) as { id: string } | undefined)?.id;
if (existing) return existing;
const id = randomUUID();
insert.run(id, game, label, credential, updatedAt);
return id;
}
for (const row of rows) {
const game = /^(t5|t6)_/.exec(row.server_id)?.[1]
?? (serverGame.get(row.server_id) as { game: string } | undefined)?.game;
if (!game) throw new Error(`Cannot determine the game for saved server key ${row.server_id}`);
const id = save(game, row.pluto_key, row.server_id, row.updated_at);
if (occupied.get({ server: row.server_id })) assignments.set(row.server_id, [id]);
}
// The old form could overwrite a saved key without changing the running
// container. Durable create/rebuild payloads are stronger evidence of
// its actual credential. Keep ambiguous delivered attempts as additional
// holds until the instance is destroyed; never free a possibly live key.
const hasTable = (name: string) => !!handle.prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name=?").get(name);
if (hasTable('ec_host_operations')) {
const payload = hasTable('ec_host_operation_payloads')
? 'COALESCE((SELECT p.payload FROM ec_host_operation_payloads p WHERE p.operation_id=o.id),o.payload)' : 'o.payload';
const command = `CASE WHEN json_valid(${payload}) THEN ${payload} ELSE '{}' END`;
const launched = handle.prepare(`SELECT n.server,o.state,o.updated_at,
json_extract(${command},'$.env.PLUTO_SERVER_KEY') AS credential,
json_extract(${command},'$.game') AS game,
json_extract(${command},'$.env.PLUTAINER_GAME') AS game_id
FROM native_instances n JOIN ec_host_operations o ON o.instance_id=n.id AND o.host_id=n.host_id
WHERE n.retired_at IS NULL AND o.kind IN ('instance.create','instance.rebuild') AND o.state!='failed'
AND (o.generation=n.generation OR json_extract(${command},'$.nextGeneration')=n.generation)
ORDER BY o.rowid DESC`).all() as
{ server: string; state: string; updated_at: string; credential: unknown; game: unknown; game_id: unknown }[];
const recovered = new Map<string, string[]>();
const complete = new Set<string>();
for (const row of launched) {
if (complete.has(row.server) || typeof row.credential !== 'string' || !row.credential) continue;
const game = row.game === 't5' || row.game === 't6' ? row.game
: row.game_id === 't5sp' ? 't5' : row.game_id === 't6zm' ? 't6'
: /^(t5|t6)_/.exec(row.server)?.[1]
?? (serverGame.get(row.server) as { game: string } | undefined)?.game;
if (!game) throw new Error(`Cannot determine the game for launched server ${row.server}`);
const ids = recovered.get(row.server) ?? [];
const id = save(game, row.credential, row.server, row.updated_at);
if (!ids.includes(id)) ids.push(id);
recovered.set(row.server, ids);
if (row.state === 'completed') complete.add(row.server);
}
for (const [server, ids] of recovered) {
// If no completed launch receipt survives, its saved assignment
// remains a possible prior credential while the attempt resolves.
if (!complete.has(server)) for (const id of assignments.get(server) ?? []) if (!ids.includes(id)) ids.push(id);
assignments.set(server, ids);
}
}
for (const [server, ids] of assignments) ids.forEach((id, index) => lease.run(server, id, index === 0 ? 1 : 0));
handle.exec('DROP TABLE legacy_server_keys');
}).immediate();
}
+8 -3
View File
@@ -13,6 +13,7 @@ import { parseServerName, zombieMapBySlug, zombieMaps } from './map-aliases.js';
import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisioning/native.js'; import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisioning/native.js';
import { nativeDeployment, nativeInstance } from '../backends/native.js'; import { nativeDeployment, nativeInstance } from '../backends/native.js';
import { launchAvailability } from '../hosts/availability.js'; import { launchAvailability } from '../hosts/availability.js';
import { availableServerKeyCount } from './server-keys.js';
function coverage(entries: BlueprintEntry[], includeHealth = true) { function coverage(entries: BlueprintEntry[], includeHealth = true) {
type Server = { name: string; game: string | null; map_slug: string | null; map: string | null }; type Server = { name: string; game: string | null; map_slug: string | null; map: string | null };
@@ -72,6 +73,8 @@ function collectionPlan(entries: BlueprintEntry[], mode: CollectionMode, include
function allocate(entries: BlueprintEntry[], counts = coverage(entries)): CollectionPreview['servers'] { function allocate(entries: BlueprintEntry[], counts = coverage(entries)): CollectionPreview['servers'] {
const occupied = db().prepare(`SELECT name FROM servers UNION const occupied = db().prepare(`SELECT name FROM servers UNION
SELECT server AS name FROM native_instances WHERE retired_at IS NULL UNION
SELECT server_id AS name FROM server_key_leases UNION
SELECT j.server AS name FROM server_creation_jobs j JOIN server_deployments d ON d.id=j.deployment_id SELECT j.server AS name FROM server_creation_jobs j JOIN server_deployments d ON d.id=j.deployment_id
WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued','running','awaiting-health') WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued','running','awaiting-health')
OR (d.action='rebuild' AND (d.state='waiting' OR (d.state IN ('failed','interrupted') AND d.stage='Removing server files and registry'))))`) OR (d.action='rebuild' AND (d.state='waiting' OR (d.state IN ('failed','interrupted') AND d.stage='Removing server files and registry'))))`)
@@ -83,7 +86,7 @@ function allocate(entries: BlueprintEntry[], counts = coverage(entries)): Collec
const taken = lobbies.get(match[1]) ?? new Set<number>(); const taken = lobbies.get(match[1]) ?? new Set<number>();
taken.add(Number(match[2])); lobbies.set(match[1], taken); taken.add(Number(match[2])); lobbies.set(match[1], taken);
} }
const savedKeys = new Set((db().prepare("SELECT server_id FROM server_keys WHERE length(trim(pluto_key))>0").all() as { server_id: string }[]).map(row => row.server_id)); const availableKeys = { t5: availableServerKeyCount('t5'), t6: availableServerKeyCount('t6') };
return entries.flatMap((entry, entryIndex) => Array.from({ length: counts[entryIndex].missing }, () => { return entries.flatMap((entry, entryIndex) => Array.from({ length: counts[entryIndex].missing }, () => {
const base = `${entry.recipe.game}_zombie_${entry.recipe.mapSlug}`; const base = `${entry.recipe.game}_zombie_${entry.recipe.mapSlug}`;
const taken = lobbies.get(base) ?? new Set<number>(); const taken = lobbies.get(base) ?? new Set<number>();
@@ -92,7 +95,9 @@ function allocate(entries: BlueprintEntry[], counts = coverage(entries)): Collec
const native = entry.recipe.native; const native = entry.recipe.native;
const port = null; const port = null;
if (lobby !== null) { taken.add(lobby); lobbies.set(base, taken); } if (lobby !== null) { taken.add(lobby); lobbies.set(base, taken); }
return { entryIndex, name, port, ...(native ? { native } : {}), hasSavedKey: !!name && savedKeys.has(name) }; const hasSavedKey = !!name && availableKeys[entry.recipe.game] > 0;
if (hasSavedKey) availableKeys[entry.recipe.game]--;
return { entryIndex, name, port, ...(native ? { native } : {}), hasSavedKey };
})); }));
} }
@@ -185,7 +190,7 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }):
const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)]; const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)];
if (blockers.length) throw new Error(blockers.join(' ')); if (blockers.length) throw new Error(blockers.join(' '));
if (entries.some((_entry, index) => counts[index].missing !== reviewed.filter(server => server.entryIndex === index).length)) { if (entries.some((_entry, index) => counts[index].missing !== reviewed.filter(server => server.entryIndex === index).length)) {
throw new Error('Fleet counts changed. Nothing was queued. Refresh the preview and review each server key.'); throw new Error('Fleet counts changed. Nothing was queued. Refresh the preview and review the launches.');
} }
const expected = new Set(servers.map(server => server.name)); const expected = new Set(servers.map(server => server.name));
for (const [name, value] of Object.entries(keys)) { for (const [name, value] of Object.entries(keys)) {
+110 -23
View File
@@ -1,35 +1,122 @@
import { randomUUID } from 'node:crypto';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { zombieMapBySlug } from './map-aliases.js';
export function validServerId(value: unknown): value is string { type KeyGame = 't5' | 't6';
if (typeof value !== 'string') return false; export interface ServerKey {
const match = /^(t5|t6)_zombie_([a-z0-9]+)_([1-9]|1[0-9]|20)$/.exec(value); id: string;
return !!match && !!zombieMapBySlug(match[1], match[2]); game: KeyGame;
label: string;
updated_at: string;
assigned_servers: string[];
} }
export function listServerKeys() { function validGame(game: unknown): asserts game is KeyGame {
return db().prepare(`SELECT k.server_id, k.updated_at, if (game !== 't5' && game !== 't6') throw new Error('Choose T5 or T6 for the server key');
EXISTS(SELECT 1 FROM servers s WHERE s.name = k.server_id) AS deployed
FROM server_keys k ORDER BY k.server_id`).all() as Array<{
server_id: string; updated_at: string; deployed: number;
}>;
} }
export function getServerKey(serverId: string): string | undefined { function normalizedKey(key: unknown): string {
return (db().prepare('SELECT pluto_key FROM server_keys WHERE server_id = ?')
.get(serverId) as { pluto_key: string } | undefined)?.pluto_key;
}
export function saveServerKey(serverId: unknown, key: unknown): void {
if (!validServerId(serverId)) throw new Error('Choose a valid T5 or T6 zombie server ID with a lobby number from 1 to 20');
if (typeof key !== 'string' || !key.trim() || key.trim().length > 512 || /\s/.test(key.trim())) { if (typeof key !== 'string' || !key.trim() || key.trim().length > 512 || /\s/.test(key.trim())) {
throw new Error('Enter a server key without whitespace (maximum 512 characters)'); throw new Error('Enter a server key without whitespace (maximum 512 characters)');
} }
db().prepare(`INSERT INTO server_keys (server_id, pluto_key) VALUES (?, ?) return key.trim();
ON CONFLICT(server_id) DO UPDATE SET pluto_key = excluded.pluto_key, updated_at = datetime('now')`)
.run(serverId, key.trim());
} }
export function removeServerKey(serverId: string): boolean { function normalizedLabel(label: unknown): string {
return db().prepare('DELETE FROM server_keys WHERE server_id = ?').run(serverId).changes > 0; if (typeof label !== 'string' || label.trim().length > 160 || /[\r\n]/.test(label)) {
throw new Error('Enter a key label with at most 160 characters');
}
return label.trim();
}
export function listServerKeys(): ServerKey[] {
const rows = db().prepare('SELECT id,game,label,updated_at FROM server_keys ORDER BY game,label,id').all() as Omit<ServerKey, 'assigned_servers'>[];
const assignments = db().prepare(`SELECT DISTINCT k.id AS key_id,l.server_id FROM server_keys k
JOIN server_keys occupied ON occupied.pluto_key=k.pluto_key JOIN server_key_leases l ON l.key_id=occupied.id
ORDER BY l.server_id`).all() as { key_id: string; server_id: string }[];
const byKey = new Map<string, string[]>();
for (const lease of assignments) {
const servers = byKey.get(lease.key_id) ?? [];
servers.push(lease.server_id);
byKey.set(lease.key_id, servers);
}
return rows.map(row => ({ ...row, assigned_servers: byKey.get(row.id) ?? [] }));
}
export function getServerKey(serverId: string): string | undefined {
return (db().prepare(`SELECT k.pluto_key FROM server_key_leases l
JOIN server_keys k ON k.id=l.key_id WHERE l.server_id=? AND l.is_current=1`).get(serverId) as { pluto_key: string } | undefined)?.pluto_key;
}
/** Saving a credential twice cannot increase available capacity. */
export function saveServerKey(game: unknown, key: unknown, label?: unknown): string {
validGame(game);
const credential = normalizedKey(key);
const keyLabel = label === undefined ? undefined : normalizedLabel(label);
return db().transaction(() => {
const existing = db().prepare('SELECT id FROM server_keys WHERE game=? AND pluto_key=?').get(game, credential) as { id: string } | undefined;
if (existing) {
if (keyLabel !== undefined) db().prepare("UPDATE server_keys SET label=?,updated_at=datetime('now') WHERE id=?").run(keyLabel, existing.id);
return existing.id;
}
const id = randomUUID();
db().prepare('INSERT INTO server_keys(id,game,label,pluto_key) VALUES(?,?,?,?)').run(id, game, keyLabel ?? '', credential);
return id;
}).immediate();
}
// Compare the credential as well as its row: a mistakenly imported copy in
// another game's pool must not permit simultaneous use of the same key.
const availableKey = `NOT EXISTS (SELECT 1 FROM server_key_leases l
JOIN server_keys occupied ON occupied.id=l.key_id
WHERE occupied.pluto_key=k.pluto_key AND (l.server_id!=@server OR l.is_current=0))`;
export function serverKeyPool(game: KeyGame, serverId?: string): { total: number; available: number } {
validGame(game);
return db().prepare(`SELECT COUNT(*) AS total,
COALESCE(SUM(CASE WHEN ${availableKey} THEN 1 ELSE 0 END),0) AS available
FROM server_keys k WHERE k.game=@game`).get({ game, server: serverId ?? '' }) as { total: number; available: number };
}
export function availableServerKeyCount(game: KeyGame): number {
return serverKeyPool(game).available;
}
/** Called within launch admission; the write lock prevents competing launches sharing a key. */
export function reserveServerKey(serverId: string, game: KeyGame, suppliedKey?: unknown): string {
validGame(game);
if (!serverId) throw new Error('Choose a server for the key reservation');
const credential = suppliedKey === undefined || suppliedKey === null || (typeof suppliedKey === 'string' && !suppliedKey.trim())
? undefined : normalizedKey(suppliedKey);
return db().transaction(() => {
const existing = db().prepare(`SELECT k.game,k.pluto_key FROM server_key_leases l
JOIN server_keys k ON k.id=l.key_id WHERE l.server_id=? AND l.is_current=1`).get(serverId) as { game: KeyGame; pluto_key: string } | undefined;
if (existing) {
if (existing.game !== game) throw new Error('This server already has a key reserved for another game');
if (credential && credential !== existing.pluto_key) throw new Error('This server already has a reserved key; destroy it before changing its key');
return existing.pluto_key;
}
const requestedId = credential ? saveServerKey(game, credential) : null;
const key = db().prepare(`SELECT k.id,k.pluto_key FROM server_keys k WHERE k.game=@game
AND (@requestedId IS NULL OR k.id=@requestedId) AND ${availableKey}
ORDER BY k.updated_at,k.id LIMIT 1`).get({ game, server: serverId, requestedId }) as { id: string; pluto_key: string } | undefined;
if (!key) throw new Error(credential ? 'This server key is already reserved by another server'
: `No available ${game.toUpperCase()} server keys. Add a key or destroy a server to release one.`);
db().prepare('INSERT INTO server_key_leases(server_id,key_id) VALUES(?,?)').run(serverId, key.id);
return key.pluto_key;
}).immediate();
}
/** Only after confirmed destruction or a creation cancelled before any host mutation. */
export function releaseServerKey(serverId: string): boolean {
return db().prepare('DELETE FROM server_key_leases WHERE server_id=?').run(serverId).changes > 0;
}
export function removeServerKey(id: string): boolean {
return db().transaction(() => {
if (db().prepare(`SELECT 1 FROM server_keys k JOIN server_keys occupied ON occupied.pluto_key=k.pluto_key
JOIN server_key_leases l ON l.key_id=occupied.id WHERE k.id=?`).get(id)) {
throw new Error('This key is reserved by a server. Destroy that server before removing its key.');
}
return db().prepare('DELETE FROM server_keys WHERE id=?').run(id).changes > 0;
}).immediate();
} }
@@ -1,5 +1,6 @@
import { pauseScalingForOperation } from '../autoscaling/ownership.js'; import { pauseScalingForOperation } from '../autoscaling/ownership.js';
import { db } from '../db/index.js'; import { db } from '../db/index.js';
import { releaseServerKey } from '../fleet/server-keys.js';
import { nativeInstance } from '../backends/native.js'; import { nativeInstance } from '../backends/native.js';
import { queueHostOperation, releaseHostPorts } from '../hosts/index.js'; import { queueHostOperation, releaseHostPorts } from '../hosts/index.js';
import { beginDeployment, cancelRemoval, deploymentStage, finishDeployment } from './history.js'; import { beginDeployment, cancelRemoval, deploymentStage, finishDeployment } from './history.js';
@@ -65,10 +66,12 @@ export function cancelCreation(id: string, triggeredBy = 'ui', pausePolicy = tru
} }
export function finishCreationCancellation(cleanupId: string): void { export function finishCreationCancellation(cleanupId: string): void {
const cancellation = db().prepare(`SELECT c.creation_id,n.operation_id FROM native_creation_cancellations c const cancellation = db().prepare(`SELECT c.creation_id,n.operation_id,d.server FROM native_creation_cancellations c
JOIN server_deployments d ON d.id=c.creation_id
LEFT JOIN native_provisioning n ON n.deployment_id=c.creation_id WHERE c.cleanup_id=?`).get(cleanupId) as LEFT JOIN native_provisioning n ON n.deployment_id=c.creation_id WHERE c.cleanup_id=?`).get(cleanupId) as
{ creation_id: string; operation_id: string | null } | undefined; { creation_id: string; operation_id: string | null; server: string } | undefined;
if (!cancellation) return; if (!cancellation) return;
releaseServerKey(cancellation.server);
const now = new Date().toISOString(); const now = new Date().toISOString();
db().prepare('UPDATE server_deployments SET server_deleted_at=? WHERE id IN (?,?)').run(now, cancellation.creation_id, cleanupId); db().prepare('UPDATE server_deployments SET server_deleted_at=? WHERE id IN (?,?)').run(now, cancellation.creation_id, cleanupId);
deploymentStage(cancellation.creation_id, 'Creation cancelled; cleanup completed', { blocker: null, deploymentStage(cancellation.creation_id, 'Creation cancelled; cleanup completed', { blocker: null,
+3 -1
View File
@@ -35,7 +35,7 @@ export interface CreateServerInput {
game?: ZombieGame; // omitted by older clients: T5 game?: ZombieGame; // omitted by older clients: T5
gameMode: GameMode; // 'zombie' — only value supported today gameMode: GameMode; // 'zombie' — only value supported today
mapSlug: string; // 'cosmodrome' mapSlug: string; // 'cosmodrome'
plutoKey?: string; // saved for this server ID when supplied plutoKey?: string; // added to the game pool and reserved for this launch when supplied
expectedName?: string; // optional preview guard; never selects a lobby expectedName?: string; // optional preview guard; never selects a lobby
previewId?: string; previewId?: string;
createdBy?: string; // 'ui' by default createdBy?: string; // 'ui' by default
@@ -44,6 +44,8 @@ export interface CreateServerInput {
export function nextLobbyNumber(baseName: string, additionalNames: string[] = []): number | null { export function nextLobbyNumber(baseName: string, additionalNames: string[] = []): number | null {
const rows = db() const rows = db()
.prepare(`SELECT name FROM servers WHERE name LIKE ? UNION .prepare(`SELECT name FROM servers WHERE name LIKE ? UNION
SELECT server AS name FROM native_instances WHERE retired_at IS NULL UNION
SELECT server_id AS name FROM server_key_leases UNION
SELECT j.server AS name FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id SELECT j.server AS name FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id
WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued', 'running', 'awaiting-health') OR (d.state = 'cancelled' AND d.submitted_at IS NOT NULL) OR (d.action = 'rebuild' AND (d.state = 'waiting' OR (d.state IN ('failed','interrupted') AND d.stage = 'Removing server files and registry'))))`) WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued', 'running', 'awaiting-health') OR (d.state = 'cancelled' AND d.submitted_at IS NOT NULL) OR (d.action = 'rebuild' AND (d.state = 'waiting' OR (d.state IN ('failed','interrupted') AND d.stage = 'Removing server files and registry'))))`)
.all(`${baseName}_%`) as Array<{ name: string }>; .all(`${baseName}_%`) as Array<{ name: string }>;
+6 -5
View File
@@ -17,7 +17,7 @@ import { getHost, listHosts, getHostOperationStatus, queueHostOperation, queueHo
import { newestReadyLibraryId, assertReadyLibrary } from '../game-files/index.js'; import { newestReadyLibraryId, assertReadyLibrary } from '../game-files/index.js';
import { blueprintRecipe, captureRecipePlan, recipeForDeployment, recipeStorage, summarizeRecipe, type LaunchRecipePlan } from '../fleet/launch-recipes.js'; import { blueprintRecipe, captureRecipePlan, recipeForDeployment, recipeStorage, summarizeRecipe, type LaunchRecipePlan } from '../fleet/launch-recipes.js';
import { prepareNativeLaunch, evaluateNativePlacement, type PreparedNativeLaunch } from './launch-preparation.js'; import { prepareNativeLaunch, evaluateNativePlacement, type PreparedNativeLaunch } from './launch-preparation.js';
import { getServerKey, saveServerKey } from '../fleet/server-keys.js'; import { getServerKey, reserveServerKey, releaseServerKey, availableServerKeyCount } from '../fleet/server-keys.js';
import { activeBetaRemovalBlocker, getServer } from '../fleet/servers.js'; import { activeBetaRemovalBlocker, getServer } from '../fleet/servers.js';
import { nextLobbyNumber, type CreateServerInput, type CreateServerResult } from './instances.js'; import { nextLobbyNumber, type CreateServerInput, type CreateServerResult } from './instances.js';
import { beginDeployment, deploymentStage, finishDeployment, REMOVING_SERVER, cancelRemoval } from './history.js'; import { beginDeployment, deploymentStage, finishDeployment, REMOVING_SERVER, cancelRemoval } from './history.js';
@@ -118,7 +118,7 @@ export function previewNativeCreation(input: Record<string, unknown>): LaunchPre
{ label: 'Host port', state: port ? 'pass' : 'blocked', detail: port ? selection.automatic ? 'UDP/TCP capacity available; port assigned at placement.' : `${selection.bindAddress}:${port} / UDP and TCP` : 'No available UDP/TCP port on this host.' }, { label: 'Host port', state: port ? 'pass' : 'blocked', detail: port ? selection.automatic ? 'UDP/TCP capacity available; port assigned at placement.' : `${selection.bindAddress}:${port} / UDP and TCP` : 'No available UDP/TCP port on this host.' },
{ label: 'Configuration', state: 'pass', detail: recipe.configSource } { label: 'Configuration', state: 'pass', detail: recipe.configSource }
]; ];
return { id, expiresAt, name, port, hasSavedKey: !!name && !!getServerKey(name), recipe: summarizeRecipe(recipe), blueprintName, checks, return { id, expiresAt, name, port, hasSavedKey: !!name && availableServerKeyCount(recipe.game) > 0, recipe: summarizeRecipe(recipe), blueprintName, checks,
ready: checks.every(check => check.state !== 'blocked') }; ready: checks.every(check => check.state !== 'blocked') };
} }
@@ -163,9 +163,7 @@ export function queueNativeCreationIntent(input: { server: string; game: 't5' |
const reserved = db().prepare(`SELECT 1 FROM server_creation_jobs j JOIN server_deployments d ON d.id=j.deployment_id WHERE j.server=? const reserved = db().prepare(`SELECT 1 FROM server_creation_jobs j JOIN server_deployments d ON d.id=j.deployment_id WHERE j.server=?
AND d.server_deleted_at IS NULL AND (d.state IN ('queued','waiting','running','awaiting-health','interrupted') OR (d.state='cancelled' AND d.submitted_at IS NOT NULL))`).get(input.server); AND d.server_deleted_at IS NULL AND (d.state IN ('queued','waiting','running','awaiting-health','interrupted') OR (d.state='cancelled' AND d.submitted_at IS NOT NULL))`).get(input.server);
if ((!input.autoscalingDecisionId && input.server !== expected) || nativeInstance(input.server) || getServer(input.server) || reserved) throw new Error('Available lobby changed. Review a fresh preview.'); if ((!input.autoscalingDecisionId && input.server !== expected) || nativeInstance(input.server) || getServer(input.server) || reserved) throw new Error('Available lobby changed. Review a fresh preview.');
const key = input.plutoKey?.trim() || getServerKey(input.server); reserveServerKey(input.server, input.game, input.plutoKey);
if (!key) throw new Error(`Save a Plutonium server key for ${input.server}.`);
saveServerKey(input.server, key);
const recipe: NativeRecipe = { ...(input.recipe ?? captureRecipePlan(input.game, input.mapSlug)), const recipe: NativeRecipe = { ...(input.recipe ?? captureRecipePlan(input.game, input.mapSlug)),
native: input.selection, image: input.selection.image! }; native: input.selection, image: input.selection.image! };
const id = beginDeployment(input.server, 'create', input.createdBy ?? 'ui'); const id = beginDeployment(input.server, 'create', input.createdBy ?? 'ui');
@@ -292,6 +290,7 @@ export function reconcileNativeProvisioningReceipt(id: string): void {
db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(removal.updatedAt, original.id); db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(removal.updatedAt, original.id);
releaseHostPorts(original.id); releaseHostPorts(original.id);
db().prepare('DELETE FROM servers WHERE name=?').run(job.server); db().prepare('DELETE FROM servers WHERE name=?').run(job.server);
releaseServerKey(job.server);
db().prepare('UPDATE ec_host_migrations SET removed_at=? WHERE deployment_id=?').run(removal.updatedAt, id); db().prepare('UPDATE ec_host_migrations SET removed_at=? WHERE deployment_id=?').run(removal.updatedAt, id);
recordProgress(id, { event: { kind: 'cleanup', message: 'Cancelled migration: source removal completed; replacement creation was not started.' } }); recordProgress(id, { event: { kind: 'cleanup', message: 'Cancelled migration: source removal completed; replacement creation was not started.' } });
} }
@@ -314,6 +313,7 @@ export function reconcileNativeProvisioningReceipt(id: string): void {
db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(operation.updatedAt, instance.id); db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(operation.updatedAt, instance.id);
releaseHostPorts(instance.id); releaseHostPorts(instance.id);
db().prepare('DELETE FROM servers WHERE name=?').run(job.server); db().prepare('DELETE FROM servers WHERE name=?').run(job.server);
releaseServerKey(job.server);
finishCreationCancellation(job.id); finishCreationCancellation(job.id);
return; return;
} }
@@ -539,6 +539,7 @@ async function processJob(job: Job, eligibility: (job: Job) => string | null) {
db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(new Date().toISOString(), instance!.id); db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(new Date().toISOString(), instance!.id);
releaseHostPorts(instance!.id); releaseHostPorts(instance!.id);
db().prepare('DELETE FROM servers WHERE name=?').run(job.server); db().prepare('DELETE FROM servers WHERE name=?').run(job.server);
releaseServerKey(job.server);
recordProgress(job.id, { checkpoint: { key: 'cleanup', label: 'Instance retired and port reservations released' } }); recordProgress(job.id, { checkpoint: { key: 'cleanup', label: 'Instance retired and port reservations released' } });
complete(job); complete(job);
finishCreationCancellation(job.id); finishCreationCancellation(job.id);
@@ -65,9 +65,17 @@
if (current === session) { if (current === session) {
const next = body as CollectionPreview; const next = body as CollectionPreview;
const names = new Set(next.servers.map(server => server.name)); const names = new Set(next.servers.map(server => server.name));
keys = Object.fromEntries(next.servers.filter(server => server.name && !server.hasSavedKey).map(server => [server.name!, keys[server.name!] ?? ''])); const enteredByGame = new Map<ZombieGame, string[]>();
if (previous && (previous.servers.length !== next.servers.length || previous.servers.some(server => !names.has(server.name)))) keyNotice = 'Server assignments changed. Keys were retained only for the exact IDs still shown below. Review the new assignments.'; for (const server of previous?.servers ?? []) {
else if (previous) keyNotice = 'Preview refreshed. Entered keys are retained for these same server IDs.'; const value = server.name ? keys[server.name]?.trim() : '';
if (!value) continue;
const game = previous!.entries[server.entryIndex].recipe.game;
enteredByGame.set(game, [...(enteredByGame.get(game) ?? []), value]);
}
keys = Object.fromEntries(next.servers.filter(server => server.name && !server.hasSavedKey)
.map(server => [server.name!, enteredByGame.get(next.entries[server.entryIndex].recipe.game)?.shift() ?? '']));
if (previous && (previous.servers.length !== next.servers.length || previous.servers.some(server => !names.has(server.name)))) keyNotice = 'Server assignments changed. Entered keys were retained for launches in the same game that still need a key.';
else if (previous) keyNotice = 'Preview refreshed. Entered keys are retained for games that still need them.';
preview = next; now = Date.now(); preview = next; now = Date.now();
} }
} catch (err) { if (current === session) error = (err as Error).message; } } catch (err) { if (current === session) error = (err as Error).message; }
@@ -108,7 +116,7 @@
{#if !preview}<p class="muted">{busy ? 'Preparing the collection…' : 'Review the fleet changes, capacity estimate, and server keys. Queued launches wait automatically when hosts need more capacity or preparation.'}</p>{#if !busy}<Button disabled={!hostReady} onclick={review}>Preview launch</Button>{/if} {#if !preview}<p class="muted">{busy ? 'Preparing the collection…' : 'Review the fleet changes, capacity estimate, and server keys. Queued launches wait automatically when hosts need more capacity or preparation.'}</p>{#if !busy}<Button disabled={!hostReady} onclick={review}>Preview launch</Button>{/if}
{:else} {:else}
{#if keyNotice}<p role="status" class="muted">{keyNotice}</p>{/if} {#if keyNotice}<p role="status" class="muted">{keyNotice}</p>{/if}
{#if expired}<p class="error" role="status">This preview expired. Refresh before launch; keys remain entered for unchanged IDs.</p>{/if} {#if expired}<p class="error" role="status">This preview expired. Refresh before launch; entered keys stay with their game.</p>{/if}
<div class="lineup"><strong>{preview.mode === 'exact' ? `${preview.retainedServers.length} keep · ${preview.servers.length} create · ${preview.removals.length} remove` : preview.servers.length ? `${preview.servers.length} missing server${preview.servers.length === 1 ? '' : 's'} to create` : 'Collection already satisfied'}</strong> <div class="lineup"><strong>{preview.mode === 'exact' ? `${preview.retainedServers.length} keep · ${preview.servers.length} create · ${preview.removals.length} remove` : preview.servers.length ? `${preview.servers.length} missing server${preview.servers.length === 1 ? '' : 's'} to create` : 'Collection already satisfied'}</strong>
<table><thead><tr><th>Map</th><th>Target</th><th>Existing / queued</th><th>To create</th></tr></thead><tbody>{#each preview.entries as entry}<tr><th>{entry.recipe.mapName} <small>{entry.recipe.game.toUpperCase()}</small></th><td>{entry.count}</td><td title={entry.existingServers.join(', ')}>{entry.existing}</td><td class="quantity">{entry.missing}</td></tr>{/each}</tbody></table> <table><thead><tr><th>Map</th><th>Target</th><th>Existing / queued</th><th>To create</th></tr></thead><tbody>{#each preview.entries as entry}<tr><th>{entry.recipe.mapName} <small>{entry.recipe.game.toUpperCase()}</small></th><td>{entry.count}</td><td title={entry.existingServers.join(', ')}>{entry.existing}</td><td class="quantity">{entry.missing}</td></tr>{/each}</tbody></table>
<p class="muted">Targets count inventory, including stopped servers; they do not mean every server is ready to play. {preview.mode === 'exact' ? 'Existing deployed lobbies are kept before queued creations, then the lowest lobby numbers. Retained servers are not changed or restarted.' : 'Only missing servers are created. Extra servers remain and existing servers are not updated.'}</p> <p class="muted">Targets count inventory, including stopped servers; they do not mean every server is ready to play. {preview.mode === 'exact' ? 'Existing deployed lobbies are kept before queued creations, then the lowest lobby numbers. Retained servers are not changed or restarted.' : 'Only missing servers are created. Extra servers remain and existing servers are not updated.'}</p>
@@ -133,7 +141,8 @@
{#each preview.blockers as blocker}<p class="error" role="alert">{blocker}</p>{/each} {#each preview.blockers as blocker}<p class="error" role="alert">{blocker}</p>{/each}
{#each preview.warnings ?? [] as warning}<p class="warning" role="status">{warning}</p>{/each} {#each preview.warnings ?? [] as warning}<p class="warning" role="status">{warning}</p>{/each}
{#if preview.servers.length || preview.removals.length} {#if preview.servers.length || preview.removals.length}
{#if preview.servers.length}<p class="muted">Saved keys are reused automatically. Add keys only where required below. Update saved keys in <a href="/server-keys" target="_blank" rel="noopener">Keys (opens in a new tab)</a>. Apply saves the reviewed jobs and opens their progress page. Each launch waits for a startup slot and host capacity.</p>{/if} {#if preview.servers.length}<p class="muted">Available keys are assigned from each game’s pool. Add an unused key for each remaining launch below. Manage pools in <a href="/server-keys" target="_blank" rel="noopener">Keys (opens in a new tab)</a>. Apply saves the reviewed jobs and opens their progress page. Each launch waits for a startup slot and host capacity.</p>{/if}
{#if preview.removals.length && preview.servers.some(server => !server.hasSavedKey)}<p class="muted">Keys held by servers awaiting removal are still in use. To reuse those keys, remove the servers first and refresh this preview after cleanup finishes.</p>{/if}
<form onsubmit={event => { event.preventDefault(); void launch(); }}> <form onsubmit={event => { event.preventDefault(); void launch(); }}>
{#if preview.removals.length} {#if preview.removals.length}
<section class="removals" aria-label="Servers to remove"> <section class="removals" aria-label="Servers to remove">
@@ -150,14 +159,14 @@
<span class="order" aria-hidden="true">{String(index + 1).padStart(2, '0')}</span> <span class="order" aria-hidden="true">{String(index + 1).padStart(2, '0')}</span>
<div class="server-identity">{#if index === 0 || preview.servers[index - 1].entryIndex !== server.entryIndex}<span class="map-group">{preview.entries[server.entryIndex].recipe.game.toUpperCase()} · {preview.entries[server.entryIndex].recipe.mapName}</span>{/if}<strong>{#if server.name}<ServerIdentity name={server.name} />{:else}{preview.entries[server.entryIndex].recipe.mapName}{/if}</strong><span class="server-id">{server.name ?? 'No available lobby'} · {server.native ? 'Host and port assigned when provisioning starts' : server.port === null ? 'No available port' : `Port ${server.port}`}</span></div> <div class="server-identity">{#if index === 0 || preview.servers[index - 1].entryIndex !== server.entryIndex}<span class="map-group">{preview.entries[server.entryIndex].recipe.game.toUpperCase()} · {preview.entries[server.entryIndex].recipe.mapName}</span>{/if}<strong>{#if server.name}<ServerIdentity name={server.name} />{:else}{preview.entries[server.entryIndex].recipe.mapName}{/if}</strong><span class="server-id">{server.name ?? 'No available lobby'} · {server.native ? 'Host and port assigned when provisioning starts' : server.port === null ? 'No available port' : `Port ${server.port}`}</span></div>
{#if server.name} {#if server.name}
{#if server.hasSavedKey}<p class="key-ready"><span aria-hidden="true">✓</span> Saved key ready</p> {#if server.hasSavedKey}<p class="key-ready"><span aria-hidden="true">✓</span> Game key available</p>
{:else}<label><span>Server key required</span><input aria-label={`Key for ${server.name}`} data-missing={!keys[server.name]?.trim()} type="password" bind:value={keys[server.name]} disabled={busy} maxlength="512" autocomplete="off" placeholder="Paste this server’s key" /></label>{/if} {:else}<label><span>{preview.entries[server.entryIndex].recipe.game.toUpperCase()} pool key required</span><input aria-label={`Unused ${preview.entries[server.entryIndex].recipe.game.toUpperCase()} key for launch ${index + 1}`} data-missing={!keys[server.name]?.trim()} type="password" bind:value={keys[server.name]} disabled={busy} maxlength="512" autocomplete="off" placeholder="Paste an unused game key" /></label>{/if}
{/if} {/if}
</li> </li>
{/each} {/each}
</ol> </ol>
{/if} {/if}
<div class="launch-footer"><div>{#if preview.servers.length}<p role="status"><strong>{keysReady} / {preview.servers.length} keys available</strong></p>{/if}<p class="caption">Key validity is checked at startup. Preview expires at {$timeFormat.minuteTime(preview.expiresAt)}.</p>{#if keysReady < preview.servers.length}<button class="text-button" type="button" onclick={nextMissing}>Jump to next missing key</button>{/if}</div><div class="actions"><Button variant="ghost" disabled={busy} onclick={() => open = false}>Cancel</Button><Button type="submit" variant={preview.removals.length ? 'danger' : 'primary'} loading={busy} disabled={busy || expired || !canLaunch}>{preview.mode === 'exact' ? `Apply: ${preview.servers.length} create, ${preview.removals.length} remove` : `Queue ${preview.servers.length} server${preview.servers.length === 1 ? '' : 's'} →`}</Button></div></div> <div class="launch-footer"><div>{#if preview.servers.length}<p role="status"><strong>{keysReady} / {preview.servers.length} keys available</strong></p>{/if}<p class="caption">Pool availability is checked again when you apply. Preview expires at {$timeFormat.minuteTime(preview.expiresAt)}.</p>{#if keysReady < preview.servers.length}<button class="text-button" type="button" onclick={nextMissing}>Jump to next missing key</button>{/if}</div><div class="actions"><Button variant="ghost" disabled={busy} onclick={() => open = false}>Cancel</Button><Button type="submit" variant={preview.removals.length ? 'danger' : 'primary'} loading={busy} disabled={busy || expired || !canLaunch}>{preview.mode === 'exact' ? `Apply: ${preview.servers.length} create, ${preview.removals.length} remove` : `Queue ${preview.servers.length} server${preview.servers.length === 1 ? '' : 's'} →`}</Button></div></div>
</form> </form>
{:else}<p class="muted">{preview.mode === 'exact' ? 'Your fleet already matches this collection.' : 'Your fleet already meets or exceeds every target in this collection.'} No servers will be created or removed.</p><div class="actions"><Button onclick={() => open = false}>Done</Button></div>{/if} {:else}<p class="muted">{preview.mode === 'exact' ? 'Your fleet already matches this collection.' : 'Your fleet already meets or exceeds every target in this collection.'} No servers will be created or removed.</p><div class="actions"><Button onclick={() => open = false}>Done</Button></div>{/if}
<details class="recipes"><summary>Setup and launch requirements</summary><p class="muted requirements">Each new server is automatically placed on an eligible host with the latest game files and prepared runtime. CPU, memory, host readiness, synced files, runtime availability, and local port reservations are checked again before deployment. Existing servers keep their current files.</p>{#each preview.entries as entry}<div class="recipe"><LaunchRecipe recipe={entry.recipe} /></div>{/each}</details> <details class="recipes"><summary>Setup and launch requirements</summary><p class="muted requirements">Each new server is automatically placed on an eligible host with the latest game files and prepared runtime. CPU, memory, host readiness, synced files, runtime availability, and local port reservations are checked again before deployment. Existing servers keep their current files.</p>{#each preview.entries as entry}<div class="recipe"><LaunchRecipe recipe={entry.recipe} /></div>{/each}</details>
@@ -97,7 +97,7 @@
const body = await res.json(); const body = await res.json();
if (!res.ok) throw new Error(body.error ?? 'Could not prepare launch preview.'); if (!res.ok) throw new Error(body.error ?? 'Could not prepare launch preview.');
if (current === session) { if (current === session) {
if (previousName && previousName !== body.name) { plutoKey = ''; keyNotice = 'The available server ID changed. Enter the key for the new ID below; the previous key was not transferred.'; } if (previousName && previousName !== body.name) keyNotice = 'The available lobby changed. Any entered key can still be used for this game.';
if (body.hasSavedKey) plutoKey = ''; if (body.hasSavedKey) plutoKey = '';
preview = body; now = Date.now(); preview = body; now = Date.now();
await tick(); heading?.focus(); await tick(); heading?.focus();
@@ -148,18 +148,18 @@
{#if error}<Button onclick={initialize}>Try again</Button>{/if} {#if error}<Button onclick={initialize}>Try again</Button>{/if}
{:else if preview} {:else if preview}
{#if keyNotice}<p role="status" class="muted">{keyNotice}</p>{/if} {#if keyNotice}<p role="status" class="muted">{keyNotice}</p>{/if}
{#if expired}<p class="error" role="status">This preview expired. Refresh it to recheck capacity. Keys stay entered when the server ID is unchanged.</p>{/if} {#if expired}<p class="error" role="status">This preview expired. Refresh it to recheck capacity. Entered keys stay with this game when you refresh.</p>{/if}
<section class="recipe-panel"><LaunchRecipe recipe={preview.recipe} />{#if preview.blueprintName}<p class="muted">From collection: {preview.blueprintName}</p>{/if}</section> <section class="recipe-panel"><LaunchRecipe recipe={preview.recipe} />{#if preview.blueprintName}<p class="muted">From collection: {preview.blueprintName}</p>{/if}</section>
<section class="preflight"><div class="section-title"><h3>Before you launch</h3><button class="text-button" disabled={busy || saving} onclick={review}>Refresh preview</button></div> <section class="preflight"><div class="section-title"><h3>Before you launch</h3><button class="text-button" disabled={busy || saving} onclick={review}>Refresh preview</button></div>
<ul>{#each preview.checks as check}<li class:blocked={check.state === 'blocked'}><span class="check" aria-label={check.state}>{check.state === 'pass' ? '✓' : check.state === 'blocked' ? '!' : 'i'}</span><div><strong>{check.label}</strong><p>{check.detail}</p></div></li>{/each}</ul> <ul>{#each preview.checks as check}<li class:blocked={check.state === 'blocked'}><span class="check" aria-label={check.state}>{check.state === 'pass' ? '✓' : check.state === 'blocked' ? '!' : 'i'}</span><div><strong>{check.label}</strong><p>{check.detail}</p></div></li>{/each}</ul>
</section> </section>
<form onsubmit={event => { event.preventDefault(); void launch(); }}> <form onsubmit={event => { event.preventDefault(); void launch(); }}>
<div class="identity"><span class="eyebrow">Your new server</span><strong>{preview.name ?? 'Waiting for an available lobby'}</strong><p>Fresh identity, with its own server key.</p></div> <div class="identity"><span class="eyebrow">Your new server</span><strong>{preview.name ?? 'Waiting for an available lobby'}</strong><p>An available key from this game’s pool is assigned when you launch.</p></div>
{#if preview.hasSavedKey}<p class="key-ready"><span aria-hidden="true">✓</span> Saved key ready</p> {#if preview.hasSavedKey}<p class="key-ready"><span aria-hidden="true">✓</span> Game key available</p>
{:else}<label>Plutonium server key<input type="password" bind:value={plutoKey} disabled={busy || saving || !preview.name} maxlength="512" autocomplete="off" placeholder="Paste the key for this server ID" /></label>{/if} {:else}<label>Plutonium server key<input type="password" bind:value={plutoKey} disabled={busy || saving || !preview.name} maxlength="512" autocomplete="off" placeholder="Paste an unused key for this game" /></label>{/if}
<p class="muted">{preview.hasSavedKey ? 'The saved key for this server ID will be reused automatically. Electric Cherry does not check whether Plutonium accepts it. Update it in Keys.' : 'Add the key here. It is saved for this exact server ID when you launch, including if provisioning fails.'} <a href={`/server-keys?server=${encodeURIComponent(preview.name ?? '')}`} target="_blank" rel="noopener">Manage saved keys in a new tab</a>; your launch stays open here.</p> <p class="muted">{preview.hasSavedKey ? 'An unused key from this game’s pool will be assigned automatically. Availability is checked again at launch.' : 'Add an unused key for this game. Launch saves it in the game pool and assigns it to this server until removal is confirmed.'} <a href={`/server-keys?game=${preview.recipe.game}`} target="_blank" rel="noopener">Manage saved keys in a new tab</a>; your launch stays open here.</p>
<div class="launch-actions"><Button variant="ghost" disabled={busy || saving} onclick={() => { preview = null; plutoKey = ''; error = ''; }}>Back</Button><Button type="submit" variant="primary" loading={busy} disabled={busy || saving || expired || !hostReady || !preview.ready || (!preview.hasSavedKey && !plutoKey.trim())}>Launch server</Button></div> <div class="launch-actions"><Button variant="ghost" disabled={busy || saving} onclick={() => { preview = null; plutoKey = ''; error = ''; }}>Back</Button><Button type="submit" variant="primary" loading={busy} disabled={busy || saving || expired || !hostReady || !preview.ready || (!preview.hasSavedKey && !plutoKey.trim())}>Launch server</Button></div>
{#if !preview.hasSavedKey && !plutoKey.trim()}<p class="muted">Enter this server’s key to enable launch.</p>{/if} {#if !preview.hasSavedKey && !plutoKey.trim()}<p class="muted">Enter an unused key for this game to enable launch.</p>{/if}
<p class="caption">Preview valid until {$timeFormat.minuteTime(preview.expiresAt)}. Capacity is checked again at launch. The host and port are assigned automatically.</p> <p class="caption">Preview valid until {$timeFormat.minuteTime(preview.expiresAt)}. Capacity is checked again at launch. The host and port are assigned automatically.</p>
</form> </form>
<details class="save"><summary>Make this a reusable collection</summary><p class="muted">Save this game and map with a target of one lobby. Future launches use the latest promoted setup and create only missing lobbies.</p> <details class="save"><summary>Make this a reusable collection</summary><p class="muted">Save this game and map with a target of one lobby. Future launches use the latest promoted setup and create only missing lobbies.</p>
+9 -5
View File
@@ -1,5 +1,5 @@
import { json } from '@sveltejs/kit'; import { json } from '@sveltejs/kit';
import { listServerKeys, saveServerKey, removeServerKey, validServerId } from '$lib/server/fleet/server-keys.js'; import { listServerKeys, saveServerKey, removeServerKey } from '$lib/server/fleet/server-keys.js';
export function GET() { export function GET() {
return json({ keys: listServerKeys() }, { headers: { 'Cache-Control': 'no-store' } }); return json({ keys: listServerKeys() }, { headers: { 'Cache-Control': 'no-store' } });
@@ -8,8 +8,8 @@ export function GET() {
export async function PUT({ request }: { request: Request }) { export async function PUT({ request }: { request: Request }) {
const body = await request.json().catch(() => null); const body = await request.json().catch(() => null);
try { try {
saveServerKey(body?.serverId, body?.plutoKey); const id = saveServerKey(body?.game, body?.plutoKey, body?.label);
return json({ ok: true }); return json({ ok: true, id });
} catch (err) { } catch (err) {
return json({ ok: false, error: (err as Error).message }, { status: 400 }); return json({ ok: false, error: (err as Error).message }, { status: 400 });
} }
@@ -17,6 +17,10 @@ export async function PUT({ request }: { request: Request }) {
export async function DELETE({ request }: { request: Request }) { export async function DELETE({ request }: { request: Request }) {
const body = await request.json().catch(() => null); const body = await request.json().catch(() => null);
if (!validServerId(body?.serverId)) return json({ error: 'Invalid server ID' }, { status: 400 }); if (typeof body?.id !== 'string' || !body.id.trim() || body.id.length > 128) return json({ error: 'Invalid key ID' }, { status: 400 });
return json({ ok: removeServerKey(body.serverId) }); try {
return json({ ok: removeServerKey(body.id) });
} catch (err) {
return json({ ok: false, error: (err as Error).message }, { status: 400 });
}
} }
+4 -4
View File
@@ -94,15 +94,15 @@
<label>Empty duration in minutes<input type="number" min="1" max="1440" required bind:value={settings.emptyMinutes} /></label> <label>Empty duration in minutes<input type="number" min="1" max="1440" required bind:value={settings.emptyMinutes} /></label>
<label>Action cooldown in minutes<input type="number" min="1" max="1440" required bind:value={settings.cooldownMinutes} /></label> <label>Action cooldown in minutes<input type="number" min="1" max="1440" required bind:value={settings.cooldownMinutes} /></label>
</div> </div>
{#if settings.mapSlug === '*'}<p>Targets apply separately to every map, including all variants. Each map needs its own keys. Actions share one cooldown and run one at a time across the game.</p>{/if} {#if settings.mapSlug === '*'}<p>Targets apply separately to every map, including all variants. All maps share this game’s key pool. Actions share one cooldown and run one at a time across the game.</p>{/if}
<p>Saved lobby keys must cover the maximum for each selected map before you can save a new policy, including Dry run.</p> <p>Each new lobby needs an available key from the game pool. Creation waits when the pool is exhausted; keys return after confirmed removal.</p>
<div class="actions"><Button type="submit" disabled={busy}>Preview changes</Button><Button onclick={() => editorOpen = false}>Close editor</Button></div> <div class="actions"><Button type="submit" disabled={busy}>Preview changes</Button><Button onclick={() => editorOpen = false}>Close editor</Button></div>
</fieldset> </fieldset>
</form> </form>
{#if preview} {#if preview}
<div class="preview" aria-live="polite"> <div class="preview" aria-live="polite">
<h3>Preview · {modeLabel(preview.policy.mode)}</h3> <h3>Preview · {modeLabel(preview.policy.mode)}</h3>
<p>Target: {preview.decision.desired} running lobbies{preview.policy.mapSlug === '*' ? ' across all maps' : ''}. {preview.snapshot.keyedNames.length} saved lobby keys. {preview.snapshot.members.length} existing or pending lobbies.</p> <p>Target: {preview.decision.desired} running lobbies{preview.policy.mapSlug === '*' ? ' across all maps' : ''}. {preview.snapshot.keyPool.available} of {preview.snapshot.keyPool.total} game keys available. {preview.snapshot.members.length} existing or pending lobbies.</p>
<p>{preview.decision.action === 'wait' ? 'Wait' : `${preview.decision.action === 'create' ? 'Create' : 'Remove'} ${preview.decision.server}`}: {preview.decision.reason}</p> <p>{preview.decision.action === 'wait' ? 'Wait' : `${preview.decision.action === 'create' ? 'Create' : 'Remove'} ${preview.decision.server}`}: {preview.decision.reason}</p>
{#if preview.policy.mode === 'create-only' || preview.policy.mode === 'create-destroy'}<p>Saving gives this policy exclusive control of the selected maps' lobby counts. Pause it before launching a competing collection or manual lifecycle action.</p>{/if} {#if preview.policy.mode === 'create-only' || preview.policy.mode === 'create-destroy'}<p>Saving gives this policy exclusive control of the selected maps' lobby counts. Pause it before launching a competing collection or manual lifecycle action.</p>{/if}
{#if preview.snapshot.members.length}<ul>{#each preview.snapshot.members as member}<li>{member.name} · {member.state}{member.blocker ? ` · ${member.blocker}` : ''}</li>{/each}</ul>{/if} {#if preview.snapshot.members.length}<ul>{#each preview.snapshot.members as member}<li>{member.name} · {member.state}{member.blocker ? ` · ${member.blocker}` : ''}</li>{/each}</ul>{/if}
@@ -118,7 +118,7 @@
<div class="heading"><h2 id={`title-${view.policy.id}`}>{view.policy.mapSlug === '*' ? 'Every map' : zombieMapBySlug(view.policy.game, view.policy.mapSlug)?.friendly ?? view.policy.mapSlug} · {view.policy.game.toUpperCase()}</h2><span>{modeLabel(view.policy.mode)}</span>{#if canOperate}<Button disabled={busy} onclick={() => edit(view.policy)}>Edit or pause</Button><Button variant="danger" disabled={busy || view.unsettled.length > 0} title={view.unsettled.length ? 'Pause the policy and settle outstanding work before deleting it.' : undefined} onclick={() => remove(view.policy)}>Delete policy</Button>{/if}</div> <div class="heading"><h2 id={`title-${view.policy.id}`}>{view.policy.mapSlug === '*' ? 'Every map' : zombieMapBySlug(view.policy.game, view.policy.mapSlug)?.friendly ?? view.policy.mapSlug} · {view.policy.game.toUpperCase()}</h2><span>{modeLabel(view.policy.mode)}</span>{#if canOperate}<Button disabled={busy} onclick={() => edit(view.policy)}>Edit or pause</Button><Button variant="danger" disabled={busy || view.unsettled.length > 0} title={view.unsettled.length ? 'Pause the policy and settle outstanding work before deleting it.' : undefined} onclick={() => remove(view.policy)}>Delete policy</Button>{/if}</div>
<p class="counts">{view.decision.occupied} occupied · {view.decision.empty} empty and ready · {view.decision.pending} pending · {view.decision.unknown} unknown or stopped</p> <p class="counts">{view.decision.occupied} occupied · {view.decision.empty} empty and ready · {view.decision.pending} pending · {view.decision.unknown} unknown or stopped</p>
{#if view.unsettled.length}<div class="banner"><strong>Work is still settling.</strong><p>Pause this policy and settle its outstanding work before deleting it.</p>{#each view.unsettled as operation}<p><a href={`/provisioning/${operation.id}`}>{operation.action} · {operation.state}</a> — {operation.stage}</p>{/each}{#if view.ownsMap}<p>This policy retains ownership of its scope until its outstanding work settles.</p>{/if}</div>{/if} {#if view.unsettled.length}<div class="banner"><strong>Work is still settling.</strong><p>Pause this policy and settle its outstanding work before deleting it.</p>{#each view.unsettled as operation}<p><a href={`/provisioning/${operation.id}`}>{operation.action} · {operation.state}</a> — {operation.stage}</p>{/each}{#if view.ownsMap}<p>This policy retains ownership of its scope until its outstanding work settles.</p>{/if}</div>{/if}
<p>{view.policy.mapSlug === '*' ? 'Per map: ' : ''}Minimum {view.policy.minimum} · Maximum {view.policy.maximum} · Spare target {view.policy.spares} · {view.snapshot.keyedNames.length} saved keys</p> <p>{view.policy.mapSlug === '*' ? 'Per map: ' : ''}Minimum {view.policy.minimum} · Maximum {view.policy.maximum} · Spare target {view.policy.spares} · {view.snapshot.keyPool.available} / {view.snapshot.keyPool.total} game keys available</p>
<p><strong>{view.policy.mode === 'dry-run' ? 'Hypothetical decision: ' : 'Current decision: '}</strong>{view.decision.action}{view.decision.server ? ` ${view.decision.server}` : ''}. {view.decision.reason}</p> <p><strong>{view.policy.mode === 'dry-run' ? 'Hypothetical decision: ' : 'Current decision: '}</strong>{view.decision.action}{view.decision.server ? ` ${view.decision.server}` : ''}. {view.decision.reason}</p>
{#if view.policy.mode === 'create-only' && view.decision.action === 'destroy'}<p>Removal is advisory in Create only mode.</p>{/if} {#if view.policy.mode === 'create-only' && view.decision.action === 'destroy'}<p>Removal is advisory in Create only mode.</p>{/if}
<p class="muted">Last evaluated: {view.lastEvaluatedAt ? $timeFormat.dateTime(new Date(view.lastEvaluatedAt).toISOString()) : 'Not evaluated yet'}{view.policy.nextActionAt > Date.now() ? ` · Next action after ${$timeFormat.dateTime(new Date(view.policy.nextActionAt).toISOString())}` : ''}</p> <p class="muted">Last evaluated: {view.lastEvaluatedAt ? $timeFormat.dateTime(new Date(view.lastEvaluatedAt).toISOString()) : 'Not evaluated yet'}{view.policy.nextActionAt > Date.now() ? ` · Next action after ${$timeFormat.dateTime(new Date(view.policy.nextActionAt).toISOString())}` : ''}</p>
+1 -1
View File
@@ -113,7 +113,7 @@
{#if detail.canResumeRebuild}<Button variant="primary" loading={view.busy === 'resume'} disabled={!!view.busy} onclick={() => controller.act('resume', 'Rebuild cleanup requested. Fresh creation waits for local cleanup to finish.')}>Resume rebuild cleanup</Button>{/if} {#if detail.canResumeRebuild}<Button variant="primary" loading={view.busy === 'resume'} disabled={!!view.busy} onclick={() => controller.act('resume', 'Rebuild cleanup requested. Fresh creation waits for local cleanup to finish.')}>Resume rebuild cleanup</Button>{/if}
{#if detail.canResumeCleanup}<Button variant="primary" loading={view.busy === 'resume'} disabled={!!view.busy} onclick={() => controller.act('resume', 'Local cleanup requested. Remote container deletion will not be repeated.')}>Resume cleanup</Button>{/if} {#if detail.canResumeCleanup}<Button variant="primary" loading={view.busy === 'resume'} disabled={!!view.busy} onclick={() => controller.act('resume', 'Local cleanup requested. Remote container deletion will not be repeated.')}>Resume cleanup</Button>{/if}
{#if detail.canCancel}<div><Button disabled={!!view.busy} loading={view.busy === 'cancel'} onclick={() => controller.act('cancel', 'Operation cancelled. Already-submitted commands may finish; no further steps will be started.')}>Cancel operation</Button><p class="action-help">Stops remaining work and health checks. Changes already made are not undone.</p></div>{/if} {#if detail.canCancel}<div><Button disabled={!!view.busy} loading={view.busy === 'cancel'} onclick={() => controller.act('cancel', 'Operation cancelled. Already-submitted commands may finish; no further steps will be started.')}>Cancel operation</Button><p class="action-help">Stops remaining work and health checks. Changes already made are not undone.</p></div>{/if}
{#if detail.canCancelCreation}<div><Button variant="danger" disabled={!!view.busy} loading={view.busy === 'cancel-creation'} onclick={() => controller.act('cancel-creation', 'Creation cancelled. Follow cleanup below until removal is confirmed.')}>Cancel creation and remove server</Button><p class="action-help">Stops this launch and deletes its container and instance files, even if player status is unknown or players have joined. Shared game files and the saved server key are kept.</p></div>{/if} {#if detail.canCancelCreation}<div><Button variant="danger" disabled={!!view.busy} loading={view.busy === 'cancel-creation'} onclick={() => controller.act('cancel-creation', 'Creation cancelled. Follow cleanup below until removal is confirmed.')}>Cancel creation and remove server</Button><p class="action-help">Stops this launch and deletes its container and instance files, even if player status is unknown or players have joined. Shared game files are kept. The key returns to its game pool after removal is confirmed.</p></div>{/if}
</div> </div>
{/if} {/if}
{#if detail.cancellation}<div class="recovery-note"><strong>{detail.cancellation.state === 'done' ? 'Cancellation cleanup completed' : 'Cancellation cleanup'}</strong><p>{detail.cancellation.error ?? detail.cancellation.stage}</p><a href={`/provisioning/${encodeURIComponent(detail.cancellation.cleanupId)}`}>View cleanup progress →</a></div>{/if} {#if detail.cancellation}<div class="recovery-note"><strong>{detail.cancellation.state === 'done' ? 'Cancellation cleanup completed' : 'Cancellation cleanup'}</strong><p>{detail.cancellation.error ?? detail.cancellation.stage}</p><a href={`/provisioning/${encodeURIComponent(detail.cancellation.cleanupId)}`}>View cleanup progress →</a></div>{/if}
+65 -67
View File
@@ -3,7 +3,7 @@
import { onMount } from 'svelte'; import { onMount } from 'svelte';
import { page } from '$app/stores'; import { page } from '$app/stores';
import { goto } from '$app/navigation'; import { goto } from '$app/navigation';
import { zombieMaps, zombieMapBySlug, type ZombieGame } from '$lib/map-aliases.js'; import type { ZombieGame } from '$lib/map-aliases.js';
import { confirmAction } from '$ui/confirm.js'; import { confirmAction } from '$ui/confirm.js';
import { serverTitle } from '$ui/server-labels.js'; import { serverTitle } from '$ui/server-labels.js';
import ServerIdentity from '$ui/components/ServerIdentity.svelte'; import ServerIdentity from '$ui/components/ServerIdentity.svelte';
@@ -15,31 +15,34 @@
import Field from '$lib/ui/components/primitives/Field.svelte'; import Field from '$lib/ui/components/primitives/Field.svelte';
import Input from '$lib/ui/components/primitives/Input.svelte'; import Input from '$lib/ui/components/primitives/Input.svelte';
interface SavedKey { server_id: string; deployed: number; updated_at: string } interface SavedKey { id: string; game: ZombieGame; label: string; assigned_servers: string[]; updated_at: string }
const games: ZombieGame[] = ['t5', 't6'];
let open = $state(false); let open = $state(false);
let search = $state(''); let search = $state('');
let gameFilter = $state('');
let limit = $state(25); let limit = $state(25);
let keys = $state<SavedKey[]>([]); let keys = $state<SavedKey[]>([]);
let serverId = $state('');
let editing = $state(false);
let rawMode = $state(false);
let game = $state<ZombieGame>('t5'); let game = $state<ZombieGame>('t5');
let mapSlug = $state('theater'); let label = $state('');
let lobby = $state(1);
const chosenId = $derived(editing || rawMode ? serverId.trim() : `${game}_zombie_${mapSlug}_${lobby}`);
const validId = $derived.by(() => {
const match = /^(t5|t6)_zombie_([a-z0-9]+)_([1-9]|1[0-9]|20)$/.exec(chosenId);
return !!match && !!zombieMapBySlug(match[1], match[2]);
});
$effect(() => { search = $page.url.searchParams.get('server') ?? ''; limit = Math.max(25, Number($page.url.searchParams.get('limit')) || 25); });
function saveSearch() { const url = new URL($page.url); if (search) url.searchParams.set('server', search); else url.searchParams.delete('server'); if (limit > 25) url.searchParams.set('limit', String(limit)); else url.searchParams.delete('limit'); void goto(url, { replaceState: true, keepFocus: true, noScroll: true }); }
let plutoKey = $state(''); let plutoKey = $state('');
let busy = $state(false); let busy = $state(false);
let loading = $state(true); let loading = $state(true);
let error = $state(''); let error = $state('');
let notice = $state(''); let notice = $state('');
const filtered = $derived(keys.filter(key => (key.server_id + ' ' + serverTitle(key.server_id)).toLowerCase().includes(search.toLowerCase()))); $effect(() => {
search = $page.url.searchParams.get('search') ?? $page.url.searchParams.get('server') ?? '';
gameFilter = $page.url.searchParams.get('game') ?? '';
limit = Math.max(25, Number($page.url.searchParams.get('limit')) || 25);
});
const filtered = $derived(keys.filter(key => (!gameFilter || key.game === gameFilter)
&& [key.game, key.label, ...key.assigned_servers.flatMap(name => [name, serverTitle(name)])].join(' ').toLowerCase().includes(search.toLowerCase())));
function saveSearch() {
const url = new URL($page.url); url.searchParams.delete('server');
if (search) url.searchParams.set('search', search); else url.searchParams.delete('search');
if (gameFilter) url.searchParams.set('game', gameFilter); else url.searchParams.delete('game');
if (limit > 25) url.searchParams.set('limit', String(limit)); else url.searchParams.delete('limit');
void goto(url, { replaceState: true, keepFocus: true, noScroll: true });
}
async function load() { async function load() {
const res = await fetch('/api/server-keys'); const res = await fetch('/api/server-keys');
if (!res.ok) throw new Error('Could not load server keys'); if (!res.ok) throw new Error('Could not load server keys');
@@ -47,39 +50,36 @@
} }
async function reload() { try { await load(); error = ''; } catch (e) { error = (e as Error).message; } finally { loading = false; } } async function reload() { try { await load(); error = ''; } catch (e) { error = (e as Error).message; } finally { loading = false; } }
onMount(() => { void reload(); }); onMount(() => { void reload(); });
function reset() { serverId = ''; plutoKey = ''; editing = false; rawMode = false; }
function addKey() { function addKey() {
reset(); error = ''; plutoKey = ''; label = ''; error = '';
const match = /^(t5|t6)_zombie_([a-z0-9]+)_([1-9]|1[0-9]|20)$/.exec($page.url.searchParams.get('server') ?? ''); if (gameFilter === 't5' || gameFilter === 't6') game = gameFilter;
if (match && zombieMapBySlug(match[1], match[2])) { game = match[1] as ZombieGame; mapSlug = match[2]; lobby = Number(match[3]); }
open = true; open = true;
} }
function savedAt(value: string) { return $timeFormat.dateTime(value); }
async function save(e: SubmitEvent) { async function save(e: SubmitEvent) {
e.preventDefault(); e.preventDefault();
if (!validId || !plutoKey.trim() || busy) return; if (!plutoKey.trim() || busy) return;
busy = true; error = ''; notice = ''; busy = true; error = ''; notice = '';
try { try {
const res = await fetch('/api/server-keys', { const res = await fetch('/api/server-keys', {
method: 'PUT', headers: { 'Content-Type': 'application/json' }, method: 'PUT', headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ serverId: chosenId, plutoKey }) body: JSON.stringify({ game, label, plutoKey })
}); });
const body = await res.json(); const body = await res.json();
if (!res.ok) throw new Error(body.error ?? 'Could not save key'); if (!res.ok) throw new Error(body.error ?? 'Could not save key');
reset(); open = false; notice = 'Server key saved.'; await load(); plutoKey = ''; label = ''; open = false; notice = 'Key saved to the game pool.'; await load();
} catch (e) { error = (e as Error).message; } } catch (e) { error = (e as Error).message; }
finally { busy = false; } finally { busy = false; }
} }
async function remove(id: string) { async function remove(key: SavedKey) {
if (!await confirmAction({title:'Remove saved key?',message:`Remove the key for ${serverTitle(id)} (${id})? Running servers are unaffected.`,confirmLabel:'Remove key',danger:true})) return; if (key.assigned_servers.length) return;
if (!await confirmAction({ title: 'Remove saved key?', message: `Remove ${key.label || 'this key'} from the ${key.game.toUpperCase()} pool?`, confirmLabel: 'Remove key', danger: true })) return;
busy = true; error = ''; notice = ''; busy = true; error = ''; notice = '';
try { try {
const res = await fetch('/api/server-keys', { const res = await fetch('/api/server-keys', {
method: 'DELETE', headers: { 'Content-Type': 'application/json' }, method: 'DELETE', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ id: key.id })
body: JSON.stringify({ serverId: id })
}); });
if (!res.ok) throw new Error('Could not remove key'); const body = await res.json();
if (serverId === id) reset(); if (!res.ok) throw new Error(body.error ?? 'Could not remove key');
notice = 'Saved key removed.'; await load(); notice = 'Saved key removed.'; await load();
} catch (e) { error = (e as Error).message; } } catch (e) { error = (e as Error).message; }
finally { busy = false; } finally { busy = false; }
@@ -88,72 +88,70 @@
<svelte:head><title>Keys · Electric Cherry</title></svelte:head> <svelte:head><title>Keys · Electric Cherry</title></svelte:head>
<div class="page"> <div class="page">
<SectionHeader title="Keys" subtitle="Save a Plutonium key for each server ID. Keys are kept when a server is destroyed."> <SectionHeader title="Keys" subtitle="Pool Plutonium keys by game. Any available key can launch any map or lobby in that game.">
{#snippet actions()}<Button variant="primary" onclick={addKey}>Add key</Button>{/snippet} {#snippet actions()}<Button variant="primary" onclick={addKey}>Add key</Button>{/snippet}
</SectionHeader> </SectionHeader>
<p class="key-help">Saved keys are reused when a server is created or rebuilt. Electric Cherry does not check whether Plutonium accepts them. Updating or removing a saved key does not change an existing server.</p> <p class="key-help">A key stays assigned while its server exists, including when stopped, and becomes available after removal is confirmed. Rebuilds keep the same key. Use the optional label to record its Plutonium hostname; saving a label does not change that hostname.</p>
<div class="ui-toolbar"><input type="search" aria-label="Search keys" bind:value={search} oninput={event => { search = event.currentTarget.value; limit = 25; saveSearch(); }} placeholder="Search servers…" /></div> {#if !loading}<div class="pools" aria-label="Game key pools">{#each games as poolGame}
{@const pool = keys.filter(key => key.game === poolGame)}
<div><strong>{poolGame === 't5' ? 'Black Ops I' : 'Black Ops II'}</strong><span>{pool.filter(key => !key.assigned_servers.length).length} available · {pool.filter(key => key.assigned_servers.length).length} assigned · {pool.length} total</span></div>
{/each}</div>{/if}
<div class="ui-toolbar">
<input type="search" aria-label="Search keys" bind:value={search} oninput={event => { search = event.currentTarget.value; limit = 25; saveSearch(); }} placeholder="Search labels or assigned servers…" />
<select aria-label="Filter keys by game" bind:value={gameFilter} onchange={() => { limit = 25; saveSearch(); }}><option value="">All games</option><option value="t5">Black Ops I</option><option value="t6">Black Ops II</option></select>
</div>
{#if error && !open}<div role="alert" class="ui-error">{error}<Button size="sm" onclick={reload}>Retry</Button></div>{/if} {#if error && !open}<div role="alert" class="ui-error">{error}<Button size="sm" onclick={reload}>Retry</Button></div>{/if}
{#if notice}<p role="status">{notice}</p>{/if} {#if notice}<p role="status">{notice}</p>{/if}
<h2>Saved keys</h2> <h2>Saved keys</h2>
{#if loading}<p>Loading keys…</p> {#if loading}<p>Loading keys…</p>
{:else if !filtered.length}<EmptyState title={keys.length ? "No matching keys" : "No saved keys"} description="Add a key here or save one when creating a server." /> {:else if !filtered.length}<EmptyState title={keys.length ? "No matching keys" : "No saved keys"} description="Add a key to a game pool here or when launching a server." />
{:else} {:else}
<div class="table-wrap"><table> <div class="table-wrap"><table>
<thead><tr><th>Server ID</th><th>Server</th><th>Last saved</th><th>Actions</th></tr></thead> <thead><tr><th>Game / label</th><th>Availability</th><th>Last saved</th><th>Actions</th></tr></thead>
<tbody>{#each filtered.slice(0,limit) as key (key.server_id)} <tbody>{#each filtered.slice(0, limit) as key (key.id)}
<tr> <tr>
<td>{#if key.deployed}<a href="/servers/{encodeURIComponent(key.server_id)}"><ServerIdentity name={key.server_id} /></a>{:else}<ServerIdentity name={key.server_id} />{/if}<code>{key.server_id}</code></td> <td><strong>{key.game.toUpperCase()}</strong><span class="key-label">{key.label || 'Unlabeled key'}</span></td>
<td>{key.deployed ? 'Server exists' : 'No server deployed'}</td> <td>{#if key.assigned_servers.length}<span>Assigned</span>{#each key.assigned_servers as name}<div class="assignment"><ServerIdentity {name} /><code>{name}</code></div>{/each}{:else}<span class="available">Available</span>{/if}</td>
<td>{savedAt(key.updated_at)}</td> <td>{$timeFormat.dateTime(key.updated_at)}</td>
<td><div class="actions"> <td><Button ariaLabel={`Remove ${key.label || key.game.toUpperCase() + ' key'}`} disabled={busy || !!key.assigned_servers.length} onclick={() => remove(key)}>Remove</Button>{#if key.assigned_servers.length}<small>In use</small>{/if}</td>
<Button ariaLabel={`Update key for ${serverTitle(key.server_id)}`} disabled={busy} onclick={() => { serverId = key.server_id; plutoKey = ''; editing = true; error = ''; notice = ''; open = true; }}>Update</Button>
<Button ariaLabel={`Remove key for ${serverTitle(key.server_id)}`} disabled={busy} onclick={() => remove(key.server_id)}>Remove</Button>
</div></td>
</tr> </tr>
{/each}</tbody> {/each}</tbody>
</table></div> </table></div>
{/if} {/if}
<ListFooter total={filtered.length} bind:limit onchange={saveSearch} label="keys" /> <ListFooter total={filtered.length} bind:limit onchange={saveSearch} label="keys" />
</div> </div>
<Dialog bind:open title={editing ? 'Update key' : 'Add key'} busy={busy}> <Dialog bind:open title="Add key" busy={busy}>
{#if error}<p role="alert" class="error">{error}</p>{/if} {#if error}<p role="alert" class="error">{error}</p>{/if}
<form onsubmit={save}> <form onsubmit={save}>
{#if !editing}<label class="raw-toggle"><input type="checkbox" bind:checked={rawMode} disabled={busy} onchange={() => { if (!rawMode) serverId = `${game}_zombie_${mapSlug}_${lobby}`; }} /> Enter a raw server ID</label>{/if} <label>Game<select bind:value={game} disabled={busy}><option value="t5">Black Ops I (T5)</option><option value="t6">Black Ops II (T6)</option></select></label>
{#if editing || rawMode}<Field label="Server ID" hint="Use a T5 or T6 map slug and lobby number from 1 to 20, e.g. t5_zombie_theater_2 or t6_zombie_tomb_1." error={!validId && serverId ? 'Choose a recognized map and lobby number 1–20.' : null}> <Field label="Label (optional)" hint="Record the hostname you chose in Plutonium, or another name that helps identify this key.">
<Input bind:value={serverId} placeholder="t5_zombie_theater_2" required readonly={editing} disabled={busy} mono /> <Input bind:value={label} placeholder="e.g. Electric Cherry 5" disabled={busy} />
</Field>{:else}<div class="id-picker"><label>Game<select bind:value={game} disabled={busy} onchange={event => mapSlug = zombieMaps(event.currentTarget.value)[0].slug}><option value="t5">Black Ops I</option><option value="t6">Black Ops II</option></select></label><label>Map<select bind:value={mapSlug} disabled={busy}>{#each zombieMaps(game) as map}<option value={map.slug}>{map.friendly}</option>{/each}</select></label><label>Lobby<input type="number" min="1" max="20" step="1" bind:value={lobby} disabled={busy} required /></label></div><p>Server ID: <code>{chosenId}</code></p>{/if} </Field>
{#if !editing && keys.some(key => key.server_id === chosenId)}<p role="status">A key is already saved for this ID. Saving replaces it for the next deployment; the running server keeps its current key.</p>{/if} <Field label="Plutonium server key" hint="Available to any new lobby in this game. Existing credentials are never displayed.">
<Field label="Plutonium server key" hint="Used when this server ID is created or rebuilt. Existing servers keep their current key.">
<Input type="password" bind:value={plutoKey} placeholder="Paste key" autocomplete="off" required disabled={busy} /> <Input type="password" bind:value={plutoKey} placeholder="Paste key" autocomplete="off" required disabled={busy} />
</Field> </Field>
<div class="actions"> <div class="actions"><Button type="submit" variant="primary" disabled={busy || loading || !plutoKey.trim()} loading={busy}>Save key</Button><Button onclick={() => open = false} disabled={busy}>Cancel</Button></div>
<Button type="submit" variant="primary" disabled={busy || loading || !validId || !plutoKey.trim()} loading={busy}>{editing ? 'Update key' : 'Save key'}</Button>
<Button onclick={() => open = false} disabled={busy}>Cancel</Button>
</div>
</form> </form>
{#if !validId || !plutoKey.trim()}<p role="status">Choose a valid server ID and enter its Plutonium key to save.</p>{/if}
</Dialog> </Dialog>
<style> <style>
a { color: var(--accent-primary-hover); } .page { display: grid; gap: 24px; }
.key-help { margin: 0; font-size: var(--text-sm); line-height: 1.65; } .key-help { margin: 0; font-size: var(--text-sm); line-height: 1.65; }
.id-picker { display: grid; grid-template-columns: 1fr 1.4fr 80px; gap: 12px; }.id-picker label { display: grid; gap: 8px; font-size: 14px; }.id-picker input, .id-picker select { width: 100%; min-width: 0; padding: 10px; border: 1px solid var(--border-control); border-radius: var(--radius-md); background: var(--surface-sunken); color: inherit; }.raw-toggle { display: flex; gap: 8px; align-items: center; font-size: 14px; } .pools { display: grid; grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); gap: 16px; }
@media (max-width: 600px) { .id-picker { grid-template-columns: 1fr; } } .pools > div { display: grid; gap: 8px; padding: 16px; border: 1px solid var(--border-subtle); border-radius: var(--radius-md); }
.pools span, .key-label, small { color: var(--text-secondary); font-size: var(--text-sm); }
.key-label, small { display: block; margin-top: 6px; }
.assignment { margin-top: 8px; }
.available { color: var(--state-online); }
h2 { font-size: var(--text-md); } h2 { font-size: var(--text-md); }
p { color: var(--text-secondary); } p { color: var(--text-secondary); }
form { display: grid; gap: var(--space-4); max-width: 620px; padding: var(--space-6); background: var(--surface); border: 1px solid var(--border-subtle); border-radius: var(--radius-lg); margin: var(--space-6) 0; } form, label { display: grid; gap: var(--space-4); }
label { gap: 8px; font-size: 14px; }
select { padding: 10px; border: 1px solid var(--border-control); border-radius: var(--radius-md); background: var(--surface-sunken); color: inherit; }
.actions { display: flex; gap: var(--space-2); } .actions { display: flex; gap: var(--space-2); }
.table-wrap { overflow-x: auto; } .table-wrap { overflow-x: auto; }
table { width: 100%; border-collapse: collapse; text-align: left; } table { width: 100%; border-collapse: collapse; text-align: left; }
th, td { padding: var(--space-4); border-bottom: 1px solid var(--border-subtle); } th, td { padding: var(--space-4); border-bottom: 1px solid var(--border-subtle); }
th { color: var(--text-tertiary); font-size: var(--text-sm); } th { color: var(--text-tertiary); font-size: var(--text-sm); }
code { font-family: var(--font-mono); } code { display: block; font-family: var(--font-mono); font-size: 12px; color: var(--text-secondary); margin-top: 6px; }
.error { color: var(--state-danger); } .error { color: var(--state-danger); }
.page { display:grid; gap:24px; }
form { margin:0; padding:0; border:0; }
code { display:block; font-size:12px; color:var(--text-secondary); margin-top:6px; }
</style> </style>
+2 -2
View File
@@ -10,7 +10,7 @@ async function setup(mode = 'dry-run') {
if (mode.startsWith('create')) await prepareNativeHost(); if (mode.startsWith('create')) await prepareNativeHost();
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const scaling = await import('../../src/lib/server/autoscaling/index.js'); const scaling = await import('../../src/lib/server/autoscaling/index.js');
db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run('t6_zombie_town_1', 'test-key'); db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('key-1','t6','test-key')").run();
const input = { ...settings, mode }; const input = { ...settings, mode };
const saved = scaling.saveScaling(input, undefined, undefined, scaling.previewScaling(input).token); const saved = scaling.saveScaling(input, undefined, undefined, scaling.previewScaling(input).token);
return { db, scaling, saved, input }; return { db, scaling, saved, input };
@@ -20,7 +20,7 @@ it('deletes policy evidence atomically while preserving lobbies, keys, and unrel
const { db, scaling, saved } = await setup(); const { db, scaling, saved } = await setup();
db().prepare("INSERT INTO servers(name,game,map_slug,map) VALUES('t6_zombie_town_1','t6','town','zm_transit')").run(); db().prepare("INSERT INTO servers(name,game,map_slug,map) VALUES('t6_zombie_town_1','t6','town','zm_transit')").run();
await scaling.runAutoscaling(); await scaling.runAutoscaling();
db().prepare("INSERT INTO server_keys(server_id,pluto_key) VALUES('t5_zombie_theater_1','other-key')").run(); db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('other-key','t5','other-key')").run();
const otherSettings = { ...settings, game: 't5', mapSlug: 'theater', mode: 'off' }; const otherSettings = { ...settings, game: 't5', mapSlug: 'theater', mode: 'off' };
const other = scaling.saveScaling(otherSettings, undefined, undefined, scaling.previewScaling(otherSettings).token); const other = scaling.saveScaling(otherSettings, undefined, undefined, scaling.previewScaling(otherSettings).token);
for (const table of ['autoscaling_decisions', 'autoscaling_members', 'autoscaling_samples']) expect(db().prepare(`SELECT COUNT(*) AS n FROM ${table} WHERE policy_id=?`).get(saved.id)).toEqual({ n: 1 }); for (const table of ['autoscaling_decisions', 'autoscaling_members', 'autoscaling_samples']) expect(db().prepare(`SELECT COUNT(*) AS n FROM ${table} WHERE policy_id=?`).get(saved.id)).toEqual({ n: 1 });
+6 -4
View File
@@ -6,7 +6,7 @@ let fixture: Fixture;
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); fixture = await newFixture([]); vi.resetModules(); fixture = await newFixture([]);
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
for (const n of [3, 8, 9, 10]) db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_town_${n}`, `secret-${n}`); for (const n of [3, 8, 9, 10]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `secret-${n}`);
}); });
afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); }); afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); });
const settings = { game: 't6', mapSlug: 'town', maximum: 4, mode: 'dry-run' }; const settings = { game: 't6', mapSlug: 'town', maximum: 4, mode: 'dry-run' };
@@ -22,7 +22,7 @@ it('persists deduplicated dry-run decisions across restart without provisioning
const view = scaling.scalingReport().policies[0]; const view = scaling.scalingReport().policies[0];
expect(view.policy.id).toBe(saved.id); expect(view.policy.id).toBe(saved.id);
expect(view.history).toHaveLength(1); expect(view.history).toHaveLength(1);
expect(view.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_3' }); expect(view.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_1' });
expect(JSON.stringify(view)).not.toContain('secret'); expect(JSON.stringify(view)).not.toContain('secret');
for (const table of ['server_deployments', 'native_creation_intents', 'native_instances']) expect(db().prepare(`SELECT COUNT(*) AS n FROM ${table}`).get()).toEqual({ n: 0 }); for (const table of ['server_deployments', 'native_creation_intents', 'native_instances']) expect(db().prepare(`SELECT COUNT(*) AS n FROM ${table}`).get()).toEqual({ n: 0 });
}); });
@@ -35,8 +35,10 @@ it('rejects duplicate policies, stale edits and changed previews', async () => {
expect(() => scaling.previewScaling(settings, saved.id, 0)).toThrow('changed'); expect(() => scaling.previewScaling(settings, saved.id, 0)).toThrow('changed');
expect(() => scaling.saveScaling({ ...settings, maximum: 5 }, saved.id, saved.revision, preview.token)).toThrow('preview changed'); expect(() => scaling.saveScaling({ ...settings, maximum: 5 }, saved.id, saved.revision, preview.token)).toThrow('preview changed');
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_10'").run(); const beforeKeyLoss = scaling.previewScaling(settings, saved.id, saved.revision);
expect(() => scaling.saveScaling({ ...settings, mode: 'create-destroy' }, saved.id, saved.revision, scaling.previewScaling({ ...settings, mode: 'create-destroy' }, saved.id, saved.revision).token)).toThrow('Save keys'); db().prepare("DELETE FROM server_keys WHERE id='key-10'").run();
expect(() => scaling.saveScaling(settings, saved.id, saved.revision, beforeKeyLoss.token)).toThrow('preview changed');
expect(scaling.saveScaling(settings, saved.id, saved.revision, scaling.previewScaling(settings, saved.id, saved.revision).token).maximum).toBe(4);
}); });
it('pauses evaluations and prunes expired observations without deleting current history', async () => { it('pauses evaluations and prunes expired observations without deleting current history', async () => {
+12 -8
View File
@@ -14,7 +14,7 @@ async function setup() {
await prepareNativeHost(); await prepareNativeHost();
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const scaling = await import('../../src/lib/server/autoscaling/index.js'); const scaling = await import('../../src/lib/server/autoscaling/index.js');
for (const map of maps) for (const n of [1, 2]) db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_${map.slug}_${n}`, `key-${map.slug}-${n}`); db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('key-1','t6','game-key')").run();
return { db, scaling }; return { db, scaling };
} }
@@ -24,7 +24,7 @@ it('applies targets and spare deficits independently to every map', () => {
const members: ScalingMember[] = maps.flatMap(map => [1, 2].map(n => ({ name: `t6_zombie_${map.slug}_${n}`, mapSlug: map.slug, const members: ScalingMember[] = maps.flatMap(map => [1, 2].map(n => ({ name: `t6_zombie_${map.slug}_${n}`, mapSlug: map.slug,
identity: `${map.slug}:${n}`, state: map.slug === 'town' ? 'occupied' : 'empty', players: map.slug === 'town' ? 1 : 0, identity: `${map.slug}:${n}`, state: map.slug === 'town' ? 'occupied' : 'empty', players: map.slug === 'town' ? 1 : 0,
observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null }))); observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null })));
const snapshot = { members, reservedNames: members.map(m => m.name), keyedNames: members.map(m => m.name), blockers: [] }; const snapshot = { members, reservedNames: members.map(m => m.name), keyPool: { total: 15, available: 1 }, blockers: [] };
// Other maps' empty lobbies cannot satisfy Town's spare target or maximum. // Other maps' empty lobbies cannot satisfy Town's spare target or maximum.
expect(planScaling(policy, snapshot, now)).toMatchObject({ action: 'destroy', server: 't6_zombie_transit_1', desired: maps.length + 1, deficit: 1 }); expect(planScaling(policy, snapshot, now)).toMatchObject({ action: 'destroy', server: 't6_zombie_transit_1', desired: maps.length + 1, deficit: 1 });
snapshot.members = members.filter(m => m.name !== 't6_zombie_town_2'); snapshot.members = members.filter(m => m.name !== 't6_zombie_town_2');
@@ -37,9 +37,9 @@ it('applies targets and spare deficits independently to every map', () => {
it('persists a game rule, queues the selected map recipe, and protects every map', async () => { it('persists a game rule, queues the selected map recipe, and protects every map', async () => {
const { db, scaling } = await setup(); const { db, scaling } = await setup();
db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run('t5_zombie_theater_1', 'other-game'); db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('other-key','t5','other-game')").run();
const preview = scaling.previewScaling(settings); const preview = scaling.previewScaling(settings);
expect(preview.snapshot.keyedNames).toHaveLength(maps.length * 2); expect(preview.snapshot.keyPool).toEqual({ total: 1, available: 1 });
const saved = scaling.saveScaling(settings, undefined, undefined, preview.token); const saved = scaling.saveScaling(settings, undefined, undefined, preview.token);
expect(() => scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token)).toThrow(); expect(() => scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token)).toThrow();
expect(() => scaling.previewScaling({ ...settings, mapSlug: 'town' }, saved.id, saved.revision)).toThrow('cannot change'); expect(() => scaling.previewScaling({ ...settings, mapSlug: 'town' }, saved.id, saved.revision)).toThrow('cannot change');
@@ -64,10 +64,14 @@ it('persists a game rule, queues the selected map recipe, and protects every map
expect(scaling.previewScaling(settings, saved.id, current.revision).blockers.join(' ')).toContain(mapPolicy.id); expect(scaling.previewScaling(settings, saved.id, current.revision).blockers.join(' ')).toContain(mapPolicy.id);
}); });
it('requires live key coverage on each map even when the game has enough keys in total', async () => { it('holds game-wide growth when its shared pool is exhausted by another map', async () => {
const { db, scaling } = await setup(); const { db, scaling } = await setup();
db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_2'").run(); db().prepare("INSERT INTO server_key_leases(server_id,key_id) VALUES('t6_zombie_town_2','key-1')").run();
const preview = scaling.previewScaling(settings); const preview = scaling.previewScaling(settings);
expect(preview.blockers.join(' ')).toContain('Town Survival'); expect(preview.snapshot.keyPool).toEqual({ total: 1, available: 0 });
expect(() => scaling.saveScaling(settings, undefined, undefined, preview.token)).toThrow('Town Survival'); expect(preview.blockers).toEqual([]);
expect(preview.decision).toMatchObject({ action: 'wait', reason: expect.stringContaining('keys are in use') });
scaling.saveScaling(settings, undefined, undefined, preview.token);
expect(await scaling.runAutoscaling()).toBe(false);
expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 0 });
}); });
+73 -57
View File
@@ -1,7 +1,6 @@
import { afterEach, beforeEach, expect, it, vi } from 'vitest'; import { afterEach, beforeEach, expect, it, vi } from 'vitest';
import { newFixture, type Fixture } from '../helpers/fixture.js'; import { newFixture, type Fixture } from '../helpers/fixture.js';
import { prepareNativeHost } from '../helpers/native-host.js'; import { prepareNativeHost } from '../helpers/native-host.js';
import { zombieMaps } from '../../src/lib/map-aliases.js';
let fixture: Fixture; let fixture: Fixture;
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); }); beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
@@ -10,73 +9,90 @@ const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'dry-run' };
async function setup() { async function setup() {
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const scaling = await import('../../src/lib/server/autoscaling/index.js'); const scaling = await import('../../src/lib/server/autoscaling/index.js');
const key = (name: string, value = `key-${name}`) => db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(name, value); const { saveServerKey, reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
return { db, scaling, key }; return { db, scaling, saveServerKey, reserveServerKey };
} }
it.each(['off', 'dry-run', 'create-only', 'create-destroy'])('rejects new %s policies without enough keys through both service and API', async mode => { it.each(['off', 'dry-run', 'create-only', 'create-destroy'])('saves a %s policy without preallocating keys for its maximum', async mode => {
const { db, scaling, key } = await setup(); if (mode.startsWith('create')) await prepareNativeHost();
key('t6_zombie_town_3'); const { db, scaling } = await setup();
const input = { ...settings, mode }; const input = { ...settings, mode, maximum: 20 };
const preview = scaling.previewScaling(input); const preview = scaling.previewScaling(input);
expect(preview.blockers[0]).toContain('1 saved; 1 more required'); expect(preview.blockers).toEqual([]);
expect(() => scaling.saveScaling(input, undefined, undefined, preview.token)).toThrow('Save keys'); expect(preview.decision.action).toBe('wait');
if (mode !== 'off') expect(preview.decision.reason).toContain('No Plutonium keys');
const { POST } = await import('../../src/routes/api/autoscaling/+server.js'); const { POST } = await import('../../src/routes/api/autoscaling/+server.js');
const response = await POST({ request: new Request('http://localhost/api/autoscaling', { method: 'POST', body: JSON.stringify({ action: 'save', settings: input, token: preview.token }) }) }); const response = await POST({ request: new Request('http://localhost/api/autoscaling', { method: 'POST', body: JSON.stringify({ action: 'save', settings: input, token: preview.token }) }) });
expect(response.status).toBe(400); expect(response.status).toBe(200);
expect(await response.json()).toMatchObject({ error: expect.stringContaining('Save keys') }); expect(db().prepare('SELECT COUNT(*) AS n FROM autoscaling_policies').get()).toEqual({ n: 1 });
expect(db().prepare('SELECT COUNT(*) AS n FROM autoscaling_policies').get()).toEqual({ n: 0 }); expect(await scaling.runAutoscaling()).toBe(false);
}); });
it('counts only supported, nonblank identities in the selected map and accepts exact nonconsecutive coverage', async () => { it('uses any free key for the selected game independently of its label or map', async () => {
const { scaling, key } = await setup(); const { scaling, saveServerKey } = await setup();
for (const name of ['t6_zombie_town_0', 't6_zombie_town_21', 't6_zombie_town_01', 't6_zombie_town', 't6_town_2', 't5_zombie_theater_1', 't6_zombie_farm_1']) key(name); saveServerKey('t5', 'other-game-key', 'Town Lobby 1');
key('t6_zombie_town_2', ' '); saveServerKey('t6', 'game-key', 'Farm Lobby 8');
key('t6_zombie_town_3');
expect(scaling.previewScaling(settings).snapshot.keyedNames).toEqual(['t6_zombie_town_3']);
key('t6_zombie_town_8');
const preview = scaling.previewScaling(settings); const preview = scaling.previewScaling(settings);
expect(preview.blockers).toEqual([]); expect(preview.snapshot.keyPool).toEqual({ total: 1, available: 1 });
expect(preview.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_1' });
expect(scaling.saveScaling(settings, undefined, undefined, preview.token).maximum).toBe(2); expect(scaling.saveScaling(settings, undefined, undefined, preview.token).maximum).toBe(2);
}); });
it('requires coverage on every game map rather than pooling excess keys from another map', async () => { it('uses one shared pool for every map without requiring a key for each maximum', async () => {
const { scaling, key } = await setup(); const { scaling, saveServerKey } = await setup();
for (const map of zombieMaps('t6')) for (const n of [3, 8]) if (map.slug !== 'town' || n === 3) key(`t6_zombie_${map.slug}_${n}`); saveServerKey('t6', 'game-key');
key('t6_zombie_farm_9'); const input = { ...settings, mapSlug: '*', maximum: 20 };
const input = { ...settings, mapSlug: '*' };
const preview = scaling.previewScaling(input); const preview = scaling.previewScaling(input);
expect(preview.blockers).toEqual([expect.stringContaining('Town Survival')]);
expect(() => scaling.saveScaling(input, undefined, undefined, preview.token)).toThrow('Town Survival');
key('t6_zombie_town_8');
expect(scaling.saveScaling(input, undefined, undefined, scaling.previewScaling(input).token).mapSlug).toBe('*');
});
it('rechecks key deletion after preview and rejects increases beyond coverage', async () => {
const { db, scaling, key } = await setup();
key('t6_zombie_town_3'); key('t6_zombie_town_8');
const preview = scaling.previewScaling(settings);
db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_8'").run();
expect(() => scaling.saveScaling(settings, undefined, undefined, preview.token)).toThrow('preview changed');
expect(() => scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token)).toThrow('Save keys');
key('t6_zombie_town_8');
const saved = scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token);
const higher = { ...settings, maximum: 3 };
expect(() => scaling.saveScaling(higher, saved.id, saved.revision, scaling.previewScaling(higher, saved.id, saved.revision).token)).toThrow('Save keys');
expect(scaling.getScalingPolicy(saved.id)?.maximum).toBe(2);
});
it('allows an existing live policy to pause and withdraw pending work after all keys are removed', async () => {
await prepareNativeHost();
const { db, scaling, key } = await setup();
key('t6_zombie_town_3'); key('t6_zombie_town_8');
const live = { ...settings, mode: 'create-only' };
const saved = scaling.saveScaling(live, undefined, undefined, scaling.previewScaling(live).token);
await scaling.runAutoscaling();
db().prepare('DELETE FROM server_keys').run();
const off = { ...settings, mode: 'off' };
const preview = scaling.previewScaling(off, saved.id, saved.revision);
expect(preview.blockers).toEqual([]); expect(preview.blockers).toEqual([]);
expect(scaling.saveScaling(off, saved.id, saved.revision, preview.token).mode).toBe('off'); expect(preview.snapshot.keyPool).toEqual({ total: 1, available: 1 });
expect(db().prepare("SELECT state FROM server_deployments WHERE action='create'").get()).toEqual({ state: 'cancelled' }); expect(scaling.saveScaling(input, undefined, undefined, preview.token).mapSlug).toBe('*');
});
it('rechecks game pool deletion and reservations after preview', async () => {
const { db, scaling, saveServerKey, reserveServerKey } = await setup();
const keyId = saveServerKey('t6', 'game-key');
const preview = scaling.previewScaling(settings);
db().prepare('DELETE FROM server_keys WHERE id=?').run(keyId);
expect(() => scaling.saveScaling(settings, undefined, undefined, preview.token)).toThrow('preview changed');
saveServerKey('t6', 'game-key');
const available = scaling.previewScaling(settings);
reserveServerKey('t6_zombie_farm_1', 't6');
expect(() => scaling.saveScaling(settings, undefined, undefined, available.token)).toThrow('preview changed');
const held = scaling.previewScaling(settings);
expect(held.decision.reason).toContain('keys are in use');
expect(held.snapshot.members).toEqual([]);
expect(scaling.saveScaling(settings, undefined, undefined, held.token).maximum).toBe(2);
});
it('shares capacity between live map policies and admits the job holding the last key', async () => {
await prepareNativeHost();
const { db, scaling, saveServerKey } = await setup();
saveServerKey('t6', 'game-key');
saveServerKey('t5', 'other-game-key');
for (const mapSlug of ['town', 'farm']) {
const input = { ...settings, mapSlug, mode: 'create-only' };
scaling.saveScaling(input, undefined, undefined, scaling.previewScaling(input).token);
}
expect(await scaling.runAutoscaling()).toBe(true);
const jobs = db().prepare('SELECT deployment_id,server FROM server_creation_jobs').all() as { deployment_id: string; server: string }[];
expect(jobs).toHaveLength(1);
expect(jobs[0].server).toBe('t6_zombie_farm_1');
const { scalingAdmission } = await import('../../src/lib/server/autoscaling/admission.js');
expect(scalingAdmission(jobs[0].deployment_id)).toBeNull();
const town = scaling.scalingReport().policies.find(p => p.policy.mapSlug === 'town')!;
expect(town.snapshot.keyPool).toEqual({ total: 1, available: 0 });
expect(town.decision.reason).toContain('keys are in use');
expect(town.policy.pausedReason).toBeNull();
expect(await scaling.runAutoscaling()).toBe(false);
expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 1 });
});
it('reserves a lobby identity for an unsettled failed launch with a retained key', async () => {
const { scaling, saveServerKey, reserveServerKey } = await setup();
saveServerKey('t6', 'held-key');
reserveServerKey('t6_zombie_town_1', 't6');
saveServerKey('t6', 'free-key');
const preview = scaling.previewScaling(settings);
expect(preview.snapshot.reservedNames).toEqual(['t6_zombie_town_1']);
expect(preview.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_2' });
}); });
+11 -9
View File
@@ -10,7 +10,7 @@ async function setup() {
await prepareNativeHost(); await prepareNativeHost();
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const scaling = await import('../../src/lib/server/autoscaling/index.js'); const scaling = await import('../../src/lib/server/autoscaling/index.js');
for (const n of [3, 8]) db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_town_${n}`, `key-${n}`); for (const n of [3, 8]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `key-${n}`);
const saved = scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token); const saved = scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token);
return { db, scaling, saved }; return { db, scaling, saved };
} }
@@ -21,7 +21,7 @@ it('admits one durable keyed create and reuses it across ticks and restart', asy
await Promise.all([scaling.runAutoscaling(), scaling.runAutoscaling()]); await Promise.all([scaling.runAutoscaling(), scaling.runAutoscaling()]);
const { closeDb } = await import('../../src/lib/server/db/index.js'); closeDb(); const { closeDb } = await import('../../src/lib/server/db/index.js'); closeDb();
await scaling.runAutoscaling(); await scaling.runAutoscaling();
expect(db().prepare('SELECT server,state FROM server_deployments WHERE action=\'create\'').all()).toEqual([{ server: 't6_zombie_town_3', state: 'queued' }]); expect(db().prepare('SELECT server,state FROM server_deployments WHERE action=\'create\'').all()).toEqual([{ server: 't6_zombie_town_1', state: 'queued' }]);
expect(db().prepare('SELECT COUNT(*) AS n FROM autoscaling_decisions WHERE resolved=0').get()).toEqual({ n: 1 }); expect(db().prepare('SELECT COUNT(*) AS n FROM autoscaling_decisions WHERE resolved=0').get()).toEqual({ n: 1 });
const { operation_id: id } = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string }; const { operation_id: id } = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string };
const { scalingAdmission } = await import('../../src/lib/server/autoscaling/admission.js'); const { scalingAdmission } = await import('../../src/lib/server/autoscaling/admission.js');
@@ -30,16 +30,19 @@ it('admits one durable keyed create and reuses it across ticks and restart', asy
expect(getDeployment(id)?.sources).toContainEqual(expect.objectContaining({ kind: 'autoscaling', href: `/autoscaling#${saved.id}` })); expect(getDeployment(id)?.sources).toContainEqual(expect.objectContaining({ kind: 'autoscaling', href: `/autoscaling#${saved.id}` }));
}); });
it('requires key coverage and withdraws an unsubmitted create when the policy pauses', async () => { it('withdraws an unsubmitted create and returns its game key when the policy pauses', async () => {
const { db, scaling, saved } = await setup(); const { db, scaling, saved } = await setup();
await scaling.runAutoscaling(); await scaling.runAutoscaling();
const off = { ...settings, mode: 'off' }; const off = { ...settings, mode: 'off' };
scaling.saveScaling(off, saved.id, saved.revision, scaling.previewScaling(off, saved.id, saved.revision).token); scaling.saveScaling(off, saved.id, saved.revision, scaling.previewScaling(off, saved.id, saved.revision).token);
expect(db().prepare("SELECT state FROM server_deployments WHERE action='create'").get()).toEqual({ state: 'cancelled' }); expect(db().prepare("SELECT state FROM server_deployments WHERE action='create'").get()).toEqual({ state: 'cancelled' });
expect(db().prepare('SELECT owns_map FROM autoscaling_policies').get()).toEqual({ owns_map: 0 }); expect(db().prepare('SELECT owns_map FROM autoscaling_policies').get()).toEqual({ owns_map: 0 });
db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_8'").run(); expect(db().prepare('SELECT COUNT(*) AS n FROM server_key_leases').get()).toEqual({ n: 0 });
db().prepare('DELETE FROM server_keys').run();
const current = scaling.getScalingPolicy(saved.id)!; const current = scaling.getScalingPolicy(saved.id)!;
expect(scaling.previewScaling(settings, saved.id, current.revision).blockers.join(' ')).toContain('Save keys'); const preview = scaling.previewScaling(settings, saved.id, current.revision);
expect(preview.blockers).toEqual([]);
expect(preview.snapshot.keyPool).toEqual({ total: 0, available: 0 });
}); });
it('blocks manual lifecycle and collection creation, including stale previews', async () => { it('blocks manual lifecycle and collection creation, including stale previews', async () => {
@@ -78,7 +81,7 @@ it('exact matching cannot remove an owned map absent from the collection', async
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const at = new Date().toISOString(); const at = new Date().toISOString();
db().prepare("INSERT INTO servers(name,game,map,map_slug,container_state,container_checked_at,last_seen_at,last_players) VALUES(?,'t6','zm_transit','town','running',?,?,0)").run(town, at, at); db().prepare("INSERT INTO servers(name,game,map,map_slug,container_state,container_checked_at,last_seen_at,last_players) VALUES(?,'t6','zm_transit','town','running',?,?,0)").run(town, at, at);
for (const n of [1, 2]) db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_town_${n}`, `key-${n}`); for (const n of [1, 2]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `key-${n}`);
const scaling = await import('../../src/lib/server/autoscaling/index.js'); const scaling = await import('../../src/lib/server/autoscaling/index.js');
scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token); scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token);
const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js'); const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js');
@@ -90,13 +93,12 @@ it('exact matching cannot remove an owned map absent from the collection', async
expect(() => queueCollection({ previewId: preview.id })).toThrow('Autoscaling'); expect(() => queueCollection({ previewId: preview.id })).toThrow('Autoscaling');
}); });
it('rechecks evidence and global pause before dispatch', async () => { it('honors global pause before dispatch while recognizing its own reserved key', async () => {
const { db, scaling } = await setup(); const { db, scaling } = await setup();
await scaling.runAutoscaling(); await scaling.runAutoscaling();
const { operation_id: id } = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string }; const { operation_id: id } = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string };
const { scalingAdmission } = await import('../../src/lib/server/autoscaling/admission.js'); const { scalingAdmission } = await import('../../src/lib/server/autoscaling/admission.js');
db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_3'").run(); expect(scalingAdmission(id)).toBeNull();
expect(scalingAdmission(id)).toContain('no longer eligible');
scaling.setScalingGlobalPause(true); scaling.setScalingGlobalPause(true);
expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(id)).toEqual({ state: 'cancelled' }); expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(id)).toEqual({ state: 'cancelled' });
expect(await scaling.runAutoscaling()).toBe(false); expect(await scaling.runAutoscaling()).toBe(false);
+8 -4
View File
@@ -7,7 +7,7 @@ const now = 2_000_000;
const policy: ScalingPolicy = { ...scalingSettings({ game: 't6', mapSlug: 'town', maximum: 4, mode: 'dry-run' }), id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' }; const policy: ScalingPolicy = { ...scalingSettings({ game: 't6', mapSlug: 'town', maximum: 4, mode: 'dry-run' }), id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' };
const name = (n: number) => `t6_zombie_town_${n}`; const name = (n: number) => `t6_zombie_town_${n}`;
const member = (n: number, state: ScalingMember['state'] = 'empty'): ScalingMember => ({ name: name(n), identity: `instance:${n}`, state, players: state === 'occupied' ? 1 : state === 'empty' ? 0 : null, observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null }); const member = (n: number, state: ScalingMember['state'] = 'empty'): ScalingMember => ({ name: name(n), identity: `instance:${n}`, state, players: state === 'occupied' ? 1 : state === 'empty' ? 0 : null, observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null });
const snapshot = (...members: ScalingMember[]): ScalingSnapshot => ({ members, reservedNames: members.map(m => m.name), keyedNames: [1, 2, 3, 4].map(name), blockers: [] }); const snapshot = (...members: ScalingMember[]): ScalingSnapshot => ({ members, reservedNames: members.map(m => m.name), keyPool: { total: 4, available: 4 - members.length }, blockers: [] });
describe('autoscaling policy planning', () => { describe('autoscaling policy planning', () => {
it('requires explicit bounded maximum and valid variant', () => { it('requires explicit bounded maximum and valid variant', () => {
@@ -19,9 +19,13 @@ describe('autoscaling policy planning', () => {
expect(planScaling(policy, snapshot(), now)).toMatchObject({ action: 'create', desired: 1 }); expect(planScaling(policy, snapshot(), now)).toMatchObject({ action: 'create', desired: 1 });
expect(planScaling(policy, snapshot(member(1, 'occupied'), member(2)), now).action).toBe('wait'); expect(planScaling(policy, snapshot(member(1, 'occupied'), member(2)), now).action).toBe('wait');
}); });
it('uses nonconsecutive keyed identities without moving keys', () => { it('selects the first free identity independently of the game key pool size', () => {
expect(planScaling(policy, { ...snapshot(), keyedNames: [name(8), name(3)] }, now).server).toBe(name(3)); expect(planScaling(policy, { ...snapshot(), reservedNames: [name(1), name(3)], keyPool: { total: 1, available: 1 } }, now).server).toBe(name(2));
expect(planScaling(policy, { ...snapshot(), keyedNames: [] }, now).reason).toContain('saved key'); expect(planScaling(policy, { ...snapshot(), keyPool: { total: 0, available: 0 } }, now).reason).toContain('No Plutonium keys');
expect(planScaling(policy, { ...snapshot(), keyPool: { total: 2, available: 0 } }, now).reason).toContain('keys are in use');
});
it('still removes eligible surplus lobbies when the game pool is exhausted', () => {
expect(planScaling(policy, { ...snapshot(member(1), member(2)), keyPool: { total: 2, available: 0 } }, now)).toMatchObject({ action: 'destroy', server: name(1) });
}); });
it('holds for pending creation, unknown, stopped, or protected members', () => { it('holds for pending creation, unknown, stopped, or protected members', () => {
for (const state of ['creating', 'removing', 'unknown', 'stopped', 'protected'] as const) expect(planScaling(policy, snapshot(member(1, state)), now).action).toBe('wait'); for (const state of ['creating', 'removing', 'unknown', 'stopped', 'protected'] as const) expect(planScaling(policy, snapshot(member(1, state)), now).action).toBe('wait');
+2 -2
View File
@@ -22,10 +22,10 @@ async function setup(gameWide = false) {
for (const name of servers) { for (const name of servers) {
const map = zombieMaps('t6').find(m => name === `t6_zombie_${m.slug}_1` || name === `t6_zombie_${m.slug}_2`)!; const map = zombieMaps('t6').find(m => name === `t6_zombie_${m.slug}_1` || name === `t6_zombie_${m.slug}_2`)!;
db().prepare("INSERT INTO servers(name,game,map,map_slug,container_state,last_players) VALUES(?,'t6',?,?,'running',0)").run(name, map.canonical, map.slug); db().prepare("INSERT INTO servers(name,game,map,map_slug,container_state,last_players) VALUES(?,'t6',?,?,'running',0)").run(name, map.canonical, map.slug);
db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(name, `key-${name}`); db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${name}`, `key-${name}`);
} }
if (gameWide) for (const map of zombieMaps('t6').filter(m => m.slug !== 'town')) { if (gameWide) for (const map of zombieMaps('t6').filter(m => m.slug !== 'town')) {
db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_${map.slug}_2`, `key-${map.slug}-2`); db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${map.slug}-2`, `key-${map.slug}-2`);
} }
const fresh = (offset: number) => { const fresh = (offset: number) => {
vi.setSystemTime(base + offset); const at = new Date().toISOString(); vi.setSystemTime(base + offset); const at = new Date().toISOString();
+2 -2
View File
@@ -31,7 +31,7 @@ it('does not create schema mutations or host work when planning an empty map', a
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { scalingSnapshot } = await import('../../src/lib/server/autoscaling/snapshot.js'); const { scalingSnapshot } = await import('../../src/lib/server/autoscaling/snapshot.js');
const { planScaling } = await import('../../src/lib/server/autoscaling/planner.js'); const { planScaling } = await import('../../src/lib/server/autoscaling/planner.js');
db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run('t6_zombie_town_8', 'private-key'); db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('key-8','t6','private-key')").run();
expect(planScaling(policy, scalingSnapshot(policy), Date.now()).server).toBe('t6_zombie_town_8'); expect(planScaling(policy, scalingSnapshot(policy), Date.now()).server).toBe('t6_zombie_town_1');
expect(db().prepare('SELECT COUNT(*) AS n FROM server_deployments').get()).toEqual({ n: 0 }); expect(db().prepare('SELECT COUNT(*) AS n FROM server_deployments').get()).toEqual({ n: 0 });
}); });
+2 -2
View File
@@ -111,11 +111,11 @@ describe('active beta destruction guard', () => {
const { requestLifecycle } = await import('../../src/lib/server/fleet/lifecycle.js'); const { requestLifecycle } = await import('../../src/lib/server/fleet/lifecycle.js');
const { cancelRemoval } = await import('../../src/lib/server/fleet/deployments.js'); const { cancelRemoval } = await import('../../src/lib/server/fleet/deployments.js');
const { cancelProvisioning } = await import('../../src/lib/server/provisioning/requests.js'); const { cancelProvisioning } = await import('../../src/lib/server/provisioning/requests.js');
const { saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); const { reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { createScript } = await import('../../src/lib/server/fleet/scripts.js'); const { createScript } = await import('../../src/lib/server/fleet/scripts.js');
const { openBeta } = await import('../../src/lib/server/fleet/beta.js'); const { openBeta } = await import('../../src/lib/server/fleet/beta.js');
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
saveServerKey(server, 'saved-key'); reserveServerKey(server, 't5', 'saved-key');
const queued = action === 'destroy' ? destroyServer(server) : requestLifecycle(server, 'rebuild'); const queued = action === 'destroy' ? destroyServer(server) : requestLifecycle(server, 'rebuild');
expect(queued.ok).toBe(true); expect(queued.ok).toBe(true);
if (!queued.ok) throw new Error('Expected queued removal'); if (!queued.ok) throw new Error('Expected queued removal');
@@ -117,6 +117,6 @@ describe('large native collection admission', () => {
.toThrow(change === 'newer game files' ? /game files changed/ : /host was not found/); .toThrow(change === 'newer game files' ? /game files changed/ : /host was not found/);
expect(db().prepare('SELECT id FROM server_launch_collections').all()).toEqual([]); expect(db().prepare('SELECT id FROM server_launch_collections').all()).toEqual([]);
expect(db().prepare('SELECT deployment_id FROM native_creation_intents').all()).toEqual([]); expect(db().prepare('SELECT deployment_id FROM native_creation_intents').all()).toEqual([]);
expect(db().prepare('SELECT server_id FROM server_keys').all()).toEqual([]); expect(db().prepare('SELECT id FROM server_keys').all()).toEqual([]);
}); });
}); });
+26 -8
View File
@@ -90,19 +90,36 @@ describe('multi-server blueprints', () => {
describe('atomic collection launches', () => { describe('atomic collection launches', () => {
it('counts available keys once per game across all maps without reserving them in a preview', async () => {
const { blueprint } = await previewLineup();
const { saveServerKey, reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const { db } = await import('../../src/lib/server/db/index.js');
saveServerKey('t5', 'pool-one'); saveServerKey('t5', 'pool-two'); saveServerKey('t6', 'pool-three');
const preview = previewCollection(blueprint.id);
expect(preview.servers.map(server => server.hasSavedKey)).toEqual([true, true, false, true, false]);
expect(db().prepare('SELECT * FROM server_key_leases').all()).toEqual([]);
reserveServerKey('t5_zombie_pentagon_1', 't5');
expect(() => queueCollection({ previewId: preview.id })).toThrow(/No available T5 server keys/);
expect(db().prepare('SELECT * FROM server_creation_jobs').all()).toEqual([]);
expect(db().prepare('SELECT * FROM server_key_leases').all()).toHaveLength(1);
});
it('rolls back all earlier jobs and key changes if the final server has no valid key', async () => { it('rolls back all earlier jobs and key changes if the final server has no valid key', async () => {
const { preview } = await previewLineup(); const { preview } = await previewLineup();
const { queueCollection } = await import('../../src/lib/server/fleet/collections.js'); const { queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { saveServerKey, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); const { saveServerKey, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
saveServerKey(preview.servers[0].name!, 'original-key'); saveServerKey('t5', 'original-key');
const keys = keysFor(preview); const keys = keysFor(preview);
delete keys[preview.servers[4].name!]; delete keys[preview.servers[4].name!];
expect(() => queueCollection({ previewId: preview.id, keys })).toThrow(/Save a Plutonium server key/); expect(() => queueCollection({ previewId: preview.id, keys })).toThrow(/No available .* server keys/);
for (const table of ['server_launch_collections', 'server_collection_members', 'server_creation_jobs', 'server_deployments', 'server_launch_previews', 'server_launch_recipes']) { for (const table of ['server_launch_collections', 'server_collection_members', 'server_creation_jobs', 'server_deployments', 'server_launch_previews', 'server_launch_recipes']) {
expect(db().prepare(`SELECT * FROM ${table}`).all()).toEqual([]); expect(db().prepare(`SELECT * FROM ${table}`).all()).toEqual([]);
} }
expect(getServerKey(preview.servers[0].name!)).toBe('original-key'); expect(getServerKey(preview.servers[0].name!)).toBeUndefined();
expect(db().prepare('SELECT pluto_key FROM server_keys').get()).toEqual({ pluto_key: 'original-key' });
expect(db().prepare('SELECT * FROM server_key_leases').all()).toEqual([]);
expect(db().prepare('SELECT * FROM server_keys').all()).toHaveLength(1); expect(db().prepare('SELECT * FROM server_keys').all()).toHaveLength(1);
keys[preview.servers[4].name!] = 'invalid key'; keys[preview.servers[4].name!] = 'invalid key';
expect(() => queueCollection({ previewId: preview.id, keys })).toThrow(/without whitespace/); expect(() => queueCollection({ previewId: preview.id, keys })).toThrow(/without whitespace/);
@@ -115,7 +132,7 @@ describe('atomic collection launches', () => {
const { queueCollection, previewCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js'); const { queueCollection, previewCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js');
const { saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); const { saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { db, closeDb } = await import('../../src/lib/server/db/index.js'); const { db, closeDb } = await import('../../src/lib/server/db/index.js');
for (const server of preview.servers) saveServerKey(server.name!, `saved-${server.name}`); for (const server of preview.servers) saveServerKey(server.name!.slice(0, 2), `saved-${server.name}`);
const ready = previewCollection(blueprint.id); const ready = previewCollection(blueprint.id);
expect(ready.servers.every(server => server.hasSavedKey)).toBe(true); expect(ready.servers.every(server => server.hasSavedKey)).toBe(true);
const queued = queueCollection({ previewId: ready.id }); const queued = queueCollection({ previewId: ready.id });
@@ -171,17 +188,18 @@ describe('matching a blueprint exactly', () => {
expect(db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(first.deploymentId)).toEqual({ state: 'queued' }); expect(db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(first.deploymentId)).toEqual({ state: 'queued' });
expect(getCollection(queued.collectionId!)?.collection).toMatchObject({ done: 1, total: 1, active: 0 }); expect(getCollection(queued.collectionId!)?.collection).toMatchObject({ done: 1, total: 1, active: 0 });
const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
expect(getServerKey(second.name)).toBe('second'); expect(getServerKey(second.name)).toBeUndefined();
expect(db().prepare("SELECT pluto_key FROM server_keys WHERE pluto_key='second'").get()).toEqual({ pluto_key: 'second' });
}); });
it('rolls back removals and creation cancellations when a required key is missing', async () => { it('rolls back removals and creation cancellations when the game pool is exhausted', async () => {
const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js'); const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js');
const old = queueCreateServer({ gameMode: 'zombie', mapSlug: 'moon', plutoKey: 'existing' }); const old = queueCreateServer({ gameMode: 'zombie', mapSlug: 'moon', plutoKey: 'existing' });
if (!old.ok) throw new Error(old.message); if (!old.ok) throw new Error(old.message);
const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 1 }]); const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 2 }]);
const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js'); const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js');
const preview = previewCollection(plan.id, 'exact'); const preview = previewCollection(plan.id, 'exact');
expect(() => queueCollection({ previewId: preview.id })).toThrow(/Save a Plutonium server key/); expect(() => queueCollection({ previewId: preview.id })).toThrow(/No available .* server keys/);
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(old.deploymentId)).toEqual({ state: 'queued' }); expect(db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(old.deploymentId)).toEqual({ state: 'queued' });
for (const table of ['server_collection_removals', 'server_launch_collections', 'server_creation_dependencies']) expect(db().prepare(`SELECT * FROM ${table}`).all()).toEqual([]); for (const table of ['server_collection_removals', 'server_launch_collections', 'server_creation_dependencies']) expect(db().prepare(`SELECT * FROM ${table}`).all()).toEqual([]);
+1 -1
View File
@@ -153,7 +153,7 @@ describe('host deletion', () => {
{ hostId: other.id, automatic: false } { hostId: other.id, automatic: false }
].entries()) { ].entries()) {
expect(queueNativeCreationIntent({ server: `t5_zombie_theater_${index + 1}`, game: 't5', mapSlug: 'theater', expect(queueNativeCreationIntent({ server: `t5_zombie_theater_${index + 1}`, game: 't5', mapSlug: 'theater',
plutoKey: 'saved-key', selection: { ...selection, libraryId: 'library', image: 'plutainer:test' } }).ok).toBe(true); plutoKey: `saved-key-${index}`, selection: { ...selection, libraryId: 'library', image: 'plutainer:test' } }).ok).toBe(true);
} }
expect(hosts.hostDeletionStatus(hosts.host.id)).toEqual({ allowed: true, blockers: [] }); expect(hosts.hostDeletionStatus(hosts.host.id)).toEqual({ allowed: true, blockers: [] });
expect(hosts.hostDeletionStatus(other.id).allowed).toBe(false); expect(hosts.hostDeletionStatus(other.id).allowed).toBe(false);
+1 -1
View File
@@ -353,7 +353,7 @@ describe('automatic host preparation', () => {
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js'); const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
for (let index = 0; index < 2; index++) { for (let index = 0; index < 2; index++) {
expect(queueNativeCreation({ game: 't5', gameMode: 'zombie', mapSlug: 'theater', libraryId: old.id, expect(queueNativeCreation({ game: 't5', gameMode: 'zombie', mapSlug: 'theater', libraryId: old.id,
hostId: automatic ? undefined : id, image, plutoKey: 'saved-key' }).ok).toBe(true); hostId: automatic ? undefined : id, image, plutoKey: `saved-key-${index}` }).ok).toBe(true);
} }
// Also cover pre-existing intents whose image is no longer cached. // Also cover pre-existing intents whose image is no longer cached.
heartbeat(id, { runtimeImages: [runtime] }); heartbeat(id, { runtimeImages: [runtime] });
+5 -4
View File
@@ -55,12 +55,12 @@ describe('native creation intents before placement', () => {
it('cancels unplaced work locally once, retains keys, and frees its logical server name', async () => { it('cancels unplaced work locally once, retains keys, and frees its logical server name', async () => {
const id = await intent(); const id = await intent();
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { saveServerKey, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); const { reserveServerKey, getServerKey, serverKeyPool } = await import('../../src/lib/server/fleet/server-keys.js');
const { cancelCreation, canCancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js'); const { cancelCreation, canCancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
const { getDeployment } = await import('../../src/lib/server/provisioning/history.js'); const { getDeployment } = await import('../../src/lib/server/provisioning/history.js');
const { nativeDeployment, unplacedNativeCreation } = await import('../../src/lib/server/backends/native.js'); const { nativeDeployment, unplacedNativeCreation } = await import('../../src/lib/server/backends/native.js');
const { nextLobbyNumber } = await import('../../src/lib/server/provisioning/instances.js'); const { nextLobbyNumber } = await import('../../src/lib/server/provisioning/instances.js');
saveServerKey(server, 'saved-key'); reserveServerKey(server, 't5', 'saved-key');
expect(canCancelCreation(id)).toBe(true); expect(canCancelCreation(id)).toBe(true);
const cleanup = cancelCreation(id)!; const cleanup = cancelCreation(id)!;
expect(cleanup).toEqual(expect.any(String)); expect(cleanup).toEqual(expect.any(String));
@@ -71,7 +71,8 @@ describe('native creation intents before placement', () => {
expect(getDeployment(cleanup)).toMatchObject({ backend: 'native', deployment: { state: 'done', server_deleted_at: expect.any(String) } }); expect(getDeployment(cleanup)).toMatchObject({ backend: 'native', deployment: { state: 'done', server_deleted_at: expect.any(String) } });
expect(unplacedNativeCreation(server)).toBeNull(); expect(unplacedNativeCreation(server)).toBeNull();
expect(nextLobbyNumber('t5_zombie_theater')).toBe(1); expect(nextLobbyNumber('t5_zombie_theater')).toBe(1);
expect(getServerKey(server)).toBe('saved-key'); expect(getServerKey(server)).toBeUndefined();
expect(serverKeyPool('t5')).toEqual({ total: 1, available: 1 });
expect(db().prepare('SELECT id FROM ec_host_operations').all()).toEqual([]); expect(db().prepare('SELECT id FROM ec_host_operations').all()).toEqual([]);
expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]); expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]);
}); });
@@ -142,7 +143,7 @@ describe('native creation intents before placement', () => {
const { queueNativeCreation, processNativeProvisioning } = await import('../../src/lib/server/provisioning/native.js'); const { queueNativeCreation, processNativeProvisioning } = await import('../../src/lib/server/provisioning/native.js');
const input = { game: 't5' as const, gameMode: 'zombie' as const, mapSlug: 'theater', libraryId: library.id, image, plutoKey: 'saved-key' }; const input = { game: 't5' as const, gameMode: 'zombie' as const, mapSlug: 'theater', libraryId: library.id, image, plutoKey: 'saved-key' };
const blocked = queueNativeCreation({ ...input, hostId: offline.id }); const blocked = queueNativeCreation({ ...input, hostId: offline.id });
const available = queueNativeCreation({ ...input, hostId: ready.id }); const available = queueNativeCreation({ ...input, hostId: ready.id, plutoKey: 'ready-key' });
expect(blocked.ok && available.ok).toBe(true); expect(blocked.ok && available.ok).toBe(true);
if (!blocked.ok || !available.ok) throw new Error('Could not queue launch fixtures.'); if (!blocked.ok || !available.ok) throw new Error('Could not queue launch fixtures.');
await processNativeProvisioning(() => null); await processNativeProvisioning(() => null);
+2 -1
View File
@@ -43,10 +43,11 @@ afterEach(async () => {
vi.unstubAllEnvs(); vi.unstubAllEnvs();
}); });
let queuedKeys = 0;
async function queue() { async function queue() {
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js'); const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
const result = queueNativeCreation({ game: 't5', gameMode: 'zombie', mapSlug: 'theater', const result = queueNativeCreation({ game: 't5', gameMode: 'zombie', mapSlug: 'theater',
hostId, libraryId, image, plutoKey: 'saved-key' }); hostId, libraryId, image, plutoKey: `saved-key-${++queuedKeys}` });
if (!result.ok) throw new Error(result.message); if (!result.ok) throw new Error(result.message);
return result; return result;
} }
+94 -14
View File
@@ -12,11 +12,12 @@ vi.mock('../../src/lib/server/integrations/rcon.js', () => ({
vi.mock('../../src/lib/server/integrations/plutonium.js', () => ({ checkRelease: vi.fn(async () => {}), freshRelease: () => 5354, probeVersion: vi.fn(async () => 5354) })); vi.mock('../../src/lib/server/integrations/plutonium.js', () => ({ checkRelease: vi.fn(async () => {}), freshRelease: () => 5354, probeVersion: vi.fn(async () => 5354) }));
vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}), kickProvisioning: vi.fn(async () => {}) })); vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}), kickProvisioning: vi.fn(async () => {}) }));
let queuedKeys = 0;
let fixture: Fixture; let hostId: string; let libraryId: string; let fixture: Fixture; let hostId: string; let libraryId: string;
const image = 'example.test/plutainer:test'; const image = 'example.test/plutainer:test';
const inventory = () => ({ cpuCount: 16, cpuUsagePercent: 10, memoryTotal: 64 * 1024 ** 3, memoryAvailable: 60 * 1024 ** 3, protocolVersion: 1, dockerRunning: true, directoriesWritable: true, storageAvailable: 100_000_000_000, runtimeImages: [image], libraries: libraryId ? [{ id: libraryId, state: 'ready' as const }] : [] }); const inventory = () => ({ cpuCount: 16, cpuUsagePercent: 10, memoryTotal: 64 * 1024 ** 3, memoryAvailable: 60 * 1024 ** 3, protocolVersion: 1, dockerRunning: true, directoriesWritable: true, storageAvailable: 100_000_000_000, runtimeImages: [image], libraries: libraryId ? [{ id: libraryId, state: 'ready' as const }] : [] });
beforeEach(async () => { beforeEach(async () => {
vi.resetModules(); vi.clearAllMocks(); remote.state = 'running'; remote.players = 0; remote.map = 'zombie_theater'; libraryId = ''; vi.resetModules(); vi.clearAllMocks(); remote.state = 'running'; remote.players = 0; remote.map = 'zombie_theater'; libraryId = ''; queuedKeys = 0;
fixture = await newFixture([]); fixture = await newFixture([]);
vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries')); vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries'));
vi.stubEnv('PLUTAINER_IMAGE', image); vi.stubEnv('RCON_PASSWORD', 'test-password'); vi.stubEnv('PLUTAINER_IMAGE', image); vi.stubEnv('RCON_PASSWORD', 'test-password');
@@ -48,7 +49,7 @@ async function queue(game: 't5' | 't6' = 't5', placement: 'pinned' | 'automatic'
await prepareLibrary(game); remote.map = game === 't5' ? 'zombie_theater' : 'zm_transit'; await prepareLibrary(game); remote.map = game === 't5' ? 'zombie_theater' : 'zm_transit';
const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js'); const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js');
const result = queueCreateServer({ game, gameMode: 'zombie', mapSlug: game === 't5' ? 'theater' : 'town', const result = queueCreateServer({ game, gameMode: 'zombie', mapSlug: game === 't5' ? 'theater' : 'town',
hostId: placement === 'pinned' ? hostId : undefined, libraryId, image, plutoKey: 'saved-key' }); hostId: placement === 'pinned' ? hostId : undefined, libraryId, image, plutoKey: ++queuedKeys === 1 ? 'saved-key' : `saved-key-${queuedKeys}` });
if (!result.ok) throw new Error(result.message); if (!result.ok) throw new Error(result.message);
return result; return result;
} }
@@ -990,8 +991,9 @@ describe('native instance provisioning', () => {
expect(db().prepare("SELECT COUNT(*) AS count FROM ec_host_operations WHERE kind LIKE 'instance.%'").get()).toEqual({ count: 0 }); expect(db().prepare("SELECT COUNT(*) AS count FROM ec_host_operations WHERE kind LIKE 'instance.%'").get()).toEqual({ count: 0 });
const { getLibraryVersion } = await import('../../src/lib/server/game-files/index.js'); const { getLibraryVersion } = await import('../../src/lib/server/game-files/index.js');
expect(getLibraryVersion(libraryId)?.state).toBe('ready'); expect(getLibraryVersion(libraryId)?.state).toBe('ready');
const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); const { getServerKey, serverKeyPool } = await import('../../src/lib/server/fleet/server-keys.js');
expect(getServerKey(job.name)).toBe('saved-key'); expect(getServerKey(job.name)).toBeUndefined();
expect(serverKeyPool('t5')).toEqual({ total: 1, available: 1 });
}); });
}); });
@@ -1083,7 +1085,7 @@ describe('automatic capacity admission and placement', () => {
const preview = previewCollection(blueprint.id); const preview = previewCollection(blueprint.id);
const { heartbeat } = await import('../../src/lib/server/hosts/index.js'); const { heartbeat } = await import('../../src/lib/server/hosts/index.js');
heartbeat(hostId, { ...inventory(), cpuUsagePercent: 100 }); heartbeat(hostId, { ...inventory(), cpuUsagePercent: 100 });
const result = queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) }); const result = queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) });
expect(result).toMatchObject({ ok: true, collectionId: expect.any(String) }); expect(result).toMatchObject({ ok: true, collectionId: expect.any(String) });
await advance(); await advance();
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
@@ -1117,7 +1119,7 @@ describe('automatic capacity admission and placement', () => {
const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js'); const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js');
const measured = { ...inventory(), cpuCount: 1, cpuUsagePercent: 50 }; const measured = { ...inventory(), cpuCount: 1, cpuUsagePercent: 50 };
heartbeat(hostId, measured); heartbeat(hostId, measured);
const next = queueNativeCreation(input); const next = queueNativeCreation({ ...input, plutoKey: 'next-key' });
expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message); expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message);
await advance(); await advance();
expect(await operationFor(next.deploymentId)).toBeNull(); expect(await operationFor(next.deploymentId)).toBeNull();
@@ -1151,7 +1153,7 @@ describe('automatic capacity admission and placement', () => {
const blueprint = saveBlueprint({ name: 'Staggered lobbies', entries: [{ game: 't5', mapSlug: 'theater', count: 3 }] }); const blueprint = saveBlueprint({ name: 'Staggered lobbies', entries: [{ game: 't5', mapSlug: 'theater', count: 3 }] });
const preview = previewCollection(blueprint.id); const preview = previewCollection(blueprint.id);
expect(preview.ready).toBe(true); expect(preview.ready).toBe(true);
expect(queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) }).ok).toBe(true); expect(queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) }).ok).toBe(true);
await advance(); await advance();
const creates = () => db().prepare("SELECT instance_id FROM ec_host_operations WHERE kind='instance.create'").all() as { instance_id: string }[]; const creates = () => db().prepare("SELECT instance_id FROM ec_host_operations WHERE kind='instance.create'").all() as { instance_id: string }[];
expect(creates()).toHaveLength(1); expect(creates()).toHaveLength(1);
@@ -1170,7 +1172,7 @@ describe('automatic capacity admission and placement', () => {
const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js'); const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
const first = queueNativeCreation(input); const first = queueNativeCreation(input);
expect(first.ok).toBe(true); if (!first.ok) return; expect(first.ok).toBe(true); if (!first.ok) return;
const second = queueNativeCreation(input); const second = queueNativeCreation({ ...input, plutoKey: 'second-key' });
expect(second).toMatchObject({ ok: true, name: 't5_zombie_theater_2', port: null }); expect(second).toMatchObject({ ok: true, name: 't5_zombie_theater_2', port: null });
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]); expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
@@ -1191,7 +1193,7 @@ describe('automatic capacity admission and placement', () => {
(host_id,sampled_at,cpu_count,cpu_usage_percent,memory_total,memory_available) VALUES(?,?,4,?,?,?)`) (host_id,sampled_at,cpu_count,cpu_usage_percent,memory_total,memory_available) VALUES(?,?,4,?,?,?)`)
.run(hostId, new Date(Date.now() - minute * 60_000).toISOString(), resource === 'cpu' ? 100 : 10, .run(hostId, new Date(Date.now() - minute * 60_000).toISOString(), resource === 'cpu' ? 100 : 10,
4 * GiB, resource === 'memory' ? GiB / 8 : Math.floor(3.8 * GiB)); 4 * GiB, resource === 'memory' ? GiB / 8 : Math.floor(3.8 * GiB));
const jobs = Array.from({ length: 3 }, () => queueNativeCreation(input)); const jobs = Array.from({ length: 3 }, (_, index) => queueNativeCreation({ ...input, plutoKey: `key-${index}` }));
expect(jobs.every(job => job.ok)).toBe(true); expect(jobs.every(job => job.ok)).toBe(true);
await advance(); await advance();
expect(await operationFor(jobs[0].ok ? jobs[0].deploymentId : 'missing')).not.toBeNull(); expect(await operationFor(jobs[0].ok ? jobs[0].deploymentId : 'missing')).not.toBeNull();
@@ -1205,7 +1207,7 @@ describe('automatic capacity admission and placement', () => {
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const GiB = 1024 ** 3; const GiB = 1024 ** 3;
heartbeat(hostId, { ...inventory(), cpuCount: 4, memoryTotal: 4 * GiB, memoryAvailable: Math.floor(3.8 * GiB) }); heartbeat(hostId, { ...inventory(), cpuCount: 4, memoryTotal: 4 * GiB, memoryAvailable: Math.floor(3.8 * GiB) });
const jobs = Array.from({ length: 3 }, () => queueNativeCreation(input)); const jobs = Array.from({ length: 3 }, (_, index) => queueNativeCreation({ ...input, plutoKey: `key-${index}` }));
expect(jobs.every(job => job.ok)).toBe(true); expect(jobs.every(job => job.ok)).toBe(true);
// Learning increases the startup allowance after admission. All three // Learning increases the startup allowance after admission. All three
// no longer fit concurrently, but this otherwise empty host fits one. // no longer fit concurrently, but this otherwise empty host fits one.
@@ -1244,7 +1246,7 @@ describe('automatic capacity admission and placement', () => {
// Adding a new 1 GiB server fits below 2.53 GiB usable memory. // Adding a new 1 GiB server fits below 2.53 GiB usable memory.
heartbeat(hostId, { ...host, memoryAvailable: 2 * GiB, instanceResources: [{ instanceId: instance.id, heartbeat(hostId, { ...host, memoryAvailable: 2 * GiB, instanceResources: [{ instanceId: instance.id,
generation: instance.generation, uptimeSeconds: 60, cpuCores: 0.1, memoryBytes: Math.floor(0.8 * GiB) }] }); generation: instance.generation, uptimeSeconds: 60, cpuCores: 0.1, memoryBytes: Math.floor(0.8 * GiB) }] });
const second = queueNativeCreation(input); const second = queueNativeCreation({ ...input, plutoKey: 'second-key' });
expect(second).toMatchObject({ ok: true }); expect(second).toMatchObject({ ok: true });
await advance(); await advance();
expect(await operationFor(second.ok ? second.deploymentId : 'missing')).not.toBeNull(); expect(await operationFor(second.ok ? second.deploymentId : 'missing')).not.toBeNull();
@@ -1265,7 +1267,7 @@ describe('automatic capacity admission and placement', () => {
heartbeat(hostId, { ...inventory(), memoryTotal: 3 * GiB, memoryAvailable: 2 * GiB, heartbeat(hostId, { ...inventory(), memoryTotal: 3 * GiB, memoryAvailable: 2 * GiB,
instanceResources: evidence === 'missing' ? [] : [{ instanceId: instance.id, generation: instance.generation + 1, instanceResources: evidence === 'missing' ? [] : [{ instanceId: instance.id, generation: instance.generation + 1,
uptimeSeconds: 60, cpuCores: 0.1, memoryBytes: Math.floor(0.8 * GiB) }] }); uptimeSeconds: 60, cpuCores: 0.1, memoryBytes: Math.floor(0.8 * GiB) }] });
const next = queueNativeCreation(input); const next = queueNativeCreation({ ...input, plutoKey: 'next-key' });
expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message); expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message);
await advance(); await advance();
expect(await operationFor(next.deploymentId)).toBeNull(); expect(await operationFor(next.deploymentId)).toBeNull();
@@ -1318,7 +1320,7 @@ describe('automatic capacity admission and placement', () => {
await advance(); await advance();
expect(nativeInstance(result.name)?.host_id).toBe(other.id); expect(nativeInstance(result.name)?.host_id).toBe(other.id);
updateHost(other.id, { mode: 'draining' }); updateHost(other.id, { mode: 'draining' });
const next = queueNativeCreation(input); const next = queueNativeCreation({ ...input, plutoKey: 'next-key' });
expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message); expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message);
await advance(); await advance();
expect(await operationFor(next.deploymentId)).toBeNull(); expect(await operationFor(next.deploymentId)).toBeNull();
@@ -1333,7 +1335,7 @@ describe('automatic capacity admission and placement', () => {
const preview = previewCollection(blueprint.id); const preview = previewCollection(blueprint.id);
expect(preview.ready).toBe(true); expect(preview.ready).toBe(true);
expect(preview.servers.map(server => server.port)).toEqual([null, null]); expect(preview.servers.map(server => server.port)).toEqual([null, null]);
expect(queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) }).ok).toBe(true); expect(queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) }).ok).toBe(true);
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]); expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]);
const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js'); const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js');
@@ -1481,6 +1483,7 @@ describe('cancelled native mutation bookkeeping', () => {
it.each(['rebuild','destroy'] as const)('reconciles a delivered %s receipt without resuming the cancelled job', async action => { it.each(['rebuild','destroy'] as const)('reconciles a delivered %s receipt without resuming the cancelled job', async action => {
const job = await queue(); await advance(); await acknowledge(job.deploymentId); const job = await queue(); await advance(); await acknowledge(job.deploymentId);
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { serverKeyPool, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js'); const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js'); const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js');
const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js'); const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js');
@@ -1491,6 +1494,7 @@ describe('cancelled native mutation bookkeeping', () => {
await advance(); const operation = (await operationFor(id))!; await advance(); const operation = (await operationFor(id))!;
pendingHostOperations(hostId); pendingHostOperations(hostId);
expect(cancelProvisioningJob(id)).toBe(true); expect(cancelProvisioningJob(id)).toBe(true);
expect(serverKeyPool('t5').available).toBe(0);
const replacement = action === 'rebuild' ? requestLifecycle(job.name, 'restart') : null; const replacement = action === 'rebuild' ? requestLifecycle(job.name, 'restart') : null;
if (replacement && !replacement.ok) throw new Error(replacement.error); if (replacement && !replacement.ok) throw new Error(replacement.error);
await advance(); await advance();
@@ -1498,6 +1502,8 @@ describe('cancelled native mutation bookkeeping', () => {
await advance(); await advance(); await advance(); await advance();
if (replacement?.ok) expect(await operationFor(replacement.deploymentId)).toMatchObject({ kind: 'instance.restart', generation: 2 }); if (replacement?.ok) expect(await operationFor(replacement.deploymentId)).toMatchObject({ kind: 'instance.restart', generation: 2 });
expect((await detail(id)).state).toBe('cancelled'); expect((await detail(id)).state).toBe('cancelled');
expect(serverKeyPool('t5').available).toBe(action === 'destroy' ? 1 : 0);
expect(getServerKey(job.name)).toBe(action === 'destroy' ? undefined : 'saved-key');
if(action === 'rebuild') expect(db().prepare('SELECT generation,retired_at FROM native_instances WHERE server=?').get(job.name)).toEqual({generation:2,retired_at:null}); if(action === 'rebuild') expect(db().prepare('SELECT generation,retired_at FROM native_instances WHERE server=?').get(job.name)).toEqual({generation:2,retired_at:null});
else { else {
expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]); expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]);
@@ -1508,6 +1514,7 @@ describe('cancelled native mutation bookkeeping', () => {
it.each([false,true])('cancels a host migration without restarting it or leaking destination reservations (delivered=%s)', async delivered => { it.each([false,true])('cancels a host migration without restarting it or leaking destination reservations (delivered=%s)', async delivered => {
const job = await queue(); await advance(); await acknowledge(job.deploymentId); const job = await queue(); await advance(); await acknowledge(job.deploymentId);
const hosts = await import('../../src/lib/server/hosts/index.js'); const hosts = await import('../../src/lib/server/hosts/index.js');
const { serverKeyPool, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { getLibraryVersion, updateHostLibraryCaches } = await import('../../src/lib/server/game-files/index.js'); const { getLibraryVersion, updateHostLibraryCaches } = await import('../../src/lib/server/game-files/index.js');
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
@@ -1521,12 +1528,15 @@ describe('cancelled native mutation bookkeeping', () => {
expect(migration.remove_operation_id).toBeTruthy(); expect(migration.remove_operation_id).toBeTruthy();
if(delivered) hosts.pendingHostOperations(hostId); if(delivered) hosts.pendingHostOperations(hostId);
expect(cancelProvisioningJob(migration.deployment_id)).toBe(true); expect(cancelProvisioningJob(migration.deployment_id)).toBe(true);
expect(serverKeyPool('t5').available).toBe(0);
if(delivered) hosts.reportHostOperation(hostId,migration.remove_operation_id,{state:'completed'}); if(delivered) hosts.reportHostOperation(hostId,migration.remove_operation_id,{state:'completed'});
await processProvisioning(); await processProvisioning(); await processProvisioning(); await processProvisioning();
expect((await detail(migration.deployment_id)).state).toBe('cancelled'); expect((await detail(migration.deployment_id)).state).toBe('cancelled');
expect(db().prepare('SELECT * FROM ec_host_ports WHERE host_id=?').all(destination.id)).toEqual([]); expect(db().prepare('SELECT * FROM ec_host_ports WHERE host_id=?').all(destination.id)).toEqual([]);
expect(db().prepare('SELECT * FROM ec_host_migrations').all()).toHaveLength(1); expect(db().prepare('SELECT * FROM ec_host_migrations').all()).toHaveLength(1);
expect(db().prepare('SELECT * FROM ec_host_ports WHERE host_id=?').all(hostId)).toHaveLength(delivered ? 0 : 2); expect(db().prepare('SELECT * FROM ec_host_ports WHERE host_id=?').all(hostId)).toHaveLength(delivered ? 0 : 2);
expect(serverKeyPool('t5').available).toBe(delivered ? 1 : 0);
expect(getServerKey(job.name)).toBe(delivered ? undefined : 'saved-key');
}); });
}); });
@@ -1547,3 +1557,73 @@ describe('additional concurrency admission', () => {
expect(await operationFor(second.deploymentId)).not.toBeNull(); expect(await operationFor(second.deploymentId)).not.toBeNull();
}); });
}); });
describe('game key pool launch lifecycle', () => {
it('reserves across maps atomically, rejects duplicate credentials, and reuses a cancelled launch key', async () => {
await prepareLibrary();
const { saveServerKey, serverKeyPool, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { queueCreateServer, destroyServer, previewCreation } = await import('../../src/lib/server/provisioning/instances.js');
saveServerKey('t5', 'shared-t5-key', 'Any T5 map');
saveServerKey('t6', 'shared-t6-key');
const input = { game: 't5' as const, gameMode: 'zombie' as const, mapSlug: 'moon', hostId, libraryId, image };
expect(previewCreation(input).hasSavedKey).toBe(true);
const first = queueCreateServer(input);
if (!first.ok) throw new Error(first.message);
expect(first.name).toBe('t5_zombie_moon_1');
expect(getServerKey(first.name)).toBe('shared-t5-key');
expect(serverKeyPool('t5')).toEqual({ total: 1, available: 0 });
expect(serverKeyPool('t6')).toEqual({ total: 1, available: 1 });
expect(previewCreation({ ...input, mapSlug: 'theater' }).hasSavedKey).toBe(false);
expect(queueCreateServer({ ...input, mapSlug: 'theater' }).ok).toBe(false);
expect(queueCreateServer({ ...input, mapSlug: 'theater', plutoKey: 'shared-t5-key' }).ok).toBe(false);
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
closeDb();
expect(serverKeyPool('t5').available).toBe(0);
expect(db().prepare('SELECT * FROM server_creation_jobs').all()).toHaveLength(1);
expect(destroyServer(first.name).ok).toBe(true);
expect(serverKeyPool('t5').available).toBe(1);
const second = queueCreateServer({ ...input, mapSlug: 'theater' });
if (!second.ok) throw new Error(second.message);
expect(getServerKey(second.name)).toBe('shared-t5-key');
expect(getServerKey(first.name)).toBeUndefined();
});
it('keeps a key through stop and rebuild, then releases it only after confirmed removal', async () => {
const job = await queue(); await advance(); await acknowledge(job.deploymentId);
const { serverKeyPool, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js');
const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js');
const stopped = requestLifecycle(job.name, 'stop');
if (!stopped.ok) throw new Error(stopped.error);
await advance(); remote.state = 'stopped'; await acknowledge(stopped.deploymentId);
expect(serverKeyPool('t5').available).toBe(0);
const rebuilt = requestLifecycle(job.name, 'rebuild');
if (!rebuilt.ok) throw new Error(rebuilt.error);
await advance(); remote.state = 'running'; await acknowledge(rebuilt.deploymentId);
expect(getServerKey(job.name)).toBe('saved-key');
expect(serverKeyPool('t5').available).toBe(0);
const removal = destroyServer(job.name);
if (!removal.ok || !removal.deploymentId) throw new Error('Expected removal');
await advance();
expect(serverKeyPool('t5').available).toBe(0);
const operation = (await operationFor(removal.deploymentId))!;
const { reportHostOperation } = await import('../../src/lib/server/hosts/index.js');
reportHostOperation(hostId, operation.id, { state: 'unknown', error: 'Receipt lost' });
await advance();
expect(serverKeyPool('t5').available).toBe(0);
await acknowledge(removal.deploymentId);
expect(serverKeyPool('t5')).toEqual({ total: 1, available: 1 });
expect(getServerKey(job.name)).toBeUndefined();
});
it('keeps the reservation for a submitted cancellation until the host confirms cleanup', async () => {
const job = await queue(); await advance();
const { serverKeyPool } = await import('../../src/lib/server/fleet/server-keys.js');
const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js');
const cleanup = cancelCreation(job.deploymentId)!;
expect(cleanup).toEqual(expect.any(String));
expect(serverKeyPool('t5').available).toBe(0);
await acknowledge(cleanup);
expect(serverKeyPool('t5')).toEqual({ total: 1, available: 1 });
});
});
@@ -12,6 +12,9 @@ it('retires only obsolete backend columns while preserving instances, history, k
const previous = new Database(fixture.dbPath); const previous = new Database(fixture.dbPath);
// Model the former required resource identity and an unrelated extension. // Model the former required resource identity and an unrelated extension.
previous.exec(schema); previous.exec(schema);
previous.exec(`DROP TABLE server_keys;
CREATE TABLE server_keys(server_id TEXT PRIMARY KEY,pluto_key TEXT NOT NULL,
updated_at TEXT NOT NULL DEFAULT (datetime('now')));`);
previous.exec(`ALTER TABLE servers ADD COLUMN komodo_type TEXT NOT NULL DEFAULT 'deployment'; previous.exec(`ALTER TABLE servers ADD COLUMN komodo_type TEXT NOT NULL DEFAULT 'deployment';
ALTER TABLE servers ADD COLUMN komodo_name TEXT NOT NULL; ALTER TABLE servers ADD COLUMN komodo_name TEXT NOT NULL;
ALTER TABLE servers ADD COLUMN komodo_server TEXT; ALTER TABLE servers ADD COLUMN komodo_server TEXT;
+1 -1
View File
@@ -81,7 +81,7 @@ describe('runtime image launch policy', () => {
expect(() => previewCollection(blueprint.id, 'fill', { image: custom })).toThrow(/prepared by an admin/); expect(() => previewCollection(blueprint.id, 'fill', { image: custom })).toThrow(/prepared by an admin/);
await cache([custom]); await cache([custom]);
const preview = previewCollection(blueprint.id, 'fill', { image: custom }); const preview = previewCollection(blueprint.id, 'fill', { image: custom });
const keys = Object.fromEntries(preview.servers.map(server => [server.name!, 'saved-key'])); const keys = Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name]));
await cache([]); await cache([]);
expect(() => queueCollection({ previewId: preview.id, keys })).toThrow(/prepared by an admin/); expect(() => queueCollection({ previewId: preview.id, keys })).toThrow(/prepared by an admin/);
const { db } = await import('../../src/lib/server/db/index.js'); const { db } = await import('../../src/lib/server/db/index.js');
+108
View File
@@ -0,0 +1,108 @@
import Database from 'better-sqlite3';
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
import schema from '../../src/lib/server/db/schema.sql?raw';
import { newFixture, type Fixture } from '../helpers/fixture.js';
let fixture: Fixture;
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
afterEach(async () => { await fixture.cleanup(); });
it('migrates idle, occupied and duplicate legacy keys without releasing unresolved assignments', async () => {
const previous = new Database(fixture.dbPath);
previous.exec(schema);
previous.exec(`DROP TABLE server_keys;
CREATE TABLE server_keys(server_id TEXT PRIMARY KEY,pluto_key TEXT NOT NULL,updated_at TEXT NOT NULL DEFAULT (datetime('now')));`);
const { migrateNativeInstances } = await import('../../src/lib/server/backends/schema.js');
migrateNativeInstances(previous);
const cases = [
['t5_zombie_theater_1', 'shared-occupied'],
['t5_zombie_pentagon_1', 'shared-occupied'],
['t5_zombie_cosmodrome_1', 'native-only'],
['t5_zombie_temple_1', 'queued-key'],
['t5_zombie_sumpf_1', 'failed-key'],
['t5_zombie_moon_1', 'idle-key'],
['t5_zombie_moon_2', 'idle-key'],
['t6_zombie_town_1', 't6-key']
];
for (const [server, credential] of cases) previous.prepare('INSERT INTO server_keys(server_id,pluto_key,updated_at) VALUES(?,?,?)').run(server, credential, '2026-01-01');
previous.exec(`INSERT INTO servers(name,game,desired_state) VALUES('t5_zombie_theater_1','t5','running'),('t5_zombie_pentagon_1','t5','stopped');
INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at)
VALUES('native','t5_zombie_cosmodrome_1','host','library','image',28960,'2026-01-01');
INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by) VALUES
('queued','t5_zombie_temple_1','create','queued','Queued','2026-01-01','fixture'),
('failed','t5_zombie_sumpf_1','create','failed','Placement failed','2026-01-01','fixture');`);
previous.close();
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
const pool = await import('../../src/lib/server/fleet/server-keys.js');
for (let pass = 0; pass < 2; pass++) {
expect(pool.serverKeyPool('t5')).toEqual({ total: 5, available: 1 });
expect(pool.serverKeyPool('t6')).toEqual({ total: 1, available: 1 });
expect(pool.getServerKey('t5_zombie_theater_1')).toBe('shared-occupied');
expect(pool.getServerKey('t5_zombie_pentagon_1')).toBe('shared-occupied');
expect(pool.getServerKey('t5_zombie_cosmodrome_1')).toBe('native-only');
expect(pool.getServerKey('t5_zombie_temple_1')).toBe('queued-key');
expect(pool.getServerKey('t5_zombie_sumpf_1')).toBe('failed-key');
expect(pool.getServerKey('t5_zombie_moon_1')).toBeUndefined();
expect(db().pragma('foreign_key_check')).toEqual([]);
closeDb();
}
const duplicate = pool.listServerKeys().find(key => key.assigned_servers.length === 2)!;
expect(duplicate.assigned_servers).toEqual(['t5_zombie_pentagon_1', 't5_zombie_theater_1']);
expect(duplicate.updated_at).toBe('2026-01-01');
expect(pool.reserveServerKey('t5_zombie_theater_2', 't5')).toBe('idle-key');
pool.releaseServerKey('t5_zombie_theater_1');
expect(pool.availableServerKeyCount('t5')).toBe(0);
expect(() => pool.removeServerKey(duplicate.id)).toThrow('reserved');
pool.releaseServerKey('t5_zombie_pentagon_1');
expect(pool.reserveServerKey('t5_zombie_theater_3', 't5')).toBe('shared-occupied');
});
it('recovers actual launch credentials and holds both sides of an unresolved legacy rebuild', async () => {
const previous = new Database(fixture.dbPath);
previous.exec(schema);
previous.exec(`DROP TABLE server_keys;
CREATE TABLE server_keys(server_id TEXT PRIMARY KEY,pluto_key TEXT NOT NULL,updated_at TEXT NOT NULL DEFAULT (datetime('now')));`);
const { migrateNativeInstances } = await import('../../src/lib/server/backends/schema.js');
const { ensureHostSchema } = await import('../../src/lib/server/hosts/schema.js');
migrateNativeInstances(previous);
ensureHostSchema(previous);
previous.exec(`INSERT INTO ec_hosts(id,name,created_at) VALUES('host','Fixture host','2026-01-01');
INSERT INTO servers(name,game) VALUES('t5_zombie_theater_1','t5'),('t5_zombie_pentagon_1','t5');
INSERT INTO server_keys(server_id,pluto_key) VALUES('t5_zombie_theater_1','edited-saved-key'),
('t5_zombie_pentagon_1','original-key'),('t5_zombie_moon_1','original-key');
INSERT INTO native_instances(id,server,host_id,generation,library_id,image,port,created_at) VALUES
('kino','t5_zombie_theater_1','host',2,'library','image',28960,'2026-01-01'),
('five','t5_zombie_pentagon_1','host',1,'library','image',28961,'2026-01-01'),
('queued','t5_zombie_cosmodrome_1','host',1,'library','image',28962,'2026-01-01');`);
function operation(id: string, instance: string, generation: number, kind: string, state: string, key: string, staged = false, nextGeneration?: number) {
const payload = JSON.stringify({ game: 't5', env: { PLUTO_SERVER_KEY: key }, nextGeneration });
previous.prepare(`INSERT INTO ec_host_operations(id,host_id,instance_id,generation,kind,state,payload,created_at,updated_at)
VALUES(?,'host',?,?,?,?,?,'2026-01-01','2026-01-01')`).run(id, instance, generation, kind, state, staged ? '{}' : payload);
if (staged) previous.prepare('INSERT INTO ec_host_operation_payloads(operation_id,payload) VALUES(?,?)').run(id, payload);
}
operation('obsolete-kino', 'kino', 1, 'instance.create', 'completed', 'obsolete-key');
operation('actual-kino', 'kino', 1, 'instance.rebuild', 'completed', 'actual-key', true, 2);
operation('original-five', 'five', 1, 'instance.create', 'completed', 'original-key');
operation('unknown-five', 'five', 1, 'instance.rebuild', 'unknown', 'pending-key', true, 2);
operation('unknown-creation', 'queued', 1, 'instance.create', 'unknown', 'unknown-creation-key');
operation('failed-new-key', 'kino', 2, 'instance.rebuild', 'failed', 'never-applied-key', true, 3);
previous.close();
const pool = await import('../../src/lib/server/fleet/server-keys.js');
expect(pool.getServerKey('t5_zombie_theater_1')).toBe('actual-key');
expect(pool.getServerKey('t5_zombie_pentagon_1')).toBe('pending-key');
expect(pool.getServerKey('t5_zombie_cosmodrome_1')).toBe('unknown-creation-key');
expect(pool.serverKeyPool('t5')).toEqual({ total: 5, available: 1 });
expect(pool.serverKeyPool('t5', 't5_zombie_pentagon_1')).toEqual({ total: 5, available: 2 });
expect(() => pool.reserveServerKey('t5_zombie_moon_2', 't5', 'original-key')).toThrow('already reserved');
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
expect(db().prepare(`SELECT k.pluto_key,l.is_current FROM server_key_leases l JOIN server_keys k ON k.id=l.key_id
WHERE l.server_id='t5_zombie_pentagon_1' ORDER BY l.is_current`).all())
.toEqual([{ pluto_key: 'original-key', is_current: 0 }, { pluto_key: 'pending-key', is_current: 1 }]);
expect(db().pragma('foreign_key_check')).toEqual([]);
closeDb();
expect(pool.serverKeyPool('t5')).toEqual({ total: 5, available: 1 });
pool.releaseServerKey('t5_zombie_pentagon_1');
expect(pool.availableServerKeyCount('t5')).toBe(3);
});
+92
View File
@@ -0,0 +1,92 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { newFixture, type Fixture } from '../helpers/fixture.js';
let fixture: Fixture;
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
afterEach(async () => { await fixture.cleanup(); });
const kino = 't5_zombie_theater_1';
const five = 't5_zombie_pentagon_1';
const town = 't6_zombie_town_1';
async function keys() { return import('../../src/lib/server/fleet/server-keys.js'); }
describe('per-game key pools', () => {
it('shares keys across maps only after their previous reservation is released', async () => {
const pool = await keys();
const id = pool.saveServerKey('t5', 'secret-t5', 'Shared T5 key');
pool.saveServerKey('t6', 'secret-t6');
expect(pool.reserveServerKey(kino, 't5')).toBe('secret-t5');
expect(pool.reserveServerKey(kino, 't5')).toBe('secret-t5');
expect(pool.serverKeyPool('t5')).toEqual({ total: 1, available: 0 });
expect(pool.serverKeyPool('t5', kino)).toEqual({ total: 1, available: 1 });
expect(() => pool.reserveServerKey(five, 't5')).toThrow('No available T5 server keys');
expect(() => pool.removeServerKey(id)).toThrow('reserved');
expect(pool.reserveServerKey(town, 't6')).toBe('secret-t6');
expect(pool.getServerKey(five)).toBeUndefined();
expect(pool.releaseServerKey(kino)).toBe(true);
expect(pool.reserveServerKey(five, 't5')).toBe('secret-t5');
expect(pool.listServerKeys().find(key => key.id === id)).toMatchObject({ label: 'Shared T5 key', assigned_servers: [five] });
pool.releaseServerKey(five);
expect(pool.removeServerKey(id)).toBe(true);
expect(pool.removeServerKey(id)).toBe(false);
});
it('deduplicates credentials and exposes only metadata', async () => {
const pool = await keys();
const id = pool.saveServerKey('t5', ' secret ', ' First ');
expect(pool.saveServerKey('t5', 'secret', 'Renamed')).toBe(id);
expect(pool.saveServerKey('t5', 'secret')).toBe(id);
expect(pool.serverKeyPool('t5')).toEqual({ total: 1, available: 1 });
expect(pool.listServerKeys()).toEqual([{ id, game: 't5', label: 'Renamed', updated_at: expect.any(String), assigned_servers: [] }]);
expect(JSON.stringify(pool.listServerKeys())).not.toContain('secret');
for (const [game, credential, label] of [['t7', 'valid', ''], ['t5', 'two words', ''], ['t5', '', ''], ['t5', 'valid', 'x'.repeat(161)]]) {
expect(() => pool.saveServerKey(game, credential, label)).toThrow();
}
});
it('reserves supplied keys atomically and never substitutes another available key', async () => {
const pool = await keys();
expect(pool.reserveServerKey(kino, 't5', 'manual-key')).toBe('manual-key');
pool.saveServerKey('t5', 'spare-key');
expect(() => pool.reserveServerKey(five, 't5', 'manual-key')).toThrow('already reserved');
expect(pool.getServerKey(five)).toBeUndefined();
expect(pool.availableServerKeyCount('t5')).toBe(1);
expect(() => pool.reserveServerKey(kino, 't5', 'replacement')).toThrow('already has a reserved key');
expect(() => pool.reserveServerKey(kino, 't6')).toThrow('another game');
expect(pool.reserveServerKey(five, 't5', '')).toBe('spare-key');
});
it('rolls back reservation and imported credentials when launch admission fails', async () => {
const pool = await keys();
const { db } = await import('../../src/lib/server/db/index.js');
expect(() => db().transaction(() => {
pool.reserveServerKey(kino, 't5', 'temporary-key');
throw new Error('Launch validation failed');
})()).toThrow('Launch validation failed');
expect(pool.listServerKeys()).toEqual([]);
expect(pool.getServerKey(kino)).toBeUndefined();
});
it('keeps reservations across restart and missing registry rows', async () => {
const pool = await keys();
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
pool.reserveServerKey(kino, 't5', 'retained-key');
db().prepare("INSERT INTO servers(name,game,desired_state) VALUES(?,'t5','stopped')").run(kino);
closeDb();
expect(pool.getServerKey(kino)).toBe('retained-key');
expect(pool.availableServerKeyCount('t5')).toBe(0);
db().prepare('DELETE FROM servers WHERE name=?').run(kino);
expect(pool.availableServerKeyCount('t5')).toBe(0);
});
it('does not allocate a duplicated credential imported into another game', async () => {
const pool = await keys();
pool.reserveServerKey(kino, 't5', 'same-secret');
const duplicate = pool.saveServerKey('t6', 'same-secret');
expect(pool.serverKeyPool('t6')).toEqual({ total: 1, available: 0 });
expect(pool.listServerKeys().find(key => key.id === duplicate)?.assigned_servers).toEqual([kino]);
expect(() => pool.removeServerKey(duplicate)).toThrow('reserved');
expect(() => pool.reserveServerKey(town, 't6')).toThrow('No available T6');
});
});
+70
View File
@@ -0,0 +1,70 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { newFixture, type Fixture } from '../helpers/fixture.js';
let fixture: Fixture;
beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); });
afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); });
function request(method: string, body: unknown) {
return { request: new Request('http://localhost/api/server-keys', {
method, headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body)
}) };
}
describe('game key pool API', () => {
it('returns pool metadata and assigned servers without credentials', async () => {
const route = await import('../../src/routes/api/server-keys/+server.js');
const { reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const saved = await route.PUT(request('PUT', { game: 't5', label: 'Cherry hostname 5', plutoKey: 'private-t5-credential' }));
expect(saved.status).toBe(200);
const { id } = await saved.json();
reserveServerKey('t5_zombie_theater_2', 't5');
const response = route.GET();
expect(response.headers.get('cache-control')).toBe('no-store');
const text = await response.text();
expect(text).not.toContain('private-t5-credential');
expect(text).not.toContain('pluto_key');
expect(text).not.toContain('plutoKey');
expect(JSON.parse(text)).toEqual({ keys: [{
id, game: 't5', label: 'Cherry hostname 5', assigned_servers: ['t5_zombie_theater_2'], updated_at: expect.any(String)
}] });
});
it.each([
{ game: 't4', plutoKey: 'private-key' },
{ serverId: 't5_zombie_theater_2', plutoKey: 'private-key' },
{ game: 't5', plutoKey: 'contains whitespace' },
{ game: 't5', plutoKey: 'private-key', label: 'x'.repeat(161) }
])('rejects invalid pool input without saving it: %j', async body => {
const route = await import('../../src/routes/api/server-keys/+server.js');
const response = await route.PUT(request('PUT', body));
expect(response.status).toBe(400);
expect(await response.json()).toMatchObject({ ok: false, error: expect.any(String) });
expect(await route.GET().json()).toEqual({ keys: [] });
});
it('deduplicates a credential in its game pool', async () => {
const route = await import('../../src/routes/api/server-keys/+server.js');
const first = await route.PUT(request('PUT', { game: 't6', plutoKey: 'private-key', label: 'First label' }));
const repeated = await route.PUT(request('PUT', { game: 't6', plutoKey: ' private-key ', label: 'Updated label' }));
expect(await repeated.json()).toEqual(await first.json());
expect((await route.GET().json()).keys).toEqual([expect.objectContaining({
game: 't6', label: 'Updated label', assigned_servers: []
})]);
});
it('rejects removal while assigned and permits it after the key is released', async () => {
const route = await import('../../src/routes/api/server-keys/+server.js');
const { reserveServerKey, releaseServerKey, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
const saved = await route.PUT(request('PUT', { game: 't5', plutoKey: 'private-key' }));
const { id } = await saved.json();
reserveServerKey('t5_zombie_moon_1', 't5');
const blocked = await route.DELETE(request('DELETE', { id }));
expect(blocked.status).toBe(400);
expect(await blocked.json()).toMatchObject({ ok: false, error: expect.stringMatching(/reserved|assigned|in use/i) });
expect(getServerKey('t5_zombie_moon_1')).toBe('private-key');
releaseServerKey('t5_zombie_moon_1');
expect(await (await route.DELETE(request('DELETE', { id }))).json()).toEqual({ ok: true });
expect(await route.GET().json()).toEqual({ keys: [] });
});
});
+4 -4
View File
@@ -31,8 +31,8 @@ beforeEach(async () => {
const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js');
await seedServerRegistry(); await seedServerRegistry();
database.db().prepare('UPDATE servers SET game_port = 28960 WHERE name = ?').run(server); database.db().prepare('UPDATE servers SET game_port = 28960 WHERE name = ?').run(server);
const { saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); const { reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js');
saveServerKey(server, 'saved-key'); reserveServerKey(server, 't5', 'saved-key');
refreshHost = (await prepareNativeHost([server])).heartbeat; refreshHost = (await prepareNativeHost([server])).heartbeat;
backend = await import('../../src/lib/server/backends/native.js'); backend = await import('../../src/lib/server/backends/native.js');
vi.spyOn(backend, 'nativeHealth').mockResolvedValue({ ok: true, state: 'running' }); vi.spyOn(backend, 'nativeHealth').mockResolvedValue({ ok: true, state: 'running' });
@@ -504,8 +504,8 @@ describe('Watchdog APIs', () => {
describe('Watchdog drift campaigns', () => { describe('Watchdog drift campaigns', () => {
it('keeps a missing rebuild key visible as attention without deleting files or retrying automatically', async () => { it('keeps a missing rebuild key visible as attention without deleting files or retrying automatically', async () => {
const { removeServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); // Simulate legacy registry damage; administrators cannot remove a leased key.
removeServerKey(server); database.db().prepare('DELETE FROM server_key_leases WHERE server_id=?').run(server);
const path = join(fixture.t5, server, 'runtime/plutonium/storage/t5/raw/scripts/sp', 'stray.gsc'); const path = join(fixture.t5, server, 'runtime/plutonium/storage/t5/raw/scripts/sp', 'stray.gsc');
await writeFile(path, 'unexpected'); await writeFile(path, 'unexpected');
await watchdog.watchdogTick(true); await watchdog.watchdogTick(true);