diff --git a/docs/guides/authentication.md b/docs/guides/authentication.md index 2570047..e49a6ae 100644 --- a/docs/guides/authentication.md +++ b/docs/guides/authentication.md @@ -42,7 +42,7 @@ guessable token. An unset setup token holds setup closed. | Create users, change roles, disable accounts and reset passwords | No | No | Yes | | Own password and sign out | Yes | Yes | Yes | -Operators can launch using server keys already saved by an administrator. They +Operators can launch using available game-pool keys already saved by an administrator. They cannot save keys indirectly through a launch request. An operator's content permissions include executable scripts and plugins; assign that role only to people trusted to operate the game fleet. diff --git a/docs/guides/autoscaling.md b/docs/guides/autoscaling.md index f0cc01b..ead83a0 100644 --- a/docs/guides/autoscaling.md +++ b/docs/guides/autoscaling.md @@ -8,7 +8,7 @@ Autoscaling is opt-in. Installing the feature creates no policies and changes no Choose **Add policy**, select the game and scope, and enter an explicit maximum. The initial editor selects Dry run, with a minimum of one running lobby, one empty spare, a 20-minute empty interval, and a five-minute action cooldown. Counts must be between 1 and 20; the minimum and spare target cannot exceed the maximum. -Choose **Preview changes** to inspect the existing lobbies, saved key coverage, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving. +Choose **Preview changes** to inspect the existing lobbies, game key availability, target, proposed action, and blockers. **Save reviewed policy** checks the policy revision and current fleet again. If the preview changed, review a new preview before saving. | Mode | Behavior | | --- | --- | @@ -17,15 +17,15 @@ Choose **Preview changes** to inspect the existing lobbies, saved key coverage, | Create only | Adds a lobby when eligible. Surplus removal remains advisory. | | Create and destroy | Adds lobbies and removes surplus lobbies after continuous empty evidence and all removal checks pass. | -Creating a policy in any mode, including Dry run or Off, requires saved Plutonium keys for enough exact lobby identities to support the configured maximum on every selected map. Preview lists each shortage, and Save rechecks coverage transactionally. Non-Off edits also require coverage; an existing policy can always be turned Off if keys have been removed. Only supported lobby identities numbered 1–20 with nonblank saved keys count. Keys may be nonconsecutive: if only lobby 3 and lobby 8 are keyed, those identities can be used. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host. +Plutonium keys belong to a game pool shared by every map, manual launch, collection, beta, and autoscaling policy for that game. An eligible creation uses the lowest available lobby number from 1–20 and reserves any free game key atomically. Key labels do not select a map or lobby. The preview shows saved and available game keys; policy maxima can exceed pool size, and a policy can be saved before any keys are added. An empty or exhausted pool holds new growth with a clear reason; it does not prevent surplus removal. Queued launches reserve their key, stopped instances retain it, and confirmed cleanup returns it to the pool. Failed or uncertain launches keep their reservation until safely reconciled. Existing lobbies and unfinished operations must be healthy and settled before activation. Automatic removal also requires agent **1.7.0 or later** advertising `instance.autoscaling-remove` on each affected host. The desired count is `min(maximum, max(minimum, occupied lobbies + empty spare target))`. The controller evaluates policies every 15 seconds and requests at most one unresolved operation per policy. Game-wide policies share one cooldown and one outstanding operation across their maps, prioritizing additions before removals. Create uses the existing startup concurrency, CPU, memory, library, port, and host admission checks. Lack of host capacity leaves one request waiting; it does not create additional requests every tick. Creates use current promoted launch files when provisioning begins. ## Ownership and blockers -A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected; dry-run policies may overlap. Game-wide rules include every catalog map and require key coverage for the maximum on each map. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish. +A live policy owns the lobby counts for its selected maps. Overlapping live game-wide and map-specific policies are rejected; dry-run policies may overlap. Game-wide rules include every catalog map and apply each map’s targets independently while sharing the game’s key pool. Pool capacity is a game-wide limit; targets across maps are not guaranteed when all keys are in use. Competing collection launches and manual Create, Start, Stop, Rebuild, or Destroy requests are rejected with a policy link. Exact collection matching also checks managed maps absent from the collection. To hand control back, set the policy to Off and wait for unsettled operations to finish. -Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and missing-key waits. +Unknown or stale occupancy, stopped or unhealthy members, active betas, campaigns, maintenance, host drains, and unresolved recovery hold the policy's decisions. Autoscaling does not repair these conditions itself. The page and linked Provisioning operation explain the current blocker, including capacity and game key waits. Current positive occupancy from either telemetry or RCON vetoes an empty claim. Empty intervals reset when players arrive, evidence becomes unknown, the instance or lifecycle changes, or observation continuity breaks. Controller downtime does not count toward the empty duration. Failed or uncertain automatic operations pause further actions until an operator inspects their Provisioning recovery and reviews the policy again. diff --git a/docs/guides/collections.md b/docs/guides/collections.md index fdaf47d..71d1ab4 100644 --- a/docs/guides/collections.md +++ b/docs/guides/collections.md @@ -32,8 +32,9 @@ It fills deficits without changing existing servers' files. 3. Choose **Save & preview** (or **Preview fleet changes** on a saved collection). Choose **Fill missing servers** or **Match collection exactly**, then review the server IDs and estimated capacity. Native host ports are assigned later. -4. Lobbies with a saved key show **Saved key ready** and reuse it automatically. - Enter a Plutonium key only for lobbies missing one. Update saved keys in **Keys**. +4. Launches covered by unused keys show **Game key available**. Each game’s + available keys are counted once across the full lineup. Add an unused Plutonium + key for each remaining launch, or manage the game pools in **Keys**. 5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact mode) and follow the collection progress page. If every target is already met, choose **Done**; no operation is created. @@ -48,7 +49,8 @@ servers keep their containers, files, ports, and configuration without a restart The option is chosen per launch; opening another preview defaults to fill mode. Review the removal list and check its confirmation before applying. Removed -servers lose their containers, files, and registrations; saved keys remain. +servers lose their containers, files, and registrations; their keys return to the +game pool only after confirmed removal. Unstarted creations are cancelled without booting a container. Running servers use the normal removal queue and require a fresh zero-player RCON response. Occupied or unreachable servers wait. Already pending removals are reused. @@ -59,6 +61,11 @@ map, port, or pending-removal change since an exact preview requires a new revie Servers added after submission are not included. Applying the same preview again returns the same collection without repeating operations. +Keys assigned to pending removals stay reserved until cleanup is confirmed. An +exact collection that replaces running servers still needs enough currently +available game keys for every new launch. If the pool is full, complete the +removals first, then refresh the preview to use the released keys. + Native launches can be queued even when the current port range is full. Removed servers retain their ports until cleanup completes; provisioning assigns newly available ports as each creation starts. A failed or cancelled removal can leave @@ -94,11 +101,11 @@ is a warning: provisioning can wait for it. The preview does not reserve resourc Apply validates the reviewed configuration, lobby IDs, keys, and fleet changes, saves all jobs in one database transaction, and returns the progress page ID. Native jobs initially have no host or port; each job performs its capacity checks asynchronously during -provisioning. If a server lacks a valid key or its reviewed identity changed, +provisioning. If a server lacks an available game key or its reviewed identity changed, no jobs or key changes from that submission are committed. Legacy submissions also revalidate their reviewed ports. Refreshing -the preview retains typed keys only for unchanged server IDs still missing a saved -key, and resets removal confirmation. +the preview retains typed keys for launches in the same game that still need +keys, and resets removal confirmation. Submitting the same preview again returns the same collection, including after a lost response or an application restart. Reusing the collection with a new @@ -138,13 +145,15 @@ remain visible in Provisioning activity and history. 2. Choose the current promoted setup and select a game/map, or select a collection. 3. Choose **Preview launch**. Review the image reference, configuration sources, managed files and versions, selected map, lobby, port, and prerequisite checks. -4. Enter the Plutonium key for the displayed server ID, or reuse its saved key. +4. Use an available key from the game pool, or enter an unused key for that game. 5. Choose **Launch server** and follow the operation to verified health. A preview lasts 30 minutes. It neither reserves capacity nor saves a key. If another launch takes the lobby, submission stops before saving the key; -refresh the preview and review the newly assigned ID. Refreshing a preview clears -the typed key so it cannot accidentally be saved against a different lobby. +refresh the preview and review the newly assigned ID. A typed key can follow +the new lobby within the same game. Pool availability is checked again at +submission, and keys are reserved atomically so concurrent launches cannot share +a credential. Host selection, game-file versions, runtime configuration and lobby availability are checked before queueing. Provisioning assigns host capacity and physical @@ -205,7 +214,8 @@ and drift reconciliation using the server's game/map scopes. Container tags can resolve to newer images; use a digest for a fixed image. Without a promoted baseline, Plutainer seeds its own defaults on first boot, which are not captured. Host, network, game-file volumes, destination directories, -and RCON/Plutonium credentials use the environment and key store at launch time. +and RCON/Plutonium credentials use the environment and assigned game-pool key at +launch time. The current main config filename and launch settings apply to new deployments. Recipes do not include the separately stored Plutonium or integration credentials. @@ -249,7 +259,9 @@ there while preserving query parameters. where mode is `fill` (default) or `exact`. Returns `mode`, `retainedServers`, `removals` (name/reason), creation `servers`, and configured `launchConcurrency`. - `POST /api/deployment-collections`: atomically queue `{ previewId, keys? }`, - where `keys` maps exact preview server IDs to optional replacement keys. + where `keys` maps preview server IDs to optional unused keys added to their + game's pool and assigned during launch. Omit entries covered by available + pool keys; these IDs do not permanently bind a key to a map or lobby. A satisfied preview returns `{ ok: true, collectionId: null }` with HTTP 200, without creating a collection, saving keys, or kicking the worker. - `GET /api/deployment-collections`: list collection progress summaries with diff --git a/docs/guides/getting-started.md b/docs/guides/getting-started.md index 3ef5c6e..81c04e4 100644 --- a/docs/guides/getting-started.md +++ b/docs/guides/getting-started.md @@ -10,7 +10,7 @@ HTTPS. Running game containers continue through controller or agent outages. - A controller machine with Docker Compose and an HTTPS reverse proxy. - Debian 12 or 13 amd64 game hosts with storage for game files and instances. - Your complete T5 or T6 game folders, including the DLC you intend to host. -- Plutonium server keys for the exact lobby identities you will launch. +- Plutonium server keys for each game, with one unused key per new concurrent lobby. The controller does not need access to a Docker socket, game-host mounts, or inbound RCON. Host agents handle local game commands and file operations. @@ -67,13 +67,18 @@ base game files; each instance has its own writable data. See ## Launch your first server -1. Save the exact Plutonium server keys under **Administration → Keys**. +1. Save Plutonium server keys in the T5 or T6 pool under **Administration → Keys**. 2. Create a server, or build a reusable map lineup in **Fleet → Collections**. 3. Review the launch and apply it. Provisioning queues the work, then assigns a host and port when capacity is available. 4. Follow the operation until readiness checks pass, then verify that players can find and join the server. +Keys are shared across maps within their game. A launch assigns one available key +until the server is removed; stopped servers keep their keys. Optional key labels +can record the hostname chosen in Plutonium without binding the key to a map or +lobby. Electric Cherry does not change the hostname registered with Plutonium. + New launches use the latest verified game files and promoted configuration, scripts, and plugins. Uploading newer game files does not restart existing servers. Disruptive operations normally wait for a fresh empty lobby or a diff --git a/docs/guides/native-hosts.md b/docs/guides/native-hosts.md index 521cc4e..ea2b720 100644 --- a/docs/guides/native-hosts.md +++ b/docs/guides/native-hosts.md @@ -33,7 +33,8 @@ the agent owns local execution and its durable operation journal. before launching. Apply saves the requested lobby IDs. Provisioning selects a host and atomically reserves UDP/TCP ports when capacity is available. - Saved Plutonium server keys remain associated with their exact server names. + An available Plutonium key from the game pool is reserved for each launch. + Its assignment survives stops and rebuilds and ends after confirmed removal. A fresh destination host downloads centrally stored game files without manual copying or per-container setup. EC does not supply the game content itself. diff --git a/docs/guides/provisioning.md b/docs/guides/provisioning.md index 0205570..7603258 100644 --- a/docs/guides/provisioning.md +++ b/docs/guides/provisioning.md @@ -30,9 +30,10 @@ or rebuild on its selected servers. Rebuild requires confirmation because **no instance data is preserved**. Its fresh files come from the latest promoted setup when creation begins, including changes -promoted while waiting. The independently saved key in **Keys** must exist before -teardown. Blueprint definitions, promoted artifacts, saved keys, and historical -activity belong to Electric Cherry, and are not instance files. +promoted while waiting. Its assigned key in **Keys** must exist before teardown +and stays reserved throughout the rebuild. Blueprint definitions, promoted +artifacts, saved keys, and historical activity belong to Electric Cherry, and are +not instance files. Rebuild reserves the original ID and port throughout teardown and creation, even when a lower lobby is available. It shares the existing creation queue and startup @@ -58,6 +59,22 @@ maintenance. A successful agent receipt is followed by actual container and game slot and are not automatically replayed. Stop requires an explicit stopped container state; an unreachable host agent is not evidence that a stop succeeded. +### Game key pools + +Each create reserves one unused Plutonium key for its game before the launch is +queued. T5 and T6 have independent pools; any map or lobby within that game can +use a free key. Stopping a server keeps the reservation, and a rebuild keeps the +same key. Confirmed removal releases it for another launch. Failed or ambiguous +creation keeps the assignment while cleanup may still be required. + +An unstarted creation cancelled before dispatch releases its key immediately. +Administrators can add and remove available keys in **Administration → Keys**; +assigned keys cannot be removed. Existing keys are migrated into their game +pool, retaining assignments for existing servers and pending work. If a legacy +rebuild has an uncertain result and used a different key, both possible keys +remain reserved until that server is destroyed. Labels are for identification +only and do not change the hostname registered with Plutonium. + ### Player checks and administrator overrides Disruptive operations require a fresh zero-player result from RCON or accepted diff --git a/docs/plans/lobby-autoscaling-plan.md b/docs/plans/lobby-autoscaling-plan.md index 705cfc1..cda850b 100644 --- a/docs/plans/lobby-autoscaling-plan.md +++ b/docs/plans/lobby-autoscaling-plan.md @@ -4,7 +4,7 @@ Status: Approved and implemented October 6, 2026. Controller support and agent 1 The design below preserves the approved scope and subsequent implementation record. Use the linked guide for current policy behavior, including game-wide -policies and saved-key requirements added after the initial design. +policies and shared game key pools added after the initial design. Introduce an opt-in policy for each game and map variant that keeps a small number of empty, ready lobbies available. When the last spare fills, request another lobby through Provisioning. Remove surplus lobbies only after sustained, confirmed inactivity. This reduces idle hosting while preserving room for new groups, including when existing Zombies sessions have free player slots but are already well into a game. @@ -19,7 +19,7 @@ Start with Create/Destroy on existing native hosts. Run the evaluator in dry run | Safe creation | Use the durable creation intent and placement path in [provisioning/native.ts](../../src/lib/server/provisioning/native.ts). Preserve CPU, memory, port, host eligibility, library, and startup concurrency checks. A blocked create remains queued. | | Safe removal | Reuse the controller occupancy checks, instance generation checks, host-local check, and cleanup checkpoints. Add autoscaling-specific admission and cancellation behavior before enabling automatic destruction. | | Requester eligibility | [provisioning/worker.ts](../../src/lib/server/provisioning/worker.ts) currently gives Create and Destroy separate paths that bypass the runtime requester eligibility check. Autoscaling ownership and policy checks must cover both paths, including delayed execution. A source label alone is insufficient. | -| Saved keys | [server-keys.ts](../../src/lib/server/fleet/server-keys.ts) stores keys by exact lobby name and accepts lobby numbers 1–20. Allocate only unused identities with saved keys; never copy a key to a different identity. | +| Saved keys | [server-keys.ts](../../src/lib/server/fleet/server-keys.ts) stores keys in game pools. Each live or pending launch reserves a key until cleanup confirms it is safe to reuse. Allocate lobby identities numbered 1–20 independently of the key label. | | Collections | [collections.ts](../../src/lib/server/fleet/collections.ts) already counts queued creations and launches promoted recipes. Exact matching can remove lobbies across the fleet, including maps omitted from the collection. Add ownership checks to preview and submission. | | Hosting insights | [metrics-insights.ts](../../src/lib/metrics-insights.ts) provides historical suggestions for hosting trials. Use these to help choose policies; the live evaluator needs current empty-lobby evidence and must not execute historical suggestions directly. | @@ -39,7 +39,7 @@ Use one policy per `(game, mapSlug)`, preserving configured variants such as T6 | Action cooldown | Default 5 minutes between new policy actions, measured from durable admission. Cancellation and safety checks do not wait for cooldown. | | Launch settings | Reuse normal map recipes and automatic placement on eligible active hosts. Use current promoted files when provisioning begins. | -Validate integer counts with `1 <= minimum <= maximum <= 20` and `1 <= spare target <= maximum`. Show when saved-key coverage limits the configured maximum; block live activation until enough usable identities have keys. Subsequent key loss blocks affected growth and does not trigger removal. Keys need not occupy consecutive lobby numbers. +Validate integer counts with `1 <= minimum <= maximum <= 20` and `1 <= spare target <= maximum`. Show the game pool’s saved and available counts independently of each map’s maximum. Policies can be saved with fewer keys than the maximum, or with no keys yet. Exhaustion holds growth across every map sharing that game, while eligible surplus removal can still return capacity. Game-wide targets are per map, but game keys are shared across all maps and launch paths. Live activation explicitly hands management of the game's map variant to the policy and adopts its eligible existing native lobbies. Show those lobbies, exclusions, current keys, and the first proposed action in the activation preview. Do not activate across unresolved provisioning, collection launches, or ambiguous identity ownership. Dry run observes without acquiring ownership or reserving names, ports, or capacity. @@ -65,7 +65,7 @@ Evaluate in this order: 1. Reconcile existing autoscaling intent and any unsettled host commands. Cancel an eligible queued removal if players arrive, evidence becomes unknown, the policy changes, a conflict appears, or removal would breach the minimum or spare target. 2. Apply mode, ownership, evidence, conflict, failure, and cooldown gates. Never submit a second policy action while a prior one remains unresolved. Count external reservations as well as policy jobs against the maximum. -3. If the ready count is below the desired count, propose one create using the lowest available keyed lobby identity. Recheck identity availability and the policy revision atomically when persisting the request. Capacity waiting must reuse this request rather than add another each tick. +3. If the ready count is below the desired count, propose one create using the lowest available lobby identity and any free key from that game. Recheck identity availability, game key availability, and the policy revision atomically when persisting the request. Admission rechecks recognize the pending launch’s own held key. Capacity waiting must reuse this request rather than add another each tick. 4. If the ready count exceeds the desired count, select one continuously empty eligible lobby whose removal preserves both the minimum and spare target. Prefer the longest-confirmed empty lobby, with lobby number as a stable tie-breaker. Create only mode records this opportunity without executing it. 5. Otherwise record a stable no-action reason. Deduplicate unchanged decisions so the 15-second timer does not flood history. @@ -99,7 +99,7 @@ Expose authenticated read and update routes using the existing fleet-management ## Operator experience and measurements -Add an Autoscaling view linked from Fleet, Collections, and Metrics. Show each map's mode, occupied and empty ready counts, pending work, minimum, maximum, spare target, key coverage, last evaluation, next eligible action, and current blocker. Detail includes the policy editor, affected lobbies, recent decisions, Provisioning links, and pause/resume controls. Dry run actions must be visibly hypothetical. +Add an Autoscaling view linked from Fleet, Collections, and Metrics. Show each map's mode, occupied and empty ready counts, pending work, minimum, maximum, spare target, game key availability, last evaluation, next eligible action, and current blocker. Detail includes the policy editor, affected lobbies, recent decisions, Provisioning links, and pause/resume controls. Dry run actions must be visibly hypothetical. Activation and edits preview the resulting target and any proposed removals. Pause stops new decisions immediately and shows which queued actions were withdrawn and which delivered operations remain unresolved. Do not claim the fleet was restored when a policy is disabled. @@ -111,7 +111,7 @@ Compare equivalent weekly periods and record game/runtime versions, player-hours Complete and validate each phase in a separate commit before starting the next. -1. **Policy model and planner.** Add additive schema changes, identity-aware snapshots, policy validation, decision types, and the pure evaluator. Keep all modes non-mutating until admission integration exists. Acceptance: deterministic fixtures cover the target formula, variant separation, incomplete evidence, saved-key gaps, bounds, and continuous empty timing. +1. **Policy model and planner.** Add additive schema changes, identity-aware snapshots, policy validation, decision types, and the pure evaluator. Keep all modes non-mutating until admission integration exists. Acceptance: deterministic fixtures cover the target formula, variant separation, incomplete evidence, game pool exhaustion and reuse, bounds, and continuous empty timing. 2. **Dry run and visibility.** Register the requester, persist bounded decision history, and add the policy UI/API with dry run enabled. Acceptance: repeated ticks and controller restarts produce understandable decisions without creating Provisioning jobs or reserving identities, ports, or hosts. Begin the two-week observation period on representative maps. 3. **Ownership and live creation.** Add transactional intent, Provisioning source eligibility for Create/Destroy, collection/manual conflict checks, and Create only mode. Acceptance: no duplicate creation under concurrent ticks or submissions; no growth beyond the maximum; no capacity bypass; blocked creates remain single durable jobs. Pilot one or two maps after reviewing dry run evidence. 4. **Safe removal.** Implement continuous-empty selection, admission rechecks, queued cancellation, and the agent authorization contract above. Acceptance: positive/unknown occupancy blocks destruction; a held command cannot remove a lobby later without renewed eligibility; policy pause, revision changes, and new protected work invalidate pending intent. Enable Create and destroy only after these checks pass. @@ -124,7 +124,8 @@ Required behavioral scenarios across these phases: | Last spare fills, including a high-round partially filled session | One create when eligible; free player slots do not count as spares. | | Create is queued, starting, or awaiting verification | Reservation counts once; it is not advertised as ready and no duplicate is queued. | | Every eligible host is full or unavailable | Existing job waits; other policies and worker tasks continue. | -| Key is missing, removed, or rejected | Clear blocker or paused failure; no alternate identity receives the same key automatically. | +| Game key pool is empty or exhausted | Hold growth with a clear reason; other games remain independent and eligible surplus removal can proceed. | +| A launch is queued, stopped, failed, or uncertain | Its key stays reserved until confirmed cleanup; no other launch can use that key concurrently. | | Occupancy evidence conflicts, expires, or arrives from an old lifecycle | Positive current occupancy wins; invalid evidence cannot establish emptiness. | | Players join before claim, after dispatch, or while the agent holds removal | Queued work is withdrawn where possible; agent safety prevents occupied destruction; held autoscaling intent cannot execute later without a fresh decision. | | Process restarts before or after intent submission | Existing operation is recovered by durable identity without replay or duplicate admission. | @@ -153,6 +154,6 @@ An isolated Firefox review exercised policy creation, preview, save, keyboard fo Game-wide policy follow-up: selecting every map applies the targets independently to each catalog map and variant. Game-wide policies serialize actions across maps and cannot overlap live map-specific ownership. Existing map policies remain compatible. -Key coverage follow-up: creation in every mode requires sufficient saved keys for the maximum on each selected map. Coverage is rechecked on save, including edits to active or dry-run policies. Turning an existing policy Off remains available after key loss. +Game key pool follow-up: keys are shared by all launch paths for a game. Lobby names and key labels are independent. Policy maxima no longer require dedicated key coverage on every map, and policies can be saved before adding keys. Snapshots account for reservations on other maps, planner creation waits when the pool is exhausted, and admission accepts its own reserved key. Keys are released only after confirmed cleanup. -Follow-up validation: 40 autoscaling tests and 118 related provisioning, collection, and cancellation tests passed, along with type checking and the production build. An isolated Firefox review verified that missing per-map keys disable Save, complete coverage permits game-wide dry-run creation, and keyboard navigation and 1440, 390, and 320 pixel layouts work. +Follow-up validation: 40 autoscaling tests and 118 related provisioning, collection, and cancellation tests passed, along with type checking and the production build. The earlier isolated Firefox review covered keyboard navigation and 1440, 390, and 320 pixel layouts. Its per-map key coverage checks described the previous behavior, superseded by game key pools. diff --git a/src/lib/autoscaling.ts b/src/lib/autoscaling.ts index 0b4947d..619c812 100644 --- a/src/lib/autoscaling.ts +++ b/src/lib/autoscaling.ts @@ -19,7 +19,7 @@ export interface ScalingMember { emptyStartedAt?: number | null; } export interface ScalingSnapshot { - members: ScalingMember[]; reservedNames: string[]; keyedNames: string[]; blockers: string[]; + members: ScalingMember[]; reservedNames: string[]; keyPool: { total: number; available: number }; blockers: string[]; } export interface ScalingDecision { action: 'create' | 'destroy' | 'wait'; reason: string; server: string | null; diff --git a/src/lib/server/autoscaling/index.ts b/src/lib/server/autoscaling/index.ts index 8f11357..cdc5f75 100644 --- a/src/lib/server/autoscaling/index.ts +++ b/src/lib/server/autoscaling/index.ts @@ -2,7 +2,7 @@ import { createHash, randomUUID } from 'node:crypto'; import { db } from '../db/index.js'; import { liveScaling, scalingSettings, type ScalingPolicy, type ScalingPreview, type ScalingReport, type ScalingView } from '../../autoscaling.js'; import { emptyObservation, planScaling } from './planner.js'; -import { zombieMaps, parseServerName } from '../../map-aliases.js'; +import { parseServerName } from '../../map-aliases.js'; import { scalingSnapshot } from './snapshot.js'; import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisioning/native.js'; import { cancelCreation } from '../provisioning/cancel-creation.js'; @@ -30,13 +30,6 @@ export function previewScaling(input: unknown, id?: string, revision?: number): nextActionAt: existing?.nextActionAt ?? 0, pausedReason: null, updatedAt: existing?.updatedAt ?? '' }; const snapshot = scalingSnapshot(policy); const blockers: string[] = []; - // Creating any policy requires full coverage; existing policies must always be pausable. - if (!existing || policy.mode !== 'off') { - for (const map of zombieMaps(policy.game).filter(m => policy.mapSlug === '*' || m.slug === policy.mapSlug)) { - const count = snapshot.keyedNames.filter(name => parseServerName(name)?.mapSlug === map.slug).length; - if (count < policy.maximum) blockers.push(`Save keys for at least ${policy.maximum} lobby identities for ${map.friendly} before saving this policy (${count} saved; ${policy.maximum - count} more required).`); - } - } if (liveScaling(policy.mode)) { const owner = scalingOwner(policy.game, policy.mapSlug); if (owner && owner !== policy.id) blockers.push(`An overlapping policy owns this lineup. Pause it and settle pending work first: /autoscaling#${owner}`); @@ -51,7 +44,7 @@ export function previewScaling(input: unknown, id?: string, revision?: number): } const decision = planScaling(policy, snapshot, Date.now()); const token = createHash('sha256').update(JSON.stringify({ settings, revision: policy.revision, globalPaused: scalingGloballyPaused(), - members: snapshot.members.map(m => [m.name, m.identity, m.state]), keys: snapshot.keyedNames.sort(), reservations: snapshot.reservedNames.sort(), blockers: snapshot.blockers, decision })).digest('hex'); + members: snapshot.members.map(m => [m.name, m.identity, m.state]), keyPool: snapshot.keyPool, reservations: snapshot.reservedNames.sort(), blockers: snapshot.blockers, decision })).digest('hex'); return { policy, snapshot, decision, token, blockers }; } @@ -154,7 +147,7 @@ export async function runAutoscaling(): Promise { ON CONFLICT(policy_id,server) DO UPDATE SET identity=excluded.identity,observation=excluded.observation`).run(policy.id, member.name, member.identity, JSON.stringify(observation)); } // Evidence includes provenance and timestamps but never keys or player rosters. - const evidence = JSON.stringify({ members: snapshot.members, reservations: snapshot.reservedNames, keyCount: snapshot.keyedNames.length, blockers: snapshot.blockers }); + const evidence = JSON.stringify({ members: snapshot.members, reservations: snapshot.reservedNames, keyPool: snapshot.keyPool, blockers: snapshot.blockers }); const fingerprint = createHash('sha256').update(JSON.stringify([policy.revision, decision, snapshot.members.map(m => [m.name, m.identity, m.state])])).digest('hex'); const last = db().prepare('SELECT id,fingerprint FROM autoscaling_decisions WHERE policy_id=? ORDER BY evaluated_at DESC,rowid DESC LIMIT 1').get(policy.id) as { id: string; fingerprint: string } | undefined; const execute = liveScaling(policy.mode) && (decision.action === 'create' || policy.mode === 'create-destroy' && decision.action === 'destroy'); diff --git a/src/lib/server/autoscaling/planner.ts b/src/lib/server/autoscaling/planner.ts index 2ecc876..5113c70 100644 --- a/src/lib/server/autoscaling/planner.ts +++ b/src/lib/server/autoscaling/planner.ts @@ -21,7 +21,7 @@ export function planScaling(policy: ScalingPolicy, snapshot: ScalingSnapshot, no const matches = (name: string) => (snapshot.members.find(m => m.name === name)?.mapSlug ?? parseServerName(name)?.mapSlug) === map.slug; return planScaling({ ...policy, mapSlug: map.slug }, { ...snapshot, members: snapshot.members.filter(m => matches(m.name)), - reservedNames: snapshot.reservedNames.filter(matches), keyedNames: snapshot.keyedNames.filter(matches) + reservedNames: snapshot.reservedNames.filter(matches) }, now); }); // Serialize game-wide work and prefer replenishing any map before trimming another. @@ -50,8 +50,12 @@ export function planScaling(policy: ScalingPolicy, snapshot: ScalingSnapshot, no if (policy.nextActionAt > now) return result('wait', 'Action cooldown is active.'); if (occupied + empty < desired) { if (snapshot.reservedNames.length >= policy.maximum) return result('wait', 'Maximum lobby reservations reached.'); - const available = snapshot.keyedNames.filter(name => !snapshot.reservedNames.includes(name)).sort((a, b) => a.localeCompare(b, 'en', { numeric: true })); - return available.length ? result('create', 'Replenish empty lobby capacity.', available[0]) : result('wait', 'No unused lobby identity has a saved key.'); + if (!snapshot.keyPool.available) return result('wait', snapshot.keyPool.total + ? `All ${policy.game.toUpperCase()} Plutonium keys are in use. Waiting for a free key.` + : `No Plutonium keys are saved for ${policy.game.toUpperCase()}.`); + const available = Array.from({ length: 20 }, (_, index) => `${policy.game}_zombie_${policy.mapSlug}_${index + 1}`) + .find(name => !snapshot.reservedNames.includes(name)); + return available ? result('create', 'Replenish empty lobby capacity.', available) : result('wait', 'No unused lobby identity is available.'); } if (occupied + empty > desired && empty > policy.spares && occupied + empty > policy.minimum) { const candidates = members.filter(m => m.state === 'empty' && m.emptySince !== null && m.observedAt! - m.emptySince >= policy.emptyMinutes * 60_000) diff --git a/src/lib/server/autoscaling/snapshot.ts b/src/lib/server/autoscaling/snapshot.ts index cf4929a..4b74c49 100644 --- a/src/lib/server/autoscaling/snapshot.ts +++ b/src/lib/server/autoscaling/snapshot.ts @@ -1,5 +1,5 @@ import { db } from '../db/index.js'; -import { validServerId } from '../fleet/server-keys.js'; +import { serverKeyPool } from '../fleet/server-keys.js'; import { listServers, type ServerRow } from '../fleet/servers.js'; import { observationReader } from '../fleet/status.js'; import { parseServerName, zombieMaps } from '../../map-aliases.js'; @@ -73,6 +73,8 @@ export function scalingSnapshot(policy: ScalingPolicy, now = Date.now(), exceptO reservedNames.add(row.name); if (!members.some(m => m.name === row.name)) blockers.push(`Unresolved native instance ${row.name}.`); } - const keyedNames = (handle.prepare("SELECT server_id AS name FROM server_keys WHERE length(trim(pluto_key))>0").all() as { name: string }[]).filter(row => validServerId(row.name) && matches(row)).map(r => r.name); - return { members, reservedNames: [...reservedNames], keyedNames, blockers }; + for (const row of handle.prepare('SELECT server_id AS name FROM server_key_leases').all() as { name: string }[]) if (matches(row)) reservedNames.add(row.name); + const ownServer = exceptOperation ? (handle.prepare('SELECT server FROM server_deployments WHERE id=?').get(exceptOperation) as { server: string } | undefined)?.server : undefined; + const { total, available } = serverKeyPool(policy.game, ownServer); + return { members, reservedNames: [...reservedNames], keyPool: { total, available }, blockers }; } diff --git a/src/lib/server/db/index.ts b/src/lib/server/db/index.ts index a4a0b7c..3c6e9f0 100644 --- a/src/lib/server/db/index.ts +++ b/src/lib/server/db/index.ts @@ -10,6 +10,7 @@ import { backfillLeaderboard } from '../fleet/leaderboard-ingest.js'; import { initializeFleetSettings } from './fleet-settings.js'; import { migrateServerHistory } from './server-history.js'; import { migrateNativeInstances } from '../backends/schema.js'; +import { migrateServerKeys } from './server-keys.js'; import { ensureAutoscalingSchema } from '../autoscaling/schema.js'; import { installPerformanceIndexes } from './performance-indexes.js'; import { migrateFileBlobs } from './file-blobs.js'; @@ -397,6 +398,7 @@ export function db(): Database.Database { backfillLeaderboard(handle); initializeFleetSettings(handle); migrationStep('native instance schema and file plans', () => migrateNativeInstances(handle)); + migrationStep('per-game server key pools', () => migrateServerKeys(handle)); migrationStep('autoscaling schema', () => ensureAutoscalingSchema(handle)); // File-blob migration removes the legacy content column altogether. // Reordering it first would unnecessarily copy every artifact twice. diff --git a/src/lib/server/db/schema.sql b/src/lib/server/db/schema.sql index 4fbc43b..2b69e43 100644 --- a/src/lib/server/db/schema.sql +++ b/src/lib/server/db/schema.sql @@ -433,11 +433,15 @@ CREATE TABLE IF NOT EXISTS deployment_scripts ( ); CREATE INDEX IF NOT EXISTS idx_deployment_scripts_dep ON deployment_scripts(deployment_id); --- Deliberately independent of servers: credentials survive server destruction. +-- Credentials belong to a game and survive server destruction. Reservations +-- are installed by the key-pool migration after native instance schema exists. CREATE TABLE IF NOT EXISTS server_keys ( - server_id TEXT PRIMARY KEY, + id TEXT PRIMARY KEY, + game TEXT NOT NULL CHECK (game IN ('t5','t6')), + label TEXT NOT NULL DEFAULT '', pluto_key TEXT NOT NULL, - updated_at TEXT NOT NULL DEFAULT (datetime('now')) + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + UNIQUE(game, pluto_key) ); CREATE TABLE IF NOT EXISTS settings ( diff --git a/src/lib/server/db/server-keys.ts b/src/lib/server/db/server-keys.ts new file mode 100644 index 0000000..b8fde45 --- /dev/null +++ b/src/lib/server/db/server-keys.ts @@ -0,0 +1,100 @@ +import { randomUUID } from 'node:crypto'; +import type Database from 'better-sqlite3'; + +const keySchema = `CREATE TABLE IF NOT EXISTS server_keys ( + id TEXT PRIMARY KEY, + game TEXT NOT NULL CHECK (game IN ('t5','t6')), + label TEXT NOT NULL DEFAULT '', + pluto_key TEXT NOT NULL, + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + UNIQUE(game, pluto_key) +);`; + +const leaseSchema = `CREATE TABLE IF NOT EXISTS server_key_leases ( + server_id TEXT NOT NULL, + key_id TEXT NOT NULL REFERENCES server_keys(id) ON DELETE RESTRICT, + is_current INTEGER NOT NULL DEFAULT 1 CHECK (is_current IN (0,1)), + reserved_at TEXT NOT NULL DEFAULT (datetime('now')), + PRIMARY KEY(server_id,key_id) +); +CREATE INDEX IF NOT EXISTS idx_server_key_leases_key ON server_key_leases(key_id); +CREATE UNIQUE INDEX IF NOT EXISTS idx_server_key_leases_current ON server_key_leases(server_id) WHERE is_current=1;`; + +/** Preserve every occupied legacy assignment, even duplicated credentials. */ +export function migrateServerKeys(handle: Database.Database): void { + handle.transaction(() => { + const legacy = (handle.prepare('PRAGMA table_info(server_keys)').all() as { name: string }[]) + .some(column => column.name === 'server_id'); + if (legacy) handle.exec('ALTER TABLE server_keys RENAME TO legacy_server_keys'); + handle.exec(keySchema + leaseSchema); + if (!legacy) return; + const rows = handle.prepare(`SELECT server_id, pluto_key, updated_at FROM legacy_server_keys ORDER BY server_id`).all() as + { server_id: string; pluto_key: string; updated_at: string }[]; + const occupied = handle.prepare(`SELECT 1 WHERE + EXISTS (SELECT 1 FROM servers WHERE name=@server AND destroyed_at IS NULL) + OR EXISTS (SELECT 1 FROM native_instances WHERE server=@server AND retired_at IS NULL) + OR EXISTS (SELECT 1 FROM server_deployments WHERE server=@server AND server_deleted_at IS NULL + AND action IN ('create','rebuild') AND state NOT IN ('done','warning','cancelled'))`); + const find = handle.prepare('SELECT id FROM server_keys WHERE game=? AND pluto_key=?'); + const serverGame = handle.prepare("SELECT game FROM servers WHERE name=? AND game IN ('t5','t6')"); + const insert = handle.prepare('INSERT INTO server_keys(id,game,label,pluto_key,updated_at) VALUES(?,?,?,?,?)'); + const lease = handle.prepare('INSERT INTO server_key_leases(server_id,key_id,is_current) VALUES(?,?,?)'); + const assignments = new Map(); + function save(game: string, credential: string, label: string, updatedAt: string): string { + const existing = (find.get(game, credential) as { id: string } | undefined)?.id; + if (existing) return existing; + const id = randomUUID(); + insert.run(id, game, label, credential, updatedAt); + return id; + } + for (const row of rows) { + const game = /^(t5|t6)_/.exec(row.server_id)?.[1] + ?? (serverGame.get(row.server_id) as { game: string } | undefined)?.game; + if (!game) throw new Error(`Cannot determine the game for saved server key ${row.server_id}`); + const id = save(game, row.pluto_key, row.server_id, row.updated_at); + if (occupied.get({ server: row.server_id })) assignments.set(row.server_id, [id]); + } + // The old form could overwrite a saved key without changing the running + // container. Durable create/rebuild payloads are stronger evidence of + // its actual credential. Keep ambiguous delivered attempts as additional + // holds until the instance is destroyed; never free a possibly live key. + const hasTable = (name: string) => !!handle.prepare("SELECT 1 FROM sqlite_master WHERE type='table' AND name=?").get(name); + if (hasTable('ec_host_operations')) { + const payload = hasTable('ec_host_operation_payloads') + ? 'COALESCE((SELECT p.payload FROM ec_host_operation_payloads p WHERE p.operation_id=o.id),o.payload)' : 'o.payload'; + const command = `CASE WHEN json_valid(${payload}) THEN ${payload} ELSE '{}' END`; + const launched = handle.prepare(`SELECT n.server,o.state,o.updated_at, + json_extract(${command},'$.env.PLUTO_SERVER_KEY') AS credential, + json_extract(${command},'$.game') AS game, + json_extract(${command},'$.env.PLUTAINER_GAME') AS game_id + FROM native_instances n JOIN ec_host_operations o ON o.instance_id=n.id AND o.host_id=n.host_id + WHERE n.retired_at IS NULL AND o.kind IN ('instance.create','instance.rebuild') AND o.state!='failed' + AND (o.generation=n.generation OR json_extract(${command},'$.nextGeneration')=n.generation) + ORDER BY o.rowid DESC`).all() as + { server: string; state: string; updated_at: string; credential: unknown; game: unknown; game_id: unknown }[]; + const recovered = new Map(); + const complete = new Set(); + for (const row of launched) { + if (complete.has(row.server) || typeof row.credential !== 'string' || !row.credential) continue; + const game = row.game === 't5' || row.game === 't6' ? row.game + : row.game_id === 't5sp' ? 't5' : row.game_id === 't6zm' ? 't6' + : /^(t5|t6)_/.exec(row.server)?.[1] + ?? (serverGame.get(row.server) as { game: string } | undefined)?.game; + if (!game) throw new Error(`Cannot determine the game for launched server ${row.server}`); + const ids = recovered.get(row.server) ?? []; + const id = save(game, row.credential, row.server, row.updated_at); + if (!ids.includes(id)) ids.push(id); + recovered.set(row.server, ids); + if (row.state === 'completed') complete.add(row.server); + } + for (const [server, ids] of recovered) { + // If no completed launch receipt survives, its saved assignment + // remains a possible prior credential while the attempt resolves. + if (!complete.has(server)) for (const id of assignments.get(server) ?? []) if (!ids.includes(id)) ids.push(id); + assignments.set(server, ids); + } + } + for (const [server, ids] of assignments) ids.forEach((id, index) => lease.run(server, id, index === 0 ? 1 : 0)); + handle.exec('DROP TABLE legacy_server_keys'); + }).immediate(); +} diff --git a/src/lib/server/fleet/collections.ts b/src/lib/server/fleet/collections.ts index 484d190..872a367 100644 --- a/src/lib/server/fleet/collections.ts +++ b/src/lib/server/fleet/collections.ts @@ -13,6 +13,7 @@ import { parseServerName, zombieMapBySlug, zombieMaps } from './map-aliases.js'; import { nativeLaunchSelection, queueNativeCreationIntent } from '../provisioning/native.js'; import { nativeDeployment, nativeInstance } from '../backends/native.js'; import { launchAvailability } from '../hosts/availability.js'; +import { availableServerKeyCount } from './server-keys.js'; function coverage(entries: BlueprintEntry[], includeHealth = true) { type Server = { name: string; game: string | null; map_slug: string | null; map: string | null }; @@ -72,6 +73,8 @@ function collectionPlan(entries: BlueprintEntry[], mode: CollectionMode, include function allocate(entries: BlueprintEntry[], counts = coverage(entries)): CollectionPreview['servers'] { const occupied = db().prepare(`SELECT name FROM servers UNION + SELECT server AS name FROM native_instances WHERE retired_at IS NULL UNION + SELECT server_id AS name FROM server_key_leases UNION SELECT j.server AS name FROM server_creation_jobs j JOIN server_deployments d ON d.id=j.deployment_id WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued','running','awaiting-health') OR (d.action='rebuild' AND (d.state='waiting' OR (d.state IN ('failed','interrupted') AND d.stage='Removing server files and registry'))))`) @@ -83,7 +86,7 @@ function allocate(entries: BlueprintEntry[], counts = coverage(entries)): Collec const taken = lobbies.get(match[1]) ?? new Set(); taken.add(Number(match[2])); lobbies.set(match[1], taken); } - const savedKeys = new Set((db().prepare("SELECT server_id FROM server_keys WHERE length(trim(pluto_key))>0").all() as { server_id: string }[]).map(row => row.server_id)); + const availableKeys = { t5: availableServerKeyCount('t5'), t6: availableServerKeyCount('t6') }; return entries.flatMap((entry, entryIndex) => Array.from({ length: counts[entryIndex].missing }, () => { const base = `${entry.recipe.game}_zombie_${entry.recipe.mapSlug}`; const taken = lobbies.get(base) ?? new Set(); @@ -92,7 +95,9 @@ function allocate(entries: BlueprintEntry[], counts = coverage(entries)): Collec const native = entry.recipe.native; const port = null; if (lobby !== null) { taken.add(lobby); lobbies.set(base, taken); } - return { entryIndex, name, port, ...(native ? { native } : {}), hasSavedKey: !!name && savedKeys.has(name) }; + const hasSavedKey = !!name && availableKeys[entry.recipe.game] > 0; + if (hasSavedKey) availableKeys[entry.recipe.game]--; + return { entryIndex, name, port, ...(native ? { native } : {}), hasSavedKey }; })); } @@ -185,7 +190,7 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }): const blockers = [...blockersFor(servers, entries), ...removalBlockersFor(removals)]; if (blockers.length) throw new Error(blockers.join(' ')); if (entries.some((_entry, index) => counts[index].missing !== reviewed.filter(server => server.entryIndex === index).length)) { - throw new Error('Fleet counts changed. Nothing was queued. Refresh the preview and review each server key.'); + throw new Error('Fleet counts changed. Nothing was queued. Refresh the preview and review the launches.'); } const expected = new Set(servers.map(server => server.name)); for (const [name, value] of Object.entries(keys)) { diff --git a/src/lib/server/fleet/server-keys.ts b/src/lib/server/fleet/server-keys.ts index 76adb8c..a50e037 100644 --- a/src/lib/server/fleet/server-keys.ts +++ b/src/lib/server/fleet/server-keys.ts @@ -1,35 +1,122 @@ +import { randomUUID } from 'node:crypto'; import { db } from '../db/index.js'; -import { zombieMapBySlug } from './map-aliases.js'; -export function validServerId(value: unknown): value is string { - if (typeof value !== 'string') return false; - const match = /^(t5|t6)_zombie_([a-z0-9]+)_([1-9]|1[0-9]|20)$/.exec(value); - return !!match && !!zombieMapBySlug(match[1], match[2]); +type KeyGame = 't5' | 't6'; +export interface ServerKey { + id: string; + game: KeyGame; + label: string; + updated_at: string; + assigned_servers: string[]; } -export function listServerKeys() { - return db().prepare(`SELECT k.server_id, k.updated_at, - EXISTS(SELECT 1 FROM servers s WHERE s.name = k.server_id) AS deployed - FROM server_keys k ORDER BY k.server_id`).all() as Array<{ - server_id: string; updated_at: string; deployed: number; - }>; +function validGame(game: unknown): asserts game is KeyGame { + if (game !== 't5' && game !== 't6') throw new Error('Choose T5 or T6 for the server key'); } -export function getServerKey(serverId: string): string | undefined { - return (db().prepare('SELECT pluto_key FROM server_keys WHERE server_id = ?') - .get(serverId) as { pluto_key: string } | undefined)?.pluto_key; -} - -export function saveServerKey(serverId: unknown, key: unknown): void { - if (!validServerId(serverId)) throw new Error('Choose a valid T5 or T6 zombie server ID with a lobby number from 1 to 20'); +function normalizedKey(key: unknown): string { if (typeof key !== 'string' || !key.trim() || key.trim().length > 512 || /\s/.test(key.trim())) { throw new Error('Enter a server key without whitespace (maximum 512 characters)'); } - db().prepare(`INSERT INTO server_keys (server_id, pluto_key) VALUES (?, ?) - ON CONFLICT(server_id) DO UPDATE SET pluto_key = excluded.pluto_key, updated_at = datetime('now')`) - .run(serverId, key.trim()); + return key.trim(); } -export function removeServerKey(serverId: string): boolean { - return db().prepare('DELETE FROM server_keys WHERE server_id = ?').run(serverId).changes > 0; +function normalizedLabel(label: unknown): string { + if (typeof label !== 'string' || label.trim().length > 160 || /[\r\n]/.test(label)) { + throw new Error('Enter a key label with at most 160 characters'); + } + return label.trim(); +} + +export function listServerKeys(): ServerKey[] { + const rows = db().prepare('SELECT id,game,label,updated_at FROM server_keys ORDER BY game,label,id').all() as Omit[]; + const assignments = db().prepare(`SELECT DISTINCT k.id AS key_id,l.server_id FROM server_keys k + JOIN server_keys occupied ON occupied.pluto_key=k.pluto_key JOIN server_key_leases l ON l.key_id=occupied.id + ORDER BY l.server_id`).all() as { key_id: string; server_id: string }[]; + const byKey = new Map(); + for (const lease of assignments) { + const servers = byKey.get(lease.key_id) ?? []; + servers.push(lease.server_id); + byKey.set(lease.key_id, servers); + } + return rows.map(row => ({ ...row, assigned_servers: byKey.get(row.id) ?? [] })); +} + +export function getServerKey(serverId: string): string | undefined { + return (db().prepare(`SELECT k.pluto_key FROM server_key_leases l + JOIN server_keys k ON k.id=l.key_id WHERE l.server_id=? AND l.is_current=1`).get(serverId) as { pluto_key: string } | undefined)?.pluto_key; +} + +/** Saving a credential twice cannot increase available capacity. */ +export function saveServerKey(game: unknown, key: unknown, label?: unknown): string { + validGame(game); + const credential = normalizedKey(key); + const keyLabel = label === undefined ? undefined : normalizedLabel(label); + return db().transaction(() => { + const existing = db().prepare('SELECT id FROM server_keys WHERE game=? AND pluto_key=?').get(game, credential) as { id: string } | undefined; + if (existing) { + if (keyLabel !== undefined) db().prepare("UPDATE server_keys SET label=?,updated_at=datetime('now') WHERE id=?").run(keyLabel, existing.id); + return existing.id; + } + const id = randomUUID(); + db().prepare('INSERT INTO server_keys(id,game,label,pluto_key) VALUES(?,?,?,?)').run(id, game, keyLabel ?? '', credential); + return id; + }).immediate(); +} + +// Compare the credential as well as its row: a mistakenly imported copy in +// another game's pool must not permit simultaneous use of the same key. +const availableKey = `NOT EXISTS (SELECT 1 FROM server_key_leases l + JOIN server_keys occupied ON occupied.id=l.key_id + WHERE occupied.pluto_key=k.pluto_key AND (l.server_id!=@server OR l.is_current=0))`; + +export function serverKeyPool(game: KeyGame, serverId?: string): { total: number; available: number } { + validGame(game); + return db().prepare(`SELECT COUNT(*) AS total, + COALESCE(SUM(CASE WHEN ${availableKey} THEN 1 ELSE 0 END),0) AS available + FROM server_keys k WHERE k.game=@game`).get({ game, server: serverId ?? '' }) as { total: number; available: number }; +} + +export function availableServerKeyCount(game: KeyGame): number { + return serverKeyPool(game).available; +} + +/** Called within launch admission; the write lock prevents competing launches sharing a key. */ +export function reserveServerKey(serverId: string, game: KeyGame, suppliedKey?: unknown): string { + validGame(game); + if (!serverId) throw new Error('Choose a server for the key reservation'); + const credential = suppliedKey === undefined || suppliedKey === null || (typeof suppliedKey === 'string' && !suppliedKey.trim()) + ? undefined : normalizedKey(suppliedKey); + return db().transaction(() => { + const existing = db().prepare(`SELECT k.game,k.pluto_key FROM server_key_leases l + JOIN server_keys k ON k.id=l.key_id WHERE l.server_id=? AND l.is_current=1`).get(serverId) as { game: KeyGame; pluto_key: string } | undefined; + if (existing) { + if (existing.game !== game) throw new Error('This server already has a key reserved for another game'); + if (credential && credential !== existing.pluto_key) throw new Error('This server already has a reserved key; destroy it before changing its key'); + return existing.pluto_key; + } + const requestedId = credential ? saveServerKey(game, credential) : null; + const key = db().prepare(`SELECT k.id,k.pluto_key FROM server_keys k WHERE k.game=@game + AND (@requestedId IS NULL OR k.id=@requestedId) AND ${availableKey} + ORDER BY k.updated_at,k.id LIMIT 1`).get({ game, server: serverId, requestedId }) as { id: string; pluto_key: string } | undefined; + if (!key) throw new Error(credential ? 'This server key is already reserved by another server' + : `No available ${game.toUpperCase()} server keys. Add a key or destroy a server to release one.`); + db().prepare('INSERT INTO server_key_leases(server_id,key_id) VALUES(?,?)').run(serverId, key.id); + return key.pluto_key; + }).immediate(); +} + +/** Only after confirmed destruction or a creation cancelled before any host mutation. */ +export function releaseServerKey(serverId: string): boolean { + return db().prepare('DELETE FROM server_key_leases WHERE server_id=?').run(serverId).changes > 0; +} + +export function removeServerKey(id: string): boolean { + return db().transaction(() => { + if (db().prepare(`SELECT 1 FROM server_keys k JOIN server_keys occupied ON occupied.pluto_key=k.pluto_key + JOIN server_key_leases l ON l.key_id=occupied.id WHERE k.id=?`).get(id)) { + throw new Error('This key is reserved by a server. Destroy that server before removing its key.'); + } + return db().prepare('DELETE FROM server_keys WHERE id=?').run(id).changes > 0; + }).immediate(); } diff --git a/src/lib/server/provisioning/cancel-creation.ts b/src/lib/server/provisioning/cancel-creation.ts index 685b835..2fb1ccb 100644 --- a/src/lib/server/provisioning/cancel-creation.ts +++ b/src/lib/server/provisioning/cancel-creation.ts @@ -1,5 +1,6 @@ import { pauseScalingForOperation } from '../autoscaling/ownership.js'; import { db } from '../db/index.js'; +import { releaseServerKey } from '../fleet/server-keys.js'; import { nativeInstance } from '../backends/native.js'; import { queueHostOperation, releaseHostPorts } from '../hosts/index.js'; import { beginDeployment, cancelRemoval, deploymentStage, finishDeployment } from './history.js'; @@ -65,10 +66,12 @@ export function cancelCreation(id: string, triggeredBy = 'ui', pausePolicy = tru } export function finishCreationCancellation(cleanupId: string): void { - const cancellation = db().prepare(`SELECT c.creation_id,n.operation_id FROM native_creation_cancellations c + const cancellation = db().prepare(`SELECT c.creation_id,n.operation_id,d.server FROM native_creation_cancellations c + JOIN server_deployments d ON d.id=c.creation_id LEFT JOIN native_provisioning n ON n.deployment_id=c.creation_id WHERE c.cleanup_id=?`).get(cleanupId) as - { creation_id: string; operation_id: string | null } | undefined; + { creation_id: string; operation_id: string | null; server: string } | undefined; if (!cancellation) return; + releaseServerKey(cancellation.server); const now = new Date().toISOString(); db().prepare('UPDATE server_deployments SET server_deleted_at=? WHERE id IN (?,?)').run(now, cancellation.creation_id, cleanupId); deploymentStage(cancellation.creation_id, 'Creation cancelled; cleanup completed', { blocker: null, diff --git a/src/lib/server/provisioning/instances.ts b/src/lib/server/provisioning/instances.ts index e03c197..a6281fd 100644 --- a/src/lib/server/provisioning/instances.ts +++ b/src/lib/server/provisioning/instances.ts @@ -35,7 +35,7 @@ export interface CreateServerInput { game?: ZombieGame; // omitted by older clients: T5 gameMode: GameMode; // 'zombie' — only value supported today mapSlug: string; // 'cosmodrome' - plutoKey?: string; // saved for this server ID when supplied + plutoKey?: string; // added to the game pool and reserved for this launch when supplied expectedName?: string; // optional preview guard; never selects a lobby previewId?: string; createdBy?: string; // 'ui' by default @@ -44,6 +44,8 @@ export interface CreateServerInput { export function nextLobbyNumber(baseName: string, additionalNames: string[] = []): number | null { const rows = db() .prepare(`SELECT name FROM servers WHERE name LIKE ? UNION + SELECT server AS name FROM native_instances WHERE retired_at IS NULL UNION + SELECT server_id AS name FROM server_key_leases UNION SELECT j.server AS name FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id WHERE d.server_deleted_at IS NULL AND (d.state IN ('queued', 'running', 'awaiting-health') OR (d.state = 'cancelled' AND d.submitted_at IS NOT NULL) OR (d.action = 'rebuild' AND (d.state = 'waiting' OR (d.state IN ('failed','interrupted') AND d.stage = 'Removing server files and registry'))))`) .all(`${baseName}_%`) as Array<{ name: string }>; diff --git a/src/lib/server/provisioning/native.ts b/src/lib/server/provisioning/native.ts index 672c5ca..c44c2cc 100644 --- a/src/lib/server/provisioning/native.ts +++ b/src/lib/server/provisioning/native.ts @@ -17,7 +17,7 @@ import { getHost, listHosts, getHostOperationStatus, queueHostOperation, queueHo import { newestReadyLibraryId, assertReadyLibrary } from '../game-files/index.js'; import { blueprintRecipe, captureRecipePlan, recipeForDeployment, recipeStorage, summarizeRecipe, type LaunchRecipePlan } from '../fleet/launch-recipes.js'; import { prepareNativeLaunch, evaluateNativePlacement, type PreparedNativeLaunch } from './launch-preparation.js'; -import { getServerKey, saveServerKey } from '../fleet/server-keys.js'; +import { getServerKey, reserveServerKey, releaseServerKey, availableServerKeyCount } from '../fleet/server-keys.js'; import { activeBetaRemovalBlocker, getServer } from '../fleet/servers.js'; import { nextLobbyNumber, type CreateServerInput, type CreateServerResult } from './instances.js'; import { beginDeployment, deploymentStage, finishDeployment, REMOVING_SERVER, cancelRemoval } from './history.js'; @@ -118,7 +118,7 @@ export function previewNativeCreation(input: Record): LaunchPre { label: 'Host port', state: port ? 'pass' : 'blocked', detail: port ? selection.automatic ? 'UDP/TCP capacity available; port assigned at placement.' : `${selection.bindAddress}:${port} / UDP and TCP` : 'No available UDP/TCP port on this host.' }, { label: 'Configuration', state: 'pass', detail: recipe.configSource } ]; - return { id, expiresAt, name, port, hasSavedKey: !!name && !!getServerKey(name), recipe: summarizeRecipe(recipe), blueprintName, checks, + return { id, expiresAt, name, port, hasSavedKey: !!name && availableServerKeyCount(recipe.game) > 0, recipe: summarizeRecipe(recipe), blueprintName, checks, ready: checks.every(check => check.state !== 'blocked') }; } @@ -163,9 +163,7 @@ export function queueNativeCreationIntent(input: { server: string; game: 't5' | const reserved = db().prepare(`SELECT 1 FROM server_creation_jobs j JOIN server_deployments d ON d.id=j.deployment_id WHERE j.server=? AND d.server_deleted_at IS NULL AND (d.state IN ('queued','waiting','running','awaiting-health','interrupted') OR (d.state='cancelled' AND d.submitted_at IS NOT NULL))`).get(input.server); if ((!input.autoscalingDecisionId && input.server !== expected) || nativeInstance(input.server) || getServer(input.server) || reserved) throw new Error('Available lobby changed. Review a fresh preview.'); - const key = input.plutoKey?.trim() || getServerKey(input.server); - if (!key) throw new Error(`Save a Plutonium server key for ${input.server}.`); - saveServerKey(input.server, key); + reserveServerKey(input.server, input.game, input.plutoKey); const recipe: NativeRecipe = { ...(input.recipe ?? captureRecipePlan(input.game, input.mapSlug)), native: input.selection, image: input.selection.image! }; const id = beginDeployment(input.server, 'create', input.createdBy ?? 'ui'); @@ -292,6 +290,7 @@ export function reconcileNativeProvisioningReceipt(id: string): void { db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(removal.updatedAt, original.id); releaseHostPorts(original.id); db().prepare('DELETE FROM servers WHERE name=?').run(job.server); + releaseServerKey(job.server); db().prepare('UPDATE ec_host_migrations SET removed_at=? WHERE deployment_id=?').run(removal.updatedAt, id); recordProgress(id, { event: { kind: 'cleanup', message: 'Cancelled migration: source removal completed; replacement creation was not started.' } }); } @@ -314,6 +313,7 @@ export function reconcileNativeProvisioningReceipt(id: string): void { db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(operation.updatedAt, instance.id); releaseHostPorts(instance.id); db().prepare('DELETE FROM servers WHERE name=?').run(job.server); + releaseServerKey(job.server); finishCreationCancellation(job.id); return; } @@ -539,6 +539,7 @@ async function processJob(job: Job, eligibility: (job: Job) => string | null) { db().prepare('UPDATE native_instances SET retired_at=? WHERE id=?').run(new Date().toISOString(), instance!.id); releaseHostPorts(instance!.id); db().prepare('DELETE FROM servers WHERE name=?').run(job.server); + releaseServerKey(job.server); recordProgress(job.id, { checkpoint: { key: 'cleanup', label: 'Instance retired and port reservations released' } }); complete(job); finishCreationCancellation(job.id); diff --git a/src/lib/ui/components/DeployCollectionDialog.svelte b/src/lib/ui/components/DeployCollectionDialog.svelte index fa082d4..f708926 100644 --- a/src/lib/ui/components/DeployCollectionDialog.svelte +++ b/src/lib/ui/components/DeployCollectionDialog.svelte @@ -65,9 +65,17 @@ if (current === session) { const next = body as CollectionPreview; const names = new Set(next.servers.map(server => server.name)); - keys = Object.fromEntries(next.servers.filter(server => server.name && !server.hasSavedKey).map(server => [server.name!, keys[server.name!] ?? ''])); - if (previous && (previous.servers.length !== next.servers.length || previous.servers.some(server => !names.has(server.name)))) keyNotice = 'Server assignments changed. Keys were retained only for the exact IDs still shown below. Review the new assignments.'; - else if (previous) keyNotice = 'Preview refreshed. Entered keys are retained for these same server IDs.'; + const enteredByGame = new Map(); + for (const server of previous?.servers ?? []) { + const value = server.name ? keys[server.name]?.trim() : ''; + if (!value) continue; + const game = previous!.entries[server.entryIndex].recipe.game; + enteredByGame.set(game, [...(enteredByGame.get(game) ?? []), value]); + } + keys = Object.fromEntries(next.servers.filter(server => server.name && !server.hasSavedKey) + .map(server => [server.name!, enteredByGame.get(next.entries[server.entryIndex].recipe.game)?.shift() ?? ''])); + if (previous && (previous.servers.length !== next.servers.length || previous.servers.some(server => !names.has(server.name)))) keyNotice = 'Server assignments changed. Entered keys were retained for launches in the same game that still need a key.'; + else if (previous) keyNotice = 'Preview refreshed. Entered keys are retained for games that still need them.'; preview = next; now = Date.now(); } } catch (err) { if (current === session) error = (err as Error).message; } @@ -108,7 +116,7 @@ {#if !preview}

{busy ? 'Preparing the collection…' : 'Review the fleet changes, capacity estimate, and server keys. Queued launches wait automatically when hosts need more capacity or preparation.'}

{#if !busy}{/if} {:else} {#if keyNotice}

{keyNotice}

{/if} - {#if expired}

This preview expired. Refresh before launch; keys remain entered for unchanged IDs.

{/if} + {#if expired}

This preview expired. Refresh before launch; entered keys stay with their game.

{/if}
{preview.mode === 'exact' ? `${preview.retainedServers.length} keep · ${preview.servers.length} create · ${preview.removals.length} remove` : preview.servers.length ? `${preview.servers.length} missing server${preview.servers.length === 1 ? '' : 's'} to create` : 'Collection already satisfied'} {#each preview.entries as entry}{/each}
MapTargetExisting / queuedTo create
{entry.recipe.mapName} {entry.recipe.game.toUpperCase()}{entry.count}{entry.existing}{entry.missing}

Targets count inventory, including stopped servers; they do not mean every server is ready to play. {preview.mode === 'exact' ? 'Existing deployed lobbies are kept before queued creations, then the lowest lobby numbers. Retained servers are not changed or restarted.' : 'Only missing servers are created. Extra servers remain and existing servers are not updated.'}

@@ -133,7 +141,8 @@ {#each preview.blockers as blocker}{/each} {#each preview.warnings ?? [] as warning}

{warning}

{/each} {#if preview.servers.length || preview.removals.length} - {#if preview.servers.length}

Saved keys are reused automatically. Add keys only where required below. Update saved keys in Keys (opens in a new tab). Apply saves the reviewed jobs and opens their progress page. Each launch waits for a startup slot and host capacity.

{/if} + {#if preview.servers.length}

Available keys are assigned from each game’s pool. Add an unused key for each remaining launch below. Manage pools in Keys (opens in a new tab). Apply saves the reviewed jobs and opens their progress page. Each launch waits for a startup slot and host capacity.

{/if} + {#if preview.removals.length && preview.servers.some(server => !server.hasSavedKey)}

Keys held by servers awaiting removal are still in use. To reuse those keys, remove the servers first and refresh this preview after cleanup finishes.

{/if}
{ event.preventDefault(); void launch(); }}> {#if preview.removals.length}
@@ -150,14 +159,14 @@
{#if index === 0 || preview.servers[index - 1].entryIndex !== server.entryIndex}{preview.entries[server.entryIndex].recipe.game.toUpperCase()} · {preview.entries[server.entryIndex].recipe.mapName}{/if}{#if server.name}{:else}{preview.entries[server.entryIndex].recipe.mapName}{/if}{server.name ?? 'No available lobby'} · {server.native ? 'Host and port assigned when provisioning starts' : server.port === null ? 'No available port' : `Port ${server.port}`}
{#if server.name} - {#if server.hasSavedKey}

Saved key ready

- {:else}{/if} + {#if server.hasSavedKey}

Game key available

+ {:else}{/if} {/if} {/each} {/if} -
{#if preview.servers.length}

{keysReady} / {preview.servers.length} keys available

{/if}

Key validity is checked at startup. Preview expires at {$timeFormat.minuteTime(preview.expiresAt)}.

{#if keysReady < preview.servers.length}{/if}
+
{#if preview.servers.length}

{keysReady} / {preview.servers.length} keys available

{/if}

Pool availability is checked again when you apply. Preview expires at {$timeFormat.minuteTime(preview.expiresAt)}.

{#if keysReady < preview.servers.length}{/if}
{:else}

{preview.mode === 'exact' ? 'Your fleet already matches this collection.' : 'Your fleet already meets or exceeds every target in this collection.'} No servers will be created or removed.

{/if}
Setup and launch requirements

Each new server is automatically placed on an eligible host with the latest game files and prepared runtime. CPU, memory, host readiness, synced files, runtime availability, and local port reservations are checked again before deployment. Existing servers keep their current files.

{#each preview.entries as entry}
{/each}
diff --git a/src/lib/ui/components/DeployServerDialog.svelte b/src/lib/ui/components/DeployServerDialog.svelte index d0a84ba..fd3bdf5 100644 --- a/src/lib/ui/components/DeployServerDialog.svelte +++ b/src/lib/ui/components/DeployServerDialog.svelte @@ -97,7 +97,7 @@ const body = await res.json(); if (!res.ok) throw new Error(body.error ?? 'Could not prepare launch preview.'); if (current === session) { - if (previousName && previousName !== body.name) { plutoKey = ''; keyNotice = 'The available server ID changed. Enter the key for the new ID below; the previous key was not transferred.'; } + if (previousName && previousName !== body.name) keyNotice = 'The available lobby changed. Any entered key can still be used for this game.'; if (body.hasSavedKey) plutoKey = ''; preview = body; now = Date.now(); await tick(); heading?.focus(); @@ -148,18 +148,18 @@ {#if error}{/if} {:else if preview} {#if keyNotice}

{keyNotice}

{/if} - {#if expired}

This preview expired. Refresh it to recheck capacity. Keys stay entered when the server ID is unchanged.

{/if} + {#if expired}

This preview expired. Refresh it to recheck capacity. Entered keys stay with this game when you refresh.

{/if}
{#if preview.blueprintName}

From collection: {preview.blueprintName}

{/if}

Before you launch

    {#each preview.checks as check}
  • {check.state === 'pass' ? '✓' : check.state === 'blocked' ? '!' : 'i'}
    {check.label}

    {check.detail}

    • {/each}
{ event.preventDefault(); void launch(); }}> -
Your new server{preview.name ?? 'Waiting for an available lobby'}

Fresh identity, with its own server key.

- {#if preview.hasSavedKey}

Saved key ready

- {:else}{/if} -

{preview.hasSavedKey ? 'The saved key for this server ID will be reused automatically. Electric Cherry does not check whether Plutonium accepts it. Update it in Keys.' : 'Add the key here. It is saved for this exact server ID when you launch, including if provisioning fails.'} Manage saved keys in a new tab; your launch stays open here.

+
Your new server{preview.name ?? 'Waiting for an available lobby'}

An available key from this game’s pool is assigned when you launch.

+ {#if preview.hasSavedKey}

Game key available

+ {:else}{/if} +

{preview.hasSavedKey ? 'An unused key from this game’s pool will be assigned automatically. Availability is checked again at launch.' : 'Add an unused key for this game. Launch saves it in the game pool and assigns it to this server until removal is confirmed.'} Manage saved keys in a new tab; your launch stays open here.

- {#if !preview.hasSavedKey && !plutoKey.trim()}

Enter this server’s key to enable launch.

{/if} + {#if !preview.hasSavedKey && !plutoKey.trim()}

Enter an unused key for this game to enable launch.

{/if}

Preview valid until {$timeFormat.minuteTime(preview.expiresAt)}. Capacity is checked again at launch. The host and port are assigned automatically.

Make this a reusable collection

Save this game and map with a target of one lobby. Future launches use the latest promoted setup and create only missing lobbies.

diff --git a/src/routes/api/server-keys/+server.ts b/src/routes/api/server-keys/+server.ts index 8a50c29..354585a 100644 --- a/src/routes/api/server-keys/+server.ts +++ b/src/routes/api/server-keys/+server.ts @@ -1,5 +1,5 @@ import { json } from '@sveltejs/kit'; -import { listServerKeys, saveServerKey, removeServerKey, validServerId } from '$lib/server/fleet/server-keys.js'; +import { listServerKeys, saveServerKey, removeServerKey } from '$lib/server/fleet/server-keys.js'; export function GET() { return json({ keys: listServerKeys() }, { headers: { 'Cache-Control': 'no-store' } }); @@ -8,8 +8,8 @@ export function GET() { export async function PUT({ request }: { request: Request }) { const body = await request.json().catch(() => null); try { - saveServerKey(body?.serverId, body?.plutoKey); - return json({ ok: true }); + const id = saveServerKey(body?.game, body?.plutoKey, body?.label); + return json({ ok: true, id }); } catch (err) { return json({ ok: false, error: (err as Error).message }, { status: 400 }); } @@ -17,6 +17,10 @@ export async function PUT({ request }: { request: Request }) { export async function DELETE({ request }: { request: Request }) { const body = await request.json().catch(() => null); - if (!validServerId(body?.serverId)) return json({ error: 'Invalid server ID' }, { status: 400 }); - return json({ ok: removeServerKey(body.serverId) }); + if (typeof body?.id !== 'string' || !body.id.trim() || body.id.length > 128) return json({ error: 'Invalid key ID' }, { status: 400 }); + try { + return json({ ok: removeServerKey(body.id) }); + } catch (err) { + return json({ ok: false, error: (err as Error).message }, { status: 400 }); + } } diff --git a/src/routes/autoscaling/+page.svelte b/src/routes/autoscaling/+page.svelte index bec435e..68d1143 100644 --- a/src/routes/autoscaling/+page.svelte +++ b/src/routes/autoscaling/+page.svelte @@ -94,15 +94,15 @@
- {#if settings.mapSlug === '*'}

Targets apply separately to every map, including all variants. Each map needs its own keys. Actions share one cooldown and run one at a time across the game.

{/if} -

Saved lobby keys must cover the maximum for each selected map before you can save a new policy, including Dry run.

+ {#if settings.mapSlug === '*'}

Targets apply separately to every map, including all variants. All maps share this game’s key pool. Actions share one cooldown and run one at a time across the game.

{/if} +

Each new lobby needs an available key from the game pool. Creation waits when the pool is exhausted; keys return after confirmed removal.

{#if preview}

Preview · {modeLabel(preview.policy.mode)}

-

Target: {preview.decision.desired} running lobbies{preview.policy.mapSlug === '*' ? ' across all maps' : ''}. {preview.snapshot.keyedNames.length} saved lobby keys. {preview.snapshot.members.length} existing or pending lobbies.

+

Target: {preview.decision.desired} running lobbies{preview.policy.mapSlug === '*' ? ' across all maps' : ''}. {preview.snapshot.keyPool.available} of {preview.snapshot.keyPool.total} game keys available. {preview.snapshot.members.length} existing or pending lobbies.

{preview.decision.action === 'wait' ? 'Wait' : `${preview.decision.action === 'create' ? 'Create' : 'Remove'} ${preview.decision.server}`}: {preview.decision.reason}

{#if preview.policy.mode === 'create-only' || preview.policy.mode === 'create-destroy'}

Saving gives this policy exclusive control of the selected maps' lobby counts. Pause it before launching a competing collection or manual lifecycle action.

{/if} {#if preview.snapshot.members.length}
    {#each preview.snapshot.members as member}
  • {member.name} · {member.state}{member.blocker ? ` · ${member.blocker}` : ''}
    • {/each}
{/if} @@ -118,7 +118,7 @@

{view.policy.mapSlug === '*' ? 'Every map' : zombieMapBySlug(view.policy.game, view.policy.mapSlug)?.friendly ?? view.policy.mapSlug} · {view.policy.game.toUpperCase()}

{modeLabel(view.policy.mode)}{#if canOperate}{/if}

{view.decision.occupied} occupied · {view.decision.empty} empty and ready · {view.decision.pending} pending · {view.decision.unknown} unknown or stopped

{#if view.unsettled.length}{/if} -

{view.policy.mapSlug === '*' ? 'Per map: ' : ''}Minimum {view.policy.minimum} · Maximum {view.policy.maximum} · Spare target {view.policy.spares} · {view.snapshot.keyedNames.length} saved keys

+

{view.policy.mapSlug === '*' ? 'Per map: ' : ''}Minimum {view.policy.minimum} · Maximum {view.policy.maximum} · Spare target {view.policy.spares} · {view.snapshot.keyPool.available} / {view.snapshot.keyPool.total} game keys available

{view.policy.mode === 'dry-run' ? 'Hypothetical decision: ' : 'Current decision: '}{view.decision.action}{view.decision.server ? ` ${view.decision.server}` : ''}. {view.decision.reason}

{#if view.policy.mode === 'create-only' && view.decision.action === 'destroy'}

Removal is advisory in Create only mode.

{/if}

Last evaluated: {view.lastEvaluatedAt ? $timeFormat.dateTime(new Date(view.lastEvaluatedAt).toISOString()) : 'Not evaluated yet'}{view.policy.nextActionAt > Date.now() ? ` · Next action after ${$timeFormat.dateTime(new Date(view.policy.nextActionAt).toISOString())}` : ''}

diff --git a/src/routes/provisioning/[id]/+page.svelte b/src/routes/provisioning/[id]/+page.svelte index 93f8298..f4b2859 100644 --- a/src/routes/provisioning/[id]/+page.svelte +++ b/src/routes/provisioning/[id]/+page.svelte @@ -113,7 +113,7 @@ {#if detail.canResumeRebuild}{/if} {#if detail.canResumeCleanup}{/if} {#if detail.canCancel}

Stops remaining work and health checks. Changes already made are not undone.

{/if} - {#if detail.canCancelCreation}

Stops this launch and deletes its container and instance files, even if player status is unknown or players have joined. Shared game files and the saved server key are kept.

{/if} + {#if detail.canCancelCreation}

Stops this launch and deletes its container and instance files, even if player status is unknown or players have joined. Shared game files are kept. The key returns to its game pool after removal is confirmed.

{/if}
{/if} {#if detail.cancellation}
{detail.cancellation.state === 'done' ? 'Cancellation cleanup completed' : 'Cancellation cleanup'}

{detail.cancellation.error ?? detail.cancellation.stage}

View cleanup progress →
{/if} diff --git a/src/routes/server-keys/+page.svelte b/src/routes/server-keys/+page.svelte index fdd5b36..22dbe9b 100644 --- a/src/routes/server-keys/+page.svelte +++ b/src/routes/server-keys/+page.svelte @@ -3,7 +3,7 @@ import { onMount } from 'svelte'; import { page } from '$app/stores'; import { goto } from '$app/navigation'; - import { zombieMaps, zombieMapBySlug, type ZombieGame } from '$lib/map-aliases.js'; + import type { ZombieGame } from '$lib/map-aliases.js'; import { confirmAction } from '$ui/confirm.js'; import { serverTitle } from '$ui/server-labels.js'; import ServerIdentity from '$ui/components/ServerIdentity.svelte'; @@ -15,31 +15,34 @@ import Field from '$lib/ui/components/primitives/Field.svelte'; import Input from '$lib/ui/components/primitives/Input.svelte'; - interface SavedKey { server_id: string; deployed: number; updated_at: string } + interface SavedKey { id: string; game: ZombieGame; label: string; assigned_servers: string[]; updated_at: string } + const games: ZombieGame[] = ['t5', 't6']; let open = $state(false); let search = $state(''); + let gameFilter = $state(''); let limit = $state(25); let keys = $state([]); - let serverId = $state(''); - let editing = $state(false); - let rawMode = $state(false); let game = $state('t5'); - let mapSlug = $state('theater'); - let lobby = $state(1); - const chosenId = $derived(editing || rawMode ? serverId.trim() : `${game}_zombie_${mapSlug}_${lobby}`); - const validId = $derived.by(() => { - const match = /^(t5|t6)_zombie_([a-z0-9]+)_([1-9]|1[0-9]|20)$/.exec(chosenId); - return !!match && !!zombieMapBySlug(match[1], match[2]); - }); - $effect(() => { search = $page.url.searchParams.get('server') ?? ''; limit = Math.max(25, Number($page.url.searchParams.get('limit')) || 25); }); - function saveSearch() { const url = new URL($page.url); if (search) url.searchParams.set('server', search); else url.searchParams.delete('server'); if (limit > 25) url.searchParams.set('limit', String(limit)); else url.searchParams.delete('limit'); void goto(url, { replaceState: true, keepFocus: true, noScroll: true }); } + let label = $state(''); let plutoKey = $state(''); let busy = $state(false); let loading = $state(true); let error = $state(''); let notice = $state(''); - const filtered = $derived(keys.filter(key => (key.server_id + ' ' + serverTitle(key.server_id)).toLowerCase().includes(search.toLowerCase()))); - + $effect(() => { + search = $page.url.searchParams.get('search') ?? $page.url.searchParams.get('server') ?? ''; + gameFilter = $page.url.searchParams.get('game') ?? ''; + limit = Math.max(25, Number($page.url.searchParams.get('limit')) || 25); + }); + const filtered = $derived(keys.filter(key => (!gameFilter || key.game === gameFilter) + && [key.game, key.label, ...key.assigned_servers.flatMap(name => [name, serverTitle(name)])].join(' ').toLowerCase().includes(search.toLowerCase()))); + function saveSearch() { + const url = new URL($page.url); url.searchParams.delete('server'); + if (search) url.searchParams.set('search', search); else url.searchParams.delete('search'); + if (gameFilter) url.searchParams.set('game', gameFilter); else url.searchParams.delete('game'); + if (limit > 25) url.searchParams.set('limit', String(limit)); else url.searchParams.delete('limit'); + void goto(url, { replaceState: true, keepFocus: true, noScroll: true }); + } async function load() { const res = await fetch('/api/server-keys'); if (!res.ok) throw new Error('Could not load server keys'); @@ -47,39 +50,36 @@ } async function reload() { try { await load(); error = ''; } catch (e) { error = (e as Error).message; } finally { loading = false; } } onMount(() => { void reload(); }); - function reset() { serverId = ''; plutoKey = ''; editing = false; rawMode = false; } function addKey() { - reset(); error = ''; - const match = /^(t5|t6)_zombie_([a-z0-9]+)_([1-9]|1[0-9]|20)$/.exec($page.url.searchParams.get('server') ?? ''); - if (match && zombieMapBySlug(match[1], match[2])) { game = match[1] as ZombieGame; mapSlug = match[2]; lobby = Number(match[3]); } + plutoKey = ''; label = ''; error = ''; + if (gameFilter === 't5' || gameFilter === 't6') game = gameFilter; open = true; } - function savedAt(value: string) { return $timeFormat.dateTime(value); } async function save(e: SubmitEvent) { e.preventDefault(); - if (!validId || !plutoKey.trim() || busy) return; + if (!plutoKey.trim() || busy) return; busy = true; error = ''; notice = ''; try { const res = await fetch('/api/server-keys', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ serverId: chosenId, plutoKey }) + body: JSON.stringify({ game, label, plutoKey }) }); const body = await res.json(); if (!res.ok) throw new Error(body.error ?? 'Could not save key'); - reset(); open = false; notice = 'Server key saved.'; await load(); + plutoKey = ''; label = ''; open = false; notice = 'Key saved to the game pool.'; await load(); } catch (e) { error = (e as Error).message; } finally { busy = false; } } - async function remove(id: string) { - if (!await confirmAction({title:'Remove saved key?',message:`Remove the key for ${serverTitle(id)} (${id})? Running servers are unaffected.`,confirmLabel:'Remove key',danger:true})) return; + async function remove(key: SavedKey) { + if (key.assigned_servers.length) return; + if (!await confirmAction({ title: 'Remove saved key?', message: `Remove ${key.label || 'this key'} from the ${key.game.toUpperCase()} pool?`, confirmLabel: 'Remove key', danger: true })) return; busy = true; error = ''; notice = ''; try { const res = await fetch('/api/server-keys', { - method: 'DELETE', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ serverId: id }) + method: 'DELETE', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ id: key.id }) }); - if (!res.ok) throw new Error('Could not remove key'); - if (serverId === id) reset(); + const body = await res.json(); + if (!res.ok) throw new Error(body.error ?? 'Could not remove key'); notice = 'Saved key removed.'; await load(); } catch (e) { error = (e as Error).message; } finally { busy = false; } @@ -88,72 +88,70 @@ Keys · Electric Cherry
- + {#snippet actions()}{/snippet} -

Saved keys are reused when a server is created or rebuilt. Electric Cherry does not check whether Plutonium accepts them. Updating or removing a saved key does not change an existing server.

-
{ search = event.currentTarget.value; limit = 25; saveSearch(); }} placeholder="Search servers…" />
- +

A key stays assigned while its server exists, including when stopped, and becomes available after removal is confirmed. Rebuilds keep the same key. Use the optional label to record its Plutonium hostname; saving a label does not change that hostname.

+{#if !loading}
{#each games as poolGame} + {@const pool = keys.filter(key => key.game === poolGame)} +
{poolGame === 't5' ? 'Black Ops I' : 'Black Ops II'}{pool.filter(key => !key.assigned_servers.length).length} available · {pool.filter(key => key.assigned_servers.length).length} assigned · {pool.length} total
+{/each}
{/if} +
+ { search = event.currentTarget.value; limit = 25; saveSearch(); }} placeholder="Search labels or assigned servers…" /> + +
{#if error && !open}{/if} {#if notice}

{notice}

{/if}

Saved keys

{#if loading}

Loading keys…

-{:else if !filtered.length} +{:else if !filtered.length} {:else}
- - {#each filtered.slice(0,limit) as key (key.server_id)} + + {#each filtered.slice(0, limit) as key (key.id)} - - - - + + + + {/each}
Server IDServerLast savedActions
Game / labelAvailabilityLast savedActions
{#if key.deployed}{:else}{/if}{key.server_id}{key.deployed ? 'Server exists' : 'No server deployed'}{savedAt(key.updated_at)}
- - -
{key.game.toUpperCase()}{key.label || 'Unlabeled key'}{#if key.assigned_servers.length}Assigned{#each key.assigned_servers as name}
{name}
{/each}{:else}Available{/if}		{$timeFormat.dateTime(key.updated_at)}{#if key.assigned_servers.length}In use{/if}
{/if} -
- + {#if error}{/if}
- {#if !editing}{/if} - {#if editing || rawMode} - - {:else}

Server ID: {chosenId}

{/if} - {#if !editing && keys.some(key => key.server_id === chosenId)}

A key is already saved for this ID. Saving replaces it for the next deployment; the running server keeps its current key.

{/if} - - - -
- - -
-
- {#if !validId || !plutoKey.trim()}

Choose a valid server ID and enter its Plutonium key to save.

{/if} + + + + + + + +
+ diff --git a/tests/unit/autoscaling-delete.test.ts b/tests/unit/autoscaling-delete.test.ts index c5bc916..8ce0f4a 100644 --- a/tests/unit/autoscaling-delete.test.ts +++ b/tests/unit/autoscaling-delete.test.ts @@ -10,7 +10,7 @@ async function setup(mode = 'dry-run') { if (mode.startsWith('create')) await prepareNativeHost(); const { db } = await import('../../src/lib/server/db/index.js'); const scaling = await import('../../src/lib/server/autoscaling/index.js'); - db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run('t6_zombie_town_1', 'test-key'); + db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('key-1','t6','test-key')").run(); const input = { ...settings, mode }; const saved = scaling.saveScaling(input, undefined, undefined, scaling.previewScaling(input).token); return { db, scaling, saved, input }; @@ -20,7 +20,7 @@ it('deletes policy evidence atomically while preserving lobbies, keys, and unrel const { db, scaling, saved } = await setup(); db().prepare("INSERT INTO servers(name,game,map_slug,map) VALUES('t6_zombie_town_1','t6','town','zm_transit')").run(); await scaling.runAutoscaling(); - db().prepare("INSERT INTO server_keys(server_id,pluto_key) VALUES('t5_zombie_theater_1','other-key')").run(); + db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('other-key','t5','other-key')").run(); const otherSettings = { ...settings, game: 't5', mapSlug: 'theater', mode: 'off' }; const other = scaling.saveScaling(otherSettings, undefined, undefined, scaling.previewScaling(otherSettings).token); for (const table of ['autoscaling_decisions', 'autoscaling_members', 'autoscaling_samples']) expect(db().prepare(`SELECT COUNT(*) AS n FROM ${table} WHERE policy_id=?`).get(saved.id)).toEqual({ n: 1 }); diff --git a/tests/unit/autoscaling-dry-run.test.ts b/tests/unit/autoscaling-dry-run.test.ts index 8d86c78..65476e9 100644 --- a/tests/unit/autoscaling-dry-run.test.ts +++ b/tests/unit/autoscaling-dry-run.test.ts @@ -6,7 +6,7 @@ let fixture: Fixture; beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); const { db } = await import('../../src/lib/server/db/index.js'); - for (const n of [3, 8, 9, 10]) db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_town_${n}`, `secret-${n}`); + for (const n of [3, 8, 9, 10]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `secret-${n}`); }); afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); }); const settings = { game: 't6', mapSlug: 'town', maximum: 4, mode: 'dry-run' }; @@ -22,7 +22,7 @@ it('persists deduplicated dry-run decisions across restart without provisioning const view = scaling.scalingReport().policies[0]; expect(view.policy.id).toBe(saved.id); expect(view.history).toHaveLength(1); - expect(view.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_3' }); + expect(view.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_1' }); expect(JSON.stringify(view)).not.toContain('secret'); for (const table of ['server_deployments', 'native_creation_intents', 'native_instances']) expect(db().prepare(`SELECT COUNT(*) AS n FROM ${table}`).get()).toEqual({ n: 0 }); }); @@ -35,8 +35,10 @@ it('rejects duplicate policies, stale edits and changed previews', async () => { expect(() => scaling.previewScaling(settings, saved.id, 0)).toThrow('changed'); expect(() => scaling.saveScaling({ ...settings, maximum: 5 }, saved.id, saved.revision, preview.token)).toThrow('preview changed'); const { db } = await import('../../src/lib/server/db/index.js'); - db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_10'").run(); - expect(() => scaling.saveScaling({ ...settings, mode: 'create-destroy' }, saved.id, saved.revision, scaling.previewScaling({ ...settings, mode: 'create-destroy' }, saved.id, saved.revision).token)).toThrow('Save keys'); + const beforeKeyLoss = scaling.previewScaling(settings, saved.id, saved.revision); + db().prepare("DELETE FROM server_keys WHERE id='key-10'").run(); + expect(() => scaling.saveScaling(settings, saved.id, saved.revision, beforeKeyLoss.token)).toThrow('preview changed'); + expect(scaling.saveScaling(settings, saved.id, saved.revision, scaling.previewScaling(settings, saved.id, saved.revision).token).maximum).toBe(4); }); it('pauses evaluations and prunes expired observations without deleting current history', async () => { diff --git a/tests/unit/autoscaling-game.test.ts b/tests/unit/autoscaling-game.test.ts index 85f599b..6d6e5c8 100644 --- a/tests/unit/autoscaling-game.test.ts +++ b/tests/unit/autoscaling-game.test.ts @@ -14,7 +14,7 @@ async function setup() { await prepareNativeHost(); const { db } = await import('../../src/lib/server/db/index.js'); const scaling = await import('../../src/lib/server/autoscaling/index.js'); - for (const map of maps) for (const n of [1, 2]) db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_${map.slug}_${n}`, `key-${map.slug}-${n}`); + db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('key-1','t6','game-key')").run(); return { db, scaling }; } @@ -24,7 +24,7 @@ it('applies targets and spare deficits independently to every map', () => { const members: ScalingMember[] = maps.flatMap(map => [1, 2].map(n => ({ name: `t6_zombie_${map.slug}_${n}`, mapSlug: map.slug, identity: `${map.slug}:${n}`, state: map.slug === 'town' ? 'occupied' : 'empty', players: map.slug === 'town' ? 1 : 0, observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null }))); - const snapshot = { members, reservedNames: members.map(m => m.name), keyedNames: members.map(m => m.name), blockers: [] }; + const snapshot = { members, reservedNames: members.map(m => m.name), keyPool: { total: 15, available: 1 }, blockers: [] }; // Other maps' empty lobbies cannot satisfy Town's spare target or maximum. expect(planScaling(policy, snapshot, now)).toMatchObject({ action: 'destroy', server: 't6_zombie_transit_1', desired: maps.length + 1, deficit: 1 }); snapshot.members = members.filter(m => m.name !== 't6_zombie_town_2'); @@ -37,9 +37,9 @@ it('applies targets and spare deficits independently to every map', () => { it('persists a game rule, queues the selected map recipe, and protects every map', async () => { const { db, scaling } = await setup(); - db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run('t5_zombie_theater_1', 'other-game'); + db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('other-key','t5','other-game')").run(); const preview = scaling.previewScaling(settings); - expect(preview.snapshot.keyedNames).toHaveLength(maps.length * 2); + expect(preview.snapshot.keyPool).toEqual({ total: 1, available: 1 }); const saved = scaling.saveScaling(settings, undefined, undefined, preview.token); expect(() => scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token)).toThrow(); expect(() => scaling.previewScaling({ ...settings, mapSlug: 'town' }, saved.id, saved.revision)).toThrow('cannot change'); @@ -64,10 +64,14 @@ it('persists a game rule, queues the selected map recipe, and protects every map expect(scaling.previewScaling(settings, saved.id, current.revision).blockers.join(' ')).toContain(mapPolicy.id); }); -it('requires live key coverage on each map even when the game has enough keys in total', async () => { +it('holds game-wide growth when its shared pool is exhausted by another map', async () => { const { db, scaling } = await setup(); - db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_2'").run(); + db().prepare("INSERT INTO server_key_leases(server_id,key_id) VALUES('t6_zombie_town_2','key-1')").run(); const preview = scaling.previewScaling(settings); - expect(preview.blockers.join(' ')).toContain('Town Survival'); - expect(() => scaling.saveScaling(settings, undefined, undefined, preview.token)).toThrow('Town Survival'); + expect(preview.snapshot.keyPool).toEqual({ total: 1, available: 0 }); + expect(preview.blockers).toEqual([]); + expect(preview.decision).toMatchObject({ action: 'wait', reason: expect.stringContaining('keys are in use') }); + scaling.saveScaling(settings, undefined, undefined, preview.token); + expect(await scaling.runAutoscaling()).toBe(false); + expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 0 }); }); diff --git a/tests/unit/autoscaling-keys.test.ts b/tests/unit/autoscaling-keys.test.ts index 2588143..f988048 100644 --- a/tests/unit/autoscaling-keys.test.ts +++ b/tests/unit/autoscaling-keys.test.ts @@ -1,7 +1,6 @@ import { afterEach, beforeEach, expect, it, vi } from 'vitest'; import { newFixture, type Fixture } from '../helpers/fixture.js'; import { prepareNativeHost } from '../helpers/native-host.js'; -import { zombieMaps } from '../../src/lib/map-aliases.js'; let fixture: Fixture; beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); }); @@ -10,73 +9,90 @@ const settings = { game: 't6', mapSlug: 'town', maximum: 2, mode: 'dry-run' }; async function setup() { const { db } = await import('../../src/lib/server/db/index.js'); const scaling = await import('../../src/lib/server/autoscaling/index.js'); - const key = (name: string, value = `key-${name}`) => db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(name, value); - return { db, scaling, key }; + const { saveServerKey, reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + return { db, scaling, saveServerKey, reserveServerKey }; } -it.each(['off', 'dry-run', 'create-only', 'create-destroy'])('rejects new %s policies without enough keys through both service and API', async mode => { - const { db, scaling, key } = await setup(); - key('t6_zombie_town_3'); - const input = { ...settings, mode }; +it.each(['off', 'dry-run', 'create-only', 'create-destroy'])('saves a %s policy without preallocating keys for its maximum', async mode => { + if (mode.startsWith('create')) await prepareNativeHost(); + const { db, scaling } = await setup(); + const input = { ...settings, mode, maximum: 20 }; const preview = scaling.previewScaling(input); - expect(preview.blockers[0]).toContain('1 saved; 1 more required'); - expect(() => scaling.saveScaling(input, undefined, undefined, preview.token)).toThrow('Save keys'); + expect(preview.blockers).toEqual([]); + expect(preview.decision.action).toBe('wait'); + if (mode !== 'off') expect(preview.decision.reason).toContain('No Plutonium keys'); const { POST } = await import('../../src/routes/api/autoscaling/+server.js'); const response = await POST({ request: new Request('http://localhost/api/autoscaling', { method: 'POST', body: JSON.stringify({ action: 'save', settings: input, token: preview.token }) }) }); - expect(response.status).toBe(400); - expect(await response.json()).toMatchObject({ error: expect.stringContaining('Save keys') }); - expect(db().prepare('SELECT COUNT(*) AS n FROM autoscaling_policies').get()).toEqual({ n: 0 }); + expect(response.status).toBe(200); + expect(db().prepare('SELECT COUNT(*) AS n FROM autoscaling_policies').get()).toEqual({ n: 1 }); + expect(await scaling.runAutoscaling()).toBe(false); }); -it('counts only supported, nonblank identities in the selected map and accepts exact nonconsecutive coverage', async () => { - const { scaling, key } = await setup(); - for (const name of ['t6_zombie_town_0', 't6_zombie_town_21', 't6_zombie_town_01', 't6_zombie_town', 't6_town_2', 't5_zombie_theater_1', 't6_zombie_farm_1']) key(name); - key('t6_zombie_town_2', ' '); - key('t6_zombie_town_3'); - expect(scaling.previewScaling(settings).snapshot.keyedNames).toEqual(['t6_zombie_town_3']); - key('t6_zombie_town_8'); +it('uses any free key for the selected game independently of its label or map', async () => { + const { scaling, saveServerKey } = await setup(); + saveServerKey('t5', 'other-game-key', 'Town Lobby 1'); + saveServerKey('t6', 'game-key', 'Farm Lobby 8'); const preview = scaling.previewScaling(settings); - expect(preview.blockers).toEqual([]); + expect(preview.snapshot.keyPool).toEqual({ total: 1, available: 1 }); + expect(preview.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_1' }); expect(scaling.saveScaling(settings, undefined, undefined, preview.token).maximum).toBe(2); }); -it('requires coverage on every game map rather than pooling excess keys from another map', async () => { - const { scaling, key } = await setup(); - for (const map of zombieMaps('t6')) for (const n of [3, 8]) if (map.slug !== 'town' || n === 3) key(`t6_zombie_${map.slug}_${n}`); - key('t6_zombie_farm_9'); - const input = { ...settings, mapSlug: '*' }; +it('uses one shared pool for every map without requiring a key for each maximum', async () => { + const { scaling, saveServerKey } = await setup(); + saveServerKey('t6', 'game-key'); + const input = { ...settings, mapSlug: '*', maximum: 20 }; const preview = scaling.previewScaling(input); - expect(preview.blockers).toEqual([expect.stringContaining('Town Survival')]); - expect(() => scaling.saveScaling(input, undefined, undefined, preview.token)).toThrow('Town Survival'); - key('t6_zombie_town_8'); - expect(scaling.saveScaling(input, undefined, undefined, scaling.previewScaling(input).token).mapSlug).toBe('*'); -}); - -it('rechecks key deletion after preview and rejects increases beyond coverage', async () => { - const { db, scaling, key } = await setup(); - key('t6_zombie_town_3'); key('t6_zombie_town_8'); - const preview = scaling.previewScaling(settings); - db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_8'").run(); - expect(() => scaling.saveScaling(settings, undefined, undefined, preview.token)).toThrow('preview changed'); - expect(() => scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token)).toThrow('Save keys'); - key('t6_zombie_town_8'); - const saved = scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token); - const higher = { ...settings, maximum: 3 }; - expect(() => scaling.saveScaling(higher, saved.id, saved.revision, scaling.previewScaling(higher, saved.id, saved.revision).token)).toThrow('Save keys'); - expect(scaling.getScalingPolicy(saved.id)?.maximum).toBe(2); -}); - -it('allows an existing live policy to pause and withdraw pending work after all keys are removed', async () => { - await prepareNativeHost(); - const { db, scaling, key } = await setup(); - key('t6_zombie_town_3'); key('t6_zombie_town_8'); - const live = { ...settings, mode: 'create-only' }; - const saved = scaling.saveScaling(live, undefined, undefined, scaling.previewScaling(live).token); - await scaling.runAutoscaling(); - db().prepare('DELETE FROM server_keys').run(); - const off = { ...settings, mode: 'off' }; - const preview = scaling.previewScaling(off, saved.id, saved.revision); expect(preview.blockers).toEqual([]); - expect(scaling.saveScaling(off, saved.id, saved.revision, preview.token).mode).toBe('off'); - expect(db().prepare("SELECT state FROM server_deployments WHERE action='create'").get()).toEqual({ state: 'cancelled' }); + expect(preview.snapshot.keyPool).toEqual({ total: 1, available: 1 }); + expect(scaling.saveScaling(input, undefined, undefined, preview.token).mapSlug).toBe('*'); +}); + +it('rechecks game pool deletion and reservations after preview', async () => { + const { db, scaling, saveServerKey, reserveServerKey } = await setup(); + const keyId = saveServerKey('t6', 'game-key'); + const preview = scaling.previewScaling(settings); + db().prepare('DELETE FROM server_keys WHERE id=?').run(keyId); + expect(() => scaling.saveScaling(settings, undefined, undefined, preview.token)).toThrow('preview changed'); + saveServerKey('t6', 'game-key'); + const available = scaling.previewScaling(settings); + reserveServerKey('t6_zombie_farm_1', 't6'); + expect(() => scaling.saveScaling(settings, undefined, undefined, available.token)).toThrow('preview changed'); + const held = scaling.previewScaling(settings); + expect(held.decision.reason).toContain('keys are in use'); + expect(held.snapshot.members).toEqual([]); + expect(scaling.saveScaling(settings, undefined, undefined, held.token).maximum).toBe(2); +}); + +it('shares capacity between live map policies and admits the job holding the last key', async () => { + await prepareNativeHost(); + const { db, scaling, saveServerKey } = await setup(); + saveServerKey('t6', 'game-key'); + saveServerKey('t5', 'other-game-key'); + for (const mapSlug of ['town', 'farm']) { + const input = { ...settings, mapSlug, mode: 'create-only' }; + scaling.saveScaling(input, undefined, undefined, scaling.previewScaling(input).token); + } + expect(await scaling.runAutoscaling()).toBe(true); + const jobs = db().prepare('SELECT deployment_id,server FROM server_creation_jobs').all() as { deployment_id: string; server: string }[]; + expect(jobs).toHaveLength(1); + expect(jobs[0].server).toBe('t6_zombie_farm_1'); + const { scalingAdmission } = await import('../../src/lib/server/autoscaling/admission.js'); + expect(scalingAdmission(jobs[0].deployment_id)).toBeNull(); + const town = scaling.scalingReport().policies.find(p => p.policy.mapSlug === 'town')!; + expect(town.snapshot.keyPool).toEqual({ total: 1, available: 0 }); + expect(town.decision.reason).toContain('keys are in use'); + expect(town.policy.pausedReason).toBeNull(); + expect(await scaling.runAutoscaling()).toBe(false); + expect(db().prepare('SELECT COUNT(*) AS n FROM server_creation_jobs').get()).toEqual({ n: 1 }); +}); + +it('reserves a lobby identity for an unsettled failed launch with a retained key', async () => { + const { scaling, saveServerKey, reserveServerKey } = await setup(); + saveServerKey('t6', 'held-key'); + reserveServerKey('t6_zombie_town_1', 't6'); + saveServerKey('t6', 'free-key'); + const preview = scaling.previewScaling(settings); + expect(preview.snapshot.reservedNames).toEqual(['t6_zombie_town_1']); + expect(preview.decision).toMatchObject({ action: 'create', server: 't6_zombie_town_2' }); }); diff --git a/tests/unit/autoscaling-live.test.ts b/tests/unit/autoscaling-live.test.ts index 1549dae..90a8af8 100644 --- a/tests/unit/autoscaling-live.test.ts +++ b/tests/unit/autoscaling-live.test.ts @@ -10,7 +10,7 @@ async function setup() { await prepareNativeHost(); const { db } = await import('../../src/lib/server/db/index.js'); const scaling = await import('../../src/lib/server/autoscaling/index.js'); - for (const n of [3, 8]) db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_town_${n}`, `key-${n}`); + for (const n of [3, 8]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `key-${n}`); const saved = scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token); return { db, scaling, saved }; } @@ -21,7 +21,7 @@ it('admits one durable keyed create and reuses it across ticks and restart', asy await Promise.all([scaling.runAutoscaling(), scaling.runAutoscaling()]); const { closeDb } = await import('../../src/lib/server/db/index.js'); closeDb(); await scaling.runAutoscaling(); - expect(db().prepare('SELECT server,state FROM server_deployments WHERE action=\'create\'').all()).toEqual([{ server: 't6_zombie_town_3', state: 'queued' }]); + expect(db().prepare('SELECT server,state FROM server_deployments WHERE action=\'create\'').all()).toEqual([{ server: 't6_zombie_town_1', state: 'queued' }]); expect(db().prepare('SELECT COUNT(*) AS n FROM autoscaling_decisions WHERE resolved=0').get()).toEqual({ n: 1 }); const { operation_id: id } = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string }; const { scalingAdmission } = await import('../../src/lib/server/autoscaling/admission.js'); @@ -30,16 +30,19 @@ it('admits one durable keyed create and reuses it across ticks and restart', asy expect(getDeployment(id)?.sources).toContainEqual(expect.objectContaining({ kind: 'autoscaling', href: `/autoscaling#${saved.id}` })); }); -it('requires key coverage and withdraws an unsubmitted create when the policy pauses', async () => { +it('withdraws an unsubmitted create and returns its game key when the policy pauses', async () => { const { db, scaling, saved } = await setup(); await scaling.runAutoscaling(); const off = { ...settings, mode: 'off' }; scaling.saveScaling(off, saved.id, saved.revision, scaling.previewScaling(off, saved.id, saved.revision).token); expect(db().prepare("SELECT state FROM server_deployments WHERE action='create'").get()).toEqual({ state: 'cancelled' }); expect(db().prepare('SELECT owns_map FROM autoscaling_policies').get()).toEqual({ owns_map: 0 }); - db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_8'").run(); + expect(db().prepare('SELECT COUNT(*) AS n FROM server_key_leases').get()).toEqual({ n: 0 }); + db().prepare('DELETE FROM server_keys').run(); const current = scaling.getScalingPolicy(saved.id)!; - expect(scaling.previewScaling(settings, saved.id, current.revision).blockers.join(' ')).toContain('Save keys'); + const preview = scaling.previewScaling(settings, saved.id, current.revision); + expect(preview.blockers).toEqual([]); + expect(preview.snapshot.keyPool).toEqual({ total: 0, available: 0 }); }); it('blocks manual lifecycle and collection creation, including stale previews', async () => { @@ -78,7 +81,7 @@ it('exact matching cannot remove an owned map absent from the collection', async const { db } = await import('../../src/lib/server/db/index.js'); const at = new Date().toISOString(); db().prepare("INSERT INTO servers(name,game,map,map_slug,container_state,container_checked_at,last_seen_at,last_players) VALUES(?,'t6','zm_transit','town','running',?,?,0)").run(town, at, at); - for (const n of [1, 2]) db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_town_${n}`, `key-${n}`); + for (const n of [1, 2]) db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${n}`, `key-${n}`); const scaling = await import('../../src/lib/server/autoscaling/index.js'); scaling.saveScaling(settings, undefined, undefined, scaling.previewScaling(settings).token); const { saveBlueprint } = await import('../../src/lib/server/fleet/launch-recipes.js'); @@ -90,13 +93,12 @@ it('exact matching cannot remove an owned map absent from the collection', async expect(() => queueCollection({ previewId: preview.id })).toThrow('Autoscaling'); }); -it('rechecks evidence and global pause before dispatch', async () => { +it('honors global pause before dispatch while recognizing its own reserved key', async () => { const { db, scaling } = await setup(); await scaling.runAutoscaling(); const { operation_id: id } = db().prepare('SELECT operation_id FROM autoscaling_decisions WHERE resolved=0').get() as { operation_id: string }; const { scalingAdmission } = await import('../../src/lib/server/autoscaling/admission.js'); - db().prepare("DELETE FROM server_keys WHERE server_id='t6_zombie_town_3'").run(); - expect(scalingAdmission(id)).toContain('no longer eligible'); + expect(scalingAdmission(id)).toBeNull(); scaling.setScalingGlobalPause(true); expect(db().prepare('SELECT state FROM server_deployments WHERE id=?').get(id)).toEqual({ state: 'cancelled' }); expect(await scaling.runAutoscaling()).toBe(false); diff --git a/tests/unit/autoscaling-planner.test.ts b/tests/unit/autoscaling-planner.test.ts index 356ded4..16712c2 100644 --- a/tests/unit/autoscaling-planner.test.ts +++ b/tests/unit/autoscaling-planner.test.ts @@ -7,7 +7,7 @@ const now = 2_000_000; const policy: ScalingPolicy = { ...scalingSettings({ game: 't6', mapSlug: 'town', maximum: 4, mode: 'dry-run' }), id: 'policy', revision: 1, nextActionAt: 0, pausedReason: null, updatedAt: '' }; const name = (n: number) => `t6_zombie_town_${n}`; const member = (n: number, state: ScalingMember['state'] = 'empty'): ScalingMember => ({ name: name(n), identity: `instance:${n}`, state, players: state === 'occupied' ? 1 : state === 'empty' ? 0 : null, observedAt: now, source: 'rcon', emptySince: 1, blocker: null, operationId: null }); -const snapshot = (...members: ScalingMember[]): ScalingSnapshot => ({ members, reservedNames: members.map(m => m.name), keyedNames: [1, 2, 3, 4].map(name), blockers: [] }); +const snapshot = (...members: ScalingMember[]): ScalingSnapshot => ({ members, reservedNames: members.map(m => m.name), keyPool: { total: 4, available: 4 - members.length }, blockers: [] }); describe('autoscaling policy planning', () => { it('requires explicit bounded maximum and valid variant', () => { @@ -19,9 +19,13 @@ describe('autoscaling policy planning', () => { expect(planScaling(policy, snapshot(), now)).toMatchObject({ action: 'create', desired: 1 }); expect(planScaling(policy, snapshot(member(1, 'occupied'), member(2)), now).action).toBe('wait'); }); - it('uses nonconsecutive keyed identities without moving keys', () => { - expect(planScaling(policy, { ...snapshot(), keyedNames: [name(8), name(3)] }, now).server).toBe(name(3)); - expect(planScaling(policy, { ...snapshot(), keyedNames: [] }, now).reason).toContain('saved key'); + it('selects the first free identity independently of the game key pool size', () => { + expect(planScaling(policy, { ...snapshot(), reservedNames: [name(1), name(3)], keyPool: { total: 1, available: 1 } }, now).server).toBe(name(2)); + expect(planScaling(policy, { ...snapshot(), keyPool: { total: 0, available: 0 } }, now).reason).toContain('No Plutonium keys'); + expect(planScaling(policy, { ...snapshot(), keyPool: { total: 2, available: 0 } }, now).reason).toContain('keys are in use'); + }); + it('still removes eligible surplus lobbies when the game pool is exhausted', () => { + expect(planScaling(policy, { ...snapshot(member(1), member(2)), keyPool: { total: 2, available: 0 } }, now)).toMatchObject({ action: 'destroy', server: name(1) }); }); it('holds for pending creation, unknown, stopped, or protected members', () => { for (const state of ['creating', 'removing', 'unknown', 'stopped', 'protected'] as const) expect(planScaling(policy, snapshot(member(1, state)), now).action).toBe('wait'); diff --git a/tests/unit/autoscaling-removal.test.ts b/tests/unit/autoscaling-removal.test.ts index 29d74b2..a299f60 100644 --- a/tests/unit/autoscaling-removal.test.ts +++ b/tests/unit/autoscaling-removal.test.ts @@ -22,10 +22,10 @@ async function setup(gameWide = false) { for (const name of servers) { const map = zombieMaps('t6').find(m => name === `t6_zombie_${m.slug}_1` || name === `t6_zombie_${m.slug}_2`)!; db().prepare("INSERT INTO servers(name,game,map,map_slug,container_state,last_players) VALUES(?,'t6',?,?,'running',0)").run(name, map.canonical, map.slug); - db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(name, `key-${name}`); + db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${name}`, `key-${name}`); } if (gameWide) for (const map of zombieMaps('t6').filter(m => m.slug !== 'town')) { - db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run(`t6_zombie_${map.slug}_2`, `key-${map.slug}-2`); + db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES(?,'t6',?)").run(`key-${map.slug}-2`, `key-${map.slug}-2`); } const fresh = (offset: number) => { vi.setSystemTime(base + offset); const at = new Date().toISOString(); diff --git a/tests/unit/autoscaling-snapshot.test.ts b/tests/unit/autoscaling-snapshot.test.ts index 0a2cb4e..26dd482 100644 --- a/tests/unit/autoscaling-snapshot.test.ts +++ b/tests/unit/autoscaling-snapshot.test.ts @@ -31,7 +31,7 @@ it('does not create schema mutations or host work when planning an empty map', a const { db } = await import('../../src/lib/server/db/index.js'); const { scalingSnapshot } = await import('../../src/lib/server/autoscaling/snapshot.js'); const { planScaling } = await import('../../src/lib/server/autoscaling/planner.js'); - db().prepare('INSERT INTO server_keys(server_id,pluto_key) VALUES(?,?)').run('t6_zombie_town_8', 'private-key'); - expect(planScaling(policy, scalingSnapshot(policy), Date.now()).server).toBe('t6_zombie_town_8'); + db().prepare("INSERT INTO server_keys(id,game,pluto_key) VALUES('key-8','t6','private-key')").run(); + expect(planScaling(policy, scalingSnapshot(policy), Date.now()).server).toBe('t6_zombie_town_1'); expect(db().prepare('SELECT COUNT(*) AS n FROM server_deployments').get()).toEqual({ n: 0 }); }); diff --git a/tests/unit/beta-destruction.test.ts b/tests/unit/beta-destruction.test.ts index e748401..a60b3bb 100644 --- a/tests/unit/beta-destruction.test.ts +++ b/tests/unit/beta-destruction.test.ts @@ -111,11 +111,11 @@ describe('active beta destruction guard', () => { const { requestLifecycle } = await import('../../src/lib/server/fleet/lifecycle.js'); const { cancelRemoval } = await import('../../src/lib/server/fleet/deployments.js'); const { cancelProvisioning } = await import('../../src/lib/server/provisioning/requests.js'); - const { saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + const { reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); const { createScript } = await import('../../src/lib/server/fleet/scripts.js'); const { openBeta } = await import('../../src/lib/server/fleet/beta.js'); const { db } = await import('../../src/lib/server/db/index.js'); - saveServerKey(server, 'saved-key'); + reserveServerKey(server, 't5', 'saved-key'); const queued = action === 'destroy' ? destroyServer(server) : requestLifecycle(server, 'rebuild'); expect(queued.ok).toBe(true); if (!queued.ok) throw new Error('Expected queued removal'); diff --git a/tests/unit/collection-launch-performance.test.ts b/tests/unit/collection-launch-performance.test.ts index 9bc4b83..b3edfb2 100644 --- a/tests/unit/collection-launch-performance.test.ts +++ b/tests/unit/collection-launch-performance.test.ts @@ -117,6 +117,6 @@ describe('large native collection admission', () => { .toThrow(change === 'newer game files' ? /game files changed/ : /host was not found/); expect(db().prepare('SELECT id FROM server_launch_collections').all()).toEqual([]); expect(db().prepare('SELECT deployment_id FROM native_creation_intents').all()).toEqual([]); - expect(db().prepare('SELECT server_id FROM server_keys').all()).toEqual([]); + expect(db().prepare('SELECT id FROM server_keys').all()).toEqual([]); }); }); diff --git a/tests/unit/deployment-collections.test.ts b/tests/unit/deployment-collections.test.ts index 85a7fd3..df97b6f 100644 --- a/tests/unit/deployment-collections.test.ts +++ b/tests/unit/deployment-collections.test.ts @@ -90,19 +90,36 @@ describe('multi-server blueprints', () => { describe('atomic collection launches', () => { + it('counts available keys once per game across all maps without reserving them in a preview', async () => { + const { blueprint } = await previewLineup(); + const { saveServerKey, reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + saveServerKey('t5', 'pool-one'); saveServerKey('t5', 'pool-two'); saveServerKey('t6', 'pool-three'); + const preview = previewCollection(blueprint.id); + expect(preview.servers.map(server => server.hasSavedKey)).toEqual([true, true, false, true, false]); + expect(db().prepare('SELECT * FROM server_key_leases').all()).toEqual([]); + reserveServerKey('t5_zombie_pentagon_1', 't5'); + expect(() => queueCollection({ previewId: preview.id })).toThrow(/No available T5 server keys/); + expect(db().prepare('SELECT * FROM server_creation_jobs').all()).toEqual([]); + expect(db().prepare('SELECT * FROM server_key_leases').all()).toHaveLength(1); + }); + it('rolls back all earlier jobs and key changes if the final server has no valid key', async () => { const { preview } = await previewLineup(); const { queueCollection } = await import('../../src/lib/server/fleet/collections.js'); const { db } = await import('../../src/lib/server/db/index.js'); const { saveServerKey, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); - saveServerKey(preview.servers[0].name!, 'original-key'); + saveServerKey('t5', 'original-key'); const keys = keysFor(preview); delete keys[preview.servers[4].name!]; - expect(() => queueCollection({ previewId: preview.id, keys })).toThrow(/Save a Plutonium server key/); + expect(() => queueCollection({ previewId: preview.id, keys })).toThrow(/No available .* server keys/); for (const table of ['server_launch_collections', 'server_collection_members', 'server_creation_jobs', 'server_deployments', 'server_launch_previews', 'server_launch_recipes']) { expect(db().prepare(`SELECT * FROM ${table}`).all()).toEqual([]); } - expect(getServerKey(preview.servers[0].name!)).toBe('original-key'); + expect(getServerKey(preview.servers[0].name!)).toBeUndefined(); + expect(db().prepare('SELECT pluto_key FROM server_keys').get()).toEqual({ pluto_key: 'original-key' }); + expect(db().prepare('SELECT * FROM server_key_leases').all()).toEqual([]); expect(db().prepare('SELECT * FROM server_keys').all()).toHaveLength(1); keys[preview.servers[4].name!] = 'invalid key'; expect(() => queueCollection({ previewId: preview.id, keys })).toThrow(/without whitespace/); @@ -115,7 +132,7 @@ describe('atomic collection launches', () => { const { queueCollection, previewCollection, getCollection } = await import('../../src/lib/server/fleet/collections.js'); const { saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); const { db, closeDb } = await import('../../src/lib/server/db/index.js'); - for (const server of preview.servers) saveServerKey(server.name!, `saved-${server.name}`); + for (const server of preview.servers) saveServerKey(server.name!.slice(0, 2), `saved-${server.name}`); const ready = previewCollection(blueprint.id); expect(ready.servers.every(server => server.hasSavedKey)).toBe(true); const queued = queueCollection({ previewId: ready.id }); @@ -171,17 +188,18 @@ describe('matching a blueprint exactly', () => { expect(db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(first.deploymentId)).toEqual({ state: 'queued' }); expect(getCollection(queued.collectionId!)?.collection).toMatchObject({ done: 1, total: 1, active: 0 }); const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); - expect(getServerKey(second.name)).toBe('second'); + expect(getServerKey(second.name)).toBeUndefined(); + expect(db().prepare("SELECT pluto_key FROM server_keys WHERE pluto_key='second'").get()).toEqual({ pluto_key: 'second' }); }); - it('rolls back removals and creation cancellations when a required key is missing', async () => { + it('rolls back removals and creation cancellations when the game pool is exhausted', async () => { const { queueCreateServer } = await import('../../src/lib/server/fleet/create.js'); const old = queueCreateServer({ gameMode: 'zombie', mapSlug: 'moon', plutoKey: 'existing' }); if (!old.ok) throw new Error(old.message); - const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 1 }]); + const plan = await blueprint([{ game: 't5', mapSlug: 'theater', count: 2 }]); const { previewCollection, queueCollection } = await import('../../src/lib/server/fleet/collections.js'); const preview = previewCollection(plan.id, 'exact'); - expect(() => queueCollection({ previewId: preview.id })).toThrow(/Save a Plutonium server key/); + expect(() => queueCollection({ previewId: preview.id })).toThrow(/No available .* server keys/); const { db } = await import('../../src/lib/server/db/index.js'); expect(db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(old.deploymentId)).toEqual({ state: 'queued' }); for (const table of ['server_collection_removals', 'server_launch_collections', 'server_creation_dependencies']) expect(db().prepare(`SELECT * FROM ${table}`).all()).toEqual([]); diff --git a/tests/unit/host-deletion.test.ts b/tests/unit/host-deletion.test.ts index 21a7787..47608e0 100644 --- a/tests/unit/host-deletion.test.ts +++ b/tests/unit/host-deletion.test.ts @@ -153,7 +153,7 @@ describe('host deletion', () => { { hostId: other.id, automatic: false } ].entries()) { expect(queueNativeCreationIntent({ server: `t5_zombie_theater_${index + 1}`, game: 't5', mapSlug: 'theater', - plutoKey: 'saved-key', selection: { ...selection, libraryId: 'library', image: 'plutainer:test' } }).ok).toBe(true); + plutoKey: `saved-key-${index}`, selection: { ...selection, libraryId: 'library', image: 'plutainer:test' } }).ok).toBe(true); } expect(hosts.hostDeletionStatus(hosts.host.id)).toEqual({ allowed: true, blockers: [] }); expect(hosts.hostDeletionStatus(other.id).allowed).toBe(false); diff --git a/tests/unit/host-preparation.test.ts b/tests/unit/host-preparation.test.ts index 8703e6b..c3939e1 100644 --- a/tests/unit/host-preparation.test.ts +++ b/tests/unit/host-preparation.test.ts @@ -353,7 +353,7 @@ describe('automatic host preparation', () => { const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js'); for (let index = 0; index < 2; index++) { expect(queueNativeCreation({ game: 't5', gameMode: 'zombie', mapSlug: 'theater', libraryId: old.id, - hostId: automatic ? undefined : id, image, plutoKey: 'saved-key' }).ok).toBe(true); + hostId: automatic ? undefined : id, image, plutoKey: `saved-key-${index}` }).ok).toBe(true); } // Also cover pre-existing intents whose image is no longer cached. heartbeat(id, { runtimeImages: [runtime] }); diff --git a/tests/unit/native-creation-intents.test.ts b/tests/unit/native-creation-intents.test.ts index 0c0d782..c2a40d3 100644 --- a/tests/unit/native-creation-intents.test.ts +++ b/tests/unit/native-creation-intents.test.ts @@ -55,12 +55,12 @@ describe('native creation intents before placement', () => { it('cancels unplaced work locally once, retains keys, and frees its logical server name', async () => { const id = await intent(); const { db } = await import('../../src/lib/server/db/index.js'); - const { saveServerKey, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + const { reserveServerKey, getServerKey, serverKeyPool } = await import('../../src/lib/server/fleet/server-keys.js'); const { cancelCreation, canCancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js'); const { getDeployment } = await import('../../src/lib/server/provisioning/history.js'); const { nativeDeployment, unplacedNativeCreation } = await import('../../src/lib/server/backends/native.js'); const { nextLobbyNumber } = await import('../../src/lib/server/provisioning/instances.js'); - saveServerKey(server, 'saved-key'); + reserveServerKey(server, 't5', 'saved-key'); expect(canCancelCreation(id)).toBe(true); const cleanup = cancelCreation(id)!; expect(cleanup).toEqual(expect.any(String)); @@ -71,7 +71,8 @@ describe('native creation intents before placement', () => { expect(getDeployment(cleanup)).toMatchObject({ backend: 'native', deployment: { state: 'done', server_deleted_at: expect.any(String) } }); expect(unplacedNativeCreation(server)).toBeNull(); expect(nextLobbyNumber('t5_zombie_theater')).toBe(1); - expect(getServerKey(server)).toBe('saved-key'); + expect(getServerKey(server)).toBeUndefined(); + expect(serverKeyPool('t5')).toEqual({ total: 1, available: 1 }); expect(db().prepare('SELECT id FROM ec_host_operations').all()).toEqual([]); expect(db().prepare('SELECT instance_id FROM ec_host_ports').all()).toEqual([]); }); @@ -142,7 +143,7 @@ describe('native creation intents before placement', () => { const { queueNativeCreation, processNativeProvisioning } = await import('../../src/lib/server/provisioning/native.js'); const input = { game: 't5' as const, gameMode: 'zombie' as const, mapSlug: 'theater', libraryId: library.id, image, plutoKey: 'saved-key' }; const blocked = queueNativeCreation({ ...input, hostId: offline.id }); - const available = queueNativeCreation({ ...input, hostId: ready.id }); + const available = queueNativeCreation({ ...input, hostId: ready.id, plutoKey: 'ready-key' }); expect(blocked.ok && available.ok).toBe(true); if (!blocked.ok || !available.ok) throw new Error('Could not queue launch fixtures.'); await processNativeProvisioning(() => null); diff --git a/tests/unit/native-launch-preparation.test.ts b/tests/unit/native-launch-preparation.test.ts index f9f7795..e71d55f 100644 --- a/tests/unit/native-launch-preparation.test.ts +++ b/tests/unit/native-launch-preparation.test.ts @@ -43,10 +43,11 @@ afterEach(async () => { vi.unstubAllEnvs(); }); +let queuedKeys = 0; async function queue() { const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js'); const result = queueNativeCreation({ game: 't5', gameMode: 'zombie', mapSlug: 'theater', - hostId, libraryId, image, plutoKey: 'saved-key' }); + hostId, libraryId, image, plutoKey: `saved-key-${++queuedKeys}` }); if (!result.ok) throw new Error(result.message); return result; } diff --git a/tests/unit/native-provisioning.test.ts b/tests/unit/native-provisioning.test.ts index d09d11e..24e4278 100644 --- a/tests/unit/native-provisioning.test.ts +++ b/tests/unit/native-provisioning.test.ts @@ -12,11 +12,12 @@ vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ vi.mock('../../src/lib/server/integrations/plutonium.js', () => ({ checkRelease: vi.fn(async () => {}), freshRelease: () => 5354, probeVersion: vi.fn(async () => 5354) })); vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}), kickProvisioning: vi.fn(async () => {}) })); +let queuedKeys = 0; let fixture: Fixture; let hostId: string; let libraryId: string; const image = 'example.test/plutainer:test'; const inventory = () => ({ cpuCount: 16, cpuUsagePercent: 10, memoryTotal: 64 * 1024 ** 3, memoryAvailable: 60 * 1024 ** 3, protocolVersion: 1, dockerRunning: true, directoriesWritable: true, storageAvailable: 100_000_000_000, runtimeImages: [image], libraries: libraryId ? [{ id: libraryId, state: 'ready' as const }] : [] }); beforeEach(async () => { - vi.resetModules(); vi.clearAllMocks(); remote.state = 'running'; remote.players = 0; remote.map = 'zombie_theater'; libraryId = ''; + vi.resetModules(); vi.clearAllMocks(); remote.state = 'running'; remote.players = 0; remote.map = 'zombie_theater'; libraryId = ''; queuedKeys = 0; fixture = await newFixture([]); vi.stubEnv('GAME_FILES_ROOT', join(fixture.root, 'libraries')); vi.stubEnv('PLUTAINER_IMAGE', image); vi.stubEnv('RCON_PASSWORD', 'test-password'); @@ -48,7 +49,7 @@ async function queue(game: 't5' | 't6' = 't5', placement: 'pinned' | 'automatic' await prepareLibrary(game); remote.map = game === 't5' ? 'zombie_theater' : 'zm_transit'; const { queueCreateServer } = await import('../../src/lib/server/provisioning/instances.js'); const result = queueCreateServer({ game, gameMode: 'zombie', mapSlug: game === 't5' ? 'theater' : 'town', - hostId: placement === 'pinned' ? hostId : undefined, libraryId, image, plutoKey: 'saved-key' }); + hostId: placement === 'pinned' ? hostId : undefined, libraryId, image, plutoKey: ++queuedKeys === 1 ? 'saved-key' : `saved-key-${queuedKeys}` }); if (!result.ok) throw new Error(result.message); return result; } @@ -990,8 +991,9 @@ describe('native instance provisioning', () => { expect(db().prepare("SELECT COUNT(*) AS count FROM ec_host_operations WHERE kind LIKE 'instance.%'").get()).toEqual({ count: 0 }); const { getLibraryVersion } = await import('../../src/lib/server/game-files/index.js'); expect(getLibraryVersion(libraryId)?.state).toBe('ready'); - const { getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); - expect(getServerKey(job.name)).toBe('saved-key'); + const { getServerKey, serverKeyPool } = await import('../../src/lib/server/fleet/server-keys.js'); + expect(getServerKey(job.name)).toBeUndefined(); + expect(serverKeyPool('t5')).toEqual({ total: 1, available: 1 }); }); }); @@ -1083,7 +1085,7 @@ describe('automatic capacity admission and placement', () => { const preview = previewCollection(blueprint.id); const { heartbeat } = await import('../../src/lib/server/hosts/index.js'); heartbeat(hostId, { ...inventory(), cpuUsagePercent: 100 }); - const result = queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) }); + const result = queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) }); expect(result).toMatchObject({ ok: true, collectionId: expect.any(String) }); await advance(); const { db } = await import('../../src/lib/server/db/index.js'); @@ -1117,7 +1119,7 @@ describe('automatic capacity admission and placement', () => { const { queueNativeCreation } = await import('../../src/lib/server/provisioning/native.js'); const measured = { ...inventory(), cpuCount: 1, cpuUsagePercent: 50 }; heartbeat(hostId, measured); - const next = queueNativeCreation(input); + const next = queueNativeCreation({ ...input, plutoKey: 'next-key' }); expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message); await advance(); expect(await operationFor(next.deploymentId)).toBeNull(); @@ -1151,7 +1153,7 @@ describe('automatic capacity admission and placement', () => { const blueprint = saveBlueprint({ name: 'Staggered lobbies', entries: [{ game: 't5', mapSlug: 'theater', count: 3 }] }); const preview = previewCollection(blueprint.id); expect(preview.ready).toBe(true); - expect(queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) }).ok).toBe(true); + expect(queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) }).ok).toBe(true); await advance(); const creates = () => db().prepare("SELECT instance_id FROM ec_host_operations WHERE kind='instance.create'").all() as { instance_id: string }[]; expect(creates()).toHaveLength(1); @@ -1170,7 +1172,7 @@ describe('automatic capacity admission and placement', () => { const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js'); const first = queueNativeCreation(input); expect(first.ok).toBe(true); if (!first.ok) return; - const second = queueNativeCreation(input); + const second = queueNativeCreation({ ...input, plutoKey: 'second-key' }); expect(second).toMatchObject({ ok: true, name: 't5_zombie_theater_2', port: null }); const { db } = await import('../../src/lib/server/db/index.js'); expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]); @@ -1191,7 +1193,7 @@ describe('automatic capacity admission and placement', () => { (host_id,sampled_at,cpu_count,cpu_usage_percent,memory_total,memory_available) VALUES(?,?,4,?,?,?)`) .run(hostId, new Date(Date.now() - minute * 60_000).toISOString(), resource === 'cpu' ? 100 : 10, 4 * GiB, resource === 'memory' ? GiB / 8 : Math.floor(3.8 * GiB)); - const jobs = Array.from({ length: 3 }, () => queueNativeCreation(input)); + const jobs = Array.from({ length: 3 }, (_, index) => queueNativeCreation({ ...input, plutoKey: `key-${index}` })); expect(jobs.every(job => job.ok)).toBe(true); await advance(); expect(await operationFor(jobs[0].ok ? jobs[0].deploymentId : 'missing')).not.toBeNull(); @@ -1205,7 +1207,7 @@ describe('automatic capacity admission and placement', () => { const { db } = await import('../../src/lib/server/db/index.js'); const GiB = 1024 ** 3; heartbeat(hostId, { ...inventory(), cpuCount: 4, memoryTotal: 4 * GiB, memoryAvailable: Math.floor(3.8 * GiB) }); - const jobs = Array.from({ length: 3 }, () => queueNativeCreation(input)); + const jobs = Array.from({ length: 3 }, (_, index) => queueNativeCreation({ ...input, plutoKey: `key-${index}` })); expect(jobs.every(job => job.ok)).toBe(true); // Learning increases the startup allowance after admission. All three // no longer fit concurrently, but this otherwise empty host fits one. @@ -1244,7 +1246,7 @@ describe('automatic capacity admission and placement', () => { // Adding a new 1 GiB server fits below 2.53 GiB usable memory. heartbeat(hostId, { ...host, memoryAvailable: 2 * GiB, instanceResources: [{ instanceId: instance.id, generation: instance.generation, uptimeSeconds: 60, cpuCores: 0.1, memoryBytes: Math.floor(0.8 * GiB) }] }); - const second = queueNativeCreation(input); + const second = queueNativeCreation({ ...input, plutoKey: 'second-key' }); expect(second).toMatchObject({ ok: true }); await advance(); expect(await operationFor(second.ok ? second.deploymentId : 'missing')).not.toBeNull(); @@ -1265,7 +1267,7 @@ describe('automatic capacity admission and placement', () => { heartbeat(hostId, { ...inventory(), memoryTotal: 3 * GiB, memoryAvailable: 2 * GiB, instanceResources: evidence === 'missing' ? [] : [{ instanceId: instance.id, generation: instance.generation + 1, uptimeSeconds: 60, cpuCores: 0.1, memoryBytes: Math.floor(0.8 * GiB) }] }); - const next = queueNativeCreation(input); + const next = queueNativeCreation({ ...input, plutoKey: 'next-key' }); expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message); await advance(); expect(await operationFor(next.deploymentId)).toBeNull(); @@ -1318,7 +1320,7 @@ describe('automatic capacity admission and placement', () => { await advance(); expect(nativeInstance(result.name)?.host_id).toBe(other.id); updateHost(other.id, { mode: 'draining' }); - const next = queueNativeCreation(input); + const next = queueNativeCreation({ ...input, plutoKey: 'next-key' }); expect(next.ok).toBe(true); if (!next.ok) throw new Error(next.message); await advance(); expect(await operationFor(next.deploymentId)).toBeNull(); @@ -1333,7 +1335,7 @@ describe('automatic capacity admission and placement', () => { const preview = previewCollection(blueprint.id); expect(preview.ready).toBe(true); expect(preview.servers.map(server => server.port)).toEqual([null, null]); - expect(queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key'])) }).ok).toBe(true); + expect(queueCollection({ previewId: preview.id, keys: Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])) }).ok).toBe(true); const { db } = await import('../../src/lib/server/db/index.js'); expect(db().prepare('SELECT id FROM native_instances').all()).toEqual([]); const { fleetSettings, saveFleetSettings } = await import('../../src/lib/server/settings.js'); @@ -1481,6 +1483,7 @@ describe('cancelled native mutation bookkeeping', () => { it.each(['rebuild','destroy'] as const)('reconciles a delivered %s receipt without resuming the cancelled job', async action => { const job = await queue(); await advance(); await acknowledge(job.deploymentId); const { db } = await import('../../src/lib/server/db/index.js'); + const { serverKeyPool, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js'); const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js'); const { cancelProvisioningJob } = await import('../../src/lib/server/provisioning/cancellation.js'); @@ -1491,6 +1494,7 @@ describe('cancelled native mutation bookkeeping', () => { await advance(); const operation = (await operationFor(id))!; pendingHostOperations(hostId); expect(cancelProvisioningJob(id)).toBe(true); + expect(serverKeyPool('t5').available).toBe(0); const replacement = action === 'rebuild' ? requestLifecycle(job.name, 'restart') : null; if (replacement && !replacement.ok) throw new Error(replacement.error); await advance(); @@ -1498,6 +1502,8 @@ describe('cancelled native mutation bookkeeping', () => { await advance(); await advance(); if (replacement?.ok) expect(await operationFor(replacement.deploymentId)).toMatchObject({ kind: 'instance.restart', generation: 2 }); expect((await detail(id)).state).toBe('cancelled'); + expect(serverKeyPool('t5').available).toBe(action === 'destroy' ? 1 : 0); + expect(getServerKey(job.name)).toBe(action === 'destroy' ? undefined : 'saved-key'); if(action === 'rebuild') expect(db().prepare('SELECT generation,retired_at FROM native_instances WHERE server=?').get(job.name)).toEqual({generation:2,retired_at:null}); else { expect(db().prepare('SELECT * FROM ec_host_ports').all()).toEqual([]); @@ -1508,6 +1514,7 @@ describe('cancelled native mutation bookkeeping', () => { it.each([false,true])('cancels a host migration without restarting it or leaking destination reservations (delivered=%s)', async delivered => { const job = await queue(); await advance(); await acknowledge(job.deploymentId); const hosts = await import('../../src/lib/server/hosts/index.js'); + const { serverKeyPool, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); const { getLibraryVersion, updateHostLibraryCaches } = await import('../../src/lib/server/game-files/index.js'); const { db } = await import('../../src/lib/server/db/index.js'); const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); @@ -1521,12 +1528,15 @@ describe('cancelled native mutation bookkeeping', () => { expect(migration.remove_operation_id).toBeTruthy(); if(delivered) hosts.pendingHostOperations(hostId); expect(cancelProvisioningJob(migration.deployment_id)).toBe(true); + expect(serverKeyPool('t5').available).toBe(0); if(delivered) hosts.reportHostOperation(hostId,migration.remove_operation_id,{state:'completed'}); await processProvisioning(); await processProvisioning(); expect((await detail(migration.deployment_id)).state).toBe('cancelled'); expect(db().prepare('SELECT * FROM ec_host_ports WHERE host_id=?').all(destination.id)).toEqual([]); expect(db().prepare('SELECT * FROM ec_host_migrations').all()).toHaveLength(1); expect(db().prepare('SELECT * FROM ec_host_ports WHERE host_id=?').all(hostId)).toHaveLength(delivered ? 0 : 2); + expect(serverKeyPool('t5').available).toBe(delivered ? 1 : 0); + expect(getServerKey(job.name)).toBe(delivered ? undefined : 'saved-key'); }); }); @@ -1547,3 +1557,73 @@ describe('additional concurrency admission', () => { expect(await operationFor(second.deploymentId)).not.toBeNull(); }); }); + +describe('game key pool launch lifecycle', () => { + it('reserves across maps atomically, rejects duplicate credentials, and reuses a cancelled launch key', async () => { + await prepareLibrary(); + const { saveServerKey, serverKeyPool, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + const { queueCreateServer, destroyServer, previewCreation } = await import('../../src/lib/server/provisioning/instances.js'); + saveServerKey('t5', 'shared-t5-key', 'Any T5 map'); + saveServerKey('t6', 'shared-t6-key'); + const input = { game: 't5' as const, gameMode: 'zombie' as const, mapSlug: 'moon', hostId, libraryId, image }; + expect(previewCreation(input).hasSavedKey).toBe(true); + const first = queueCreateServer(input); + if (!first.ok) throw new Error(first.message); + expect(first.name).toBe('t5_zombie_moon_1'); + expect(getServerKey(first.name)).toBe('shared-t5-key'); + expect(serverKeyPool('t5')).toEqual({ total: 1, available: 0 }); + expect(serverKeyPool('t6')).toEqual({ total: 1, available: 1 }); + expect(previewCreation({ ...input, mapSlug: 'theater' }).hasSavedKey).toBe(false); + expect(queueCreateServer({ ...input, mapSlug: 'theater' }).ok).toBe(false); + expect(queueCreateServer({ ...input, mapSlug: 'theater', plutoKey: 'shared-t5-key' }).ok).toBe(false); + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + closeDb(); + expect(serverKeyPool('t5').available).toBe(0); + expect(db().prepare('SELECT * FROM server_creation_jobs').all()).toHaveLength(1); + expect(destroyServer(first.name).ok).toBe(true); + expect(serverKeyPool('t5').available).toBe(1); + const second = queueCreateServer({ ...input, mapSlug: 'theater' }); + if (!second.ok) throw new Error(second.message); + expect(getServerKey(second.name)).toBe('shared-t5-key'); + expect(getServerKey(first.name)).toBeUndefined(); + }); + + it('keeps a key through stop and rebuild, then releases it only after confirmed removal', async () => { + const job = await queue(); await advance(); await acknowledge(job.deploymentId); + const { serverKeyPool, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js'); + const { destroyServer } = await import('../../src/lib/server/provisioning/instances.js'); + const stopped = requestLifecycle(job.name, 'stop'); + if (!stopped.ok) throw new Error(stopped.error); + await advance(); remote.state = 'stopped'; await acknowledge(stopped.deploymentId); + expect(serverKeyPool('t5').available).toBe(0); + const rebuilt = requestLifecycle(job.name, 'rebuild'); + if (!rebuilt.ok) throw new Error(rebuilt.error); + await advance(); remote.state = 'running'; await acknowledge(rebuilt.deploymentId); + expect(getServerKey(job.name)).toBe('saved-key'); + expect(serverKeyPool('t5').available).toBe(0); + const removal = destroyServer(job.name); + if (!removal.ok || !removal.deploymentId) throw new Error('Expected removal'); + await advance(); + expect(serverKeyPool('t5').available).toBe(0); + const operation = (await operationFor(removal.deploymentId))!; + const { reportHostOperation } = await import('../../src/lib/server/hosts/index.js'); + reportHostOperation(hostId, operation.id, { state: 'unknown', error: 'Receipt lost' }); + await advance(); + expect(serverKeyPool('t5').available).toBe(0); + await acknowledge(removal.deploymentId); + expect(serverKeyPool('t5')).toEqual({ total: 1, available: 1 }); + expect(getServerKey(job.name)).toBeUndefined(); + }); + + it('keeps the reservation for a submitted cancellation until the host confirms cleanup', async () => { + const job = await queue(); await advance(); + const { serverKeyPool } = await import('../../src/lib/server/fleet/server-keys.js'); + const { cancelCreation } = await import('../../src/lib/server/provisioning/cancel-creation.js'); + const cleanup = cancelCreation(job.deploymentId)!; + expect(cleanup).toEqual(expect.any(String)); + expect(serverKeyPool('t5').available).toBe(0); + await acknowledge(cleanup); + expect(serverKeyPool('t5')).toEqual({ total: 1, available: 1 }); + }); +}); diff --git a/tests/unit/native-registry-migration.test.ts b/tests/unit/native-registry-migration.test.ts index a5976fa..6b72a44 100644 --- a/tests/unit/native-registry-migration.test.ts +++ b/tests/unit/native-registry-migration.test.ts @@ -12,6 +12,9 @@ it('retires only obsolete backend columns while preserving instances, history, k const previous = new Database(fixture.dbPath); // Model the former required resource identity and an unrelated extension. previous.exec(schema); + previous.exec(`DROP TABLE server_keys; + CREATE TABLE server_keys(server_id TEXT PRIMARY KEY,pluto_key TEXT NOT NULL, + updated_at TEXT NOT NULL DEFAULT (datetime('now')));`); previous.exec(`ALTER TABLE servers ADD COLUMN komodo_type TEXT NOT NULL DEFAULT 'deployment'; ALTER TABLE servers ADD COLUMN komodo_name TEXT NOT NULL; ALTER TABLE servers ADD COLUMN komodo_server TEXT; diff --git a/tests/unit/runtime-image-policy.test.ts b/tests/unit/runtime-image-policy.test.ts index 9e4c23b..1b33daf 100644 --- a/tests/unit/runtime-image-policy.test.ts +++ b/tests/unit/runtime-image-policy.test.ts @@ -81,7 +81,7 @@ describe('runtime image launch policy', () => { expect(() => previewCollection(blueprint.id, 'fill', { image: custom })).toThrow(/prepared by an admin/); await cache([custom]); const preview = previewCollection(blueprint.id, 'fill', { image: custom }); - const keys = Object.fromEntries(preview.servers.map(server => [server.name!, 'saved-key'])); + const keys = Object.fromEntries(preview.servers.map(server => [server.name!, 'key-' + server.name])); await cache([]); expect(() => queueCollection({ previewId: preview.id, keys })).toThrow(/prepared by an admin/); const { db } = await import('../../src/lib/server/db/index.js'); diff --git a/tests/unit/server-key-migration.test.ts b/tests/unit/server-key-migration.test.ts new file mode 100644 index 0000000..518353a --- /dev/null +++ b/tests/unit/server-key-migration.test.ts @@ -0,0 +1,108 @@ +import Database from 'better-sqlite3'; +import { afterEach, beforeEach, expect, it, vi } from 'vitest'; +import schema from '../../src/lib/server/db/schema.sql?raw'; +import { newFixture, type Fixture } from '../helpers/fixture.js'; + +let fixture: Fixture; +beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); }); +afterEach(async () => { await fixture.cleanup(); }); + +it('migrates idle, occupied and duplicate legacy keys without releasing unresolved assignments', async () => { + const previous = new Database(fixture.dbPath); + previous.exec(schema); + previous.exec(`DROP TABLE server_keys; + CREATE TABLE server_keys(server_id TEXT PRIMARY KEY,pluto_key TEXT NOT NULL,updated_at TEXT NOT NULL DEFAULT (datetime('now')));`); + const { migrateNativeInstances } = await import('../../src/lib/server/backends/schema.js'); + migrateNativeInstances(previous); + const cases = [ + ['t5_zombie_theater_1', 'shared-occupied'], + ['t5_zombie_pentagon_1', 'shared-occupied'], + ['t5_zombie_cosmodrome_1', 'native-only'], + ['t5_zombie_temple_1', 'queued-key'], + ['t5_zombie_sumpf_1', 'failed-key'], + ['t5_zombie_moon_1', 'idle-key'], + ['t5_zombie_moon_2', 'idle-key'], + ['t6_zombie_town_1', 't6-key'] + ]; + for (const [server, credential] of cases) previous.prepare('INSERT INTO server_keys(server_id,pluto_key,updated_at) VALUES(?,?,?)').run(server, credential, '2026-01-01'); + previous.exec(`INSERT INTO servers(name,game,desired_state) VALUES('t5_zombie_theater_1','t5','running'),('t5_zombie_pentagon_1','t5','stopped'); + INSERT INTO native_instances(id,server,host_id,library_id,image,port,created_at) + VALUES('native','t5_zombie_cosmodrome_1','host','library','image',28960,'2026-01-01'); + INSERT INTO server_deployments(id,server,action,state,stage,started_at,triggered_by) VALUES + ('queued','t5_zombie_temple_1','create','queued','Queued','2026-01-01','fixture'), + ('failed','t5_zombie_sumpf_1','create','failed','Placement failed','2026-01-01','fixture');`); + previous.close(); + + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + const pool = await import('../../src/lib/server/fleet/server-keys.js'); + for (let pass = 0; pass < 2; pass++) { + expect(pool.serverKeyPool('t5')).toEqual({ total: 5, available: 1 }); + expect(pool.serverKeyPool('t6')).toEqual({ total: 1, available: 1 }); + expect(pool.getServerKey('t5_zombie_theater_1')).toBe('shared-occupied'); + expect(pool.getServerKey('t5_zombie_pentagon_1')).toBe('shared-occupied'); + expect(pool.getServerKey('t5_zombie_cosmodrome_1')).toBe('native-only'); + expect(pool.getServerKey('t5_zombie_temple_1')).toBe('queued-key'); + expect(pool.getServerKey('t5_zombie_sumpf_1')).toBe('failed-key'); + expect(pool.getServerKey('t5_zombie_moon_1')).toBeUndefined(); + expect(db().pragma('foreign_key_check')).toEqual([]); + closeDb(); + } + const duplicate = pool.listServerKeys().find(key => key.assigned_servers.length === 2)!; + expect(duplicate.assigned_servers).toEqual(['t5_zombie_pentagon_1', 't5_zombie_theater_1']); + expect(duplicate.updated_at).toBe('2026-01-01'); + expect(pool.reserveServerKey('t5_zombie_theater_2', 't5')).toBe('idle-key'); + pool.releaseServerKey('t5_zombie_theater_1'); + expect(pool.availableServerKeyCount('t5')).toBe(0); + expect(() => pool.removeServerKey(duplicate.id)).toThrow('reserved'); + pool.releaseServerKey('t5_zombie_pentagon_1'); + expect(pool.reserveServerKey('t5_zombie_theater_3', 't5')).toBe('shared-occupied'); +}); + +it('recovers actual launch credentials and holds both sides of an unresolved legacy rebuild', async () => { + const previous = new Database(fixture.dbPath); + previous.exec(schema); + previous.exec(`DROP TABLE server_keys; + CREATE TABLE server_keys(server_id TEXT PRIMARY KEY,pluto_key TEXT NOT NULL,updated_at TEXT NOT NULL DEFAULT (datetime('now')));`); + const { migrateNativeInstances } = await import('../../src/lib/server/backends/schema.js'); + const { ensureHostSchema } = await import('../../src/lib/server/hosts/schema.js'); + migrateNativeInstances(previous); + ensureHostSchema(previous); + previous.exec(`INSERT INTO ec_hosts(id,name,created_at) VALUES('host','Fixture host','2026-01-01'); + INSERT INTO servers(name,game) VALUES('t5_zombie_theater_1','t5'),('t5_zombie_pentagon_1','t5'); + INSERT INTO server_keys(server_id,pluto_key) VALUES('t5_zombie_theater_1','edited-saved-key'), + ('t5_zombie_pentagon_1','original-key'),('t5_zombie_moon_1','original-key'); + INSERT INTO native_instances(id,server,host_id,generation,library_id,image,port,created_at) VALUES + ('kino','t5_zombie_theater_1','host',2,'library','image',28960,'2026-01-01'), + ('five','t5_zombie_pentagon_1','host',1,'library','image',28961,'2026-01-01'), + ('queued','t5_zombie_cosmodrome_1','host',1,'library','image',28962,'2026-01-01');`); + function operation(id: string, instance: string, generation: number, kind: string, state: string, key: string, staged = false, nextGeneration?: number) { + const payload = JSON.stringify({ game: 't5', env: { PLUTO_SERVER_KEY: key }, nextGeneration }); + previous.prepare(`INSERT INTO ec_host_operations(id,host_id,instance_id,generation,kind,state,payload,created_at,updated_at) + VALUES(?,'host',?,?,?,?,?,'2026-01-01','2026-01-01')`).run(id, instance, generation, kind, state, staged ? '{}' : payload); + if (staged) previous.prepare('INSERT INTO ec_host_operation_payloads(operation_id,payload) VALUES(?,?)').run(id, payload); + } + operation('obsolete-kino', 'kino', 1, 'instance.create', 'completed', 'obsolete-key'); + operation('actual-kino', 'kino', 1, 'instance.rebuild', 'completed', 'actual-key', true, 2); + operation('original-five', 'five', 1, 'instance.create', 'completed', 'original-key'); + operation('unknown-five', 'five', 1, 'instance.rebuild', 'unknown', 'pending-key', true, 2); + operation('unknown-creation', 'queued', 1, 'instance.create', 'unknown', 'unknown-creation-key'); + operation('failed-new-key', 'kino', 2, 'instance.rebuild', 'failed', 'never-applied-key', true, 3); + previous.close(); + + const pool = await import('../../src/lib/server/fleet/server-keys.js'); + expect(pool.getServerKey('t5_zombie_theater_1')).toBe('actual-key'); + expect(pool.getServerKey('t5_zombie_pentagon_1')).toBe('pending-key'); + expect(pool.getServerKey('t5_zombie_cosmodrome_1')).toBe('unknown-creation-key'); + expect(pool.serverKeyPool('t5')).toEqual({ total: 5, available: 1 }); + expect(pool.serverKeyPool('t5', 't5_zombie_pentagon_1')).toEqual({ total: 5, available: 2 }); + expect(() => pool.reserveServerKey('t5_zombie_moon_2', 't5', 'original-key')).toThrow('already reserved'); + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + expect(db().prepare(`SELECT k.pluto_key,l.is_current FROM server_key_leases l JOIN server_keys k ON k.id=l.key_id + WHERE l.server_id='t5_zombie_pentagon_1' ORDER BY l.is_current`).all()) + .toEqual([{ pluto_key: 'original-key', is_current: 0 }, { pluto_key: 'pending-key', is_current: 1 }]); + expect(db().pragma('foreign_key_check')).toEqual([]); + closeDb(); + expect(pool.serverKeyPool('t5')).toEqual({ total: 5, available: 1 }); + pool.releaseServerKey('t5_zombie_pentagon_1'); + expect(pool.availableServerKeyCount('t5')).toBe(3); +}); diff --git a/tests/unit/server-key-pool.test.ts b/tests/unit/server-key-pool.test.ts new file mode 100644 index 0000000..24b7776 --- /dev/null +++ b/tests/unit/server-key-pool.test.ts @@ -0,0 +1,92 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { newFixture, type Fixture } from '../helpers/fixture.js'; + +let fixture: Fixture; +beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); }); +afterEach(async () => { await fixture.cleanup(); }); + +const kino = 't5_zombie_theater_1'; +const five = 't5_zombie_pentagon_1'; +const town = 't6_zombie_town_1'; + +async function keys() { return import('../../src/lib/server/fleet/server-keys.js'); } + +describe('per-game key pools', () => { + it('shares keys across maps only after their previous reservation is released', async () => { + const pool = await keys(); + const id = pool.saveServerKey('t5', 'secret-t5', 'Shared T5 key'); + pool.saveServerKey('t6', 'secret-t6'); + expect(pool.reserveServerKey(kino, 't5')).toBe('secret-t5'); + expect(pool.reserveServerKey(kino, 't5')).toBe('secret-t5'); + expect(pool.serverKeyPool('t5')).toEqual({ total: 1, available: 0 }); + expect(pool.serverKeyPool('t5', kino)).toEqual({ total: 1, available: 1 }); + expect(() => pool.reserveServerKey(five, 't5')).toThrow('No available T5 server keys'); + expect(() => pool.removeServerKey(id)).toThrow('reserved'); + expect(pool.reserveServerKey(town, 't6')).toBe('secret-t6'); + expect(pool.getServerKey(five)).toBeUndefined(); + expect(pool.releaseServerKey(kino)).toBe(true); + expect(pool.reserveServerKey(five, 't5')).toBe('secret-t5'); + expect(pool.listServerKeys().find(key => key.id === id)).toMatchObject({ label: 'Shared T5 key', assigned_servers: [five] }); + pool.releaseServerKey(five); + expect(pool.removeServerKey(id)).toBe(true); + expect(pool.removeServerKey(id)).toBe(false); + }); + + it('deduplicates credentials and exposes only metadata', async () => { + const pool = await keys(); + const id = pool.saveServerKey('t5', ' secret ', ' First '); + expect(pool.saveServerKey('t5', 'secret', 'Renamed')).toBe(id); + expect(pool.saveServerKey('t5', 'secret')).toBe(id); + expect(pool.serverKeyPool('t5')).toEqual({ total: 1, available: 1 }); + expect(pool.listServerKeys()).toEqual([{ id, game: 't5', label: 'Renamed', updated_at: expect.any(String), assigned_servers: [] }]); + expect(JSON.stringify(pool.listServerKeys())).not.toContain('secret'); + for (const [game, credential, label] of [['t7', 'valid', ''], ['t5', 'two words', ''], ['t5', '', ''], ['t5', 'valid', 'x'.repeat(161)]]) { + expect(() => pool.saveServerKey(game, credential, label)).toThrow(); + } + }); + + it('reserves supplied keys atomically and never substitutes another available key', async () => { + const pool = await keys(); + expect(pool.reserveServerKey(kino, 't5', 'manual-key')).toBe('manual-key'); + pool.saveServerKey('t5', 'spare-key'); + expect(() => pool.reserveServerKey(five, 't5', 'manual-key')).toThrow('already reserved'); + expect(pool.getServerKey(five)).toBeUndefined(); + expect(pool.availableServerKeyCount('t5')).toBe(1); + expect(() => pool.reserveServerKey(kino, 't5', 'replacement')).toThrow('already has a reserved key'); + expect(() => pool.reserveServerKey(kino, 't6')).toThrow('another game'); + expect(pool.reserveServerKey(five, 't5', '')).toBe('spare-key'); + }); + + it('rolls back reservation and imported credentials when launch admission fails', async () => { + const pool = await keys(); + const { db } = await import('../../src/lib/server/db/index.js'); + expect(() => db().transaction(() => { + pool.reserveServerKey(kino, 't5', 'temporary-key'); + throw new Error('Launch validation failed'); + })()).toThrow('Launch validation failed'); + expect(pool.listServerKeys()).toEqual([]); + expect(pool.getServerKey(kino)).toBeUndefined(); + }); + + it('keeps reservations across restart and missing registry rows', async () => { + const pool = await keys(); + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + pool.reserveServerKey(kino, 't5', 'retained-key'); + db().prepare("INSERT INTO servers(name,game,desired_state) VALUES(?,'t5','stopped')").run(kino); + closeDb(); + expect(pool.getServerKey(kino)).toBe('retained-key'); + expect(pool.availableServerKeyCount('t5')).toBe(0); + db().prepare('DELETE FROM servers WHERE name=?').run(kino); + expect(pool.availableServerKeyCount('t5')).toBe(0); + }); + + it('does not allocate a duplicated credential imported into another game', async () => { + const pool = await keys(); + pool.reserveServerKey(kino, 't5', 'same-secret'); + const duplicate = pool.saveServerKey('t6', 'same-secret'); + expect(pool.serverKeyPool('t6')).toEqual({ total: 1, available: 0 }); + expect(pool.listServerKeys().find(key => key.id === duplicate)?.assigned_servers).toEqual([kino]); + expect(() => pool.removeServerKey(duplicate)).toThrow('reserved'); + expect(() => pool.reserveServerKey(town, 't6')).toThrow('No available T6'); + }); +}); diff --git a/tests/unit/server-keys-api.test.ts b/tests/unit/server-keys-api.test.ts new file mode 100644 index 0000000..4c80b05 --- /dev/null +++ b/tests/unit/server-keys-api.test.ts @@ -0,0 +1,70 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { newFixture, type Fixture } from '../helpers/fixture.js'; + +let fixture: Fixture; +beforeEach(async () => { vi.resetModules(); fixture = await newFixture([]); }); +afterEach(async () => { await fixture.cleanup(); vi.unstubAllEnvs(); }); + +function request(method: string, body: unknown) { + return { request: new Request('http://localhost/api/server-keys', { + method, headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) + }) }; +} + +describe('game key pool API', () => { + it('returns pool metadata and assigned servers without credentials', async () => { + const route = await import('../../src/routes/api/server-keys/+server.js'); + const { reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + const saved = await route.PUT(request('PUT', { game: 't5', label: 'Cherry hostname 5', plutoKey: 'private-t5-credential' })); + expect(saved.status).toBe(200); + const { id } = await saved.json(); + reserveServerKey('t5_zombie_theater_2', 't5'); + const response = route.GET(); + expect(response.headers.get('cache-control')).toBe('no-store'); + const text = await response.text(); + expect(text).not.toContain('private-t5-credential'); + expect(text).not.toContain('pluto_key'); + expect(text).not.toContain('plutoKey'); + expect(JSON.parse(text)).toEqual({ keys: [{ + id, game: 't5', label: 'Cherry hostname 5', assigned_servers: ['t5_zombie_theater_2'], updated_at: expect.any(String) + }] }); + }); + + it.each([ + { game: 't4', plutoKey: 'private-key' }, + { serverId: 't5_zombie_theater_2', plutoKey: 'private-key' }, + { game: 't5', plutoKey: 'contains whitespace' }, + { game: 't5', plutoKey: 'private-key', label: 'x'.repeat(161) } + ])('rejects invalid pool input without saving it: %j', async body => { + const route = await import('../../src/routes/api/server-keys/+server.js'); + const response = await route.PUT(request('PUT', body)); + expect(response.status).toBe(400); + expect(await response.json()).toMatchObject({ ok: false, error: expect.any(String) }); + expect(await route.GET().json()).toEqual({ keys: [] }); + }); + + it('deduplicates a credential in its game pool', async () => { + const route = await import('../../src/routes/api/server-keys/+server.js'); + const first = await route.PUT(request('PUT', { game: 't6', plutoKey: 'private-key', label: 'First label' })); + const repeated = await route.PUT(request('PUT', { game: 't6', plutoKey: ' private-key ', label: 'Updated label' })); + expect(await repeated.json()).toEqual(await first.json()); + expect((await route.GET().json()).keys).toEqual([expect.objectContaining({ + game: 't6', label: 'Updated label', assigned_servers: [] + })]); + }); + + it('rejects removal while assigned and permits it after the key is released', async () => { + const route = await import('../../src/routes/api/server-keys/+server.js'); + const { reserveServerKey, releaseServerKey, getServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + const saved = await route.PUT(request('PUT', { game: 't5', plutoKey: 'private-key' })); + const { id } = await saved.json(); + reserveServerKey('t5_zombie_moon_1', 't5'); + const blocked = await route.DELETE(request('DELETE', { id })); + expect(blocked.status).toBe(400); + expect(await blocked.json()).toMatchObject({ ok: false, error: expect.stringMatching(/reserved|assigned|in use/i) }); + expect(getServerKey('t5_zombie_moon_1')).toBe('private-key'); + releaseServerKey('t5_zombie_moon_1'); + expect(await (await route.DELETE(request('DELETE', { id }))).json()).toEqual({ ok: true }); + expect(await route.GET().json()).toEqual({ keys: [] }); + }); +}); diff --git a/tests/unit/watchdog.test.ts b/tests/unit/watchdog.test.ts index 37b6240..7db461b 100644 --- a/tests/unit/watchdog.test.ts +++ b/tests/unit/watchdog.test.ts @@ -31,8 +31,8 @@ beforeEach(async () => { const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); await seedServerRegistry(); database.db().prepare('UPDATE servers SET game_port = 28960 WHERE name = ?').run(server); - const { saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); - saveServerKey(server, 'saved-key'); + const { reserveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + reserveServerKey(server, 't5', 'saved-key'); refreshHost = (await prepareNativeHost([server])).heartbeat; backend = await import('../../src/lib/server/backends/native.js'); vi.spyOn(backend, 'nativeHealth').mockResolvedValue({ ok: true, state: 'running' }); @@ -504,8 +504,8 @@ describe('Watchdog APIs', () => { describe('Watchdog drift campaigns', () => { it('keeps a missing rebuild key visible as attention without deleting files or retrying automatically', async () => { - const { removeServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); - removeServerKey(server); + // Simulate legacy registry damage; administrators cannot remove a leased key. + database.db().prepare('DELETE FROM server_key_leases WHERE server_id=?').run(server); const path = join(fixture.t5, server, 'runtime/plutonium/storage/t5/raw/scripts/sp', 'stray.gsc'); await writeFile(path, 'unexpected'); await watchdog.watchdogTick(true);