From 0b0e5b5bdbced29e3e05a362aa4a7f5c18060b29 Mon Sep 17 00:00:00 2001 From: Josh Wright Date: Sun, 27 Sep 2026 13:36:08 -0400 Subject: [PATCH] feat: match fleet to blueprint targets exactly Add an optional exact blueprint mode with keep/create/remove previews, atomic queueing, cancellation of unstarted excess creations, and persistent dependencies for safe port reuse. Track removals alongside creations and preserve fill mode as the default. Validation: type check, production build, and 103 deployment tests passed. The full suite has one telemetry assertion failure also reproduced on the previous commit. Fixes #42 --- README.md | 6 + docs/deployment-blueprints.md | 54 +++- src/lib/deployment-plan.ts | 9 +- src/lib/server/db/index.ts | 8 + src/lib/server/db/schema.sql | 21 +- src/lib/server/fleet/collections.ts | 112 +++++++-- src/lib/server/fleet/create.ts | 43 +++- src/lib/server/fleet/deployments.ts | 4 +- .../components/DeployCollectionDialog.svelte | 53 +++- .../deployment-collections/preview/+server.ts | 2 +- src/routes/deployments/+page.svelte | 10 +- .../deployments/collections/[id]/+page.svelte | 47 ++-- tests/unit/deployment-collections.test.ts | 231 ++++++++++++++++++ 13 files changed, 517 insertions(+), 83 deletions(-) diff --git a/README.md b/README.md index d07e777..ba09ff3 100644 --- a/README.md +++ b/README.md @@ -147,6 +147,12 @@ multiple T5/T6 maps and set the desired total lobbies for each. **Conform fleet* counts existing and queued servers and creates only the shortfall. The preview shows target/existing/missing counts and IDs, ports, and key requirements for new servers only. Already-satisfied targets create nothing; excess servers are kept. +Choose **Match blueprint exactly** in the preview to also remove excess lobbies +and maps outside the blueprint. Review the keep/create/remove lists and confirm +removals. Matching servers stay untouched; unstarted excess creations are cancelled, +and running servers wait for a fresh zero-player check before removal. New lobbies +can reserve ports from these removals and wait until they are released. The collection +page tracks both actions. **Fill missing servers** remains the default. All jobs and capacity reservations are committed together before provisioning; one collection page tracks each server's independent progress and failures. New servers start one at a time by default across all collections and individual diff --git a/docs/deployment-blueprints.md b/docs/deployment-blueprints.md index 25c889a..2777ffe 100644 --- a/docs/deployment-blueprints.md +++ b/docs/deployment-blueprints.md @@ -17,20 +17,53 @@ they appear in both the server registry and active creation queue. Registered servers count even if stopped or unhealthy; conforming does not restart or reconfigure them. Failed creation jobs without a registered server, -destroyed records, and servers awaiting removal do not count. Excess servers and -maps outside the blueprint are kept. This action fills deficits; it does not -destroy servers or change existing servers' files. +destroyed records, and servers awaiting removal do not count. The default +**Fill missing servers** mode keeps excess servers and maps outside the blueprint. +It fills deficits without changing existing servers' files. ## Build and launch a collection 1. Choose **Create blueprint** on Deployments and name the lineup. 2. Add maps from Black Ops I or II and set the desired total lobbies for each. -3. Choose **Save & preview** to review every new server ID and game port. +3. Choose **Save & preview** (or **Preview fleet changes** on a saved blueprint). + Choose **Fill missing servers** or **Match blueprint exactly**, then review + the server IDs and game ports. 4. Lobbies with a saved key show **Saved key ready** and reuse it automatically. Enter a Plutonium key only for lobbies missing one. Update saved keys in **Keys**. -5. Choose **Queue N servers** and follow their collection progress page. +5. Choose **Queue N servers** (or confirm removals and choose **Apply** in exact + mode) and follow the collection progress page. If every target is already met, choose **Done**; no operation is created. +### Match blueprint exactly + +Exact mode applies to the entire registered and queued fleet across both games. +The preview lists servers to keep, create, and remove. It keeps deployed lobbies +before unstarted creations, then the lowest lobby numbers within each game/map +variant. It removes excess lobbies and maps outside the blueprint. Retained +servers keep their containers, files, ports, and configuration without a restart. +The option is chosen per launch; opening another preview defaults to fill mode. + +Review the removal list and check its confirmation before applying. Removed +servers lose their containers, files, and registrations; saved keys remain. +Unstarted creations are cancelled without booting a container. Running servers +use the normal removal queue and require a fresh zero-player RCON response. +Occupied or unreachable servers wait. Already pending removals are reused. + +Creation reservations and removals are queued in one database transaction. A +missing key or changed fleet rejects the entire request. Any fleet identity, +map, port, or pending-removal change since an exact preview requires a new review. +Servers added after submission are not included. Applying the same preview again +returns the same collection without repeating operations. + +Ports belonging to reviewed removals can be reserved for new lobbies even when +the current port range is full. A new lobby waits until its port's removal has +completed; a failed, interrupted, or cancelled removal keeps that launch blocked, +with the reason shown in its progress. Other launches with available capacity can +proceed. Reservations and dependencies survive app restarts. The progress page +tracks creations and removals, including removal-only plans, and lists the servers +that were retained. The existing operation detail page offers removal cancellation +and cleanup recovery where applicable. + Blueprints support up to 40 distinct game/map selections, 1–20 lobbies per map, and 100 servers total. The host's configured port range and existing fleet may allow fewer. A blueprint can be saved even when there is insufficient capacity @@ -48,7 +81,8 @@ A collection preview does not reserve capacity. At submission, all lobby IDs, ports, and keys are validated again. The entire collection is then queued in one database transaction. If any server lacks a valid key or capacity has changed, no jobs, reservations, or key changes from that submission are committed. Refreshing -the preview clears typed keys so they cannot be applied to changed server IDs. +the preview retains typed keys only for unchanged server IDs still missing a saved +key, and resets removal confirmation. Submitting the same preview again returns the same collection, including after a lost response or an application restart. Reusing the blueprint with a new @@ -174,15 +208,17 @@ repeats container creation or deployment as part of this recovery. - `PUT /api/blueprints/:id`: edit `{ name, description?, entries }`. All entries use the latest promoted setup when launched. - `DELETE /api/blueprints/:id`: remove a library entry. -- `POST /api/deployment-collections/preview`: preview `{ blueprintId }`, including - the configured `launchConcurrency`. +- `POST /api/deployment-collections/preview`: preview `{ blueprintId, mode? }`, + where mode is `fill` (default) or `exact`. Returns `mode`, `retainedServers`, + `removals` (name/reason), creation `servers`, and configured `launchConcurrency`. - `POST /api/deployment-collections`: atomically queue `{ previewId, keys? }`, where `keys` maps exact preview server IDs to optional replacement keys. A satisfied preview returns `{ ok: true, collectionId: null }` with HTTP 200, without creating a collection, saving keys, or kicking the worker. - `GET /api/deployment-collections`: list collection progress summaries with separate `queued`, `provisioning`, and `verifying` counts. -- `GET /api/deployment-collections/:id`: collection and per-server progress, +- `GET /api/deployment-collections/:id`: collection mode, retained server IDs, + per-operation progress with `action: create | destroy`, each waiting server's global `queuePosition`, and the shared queue's concurrency, active startups, and queued count. - `POST /api/deployments/preview`: preview `{ game, mapSlug }` or `{ deploymentId }` diff --git a/src/lib/deployment-plan.ts b/src/lib/deployment-plan.ts index c9c6572..c8efba8 100644 --- a/src/lib/deployment-plan.ts +++ b/src/lib/deployment-plan.ts @@ -32,11 +32,16 @@ export interface BlueprintSummary { totalServers: number; } +export type CollectionMode = 'fill' | 'exact'; + export interface CollectionPreview { id: string; expiresAt: string; blueprintName: string; launchConcurrency: number; + mode: CollectionMode; + retainedServers: string[]; + removals: { name: string; reason: string }[]; entries: { count: number; existing: number; missing: number; existingServers: string[]; coverage: { name: string; health: string; href: string }[]; recipe: LaunchRecipeSummary }[]; servers: { entryIndex: number; name: string | null; port: number | null; hasSavedKey: boolean }[]; @@ -55,12 +60,14 @@ export interface CollectionSummary { queued: number; provisioning: number; verifying: number; + mode: CollectionMode; } export interface CollectionDetail { collection: CollectionSummary; + retainedServers: string[]; queue: { concurrency: number; active: { id: string; server: string; state: string; stage: string }[]; queued: number }; - servers: { id: string; server: string; port: number; map_slug: string; state: string; stage: string; + servers: { id: string; server: string; action: 'create' | 'destroy'; port: number | null; map_slug: string | null; state: string; stage: string; started_at: string; finished_at: string | null; error: string | null; server_exists: number; queuePosition: number | null }[]; } diff --git a/src/lib/server/db/index.ts b/src/lib/server/db/index.ts index bc600a0..0fbe36b 100644 --- a/src/lib/server/db/index.ts +++ b/src/lib/server/db/index.ts @@ -249,6 +249,14 @@ export function db(): Database.Database { migrateTable(handle, 'beta_scripts', BETA_SCRIPTS_COLUMNS); migrateTable(handle, 'deployment_scripts', DEPLOYMENT_SCRIPTS_COLUMNS); migrateTable(handle, 'deployments', DEPLOYMENTS_COLUMNS); + migrateTable(handle, 'server_launch_collections', [ + { name: 'mode', ddl: "ALTER TABLE server_launch_collections ADD COLUMN mode TEXT NOT NULL DEFAULT 'fill'" }, + { name: 'retained_servers', ddl: "ALTER TABLE server_launch_collections ADD COLUMN retained_servers TEXT NOT NULL DEFAULT '[]'" } + ]); + migrateTable(handle, 'server_collection_previews', [ + { name: 'mode', ddl: "ALTER TABLE server_collection_previews ADD COLUMN mode TEXT NOT NULL DEFAULT 'fill'" }, + { name: 'fleet_snapshot', ddl: 'ALTER TABLE server_collection_previews ADD COLUMN fleet_snapshot TEXT' } + ]); migrateTable(handle, 'server_telemetry', [{ name: 'payload', ddl: 'ALTER TABLE server_telemetry ADD COLUMN payload TEXT' }]); // Kind-referencing indexes: safe here because migrateTable above just // ALTER'd the column into existence if it was missing. diff --git a/src/lib/server/db/schema.sql b/src/lib/server/db/schema.sql index 7743733..86351e7 100644 --- a/src/lib/server/db/schema.sql +++ b/src/lib/server/db/schema.sql @@ -467,7 +467,9 @@ CREATE TABLE IF NOT EXISTS server_launch_recipes ( CREATE TABLE IF NOT EXISTS server_launch_collections ( id TEXT PRIMARY KEY, name TEXT NOT NULL, - created_at TEXT NOT NULL + created_at TEXT NOT NULL, + mode TEXT NOT NULL DEFAULT 'fill', + retained_servers TEXT NOT NULL DEFAULT '[]' ); CREATE TABLE IF NOT EXISTS server_collection_members ( @@ -483,9 +485,26 @@ CREATE TABLE IF NOT EXISTS server_collection_previews ( blueprint_name TEXT NOT NULL, entries TEXT NOT NULL, servers TEXT NOT NULL, + mode TEXT NOT NULL DEFAULT 'fill', + fleet_snapshot TEXT, collection_id TEXT REFERENCES server_launch_collections(id) ON DELETE CASCADE ); +-- A pending removal can be shared by multiple reviewed collection requests. +CREATE TABLE IF NOT EXISTS server_collection_removals ( + collection_id TEXT NOT NULL REFERENCES server_launch_collections(id) ON DELETE CASCADE, + deployment_id TEXT NOT NULL REFERENCES server_deployments(id) ON DELETE CASCADE, + position INTEGER NOT NULL, + PRIMARY KEY (collection_id, deployment_id) +); + +-- Reusing a port is safe only after the reviewed removal has completed. +CREATE TABLE IF NOT EXISTS server_creation_dependencies ( + deployment_id TEXT NOT NULL REFERENCES server_deployments(id) ON DELETE CASCADE, + removal_id TEXT NOT NULL REFERENCES server_deployments(id) ON DELETE CASCADE, + PRIMARY KEY (deployment_id, removal_id) +); + -- Persist state transitions atomically, including changes made by background workers. -- Polls with unchanged status/error do not produce duplicate activity entries. CREATE TABLE IF NOT EXISTS beta_events ( diff --git a/src/lib/server/fleet/collections.ts b/src/lib/server/fleet/collections.ts index 93331ae..5801fe1 100644 --- a/src/lib/server/fleet/collections.ts +++ b/src/lib/server/fleet/collections.ts @@ -1,9 +1,11 @@ import { randomUUID } from 'node:crypto'; -import type { CollectionDetail, CollectionPreview, CollectionSummary } from '$lib/deployment-plan.js'; +import type { CollectionDetail, CollectionMode, CollectionPreview, CollectionSummary } from '$lib/deployment-plan.js'; import { db } from '../db/index.js'; import { config, komodoConfigured, rconConfigured } from '../config.js'; import { blueprintEntries, summarizeRecipe, type BlueprintEntry } from './launch-recipes.js'; -import { creationQueue, nextAvailablePort, nextLobbyNumber, queueCreateServer } from './create.js'; +import { creationQueue, destroyServer, nextAvailablePort, nextLobbyNumber, queueCreateServer } from './create.js'; +import { deploymentStage, finishDeployment, recordDeploymentEvent } from './deployments.js'; +import { housePartyTargets } from './house-party.js'; import { getServerKey } from './server-keys.js'; import { classifyHealth } from './health.js'; import { getServer } from './servers.js'; @@ -43,7 +45,29 @@ function coverage(entries: BlueprintEntry[]) { }); } -function allocate(entries: BlueprintEntry[]): CollectionPreview['servers'] { +function fleetSnapshot() { + return housePartyTargets().map(name => ({ name, + server: db().prepare('SELECT game, map_slug, map, game_port FROM servers WHERE name = ?').get(name) ?? null, + creation: db().prepare("SELECT id FROM server_deployments WHERE server = ? AND action = 'create' ORDER BY rowid DESC LIMIT 1").get(name) ?? null, + removal: db().prepare("SELECT id FROM server_deployments WHERE server = ? AND action = 'destroy' AND state IN ('waiting', 'running')").get(name) ?? null + })); +} + +function collectionPlan(entries: BlueprintEntry[], mode: CollectionMode) { + const counts = coverage(entries); + // Preserve deployed lobbies before queued ones, then the lowest lobby IDs. + const retainedServers = counts.flatMap((entry, index) => [...entry.existingServers].sort((a, b) => + Number(!!getServer(b)) - Number(!!getServer(a)) || a.localeCompare(b, 'en', { numeric: true }) + ).slice(0, mode === 'exact' ? entries[index].count : undefined)); + const retained = new Set(retainedServers); + const matching = new Set(counts.flatMap(entry => entry.existingServers)); + const removals = mode === 'exact' ? housePartyTargets().filter(name => !retained.has(name)).map(name => ({ name, + reason: matching.has(name) ? 'Exceeds the target lobby count' : 'Outside this blueprint or already awaiting removal' + })) : []; + return { counts, retainedServers, removals }; +} + +function allocate(entries: BlueprintEntry[], releasingServers: string[] = []): CollectionPreview['servers'] { const names: string[] = []; const ports: number[] = []; const counts = coverage(entries); @@ -51,7 +75,7 @@ function allocate(entries: BlueprintEntry[]): CollectionPreview['servers'] { const base = `${entry.recipe.game}_zombie_${entry.recipe.mapSlug}`; const lobby = nextLobbyNumber(base, names); const name = lobby === null ? null : `${base}_${lobby}`; - const port = nextAvailablePort(ports); + const port = nextAvailablePort(ports, releasingServers); if (name) names.push(name); if (port !== null) ports.push(port); return { entryIndex, name, port, hasSavedKey: !!name && !!getServerKey(name) }; @@ -68,19 +92,21 @@ function blockersFor(servers: CollectionPreview['servers']): string[] { return blockers; } -export function previewCollection(blueprintId: unknown): CollectionPreview { +export function previewCollection(blueprintId: unknown, mode: unknown = 'fill'): CollectionPreview { if (typeof blueprintId !== 'string' || !blueprintId) throw new Error('Choose a blueprint to launch.'); - return db().transaction(() => { + if (mode !== 'fill' && mode !== 'exact') throw new Error('Choose fill or exact blueprint mode.'); + return db().transaction((): CollectionPreview => { const { entries, blueprintName } = blueprintEntries(blueprintId); - const servers = allocate(entries); - const counts = coverage(entries); + const { counts, retainedServers, removals } = collectionPlan(entries, mode); + const servers = allocate(entries, removals.map(server => server.name)); const blockers = blockersFor(servers); + if (removals.length && (!komodoConfigured() || !rconConfigured())) blockers.push('Configure Komodo and RCON before removing servers.'); const id = randomUUID(); const expiresAt = new Date(Date.now() + 30 * 60_000).toISOString(); db().prepare('DELETE FROM server_collection_previews WHERE expires_at < ? AND collection_id IS NULL').run(new Date().toISOString()); - db().prepare('INSERT INTO server_collection_previews (id, expires_at, blueprint_name, entries, servers) VALUES (?, ?, ?, ?, ?)') - .run(id, expiresAt, blueprintName, JSON.stringify(entries), JSON.stringify(servers)); - return { id, expiresAt, blueprintName, launchConcurrency: config.createConcurrency, servers, blockers, ready: blockers.length === 0, + db().prepare('INSERT INTO server_collection_previews (id, expires_at, blueprint_name, entries, servers, mode, fleet_snapshot) VALUES (?, ?, ?, ?, ?, ?, ?)') + .run(id, expiresAt, blueprintName, JSON.stringify(entries), JSON.stringify(servers), mode, mode === 'exact' ? JSON.stringify(fleetSnapshot()) : null); + return { id, expiresAt, blueprintName, mode, retainedServers, removals, launchConcurrency: config.createConcurrency, servers, blockers, ready: blockers.length === 0, entries: entries.map((entry, index) => ({ count: entry.count, ...counts[index], recipe: summarizeRecipe(entry.recipe) })) }; })(); } @@ -92,15 +118,21 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }): const keys = (input.keys ?? {}) as Record; return db().transaction(() => { const preview = db().prepare('SELECT * FROM server_collection_previews WHERE id = ?').get(input.previewId) as { - id: string; expires_at: string; blueprint_name: string; entries: string; servers: string; collection_id: string | null; + id: string; expires_at: string; blueprint_name: string; entries: string; servers: string; collection_id: string | null; mode: CollectionMode; fleet_snapshot: string | null; } | undefined; if (!preview) throw new Error('Collection preview expired or was not found. Refresh the preview.'); if (preview.collection_id) return { ok: true as const, collectionId: preview.collection_id }; if (preview.expires_at <= new Date().toISOString()) throw new Error('Collection preview expired. Refresh it before launching.'); const entries: BlueprintEntry[] = JSON.parse(preview.entries); const reviewed: CollectionPreview['servers'] = JSON.parse(preview.servers); - const servers = allocate(entries); + if (preview.mode === 'exact' && preview.fleet_snapshot !== JSON.stringify(fleetSnapshot())) { + throw new Error('The fleet changed. Nothing was queued. Refresh the preview and review the servers to keep and remove.'); + } + const { retainedServers, removals } = collectionPlan(entries, preview.mode); + const releasingServers = removals.map(server => server.name); + const servers = allocate(entries, releasingServers); const blockers = blockersFor(servers); + if (removals.length && (!komodoConfigured() || !rconConfigured())) blockers.push('Configure Komodo and RCON before removing servers.'); if (blockers.length) throw new Error(blockers.join(' ')); if (servers.length !== reviewed.length || servers.some((server, i) => server.entryIndex !== reviewed[i]?.entryIndex || server.name !== reviewed[i]?.name || server.port !== reviewed[i]?.port)) { throw new Error('Fleet counts, lobby or port availability changed. Nothing was queued. Refresh the preview and review each server key.'); @@ -109,36 +141,65 @@ export function queueCollection(input: { previewId?: unknown; keys?: unknown }): for (const [name, value] of Object.entries(keys)) { if (!expected.has(name) || typeof value !== 'string') throw new Error('Keys must be strings for server IDs shown in this preview.'); } - if (!servers.length) return { ok: true as const, collectionId: null }; + if (!servers.length && !removals.length) return { ok: true as const, collectionId: null }; const collectionId = randomUUID(); - db().prepare('INSERT INTO server_launch_collections (id, name, created_at) VALUES (?, ?, ?)') - .run(collectionId, preview.blueprint_name, new Date().toISOString()); + db().prepare('INSERT INTO server_launch_collections (id, name, created_at, mode, retained_servers) VALUES (?, ?, ?, ?, ?)') + .run(collectionId, preview.blueprint_name, new Date().toISOString(), preview.mode, JSON.stringify(retainedServers)); + const removalJobs = removals.map((server, position) => { + const ports = (db().prepare(`SELECT game_port AS port FROM servers WHERE name = ? UNION + SELECT j.port FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id + WHERE j.server = ? AND d.state IN ('queued', 'running', 'awaiting-health')`).all(server.name, server.name) as { port: number | null }[]).map(row => row.port); + const queued = db().prepare("SELECT id FROM server_deployments WHERE server = ? AND action = 'create' AND state = 'queued'").get(server.name) as { id: string } | undefined; + let id: string; + if (queued && !getServer(server.name)) { + // No container or files exist yet: cancel the reservation instead of booting just to remove it. + const result = destroyServer(server.name, 'blueprint-exact'); + if (!result.ok || !result.deploymentId) throw new Error('Could not queue removal for ' + server.name); + id = result.deploymentId; + db().prepare("UPDATE server_deployments SET state = 'cancelled', stage = 'Cancelled by exact blueprint', finished_at = ? WHERE id = ?").run(new Date().toISOString(), queued.id); + recordDeploymentEvent(queued.id, 'Queued creation cancelled by exact blueprint; saved key retained.'); + deploymentStage(id, 'Cancelled queued creation; no server was started'); + finishDeployment(id, { ok: true }); + } else { + const result = destroyServer(server.name, 'blueprint-exact'); + if (!result.ok || !result.deploymentId) throw new Error('Could not queue removal for ' + server.name); + id = result.deploymentId; + } + db().prepare('INSERT INTO server_collection_removals (collection_id, deployment_id, position) VALUES (?, ?, ?)').run(collectionId, id, position); + return { id, ports }; + }); servers.forEach((server, index) => { const recipe = entries[server.entryIndex].recipe; const childPreviewId = randomUUID(); db().prepare('INSERT INTO server_launch_previews (id, expires_at, server, port, recipe, blueprint_name) VALUES (?, ?, ?, ?, ?, ?)') .run(childPreviewId, preview.expires_at, server.name, server.port, JSON.stringify(recipe), preview.blueprint_name); const result = queueCreateServer({ game: recipe.game, gameMode: 'zombie', mapSlug: recipe.mapSlug, - expectedName: server.name!, previewId: childPreviewId, plutoKey: keys[server.name!] as string | undefined, createdBy: 'ui' }); + expectedName: server.name!, previewId: childPreviewId, plutoKey: keys[server.name!] as string | undefined, createdBy: 'ui' }, releasingServers); // Throw so the outer transaction also reverses all earlier keys and children. if (!result.ok) throw new Error(`${server.name}: ${result.message} Nothing was queued.`); db().prepare('INSERT INTO server_collection_members (collection_id, deployment_id, position) VALUES (?, ?, ?)') .run(collectionId, result.deploymentId, index); + for (const removal of removalJobs.filter(removal => removal.ports.includes(server.port))) { + db().prepare('INSERT INTO server_creation_dependencies (deployment_id, removal_id) VALUES (?, ?)').run(result.deploymentId, removal.id); + } }); db().prepare('UPDATE server_collection_previews SET collection_id = ? WHERE id = ?').run(collectionId, preview.id); return { ok: true as const, collectionId }; })(); } -const summarySql = `SELECT c.id, c.name, c.created_at, COUNT(d.id) AS total, +const membershipSql = `SELECT collection_id, deployment_id, position FROM server_collection_members + UNION ALL SELECT collection_id, deployment_id, position FROM server_collection_removals`; + +const summarySql = `SELECT c.id, c.name, c.created_at, c.mode, COUNT(d.id) AS total, COALESCE(SUM(d.state IN ('done', 'warning')), 0) AS done, - COALESCE(SUM(d.state IN ('failed', 'interrupted')), 0) AS failed, + COALESCE(SUM(d.state IN ('failed', 'interrupted', 'cancelled')), 0) AS failed, COALESCE(SUM(d.state IN ('queued', 'running', 'awaiting-health', 'waiting')), 0) AS active, - COALESCE(SUM(d.state = 'queued'), 0) AS queued, + COALESCE(SUM(d.state IN ('queued', 'waiting')), 0) AS queued, COALESCE(SUM(d.state = 'running'), 0) AS provisioning, COALESCE(SUM(d.state = 'awaiting-health'), 0) AS verifying FROM server_launch_collections c - LEFT JOIN server_collection_members m ON m.collection_id = c.id + LEFT JOIN (${membershipSql}) m ON m.collection_id = c.id LEFT JOIN server_deployments d ON d.id = m.deployment_id`; export function listCollections(): CollectionSummary[] { @@ -148,12 +209,13 @@ export function listCollections(): CollectionSummary[] { export function getCollection(id: string): CollectionDetail | null { const collection = db().prepare(summarySql + ' WHERE c.id = ? GROUP BY c.id').get(id) as CollectionSummary | undefined; if (!collection) return null; - const servers = db().prepare(`SELECT d.id, d.server, d.state, d.stage, d.started_at, d.finished_at, d.error, j.port, j.map_slug, + const servers = db().prepare(`SELECT d.id, d.server, d.action, d.state, d.stage, d.started_at, d.finished_at, d.error, j.port, j.map_slug, EXISTS(SELECT 1 FROM servers s WHERE s.name = d.server) AS server_exists - FROM server_collection_members m JOIN server_deployments d ON d.id = m.deployment_id - JOIN server_creation_jobs j ON j.deployment_id = d.id WHERE m.collection_id = ? ORDER BY m.position`).all(id) as CollectionDetail['servers']; + FROM (${membershipSql}) m JOIN server_deployments d ON d.id = m.deployment_id + LEFT JOIN server_creation_jobs j ON j.deployment_id = d.id WHERE m.collection_id = ? ORDER BY d.action DESC, m.position`).all(id) as CollectionDetail['servers']; const queue = creationQueue(); const positions = new Map(queue.queued.map((job, index) => [job.id, index + 1])); - return { collection, queue: { concurrency: config.createConcurrency, active: queue.active, queued: queue.queued.length }, + const stored = db().prepare('SELECT retained_servers FROM server_launch_collections WHERE id = ?').get(id) as { retained_servers: string }; + return { collection, retainedServers: JSON.parse(stored.retained_servers), queue: { concurrency: config.createConcurrency, active: queue.active, queued: queue.queued.length }, servers: servers.map(server => ({ ...server, queuePosition: positions.get(server.id) ?? null })) }; } diff --git a/src/lib/server/fleet/create.ts b/src/lib/server/fleet/create.ts index b3857db..a05be56 100644 --- a/src/lib/server/fleet/create.ts +++ b/src/lib/server/fleet/create.ts @@ -83,13 +83,13 @@ export function nextLobbyNumber(baseName: string, additionalNames: string[] = [] * from the DB (which EC controls end-to-end under the new naming scheme, so * the DB is authoritative — no need to ask Komodo). */ -export function nextAvailablePort(additionalPorts: number[] = []): number | null { +export function nextAvailablePort(additionalPorts: number[] = [], releasingServers: string[] = []): number | null { const rows = db() - .prepare(`SELECT game_port FROM servers WHERE game_port IS NOT NULL UNION - SELECT j.port AS game_port FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id + .prepare(`SELECT name, game_port FROM servers WHERE game_port IS NOT NULL UNION + SELECT j.server AS name, j.port AS game_port FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id WHERE d.state IN ('queued', 'running', 'awaiting-health')`) - .all() as Array<{ game_port: number }>; - const taken = new Set([...rows.map((r) => r.game_port), ...additionalPorts]); + .all() as Array<{ name: string; game_port: number }>; + const taken = new Set([...rows.filter(row => !releasingServers.includes(row.name)).map((r) => r.game_port), ...additionalPorts]); for (let p = config.portRange.min; p <= config.portRange.max; p++) { if (!taken.has(p)) return p; } @@ -195,7 +195,8 @@ export function previewCreation(input: { game?: unknown; mapSlug?: unknown; blue checks, ready: checks.every(check => check.state !== 'blocked') }; } -export function queueCreateServer(input: CreateServerInput): CreateServerResult { +// releasingServers is internal to atomic collection queueing; HTTP callers cannot set it. +export function queueCreateServer(input: CreateServerInput, releasingServers: string[] = []): CreateServerResult { return db().transaction((): CreateServerResult => { const preview = input.previewId ? db().prepare('SELECT * FROM server_launch_previews WHERE id = ?').get(input.previewId) as PreviewRow | undefined : undefined; if (input.previewId && !preview) return { ok: false, kind: 'bad-request', message: 'Launch preview expired or was not found. Review a fresh preview.' }; @@ -206,13 +207,13 @@ export function queueCreateServer(input: CreateServerInput): CreateServerResult // A repeated submit after a lost response returns the same operation. if (preview?.deployment_id) return { ok: true, name: preview.server!, port: preview.port!, deploymentId: preview.deployment_id }; if (preview && preview.expires_at <= new Date().toISOString()) return { ok: false, kind: 'bad-request', message: 'Launch preview expired. Review a fresh preview.' }; - if (preview && (!preview.server || !preview.port || preview.server !== nextName(input) || preview.port !== nextAvailablePort())) { + if (preview && (!preview.server || !preview.port || preview.server !== nextName(input) || preview.port !== nextAvailablePort([], releasingServers))) { return { ok: false, kind: 'no-slot', message: 'Available lobby or port changed. Refresh the preview and review the new server key.' }; } if (recipe && !config.rcon.password) { return { ok: false, kind: 'unconfigured', message: 'Health-check settings changed. Refresh the preview.' }; } - const prepared = prepareCreation(input); + const prepared = prepareCreation(input, releasingServers); if (!prepared.ok) return prepared; // A preview records choices, not version pins. Promotions remain authoritative. const currentRecipe = captureRecipe(input.game ?? 't5', prepared.mapEntry.slug); @@ -247,7 +248,7 @@ export async function createServer(input: CreateServerInput): Promise { +function prepareCreation(input: CreateServerInput, releasingServers: string[] = []): PreparedCreation | Extract { const game = input.game ?? 't5'; if (game !== 't5' && game !== 't6') { return { ok: false, kind: 'bad-request', message: 'Supported games are t5 and t6' }; @@ -279,7 +280,7 @@ function prepareCreation(input: CreateServerInput): PreparedCreation | Extract job.state !== 'queued'), queued: jobs.filter(job => job.state === 'queued') }; } +function creationBlocker(id: string): string | null { + const dependency = db().prepare(`SELECT d.server, d.state FROM server_creation_dependencies dep + JOIN server_deployments d ON d.id = dep.removal_id + WHERE dep.deployment_id = ? AND d.state NOT IN ('done', 'warning') LIMIT 1`).get(id) as { server: string; state: string } | undefined; + if (dependency) return `Waiting for removal of ${dependency.server} (${dependency.state}) before reusing its port`; + const occupied = db().prepare(`SELECT s.name FROM servers s JOIN server_creation_jobs j ON s.game_port = j.port + WHERE j.deployment_id = ? LIMIT 1`).get(id) as { name: string } | undefined; + return occupied ? `Waiting for port to be released by ${occupied.name}` : null; +} + async function runCreations(): Promise { const outcomes: CreationOutcomes = new Map(); if (creating || destroying.size > 0) return outcomes; @@ -696,12 +707,22 @@ async function runCreations(): Promise { // worker pass into an unbounded stream of expensive container startups. const queue = creationQueue(); const available = Math.max(0, config.createConcurrency - queue.active.length); - for (const { id } of queue.queued.slice(0, available)) { + const eligible = queue.queued.filter(({ id }) => { + const blocker = creationBlocker(id); + if (blocker) deploymentStage(id, blocker); + return !blocker; + }); + for (const { id } of eligible.slice(0, available)) { if (creating || destroying.size > 0) break; // API calls can cancel a selected job or resume an older startup while // a previous provision awaits Komodo. Recheck without expanding this pass. if (creationQueue().active.length >= config.createConcurrency) break; if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND action = 'create' AND state = 'queued'").get(id)) continue; + const blocker = creationBlocker(id); + if (blocker) { + deploymentStage(id, blocker); + continue; + } try { outcomes.set(id, await provisionCreation(id)); } catch (err) { const failure = err instanceof Error ? err : new Error(String(err)); diff --git a/src/lib/server/fleet/deployments.ts b/src/lib/server/fleet/deployments.ts index 3f19825..a0131b3 100644 --- a/src/lib/server/fleet/deployments.ts +++ b/src/lib/server/fleet/deployments.ts @@ -27,7 +27,9 @@ export function getDeployment(id: string) { if (!deployment) return null; const events = db().prepare('SELECT id, created_at, message FROM server_deployment_events WHERE deployment_id = ? ORDER BY id').all(id); const launch = recipeForDeployment(id); - const collection = db().prepare(`SELECT c.id, c.name FROM server_collection_members m + const collection = db().prepare(`SELECT c.id, c.name FROM ( + SELECT collection_id, deployment_id FROM server_collection_members + UNION ALL SELECT collection_id, deployment_id FROM server_collection_removals) m JOIN server_launch_collections c ON c.id = m.collection_id WHERE m.deployment_id = ?`).get(id) ?? null; return { deployment, events, recipe: launch ? summarizeRecipe(launch.recipe) : null, blueprintName: launch?.blueprintName ?? null, canResumeVerification: canResumeVerification(id), canResumeCleanup: canResumeCleanup(id), canCancelRemoval: canCancelRemoval(id), collection }; diff --git a/src/lib/ui/components/DeployCollectionDialog.svelte b/src/lib/ui/components/DeployCollectionDialog.svelte index ca19aaa..fefa32d 100644 --- a/src/lib/ui/components/DeployCollectionDialog.svelte +++ b/src/lib/ui/components/DeployCollectionDialog.svelte @@ -6,11 +6,13 @@ import Button from './primitives/Button.svelte'; import LaunchRecipe from './LaunchRecipe.svelte'; import ServerIdentity from './ServerIdentity.svelte'; - import type { CollectionPreview } from '$lib/deployment-plan.js'; + import type { CollectionMode, CollectionPreview } from '$lib/deployment-plan.js'; let { open = $bindable(false), blueprintId = '' }: { open?: boolean; blueprintId?: string } = $props(); let dialog: HTMLDialogElement | undefined = $state(); let preview = $state(null); + let mode = $state('fill'); + let confirmRemovals = $state(false); let keys = $state>({}); let busy = $state(false); let error = $state(''); @@ -21,20 +23,21 @@ const expired = $derived(!!preview && new Date(preview.expiresAt).getTime() <= now); const keysReady = $derived(preview?.servers.filter(server => server.name && (server.hasSavedKey || keys[server.name]?.trim())).length ?? 0); onMount(() => { const timer = setInterval(() => { now = Date.now(); }, 10_000); return () => clearInterval(timer); }); - let canLaunch = $derived(preview?.ready && preview.servers.length > 0 && preview.servers.every(server => server.name && (server.hasSavedKey || keys[server.name]?.trim()))); + let canLaunch = $derived(preview?.ready && preview.mode === mode && (preview.servers.length > 0 || preview.removals.length > 0) + && (!preview.removals.length || confirmRemovals) && preview.servers.every(server => server.name && (server.hasSavedKey || keys[server.name]?.trim()))); $effect(() => { if (!dialog) return; - if (open && !wasOpen) { wasOpen = true; keys = {}; preview = null; keyNotice = ''; dialog.showModal(); void review(); } + if (open && !wasOpen) { wasOpen = true; mode = 'fill'; keys = {}; preview = null; keyNotice = ''; dialog.showModal(); void review(); } else if (!open && wasOpen) { wasOpen = false; session++; dialog.close(); } }); async function review() { const current = ++session; const previous = preview; - busy = true; error = ''; keyNotice = ''; + busy = true; error = ''; keyNotice = ''; confirmRemovals = false; try { - const res = await fetch('/api/deployment-collections/preview', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ blueprintId }) }); + const res = await fetch('/api/deployment-collections/preview', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ blueprintId, mode }) }); const body = await res.json(); if (!res.ok) throw new Error(body.error ?? 'Could not preview the collection.'); if (current === session) { @@ -71,21 +74,38 @@ { if (isDialogBackdrop(event, dialog) && !busy) open = false; }} oncancel={event => { if (busy) event.preventDefault(); }} onclose={() => open = false}>
Collection launch

{preview?.blueprintName ?? 'Preparing your collection'}

+ +

{mode === 'exact' ? 'Keep matching lobbies, add missing ones, and remove excess lobbies and maps outside this blueprint across both games.' : 'Create missing lobbies and keep all existing servers.'}

{#if error}{/if} {#if !preview}

{busy ? 'Finding a lobby and port for every server…' : 'Refresh to prepare another preview.'}

{#if !busy}{/if} {:else} {#if keyNotice}

{keyNotice}

{/if} {#if expired}

This preview expired. Refresh before launch; keys remain entered for unchanged IDs.

{/if} -
{preview.servers.length ? `${preview.servers.length} missing server${preview.servers.length === 1 ? '' : 's'} to create` : 'Blueprint already satisfied'} +
{preview.mode === 'exact' ? `${preview.retainedServers.length} keep · ${preview.servers.length} create · ${preview.removals.length} remove` : preview.servers.length ? `${preview.servers.length} missing server${preview.servers.length === 1 ? '' : 's'} to create` : 'Blueprint already satisfied'} {#each preview.entries as entry}{/each}
MapTargetExisting / queuedTo create
{entry.recipe.mapName} {entry.recipe.game.toUpperCase()}{entry.count}{entry.existing}{entry.missing}
-

Targets count inventory, including stopped servers; they do not mean every server is ready to play. Only missing servers are created. Extra servers remain and existing servers are not updated.

- {#if preview.entries.some(entry => entry.existing)}
Review existing servers{#each preview.entries.filter(entry => entry.existing) as entry}
{entry.recipe.mapName}: {entry.existing} existing

{[...new Set((entry.coverage ?? []).map(server => server.health))].map(health => `${entry.coverage.filter(server => server.health === health).length} ${health}`).join(', ')}

    {#each entry.coverage ?? [] as server}
  • {server.health === 'online' ? 'Healthy' : server.health}
    • {/each}
{/each}
{/if} +

Targets count inventory, including stopped servers; they do not mean every server is ready to play. {preview.mode === 'exact' ? 'Existing deployed lobbies are kept before queued creations, then the lowest lobby numbers. Retained servers are not changed or restarted.' : 'Only missing servers are created. Extra servers remain and existing servers are not updated.'}

+ {#if preview.mode === 'exact' && preview.retainedServers.length}
Servers to keep ({preview.retainedServers.length})
    {#each preview.retainedServers as name}
  • {name}
    • {/each}
{/if} + {#if preview.mode === 'fill' && preview.entries.some(entry => entry.existing)}
Review existing servers{#each preview.entries.filter(entry => entry.existing) as entry}
{entry.recipe.mapName}: {entry.existing} existing

{[...new Set((entry.coverage ?? []).map(server => server.health))].map(health => `${entry.coverage.filter(server => server.health === health).length} ${health}`).join(', ')}

    {#each entry.coverage ?? [] as server}
  • {server.health === 'online' ? 'Healthy' : server.health}
    • {/each}
{/each}
{/if}
-

{preview.servers.length ? 'Your launch order' : 'Nothing to create'}

+

{preview.mode === 'exact' ? 'Review fleet changes' : preview.servers.length ? 'Your launch order' : 'Nothing to create'}

{#each preview.blockers as blocker}{/each} - {#if preview.servers.length} -

Saved keys are reused automatically. Add keys only where required below. Update saved keys in Keys (opens in a new tab). Fleet counts and available slots are checked again when you queue the launch.

+ {#if preview.servers.length || preview.removals.length} + {#if preview.servers.length}

Saved keys are reused automatically. Add keys only where required below. Update saved keys in Keys (opens in a new tab). Fleet counts and available slots are checked again when you queue the launch.

{/if}
{ event.preventDefault(); void launch(); }}> + {#if preview.removals.length} +
+

Remove {preview.removals.length} server{preview.removals.length === 1 ? '' : 's'}

+

Containers, files, and registrations will be removed after a fresh zero-player check. Busy or unreachable servers wait. Unstarted creations are cancelled. Saved keys are kept. New servers reusing these ports wait for removal to finish.

+
    {#each preview.removals as server}
  • {server.name}{server.reason}
    • {/each}
+ +
+ {/if} + {#if preview.servers.length}
    {#each preview.servers as server, index (server.name ?? `unavailable-${index}`)}
  1. @@ -98,15 +118,22 @@
    2. {/each}
-

{keysReady} / {preview.servers.length} keys available

Validity is checked at startup. Preview expires at {$timeFormat.minuteTime(preview.expiresAt)}.

{#if keysReady < preview.servers.length}{/if}
+ {/if} +
{#if preview.servers.length}

{keysReady} / {preview.servers.length} keys available

{/if}

Key validity is checked at startup. Preview expires at {$timeFormat.minuteTime(preview.expiresAt)}.

{#if keysReady < preview.servers.length}{/if}
- {:else}

Your fleet already meets or exceeds every target in this blueprint. No servers will be created or removed.

{/if} + {:else}

{preview.mode === 'exact' ? 'Your fleet already matches this blueprint.' : 'Your fleet already meets or exceeds every target in this blueprint.'} No servers will be created or removed.

{/if}
Setup and launch requirements

Each server uses the latest promoted setup when its turn begins. The host must have the selected games and DLC. This preview checks settings and capacity; it does not probe game files, images, credentials, or network access.

{#each preview.entries as entry}
{/each}
{/if}