From cfed584614236c5d79ddb4b2a5898cd3ca1a1caa Mon Sep 17 00:00:00 2001 From: Josh Wright Date: Sun, 27 Sep 2026 14:20:04 -0400 Subject: [PATCH] feat: add provisioning lifecycle actions Rename Deployments to Provisioning with compatible redirects and APIs. Add durable Start, Restart, Stop, and Rebuild operations alongside Create and Destroy; rebuild wipes the instance and recreates it in the same slot and port. Add lifecycle, migration, recovery, and Komodo command tests. Type checks and production build pass; the full suite retains a telemetry assertion failure reproduced on the unchanged branch. Refs #39 --- README.md | 27 +- docs/deployment-blueprints.md | 6 +- docs/provisioning.md | 64 +++++ docs/ui-ux-review.md | 4 +- src/lib/provisioning.ts | 17 ++ src/lib/server/db/index.ts | 21 ++ src/lib/server/db/schema.sql | 7 +- src/lib/server/fleet/collections.ts | 6 +- src/lib/server/fleet/create.ts | 87 ++++++- src/lib/server/fleet/deployments.ts | 25 +- src/lib/server/fleet/house-party.ts | 2 +- src/lib/server/fleet/lifecycle.ts | 117 +++++++++ src/lib/server/fleet/overview.ts | 6 +- src/lib/server/fleet/provisioning.ts | 2 +- src/lib/server/integrations/komodo.ts | 15 ++ src/lib/server/pipeline/restart-rollout.ts | 41 ++- src/lib/server/pipeline/worker.ts | 3 + .../components/DeployCollectionDialog.svelte | 2 +- .../ui/components/DeployServerDialog.svelte | 6 +- .../ui/components/DestroyServerModal.svelte | 6 +- src/lib/ui/components/HousePartyDialog.svelte | 6 +- src/lib/ui/components/LifecycleDialog.svelte | 70 +++++ src/routes/+layout.svelte | 10 +- src/routes/+page.svelte | 2 +- .../api/deployments/[id]/resume/+server.ts | 4 +- src/routes/api/provisioning/+server.ts | 20 ++ src/routes/api/provisioning/[id]/+server.ts | 1 + .../api/provisioning/[id]/cancel/+server.ts | 1 + .../api/provisioning/[id]/resume/+server.ts | 1 + .../api/provisioning/house-party/+server.ts | 1 + .../api/provisioning/preview/+server.ts | 1 + src/routes/api/servers/[name]/+server.ts | 2 +- src/routes/api/system-status/+server.ts | 2 +- .../deployments/[...path]/+page.server.ts | 6 + .../+page.svelte | 37 +-- .../[id]/+page.server.ts | 2 +- .../[id]/+page.svelte | 67 +++-- .../collections/[id]/+page.server.ts | 0 .../collections/[id]/+page.svelte | 8 +- src/routes/servers/[name]/+page.svelte | 7 +- tests/unit/komodo-runtime.test.ts | 36 +++ tests/unit/provisioning.test.ts | 246 ++++++++++++++++++ 42 files changed, 886 insertions(+), 108 deletions(-) create mode 100644 docs/provisioning.md create mode 100644 src/lib/provisioning.ts create mode 100644 src/lib/server/fleet/lifecycle.ts create mode 100644 src/lib/ui/components/LifecycleDialog.svelte create mode 100644 src/routes/api/provisioning/+server.ts create mode 100644 src/routes/api/provisioning/[id]/+server.ts create mode 100644 src/routes/api/provisioning/[id]/cancel/+server.ts create mode 100644 src/routes/api/provisioning/[id]/resume/+server.ts create mode 100644 src/routes/api/provisioning/house-party/+server.ts create mode 100644 src/routes/api/provisioning/preview/+server.ts create mode 100644 src/routes/deployments/[...path]/+page.server.ts rename src/routes/{deployments => provisioning}/+page.svelte (91%) rename src/routes/{deployments => provisioning}/[id]/+page.server.ts (89%) rename src/routes/{deployments => provisioning}/[id]/+page.svelte (67%) rename src/routes/{deployments => provisioning}/collections/[id]/+page.server.ts (100%) rename src/routes/{deployments => provisioning}/collections/[id]/+page.svelte (97%) create mode 100644 tests/unit/komodo-runtime.test.ts create mode 100644 tests/unit/provisioning.test.ts diff --git a/README.md b/README.md index ba09ff3..05dc07a 100644 --- a/README.md +++ b/README.md @@ -127,9 +127,9 @@ RCON, and selected-map verification. No token, no session, no auth freeze. If Komodo or RCON aren't configured, the sweep marks pending rows `manual` so nothing hangs forever. -## Deployment blueprints and launch previews +## Provisioning blueprints and launch previews -Deployments also offers **House Party Protocol**: review the entire fleet and queue +Provisioning also offers **House Party Protocol**: review the entire fleet and queue destruction in one action. This includes queued creations and reuses pending destruction requests. Each server waits for a fresh zero-player reading; occupied or unreachable servers remain queued. Containers, server files, and registrations @@ -142,7 +142,7 @@ If deletion fails after Komodo removal, the operation detail page offers repeating Komodo deletion. Requesting destruction again also resumes that cleanup. Recovery is blocked if a newer operation exists for the same server name. -Deployments includes a blueprint builder for whole server collections: choose +Provisioning includes a blueprint builder for whole server collections: choose multiple T5/T6 maps and set the desired total lobbies for each. **Conform fleet** counts existing and queued servers and creates only the shortfall. The preview shows target/existing/missing counts and IDs, ports, and key requirements for new @@ -348,10 +348,25 @@ RCON parsing (fake UDP socket). 5. Load the app on the LAN or over tailscale (no web-app auth by design). -## Campaigns and server deployments +## Campaigns and provisioning Campaigns (`/campaigns`) tracks script/config updates, drift reconciliation, restart progress, file diffs, and rollbacks. Existing campaign records are preserved in the original database tables. The campaign API lives at `/api/campaigns`; legacy detail API endpoints remain available for compatibility. -Deployments (`/deployments`, `/api/deployments`) now tracks server creations and destructions with live stages and persistent history, including failures, warnings, and interrupted operations. History starts with operations performed after this update and survives server removal. New creation requests return HTTP 202 with a deployment ID and open the deployment detail page. Creation is complete only when Komodo reports healthy, RCON responds successfully, and the RCON map exactly matches the selected map. Queued requests reserve their server name and port; queued creation and pending health checks survive application restarts. The detail page records startup progress and explains any health or map mismatch that is still blocking completion. +Provisioning (`/provisioning`, `/api/provisioning`) tracks Create, Start, Restart, Rebuild, Stop, and Destroy operations with live stages and persistent history, including failures, warnings, and interrupted operations. History starts with operations performed after this update and survives server removal. New creation requests return HTTP 202 with an operation ID and open the provisioning detail page. Creation is complete only when Komodo reports healthy, RCON responds successfully, and the RCON map exactly matches the selected map. Queued requests reserve their server name and port; queued creation and pending health checks survive application restarts. The detail page records startup progress and explains any health or map mismatch that is still blocking completion. -Each server deployment opens a detail page with its flow and timestamped activity. Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh RCON player count of zero before teardown; occupied or unreachable servers remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the deployment detail page. +Each server deployment opens a detail page with its flow and timestamped activity. Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh RCON player count of zero before teardown; occupied or unreachable servers remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the provisioning detail page. + + +Use **Provisioning → Manage server**, or a server’s **Server actions**, to start, +restart, stop, or rebuild an existing instance. **Rebuild destroys and creates in +place**: the old container, Komodo deployment, server registration, and all server +files are deleted. A fresh instance uses the same server ID, map, and port with +the latest promoted setup. No old instance files are carried forward. The key +stored separately in **Keys** must be available before teardown. Stop retains the +instance so Start can bring it back. Disruptive operations wait for a fresh empty +reading or a confirmed stopped container, and completion requires verification. +See [Provisioning](docs/provisioning.md) for lifecycle behavior and recovery. + +Existing `/deployments` page links redirect to `/provisioning`, preserving filters +and operation IDs. Legacy `/api/deployments` endpoints and stored history remain +compatible. diff --git a/docs/deployment-blueprints.md b/docs/deployment-blueprints.md index 2777ffe..754b7d5 100644 --- a/docs/deployment-blueprints.md +++ b/docs/deployment-blueprints.md @@ -1,6 +1,6 @@ # Deployment blueprints -The Deployments page is the starting point for reusable server collections. A +The Provisioning page is the starting point for reusable server collections. A blueprint describes a whole lineup: for example, two Kino lobbies, one Moon lobby, and two Town lobbies. Counts are desired fleet totals: existing and queued servers count toward each target, and only missing servers are created. @@ -23,7 +23,7 @@ It fills deficits without changing existing servers' files. ## Build and launch a collection -1. Choose **Create blueprint** on Deployments and name the lineup. +1. Choose **Create blueprint** on Provisioning and name the lineup. 2. Add maps from Black Ops I or II and set the desired total lobbies for each. 3. Choose **Save & preview** (or **Preview fleet changes** on a saved blueprint). Choose **Fill missing servers** or **Match blueprint exactly**, then review @@ -113,7 +113,7 @@ operation links back to its collection. ## Create and review a launch -1. Choose **Single server** on Deployments (or the deploy action on the fleet). +1. Choose **Single server** on Provisioning (or the deploy action on the fleet). 2. Choose the current promoted setup and select a game/map, or select a blueprint. 3. Choose **Preview launch**. Review the image reference, configuration sources, managed files and versions, selected map, lobby, port, and prerequisite checks. diff --git a/docs/provisioning.md b/docs/provisioning.md new file mode 100644 index 0000000..6a256d3 --- /dev/null +++ b/docs/provisioning.md @@ -0,0 +1,64 @@ +# Provisioning + +Open **Operations → Provisioning** for instance lifecycle work, blueprints, and +persistent activity. Create a server using **Create server**. For an existing +server, choose **Manage server** or open its **Server actions** menu. + +| Action | Behavior | +| --- | --- | +| Create | Create a new instance using the latest promoted files and verify container health, RCON, and the selected map. | +| Start | Start the existing container and verify container health and RCON. | +| Restart | Wait until empty, restart the existing container, and verify container health and RCON. | +| Rebuild | Wait until empty, destroy the instance, delete all server files and its registration, then create a fresh instance in the same server slot and port. Verify container health, RCON, and map. | +| Stop | Wait until empty, stop the container, and verify that Komodo reports it stopped. Keep the instance for a later Start. | +| Destroy | Wait until empty, remove the container and Komodo deployment, then delete server files and registration. | + +Rebuild requires confirmation because **no instance data is preserved**. Its fresh +files come from the latest promoted setup when creation begins, including changes +promoted while waiting. The independently saved key in **Keys** must exist before +teardown. Blueprint definitions, promoted artifacts, saved keys, and historical +activity belong to Electric Cherry, and are not instance files. + +Rebuild reserves the original ID and port throughout teardown and creation, even +when a lower lobby is available. It shares the existing creation queue and startup +limit. A failed removal never proceeds to creation. Once Komodo deletion is +confirmed, failed local cleanup offers **Resume rebuild cleanup**; creation cannot +proceed until the old files are fully removed. An interrupted fresh startup can +resume health verification, without repeating teardown or creation. Inspect other +failures before making another request. + +Start and Restart share the rolling restart limit with campaigns, betas, and +maintenance. A successful API reply is followed by a recovery delay and actual +container/RCON checks. Failed or ambiguous restart requests retain their recovery +slot and are not automatically replayed. Automated restart submissions also appear +in Provisioning history. Stop requires an explicit stopped container state; an +unreachable Komodo API is not evidence that a stop succeeded. + +Disruptive operations require a fresh zero-player RCON result, unless Komodo +positively reports a stopped container. Runtime actions wait for conflicting +campaign, beta, maintenance, or provisioning work. Duplicate pending requests for +the same action return the existing operation; a different action returns a +conflict. Requests and events survive application restarts. + +## Routes and compatibility + +- `/provisioning`: active operations, blueprints, and history filtered by action. +- `/provisioning/:id`: operation stages, errors, timestamps, and activity. +- `/provisioning/collections/:id`: collection progress. +- `GET /api/provisioning`: lifecycle history (`deployments` response property). +- `POST /api/provisioning`: queue `{ "server": "t5_zombie_theater_1", "action": "start" }`. + Supported actions are `start`, `restart`, `rebuild`, and `stop`. Returns HTTP 202 + with `deploymentId`. Create continues through `POST /api/servers`; Destroy uses + `DELETE /api/servers/:name`. +- `/api/provisioning/:id`, `/:id/resume`, `/:id/cancel`, `/preview`, and + `/house-party` preserve the existing operation, recovery, removal cancellation, + launch preview, and fleet destruction APIs. + +Old `/deployments` pages redirect to their `/provisioning` equivalents, preserving +query parameters. `/api/deployments` remains compatible, including old campaign +detail endpoints. No history or IDs are renamed or discarded. + +Start and Stop use Komodo's corresponding runtime commands. Rebuild uses full +destruction and creation, not Komodo's redeploy command. See the +[Komodo execute API](https://docs.rs/komodo_client/latest/komodo_client/api/execute/index.html) +for the underlying container operations. diff --git a/docs/ui-ux-review.md b/docs/ui-ux-review.md index 5855bdc..891c391 100644 --- a/docs/ui-ux-review.md +++ b/docs/ui-ux-review.md @@ -122,7 +122,7 @@ Add **Cancel queued removal** while the operation is still waiting, with a preci Acceptance: an operator can withdraw a waiting removal, and a race with worker execution cannot leave the UI claiming cancellation after removal has begun. -Sources: [DestroyServerModal](../src/lib/ui/components/DestroyServerModal.svelte), lines 36–40; [HousePartyDialog](../src/lib/ui/components/HousePartyDialog.svelte), line 51; [deployment detail](../src/routes/deployments/[id]/+page.svelte), lines 110–123. Existing consequence confirmations should remain. +Sources: [DestroyServerModal](../src/lib/ui/components/DestroyServerModal.svelte), lines 36–40; [HousePartyDialog](../src/lib/ui/components/HousePartyDialog.svelte), line 51; [deployment detail](../src/routes/provisioning/[id]/+page.svelte), lines 110–123. Existing consequence confirmations should remain. ### 8. Report partial success as a distinct outcome — P1 · Source @@ -310,7 +310,7 @@ Use “server” consistently for the managed unit; introduce “lobby” as its Acceptance: primary tasks are identifiable by placement and wording, and multiple browser tabs are distinguishable. Contrast should be measured before claiming a conformance result. -Sources: [tokens](../src/lib/ui/styles/tokens.css), lines 57–71, 86–87; [BetaPlanBuilder](../src/lib/ui/components/BetaPlanBuilder.svelte); [navigation and shortcuts](../src/routes/+layout.svelte), lines 37–65; [deployment blueprint actions](../src/routes/deployments/+page.svelte), lines 125–132. +Sources: [tokens](../src/lib/ui/styles/tokens.css), lines 57–71, 86–87; [BetaPlanBuilder](../src/lib/ui/components/BetaPlanBuilder.svelte); [navigation and shortcuts](../src/routes/+layout.svelte), lines 37–65; [deployment blueprint actions](../src/routes/provisioning/+page.svelte), lines 125–132. ### 26. Make investigation and handoffs easier over time — P3 · Opportunity diff --git a/src/lib/provisioning.ts b/src/lib/provisioning.ts new file mode 100644 index 0000000..bf71dc6 --- /dev/null +++ b/src/lib/provisioning.ts @@ -0,0 +1,17 @@ +export const provisioningActions = ['create', 'start', 'restart', 'rebuild', 'stop', 'destroy'] as const; +export type ProvisioningAction = typeof provisioningActions[number]; +export type RuntimeAction = Exclude; +export const actionLabels: Record = { + create: 'Create', start: 'Start', restart: 'Restart', rebuild: 'Rebuild', stop: 'Stop', destroy: 'Destroy' +}; +export const actionCompleted: Record = { + create: 'Server creation completed', start: 'Server started', restart: 'Server restarted', + rebuild: 'Server rebuilt', stop: 'Server stopped', destroy: 'Server destroyed' +}; +export const runtimeActions: RuntimeAction[] = ['start', 'restart', 'rebuild', 'stop']; +export const runtimeDescriptions: Record = { + start: 'Start the existing container and verify its health.', + restart: 'Restart the container after the server is empty, then verify its health.', + rebuild: 'Destroy the instance and delete all of its server files, then create a fresh instance in the same server slot and port with the latest promoted setup. No instance data is retained. Uses the key saved separately in Keys. Waits until empty, then verifies health and map.', + stop: 'Stop the container after the server is empty. Its files, key, and registration are retained so you can start it again.' +}; diff --git a/src/lib/server/db/index.ts b/src/lib/server/db/index.ts index 0fbe36b..52334ed 100644 --- a/src/lib/server/db/index.ts +++ b/src/lib/server/db/index.ts @@ -227,6 +227,25 @@ function migrateGameTelemetry(handle: Database.Database): void { })(); } +function migrateProvisioningActions(handle: Database.Database): void { + const table = handle.prepare("SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'server_deployments'").get() as { sql: string }; + if (table.sql.includes("'rebuild'")) return; + // Rebuild the CHECK constraint without cascading into events, recipes, or + // collection members. Preserve rowids: they order same-server operations. + handle.pragma('foreign_keys = OFF'); + try { + handle.transaction(() => { + handle.exec(table.sql.replace('server_deployments', 'server_deployments_new') + .replace("'create', 'destroy'", "'create', 'start', 'restart', 'rebuild', 'stop', 'destroy'")); + handle.exec(`INSERT INTO server_deployments_new (rowid, id, server, action, state, stage, started_at, finished_at, triggered_by, error, warnings) + SELECT rowid, id, server, action, state, stage, started_at, finished_at, triggered_by, error, warnings FROM server_deployments; + DROP TABLE server_deployments; + ALTER TABLE server_deployments_new RENAME TO server_deployments;`); + if ((handle.pragma('foreign_key_check') as unknown[]).length) throw new Error('Provisioning migration found broken references'); + })(); + } finally { handle.pragma('foreign_keys = ON'); } +} + export function db(): Database.Database { if (_db) return _db; @@ -240,6 +259,8 @@ export function db(): Database.Database { handle.pragma('foreign_keys = ON'); dropLegacyBetaTables(handle); handle.exec(schemaSql); + migrateProvisioningActions(handle); + migrateTable(handle, 'restart_rollout', [{ name: 'operation_id', ddl: 'ALTER TABLE restart_rollout ADD COLUMN operation_id TEXT' }]); migrateServers(handle); // Column additions must run BEFORE any statement that references the // new columns. schema.sql cannot include those references — see the diff --git a/src/lib/server/db/schema.sql b/src/lib/server/db/schema.sql index 86351e7..e1f0b75 100644 --- a/src/lib/server/db/schema.sql +++ b/src/lib/server/db/schema.sql @@ -409,7 +409,7 @@ CREATE INDEX IF NOT EXISTS idx_telemetry_requests_received ON telemetry_requests CREATE TABLE IF NOT EXISTS server_deployments ( id TEXT PRIMARY KEY, server TEXT NOT NULL, - action TEXT NOT NULL CHECK (action IN ('create', 'destroy')), + action TEXT NOT NULL CHECK (action IN ('create', 'start', 'restart', 'rebuild', 'stop', 'destroy')), state TEXT NOT NULL, stage TEXT NOT NULL, started_at TEXT NOT NULL, @@ -436,6 +436,11 @@ CREATE TABLE IF NOT EXISTS server_creation_jobs ( expected_map TEXT NOT NULL ); +CREATE TABLE IF NOT EXISTS server_rebuild_jobs ( + deployment_id TEXT PRIMARY KEY REFERENCES server_deployments(id) ON DELETE CASCADE, + removed_at TEXT +); + -- Blueprints retain map/count selections only. The legacy column name remains -- compatible with older snapshot-based definitions, read as selections at runtime. CREATE TABLE IF NOT EXISTS server_blueprints ( diff --git a/src/lib/server/fleet/collections.ts b/src/lib/server/fleet/collections.ts index 5801fe1..cbfc48e 100644 --- a/src/lib/server/fleet/collections.ts +++ b/src/lib/server/fleet/collections.ts @@ -34,12 +34,12 @@ function coverage(entries: BlueprintEntry[]) { return matches.length === 1 && matches[0].slug === entry.recipe.mapSlug; }).map(server => server.name).sort(); const coverage = existingServers.map(name => { - const operation = db().prepare("SELECT id, state FROM server_deployments WHERE server = ? AND action = 'create' AND state IN ('queued', 'running', 'awaiting-health') ORDER BY rowid DESC LIMIT 1").get(name) as { id: string; state: string } | undefined; + const operation = db().prepare("SELECT id, state FROM server_deployments WHERE server = ? AND action IN ('create','rebuild') AND state IN ('queued', 'running', 'awaiting-health') ORDER BY rowid DESC LIMIT 1").get(name) as { id: string; state: string } | undefined; const server = getServer(name); return { name, health: operation ? operation.state === 'queued' ? 'queued' : 'starting' : server?.container_state && !['running', 'starting', 'restarting'].includes(server.container_state) ? 'stopped' : classifyHealth(server?.last_seen_at ?? null, server?.last_error ?? null, server?.container_state), - href: operation ? `/deployments/${encodeURIComponent(operation.id)}` : `/servers/${encodeURIComponent(name)}` }; + href: operation ? `/provisioning/${encodeURIComponent(operation.id)}` : `/servers/${encodeURIComponent(name)}` }; }); return { existingServers, coverage, existing: existingServers.length, missing: Math.max(0, entry.count - existingServers.length) }; }); @@ -48,7 +48,7 @@ function coverage(entries: BlueprintEntry[]) { function fleetSnapshot() { return housePartyTargets().map(name => ({ name, server: db().prepare('SELECT game, map_slug, map, game_port FROM servers WHERE name = ?').get(name) ?? null, - creation: db().prepare("SELECT id FROM server_deployments WHERE server = ? AND action = 'create' ORDER BY rowid DESC LIMIT 1").get(name) ?? null, + creation: db().prepare("SELECT id FROM server_deployments WHERE server = ? AND action IN ('create','rebuild') ORDER BY rowid DESC LIMIT 1").get(name) ?? null, removal: db().prepare("SELECT id FROM server_deployments WHERE server = ? AND action = 'destroy' AND state IN ('waiting', 'running')").get(name) ?? null })); } diff --git a/src/lib/server/fleet/create.ts b/src/lib/server/fleet/create.ts index a05be56..92fa564 100644 --- a/src/lib/server/fleet/create.ts +++ b/src/lib/server/fleet/create.ts @@ -42,7 +42,7 @@ export type CreateServerResult = export type DestroyServerResult = | { ok: true; warnings: string[] } - | { ok: false; kind: 'not-found' | 'komodo' | 'fs' | 'db'; message: string; warnings: string[] }; + | { ok: false; kind: 'not-found' | 'conflict' | 'komodo' | 'fs' | 'db'; message: string; warnings: string[] }; export interface CreateServerInput { game?: ZombieGame; // omitted by older clients: T5 @@ -63,7 +63,7 @@ export function nextLobbyNumber(baseName: string, additionalNames: string[] = [] const rows = db() .prepare(`SELECT name FROM servers WHERE name LIKE ? UNION SELECT j.server AS name FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id - WHERE d.state IN ('queued', 'running', 'awaiting-health')`) + WHERE d.state IN ('queued', 'running', 'awaiting-health') OR (d.action = 'rebuild' AND (d.state = 'waiting' OR (d.state IN ('failed','interrupted') AND d.stage = 'Removing server files and registry')))`) .all(`${baseName}_%`) as Array<{ name: string }>; const taken = new Set(); const prefix = `${baseName}_`; @@ -87,7 +87,7 @@ export function nextAvailablePort(additionalPorts: number[] = [], releasingServe const rows = db() .prepare(`SELECT name, game_port FROM servers WHERE game_port IS NOT NULL UNION SELECT j.server AS name, j.port AS game_port FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id - WHERE d.state IN ('queued', 'running', 'awaiting-health')`) + WHERE d.state IN ('queued', 'running', 'awaiting-health') OR (d.action = 'rebuild' AND (d.state = 'waiting' OR (d.state IN ('failed','interrupted') AND d.stage = 'Removing server files and registry')))`) .all() as Array<{ name: string; game_port: number }>; const taken = new Set([...rows.filter(row => !releasingServers.includes(row.name)).map((r) => r.game_port), ...additionalPorts]); for (let p = config.portRange.min; p <= config.portRange.max; p++) { @@ -229,6 +229,63 @@ export function queueCreateServer(input: CreateServerInput, releasingServers: st })(); } +/** Reserve the same identity and port before removing any part of the instance. */ +export function queueRebuildServer(name: string): CreateServerResult { + const server = getServer(name); + const game = gameOfServer(name); + const map = server?.map_slug && zombieMapBySlug(game, server.map_slug); + if (!server || !map || !server.game_port || server.komodo_type !== 'deployment' || server.komodo_name !== name + || (server.host_dir && server.host_dir !== serverDir(name))) { + return { ok: false, kind: 'bad-request', message: 'Rebuild requires a managed server with its original map, directory, and port.' }; + } + if (!config.komodo.targetServer || !config.rcon.password) return { ok: false, kind: 'unconfigured', message: 'Configure the Komodo target host and RCON before rebuilding.' }; + if (!getServerKey(name)) return { ok: false, kind: 'bad-request', message: `Save a server key for ${name} in Keys before rebuilding.` }; + // Validate that the replacement setup can be composed before destruction. + const recipe = captureRecipe(game, map.slug); + return db().transaction(() => { + const id = beginDeployment(name, 'rebuild'); + db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); + db().prepare('INSERT INTO server_creation_jobs (deployment_id, server, port, game_mode, map_slug, expected_map) VALUES (?, ?, ?, ?, ?, ?)') + .run(id, name, server.game_port, 'zombie', map.slug, map.canonical); + db().prepare('INSERT INTO server_rebuild_jobs (deployment_id) VALUES (?)').run(id); + db().prepare('INSERT INTO server_launch_recipes (deployment_id, recipe) VALUES (?, ?)').run(id, JSON.stringify(recipe)); + deploymentStage(id, 'Waiting to rebuild the empty server'); + return { ok: true as const, name, port: server.game_port!, deploymentId: id }; + })(); +} + +/** Caller has just verified emptiness and excluded conflicting work. */ +export async function rebuildServer(id: string): Promise { + const job = db().prepare(`SELECT j.*, d.stage FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id + WHERE d.id = ? AND d.action = 'rebuild' AND d.state = 'waiting'`).get(id) as (CreationJob & { stage: string }) | undefined; + if (!job) return; + if (creating || destroying.size || creationQueue().active.length >= config.createConcurrency) { + // The cleanup checkpoint must survive capacity waits and app restarts. + if (job.stage !== REMOVING_SERVER) deploymentStage(id, 'Waiting for a server creation slot before rebuilding'); + return; + } + // Recheck prerequisites while the original instance still exists. + if (!getServerKey(job.server)) { finishDeployment(id, { ok: false, message: 'Saved server key was removed; rebuild did not begin.' }); return; } + try { captureRecipe(gameOfServer(job.server), job.map_slug); } + catch (err) { finishDeployment(id, { ok: false, message: (err as Error).message }); return; } + db().prepare("UPDATE server_deployments SET state = 'running' WHERE id = ? AND state = 'waiting'").run(id); + destroying.add(job.server); + try { + const result = job.stage === REMOVING_SERVER + ? await cleanupDestroyedServer(job.server, id) : await destroyServerUnlocked(job.server, id); + if (!result.ok || result.warnings.length) { + finishDeployment(id, { ok: false, message: result.ok ? 'Old server files could not be completely removed.' : result.message, warnings: result.warnings }); + return; + } + db().transaction(() => { + db().prepare('UPDATE server_rebuild_jobs SET removed_at = ? WHERE deployment_id = ?').run(new Date().toISOString(), id); + db().prepare("UPDATE server_deployments SET state = 'queued' WHERE id = ?").run(id); + deploymentStage(id, 'Queued for server creation'); + })(); + } catch (err) { finishDeployment(id, { ok: false, message: (err as Error).message }); } + finally { destroying.delete(job.server); } +} + function nextName(input: CreateServerInput): string | null { const base = `${input.game ?? 't5'}_${input.gameMode}_${input.mapSlug}`; const lobby = nextLobbyNumber(base); @@ -437,6 +494,10 @@ function hasActiveCreation(name: string): boolean { } export function destroyServer(name: string, triggeredBy = 'ui'): DestroyServerResult & { deploymentId?: string } { + if (db().prepare(`SELECT 1 FROM server_deployments WHERE server = ? AND action IN ('start','restart','stop','rebuild') + AND state IN ('waiting','queued','running')`).get(name)) { + return { ok: false, kind: 'conflict', message: 'Wait for the current provisioning operation before destroying this server.', warnings: [] }; + } const existing = db().prepare("SELECT id FROM server_deployments WHERE server = ? AND action = 'destroy' AND state IN ('waiting', 'running')").get(name) as { id: string } | undefined; if (existing) return { ok: true, warnings: [], deploymentId: existing.id }; const previous = db().prepare("SELECT id FROM server_deployments WHERE server = ? ORDER BY rowid DESC LIMIT 1").get(name) as { id: string } | undefined; @@ -485,7 +546,11 @@ async function runDestructions(): Promise { continue; } // A fresh successful zero is required; cached occupancy and timeouts are not evidence of emptiness. - const players = await rconGetPlayerCount(operation.server); + let players = await rconGetPlayerCount(operation.server); + if (players === null) { + const health = await komodoInspectHealthy(operation.server); + if (['stopped','exited','created','dead'].includes(health.state?.toLowerCase() ?? '')) players = 0; + } if (!db().prepare("SELECT id FROM server_deployments WHERE id = ? AND state = 'waiting'").get(operation.id)) continue; if (players !== 0) { deploymentStage(operation.id, players === null @@ -580,10 +645,12 @@ async function cleanupDestroyedServer(name: string, operation: string): Promise< }; } - const unfinished = db().prepare("SELECT id FROM server_deployments WHERE server = ? AND action = 'create' AND state = 'awaiting-health'").all(name) as { id: string }[]; - for (const { id } of unfinished) finishDeployment(id, { ok: false, message: 'Server destroyed before creation health checks completed' }); + const unfinished = db().prepare("SELECT id, action FROM server_deployments WHERE server = ? AND state = 'awaiting-health'").all(name) as { id: string; action: string }[]; + for (const { id, action } of unfinished) finishDeployment(id, { ok: false, message: action === 'create' + ? 'Server destroyed before creation health checks completed' : 'Server destroyed before operation verification completed' }); return { ok: true, warnings }; -} interface CreationJob { +} +interface CreationJob { deployment_id: string; server: string; port: number; game_mode: GameMode; map_slug: string; expected_map: string; triggered_by?: string; } @@ -636,7 +703,7 @@ export async function processCreations(): Promise { export function creationQueue() { type Entry = { id: string; server: string; state: string; stage: string }; const jobs = db().prepare(`SELECT id, server, state, stage FROM server_deployments - WHERE action = 'create' AND state IN ('queued', 'running', 'awaiting-health') + WHERE action IN ('create', 'rebuild') AND state IN ('queued', 'running', 'awaiting-health') ORDER BY started_at, rowid`).all() as Entry[]; return { active: jobs.filter(job => job.state !== 'queued'), queued: jobs.filter(job => job.state === 'queued') }; } @@ -656,7 +723,7 @@ async function runCreations(): Promise { if (creating || destroying.size > 0) return outcomes; // Restart recovery is observation-only: never repeat external creation or // deployment. A known interrupted startup must retain its readiness gate. - const interrupted = db().prepare("SELECT id FROM server_deployments WHERE action = 'create' AND state = 'interrupted' ORDER BY started_at, rowid").all() as { id: string }[]; + const interrupted = db().prepare("SELECT id FROM server_deployments WHERE action IN ('create', 'rebuild') AND state = 'interrupted' ORDER BY started_at, rowid").all() as { id: string }[]; for (const { id } of interrupted) { if (resumeVerification(id)) continue; // A queued removal does not make the interrupted container disappear. @@ -717,7 +784,7 @@ async function runCreations(): Promise { // API calls can cancel a selected job or resume an older startup while // a previous provision awaits Komodo. Recheck without expanding this pass. if (creationQueue().active.length >= config.createConcurrency) break; - if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND action = 'create' AND state = 'queued'").get(id)) continue; + if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND action IN ('create', 'rebuild') AND state = 'queued'").get(id)) continue; const blocker = creationBlocker(id); if (blocker) { deploymentStage(id, blocker); diff --git a/src/lib/server/fleet/deployments.ts b/src/lib/server/fleet/deployments.ts index a0131b3..6a838f1 100644 --- a/src/lib/server/fleet/deployments.ts +++ b/src/lib/server/fleet/deployments.ts @@ -1,12 +1,14 @@ import { randomUUID } from 'node:crypto'; import { db } from '../db/index.js'; import { recipeForDeployment, summarizeRecipe } from './launch-recipes.js'; +import { actionLabels, type ProvisioningAction } from '../../provisioning.js'; -export function beginDeployment(server: string, action: 'create' | 'destroy', triggeredBy = 'ui'): string { +export function beginDeployment(server: string, action: ProvisioningAction, triggeredBy = 'ui'): string { const id = randomUUID(); db().prepare(`INSERT INTO server_deployments (id, server, action, state, stage, started_at, triggered_by) VALUES (?, ?, ?, 'running', 'Starting', ?, ?)`).run(id, server, action, new Date().toISOString(), triggeredBy); - recordDeploymentEvent(id, action === 'create' ? 'Server creation requested' : 'Server destruction requested'); + recordDeploymentEvent(id, action === 'create' ? 'Server creation requested' + : action === 'destroy' ? 'Server destruction requested' : `${actionLabels[action]} server requested`); return id; } @@ -32,7 +34,7 @@ export function getDeployment(id: string) { UNION ALL SELECT collection_id, deployment_id FROM server_collection_removals) m JOIN server_launch_collections c ON c.id = m.collection_id WHERE m.deployment_id = ?`).get(id) ?? null; return { deployment, events, recipe: launch ? summarizeRecipe(launch.recipe) : null, blueprintName: launch?.blueprintName ?? null, - canResumeVerification: canResumeVerification(id), canResumeCleanup: canResumeCleanup(id), canCancelRemoval: canCancelRemoval(id), collection }; + canResumeVerification: canResumeVerification(id), canResumeCleanup: canResumeCleanup(id), canResumeRebuild: canResumeRebuild(id), canCancelRemoval: canCancelRemoval(id), collection }; } // This stage is written only after Komodo has confirmed deployment deletion. @@ -77,11 +79,26 @@ export function canResumeVerification(id: string): boolean { return !!db().prepare(`SELECT d.id FROM server_deployments d JOIN server_creation_jobs j ON j.deployment_id = d.id JOIN servers s ON s.name = j.server AND s.game_port = j.port - WHERE d.id = ? AND d.action = 'create' AND d.state IN ('failed', 'interrupted') + WHERE d.id = ? AND (d.action = 'create' OR (d.action = 'rebuild' AND EXISTS ( + SELECT 1 FROM server_rebuild_jobs r WHERE r.deployment_id = d.id AND r.removed_at IS NOT NULL))) AND d.state IN ('failed', 'interrupted') AND NOT EXISTS (SELECT 1 FROM server_deployments newer WHERE newer.server = d.server AND newer.rowid > d.rowid)`) .get(id); } +export function canResumeRebuild(id: string): boolean { + return !!db().prepare(`SELECT d.id FROM server_deployments d JOIN server_rebuild_jobs r ON r.deployment_id = d.id + WHERE d.id = ? AND d.state IN ('failed','interrupted') AND d.stage = ? AND r.removed_at IS NULL + AND NOT EXISTS (SELECT 1 FROM server_deployments newer WHERE newer.server = d.server AND newer.rowid > d.rowid)`) + .get(id, REMOVING_SERVER); +} + +export function resumeRebuild(id: string): boolean { + if (!canResumeRebuild(id)) return false; + db().prepare("UPDATE server_deployments SET state = 'waiting', finished_at = NULL, error = NULL WHERE id = ?").run(id); + recordDeploymentEvent(id, 'Resumed rebuild cleanup after confirmed Komodo deletion'); + return true; +} + /** Resume observation only. Never repeat a container creation or restart. */ export function resumeVerification(id: string): boolean { return db().transaction(() => { diff --git a/src/lib/server/fleet/house-party.ts b/src/lib/server/fleet/house-party.ts index ef37cb1..7a2c033 100644 --- a/src/lib/server/fleet/house-party.ts +++ b/src/lib/server/fleet/house-party.ts @@ -6,7 +6,7 @@ export function housePartyTargets(): string[] { SELECT name FROM servers WHERE destroyed_at IS NULL UNION SELECT server AS name FROM server_deployments - WHERE action = 'create' AND state IN ('queued', 'running', 'awaiting-health') + WHERE action IN ('create', 'rebuild') AND state IN ('queued', 'running', 'awaiting-health') ORDER BY name `).all() as { name: string }[]).map(row => row.name); } diff --git a/src/lib/server/fleet/lifecycle.ts b/src/lib/server/fleet/lifecycle.ts new file mode 100644 index 0000000..1515614 --- /dev/null +++ b/src/lib/server/fleet/lifecycle.ts @@ -0,0 +1,117 @@ +import { db } from '../db/index.js'; +import { komodoConfigured, rconConfigured } from '../config.js'; +import { getServer } from './servers.js'; +import { beginDeployment, deploymentStage, finishDeployment, REMOVING_SERVER } from './deployments.js'; +import { queueRebuildServer, rebuildServer } from './create.js'; +import { komodoInspectHealthy, komodoRuntimeAction } from '../integrations/komodo.js'; +import { rconGetPlayerCount } from '../integrations/rcon.js'; +import { refreshRestartRollout, restartInRollout, rolloutWaitReason } from '../pipeline/restart-rollout.js'; +import { runtimeActions, type RuntimeAction } from '../../provisioning.js'; + +const stoppedStates = new Set(['stopped', 'exited', 'created', 'dead']); +export const isStopped = (state?: string) => stoppedStates.has(state?.toLowerCase() ?? ''); + +function blocker(server: string, id = ''): string | null { + if (db().prepare(`SELECT 1 FROM server_deployments WHERE server = ? AND id != ? + AND state IN ('queued','waiting','running','awaiting-health')`).get(server, id)) return 'Waiting for another provisioning operation on this server'; + if (db().prepare(`SELECT 1 FROM deployment_servers WHERE server = ? AND restart_state IN ('pending','failed','held') + UNION ALL SELECT 1 FROM beta_servers bs JOIN betas b ON b.id = bs.beta_id + WHERE bs.server = ? AND b.state IN ('running','ready-promote') + UNION ALL SELECT 1 FROM maintenance_jobs WHERE server = ? AND state IN ('queued','restarting') LIMIT 1`).get(server, server, server)) { + return 'Waiting for campaign, beta, or maintenance work on this server'; + } + return rolloutWaitReason(server, id); +} + +export function requestLifecycle(server: string, action: RuntimeAction) { + if (!runtimeActions.includes(action)) return { ok: false as const, status: 400, error: 'Choose start, restart, rebuild, or stop.' }; + if (!komodoConfigured() || !rconConfigured()) return { ok: false as const, status: 503, error: 'Configure Komodo and RCON before provisioning.' }; + return db().transaction(() => { + const existing = db().prepare(`SELECT id, action FROM server_deployments WHERE server = ? + AND state IN ('queued','waiting','running','awaiting-health') ORDER BY rowid DESC LIMIT 1`).get(server) as { id: string; action: string } | undefined; + if (existing) return existing.action === action + ? { ok: true as const, deploymentId: existing.id } + : { ok: false as const, status: 409, error: 'Another provisioning operation is already pending for this server.' }; + const row = getServer(server); + if (!row || row.destroyed_at) return { ok: false as const, status: 404, error: 'Server not found.' }; + if (action === 'rebuild') { + const result = queueRebuildServer(server); + return result.ok ? { ok: true as const, deploymentId: result.deploymentId } + : { ok: false as const, status: 400, error: result.message }; + } + const id = beginDeployment(server, action); + db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); + deploymentStage(id, action === 'start' ? 'Waiting to start the container' : 'Waiting to verify the server is empty'); + return { ok: true as const, deploymentId: id }; + })(); +} + +interface Operation { id: string; server: string; action: RuntimeAction; state: string; stage: string } +let ticking: Promise | null = null; +export function processLifecycle(): Promise { + if (ticking) return ticking; + ticking = run().finally(() => { ticking = null; }); + return ticking; +} + +async function run(): Promise { + await refreshRestartRollout(); + const operations = db().prepare(`SELECT id, server, action, state, stage FROM server_deployments + WHERE action IN ('start','restart','stop','rebuild') AND (state = 'waiting' OR (action = 'stop' AND state = 'awaiting-health')) + ORDER BY started_at, rowid`).all() as Operation[]; + for (const operation of operations) { + try { await processOperation(operation); } + catch (err) { + // Observation failures are retried; remote mutations are never replayed. + deploymentStage(operation.id, 'Waiting for verification: ' + (err as Error).message); + } + } +} + +async function processOperation(operation: Operation): Promise { + const { id, server, action } = operation; + if (action === 'rebuild' && operation.stage === REMOVING_SERVER) { await rebuildServer(id); return; } + if (!getServer(server)) { finishDeployment(id, { ok: false, message: 'Server is no longer registered.' }); return; } + if (!komodoConfigured() || !rconConfigured()) { deploymentStage(id, 'Waiting for Komodo and RCON configuration'); return; } + if (operation.state === 'awaiting-health') { + const health = await komodoInspectHealthy(server); + if (isStopped(health.state)) { + db().prepare("UPDATE servers SET container_state = ?, last_players = NULL, last_roster = NULL, game_started_at = NULL, game_ended_at = NULL WHERE name = ?").run(health.state, server); + deploymentStage(id, 'Verified container stopped'); + finishDeployment(id, { ok: true }); + } else deploymentStage(id, 'Waiting for container to stop' + (!health.ok ? ': ' + health.reason : '')); + return; + } + const reason = blocker(server, id); + if (reason) { deploymentStage(id, reason); return; } + const health = await komodoInspectHealthy(server); + if (action !== 'start' && !isStopped(health.state)) { + const players = await rconGetPlayerCount(server); + if (players !== 0) { + deploymentStage(id, players === null ? 'Waiting for a fresh player count' : `Waiting for ${players} player(s) to leave`); + return; + } + } + // Requests can arrive during the probes. Recheck before claiming the action. + if (blocker(server, id)) return; + if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND state = 'waiting'").get(id)) return; + if (action === 'rebuild') { await rebuildServer(id); return; } + const claim = db().prepare("UPDATE server_deployments SET state = 'running' WHERE id = ? AND state = 'waiting'").run(id); + if (!claim.changes) return; + if (action === 'start' || action === 'restart') { + const result = await restartInRollout(server, { operationId: id, action }); + if (result.kind === 'queued') { + db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); + deploymentStage(id, result.message); + } + return; + } + deploymentStage(id, 'Stopping container'); + // Persist intent first; a lost reply must not issue another stop later. + try { + const result = await komodoRuntimeAction(server, 'stop'); + if (!result.ok) { finishDeployment(id, { ok: false, message: result.message }); return; } + db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(id); + deploymentStage(id, 'Waiting for container to stop'); + } catch (err) { finishDeployment(id, { ok: false, message: (err as Error).message }); } +} diff --git a/src/lib/server/fleet/overview.ts b/src/lib/server/fleet/overview.ts index 4248491..f610cfa 100644 --- a/src/lib/server/fleet/overview.ts +++ b/src/lib/server/fleet/overview.ts @@ -1,3 +1,4 @@ +import { actionLabels, type ProvisioningAction } from '../../provisioning.js'; import { db } from '../db/index.js'; import type { FleetOperation } from '../../fleet-overview.js'; @@ -41,13 +42,14 @@ export function fleetOperations(now = Date.now()): FleetOperation[] { .all(since) as { id: string; server: string; action: string; state: string; stage: string; error: string | null; started_at: string }[]; for (const row of lifecycle) { + if (row.state === 'awaiting-health') representedRecovery.add(row.server); const state = ['done', 'warning', 'cancelled'].includes(row.state) ? 'done' : ['failed', 'interrupted'].includes(row.state) ? 'attention' : 'active'; operations.push({ key: `deployment:${row.id}`, group: `deployment:${row.id}`, kind: 'deployment', server: row.server, state, startedAt: row.started_at, label: state === 'done' ? row.state === 'cancelled' ? 'Removal cancelled' : row.state === 'warning' ? 'Completed with warnings' : 'Complete' - : state === 'attention' ? `${row.action === 'create' ? 'Launch' : 'Removal'} needs attention` : row.stage, - detail: row.error ?? row.stage, href: `/deployments/${encodeURIComponent(row.id)}` }); + : state === 'attention' ? `${actionLabels[row.action as ProvisioningAction]} needs attention` : row.stage, + detail: row.error ?? row.stage, href: `/provisioning/${encodeURIComponent(row.id)}` }); } const betas = handle.prepare(`SELECT b.id, b.opened_at, bs.server, bs.stage, bs.error FROM betas b JOIN beta_servers bs ON bs.beta_id = b.id WHERE b.state IN ('running', 'ready-promote')`) diff --git a/src/lib/server/fleet/provisioning.ts b/src/lib/server/fleet/provisioning.ts index 58c4409..cbbac8e 100644 --- a/src/lib/server/fleet/provisioning.ts +++ b/src/lib/server/fleet/provisioning.ts @@ -5,5 +5,5 @@ export function isProvisioning(server: string): boolean { const operation = db().prepare(`SELECT action, state FROM server_deployments WHERE server = ? ORDER BY rowid DESC LIMIT 1`).get(server) as { action: string; state: string } | undefined; - return operation?.action === 'create' && ['queued', 'running', 'awaiting-health'].includes(operation.state); + return !!operation && ['create', 'rebuild', 'start'].includes(operation.action) && ['queued', 'running', 'awaiting-health'].includes(operation.state); } diff --git a/src/lib/server/integrations/komodo.ts b/src/lib/server/integrations/komodo.ts index afaee6f..921ed08 100644 --- a/src/lib/server/integrations/komodo.ts +++ b/src/lib/server/integrations/komodo.ts @@ -520,6 +520,21 @@ export async function komodoDeploy(deployment: string): Promise { + if (!komodoConfigured()) return { ok: false, kind: 'unconfigured', message: 'komodo not configured' }; + const row = getServer(server); + if (!row?.komodo_name) return { ok: false, kind: 'no-name', message: 'no komodo_name set for server' }; + const type = row.komodo_type ?? 'deployment'; + const verb = action === 'start' ? 'Start' : 'Stop'; + if (type === 'container') { + if (!row.komodo_server) return { ok: false, kind: 'no-name', message: 'container type needs komodo_server' }; + return postAndClassify(`execute/${verb}Container`, { server: row.komodo_server, container: row.komodo_name }); + } + if (type === 'stack') return postAndClassify(`execute/${verb}Stack`, { stack: row.komodo_name }); + return postAndClassify(`execute/${verb}Deployment`, { deployment: row.komodo_name }); +} + /** Stop + remove the container backing a Komodo deployment. Entity remains. */ export async function komodoDestroyDeployment(deployment: string): Promise { if (!komodoConfigured()) { diff --git a/src/lib/server/pipeline/restart-rollout.ts b/src/lib/server/pipeline/restart-rollout.ts index 2e0f9c8..1bea23b 100644 --- a/src/lib/server/pipeline/restart-rollout.ts +++ b/src/lib/server/pipeline/restart-rollout.ts @@ -1,7 +1,8 @@ import { config } from '../config.js'; import { db } from '../db/index.js'; -import { komodoInspectHealthy, komodoRestart } from '../integrations/komodo.js'; +import { komodoInspectHealthy, komodoRestart, komodoRuntimeAction } from '../integrations/komodo.js'; import { rconStatusResult } from '../integrations/rcon.js'; +import { beginDeployment, deploymentStage, finishDeployment } from '../fleet/deployments.js'; // Komodo queues restart work. Don't mistake the still-running old container // for a completed restart when an API kick immediately follows the request. @@ -15,6 +16,7 @@ interface RestartSlot { healthy_at: string | null; players: number | null; error: string | null; + operation_id: string | null; } function slotFor(server: string): RestartSlot | undefined { @@ -28,7 +30,12 @@ export function restartSettling(server: string): boolean { && Date.now() - Date.parse(slot.requested_at) < HEALTH_POLL_MS); } -export function rolloutWaitReason(server: string): string | null { +export function rolloutWaitReason(server: string, operationId = ''): string | null { + if (db().prepare(`SELECT 1 FROM server_deployments WHERE server = ? AND id != ? + AND ((action IN ('start','stop','rebuild') AND state IN ('running','awaiting-health')) + OR (action = 'destroy' AND state IN ('waiting','running')))`).get(server, operationId)) { + return 'Waiting for the current provisioning operation'; + } const slot = slotFor(server); if (slot?.healthy_at === null) { return `Waiting for the previous restart of this server to recover${slot.error ? `: ${slot.error}` : ''}`; @@ -43,14 +50,17 @@ export type RolloutRestartResult = | { kind: 'failed'; message: string }; /** Reserve capacity before any async I/O; callers still check for players first. */ -export async function restartInRollout(server: string): Promise { +export async function restartInRollout(server: string, options: { operationId?: string; action?: 'start' | 'restart'; triggeredBy?: string } = {}): Promise { + let operationId = options.operationId; const waitReason = db().transaction(() => { - const reason = rolloutWaitReason(server); + const reason = rolloutWaitReason(server, operationId); if (reason) return reason; - db().prepare(`INSERT INTO restart_rollout (server, requested_at) VALUES (?, ?) + operationId ??= beginDeployment(server, options.action ?? 'restart', options.triggeredBy ?? 'automation'); + db().prepare(`INSERT INTO restart_rollout (server, requested_at, operation_id) VALUES (?, ?, ?) ON CONFLICT(server) DO UPDATE SET requested_at = excluded.requested_at, - checked_at = NULL, healthy_at = NULL, players = NULL, error = NULL`) - .run(server, new Date().toISOString()); + checked_at = NULL, healthy_at = NULL, players = NULL, error = NULL, operation_id = excluded.operation_id`) + .run(server, new Date().toISOString(), operationId); + deploymentStage(operationId, options.action === 'start' ? 'Starting container' : 'Restarting container'); return null; }).immediate(); if (waitReason) return { kind: 'queued', message: waitReason }; @@ -59,8 +69,12 @@ export async function restartInRollout(server: string): Promise { db().prepare(`UPDATE restart_rollout SET checked_at = ?, healthy_at = ?, players = ?, error = ? WHERE server = ? AND requested_at = ? AND healthy_at IS NULL`) .run(now, error ? null : now, players, error, slot.server, slot.requested_at); + if (slot.operation_id && db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND state = 'awaiting-health'").get(slot.operation_id)) { + if (error) deploymentStage(slot.operation_id, 'Waiting for health: ' + error); + else { + deploymentStage(slot.operation_id, 'Verified container and RCON health'); + finishDeployment(slot.operation_id, { ok: true }); + } + } } } diff --git a/src/lib/server/pipeline/worker.ts b/src/lib/server/pipeline/worker.ts index 99b6789..956f31d 100644 --- a/src/lib/server/pipeline/worker.ts +++ b/src/lib/server/pipeline/worker.ts @@ -1,4 +1,5 @@ import { processDestructions, processCreations } from '../fleet/create.js'; +import { processLifecycle } from '../fleet/lifecycle.js'; import { sweep } from './sweep.js'; import { betaTick } from './beta-runner.js'; import { emit } from './bus.js'; @@ -37,6 +38,8 @@ async function runTick(): Promise { catch (err) { console.error('[worker] creation error:', err); } try { await processDestructions(); } catch (err) { console.error('[worker] destruction error:', err); } + try { await processLifecycle(); } + catch (err) { console.error('[worker] provisioning error:', err); } try { await sweep(); } catch (err) { diff --git a/src/lib/ui/components/DeployCollectionDialog.svelte b/src/lib/ui/components/DeployCollectionDialog.svelte index fefa32d..dca3445 100644 --- a/src/lib/ui/components/DeployCollectionDialog.svelte +++ b/src/lib/ui/components/DeployCollectionDialog.svelte @@ -60,7 +60,7 @@ const body = await res.json(); if (!res.ok) throw new Error(body.error ?? 'Could not queue the collection.'); open = false; - if (body.collectionId) await goto(`/deployments/collections/${encodeURIComponent(body.collectionId)}`); + if (body.collectionId) await goto(`/provisioning/collections/${encodeURIComponent(body.collectionId)}`); } catch (err) { error = (err as Error).message; } finally { busy = false; } } diff --git a/src/lib/ui/components/DeployServerDialog.svelte b/src/lib/ui/components/DeployServerDialog.svelte index 4ba50e0..c64d528 100644 --- a/src/lib/ui/components/DeployServerDialog.svelte +++ b/src/lib/ui/components/DeployServerDialog.svelte @@ -73,7 +73,7 @@ const previousName = preview?.name; busy = true; error = ''; savedName = ''; keyNotice = ''; try { - const res = await fetch('/api/deployments/preview', { method: 'POST', headers: { 'Content-Type': 'application/json' }, + const res = await fetch('/api/provisioning/preview', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(sourceDeploymentId ? { deploymentId: sourceDeploymentId } : blueprintId ? { blueprintId } : { game, mapSlug }) }); const body = await res.json(); if (!res.ok) throw new Error(body.error ?? 'Could not prepare launch preview.'); @@ -97,7 +97,7 @@ const body = await res.json(); if (!res.ok || !body.ok) throw new Error(body.error ?? 'Could not queue this launch.'); open = false; - await goto(`/deployments/${encodeURIComponent(body.deploymentId)}`); + await goto(`/provisioning/${encodeURIComponent(body.deploymentId)}`); } catch (err) { error = (err as Error).message; } finally { busy = false; } } @@ -119,7 +119,7 @@ { if (isDialogBackdrop(event, dialog) && !busy && !saving) open = false; }} oncancel={event => { if (busy || saving) event.preventDefault(); }} onclose={() => open = false}>
-

{preview ? 'Review server launch' : 'Create server'}

+

{preview ? 'Review server launch' : 'Create server'}

  1. 01 Choose a starting point
  2. 02 Review & launch
{#if error}{/if} {#if !loaded} diff --git a/src/lib/ui/components/DestroyServerModal.svelte b/src/lib/ui/components/DestroyServerModal.svelte index f15948e..ed1f3ac 100644 --- a/src/lib/ui/components/DestroyServerModal.svelte +++ b/src/lib/ui/components/DestroyServerModal.svelte @@ -20,7 +20,7 @@ const res = await fetch(`/api/servers/${encodeURIComponent(name)}`, { method: 'DELETE' }); const body = await res.json(); if (!res.ok || !body.ok || !body.deploymentId) throw new Error(body.error ?? 'Could not queue destruction. Please retry.'); - await goto(`/deployments/${encodeURIComponent(body.deploymentId)}`); + await goto(`/provisioning/${encodeURIComponent(body.deploymentId)}`); open = false; } catch (err) { error = (err as Error).message; } finally { submitting = false; } @@ -32,10 +32,10 @@
{ event.preventDefault(); void destroy(); }}>

Queue server removal?

{name}

-

This stops the container, deletes its Komodo deployment, and removes its server files and campaign details. Deployment history and the saved server key are retained.

+

This stops the container, deletes its Komodo deployment, and removes its server files and campaign details. Provisioning history and the saved server key are retained.

{#if players !== null && players > 0}{players} player{players === 1 ? '' : 's'} currently online.{/if} - Removal waits until a live check confirms the server is empty. Follow progress or cancel the waiting request on the deployment page. + Removal waits until a live check confirms the server is empty. Follow progress or cancel the waiting request on the provisioning page.

You can cancel while waiting. Once removal begins, it cannot be undone.

{#if error}{/if} diff --git a/src/lib/ui/components/HousePartyDialog.svelte b/src/lib/ui/components/HousePartyDialog.svelte index a95b027..391acdb 100644 --- a/src/lib/ui/components/HousePartyDialog.svelte +++ b/src/lib/ui/components/HousePartyDialog.svelte @@ -18,7 +18,7 @@ async function review() { busy = true; error = ''; servers = null; try { - const res = await fetch('/api/deployments/house-party'); + const res = await fetch('/api/provisioning/house-party'); if (!res.ok) throw new Error('Could not load the fleet.'); servers = (await res.json()).servers; } catch (err) { error = (err as Error).message; } @@ -29,7 +29,7 @@ if (busy || !servers?.length) return; busy = true; error = ''; try { - const res = await fetch('/api/deployments/house-party', { + const res = await fetch('/api/provisioning/house-party', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ servers }) }); @@ -50,7 +50,7 @@

Queue fleet removal

Queue destruction of every server, including servers being created. Each waits until a fresh player count confirms it is empty. Unreachable servers stay queued.

Destruction removes containers, server files, and fleet registrations. Saved keys and blueprints are kept.

-

In Active deployments, cancel individual waiting requests or all waiting removals. Once a server begins removal, that removal cannot be cancelled or undone.

+

In Provisioning’s Active tab, cancel individual waiting requests or all waiting removals. Once a server begins removal, that removal cannot be cancelled or undone.

{#if error}{/if} {#if servers} {servers.length ? `${servers.length} server${servers.length === 1 ? '' : 's'} to destroy` : 'The house is already empty.'} diff --git a/src/lib/ui/components/LifecycleDialog.svelte b/src/lib/ui/components/LifecycleDialog.svelte new file mode 100644 index 0000000..7796f81 --- /dev/null +++ b/src/lib/ui/components/LifecycleDialog.svelte @@ -0,0 +1,70 @@ + + + +
+ + +

{action === 'destroy' ? 'Destroy the container, delete its Komodo deployment, and remove all server files and registration after the server is empty. Provisioning history and the key saved separately in Keys are retained.' : runtimeDescriptions[action]}

+ {#if destructive}{/if} + {#if error}{/if} +
+ {#snippet actions()} + + + {/snippet} + + + diff --git a/src/routes/+layout.svelte b/src/routes/+layout.svelte index af94d62..c2bffef 100644 --- a/src/routes/+layout.svelte +++ b/src/routes/+layout.svelte @@ -49,10 +49,10 @@ const stopServers = startServerStream(); const stopFleetMonitor = startFleetMonitor(); - // Two-key nav shortcuts (Gmail-style): `g f` fleet, `g d` deployments, + // Two-key nav shortcuts (Gmail-style): `g f` fleet, `g d` provisioning, // `g b` betas, `g s` scripts. Ignored while typing in inputs. const shortcuts: Record = { - f: '/', d: '/deployments', a: '/campaigns', b: '/betas', + f: '/', d: '/provisioning', a: '/campaigns', b: '/betas', k: '/server-keys', s: '/scripts', c: '/configs', p: '/plugins', l: '/leaderboard', h: '/settings', m: '/metrics' }; let armed = false; @@ -99,7 +99,7 @@ const tabs = [ { href: '/', label: 'Fleet', match: (p: string) => p === '/' || p.startsWith('/servers/') }, { href: '/scripts', label: 'Library', match: (p: string) => /^\/(scripts|configs|plugins)(\/|$)/.test(p) }, - { href: '/deployments', label: 'Operations', match: (p: string) => /^\/(deployments|campaigns|server-keys)(\/|$)/.test(p) }, + { href: '/provisioning', label: 'Operations', match: (p: string) => /^\/(provisioning|deployments|campaigns|server-keys)(\/|$)/.test(p) }, { href: '/betas', label: 'Betas', match: (p: string) => p.startsWith('/betas') }, { href: '/settings', label: 'Settings', match: (p: string) => p.startsWith('/settings') }, { href: '/leaderboard', label: 'Leaderboard', match: (p: string) => p.startsWith('/leaderboard') }, @@ -109,7 +109,7 @@ const secondary = $derived(tabs[1].match($page.url.pathname) ? [{ href: '/scripts', label: 'Scripts' }, { href: '/configs', label: 'Configs' }, { href: '/plugins', label: 'Plugins' }] : tabs[2].match($page.url.pathname) - ? [{ href: '/deployments', label: 'Deployments' }, { href: '/campaigns', label: 'Campaigns' }, { href: '/server-keys', label: 'Keys' }] + ? [{ href: '/provisioning', label: 'Provisioning' }, { href: '/campaigns', label: 'Campaigns' }, { href: '/server-keys', label: 'Keys' }] : tabs[4].match($page.url.pathname) ? [{ href: '/settings', label: 'Preferences' }] : []); @@ -149,7 +149,7 @@ -
{#each [['g f','Fleet'],['g s','Scripts'],['g c','Configs'],['g p','Plugins'],['g d','Deployments'],['g a','Campaigns'],['g b','Betas'],['g h','Settings'],['g k','Keys'],['g l','Leaderboard'],['g m','Metrics'],['?','This help']] as [keys,label]}
{keys}
{label}
{/each}
+
{#each [['g f','Fleet'],['g s','Scripts'],['g c','Configs'],['g p','Plugins'],['g d','Provisioning'],['g a','Campaigns'],['g b','Betas'],['g h','Settings'],['g k','Keys'],['g l','Leaderboard'],['g m','Metrics'],['?','This help']] as [keys,label]}
{keys}
{label}
{/each}
diff --git a/src/routes/+page.svelte b/src/routes/+page.svelte index e45647e..fb50069 100644 --- a/src/routes/+page.svelte +++ b/src/routes/+page.svelte @@ -173,7 +173,7 @@ {/if} {#if servers.length === 0} - {#snippet actions()}{/snippet} + {#snippet actions()}{/snippet} {:else}
diff --git a/src/routes/api/deployments/[id]/resume/+server.ts b/src/routes/api/deployments/[id]/resume/+server.ts index b033ea1..01bfa13 100644 --- a/src/routes/api/deployments/[id]/resume/+server.ts +++ b/src/routes/api/deployments/[id]/resume/+server.ts @@ -1,9 +1,9 @@ import { json } from '@sveltejs/kit'; -import { resumeVerification, resumeCleanup } from '$lib/server/fleet/deployments.js'; +import { resumeVerification, resumeCleanup, resumeRebuild } from '$lib/server/fleet/deployments.js'; import { kick } from '$lib/server/pipeline/worker.js'; export function POST({ params }: { params: { id: string } }) { - if (!resumeVerification(params.id) && !resumeCleanup(params.id)) return json({ error: 'This operation cannot resume. Refresh and inspect its server.' }, { status: 409 }); + if (!resumeVerification(params.id) && !resumeCleanup(params.id) && !resumeRebuild(params.id)) return json({ error: 'This operation cannot resume. Refresh and inspect its server.' }, { status: 409 }); void kick().catch(err => console.error('[deployment] verification kick failed:', err)); return json({ ok: true }, { status: 202 }); } diff --git a/src/routes/api/provisioning/+server.ts b/src/routes/api/provisioning/+server.ts new file mode 100644 index 0000000..d0cbff7 --- /dev/null +++ b/src/routes/api/provisioning/+server.ts @@ -0,0 +1,20 @@ +import { json } from '@sveltejs/kit'; +import { listDeployments } from '$lib/server/fleet/deployments.js'; +import { requestLifecycle } from '$lib/server/fleet/lifecycle.js'; +import { runtimeActions, type RuntimeAction } from '$lib/provisioning.js'; +import { kick } from '$lib/server/pipeline/worker.js'; + +export function GET() { return json({ deployments: listDeployments() }); } + +export async function POST({ request }: { request: Request }) { + const body = await request.json().catch(() => null); + if (!body || typeof body.server !== 'string' || !body.server || !runtimeActions.includes(body.action)) { + return json({ error: 'server and a valid lifecycle action are required.' }, { status: 400 }); + } + try { + const result = requestLifecycle(body.server, body.action as RuntimeAction); + if (!result.ok) return json({ error: result.error }, { status: result.status }); + void kick().catch(err => console.error('[provisioning] worker kick failed:', err)); + return json(result, { status: 202 }); + } catch (err) { return json({ error: (err as Error).message }, { status: 400 }); } +} diff --git a/src/routes/api/provisioning/[id]/+server.ts b/src/routes/api/provisioning/[id]/+server.ts new file mode 100644 index 0000000..fab30d7 --- /dev/null +++ b/src/routes/api/provisioning/[id]/+server.ts @@ -0,0 +1 @@ +export * from '../../deployments/[id]/+server.js'; diff --git a/src/routes/api/provisioning/[id]/cancel/+server.ts b/src/routes/api/provisioning/[id]/cancel/+server.ts new file mode 100644 index 0000000..bb0aa4a --- /dev/null +++ b/src/routes/api/provisioning/[id]/cancel/+server.ts @@ -0,0 +1 @@ +export * from '../../../deployments/[id]/cancel/+server.js'; diff --git a/src/routes/api/provisioning/[id]/resume/+server.ts b/src/routes/api/provisioning/[id]/resume/+server.ts new file mode 100644 index 0000000..903892c --- /dev/null +++ b/src/routes/api/provisioning/[id]/resume/+server.ts @@ -0,0 +1 @@ +export * from '../../../deployments/[id]/resume/+server.js'; diff --git a/src/routes/api/provisioning/house-party/+server.ts b/src/routes/api/provisioning/house-party/+server.ts new file mode 100644 index 0000000..0ecc35f --- /dev/null +++ b/src/routes/api/provisioning/house-party/+server.ts @@ -0,0 +1 @@ +export * from '../../deployments/house-party/+server.js'; diff --git a/src/routes/api/provisioning/preview/+server.ts b/src/routes/api/provisioning/preview/+server.ts new file mode 100644 index 0000000..e46b526 --- /dev/null +++ b/src/routes/api/provisioning/preview/+server.ts @@ -0,0 +1 @@ +export * from '../../deployments/preview/+server.js'; diff --git a/src/routes/api/servers/[name]/+server.ts b/src/routes/api/servers/[name]/+server.ts index 6afc3ff..9b0bae8 100644 --- a/src/routes/api/servers/[name]/+server.ts +++ b/src/routes/api/servers/[name]/+server.ts @@ -9,7 +9,7 @@ export async function DELETE({ params }: { params: { name: string } }) { } const result = await destroyServer(name); if (!result.ok) { - const status = result.kind === 'not-found' ? 404 : 500; + const status = result.kind === 'not-found' ? 404 : result.kind === 'conflict' ? 409 : 500; return json({ ok: false, error: result.message, kind: result.kind, warnings: result.warnings }, { status }); } void kick().catch((err) => console.error('[destroy] worker kick failed:', err)); diff --git a/src/routes/api/system-status/+server.ts b/src/routes/api/system-status/+server.ts index 60ebb69..91f72aa 100644 --- a/src/routes/api/system-status/+server.ts +++ b/src/routes/api/system-status/+server.ts @@ -31,7 +31,7 @@ export async function GET({ url }: { url: URL }) { { id: 'rcon', title: 'Game server connection', state: !rconConfigured() ? 'unconfigured' : readyServers ? 'ready' : 'unverified', detail: !rconConfigured() ? 'Set RCON_PASSWORD to the password used by your game servers.' : readyServers ? `${readyServers} servers have fresh successful RCON reports.` : 'RCON password configured. A running server is needed to verify a response.', href: servers.length ? '/' : '#setup-guide', action: servers.length ? 'Review fleet' : 'Setup instructions' }, { id: 'game-files', title: 'Game files on the host', state: 'unverified', detail: 'T5 and T6 game-file volumes must be populated on the Komodo host. Their contents cannot be verified from this app.', href: '#game-files', action: 'Game-file instructions' }, { id: 'keys', title: 'Saved server keys', state: savedKeys ? 'ready' : 'attention', detail: `${savedKeys} saved ${savedKeys === 1 ? 'key' : 'keys'}. Key presence does not confirm validity with Plutonium.`, href: '/server-keys', action: savedKeys ? 'Manage keys' : 'Add a key' }, - { id: 'first-server', title: 'First server ready', state: readyServers ? 'ready' : 'attention', detail: readyServers ? `${readyServers} of ${servers.length} servers are responding.` : servers.length ? 'Servers are registered but none has a fresh healthy report. Open diagnostics to investigate.' : 'Create your first server after completing the setup checks.', href: servers.length ? '/' : '/deployments', action: servers.length ? 'Review fleet' : 'Create a server' } + { id: 'first-server', title: 'First server ready', state: readyServers ? 'ready' : 'attention', detail: readyServers ? `${readyServers} of ${servers.length} servers are responding.` : servers.length ? 'Servers are registered but none has a fresh healthy report. Open diagnostics to investigate.' : 'Create your first server after completing the setup checks.', href: servers.length ? '/' : '/provisioning', action: servers.length ? 'Review fleet' : 'Create a server' } ]; const body: SystemStatus = { checkedAt: new Date().toISOString(), checks, readyServers, totalServers: servers.length, savedKeys }; return json(body, { headers: { 'cache-control': 'no-store' } }); diff --git a/src/routes/deployments/[...path]/+page.server.ts b/src/routes/deployments/[...path]/+page.server.ts new file mode 100644 index 0000000..a8391ad --- /dev/null +++ b/src/routes/deployments/[...path]/+page.server.ts @@ -0,0 +1,6 @@ +import { redirect } from '@sveltejs/kit'; + +/** Preserve bookmarked operation links and list filters. */ +export function load({ url }: { url: URL }) { + redirect(308, url.pathname.replace(/^\/deployments(?=\/|$)/, '/provisioning') + url.search); +} diff --git a/src/routes/deployments/+page.svelte b/src/routes/provisioning/+page.svelte similarity index 91% rename from src/routes/deployments/+page.svelte rename to src/routes/provisioning/+page.svelte index c09721a..21e1352 100644 --- a/src/routes/deployments/+page.svelte +++ b/src/routes/provisioning/+page.svelte @@ -20,8 +20,12 @@ import HousePartyDialog from '$ui/components/HousePartyDialog.svelte'; import type { BlueprintSummary, CollectionSummary } from '$lib/deployment-plan.js'; + import LifecycleDialog from '$ui/components/LifecycleDialog.svelte'; + import { provisioningActions, actionLabels, actionCompleted, type ProvisioningAction } from '$lib/provisioning.js'; + + let lifecycleOpen = $state(false); interface Deployment { - id: string; server: string; action: 'create' | 'destroy'; + id: string; server: string; action: ProvisioningAction; state: 'queued' | 'awaiting-health' | 'waiting' | 'running' | 'done' | 'failed' | 'warning' | 'interrupted' | 'cancelled'; stage: string; started_at: string; finished_at: string | null; triggered_by: string; error: string | null; warnings: string; server_exists: number; @@ -51,7 +55,7 @@ $effect(() => { const params = $page.url.searchParams; tab = ['active', 'blueprints', 'history'].includes(params.get('view') ?? '') ? params.get('view')! : 'active'; - action = ['create', 'destroy'].includes(params.get('action') ?? '') ? params.get('action')! : 'all'; + action = provisioningActions.some(value => value === params.get('action')) ? params.get('action')! : 'all'; historyState = params.get('status') === 'attention' ? 'attention' : 'all'; search = params.get('server') ?? ''; limit = Math.max(25, Number(params.get('limit')) || 25); @@ -68,7 +72,7 @@ async function cancelRequests(ids: string[]) { cancelling = true; cancellationNotice = ''; let cancelled = 0; const unavailable: string[] = []; for (const id of ids) { - try { const response = await fetch(`/api/deployments/${encodeURIComponent(id)}/cancel`, { method: 'POST' }); if (response.ok) cancelled++; else unavailable.push(id); } + try { const response = await fetch(`/api/provisioning/${encodeURIComponent(id)}/cancel`, { method: 'POST' }); if (response.ok) cancelled++; else unavailable.push(id); } catch { unavailable.push(id); } } cancellationNotice = `${cancelled} queued removal${cancelled === 1 ? '' : 's'} cancelled.${cancelled ? ' The servers covered by those cancelled requests and their files were kept.' : ''}${unavailable.length ? ` ${unavailable.length} could not be cancelled; refresh their operations to see whether removal has begun.` : ''}`; @@ -90,8 +94,8 @@ if (refreshing) return; refreshing = true; try { - const [res, collectionRes] = await Promise.all([fetch('/api/deployments'), fetch('/api/deployment-collections')]); - if (!res.ok || !collectionRes.ok) throw new Error('Could not load server deployments.'); + const [res, collectionRes] = await Promise.all([fetch('/api/provisioning'), fetch('/api/deployment-collections')]); + if (!res.ok || !collectionRes.ok) throw new Error('Could not load provisioning activity.'); deployments = (await res.json()).deployments; collections = (await collectionRes.json()).collections; error = ''; @@ -126,12 +130,13 @@ function warnings(d: Deployment): string[] { try { return JSON.parse(d.warnings); } catch { return []; } } -Deployments | Electric Cherry +Provisioning | Electric Cherry
- - {#snippet actions()}
Fleet actions
{/snippet} + + {#snippet actions()}
Fleet actions
{/snippet} - saveContext(true)} ariaLabel="Deployment views" idPrefix="deployment-tabs" panelId="deployment-panel" /> + saveContext(true)} ariaLabel="Provisioning views" idPrefix="deployment-tabs" panelId="deployment-panel" /> + @@ -174,7 +179,7 @@

Collection launches

{#each visibleCollections.slice(0,collectionLimit) as collection (collection.id)} - +
{collection.name}{collection.active ? collection.provisioning + collection.verifying ? collection.mode === 'exact' ? 'Applying blueprint' : 'Launching' : 'Queued' : collection.failed ? 'Needs attention' : 'Completed'}
{collection.done} / {collection.total} {collection.mode === 'exact' ? 'completed' : 'ready'}{#if collection.provisioning + collection.verifying}{collection.provisioning + collection.verifying} {collection.mode === 'exact' ? 'in progress' : 'launching'}{/if}{#if collection.queued}{collection.queued} queued{/if}{#if collection.failed}{collection.failed} need attention{/if}
@@ -189,15 +194,15 @@ {#if active.some(d => d.canCancelRemoval)}
Waiting removals can be withdrawn until removal begins. Running and completed removals cannot be reversed.
{/if}

In flight {active.length}

- {#if loading}

Loading deployments...

- {:else if active.length === 0} + {#if loading}

Loading provisioning activity...

+ {:else if active.length === 0} {:else}
{#each active as d (d.id)}{@render row(d)}{/each}
{/if}
{:else}

History

- +
@@ -211,14 +216,14 @@
{#snippet row(d: Deployment)} -
+
-
{d.action === 'create' ? 'Creation' : 'Destruction'} +
{actionLabels[d.action]}
-

{d.state === 'done' ? (d.action === 'create' ? 'Server creation completed.' : 'Server destroyed.') : d.stage}

+

{d.state === 'done' ? actionCompleted[d.action] : d.stage}

Started {fmt(d.started_at)}{#if d.finished_at}Finished {fmt(d.finished_at)}{/if}By {d.triggered_by}
{#if d.error}

{d.error}

{/if} {#each warnings(d) as warning}

{warning}

{/each} diff --git a/src/routes/deployments/[id]/+page.server.ts b/src/routes/provisioning/[id]/+page.server.ts similarity index 89% rename from src/routes/deployments/[id]/+page.server.ts rename to src/routes/provisioning/[id]/+page.server.ts index ee55b8f..ee09dbf 100644 --- a/src/routes/deployments/[id]/+page.server.ts +++ b/src/routes/provisioning/[id]/+page.server.ts @@ -7,5 +7,5 @@ export function load({ params }: { params: { id: string } }) { if (db().prepare('SELECT id FROM deployments WHERE id = ?').get(params.id)) { redirect(307, '/campaigns/' + encodeURIComponent(params.id)); } - error(404, 'Deployment not found'); + error(404, 'Provisioning operation not found'); } diff --git a/src/routes/deployments/[id]/+page.svelte b/src/routes/provisioning/[id]/+page.svelte similarity index 67% rename from src/routes/deployments/[id]/+page.svelte rename to src/routes/provisioning/[id]/+page.svelte index 60ccaec..d8c9a39 100644 --- a/src/routes/deployments/[id]/+page.svelte +++ b/src/routes/provisioning/[id]/+page.svelte @@ -15,12 +15,14 @@ import LaunchRecipe from '$ui/components/LaunchRecipe.svelte'; import type { LaunchRecipeSummary } from '$lib/deployment-plan.js'; + import { actionLabels, actionCompleted, type ProvisioningAction } from '$lib/provisioning.js'; + interface Deployment { - id: string; expected_map?: string; server: string; action: 'create' | 'destroy'; state: string; + id: string; expected_map?: string; server: string; action: ProvisioningAction; state: string; stage: string; started_at: string; finished_at: string | null; triggered_by: string; error: string | null; warnings: string; server_exists: number; } - interface Detail { deployment: Deployment; events: { id: number; created_at: string; message: string }[]; recipe: LaunchRecipeSummary | null; blueprintName: string | null; canResumeVerification: boolean; canResumeCleanup: boolean; canCancelRemoval: boolean; collection: { id: string; name: string } | null } + interface Detail { deployment: Deployment; events: { id: number; created_at: string; message: string }[]; recipe: LaunchRecipeSummary | null; blueprintName: string | null; canResumeVerification: boolean; canResumeCleanup: boolean; canResumeRebuild: boolean; canCancelRemoval: boolean; collection: { id: string; name: string } | null } let detail = $state(null); let error = $state(''); let refreshing = false; @@ -32,25 +34,40 @@ let warnings = $derived.by(() => { try { return JSON.parse(deployment?.warnings ?? '[]') as string[]; } catch { return []; } }); - let steps = $derived(deployment?.action === 'destroy' - ? ['Request destruction', 'Wait for empty server', 'Stop and remove container', 'Delete Komodo deployment', 'Remove server files and registry'] - : ['Request creation', 'Prepare server files', 'Create Komodo deployment', 'Start container', 'Verify Komodo health', 'Verify RCON health', 'Verify selected map']); + const createSteps = ['Prepare server files', 'Create Komodo deployment', 'Start container', 'Verify Komodo health', 'Verify RCON health', 'Verify selected map']; + const createStages = ['Preparing server files', 'Creating Komodo deployment', 'Starting container', 'Waiting for Komodo health', 'Waiting for RCON health', 'Waiting for selected map']; + const destroySteps = ['Wait for empty server', 'Stop and remove container', 'Delete Komodo deployment', 'Remove server files and registry']; + const destroyStages = ['Waiting to verify', 'Stopping and removing container', 'Deleting Komodo deployment', 'Removing server files and registry']; + let steps = $derived(deployment?.action === 'destroy' ? ['Request destruction', ...destroySteps] + : deployment?.action === 'rebuild' ? ['Request rebuild', ...destroySteps, 'Queue fresh instance', ...createSteps] + : deployment?.action === 'create' ? ['Request creation', ...createSteps] + : ['Request operation', 'Wait for eligibility', `${deployment ? actionLabels[deployment.action] : 'Start'} container`, deployment?.action === 'stop' ? 'Verify container stopped' : 'Verify container and RCON health']); let currentStep = $derived.by(() => { if (!deployment) return 0; - const stages = deployment.action === 'destroy' - ? ['Starting', 'Waiting', 'Stopping and removing container', 'Deleting Komodo deployment', 'Removing server files and registry'] - : ['Starting', 'Preparing server files', 'Creating Komodo deployment', 'Starting container', 'Waiting for Komodo health', 'Waiting for RCON health', 'Waiting for selected map']; - if (deployment.state === 'waiting' && deployment.stage !== 'Removing server files and registry') return 1; - if (deployment.state === 'awaiting-health' && deployment.stage.startsWith('Waiting for health verification')) return 4; - if (deployment.state === 'done' || deployment.state === 'warning') return stages.length; - return Math.max(0, stages.findIndex(stage => deployment.stage.startsWith(stage))); + if (deployment.state === 'done' || deployment.state === 'warning') return steps.length; + const { action, state, stage } = deployment; + if (state === 'waiting' && stage !== 'Removing server files and registry') return 1; + if (action === 'start' || action === 'restart' || action === 'stop') return state === 'awaiting-health' ? 3 : state === 'running' ? 2 : stage.includes('container') ? 2 : 1; + if (state === 'queued') return action === 'rebuild' ? 5 : 0; + const stages = action === 'destroy' ? ['Starting', ...destroyStages] + : action === 'rebuild' ? ['Starting', ...destroyStages, 'Queued for server creation', ...createStages] + : ['Starting', ...createStages]; + const index = stages.findIndex(value => stage === value || stage.startsWith(value + ':')); + if (index < 0 && state === 'awaiting-health') return action === 'rebuild' ? 9 : 4; + return Math.max(0, index); }); let explanation = $derived.by(() => { if (!deployment) return ''; + if (detail?.canResumeRebuild) return 'The old Komodo instance was deleted. Resume file cleanup, then create a fresh instance in the reserved server slot.'; + if (['start', 'restart', 'stop'].includes(deployment.action)) { + if (deployment.state === 'done') return deployment.action === 'stop' ? 'Komodo confirmed the container is stopped. Start it again when ready.' : 'The container is healthy and RCON responds.'; + if (['failed', 'interrupted'].includes(deployment.state)) return 'Inspect the server before submitting another operation. An uncertain response may have reached Komodo.'; + return deployment.state === 'waiting' ? 'Waiting for other work and a safe server state before continuing.' : 'The request is saved. Verifying the resulting container state automatically.'; + } if (deployment.state === 'cancelled') return 'The queued removal was cancelled before removal began. The server and its files were kept.'; if (detail?.canResumeCleanup) return 'Komodo deletion succeeded. Resume the remaining file and database cleanup; the removed container does not need to answer RCON.'; if (deployment.state === 'waiting' && deployment.stage === 'Removing server files and registry') return 'Local cleanup is queued. Komodo deletion has already succeeded.'; - if (deployment.state === 'done') return deployment.action === 'create' + if (deployment.state === 'done') return ['create', 'rebuild'].includes(deployment.action) ? 'The container is healthy, RCON responds, and the reported engine map matches your selection.' : 'The container, deployment, and server registration have been removed.'; if (deployment.state === 'warning') return 'The operation finished with the warnings below. Review any remaining cleanup.'; @@ -58,11 +75,12 @@ ? 'The server is still registered. Inspect its status, then resume health verification without creating or restarting a container.' : 'Inspect the error and any remaining container or files before launching again. The activity below records how far this operation reached.'; if (deployment.state === 'queued') return 'Your request is saved in the launch queue. A startup slot opens after an earlier server passes its container, RCON, and map checks.'; + if (deployment.action === 'rebuild' && deployment.state === 'waiting') return 'Waiting until the instance is empty and a creation slot is available. The old instance and all of its files will be deleted before a fresh instance is created.'; if (deployment.state === 'waiting') return 'The server stays online while players finish. Removal begins only after a live check confirms it is empty.'; if (deployment.stage.startsWith('Waiting for selected map')) return 'Container and RCON checks passed. Waiting for RCON to report the selected engine map. Inspect the server configuration if this persists.'; if (deployment.stage.startsWith('Waiting for RCON')) return 'The container is healthy. Waiting for the game to answer RCON; check its startup logs, password, and network if this persists.'; if (deployment.state === 'awaiting-health') return 'Waiting for the container to become healthy. Verification runs automatically; startup logs can help explain a delay.'; - return deployment.action === 'create' ? 'Installing the latest promoted setup and starting its container. Health checks follow automatically.' : 'Removing the empty server and recording cleanup results.'; + return ['create', 'rebuild'].includes(deployment.action) ? 'Installing the latest promoted setup and starting its container. Health checks follow automatically.' : 'Removing the empty server and recording cleanup results.'; }); function duration(start: string, end: number) { const seconds = Math.max(0, Math.floor((end - new Date(start).getTime()) / 1000)); @@ -75,7 +93,7 @@ if (!deployment || resuming) return; resuming = true; try { - const res = await fetch(`/api/deployments/${encodeURIComponent(deployment.id)}/resume`, { method: 'POST' }); + const res = await fetch(`/api/provisioning/${encodeURIComponent(deployment.id)}/resume`, { method: 'POST' }); const body = await res.json(); if (!res.ok) throw new Error(body.error ?? 'Could not resume the operation.'); await load(); @@ -86,7 +104,7 @@ if (!deployment || resuming) return; resuming = true; error = ''; try { - const response = await fetch(`/api/deployments/${encodeURIComponent(deployment.id)}/cancel`, { method: 'POST' }); + const response = await fetch(`/api/provisioning/${encodeURIComponent(deployment.id)}/cancel`, { method: 'POST' }); const body = await response.json(); if (!response.ok) throw new Error(body.error ?? 'Could not cancel this request.'); await load(); @@ -98,8 +116,8 @@ refreshing = true; const id = $page.params.id ?? ''; try { - const res = await fetch('/api/deployments/' + encodeURIComponent(id)); - if (!res.ok) throw new Error(res.status === 404 ? 'Deployment not found.' : 'Could not refresh deployment.'); + const res = await fetch('/api/provisioning/' + encodeURIComponent(id)); + if (!res.ok) throw new Error(res.status === 404 ? 'Provisioning operation not found.' : 'Could not refresh provisioning operation.'); const body = await res.json(); if ($page.params.id === id) { detail = body; error = ''; } } catch (err) { error = (err as Error).message; } @@ -110,22 +128,23 @@ function fmt(iso: string) { return $timeFormat.dateTime(iso); } -{deployment ? (deployment.action === 'destroy' ? 'Destroy ' : 'Create ') + deployment.server : 'Deployment'} | Electric Cherry +{deployment ? actionLabels[deployment.action] + ' ' + deployment.server : 'Provisioning'} | Electric Cherry
- - {#snippet crumbs()}{/snippet} + + {#snippet crumbs()}{/snippet} {#if error}{/if} - {#if detail?.collection}← Back to {detail.collection.name} collection{/if} + {#if detail?.collection}← Back to {detail.collection.name} collection{/if} {#if deployment}
{#if deployment.server_exists}View server{/if}
-

{deployment.state === 'done' ? (deployment.action === 'destroy' ? 'Server destroyed' : 'Server creation completed') : deployment.stage}

+

{deployment.state === 'done' ? actionCompleted[deployment.action] : deployment.stage}

{explanation}

Total time {duration(deployment.started_at, deployment.finished_at ? new Date(deployment.finished_at).getTime() : now)}{#if active && stageStarted}This step {duration(stageStarted, now)}{/if}
{#if detail?.canResumeVerification}
{/if} + {#if detail?.canResumeRebuild}
{/if} {#if detail?.canResumeCleanup}
{/if} {#if detail?.canCancelRemoval}

Available until removal begins. Cancelling keeps the server and its files.

{/if} {#if deployment.server_exists && deployment.state !== 'done' && deployment.state !== 'cancelled'}
Open startup logsDiagnose connection{#if deployment.stage.includes('map')}Review map configuration{/if}
{/if} @@ -137,7 +156,7 @@ {#if detail?.recipe}

{detail.blueprintName ? `Launched from ${detail.blueprintName}` : 'Recorded launch files'}

Launch this map with the latest promoted files using a fresh server ID and port. This does not repair the existing server.

{/if} ({label,state:deployment.state === 'cancelled' ? 'pending' : index < currentStep ? 'done' : index === currentStep ? (!active && ['failed','interrupted'].includes(deployment.state) ? 'failed' : 'current') : 'pending'}))} events={detail?.events ?? []} {active} /> - {:else if !error}

Loading deployment...

{/if} + {:else if !error}

Loading provisioning operation...

{/if}