diff --git a/README.md b/README.md index 382a9d7..cc700c6 100644 --- a/README.md +++ b/README.md @@ -81,7 +81,7 @@ The schedule and jobs survive app restarts. Missed days become one run, and a server can have only one pending maintenance restart. Disabling the schedule does not cancel work already queued; use **Cancel** for those jobs. -Maintenance waits for a fresh zero-player RCON response and for existing +Maintenance waits for a fresh zero-player RCON or telemetry report and for existing deployment/beta work, then uses the shared rolling restart limit. Unreachable or occupied servers remain queued. After container and RCON recovery, Electric Cherry queries the running revision again. If a fresh published revision was @@ -100,6 +100,13 @@ behavior on your live T5/T6 servers before relying on unattended maintenance. ## How restarts stay safe +**Operations → Watchdog** automatically detects unhealthy servers and file drift. +Health incidents receive one restart followed by one rebuild if recovery fails; +drift queues a rebuild with the promoted setup and a linked Campaign. Incidents, +evidence, and attempts survive application restarts. Pause automation, inspect +failures, and explicitly retry from Watchdog. Intentional stops and active beta or +campaign work are respected. See [Watchdog](docs/watchdog.md) for timing and recovery behavior. + Betas support configurable gameplay, participation, stability and manual-review requirements, saved presets, coverage rules and staged waves. See [Beta test plans](docs/beta-test-plans.md) for usage and the four-event game telemetry contract. @@ -300,8 +307,10 @@ the full player data, updates Fleet, and applies gameplay progress to active betas automatically. No discovery request or beta-specific fields are needed. See [Game reporter protocol](docs/beta-test-plans.md#game-reporter-protocol) for -complete payload examples. Telemetry does not replace RCON's empty-server -check before an automated restart. +complete payload examples. Fresh telemetry can verify player count and game +activity alongside RCON during provisioning and Watchdog recovery. Occupied +reports prevent interruptions; stale reports cannot authorize them. Komodo still +verifies container readiness. For a missing scoreboard, open the server's **Diagnostics → Telemetry API requests**. The log shows received requests for that exact server name, including accepted diff --git a/docs/provisioning.md b/docs/provisioning.md index 206d3a5..63bd6d5 100644 --- a/docs/provisioning.md +++ b/docs/provisioning.md @@ -30,12 +30,16 @@ failures before making another request. Start and Restart share the rolling restart limit with campaigns, betas, and maintenance. A successful API reply is followed by a recovery delay and actual -container/RCON checks. Failed or ambiguous restart requests retain their recovery +container and game-status checks. Failed or ambiguous restart requests retain their recovery slot and are not automatically replayed. Stop requires an explicit stopped container state; an unreachable Komodo API is not evidence that a stop succeeded. -Disruptive operations require a fresh zero-player RCON result, unless Komodo -positively reports a stopped container. Runtime actions wait for conflicting +Disruptive operations require a fresh zero-player result from RCON or accepted +telemetry, unless Komodo positively reports a stopped container. Fresh occupied +telemetry vetoes a conflicting empty RCON reading. Reports must be received within +45 seconds; recovery verification also excludes reports from before the submitted +operation. Telemetry can verify the game's activity and selected map after Komodo +reports healthy. Runtime actions wait for conflicting campaign, beta, maintenance, or provisioning work. Duplicate pending requests for the same action return the existing operation; a different action returns a conflict. Requests and events survive application restarts. @@ -48,6 +52,12 @@ health verification. The callers retain file deployment, beta evidence, and Plutonium revision verification. Jobs remain visible while waiting for players, other work, or integration configuration. +[Watchdog](watchdog.md) also submits recovery operations and links each attempt +to its incident. A failed Watchdog restart can hand its reserved slot to a rebuild +after five minutes; unrelated recovery slots remain held. Drift campaigns follow +their linked rebuild without requesting another restart. Their file evidence and +history remain available after the instance is removed. + Campaign changes queued before submission can share one restart. Changes arriving after submission require a new operation. A failed or interrupted request keeps its original ID and never automatically replays. An explicit beta retry starts a diff --git a/docs/watchdog.md b/docs/watchdog.md new file mode 100644 index 0000000..b83f876 --- /dev/null +++ b/docs/watchdog.md @@ -0,0 +1,80 @@ +# Watchdog + +**Operations → Watchdog** detects unhealthy servers and drift, records each +incident, and links its recovery attempts to Provisioning. File drift also opens +a Campaign with the observed file differences. Automation is enabled by default; +**Pause automation** persists across application restarts. Scanning and verification +continue while paused, but new actions and queued Watchdog actions are held. +Already submitted actions continue to completion. + +The worker scans once per minute while Electric Cherry is running. **Scan now** +requests a scan immediately. Repeated button presses do not count as independent +health failures. A health incident must have three observations spanning at least +two minutes before an automatic restart is queued. Starting or restarting +containers get five minutes before recovery is requested. Unknown Komodo responses +are recorded without interpreting them as a stopped container. + +## Recovery + +A health incident gets one restart, then one rebuild if the restart has not +recovered within five minutes. Both are normal Provisioning jobs with durable +history and shared capacity limits. Escalation transfers only that incident's +restart recovery slot to the rebuild. An unsuccessful rebuild requires operator +attention; Watchdog does not start an endless restart/rebuild loop. Verification +of an already submitted rebuild continues, including after the incident is marked +as needing attention. + +An intentional **Stop** is retained as the server's desired state. Watchdog leaves +it stopped until an explicit Start, Restart, or Rebuild request changes that intent. +Active creations, removals, betas, and campaigns defer new Watchdog actions. + +Provisioning checks current container health before using game status. A disruptive +action requires a confirmed stopped container or a fresh zero-player reading from +RCON or accepted telemetry. A positive player count from either source prevents +an interruption. Telemetry evidence must have been received within 45 seconds; +future, malformed, stale, and pre-restart reports cannot verify a new process. +A game-end event retains its reported roster; it does not imply zero players. +Fresh telemetry can also verify game activity and the selected engine map after +Komodo reports the container healthy. A healthy container with neither current +RCON status nor current telemetry remains queued for player-count verification. + +## Drift + +Drift includes missing promoted files, changed bytes, and extra files in managed +script, config, and plugin destinations. Config comparison uses the composed +baseline and overlays with the server's selected map. Plutainer's unmanaged default +config remains exempt until a baseline is promoted. Active betas are deferred; +missing server mounts are reported as scan errors instead of being treated as +empty directories. + +Watchdog rebuilds a drifting server in place using the same identity, map, port, +and saved server key. **Rebuild deletes all old instance files.** The new instance +uses the latest promoted setup when provisioning starts. A missing saved key +leaves the incident needing attention without deleting the old server. + +The linked Campaign captures before/after evidence at detection time. Its activity +identifies this as the detected plan, and it follows the rebuild's outcome without +queuing an additional restart. Promotions can change while the rebuild is queued; +the Provisioning launch recipe records the files actually selected for that launch. +Campaign evidence and Watchdog history survive subsequent rebuilds and removals. + +## Review and retry + +The list can be filtered by server, incident type, and status. Each detail page +shows the triggering evidence, timestamped events, and links to every attempt. +**Dismiss** suppresses further automatic attempts until the condition clears. +**Retry** explicitly starts a new bounded attempt after inspecting a dismissed or +failed incident. Active Provisioning operations must finish or be cancelled first. +A verified recovery resolves the incident automatically. + +## API + +- `GET /api/watchdog`: automation setting, last scan, scan errors, and latest incidents. +- `POST /api/watchdog` with `{ "action": "scan" }`: run a scan now. +- `POST /api/watchdog` with `{ "enabled": false }` (or `true`): pause/resume automation. +- `GET /api/watchdog/:id`: incident, event history, and linked actions. +- `POST /api/watchdog/:id` with `{ "action": "retry" }` or `{ "action": "dismiss" }`. + +The legacy `/api/drift` GET and dry-run POST still return previews. Mutating POST +requests return HTTP 410 and direct clients to Watchdog; file repair is now a +provisioning rebuild instead of an immediate rewrite. diff --git a/src/lib/server/db/index.ts b/src/lib/server/db/index.ts index 9cf579b..4f6e45b 100644 --- a/src/lib/server/db/index.ts +++ b/src/lib/server/db/index.ts @@ -16,6 +16,7 @@ let _db: Database.Database | null = null; // backfill columns; this can. const SERVERS_COLUMNS: Array<{ name: string; ddl: string }> = [ { name: 'container_state', ddl: 'ALTER TABLE servers ADD COLUMN container_state TEXT' }, + { name: 'desired_state', ddl: "ALTER TABLE servers ADD COLUMN desired_state TEXT NOT NULL DEFAULT 'running' CHECK (desired_state IN ('running','stopped'))" }, // Base columns that shipped in the very first schema stay here for parity // with fresh installs — no ALTER is issued if they're already present. { name: 'komodo_server', ddl: 'ALTER TABLE servers ADD COLUMN komodo_server TEXT' }, @@ -246,6 +247,44 @@ function migrateProvisioningActions(handle: Database.Database): void { } finally { handle.pragma('foreign_keys = ON'); } } +/** Rebuild constrained tables while preserving links, indexes, triggers, and row order. */ +function migrateWatchdogConstraints(handle: Database.Database): void { + for (const name of ['provisioning_sources', 'deployment_servers']) { + const { sql } = handle.prepare("SELECT sql FROM sqlite_master WHERE type = 'table' AND name = ?").get(name) as { sql: string }; + const replacement = name === 'provisioning_sources' + ? (sql.includes("'watchdog'") ? sql : sql.replace("'campaign','beta','maintenance'", "'campaign','beta','maintenance','watchdog'")) + : sql.replace(/server\s+TEXT NOT NULL REFERENCES servers\(name\)/, 'server TEXT NOT NULL'); + if (sql === replacement) continue; + const dependent = handle.prepare("SELECT sql FROM sqlite_master WHERE tbl_name = ? AND type IN ('index','trigger') AND sql IS NOT NULL") + .all(name) as { sql: string }[]; + handle.pragma('foreign_keys = OFF'); + try { + handle.transaction(() => { + handle.exec(replacement.replace(name, name + '_watchdog_new')); + handle.exec(`INSERT INTO ${name}_watchdog_new (rowid, ${ + (handle.prepare(`PRAGMA table_info(${name})`).all() as { name: string }[]).map(col => col.name).join(', ') + }) SELECT rowid, ${ + (handle.prepare(`PRAGMA table_info(${name})`).all() as { name: string }[]).map(col => col.name).join(', ') + } FROM ${name}; DROP TABLE ${name}; ALTER TABLE ${name}_watchdog_new RENAME TO ${name};`); + for (const object of dependent) handle.exec(object.sql); + if ((handle.pragma('foreign_key_check') as unknown[]).length) throw new Error('Watchdog migration found broken references'); + })(); + } finally { handle.pragma('foreign_keys = ON'); } + } +} + +function backfillDesiredState(handle: Database.Database): void { + if (handle.prepare("SELECT 1 FROM settings WHERE key = 'watchdog_desired_state_v1'").get()) return; + handle.transaction(() => { + // A failed stop may have reached Komodo. Preserve that explicit intent. + handle.exec(`UPDATE servers SET desired_state = 'stopped' WHERE 'stop' = ( + SELECT action FROM server_deployments d WHERE d.server = servers.name AND d.state != 'cancelled' + AND d.action IN ('create','start','restart','rebuild','stop') + ORDER BY d.rowid DESC LIMIT 1); + INSERT INTO settings (key, value) VALUES ('watchdog_desired_state_v1', '1');`); + })(); +} + export function db(): Database.Database { if (_db) return _db; @@ -260,6 +299,7 @@ export function db(): Database.Database { dropLegacyBetaTables(handle); handle.exec(schemaSql); migrateProvisioningActions(handle); + migrateWatchdogConstraints(handle); migrateTable(handle, 'restart_rollout', [{ name: 'operation_id', ddl: 'ALTER TABLE restart_rollout ADD COLUMN operation_id TEXT' }]); migrateTable(handle, 'server_deployments', [ { name: 'submitted_at', ddl: 'ALTER TABLE server_deployments ADD COLUMN submitted_at TEXT' }, @@ -270,6 +310,7 @@ export function db(): Database.Database { migrateTable(handle, table, [{ name: 'provisioning_id', ddl: `ALTER TABLE ${table} ADD COLUMN provisioning_id TEXT` }]); } migrateServers(handle); + backfillDesiredState(handle); // Column additions must run BEFORE any statement that references the // new columns. schema.sql cannot include those references — see the // comment there. diff --git a/src/lib/server/db/schema.sql b/src/lib/server/db/schema.sql index e6fefb5..f42f9a9 100644 --- a/src/lib/server/db/schema.sql +++ b/src/lib/server/db/schema.sql @@ -64,6 +64,7 @@ CREATE TABLE IF NOT EXISTS servers ( komodo_name TEXT NOT NULL, komodo_server TEXT, container_state TEXT, -- last observed Komodo runtime state + desired_state TEXT NOT NULL DEFAULT 'running' CHECK (desired_state IN ('running','stopped')), game_port INTEGER, -- UDP RCON port; lazy-filled from Komodo rcon_host TEXT, -- optional per-server override of RCON_HOST game TEXT, -- 't5' etc.; static per-server (no map rotation) @@ -299,7 +300,7 @@ CREATE INDEX IF NOT EXISTS idx_deployments_started ON deployments(started_at DES CREATE TABLE IF NOT EXISTS deployment_servers ( deployment_id TEXT NOT NULL REFERENCES deployments(id) ON DELETE CASCADE, - server TEXT NOT NULL REFERENCES servers(name), + server TEXT NOT NULL, -- Campaign observation survives a rebuild's registry gap. files_changed INTEGER NOT NULL, restart_state TEXT NOT NULL, -- pending | done | failed | manual | held players INTEGER, @@ -445,7 +446,7 @@ CREATE TABLE IF NOT EXISTS server_rebuild_jobs ( -- history retention, while current pointers live on the caller's server rows. CREATE TABLE IF NOT EXISTS provisioning_sources ( operation_id TEXT NOT NULL REFERENCES server_deployments(id) ON DELETE CASCADE, - kind TEXT NOT NULL CHECK (kind IN ('campaign','beta','maintenance')), + kind TEXT NOT NULL CHECK (kind IN ('campaign','beta','maintenance','watchdog')), source_id TEXT NOT NULL, server TEXT NOT NULL, PRIMARY KEY (operation_id, kind, source_id, server) @@ -622,3 +623,36 @@ CREATE TABLE IF NOT EXISTS metric_samples ( capacity INTEGER, PRIMARY KEY (at, server) ); + +-- Watchdog history deliberately survives server destruction/rebuild. +CREATE TABLE IF NOT EXISTS watchdog_incidents ( + id TEXT PRIMARY KEY, + server TEXT NOT NULL, + kind TEXT NOT NULL CHECK (kind IN ('health','drift')), + state TEXT NOT NULL CHECK (state IN ('open','recovering','attention','resolved','dismissed')), + summary TEXT NOT NULL, + details TEXT NOT NULL DEFAULT '{}', + first_seen_at TEXT NOT NULL, + last_seen_at TEXT NOT NULL, + finished_at TEXT, + failures INTEGER NOT NULL DEFAULT 1, + cycle INTEGER NOT NULL DEFAULT 1 +); +CREATE UNIQUE INDEX IF NOT EXISTS idx_watchdog_active ON watchdog_incidents(server,kind) WHERE state != 'resolved'; +CREATE TABLE IF NOT EXISTS watchdog_events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + incident_id TEXT NOT NULL REFERENCES watchdog_incidents(id) ON DELETE CASCADE, + created_at TEXT NOT NULL, + message TEXT NOT NULL +); +CREATE INDEX IF NOT EXISTS idx_watchdog_events ON watchdog_events(incident_id,id); +CREATE TABLE IF NOT EXISTS watchdog_actions ( + id TEXT PRIMARY KEY, + incident_id TEXT NOT NULL REFERENCES watchdog_incidents(id) ON DELETE CASCADE, + action TEXT NOT NULL CHECK (action IN ('restart','rebuild')), + operation_id TEXT NOT NULL UNIQUE, + campaign_id TEXT, + created_at TEXT NOT NULL, + cycle INTEGER NOT NULL DEFAULT 1, + UNIQUE(incident_id,cycle,action) +); diff --git a/src/lib/server/fleet/drift.ts b/src/lib/server/fleet/drift.ts index 52acf37..2d46cc2 100644 --- a/src/lib/server/fleet/drift.ts +++ b/src/lib/server/fleet/drift.ts @@ -1,22 +1,20 @@ import { mkdir, readdir, readFile, rm, stat, writeFile } from 'node:fs/promises'; import { randomUUID } from 'node:crypto'; import type { DriftPlan, DriftPlanFile } from '$lib/drift-plan.js'; -import { existsSync } from 'node:fs'; import { dirname, join, posix, relative, sep } from 'node:path'; -import { configFileOf, destOfKind, type ArtifactKind } from '../config.js'; +import { configFileOf, destOfKind, serverDir, type ArtifactKind } from '../config.js'; import { db } from '../db/index.js'; import { emit } from '../pipeline/bus.js'; import { sha256Buffer } from './files.js'; import { getServerScript, - listServerScripts, recordScript, removeScript } from './inventory.js'; import { getBaselineForScope, getVersion } from './scripts.js'; import { getPromotedServerConfig, getServerMapConfig, MAP_CONFIG_FILE } from './server-config.js'; import { selectedZombieMap } from './map-aliases.js'; -import { serverDirectories, getServer, listServers } from './servers.js'; +import { getServer, listServers } from './servers.js'; import { promotedArtifacts } from './promoted-artifacts.js'; // Destinations drift scans, in a stable order so the UI shows scripts first. @@ -54,14 +52,35 @@ function toPosix(p: string): string { return p.split(sep).join(posix.sep); } +function hasActiveBeta(server: string): boolean { + return !!db().prepare(`SELECT 1 FROM beta_servers bs JOIN betas b ON b.id = bs.beta_id + WHERE bs.server = ? AND b.state IN ('running', 'ready-promote') LIMIT 1`).get(server); +} + +async function requireServerDirectory(server: string): Promise { + try { + if (!(await stat(serverDir(server))).isDirectory()) throw new Error('path is not a directory'); + } catch (error) { + throw new Error(`Server files unavailable for ${server}: ${(error as Error).message}`); + } +} + +async function readExistingFile(path: string): Promise { + return readFile(path).catch((error: NodeJS.ErrnoException) => { + if (error.code === 'ENOENT') return null; + throw error; + }); +} + async function walkFiles(root: string): Promise { const out: string[] = []; async function recur(dir: string): Promise { let entries; try { entries = await readdir(dir, { withFileTypes: true }); - } catch { - return; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return; + throw error; } for (const e of entries) { if (DRIFT_IGNORE_FILENAMES.has(e.name)) continue; @@ -76,27 +95,28 @@ async function walkFiles(root: string): Promise { /** * Scan every managed destination tree on a server (scripts, configs, plugins) - * and return every file that isn't tracked in server_scripts. During a beta, - * the beta bytes live at the actual target filename and the inventory row is - * updated to reflect them, so beta files register as tracked with no - * special-casing (pre-beta bytes are snapshotted to beta_file_snapshots for - * rollback rather than left on disk as .OLD). - * - * The inventory table (server_scripts) keys on (server, rel_path) with no - * destination discriminator. Rel_paths across destinations don't collide in - * practice (`.gsc` for scripts, `.cfg` for configs, `.dll` for plugins), so - * a shared tracked-set is safe. + * and compare disk contents with promoted artifacts and inventory. An active + * beta owns its server's files until promotion or rollback; defer the whole + * server so reconciliation cannot replace experimental config or missing beta + * artifacts with the promoted versions. */ export async function scanServerDrift(server: string): Promise { - const tracked = new Set(listServerScripts(server).map((r) => r.rel_path)); - const drift = await scanManagedForServer(server); - const managed = new Set(drift.map((entry) => `${entry.kind}:${entry.relPath}`)); const srv = getServer(server); + if (!srv || srv.destroyed_at || hasActiveBeta(server)) return []; + // A missing mount is unavailable evidence, not an empty managed tree. + await requireServerDirectory(server); + const drift = await scanManagedForServer(server); + // Previous inventory is an audit of installed bytes, not permission to + // retain artifacts whose scope was moved or removed from the library. + const managed = new Set(promotedArtifacts(srv.game, srv.map) + .map(({ kind, script }) => `${kind}:${script.target_rel_path}`)); + const baseline = srv.game ? getBaselineForScope('game', srv.game) : undefined; + if (baseline?.current_version_id) managed.add(`config:${configFileOf(server)}`); // Without a promoted baseline, Plutainer owns the seeded config. Keep it // until a baseline is promoted; startup arguments handle map selection. const keepDefaultConfig = srv?.game && selectedZombieMap(srv.game, srv.map, srv.map_slug) - && !getBaselineForScope('game', srv.game)?.current_version_id; + && !baseline?.current_version_id; // Scan for untracked files only after resolving managed targets. for (const kind of DRIFT_DESTS) { @@ -105,7 +125,7 @@ export async function scanServerDrift(server: string): Promise { if (onDisk.length === 0) continue; for (const relPath of onDisk) { - if (tracked.has(relPath) || managed.has(`${kind}:${relPath}`)) continue; + if (managed.has(`${kind}:${relPath}`)) continue; if (keepDefaultConfig && kind === 'config' && relPath === configFileOf(server)) continue; if (kind === 'config' && relPath === MAP_CONFIG_FILE && srv?.created_by && srv.game_mode === 'zombie' && getServerMapConfig(srv.game, srv.map, srv.map_slug)) continue; @@ -120,13 +140,15 @@ export async function scanServerDrift(server: string): Promise { kind, driftKind: 'untracked' }); - } catch { - /* file disappeared between walk and stat — skip */ + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + // A file disappeared between walk and stat; the next scan sees it. } } } - return drift; + await requireServerDirectory(server); + return hasActiveBeta(server) ? [] : drift; } // Every promoted artifact whose scope matches this server's game/map should @@ -143,21 +165,23 @@ async function scanManagedForServer(server: string): Promise { ? getServerMapConfig(srv.game, srv.map, srv.map_slug) : null; if (mapConfig) { const abs = join(destOfKind(server, 'config'), MAP_CONFIG_FILE); - const content = await readFile(abs).catch(() => null); + const content = await readExistingFile(abs); if (!content?.equals(mapConfig)) { out.push({ server, relPath: MAP_CONFIG_FILE, kind: 'config', driftKind: content ? 'replace' : 'missing', size: mapConfig.byteLength, sha256: sha256Buffer(mapConfig) }); } } - // Existing files without inventory must be installed/tracked in one pass, - // never deleted first. Preserve tracked files (including beta versions). + // Inventory records the last managed write, not the current disk bytes. + // Check both so edits and outdated versions are detected on tracked files. for (const { kind, script, version } of promotedArtifacts(srv.game, srv.map)) { - const present = existsSync(join(destOfKind(server, kind), script.target_rel_path)); - if (present && getServerScript(server, script.target_rel_path)) continue; + const content = await readExistingFile(join(destOfKind(server, kind), script.target_rel_path)); + const inventory = getServerScript(server, script.target_rel_path); + if (content?.equals(version.content) && inventory?.sha256 === version.sha256 + && inventory.script_version_id === version.id) continue; out.push({ server, relPath: script.target_rel_path, kind, - driftKind: present ? 'replace' : 'missing', + driftKind: content ? 'replace' : 'missing', size: version.size, sha256: version.sha256, scriptId: script.id, scriptVersionId: version.id }); @@ -176,13 +200,14 @@ async function scanManagedForServer(server: string): Promise { const targetRel = configFileOf(server); const abs = join(dest, targetRel); const inv = getServerScript(server, targetRel); - const onDiskMatches = existsSync(abs) && inv?.sha256 === composed.sha256; + const content = await readExistingFile(abs); + const onDiskMatches = content?.equals(composed.content) && inv?.sha256 === composed.sha256; if (!onDiskMatches) { out.push({ server, relPath: targetRel, kind: 'config', - driftKind: existsSync(abs) ? 'replace' : 'missing', + driftKind: content ? 'replace' : 'missing', size: composed.content.byteLength, sha256: composed.sha256, scriptId: baseline.id, @@ -230,12 +255,8 @@ function composeExpectedCfg( */ export async function scanFleetDrift(): Promise { const servers = listServers().map((s) => s.name); - const registered = new Set(servers); - const allDirs = new Set(await serverDirectories()); - // Only touch server dirs that are both registered AND present on disk. - const eligible = servers.filter((n) => registered.has(n) && allDirs.has(n)); const out: DriftEntry[] = []; - for (const name of eligible) { + for (const name of servers) { const rows = await scanServerDrift(name); out.push(...rows); } @@ -271,6 +292,10 @@ async function describePlan(servers: string[], entries: DriftEntry[]): Promise `${a.server}:${a.kind}:${a.relPath}`.localeCompare(`${b.server}:${b.kind}:${b.relPath}`)); return { servers, files, scannedAt: new Date().toISOString(), fingerprint: sha256Buffer(Buffer.from(JSON.stringify({ servers, files }))) }; } @@ -298,6 +323,62 @@ function deploymentIdNow(): string { return `dep-${stamp}-drift-${randomUUID().slice(0, 8)}`; } +export interface PreparedDriftCampaign { + plan: DriftPlan; + files: Array<{ file: DriftPlanFile; before: Buffer | null; after: Buffer | null }>; +} + +/** Capture a rebuild's file evidence without changing the running instance. */ +export async function prepareDriftCampaign(server: string): Promise { + const servers = driftTargets([server]); + const entries = await scanServerDrift(server); + const files: PreparedDriftCampaign['files'] = []; + for (const entry of entries) { + const before = await readExistingFile(join(destOfKind(server, entry.kind), entry.relPath)); + const after = entry.driftKind === 'untracked' ? null : await resolveManagedEntry(server, entry); + if (entry.driftKind !== 'untracked' && !after) throw new Error(`Cannot resolve promoted content for ${entry.relPath}`); + files.push({ before, after, file: { server, relPath: entry.relPath, kind: entry.kind, driftKind: entry.driftKind, + beforeSha256: before ? sha256Buffer(before) : null, afterSha256: after ? sha256Buffer(after) : null, + beforeSize: before?.byteLength ?? null, afterSize: after?.byteLength ?? null } }); + } + await requireServerDirectory(server); + if (hasActiveBeta(server)) throw new Error('Active beta owns this server; reconciliation deferred'); + return { plan: buildPlan(servers, files.map(({ file }) => file)), files }; +} + +/** Record the audited plan atomically with a queued rebuild, never a second restart. */ +export function createDriftRebuildCampaign( + server: string, operationId: string, incidentId: string | number, prepared: PreparedDriftCampaign +): string | null { + if (!prepared.files.length) return null; + if (prepared.plan.servers.length !== 1 || prepared.plan.servers[0] !== server + || prepared.files.some(({ file }) => file.server !== server)) throw new Error('Drift evidence belongs to another server'); + if (hasActiveBeta(server)) throw new Error('Active beta owns this server; reconciliation deferred'); + return db().transaction(() => { + if (!db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND server = ? AND action = 'rebuild'").get(operationId, server)) { + throw new Error('Drift campaign requires the matching rebuild operation'); + } + const id = deploymentIdNow(); + db().prepare("INSERT INTO deployments (id, action, started_at, triggered_by) VALUES (?, 'drift', ?, 'watchdog')") + .run(id, new Date().toISOString()); + const insertFile = db().prepare(`INSERT INTO deployment_files + (deployment_id, server, verb, rel_path, sha256_before, sha256_after, content_before, content_after) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`); + for (const { file, before, after } of prepared.files) { + insertFile.run(id, server, after === null ? 'removed' : before === null ? 'added' : 'updated', + file.relPath, file.beforeSha256, file.afterSha256, before, after); + } + db().prepare(`INSERT INTO deployment_servers (deployment_id, server, files_changed, restart_state, provisioning_id) + VALUES (?, ?, ?, 'pending', ?)`).run(id, server, prepared.files.length, operationId); + db().prepare("INSERT INTO provisioning_sources (operation_id, kind, source_id, server) VALUES (?, 'campaign', ?, ?)") + .run(operationId, id, server); + db().prepare('INSERT INTO campaign_events (deployment_id, message) VALUES (?, ?)') + .run(id, `Watchdog ${incidentId}: captured drift evidence; rebuild ${operationId} will install the latest promoted setup. Files remain unchanged until provisioning rebuilds the instance.`); + emit({ type: 'deployment-opened', deploymentId: id, action: 'drift', servers: [server] }); + return id; + })(); +} + /** * Detect drift on the requested servers (or the whole fleet) and either * report the plan (dryRun) or remove untracked files and install managed @@ -368,10 +449,19 @@ export async function runDriftDeployment(input: RunDriftInput = {}): Promise file.server === s.server && file.kind === entry.kind && file.relPath === entry.relPath); if (reviewed) { const current = await readFile(abs).catch((error: NodeJS.ErrnoException) => { if (error.code === 'ENOENT') return null; throw error; }); @@ -383,6 +473,7 @@ export async function runDriftDeployment(input: RunDriftInput = {}): Promise ONLINE_MAX_MS) return null; + if (after && (!Number.isFinite(Date.parse(after)) || received <= Date.parse(after))) return null; + try { + const event = parseGameEvent(JSON.parse(row.payload)); + if (event.server !== server) return null; + return { players: event.players.length, map: event.map, receivedAt: row.received_at }; + } catch { return null; } +} + +/** Positive occupancy from either source always vetoes an empty-server claim. */ +export async function probePlayerCount(server: string): Promise { + const players = await rconGetPlayerCount(server).catch(() => null); + // Read after the network request: a player may join while RCON is pending. + const telemetry = telemetryEvidence(server); + if (players !== null && players > 0) return players; + if (telemetry && telemetry.players > 0) return telemetry.players; + return players === 0 || telemetry?.players === 0 ? 0 : null; +} + +/** Komodo readiness remains a caller prerequisite. Old-process telemetry is excluded. */ +export async function probeServerStatus(server: string, after?: string | null): Promise< + { ok: true; players: number; map: string | null; source: 'rcon' | 'telemetry' } | { ok: false; reason: string } +> { + let reason = 'RCON unavailable'; + try { + const rcon = await rconStatusResult(server); + if (rcon.ok) return { ok: true, players: Math.max(rcon.status.players, telemetryEvidence(server, after)?.players ?? 0), map: rcon.status.map, source: 'rcon' }; + reason = rcon.reason; + } catch (err) { reason = (err as Error).message; } + const telemetry = telemetryEvidence(server, after); + return telemetry ? { ok: true, ...telemetry, source: 'telemetry' } : { ok: false, reason }; +} diff --git a/src/lib/server/pipeline/sweep.ts b/src/lib/server/pipeline/sweep.ts index c7acac7..f45705f 100644 --- a/src/lib/server/pipeline/sweep.ts +++ b/src/lib/server/pipeline/sweep.ts @@ -41,7 +41,8 @@ async function runSweep(): Promise { } db().prepare(`UPDATE deployments SET finished_at = ? WHERE finished_at IS NULL AND NOT EXISTS ( SELECT 1 FROM deployment_servers WHERE deployment_id = deployments.id AND restart_state IN ('pending','failed','held'))`).run(nowIso()); - db().prepare(`DELETE FROM deployments WHERE started_at < ? AND NOT EXISTS ( + db().prepare(`DELETE FROM deployments WHERE started_at < ? + AND NOT EXISTS (SELECT 1 FROM watchdog_actions WHERE campaign_id = deployments.id) AND NOT EXISTS ( SELECT 1 FROM deployment_servers WHERE deployment_id = deployments.id AND restart_state IN ('pending','failed','held'))`) .run(new Date(Date.now() - config.deployKeepDays * 86_400_000).toISOString()); emit({ type: 'sweep-tick', frozen: false }); diff --git a/src/lib/server/pipeline/worker.ts b/src/lib/server/pipeline/worker.ts index 2c396c4..203ad6c 100644 --- a/src/lib/server/pipeline/worker.ts +++ b/src/lib/server/pipeline/worker.ts @@ -14,6 +14,7 @@ import { STATUS_REFRESH_MS } from '../../server-freshness.js'; import { recordMetrics } from '../fleet/metrics.js'; import { selectedZombieMap } from '../../map-aliases.js'; import { maintenanceTick } from '../fleet/plutonium.js'; +import { watchdogTick } from '../watchdog/index.js'; const TICK_MS = 15_000; // Cap concurrent UDP RCON calls so a 15-server fleet doesn't blast the @@ -36,6 +37,7 @@ async function reconcileRequesters(): Promise { try { await sweep(); } catch (err) { console.error('[worker] campaign error:', err); } try { await betaTick(); } catch (err) { console.error('[worker] beta error:', err); } try { await maintenanceTick(); } catch (err) { console.error('[worker] maintenance error:', err); } + try { await watchdogTick(); } catch (err) { console.error('[worker] watchdog error:', err); } } async function runTick(): Promise { diff --git a/src/lib/server/provisioning/index.ts b/src/lib/server/provisioning/index.ts index d1d0af9..bf5d7c4 100644 --- a/src/lib/server/provisioning/index.ts +++ b/src/lib/server/provisioning/index.ts @@ -1,6 +1,6 @@ // Request/observation boundary used by APIs, campaigns, beta tests, and schedules. // Execution is driven exclusively by the provisioning worker. -export { requestLifecycle, requestRestart, provisioningOperation, cancelProvisioning, releaseRestartRequest } from './requests.js'; +export { requestLifecycle, requestRestart, requestWatchdogLifecycle, escalateWatchdogRestart, provisioningOperation, cancelProvisioning, releaseRestartRequest } from './requests.js'; export type { LifecycleOperation, RestartSource } from './requests.js'; export { queueCreateServer, destroyServer, previewCreation, creationQueue, nextAvailablePort, nextLobbyNumber } from './instances.js'; export { getDeployment, listDeployments, resumeVerification, resumeCleanup, resumeRebuild, cancelRemoval } from './history.js'; diff --git a/src/lib/server/provisioning/instances.ts b/src/lib/server/provisioning/instances.ts index 7db5f75..48d33dd 100644 --- a/src/lib/server/provisioning/instances.ts +++ b/src/lib/server/provisioning/instances.ts @@ -1,4 +1,4 @@ -import { rconGetPlayerCount, rconStatusResult } from '../integrations/rcon.js'; +import { probePlayerCount, probeServerStatus } from '../fleet/status-evidence.js'; import { beginDeployment, deploymentStage, finishDeployment, recordDeploymentEvent, REMOVING_SERVER, resumeCleanup, resumeVerification, claimRemoval } from './history.js'; import { randomUUID } from 'node:crypto'; import type { LaunchPreview } from '$lib/deployment-plan.js'; @@ -229,7 +229,7 @@ export function queueCreateServer(input: CreateServerInput, releasingServers: st } /** Reserve the same identity and port before removing any part of the instance. */ -export function queueRebuildServer(name: string): CreateServerResult { +export function queueRebuildServer(name: string, triggeredBy = 'ui'): CreateServerResult { const server = getServer(name); const game = gameOfServer(name); const map = server?.map_slug && zombieMapBySlug(game, server.map_slug); @@ -237,12 +237,12 @@ export function queueRebuildServer(name: string): CreateServerResult { || (server.host_dir && server.host_dir !== serverDir(name))) { return { ok: false, kind: 'bad-request', message: 'Rebuild requires a managed server with its original map, directory, and port.' }; } - if (!config.komodo.targetServer || !config.rcon.password) return { ok: false, kind: 'unconfigured', message: 'Configure the Komodo target host and RCON before rebuilding.' }; + if (!config.komodo.targetServer || (!config.rcon.password && !triggeredBy.startsWith('watchdog:'))) return { ok: false, kind: 'unconfigured', message: 'Configure the Komodo target host and RCON before rebuilding.' }; if (!getServerKey(name)) return { ok: false, kind: 'bad-request', message: `Save a server key for ${name} in Keys before rebuilding.` }; // Validate that the replacement setup can be composed before destruction. const recipe = captureRecipe(game, map.slug); return db().transaction(() => { - const id = beginDeployment(name, 'rebuild'); + const id = beginDeployment(name, 'rebuild', triggeredBy); db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); db().prepare('INSERT INTO server_creation_jobs (deployment_id, server, port, game_mode, map_slug, expected_map) VALUES (?, ?, ?, ?, ?, ?)') .run(id, name, server.game_port, 'zombie', map.slug, map.canonical); @@ -436,6 +436,7 @@ async function createServerUnlocked(input: CreateServerInput, prepared: Prepared // Step 5: execute/Deploy — actually start the container. deploymentStage(operation, 'Starting container'); + db().prepare('UPDATE server_deployments SET submitted_at = ? WHERE id = ?').run(new Date().toISOString(), operation); const deployRes = await komodoDeploy(name); if (!deployRes.ok) { if (deployRes.kind === 'unreachable' || (deployRes.httpStatus !== undefined && deployRes.httpStatus >= 500)) { @@ -541,7 +542,7 @@ async function runDestructions(): Promise { continue; } // A fresh successful zero is required; cached occupancy and timeouts are not evidence of emptiness. - let players = await rconGetPlayerCount(operation.server); + let players = await probePlayerCount(operation.server); if (players === null) { const health = await komodoInspectHealthy(operation.server); if (['stopped','exited','created','dead'].includes(health.state?.toLowerCase() ?? '')) players = 0; @@ -621,8 +622,13 @@ async function cleanupDestroyedServer(name: string, operation: string): Promise< // Test evidence cascades from runs; inventory and snapshots cascade from servers. try { const tx = db().transaction((n: string) => { - db().prepare('DELETE FROM deployment_servers WHERE server = ?').run(n); - db().prepare('DELETE FROM deployment_files WHERE server = ?').run(n); + // The Watchdog's rebuild campaign observes this exact operation + // through the registry gap and retains its before/after file audit. + for (const table of ['deployment_servers', 'deployment_files']) { + db().prepare(`DELETE FROM ${table} WHERE server = ? AND deployment_id NOT IN ( + SELECT campaign_id FROM watchdog_actions WHERE campaign_id IS NOT NULL)`) + .run(n); + } db().prepare('DELETE FROM beta_servers WHERE server = ?').run(n); db().prepare('DELETE FROM beta_server_scripts WHERE server = ?').run(n); db().prepare('DELETE FROM beta_test_runs WHERE server = ?').run(n); @@ -647,6 +653,7 @@ async function cleanupDestroyedServer(name: string, operation: string): Promise< } interface CreationJob { deployment_id: string; server: string; port: number; game_mode: GameMode; map_slug: string; expected_map: string; triggered_by?: string; + submitted_at?: string | null; } async function provisionCreation(id: string): Promise { @@ -739,7 +746,7 @@ async function runCreations(): Promise { // Check only previously started servers. Newly launched servers keep their // slot until a later pass, even if a health probe would answer immediately. - const checking = db().prepare(`SELECT j.* FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id + const checking = db().prepare(`SELECT j.*, COALESCE(d.submitted_at, d.started_at) AS submitted_at FROM server_creation_jobs j JOIN server_deployments d ON d.id = j.deployment_id WHERE d.state = 'awaiting-health' ORDER BY d.started_at, d.rowid`).all() as CreationJob[]; for (const job of checking) { const id = job.deployment_id; @@ -751,14 +758,14 @@ async function runCreations(): Promise { try { const komodo = await komodoInspectHealthy(job.server); if (!komodo.ok) { deploymentStage(id, 'Waiting for Komodo health: ' + komodo.reason); continue; } - const rcon = await rconStatusResult(job.server); - if (!rcon.ok) { deploymentStage(id, 'Waiting for RCON health: ' + rcon.reason); continue; } - const actualMap = rcon.status.map; + const status = await probeServerStatus(job.server, job.submitted_at); + if (!status.ok) { deploymentStage(id, 'Waiting for RCON or fresh telemetry health: ' + status.reason); continue; } + const actualMap = status.map; if (actualMap !== job.expected_map) { - deploymentStage(id, 'Waiting for selected map: expected ' + job.expected_map + ', RCON reports ' + (actualMap ?? 'no map')); + deploymentStage(id, 'Waiting for selected map: expected ' + job.expected_map + ', ' + (status.source === 'rcon' ? 'RCON' : 'telemetry') + ' reports ' + (actualMap ?? 'no map')); continue; } - deploymentStage(id, 'Verified Komodo healthy, RCON healthy, and map ' + actualMap); + deploymentStage(id, 'Verified Komodo healthy, ' + (status.source === 'rcon' ? 'RCON healthy' : 'fresh telemetry') + ', and map ' + actualMap); finishDeployment(id, { ok: true }); } catch (err) { deploymentStage(id, 'Waiting for health verification: ' + (err as Error).message); diff --git a/src/lib/server/provisioning/requests.ts b/src/lib/server/provisioning/requests.ts index f8614a7..5901794 100644 --- a/src/lib/server/provisioning/requests.ts +++ b/src/lib/server/provisioning/requests.ts @@ -1,14 +1,16 @@ import { db } from '../db/index.js'; import { komodoConfigured, rconConfigured } from '../config.js'; import { getServer } from '../fleet/servers.js'; -import { beginDeployment, deploymentStage, recordDeploymentEvent } from './history.js'; +import { beginDeployment, deploymentStage, finishDeployment, recordDeploymentEvent } from './history.js'; import { queueRebuildServer } from './instances.js'; +import { restartRequestInFlight } from './restarts.js'; import { runtimeActions, type RuntimeAction } from '../../provisioning.js'; export interface LifecycleOperation { id: string; server: string; action: string; state: string; stage: string; submitted_at: string | null; finished_at: string | null; checked_at: string | null; players: number | null; error: string | null; + started_at: string; } export interface RestartSource { kind: 'campaign' | 'beta' | 'maintenance'; id: string } export function provisioningOperation(id: string): LifecycleOperation | undefined { @@ -115,19 +117,131 @@ export function requestLifecycle(server: string, action: RuntimeAction) { return db().transaction(() => { const existing = db().prepare(`SELECT id, action FROM server_deployments WHERE server = ? AND state IN ('queued','waiting','running','awaiting-health') ORDER BY rowid DESC LIMIT 1`).get(server) as { id: string; action: string } | undefined; - if (existing) return existing.action === action - ? { ok: true as const, deploymentId: existing.id } - : { ok: false as const, status: 409, error: 'Another provisioning operation is already pending for this server.' }; + if (existing) { + if (existing.action !== action) return { ok: false as const, status: 409, error: 'Another provisioning operation is already pending for this server.' }; + db().prepare('UPDATE servers SET desired_state = ? WHERE name = ?').run(action === 'stop' ? 'stopped' : 'running', server); + return { ok: true as const, deploymentId: existing.id }; + } const row = getServer(server); if (!row || row.destroyed_at) return { ok: false as const, status: 404, error: 'Server not found.' }; if (action === 'rebuild') { const result = queueRebuildServer(server); + if (result.ok) db().prepare("UPDATE servers SET desired_state = 'running' WHERE name = ?").run(server); return result.ok ? { ok: true as const, deploymentId: result.deploymentId } : { ok: false as const, status: 400, error: result.message }; } const id = beginDeployment(server, action); + db().prepare('UPDATE servers SET desired_state = ? WHERE name = ?').run(action === 'stop' ? 'stopped' : 'running', server); db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); deploymentStage(id, action === 'start' ? 'Waiting to start the container' : 'Waiting to verify the server is empty'); return { ok: true as const, deploymentId: id }; })(); } + +type LifecycleRequestResult = { ok: true; deploymentId: string } | { ok: false; status: number; error: string }; + +function watchdogRequestGuard(server: string, incidentId: string): Extract | null { + const incident = db().prepare("SELECT 1 FROM watchdog_incidents WHERE id = ? AND server = ? AND state IN ('open','recovering')").get(incidentId, server); + if (!incident) return { ok: false, status: 409, error: 'The Watchdog incident is no longer active.' }; + if (db().prepare("SELECT 1 FROM settings WHERE key = 'watchdog_enabled' AND value = '0'").get()) { + return { ok: false, status: 409, error: 'Watchdog is paused.' }; + } + const row = getServer(server); + if (!row || row.destroyed_at) return { ok: false, status: 404, error: 'Server not found.' }; + if (row.desired_state === 'stopped') return { ok: false, status: 409, error: 'This server is intentionally stopped.' }; + if (!komodoConfigured()) return { ok: false, status: 503, error: 'Configure Komodo before provisioning.' }; + return null; +} + +function existingWatchdogAction(incidentId: string, action: 'restart' | 'rebuild'): string | undefined { + return (db().prepare(`SELECT a.operation_id FROM watchdog_actions a JOIN watchdog_incidents i ON i.id = a.incident_id + WHERE a.incident_id = ? AND a.action = ? AND a.cycle = i.cycle`).get(incidentId, action) as { operation_id: string } | undefined)?.operation_id; +} + +function queueWatchdogAction(server: string, action: 'restart' | 'rebuild', incidentId: string): LifecycleRequestResult { + let id: string; + if (action === 'rebuild') { + const result = queueRebuildServer(server, `watchdog:${incidentId}`); + if (!result.ok) return { ok: false, status: 400, error: result.message }; + id = result.deploymentId; + } else { + id = beginDeployment(server, action, `watchdog:${incidentId}`); + db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(id); + deploymentStage(id, 'Waiting to verify the server is empty'); + } + db().prepare("INSERT INTO provisioning_sources (operation_id, kind, source_id, server) VALUES (?, 'watchdog', ?, ?)").run(id, incidentId, server); + return { ok: true, deploymentId: id }; +} + +/** Watchdog submits durable work; Provisioning remains the only execution owner. */ +export function requestWatchdogLifecycle(server: string, action: 'restart' | 'rebuild', incidentId: string): LifecycleRequestResult { + return db().transaction(() => { + const previous = existingWatchdogAction(incidentId, action); + if (previous) return { ok: true as const, deploymentId: previous }; + const error = watchdogRequestGuard(server, incidentId); + if (error) return error; + const pending = db().prepare(`SELECT d.id, d.action, s.source_id FROM server_deployments d + LEFT JOIN provisioning_sources s ON s.operation_id = d.id AND s.kind = 'watchdog' + WHERE d.server = ? AND d.state IN ('queued','waiting','running','awaiting-health') ORDER BY d.rowid DESC LIMIT 1`) + .get(server) as { id: string; action: string; source_id: string | null } | undefined; + if (pending) return pending.source_id === incidentId && pending.action === action + ? { ok: true as const, deploymentId: pending.id } + : { ok: false as const, status: 409, error: 'Another provisioning operation is already pending for this server.' }; + const next = queueWatchdogAction(server, action, incidentId); + if (!next.ok) return next; + // An operator retry starts a new incident cycle. Keep the previous + // failed attempt's capacity, but let that new attempt use it instead + // of waiting forever behind its own incident's terminal operation. + const retained = db().prepare(`SELECT r.operation_id FROM restart_rollout r + JOIN server_deployments d ON d.id = r.operation_id + JOIN watchdog_actions a ON a.operation_id = d.id AND a.incident_id = ? + JOIN watchdog_incidents i ON i.id = a.incident_id + WHERE r.server = ? AND r.healthy_at IS NULL AND d.state IN ('failed','interrupted','cancelled') + AND a.cycle < i.cycle`).get(incidentId, server) as { operation_id: string } | undefined; + if (retained && !restartRequestInFlight(server)) { + db().prepare('UPDATE restart_rollout SET operation_id = ? WHERE server = ? AND operation_id = ? AND healthy_at IS NULL') + .run(next.deploymentId, server, retained.operation_id); + recordDeploymentEvent(retained.operation_id, 'Recovery capacity handed to explicit Watchdog retry ' + next.deploymentId); + recordDeploymentEvent(next.deploymentId, 'Retained recovery capacity from prior incident attempt ' + retained.operation_id); + } + return next; + })(); +} + +/** Atomically reserve a replacement before handing it the failed restart's capacity. */ +export function escalateWatchdogRestart(incidentId: string, operationId: string, recoveryTimeoutMs = 300_000): LifecycleRequestResult { + return db().transaction(() => { + const previous = existingWatchdogAction(incidentId, 'rebuild'); + if (previous) return { ok: true as const, deploymentId: previous }; + const operation = provisioningOperation(operationId); + if (!operation || operation.action !== 'restart' || !['awaiting-health','failed'].includes(operation.state) + || !db().prepare("SELECT 1 FROM provisioning_sources WHERE operation_id = ? AND kind = 'watchdog' AND source_id = ?").get(operationId, incidentId)) { + return { ok: false as const, status: 409, error: 'The restart is not eligible for Watchdog escalation.' }; + } + const error = watchdogRequestGuard(operation.server, incidentId); + if (error) return error; + const pending = db().prepare(`SELECT d.id, d.action, s.source_id FROM server_deployments d + LEFT JOIN provisioning_sources s ON s.operation_id = d.id AND s.kind = 'watchdog' + WHERE d.server = ? AND d.id != ? AND d.state IN ('queued','waiting','running','awaiting-health') LIMIT 1`) + .get(operation.server, operationId) as { id: string; action: string; source_id: string | null } | undefined; + if (pending) return pending.action === 'rebuild' && pending.source_id === incidentId + ? { ok: true as const, deploymentId: pending.id } + : { ok: false as const, status: 409, error: 'Another provisioning operation is already pending for this server.' }; + const slot = db().prepare('SELECT operation_id, requested_at, healthy_at FROM restart_rollout WHERE server = ?').get(operation.server) as + { operation_id: string | null; requested_at: string; healthy_at: string | null } | undefined; + const anchor = Date.parse(slot?.requested_at ?? operation.submitted_at ?? operation.started_at); + if (restartRequestInFlight(operation.server) || (slot && (slot.operation_id !== operationId || slot.healthy_at)) + || !Number.isFinite(anchor) || Date.now() - anchor < Math.max(300_000, recoveryTimeoutMs)) { + return { ok: false as const, status: 409, error: 'Waiting for the submitted restart to finish its recovery window.' }; + } + const replacement = queueWatchdogAction(operation.server, 'rebuild', incidentId); + if (!replacement.ok) return replacement; + const reason = 'Watchdog restart recovery timed out; handed recovery to rebuild ' + replacement.deploymentId; + if (operation.state === 'awaiting-health') finishDeployment(operationId, { ok: false, message: reason }); + else recordDeploymentEvent(operationId, reason); + if (slot) db().prepare('UPDATE restart_rollout SET operation_id = ?, error = ? WHERE server = ? AND operation_id = ? AND healthy_at IS NULL') + .run(replacement.deploymentId, reason, operation.server, operationId); + recordDeploymentEvent(replacement.deploymentId, 'Recovery handed off from restart ' + operationId + '; waiting for a fresh empty or stopped server'); + return replacement; + })(); +} diff --git a/src/lib/server/provisioning/restarts.ts b/src/lib/server/provisioning/restarts.ts index 146abb3..da1bf5f 100644 --- a/src/lib/server/provisioning/restarts.ts +++ b/src/lib/server/provisioning/restarts.ts @@ -1,13 +1,14 @@ import { config } from '../config.js'; import { db } from '../db/index.js'; import { komodoInspectHealthy, komodoRestart, komodoRuntimeAction } from '../integrations/komodo.js'; -import { rconStatusResult } from '../integrations/rcon.js'; +import { probeServerStatus } from '../fleet/status-evidence.js'; import { deploymentStage, finishDeployment } from './history.js'; // Komodo queues restart work. Don't mistake the still-running old container // for a completed restart when an API kick immediately follows the request. const HEALTH_POLL_MS = 15_000; const requestsInFlight = new Set(); +export const restartRequestInFlight = (server: string) => requestsInFlight.has(server); interface RestartSlot { server: string; @@ -41,11 +42,14 @@ export function rolloutWaitReason(server: string, operationId = ''): string | nu return 'Waiting for the current provisioning operation'; } const slot = slotFor(server); - if (slot?.healthy_at === null) { + const ownHandoff = slot?.operation_id === operationId && !!db().prepare(`SELECT 1 FROM server_deployments d + WHERE d.id = ? AND d.action IN ('restart','rebuild') AND EXISTS ( + SELECT 1 FROM provisioning_sources s WHERE s.operation_id = d.id AND s.kind = 'watchdog')`).get(operationId); + if (slot?.healthy_at === null && !ownHandoff) { return `Waiting for the previous restart of this server to recover${slot.error ? `: ${slot.error}` : ''}`; } const { n } = db().prepare('SELECT COUNT(*) AS n FROM restart_rollout WHERE healthy_at IS NULL').get() as { n: number }; - return n >= config.restartConcurrency ? 'Queued for a rolling restart; waiting for another server to recover' : null; + return n - (ownHandoff && slot?.healthy_at === null ? 1 : 0) >= config.restartConcurrency ? 'Queued for a rolling restart; waiting for another server to recover' : null; } export type RolloutRestartResult = @@ -76,7 +80,7 @@ export async function restartInRollout(server: string, options: { operationId: s const result = options.action === 'start' ? await komodoRuntimeAction(server, 'start') : await komodoRestart(server); if (result.ok) { db().prepare("UPDATE server_deployments SET state = 'awaiting-health' WHERE id = ?").run(operationId); - deploymentStage(operationId!, 'Waiting for container and RCON health'); + deploymentStage(operationId!, 'Waiting for container and game health (RCON or fresh telemetry)'); return { kind: 'started' }; } // These failures cannot have queued a remote restart. Other failures @@ -115,6 +119,13 @@ async function refresh(): Promise { .all() as RestartSlot[]; for (const slot of slots) { if (requestsInFlight.has(slot.server)) continue; + // Recovery capacity transferred to a rebuild remains reserved until + // teardown. Its health is verified by the creation worker, never here. + if (slot.operation_id && db().prepare(`SELECT 1 FROM server_deployments d WHERE d.id = ? AND ( + (d.action = 'rebuild' AND d.state IN ('waiting','queued','running','awaiting-health')) + OR (d.action = 'restart' AND d.state = 'waiting' AND d.submitted_at IS NULL AND EXISTS ( + SELECT 1 FROM provisioning_sources s WHERE s.operation_id = d.id AND s.kind = 'watchdog')))`) + .get(slot.operation_id)) continue; if (Date.now() - Date.parse(slot.checked_at ?? slot.requested_at) < HEALTH_POLL_MS) continue; let error: string | null = null; let players: number | null = null; @@ -123,22 +134,23 @@ async function refresh(): Promise { if (!health.ok) { error = `unhealthy: ${health.reason}${health.snippet ? ` snippet=${health.snippet.slice(0, 400)}` : ''}`; } else { - const rcon = await rconStatusResult(slot.server); - if (!rcon.ok) error = `rcon not answering: ${rcon.reason}`; - else players = rcon.status.players; + const status = await probeServerStatus(slot.server, slot.requested_at); + if (!status.ok) error = `rcon not answering and no fresh telemetry: ${status.reason}`; + else players = status.players; } } catch (err) { error = `health check: ${(err as Error).message}`; } const now = new Date().toISOString(); - db().prepare(`UPDATE restart_rollout SET checked_at = ?, healthy_at = ?, players = ?, error = ? - WHERE server = ? AND requested_at = ? AND healthy_at IS NULL`) - .run(now, error ? null : now, players, error, slot.server, slot.requested_at); + const checked = db().prepare(`UPDATE restart_rollout SET checked_at = ?, healthy_at = ?, players = ?, error = ? + WHERE server = ? AND requested_at = ? AND operation_id IS ? AND healthy_at IS NULL`) + .run(now, error ? null : now, players, error, slot.server, slot.requested_at, slot.operation_id); + if (!checked.changes) continue; if (slot.operation_id && db().prepare("SELECT 1 FROM server_deployments WHERE id = ? AND state = 'awaiting-health'").get(slot.operation_id)) { db().prepare('UPDATE server_deployments SET checked_at = ?, players = ? WHERE id = ?').run(now, players, slot.operation_id); if (error) deploymentStage(slot.operation_id, 'Waiting for health: ' + error); else { - deploymentStage(slot.operation_id, 'Verified container and RCON health'); + deploymentStage(slot.operation_id, 'Verified container and game health (RCON or fresh telemetry)'); finishDeployment(slot.operation_id, { ok: true }); } } diff --git a/src/lib/server/provisioning/worker.ts b/src/lib/server/provisioning/worker.ts index 1ef4b28..1db5798 100644 --- a/src/lib/server/provisioning/worker.ts +++ b/src/lib/server/provisioning/worker.ts @@ -4,7 +4,7 @@ import { getServer } from '../fleet/servers.js'; import { deploymentStage, finishDeployment, REMOVING_SERVER } from './history.js'; import { processCreations, processDestructions, rebuildServer } from './instances.js'; import { komodoInspectHealthy, komodoRuntimeAction } from '../integrations/komodo.js'; -import { rconGetPlayerCount } from '../integrations/rcon.js'; +import { probePlayerCount } from '../fleet/status-evidence.js'; import { refreshRestartRollout, restartInRollout, rolloutWaitReason } from './restarts.js'; import { cancelProvisioning } from './requests.js'; import type { RuntimeAction } from '../../provisioning.js'; @@ -12,14 +12,20 @@ import type { RuntimeAction } from '../../provisioning.js'; const stoppedStates = new Set(['stopped', 'exited', 'created', 'dead']); export const isStopped = (state?: string) => stoppedStates.has(state?.toLowerCase() ?? ''); -interface Source { kind: 'campaign' | 'beta' | 'maintenance'; source_id: string } +interface Source { kind: 'campaign' | 'beta' | 'maintenance' | 'watchdog'; source_id: string } const sourcesFor = (id: string) => db().prepare('SELECT kind, source_id FROM provisioning_sources WHERE operation_id = ?').all(id) as Source[]; -const priority = (sources: Source[]) => sources.some(s => s.kind === 'beta') ? 0 : sources.some(s => s.kind === 'campaign') ? 1 : sources.some(s => s.kind === 'maintenance') ? 3 : 2; +const priority = (sources: Source[]) => sources.some(s => s.kind === 'watchdog') ? 2 : sources.some(s => s.kind === 'beta') ? 0 : sources.some(s => s.kind === 'campaign') ? 1 : sources.some(s => s.kind === 'maintenance') ? 3 : 2; /** Source work may hold or withdraw a request, but never executes it. */ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled' { const sources = sourcesFor(operation.id); if (!sources.length) return 'ready'; + // Watchdog intent gates every linked source, including its drift campaign. + for (const source of sources.filter(source => source.kind === 'watchdog')) { + if (!db().prepare("SELECT 1 FROM watchdog_incidents WHERE id = ? AND server = ? AND state IN ('open','recovering')").get(source.source_id, operation.server) + || getServer(operation.server)?.desired_state === 'stopped') return 'cancelled'; + if (db().prepare("SELECT 1 FROM settings WHERE key = 'watchdog_enabled' AND value = '0'").get()) return 'held'; + } let held = false; for (const source of sources) { if (source.kind === 'campaign') { @@ -36,10 +42,10 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled' if (!['awaiting-empty','restarting','awaiting-health'].includes(row.stage)) continue; if (!row.outcome || ['active','passed'].includes(row.outcome)) return 'ready'; if (row.outcome === 'paused') held = true; - } else { + } else if (source.kind === 'maintenance') { const row = db().prepare('SELECT state, provisioning_id FROM maintenance_jobs WHERE id = ?').get(source.source_id) as { state: string; provisioning_id: string } | undefined; if (row?.provisioning_id === operation.id && ['queued','restarting'].includes(row.state)) return 'ready'; - } + } else return 'ready'; } return held ? 'held' : 'cancelled'; } @@ -61,7 +67,8 @@ function blocker(server: string, id: string): string | null { if (rank > 1 && db().prepare(`SELECT 1 FROM deployment_servers ds WHERE ds.server = ? AND ds.restart_state IN ('pending','failed','held') AND (ds.provisioning_id IS NULL OR EXISTS ( SELECT 1 FROM server_deployments d WHERE d.id = ds.provisioning_id AND d.state IN ('queued','waiting','running','awaiting-health'))) - LIMIT 1`).get(server)) return 'Waiting for campaign work on this server'; + AND NOT EXISTS (SELECT 1 FROM watchdog_actions a WHERE a.operation_id = ? AND a.campaign_id = ds.deployment_id) + LIMIT 1`).get(server, id)) return 'Waiting for campaign work on this server'; return rolloutWaitReason(server, id); } @@ -93,14 +100,18 @@ async function run(): Promise { async function processOperation(operation: Operation): Promise { const { id, server, action } = operation; + // An explicit cleanup resume follows confirmed remote deletion. Complete + // that durable checkpoint even if the incident now needs attention or the + // Watchdog is paused; never strand the reserved replacement identity. + if (action === 'rebuild' && operation.stage === REMOVING_SERVER) { await rebuildServer(id); return; } if (operation.state === 'waiting') { const eligibility = sourceEligibility(operation); if (eligibility === 'cancelled') { cancelProvisioning(id, 'Requester cancelled or replaced this operation'); return; } if (eligibility === 'held') { deploymentStage(id, 'Waiting for the requester to resume'); return; } } - if (action === 'rebuild' && operation.stage === REMOVING_SERVER) { await rebuildServer(id); return; } if (!getServer(server)) { finishDeployment(id, { ok: false, message: 'Server is no longer registered.' }); return; } - if (!komodoConfigured() || !rconConfigured()) { deploymentStage(id, 'Waiting for Komodo and RCON configuration'); return; } + const watchdog = sourcesFor(id).some(source => source.kind === 'watchdog'); + if (!komodoConfigured() || (!rconConfigured() && !watchdog)) { deploymentStage(id, 'Waiting for Komodo and RCON configuration'); return; } if (operation.state === 'awaiting-health') { const health = await komodoInspectHealthy(server); if (isStopped(health.state)) { @@ -114,7 +125,7 @@ async function processOperation(operation: Operation): Promise { if (reason) { deploymentStage(id, reason); return; } const health = await komodoInspectHealthy(server); if (action !== 'start' && !isStopped(health.state)) { - const players = await rconGetPlayerCount(server); + const players = await probePlayerCount(server); db().prepare('UPDATE server_deployments SET players = ?, checked_at = ? WHERE id = ?').run(players, new Date().toISOString(), id); if (players !== 0) { deploymentStage(id, players === null ? 'Waiting for a fresh player count' : `Waiting for ${players} player(s) to leave`); diff --git a/src/lib/server/watchdog/index.ts b/src/lib/server/watchdog/index.ts new file mode 100644 index 0000000..dbfe91a --- /dev/null +++ b/src/lib/server/watchdog/index.ts @@ -0,0 +1,233 @@ +import { randomUUID } from 'node:crypto'; +import { db } from '../db/index.js'; +import { komodoConfigured } from '../config.js'; +import { getServer, listServers } from '../fleet/servers.js'; +import { komodoInspectHealthy } from '../integrations/komodo.js'; +import { rconStatusResult } from '../integrations/rcon.js'; +import { telemetryEvidence } from '../fleet/status-evidence.js'; +import { prepareDriftCampaign, createDriftRebuildCampaign, previewDrift } from '../fleet/drift.js'; +import { requestWatchdogLifecycle, escalateWatchdogRestart } from '../provisioning/requests.js'; +import type { WatchdogAction, WatchdogDetail, WatchdogEvidence, WatchdogIncident, WatchdogOverview, WatchdogState } from '../../watchdog.js'; + +export const WATCHDOG_SCAN_MS = 60_000; +export const WATCHDOG_FAILURE_MS = 120_000; +export const WATCHDOG_RECOVERY_MS = 300_000; +const iso = () => new Date().toISOString(); +const pendingStates = ['queued', 'waiting', 'running', 'awaiting-health']; +const setting = (key: string) => (db().prepare('SELECT value FROM settings WHERE key = ?').get(key) as { value: string } | undefined)?.value ?? null; +function setSetting(key: string, value: string) { db().prepare('INSERT INTO settings(key,value) VALUES (?,?) ON CONFLICT(key) DO UPDATE SET value=excluded.value').run(key, value); } +export const watchdogEnabled = () => setting('watchdog_enabled') !== '0'; +export function setWatchdogEnabled(enabled: boolean) { setSetting('watchdog_enabled', enabled ? '1' : '0'); } + +function incident(id: string) { return db().prepare('SELECT * FROM watchdog_incidents WHERE id = ?').get(id) as WatchdogIncident | undefined; } +function active(server: string, kind: 'health' | 'drift') { return db().prepare("SELECT * FROM watchdog_incidents WHERE server = ? AND kind = ? AND state != 'resolved'").get(server, kind) as WatchdogIncident | undefined; } +function event(id: string, message: string) { db().prepare('INSERT INTO watchdog_events(incident_id,created_at,message) VALUES (?,?,?)').run(id, iso(), message); } +function transition(id: string, state: WatchdogState, summary: string) { + const previous = incident(id); + if (!previous || (previous.state === state && previous.summary === summary)) return; + db().prepare('UPDATE watchdog_incidents SET state=?, summary=?, finished_at=? WHERE id=?') + .run(state, summary, state === 'resolved' || state === 'dismissed' ? iso() : null, id); + event(id, summary); +} +function detect(server: string, kind: 'health' | 'drift', summary: string, details: WatchdogEvidence): WatchdogIncident { + const previous = active(server, kind); + if (previous) { + // Repeated UI scans must not manufacture a sustained failure. + const newSample = Date.now() - Date.parse(previous.last_seen_at) >= WATCHDOG_SCAN_MS; + const actionable = (details as WatchdogEvidence & { actionable?: boolean }).actionable; + const priorActionable = (JSON.parse(previous.details) as { actionable?: boolean }).actionable; + const reset = kind === 'health' && actionable === true && priorActionable !== true; + db().prepare('UPDATE watchdog_incidents SET details=?, last_seen_at=?, failures=?, first_seen_at=? WHERE id=?') + .run(JSON.stringify(details), newSample || reset ? iso() : previous.last_seen_at, + reset ? 1 : previous.failures + (newSample ? 1 : 0), reset ? iso() : previous.first_seen_at, previous.id); + if (previous.state === 'open') transition(previous.id, 'open', summary); + return incident(previous.id)!; + } + const id = randomUUID(); + db().prepare(`INSERT INTO watchdog_incidents(id,server,kind,state,summary,details,first_seen_at,last_seen_at) + VALUES (?,?,?,'open',?,?,?,?)`).run(id, server, kind, summary, JSON.stringify(details), iso(), iso()); + event(id, summary); + return incident(id)!; +} +function actions(id: string): WatchdogAction[] { + return db().prepare(`SELECT a.*, d.state AS operation_state, d.stage AS operation_stage, d.error AS operation_error + FROM watchdog_actions a LEFT JOIN server_deployments d ON d.id=a.operation_id + WHERE a.incident_id=? ORDER BY a.rowid`).all(id) as WatchdogAction[]; +} +export function watchdogOverview(): WatchdogOverview { + return { enabled: watchdogEnabled(), lastScanAt: setting('watchdog_last_scan'), scanError: setting('watchdog_scan_error') || null, + incidents: db().prepare(`SELECT * FROM watchdog_incidents WHERE state != 'resolved' OR id IN ( + SELECT id FROM watchdog_incidents WHERE state = 'resolved' ORDER BY first_seen_at DESC, rowid DESC LIMIT 100) + ORDER BY CASE WHEN state='resolved' THEN 1 ELSE 0 END, first_seen_at DESC, rowid DESC`).all() as WatchdogIncident[] }; +} +export function watchdogDetail(id: string): WatchdogDetail | null { + const row = incident(id); + return row ? { incident: row, actions: actions(id), events: db().prepare('SELECT id,created_at,message FROM watchdog_events WHERE incident_id=? ORDER BY id').all(id) as WatchdogDetail['events'] } : null; +} +function hasPending(server: string) { + return !!db().prepare("SELECT 1 FROM server_deployments WHERE server=? AND state IN ('queued','waiting','running','awaiting-health') LIMIT 1").get(server); +} +function protectedWork(server: string): boolean { + return !!db().prepare(`SELECT 1 FROM beta_servers bs JOIN betas b ON b.id=bs.beta_id WHERE bs.server=? AND b.state IN ('running','ready-promote') LIMIT 1`).get(server) + || !!db().prepare(`SELECT 1 FROM deployment_servers ds WHERE ds.server=? AND ds.restart_state IN ('pending','held','failed','manual') + AND NOT EXISTS (SELECT 1 FROM watchdog_actions a WHERE a.campaign_id=ds.deployment_id) LIMIT 1`).get(server); +} +function addAction(row: WatchdogIncident, operationId: string, action: 'restart' | 'rebuild', campaignId: string | null = null) { + db().prepare('INSERT INTO watchdog_actions(id,incident_id,action,operation_id,campaign_id,created_at,cycle) VALUES (?,?,?,?,?,?,?)') + .run(randomUUID(), row.id, action, operationId, campaignId, iso(), row.cycle); + transition(row.id, 'recovering', action === 'restart' ? 'Restart queued; waiting for safe recovery' : 'Rebuild queued with the latest promoted setup'); + event(row.id, `${action === 'restart' ? 'Restart' : 'Rebuild'} requested: ${operationId}${campaignId ? `; campaign ${campaignId}` : ''}`); +} +export function updateWatchdogIncident(id: string, action: 'retry' | 'dismiss') { + return db().transaction(() => { + const row = incident(id); + if (!row) return { ok: false as const, status: 404, error: 'Watchdog incident not found.' }; + if (actions(id).some(a => pendingStates.includes(a.operation_state ?? ''))) return { ok: false as const, status: 409, error: 'Provisioning is still active. Complete or cancel that operation first.' }; + if (action === 'dismiss') { + if (!['open','attention'].includes(row.state)) return { ok: false as const, status: 409, error: 'Only detected incidents or incidents needing attention can be dismissed.' }; + transition(id, 'dismissed', 'Dismissed by operator; automatic attempts remain suppressed until the condition clears'); + } else { + if (!['attention','dismissed'].includes(row.state)) return { ok: false as const, status: 409, error: 'Only incidents needing attention or dismissed incidents can be retried.' }; + if (!getServer(row.server)) return { ok: false as const, status: 409, error: 'The server is no longer registered; inspect its provisioning history.' }; + db().prepare('UPDATE watchdog_incidents SET cycle=cycle+1, failures=1, first_seen_at=?, last_seen_at=? WHERE id=?').run(iso(), iso(), id); + transition(id, 'open', 'Operator requested a fresh recovery attempt; checking current conditions'); + } + return { ok: true as const }; + })(); +} + +async function reconcileActions(): Promise { + const rows = db().prepare("SELECT * FROM watchdog_incidents WHERE state IN ('recovering','attention')").all() as WatchdogIncident[]; + for (const row of rows) { + const action = actions(row.id).filter(a => a.cycle === row.cycle).at(-1); + if (!action) continue; + const operation = db().prepare('SELECT state,submitted_at,started_at,stage,error FROM server_deployments WHERE id=?').get(action.operation_id) as + { state: string; submitted_at: string | null; started_at: string; stage: string; error: string | null } | undefined; + if (!operation) { transition(row.id, 'attention', 'Provisioning history is missing; inspect the server before retrying'); continue; } + if (['done','warning'].includes(operation.state)) { + if (row.kind === 'drift') { + try { + const plan = await previewDrift([row.server]); + if (incident(row.id)?.cycle !== row.cycle) continue; + if (plan.files.length) { transition(row.id, 'attention', 'Rebuild completed but drift remains; inspect the campaign and server files'); continue; } + } catch (err) { transition(row.id, 'attention', `Cannot verify rebuilt files: ${(err as Error).message}`); continue; } + } + transition(row.id, 'resolved', action.action === 'restart' ? 'Server recovered after restart' : 'Server recovered after rebuild'); + continue; + } + if (row.state === 'attention') continue; // Exhausted attempts stay latched, including across app restarts. + if (['cancelled','interrupted'].includes(operation.state)) { + transition(row.id, 'attention', `Provisioning ${operation.state}; inspect the recorded operation before retrying`); continue; + } + if (action.action === 'rebuild') { + if (operation.state === 'failed') transition(row.id, 'attention', `Rebuild failed: ${operation.error ?? operation.stage}`); + else if (operation.state === 'awaiting-health' && Date.now() - Date.parse(operation.submitted_at ?? operation.started_at) >= WATCHDOG_RECOVERY_MS) + transition(row.id, 'attention', 'Rebuild has not recovered within five minutes; provisioning continues verification'); + continue; + } + if (!['awaiting-health','failed'].includes(operation.state)) continue; + if (Date.now() - Date.parse(operation.submitted_at ?? operation.started_at) < WATCHDOG_RECOVERY_MS) continue; + if (!watchdogEnabled()) continue; + const current = getServer(row.server); + if (!current || current.desired_state === 'stopped' || protectedWork(row.server)) continue; + const result = db().transaction(() => { + const next = escalateWatchdogRestart(row.id, action.operation_id, WATCHDOG_RECOVERY_MS); + if (next.ok) addAction(row, next.deploymentId, 'rebuild'); + return next; + })(); + if (!result.ok && result.status !== 409 && result.status !== 503) transition(row.id, 'attention', `Could not queue rebuild: ${result.error}`); + } +} + +interface Observation { healthy: boolean; actionable: boolean; reason: string; containerState: string | null } +async function observe(server: string): Promise { + const health = await komodoInspectHealthy(server); + const state = health.state?.toLowerCase() ?? null; + if (!health.ok) return { healthy: false, actionable: !!state && ['stopped','exited','created','dead','unhealthy','restarting','starting'].includes(state), reason: health.reason, containerState: state }; + if (telemetryEvidence(server)) return { healthy: true, actionable: false, reason: 'Komodo healthy with fresh telemetry', containerState: state }; + const rcon = await rconStatusResult(server); + if (rcon.ok || telemetryEvidence(server)) return { healthy: true, actionable: false, reason: 'Komodo healthy with fresh game status', containerState: state }; + return { healthy: false, actionable: true, reason: `Game status unavailable: ${rcon.reason}; waiting for player-count verification before recovery`, containerState: state }; +} +async function scanServer(server: string): Promise { + const row = getServer(server); + if (!row || row.desired_state === 'stopped') return; + if (hasPending(server) || protectedWork(server)) return; + const observation = await observe(server); + // Probes can overlap an API request; recheck ownership before queuing anything. + if (hasPending(server) || protectedWork(server) || getServer(server)?.desired_state === 'stopped') return; + if (observation.healthy) { + const current = active(server, 'health'); + if (current) transition(current.id, 'resolved', 'Fresh game status and Komodo health verified'); + } else { + const problem = detect(server, 'health', observation.reason, observation); + const startup = ['starting','restarting'].includes(observation.containerState ?? ''); + const oldEnough = Date.now() - Date.parse(problem.first_seen_at) >= (startup ? WATCHDOG_RECOVERY_MS : WATCHDOG_FAILURE_MS); + if (problem.state === 'open' && observation.actionable && oldEnough && problem.failures >= 3 && watchdogEnabled()) { + const result = db().transaction(() => { + const next = requestWatchdogLifecycle(server, 'restart', problem.id); + if (next.ok) addAction(problem, next.deploymentId, 'restart'); + return next; + })(); + if (!result.ok && result.status !== 409 && result.status !== 503) transition(problem.id, 'attention', result.error); + } + // Unknown container health is an observation failure, not proof that a rebuild is safe. + if (!observation.actionable || hasPending(server)) return; + } + const prepared = await prepareDriftCampaign(server); + if (hasPending(server) || protectedWork(server) || getServer(server)?.desired_state === 'stopped') return; + if (!prepared.plan.files.length) { + const current = active(server, 'drift'); + if (current) transition(current.id, 'resolved', 'Server files match the promoted setup'); + return; + } + const problem = detect(server, 'drift', `${prepared.plan.files.length} file difference(s) from the promoted setup`, { files: prepared.plan.files, fingerprint: prepared.plan.fingerprint }); + if (problem.state !== 'open' || !watchdogEnabled()) return; + const result = db().transaction(() => { + const next = requestWatchdogLifecycle(server, 'rebuild', problem.id); + if (next.ok) { + const campaignId = createDriftRebuildCampaign(server, next.deploymentId, problem.id, prepared); + addAction(problem, next.deploymentId, 'rebuild', campaignId); + } + return next; + })(); + if (!result.ok && result.status !== 409 && result.status !== 503) transition(problem.id, 'attention', `Could not queue drift rebuild: ${result.error}`); +} + +let inFlight: Promise | null = null; +export function watchdogTick(forceScan = false): Promise { + if (inFlight) return inFlight; + inFlight = run(forceScan).finally(() => { inFlight = null; }); + return inFlight; +} +async function run(forceScan: boolean): Promise { + await reconcileActions(); + const lastScan = setting('watchdog_last_scan'); + if (!forceScan && lastScan && Date.now() - Date.parse(lastScan) < WATCHDOG_SCAN_MS) return; + setSetting('watchdog_last_scan', iso()); + if (!komodoConfigured()) { + setSetting('watchdog_scan_error', 'Configure Komodo before automatic recovery. RCON or fresh telemetry verifies game status.'); return; + } + const errors: string[] = []; + const servers = listServers(); + let cursor = 0; + async function pump() { + while (cursor < servers.length) { + const server = servers[cursor++]; + try { await scanServer(server.name); } + catch (err) { errors.push(`${server.name}: ${(err as Error).message}`); } + } + } + await Promise.all(Array.from({ length: Math.min(4, servers.length) }, pump)); + // Preserve incidents for temporarily absent rebuild instances, but close retired registrations. + const abandoned = db().prepare("SELECT * FROM watchdog_incidents WHERE state != 'resolved'").all() as WatchdogIncident[]; + for (const row of abandoned) { + if (getServer(row.server) || hasPending(row.server)) continue; + const last = db().prepare('SELECT action,state FROM server_deployments WHERE server=? ORDER BY rowid DESC LIMIT 1').get(row.server) as { action: string; state: string } | undefined; + // An absent registry can also mean a failed rebuild. Only a completed + // destruction retires that failure without requiring inspection. + if (['open','dismissed'].includes(row.state) || (last?.action === 'destroy' && ['done','warning'].includes(last.state))) + transition(row.id, 'resolved', 'Server is no longer registered'); + } + setSetting('watchdog_scan_error', errors.join('\n')); +} diff --git a/src/lib/ui/components/ArtifactDetail.svelte b/src/lib/ui/components/ArtifactDetail.svelte index 729248e..6ff6327 100644 --- a/src/lib/ui/components/ArtifactDetail.svelte +++ b/src/lib/ui/components/ArtifactDetail.svelte @@ -237,7 +237,7 @@ flash = body.error ?? `HTTP ${res.status}`; return; } - scopeComplete = { count: scopeImpact.affected.length, composed: scopeImpact.composed, detached: body.detached?.length ?? 0, href: scopeImpact.affected.length === 1 ? '/campaigns?server=' + encodeURIComponent(scopeImpact.affected[0]) : '/campaigns' }; + scopeComplete = { count: scopeImpact.affected.length, composed: scopeImpact.composed, detached: body.detached?.length ?? 0, href: scopeImpact.affected.length === 1 ? '/watchdog?kind=drift&server=' + encodeURIComponent(scopeImpact.affected[0]) : '/watchdog?kind=drift' }; scopeReviewOpen = false; scopeDialog?.close(); await load(); } catch (error) { flash = (error as Error).message; } finally { @@ -286,7 +286,7 @@ {/snippet} - {#if scopeComplete}
Scope saved. {scopeComplete.composed ? `Intended configuration may change on ${scopeComplete.count} eligible servers.` : scopeComplete.detached ? `Inventory tracking detached on ${scopeComplete.detached} servers; live files remain.` : `${scopeComplete.count} servers are in the old or new scope.`} Running servers have not been changed. Review reconciliation →
{/if} + {#if scopeComplete}
Scope saved. {scopeComplete.composed ? `Intended configuration may change on ${scopeComplete.count} eligible servers.` : scopeComplete.detached ? `Inventory tracking detached on ${scopeComplete.detached} servers; live files remain.` : `${scopeComplete.count} servers are in the old or new scope.`} Watchdog detects resulting drift and rebuilds eligible servers when automation is enabled. View drift recovery →
{/if} {#if latestVersion}

Latest {artifactVersionLabel(latestVersion)}

{latestVersion.notes || 'No version notes provided.'}

{#if runningBetaId !== null}{:else if latestVersion.id !== a.current_version_id}{:else}Latest version is promoted{/if}
{/if} {#if runningBetaId} @@ -417,7 +417,7 @@ {/if} - {#if scopePreviewLoading}

Reviewing eligible servers…

{:else if scopePreviewError}

{scopePreviewError}

{:else if scopeImpact}
{scopeImpact.after.length} eligible · {scopeImpact.added.length} added · {scopeImpact.removed.length} removed{#if scopeImpact.added.length}

Added: {scopeImpact.added.join(', ')}

{/if}{#if scopeImpact.removed.length}

Removed: {scopeImpact.removed.join(', ')}

{/if}

{scopeImpact.composed ? 'This changes intended configuration composition. Live configuration files remain unchanged until a deployment or reconciliation.' : scopeType === 'none' ? `${scopeImpact.detached.length} recorded installations will become untracked. Files remain on servers; review reconciliation to remove orphaned files.` : 'Scope describes eligibility. Saving does not install files or restart servers; review reconciliation for existing promoted files.'}

{/if} + {#if scopePreviewLoading}

Reviewing eligible servers…

{:else if scopePreviewError}

{scopePreviewError}

{:else if scopeImpact}
{scopeImpact.after.length} eligible · {scopeImpact.added.length} added · {scopeImpact.removed.length} removed{#if scopeImpact.added.length}

Added: {scopeImpact.added.join(', ')}

{/if}{#if scopeImpact.removed.length}

Removed: {scopeImpact.removed.join(', ')}

{/if}

{scopeImpact.composed ? 'This changes intended configuration composition.' : scopeType === 'none' ? `${scopeImpact.detached.length} recorded installations will become untracked. Watchdog will detect orphaned files as drift.` : 'Scope describes eligibility for promoted files.'}

When automation is enabled, Watchdog rebuilds servers with drift after verifying a safe server state. Rebuilds delete the existing instance and all of its files.

{/if}
diff --git a/src/routes/api/campaigns/[id]/+server.ts b/src/routes/api/campaigns/[id]/+server.ts index bbbb4fe..63bd69b 100644 --- a/src/routes/api/campaigns/[id]/+server.ts +++ b/src/routes/api/campaigns/[id]/+server.ts @@ -39,5 +39,6 @@ export async function GET({ params }: { params: { id: string } }) { .all(params.id); const events = db().prepare('SELECT id, created_at, message FROM campaign_events WHERE deployment_id = ? ORDER BY id').all(params.id); - return json({ deployment: dep, servers, files, scripts, events }); + const watchdog = db().prepare('SELECT incident_id FROM watchdog_actions WHERE campaign_id = ? LIMIT 1').get(params.id) as { incident_id: string } | undefined; + return json({ deployment: dep, servers, files, scripts, events, watchdogIncidentId: watchdog?.incident_id ?? null }); } diff --git a/src/routes/api/drift/+server.ts b/src/routes/api/drift/+server.ts index 3edb26d..8bb8904 100644 --- a/src/routes/api/drift/+server.ts +++ b/src/routes/api/drift/+server.ts @@ -1,6 +1,5 @@ import { json } from '@sveltejs/kit'; -import { DriftPlanChanged, previewDrift, runDriftDeployment } from '$lib/server/fleet/drift.js'; -import { kick } from '$lib/server/pipeline/worker.js'; +import { previewDrift } from '$lib/server/fleet/drift.js'; // GET returns the current drift plan for the whole fleet without touching disk. export async function GET() { @@ -27,7 +26,7 @@ export async function GET() { }); } -// POST removes drift and creates a drift deployment. Body: { servers?: string[], dryRun?: boolean } +// Legacy previews remain readable. Watchdog owns all new reconciliation requests. export async function POST({ request }: { request: Request }) { const body = await request.json().catch(() => null); if (!body || (body.servers !== undefined && (!Array.isArray(body.servers) || !body.servers.every((server: unknown) => typeof server === 'string')))) { @@ -35,12 +34,8 @@ export async function POST({ request }: { request: Request }) { } try { if (body.dryRun === true) return json({ ok: true, plan: await previewDrift(body.servers) }); - if (typeof body.expectedFingerprint !== 'string' || !body.expectedFingerprint) return json({ ok: false, error: 'Review a reconciliation plan before applying changes.' }, { status: 400 }); - const result = await runDriftDeployment({ servers: body.servers, expectedFingerprint: body.expectedFingerprint, triggeredBy: 'ui' }); - if (result.deploymentId) void kick(); - return json({ ok: true, result }); + return json({ ok: false, error: 'Drift reconciliation has moved to Watchdog. Scan there to queue a rebuild and campaign.', watchdogUrl: '/watchdog' }, { status: 410 }); } catch (error) { - if (error instanceof DriftPlanChanged) return json({ ok: false, error: error.message, plan: error.plan }, { status: 409 }); return json({ ok: false, error: (error as Error).message }, { status: 400 }); } } diff --git a/src/routes/api/watchdog/+server.ts b/src/routes/api/watchdog/+server.ts new file mode 100644 index 0000000..c540e05 --- /dev/null +++ b/src/routes/api/watchdog/+server.ts @@ -0,0 +1,18 @@ +import { json } from '@sveltejs/kit'; +import { watchdogOverview, watchdogTick, setWatchdogEnabled } from '$lib/server/watchdog/index.js'; +import { kick } from '$lib/server/pipeline/worker.js'; + +export function GET() { return json(watchdogOverview()); } +export async function POST({ request }: { request: Request }) { + const body = await request.json().catch(() => null); + if (!body || typeof body !== 'object' || Array.isArray(body) + || (body.enabled !== undefined && typeof body.enabled !== 'boolean') + || (body.action !== 'scan' && typeof body.enabled !== 'boolean') + || (body.action !== undefined && body.action !== 'scan')) { + return json({ error: 'Provide action: scan or an enabled boolean.' }, { status: 400 }); + } + if (typeof body.enabled === 'boolean') setWatchdogEnabled(body.enabled); + if (body.action === 'scan') await watchdogTick(true); + void kick().catch(err => console.error('[watchdog] worker kick failed:', err)); + return json(watchdogOverview()); +} diff --git a/src/routes/api/watchdog/[id]/+server.ts b/src/routes/api/watchdog/[id]/+server.ts new file mode 100644 index 0000000..eea8550 --- /dev/null +++ b/src/routes/api/watchdog/[id]/+server.ts @@ -0,0 +1,16 @@ +import { json } from '@sveltejs/kit'; +import { watchdogDetail, updateWatchdogIncident } from '$lib/server/watchdog/index.js'; +import { kick } from '$lib/server/pipeline/worker.js'; + +export function GET({ params }: { params: { id: string } }) { + const detail = watchdogDetail(params.id); + return detail ? json(detail) : json({ error: 'Watchdog incident not found.' }, { status: 404 }); +} +export async function POST({ params, request }: { params: { id: string }; request: Request }) { + const body = await request.json().catch(() => null); + if (!body || !['retry','dismiss'].includes(body.action)) return json({ error: 'Choose retry or dismiss.' }, { status: 400 }); + const result = updateWatchdogIncident(params.id, body.action); + if (!result.ok) return json({ error: result.error }, { status: result.status }); + void kick().catch(err => console.error('[watchdog] worker kick failed:', err)); + return json(watchdogDetail(params.id)); +} diff --git a/src/routes/campaigns/+page.svelte b/src/routes/campaigns/+page.svelte index cb5d2af..d2c8d86 100644 --- a/src/routes/campaigns/+page.svelte +++ b/src/routes/campaigns/+page.svelte @@ -3,12 +3,9 @@ import { onMount } from 'svelte'; import { page } from '$app/stores'; import { goto } from '$app/navigation'; - import Dialog from '$ui/components/primitives/Dialog.svelte'; - import type { DriftPlan, DriftResult } from '$lib/drift-plan.js'; import { pollVisible } from '$ui/poll.js'; import { confirmAction } from '$ui/confirm.js'; import { serverTitle } from '$ui/server-labels.js'; - import ServerIdentity from '$ui/components/ServerIdentity.svelte'; import StatusBadge from '$ui/components/primitives/StatusBadge.svelte'; import ListFooter from '$ui/components/primitives/ListFooter.svelte'; import Card from '$ui/components/primitives/Card.svelte'; @@ -48,45 +45,23 @@ prior_version: number | null; scripts: DepScript[]; } - interface DriftFile { - relPath: string; - size: number; - sha256: string; - kind: 'script' | 'config' | 'plugin' | 'baseline'; - driftKind: 'untracked' | 'missing' | 'replace'; - } - interface DriftServer { server: string; files: DriftFile[] } - interface DriftPreview { totalDrift: number; servers: DriftServer[]; scannedAt: string } - let loadError = $state(''); let limit = $state(25); - let driftLimit = $state(25); - $effect(() => { driftFilter; driftLimit = 25; }); let deployments = $state([]); let loading = $state(true); - let drift = $state(null); - let scanning = $state(false); - let running = $state(false); let rollingBack = $state(null); let flash = $state(null); let flashError = $state(false); let flashId = $state(''); - let selected = $state([]); - let reviewOpen = $state(false); - let plan = $state(null); - let planError = $state(''); - let reviewing = $state(false); - let reviewAll = $state(false); - let result = $state(null); - const skipped = $derived(result?.perServer.filter(server => server.skipped.length) ?? []); - let driftError = $state(''); - // Drift panel opens automatically when there's a problem; collapses when clean. - let driftOpen = $state(false); - let driftFilter = $state(''); - $effect(() => { driftFilter = $page.url.searchParams.get('server') ?? ''; limit = Math.max(25, Number($page.url.searchParams.get('limit')) || 25); }); - function saveContext() { + let serverFilter = $state(''); + $effect(() => { + serverFilter = $page.url.searchParams.get('server') ?? ''; + limit = Math.max(25, Number($page.url.searchParams.get('limit')) || 25); + }); + function saveContext(reset = false) { + if (reset) limit = 25; const url = new URL($page.url); - if (driftFilter) url.searchParams.set('server', driftFilter); else url.searchParams.delete('server'); + if (serverFilter) url.searchParams.set('server', serverFilter); else url.searchParams.delete('server'); if (limit > 25) url.searchParams.set('limit', String(limit)); else url.searchParams.delete('limit'); void goto(url, { replaceState: true, noScroll: true, keepFocus: true }); } @@ -99,69 +74,7 @@ } catch (error) { loadError = (error as Error).message; } finally { loading = false; } } - async function scanDrift(): Promise { - if (scanning) return; - scanning = true; - try { - const res = await fetch('/api/drift'); - if (!res.ok) throw new Error('Could not scan drift.'); - drift = (await res.json()) as DriftPreview; - selected = selected.filter(name => drift!.servers.some(server => server.server === name)); - driftError = ''; - } catch (error) { driftError = (error as Error).message; - } finally { - scanning = false; - } - } - - let driftCounts = $derived.by(() => { - if (!drift) return { untracked: 0, missing: 0, replace: 0 }; - let u = 0, m = 0, r = 0; - for (const s of drift.servers) for (const f of s.files) { - if (f.driftKind === 'untracked') u++; else if (f.driftKind === 'replace') r++; else m++; - } - return { untracked: u, missing: m, replace: r }; - }); - - // Auto-open on first scan if there's drift; leave user's toggle alone after. - let firstScan = $state(true); - $effect(() => { - if (drift && firstScan) { - firstScan = false; - if (drift.totalDrift > 0) driftOpen = true; - } - }); - - async function reviewDrift(all: boolean, targets = selected): Promise { - reviewOpen = true; reviewing = true; plan = null; planError = ''; reviewAll = all; - try { - const response = await fetch('/api/drift', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ dryRun: true, servers: all ? undefined : targets }) }); - const body = await response.json(); - if (!response.ok) throw new Error(body.error ?? 'Could not prepare the reconciliation plan.'); - plan = body.plan; - } catch (error) { planError = (error as Error).message; } - finally { reviewing = false; } - } - - async function runDrift(): Promise { - if (!plan || running) return; - running = true; planError = ''; result = null; - try { - const res = await fetch('/api/drift', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ servers: reviewAll ? undefined : plan.servers, expectedFingerprint: plan.fingerprint }) - }); - const body = await res.json(); - if (!res.ok || !body.ok) { if (body.plan) plan = body.plan; throw new Error(body.error ?? 'Reconciliation failed.'); } - result = body.result; reviewOpen = false; - await Promise.all([loadCampaigns(), scanDrift()]); - } catch (error) { planError = (error as Error).message; } finally { - running = false; - } - } - - onMount(() => { void scanDrift(); return pollVisible(loadCampaigns, 5000); }); + onMount(() => pollVisible(loadCampaigns, 5000)); function isActive(d: Deployment): boolean { return d.servers.some((s) => s.restart_state === 'pending' || s.restart_state === 'failed'); @@ -174,8 +87,11 @@ return `${Math.floor(d / 86_400_000)}d ago`; } - let active = $derived(deployments.filter(isActive)); - let history = $derived(deployments.filter((d) => !isActive(d))); + const matching = $derived(deployments.filter(d => !serverFilter || d.servers.some(server => + (server.server + ' ' + serverTitle(server.server)).toLowerCase().includes(serverFilter.toLowerCase()) + ))); + let active = $derived(matching.filter(isActive)); + let history = $derived(matching.filter((d) => !isActive(d))); function scriptSummary(d: Deployment): string { if (d.scripts && d.scripts.length > 0) { @@ -218,12 +134,6 @@ return Array.from(groups.entries()); }); - // Drift filtering — spare the operator's eyes on huge fleets. - let filteredDriftServers = $derived( - drift?.servers.filter((s) => - !driftFilter || (s.server + ' ' + serverTitle(s.server)).toLowerCase().includes(driftFilter.toLowerCase()) - ) ?? [] - ); Campaigns | Electric Cherry @@ -231,106 +141,20 @@
{#if loadError}{/if} {#if loading}
Loading…
{:else} - - {#if driftError}{/if} -
- -
- - - -
+
+

Automatic drift repair

Watchdog detects drift and rebuilds affected servers with the promoted setup. Repair campaigns remain in this history.

+
- - {#if driftOpen} -
- {#if drift}

Scanned {$timeFormat.dateTime(drift.scannedAt)}. Selection controls the action scope; search only changes the visible list.

{/if} - {#if drift && drift.totalDrift > 0} - { driftFilter = event.currentTarget.value; saveContext(); }} - /> -
{selected.length} selected
- {#each filteredDriftServers.slice(0,driftLimit) as s (s.server)} -
-
- - {s.files.length} entr{s.files.length === 1 ? 'y' : 'ies'} -
-
- {#each s.files as f (f.driftKind + ':' + f.kind + ':' + f.relPath)} - - [{f.driftKind === 'missing' ? '+' : f.driftKind === 'replace' ? '↻' : '−'} {f.kind}] {f.relPath} - - {/each} -
-
- {/each} - - {:else if drift?.totalDrift === 0} - - {:else} - - {/if} -
- {/if} - - {#if result}
0} role={skipped.length ? 'alert' : 'status'}> - {result.totalServers} server{result.totalServers === 1 ? '' : 's'} updated{skipped.length ? `, ${skipped.length} with skipped files` : ''} -

{result.totalRemoved} files removed; {result.totalRestored} installed. {result.totalServers ? 'Affected servers restart when empty, then health is verified.' : skipped.length ? 'No changes could be applied. Review the reasons below.' : 'The reviewed servers already match the intended files.'}

- {#if result.deploymentId}View campaign →{/if} - {#each skipped as server}
    {#each server.skipped as reason}
  • {reason}
    • {/each}
{/each} - {#if skipped.length}{/if} -
{/if} +
@@ -376,22 +200,10 @@ {/each} {/if}
- + saveContext()} label="campaigns" /> {/if}
- - {#if planError}{/if} - {#if reviewing}

Comparing current files with their intended versions…

- {:else if plan} -

{reviewAll ? 'Entire fleet' : 'Selected servers'}: {plan.servers.length} server{plan.servers.length === 1 ? '' : 's'}, {plan.files.length} file change{plan.files.length === 1 ? '' : 's'}.

-

Files change immediately. Each affected server restarts after a fresh check confirms it is empty, then waits for health verification. Untracked files will be removed.

-

Reviewed {$timeFormat.dateTime(plan.scannedAt)}. Changes to this plan require another review.

-
{#each plan.servers as server}{@const files = plan.files.filter(file => file.server === server)}{#if files.length}

{#each files as file}
{file.driftKind === 'untracked' ? 'Remove' : file.driftKind === 'missing' ? 'Restore' : 'Replace'} {file.relPath}{file.beforeSize === null ? 'Absent' : `${file.beforeSize} bytes`} → {file.afterSize === null ? 'Removed' : `${file.afterSize} bytes`}
File fingerprints

Before: {file.beforeSha256 ?? 'Absent'}
After: {file.afterSha256 ?? 'Removed'}

{/each}
{/if}{/each}
- {/if} - {#snippet actions()}{/snippet} - - {#snippet deployRow(d: Deployment)}
{#if flash && flashId === d.id}
{flash}
{/if} @@ -447,16 +259,13 @@ {/snippet} diff --git a/src/routes/campaigns/[id]/+page.svelte b/src/routes/campaigns/[id]/+page.svelte index c2d7d1f..f5b8c55 100644 --- a/src/routes/campaigns/[id]/+page.svelte +++ b/src/routes/campaigns/[id]/+page.svelte @@ -35,6 +35,7 @@ servers: ServerRow[]; files: FileRow[]; events: ActivityEvent[]; + watchdogIncidentId?: string | null; } let detail = $state(null); @@ -121,7 +122,9 @@ {/snippet} -

{completed} / {detail.servers.length} servers activated{unresolved ? ` · ${unresolved} unresolved` : ''}

Files are staged before activation. Servers restart when empty, then their health is verified.

{#if unresolved}

Review failed or waiting servers below. Diagnose connectivity or inspect startup logs from the affected row.

{:else}

Every recorded server completed activation.

{/if}
+

{completed} / {detail.servers.length} servers activated{unresolved ? ` · ${unresolved} unresolved` : ''}

{detail.watchdogIncidentId ? 'Watchdog repairs drift by deleting the existing instance files and rebuilding with the latest promoted setup after verifying a safe server state.' : 'Files are staged before activation. Servers restart when empty, then their health is verified.'}

{#if unresolved}

Review failed or waiting servers below. Diagnose connectivity or inspect startup logs from the affected row.

{:else}

Every recorded server completed activation.

{/if}
+ + {#if detail.watchdogIncidentId}

Started by Watchdog · View incident and recovery history

{/if}
@@ -204,6 +207,7 @@ {/if} diff --git a/src/routes/watchdog/[id]/+page.svelte b/src/routes/watchdog/[id]/+page.svelte new file mode 100644 index 0000000..a74c1b7 --- /dev/null +++ b/src/routes/watchdog/[id]/+page.svelte @@ -0,0 +1,205 @@ + + +{incident ? `${serverTitle(incident.server)} · Watchdog` : 'Watchdog incident'} | Electric Cherry + +
+ + {#snippet crumbs()}{/snippet} + {#snippet actions()} + {#if canRetry}{/if} + {#if canDismiss}{/if} + {/snippet} + + + {#if loadError}{/if} + {#if actionError}{/if} + {#if notice}

{notice}

{/if} + + {#if incident && detail} +
+
View server
+

{incident.summary}

+

{explanation}

+ {#if activeOperation && (canRetry || canDismiss)}

Complete or cancel the active provisioning operation before retrying or dismissing this incident.

{/if} + {#if incident.kind === 'drift'}

Drift repair deletes the existing instance and all of its files, then rebuilds it with the latest promoted setup. The repair is recorded as a campaign.

{:else if incident.state !== 'resolved'}

Automatic health recovery first tries a restart, then rebuilds if needed. A rebuild deletes the existing instance and all of its files and installs the latest promoted setup.

{/if} +
+
First detected
{$timeFormat.dateTime(incident.first_seen_at)}
+
Last observed
{$timeFormat.dateTime(incident.last_seen_at)}
+ {#if incident.finished_at}
Finished
{$timeFormat.dateTime(incident.finished_at)}
{/if} + {#if evidence.containerState}
Observed Komodo state
{evidence.containerState}
{/if} +
+ {#if evidence.reason && evidence.reason !== incident.summary}

Observed reason: {evidence.reason}

{/if} +
Diagnose connectionOpen server logs
+
+ +
+

Recovery operations

+ {#if !actions.length} + {:else}
+ {#each actions as action (action.id)} +
+

{action.action === 'restart' ? 'Restart server' : 'Rebuild server'}

{#if action.operation_state}{/if}
+ {#if action.operation_stage}

{action.operation_stage}

{/if} + {#if action.operation_error}

{action.operation_error}

{/if} + +
View provisioning{#if action.campaign_id}View repair campaign{/if}
+
+ {/each} +
{/if} +
+ + {#if incident.kind === 'drift'} +
+

Observed file drift {files.length}

+

These differences describe the observed problem. Recovery rebuilds the whole server instance.

+ {#if !files.length} + {:else}
+ {#each files.slice(0, fileLimit) as file (`${file.kind}:${file.driftKind}:${file.relPath}`)} +
+
{file.driftKind === 'replace' ? 'Modified' : file.driftKind === 'missing' ? 'Missing' : 'Untracked'}{file.relPath}{file.kind}
+
Observed and expected file
+
Observed
{file.beforeSize === null ? 'Absent' : `${file.beforeSize} bytes`}{#if file.beforeSha256}{file.beforeSha256}{/if}
+
Expected
{file.afterSize === null ? 'Absent' : `${file.afterSize} bytes`}{#if file.afterSha256}{file.afterSha256}{/if}
+
+
+ {/each} +
{/if} +
+ {/if} + +
+

Activity history {events.length}

+

Newest first · updates automatically

+ {#if !events.length} + {:else}
    + {#each events.slice(0, eventLimit) as event (event.id)} +

  1. {event.message}

    2. + {/each} +
{/if} +
+ {:else if !loadError}

Loading incident…

{/if} +
+ + diff --git a/tests/unit/beta-plans.test.ts b/tests/unit/beta-plans.test.ts index e762d3e..825a6a8 100644 --- a/tests/unit/beta-plans.test.ts +++ b/tests/unit/beta-plans.test.ts @@ -54,6 +54,19 @@ async function tick(seconds = 15, players = 4, healthy = true) { const scriptPath = (name: string) => join(fixture.t5, name, 'runtime/plutonium/storage/t5/raw/scripts/sp/test.gsc'); describe('durable beta test plans', () => { + it('waits for occupied telemetry to clear before restarting a beta despite an empty RCON response', async () => { + const id = await openPlan(defaultPlan()); + const { listBetaServers } = await import('../../src/lib/server/fleet/beta.js'); + const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js'); + await postTelemetry(gameEvent('heartbeat', 4, 2)); + await tick(); + expect(komodoRestart).not.toHaveBeenCalled(); + expect(listBetaServers(id)[0].stage).toBe('awaiting-empty'); + await postTelemetry(gameEvent('heartbeat', 4, 0)); + await tick(); + expect(komodoRestart).toHaveBeenCalledTimes(1); + }); + it('holds file deployment until each wave passes, and rollback leaves undeployed files alone', async () => { await writeFile(scriptPath(names[1]), 'original untracked file'); const id = await openPlan({ ...defaultPlan(), waveSize: 1 }, names); @@ -81,7 +94,7 @@ describe('durable beta test plans', () => { it('uses script events instead of logs and keeps match progress across app restarts', async () => { await postTelemetry(gameEvent('game_end')); const id = await openPlan({ ...defaultPlan(), requirements: [requirement('matches', 'matches')] }); - await tick(); await tick(); + await tick(46); await tick(); const { testView } = await import('../../src/lib/server/fleet/beta-tests.js'); await appendFile(join(fixture.t5, names[0], 'logs/games_sp.log'), 'InitGame:\n 0:00 J;new;1;Josh\nExitLevel: executed\n'); await tick(); @@ -98,7 +111,7 @@ describe('durable beta test plans', () => { it('automatically credits ordinary rounds only to active deployed beta servers', async () => { const id = await openPlan({ ...defaultPlan(), waveSize: 1, requirements: [requirement('round', 'round')] }, names); await postTelemetry(gameEvent('round_change')); - await tick(); await tick(); + await tick(46); await tick(); const { testView } = await import('../../src/lib/server/fleet/beta-tests.js'); expect(testView(id)?.servers[0].progress[0].current).toBe(0); await postTelemetry(gameEvent('round_change', 99, 4, names[1])); @@ -117,7 +130,7 @@ describe('durable beta test plans', () => { const first = currentRun(id, names[0])!; expect(testView(id)?.servers[0].progress[0].current).toBe(0); await postTelemetry(gameEvent('game_start')); - await tick(); await tick(); + await tick(46); await tick(); expect(currentRun(id, names[0])?.id).not.toBe(first.id); expect(currentRun(id, names[0])?.phase).toBe(1); expect(testView(id)?.servers[0].progress[0].current).toBe(0); @@ -212,7 +225,7 @@ describe('durable beta test plans', () => { expect(retry.status).toBe(200); await postTelemetry(gameEvent('round_change')); expect(testView(id)?.servers).toHaveLength(0); - await tick(); await tick(); + await tick(46); await tick(); expect(testView(id)?.servers[0].progress[0].current).toBe(0); await postTelemetry(gameEvent('round_change')); expect(testView(id)?.servers[0].progress[0].current).toBe(25); diff --git a/tests/unit/create-server.test.ts b/tests/unit/create-server.test.ts index 5b3255e..2963d9d 100644 --- a/tests/unit/create-server.test.ts +++ b/tests/unit/create-server.test.ts @@ -435,7 +435,7 @@ describe('server creation map selection', () => { vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: true, state: 'running' }); vi.mocked(rconStatusResult).mockResolvedValueOnce({ ok: false, endpoint: null, reason: 'timeout', detail: 'not ready' }); await processCreations(); - expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', stage: 'Waiting for RCON health: timeout' }); + expect(getDeployment(id)?.deployment).toMatchObject({ state: 'awaiting-health', stage: 'Waiting for RCON or fresh telemetry health: timeout' }); for (const map of [null, 'zombie_moon']) { vi.mocked(komodoInspectHealthy).mockResolvedValueOnce({ ok: true, state: 'running' }); vi.mocked(rconStatusResult).mockResolvedValueOnce({ ok: true, endpoint: { host: 'localhost', port: created.port }, rawReply: '', status: { players: 0, map, gametype: null, roster: [] } }); diff --git a/tests/unit/creation-queue.test.ts b/tests/unit/creation-queue.test.ts index 548a498..da6ca55 100644 --- a/tests/unit/creation-queue.test.ts +++ b/tests/unit/creation-queue.test.ts @@ -94,7 +94,7 @@ describe('shared creation queue', () => { expect(getDeployment(jobs[0].deploymentId)?.deployment).toMatchObject({ stage: 'Waiting for Komodo health: starting' }); vi.mocked(rconStatusResult).mockResolvedValueOnce({ ok: false, reason: 'timeout', endpoint: null, detail: 'booting' }); await processCreations(); - expect(getDeployment(jobs[0].deploymentId)?.deployment).toMatchObject({ stage: 'Waiting for RCON health: timeout' }); + expect(getDeployment(jobs[0].deploymentId)?.deployment).toMatchObject({ stage: 'Waiting for RCON or fresh telemetry health: timeout' }); vi.mocked(rconStatusResult).mockResolvedValueOnce(status(jobs[0].name, 'zombie_moon')); await processCreations(); expect(getDeployment(jobs[0].deploymentId)?.deployment).toMatchObject({ stage: expect.stringContaining('expected zombie_theater') }); diff --git a/tests/unit/drift-kinds.test.ts b/tests/unit/drift-kinds.test.ts index 0963332..f45f6bb 100644 --- a/tests/unit/drift-kinds.test.ts +++ b/tests/unit/drift-kinds.test.ts @@ -73,6 +73,112 @@ describe('reviewed drift plans', () => { }); describe('drift scanning', () => { + it.each([ + ['script', 'moved'], ['script', 'unset'], ['plugin', 'moved'], ['plugin', 'unset'] + ] as const)('detects formerly tracked %s files after their scope is %s', async (kind, change) => { + const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); + const { createScript, setScope } = await import('../../src/lib/server/fleet/scripts.js'); + const { scanServerDrift, runDriftDeployment } = await import('../../src/lib/server/fleet/drift.js'); + const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + const { destOfKind } = await import('../../src/lib/server/config.js'); + await seedServerRegistry(); + const server = 't5_zombie_theater_1'; + const relPath = kind === 'script' ? 'outdated.gsc' : 'outdated.dll'; + const { script, version } = createScript({ name: `outdated ${kind}`, kind, scopeType: 'game', scopeValue: 't5', targetRelPath: relPath, content: Buffer.from('previously promoted') }); + db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(version.id, script.id); + await runDriftDeployment({ servers: [server] }); + setScope({ scriptId: script.id, scopeType: change === 'moved' ? 'game' : null, scopeValue: change === 'moved' ? 't6' : null }); + expect(getServerScript(server, relPath)).toBeDefined(); + expect(await scanServerDrift(server)).toEqual([expect.objectContaining({ kind, relPath, driftKind: 'untracked' })]); + await runDriftDeployment({ servers: [server] }); + expect(existsSync(join(destOfKind(server, kind), relPath))).toBe(false); + expect(getServerScript(server, relPath)).toBeUndefined(); + }); + + it.each(['script', 'plugin'] as const)('detects edited tracked %s bytes and restores their promoted version', async (kind) => { + const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); + const { createScript } = await import('../../src/lib/server/fleet/scripts.js'); + const { scanServerDrift, runDriftDeployment } = await import('../../src/lib/server/fleet/drift.js'); + const { getServerScript } = await import('../../src/lib/server/fleet/inventory.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + const { destOfKind } = await import('../../src/lib/server/config.js'); + await seedServerRegistry(); + const server = 't5_zombie_theater_1'; + const relPath = kind === 'script' ? 'managed.gsc' : 'managed.dll'; + const content = Buffer.from('promoted content'); + const { script, version } = createScript({ name: `managed ${kind}`, kind, scopeType: 'game', scopeValue: 't5', targetRelPath: relPath, content }); + db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(version.id, script.id); + await runDriftDeployment({ servers: [server] }); + const inventory = getServerScript(server, relPath); + const path = join(destOfKind(server, kind), relPath); + await writeFile(path, 'changed outside Electric Cherry'); + expect(getServerScript(server, relPath)).toEqual(inventory); + expect(await scanServerDrift(server)).toEqual([expect.objectContaining({ relPath, kind, driftKind: 'replace' })]); + await runDriftDeployment({ servers: [server] }); + expect(await readFile(path)).toEqual(content); + expect(await scanServerDrift(server)).toEqual([]); + }); + + it('compares composed config with disk even when inventory still has the promoted hash', async () => { + const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); + const { createScript } = await import('../../src/lib/server/fleet/scripts.js'); + const { scanServerDrift, runDriftDeployment } = await import('../../src/lib/server/fleet/drift.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + const { getPromotedServerConfig } = await import('../../src/lib/server/fleet/server-config.js'); + await seedServerRegistry(); + const server = 't5_zombie_theater_1'; + const { script, version } = createScript({ name: 'baseline', kind: 'baseline', scopeType: 'game', scopeValue: 't5', content: Buffer.from('set sv_voice "1"\n') }); + db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(version.id, script.id); + await runDriftDeployment({ servers: [server] }); + const path = join(fixture.t5, server, 'configs', 'dedicated_sp.cfg'); + await writeFile(path, 'changed outside Electric Cherry'); + expect(await scanServerDrift(server)).toEqual([expect.objectContaining({ relPath: 'dedicated_sp.cfg', driftKind: 'replace' })]); + await runDriftDeployment({ servers: [server] }); + expect(await readFile(path)).toEqual(getPromotedServerConfig('t5', 'zombie_theater')); + }); + + it.each(['running', 'ready-promote'])('defers all drift for a server with a %s beta', async (state) => { + const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); + const { createScript, uploadNewVersion } = await import('../../src/lib/server/fleet/scripts.js'); + const { openBeta } = await import('../../src/lib/server/fleet/beta.js'); + const { scanServerDrift, runDriftDeployment } = await import('../../src/lib/server/fleet/drift.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + await seedServerRegistry(); + const server = 't5_zombie_theater_1'; + const { script, version } = createScript({ name: 'baseline', kind: 'baseline', scopeType: 'game', scopeValue: 't5', content: Buffer.from('set sv_voice "1"\n') }); + db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(version.id, script.id); + const experimental = uploadNewVersion(script.id, Buffer.from('set sv_voice "0"\n')); + const { betaId } = await openBeta({ scripts: [{ scriptId: script.id, scriptVersionId: experimental.id }], servers: [server], waitForHealth: false, waitForMatch: false }); + db().prepare('UPDATE betas SET state = ? WHERE id = ?').run(state, betaId); + const path = join(fixture.t5, server, 'configs', 'dedicated_sp.cfg'); + const betaBytes = await readFile(path); + const stray = join(fixture.t5, server, 'configs', 'stray.cfg'); + await writeFile(stray, 'also deferred'); + expect(await scanServerDrift(server)).toEqual([]); + expect(await runDriftDeployment({ servers: [server] })).toMatchObject({ deploymentId: null, totalServers: 0 }); + expect(await readFile(path)).toEqual(betaBytes); + expect(await readFile(stray, 'utf8')).toBe('also deferred'); + db().prepare("UPDATE betas SET state = 'rolled-back' WHERE id = ?").run(betaId); + expect((await scanServerDrift(server)).map(entry => entry.relPath)).toEqual(['dedicated_sp.cfg', 'stray.cfg']); + }); + + it('reports an unavailable registered directory without hiding drift on another game root', async () => { + const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); + const { scanServerDrift, runDriftDeployment } = await import('../../src/lib/server/fleet/drift.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + const other = 't6_zombie_tranzit_1'; + await mkdir(join(fixture.t6, other, 'configs'), { recursive: true }); + await writeFile(join(fixture.t6, other, 'configs', 'stray.cfg'), 'untracked'); + await seedServerRegistry(); + await rm(fixture.t5, { recursive: true }); + await expect(scanServerDrift('t5_zombie_theater_1')).rejects.toThrow('Server files unavailable'); + await expect(runDriftDeployment({ servers: ['t5_zombie_theater_1'] })).rejects.toThrow('Server files unavailable'); + expect(existsSync(fixture.t5)).toBe(false); + expect(db().prepare('SELECT * FROM deployments').all()).toEqual([]); + expect(await scanServerDrift(other)).toEqual([expect.objectContaining({ server: other, relPath: 'stray.cfg', driftKind: 'untracked' })]); + }); + it('replaces a seeded config once instead of planning both removal and installation', async () => { const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); const { createScript } = await import('../../src/lib/server/fleet/scripts.js'); @@ -152,6 +258,47 @@ describe('drift scanning', () => { }); }); +describe('drift rebuild campaigns', () => { + it('captures intended changes without writing and follows the linked rebuild without another restart', async () => { + const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); + const { createScript } = await import('../../src/lib/server/fleet/scripts.js'); + const { prepareDriftCampaign, createDriftRebuildCampaign } = await import('../../src/lib/server/fleet/drift.js'); + const { beginDeployment } = await import('../../src/lib/server/provisioning/history.js'); + const { sweep } = await import('../../src/lib/server/pipeline/sweep.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + const { destOfKind } = await import('../../src/lib/server/config.js'); + await seedServerRegistry(); + const server = 't5_zombie_theater_1'; + const { script, version } = createScript({ name: 'managed', kind: 'script', scopeType: 'game', scopeValue: 't5', targetRelPath: 'managed.gsc', content: Buffer.from('promoted content') }); + db().prepare('UPDATE scripts SET current_version_id = ? WHERE id = ?').run(version.id, script.id); + const target = join(destOfKind(server, 'script'), 'managed.gsc'); + await writeFile(target, 'old content'); + const stray = join(destOfKind(server, 'config'), 'stray.cfg'); + await writeFile(stray, 'untracked'); + const prepared = await prepareDriftCampaign(server); + const operationId = beginDeployment(server, 'rebuild', 'watchdog'); + db().prepare("UPDATE server_deployments SET state = 'waiting' WHERE id = ?").run(operationId); + const campaignId = createDriftRebuildCampaign(server, operationId, 42, prepared); + expect(campaignId).toBeTruthy(); + expect(await readFile(target, 'utf8')).toBe('old content'); + expect(await readFile(stray, 'utf8')).toBe('untracked'); + expect(db().prepare('SELECT rel_path, verb, content_before, content_after FROM deployment_files WHERE deployment_id = ? ORDER BY rel_path').all(campaignId)) + .toEqual([ + { rel_path: 'managed.gsc', verb: 'updated', content_before: Buffer.from('old content'), content_after: version.content }, + { rel_path: 'stray.cfg', verb: 'removed', content_before: Buffer.from('untracked'), content_after: null } + ]); + await sweep(); + expect(db().prepare('SELECT action FROM server_deployments').all()).toEqual([{ action: 'rebuild' }]); + expect(db().prepare('SELECT provisioning_id, restart_state FROM deployment_servers WHERE deployment_id = ?').get(campaignId)) + .toEqual({ provisioning_id: operationId, restart_state: 'pending' }); + db().prepare("UPDATE server_deployments SET state = 'done', submitted_at = ?, finished_at = ? WHERE id = ?") + .run(new Date().toISOString(), new Date().toISOString(), operationId); + await sweep(); + expect(db().prepare('SELECT restart_state FROM deployment_servers WHERE deployment_id = ?').get(campaignId)).toEqual({ restart_state: 'done' }); + expect(db().prepare('SELECT finished_at FROM deployments WHERE id = ?').get(campaignId)).toMatchObject({ finished_at: expect.any(String) }); + }); +}); + // Deploys a promoted artifact to a scope-matching server, then deletes it // off disk so drift's "missing" pass has something to find. async function promoteAndOrphan( diff --git a/tests/unit/provisioning.test.ts b/tests/unit/provisioning.test.ts index 8438dae..d2a6fa7 100644 --- a/tests/unit/provisioning.test.ts +++ b/tests/unit/provisioning.test.ts @@ -29,10 +29,13 @@ beforeEach(async () => { // Exercise the upgrade path, with history and references present before migration. const old = new Database(fixture.dbPath); const schema = await readFile(new URL('../../src/lib/server/db/schema.sql', import.meta.url), 'utf8'); - old.exec(schema.replace("'create', 'start', 'restart', 'rebuild', 'stop', 'destroy'", "'create', 'destroy'")); + old.exec(schema.replace("'create', 'start', 'restart', 'rebuild', 'stop', 'destroy'", "'create', 'destroy'") + .replace("'campaign','beta','maintenance','watchdog'", "'campaign','beta','maintenance'") + .replace("server TEXT NOT NULL, -- Campaign observation survives a rebuild's registry gap.", 'server TEXT NOT NULL REFERENCES servers(name),')); old.prepare(`INSERT INTO server_deployments (rowid,id,server,action,state,stage,started_at,triggered_by) VALUES (42,'old','old-server','create','done','Complete','2026-09-01','test')`).run(); old.prepare("INSERT INTO server_deployment_events (deployment_id,created_at,message) VALUES ('old','2026-09-01','Old activity')").run(); old.prepare("INSERT INTO server_creation_jobs VALUES ('old','old-server',28961,'zombie','theater','zombie_theater')").run(); + old.prepare("INSERT INTO provisioning_sources VALUES ('old','campaign','legacy-campaign','old-server')").run(); old.close(); const { db } = await import('../../src/lib/server/db/index.js'); db().prepare(`INSERT INTO servers (name,komodo_type,komodo_name,game,map,game_mode,map_slug,game_port,host_dir) @@ -73,6 +76,7 @@ describe('provisioning operations', () => { expect(db().prepare("SELECT rowid FROM server_deployments WHERE id = 'old'").get()).toEqual({ rowid: 42 }); expect(db().prepare("SELECT message FROM server_deployment_events WHERE deployment_id = 'old'").get()).toEqual({ message: 'Old activity' }); expect(db().prepare("SELECT server FROM server_creation_jobs WHERE deployment_id = 'old'").get()).toEqual({ server: 'old-server' }); + expect(db().prepare("SELECT kind, source_id FROM provisioning_sources WHERE operation_id = 'old'").get()).toEqual({ kind: 'campaign', source_id: 'legacy-campaign' }); expect(db().pragma('foreign_key_check')).toEqual([]); expect(db().pragma('foreign_keys', { simple: true })).toBe(1); }); @@ -244,3 +248,279 @@ describe('provisioning operations', () => { expect(komodoCreateDeployment).not.toHaveBeenCalled(); }); }); + +async function incident(id = 'incident') { + const { db } = await import('../../src/lib/server/db/index.js'); + const now = new Date().toISOString(); + db().prepare(`INSERT INTO watchdog_incidents (id,server,kind,state,summary,first_seen_at,last_seen_at) + VALUES (?,?,'health','open','Server down',?,?)`).run(id, server, now, now); + return id; +} + +async function report(players: number, timestamp = new Date().toISOString(), map = 'zombie_theater') { + const { db } = await import('../../src/lib/server/db/index.js'); + const { gameEvent } = await import('../helpers/telemetry.js'); + db().prepare(`INSERT INTO server_telemetry (server,round,players,received_at,payload) VALUES (?,1,?,?,?) + ON CONFLICT(server) DO UPDATE SET players=excluded.players, received_at=excluded.received_at, payload=excluded.payload`) + .run(server, players, timestamp, JSON.stringify({ ...gameEvent('heartbeat', 1, players, server, map), receivedAt: timestamp })); +} + +describe('Watchdog provisioning and telemetry evidence', () => { + it('keeps the original running intent when a queued removal is cancelled', async () => { + const { destroyServer, cancelRemoval } = await import('../../src/lib/server/provisioning/index.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + const removal = destroyServer(server); + if (!removal.ok || !removal.deploymentId) throw new Error('Could not queue removal'); + expect(cancelRemoval(removal.deploymentId)).toBe(true); + expect(db().prepare('SELECT desired_state FROM servers WHERE name = ?').get(server)).toEqual({ desired_state: 'running' }); + }); + + it('preserves stop intent across process restarts, including an uncertain stop request', async () => { + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + const { finishDeployment } = await import('../../src/lib/server/provisioning/history.js'); + const id = await queue('stop'); + finishDeployment(id, { ok: false, message: 'lost response' }); + expect(db().prepare('SELECT desired_state FROM servers WHERE name = ?').get(server)).toEqual({ desired_state: 'stopped' }); + // Exercise a legacy upgrade where the stop exists but intent was not stored. + db().prepare("UPDATE servers SET desired_state = 'running'").run(); + db().prepare("DELETE FROM settings WHERE key = 'watchdog_desired_state_v1'").run(); + closeDb(); + expect(db().prepare('SELECT desired_state FROM servers WHERE name = ?').get(server)).toEqual({ desired_state: 'stopped' }); + const { requestWatchdogLifecycle } = await import('../../src/lib/server/provisioning/index.js'); + await incident(); + expect(requestWatchdogLifecycle(server, 'restart', 'incident')).toMatchObject({ ok: false, error: 'This server is intentionally stopped.' }); + await queue('start'); + expect(db().prepare('SELECT desired_state FROM servers WHERE name = ?').get(server)).toEqual({ desired_state: 'running' }); + }); + + it('deduplicates Watchdog requests, records attribution, and holds paused work', async () => { + const { requestWatchdogLifecycle, getDeployment } = await import('../../src/lib/server/provisioning/index.js'); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + await incident(); + const result = requestWatchdogLifecycle(server, 'restart', 'incident'); + expect(result.ok).toBe(true); + if (!result.ok) throw new Error(result.error); + expect(requestWatchdogLifecycle(server, 'restart', 'incident')).toEqual(result); + expect(getDeployment(result.deploymentId)?.sources).toEqual([{ kind: 'watchdog', source_id: 'incident', server }]); + expect(await operation(result.deploymentId)).toMatchObject({ triggered_by: 'watchdog:incident' }); + db().prepare("INSERT INTO settings (key,value) VALUES ('watchdog_enabled','0')").run(); + await processProvisioning(); + expect(komodoRestart).not.toHaveBeenCalled(); + db().prepare("UPDATE settings SET value = '1' WHERE key = 'watchdog_enabled'").run(); + await processProvisioning(); + expect(komodoRestart).toHaveBeenCalledTimes(1); + }); + + it('accepts fresh telemetry for occupancy and restart recovery without RCON, while rejecting old-process reports', async () => { + const { requestWatchdogLifecycle } = await import('../../src/lib/server/provisioning/index.js'); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { rconGetPlayerCount, rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js'); + const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js'); + const { config } = await import('../../src/lib/server/config.js'); + vi.spyOn(config.rcon, 'password', 'get').mockReturnValue(''); + vi.mocked(rconGetPlayerCount).mockResolvedValue(null); + vi.mocked(rconStatusResult).mockResolvedValue({ ok: false, reason: 'no-config', detail: 'No RCON', endpoint: null }); + await incident(); await report(0); + const result = requestWatchdogLifecycle(server, 'restart', 'incident'); + if (!result.ok) throw new Error(result.error); + await processProvisioning(); + expect(komodoRestart).toHaveBeenCalledTimes(1); + vi.setSystemTime(Date.now() + 15_000); + await processProvisioning(); + expect(await operation(result.deploymentId)).toMatchObject({ state: 'awaiting-health' }); + await report(0); + vi.setSystemTime(Date.now() + 15_000); + await processProvisioning(); + expect(await operation(result.deploymentId)).toMatchObject({ state: 'done' }); + }); + + it('uses fresh occupied telemetry arriving during RCON to veto a zero-player restart', async () => { + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js'); + const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js'); + const id = await queue('restart'); + vi.mocked(rconGetPlayerCount).mockImplementationOnce(async () => { await report(2); return 0; }); + await processProvisioning(); + expect(komodoRestart).not.toHaveBeenCalled(); + expect(await operation(id)).toMatchObject({ state: 'waiting', players: 2 }); + await report(0); + vi.mocked(rconGetPlayerCount).mockResolvedValueOnce(3); + await processProvisioning(); + expect(komodoRestart).not.toHaveBeenCalled(); + expect(await operation(id)).toMatchObject({ state: 'waiting', players: 3 }); + }); + + it('rejects stale, future, pre-submission, and invalid telemetry evidence', async () => { + const { telemetryEvidence, probePlayerCount } = await import('../../src/lib/server/fleet/status-evidence.js'); + const { rconGetPlayerCount } = await import('../../src/lib/server/integrations/rcon.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + vi.mocked(rconGetPlayerCount).mockResolvedValue(null); + for (const offset of [-45_001, 1]) { + await report(0, new Date(Date.now() + offset).toISOString()); + expect(telemetryEvidence(server)).toBeNull(); + expect(await probePlayerCount(server)).toBeNull(); + } + await report(0); + expect(telemetryEvidence(server, new Date().toISOString())).toBeNull(); + db().prepare("UPDATE server_telemetry SET payload = '{}' WHERE server = ?").run(server); + expect(telemetryEvidence(server)).toBeNull(); + }); + + it('reserves a rebuild before transferring its exact restart slot and requires fresh emptiness', async () => { + const { requestWatchdogLifecycle, escalateWatchdogRestart } = await import('../../src/lib/server/provisioning/index.js'); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { komodoInspectHealthy, komodoDestroyDeployment, komodoRestart } = await import('../../src/lib/server/integrations/komodo.js'); + const { rconGetPlayerCount, rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js'); + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + await incident(); + const restart = requestWatchdogLifecycle(server, 'restart', 'incident'); + if (!restart.ok) throw new Error(restart.error); + await processProvisioning(); + expect(escalateWatchdogRestart('incident', restart.deploymentId)).toMatchObject({ ok: false }); + vi.setSystemTime(Date.now() + 300_000); + const rebuild = escalateWatchdogRestart('incident', restart.deploymentId); + if (!rebuild.ok) throw new Error(rebuild.error); + expect(escalateWatchdogRestart('incident', restart.deploymentId)).toEqual(rebuild); + expect(await operation(restart.deploymentId)).toMatchObject({ state: 'failed' }); + expect(db().prepare('SELECT operation_id, healthy_at FROM restart_rollout WHERE server = ?').get(server)) + .toEqual({ operation_id: rebuild.deploymentId, healthy_at: null }); + closeDb(); + vi.mocked(rconGetPlayerCount).mockResolvedValue(null); + vi.mocked(rconStatusResult).mockResolvedValue({ ok: false, reason: 'timeout', detail: 'No reply', endpoint: null }); + await processProvisioning(); + expect(komodoDestroyDeployment).not.toHaveBeenCalled(); + expect(await operation(rebuild.deploymentId)).toMatchObject({ state: 'waiting' }); + expect(komodoRestart).toHaveBeenCalledTimes(1); + vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'Stopped' }); + await processProvisioning(); + expect(komodoDestroyDeployment).toHaveBeenCalledTimes(1); + expect(await operation(rebuild.deploymentId)).toMatchObject({ state: 'queued' }); + expect(db().prepare('SELECT 1 FROM restart_rollout WHERE server = ?').get(server)).toBeUndefined(); + }); + + it('retains recovery capacity when a rebuild prerequisite fails or a different operation owns the slot', async () => { + const { requestWatchdogLifecycle, escalateWatchdogRestart } = await import('../../src/lib/server/provisioning/index.js'); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { removeServerKey, saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + await incident(); + const restart = requestWatchdogLifecycle(server, 'restart', 'incident'); + if (!restart.ok) throw new Error(restart.error); + await processProvisioning(); vi.setSystemTime(Date.now() + 300_000); + removeServerKey(server); + expect(escalateWatchdogRestart('incident', restart.deploymentId)).toMatchObject({ ok: false }); + expect(await operation(restart.deploymentId)).toMatchObject({ state: 'awaiting-health' }); + expect(db().prepare('SELECT operation_id FROM restart_rollout WHERE server = ?').get(server)).toEqual({ operation_id: restart.deploymentId }); + saveServerKey(server, 'saved-key'); + db().prepare("UPDATE restart_rollout SET operation_id = 'another-operation' WHERE server = ?").run(server); + expect(escalateWatchdogRestart('incident', restart.deploymentId)).toMatchObject({ ok: false }); + expect(db().prepare('SELECT operation_id FROM restart_rollout WHERE server = ?').get(server)).toEqual({ operation_id: 'another-operation' }); + expect(db().prepare("SELECT COUNT(*) AS n FROM server_deployments WHERE action = 'rebuild'").get()).toEqual({ n: 0 }); + }); + + it('does not let an in-flight old restart observation release a transferred rebuild slot', async () => { + const { requestWatchdogLifecycle, escalateWatchdogRestart } = await import('../../src/lib/server/provisioning/index.js'); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { refreshRestartRollout } = await import('../../src/lib/server/provisioning/restarts.js'); + const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + await incident(); + const restart = requestWatchdogLifecycle(server, 'restart', 'incident'); + if (!restart.ok) throw new Error(restart.error); + await processProvisioning(); vi.setSystemTime(Date.now() + 300_000); + let reportHealthy!: (value: { ok: true; state: string }) => void; + vi.mocked(komodoInspectHealthy).mockImplementationOnce(() => new Promise(resolve => { reportHealthy = resolve; })); + const observation = refreshRestartRollout(); + const rebuild = escalateWatchdogRestart('incident', restart.deploymentId); + if (!rebuild.ok) throw new Error(rebuild.error); + reportHealthy({ ok: true, state: 'running' }); + await observation; + expect(db().prepare('SELECT operation_id,healthy_at FROM restart_rollout WHERE server = ?').get(server)) + .toEqual({ operation_id: rebuild.deploymentId, healthy_at: null }); + expect(await operation(restart.deploymentId)).toMatchObject({ state: 'failed' }); + expect(await operation(rebuild.deploymentId)).toMatchObject({ state: 'waiting' }); + }); + + it('resumes explicitly requested rebuild cleanup after the incident needs attention, even when Watchdog is paused', async () => { + const { requestWatchdogLifecycle, resumeRebuild } = await import('../../src/lib/server/provisioning/index.js'); + const { deploymentStage, finishDeployment, REMOVING_SERVER } = await import('../../src/lib/server/provisioning/history.js'); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { komodoDestroyDeployment } = await import('../../src/lib/server/integrations/komodo.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + await incident(); + const rebuild = requestWatchdogLifecycle(server, 'rebuild', 'incident'); + if (!rebuild.ok) throw new Error(rebuild.error); + deploymentStage(rebuild.deploymentId, REMOVING_SERVER); + finishDeployment(rebuild.deploymentId, { ok: false, message: 'Cleanup interrupted' }); + db().prepare("UPDATE watchdog_incidents SET state = 'attention' WHERE id = 'incident'").run(); + db().prepare("INSERT INTO settings (key,value) VALUES ('watchdog_enabled','0')").run(); + db().prepare('DELETE FROM servers WHERE name = ?').run(server); + expect(resumeRebuild(rebuild.deploymentId)).toBe(true); + await processProvisioning(); + expect(await operation(rebuild.deploymentId)).toMatchObject({ state: 'queued' }); + expect(komodoDestroyDeployment).not.toHaveBeenCalled(); + }); + + it.each(['restart','rebuild'] as const)('hands retained failed-rebuild capacity to an explicit new-cycle %s attempt', async action => { + const { requestWatchdogLifecycle, escalateWatchdogRestart } = await import('../../src/lib/server/provisioning/index.js'); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { refreshRestartRollout } = await import('../../src/lib/server/provisioning/restarts.js'); + const { komodoDestroyDeployment, komodoInspectHealthy, komodoRestart } = await import('../../src/lib/server/integrations/komodo.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + await incident(); + const restart = requestWatchdogLifecycle(server, 'restart', 'incident'); + if (!restart.ok) throw new Error(restart.error); + await processProvisioning(); vi.setSystemTime(Date.now() + 300_000); + const rebuild = escalateWatchdogRestart('incident', restart.deploymentId); + if (!rebuild.ok) throw new Error(rebuild.error); + db().prepare("INSERT INTO watchdog_actions (id,incident_id,action,operation_id,created_at) VALUES ('failed-rebuild','incident','rebuild',?,?)") + .run(rebuild.deploymentId, new Date().toISOString()); + vi.mocked(komodoDestroyDeployment).mockResolvedValueOnce({ ok: false, kind: 'unreachable', message: 'Teardown failed' }); + await processProvisioning(); + expect(await operation(rebuild.deploymentId)).toMatchObject({ state: 'failed' }); + db().prepare("UPDATE watchdog_incidents SET cycle = 2, state = 'open' WHERE id = 'incident'").run(); + const retry = requestWatchdogLifecycle(server, action, 'incident'); + if (!retry.ok) throw new Error(retry.error); + expect(db().prepare('SELECT operation_id,healthy_at FROM restart_rollout WHERE server = ?').get(server)) + .toEqual({ operation_id: retry.deploymentId, healthy_at: null }); + await refreshRestartRollout(); + // Old process health cannot release capacity before this retry submits. + expect(db().prepare('SELECT healthy_at FROM restart_rollout WHERE server = ?').get(server)).toEqual({ healthy_at: null }); + if (action === 'rebuild') vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'Stopped' }); + await processProvisioning(); + expect(await operation(retry.deploymentId)).toMatchObject({ state: action === 'restart' ? 'awaiting-health' : 'queued' }); + expect(komodoRestart).toHaveBeenCalledTimes(action === 'restart' ? 2 : 1); + }); + + it('keeps its linked drift campaign and file audit throughout rebuild, and verifies the replacement through telemetry', async () => { + const { requestWatchdogLifecycle } = await import('../../src/lib/server/provisioning/index.js'); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); + const { rconStatusResult } = await import('../../src/lib/server/integrations/rcon.js'); + const { db } = await import('../../src/lib/server/db/index.js'); + await incident(); + const rebuild = requestWatchdogLifecycle(server, 'rebuild', 'incident'); + if (!rebuild.ok) throw new Error(rebuild.error); + const now = new Date().toISOString(); + db().prepare("INSERT INTO deployments (id,action,started_at) VALUES ('drift-audit','drift',?)").run(now); + db().prepare("INSERT INTO deployment_servers (deployment_id,server,files_changed,restart_state,provisioning_id) VALUES ('drift-audit',?,1,'pending',?)").run(server, rebuild.deploymentId); + db().prepare("INSERT INTO deployment_files (deployment_id,server,verb,rel_path) VALUES ('drift-audit',?,'updated','script.gsc')").run(server); + db().prepare("INSERT INTO provisioning_sources VALUES (?,'campaign','drift-audit',?)").run(rebuild.deploymentId, server); + db().prepare("INSERT INTO watchdog_actions (id,incident_id,action,operation_id,campaign_id,created_at) VALUES ('action','incident','rebuild',?,'drift-audit',?)").run(rebuild.deploymentId, now); + vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'Stopped' }); + await processProvisioning(); + expect(await operation(rebuild.deploymentId)).toMatchObject({ state: 'queued' }); + expect(db().prepare("SELECT provisioning_id FROM deployment_servers WHERE deployment_id = 'drift-audit'").get()).toEqual({ provisioning_id: rebuild.deploymentId }); + expect(db().prepare("SELECT COUNT(*) AS n FROM deployment_files WHERE deployment_id = 'drift-audit'").get()).toEqual({ n: 1 }); + expect(db().pragma('foreign_key_check')).toEqual([]); + await processProvisioning(); + expect(await operation(rebuild.deploymentId)).toMatchObject({ state: 'awaiting-health' }); + vi.setSystemTime(Date.now() + 15_000); await report(0); + vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' }); + vi.mocked(rconStatusResult).mockResolvedValue({ ok: false, reason: 'timeout', detail: 'No reply', endpoint: null }); + await processProvisioning(); + expect(await operation(rebuild.deploymentId)).toMatchObject({ state: 'done' }); + }); +}); diff --git a/tests/unit/telemetry-diagnostics.test.ts b/tests/unit/telemetry-diagnostics.test.ts index 46154c4..940238e 100644 --- a/tests/unit/telemetry-diagnostics.test.ts +++ b/tests/unit/telemetry-diagnostics.test.ts @@ -183,9 +183,9 @@ it('still accepts and broadcasts telemetry if storing diagnostics fails', async expect(response.status).toBe(200); const { listTelemetry } = await import('../../src/lib/server/fleet/telemetry.js'); - expect(listTelemetry()).toEqual([{ ...payload, receivedAt: expect.any(String) }]); + expect(listTelemetry()).toEqual([expect.objectContaining({ ...payload, receivedAt: expect.any(String) })]); expect(listener).toHaveBeenCalledWith({ - type: 'server-telemetry', entries: [{ ...payload, receivedAt: expect.any(String) }] + type: 'server-telemetry', entries: [expect.objectContaining({ ...payload, receivedAt: expect.any(String) })] }); }); diff --git a/tests/unit/watchdog.test.ts b/tests/unit/watchdog.test.ts new file mode 100644 index 0000000..3379965 --- /dev/null +++ b/tests/unit/watchdog.test.ts @@ -0,0 +1,397 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { access, mkdir, readFile, writeFile } from 'node:fs/promises'; +import { join } from 'node:path'; +import { newFixture, type Fixture } from '../helpers/fixture.js'; +import { gameEvent } from '../helpers/telemetry.js'; + +vi.mock('../../src/lib/server/integrations/komodo.js', () => ({ + komodoRestart: vi.fn(), komodoRuntimeAction: vi.fn(), komodoInspectHealthy: vi.fn(), + komodoCreateDeployment: vi.fn(), komodoDeploy: vi.fn(), komodoDestroyDeployment: vi.fn(), komodoDeleteDeployment: vi.fn(), + komodoEnsureTag: vi.fn(), komodoUpdateTagsOnResource: vi.fn() +})); +vi.mock('../../src/lib/server/integrations/rcon.js', () => ({ rconGetPlayerCount: vi.fn(), rconStatusResult: vi.fn() })); +vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn(async () => {}) })); + +const server = 't5_zombie_theater_1'; +const healthyRcon = { ok: true as const, endpoint: { host: 'test', port: 28960 }, rawReply: '', + status: { players: 0, map: 'zombie_theater', gametype: 'zom', roster: [] } }; +const missingRcon = { ok: false as const, endpoint: { host: 'test', port: 28960 }, reason: 'timeout' as const, detail: 'No reply' }; +let fixture: Fixture; +type Engine = typeof import('../../src/lib/server/watchdog/index.js'); +let watchdog: Engine; +let database: typeof import('../../src/lib/server/db/index.js'); +let komodo: typeof import('../../src/lib/server/integrations/komodo.js'); +let rcon: typeof import('../../src/lib/server/integrations/rcon.js'); + +beforeEach(async () => { + vi.resetModules(); vi.resetAllMocks(); + vi.useFakeTimers({ toFake: ['Date'] }); + vi.setSystemTime(new Date('2026-09-27T12:00:00Z')); + fixture = await newFixture([server]); + for (const [key, value] of Object.entries({ KOMODO_ADDRESS: 'http://komodo.test', KOMODO_API_KEY: 'test', KOMODO_API_SECRET: 'test', + KOMODO_TARGET_SERVER: 'host', RCON_PASSWORD: 'test', DEPLOYMENT_CREATE_CONCURRENCY: '1', DEPLOYMENT_RESTART_CONCURRENCY: '1', + SERVER_PORT_MIN: '28960', SERVER_PORT_MAX: '28970' })) vi.stubEnv(key, value); + database = await import('../../src/lib/server/db/index.js'); + const { seedServerRegistry } = await import('../../src/lib/server/fleet/servers.js'); + await seedServerRegistry(); + database.db().prepare('UPDATE servers SET game_port = 28960 WHERE name = ?').run(server); + const { saveServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + saveServerKey(server, 'saved-key'); + komodo = await import('../../src/lib/server/integrations/komodo.js'); + for (const name of ['komodoRestart', 'komodoRuntimeAction', 'komodoCreateDeployment', 'komodoDeploy', 'komodoDestroyDeployment', 'komodoDeleteDeployment'] as const) { + vi.mocked(komodo[name]).mockResolvedValue({ ok: true }); + } + vi.mocked(komodo.komodoEnsureTag).mockImplementation(async name => ({ ok: true, id: name })); + vi.mocked(komodo.komodoUpdateTagsOnResource).mockResolvedValue({ ok: true }); + vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' }); + rcon = await import('../../src/lib/server/integrations/rcon.js'); + vi.mocked(rcon.rconGetPlayerCount).mockResolvedValue(0); + vi.mocked(rcon.rconStatusResult).mockResolvedValue(healthyRcon); + watchdog = await import('../../src/lib/server/watchdog/index.js'); +}); +afterEach(async () => { await fixture.cleanup(); vi.useRealTimers(); vi.unstubAllEnvs(); }); + +const incidents = () => watchdog.watchdogOverview().incidents; +const onlyIncident = () => { expect(incidents()).toHaveLength(1); return incidents()[0]; }; +const operations = () => database.db().prepare('SELECT id, action, state FROM server_deployments ORDER BY rowid').all() as { id: string; action: string; state: string }[]; +function advance(ms: number) { vi.setSystemTime(Date.now() + ms); } +function down() { vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'exited', reason: 'Container exited' }); } +async function sustainedFailure() { + down(); + await watchdog.watchdogTick(true); + advance(watchdog.WATCHDOG_SCAN_MS); await watchdog.watchdogTick(true); + advance(watchdog.WATCHDOG_SCAN_MS); await watchdog.watchdogTick(true); + expect(operations()).toEqual([expect.objectContaining({ action: 'restart', state: 'waiting' })]); + return onlyIncident(); +} + +describe('Watchdog detection and ownership', () => { + it('defaults to automatic recovery and debounces manual scans without accelerating the failure window', async () => { + expect(watchdog.watchdogOverview().enabled).toBe(true); + down(); + await watchdog.watchdogTick(true); + expect(onlyIncident()).toMatchObject({ state: 'open', failures: 1 }); + for (let n = 0; n < 5; n++) await watchdog.watchdogTick(true); + expect(onlyIncident().failures).toBe(1); + expect(operations()).toEqual([]); + advance(watchdog.WATCHDOG_SCAN_MS); + await watchdog.watchdogTick(); + expect(onlyIncident().failures).toBe(2); + advance(watchdog.WATCHDOG_SCAN_MS - 1); + await watchdog.watchdogTick(true); + expect(operations()).toEqual([]); + advance(1); + await watchdog.watchdogTick(true); + const row = onlyIncident(); + expect(row).toMatchObject({ state: 'recovering', failures: 3 }); + expect(watchdog.watchdogDetail(row.id)?.actions).toEqual([expect.objectContaining({ action: 'restart', operation_state: 'waiting' })]); + await watchdog.watchdogTick(true); + expect(operations()).toHaveLength(1); + expect(komodo.komodoRestart).not.toHaveBeenCalled(); + }); + + it('keeps paused recovery observational and resumes one pending incident', async () => { + watchdog.setWatchdogEnabled(false); + down(); + for (let n = 0; n < 4; n++) { await watchdog.watchdogTick(true); advance(watchdog.WATCHDOG_SCAN_MS); } + expect(onlyIncident()).toMatchObject({ state: 'open', failures: 4 }); + expect(operations()).toEqual([]); + database.closeDb(); + expect(watchdog.watchdogOverview().enabled).toBe(false); + watchdog.setWatchdogEnabled(true); + await watchdog.watchdogTick(true); + expect(operations()).toEqual([expect.objectContaining({ action: 'restart' })]); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + watchdog.setWatchdogEnabled(false); + await processProvisioning(); + expect(operations()[0].state).toBe('waiting'); + expect(komodo.komodoRestart).not.toHaveBeenCalled(); + watchdog.setWatchdogEnabled(true); + await processProvisioning(); + expect(komodo.komodoRestart).toHaveBeenCalledTimes(1); + }); + + it('does not restart an unknown Komodo state even after repeated sustained observations', async () => { + vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: false, reason: 'Komodo request timed out' }); + for (let n = 0; n < 8; n++) { await watchdog.watchdogTick(true); advance(watchdog.WATCHDOG_SCAN_MS); } + expect(onlyIncident()).toMatchObject({ kind: 'health', state: 'open' }); + expect(operations()).toEqual([]); + expect(rcon.rconStatusResult).not.toHaveBeenCalled(); + down(); + await watchdog.watchdogTick(true); + expect(onlyIncident().failures).toBe(1); + expect(operations()).toEqual([]); + advance(watchdog.WATCHDOG_SCAN_MS); await watchdog.watchdogTick(true); + expect(operations()).toEqual([]); + advance(watchdog.WATCHDOG_SCAN_MS); await watchdog.watchdogTick(true); + expect(operations()).toEqual([expect.objectContaining({ action: 'restart' })]); + }); + + it('allows a longer initial startup grace period before requesting recovery', async () => { + vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'starting', reason: 'Starting' }); + for (let n = 0; n < 5; n++) { await watchdog.watchdogTick(true); advance(watchdog.WATCHDOG_SCAN_MS); } + expect(operations()).toEqual([]); + await watchdog.watchdogTick(true); + expect(operations()).toEqual([expect.objectContaining({ action: 'restart' })]); + }); + + it('accepts fresh validated telemetry when RCON is unavailable and rechecks after it becomes stale', async () => { + const { saveTelemetry } = await import('../../src/lib/server/fleet/telemetry.js'); + vi.mocked(rcon.rconStatusResult).mockResolvedValue(missingRcon); + saveTelemetry({ ...gameEvent('heartbeat', 17, 4, server), receivedAt: new Date().toISOString() }); + await watchdog.watchdogTick(true); + expect(incidents()).toEqual([]); + expect(rcon.rconStatusResult).not.toHaveBeenCalled(); + advance(60_000); + await watchdog.watchdogTick(true); + expect(onlyIncident()).toMatchObject({ kind: 'health', state: 'open', failures: 1 }); + saveTelemetry({ ...gameEvent('heartbeat', 18, 4, server), receivedAt: new Date().toISOString() }); + await watchdog.watchdogTick(true); + expect(onlyIncident().state).toBe('resolved'); + expect(operations()).toEqual([]); + }); + + it('preserves intentional stop intent after completion and database reopening', async () => { + const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js'); + const stop = requestLifecycle(server, 'stop'); + expect(stop.ok).toBe(true); + if (!stop.ok) throw new Error(stop.error); + database.db().prepare("UPDATE server_deployments SET state = 'done', finished_at = ? WHERE id = ?").run(new Date().toISOString(), stop.deploymentId); + down(); + database.closeDb(); + for (let n = 0; n < 4; n++) { await watchdog.watchdogTick(true); advance(watchdog.WATCHDOG_SCAN_MS); } + expect(incidents()).toEqual([]); + expect(operations()).toEqual([expect.objectContaining({ action: 'stop' })]); + expect(komodo.komodoInspectHealthy).not.toHaveBeenCalled(); + }); + + it.each(['beta', 'campaign'] as const)('defers a server owned by %s work', async (owner) => { + if (owner === 'beta') { + const { createScript } = await import('../../src/lib/server/fleet/scripts.js'); + const { openBeta } = await import('../../src/lib/server/fleet/beta.js'); + const { script, version } = createScript({ name: 'beta', content: Buffer.from('experimental'), targetRelPath: 'beta.gsc' }); + await openBeta({ scripts: [{ scriptId: script.id, scriptVersionId: version.id }], servers: [server], waitForHealth: false, waitForMatch: false }); + } else { + database.db().prepare("INSERT INTO deployments (id, action, started_at) VALUES ('campaign', 'promote', ?)").run(new Date().toISOString()); + database.db().prepare("INSERT INTO deployment_servers (deployment_id, server, files_changed, restart_state) VALUES ('campaign', ?, 1, 'held')").run(server); + } + down(); + for (let n = 0; n < 4; n++) { await watchdog.watchdogTick(true); advance(watchdog.WATCHDOG_SCAN_MS); } + expect(incidents()).toEqual([]); + expect(operations()).toEqual([]); + expect(komodo.komodoInspectHealthy).not.toHaveBeenCalled(); + }); + + it('rechecks lifecycle ownership after a health probe yields to an operator stop', async () => { + const { requestLifecycle } = await import('../../src/lib/server/provisioning/requests.js'); + vi.mocked(komodo.komodoInspectHealthy).mockImplementationOnce(async () => { + expect(requestLifecycle(server, 'stop').ok).toBe(true); + return { ok: false, state: 'exited', reason: 'Container exited' }; + }); + await watchdog.watchdogTick(true); + expect(incidents()).toEqual([]); + expect(operations()).toEqual([expect.objectContaining({ action: 'stop' })]); + }); +}); + +describe('Watchdog recovery persistence', () => { + it('resolves an attention incident after an intentional successful destroy', async () => { + const { removeServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + const { destroyServer, processDestructions } = await import('../../src/lib/server/provisioning/instances.js'); + removeServerKey(server); + await writeFile(join(fixture.t5, server, 'runtime/plutonium/storage/t5/raw/scripts/sp', 'stray.gsc'), 'unexpected'); + await watchdog.watchdogTick(true); + const id = onlyIncident().id; + expect(onlyIncident().state).toBe('attention'); + expect(destroyServer(server)).toMatchObject({ ok: true }); + down(); + await processDestructions(); + expect(operations()).toEqual([expect.objectContaining({ action: 'destroy', state: 'done' })]); + expect(database.db().prepare('SELECT name FROM servers WHERE name = ?').get(server)).toBeUndefined(); + await watchdog.watchdogTick(true); + expect(watchdog.watchdogDetail(id)?.incident).toMatchObject({ state: 'resolved', finished_at: expect.any(String) }); + expect(watchdog.watchdogDetail(id)?.events.at(-1)?.message).toMatch(/destroyed|registered|retired/i); + }); + + it('requests one restart, escalates once to rebuild, and retains exhausted attention across reopening', async () => { + const row = await sustainedFailure(); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + await processProvisioning(); + expect(komodo.komodoRestart).toHaveBeenCalledTimes(1); + expect(operations()[0].state).toBe('awaiting-health'); + advance(watchdog.WATCHDOG_RECOVERY_MS - 1); + await watchdog.watchdogTick(true); + expect(operations()).toHaveLength(1); + advance(1); + await watchdog.watchdogTick(true); + expect(operations().map(op => op.action)).toEqual(['restart', 'rebuild']); + expect(watchdog.watchdogDetail(row.id)?.actions).toHaveLength(2); + vi.mocked(komodo.komodoDestroyDeployment).mockResolvedValue({ ok: false, kind: 'unreachable', message: 'Host unavailable' }); + await processProvisioning(); + expect(operations()[1].state).toBe('failed'); + await watchdog.watchdogTick(true); + expect(watchdog.watchdogDetail(row.id)?.incident.state).toBe('attention'); + database.closeDb(); + for (let n = 0; n < 4; n++) { advance(watchdog.WATCHDOG_RECOVERY_MS); await watchdog.watchdogTick(true); } + expect(operations().map(op => op.action)).toEqual(['restart', 'rebuild']); + expect(watchdog.watchdogDetail(row.id)?.incident.state).toBe('attention'); + expect(komodo.komodoRestart).toHaveBeenCalledTimes(1); + expect(komodo.komodoDestroyDeployment).toHaveBeenCalledTimes(1); + }); + + it('does not escalate a timed out restart while automatic recovery is paused', async () => { + await sustainedFailure(); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + await processProvisioning(); + watchdog.setWatchdogEnabled(false); + advance(watchdog.WATCHDOG_RECOVERY_MS); + await watchdog.watchdogTick(true); + expect(operations()).toHaveLength(1); + watchdog.setWatchdogEnabled(true); + await watchdog.watchdogTick(true); + expect(operations().map(op => op.action)).toEqual(['restart', 'rebuild']); + }); + + it('resumes a persisted detection window and does not duplicate queued work after reopening', async () => { + down(); + await watchdog.watchdogTick(true); + advance(watchdog.WATCHDOG_SCAN_MS); await watchdog.watchdogTick(true); + const id = onlyIncident().id; + database.closeDb(); + advance(watchdog.WATCHDOG_SCAN_MS); await watchdog.watchdogTick(true); + expect(onlyIncident().id).toBe(id); + const queued = operations(); + expect(queued).toHaveLength(1); + database.closeDb(); + advance(watchdog.WATCHDOG_SCAN_MS); await watchdog.watchdogTick(true); + expect(operations()).toEqual(queued); + expect(watchdog.watchdogDetail(id)?.actions).toHaveLength(1); + }); + + it('permits an explicit new attempt after dismissal without silently rearming a sustained condition', async () => { + down(); + await watchdog.watchdogTick(true); + const id = onlyIncident().id; + expect(watchdog.updateWatchdogIncident(id, 'dismiss')).toEqual({ ok: true }); + for (let n = 0; n < 4; n++) { advance(watchdog.WATCHDOG_SCAN_MS); await watchdog.watchdogTick(true); } + expect(operations()).toEqual([]); + expect(onlyIncident().state).toBe('dismissed'); + expect(watchdog.updateWatchdogIncident(id, 'retry')).toEqual({ ok: true }); + await watchdog.watchdogTick(true); + expect(operations()).toEqual([]); + advance(watchdog.WATCHDOG_SCAN_MS); await watchdog.watchdogTick(true); + advance(watchdog.WATCHDOG_SCAN_MS); await watchdog.watchdogTick(true); + expect(onlyIncident()).toMatchObject({ id, state: 'recovering', cycle: 2 }); + expect(watchdog.watchdogDetail(id)?.actions).toEqual([expect.objectContaining({ action: 'restart', cycle: 2 })]); + }); +}); + +describe('Watchdog APIs', () => { + const request = (body: unknown, raw = false) => new Request('http://localhost/api/watchdog', { + method: 'POST', headers: { 'Content-Type': 'application/json' }, body: raw ? String(body) : JSON.stringify(body) + }); + + it('returns overview state and validates scan actions and pause booleans', async () => { + const { GET, POST } = await import('../../src/routes/api/watchdog/+server.js'); + expect(await GET().json()).toEqual({ enabled: true, lastScanAt: null, scanError: null, incidents: [] }); + expect((await POST({ request: request('{', true) })).status).toBe(400); + for (const body of [null, [], {}, { action: 'restart' }, { enabled: 'false' }, { enabled: 0 }, { action: 'scan', enabled: 'false' }]) { + expect((await POST({ request: request(body) })).status).toBe(400); + } + expect(await (await POST({ request: request({ enabled: false }) })).json()).toMatchObject({ enabled: false }); + database.closeDb(); + expect(await GET().json()).toMatchObject({ enabled: false }); + down(); + const scanned = await POST({ request: request({ action: 'scan' }) }); + expect(scanned.status).toBe(200); + expect(await scanned.json()).toMatchObject({ enabled: false, lastScanAt: expect.any(String), incidents: [expect.objectContaining({ kind: 'health', state: 'open' })] }); + expect(operations()).toEqual([]); + }); + + it('validates incident actions and refuses retries while provisioning is pending', async () => { + const { GET, POST } = await import('../../src/routes/api/watchdog/[id]/+server.js'); + expect(GET({ params: { id: 'missing' } }).status).toBe(404); + expect((await POST({ params: { id: 'missing' }, request: request({ action: 'retry' }) })).status).toBe(404); + for (const body of [null, {}, [], { action: 'restart' }]) { + expect((await POST({ params: { id: 'missing' }, request: request(body) })).status).toBe(400); + } + expect((await POST({ params: { id: 'missing' }, request: request('{', true) })).status).toBe(400); + const row = await sustainedFailure(); + expect(await GET({ params: { id: row.id } }).json()).toMatchObject({ + incident: { id: row.id, state: 'recovering' }, actions: [expect.objectContaining({ action: 'restart' })], events: expect.any(Array) + }); + for (const action of ['retry', 'dismiss']) { + const response = await POST({ params: { id: row.id }, request: request({ action }) }); + expect(response.status).toBe(409); + expect(await response.json()).toMatchObject({ error: expect.stringContaining('Provisioning is still active') }); + } + expect(operations()).toHaveLength(1); + }); + + it('keeps legacy drift previews readable and directs mutations to Watchdog without changing files', async () => { + const { GET, POST } = await import('../../src/routes/api/drift/+server.js'); + const path = join(fixture.t5, server, 'runtime/plutonium/storage/t5/raw/scripts/sp', 'stray.gsc'); + await writeFile(path, 'unexpected content'); + expect(await (await GET()).json()).toMatchObject({ totalDrift: 1, servers: [{ server, files: [expect.objectContaining({ relPath: 'stray.gsc' })] }] }); + const preview = await POST({ request: request({ dryRun: true, servers: [server] }) }); + expect(preview.status).toBe(200); + const body = await preview.json(); + expect(body).toMatchObject({ ok: true, plan: { servers: [server], fingerprint: expect.any(String), files: [expect.objectContaining({ relPath: 'stray.gsc' })] } }); + const mutation = await POST({ request: request({ servers: [server], expectedFingerprint: body.plan.fingerprint }) }); + expect(mutation.status).toBe(410); + expect(await mutation.json()).toMatchObject({ ok: false, watchdogUrl: '/watchdog', error: expect.stringContaining('Watchdog') }); + expect(await readFile(path, 'utf8')).toBe('unexpected content'); + expect(operations()).toEqual([]); + expect(database.db().prepare('SELECT id FROM deployments').all()).toEqual([]); + }); +}); + +describe('Watchdog drift campaigns', () => { + it('queues an audited rebuild without changing files until provisioning executes and retains its campaign through cleanup', async () => { + await mkdir(join(fixture.t5, server, 'configs'), { recursive: true }); + const path = join(fixture.t5, server, 'configs', 'stray.cfg'); + await writeFile(path, 'unexpected content'); + await watchdog.watchdogTick(true); + const row = onlyIncident(); + expect(row).toMatchObject({ kind: 'drift', state: 'recovering' }); + const action = watchdog.watchdogDetail(row.id)!.actions[0]; + expect(action).toMatchObject({ action: 'rebuild', campaign_id: expect.any(String), operation_state: 'waiting' }); + expect(await readFile(path, 'utf8')).toBe('unexpected content'); + expect(komodo.komodoDestroyDeployment).not.toHaveBeenCalled(); + expect(database.db().prepare('SELECT rel_path, content_before, content_after FROM deployment_files WHERE deployment_id = ?').all(action.campaign_id)) + .toEqual([{ rel_path: 'stray.cfg', content_before: Buffer.from('unexpected content'), content_after: null }]); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { processCreations } = await import('../../src/lib/server/provisioning/instances.js'); + const { sweep } = await import('../../src/lib/server/pipeline/sweep.js'); + down(); + await processProvisioning(); + expect(operations()).toEqual([expect.objectContaining({ action: 'rebuild', state: 'queued' })]); + await expect(access(path)).rejects.toThrow(); + expect(database.db().prepare('SELECT name FROM servers WHERE name = ?').get(server)).toBeUndefined(); + database.closeDb(); + await sweep(); await watchdog.watchdogTick(true); + expect(watchdog.watchdogDetail(row.id)?.incident.state).toBe('recovering'); + expect(database.db().prepare('SELECT provisioning_id, restart_state FROM deployment_servers WHERE deployment_id = ?').get(action.campaign_id)) + .toEqual({ provisioning_id: action.operation_id, restart_state: 'pending' }); + await processCreations(); + vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' }); + await processCreations(); await sweep(); await watchdog.watchdogTick(true); + expect(watchdog.watchdogDetail(row.id)?.incident.state).toBe('resolved'); + expect(database.db().prepare('SELECT restart_state FROM deployment_servers WHERE deployment_id = ?').get(action.campaign_id)).toEqual({ restart_state: 'done' }); + expect(operations()).toEqual([expect.objectContaining({ action: 'rebuild', state: 'done' })]); + expect(komodo.komodoRestart).not.toHaveBeenCalled(); + }); + + it('keeps a missing rebuild key visible as attention without deleting files or retrying automatically', async () => { + const { removeServerKey } = await import('../../src/lib/server/fleet/server-keys.js'); + removeServerKey(server); + const path = join(fixture.t5, server, 'runtime/plutonium/storage/t5/raw/scripts/sp', 'stray.gsc'); + await writeFile(path, 'unexpected'); + await watchdog.watchdogTick(true); + expect(onlyIncident()).toMatchObject({ kind: 'drift', state: 'attention', summary: expect.stringContaining('Save a server key') }); + advance(watchdog.WATCHDOG_SCAN_MS); await watchdog.watchdogTick(true); + expect(operations()).toEqual([]); + expect(await readFile(path, 'utf8')).toBe('unexpected'); + expect(komodo.komodoDestroyDeployment).not.toHaveBeenCalled(); + }); +});