diff --git a/README.md b/README.md index c940327..9590bef 100644 --- a/README.md +++ b/README.md @@ -67,56 +67,39 @@ status between refreshes. Failed refreshes keep the last snapshot visible. Plugins support named release versions, GitHub DLL imports and update discovery. See [Plugin releases](docs/plugin-releases.md) for the import and beta workflow. -## Plutonium updates and scheduled restarts +## Maintenance -Open **Settings → Plutonium updates & restarts** to check the latest published -revision, inspect each server's running revision, and queue a server or fleet -restart. Release and running-version checks also run hourly. The published -revision comes from the [official production updater manifest](https://cdn.plutoniummod.com/updater/prod/info.json), -which exposes the game revision directly and avoids the changelog website's browser challenge; -running revisions come from RCON `version`. Unsupported replies and failed -queries appear as unknown rather than being treated as up to date. A failed -manifest request retains the last release with a visible error. +Open **Operations → Maintenance** for health recovery, configuration drift, +Plutonium updates, and scheduled restarts. Overview shows active work, attention, +version coverage, and upcoming restarts. Policies & schedules controls each +capability independently; Activity links findings and results to Provisioning. +Fleet and server pages show runtime revision indicators and maintenance links. -Daily fleet restarts are optional and disabled by default. Choose a queue time -in your **selected timezone** and save the schedule. The timezone is saved with -the schedule so the daily time follows daylight saving changes even when your -browser is closed. Skipped times run at the first valid minute after the gap; -repeated times run once on their first occurrence. Existing UTC schedules retain -their original timing until saved again. The form displays their next occurrence -in your timezone and identifies the saved zone if it differs. Changing a browser's -display preference does not silently change a fleet schedule; save to apply it. -Every due run queues the current fleet, regardless of whether an update exists. -The schedule and jobs survive app restarts. Missed days become one run, and a -server can have only one pending maintenance restart. Disabling the schedule -does not cancel work already queued; use **Cancel** for those jobs. +Maintenance decides when work is needed and verifies the result. Provisioning +executes every lifecycle operation using the shared safety checks and queues. +Health recovery uses one restart followed by one rebuild if needed; drift repair +requests a rebuild with a linked Campaign. **Rebuild deletes all instance files.** -Maintenance waits for a fresh zero-player RCON or telemetry report and for existing -deployment/beta work, then uses the shared rolling restart limit. Unreachable or -occupied servers remain queued. After container and game-status recovery, Electric -Cherry queries the running revision again. If a fresh published revision was -known when queued, recovery on an older or unknown revision is reported as a -failed verification, without an automatic restart loop. Ambiguous restart -failures likewise require inspection before a new request. Activity is retained -for 30 days, with pending jobs and the latest 100 records shown in Settings. +Plutonium updates are applied by Plutainer's startup updater +(`PLUTAINER_AUTO_UPDATE` must be enabled). Check versions and choose **Update on +restart**, or opt into automatic updates for verified outdated servers. Revision +checks run hourly. Unknown or stale revisions remain unverified. A healthy restart +can still fail update verification; Maintenance reports that outcome without +rebuilding or repeatedly restarting the server. Electric Cherry does not download +binaries or pin a runtime revision. -Updates rely on Plutainer's startup updater (`PLUTAINER_AUTO_UPDATE` must be -enabled); Electric Cherry does not download binaries or pull a container image. -See [Plutainer's startup flow](https://github.com/Ayymoss/Plutainer/blob/main/CLAUDE.md) -and [environment migration reference](https://github.com/Ayymoss/Plutainer/blob/main/MIGRATION.md). -The worker starts on the first application request, so keep the app running and -reachable for scheduled maintenance. Validate RCON revision replies and updater -behavior on your live T5/T6 servers before relying on unattended maintenance. +Optional daily restarts are independent of updates. Schedules use the saved +timezone, follow daylight saving changes, and coalesce missed days into one run. +Intentionally stopped servers are excluded. Existing schedules, pending jobs, +operation IDs, and history survive the move from Settings and Watchdog. + +**Pause all maintenance** holds new and queued maintenance while checks and +submitted recovery continue. Each capability retains its own policy. Automatic +updates remain disabled until explicitly enabled. See [Maintenance](docs/maintenance.md) +for verification, retry, scheduling, and compatibility details. ## How restarts stay safe -**Operations → Watchdog** automatically detects unhealthy servers and file drift. -Health incidents receive one restart followed by one rebuild if recovery fails; -drift queues a rebuild with the promoted setup and a linked Campaign. Incidents, -evidence, and attempts survive application restarts. Pause automation, inspect -failures, and explicitly retry from Watchdog. Intentional stops and active beta or -campaign work are respected. See [Watchdog](docs/watchdog.md) for timing and recovery behavior. - Betas support configurable gameplay, participation, stability and manual-review requirements, saved presets, coverage rules and staged waves. See [Beta test plans](docs/beta-test-plans.md) for usage and the four-event game telemetry contract. diff --git a/docs/maintenance.md b/docs/maintenance.md new file mode 100644 index 0000000..f73fdb4 --- /dev/null +++ b/docs/maintenance.md @@ -0,0 +1,138 @@ +# Maintenance + +Open **Operations → Maintenance** to keep existing servers healthy and current. +Maintenance owns observations, policies, schedules, requests, and verification of +maintenance outcomes. Provisioning owns lifecycle execution, player checks, +conflicts, rolling capacity, container recovery, and persistent operation history. +Campaigns retain managed-file deployment evidence; Collections define lineups. + +## Views + +- **Overview** shows active work, attention, upcoming restarts, and Plutonium version + coverage. Filter by capability or server. **Check now** refreshes health, drift, + the production release, and running revisions. +- **Policies & schedules** controls health recovery, configuration drift, automatic + Plutonium updates, and the optional daily restart schedule independently. +- **Activity** combines recovery findings, update results, and scheduled restarts. + Follow a finding for evidence and attempts, or its Provisioning operation for + execution progress. Cancel unsubmitted runtime restarts here. + +Fleet and server details link their revision indicators to Maintenance. Failed +runtime maintenance appears in Fleet attention, even when the underlying restart +succeeded. Provisioning and repair Campaigns link back to the requesting work. + +## Policies and pause + +Health recovery and drift repair inherit the previous Watchdog enabled setting. +Automatic Plutonium updates are **disabled by default**. Existing daily schedules +retain their enabled state, timezone, and next occurrence. + +**Pause all maintenance** holds new remediation, scheduled runs, and unsubmitted +maintenance operations. Scans and result verification continue; submitted +operations finish recovery. Campaigns, betas, and manually requested Provisioning +operations continue independently. Resuming keeps each policy's saved setting. + +Disabling health recovery or drift repair holds queued work for that capability +and prevents further automatic attempts, including health escalation. Disabling +automatic updates holds queued automatic update requests. Explicitly requested +updates remain operator-authorized work. Disabling a daily schedule prevents +future runs but does not cancel requests it already created. + +All capabilities respect intentionally stopped servers. Start them explicitly +through Provisioning before expecting maintenance to act on them. + +## Plutonium updates + +The published revision comes from the official production updater manifest. +Running revisions come from RCON `version` after checking container readiness. +Checks run hourly while the application is running. Failed release checks retain +the cached release with an error. Unsupported replies, stale checks, unhealthy +containers, and evidence from before a lifecycle submission remain unverified. +A missing revision is never treated as current or as an automatic update target. + +**Update on restart** requests ordinary Provisioning restarts for verified outdated +servers. **Update outdated servers on restart** applies to the filtered list. +Automatic updates, when enabled, apply to verified outdated servers across the +fleet, with one attempt per server and published revision. Existing pending +maintenance is reused. Occupied servers and servers with conflicting beta or +campaign work wait without consuming a restart slot. + +Plutainer's startup updater must be enabled (`PLUTAINER_AUTO_UPDATE`). Electric +Cherry does not download binaries, pull a container image, or pin the runtime to +a particular revision. The observed published revision is recorded as the minimum +expected result; the actual revision is queried after recovery. + +A healthy restart and a successful update are separate outcomes. If the server +recovers on an older or unknown revision, the restart is successful in Provisioning +and the update needs attention in Maintenance. Update verification never triggers +a rebuild or an automatic restart loop. Inspect the updater configuration and +server logs, then use **Retry after inspection** to create a new attempt. Ambiguous +or failed lifecycle requests require inspection of the existing Provisioning +operation and its retained recovery slot first. + +Update attempts persist as suppression evidence across app restarts and history +retention. Cancelling an automatic attempt suppresses another automatic attempt for +that same revision; an operator can request an update explicitly. + +## Scheduled restarts + +Daily restarts are independent of update availability. At the saved queue time, +Maintenance requests a restart for each server whose desired state is running. +These requests have no revision requirement. Busy servers can restart later. + +Saving a schedule uses the selected timezone and follows daylight saving changes. +Skipped times run at the first valid minute after the gap; repeated times run once +on their first occurrence. Legacy UTC schedules keep their timing until saved. +Changing the browser display timezone does not change the saved schedule. + +Schedules survive application restarts. Missed days, including while globally +paused, become one run on resume. Pending work is deduplicated per server. The +worker starts on the first application request, so keep the app running and +reachable. Finished scheduled and legacy maintenance activity is retained for +30 days; update attempt records remain durable. + +## Health recovery and configuration drift + +The health observer (the watchdog) scans once per minute. Three observations over +at least two minutes are required before an automatic health restart; starting or +restarting containers get five minutes. Unknown Komodo responses are observations, +not proof of a stopped container. Repeated manual checks do not accelerate this +window. + +A health finding gets one restart, then one rebuild if recovery fails after five +minutes. Escalation transfers only that finding's restart slot. A failed rebuild +requires operator attention. **Rebuild deletes all instance files** and uses the +latest promoted setup, the same server identity and port, and the saved server key. +Missing keys block teardown. Submitted rebuild verification continues while paused. + +Drift includes missing promoted files, changed bytes, and extra files in managed +script, config, and plugin directories. Config comparison includes the composed +baseline, overlays, and the selected map. Unmanaged default config is exempt until +a baseline is promoted. Missing mounts are scan errors, not empty directories. + +Drift repair creates a Campaign capturing observed differences and requests a +Provisioning rebuild. The Campaign observes that rebuild without requesting +another restart. Its detection evidence and the actual launch recipe remain +available even if promotions change while the work waits. Histories survive +instance removal. Active beta and campaign work defers new recovery. + +**Dismiss** suppresses health/drift attempts until the condition clears. **Retry +recovery** begins a fresh bounded attempt after inspection. Existing operations +must finish or be cancelled first. Fresh verification resolves the finding. + +## Compatibility and API + +Existing job, incident, and operation IDs are retained. Upgrade migration adds +request metadata without replaying commands; existing pending requests keep their +original verification requirements. `/watchdog` and `/watchdog/:id` redirect to +Maintenance, preserving query parameters. Old Watchdog and Plutonium APIs remain +compatible. The previous Watchdog enabled switch controls health and drift only. + +- `GET /api/maintenance`: policies, recovery findings, release/version coverage, + schedule, and recent runtime activity with linked operation stages. +- `POST /api/maintenance`: actions `scan`, `pause` (`paused` boolean), `policy` + (`policy`: `health`, `drift`, or `updates`; `enabled` boolean), `update` (optional + `servers` array), `schedule` (`enabled`, `time`, `timeZone`), `cancel` or `retry` + (`id`, the numeric runtime maintenance activity ID). +- `/api/maintenance/:id`: GET recovery evidence and attempts; POST `retry` or + `dismiss` for a health/drift finding. diff --git a/docs/provisioning.md b/docs/provisioning.md index 63bd6d5..5ff27a0 100644 --- a/docs/provisioning.md +++ b/docs/provisioning.md @@ -49,11 +49,11 @@ Provisioning and observe the resulting operation by ID. Their pages link to that operation, and its history links back to the requesting work. Provisioning owns the empty-server check, conflict handling, restart capacity, remote command, and health verification. The callers retain file deployment, beta evidence, and -Plutonium revision verification. Jobs remain visible while waiting for players, +maintenance outcome verification. Jobs remain visible while waiting for players, other work, or integration configuration. -[Watchdog](watchdog.md) also submits recovery operations and links each attempt -to its incident. A failed Watchdog restart can hand its reserved slot to a rebuild +[Maintenance](maintenance.md) also submits recovery operations and links each attempt +to its incident. A failed Maintenance restart can hand its reserved slot to a rebuild after five minutes; unrelated recovery slots remain held. Drift campaigns follow their linked rebuild without requesting another restart. Their file evidence and history remain available after the instance is removed. diff --git a/docs/watchdog.md b/docs/watchdog.md index b83f876..b19a974 100644 --- a/docs/watchdog.md +++ b/docs/watchdog.md @@ -1,80 +1,7 @@ -# Watchdog +# Watchdog has moved to Maintenance -**Operations → Watchdog** detects unhealthy servers and drift, records each -incident, and links its recovery attempts to Provisioning. File drift also opens -a Campaign with the observed file differences. Automation is enabled by default; -**Pause automation** persists across application restarts. Scanning and verification -continue while paused, but new actions and queued Watchdog actions are held. -Already submitted actions continue to completion. +Open **Operations → Maintenance** for health recovery and drift repair, together +with Plutonium updates and scheduled restarts. Existing Watchdog URLs redirect +to Maintenance; incidents, operation IDs, and history are preserved. -The worker scans once per minute while Electric Cherry is running. **Scan now** -requests a scan immediately. Repeated button presses do not count as independent -health failures. A health incident must have three observations spanning at least -two minutes before an automatic restart is queued. Starting or restarting -containers get five minutes before recovery is requested. Unknown Komodo responses -are recorded without interpreting them as a stopped container. - -## Recovery - -A health incident gets one restart, then one rebuild if the restart has not -recovered within five minutes. Both are normal Provisioning jobs with durable -history and shared capacity limits. Escalation transfers only that incident's -restart recovery slot to the rebuild. An unsuccessful rebuild requires operator -attention; Watchdog does not start an endless restart/rebuild loop. Verification -of an already submitted rebuild continues, including after the incident is marked -as needing attention. - -An intentional **Stop** is retained as the server's desired state. Watchdog leaves -it stopped until an explicit Start, Restart, or Rebuild request changes that intent. -Active creations, removals, betas, and campaigns defer new Watchdog actions. - -Provisioning checks current container health before using game status. A disruptive -action requires a confirmed stopped container or a fresh zero-player reading from -RCON or accepted telemetry. A positive player count from either source prevents -an interruption. Telemetry evidence must have been received within 45 seconds; -future, malformed, stale, and pre-restart reports cannot verify a new process. -A game-end event retains its reported roster; it does not imply zero players. -Fresh telemetry can also verify game activity and the selected engine map after -Komodo reports the container healthy. A healthy container with neither current -RCON status nor current telemetry remains queued for player-count verification. - -## Drift - -Drift includes missing promoted files, changed bytes, and extra files in managed -script, config, and plugin destinations. Config comparison uses the composed -baseline and overlays with the server's selected map. Plutainer's unmanaged default -config remains exempt until a baseline is promoted. Active betas are deferred; -missing server mounts are reported as scan errors instead of being treated as -empty directories. - -Watchdog rebuilds a drifting server in place using the same identity, map, port, -and saved server key. **Rebuild deletes all old instance files.** The new instance -uses the latest promoted setup when provisioning starts. A missing saved key -leaves the incident needing attention without deleting the old server. - -The linked Campaign captures before/after evidence at detection time. Its activity -identifies this as the detected plan, and it follows the rebuild's outcome without -queuing an additional restart. Promotions can change while the rebuild is queued; -the Provisioning launch recipe records the files actually selected for that launch. -Campaign evidence and Watchdog history survive subsequent rebuilds and removals. - -## Review and retry - -The list can be filtered by server, incident type, and status. Each detail page -shows the triggering evidence, timestamped events, and links to every attempt. -**Dismiss** suppresses further automatic attempts until the condition clears. -**Retry** explicitly starts a new bounded attempt after inspecting a dismissed or -failed incident. Active Provisioning operations must finish or be cancelled first. -A verified recovery resolves the incident automatically. - -## API - -- `GET /api/watchdog`: automation setting, last scan, scan errors, and latest incidents. -- `POST /api/watchdog` with `{ "action": "scan" }`: run a scan now. -- `POST /api/watchdog` with `{ "enabled": false }` (or `true`): pause/resume automation. -- `GET /api/watchdog/:id`: incident, event history, and linked actions. -- `POST /api/watchdog/:id` with `{ "action": "retry" }` or `{ "action": "dismiss" }`. - -The legacy `/api/drift` GET and dry-run POST still return previews. Mutating POST -requests return HTTP 410 and direct clients to Watchdog; file repair is now a -provisioning rebuild instead of an immediate rewrite. +See [Maintenance](maintenance.md) for policies, recovery behavior, and the API. diff --git a/src/lib/fleet-overview.ts b/src/lib/fleet-overview.ts index ef603a6..b0e276c 100644 --- a/src/lib/fleet-overview.ts +++ b/src/lib/fleet-overview.ts @@ -7,6 +7,7 @@ import type { Health, RosterPlayer } from './ui/stores/servers.js'; import { mapTitle, serverTitle } from './ui/server-labels.js'; export interface FleetServer { + plutonium?: import('./plutonium.js').PlutoniumVersion; container_state?: string | null; container_checked_at?: string | null; last_checked_at?: string | null; @@ -37,7 +38,7 @@ export interface FleetOperation { key: string; group: string; server: string; - kind: 'campaign' | 'deployment' | 'beta' | 'recovery'; + kind: 'campaign' | 'deployment' | 'beta' | 'recovery' | 'maintenance'; label: string; detail: string; href: string; diff --git a/src/lib/maintenance.ts b/src/lib/maintenance.ts new file mode 100644 index 0000000..7f963e6 --- /dev/null +++ b/src/lib/maintenance.ts @@ -0,0 +1,10 @@ +import type { WatchdogOverview } from './watchdog.js'; +import type { PlutoniumStatus } from './plutonium.js'; + +export type MaintenancePolicy = 'health' | 'drift' | 'updates'; +export interface MaintenancePolicies { paused: boolean; health: boolean; drift: boolean; updates: boolean } +export interface MaintenanceOverview { + policies: MaintenancePolicies; + recovery: WatchdogOverview; + runtime: PlutoniumStatus; +} diff --git a/src/lib/plutonium.ts b/src/lib/plutonium.ts index d34594d..4b93c2a 100644 --- a/src/lib/plutonium.ts +++ b/src/lib/plutonium.ts @@ -16,6 +16,8 @@ export interface PlutoniumVersion { revision: number | null; checked_at: string | null; error: string | null; + stopped?: boolean; + update_status?: 'outdated' | 'current' | 'ahead' | 'unknown' | 'stopped'; } export interface MaintenanceJob { id: number; @@ -27,6 +29,10 @@ export interface MaintenanceJob { provisioning_id: string | null; finished_at: string | null; detail: string | null; + kind: 'legacy' | 'update' | 'schedule'; + origin: 'manual' | 'automatic' | 'schedule'; + operation_stage?: string | null; + operation_state?: string | null; } export interface PlutoniumStatus { release: PlutoniumRelease; diff --git a/src/lib/server/db/index.ts b/src/lib/server/db/index.ts index 93114e9..ac0a11b 100644 --- a/src/lib/server/db/index.ts +++ b/src/lib/server/db/index.ts @@ -310,6 +310,10 @@ export function db(): Database.Database { for (const table of ['deployment_servers', 'beta_servers', 'maintenance_jobs']) { migrateTable(handle, table, [{ name: 'provisioning_id', ddl: `ALTER TABLE ${table} ADD COLUMN provisioning_id TEXT` }]); } + migrateTable(handle, 'maintenance_jobs', [ + { name: 'kind', ddl: "ALTER TABLE maintenance_jobs ADD COLUMN kind TEXT NOT NULL DEFAULT 'legacy'" }, + { name: 'origin', ddl: "ALTER TABLE maintenance_jobs ADD COLUMN origin TEXT NOT NULL DEFAULT 'manual'" } + ]); migrateServers(handle); backfillDesiredState(handle); // Column additions must run BEFORE any statement that references the diff --git a/src/lib/server/fleet/overview.ts b/src/lib/server/fleet/overview.ts index f610cfa..3aa18d1 100644 --- a/src/lib/server/fleet/overview.ts +++ b/src/lib/server/fleet/overview.ts @@ -64,6 +64,15 @@ export function fleetOperations(now = Date.now()): FleetOperation[] { state: row.stage === 'failed' ? 'attention' : 'active', label: stages[row.stage] ?? row.stage, detail: row.error ?? recovering.get(row.server)?.error ?? `Beta ${row.id}`, href: `/betas/${row.id}` }); } + const maintenance = handle.prepare(`SELECT m.* FROM maintenance_jobs m WHERE m.state = 'failed' + AND NOT EXISTS (SELECT 1 FROM maintenance_jobs n WHERE n.server = m.server AND n.kind = m.kind AND n.id > m.id)`) + .all() as { id: number; server: string; kind: string; detail: string | null; requested_at: string }[]; + for (const job of maintenance) { + operations.push({ key: `maintenance:${job.id}`, group: `maintenance:${job.id}`, kind: 'maintenance', + server: job.server, state: 'attention', startedAt: job.requested_at, + label: job.kind === 'update' ? 'Update verification needs attention' : 'Maintenance needs attention', + detail: job.detail ?? 'Inspect the maintenance result', href: `/maintenance?view=activity&server=${encodeURIComponent(job.server)}#job-${job.id}` }); + } for (const slot of slots) { if (representedRecovery.has(slot.server)) continue; operations.push({ key: `recovery:${slot.server}:${slot.requested_at}`, group: `recovery:${slot.server}`, diff --git a/src/lib/server/fleet/plutonium.ts b/src/lib/server/fleet/plutonium.ts index 5766be3..67b9bd6 100644 --- a/src/lib/server/fleet/plutonium.ts +++ b/src/lib/server/fleet/plutonium.ts @@ -1,156 +1,2 @@ -import { db } from '../db/index.js'; -import { config } from '../config.js'; -import { getServer, listServers } from './servers.js'; -import { rconPlutoniumVersion } from '../integrations/rcon.js'; -import { requestRestart, cancelProvisioning, restartRecoveryPending } from '../provisioning/index.js'; -import { nextMaintenanceAt, parsePlutoniumManifest, PLUTONIUM_MANIFEST_URL } from '../../plutonium.js'; -import type { MaintenanceJob, MaintenanceSchedule, PlutoniumRelease, PlutoniumStatus, PlutoniumVersion } from '../../plutonium.js'; - -const HOUR = 60 * 60 * 1000; -const nowIso = () => new Date().toISOString(); -function readSetting(key: string, fallback: T): T { - const row = db().prepare('SELECT value FROM settings WHERE key = ?').get(key) as { value: string } | undefined; - return row ? JSON.parse(row.value) : fallback; -} -function writeSetting(key: string, value: unknown): void { - db().prepare('INSERT INTO settings (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value') - .run(key, JSON.stringify(value)); -} -function release(): PlutoniumRelease { - return readSetting('plutonium_release', { revision: null, checkedAt: null, error: null }); -} -function schedule(): MaintenanceSchedule { - // Legacy schedules were explicitly UTC. Preserve their timing until saved. - return { timeZone: 'UTC', ...readSetting('plutonium_schedule', { enabled: false, time: '06:00', nextAt: null }) }; -} -export function saveMaintenanceSchedule(enabled: boolean, time: string, timeZone: string): void { - const nextAt = nextMaintenanceAt(time, new Date(), timeZone); - writeSetting('plutonium_schedule', { enabled, time, timeZone, nextAt: enabled ? nextAt : null }); -} -export function plutoniumStatus(): PlutoniumStatus { - const versions = db().prepare('SELECT * FROM plutonium_versions').all() as PlutoniumVersion[]; - return { - release: release(), schedule: schedule(), - servers: listServers().map(s => versions.find(v => v.server === s.name) - ?? { server: s.name, revision: null, checked_at: null, error: null }), - jobs: db().prepare(`SELECT * FROM maintenance_jobs WHERE state IN ('queued','restarting') - OR id IN (SELECT id FROM maintenance_jobs ORDER BY id DESC LIMIT 100) ORDER BY id DESC`).all() as MaintenanceJob[] - }; -} - -async function probeVersion(server: string): Promise { - const recovering = restartRecoveryPending(server); - const result = recovering ? { revision: null, error: 'Waiting for restart recovery.' } - : await rconPlutoniumVersion(server).catch(() => ({ revision: null, error: 'Version check failed.' })); - // Clear the old revision on failure: it is no longer evidence of what is running. - db().prepare(`INSERT INTO plutonium_versions (server, revision, checked_at, error) VALUES (?, ?, ?, ?) - ON CONFLICT(server) DO UPDATE SET revision = excluded.revision, checked_at = excluded.checked_at, error = excluded.error`) - .run(server, result.revision, nowIso(), result.error); - return result.revision; -} - -let checking: Promise | null = null; -export function checkPlutonium(): Promise { - if (checking) return checking; - checking = check().finally(() => { checking = null; }); - return checking; -} -async function check(): Promise { - const previous = release(); - try { - const response = await fetch(PLUTONIUM_MANIFEST_URL, { signal: AbortSignal.timeout(config.httpTimeoutMs), redirect: 'error' }); - if (!response.ok) throw new Error(`Updater manifest request failed (HTTP ${response.status}).`); - const body = await response.text(); - if (body.length > 2_000_000) throw new Error('Updater manifest response was too large.'); - writeSetting('plutonium_release', { revision: parsePlutoniumManifest(JSON.parse(body)), checkedAt: nowIso(), error: null }); - } catch (err) { - writeSetting('plutonium_release', { ...previous, checkedAt: nowIso(), error: (err as Error).message }); - } - const servers = listServers(); - for (let i = 0; i < servers.length; i += 4) { - await Promise.all(servers.slice(i, i + 4).map(s => probeVersion(s.name))); - } -} - -export function queueMaintenance(servers: string[]): number { - const names = new Set(listServers().map(s => s.name)); - if (!servers.length || servers.some(s => !names.has(s))) throw new Error('Choose existing fleet servers.'); - return db().transaction(() => { - const latest = release(); - // An old cached release is still displayed, but never used as a fresh target. - const target = !latest.error && latest.checkedAt && Date.now() - Date.parse(latest.checkedAt) < HOUR - ? latest.revision : null; - let added = 0; - for (const server of new Set(servers)) { - added += db().prepare(`INSERT OR IGNORE INTO maintenance_jobs (server, target_revision, requested_at) - VALUES (?, ?, ?)`).run(server, target, nowIso()).changes; - const job = db().prepare("SELECT id FROM maintenance_jobs WHERE server = ? AND state IN ('queued','restarting')").get(server) as { id: number }; - requestRestart(server, { kind: 'maintenance', id: String(job.id) }); - } - return added; - })(); -} -export function cancelMaintenance(id: number): boolean { - return db().transaction(() => { - const job = db().prepare("SELECT provisioning_id FROM maintenance_jobs WHERE id = ? AND state = 'queued'").get(id) as { provisioning_id: string | null } | undefined; - if (!job) return false; - if (job.provisioning_id && !cancelProvisioning(job.provisioning_id)) return false; - return db().prepare(`UPDATE maintenance_jobs SET state = 'cancelled', finished_at = ?, detail = 'Cancelled by operator' - WHERE id = ? AND state = 'queued'`).run(nowIso(), id).changes > 0; - })(); -} - -function finish(job: MaintenanceJob, state: 'done' | 'failed' | 'cancelled', detail: string) { - db().prepare('UPDATE maintenance_jobs SET state = ?, detail = ?, finished_at = ? WHERE id = ?') - .run(state, detail, nowIso(), job.id); -} -function hold(job: MaintenanceJob, detail: string) { - db().prepare('UPDATE maintenance_jobs SET detail = ? WHERE id = ?').run(detail, job.id); -} - -let ticking: Promise | null = null; -export function maintenanceTick(): Promise { - if (ticking) return ticking; - ticking = runTick().finally(() => { ticking = null; }); - return ticking; -} -async function runTick(): Promise { - const latest = release(); - if (!latest.checkedAt || Date.now() - Date.parse(latest.checkedAt) >= HOUR) await checkPlutonium(); - db().transaction(() => { - const plan = schedule(); - if (plan.enabled && plan.nextAt && plan.nextAt <= nowIso()) { - const names = listServers().map(s => s.name); - if (names.length) queueMaintenance(names); - // Coalesce missed days after downtime into one run; no restart backlog. - writeSetting('plutonium_schedule', { ...plan, nextAt: nextMaintenanceAt(plan.time, new Date(), plan.timeZone) }); - } - })(); - const jobs = db().prepare(`SELECT * FROM maintenance_jobs WHERE state IN ('queued','restarting') ORDER BY id`).all() as MaintenanceJob[]; - for (const job of jobs) { - try { await processJob(job); } - catch { hold(job, 'Maintenance check failed; waiting for the next check.'); } - } - db().prepare(`DELETE FROM maintenance_jobs WHERE finished_at < ? AND state IN ('done','failed','cancelled')`) - .run(new Date(Date.now() - 30 * 24 * HOUR).toISOString()); -} -async function processJob(job: MaintenanceJob): Promise { - const server = getServer(job.server); - if (!server || server.destroyed_at) return finish(job, 'cancelled', 'Server removed.'); - const operation = requestRestart(job.server, { kind: 'maintenance', id: String(job.id) }); - if (['failed','interrupted','cancelled'].includes(operation.state)) { - return finish(job, operation.state === 'cancelled' ? 'cancelled' : 'failed', operation.error ?? operation.stage); - } - if (operation.state !== 'done') { - if (operation.submitted_at) { - db().prepare("UPDATE maintenance_jobs SET state = 'restarting', restarted_at = ? WHERE id = ?").run(operation.submitted_at, job.id); - db().prepare('UPDATE plutonium_versions SET revision = NULL, error = ? WHERE server = ?').run('Waiting for restart recovery.', job.server); - } - return hold(job, operation.stage); - } - const revision = await probeVersion(job.server); - if (job.target_revision && (revision === null || revision < job.target_revision)) { - return finish(job, 'failed', `Server recovered, but r${job.target_revision} was not verified (${revision === null ? 'version unknown' : `running r${revision}`}). Check Plutainer auto-update settings.`); - } - return finish(job, 'done', `Server recovered; ${revision === null ? 'running revision unknown' : `running r${revision}`}.`); -} +// Compatibility for existing jobs and API clients. +export * from "../maintenance/runtime.js"; diff --git a/src/lib/server/integrations/plutonium.ts b/src/lib/server/integrations/plutonium.ts new file mode 100644 index 0000000..a97988d --- /dev/null +++ b/src/lib/server/integrations/plutonium.ts @@ -0,0 +1,76 @@ +import { db } from '../db/index.js'; +import { config } from '../config.js'; +import { getServer, listServers } from '../fleet/servers.js'; +import { latestGameSubmission } from '../fleet/status-evidence.js'; +import { rconPlutoniumVersion } from './rcon.js'; +import { komodoInspectHealthy } from './komodo.js'; +import { restartRecoveryPending } from '../provisioning/index.js'; +import { parsePlutoniumManifest, PLUTONIUM_MANIFEST_URL } from '../../plutonium.js'; +import type { PlutoniumRelease, PlutoniumVersion } from '../../plutonium.js'; +import { readSetting, writeSetting } from '../maintenance/settings.js'; +export const HOUR = 60 * 60 * 1000; +const nowIso = () => new Date().toISOString(); +export function release(): PlutoniumRelease { + return readSetting('plutonium_release', { revision: null, checkedAt: null, error: null }); +} +export function freshRelease(): number | null { + const latest = release(); + const age = latest.checkedAt ? Date.now() - Date.parse(latest.checkedAt) : NaN; + return !latest.error && age >= 0 && age < HOUR ? latest.revision : null; +} +export function runningVersions(): PlutoniumVersion[] { + const versions = db().prepare('SELECT * FROM plutonium_versions').all() as PlutoniumVersion[]; + const target = freshRelease(); + return listServers().map(server => { + const value = versions.find(v => v.server === server.name) + ?? { server: server.name, revision: null, checked_at: null, error: null }; + const age = value.checked_at ? Date.now() - Date.parse(value.checked_at) : NaN; + const epoch = latestGameSubmission(server.name); + const stopped = server.desired_state === 'stopped'; + const valid = !value.error && age >= 0 && age < HOUR && (!epoch || value.checked_at! > epoch) && !restartRecoveryPending(server.name); + const revision = valid && !stopped ? value.revision : null; + return { ...value, revision, stopped, + update_status: stopped ? 'stopped' : revision === null || target === null ? 'unknown' + : revision < target ? 'outdated' : revision > target ? 'ahead' : 'current' }; + }); +} +export async function probeVersion(server: string): Promise { + const epoch = latestGameSubmission(server); + const recovering = restartRecoveryPending(server); + const health = !recovering && getServer(server)?.desired_state !== 'stopped' + ? await komodoInspectHealthy(server).catch(() => ({ ok: false })) : { ok: false }; + let result = !health.ok ? { revision: null, error: 'Waiting for a healthy running server.' } + : await rconPlutoniumVersion(server).catch(() => ({ revision: null, error: 'Version check failed.' })); + if (!getServer(server)) return null; + if (epoch !== latestGameSubmission(server) || restartRecoveryPending(server)) { + result = { revision: null, error: 'Waiting for restart recovery.' }; + } + // Clear the old revision on failure: it is no longer evidence of what is running. + db().prepare(`INSERT INTO plutonium_versions (server, revision, checked_at, error) VALUES (?, ?, ?, ?) + ON CONFLICT(server) DO UPDATE SET revision = excluded.revision, checked_at = excluded.checked_at, error = excluded.error`) + .run(server, result.revision, nowIso(), result.error); + return result.revision; +} + +let checking: Promise | null = null; +export function checkPlutonium(): Promise { + if (checking) return checking; + checking = check().finally(() => { checking = null; }); + return checking; +} +async function check(): Promise { + const previous = release(); + try { + const response = await fetch(PLUTONIUM_MANIFEST_URL, { signal: AbortSignal.timeout(config.httpTimeoutMs), redirect: 'error' }); + if (!response.ok) throw new Error(`Updater manifest request failed (HTTP ${response.status}).`); + const body = await response.text(); + if (body.length > 2_000_000) throw new Error('Updater manifest response was too large.'); + writeSetting('plutonium_release', { revision: parsePlutoniumManifest(JSON.parse(body)), checkedAt: nowIso(), error: null }); + } catch (err) { + writeSetting('plutonium_release', { ...previous, checkedAt: nowIso(), error: (err as Error).message }); + } + const servers = listServers(); + for (let i = 0; i < servers.length; i += 4) { + await Promise.all(servers.slice(i, i + 4).map(s => probeVersion(s.name))); + } +} diff --git a/src/lib/server/maintenance/index.ts b/src/lib/server/maintenance/index.ts new file mode 100644 index 0000000..3b7698c --- /dev/null +++ b/src/lib/server/maintenance/index.ts @@ -0,0 +1,13 @@ +import { maintenancePolicies } from './policies.js'; +import { watchdogOverview, watchdogTick } from './recovery.js'; +import { maintenanceTick, plutoniumStatus } from './runtime.js'; +import type { MaintenanceOverview } from '../../maintenance.js'; + +export function maintenanceOverview(): MaintenanceOverview { + return { policies: maintenancePolicies(), recovery: watchdogOverview(), runtime: plutoniumStatus() }; +} +export async function runMaintenance(): Promise { + // One failing observer must not prevent the other capabilities from reconciling. + const results = await Promise.allSettled([maintenanceTick(), watchdogTick()]); + for (const result of results) if (result.status === 'rejected') console.error('[maintenance]', result.reason); +} diff --git a/src/lib/server/maintenance/policies.ts b/src/lib/server/maintenance/policies.ts new file mode 100644 index 0000000..c059df0 --- /dev/null +++ b/src/lib/server/maintenance/policies.ts @@ -0,0 +1,25 @@ +import { db } from '../db/index.js'; +import type { MaintenancePolicy, MaintenancePolicies } from '../../maintenance.js'; + +function setting(key: string): string | undefined { + return (db().prepare('SELECT value FROM settings WHERE key = ?').get(key) as { value: string } | undefined)?.value; +} +function save(key: string, value: boolean) { + db().prepare('INSERT INTO settings(key,value) VALUES (?,?) ON CONFLICT(key) DO UPDATE SET value=excluded.value').run(key, value ? '1' : '0'); +} +export function maintenancePolicies(): MaintenancePolicies { + // Preserve the old recovery switch without opting anyone into runtime upgrades. + const legacy = setting('watchdog_enabled') !== '0'; + return { + paused: setting('maintenance_paused') === '1', + health: setting('maintenance_health') === undefined ? legacy : setting('maintenance_health') === '1', + drift: setting('maintenance_drift') === undefined ? legacy : setting('maintenance_drift') === '1', + updates: setting('maintenance_updates') === '1' + }; +} +export function maintenanceAllowed(policy?: MaintenancePolicy): boolean { + const policies = maintenancePolicies(); + return !policies.paused && (!policy || policies[policy]); +} +export function setMaintenancePolicy(policy: MaintenancePolicy, enabled: boolean) { save(`maintenance_${policy}`, enabled); } +export function pauseMaintenance(paused: boolean) { save('maintenance_paused', paused); } diff --git a/src/lib/server/maintenance/recovery.ts b/src/lib/server/maintenance/recovery.ts new file mode 100644 index 0000000..5ca503c --- /dev/null +++ b/src/lib/server/maintenance/recovery.ts @@ -0,0 +1,242 @@ +import { randomUUID } from 'node:crypto'; +import { db } from '../db/index.js'; +import { komodoConfigured } from '../config.js'; +import { getServer, listServers } from '../fleet/servers.js'; +import { komodoInspectHealthy } from '../integrations/komodo.js'; +import { rconStatusResult } from '../integrations/rcon.js'; +import { telemetryEvidence } from '../fleet/status-evidence.js'; +import { prepareDriftCampaign, createDriftRebuildCampaign, previewDrift } from '../fleet/drift.js'; +import { requestWatchdogLifecycle, escalateWatchdogRestart } from '../provisioning/requests.js'; +import type { WatchdogAction, WatchdogDetail, WatchdogEvidence, WatchdogIncident, WatchdogOverview, WatchdogState } from '../../watchdog.js'; + +import { maintenanceAllowed, setMaintenancePolicy } from './policies.js'; + +export const WATCHDOG_SCAN_MS = 60_000; +export const WATCHDOG_FAILURE_MS = 120_000; +export const WATCHDOG_RECOVERY_MS = 300_000; +const iso = () => new Date().toISOString(); +const pendingStates = ['queued', 'waiting', 'running', 'awaiting-health']; +const setting = (key: string) => (db().prepare('SELECT value FROM settings WHERE key = ?').get(key) as { value: string } | undefined)?.value ?? null; +function setSetting(key: string, value: string) { db().prepare('INSERT INTO settings(key,value) VALUES (?,?) ON CONFLICT(key) DO UPDATE SET value=excluded.value').run(key, value); } +export const watchdogEnabled = () => maintenanceAllowed('health') || maintenanceAllowed('drift'); +export function setWatchdogEnabled(enabled: boolean) { + setSetting('watchdog_enabled', enabled ? '1' : '0'); + setMaintenancePolicy('health', enabled); + setMaintenancePolicy('drift', enabled); +} + +function incident(id: string) { return db().prepare('SELECT * FROM watchdog_incidents WHERE id = ?').get(id) as WatchdogIncident | undefined; } +function active(server: string, kind: 'health' | 'drift') { return db().prepare("SELECT * FROM watchdog_incidents WHERE server = ? AND kind = ? AND state != 'resolved'").get(server, kind) as WatchdogIncident | undefined; } +function event(id: string, message: string) { db().prepare('INSERT INTO watchdog_events(incident_id,created_at,message) VALUES (?,?,?)').run(id, iso(), message); } +function transition(id: string, state: WatchdogState, summary: string) { + const previous = incident(id); + if (!previous || (previous.state === state && previous.summary === summary)) return; + db().prepare('UPDATE watchdog_incidents SET state=?, summary=?, finished_at=? WHERE id=?') + .run(state, summary, state === 'resolved' || state === 'dismissed' ? iso() : null, id); + event(id, summary); +} +function detect(server: string, kind: 'health' | 'drift', summary: string, details: WatchdogEvidence): WatchdogIncident { + const previous = active(server, kind); + if (previous) { + // Repeated UI scans must not manufacture a sustained failure. + const newSample = Date.now() - Date.parse(previous.last_seen_at) >= WATCHDOG_SCAN_MS; + const actionable = (details as WatchdogEvidence & { actionable?: boolean }).actionable; + const priorActionable = (JSON.parse(previous.details) as { actionable?: boolean }).actionable; + const reset = kind === 'health' && actionable === true && priorActionable !== true; + db().prepare('UPDATE watchdog_incidents SET details=?, last_seen_at=?, failures=?, first_seen_at=? WHERE id=?') + .run(JSON.stringify(details), newSample || reset ? iso() : previous.last_seen_at, + reset ? 1 : previous.failures + (newSample ? 1 : 0), reset ? iso() : previous.first_seen_at, previous.id); + if (previous.state === 'open') transition(previous.id, 'open', summary); + return incident(previous.id)!; + } + const id = randomUUID(); + db().prepare(`INSERT INTO watchdog_incidents(id,server,kind,state,summary,details,first_seen_at,last_seen_at) + VALUES (?,?,?,'open',?,?,?,?)`).run(id, server, kind, summary, JSON.stringify(details), iso(), iso()); + event(id, summary); + return incident(id)!; +} +function actions(id: string): WatchdogAction[] { + return db().prepare(`SELECT a.*, d.state AS operation_state, d.stage AS operation_stage, d.error AS operation_error + FROM watchdog_actions a LEFT JOIN server_deployments d ON d.id=a.operation_id + WHERE a.incident_id=? ORDER BY a.rowid`).all(id) as WatchdogAction[]; +} +export function watchdogOverview(): WatchdogOverview { + return { enabled: watchdogEnabled(), lastScanAt: setting('watchdog_last_scan'), scanError: setting('watchdog_scan_error') || null, + incidents: db().prepare(`SELECT i.*, (SELECT d.stage FROM watchdog_actions a + JOIN server_deployments d ON d.id = a.operation_id WHERE a.incident_id = i.id AND a.cycle = i.cycle + ORDER BY a.rowid DESC LIMIT 1) AS operation_stage + FROM watchdog_incidents i WHERE state != 'resolved' OR id IN ( + SELECT id FROM watchdog_incidents WHERE state = 'resolved' ORDER BY first_seen_at DESC, rowid DESC LIMIT 100) + ORDER BY CASE WHEN state='resolved' THEN 1 ELSE 0 END, first_seen_at DESC, rowid DESC`).all() as WatchdogIncident[] }; +} +export function watchdogDetail(id: string): WatchdogDetail | null { + const row = incident(id); + return row ? { incident: row, actions: actions(id), events: db().prepare('SELECT id,created_at,message FROM watchdog_events WHERE incident_id=? ORDER BY id').all(id) as WatchdogDetail['events'] } : null; +} +function hasPending(server: string) { + return !!db().prepare("SELECT 1 FROM server_deployments WHERE server=? AND state IN ('queued','waiting','running','awaiting-health') LIMIT 1").get(server); +} +function protectedWork(server: string): boolean { + return !!db().prepare(`SELECT 1 FROM beta_servers bs JOIN betas b ON b.id=bs.beta_id WHERE bs.server=? AND b.state IN ('running','ready-promote') LIMIT 1`).get(server) + || !!db().prepare(`SELECT 1 FROM deployment_servers ds WHERE ds.server=? AND ds.restart_state IN ('pending','held','failed','manual') + AND NOT EXISTS (SELECT 1 FROM watchdog_actions a WHERE a.campaign_id=ds.deployment_id) LIMIT 1`).get(server); +} +function addAction(row: WatchdogIncident, operationId: string, action: 'restart' | 'rebuild', campaignId: string | null = null) { + db().prepare('INSERT INTO watchdog_actions(id,incident_id,action,operation_id,campaign_id,created_at,cycle) VALUES (?,?,?,?,?,?,?)') + .run(randomUUID(), row.id, action, operationId, campaignId, iso(), row.cycle); + transition(row.id, 'recovering', action === 'restart' ? 'Restart queued; waiting for safe recovery' : 'Rebuild queued with the latest promoted setup'); + event(row.id, `${action === 'restart' ? 'Restart' : 'Rebuild'} requested: ${operationId}${campaignId ? `; campaign ${campaignId}` : ''}`); +} +export function updateWatchdogIncident(id: string, action: 'retry' | 'dismiss') { + return db().transaction(() => { + const row = incident(id); + if (!row) return { ok: false as const, status: 404, error: 'Watchdog incident not found.' }; + if (actions(id).some(a => pendingStates.includes(a.operation_state ?? ''))) return { ok: false as const, status: 409, error: 'Provisioning is still active. Complete or cancel that operation first.' }; + if (action === 'dismiss') { + if (!['open','attention'].includes(row.state)) return { ok: false as const, status: 409, error: 'Only detected incidents or incidents needing attention can be dismissed.' }; + transition(id, 'dismissed', 'Dismissed by operator; automatic attempts remain suppressed until the condition clears'); + } else { + if (!['attention','dismissed'].includes(row.state)) return { ok: false as const, status: 409, error: 'Only incidents needing attention or dismissed incidents can be retried.' }; + if (!getServer(row.server)) return { ok: false as const, status: 409, error: 'The server is no longer registered; inspect its provisioning history.' }; + db().prepare('UPDATE watchdog_incidents SET cycle=cycle+1, failures=1, first_seen_at=?, last_seen_at=? WHERE id=?').run(iso(), iso(), id); + transition(id, 'open', 'Operator requested a fresh recovery attempt; checking current conditions'); + } + return { ok: true as const }; + })(); +} + +async function reconcileActions(): Promise { + const rows = db().prepare("SELECT * FROM watchdog_incidents WHERE state IN ('recovering','attention')").all() as WatchdogIncident[]; + for (const row of rows) { + const action = actions(row.id).filter(a => a.cycle === row.cycle).at(-1); + if (!action) continue; + const operation = db().prepare('SELECT state,submitted_at,started_at,stage,error FROM server_deployments WHERE id=?').get(action.operation_id) as + { state: string; submitted_at: string | null; started_at: string; stage: string; error: string | null } | undefined; + if (!operation) { transition(row.id, 'attention', 'Provisioning history is missing; inspect the server before retrying'); continue; } + if (['done','warning'].includes(operation.state)) { + if (row.kind === 'drift') { + try { + const plan = await previewDrift([row.server]); + if (incident(row.id)?.cycle !== row.cycle) continue; + if (plan.files.length) { transition(row.id, 'attention', 'Rebuild completed but drift remains; inspect the campaign and server files'); continue; } + } catch (err) { transition(row.id, 'attention', `Cannot verify rebuilt files: ${(err as Error).message}`); continue; } + } + transition(row.id, 'resolved', action.action === 'restart' ? 'Server recovered after restart' : 'Server recovered after rebuild'); + continue; + } + if (row.state === 'attention') continue; // Exhausted attempts stay latched, including across app restarts. + if (['cancelled','interrupted'].includes(operation.state)) { + transition(row.id, 'attention', `Provisioning ${operation.state}; inspect the recorded operation before retrying`); continue; + } + if (action.action === 'rebuild') { + if (operation.state === 'failed') transition(row.id, 'attention', `Rebuild failed: ${operation.error ?? operation.stage}`); + else if (operation.state === 'awaiting-health' && Date.now() - Date.parse(operation.submitted_at ?? operation.started_at) >= WATCHDOG_RECOVERY_MS) + transition(row.id, 'attention', 'Rebuild has not recovered within five minutes; provisioning continues verification'); + continue; + } + if (!['awaiting-health','failed'].includes(operation.state)) continue; + if (Date.now() - Date.parse(operation.submitted_at ?? operation.started_at) < WATCHDOG_RECOVERY_MS) continue; + if (!maintenanceAllowed(row.kind)) continue; + const current = getServer(row.server); + if (!current || current.desired_state === 'stopped' || protectedWork(row.server)) continue; + const result = db().transaction(() => { + const next = escalateWatchdogRestart(row.id, action.operation_id, WATCHDOG_RECOVERY_MS); + if (next.ok) addAction(row, next.deploymentId, 'rebuild'); + return next; + })(); + if (!result.ok && result.status !== 409 && result.status !== 503) transition(row.id, 'attention', `Could not queue rebuild: ${result.error}`); + } +} + +interface Observation { healthy: boolean; actionable: boolean; reason: string; containerState: string | null } +async function observe(server: string): Promise { + const health = await komodoInspectHealthy(server); + const state = health.state?.toLowerCase() ?? null; + if (!health.ok) return { healthy: false, actionable: !!state && ['stopped','exited','created','dead','unhealthy','restarting','starting'].includes(state), reason: health.reason, containerState: state }; + if (telemetryEvidence(server)) return { healthy: true, actionable: false, reason: 'Komodo healthy with fresh telemetry', containerState: state }; + const rcon = await rconStatusResult(server); + if (rcon.ok || telemetryEvidence(server)) return { healthy: true, actionable: false, reason: 'Komodo healthy with fresh game status', containerState: state }; + return { healthy: false, actionable: true, reason: `Game status unavailable: ${rcon.reason}; waiting for player-count verification before recovery`, containerState: state }; +} +async function scanServer(server: string): Promise { + const row = getServer(server); + if (!row || row.desired_state === 'stopped') return; + if (hasPending(server) || protectedWork(server)) return; + const observation = await observe(server); + // Probes can overlap an API request; recheck ownership before queuing anything. + if (hasPending(server) || protectedWork(server) || getServer(server)?.desired_state === 'stopped') return; + if (observation.healthy) { + const current = active(server, 'health'); + if (current) transition(current.id, 'resolved', 'Fresh game status and Komodo health verified'); + } else { + const problem = detect(server, 'health', observation.reason, observation); + const startup = ['starting','restarting'].includes(observation.containerState ?? ''); + const oldEnough = Date.now() - Date.parse(problem.first_seen_at) >= (startup ? WATCHDOG_RECOVERY_MS : WATCHDOG_FAILURE_MS); + if (problem.state === 'open' && observation.actionable && oldEnough && problem.failures >= 3 && maintenanceAllowed('health')) { + const result = db().transaction(() => { + const next = requestWatchdogLifecycle(server, 'restart', problem.id); + if (next.ok) addAction(problem, next.deploymentId, 'restart'); + return next; + })(); + if (!result.ok && result.status !== 409 && result.status !== 503) transition(problem.id, 'attention', result.error); + } + // Unknown container health is an observation failure, not proof that a rebuild is safe. + if (!observation.actionable || hasPending(server)) return; + } + const prepared = await prepareDriftCampaign(server); + if (hasPending(server) || protectedWork(server) || getServer(server)?.desired_state === 'stopped') return; + if (!prepared.plan.files.length) { + const current = active(server, 'drift'); + if (current) transition(current.id, 'resolved', 'Server files match the promoted setup'); + return; + } + const problem = detect(server, 'drift', `${prepared.plan.files.length} file difference(s) from the promoted setup`, { files: prepared.plan.files, fingerprint: prepared.plan.fingerprint }); + if (problem.state !== 'open' || !maintenanceAllowed('drift')) return; + const result = db().transaction(() => { + const next = requestWatchdogLifecycle(server, 'rebuild', problem.id); + if (next.ok) { + const campaignId = createDriftRebuildCampaign(server, next.deploymentId, problem.id, prepared); + addAction(problem, next.deploymentId, 'rebuild', campaignId); + } + return next; + })(); + if (!result.ok && result.status !== 409 && result.status !== 503) transition(problem.id, 'attention', `Could not queue drift rebuild: ${result.error}`); +} + +let inFlight: Promise | null = null; +export function watchdogTick(forceScan = false): Promise { + if (inFlight) return inFlight; + inFlight = run(forceScan).finally(() => { inFlight = null; }); + return inFlight; +} +async function run(forceScan: boolean): Promise { + await reconcileActions(); + const lastScan = setting('watchdog_last_scan'); + if (!forceScan && lastScan && Date.now() - Date.parse(lastScan) < WATCHDOG_SCAN_MS) return; + setSetting('watchdog_last_scan', iso()); + if (!komodoConfigured()) { + setSetting('watchdog_scan_error', 'Configure Komodo before automatic recovery. RCON or fresh telemetry verifies game status.'); return; + } + const errors: string[] = []; + const servers = listServers(); + let cursor = 0; + async function pump() { + while (cursor < servers.length) { + const server = servers[cursor++]; + try { await scanServer(server.name); } + catch (err) { errors.push(`${server.name}: ${(err as Error).message}`); } + } + } + await Promise.all(Array.from({ length: Math.min(4, servers.length) }, pump)); + // Preserve incidents for temporarily absent rebuild instances, but close retired registrations. + const abandoned = db().prepare("SELECT * FROM watchdog_incidents WHERE state != 'resolved'").all() as WatchdogIncident[]; + for (const row of abandoned) { + if (getServer(row.server) || hasPending(row.server)) continue; + const last = db().prepare('SELECT action,state FROM server_deployments WHERE server=? ORDER BY rowid DESC LIMIT 1').get(row.server) as { action: string; state: string } | undefined; + // An absent registry can also mean a failed rebuild. Only a completed + // destruction retires that failure without requiring inspection. + if (['open','dismissed'].includes(row.state) || (last?.action === 'destroy' && ['done','warning'].includes(last.state))) + transition(row.id, 'resolved', 'Server is no longer registered'); + } + setSetting('watchdog_scan_error', errors.join('\n')); +} diff --git a/src/lib/server/maintenance/runtime.ts b/src/lib/server/maintenance/runtime.ts new file mode 100644 index 0000000..1b0e3a4 --- /dev/null +++ b/src/lib/server/maintenance/runtime.ts @@ -0,0 +1,139 @@ +import { db } from '../db/index.js'; +import { getServer, listServers } from '../fleet/servers.js'; +import { requestRestart, cancelProvisioning } from '../provisioning/index.js'; +import { nextMaintenanceAt } from '../../plutonium.js'; +import type { MaintenanceJob, MaintenanceSchedule, PlutoniumStatus } from '../../plutonium.js'; + +import { readSetting, writeSetting } from './settings.js'; +import { checkPlutonium, probeVersion, release, freshRelease, runningVersions, HOUR } from '../integrations/plutonium.js'; +import { maintenanceAllowed } from './policies.js'; +export { checkPlutonium } from '../integrations/plutonium.js'; +const nowIso = () => new Date().toISOString(); +function schedule(): MaintenanceSchedule { + // Legacy schedules were explicitly UTC. Preserve their timing until saved. + return { timeZone: 'UTC', ...readSetting('plutonium_schedule', { enabled: false, time: '06:00', nextAt: null }) }; +} +export function saveMaintenanceSchedule(enabled: boolean, time: string, timeZone: string): void { + const nextAt = nextMaintenanceAt(time, new Date(), timeZone); + writeSetting('plutonium_schedule', { enabled, time, timeZone, nextAt: enabled ? nextAt : null }); +} +export function plutoniumStatus(): PlutoniumStatus { + return { + release: release(), schedule: schedule(), servers: runningVersions(), + jobs: db().prepare(`SELECT m.*, d.stage AS operation_stage, d.state AS operation_state + FROM maintenance_jobs m LEFT JOIN server_deployments d ON d.id = m.provisioning_id + WHERE m.state IN ('queued','restarting') + OR (m.state = 'failed' AND NOT EXISTS (SELECT 1 FROM maintenance_jobs n WHERE n.server = m.server AND n.kind = m.kind AND n.id > m.id)) + OR m.id IN (SELECT id FROM maintenance_jobs ORDER BY id DESC LIMIT 100) ORDER BY m.id DESC`).all() as MaintenanceJob[] + }; +} + +export function queueMaintenance(servers: string[], kind: MaintenanceJob['kind'] = 'legacy', origin: MaintenanceJob['origin'] = 'manual'): number { + const names = new Set(listServers().map(s => s.name)); + if (!servers.length || servers.some(s => !names.has(s))) throw new Error('Choose existing fleet servers.'); + return db().transaction(() => { + const target = kind === 'schedule' ? null : freshRelease(); + if (kind === 'update' && target === null) throw new Error('Check the published revision before requesting an update.'); + let added = 0; + for (const server of new Set(servers)) { + if (getServer(server)?.desired_state === 'stopped') continue; + added += db().prepare(`INSERT OR IGNORE INTO maintenance_jobs (server, target_revision, requested_at, kind, origin) + VALUES (?, ?, ?, ?, ?)`).run(server, target, nowIso(), kind, origin).changes; + const job = db().prepare("SELECT id FROM maintenance_jobs WHERE server = ? AND state IN ('queued','restarting')").get(server) as { id: number }; + requestRestart(server, { kind: 'maintenance', id: String(job.id) }); + } + return added; + })(); +} + +/** One automatic attempt per server/revision. Explicit retries preserve the old outcome. */ +export function queueUpdates(servers?: string[], automatic = false): number { + if (!maintenanceAllowed()) throw new Error('Maintenance is paused. Resume it before requesting updates.'); + if (automatic && !maintenanceAllowed('updates')) return 0; + if (freshRelease() === null) throw new Error('Check the published revision before requesting an update.'); + const versions = runningVersions(); + if (servers && (!servers.length || servers.some(server => !versions.some(v => v.server === server)))) throw new Error('Choose existing fleet servers.'); + const candidates = versions.filter(v => v.update_status === 'outdated' && (!servers || servers.includes(v.server))); + const names = candidates.filter(v => !automatic || !db().prepare(`SELECT 1 FROM maintenance_jobs + WHERE server = ? AND target_revision = ? AND kind = 'update'`).get(v.server, freshRelease())).map(v => v.server); + return names.length ? queueMaintenance(names, 'update', automatic ? 'automatic' : 'manual') : 0; +} + +export function retryMaintenance(id: number): number { + const job = db().prepare("SELECT * FROM maintenance_jobs WHERE id = ? AND state = 'failed'").get(id) as MaintenanceJob | undefined; + if (!job) throw new Error('Only failed maintenance can be retried.'); + if (!maintenanceAllowed()) throw new Error('Maintenance is paused.'); + if (getServer(job.server)?.desired_state === 'stopped') throw new Error('This server is intentionally stopped.'); + // Ambiguous lifecycle failures retain their slot and require diagnostics, not another request. + if (job.provisioning_id) { + const operation = db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(job.provisioning_id) as { state: string } | undefined; + if (operation?.state !== 'done') throw new Error('Inspect and recover the existing Provisioning operation before retrying.'); + } + return queueMaintenance([job.server], job.kind, 'manual'); +} +export function cancelMaintenance(id: number): boolean { + return db().transaction(() => { + const job = db().prepare("SELECT provisioning_id FROM maintenance_jobs WHERE id = ? AND state = 'queued'").get(id) as { provisioning_id: string | null } | undefined; + if (!job) return false; + if (job.provisioning_id && !cancelProvisioning(job.provisioning_id)) return false; + return db().prepare(`UPDATE maintenance_jobs SET state = 'cancelled', finished_at = ?, detail = 'Cancelled by operator' + WHERE id = ? AND state = 'queued'`).run(nowIso(), id).changes > 0; + })(); +} + +function finish(job: MaintenanceJob, state: 'done' | 'failed' | 'cancelled', detail: string) { + db().prepare('UPDATE maintenance_jobs SET state = ?, detail = ?, finished_at = ? WHERE id = ?') + .run(state, detail, nowIso(), job.id); +} +function hold(job: MaintenanceJob, detail: string) { + db().prepare('UPDATE maintenance_jobs SET detail = ? WHERE id = ?').run(detail, job.id); +} + +let ticking: Promise | null = null; +export function maintenanceTick(): Promise { + if (ticking) return ticking; + ticking = runTick().finally(() => { ticking = null; }); + return ticking; +} +async function runTick(): Promise { + const latest = release(); + if (!latest.checkedAt || Date.now() - Date.parse(latest.checkedAt) >= HOUR) await checkPlutonium(); + if (maintenanceAllowed('updates') && freshRelease() !== null) queueUpdates(undefined, true); + db().transaction(() => { + const plan = schedule(); + if (maintenanceAllowed() && plan.enabled && plan.nextAt && plan.nextAt <= nowIso()) { + const names = listServers().filter(s => s.desired_state !== 'stopped').map(s => s.name); + if (names.length) queueMaintenance(names, 'schedule', 'schedule'); + // Coalesce missed days after downtime into one run; no restart backlog. + writeSetting('plutonium_schedule', { ...plan, nextAt: nextMaintenanceAt(plan.time, new Date(), plan.timeZone) }); + } + })(); + const jobs = db().prepare(`SELECT * FROM maintenance_jobs WHERE state IN ('queued','restarting') ORDER BY id`).all() as MaintenanceJob[]; + for (const job of jobs) { + try { await processJob(job); } + catch { hold(job, 'Maintenance check failed; waiting for the next check.'); } + } + // Update attempts are also durable suppression evidence; never age them into an automatic retry. + db().prepare(`DELETE FROM maintenance_jobs WHERE kind != 'update' AND finished_at < ? AND state IN ('done','failed','cancelled')`) + .run(new Date(Date.now() - 30 * 24 * HOUR).toISOString()); +} +async function processJob(job: MaintenanceJob): Promise { + const server = getServer(job.server); + if (!server || server.destroyed_at) return finish(job, 'cancelled', 'Server removed.'); + const operation = requestRestart(job.server, { kind: 'maintenance', id: String(job.id) }); + if (['failed','interrupted','cancelled'].includes(operation.state)) { + return finish(job, operation.state === 'cancelled' ? 'cancelled' : 'failed', operation.error ?? operation.stage); + } + if (operation.state !== 'done') { + if (operation.submitted_at) { + db().prepare("UPDATE maintenance_jobs SET state = 'restarting', restarted_at = ? WHERE id = ?").run(operation.submitted_at, job.id); + db().prepare('UPDATE plutonium_versions SET revision = NULL, error = ? WHERE server = ?').run('Waiting for restart recovery.', job.server); + } + return hold(job, operation.stage); + } + const revision = await probeVersion(job.server); + if (job.target_revision && (revision === null || revision < job.target_revision)) { + return finish(job, 'failed', `Server recovered, but r${job.target_revision} was not verified (${revision === null ? 'version unknown' : `running r${revision}`}). Check Plutainer auto-update settings.`); + } + return finish(job, 'done', `Server recovered; ${revision === null ? 'running revision unknown' : `running r${revision}`}.`); +} diff --git a/src/lib/server/maintenance/settings.ts b/src/lib/server/maintenance/settings.ts new file mode 100644 index 0000000..13a107d --- /dev/null +++ b/src/lib/server/maintenance/settings.ts @@ -0,0 +1,9 @@ +import { db } from '../db/index.js'; +export function readSetting(key: string, fallback: T): T { + const row = db().prepare('SELECT value FROM settings WHERE key = ?').get(key) as { value: string } | undefined; + return row ? JSON.parse(row.value) : fallback; +} +export function writeSetting(key: string, value: unknown): void { + db().prepare('INSERT INTO settings (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value') + .run(key, JSON.stringify(value)); +} diff --git a/src/lib/server/pipeline/worker.ts b/src/lib/server/pipeline/worker.ts index d42b2d5..ca9adfe 100644 --- a/src/lib/server/pipeline/worker.ts +++ b/src/lib/server/pipeline/worker.ts @@ -15,8 +15,7 @@ import { isProvisioning } from '../fleet/provisioning.js'; import { STATUS_REFRESH_MS } from '../../server-freshness.js'; import { recordMetrics } from '../fleet/metrics.js'; import { selectedZombieMap } from '../../map-aliases.js'; -import { maintenanceTick } from '../fleet/plutonium.js'; -import { watchdogTick } from '../watchdog/index.js'; +import { runMaintenance } from '../maintenance/index.js'; const TICK_MS = 15_000; // Cap concurrent UDP RCON calls so a 15-server fleet doesn't blast the @@ -38,8 +37,7 @@ function tick(): Promise { async function reconcileRequesters(): Promise { try { await sweep(); } catch (err) { console.error('[worker] campaign error:', err); } try { await betaTick(); } catch (err) { console.error('[worker] beta error:', err); } - try { await maintenanceTick(); } catch (err) { console.error('[worker] maintenance error:', err); } - try { await watchdogTick(); } catch (err) { console.error('[worker] watchdog error:', err); } + try { await runMaintenance(); } catch (err) { console.error('[worker] maintenance error:', err); } } async function runTick(): Promise { diff --git a/src/lib/server/provisioning/requests.ts b/src/lib/server/provisioning/requests.ts index 5901794..20d9c5f 100644 --- a/src/lib/server/provisioning/requests.ts +++ b/src/lib/server/provisioning/requests.ts @@ -1,3 +1,4 @@ +import { maintenanceAllowed } from '../maintenance/policies.js'; import { db } from '../db/index.js'; import { komodoConfigured, rconConfigured } from '../config.js'; import { getServer } from '../fleet/servers.js'; @@ -141,10 +142,10 @@ export function requestLifecycle(server: string, action: RuntimeAction) { type LifecycleRequestResult = { ok: true; deploymentId: string } | { ok: false; status: number; error: string }; function watchdogRequestGuard(server: string, incidentId: string): Extract | null { - const incident = db().prepare("SELECT 1 FROM watchdog_incidents WHERE id = ? AND server = ? AND state IN ('open','recovering')").get(incidentId, server); + const incident = db().prepare("SELECT kind FROM watchdog_incidents WHERE id = ? AND server = ? AND state IN ('open','recovering')").get(incidentId, server) as { kind: 'health' | 'drift' } | undefined; if (!incident) return { ok: false, status: 409, error: 'The Watchdog incident is no longer active.' }; - if (db().prepare("SELECT 1 FROM settings WHERE key = 'watchdog_enabled' AND value = '0'").get()) { - return { ok: false, status: 409, error: 'Watchdog is paused.' }; + if (!maintenanceAllowed(incident.kind)) { + return { ok: false, status: 409, error: 'This maintenance policy is paused.' }; } const row = getServer(server); if (!row || row.destroyed_at) return { ok: false, status: 404, error: 'Server not found.' }; diff --git a/src/lib/server/provisioning/worker.ts b/src/lib/server/provisioning/worker.ts index 1db5798..a3d7549 100644 --- a/src/lib/server/provisioning/worker.ts +++ b/src/lib/server/provisioning/worker.ts @@ -1,3 +1,4 @@ +import { maintenanceAllowed } from '../maintenance/policies.js'; import { db } from '../db/index.js'; import { komodoConfigured, rconConfigured } from '../config.js'; import { getServer } from '../fleet/servers.js'; @@ -24,7 +25,8 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled' for (const source of sources.filter(source => source.kind === 'watchdog')) { if (!db().prepare("SELECT 1 FROM watchdog_incidents WHERE id = ? AND server = ? AND state IN ('open','recovering')").get(source.source_id, operation.server) || getServer(operation.server)?.desired_state === 'stopped') return 'cancelled'; - if (db().prepare("SELECT 1 FROM settings WHERE key = 'watchdog_enabled' AND value = '0'").get()) return 'held'; + const incident = db().prepare('SELECT kind FROM watchdog_incidents WHERE id = ?').get(source.source_id) as { kind: 'health' | 'drift' }; + if (!maintenanceAllowed(incident.kind)) return 'held'; } let held = false; for (const source of sources) { @@ -43,8 +45,12 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled' if (!row.outcome || ['active','passed'].includes(row.outcome)) return 'ready'; if (row.outcome === 'paused') held = true; } else if (source.kind === 'maintenance') { - const row = db().prepare('SELECT state, provisioning_id FROM maintenance_jobs WHERE id = ?').get(source.source_id) as { state: string; provisioning_id: string } | undefined; - if (row?.provisioning_id === operation.id && ['queued','restarting'].includes(row.state)) return 'ready'; + const row = db().prepare('SELECT state, provisioning_id, kind, origin FROM maintenance_jobs WHERE id = ?').get(source.source_id) as { state: string; provisioning_id: string; kind: string; origin: string } | undefined; + if (row?.provisioning_id === operation.id && ['queued','restarting'].includes(row.state)) { + if (getServer(operation.server)?.desired_state === 'stopped') return 'cancelled'; + if (!maintenanceAllowed() || (row.kind === 'update' && row.origin === 'automatic' && !maintenanceAllowed('updates'))) return 'held'; + return 'ready'; + } } else return 'ready'; } return held ? 'held' : 'cancelled'; diff --git a/src/lib/server/watchdog/index.ts b/src/lib/server/watchdog/index.ts index dbfe91a..7d42b1d 100644 --- a/src/lib/server/watchdog/index.ts +++ b/src/lib/server/watchdog/index.ts @@ -1,233 +1,2 @@ -import { randomUUID } from 'node:crypto'; -import { db } from '../db/index.js'; -import { komodoConfigured } from '../config.js'; -import { getServer, listServers } from '../fleet/servers.js'; -import { komodoInspectHealthy } from '../integrations/komodo.js'; -import { rconStatusResult } from '../integrations/rcon.js'; -import { telemetryEvidence } from '../fleet/status-evidence.js'; -import { prepareDriftCampaign, createDriftRebuildCampaign, previewDrift } from '../fleet/drift.js'; -import { requestWatchdogLifecycle, escalateWatchdogRestart } from '../provisioning/requests.js'; -import type { WatchdogAction, WatchdogDetail, WatchdogEvidence, WatchdogIncident, WatchdogOverview, WatchdogState } from '../../watchdog.js'; - -export const WATCHDOG_SCAN_MS = 60_000; -export const WATCHDOG_FAILURE_MS = 120_000; -export const WATCHDOG_RECOVERY_MS = 300_000; -const iso = () => new Date().toISOString(); -const pendingStates = ['queued', 'waiting', 'running', 'awaiting-health']; -const setting = (key: string) => (db().prepare('SELECT value FROM settings WHERE key = ?').get(key) as { value: string } | undefined)?.value ?? null; -function setSetting(key: string, value: string) { db().prepare('INSERT INTO settings(key,value) VALUES (?,?) ON CONFLICT(key) DO UPDATE SET value=excluded.value').run(key, value); } -export const watchdogEnabled = () => setting('watchdog_enabled') !== '0'; -export function setWatchdogEnabled(enabled: boolean) { setSetting('watchdog_enabled', enabled ? '1' : '0'); } - -function incident(id: string) { return db().prepare('SELECT * FROM watchdog_incidents WHERE id = ?').get(id) as WatchdogIncident | undefined; } -function active(server: string, kind: 'health' | 'drift') { return db().prepare("SELECT * FROM watchdog_incidents WHERE server = ? AND kind = ? AND state != 'resolved'").get(server, kind) as WatchdogIncident | undefined; } -function event(id: string, message: string) { db().prepare('INSERT INTO watchdog_events(incident_id,created_at,message) VALUES (?,?,?)').run(id, iso(), message); } -function transition(id: string, state: WatchdogState, summary: string) { - const previous = incident(id); - if (!previous || (previous.state === state && previous.summary === summary)) return; - db().prepare('UPDATE watchdog_incidents SET state=?, summary=?, finished_at=? WHERE id=?') - .run(state, summary, state === 'resolved' || state === 'dismissed' ? iso() : null, id); - event(id, summary); -} -function detect(server: string, kind: 'health' | 'drift', summary: string, details: WatchdogEvidence): WatchdogIncident { - const previous = active(server, kind); - if (previous) { - // Repeated UI scans must not manufacture a sustained failure. - const newSample = Date.now() - Date.parse(previous.last_seen_at) >= WATCHDOG_SCAN_MS; - const actionable = (details as WatchdogEvidence & { actionable?: boolean }).actionable; - const priorActionable = (JSON.parse(previous.details) as { actionable?: boolean }).actionable; - const reset = kind === 'health' && actionable === true && priorActionable !== true; - db().prepare('UPDATE watchdog_incidents SET details=?, last_seen_at=?, failures=?, first_seen_at=? WHERE id=?') - .run(JSON.stringify(details), newSample || reset ? iso() : previous.last_seen_at, - reset ? 1 : previous.failures + (newSample ? 1 : 0), reset ? iso() : previous.first_seen_at, previous.id); - if (previous.state === 'open') transition(previous.id, 'open', summary); - return incident(previous.id)!; - } - const id = randomUUID(); - db().prepare(`INSERT INTO watchdog_incidents(id,server,kind,state,summary,details,first_seen_at,last_seen_at) - VALUES (?,?,?,'open',?,?,?,?)`).run(id, server, kind, summary, JSON.stringify(details), iso(), iso()); - event(id, summary); - return incident(id)!; -} -function actions(id: string): WatchdogAction[] { - return db().prepare(`SELECT a.*, d.state AS operation_state, d.stage AS operation_stage, d.error AS operation_error - FROM watchdog_actions a LEFT JOIN server_deployments d ON d.id=a.operation_id - WHERE a.incident_id=? ORDER BY a.rowid`).all(id) as WatchdogAction[]; -} -export function watchdogOverview(): WatchdogOverview { - return { enabled: watchdogEnabled(), lastScanAt: setting('watchdog_last_scan'), scanError: setting('watchdog_scan_error') || null, - incidents: db().prepare(`SELECT * FROM watchdog_incidents WHERE state != 'resolved' OR id IN ( - SELECT id FROM watchdog_incidents WHERE state = 'resolved' ORDER BY first_seen_at DESC, rowid DESC LIMIT 100) - ORDER BY CASE WHEN state='resolved' THEN 1 ELSE 0 END, first_seen_at DESC, rowid DESC`).all() as WatchdogIncident[] }; -} -export function watchdogDetail(id: string): WatchdogDetail | null { - const row = incident(id); - return row ? { incident: row, actions: actions(id), events: db().prepare('SELECT id,created_at,message FROM watchdog_events WHERE incident_id=? ORDER BY id').all(id) as WatchdogDetail['events'] } : null; -} -function hasPending(server: string) { - return !!db().prepare("SELECT 1 FROM server_deployments WHERE server=? AND state IN ('queued','waiting','running','awaiting-health') LIMIT 1").get(server); -} -function protectedWork(server: string): boolean { - return !!db().prepare(`SELECT 1 FROM beta_servers bs JOIN betas b ON b.id=bs.beta_id WHERE bs.server=? AND b.state IN ('running','ready-promote') LIMIT 1`).get(server) - || !!db().prepare(`SELECT 1 FROM deployment_servers ds WHERE ds.server=? AND ds.restart_state IN ('pending','held','failed','manual') - AND NOT EXISTS (SELECT 1 FROM watchdog_actions a WHERE a.campaign_id=ds.deployment_id) LIMIT 1`).get(server); -} -function addAction(row: WatchdogIncident, operationId: string, action: 'restart' | 'rebuild', campaignId: string | null = null) { - db().prepare('INSERT INTO watchdog_actions(id,incident_id,action,operation_id,campaign_id,created_at,cycle) VALUES (?,?,?,?,?,?,?)') - .run(randomUUID(), row.id, action, operationId, campaignId, iso(), row.cycle); - transition(row.id, 'recovering', action === 'restart' ? 'Restart queued; waiting for safe recovery' : 'Rebuild queued with the latest promoted setup'); - event(row.id, `${action === 'restart' ? 'Restart' : 'Rebuild'} requested: ${operationId}${campaignId ? `; campaign ${campaignId}` : ''}`); -} -export function updateWatchdogIncident(id: string, action: 'retry' | 'dismiss') { - return db().transaction(() => { - const row = incident(id); - if (!row) return { ok: false as const, status: 404, error: 'Watchdog incident not found.' }; - if (actions(id).some(a => pendingStates.includes(a.operation_state ?? ''))) return { ok: false as const, status: 409, error: 'Provisioning is still active. Complete or cancel that operation first.' }; - if (action === 'dismiss') { - if (!['open','attention'].includes(row.state)) return { ok: false as const, status: 409, error: 'Only detected incidents or incidents needing attention can be dismissed.' }; - transition(id, 'dismissed', 'Dismissed by operator; automatic attempts remain suppressed until the condition clears'); - } else { - if (!['attention','dismissed'].includes(row.state)) return { ok: false as const, status: 409, error: 'Only incidents needing attention or dismissed incidents can be retried.' }; - if (!getServer(row.server)) return { ok: false as const, status: 409, error: 'The server is no longer registered; inspect its provisioning history.' }; - db().prepare('UPDATE watchdog_incidents SET cycle=cycle+1, failures=1, first_seen_at=?, last_seen_at=? WHERE id=?').run(iso(), iso(), id); - transition(id, 'open', 'Operator requested a fresh recovery attempt; checking current conditions'); - } - return { ok: true as const }; - })(); -} - -async function reconcileActions(): Promise { - const rows = db().prepare("SELECT * FROM watchdog_incidents WHERE state IN ('recovering','attention')").all() as WatchdogIncident[]; - for (const row of rows) { - const action = actions(row.id).filter(a => a.cycle === row.cycle).at(-1); - if (!action) continue; - const operation = db().prepare('SELECT state,submitted_at,started_at,stage,error FROM server_deployments WHERE id=?').get(action.operation_id) as - { state: string; submitted_at: string | null; started_at: string; stage: string; error: string | null } | undefined; - if (!operation) { transition(row.id, 'attention', 'Provisioning history is missing; inspect the server before retrying'); continue; } - if (['done','warning'].includes(operation.state)) { - if (row.kind === 'drift') { - try { - const plan = await previewDrift([row.server]); - if (incident(row.id)?.cycle !== row.cycle) continue; - if (plan.files.length) { transition(row.id, 'attention', 'Rebuild completed but drift remains; inspect the campaign and server files'); continue; } - } catch (err) { transition(row.id, 'attention', `Cannot verify rebuilt files: ${(err as Error).message}`); continue; } - } - transition(row.id, 'resolved', action.action === 'restart' ? 'Server recovered after restart' : 'Server recovered after rebuild'); - continue; - } - if (row.state === 'attention') continue; // Exhausted attempts stay latched, including across app restarts. - if (['cancelled','interrupted'].includes(operation.state)) { - transition(row.id, 'attention', `Provisioning ${operation.state}; inspect the recorded operation before retrying`); continue; - } - if (action.action === 'rebuild') { - if (operation.state === 'failed') transition(row.id, 'attention', `Rebuild failed: ${operation.error ?? operation.stage}`); - else if (operation.state === 'awaiting-health' && Date.now() - Date.parse(operation.submitted_at ?? operation.started_at) >= WATCHDOG_RECOVERY_MS) - transition(row.id, 'attention', 'Rebuild has not recovered within five minutes; provisioning continues verification'); - continue; - } - if (!['awaiting-health','failed'].includes(operation.state)) continue; - if (Date.now() - Date.parse(operation.submitted_at ?? operation.started_at) < WATCHDOG_RECOVERY_MS) continue; - if (!watchdogEnabled()) continue; - const current = getServer(row.server); - if (!current || current.desired_state === 'stopped' || protectedWork(row.server)) continue; - const result = db().transaction(() => { - const next = escalateWatchdogRestart(row.id, action.operation_id, WATCHDOG_RECOVERY_MS); - if (next.ok) addAction(row, next.deploymentId, 'rebuild'); - return next; - })(); - if (!result.ok && result.status !== 409 && result.status !== 503) transition(row.id, 'attention', `Could not queue rebuild: ${result.error}`); - } -} - -interface Observation { healthy: boolean; actionable: boolean; reason: string; containerState: string | null } -async function observe(server: string): Promise { - const health = await komodoInspectHealthy(server); - const state = health.state?.toLowerCase() ?? null; - if (!health.ok) return { healthy: false, actionable: !!state && ['stopped','exited','created','dead','unhealthy','restarting','starting'].includes(state), reason: health.reason, containerState: state }; - if (telemetryEvidence(server)) return { healthy: true, actionable: false, reason: 'Komodo healthy with fresh telemetry', containerState: state }; - const rcon = await rconStatusResult(server); - if (rcon.ok || telemetryEvidence(server)) return { healthy: true, actionable: false, reason: 'Komodo healthy with fresh game status', containerState: state }; - return { healthy: false, actionable: true, reason: `Game status unavailable: ${rcon.reason}; waiting for player-count verification before recovery`, containerState: state }; -} -async function scanServer(server: string): Promise { - const row = getServer(server); - if (!row || row.desired_state === 'stopped') return; - if (hasPending(server) || protectedWork(server)) return; - const observation = await observe(server); - // Probes can overlap an API request; recheck ownership before queuing anything. - if (hasPending(server) || protectedWork(server) || getServer(server)?.desired_state === 'stopped') return; - if (observation.healthy) { - const current = active(server, 'health'); - if (current) transition(current.id, 'resolved', 'Fresh game status and Komodo health verified'); - } else { - const problem = detect(server, 'health', observation.reason, observation); - const startup = ['starting','restarting'].includes(observation.containerState ?? ''); - const oldEnough = Date.now() - Date.parse(problem.first_seen_at) >= (startup ? WATCHDOG_RECOVERY_MS : WATCHDOG_FAILURE_MS); - if (problem.state === 'open' && observation.actionable && oldEnough && problem.failures >= 3 && watchdogEnabled()) { - const result = db().transaction(() => { - const next = requestWatchdogLifecycle(server, 'restart', problem.id); - if (next.ok) addAction(problem, next.deploymentId, 'restart'); - return next; - })(); - if (!result.ok && result.status !== 409 && result.status !== 503) transition(problem.id, 'attention', result.error); - } - // Unknown container health is an observation failure, not proof that a rebuild is safe. - if (!observation.actionable || hasPending(server)) return; - } - const prepared = await prepareDriftCampaign(server); - if (hasPending(server) || protectedWork(server) || getServer(server)?.desired_state === 'stopped') return; - if (!prepared.plan.files.length) { - const current = active(server, 'drift'); - if (current) transition(current.id, 'resolved', 'Server files match the promoted setup'); - return; - } - const problem = detect(server, 'drift', `${prepared.plan.files.length} file difference(s) from the promoted setup`, { files: prepared.plan.files, fingerprint: prepared.plan.fingerprint }); - if (problem.state !== 'open' || !watchdogEnabled()) return; - const result = db().transaction(() => { - const next = requestWatchdogLifecycle(server, 'rebuild', problem.id); - if (next.ok) { - const campaignId = createDriftRebuildCampaign(server, next.deploymentId, problem.id, prepared); - addAction(problem, next.deploymentId, 'rebuild', campaignId); - } - return next; - })(); - if (!result.ok && result.status !== 409 && result.status !== 503) transition(problem.id, 'attention', `Could not queue drift rebuild: ${result.error}`); -} - -let inFlight: Promise | null = null; -export function watchdogTick(forceScan = false): Promise { - if (inFlight) return inFlight; - inFlight = run(forceScan).finally(() => { inFlight = null; }); - return inFlight; -} -async function run(forceScan: boolean): Promise { - await reconcileActions(); - const lastScan = setting('watchdog_last_scan'); - if (!forceScan && lastScan && Date.now() - Date.parse(lastScan) < WATCHDOG_SCAN_MS) return; - setSetting('watchdog_last_scan', iso()); - if (!komodoConfigured()) { - setSetting('watchdog_scan_error', 'Configure Komodo before automatic recovery. RCON or fresh telemetry verifies game status.'); return; - } - const errors: string[] = []; - const servers = listServers(); - let cursor = 0; - async function pump() { - while (cursor < servers.length) { - const server = servers[cursor++]; - try { await scanServer(server.name); } - catch (err) { errors.push(`${server.name}: ${(err as Error).message}`); } - } - } - await Promise.all(Array.from({ length: Math.min(4, servers.length) }, pump)); - // Preserve incidents for temporarily absent rebuild instances, but close retired registrations. - const abandoned = db().prepare("SELECT * FROM watchdog_incidents WHERE state != 'resolved'").all() as WatchdogIncident[]; - for (const row of abandoned) { - if (getServer(row.server) || hasPending(row.server)) continue; - const last = db().prepare('SELECT action,state FROM server_deployments WHERE server=? ORDER BY rowid DESC LIMIT 1').get(row.server) as { action: string; state: string } | undefined; - // An absent registry can also mean a failed rebuild. Only a completed - // destruction retires that failure without requiring inspection. - if (['open','dismissed'].includes(row.state) || (last?.action === 'destroy' && ['done','warning'].includes(last.state))) - transition(row.id, 'resolved', 'Server is no longer registered'); - } - setSetting('watchdog_scan_error', errors.join('\n')); -} +// Compatibility for existing callers; Maintenance owns recovery. +export * from "../maintenance/recovery.js"; diff --git a/src/lib/ui/components/ArtifactDetail.svelte b/src/lib/ui/components/ArtifactDetail.svelte index 6ff6327..b5445f3 100644 --- a/src/lib/ui/components/ArtifactDetail.svelte +++ b/src/lib/ui/components/ArtifactDetail.svelte @@ -237,7 +237,7 @@ flash = body.error ?? `HTTP ${res.status}`; return; } - scopeComplete = { count: scopeImpact.affected.length, composed: scopeImpact.composed, detached: body.detached?.length ?? 0, href: scopeImpact.affected.length === 1 ? '/watchdog?kind=drift&server=' + encodeURIComponent(scopeImpact.affected[0]) : '/watchdog?kind=drift' }; + scopeComplete = { count: scopeImpact.affected.length, composed: scopeImpact.composed, detached: body.detached?.length ?? 0, href: scopeImpact.affected.length === 1 ? '/maintenance?kind=drift&server=' + encodeURIComponent(scopeImpact.affected[0]) : '/maintenance?kind=drift' }; scopeReviewOpen = false; scopeDialog?.close(); await load(); } catch (error) { flash = (error as Error).message; } finally { @@ -286,7 +286,7 @@ {/snippet} - {#if scopeComplete}
Scope saved. {scopeComplete.composed ? `Intended configuration may change on ${scopeComplete.count} eligible servers.` : scopeComplete.detached ? `Inventory tracking detached on ${scopeComplete.detached} servers; live files remain.` : `${scopeComplete.count} servers are in the old or new scope.`} Watchdog detects resulting drift and rebuilds eligible servers when automation is enabled. View drift recovery →
{/if} + {#if scopeComplete}
Scope saved. {scopeComplete.composed ? `Intended configuration may change on ${scopeComplete.count} eligible servers.` : scopeComplete.detached ? `Inventory tracking detached on ${scopeComplete.detached} servers; live files remain.` : `${scopeComplete.count} servers are in the old or new scope.`} Maintenance detects resulting drift and rebuilds eligible servers when automation is enabled. View drift recovery →
{/if} {#if latestVersion}

Latest {artifactVersionLabel(latestVersion)}

{latestVersion.notes || 'No version notes provided.'}

{#if runningBetaId !== null}{:else if latestVersion.id !== a.current_version_id}{:else}Latest version is promoted{/if}
{/if} {#if runningBetaId} @@ -417,7 +417,7 @@ {/if} - {#if scopePreviewLoading}

Reviewing eligible servers…

{:else if scopePreviewError}

{scopePreviewError}

{:else if scopeImpact}
{scopeImpact.after.length} eligible · {scopeImpact.added.length} added · {scopeImpact.removed.length} removed{#if scopeImpact.added.length}

Added: {scopeImpact.added.join(', ')}

{/if}{#if scopeImpact.removed.length}

Removed: {scopeImpact.removed.join(', ')}

{/if}

{scopeImpact.composed ? 'This changes intended configuration composition.' : scopeType === 'none' ? `${scopeImpact.detached.length} recorded installations will become untracked. Watchdog will detect orphaned files as drift.` : 'Scope describes eligibility for promoted files.'}

When automation is enabled, Watchdog rebuilds servers with drift after verifying a safe server state. Rebuilds delete the existing instance and all of its files.

{/if} + {#if scopePreviewLoading}

Reviewing eligible servers…

{:else if scopePreviewError}

{scopePreviewError}

{:else if scopeImpact}
{scopeImpact.after.length} eligible · {scopeImpact.added.length} added · {scopeImpact.removed.length} removed{#if scopeImpact.added.length}

Added: {scopeImpact.added.join(', ')}

{/if}{#if scopeImpact.removed.length}

Removed: {scopeImpact.removed.join(', ')}

{/if}

{scopeImpact.composed ? 'This changes intended configuration composition.' : scopeType === 'none' ? `${scopeImpact.detached.length} recorded installations will become untracked. Maintenance will detect orphaned files as drift.` : 'Scope describes eligibility for promoted files.'}

When automation is enabled, Maintenance rebuilds servers with drift after verifying a safe server state. Rebuilds delete the existing instance and all of its files.

{/if}
-
-

Plutainer applies updates on startup when auto-update is enabled. Restarts wait for a fresh player count of zero and share the fleet’s rolling restart limit.

- {#if error}{/if} - {#if notice}

{notice}

{/if} - {#if data} -

Latest published revision: {data.release.revision ? `r${data.release.revision}` : 'Unknown'} - · Production manifest - · Changelog - {#if data.release.checkedAt} · Checked {$timeFormat.dateTime(data.release.checkedAt)}{/if} -

- {#if data.release.error}

Update check failed: {data.release.error} Any displayed release is cached.

{/if} -
{ event.preventDefault(); void refresh({ action: 'schedule', enabled, time, timeZone: $timeZone }); }}> - - - -
-

Saving uses your selected timezone and follows its daylight saving changes. If the time is skipped, the queue starts at the next valid minute; repeated times run once.

- {#if data.schedule.enabled && data.schedule.timeZone !== $timeZone}

The saved schedule is {data.schedule.time} in {data.schedule.timeZone}. Save to use {$timeZone} instead.

{/if} -

This restarts all fleet servers daily, even without a new release. Busy servers may restart later. Disabling the schedule stops future runs; cancel existing queued restarts below.

- {#if data.schedule.nextAt}

Next run: {$timeFormat.dateTime(data.schedule.nextAt)}

{/if} -
- - {#each data.servers as server (server.server)} - - - - - {:else}{/each} -
ServerRunning revisionUpdate statusAction
{server.server}{server.revision ? `r${server.revision}` : 'Unknown'} - {#if server.checked_at}Checked {$timeFormat.dateTime(server.checked_at)}{/if} - {#if server.error}{server.error}{/if} - {server.revision === null || data.release.revision === null || data.release.error ? 'Unverified' - : server.revision < data.release.revision ? 'Update available' : server.revision > data.release.revision ? 'Ahead of published release' : 'Up to date'}
No fleet servers yet.
- {#if data.servers.length}{/if} - {#if data.jobs.length} -

Restart activity

-
    {#each data.jobs as job (job.id)}
  • - {job.server} · {job.state} · {$timeFormat.dateTime(job.requested_at)} - {#if job.target_revision} · Target r{job.target_revision}{/if} - {#if job.detail}{job.detail}{/if} - {#if job.provisioning_id}View provisioning{/if} - {#if job.state === 'queued'}{/if} -
    • {/each}
- {/if} - {:else}

{busy ? 'Loading maintenance status…' : 'Maintenance status unavailable.'}

{/if} - - - diff --git a/src/lib/ui/components/RuntimeVersion.svelte b/src/lib/ui/components/RuntimeVersion.svelte new file mode 100644 index 0000000..b459e4f --- /dev/null +++ b/src/lib/ui/components/RuntimeVersion.svelte @@ -0,0 +1,18 @@ + + + + Plutonium {revision ? `r${revision}` : 'unverified'}{outdated ? ' · Update available' : ''} + + + diff --git a/src/lib/watchdog.ts b/src/lib/watchdog.ts index 5440929..d2b8e52 100644 --- a/src/lib/watchdog.ts +++ b/src/lib/watchdog.ts @@ -13,6 +13,7 @@ export interface WatchdogIncident { finished_at: string | null; failures: number; cycle: number; + operation_stage?: string | null; } export interface WatchdogEvidence { reason?: string; diff --git a/src/routes/+layout.svelte b/src/routes/+layout.svelte index a86916a..54b3a1d 100644 --- a/src/routes/+layout.svelte +++ b/src/routes/+layout.svelte @@ -52,7 +52,7 @@ // Two-key nav shortcuts (Gmail-style): `g f` fleet, `g d` provisioning, // `g b` betas, `g s` scripts. Ignored while typing in inputs. const shortcuts: Record = { - f: '/', d: '/provisioning', a: '/campaigns', w: '/watchdog', b: '/betas', + f: '/', d: '/provisioning', a: '/campaigns', w: '/maintenance', b: '/betas', k: '/server-keys', s: '/scripts', c: '/configs', p: '/plugins', l: '/leaderboard', h: '/settings', m: '/metrics' }; let armed = false; @@ -99,7 +99,7 @@ const tabs = [ { href: '/', label: 'Fleet', match: (p: string) => p === '/' || p.startsWith('/servers/') }, { href: '/scripts', label: 'Library', match: (p: string) => /^\/(scripts|configs|plugins)(\/|$)/.test(p) }, - { href: '/provisioning', label: 'Operations', match: (p: string) => /^\/(provisioning|deployments|collections|campaigns|watchdog|server-keys)(\/|$)/.test(p) }, + { href: '/provisioning', label: 'Operations', match: (p: string) => /^\/(provisioning|deployments|collections|campaigns|maintenance|watchdog|server-keys)(\/|$)/.test(p) }, { href: '/betas', label: 'Betas', match: (p: string) => p.startsWith('/betas') }, { href: '/settings', label: 'Settings', match: (p: string) => p.startsWith('/settings') }, { href: '/leaderboard', label: 'Leaderboard', match: (p: string) => p.startsWith('/leaderboard') }, @@ -109,7 +109,7 @@ const secondary = $derived(tabs[1].match($page.url.pathname) ? [{ href: '/scripts', label: 'Scripts' }, { href: '/configs', label: 'Configs' }, { href: '/plugins', label: 'Plugins' }] : tabs[2].match($page.url.pathname) - ? [{ href: '/provisioning', label: 'Provisioning' }, { href: '/collections', label: 'Collections' }, { href: '/campaigns', label: 'Campaigns' }, { href: '/watchdog', label: 'Watchdog' }, { href: '/server-keys', label: 'Keys' }] + ? [{ href: '/provisioning', label: 'Provisioning' }, { href: '/collections', label: 'Collections' }, { href: '/campaigns', label: 'Campaigns' }, { href: '/maintenance', label: 'Maintenance' }, { href: '/server-keys', label: 'Keys' }] : tabs[4].match($page.url.pathname) ? [{ href: '/settings', label: 'Preferences' }] : []); @@ -149,7 +149,7 @@ -
{#each [['g f','Fleet'],['g s','Scripts'],['g c','Configs'],['g p','Plugins'],['g d','Provisioning'],['g a','Campaigns'],['g w','Watchdog'],['g b','Betas'],['g h','Settings'],['g k','Keys'],['g l','Leaderboard'],['g m','Metrics'],['?','This help']] as [keys,label]}
{keys}
{label}
{/each}
+
{#each [['g f','Fleet'],['g s','Scripts'],['g c','Configs'],['g p','Plugins'],['g d','Provisioning'],['g a','Campaigns'],['g w','Maintenance'],['g b','Betas'],['g h','Settings'],['g k','Keys'],['g l','Leaderboard'],['g m','Metrics'],['?','This help']] as [keys,label]}
{keys}
{label}
{/each}
 diff --git a/src/routes/+page.svelte b/src/routes/+page.svelte index d0dbc9e..392aa85 100644 --- a/src/routes/+page.svelte +++ b/src/routes/+page.svelte @@ -2,6 +2,7 @@ import { onMount, untrack } from 'svelte'; import { afterNavigate, replaceState } from '$app/navigation'; import { page } from '$app/stores'; + import RuntimeVersion from '$ui/components/RuntimeVersion.svelte'; import ServerCard from '$ui/components/ServerCard.svelte'; import DeployServerDialog from '$ui/components/DeployServerDialog.svelte'; import Button from '$ui/components/primitives/Button.svelte'; @@ -239,6 +240,7 @@ {/snippet} {#snippet serverActions(server: FleetServer)} + {/snippet} diff --git a/src/routes/api/maintenance/+server.ts b/src/routes/api/maintenance/+server.ts new file mode 100644 index 0000000..9676165 --- /dev/null +++ b/src/routes/api/maintenance/+server.ts @@ -0,0 +1,47 @@ +import { json } from '@sveltejs/kit'; +import { maintenanceOverview } from '$lib/server/maintenance/index.js'; +import { pauseMaintenance, setMaintenancePolicy } from '$lib/server/maintenance/policies.js'; +import { watchdogTick } from '$lib/server/maintenance/recovery.js'; +import { cancelMaintenance, checkPlutonium, queueUpdates, retryMaintenance, saveMaintenanceSchedule } from '$lib/server/maintenance/runtime.js'; +import { validTimeZone } from '$lib/time-zone.js'; +import { kick } from '$lib/server/pipeline/worker.js'; + +export function GET() { return json(maintenanceOverview()); } +export async function POST({ request }: { request: Request }) { + const body = await request.json().catch(() => null); + if (!body || typeof body !== 'object' || Array.isArray(body)) return json({ error: 'Invalid request.' }, { status: 400 }); + let notice = ''; + try { + switch (body.action) { + case 'pause': + if (typeof body.paused !== 'boolean') throw new Error('Provide a paused boolean.'); + pauseMaintenance(body.paused); + notice = body.paused ? 'Maintenance paused. Submitted operations continue recovery.' : 'Maintenance resumed.'; + break; + case 'policy': + if (!['health', 'drift', 'updates'].includes(body.policy) || typeof body.enabled !== 'boolean') throw new Error('Choose a policy and an enabled boolean.'); + setMaintenancePolicy(body.policy, body.enabled); notice = 'Policy saved.'; + break; + case 'scan': + await Promise.all([watchdogTick(true), checkPlutonium()]); notice = 'Checks finished.'; + break; + case 'update': + if (body.servers !== undefined && (!Array.isArray(body.servers) || !body.servers.length || body.servers.some((v: unknown) => typeof v !== 'string'))) throw new Error('Choose existing fleet servers.'); + notice = `${queueUpdates(body.servers)} update restart(s) queued. Occupied servers wait until empty.`; + break; + case 'schedule': + if (typeof body.enabled !== 'boolean' || typeof body.time !== 'string' || !validTimeZone(body.timeZone)) throw new Error('Provide enabled, a time, and a valid timezone.'); + saveMaintenanceSchedule(body.enabled, body.time, body.timeZone); notice = 'Schedule saved. Existing requests are retained.'; + break; + case 'cancel': + if (!Number.isSafeInteger(body.id) || !cancelMaintenance(body.id)) return json({ error: 'Only unsubmitted restarts can be cancelled.' }, { status: 409 }); + notice = 'Queued restart cancelled.'; break; + case 'retry': + if (!Number.isSafeInteger(body.id)) throw new Error('Choose a maintenance activity.'); + notice = `${retryMaintenance(body.id)} restart(s) queued for verification.`; break; + default: throw new Error('Unknown maintenance action.'); + } + } catch (error) { return json({ error: (error as Error).message }, { status: 400 }); } + void kick().catch(error => console.error('[maintenance] worker kick failed:', error)); + return json({ ...maintenanceOverview(), notice }); +} diff --git a/src/routes/api/maintenance/[id]/+server.ts b/src/routes/api/maintenance/[id]/+server.ts new file mode 100644 index 0000000..7bba946 --- /dev/null +++ b/src/routes/api/maintenance/[id]/+server.ts @@ -0,0 +1,16 @@ +import { json } from '@sveltejs/kit'; +import { watchdogDetail, updateWatchdogIncident } from '$lib/server/maintenance/recovery.js'; +import { kick } from '$lib/server/pipeline/worker.js'; + +export function GET({ params }: { params: { id: string } }) { + const detail = watchdogDetail(params.id); + return detail ? json(detail) : json({ error: 'Maintenance finding not found.' }, { status: 404 }); +} +export async function POST({ params, request }: { params: { id: string }; request: Request }) { + const body = await request.json().catch(() => null); + if (!body || !['retry','dismiss'].includes(body.action)) return json({ error: 'Choose retry or dismiss.' }, { status: 400 }); + const result = updateWatchdogIncident(params.id, body.action); + if (!result.ok) return json({ error: result.error }, { status: result.status }); + void kick().catch(err => console.error('[maintenance] worker kick failed:', err)); + return json(watchdogDetail(params.id)); +} diff --git a/src/routes/api/servers/+server.ts b/src/routes/api/servers/+server.ts index fb448f4..66462d4 100644 --- a/src/routes/api/servers/+server.ts +++ b/src/routes/api/servers/+server.ts @@ -6,12 +6,15 @@ import { serverObservation } from '$lib/server/fleet/status.js'; import { queueCreateServer } from '$lib/server/provisioning/index.js'; import { config, komodoConfigured, rconConfigured } from '$lib/server/config.js'; import type { GameMode, ZombieGame } from '$lib/server/fleet/map-aliases.js'; +import { runningVersions } from '$lib/server/integrations/plutonium.js'; export async function GET() { const rconHost = config.rcon.host; + const versions = new Map(runningVersions().map(version => [version.server, version])); const servers = listServers().map((s) => { return { ...serverObservation(s), + plutonium: versions.get(s.name), // Derived endpoint — same logic as endpointFor() but without asking // Komodo. If we don't have a port yet, endpoint is null. endpoint: s.game_port ? { host: s.rcon_host ?? rconHost, port: s.game_port } : null, diff --git a/src/routes/campaigns/[id]/+page.svelte b/src/routes/campaigns/[id]/+page.svelte index f5b8c55..6a345cd 100644 --- a/src/routes/campaigns/[id]/+page.svelte +++ b/src/routes/campaigns/[id]/+page.svelte @@ -122,9 +122,9 @@ {/snippet} -

{completed} / {detail.servers.length} servers activated{unresolved ? ` · ${unresolved} unresolved` : ''}

{detail.watchdogIncidentId ? 'Watchdog repairs drift by deleting the existing instance files and rebuilding with the latest promoted setup after verifying a safe server state.' : 'Files are staged before activation. Servers restart when empty, then their health is verified.'}

{#if unresolved}

Review failed or waiting servers below. Diagnose connectivity or inspect startup logs from the affected row.

{:else}

Every recorded server completed activation.

{/if}
+

{completed} / {detail.servers.length} servers activated{unresolved ? ` · ${unresolved} unresolved` : ''}

{detail.watchdogIncidentId ? 'Maintenance repairs drift by deleting the existing instance files and rebuilding with the latest promoted setup after verifying a safe server state.' : 'Files are staged before activation. Servers restart when empty, then their health is verified.'}

{#if unresolved}

Review failed or waiting servers below. Diagnose connectivity or inspect startup logs from the affected row.

{:else}

Every recorded server completed activation.

{/if}
- {#if detail.watchdogIncidentId}

Started by Watchdog · View incident and recovery history

{/if} + {#if detail.watchdogIncidentId}

Started by Maintenance · View incident and recovery history

{/if}
diff --git a/src/routes/maintenance/+page.svelte b/src/routes/maintenance/+page.svelte new file mode 100644 index 0000000..bf6bab0 --- /dev/null +++ b/src/routes/maintenance/+page.svelte @@ -0,0 +1,221 @@ + + +Maintenance | Electric Cherry +
+ + {#snippet actions()} + + {#if data}{/if} + {/snippet} + + {#if error}{/if} + {#if notice}

{notice}

{/if} + {#if data?.policies.paused}
All maintenance is paused. Checks and verification continue. New actions, scheduled runs, and queued maintenance are held. Submitted operations finish recovery. Campaigns, betas, and manual Provisioning actions continue.
{/if} + saveContext()} ariaLabel="Maintenance views" idPrefix="maintenance-tabs" panelId="maintenance-panel" /> +
+ {#if data} + {#if view === 'policies'} +

Policies apply to the current fleet and respect intentionally stopped servers. Disabling a recovery or automatic update policy holds its queued actions. Already submitted operations continue verification.

+
{#each policies as policy} +
+

{policy.title}

{data.policies[policy.id] ? 'Enabled' : 'Disabled'}
+

{policy.description}

+ +
+ {/each}
+
+

Scheduled restarts

+

Restart the running fleet daily, even without a new release. Intentionally stopped servers are excluded. Busy servers wait until empty and may restart later.

+
{ event.preventDefault(); void act({ action: 'schedule', enabled: scheduleEnabled, time: scheduleTime, timeZone: $timeZone }); }}> + + + +
+

Saving follows daylight saving changes in your selected timezone. Skipped times run at the next valid minute; repeated times run once. Missed days become one run on resume.

+ {#if data.runtime.schedule.enabled && data.runtime.schedule.timeZone !== $timeZone}

Saved for {data.runtime.schedule.time} in {data.runtime.schedule.timeZone}. Save to use {$timeZone}.

{/if} + {#if data.runtime.schedule.nextAt}

Next run: {$timeFormat.dateTime(data.runtime.schedule.nextAt)}{data.policies.paused ? ' · Held while maintenance is paused' : ''}

{/if} +

Disabling the schedule stops future runs. Existing requests remain in Activity, where unsubmitted restarts can be cancelled.

+
+ {:else} + {#if view === 'overview'} +
{attention} need attention{waiting} active{outdated.length} outdated{unknown.length} versions unverified
+ {#if data.runtime.schedule.nextAt}

Next scheduled restart: {$timeFormat.dateTime(data.runtime.schedule.nextAt)} · Manage schedule

{/if} + {#if data.recovery.lastScanAt}

Health and drift checked {$timeFormat.dateTime(data.recovery.lastScanAt)}

{/if} + {#if data.recovery.scanError}{/if} + {/if} +
+ + + {#if view === 'activity'}{/if} +
+ {#if view === 'overview' && ['all', 'updates'].includes(kind)} +
+

Plutonium updates

+

Published revision: {data.runtime.release.revision ? `r${data.runtime.release.revision}` : 'Unknown'} · Production manifest · Changelog

+ {#if data.runtime.release.checkedAt}

Checked {$timeFormat.dateTime(data.runtime.release.checkedAt)}

{/if} + {#if data.runtime.release.error}{/if} +

Plutainer applies the available release at startup when auto-update is enabled. Maintenance verifies the running revision after recovery. Unknown versions require a successful check before automatic updates.

+
+ {#each versions as version (version.server)} + {@const pending = activeJobs.find(j => j.server === version.server)} + + + + {:else}{/each} +
ServerRunning revisionStatusAction
{version.revision ? `r${version.revision}` : 'Unknown'}{#if version.checked_at}{$timeFormat.dateTime(version.checked_at)}{/if}{#if version.error}{version.error}{/if}{version.update_status === 'outdated' ? 'Update available' : version.update_status === 'current' ? 'Up to date' : version.update_status === 'ahead' ? 'Ahead of published release' : version.update_status === 'stopped' ? 'Intentionally stopped' : 'Unverified'}{#if pending}Restart pending{:else if version.update_status === 'outdated'}{/if}
No matching fleet servers.
+
+ {/if} +
+

{view === 'overview' ? 'Active work & attention' : 'Maintenance activity'}

+
{#each matching.slice(0, limit) as item (item.key)} +
+

{item.title} ·

{item.label}
+ {#if item.detail}

{item.detail}

{/if} +

{$timeFormat.dateTime(item.at)}{item.job?.target_revision ? ` · Expected r${item.job.target_revision} or newer` : ''}

+
{#if item.href}{item.job ? 'View restart operation' : 'View evidence and recovery'}{/if} + {#if item.job?.state === 'queued'}{/if} + {#if item.job?.state === 'failed' && item.job.operation_state === 'done'}{/if} +
+
+ {:else}{/each}
+ saveContext(false)} label="activities" /> +
+ {/if} + {:else if !error}

Loading maintenance…

{/if} +
+
+ + diff --git a/src/routes/watchdog/[id]/+page.svelte b/src/routes/maintenance/[id]/+page.svelte similarity index 94% rename from src/routes/watchdog/[id]/+page.svelte rename to src/routes/maintenance/[id]/+page.svelte index a74c1b7..f0cf845 100644 --- a/src/routes/watchdog/[id]/+page.svelte +++ b/src/routes/maintenance/[id]/+page.svelte @@ -41,7 +41,7 @@ if (incident.state === 'dismissed') return 'This incident was dismissed. Retry to allow automatic recovery again.'; if (incident.state === 'attention') return 'Automatic recovery needs attention. Review the failed operation and server diagnostics before retrying.'; if (incident.state === 'recovering') return 'Recovery is in progress. Follow the provisioning operations below for eligibility checks, progress, and health verification.'; - return 'Watchdog detected this problem. Automatic recovery requires a confirmed stopped container or a fresh zero-player count from RCON or telemetry before interrupting the server.'; + return 'Maintenance detected this problem. Automatic recovery requires a confirmed stopped container or a fresh zero-player count from RCON or telemetry before interrupting the server.'; }); async function load() { @@ -49,8 +49,8 @@ const id = $page.params.id ?? ''; const current = ++request; try { - const response = await fetch(`/api/watchdog/${encodeURIComponent(id)}`); - if (!response.ok) throw new Error(response.status === 404 ? 'Watchdog incident not found.' : 'Could not refresh this incident.'); + const response = await fetch(`/api/maintenance/${encodeURIComponent(id)}`); + if (!response.ok) throw new Error(response.status === 404 ? 'Maintenance finding not found.' : 'Could not refresh this incident.'); const body = await response.json() as WatchdogDetail; if (current !== request || id !== $page.params.id) return; detail = body; loadError = ''; @@ -65,14 +65,14 @@ const current = ++request; busy = action; actionError = ''; notice = ''; try { - const response = await fetch(`/api/watchdog/${encodeURIComponent(id)}`, { + const response = await fetch(`/api/maintenance/${encodeURIComponent(id)}`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ action }) }); const body = await response.json(); if (!response.ok) throw new Error(body.error ?? 'Could not update this incident.'); if (current !== request || id !== $page.params.id) return; detail = body; loadError = ''; - notice = action === 'retry' ? 'Recovery retry requested. Watchdog will recheck the server before taking action.' : 'Incident dismissed.'; + notice = action === 'retry' ? 'Recovery retry requested. Maintenance will recheck the server before taking action.' : 'Incident dismissed.'; } catch (error) { if (current === request && id === $page.params.id) actionError = (error as Error).message; } finally { if (current === request) busy = null; } @@ -90,11 +90,11 @@ }); -{incident ? `${serverTitle(incident.server)} · Watchdog` : 'Watchdog incident'} | Electric Cherry +{incident ? `${serverTitle(incident.server)} · Maintenance` : 'Maintenance finding'} | Electric Cherry
- - {#snippet crumbs()}{/snippet} + + {#snippet crumbs()}{/snippet} {#snippet actions()} {#if canRetry}{/if} {#if canDismiss}{/if} diff --git a/src/routes/provisioning/[id]/+page.svelte b/src/routes/provisioning/[id]/+page.svelte index b1a35e6..453c0f6 100644 --- a/src/routes/provisioning/[id]/+page.svelte +++ b/src/routes/provisioning/[id]/+page.svelte @@ -152,7 +152,7 @@ {#each warnings as warning}

{warning}

{/each} {#if deployment.expected_map}

Expected map: {deployment.expected_map}

{/if}
Started
{fmt(deployment.started_at)}
Finished
{deployment.finished_at ? fmt(deployment.finished_at) : 'In progress'}
Requested by
{deployment.triggered_by}
- {#if detail?.sources.length}
{#each detail.sources as source}{source.kind === 'campaign' ? 'Campaign' : source.kind === 'beta' ? 'Beta' : source.kind === 'watchdog' ? 'Watchdog incident' : 'Maintenance'} {source.source_id}{/each}
{/if} + {#if detail?.sources.length}
{#each detail.sources as source}{source.kind === 'campaign' ? 'Campaign' : source.kind === 'beta' ? 'Beta' : source.kind === 'watchdog' ? 'Maintenance finding' : 'Maintenance'} {source.source_id}{/each}
{/if} {#if detail?.recipe}

{detail.blueprintName ? `Launched from ${detail.blueprintName}` : 'Recorded launch files'}

Launch this map with the latest promoted files using a fresh server ID and port. This does not repair the existing server.

{/if} ({label,state:deployment.state === 'cancelled' ? 'pending' : index < currentStep ? 'done' : index === currentStep ? (!active && ['failed','interrupted'].includes(deployment.state) ? 'failed' : 'current') : 'pending'}))} events={detail?.events ?? []} {active} /> diff --git a/src/routes/servers/[name]/+page.svelte b/src/routes/servers/[name]/+page.svelte index 6ca09c5..d4a5331 100644 --- a/src/routes/servers/[name]/+page.svelte +++ b/src/routes/servers/[name]/+page.svelte @@ -1,4 +1,5 @@ - -Watchdog | Electric Cherry - -
- - {#snippet actions()} - - - {/snippet} - - - {#if loadError}{/if} - {#if actionError}{/if} - {#if notice}

{notice}

{/if} - - {#if overview} -
-

Automatic recovery

{overview.enabled ? 'Enabled' : 'Paused'}
-

{overview.enabled ? 'Watchdog restarts unhealthy servers, then rebuilds if recovery fails.' : 'Watchdog continues detecting incidents while automatic recovery is paused. Queued Watchdog recovery is held; already submitted operations continue.'}

-

Drift repair rebuilds the server with the latest promoted setup and records a campaign. Rebuilds delete the existing instance and all of its files. Interruption requires a confirmed stopped container or a fresh zero-player count from RCON or telemetry.

-
{active} active incident{active === 1 ? '' : 's'} · {attention} need attentionLast scan: {overview.lastScanAt ? $timeFormat.dateTime(overview.lastScanAt) : 'Not yet scanned'}
- {#if overview.scanError}{/if} -
- -
-
- - - -
-

Incidents {matching.length}

- {#if !matching.length} - - {:else} -
- {#each matching.slice(0, limit) as incident (incident.id)} - - {/each} -
- saveContext()} label="incidents" /> - {/if} -
- {:else if !loadError}

Loading Watchdog…

{/if} -
- - diff --git a/src/routes/watchdog/[id]/+page.server.ts b/src/routes/watchdog/[id]/+page.server.ts new file mode 100644 index 0000000..15b5ed2 --- /dev/null +++ b/src/routes/watchdog/[id]/+page.server.ts @@ -0,0 +1,2 @@ +import { redirect } from '@sveltejs/kit'; +export function load({ params, url }: { params: { id: string }; url: URL }) { redirect(308, `/maintenance/${encodeURIComponent(params.id)}${url.search}`); } diff --git a/tests/unit/fleet-overview.test.ts b/tests/unit/fleet-overview.test.ts index e358614..d1574a3 100644 --- a/tests/unit/fleet-overview.test.ts +++ b/tests/unit/fleet-overview.test.ts @@ -60,6 +60,16 @@ describe('persisted Fleet operations', () => { beforeEach(async () => { vi.resetModules(); fixture = await newFixture(); }); afterEach(async () => { await fixture.cleanup(); }); + it('shows failed update verification even after a healthy restart and clears attention on a new attempt', async () => { + const { db } = await import('../../src/lib/server/db/index.js'); + const handle = db(); + handle.prepare("INSERT INTO maintenance_jobs(server,kind,state,target_revision,requested_at,detail) VALUES ('server','update','failed',5354,?,'Revision not verified')").run(iso()); + const { fleetOperations } = await import('../../src/lib/server/fleet/overview.js'); + expect(fleetOperations(now)).toEqual([expect.objectContaining({ kind: 'maintenance', state: 'attention', detail: 'Revision not verified' })]); + handle.prepare("INSERT INTO maintenance_jobs(server,kind,state,target_revision,requested_at) VALUES ('server','update','queued',5354,?)").run(iso(1000)); + expect(fleetOperations(now)).toEqual([]); + }); + it('recovers waiting and recovering campaigns without needing a live event', async () => { const { db, closeDb } = await import('../../src/lib/server/db/index.js'); const handle = db(); diff --git a/tests/unit/maintenance-api.test.ts b/tests/unit/maintenance-api.test.ts new file mode 100644 index 0000000..e3b4d7b --- /dev/null +++ b/tests/unit/maintenance-api.test.ts @@ -0,0 +1,39 @@ +import { beforeEach, afterEach, describe, it, expect, vi } from 'vitest'; +import { newFixture, type Fixture } from '../helpers/fixture.js'; +vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn().mockResolvedValue(undefined) })); +let fixture: Fixture; +beforeEach(async () => { vi.resetModules(); fixture = await newFixture(); }); +afterEach(async () => { await fixture.cleanup(); }); +const request = (body: unknown) => new Request('http://localhost/api/maintenance', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) }); + +describe('Maintenance API', () => { + it('saves independent policies and the global pause without changing the restart schedule', async () => { + const { POST, GET } = await import('../../src/routes/api/maintenance/+server.js'); + expect((await POST({ request: request({ action: 'schedule', enabled: true, time: '06:00', timeZone: 'America/Indiana/Indianapolis' }) })).status).toBe(200); + const before = (await (GET()).json()).runtime.schedule; + for (const body of [{ action: 'policy', policy: 'health', enabled: false }, { action: 'policy', policy: 'updates', enabled: true }, { action: 'pause', paused: true }]) { + expect((await POST({ request: request(body) })).status).toBe(200); + } + const data = await GET().json(); + expect(data.policies).toEqual({ paused: true, health: false, drift: true, updates: true }); + expect(data.runtime.schedule).toEqual(before); + expect((await POST({ request: request({ action: 'update' }) })).status).toBe(400); + }); + it('rejects malformed actions without changing policies or queuing work', async () => { + const { POST, GET } = await import('../../src/routes/api/maintenance/+server.js'); + const before = await GET().json(); + for (const body of [null, [], {}, { action: 'pause', paused: 'yes' }, { action: 'policy', policy: 'unknown', enabled: true }, + { action: 'policy', policy: 'updates', enabled: 1 }, { action: 'schedule', enabled: true, time: '25:00', timeZone: 'UTC' }, + { action: 'schedule', enabled: true, time: '06:00', timeZone: 'invalid' }, { action: 'update', servers: [] }, + { action: 'update', servers: [1] }, { action: 'retry', id: '1' }]) { + expect((await POST({ request: request(body) })).status).toBe(400); + } + expect(await GET().json()).toEqual(before); + }); + it('keeps old Watchdog URLs working through query-preserving redirects', async () => { + const index = await import('../../src/routes/watchdog/+page.server.js'); + const detail = await import('../../src/routes/watchdog/[id]/+page.server.js'); + expect(() => index.load({ url: new URL('http://localhost/watchdog?kind=drift&server=test') })).toThrow(expect.objectContaining({ status: 308, location: '/maintenance?kind=drift&server=test' })); + expect(() => detail.load({ params: { id: 'abc' }, url: new URL('http://localhost/watchdog/abc?test=1') })).toThrow(expect.objectContaining({ status: 308, location: '/maintenance/abc?test=1' })); + }); +}); diff --git a/tests/unit/plutonium.test.ts b/tests/unit/plutonium.test.ts index 6b07f80..4de52f5 100644 --- a/tests/unit/plutonium.test.ts +++ b/tests/unit/plutonium.test.ts @@ -233,3 +233,117 @@ describe('persistent maintenance', () => { expect(m.plutoniumStatus().schedule.nextAt).toBeNull(); }); }); + +describe('Maintenance policies and runtime updates', () => { + async function outdated() { + const rcon = await import('../../src/lib/server/integrations/rcon.js'); + vi.mocked(rcon.rconPlutoniumVersion).mockResolvedValue({ revision: 5300, error: null }); + const m = await api(); await m.checkPlutonium(); return m; + } + it('preserves the legacy recovery preference without enabling updates', async () => { + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + db().prepare("INSERT INTO settings(key,value) VALUES ('watchdog_enabled','0')").run(); + const p = await import('../../src/lib/server/maintenance/policies.js'); + expect(p.maintenancePolicies()).toEqual({ paused: false, health: false, drift: false, updates: false }); + p.setMaintenancePolicy('health', true); p.pauseMaintenance(true); closeDb(); + expect(p.maintenancePolicies()).toEqual({ paused: true, health: true, drift: false, updates: false }); + expect(p.maintenanceAllowed('health')).toBe(false); + }); + it('only automatically updates verified outdated servers and requires opt-in', async () => { + const m = await outdated(); + const { db } = await import('../../src/lib/server/db/index.js'); + db().prepare('UPDATE plutonium_versions SET revision = NULL WHERE server = ?').run(servers[1]); + await m.maintenanceTick(); + expect(m.plutoniumStatus().jobs).toEqual([]); + const p = await import('../../src/lib/server/maintenance/policies.js'); p.setMaintenancePolicy('updates', true); + await m.maintenanceTick(); await m.maintenanceTick(); + expect(m.plutoniumStatus().jobs).toEqual([expect.objectContaining({ server: servers[0], kind: 'update', origin: 'automatic', target_revision: 5354 })]); + expect(m.plutoniumStatus().servers.find(s => s.server === servers[1])?.update_status).toBe('unknown'); + }); + it('holds queued automatic updates when their policy is disabled, but finishes submitted recovery while paused', async () => { + const m = await outdated(); + const p = await import('../../src/lib/server/maintenance/policies.js'); p.setMaintenancePolicy('updates', true); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js'); + m.queueUpdates([servers[0]], true); p.setMaintenancePolicy('updates', false); + await processProvisioning(); expect(komodoRestart).not.toHaveBeenCalled(); + p.setMaintenancePolicy('updates', true); p.pauseMaintenance(true); + await processProvisioning(); expect(komodoRestart).not.toHaveBeenCalled(); + p.pauseMaintenance(false); await processProvisioning(); expect(komodoRestart).toHaveBeenCalledTimes(1); + p.pauseMaintenance(true); vi.setSystemTime(Date.now() + 15_000); + await processProvisioning(); await m.maintenanceTick(); + expect(m.plutoniumStatus().jobs[0]).toMatchObject({ state: 'failed', operation_state: 'done' }); + expect(komodoRestart).toHaveBeenCalledTimes(1); + }); + it('latches failed update verification across restarts and retention, and supports an explicit retry', async () => { + const m = await outdated(); + const p = await import('../../src/lib/server/maintenance/policies.js'); p.setMaintenancePolicy('updates', true); + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + db().prepare("UPDATE servers SET desired_state = 'stopped' WHERE name = ?").run(servers[1]); + await maintenanceTick(); vi.setSystemTime(Date.now() + 15_000); await maintenanceTick(); + const failed = m.plutoniumStatus().jobs[0]; + expect(failed).toMatchObject({ kind: 'update', state: 'failed', operation_state: 'done' }); + closeDb(); vi.setSystemTime(Date.now() + 31 * 86_400_000); await maintenanceTick(); + expect(m.plutoniumStatus().jobs).toHaveLength(1); + expect(m.retryMaintenance(failed.id)).toBe(1); + expect(m.plutoniumStatus().jobs).toHaveLength(2); + const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js'); + vi.mocked(rconPlutoniumVersion).mockResolvedValue({ revision: 5354, error: null }); + await maintenanceTick(); vi.setSystemTime(Date.now() + 15_000); await maintenanceTick(); + expect(m.plutoniumStatus().jobs[0]).toMatchObject({ state: 'done', origin: 'manual' }); + expect(db().prepare("SELECT 1 FROM server_deployments WHERE action = 'rebuild'").get()).toBeUndefined(); + }); + it('does not restart intentionally stopped servers, including jobs queued before the intent changed', async () => { + const m = await outdated(); m.queueUpdates([servers[0]]); + const { db } = await import('../../src/lib/server/db/index.js'); + db().prepare("UPDATE servers SET desired_state = 'stopped'").run(); + await maintenanceTick(); + const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js'); + expect(komodoRestart).not.toHaveBeenCalled(); + expect(m.plutoniumStatus().jobs[0].state).toBe('cancelled'); + expect(m.queueUpdates()).toBe(0); + expect(m.plutoniumStatus().servers.every(s => s.update_status === 'stopped')).toBe(true); + }); + it('separates scheduled restart success from revision verification and coalesces paused runs', async () => { + const m = await outdated(); m.saveMaintenanceSchedule(true, '06:00', 'UTC'); + const p = await import('../../src/lib/server/maintenance/policies.js'); p.pauseMaintenance(true); + vi.setSystemTime(new Date('2026-09-24T07:00:00Z')); await m.maintenanceTick(); + expect(m.plutoniumStatus().jobs).toHaveLength(0); + p.pauseMaintenance(false); await maintenanceTick(); + expect(m.plutoniumStatus().jobs).toHaveLength(2); + expect(m.plutoniumStatus().jobs.every(j => j.kind === 'schedule' && j.target_revision === null)).toBe(true); + const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js'); + vi.mocked(rconPlutoniumVersion).mockResolvedValue({ revision: null, error: 'Unknown reply' }); + vi.setSystemTime(Date.now() + 15_000); await maintenanceTick(); + expect(m.plutoniumStatus().jobs.find(j => j.server === servers[0])?.state).toBe('done'); + expect(m.plutoniumStatus().schedule.nextAt).toBe('2026-09-25T06:00:00.000Z'); + }); + it('does not treat stale, pre-restart, or unhealthy observations as current runtime evidence', async () => { + const m = await outdated(); + const { db } = await import('../../src/lib/server/db/index.js'); + db().prepare('INSERT INTO restart_rollout(server,requested_at,healthy_at) VALUES (?,?,?)').run(servers[0], new Date().toISOString(), new Date().toISOString()); + expect(m.plutoniumStatus().servers[0].update_status).toBe('unknown'); + vi.setSystemTime(Date.now() + 3_600_000); + expect(() => m.queueUpdates()).toThrow('published revision'); + const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js'); + const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js'); + vi.mocked(rconPlutoniumVersion).mockClear(); + vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'starting', reason: 'Starting' }); + await m.checkPlutonium(); + expect(rconPlutoniumVersion).not.toHaveBeenCalled(); + expect(m.plutoniumStatus().servers.every(s => s.update_status === 'unknown')).toBe(true); + }); + it('adopts existing maintenance jobs without changing their IDs, schedules, or submitted operations', async () => { + const m = await outdated(); m.saveMaintenanceSchedule(true, '06:00', 'America/New_York'); + m.queueMaintenance([servers[0]]); await maintenanceTick(); + const before = m.plutoniumStatus(); + const { db, closeDb } = await import('../../src/lib/server/db/index.js'); + db().exec('ALTER TABLE maintenance_jobs DROP COLUMN kind; ALTER TABLE maintenance_jobs DROP COLUMN origin;'); closeDb(); + const after = m.plutoniumStatus(); + expect(after.schedule).toEqual(before.schedule); + expect(after.jobs[0]).toMatchObject({ id: before.jobs[0].id, provisioning_id: before.jobs[0].provisioning_id, kind: 'legacy', origin: 'manual' }); + vi.setSystemTime(Date.now() + 15_000); await maintenanceTick(); + const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js'); + expect(komodoRestart).toHaveBeenCalledTimes(1); + }); +}); diff --git a/tests/unit/watchdog.test.ts b/tests/unit/watchdog.test.ts index 3379965..cc0a310 100644 --- a/tests/unit/watchdog.test.ts +++ b/tests/unit/watchdog.test.ts @@ -395,3 +395,36 @@ describe('Watchdog drift campaigns', () => { expect(komodo.komodoDestroyDeployment).not.toHaveBeenCalled(); }); }); + +describe('Maintenance recovery policy boundaries', () => { + it('separates health recovery from drift repair and holds an already queued drift rebuild', async () => { + const policies = await import('../../src/lib/server/maintenance/policies.js'); + policies.setMaintenancePolicy('health', false); + down(); + for (let n = 0; n < 3; n++) { await watchdog.watchdogTick(true); advance(watchdog.WATCHDOG_SCAN_MS); } + expect(operations()).toEqual([]); + expect(onlyIncident().kind).toBe('health'); + vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' }); + await writeFile(join(fixture.t5, server, 'runtime/plutonium/storage/t5/raw/scripts/sp', 'stray.gsc'), 'drift'); + await watchdog.watchdogTick(true); + expect(operations()).toEqual([expect.objectContaining({ action: 'rebuild', state: 'waiting' })]); + policies.setMaintenancePolicy('drift', false); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + await processProvisioning(); + expect(operations()[0].state).toBe('waiting'); + expect(komodo.komodoDestroyDeployment).not.toHaveBeenCalled(); + }); + it('global pause blocks escalation without disabling the saved health policy', async () => { + await sustainedFailure(); + const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js'); + await processProvisioning(); + const policies = await import('../../src/lib/server/maintenance/policies.js'); + policies.pauseMaintenance(true); + advance(watchdog.WATCHDOG_RECOVERY_MS); await watchdog.watchdogTick(true); + expect(operations()).toHaveLength(1); + expect(policies.maintenancePolicies().health).toBe(true); + policies.pauseMaintenance(false); await watchdog.watchdogTick(true); + expect(operations()).toHaveLength(2); + expect(operations()[1].action).toBe('rebuild'); + }); +});