feat: unify recovery, updates, and schedules under Maintenance
This commit is contained in:
@@ -67,56 +67,39 @@ status between refreshes. Failed refreshes keep the last snapshot visible.
|
||||
Plugins support named release versions, GitHub DLL imports and update discovery.
|
||||
See [Plugin releases](docs/plugin-releases.md) for the import and beta workflow.
|
||||
|
||||
## Plutonium updates and scheduled restarts
|
||||
## Maintenance
|
||||
|
||||
Open **Settings → Plutonium updates & restarts** to check the latest published
|
||||
revision, inspect each server's running revision, and queue a server or fleet
|
||||
restart. Release and running-version checks also run hourly. The published
|
||||
revision comes from the [official production updater manifest](https://cdn.plutoniummod.com/updater/prod/info.json),
|
||||
which exposes the game revision directly and avoids the changelog website's browser challenge;
|
||||
running revisions come from RCON `version`. Unsupported replies and failed
|
||||
queries appear as unknown rather than being treated as up to date. A failed
|
||||
manifest request retains the last release with a visible error.
|
||||
Open **Operations → Maintenance** for health recovery, configuration drift,
|
||||
Plutonium updates, and scheduled restarts. Overview shows active work, attention,
|
||||
version coverage, and upcoming restarts. Policies & schedules controls each
|
||||
capability independently; Activity links findings and results to Provisioning.
|
||||
Fleet and server pages show runtime revision indicators and maintenance links.
|
||||
|
||||
Daily fleet restarts are optional and disabled by default. Choose a queue time
|
||||
in your **selected timezone** and save the schedule. The timezone is saved with
|
||||
the schedule so the daily time follows daylight saving changes even when your
|
||||
browser is closed. Skipped times run at the first valid minute after the gap;
|
||||
repeated times run once on their first occurrence. Existing UTC schedules retain
|
||||
their original timing until saved again. The form displays their next occurrence
|
||||
in your timezone and identifies the saved zone if it differs. Changing a browser's
|
||||
display preference does not silently change a fleet schedule; save to apply it.
|
||||
Every due run queues the current fleet, regardless of whether an update exists.
|
||||
The schedule and jobs survive app restarts. Missed days become one run, and a
|
||||
server can have only one pending maintenance restart. Disabling the schedule
|
||||
does not cancel work already queued; use **Cancel** for those jobs.
|
||||
Maintenance decides when work is needed and verifies the result. Provisioning
|
||||
executes every lifecycle operation using the shared safety checks and queues.
|
||||
Health recovery uses one restart followed by one rebuild if needed; drift repair
|
||||
requests a rebuild with a linked Campaign. **Rebuild deletes all instance files.**
|
||||
|
||||
Maintenance waits for a fresh zero-player RCON or telemetry report and for existing
|
||||
deployment/beta work, then uses the shared rolling restart limit. Unreachable or
|
||||
occupied servers remain queued. After container and game-status recovery, Electric
|
||||
Cherry queries the running revision again. If a fresh published revision was
|
||||
known when queued, recovery on an older or unknown revision is reported as a
|
||||
failed verification, without an automatic restart loop. Ambiguous restart
|
||||
failures likewise require inspection before a new request. Activity is retained
|
||||
for 30 days, with pending jobs and the latest 100 records shown in Settings.
|
||||
Plutonium updates are applied by Plutainer's startup updater
|
||||
(`PLUTAINER_AUTO_UPDATE` must be enabled). Check versions and choose **Update on
|
||||
restart**, or opt into automatic updates for verified outdated servers. Revision
|
||||
checks run hourly. Unknown or stale revisions remain unverified. A healthy restart
|
||||
can still fail update verification; Maintenance reports that outcome without
|
||||
rebuilding or repeatedly restarting the server. Electric Cherry does not download
|
||||
binaries or pin a runtime revision.
|
||||
|
||||
Updates rely on Plutainer's startup updater (`PLUTAINER_AUTO_UPDATE` must be
|
||||
enabled); Electric Cherry does not download binaries or pull a container image.
|
||||
See [Plutainer's startup flow](https://github.com/Ayymoss/Plutainer/blob/main/CLAUDE.md)
|
||||
and [environment migration reference](https://github.com/Ayymoss/Plutainer/blob/main/MIGRATION.md).
|
||||
The worker starts on the first application request, so keep the app running and
|
||||
reachable for scheduled maintenance. Validate RCON revision replies and updater
|
||||
behavior on your live T5/T6 servers before relying on unattended maintenance.
|
||||
Optional daily restarts are independent of updates. Schedules use the saved
|
||||
timezone, follow daylight saving changes, and coalesce missed days into one run.
|
||||
Intentionally stopped servers are excluded. Existing schedules, pending jobs,
|
||||
operation IDs, and history survive the move from Settings and Watchdog.
|
||||
|
||||
**Pause all maintenance** holds new and queued maintenance while checks and
|
||||
submitted recovery continue. Each capability retains its own policy. Automatic
|
||||
updates remain disabled until explicitly enabled. See [Maintenance](docs/maintenance.md)
|
||||
for verification, retry, scheduling, and compatibility details.
|
||||
|
||||
## How restarts stay safe
|
||||
|
||||
**Operations → Watchdog** automatically detects unhealthy servers and file drift.
|
||||
Health incidents receive one restart followed by one rebuild if recovery fails;
|
||||
drift queues a rebuild with the promoted setup and a linked Campaign. Incidents,
|
||||
evidence, and attempts survive application restarts. Pause automation, inspect
|
||||
failures, and explicitly retry from Watchdog. Intentional stops and active beta or
|
||||
campaign work are respected. See [Watchdog](docs/watchdog.md) for timing and recovery behavior.
|
||||
|
||||
Betas support configurable gameplay, participation, stability and manual-review
|
||||
requirements, saved presets, coverage rules and staged waves. See
|
||||
[Beta test plans](docs/beta-test-plans.md) for usage and the four-event game telemetry contract.
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
# Maintenance
|
||||
|
||||
Open **Operations → Maintenance** to keep existing servers healthy and current.
|
||||
Maintenance owns observations, policies, schedules, requests, and verification of
|
||||
maintenance outcomes. Provisioning owns lifecycle execution, player checks,
|
||||
conflicts, rolling capacity, container recovery, and persistent operation history.
|
||||
Campaigns retain managed-file deployment evidence; Collections define lineups.
|
||||
|
||||
## Views
|
||||
|
||||
- **Overview** shows active work, attention, upcoming restarts, and Plutonium version
|
||||
coverage. Filter by capability or server. **Check now** refreshes health, drift,
|
||||
the production release, and running revisions.
|
||||
- **Policies & schedules** controls health recovery, configuration drift, automatic
|
||||
Plutonium updates, and the optional daily restart schedule independently.
|
||||
- **Activity** combines recovery findings, update results, and scheduled restarts.
|
||||
Follow a finding for evidence and attempts, or its Provisioning operation for
|
||||
execution progress. Cancel unsubmitted runtime restarts here.
|
||||
|
||||
Fleet and server details link their revision indicators to Maintenance. Failed
|
||||
runtime maintenance appears in Fleet attention, even when the underlying restart
|
||||
succeeded. Provisioning and repair Campaigns link back to the requesting work.
|
||||
|
||||
## Policies and pause
|
||||
|
||||
Health recovery and drift repair inherit the previous Watchdog enabled setting.
|
||||
Automatic Plutonium updates are **disabled by default**. Existing daily schedules
|
||||
retain their enabled state, timezone, and next occurrence.
|
||||
|
||||
**Pause all maintenance** holds new remediation, scheduled runs, and unsubmitted
|
||||
maintenance operations. Scans and result verification continue; submitted
|
||||
operations finish recovery. Campaigns, betas, and manually requested Provisioning
|
||||
operations continue independently. Resuming keeps each policy's saved setting.
|
||||
|
||||
Disabling health recovery or drift repair holds queued work for that capability
|
||||
and prevents further automatic attempts, including health escalation. Disabling
|
||||
automatic updates holds queued automatic update requests. Explicitly requested
|
||||
updates remain operator-authorized work. Disabling a daily schedule prevents
|
||||
future runs but does not cancel requests it already created.
|
||||
|
||||
All capabilities respect intentionally stopped servers. Start them explicitly
|
||||
through Provisioning before expecting maintenance to act on them.
|
||||
|
||||
## Plutonium updates
|
||||
|
||||
The published revision comes from the official production updater manifest.
|
||||
Running revisions come from RCON `version` after checking container readiness.
|
||||
Checks run hourly while the application is running. Failed release checks retain
|
||||
the cached release with an error. Unsupported replies, stale checks, unhealthy
|
||||
containers, and evidence from before a lifecycle submission remain unverified.
|
||||
A missing revision is never treated as current or as an automatic update target.
|
||||
|
||||
**Update on restart** requests ordinary Provisioning restarts for verified outdated
|
||||
servers. **Update outdated servers on restart** applies to the filtered list.
|
||||
Automatic updates, when enabled, apply to verified outdated servers across the
|
||||
fleet, with one attempt per server and published revision. Existing pending
|
||||
maintenance is reused. Occupied servers and servers with conflicting beta or
|
||||
campaign work wait without consuming a restart slot.
|
||||
|
||||
Plutainer's startup updater must be enabled (`PLUTAINER_AUTO_UPDATE`). Electric
|
||||
Cherry does not download binaries, pull a container image, or pin the runtime to
|
||||
a particular revision. The observed published revision is recorded as the minimum
|
||||
expected result; the actual revision is queried after recovery.
|
||||
|
||||
A healthy restart and a successful update are separate outcomes. If the server
|
||||
recovers on an older or unknown revision, the restart is successful in Provisioning
|
||||
and the update needs attention in Maintenance. Update verification never triggers
|
||||
a rebuild or an automatic restart loop. Inspect the updater configuration and
|
||||
server logs, then use **Retry after inspection** to create a new attempt. Ambiguous
|
||||
or failed lifecycle requests require inspection of the existing Provisioning
|
||||
operation and its retained recovery slot first.
|
||||
|
||||
Update attempts persist as suppression evidence across app restarts and history
|
||||
retention. Cancelling an automatic attempt suppresses another automatic attempt for
|
||||
that same revision; an operator can request an update explicitly.
|
||||
|
||||
## Scheduled restarts
|
||||
|
||||
Daily restarts are independent of update availability. At the saved queue time,
|
||||
Maintenance requests a restart for each server whose desired state is running.
|
||||
These requests have no revision requirement. Busy servers can restart later.
|
||||
|
||||
Saving a schedule uses the selected timezone and follows daylight saving changes.
|
||||
Skipped times run at the first valid minute after the gap; repeated times run once
|
||||
on their first occurrence. Legacy UTC schedules keep their timing until saved.
|
||||
Changing the browser display timezone does not change the saved schedule.
|
||||
|
||||
Schedules survive application restarts. Missed days, including while globally
|
||||
paused, become one run on resume. Pending work is deduplicated per server. The
|
||||
worker starts on the first application request, so keep the app running and
|
||||
reachable. Finished scheduled and legacy maintenance activity is retained for
|
||||
30 days; update attempt records remain durable.
|
||||
|
||||
## Health recovery and configuration drift
|
||||
|
||||
The health observer (the watchdog) scans once per minute. Three observations over
|
||||
at least two minutes are required before an automatic health restart; starting or
|
||||
restarting containers get five minutes. Unknown Komodo responses are observations,
|
||||
not proof of a stopped container. Repeated manual checks do not accelerate this
|
||||
window.
|
||||
|
||||
A health finding gets one restart, then one rebuild if recovery fails after five
|
||||
minutes. Escalation transfers only that finding's restart slot. A failed rebuild
|
||||
requires operator attention. **Rebuild deletes all instance files** and uses the
|
||||
latest promoted setup, the same server identity and port, and the saved server key.
|
||||
Missing keys block teardown. Submitted rebuild verification continues while paused.
|
||||
|
||||
Drift includes missing promoted files, changed bytes, and extra files in managed
|
||||
script, config, and plugin directories. Config comparison includes the composed
|
||||
baseline, overlays, and the selected map. Unmanaged default config is exempt until
|
||||
a baseline is promoted. Missing mounts are scan errors, not empty directories.
|
||||
|
||||
Drift repair creates a Campaign capturing observed differences and requests a
|
||||
Provisioning rebuild. The Campaign observes that rebuild without requesting
|
||||
another restart. Its detection evidence and the actual launch recipe remain
|
||||
available even if promotions change while the work waits. Histories survive
|
||||
instance removal. Active beta and campaign work defers new recovery.
|
||||
|
||||
**Dismiss** suppresses health/drift attempts until the condition clears. **Retry
|
||||
recovery** begins a fresh bounded attempt after inspection. Existing operations
|
||||
must finish or be cancelled first. Fresh verification resolves the finding.
|
||||
|
||||
## Compatibility and API
|
||||
|
||||
Existing job, incident, and operation IDs are retained. Upgrade migration adds
|
||||
request metadata without replaying commands; existing pending requests keep their
|
||||
original verification requirements. `/watchdog` and `/watchdog/:id` redirect to
|
||||
Maintenance, preserving query parameters. Old Watchdog and Plutonium APIs remain
|
||||
compatible. The previous Watchdog enabled switch controls health and drift only.
|
||||
|
||||
- `GET /api/maintenance`: policies, recovery findings, release/version coverage,
|
||||
schedule, and recent runtime activity with linked operation stages.
|
||||
- `POST /api/maintenance`: actions `scan`, `pause` (`paused` boolean), `policy`
|
||||
(`policy`: `health`, `drift`, or `updates`; `enabled` boolean), `update` (optional
|
||||
`servers` array), `schedule` (`enabled`, `time`, `timeZone`), `cancel` or `retry`
|
||||
(`id`, the numeric runtime maintenance activity ID).
|
||||
- `/api/maintenance/:id`: GET recovery evidence and attempts; POST `retry` or
|
||||
`dismiss` for a health/drift finding.
|
||||
@@ -49,11 +49,11 @@ Provisioning and observe the resulting operation by ID. Their pages link to that
|
||||
operation, and its history links back to the requesting work. Provisioning owns
|
||||
the empty-server check, conflict handling, restart capacity, remote command, and
|
||||
health verification. The callers retain file deployment, beta evidence, and
|
||||
Plutonium revision verification. Jobs remain visible while waiting for players,
|
||||
maintenance outcome verification. Jobs remain visible while waiting for players,
|
||||
other work, or integration configuration.
|
||||
|
||||
[Watchdog](watchdog.md) also submits recovery operations and links each attempt
|
||||
to its incident. A failed Watchdog restart can hand its reserved slot to a rebuild
|
||||
[Maintenance](maintenance.md) also submits recovery operations and links each attempt
|
||||
to its incident. A failed Maintenance restart can hand its reserved slot to a rebuild
|
||||
after five minutes; unrelated recovery slots remain held. Drift campaigns follow
|
||||
their linked rebuild without requesting another restart. Their file evidence and
|
||||
history remain available after the instance is removed.
|
||||
|
||||
+5
-78
@@ -1,80 +1,7 @@
|
||||
# Watchdog
|
||||
# Watchdog has moved to Maintenance
|
||||
|
||||
**Operations → Watchdog** detects unhealthy servers and drift, records each
|
||||
incident, and links its recovery attempts to Provisioning. File drift also opens
|
||||
a Campaign with the observed file differences. Automation is enabled by default;
|
||||
**Pause automation** persists across application restarts. Scanning and verification
|
||||
continue while paused, but new actions and queued Watchdog actions are held.
|
||||
Already submitted actions continue to completion.
|
||||
Open **Operations → Maintenance** for health recovery and drift repair, together
|
||||
with Plutonium updates and scheduled restarts. Existing Watchdog URLs redirect
|
||||
to Maintenance; incidents, operation IDs, and history are preserved.
|
||||
|
||||
The worker scans once per minute while Electric Cherry is running. **Scan now**
|
||||
requests a scan immediately. Repeated button presses do not count as independent
|
||||
health failures. A health incident must have three observations spanning at least
|
||||
two minutes before an automatic restart is queued. Starting or restarting
|
||||
containers get five minutes before recovery is requested. Unknown Komodo responses
|
||||
are recorded without interpreting them as a stopped container.
|
||||
|
||||
## Recovery
|
||||
|
||||
A health incident gets one restart, then one rebuild if the restart has not
|
||||
recovered within five minutes. Both are normal Provisioning jobs with durable
|
||||
history and shared capacity limits. Escalation transfers only that incident's
|
||||
restart recovery slot to the rebuild. An unsuccessful rebuild requires operator
|
||||
attention; Watchdog does not start an endless restart/rebuild loop. Verification
|
||||
of an already submitted rebuild continues, including after the incident is marked
|
||||
as needing attention.
|
||||
|
||||
An intentional **Stop** is retained as the server's desired state. Watchdog leaves
|
||||
it stopped until an explicit Start, Restart, or Rebuild request changes that intent.
|
||||
Active creations, removals, betas, and campaigns defer new Watchdog actions.
|
||||
|
||||
Provisioning checks current container health before using game status. A disruptive
|
||||
action requires a confirmed stopped container or a fresh zero-player reading from
|
||||
RCON or accepted telemetry. A positive player count from either source prevents
|
||||
an interruption. Telemetry evidence must have been received within 45 seconds;
|
||||
future, malformed, stale, and pre-restart reports cannot verify a new process.
|
||||
A game-end event retains its reported roster; it does not imply zero players.
|
||||
Fresh telemetry can also verify game activity and the selected engine map after
|
||||
Komodo reports the container healthy. A healthy container with neither current
|
||||
RCON status nor current telemetry remains queued for player-count verification.
|
||||
|
||||
## Drift
|
||||
|
||||
Drift includes missing promoted files, changed bytes, and extra files in managed
|
||||
script, config, and plugin destinations. Config comparison uses the composed
|
||||
baseline and overlays with the server's selected map. Plutainer's unmanaged default
|
||||
config remains exempt until a baseline is promoted. Active betas are deferred;
|
||||
missing server mounts are reported as scan errors instead of being treated as
|
||||
empty directories.
|
||||
|
||||
Watchdog rebuilds a drifting server in place using the same identity, map, port,
|
||||
and saved server key. **Rebuild deletes all old instance files.** The new instance
|
||||
uses the latest promoted setup when provisioning starts. A missing saved key
|
||||
leaves the incident needing attention without deleting the old server.
|
||||
|
||||
The linked Campaign captures before/after evidence at detection time. Its activity
|
||||
identifies this as the detected plan, and it follows the rebuild's outcome without
|
||||
queuing an additional restart. Promotions can change while the rebuild is queued;
|
||||
the Provisioning launch recipe records the files actually selected for that launch.
|
||||
Campaign evidence and Watchdog history survive subsequent rebuilds and removals.
|
||||
|
||||
## Review and retry
|
||||
|
||||
The list can be filtered by server, incident type, and status. Each detail page
|
||||
shows the triggering evidence, timestamped events, and links to every attempt.
|
||||
**Dismiss** suppresses further automatic attempts until the condition clears.
|
||||
**Retry** explicitly starts a new bounded attempt after inspecting a dismissed or
|
||||
failed incident. Active Provisioning operations must finish or be cancelled first.
|
||||
A verified recovery resolves the incident automatically.
|
||||
|
||||
## API
|
||||
|
||||
- `GET /api/watchdog`: automation setting, last scan, scan errors, and latest incidents.
|
||||
- `POST /api/watchdog` with `{ "action": "scan" }`: run a scan now.
|
||||
- `POST /api/watchdog` with `{ "enabled": false }` (or `true`): pause/resume automation.
|
||||
- `GET /api/watchdog/:id`: incident, event history, and linked actions.
|
||||
- `POST /api/watchdog/:id` with `{ "action": "retry" }` or `{ "action": "dismiss" }`.
|
||||
|
||||
The legacy `/api/drift` GET and dry-run POST still return previews. Mutating POST
|
||||
requests return HTTP 410 and direct clients to Watchdog; file repair is now a
|
||||
provisioning rebuild instead of an immediate rewrite.
|
||||
See [Maintenance](maintenance.md) for policies, recovery behavior, and the API.
|
||||
|
||||
@@ -7,6 +7,7 @@ import type { Health, RosterPlayer } from './ui/stores/servers.js';
|
||||
import { mapTitle, serverTitle } from './ui/server-labels.js';
|
||||
|
||||
export interface FleetServer {
|
||||
plutonium?: import('./plutonium.js').PlutoniumVersion;
|
||||
container_state?: string | null;
|
||||
container_checked_at?: string | null;
|
||||
last_checked_at?: string | null;
|
||||
@@ -37,7 +38,7 @@ export interface FleetOperation {
|
||||
key: string;
|
||||
group: string;
|
||||
server: string;
|
||||
kind: 'campaign' | 'deployment' | 'beta' | 'recovery';
|
||||
kind: 'campaign' | 'deployment' | 'beta' | 'recovery' | 'maintenance';
|
||||
label: string;
|
||||
detail: string;
|
||||
href: string;
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
import type { WatchdogOverview } from './watchdog.js';
|
||||
import type { PlutoniumStatus } from './plutonium.js';
|
||||
|
||||
export type MaintenancePolicy = 'health' | 'drift' | 'updates';
|
||||
export interface MaintenancePolicies { paused: boolean; health: boolean; drift: boolean; updates: boolean }
|
||||
export interface MaintenanceOverview {
|
||||
policies: MaintenancePolicies;
|
||||
recovery: WatchdogOverview;
|
||||
runtime: PlutoniumStatus;
|
||||
}
|
||||
@@ -16,6 +16,8 @@ export interface PlutoniumVersion {
|
||||
revision: number | null;
|
||||
checked_at: string | null;
|
||||
error: string | null;
|
||||
stopped?: boolean;
|
||||
update_status?: 'outdated' | 'current' | 'ahead' | 'unknown' | 'stopped';
|
||||
}
|
||||
export interface MaintenanceJob {
|
||||
id: number;
|
||||
@@ -27,6 +29,10 @@ export interface MaintenanceJob {
|
||||
provisioning_id: string | null;
|
||||
finished_at: string | null;
|
||||
detail: string | null;
|
||||
kind: 'legacy' | 'update' | 'schedule';
|
||||
origin: 'manual' | 'automatic' | 'schedule';
|
||||
operation_stage?: string | null;
|
||||
operation_state?: string | null;
|
||||
}
|
||||
export interface PlutoniumStatus {
|
||||
release: PlutoniumRelease;
|
||||
|
||||
@@ -310,6 +310,10 @@ export function db(): Database.Database {
|
||||
for (const table of ['deployment_servers', 'beta_servers', 'maintenance_jobs']) {
|
||||
migrateTable(handle, table, [{ name: 'provisioning_id', ddl: `ALTER TABLE ${table} ADD COLUMN provisioning_id TEXT` }]);
|
||||
}
|
||||
migrateTable(handle, 'maintenance_jobs', [
|
||||
{ name: 'kind', ddl: "ALTER TABLE maintenance_jobs ADD COLUMN kind TEXT NOT NULL DEFAULT 'legacy'" },
|
||||
{ name: 'origin', ddl: "ALTER TABLE maintenance_jobs ADD COLUMN origin TEXT NOT NULL DEFAULT 'manual'" }
|
||||
]);
|
||||
migrateServers(handle);
|
||||
backfillDesiredState(handle);
|
||||
// Column additions must run BEFORE any statement that references the
|
||||
|
||||
@@ -64,6 +64,15 @@ export function fleetOperations(now = Date.now()): FleetOperation[] {
|
||||
state: row.stage === 'failed' ? 'attention' : 'active', label: stages[row.stage] ?? row.stage,
|
||||
detail: row.error ?? recovering.get(row.server)?.error ?? `Beta ${row.id}`, href: `/betas/${row.id}` });
|
||||
}
|
||||
const maintenance = handle.prepare(`SELECT m.* FROM maintenance_jobs m WHERE m.state = 'failed'
|
||||
AND NOT EXISTS (SELECT 1 FROM maintenance_jobs n WHERE n.server = m.server AND n.kind = m.kind AND n.id > m.id)`)
|
||||
.all() as { id: number; server: string; kind: string; detail: string | null; requested_at: string }[];
|
||||
for (const job of maintenance) {
|
||||
operations.push({ key: `maintenance:${job.id}`, group: `maintenance:${job.id}`, kind: 'maintenance',
|
||||
server: job.server, state: 'attention', startedAt: job.requested_at,
|
||||
label: job.kind === 'update' ? 'Update verification needs attention' : 'Maintenance needs attention',
|
||||
detail: job.detail ?? 'Inspect the maintenance result', href: `/maintenance?view=activity&server=${encodeURIComponent(job.server)}#job-${job.id}` });
|
||||
}
|
||||
for (const slot of slots) {
|
||||
if (representedRecovery.has(slot.server)) continue;
|
||||
operations.push({ key: `recovery:${slot.server}:${slot.requested_at}`, group: `recovery:${slot.server}`,
|
||||
|
||||
@@ -1,156 +1,2 @@
|
||||
import { db } from '../db/index.js';
|
||||
import { config } from '../config.js';
|
||||
import { getServer, listServers } from './servers.js';
|
||||
import { rconPlutoniumVersion } from '../integrations/rcon.js';
|
||||
import { requestRestart, cancelProvisioning, restartRecoveryPending } from '../provisioning/index.js';
|
||||
import { nextMaintenanceAt, parsePlutoniumManifest, PLUTONIUM_MANIFEST_URL } from '../../plutonium.js';
|
||||
import type { MaintenanceJob, MaintenanceSchedule, PlutoniumRelease, PlutoniumStatus, PlutoniumVersion } from '../../plutonium.js';
|
||||
|
||||
const HOUR = 60 * 60 * 1000;
|
||||
const nowIso = () => new Date().toISOString();
|
||||
function readSetting<T>(key: string, fallback: T): T {
|
||||
const row = db().prepare('SELECT value FROM settings WHERE key = ?').get(key) as { value: string } | undefined;
|
||||
return row ? JSON.parse(row.value) : fallback;
|
||||
}
|
||||
function writeSetting(key: string, value: unknown): void {
|
||||
db().prepare('INSERT INTO settings (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value')
|
||||
.run(key, JSON.stringify(value));
|
||||
}
|
||||
function release(): PlutoniumRelease {
|
||||
return readSetting('plutonium_release', { revision: null, checkedAt: null, error: null });
|
||||
}
|
||||
function schedule(): MaintenanceSchedule {
|
||||
// Legacy schedules were explicitly UTC. Preserve their timing until saved.
|
||||
return { timeZone: 'UTC', ...readSetting('plutonium_schedule', { enabled: false, time: '06:00', nextAt: null }) };
|
||||
}
|
||||
export function saveMaintenanceSchedule(enabled: boolean, time: string, timeZone: string): void {
|
||||
const nextAt = nextMaintenanceAt(time, new Date(), timeZone);
|
||||
writeSetting('plutonium_schedule', { enabled, time, timeZone, nextAt: enabled ? nextAt : null });
|
||||
}
|
||||
export function plutoniumStatus(): PlutoniumStatus {
|
||||
const versions = db().prepare('SELECT * FROM plutonium_versions').all() as PlutoniumVersion[];
|
||||
return {
|
||||
release: release(), schedule: schedule(),
|
||||
servers: listServers().map(s => versions.find(v => v.server === s.name)
|
||||
?? { server: s.name, revision: null, checked_at: null, error: null }),
|
||||
jobs: db().prepare(`SELECT * FROM maintenance_jobs WHERE state IN ('queued','restarting')
|
||||
OR id IN (SELECT id FROM maintenance_jobs ORDER BY id DESC LIMIT 100) ORDER BY id DESC`).all() as MaintenanceJob[]
|
||||
};
|
||||
}
|
||||
|
||||
async function probeVersion(server: string): Promise<number | null> {
|
||||
const recovering = restartRecoveryPending(server);
|
||||
const result = recovering ? { revision: null, error: 'Waiting for restart recovery.' }
|
||||
: await rconPlutoniumVersion(server).catch(() => ({ revision: null, error: 'Version check failed.' }));
|
||||
// Clear the old revision on failure: it is no longer evidence of what is running.
|
||||
db().prepare(`INSERT INTO plutonium_versions (server, revision, checked_at, error) VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(server) DO UPDATE SET revision = excluded.revision, checked_at = excluded.checked_at, error = excluded.error`)
|
||||
.run(server, result.revision, nowIso(), result.error);
|
||||
return result.revision;
|
||||
}
|
||||
|
||||
let checking: Promise<void> | null = null;
|
||||
export function checkPlutonium(): Promise<void> {
|
||||
if (checking) return checking;
|
||||
checking = check().finally(() => { checking = null; });
|
||||
return checking;
|
||||
}
|
||||
async function check(): Promise<void> {
|
||||
const previous = release();
|
||||
try {
|
||||
const response = await fetch(PLUTONIUM_MANIFEST_URL, { signal: AbortSignal.timeout(config.httpTimeoutMs), redirect: 'error' });
|
||||
if (!response.ok) throw new Error(`Updater manifest request failed (HTTP ${response.status}).`);
|
||||
const body = await response.text();
|
||||
if (body.length > 2_000_000) throw new Error('Updater manifest response was too large.');
|
||||
writeSetting('plutonium_release', { revision: parsePlutoniumManifest(JSON.parse(body)), checkedAt: nowIso(), error: null });
|
||||
} catch (err) {
|
||||
writeSetting('plutonium_release', { ...previous, checkedAt: nowIso(), error: (err as Error).message });
|
||||
}
|
||||
const servers = listServers();
|
||||
for (let i = 0; i < servers.length; i += 4) {
|
||||
await Promise.all(servers.slice(i, i + 4).map(s => probeVersion(s.name)));
|
||||
}
|
||||
}
|
||||
|
||||
export function queueMaintenance(servers: string[]): number {
|
||||
const names = new Set(listServers().map(s => s.name));
|
||||
if (!servers.length || servers.some(s => !names.has(s))) throw new Error('Choose existing fleet servers.');
|
||||
return db().transaction(() => {
|
||||
const latest = release();
|
||||
// An old cached release is still displayed, but never used as a fresh target.
|
||||
const target = !latest.error && latest.checkedAt && Date.now() - Date.parse(latest.checkedAt) < HOUR
|
||||
? latest.revision : null;
|
||||
let added = 0;
|
||||
for (const server of new Set(servers)) {
|
||||
added += db().prepare(`INSERT OR IGNORE INTO maintenance_jobs (server, target_revision, requested_at)
|
||||
VALUES (?, ?, ?)`).run(server, target, nowIso()).changes;
|
||||
const job = db().prepare("SELECT id FROM maintenance_jobs WHERE server = ? AND state IN ('queued','restarting')").get(server) as { id: number };
|
||||
requestRestart(server, { kind: 'maintenance', id: String(job.id) });
|
||||
}
|
||||
return added;
|
||||
})();
|
||||
}
|
||||
export function cancelMaintenance(id: number): boolean {
|
||||
return db().transaction(() => {
|
||||
const job = db().prepare("SELECT provisioning_id FROM maintenance_jobs WHERE id = ? AND state = 'queued'").get(id) as { provisioning_id: string | null } | undefined;
|
||||
if (!job) return false;
|
||||
if (job.provisioning_id && !cancelProvisioning(job.provisioning_id)) return false;
|
||||
return db().prepare(`UPDATE maintenance_jobs SET state = 'cancelled', finished_at = ?, detail = 'Cancelled by operator'
|
||||
WHERE id = ? AND state = 'queued'`).run(nowIso(), id).changes > 0;
|
||||
})();
|
||||
}
|
||||
|
||||
function finish(job: MaintenanceJob, state: 'done' | 'failed' | 'cancelled', detail: string) {
|
||||
db().prepare('UPDATE maintenance_jobs SET state = ?, detail = ?, finished_at = ? WHERE id = ?')
|
||||
.run(state, detail, nowIso(), job.id);
|
||||
}
|
||||
function hold(job: MaintenanceJob, detail: string) {
|
||||
db().prepare('UPDATE maintenance_jobs SET detail = ? WHERE id = ?').run(detail, job.id);
|
||||
}
|
||||
|
||||
let ticking: Promise<void> | null = null;
|
||||
export function maintenanceTick(): Promise<void> {
|
||||
if (ticking) return ticking;
|
||||
ticking = runTick().finally(() => { ticking = null; });
|
||||
return ticking;
|
||||
}
|
||||
async function runTick(): Promise<void> {
|
||||
const latest = release();
|
||||
if (!latest.checkedAt || Date.now() - Date.parse(latest.checkedAt) >= HOUR) await checkPlutonium();
|
||||
db().transaction(() => {
|
||||
const plan = schedule();
|
||||
if (plan.enabled && plan.nextAt && plan.nextAt <= nowIso()) {
|
||||
const names = listServers().map(s => s.name);
|
||||
if (names.length) queueMaintenance(names);
|
||||
// Coalesce missed days after downtime into one run; no restart backlog.
|
||||
writeSetting('plutonium_schedule', { ...plan, nextAt: nextMaintenanceAt(plan.time, new Date(), plan.timeZone) });
|
||||
}
|
||||
})();
|
||||
const jobs = db().prepare(`SELECT * FROM maintenance_jobs WHERE state IN ('queued','restarting') ORDER BY id`).all() as MaintenanceJob[];
|
||||
for (const job of jobs) {
|
||||
try { await processJob(job); }
|
||||
catch { hold(job, 'Maintenance check failed; waiting for the next check.'); }
|
||||
}
|
||||
db().prepare(`DELETE FROM maintenance_jobs WHERE finished_at < ? AND state IN ('done','failed','cancelled')`)
|
||||
.run(new Date(Date.now() - 30 * 24 * HOUR).toISOString());
|
||||
}
|
||||
async function processJob(job: MaintenanceJob): Promise<void> {
|
||||
const server = getServer(job.server);
|
||||
if (!server || server.destroyed_at) return finish(job, 'cancelled', 'Server removed.');
|
||||
const operation = requestRestart(job.server, { kind: 'maintenance', id: String(job.id) });
|
||||
if (['failed','interrupted','cancelled'].includes(operation.state)) {
|
||||
return finish(job, operation.state === 'cancelled' ? 'cancelled' : 'failed', operation.error ?? operation.stage);
|
||||
}
|
||||
if (operation.state !== 'done') {
|
||||
if (operation.submitted_at) {
|
||||
db().prepare("UPDATE maintenance_jobs SET state = 'restarting', restarted_at = ? WHERE id = ?").run(operation.submitted_at, job.id);
|
||||
db().prepare('UPDATE plutonium_versions SET revision = NULL, error = ? WHERE server = ?').run('Waiting for restart recovery.', job.server);
|
||||
}
|
||||
return hold(job, operation.stage);
|
||||
}
|
||||
const revision = await probeVersion(job.server);
|
||||
if (job.target_revision && (revision === null || revision < job.target_revision)) {
|
||||
return finish(job, 'failed', `Server recovered, but r${job.target_revision} was not verified (${revision === null ? 'version unknown' : `running r${revision}`}). Check Plutainer auto-update settings.`);
|
||||
}
|
||||
return finish(job, 'done', `Server recovered; ${revision === null ? 'running revision unknown' : `running r${revision}`}.`);
|
||||
}
|
||||
// Compatibility for existing jobs and API clients.
|
||||
export * from "../maintenance/runtime.js";
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
import { db } from '../db/index.js';
|
||||
import { config } from '../config.js';
|
||||
import { getServer, listServers } from '../fleet/servers.js';
|
||||
import { latestGameSubmission } from '../fleet/status-evidence.js';
|
||||
import { rconPlutoniumVersion } from './rcon.js';
|
||||
import { komodoInspectHealthy } from './komodo.js';
|
||||
import { restartRecoveryPending } from '../provisioning/index.js';
|
||||
import { parsePlutoniumManifest, PLUTONIUM_MANIFEST_URL } from '../../plutonium.js';
|
||||
import type { PlutoniumRelease, PlutoniumVersion } from '../../plutonium.js';
|
||||
import { readSetting, writeSetting } from '../maintenance/settings.js';
|
||||
export const HOUR = 60 * 60 * 1000;
|
||||
const nowIso = () => new Date().toISOString();
|
||||
export function release(): PlutoniumRelease {
|
||||
return readSetting('plutonium_release', { revision: null, checkedAt: null, error: null });
|
||||
}
|
||||
export function freshRelease(): number | null {
|
||||
const latest = release();
|
||||
const age = latest.checkedAt ? Date.now() - Date.parse(latest.checkedAt) : NaN;
|
||||
return !latest.error && age >= 0 && age < HOUR ? latest.revision : null;
|
||||
}
|
||||
export function runningVersions(): PlutoniumVersion[] {
|
||||
const versions = db().prepare('SELECT * FROM plutonium_versions').all() as PlutoniumVersion[];
|
||||
const target = freshRelease();
|
||||
return listServers().map(server => {
|
||||
const value = versions.find(v => v.server === server.name)
|
||||
?? { server: server.name, revision: null, checked_at: null, error: null };
|
||||
const age = value.checked_at ? Date.now() - Date.parse(value.checked_at) : NaN;
|
||||
const epoch = latestGameSubmission(server.name);
|
||||
const stopped = server.desired_state === 'stopped';
|
||||
const valid = !value.error && age >= 0 && age < HOUR && (!epoch || value.checked_at! > epoch) && !restartRecoveryPending(server.name);
|
||||
const revision = valid && !stopped ? value.revision : null;
|
||||
return { ...value, revision, stopped,
|
||||
update_status: stopped ? 'stopped' : revision === null || target === null ? 'unknown'
|
||||
: revision < target ? 'outdated' : revision > target ? 'ahead' : 'current' };
|
||||
});
|
||||
}
|
||||
export async function probeVersion(server: string): Promise<number | null> {
|
||||
const epoch = latestGameSubmission(server);
|
||||
const recovering = restartRecoveryPending(server);
|
||||
const health = !recovering && getServer(server)?.desired_state !== 'stopped'
|
||||
? await komodoInspectHealthy(server).catch(() => ({ ok: false })) : { ok: false };
|
||||
let result = !health.ok ? { revision: null, error: 'Waiting for a healthy running server.' }
|
||||
: await rconPlutoniumVersion(server).catch(() => ({ revision: null, error: 'Version check failed.' }));
|
||||
if (!getServer(server)) return null;
|
||||
if (epoch !== latestGameSubmission(server) || restartRecoveryPending(server)) {
|
||||
result = { revision: null, error: 'Waiting for restart recovery.' };
|
||||
}
|
||||
// Clear the old revision on failure: it is no longer evidence of what is running.
|
||||
db().prepare(`INSERT INTO plutonium_versions (server, revision, checked_at, error) VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(server) DO UPDATE SET revision = excluded.revision, checked_at = excluded.checked_at, error = excluded.error`)
|
||||
.run(server, result.revision, nowIso(), result.error);
|
||||
return result.revision;
|
||||
}
|
||||
|
||||
let checking: Promise<void> | null = null;
|
||||
export function checkPlutonium(): Promise<void> {
|
||||
if (checking) return checking;
|
||||
checking = check().finally(() => { checking = null; });
|
||||
return checking;
|
||||
}
|
||||
async function check(): Promise<void> {
|
||||
const previous = release();
|
||||
try {
|
||||
const response = await fetch(PLUTONIUM_MANIFEST_URL, { signal: AbortSignal.timeout(config.httpTimeoutMs), redirect: 'error' });
|
||||
if (!response.ok) throw new Error(`Updater manifest request failed (HTTP ${response.status}).`);
|
||||
const body = await response.text();
|
||||
if (body.length > 2_000_000) throw new Error('Updater manifest response was too large.');
|
||||
writeSetting('plutonium_release', { revision: parsePlutoniumManifest(JSON.parse(body)), checkedAt: nowIso(), error: null });
|
||||
} catch (err) {
|
||||
writeSetting('plutonium_release', { ...previous, checkedAt: nowIso(), error: (err as Error).message });
|
||||
}
|
||||
const servers = listServers();
|
||||
for (let i = 0; i < servers.length; i += 4) {
|
||||
await Promise.all(servers.slice(i, i + 4).map(s => probeVersion(s.name)));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
import { maintenancePolicies } from './policies.js';
|
||||
import { watchdogOverview, watchdogTick } from './recovery.js';
|
||||
import { maintenanceTick, plutoniumStatus } from './runtime.js';
|
||||
import type { MaintenanceOverview } from '../../maintenance.js';
|
||||
|
||||
export function maintenanceOverview(): MaintenanceOverview {
|
||||
return { policies: maintenancePolicies(), recovery: watchdogOverview(), runtime: plutoniumStatus() };
|
||||
}
|
||||
export async function runMaintenance(): Promise<void> {
|
||||
// One failing observer must not prevent the other capabilities from reconciling.
|
||||
const results = await Promise.allSettled([maintenanceTick(), watchdogTick()]);
|
||||
for (const result of results) if (result.status === 'rejected') console.error('[maintenance]', result.reason);
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
import { db } from '../db/index.js';
|
||||
import type { MaintenancePolicy, MaintenancePolicies } from '../../maintenance.js';
|
||||
|
||||
function setting(key: string): string | undefined {
|
||||
return (db().prepare('SELECT value FROM settings WHERE key = ?').get(key) as { value: string } | undefined)?.value;
|
||||
}
|
||||
function save(key: string, value: boolean) {
|
||||
db().prepare('INSERT INTO settings(key,value) VALUES (?,?) ON CONFLICT(key) DO UPDATE SET value=excluded.value').run(key, value ? '1' : '0');
|
||||
}
|
||||
export function maintenancePolicies(): MaintenancePolicies {
|
||||
// Preserve the old recovery switch without opting anyone into runtime upgrades.
|
||||
const legacy = setting('watchdog_enabled') !== '0';
|
||||
return {
|
||||
paused: setting('maintenance_paused') === '1',
|
||||
health: setting('maintenance_health') === undefined ? legacy : setting('maintenance_health') === '1',
|
||||
drift: setting('maintenance_drift') === undefined ? legacy : setting('maintenance_drift') === '1',
|
||||
updates: setting('maintenance_updates') === '1'
|
||||
};
|
||||
}
|
||||
export function maintenanceAllowed(policy?: MaintenancePolicy): boolean {
|
||||
const policies = maintenancePolicies();
|
||||
return !policies.paused && (!policy || policies[policy]);
|
||||
}
|
||||
export function setMaintenancePolicy(policy: MaintenancePolicy, enabled: boolean) { save(`maintenance_${policy}`, enabled); }
|
||||
export function pauseMaintenance(paused: boolean) { save('maintenance_paused', paused); }
|
||||
@@ -0,0 +1,242 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { db } from '../db/index.js';
|
||||
import { komodoConfigured } from '../config.js';
|
||||
import { getServer, listServers } from '../fleet/servers.js';
|
||||
import { komodoInspectHealthy } from '../integrations/komodo.js';
|
||||
import { rconStatusResult } from '../integrations/rcon.js';
|
||||
import { telemetryEvidence } from '../fleet/status-evidence.js';
|
||||
import { prepareDriftCampaign, createDriftRebuildCampaign, previewDrift } from '../fleet/drift.js';
|
||||
import { requestWatchdogLifecycle, escalateWatchdogRestart } from '../provisioning/requests.js';
|
||||
import type { WatchdogAction, WatchdogDetail, WatchdogEvidence, WatchdogIncident, WatchdogOverview, WatchdogState } from '../../watchdog.js';
|
||||
|
||||
import { maintenanceAllowed, setMaintenancePolicy } from './policies.js';
|
||||
|
||||
export const WATCHDOG_SCAN_MS = 60_000;
|
||||
export const WATCHDOG_FAILURE_MS = 120_000;
|
||||
export const WATCHDOG_RECOVERY_MS = 300_000;
|
||||
const iso = () => new Date().toISOString();
|
||||
const pendingStates = ['queued', 'waiting', 'running', 'awaiting-health'];
|
||||
const setting = (key: string) => (db().prepare('SELECT value FROM settings WHERE key = ?').get(key) as { value: string } | undefined)?.value ?? null;
|
||||
function setSetting(key: string, value: string) { db().prepare('INSERT INTO settings(key,value) VALUES (?,?) ON CONFLICT(key) DO UPDATE SET value=excluded.value').run(key, value); }
|
||||
export const watchdogEnabled = () => maintenanceAllowed('health') || maintenanceAllowed('drift');
|
||||
export function setWatchdogEnabled(enabled: boolean) {
|
||||
setSetting('watchdog_enabled', enabled ? '1' : '0');
|
||||
setMaintenancePolicy('health', enabled);
|
||||
setMaintenancePolicy('drift', enabled);
|
||||
}
|
||||
|
||||
function incident(id: string) { return db().prepare('SELECT * FROM watchdog_incidents WHERE id = ?').get(id) as WatchdogIncident | undefined; }
|
||||
function active(server: string, kind: 'health' | 'drift') { return db().prepare("SELECT * FROM watchdog_incidents WHERE server = ? AND kind = ? AND state != 'resolved'").get(server, kind) as WatchdogIncident | undefined; }
|
||||
function event(id: string, message: string) { db().prepare('INSERT INTO watchdog_events(incident_id,created_at,message) VALUES (?,?,?)').run(id, iso(), message); }
|
||||
function transition(id: string, state: WatchdogState, summary: string) {
|
||||
const previous = incident(id);
|
||||
if (!previous || (previous.state === state && previous.summary === summary)) return;
|
||||
db().prepare('UPDATE watchdog_incidents SET state=?, summary=?, finished_at=? WHERE id=?')
|
||||
.run(state, summary, state === 'resolved' || state === 'dismissed' ? iso() : null, id);
|
||||
event(id, summary);
|
||||
}
|
||||
function detect(server: string, kind: 'health' | 'drift', summary: string, details: WatchdogEvidence): WatchdogIncident {
|
||||
const previous = active(server, kind);
|
||||
if (previous) {
|
||||
// Repeated UI scans must not manufacture a sustained failure.
|
||||
const newSample = Date.now() - Date.parse(previous.last_seen_at) >= WATCHDOG_SCAN_MS;
|
||||
const actionable = (details as WatchdogEvidence & { actionable?: boolean }).actionable;
|
||||
const priorActionable = (JSON.parse(previous.details) as { actionable?: boolean }).actionable;
|
||||
const reset = kind === 'health' && actionable === true && priorActionable !== true;
|
||||
db().prepare('UPDATE watchdog_incidents SET details=?, last_seen_at=?, failures=?, first_seen_at=? WHERE id=?')
|
||||
.run(JSON.stringify(details), newSample || reset ? iso() : previous.last_seen_at,
|
||||
reset ? 1 : previous.failures + (newSample ? 1 : 0), reset ? iso() : previous.first_seen_at, previous.id);
|
||||
if (previous.state === 'open') transition(previous.id, 'open', summary);
|
||||
return incident(previous.id)!;
|
||||
}
|
||||
const id = randomUUID();
|
||||
db().prepare(`INSERT INTO watchdog_incidents(id,server,kind,state,summary,details,first_seen_at,last_seen_at)
|
||||
VALUES (?,?,?,'open',?,?,?,?)`).run(id, server, kind, summary, JSON.stringify(details), iso(), iso());
|
||||
event(id, summary);
|
||||
return incident(id)!;
|
||||
}
|
||||
function actions(id: string): WatchdogAction[] {
|
||||
return db().prepare(`SELECT a.*, d.state AS operation_state, d.stage AS operation_stage, d.error AS operation_error
|
||||
FROM watchdog_actions a LEFT JOIN server_deployments d ON d.id=a.operation_id
|
||||
WHERE a.incident_id=? ORDER BY a.rowid`).all(id) as WatchdogAction[];
|
||||
}
|
||||
export function watchdogOverview(): WatchdogOverview {
|
||||
return { enabled: watchdogEnabled(), lastScanAt: setting('watchdog_last_scan'), scanError: setting('watchdog_scan_error') || null,
|
||||
incidents: db().prepare(`SELECT i.*, (SELECT d.stage FROM watchdog_actions a
|
||||
JOIN server_deployments d ON d.id = a.operation_id WHERE a.incident_id = i.id AND a.cycle = i.cycle
|
||||
ORDER BY a.rowid DESC LIMIT 1) AS operation_stage
|
||||
FROM watchdog_incidents i WHERE state != 'resolved' OR id IN (
|
||||
SELECT id FROM watchdog_incidents WHERE state = 'resolved' ORDER BY first_seen_at DESC, rowid DESC LIMIT 100)
|
||||
ORDER BY CASE WHEN state='resolved' THEN 1 ELSE 0 END, first_seen_at DESC, rowid DESC`).all() as WatchdogIncident[] };
|
||||
}
|
||||
export function watchdogDetail(id: string): WatchdogDetail | null {
|
||||
const row = incident(id);
|
||||
return row ? { incident: row, actions: actions(id), events: db().prepare('SELECT id,created_at,message FROM watchdog_events WHERE incident_id=? ORDER BY id').all(id) as WatchdogDetail['events'] } : null;
|
||||
}
|
||||
function hasPending(server: string) {
|
||||
return !!db().prepare("SELECT 1 FROM server_deployments WHERE server=? AND state IN ('queued','waiting','running','awaiting-health') LIMIT 1").get(server);
|
||||
}
|
||||
function protectedWork(server: string): boolean {
|
||||
return !!db().prepare(`SELECT 1 FROM beta_servers bs JOIN betas b ON b.id=bs.beta_id WHERE bs.server=? AND b.state IN ('running','ready-promote') LIMIT 1`).get(server)
|
||||
|| !!db().prepare(`SELECT 1 FROM deployment_servers ds WHERE ds.server=? AND ds.restart_state IN ('pending','held','failed','manual')
|
||||
AND NOT EXISTS (SELECT 1 FROM watchdog_actions a WHERE a.campaign_id=ds.deployment_id) LIMIT 1`).get(server);
|
||||
}
|
||||
function addAction(row: WatchdogIncident, operationId: string, action: 'restart' | 'rebuild', campaignId: string | null = null) {
|
||||
db().prepare('INSERT INTO watchdog_actions(id,incident_id,action,operation_id,campaign_id,created_at,cycle) VALUES (?,?,?,?,?,?,?)')
|
||||
.run(randomUUID(), row.id, action, operationId, campaignId, iso(), row.cycle);
|
||||
transition(row.id, 'recovering', action === 'restart' ? 'Restart queued; waiting for safe recovery' : 'Rebuild queued with the latest promoted setup');
|
||||
event(row.id, `${action === 'restart' ? 'Restart' : 'Rebuild'} requested: ${operationId}${campaignId ? `; campaign ${campaignId}` : ''}`);
|
||||
}
|
||||
export function updateWatchdogIncident(id: string, action: 'retry' | 'dismiss') {
|
||||
return db().transaction(() => {
|
||||
const row = incident(id);
|
||||
if (!row) return { ok: false as const, status: 404, error: 'Watchdog incident not found.' };
|
||||
if (actions(id).some(a => pendingStates.includes(a.operation_state ?? ''))) return { ok: false as const, status: 409, error: 'Provisioning is still active. Complete or cancel that operation first.' };
|
||||
if (action === 'dismiss') {
|
||||
if (!['open','attention'].includes(row.state)) return { ok: false as const, status: 409, error: 'Only detected incidents or incidents needing attention can be dismissed.' };
|
||||
transition(id, 'dismissed', 'Dismissed by operator; automatic attempts remain suppressed until the condition clears');
|
||||
} else {
|
||||
if (!['attention','dismissed'].includes(row.state)) return { ok: false as const, status: 409, error: 'Only incidents needing attention or dismissed incidents can be retried.' };
|
||||
if (!getServer(row.server)) return { ok: false as const, status: 409, error: 'The server is no longer registered; inspect its provisioning history.' };
|
||||
db().prepare('UPDATE watchdog_incidents SET cycle=cycle+1, failures=1, first_seen_at=?, last_seen_at=? WHERE id=?').run(iso(), iso(), id);
|
||||
transition(id, 'open', 'Operator requested a fresh recovery attempt; checking current conditions');
|
||||
}
|
||||
return { ok: true as const };
|
||||
})();
|
||||
}
|
||||
|
||||
async function reconcileActions(): Promise<void> {
|
||||
const rows = db().prepare("SELECT * FROM watchdog_incidents WHERE state IN ('recovering','attention')").all() as WatchdogIncident[];
|
||||
for (const row of rows) {
|
||||
const action = actions(row.id).filter(a => a.cycle === row.cycle).at(-1);
|
||||
if (!action) continue;
|
||||
const operation = db().prepare('SELECT state,submitted_at,started_at,stage,error FROM server_deployments WHERE id=?').get(action.operation_id) as
|
||||
{ state: string; submitted_at: string | null; started_at: string; stage: string; error: string | null } | undefined;
|
||||
if (!operation) { transition(row.id, 'attention', 'Provisioning history is missing; inspect the server before retrying'); continue; }
|
||||
if (['done','warning'].includes(operation.state)) {
|
||||
if (row.kind === 'drift') {
|
||||
try {
|
||||
const plan = await previewDrift([row.server]);
|
||||
if (incident(row.id)?.cycle !== row.cycle) continue;
|
||||
if (plan.files.length) { transition(row.id, 'attention', 'Rebuild completed but drift remains; inspect the campaign and server files'); continue; }
|
||||
} catch (err) { transition(row.id, 'attention', `Cannot verify rebuilt files: ${(err as Error).message}`); continue; }
|
||||
}
|
||||
transition(row.id, 'resolved', action.action === 'restart' ? 'Server recovered after restart' : 'Server recovered after rebuild');
|
||||
continue;
|
||||
}
|
||||
if (row.state === 'attention') continue; // Exhausted attempts stay latched, including across app restarts.
|
||||
if (['cancelled','interrupted'].includes(operation.state)) {
|
||||
transition(row.id, 'attention', `Provisioning ${operation.state}; inspect the recorded operation before retrying`); continue;
|
||||
}
|
||||
if (action.action === 'rebuild') {
|
||||
if (operation.state === 'failed') transition(row.id, 'attention', `Rebuild failed: ${operation.error ?? operation.stage}`);
|
||||
else if (operation.state === 'awaiting-health' && Date.now() - Date.parse(operation.submitted_at ?? operation.started_at) >= WATCHDOG_RECOVERY_MS)
|
||||
transition(row.id, 'attention', 'Rebuild has not recovered within five minutes; provisioning continues verification');
|
||||
continue;
|
||||
}
|
||||
if (!['awaiting-health','failed'].includes(operation.state)) continue;
|
||||
if (Date.now() - Date.parse(operation.submitted_at ?? operation.started_at) < WATCHDOG_RECOVERY_MS) continue;
|
||||
if (!maintenanceAllowed(row.kind)) continue;
|
||||
const current = getServer(row.server);
|
||||
if (!current || current.desired_state === 'stopped' || protectedWork(row.server)) continue;
|
||||
const result = db().transaction(() => {
|
||||
const next = escalateWatchdogRestart(row.id, action.operation_id, WATCHDOG_RECOVERY_MS);
|
||||
if (next.ok) addAction(row, next.deploymentId, 'rebuild');
|
||||
return next;
|
||||
})();
|
||||
if (!result.ok && result.status !== 409 && result.status !== 503) transition(row.id, 'attention', `Could not queue rebuild: ${result.error}`);
|
||||
}
|
||||
}
|
||||
|
||||
interface Observation { healthy: boolean; actionable: boolean; reason: string; containerState: string | null }
|
||||
async function observe(server: string): Promise<Observation> {
|
||||
const health = await komodoInspectHealthy(server);
|
||||
const state = health.state?.toLowerCase() ?? null;
|
||||
if (!health.ok) return { healthy: false, actionable: !!state && ['stopped','exited','created','dead','unhealthy','restarting','starting'].includes(state), reason: health.reason, containerState: state };
|
||||
if (telemetryEvidence(server)) return { healthy: true, actionable: false, reason: 'Komodo healthy with fresh telemetry', containerState: state };
|
||||
const rcon = await rconStatusResult(server);
|
||||
if (rcon.ok || telemetryEvidence(server)) return { healthy: true, actionable: false, reason: 'Komodo healthy with fresh game status', containerState: state };
|
||||
return { healthy: false, actionable: true, reason: `Game status unavailable: ${rcon.reason}; waiting for player-count verification before recovery`, containerState: state };
|
||||
}
|
||||
async function scanServer(server: string): Promise<void> {
|
||||
const row = getServer(server);
|
||||
if (!row || row.desired_state === 'stopped') return;
|
||||
if (hasPending(server) || protectedWork(server)) return;
|
||||
const observation = await observe(server);
|
||||
// Probes can overlap an API request; recheck ownership before queuing anything.
|
||||
if (hasPending(server) || protectedWork(server) || getServer(server)?.desired_state === 'stopped') return;
|
||||
if (observation.healthy) {
|
||||
const current = active(server, 'health');
|
||||
if (current) transition(current.id, 'resolved', 'Fresh game status and Komodo health verified');
|
||||
} else {
|
||||
const problem = detect(server, 'health', observation.reason, observation);
|
||||
const startup = ['starting','restarting'].includes(observation.containerState ?? '');
|
||||
const oldEnough = Date.now() - Date.parse(problem.first_seen_at) >= (startup ? WATCHDOG_RECOVERY_MS : WATCHDOG_FAILURE_MS);
|
||||
if (problem.state === 'open' && observation.actionable && oldEnough && problem.failures >= 3 && maintenanceAllowed('health')) {
|
||||
const result = db().transaction(() => {
|
||||
const next = requestWatchdogLifecycle(server, 'restart', problem.id);
|
||||
if (next.ok) addAction(problem, next.deploymentId, 'restart');
|
||||
return next;
|
||||
})();
|
||||
if (!result.ok && result.status !== 409 && result.status !== 503) transition(problem.id, 'attention', result.error);
|
||||
}
|
||||
// Unknown container health is an observation failure, not proof that a rebuild is safe.
|
||||
if (!observation.actionable || hasPending(server)) return;
|
||||
}
|
||||
const prepared = await prepareDriftCampaign(server);
|
||||
if (hasPending(server) || protectedWork(server) || getServer(server)?.desired_state === 'stopped') return;
|
||||
if (!prepared.plan.files.length) {
|
||||
const current = active(server, 'drift');
|
||||
if (current) transition(current.id, 'resolved', 'Server files match the promoted setup');
|
||||
return;
|
||||
}
|
||||
const problem = detect(server, 'drift', `${prepared.plan.files.length} file difference(s) from the promoted setup`, { files: prepared.plan.files, fingerprint: prepared.plan.fingerprint });
|
||||
if (problem.state !== 'open' || !maintenanceAllowed('drift')) return;
|
||||
const result = db().transaction(() => {
|
||||
const next = requestWatchdogLifecycle(server, 'rebuild', problem.id);
|
||||
if (next.ok) {
|
||||
const campaignId = createDriftRebuildCampaign(server, next.deploymentId, problem.id, prepared);
|
||||
addAction(problem, next.deploymentId, 'rebuild', campaignId);
|
||||
}
|
||||
return next;
|
||||
})();
|
||||
if (!result.ok && result.status !== 409 && result.status !== 503) transition(problem.id, 'attention', `Could not queue drift rebuild: ${result.error}`);
|
||||
}
|
||||
|
||||
let inFlight: Promise<void> | null = null;
|
||||
export function watchdogTick(forceScan = false): Promise<void> {
|
||||
if (inFlight) return inFlight;
|
||||
inFlight = run(forceScan).finally(() => { inFlight = null; });
|
||||
return inFlight;
|
||||
}
|
||||
async function run(forceScan: boolean): Promise<void> {
|
||||
await reconcileActions();
|
||||
const lastScan = setting('watchdog_last_scan');
|
||||
if (!forceScan && lastScan && Date.now() - Date.parse(lastScan) < WATCHDOG_SCAN_MS) return;
|
||||
setSetting('watchdog_last_scan', iso());
|
||||
if (!komodoConfigured()) {
|
||||
setSetting('watchdog_scan_error', 'Configure Komodo before automatic recovery. RCON or fresh telemetry verifies game status.'); return;
|
||||
}
|
||||
const errors: string[] = [];
|
||||
const servers = listServers();
|
||||
let cursor = 0;
|
||||
async function pump() {
|
||||
while (cursor < servers.length) {
|
||||
const server = servers[cursor++];
|
||||
try { await scanServer(server.name); }
|
||||
catch (err) { errors.push(`${server.name}: ${(err as Error).message}`); }
|
||||
}
|
||||
}
|
||||
await Promise.all(Array.from({ length: Math.min(4, servers.length) }, pump));
|
||||
// Preserve incidents for temporarily absent rebuild instances, but close retired registrations.
|
||||
const abandoned = db().prepare("SELECT * FROM watchdog_incidents WHERE state != 'resolved'").all() as WatchdogIncident[];
|
||||
for (const row of abandoned) {
|
||||
if (getServer(row.server) || hasPending(row.server)) continue;
|
||||
const last = db().prepare('SELECT action,state FROM server_deployments WHERE server=? ORDER BY rowid DESC LIMIT 1').get(row.server) as { action: string; state: string } | undefined;
|
||||
// An absent registry can also mean a failed rebuild. Only a completed
|
||||
// destruction retires that failure without requiring inspection.
|
||||
if (['open','dismissed'].includes(row.state) || (last?.action === 'destroy' && ['done','warning'].includes(last.state)))
|
||||
transition(row.id, 'resolved', 'Server is no longer registered');
|
||||
}
|
||||
setSetting('watchdog_scan_error', errors.join('\n'));
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
import { db } from '../db/index.js';
|
||||
import { getServer, listServers } from '../fleet/servers.js';
|
||||
import { requestRestart, cancelProvisioning } from '../provisioning/index.js';
|
||||
import { nextMaintenanceAt } from '../../plutonium.js';
|
||||
import type { MaintenanceJob, MaintenanceSchedule, PlutoniumStatus } from '../../plutonium.js';
|
||||
|
||||
import { readSetting, writeSetting } from './settings.js';
|
||||
import { checkPlutonium, probeVersion, release, freshRelease, runningVersions, HOUR } from '../integrations/plutonium.js';
|
||||
import { maintenanceAllowed } from './policies.js';
|
||||
export { checkPlutonium } from '../integrations/plutonium.js';
|
||||
const nowIso = () => new Date().toISOString();
|
||||
function schedule(): MaintenanceSchedule {
|
||||
// Legacy schedules were explicitly UTC. Preserve their timing until saved.
|
||||
return { timeZone: 'UTC', ...readSetting('plutonium_schedule', { enabled: false, time: '06:00', nextAt: null }) };
|
||||
}
|
||||
export function saveMaintenanceSchedule(enabled: boolean, time: string, timeZone: string): void {
|
||||
const nextAt = nextMaintenanceAt(time, new Date(), timeZone);
|
||||
writeSetting('plutonium_schedule', { enabled, time, timeZone, nextAt: enabled ? nextAt : null });
|
||||
}
|
||||
export function plutoniumStatus(): PlutoniumStatus {
|
||||
return {
|
||||
release: release(), schedule: schedule(), servers: runningVersions(),
|
||||
jobs: db().prepare(`SELECT m.*, d.stage AS operation_stage, d.state AS operation_state
|
||||
FROM maintenance_jobs m LEFT JOIN server_deployments d ON d.id = m.provisioning_id
|
||||
WHERE m.state IN ('queued','restarting')
|
||||
OR (m.state = 'failed' AND NOT EXISTS (SELECT 1 FROM maintenance_jobs n WHERE n.server = m.server AND n.kind = m.kind AND n.id > m.id))
|
||||
OR m.id IN (SELECT id FROM maintenance_jobs ORDER BY id DESC LIMIT 100) ORDER BY m.id DESC`).all() as MaintenanceJob[]
|
||||
};
|
||||
}
|
||||
|
||||
export function queueMaintenance(servers: string[], kind: MaintenanceJob['kind'] = 'legacy', origin: MaintenanceJob['origin'] = 'manual'): number {
|
||||
const names = new Set(listServers().map(s => s.name));
|
||||
if (!servers.length || servers.some(s => !names.has(s))) throw new Error('Choose existing fleet servers.');
|
||||
return db().transaction(() => {
|
||||
const target = kind === 'schedule' ? null : freshRelease();
|
||||
if (kind === 'update' && target === null) throw new Error('Check the published revision before requesting an update.');
|
||||
let added = 0;
|
||||
for (const server of new Set(servers)) {
|
||||
if (getServer(server)?.desired_state === 'stopped') continue;
|
||||
added += db().prepare(`INSERT OR IGNORE INTO maintenance_jobs (server, target_revision, requested_at, kind, origin)
|
||||
VALUES (?, ?, ?, ?, ?)`).run(server, target, nowIso(), kind, origin).changes;
|
||||
const job = db().prepare("SELECT id FROM maintenance_jobs WHERE server = ? AND state IN ('queued','restarting')").get(server) as { id: number };
|
||||
requestRestart(server, { kind: 'maintenance', id: String(job.id) });
|
||||
}
|
||||
return added;
|
||||
})();
|
||||
}
|
||||
|
||||
/** One automatic attempt per server/revision. Explicit retries preserve the old outcome. */
|
||||
export function queueUpdates(servers?: string[], automatic = false): number {
|
||||
if (!maintenanceAllowed()) throw new Error('Maintenance is paused. Resume it before requesting updates.');
|
||||
if (automatic && !maintenanceAllowed('updates')) return 0;
|
||||
if (freshRelease() === null) throw new Error('Check the published revision before requesting an update.');
|
||||
const versions = runningVersions();
|
||||
if (servers && (!servers.length || servers.some(server => !versions.some(v => v.server === server)))) throw new Error('Choose existing fleet servers.');
|
||||
const candidates = versions.filter(v => v.update_status === 'outdated' && (!servers || servers.includes(v.server)));
|
||||
const names = candidates.filter(v => !automatic || !db().prepare(`SELECT 1 FROM maintenance_jobs
|
||||
WHERE server = ? AND target_revision = ? AND kind = 'update'`).get(v.server, freshRelease())).map(v => v.server);
|
||||
return names.length ? queueMaintenance(names, 'update', automatic ? 'automatic' : 'manual') : 0;
|
||||
}
|
||||
|
||||
export function retryMaintenance(id: number): number {
|
||||
const job = db().prepare("SELECT * FROM maintenance_jobs WHERE id = ? AND state = 'failed'").get(id) as MaintenanceJob | undefined;
|
||||
if (!job) throw new Error('Only failed maintenance can be retried.');
|
||||
if (!maintenanceAllowed()) throw new Error('Maintenance is paused.');
|
||||
if (getServer(job.server)?.desired_state === 'stopped') throw new Error('This server is intentionally stopped.');
|
||||
// Ambiguous lifecycle failures retain their slot and require diagnostics, not another request.
|
||||
if (job.provisioning_id) {
|
||||
const operation = db().prepare('SELECT state FROM server_deployments WHERE id = ?').get(job.provisioning_id) as { state: string } | undefined;
|
||||
if (operation?.state !== 'done') throw new Error('Inspect and recover the existing Provisioning operation before retrying.');
|
||||
}
|
||||
return queueMaintenance([job.server], job.kind, 'manual');
|
||||
}
|
||||
export function cancelMaintenance(id: number): boolean {
|
||||
return db().transaction(() => {
|
||||
const job = db().prepare("SELECT provisioning_id FROM maintenance_jobs WHERE id = ? AND state = 'queued'").get(id) as { provisioning_id: string | null } | undefined;
|
||||
if (!job) return false;
|
||||
if (job.provisioning_id && !cancelProvisioning(job.provisioning_id)) return false;
|
||||
return db().prepare(`UPDATE maintenance_jobs SET state = 'cancelled', finished_at = ?, detail = 'Cancelled by operator'
|
||||
WHERE id = ? AND state = 'queued'`).run(nowIso(), id).changes > 0;
|
||||
})();
|
||||
}
|
||||
|
||||
function finish(job: MaintenanceJob, state: 'done' | 'failed' | 'cancelled', detail: string) {
|
||||
db().prepare('UPDATE maintenance_jobs SET state = ?, detail = ?, finished_at = ? WHERE id = ?')
|
||||
.run(state, detail, nowIso(), job.id);
|
||||
}
|
||||
function hold(job: MaintenanceJob, detail: string) {
|
||||
db().prepare('UPDATE maintenance_jobs SET detail = ? WHERE id = ?').run(detail, job.id);
|
||||
}
|
||||
|
||||
let ticking: Promise<void> | null = null;
|
||||
export function maintenanceTick(): Promise<void> {
|
||||
if (ticking) return ticking;
|
||||
ticking = runTick().finally(() => { ticking = null; });
|
||||
return ticking;
|
||||
}
|
||||
async function runTick(): Promise<void> {
|
||||
const latest = release();
|
||||
if (!latest.checkedAt || Date.now() - Date.parse(latest.checkedAt) >= HOUR) await checkPlutonium();
|
||||
if (maintenanceAllowed('updates') && freshRelease() !== null) queueUpdates(undefined, true);
|
||||
db().transaction(() => {
|
||||
const plan = schedule();
|
||||
if (maintenanceAllowed() && plan.enabled && plan.nextAt && plan.nextAt <= nowIso()) {
|
||||
const names = listServers().filter(s => s.desired_state !== 'stopped').map(s => s.name);
|
||||
if (names.length) queueMaintenance(names, 'schedule', 'schedule');
|
||||
// Coalesce missed days after downtime into one run; no restart backlog.
|
||||
writeSetting('plutonium_schedule', { ...plan, nextAt: nextMaintenanceAt(plan.time, new Date(), plan.timeZone) });
|
||||
}
|
||||
})();
|
||||
const jobs = db().prepare(`SELECT * FROM maintenance_jobs WHERE state IN ('queued','restarting') ORDER BY id`).all() as MaintenanceJob[];
|
||||
for (const job of jobs) {
|
||||
try { await processJob(job); }
|
||||
catch { hold(job, 'Maintenance check failed; waiting for the next check.'); }
|
||||
}
|
||||
// Update attempts are also durable suppression evidence; never age them into an automatic retry.
|
||||
db().prepare(`DELETE FROM maintenance_jobs WHERE kind != 'update' AND finished_at < ? AND state IN ('done','failed','cancelled')`)
|
||||
.run(new Date(Date.now() - 30 * 24 * HOUR).toISOString());
|
||||
}
|
||||
async function processJob(job: MaintenanceJob): Promise<void> {
|
||||
const server = getServer(job.server);
|
||||
if (!server || server.destroyed_at) return finish(job, 'cancelled', 'Server removed.');
|
||||
const operation = requestRestart(job.server, { kind: 'maintenance', id: String(job.id) });
|
||||
if (['failed','interrupted','cancelled'].includes(operation.state)) {
|
||||
return finish(job, operation.state === 'cancelled' ? 'cancelled' : 'failed', operation.error ?? operation.stage);
|
||||
}
|
||||
if (operation.state !== 'done') {
|
||||
if (operation.submitted_at) {
|
||||
db().prepare("UPDATE maintenance_jobs SET state = 'restarting', restarted_at = ? WHERE id = ?").run(operation.submitted_at, job.id);
|
||||
db().prepare('UPDATE plutonium_versions SET revision = NULL, error = ? WHERE server = ?').run('Waiting for restart recovery.', job.server);
|
||||
}
|
||||
return hold(job, operation.stage);
|
||||
}
|
||||
const revision = await probeVersion(job.server);
|
||||
if (job.target_revision && (revision === null || revision < job.target_revision)) {
|
||||
return finish(job, 'failed', `Server recovered, but r${job.target_revision} was not verified (${revision === null ? 'version unknown' : `running r${revision}`}). Check Plutainer auto-update settings.`);
|
||||
}
|
||||
return finish(job, 'done', `Server recovered; ${revision === null ? 'running revision unknown' : `running r${revision}`}.`);
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
import { db } from '../db/index.js';
|
||||
export function readSetting<T>(key: string, fallback: T): T {
|
||||
const row = db().prepare('SELECT value FROM settings WHERE key = ?').get(key) as { value: string } | undefined;
|
||||
return row ? JSON.parse(row.value) : fallback;
|
||||
}
|
||||
export function writeSetting(key: string, value: unknown): void {
|
||||
db().prepare('INSERT INTO settings (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value')
|
||||
.run(key, JSON.stringify(value));
|
||||
}
|
||||
@@ -15,8 +15,7 @@ import { isProvisioning } from '../fleet/provisioning.js';
|
||||
import { STATUS_REFRESH_MS } from '../../server-freshness.js';
|
||||
import { recordMetrics } from '../fleet/metrics.js';
|
||||
import { selectedZombieMap } from '../../map-aliases.js';
|
||||
import { maintenanceTick } from '../fleet/plutonium.js';
|
||||
import { watchdogTick } from '../watchdog/index.js';
|
||||
import { runMaintenance } from '../maintenance/index.js';
|
||||
|
||||
const TICK_MS = 15_000;
|
||||
// Cap concurrent UDP RCON calls so a 15-server fleet doesn't blast the
|
||||
@@ -38,8 +37,7 @@ function tick(): Promise<void> {
|
||||
async function reconcileRequesters(): Promise<void> {
|
||||
try { await sweep(); } catch (err) { console.error('[worker] campaign error:', err); }
|
||||
try { await betaTick(); } catch (err) { console.error('[worker] beta error:', err); }
|
||||
try { await maintenanceTick(); } catch (err) { console.error('[worker] maintenance error:', err); }
|
||||
try { await watchdogTick(); } catch (err) { console.error('[worker] watchdog error:', err); }
|
||||
try { await runMaintenance(); } catch (err) { console.error('[worker] maintenance error:', err); }
|
||||
}
|
||||
|
||||
async function runTick(): Promise<void> {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { maintenanceAllowed } from '../maintenance/policies.js';
|
||||
import { db } from '../db/index.js';
|
||||
import { komodoConfigured, rconConfigured } from '../config.js';
|
||||
import { getServer } from '../fleet/servers.js';
|
||||
@@ -141,10 +142,10 @@ export function requestLifecycle(server: string, action: RuntimeAction) {
|
||||
type LifecycleRequestResult = { ok: true; deploymentId: string } | { ok: false; status: number; error: string };
|
||||
|
||||
function watchdogRequestGuard(server: string, incidentId: string): Extract<LifecycleRequestResult, { ok: false }> | null {
|
||||
const incident = db().prepare("SELECT 1 FROM watchdog_incidents WHERE id = ? AND server = ? AND state IN ('open','recovering')").get(incidentId, server);
|
||||
const incident = db().prepare("SELECT kind FROM watchdog_incidents WHERE id = ? AND server = ? AND state IN ('open','recovering')").get(incidentId, server) as { kind: 'health' | 'drift' } | undefined;
|
||||
if (!incident) return { ok: false, status: 409, error: 'The Watchdog incident is no longer active.' };
|
||||
if (db().prepare("SELECT 1 FROM settings WHERE key = 'watchdog_enabled' AND value = '0'").get()) {
|
||||
return { ok: false, status: 409, error: 'Watchdog is paused.' };
|
||||
if (!maintenanceAllowed(incident.kind)) {
|
||||
return { ok: false, status: 409, error: 'This maintenance policy is paused.' };
|
||||
}
|
||||
const row = getServer(server);
|
||||
if (!row || row.destroyed_at) return { ok: false, status: 404, error: 'Server not found.' };
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { maintenanceAllowed } from '../maintenance/policies.js';
|
||||
import { db } from '../db/index.js';
|
||||
import { komodoConfigured, rconConfigured } from '../config.js';
|
||||
import { getServer } from '../fleet/servers.js';
|
||||
@@ -24,7 +25,8 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled'
|
||||
for (const source of sources.filter(source => source.kind === 'watchdog')) {
|
||||
if (!db().prepare("SELECT 1 FROM watchdog_incidents WHERE id = ? AND server = ? AND state IN ('open','recovering')").get(source.source_id, operation.server)
|
||||
|| getServer(operation.server)?.desired_state === 'stopped') return 'cancelled';
|
||||
if (db().prepare("SELECT 1 FROM settings WHERE key = 'watchdog_enabled' AND value = '0'").get()) return 'held';
|
||||
const incident = db().prepare('SELECT kind FROM watchdog_incidents WHERE id = ?').get(source.source_id) as { kind: 'health' | 'drift' };
|
||||
if (!maintenanceAllowed(incident.kind)) return 'held';
|
||||
}
|
||||
let held = false;
|
||||
for (const source of sources) {
|
||||
@@ -43,8 +45,12 @@ function sourceEligibility(operation: Operation): 'ready' | 'held' | 'cancelled'
|
||||
if (!row.outcome || ['active','passed'].includes(row.outcome)) return 'ready';
|
||||
if (row.outcome === 'paused') held = true;
|
||||
} else if (source.kind === 'maintenance') {
|
||||
const row = db().prepare('SELECT state, provisioning_id FROM maintenance_jobs WHERE id = ?').get(source.source_id) as { state: string; provisioning_id: string } | undefined;
|
||||
if (row?.provisioning_id === operation.id && ['queued','restarting'].includes(row.state)) return 'ready';
|
||||
const row = db().prepare('SELECT state, provisioning_id, kind, origin FROM maintenance_jobs WHERE id = ?').get(source.source_id) as { state: string; provisioning_id: string; kind: string; origin: string } | undefined;
|
||||
if (row?.provisioning_id === operation.id && ['queued','restarting'].includes(row.state)) {
|
||||
if (getServer(operation.server)?.desired_state === 'stopped') return 'cancelled';
|
||||
if (!maintenanceAllowed() || (row.kind === 'update' && row.origin === 'automatic' && !maintenanceAllowed('updates'))) return 'held';
|
||||
return 'ready';
|
||||
}
|
||||
} else return 'ready';
|
||||
}
|
||||
return held ? 'held' : 'cancelled';
|
||||
|
||||
@@ -1,233 +1,2 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { db } from '../db/index.js';
|
||||
import { komodoConfigured } from '../config.js';
|
||||
import { getServer, listServers } from '../fleet/servers.js';
|
||||
import { komodoInspectHealthy } from '../integrations/komodo.js';
|
||||
import { rconStatusResult } from '../integrations/rcon.js';
|
||||
import { telemetryEvidence } from '../fleet/status-evidence.js';
|
||||
import { prepareDriftCampaign, createDriftRebuildCampaign, previewDrift } from '../fleet/drift.js';
|
||||
import { requestWatchdogLifecycle, escalateWatchdogRestart } from '../provisioning/requests.js';
|
||||
import type { WatchdogAction, WatchdogDetail, WatchdogEvidence, WatchdogIncident, WatchdogOverview, WatchdogState } from '../../watchdog.js';
|
||||
|
||||
export const WATCHDOG_SCAN_MS = 60_000;
|
||||
export const WATCHDOG_FAILURE_MS = 120_000;
|
||||
export const WATCHDOG_RECOVERY_MS = 300_000;
|
||||
const iso = () => new Date().toISOString();
|
||||
const pendingStates = ['queued', 'waiting', 'running', 'awaiting-health'];
|
||||
const setting = (key: string) => (db().prepare('SELECT value FROM settings WHERE key = ?').get(key) as { value: string } | undefined)?.value ?? null;
|
||||
function setSetting(key: string, value: string) { db().prepare('INSERT INTO settings(key,value) VALUES (?,?) ON CONFLICT(key) DO UPDATE SET value=excluded.value').run(key, value); }
|
||||
export const watchdogEnabled = () => setting('watchdog_enabled') !== '0';
|
||||
export function setWatchdogEnabled(enabled: boolean) { setSetting('watchdog_enabled', enabled ? '1' : '0'); }
|
||||
|
||||
function incident(id: string) { return db().prepare('SELECT * FROM watchdog_incidents WHERE id = ?').get(id) as WatchdogIncident | undefined; }
|
||||
function active(server: string, kind: 'health' | 'drift') { return db().prepare("SELECT * FROM watchdog_incidents WHERE server = ? AND kind = ? AND state != 'resolved'").get(server, kind) as WatchdogIncident | undefined; }
|
||||
function event(id: string, message: string) { db().prepare('INSERT INTO watchdog_events(incident_id,created_at,message) VALUES (?,?,?)').run(id, iso(), message); }
|
||||
function transition(id: string, state: WatchdogState, summary: string) {
|
||||
const previous = incident(id);
|
||||
if (!previous || (previous.state === state && previous.summary === summary)) return;
|
||||
db().prepare('UPDATE watchdog_incidents SET state=?, summary=?, finished_at=? WHERE id=?')
|
||||
.run(state, summary, state === 'resolved' || state === 'dismissed' ? iso() : null, id);
|
||||
event(id, summary);
|
||||
}
|
||||
function detect(server: string, kind: 'health' | 'drift', summary: string, details: WatchdogEvidence): WatchdogIncident {
|
||||
const previous = active(server, kind);
|
||||
if (previous) {
|
||||
// Repeated UI scans must not manufacture a sustained failure.
|
||||
const newSample = Date.now() - Date.parse(previous.last_seen_at) >= WATCHDOG_SCAN_MS;
|
||||
const actionable = (details as WatchdogEvidence & { actionable?: boolean }).actionable;
|
||||
const priorActionable = (JSON.parse(previous.details) as { actionable?: boolean }).actionable;
|
||||
const reset = kind === 'health' && actionable === true && priorActionable !== true;
|
||||
db().prepare('UPDATE watchdog_incidents SET details=?, last_seen_at=?, failures=?, first_seen_at=? WHERE id=?')
|
||||
.run(JSON.stringify(details), newSample || reset ? iso() : previous.last_seen_at,
|
||||
reset ? 1 : previous.failures + (newSample ? 1 : 0), reset ? iso() : previous.first_seen_at, previous.id);
|
||||
if (previous.state === 'open') transition(previous.id, 'open', summary);
|
||||
return incident(previous.id)!;
|
||||
}
|
||||
const id = randomUUID();
|
||||
db().prepare(`INSERT INTO watchdog_incidents(id,server,kind,state,summary,details,first_seen_at,last_seen_at)
|
||||
VALUES (?,?,?,'open',?,?,?,?)`).run(id, server, kind, summary, JSON.stringify(details), iso(), iso());
|
||||
event(id, summary);
|
||||
return incident(id)!;
|
||||
}
|
||||
function actions(id: string): WatchdogAction[] {
|
||||
return db().prepare(`SELECT a.*, d.state AS operation_state, d.stage AS operation_stage, d.error AS operation_error
|
||||
FROM watchdog_actions a LEFT JOIN server_deployments d ON d.id=a.operation_id
|
||||
WHERE a.incident_id=? ORDER BY a.rowid`).all(id) as WatchdogAction[];
|
||||
}
|
||||
export function watchdogOverview(): WatchdogOverview {
|
||||
return { enabled: watchdogEnabled(), lastScanAt: setting('watchdog_last_scan'), scanError: setting('watchdog_scan_error') || null,
|
||||
incidents: db().prepare(`SELECT * FROM watchdog_incidents WHERE state != 'resolved' OR id IN (
|
||||
SELECT id FROM watchdog_incidents WHERE state = 'resolved' ORDER BY first_seen_at DESC, rowid DESC LIMIT 100)
|
||||
ORDER BY CASE WHEN state='resolved' THEN 1 ELSE 0 END, first_seen_at DESC, rowid DESC`).all() as WatchdogIncident[] };
|
||||
}
|
||||
export function watchdogDetail(id: string): WatchdogDetail | null {
|
||||
const row = incident(id);
|
||||
return row ? { incident: row, actions: actions(id), events: db().prepare('SELECT id,created_at,message FROM watchdog_events WHERE incident_id=? ORDER BY id').all(id) as WatchdogDetail['events'] } : null;
|
||||
}
|
||||
function hasPending(server: string) {
|
||||
return !!db().prepare("SELECT 1 FROM server_deployments WHERE server=? AND state IN ('queued','waiting','running','awaiting-health') LIMIT 1").get(server);
|
||||
}
|
||||
function protectedWork(server: string): boolean {
|
||||
return !!db().prepare(`SELECT 1 FROM beta_servers bs JOIN betas b ON b.id=bs.beta_id WHERE bs.server=? AND b.state IN ('running','ready-promote') LIMIT 1`).get(server)
|
||||
|| !!db().prepare(`SELECT 1 FROM deployment_servers ds WHERE ds.server=? AND ds.restart_state IN ('pending','held','failed','manual')
|
||||
AND NOT EXISTS (SELECT 1 FROM watchdog_actions a WHERE a.campaign_id=ds.deployment_id) LIMIT 1`).get(server);
|
||||
}
|
||||
function addAction(row: WatchdogIncident, operationId: string, action: 'restart' | 'rebuild', campaignId: string | null = null) {
|
||||
db().prepare('INSERT INTO watchdog_actions(id,incident_id,action,operation_id,campaign_id,created_at,cycle) VALUES (?,?,?,?,?,?,?)')
|
||||
.run(randomUUID(), row.id, action, operationId, campaignId, iso(), row.cycle);
|
||||
transition(row.id, 'recovering', action === 'restart' ? 'Restart queued; waiting for safe recovery' : 'Rebuild queued with the latest promoted setup');
|
||||
event(row.id, `${action === 'restart' ? 'Restart' : 'Rebuild'} requested: ${operationId}${campaignId ? `; campaign ${campaignId}` : ''}`);
|
||||
}
|
||||
export function updateWatchdogIncident(id: string, action: 'retry' | 'dismiss') {
|
||||
return db().transaction(() => {
|
||||
const row = incident(id);
|
||||
if (!row) return { ok: false as const, status: 404, error: 'Watchdog incident not found.' };
|
||||
if (actions(id).some(a => pendingStates.includes(a.operation_state ?? ''))) return { ok: false as const, status: 409, error: 'Provisioning is still active. Complete or cancel that operation first.' };
|
||||
if (action === 'dismiss') {
|
||||
if (!['open','attention'].includes(row.state)) return { ok: false as const, status: 409, error: 'Only detected incidents or incidents needing attention can be dismissed.' };
|
||||
transition(id, 'dismissed', 'Dismissed by operator; automatic attempts remain suppressed until the condition clears');
|
||||
} else {
|
||||
if (!['attention','dismissed'].includes(row.state)) return { ok: false as const, status: 409, error: 'Only incidents needing attention or dismissed incidents can be retried.' };
|
||||
if (!getServer(row.server)) return { ok: false as const, status: 409, error: 'The server is no longer registered; inspect its provisioning history.' };
|
||||
db().prepare('UPDATE watchdog_incidents SET cycle=cycle+1, failures=1, first_seen_at=?, last_seen_at=? WHERE id=?').run(iso(), iso(), id);
|
||||
transition(id, 'open', 'Operator requested a fresh recovery attempt; checking current conditions');
|
||||
}
|
||||
return { ok: true as const };
|
||||
})();
|
||||
}
|
||||
|
||||
async function reconcileActions(): Promise<void> {
|
||||
const rows = db().prepare("SELECT * FROM watchdog_incidents WHERE state IN ('recovering','attention')").all() as WatchdogIncident[];
|
||||
for (const row of rows) {
|
||||
const action = actions(row.id).filter(a => a.cycle === row.cycle).at(-1);
|
||||
if (!action) continue;
|
||||
const operation = db().prepare('SELECT state,submitted_at,started_at,stage,error FROM server_deployments WHERE id=?').get(action.operation_id) as
|
||||
{ state: string; submitted_at: string | null; started_at: string; stage: string; error: string | null } | undefined;
|
||||
if (!operation) { transition(row.id, 'attention', 'Provisioning history is missing; inspect the server before retrying'); continue; }
|
||||
if (['done','warning'].includes(operation.state)) {
|
||||
if (row.kind === 'drift') {
|
||||
try {
|
||||
const plan = await previewDrift([row.server]);
|
||||
if (incident(row.id)?.cycle !== row.cycle) continue;
|
||||
if (plan.files.length) { transition(row.id, 'attention', 'Rebuild completed but drift remains; inspect the campaign and server files'); continue; }
|
||||
} catch (err) { transition(row.id, 'attention', `Cannot verify rebuilt files: ${(err as Error).message}`); continue; }
|
||||
}
|
||||
transition(row.id, 'resolved', action.action === 'restart' ? 'Server recovered after restart' : 'Server recovered after rebuild');
|
||||
continue;
|
||||
}
|
||||
if (row.state === 'attention') continue; // Exhausted attempts stay latched, including across app restarts.
|
||||
if (['cancelled','interrupted'].includes(operation.state)) {
|
||||
transition(row.id, 'attention', `Provisioning ${operation.state}; inspect the recorded operation before retrying`); continue;
|
||||
}
|
||||
if (action.action === 'rebuild') {
|
||||
if (operation.state === 'failed') transition(row.id, 'attention', `Rebuild failed: ${operation.error ?? operation.stage}`);
|
||||
else if (operation.state === 'awaiting-health' && Date.now() - Date.parse(operation.submitted_at ?? operation.started_at) >= WATCHDOG_RECOVERY_MS)
|
||||
transition(row.id, 'attention', 'Rebuild has not recovered within five minutes; provisioning continues verification');
|
||||
continue;
|
||||
}
|
||||
if (!['awaiting-health','failed'].includes(operation.state)) continue;
|
||||
if (Date.now() - Date.parse(operation.submitted_at ?? operation.started_at) < WATCHDOG_RECOVERY_MS) continue;
|
||||
if (!watchdogEnabled()) continue;
|
||||
const current = getServer(row.server);
|
||||
if (!current || current.desired_state === 'stopped' || protectedWork(row.server)) continue;
|
||||
const result = db().transaction(() => {
|
||||
const next = escalateWatchdogRestart(row.id, action.operation_id, WATCHDOG_RECOVERY_MS);
|
||||
if (next.ok) addAction(row, next.deploymentId, 'rebuild');
|
||||
return next;
|
||||
})();
|
||||
if (!result.ok && result.status !== 409 && result.status !== 503) transition(row.id, 'attention', `Could not queue rebuild: ${result.error}`);
|
||||
}
|
||||
}
|
||||
|
||||
interface Observation { healthy: boolean; actionable: boolean; reason: string; containerState: string | null }
|
||||
async function observe(server: string): Promise<Observation> {
|
||||
const health = await komodoInspectHealthy(server);
|
||||
const state = health.state?.toLowerCase() ?? null;
|
||||
if (!health.ok) return { healthy: false, actionable: !!state && ['stopped','exited','created','dead','unhealthy','restarting','starting'].includes(state), reason: health.reason, containerState: state };
|
||||
if (telemetryEvidence(server)) return { healthy: true, actionable: false, reason: 'Komodo healthy with fresh telemetry', containerState: state };
|
||||
const rcon = await rconStatusResult(server);
|
||||
if (rcon.ok || telemetryEvidence(server)) return { healthy: true, actionable: false, reason: 'Komodo healthy with fresh game status', containerState: state };
|
||||
return { healthy: false, actionable: true, reason: `Game status unavailable: ${rcon.reason}; waiting for player-count verification before recovery`, containerState: state };
|
||||
}
|
||||
async function scanServer(server: string): Promise<void> {
|
||||
const row = getServer(server);
|
||||
if (!row || row.desired_state === 'stopped') return;
|
||||
if (hasPending(server) || protectedWork(server)) return;
|
||||
const observation = await observe(server);
|
||||
// Probes can overlap an API request; recheck ownership before queuing anything.
|
||||
if (hasPending(server) || protectedWork(server) || getServer(server)?.desired_state === 'stopped') return;
|
||||
if (observation.healthy) {
|
||||
const current = active(server, 'health');
|
||||
if (current) transition(current.id, 'resolved', 'Fresh game status and Komodo health verified');
|
||||
} else {
|
||||
const problem = detect(server, 'health', observation.reason, observation);
|
||||
const startup = ['starting','restarting'].includes(observation.containerState ?? '');
|
||||
const oldEnough = Date.now() - Date.parse(problem.first_seen_at) >= (startup ? WATCHDOG_RECOVERY_MS : WATCHDOG_FAILURE_MS);
|
||||
if (problem.state === 'open' && observation.actionable && oldEnough && problem.failures >= 3 && watchdogEnabled()) {
|
||||
const result = db().transaction(() => {
|
||||
const next = requestWatchdogLifecycle(server, 'restart', problem.id);
|
||||
if (next.ok) addAction(problem, next.deploymentId, 'restart');
|
||||
return next;
|
||||
})();
|
||||
if (!result.ok && result.status !== 409 && result.status !== 503) transition(problem.id, 'attention', result.error);
|
||||
}
|
||||
// Unknown container health is an observation failure, not proof that a rebuild is safe.
|
||||
if (!observation.actionable || hasPending(server)) return;
|
||||
}
|
||||
const prepared = await prepareDriftCampaign(server);
|
||||
if (hasPending(server) || protectedWork(server) || getServer(server)?.desired_state === 'stopped') return;
|
||||
if (!prepared.plan.files.length) {
|
||||
const current = active(server, 'drift');
|
||||
if (current) transition(current.id, 'resolved', 'Server files match the promoted setup');
|
||||
return;
|
||||
}
|
||||
const problem = detect(server, 'drift', `${prepared.plan.files.length} file difference(s) from the promoted setup`, { files: prepared.plan.files, fingerprint: prepared.plan.fingerprint });
|
||||
if (problem.state !== 'open' || !watchdogEnabled()) return;
|
||||
const result = db().transaction(() => {
|
||||
const next = requestWatchdogLifecycle(server, 'rebuild', problem.id);
|
||||
if (next.ok) {
|
||||
const campaignId = createDriftRebuildCampaign(server, next.deploymentId, problem.id, prepared);
|
||||
addAction(problem, next.deploymentId, 'rebuild', campaignId);
|
||||
}
|
||||
return next;
|
||||
})();
|
||||
if (!result.ok && result.status !== 409 && result.status !== 503) transition(problem.id, 'attention', `Could not queue drift rebuild: ${result.error}`);
|
||||
}
|
||||
|
||||
let inFlight: Promise<void> | null = null;
|
||||
export function watchdogTick(forceScan = false): Promise<void> {
|
||||
if (inFlight) return inFlight;
|
||||
inFlight = run(forceScan).finally(() => { inFlight = null; });
|
||||
return inFlight;
|
||||
}
|
||||
async function run(forceScan: boolean): Promise<void> {
|
||||
await reconcileActions();
|
||||
const lastScan = setting('watchdog_last_scan');
|
||||
if (!forceScan && lastScan && Date.now() - Date.parse(lastScan) < WATCHDOG_SCAN_MS) return;
|
||||
setSetting('watchdog_last_scan', iso());
|
||||
if (!komodoConfigured()) {
|
||||
setSetting('watchdog_scan_error', 'Configure Komodo before automatic recovery. RCON or fresh telemetry verifies game status.'); return;
|
||||
}
|
||||
const errors: string[] = [];
|
||||
const servers = listServers();
|
||||
let cursor = 0;
|
||||
async function pump() {
|
||||
while (cursor < servers.length) {
|
||||
const server = servers[cursor++];
|
||||
try { await scanServer(server.name); }
|
||||
catch (err) { errors.push(`${server.name}: ${(err as Error).message}`); }
|
||||
}
|
||||
}
|
||||
await Promise.all(Array.from({ length: Math.min(4, servers.length) }, pump));
|
||||
// Preserve incidents for temporarily absent rebuild instances, but close retired registrations.
|
||||
const abandoned = db().prepare("SELECT * FROM watchdog_incidents WHERE state != 'resolved'").all() as WatchdogIncident[];
|
||||
for (const row of abandoned) {
|
||||
if (getServer(row.server) || hasPending(row.server)) continue;
|
||||
const last = db().prepare('SELECT action,state FROM server_deployments WHERE server=? ORDER BY rowid DESC LIMIT 1').get(row.server) as { action: string; state: string } | undefined;
|
||||
// An absent registry can also mean a failed rebuild. Only a completed
|
||||
// destruction retires that failure without requiring inspection.
|
||||
if (['open','dismissed'].includes(row.state) || (last?.action === 'destroy' && ['done','warning'].includes(last.state)))
|
||||
transition(row.id, 'resolved', 'Server is no longer registered');
|
||||
}
|
||||
setSetting('watchdog_scan_error', errors.join('\n'));
|
||||
}
|
||||
// Compatibility for existing callers; Maintenance owns recovery.
|
||||
export * from "../maintenance/recovery.js";
|
||||
|
||||
@@ -237,7 +237,7 @@
|
||||
flash = body.error ?? `HTTP ${res.status}`;
|
||||
return;
|
||||
}
|
||||
scopeComplete = { count: scopeImpact.affected.length, composed: scopeImpact.composed, detached: body.detached?.length ?? 0, href: scopeImpact.affected.length === 1 ? '/watchdog?kind=drift&server=' + encodeURIComponent(scopeImpact.affected[0]) : '/watchdog?kind=drift' };
|
||||
scopeComplete = { count: scopeImpact.affected.length, composed: scopeImpact.composed, detached: body.detached?.length ?? 0, href: scopeImpact.affected.length === 1 ? '/maintenance?kind=drift&server=' + encodeURIComponent(scopeImpact.affected[0]) : '/maintenance?kind=drift' };
|
||||
scopeReviewOpen = false; scopeDialog?.close();
|
||||
await load();
|
||||
} catch (error) { flash = (error as Error).message; } finally {
|
||||
@@ -286,7 +286,7 @@
|
||||
{/snippet}
|
||||
</SectionHeader>
|
||||
|
||||
{#if scopeComplete}<div class="scope-complete" role="status"><strong>Scope saved.</strong> {scopeComplete.composed ? `Intended configuration may change on ${scopeComplete.count} eligible servers.` : scopeComplete.detached ? `Inventory tracking detached on ${scopeComplete.detached} servers; live files remain.` : `${scopeComplete.count} servers are in the old or new scope.`} Watchdog detects resulting drift and rebuilds eligible servers when automation is enabled. <a href={scopeComplete.href}>View drift recovery →</a></div>{/if}
|
||||
{#if scopeComplete}<div class="scope-complete" role="status"><strong>Scope saved.</strong> {scopeComplete.composed ? `Intended configuration may change on ${scopeComplete.count} eligible servers.` : scopeComplete.detached ? `Inventory tracking detached on ${scopeComplete.detached} servers; live files remain.` : `${scopeComplete.count} servers are in the old or new scope.`} Maintenance detects resulting drift and rebuilds eligible servers when automation is enabled. <a href={scopeComplete.href}>View drift recovery →</a></div>{/if}
|
||||
{#if latestVersion}<Card padding="lg"><div class="release-summary"><div><h2>Latest {artifactVersionLabel(latestVersion)}</h2><StatusBadge state={latestVersion.id === a.current_version_id ? 'promoted' : latestVersion.state} /><p>{latestVersion.notes || 'No version notes provided.'}</p></div>{#if runningBetaId !== null}<Button variant="primary" href={`/betas/${runningBetaId}`}>View active beta</Button>{:else if latestVersion.id !== a.current_version_id}<Button variant="primary" href={betaSelectionHref([{ scriptId: id, versionId: latestVersion.id }])}>Test latest version</Button>{:else}<span class="dim">Latest version is promoted</span>{/if}</div></Card>{/if}
|
||||
|
||||
{#if runningBetaId}
|
||||
@@ -417,7 +417,7 @@
|
||||
{/if}
|
||||
</div>
|
||||
</Field>
|
||||
{#if scopePreviewLoading}<p role="status">Reviewing eligible servers…</p>{:else if scopePreviewError}<p role="alert">{scopePreviewError}</p>{:else if scopeImpact}<section class="scope-impact" aria-label="Scope change review"><strong>{scopeImpact.after.length} eligible · {scopeImpact.added.length} added · {scopeImpact.removed.length} removed</strong>{#if scopeImpact.added.length}<p>Added: {scopeImpact.added.join(', ')}</p>{/if}{#if scopeImpact.removed.length}<p>Removed: {scopeImpact.removed.join(', ')}</p>{/if}<p>{scopeImpact.composed ? 'This changes intended configuration composition.' : scopeType === 'none' ? `${scopeImpact.detached.length} recorded installations will become untracked. Watchdog will detect orphaned files as drift.` : 'Scope describes eligibility for promoted files.'}</p><p>When automation is enabled, Watchdog rebuilds servers with drift after verifying a safe server state. Rebuilds delete the existing instance and all of its files.</p></section>{/if}
|
||||
{#if scopePreviewLoading}<p role="status">Reviewing eligible servers…</p>{:else if scopePreviewError}<p role="alert">{scopePreviewError}</p>{:else if scopeImpact}<section class="scope-impact" aria-label="Scope change review"><strong>{scopeImpact.after.length} eligible · {scopeImpact.added.length} added · {scopeImpact.removed.length} removed</strong>{#if scopeImpact.added.length}<p>Added: {scopeImpact.added.join(', ')}</p>{/if}{#if scopeImpact.removed.length}<p>Removed: {scopeImpact.removed.join(', ')}</p>{/if}<p>{scopeImpact.composed ? 'This changes intended configuration composition.' : scopeType === 'none' ? `${scopeImpact.detached.length} recorded installations will become untracked. Maintenance will detect orphaned files as drift.` : 'Scope describes eligibility for promoted files.'}</p><p>When automation is enabled, Maintenance rebuilds servers with drift after verifying a safe server state. Rebuilds delete the existing instance and all of its files.</p></section>{/if}
|
||||
<div class="dlg-actions">
|
||||
<Button variant="ghost" onclick={cancelScopeModal} disabled={updatingScope}>Cancel</Button>
|
||||
<Button type="submit" variant="primary" disabled={updatingScope || scopePreviewLoading || !scopeImpact} loading={updatingScope}>
|
||||
|
||||
@@ -1,121 +0,0 @@
|
||||
<script lang="ts">
|
||||
import { onMount } from 'svelte';
|
||||
import type { PlutoniumStatus } from '$lib/plutonium.js';
|
||||
import { maintenanceTimeInZone, PLUTONIUM_MANIFEST_URL } from '$lib/plutonium.js';
|
||||
import { timeFormat, timeZone } from '$ui/stores/time-zone.js';
|
||||
import Button from './primitives/Button.svelte';
|
||||
|
||||
let data = $state<PlutoniumStatus | null>(null);
|
||||
let busy = $state(false);
|
||||
let error = $state('');
|
||||
let notice = $state('');
|
||||
let enabled = $state(false);
|
||||
let time = $state('06:00');
|
||||
let initialized = false;
|
||||
let displayedZone = $state('');
|
||||
$effect(() => {
|
||||
if (data && displayedZone !== $timeZone) {
|
||||
time = maintenanceTimeInZone(data.schedule, $timeZone);
|
||||
displayedZone = $timeZone;
|
||||
}
|
||||
});
|
||||
async function refresh(action?: Record<string, unknown>) {
|
||||
if (busy) return;
|
||||
busy = true;
|
||||
try {
|
||||
const response = await fetch('/api/plutonium', action ? {
|
||||
method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(action)
|
||||
} : undefined);
|
||||
const result = await response.json();
|
||||
if (!response.ok) throw new Error(result.error ?? 'Could not load Plutonium status.');
|
||||
data = result;
|
||||
error = '';
|
||||
if (!initialized) {
|
||||
enabled = result.schedule.enabled; initialized = true;
|
||||
}
|
||||
if (action) notice = action.action === 'schedule' ? 'Schedule saved.'
|
||||
: action.action === 'restart' ? 'Restart queued. Occupied servers will wait until empty.'
|
||||
: action.action === 'cancel' ? 'Queued restart cancelled.'
|
||||
: result.release.error ? 'Version checks finished with an update-source error.' : 'Version checks finished.';
|
||||
} catch (cause) { error = (cause as Error).message; }
|
||||
finally { busy = false; }
|
||||
}
|
||||
function pending(server: string) {
|
||||
return data?.jobs.some(j => j.server === server && ['queued', 'restarting'].includes(j.state));
|
||||
}
|
||||
onMount(() => {
|
||||
void refresh();
|
||||
const timer = setInterval(() => { if (!document.hidden) void refresh(); }, 5000);
|
||||
return () => clearInterval(timer);
|
||||
});
|
||||
</script>
|
||||
|
||||
<section id="plutonium" aria-labelledby="plutonium-heading">
|
||||
<div class="heading"><h2 id="plutonium-heading">Plutonium updates & restarts</h2>
|
||||
<Button disabled={busy} onclick={() => refresh({ action: 'check' })}>Check versions</Button>
|
||||
</div>
|
||||
<p>Plutainer applies updates on startup when auto-update is enabled. Restarts wait for a fresh player count of zero and share the fleet’s rolling restart limit.</p>
|
||||
{#if error}<p class="ui-error" role="alert">{error}</p>{/if}
|
||||
{#if notice}<p role="status">{notice}</p>{/if}
|
||||
{#if data}
|
||||
<p>Latest published revision: <strong>{data.release.revision ? `r${data.release.revision}` : 'Unknown'}</strong>
|
||||
· <a href={PLUTONIUM_MANIFEST_URL} target="_blank" rel="noreferrer">Production manifest</a>
|
||||
· <a href="https://plutonium.pw/docs/changelog/" target="_blank" rel="noreferrer">Changelog</a>
|
||||
{#if data.release.checkedAt} · Checked {$timeFormat.dateTime(data.release.checkedAt)}{/if}
|
||||
</p>
|
||||
{#if data.release.error}<p role="alert">Update check failed: {data.release.error} Any displayed release is cached.</p>{/if}
|
||||
<form onsubmit={(event) => { event.preventDefault(); void refresh({ action: 'schedule', enabled, time, timeZone: $timeZone }); }}>
|
||||
<label class="toggle"><input type="checkbox" bind:checked={enabled} /> Daily fleet restart</label>
|
||||
<label>Queue time ({$timeZone})<input type="time" bind:value={time} required /></label>
|
||||
<Button type="submit" disabled={busy}>Save schedule</Button>
|
||||
</form>
|
||||
<p>Saving uses your selected timezone and follows its daylight saving changes. If the time is skipped, the queue starts at the next valid minute; repeated times run once.</p>
|
||||
{#if data.schedule.enabled && data.schedule.timeZone !== $timeZone}<p>The saved schedule is {data.schedule.time} in {data.schedule.timeZone}. Save to use {$timeZone} instead.</p>{/if}
|
||||
<p>This restarts all fleet servers daily, even without a new release. Busy servers may restart later. Disabling the schedule stops future runs; cancel existing queued restarts below.</p>
|
||||
{#if data.schedule.nextAt}<p>Next run: {$timeFormat.dateTime(data.schedule.nextAt)}</p>{/if}
|
||||
<div class="table-wrap"><table>
|
||||
<thead><tr><th>Server</th><th>Running revision</th><th>Update status</th><th>Action</th></tr></thead>
|
||||
<tbody>{#each data.servers as server (server.server)}
|
||||
<tr><td><a href={`/servers/${encodeURIComponent(server.server)}`}>{server.server}</a></td>
|
||||
<td>{server.revision ? `r${server.revision}` : 'Unknown'}
|
||||
{#if server.checked_at}<small>Checked {$timeFormat.dateTime(server.checked_at)}</small>{/if}
|
||||
{#if server.error}<small>{server.error}</small>{/if}
|
||||
</td>
|
||||
<td>{server.revision === null || data.release.revision === null || data.release.error ? 'Unverified'
|
||||
: server.revision < data.release.revision ? 'Update available' : server.revision > data.release.revision ? 'Ahead of published release' : 'Up to date'}</td>
|
||||
<td><Button size="sm" disabled={busy || pending(server.server)} onclick={() => refresh({ action: 'restart', servers: [server.server] })}>
|
||||
{pending(server.server) ? 'Restart pending' : 'Queue restart'}
|
||||
</Button></td></tr>
|
||||
{:else}<tr><td colspan="4">No fleet servers yet.</td></tr>{/each}</tbody>
|
||||
</table></div>
|
||||
{#if data.servers.length}<Button disabled={busy || data.servers.every(s => pending(s.server))} onclick={() => refresh({ action: 'restart', servers: data!.servers.map(s => s.server) })}>Queue fleet restart</Button>{/if}
|
||||
{#if data.jobs.length}
|
||||
<h3>Restart activity</h3>
|
||||
<ul>{#each data.jobs as job (job.id)}<li>
|
||||
<strong>{job.server}</strong> · {job.state} · {$timeFormat.dateTime(job.requested_at)}
|
||||
{#if job.target_revision} · Target r{job.target_revision}{/if}
|
||||
{#if job.detail}<small>{job.detail}</small>{/if}
|
||||
{#if job.provisioning_id}<a href="/provisioning/{encodeURIComponent(job.provisioning_id)}">View provisioning</a>{/if}
|
||||
{#if job.state === 'queued'}<Button size="sm" disabled={busy} onclick={() => refresh({ action: 'cancel', id: job.id })}>Cancel</Button>{/if}
|
||||
</li>{/each}</ul>
|
||||
{/if}
|
||||
{:else}<p role="status">{busy ? 'Loading maintenance status…' : 'Maintenance status unavailable.'}</p>{/if}
|
||||
</section>
|
||||
|
||||
<style>
|
||||
section { padding: 20px; border: 1px solid var(--border-strong); border-radius: var(--radius-md); background: var(--surface); scroll-margin-top: 90px; }
|
||||
.heading, form { display: flex; flex-wrap: wrap; gap: 16px; align-items: center; justify-content: space-between; }
|
||||
h2 { font-size: var(--text-md); margin: 0; }
|
||||
h3 { font-size: var(--text-sm); margin-top: 24px; }
|
||||
p, td, th, label, li { font-size: var(--text-sm); line-height: 1.6; }
|
||||
p, small { color: var(--text-secondary); }
|
||||
label { display: grid; gap: 6px; }
|
||||
.toggle { display: flex; align-items: center; }
|
||||
input[type='time'] { padding: 8px; color: var(--text-primary); background: var(--surface); border: 1px solid var(--border-strong); border-radius: var(--radius-sm); }
|
||||
.table-wrap { overflow-x: auto; margin: 16px 0; }
|
||||
table { width: 100%; border-collapse: collapse; text-align: left; }
|
||||
th, td { padding: 10px; border-bottom: 1px solid var(--border-strong); }
|
||||
small { display: block; }
|
||||
ul { padding-left: 20px; }
|
||||
li { margin-bottom: 12px; }
|
||||
</style>
|
||||
@@ -0,0 +1,18 @@
|
||||
<script lang="ts">
|
||||
import type { PlutoniumVersion } from '$lib/plutonium.js';
|
||||
import { uiNow } from '$ui/stores/servers.js';
|
||||
let { name, version, after, ready = true }: { name: string; version?: PlutoniumVersion; after?: string | null; ready?: boolean } = $props();
|
||||
const fresh = $derived(ready && version?.checked_at && $uiNow - Date.parse(version.checked_at) < 3_600_000 && (!after || version.checked_at > after));
|
||||
const revision = $derived(fresh ? version?.revision : null);
|
||||
const outdated = $derived(fresh && version?.update_status === 'outdated');
|
||||
</script>
|
||||
|
||||
<a class:outdated href={`/maintenance?kind=updates&server=${encodeURIComponent(name)}`} title="View Plutonium updates and verification">
|
||||
Plutonium {revision ? `r${revision}` : 'unverified'}{outdated ? ' · Update available' : ''}
|
||||
</a>
|
||||
|
||||
<style>
|
||||
a { font-size: var(--text-xs); color: var(--text-secondary); text-decoration: none; }
|
||||
a:hover { color: var(--accent-primary-hover); text-decoration: underline; }
|
||||
.outdated { color: var(--state-warning); }
|
||||
</style>
|
||||
@@ -13,6 +13,7 @@ export interface WatchdogIncident {
|
||||
finished_at: string | null;
|
||||
failures: number;
|
||||
cycle: number;
|
||||
operation_stage?: string | null;
|
||||
}
|
||||
export interface WatchdogEvidence {
|
||||
reason?: string;
|
||||
|
||||
@@ -52,7 +52,7 @@
|
||||
// Two-key nav shortcuts (Gmail-style): `g f` fleet, `g d` provisioning,
|
||||
// `g b` betas, `g s` scripts. Ignored while typing in inputs.
|
||||
const shortcuts: Record<string, string> = {
|
||||
f: '/', d: '/provisioning', a: '/campaigns', w: '/watchdog', b: '/betas',
|
||||
f: '/', d: '/provisioning', a: '/campaigns', w: '/maintenance', b: '/betas',
|
||||
k: '/server-keys', s: '/scripts', c: '/configs', p: '/plugins', l: '/leaderboard', h: '/settings', m: '/metrics'
|
||||
};
|
||||
let armed = false;
|
||||
@@ -99,7 +99,7 @@
|
||||
const tabs = [
|
||||
{ href: '/', label: 'Fleet', match: (p: string) => p === '/' || p.startsWith('/servers/') },
|
||||
{ href: '/scripts', label: 'Library', match: (p: string) => /^\/(scripts|configs|plugins)(\/|$)/.test(p) },
|
||||
{ href: '/provisioning', label: 'Operations', match: (p: string) => /^\/(provisioning|deployments|collections|campaigns|watchdog|server-keys)(\/|$)/.test(p) },
|
||||
{ href: '/provisioning', label: 'Operations', match: (p: string) => /^\/(provisioning|deployments|collections|campaigns|maintenance|watchdog|server-keys)(\/|$)/.test(p) },
|
||||
{ href: '/betas', label: 'Betas', match: (p: string) => p.startsWith('/betas') },
|
||||
{ href: '/settings', label: 'Settings', match: (p: string) => p.startsWith('/settings') },
|
||||
{ href: '/leaderboard', label: 'Leaderboard', match: (p: string) => p.startsWith('/leaderboard') },
|
||||
@@ -109,7 +109,7 @@
|
||||
const secondary = $derived(tabs[1].match($page.url.pathname)
|
||||
? [{ href: '/scripts', label: 'Scripts' }, { href: '/configs', label: 'Configs' }, { href: '/plugins', label: 'Plugins' }]
|
||||
: tabs[2].match($page.url.pathname)
|
||||
? [{ href: '/provisioning', label: 'Provisioning' }, { href: '/collections', label: 'Collections' }, { href: '/campaigns', label: 'Campaigns' }, { href: '/watchdog', label: 'Watchdog' }, { href: '/server-keys', label: 'Keys' }]
|
||||
? [{ href: '/provisioning', label: 'Provisioning' }, { href: '/collections', label: 'Collections' }, { href: '/campaigns', label: 'Campaigns' }, { href: '/maintenance', label: 'Maintenance' }, { href: '/server-keys', label: 'Keys' }]
|
||||
: tabs[4].match($page.url.pathname) ? [{ href: '/settings', label: 'Preferences' }] : []);
|
||||
</script>
|
||||
|
||||
@@ -149,7 +149,7 @@
|
||||
<Confirmation />
|
||||
<NotificationHistory bind:open={notificationsOpen} />
|
||||
<Dialog bind:open={helpOpen} title="Keyboard shortcuts" description="Press g, then the destination key. Shortcuts are inactive while typing or using a dialog.">
|
||||
<dl class="shortcuts">{#each [['g f','Fleet'],['g s','Scripts'],['g c','Configs'],['g p','Plugins'],['g d','Provisioning'],['g a','Campaigns'],['g w','Watchdog'],['g b','Betas'],['g h','Settings'],['g k','Keys'],['g l','Leaderboard'],['g m','Metrics'],['?','This help']] as [keys,label]}<div><dt><kbd>{keys}</kbd></dt><dd>{label}</dd></div>{/each}</dl>
|
||||
<dl class="shortcuts">{#each [['g f','Fleet'],['g s','Scripts'],['g c','Configs'],['g p','Plugins'],['g d','Provisioning'],['g a','Campaigns'],['g w','Maintenance'],['g b','Betas'],['g h','Settings'],['g k','Keys'],['g l','Leaderboard'],['g m','Metrics'],['?','This help']] as [keys,label]}<div><dt><kbd>{keys}</kbd></dt><dd>{label}</dd></div>{/each}</dl>
|
||||
</Dialog>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
import { onMount, untrack } from 'svelte';
|
||||
import { afterNavigate, replaceState } from '$app/navigation';
|
||||
import { page } from '$app/stores';
|
||||
import RuntimeVersion from '$ui/components/RuntimeVersion.svelte';
|
||||
import ServerCard from '$ui/components/ServerCard.svelte';
|
||||
import DeployServerDialog from '$ui/components/DeployServerDialog.svelte';
|
||||
import Button from '$ui/components/primitives/Button.svelte';
|
||||
@@ -239,6 +240,7 @@
|
||||
{/snippet}
|
||||
|
||||
{#snippet serverActions(server: FleetServer)}
|
||||
<RuntimeVersion name={server.name} version={server.plutonium} after={server.telemetry_after} ready={server.health === 'online'} />
|
||||
<button class="pin" class:selected={pinned.includes(server.name)} aria-pressed={pinned.includes(server.name)} aria-label="Pin {serverTitle(server.name)}" onclick={() => togglePin(server.name)}>{pinned.includes(server.name) ? '★ Pinned' : '☆ Pin'}</button>
|
||||
{/snippet}
|
||||
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
import { json } from '@sveltejs/kit';
|
||||
import { maintenanceOverview } from '$lib/server/maintenance/index.js';
|
||||
import { pauseMaintenance, setMaintenancePolicy } from '$lib/server/maintenance/policies.js';
|
||||
import { watchdogTick } from '$lib/server/maintenance/recovery.js';
|
||||
import { cancelMaintenance, checkPlutonium, queueUpdates, retryMaintenance, saveMaintenanceSchedule } from '$lib/server/maintenance/runtime.js';
|
||||
import { validTimeZone } from '$lib/time-zone.js';
|
||||
import { kick } from '$lib/server/pipeline/worker.js';
|
||||
|
||||
export function GET() { return json(maintenanceOverview()); }
|
||||
export async function POST({ request }: { request: Request }) {
|
||||
const body = await request.json().catch(() => null);
|
||||
if (!body || typeof body !== 'object' || Array.isArray(body)) return json({ error: 'Invalid request.' }, { status: 400 });
|
||||
let notice = '';
|
||||
try {
|
||||
switch (body.action) {
|
||||
case 'pause':
|
||||
if (typeof body.paused !== 'boolean') throw new Error('Provide a paused boolean.');
|
||||
pauseMaintenance(body.paused);
|
||||
notice = body.paused ? 'Maintenance paused. Submitted operations continue recovery.' : 'Maintenance resumed.';
|
||||
break;
|
||||
case 'policy':
|
||||
if (!['health', 'drift', 'updates'].includes(body.policy) || typeof body.enabled !== 'boolean') throw new Error('Choose a policy and an enabled boolean.');
|
||||
setMaintenancePolicy(body.policy, body.enabled); notice = 'Policy saved.';
|
||||
break;
|
||||
case 'scan':
|
||||
await Promise.all([watchdogTick(true), checkPlutonium()]); notice = 'Checks finished.';
|
||||
break;
|
||||
case 'update':
|
||||
if (body.servers !== undefined && (!Array.isArray(body.servers) || !body.servers.length || body.servers.some((v: unknown) => typeof v !== 'string'))) throw new Error('Choose existing fleet servers.');
|
||||
notice = `${queueUpdates(body.servers)} update restart(s) queued. Occupied servers wait until empty.`;
|
||||
break;
|
||||
case 'schedule':
|
||||
if (typeof body.enabled !== 'boolean' || typeof body.time !== 'string' || !validTimeZone(body.timeZone)) throw new Error('Provide enabled, a time, and a valid timezone.');
|
||||
saveMaintenanceSchedule(body.enabled, body.time, body.timeZone); notice = 'Schedule saved. Existing requests are retained.';
|
||||
break;
|
||||
case 'cancel':
|
||||
if (!Number.isSafeInteger(body.id) || !cancelMaintenance(body.id)) return json({ error: 'Only unsubmitted restarts can be cancelled.' }, { status: 409 });
|
||||
notice = 'Queued restart cancelled.'; break;
|
||||
case 'retry':
|
||||
if (!Number.isSafeInteger(body.id)) throw new Error('Choose a maintenance activity.');
|
||||
notice = `${retryMaintenance(body.id)} restart(s) queued for verification.`; break;
|
||||
default: throw new Error('Unknown maintenance action.');
|
||||
}
|
||||
} catch (error) { return json({ error: (error as Error).message }, { status: 400 }); }
|
||||
void kick().catch(error => console.error('[maintenance] worker kick failed:', error));
|
||||
return json({ ...maintenanceOverview(), notice });
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
import { json } from '@sveltejs/kit';
|
||||
import { watchdogDetail, updateWatchdogIncident } from '$lib/server/maintenance/recovery.js';
|
||||
import { kick } from '$lib/server/pipeline/worker.js';
|
||||
|
||||
export function GET({ params }: { params: { id: string } }) {
|
||||
const detail = watchdogDetail(params.id);
|
||||
return detail ? json(detail) : json({ error: 'Maintenance finding not found.' }, { status: 404 });
|
||||
}
|
||||
export async function POST({ params, request }: { params: { id: string }; request: Request }) {
|
||||
const body = await request.json().catch(() => null);
|
||||
if (!body || !['retry','dismiss'].includes(body.action)) return json({ error: 'Choose retry or dismiss.' }, { status: 400 });
|
||||
const result = updateWatchdogIncident(params.id, body.action);
|
||||
if (!result.ok) return json({ error: result.error }, { status: result.status });
|
||||
void kick().catch(err => console.error('[maintenance] worker kick failed:', err));
|
||||
return json(watchdogDetail(params.id));
|
||||
}
|
||||
@@ -6,12 +6,15 @@ import { serverObservation } from '$lib/server/fleet/status.js';
|
||||
import { queueCreateServer } from '$lib/server/provisioning/index.js';
|
||||
import { config, komodoConfigured, rconConfigured } from '$lib/server/config.js';
|
||||
import type { GameMode, ZombieGame } from '$lib/server/fleet/map-aliases.js';
|
||||
import { runningVersions } from '$lib/server/integrations/plutonium.js';
|
||||
|
||||
export async function GET() {
|
||||
const rconHost = config.rcon.host;
|
||||
const versions = new Map(runningVersions().map(version => [version.server, version]));
|
||||
const servers = listServers().map((s) => {
|
||||
return {
|
||||
...serverObservation(s),
|
||||
plutonium: versions.get(s.name),
|
||||
// Derived endpoint — same logic as endpointFor() but without asking
|
||||
// Komodo. If we don't have a port yet, endpoint is null.
|
||||
endpoint: s.game_port ? { host: s.rcon_host ?? rconHost, port: s.game_port } : null,
|
||||
|
||||
@@ -122,9 +122,9 @@
|
||||
{/snippet}
|
||||
</SectionHeader>
|
||||
|
||||
<section class="outcome" aria-label="Campaign outcome"><h2>{completed} / {detail.servers.length} servers activated{unresolved ? ` · ${unresolved} unresolved` : ''}</h2><p>{detail.watchdogIncidentId ? 'Watchdog repairs drift by deleting the existing instance files and rebuilding with the latest promoted setup after verifying a safe server state.' : 'Files are staged before activation. Servers restart when empty, then their health is verified.'}</p>{#if unresolved}<p>Review failed or waiting servers below. Diagnose connectivity or inspect startup logs from the affected row.</p>{:else}<p>Every recorded server completed activation.</p>{/if}</section>
|
||||
<section class="outcome" aria-label="Campaign outcome"><h2>{completed} / {detail.servers.length} servers activated{unresolved ? ` · ${unresolved} unresolved` : ''}</h2><p>{detail.watchdogIncidentId ? 'Maintenance repairs drift by deleting the existing instance files and rebuilding with the latest promoted setup after verifying a safe server state.' : 'Files are staged before activation. Servers restart when empty, then their health is verified.'}</p>{#if unresolved}<p>Review failed or waiting servers below. Diagnose connectivity or inspect startup logs from the affected row.</p>{:else}<p>Every recorded server completed activation.</p>{/if}</section>
|
||||
|
||||
{#if detail.watchdogIncidentId}<p class="watchdog-source">Started by Watchdog · <a href={`/watchdog/${encodeURIComponent(detail.watchdogIncidentId)}`}>View incident and recovery history</a></p>{/if}
|
||||
{#if detail.watchdogIncidentId}<p class="watchdog-source">Started by Maintenance · <a href={`/maintenance/${encodeURIComponent(detail.watchdogIncidentId)}`}>View incident and recovery history</a></p>{/if}
|
||||
|
||||
<Card padding="lg">
|
||||
<div class="meta-grid">
|
||||
|
||||
@@ -0,0 +1,221 @@
|
||||
<script lang="ts">
|
||||
import { onMount } from 'svelte';
|
||||
import { page } from '$app/stores';
|
||||
import { goto } from '$app/navigation';
|
||||
import type { MaintenanceOverview, MaintenancePolicy } from '$lib/maintenance.js';
|
||||
import { maintenanceTimeInZone, PLUTONIUM_MANIFEST_URL } from '$lib/plutonium.js';
|
||||
import { watchdogStateLabels } from '$lib/watchdog.js';
|
||||
import { pollVisible } from '$ui/poll.js';
|
||||
import { timeFormat, timeZone } from '$ui/stores/time-zone.js';
|
||||
import { serverTitle } from '$ui/server-labels.js';
|
||||
import ServerIdentity from '$ui/components/ServerIdentity.svelte';
|
||||
import SectionHeader from '$ui/components/primitives/SectionHeader.svelte';
|
||||
import Tabs from '$ui/components/primitives/Tabs.svelte';
|
||||
import Badge from '$ui/components/primitives/Badge.svelte';
|
||||
import Button from '$ui/components/primitives/Button.svelte';
|
||||
import EmptyState from '$ui/components/primitives/EmptyState.svelte';
|
||||
import ListFooter from '$ui/components/primitives/ListFooter.svelte';
|
||||
|
||||
let data = $state<MaintenanceOverview | null>(null);
|
||||
let error = $state('');
|
||||
let notice = $state('');
|
||||
let busy = $state(false);
|
||||
let refreshing = false;
|
||||
let view = $state('overview');
|
||||
let kind = $state('all');
|
||||
let server = $state('');
|
||||
let status = $state('all');
|
||||
let limit = $state(25);
|
||||
let scheduleEnabled = $state(false);
|
||||
let scheduleTime = $state('06:00');
|
||||
let scheduleLoaded = false;
|
||||
let displayedZone = $state('');
|
||||
let request = 0;
|
||||
const tabs = [{ id: 'overview', label: 'Overview' }, { id: 'policies', label: 'Policies & schedules' }, { id: 'activity', label: 'Activity' }];
|
||||
const policies: { id: MaintenancePolicy; title: string; description: string }[] = [
|
||||
{ id: 'health', title: 'Health recovery', description: 'Restart unhealthy servers, then rebuild once if recovery fails. Rebuilding deletes instance files and installs the latest promoted setup.' },
|
||||
{ id: 'drift', title: 'Configuration drift', description: 'Rebuild servers whose managed files differ from the promoted setup. Record the differences in a linked Campaign. Rebuilding deletes instance files.' },
|
||||
{ id: 'updates', title: 'Automatic Plutonium updates', description: 'Update outdated running servers on restart. One automatic attempt per published revision; failed verification requires inspection. Disabled by default.' }
|
||||
];
|
||||
const versions = $derived(data?.runtime.servers.filter(v => !server || `${v.server} ${serverTitle(v.server)}`.toLowerCase().includes(server.toLowerCase())) ?? []);
|
||||
const outdated = $derived(versions.filter(v => v.update_status === 'outdated'));
|
||||
const unknown = $derived(versions.filter(v => v.update_status === 'unknown'));
|
||||
const activeJobs = $derived(data?.runtime.jobs.filter(j => ['queued', 'restarting'].includes(j.state)) ?? []);
|
||||
const activities = $derived([
|
||||
...(data?.recovery.incidents ?? []).map(i => ({
|
||||
key: i.id, kind: i.kind, server: i.server, at: i.first_seen_at,
|
||||
title: i.kind === 'health' ? 'Health recovery' : 'Configuration drift',
|
||||
state: ['open', 'recovering'].includes(i.state) ? 'active' : i.state === 'attention' ? 'attention' : 'done',
|
||||
label: watchdogStateLabels[i.state], detail: i.state === 'recovering' ? i.operation_stage ?? i.summary : i.summary, href: `/maintenance/${encodeURIComponent(i.id)}`, job: null
|
||||
})),
|
||||
...(data?.runtime.jobs ?? []).map(j => ({
|
||||
key: `job-${j.id}`, kind: j.kind === 'schedule' ? 'schedule' : 'updates', server: j.server, at: j.requested_at,
|
||||
title: j.kind === 'schedule' ? 'Scheduled restart' : j.kind === 'update' ? 'Plutonium update' : 'Legacy maintenance',
|
||||
state: ['queued', 'restarting'].includes(j.state) ? 'active' : j.state === 'failed' ? 'attention' : 'done',
|
||||
label: j.state === 'failed' ? 'Needs attention' : j.state === 'done' ? 'Completed' : j.state === 'cancelled' ? 'Cancelled' : j.state === 'restarting' ? 'Recovering' : 'Queued',
|
||||
detail: ['queued', 'restarting'].includes(j.state) ? j.operation_stage ?? j.detail : j.detail,
|
||||
href: j.provisioning_id ? `/provisioning/${encodeURIComponent(j.provisioning_id)}` : '', job: j
|
||||
}))
|
||||
].sort((a, b) => b.at.localeCompare(a.at)));
|
||||
const isLatest = (item: (typeof activities)[number]) => !item.job || !data?.runtime.jobs.some(j => j.server === item.server && j.kind === item.job!.kind && j.id > item.job!.id);
|
||||
const attention = $derived(activities.filter(a => a.state === 'attention' && isLatest(a)).length);
|
||||
const waiting = $derived(activities.filter(a => a.state === 'active').length);
|
||||
const matching = $derived(activities.filter(a =>
|
||||
(kind === 'all' || a.kind === kind) &&
|
||||
(!server || `${a.server} ${serverTitle(a.server)}`.toLowerCase().includes(server.toLowerCase())) &&
|
||||
(view === 'overview' ? a.state !== 'done' && isLatest(a) : status === 'all' || a.state === status)
|
||||
));
|
||||
$effect(() => {
|
||||
const params = $page.url.searchParams;
|
||||
view = ['overview', 'policies', 'activity'].includes(params.get('view') ?? '') ? params.get('view')! : 'overview';
|
||||
kind = ['health', 'drift', 'updates', 'schedule'].includes(params.get('kind') ?? '') ? params.get('kind')! : 'all';
|
||||
server = params.get('server') ?? '';
|
||||
status = ['active', 'attention', 'done'].includes(params.get('status') ?? '') ? params.get('status')! : 'all';
|
||||
limit = Math.max(25, Number(params.get('limit')) || 25);
|
||||
});
|
||||
$effect(() => {
|
||||
if (data && displayedZone !== $timeZone) {
|
||||
scheduleTime = maintenanceTimeInZone(data.runtime.schedule, $timeZone);
|
||||
displayedZone = $timeZone;
|
||||
}
|
||||
});
|
||||
function saveContext(reset = true) {
|
||||
if (reset) limit = 25;
|
||||
const url = new URL($page.url);
|
||||
for (const [key, value] of Object.entries({ view: view === 'overview' ? '' : view, kind: kind === 'all' ? '' : kind, server, status: status === 'all' ? '' : status, limit: limit > 25 ? String(limit) : '' })) {
|
||||
if (value) url.searchParams.set(key, value); else url.searchParams.delete(key);
|
||||
}
|
||||
void goto(url, { replaceState: true, noScroll: true, keepFocus: true });
|
||||
}
|
||||
function receive(body: MaintenanceOverview) {
|
||||
data = body;
|
||||
if (!scheduleLoaded) { scheduleEnabled = body.runtime.schedule.enabled; scheduleLoaded = true; }
|
||||
}
|
||||
async function load() {
|
||||
if (busy || refreshing) return;
|
||||
refreshing = true;
|
||||
const current = ++request;
|
||||
try {
|
||||
const response = await fetch('/api/maintenance');
|
||||
if (!response.ok) throw new Error('Could not refresh maintenance.');
|
||||
const body = await response.json();
|
||||
if (current === request) { receive(body); error = ''; }
|
||||
} catch (cause) { if (current === request) error = (cause as Error).message; }
|
||||
finally { refreshing = false; }
|
||||
}
|
||||
async function act(body: Record<string, unknown>) {
|
||||
if (busy) return;
|
||||
busy = true; notice = ''; error = ''; request++;
|
||||
try {
|
||||
const response = await fetch('/api/maintenance', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
|
||||
const result = await response.json();
|
||||
if (!response.ok) throw new Error(result.error ?? 'Could not update maintenance.');
|
||||
receive(result); notice = result.notice;
|
||||
} catch (cause) { error = (cause as Error).message; }
|
||||
finally { busy = false; }
|
||||
}
|
||||
onMount(() => pollVisible(load, 5000));
|
||||
</script>
|
||||
|
||||
<svelte:head><title>Maintenance | Electric Cherry</title></svelte:head>
|
||||
<div class="page">
|
||||
<SectionHeader title="Maintenance" subtitle="Keep servers healthy, current, and ready for the next match.">
|
||||
{#snippet actions()}
|
||||
<Button disabled={busy} onclick={() => act({ action: 'scan' })}>Check now</Button>
|
||||
{#if data}<Button variant="ghost" disabled={busy} onclick={() => act({ action: 'pause', paused: !data!.policies.paused })}>{data.policies.paused ? 'Resume maintenance' : 'Pause all maintenance'}</Button>{/if}
|
||||
{/snippet}
|
||||
</SectionHeader>
|
||||
{#if error}<p class="ui-error" role="alert">{error} {data ? 'Showing the last successful refresh.' : ''} <Button size="sm" onclick={load} disabled={busy}>Retry refresh</Button></p>{/if}
|
||||
{#if notice}<p role="status">{notice}</p>{/if}
|
||||
{#if data?.policies.paused}<div class="paused"><strong>All maintenance is paused.</strong> Checks and verification continue. New actions, scheduled runs, and queued maintenance are held. Submitted operations finish recovery. Campaigns, betas, and manual Provisioning actions continue.</div>{/if}
|
||||
<Tabs {tabs} bind:value={view} onchange={() => saveContext()} ariaLabel="Maintenance views" idPrefix="maintenance-tabs" panelId="maintenance-panel" />
|
||||
<div id="maintenance-panel" role="tabpanel" aria-labelledby={`maintenance-tabs-${view}`} class="panels">
|
||||
{#if data}
|
||||
{#if view === 'policies'}
|
||||
<p class="muted">Policies apply to the current fleet and respect intentionally stopped servers. Disabling a recovery or automatic update policy holds its queued actions. Already submitted operations continue verification.</p>
|
||||
<div class="policies">{#each policies as policy}
|
||||
<section class="card">
|
||||
<div class="row-heading"><h2>{policy.title}</h2><Badge variant={data.policies[policy.id] ? 'info' : 'neutral'}>{data.policies[policy.id] ? 'Enabled' : 'Disabled'}</Badge></div>
|
||||
<p>{policy.description}</p>
|
||||
<Button disabled={busy} onclick={() => act({ action: 'policy', policy: policy.id, enabled: !data!.policies[policy.id] })}>{data.policies[policy.id] ? 'Disable' : 'Enable'} {policy.title.toLowerCase()}</Button>
|
||||
</section>
|
||||
{/each}</div>
|
||||
<section class="card" id="schedule">
|
||||
<h2>Scheduled restarts</h2>
|
||||
<p>Restart the running fleet daily, even without a new release. Intentionally stopped servers are excluded. Busy servers wait until empty and may restart later.</p>
|
||||
<form onsubmit={event => { event.preventDefault(); void act({ action: 'schedule', enabled: scheduleEnabled, time: scheduleTime, timeZone: $timeZone }); }}>
|
||||
<label class="toggle"><input type="checkbox" bind:checked={scheduleEnabled} /> Daily fleet restart</label>
|
||||
<label>Queue time ({$timeZone})<input type="time" bind:value={scheduleTime} required /></label>
|
||||
<Button type="submit" disabled={busy}>Save schedule</Button>
|
||||
</form>
|
||||
<p class="muted">Saving follows daylight saving changes in your selected timezone. Skipped times run at the next valid minute; repeated times run once. Missed days become one run on resume.</p>
|
||||
{#if data.runtime.schedule.enabled && data.runtime.schedule.timeZone !== $timeZone}<p>Saved for {data.runtime.schedule.time} in {data.runtime.schedule.timeZone}. Save to use {$timeZone}.</p>{/if}
|
||||
{#if data.runtime.schedule.nextAt}<p>Next run: {$timeFormat.dateTime(data.runtime.schedule.nextAt)}{data.policies.paused ? ' · Held while maintenance is paused' : ''}</p>{/if}
|
||||
<p class="muted">Disabling the schedule stops future runs. Existing requests remain in Activity, where unsubmitted restarts can be cancelled.</p>
|
||||
</section>
|
||||
{:else}
|
||||
{#if view === 'overview'}
|
||||
<div class="summary"><span><strong>{attention}</strong> need attention</span><span><strong>{waiting}</strong> active</span><span><strong>{outdated.length}</strong> outdated</span><span><strong>{unknown.length}</strong> versions unverified</span></div>
|
||||
{#if data.runtime.schedule.nextAt}<p class="muted">Next scheduled restart: {$timeFormat.dateTime(data.runtime.schedule.nextAt)} · <a href="/maintenance?view=policies#schedule">Manage schedule</a></p>{/if}
|
||||
{#if data.recovery.lastScanAt}<p class="muted">Health and drift checked {$timeFormat.dateTime(data.recovery.lastScanAt)}</p>{/if}
|
||||
{#if data.recovery.scanError}<p class="ui-error" role="alert">{data.recovery.scanError}</p>{/if}
|
||||
{/if}
|
||||
<div class="filters">
|
||||
<label>Capability<select bind:value={kind} onchange={() => saveContext()}><option value="all">All capabilities</option><option value="health">Health recovery</option><option value="drift">Configuration drift</option><option value="updates">Plutonium updates</option><option value="schedule">Scheduled restarts</option></select></label>
|
||||
<label>Server<input bind:value={server} oninput={() => saveContext()} placeholder="Filter servers" /></label>
|
||||
{#if view === 'activity'}<label>Status<select bind:value={status} onchange={() => saveContext()}><option value="all">All activity</option><option value="active">Active</option><option value="attention">Needs attention</option><option value="done">Finished</option></select></label>{/if}
|
||||
</div>
|
||||
{#if view === 'overview' && ['all', 'updates'].includes(kind)}
|
||||
<section class="card" id="updates">
|
||||
<div class="row-heading"><h2>Plutonium updates</h2><Button disabled={busy || data.policies.paused || !outdated.some(v => !activeJobs.some(j => j.server === v.server))} onclick={() => act({ action: 'update', servers: outdated.map(v => v.server) })}>Update outdated servers on restart</Button></div>
|
||||
<p>Published revision: <strong>{data.runtime.release.revision ? `r${data.runtime.release.revision}` : 'Unknown'}</strong> · <a href={PLUTONIUM_MANIFEST_URL} target="_blank" rel="noreferrer">Production manifest</a> · <a href="https://plutonium.pw/docs/changelog/" target="_blank" rel="noreferrer">Changelog</a></p>
|
||||
{#if data.runtime.release.checkedAt}<p class="muted">Checked {$timeFormat.dateTime(data.runtime.release.checkedAt)}</p>{/if}
|
||||
{#if data.runtime.release.error}<p class="ui-error" role="alert">{data.runtime.release.error} Displayed release information is cached.</p>{/if}
|
||||
<p class="muted">Plutainer applies the available release at startup when auto-update is enabled. Maintenance verifies the running revision after recovery. Unknown versions require a successful check before automatic updates.</p>
|
||||
<div class="table-wrap"><table><thead><tr><th>Server</th><th>Running revision</th><th>Status</th><th>Action</th></tr></thead><tbody>
|
||||
{#each versions as version (version.server)}
|
||||
{@const pending = activeJobs.find(j => j.server === version.server)}
|
||||
<tr><td><a href={`/servers/${encodeURIComponent(version.server)}`}><ServerIdentity name={version.server} /></a></td><td>{version.revision ? `r${version.revision}` : 'Unknown'}{#if version.checked_at}<small>{$timeFormat.dateTime(version.checked_at)}</small>{/if}{#if version.error}<small>{version.error}</small>{/if}</td>
|
||||
<td>{version.update_status === 'outdated' ? 'Update available' : version.update_status === 'current' ? 'Up to date' : version.update_status === 'ahead' ? 'Ahead of published release' : version.update_status === 'stopped' ? 'Intentionally stopped' : 'Unverified'}</td>
|
||||
<td>{#if pending}<a href={pending.provisioning_id ? `/provisioning/${encodeURIComponent(pending.provisioning_id)}` : '#activity'}>Restart pending</a>{:else if version.update_status === 'outdated'}<Button size="sm" disabled={busy || data.policies.paused} onclick={() => act({ action: 'update', servers: [version.server] })}>Update on restart</Button>{/if}</td></tr>
|
||||
{:else}<tr><td colspan="4">No matching fleet servers.</td></tr>{/each}
|
||||
</tbody></table></div>
|
||||
</section>
|
||||
{/if}
|
||||
<section id="activity">
|
||||
<h2>{view === 'overview' ? 'Active work & attention' : 'Maintenance activity'}</h2>
|
||||
<div class="activity">{#each matching.slice(0, limit) as item (item.key)}
|
||||
<article class="card" id={item.key}>
|
||||
<div class="row-heading"><h3>{item.title} · <ServerIdentity name={item.server} /></h3><Badge variant={item.state === 'attention' ? 'danger' : item.state === 'active' ? 'info' : 'neutral'}>{item.label}</Badge></div>
|
||||
{#if item.detail}<p>{item.detail}</p>{/if}
|
||||
<p class="muted">{$timeFormat.dateTime(item.at)}{item.job?.target_revision ? ` · Expected r${item.job.target_revision} or newer` : ''}</p>
|
||||
<div class="links">{#if item.href}<a href={item.href}>{item.job ? 'View restart operation' : 'View evidence and recovery'}</a>{/if}
|
||||
{#if item.job?.state === 'queued'}<Button size="sm" disabled={busy} onclick={() => act({ action: 'cancel', id: item.job!.id })}>Cancel restart</Button>{/if}
|
||||
{#if item.job?.state === 'failed' && item.job.operation_state === 'done'}<Button size="sm" disabled={busy || data.policies.paused} onclick={() => act({ action: 'retry', id: item.job!.id })}>Retry after inspection</Button>{/if}
|
||||
</div>
|
||||
</article>
|
||||
{:else}<EmptyState title={view === 'overview' ? 'No active maintenance in this view' : 'No matching activity'} description="Findings, scheduled restarts, and update verification appear here." />{/each}</div>
|
||||
<ListFooter total={matching.length} bind:limit onchange={() => saveContext(false)} label="activities" />
|
||||
</section>
|
||||
{/if}
|
||||
{:else if !error}<p role="status">Loading maintenance…</p>{/if}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.page, .panels, section, .activity, .policies { display: grid; gap: 20px; }
|
||||
.page { gap: 28px; }
|
||||
.card { padding: 24px; border: 1px solid var(--border-subtle); border-radius: var(--radius-md); background: var(--surface); display: grid; gap: 16px; }
|
||||
.row-heading, .summary, .filters, form, .links { display: flex; gap: 16px 24px; align-items: center; flex-wrap: wrap; }
|
||||
.row-heading { justify-content: space-between; }
|
||||
.summary, .paused { padding: 20px; background: var(--surface-raised); border: 1px solid var(--border-strong); border-radius: var(--radius-md); }
|
||||
.paused { line-height: 1.7; }
|
||||
h2 { font-size: var(--text-md); } h3 { font-size: var(--text-sm); } h2, h3, p { margin: 0; }
|
||||
p, label, td, th { font-size: var(--text-sm); line-height: 1.6; }
|
||||
.muted, small { color: var(--text-secondary); } small { display: block; }
|
||||
a { color: var(--accent-primary-hover); } label { display: grid; gap: 6px; } .toggle { display: flex; align-items: center; }
|
||||
input:not([type='checkbox']), select { color: inherit; background: var(--surface); padding: 9px; border: 1px solid var(--border-strong); border-radius: var(--radius-sm); max-width: 100%; }
|
||||
.table-wrap { overflow-x: auto; } table { width: 100%; border-collapse: collapse; text-align: left; } td, th { padding: 12px 8px; border-bottom: 1px solid var(--border-subtle); }
|
||||
.activity article, section { scroll-margin-top: 100px; }
|
||||
@media (max-width: 640px) { .card { padding: 18px; } }
|
||||
</style>
|
||||
@@ -41,7 +41,7 @@
|
||||
if (incident.state === 'dismissed') return 'This incident was dismissed. Retry to allow automatic recovery again.';
|
||||
if (incident.state === 'attention') return 'Automatic recovery needs attention. Review the failed operation and server diagnostics before retrying.';
|
||||
if (incident.state === 'recovering') return 'Recovery is in progress. Follow the provisioning operations below for eligibility checks, progress, and health verification.';
|
||||
return 'Watchdog detected this problem. Automatic recovery requires a confirmed stopped container or a fresh zero-player count from RCON or telemetry before interrupting the server.';
|
||||
return 'Maintenance detected this problem. Automatic recovery requires a confirmed stopped container or a fresh zero-player count from RCON or telemetry before interrupting the server.';
|
||||
});
|
||||
|
||||
async function load() {
|
||||
@@ -49,8 +49,8 @@
|
||||
const id = $page.params.id ?? '';
|
||||
const current = ++request;
|
||||
try {
|
||||
const response = await fetch(`/api/watchdog/${encodeURIComponent(id)}`);
|
||||
if (!response.ok) throw new Error(response.status === 404 ? 'Watchdog incident not found.' : 'Could not refresh this incident.');
|
||||
const response = await fetch(`/api/maintenance/${encodeURIComponent(id)}`);
|
||||
if (!response.ok) throw new Error(response.status === 404 ? 'Maintenance finding not found.' : 'Could not refresh this incident.');
|
||||
const body = await response.json() as WatchdogDetail;
|
||||
if (current !== request || id !== $page.params.id) return;
|
||||
detail = body; loadError = '';
|
||||
@@ -65,14 +65,14 @@
|
||||
const current = ++request;
|
||||
busy = action; actionError = ''; notice = '';
|
||||
try {
|
||||
const response = await fetch(`/api/watchdog/${encodeURIComponent(id)}`, {
|
||||
const response = await fetch(`/api/maintenance/${encodeURIComponent(id)}`, {
|
||||
method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ action })
|
||||
});
|
||||
const body = await response.json();
|
||||
if (!response.ok) throw new Error(body.error ?? 'Could not update this incident.');
|
||||
if (current !== request || id !== $page.params.id) return;
|
||||
detail = body; loadError = '';
|
||||
notice = action === 'retry' ? 'Recovery retry requested. Watchdog will recheck the server before taking action.' : 'Incident dismissed.';
|
||||
notice = action === 'retry' ? 'Recovery retry requested. Maintenance will recheck the server before taking action.' : 'Incident dismissed.';
|
||||
} catch (error) {
|
||||
if (current === request && id === $page.params.id) actionError = (error as Error).message;
|
||||
} finally { if (current === request) busy = null; }
|
||||
@@ -90,11 +90,11 @@
|
||||
});
|
||||
</script>
|
||||
|
||||
<svelte:head><title>{incident ? `${serverTitle(incident.server)} · Watchdog` : 'Watchdog incident'} | Electric Cherry</title></svelte:head>
|
||||
<svelte:head><title>{incident ? `${serverTitle(incident.server)} · Maintenance` : 'Maintenance finding'} | Electric Cherry</title></svelte:head>
|
||||
|
||||
<div class="page">
|
||||
<SectionHeader title={incident ? `${incident.kind === 'drift' ? 'File drift' : 'Server health'} · ${serverTitle(incident.server)}` : 'Watchdog incident'} subtitle={incident?.id}>
|
||||
{#snippet crumbs()}<Breadcrumbs items={[{ label: 'Watchdog', href: '/watchdog' }, { label: 'Incident' }]} />{/snippet}
|
||||
<SectionHeader title={incident ? `${incident.kind === 'drift' ? 'File drift' : 'Server health'} · ${serverTitle(incident.server)}` : 'Maintenance finding'} subtitle={incident?.id}>
|
||||
{#snippet crumbs()}<Breadcrumbs items={[{ label: 'Maintenance', href: '/maintenance' }, { label: 'Incident' }]} />{/snippet}
|
||||
{#snippet actions()}
|
||||
{#if canRetry}<Button disabled={!!busy || activeOperation} loading={busy === 'retry'} onclick={() => update('retry')}>Retry recovery</Button>{/if}
|
||||
{#if canDismiss}<Button variant="ghost" disabled={!!busy || activeOperation} loading={busy === 'dismiss'} onclick={() => update('dismiss')}>Dismiss incident</Button>{/if}
|
||||
@@ -152,7 +152,7 @@
|
||||
{#each warnings as warning}<p class="warning">{warning}</p>{/each}
|
||||
{#if deployment.expected_map}<p>Expected map: <strong>{deployment.expected_map}</strong></p>{/if}
|
||||
<dl><div><dt>Started</dt><dd>{fmt(deployment.started_at)}</dd></div><div><dt>Finished</dt><dd>{deployment.finished_at ? fmt(deployment.finished_at) : 'In progress'}</dd></div><div><dt>Requested by</dt><dd>{deployment.triggered_by}</dd></div></dl>
|
||||
{#if detail?.sources.length}<div class="recovery-links" aria-label="Requesting work">{#each detail.sources as source}<a href={source.kind === 'campaign' ? `/campaigns/${encodeURIComponent(source.source_id)}` : source.kind === 'beta' ? `/betas/${encodeURIComponent(source.source_id)}` : source.kind === 'watchdog' ? `/watchdog/${encodeURIComponent(source.source_id)}` : '/settings#plutonium'}>{source.kind === 'campaign' ? 'Campaign' : source.kind === 'beta' ? 'Beta' : source.kind === 'watchdog' ? 'Watchdog incident' : 'Maintenance'} {source.source_id}</a>{/each}</div>{/if}
|
||||
{#if detail?.sources.length}<div class="recovery-links" aria-label="Requesting work">{#each detail.sources as source}<a href={source.kind === 'campaign' ? `/campaigns/${encodeURIComponent(source.source_id)}` : source.kind === 'beta' ? `/betas/${encodeURIComponent(source.source_id)}` : source.kind === 'watchdog' ? `/maintenance/${encodeURIComponent(source.source_id)}` : `/maintenance?view=activity#job-${source.source_id}`}>{source.kind === 'campaign' ? 'Campaign' : source.kind === 'beta' ? 'Beta' : source.kind === 'watchdog' ? 'Maintenance finding' : 'Maintenance'} {source.source_id}</a>{/each}</div>{/if}
|
||||
</section>
|
||||
{#if detail?.recipe}<section class="saved-recipe"><div class="top"><h2>{detail.blueprintName ? `Launched from ${detail.blueprintName}` : 'Recorded launch files'}</h2><Button variant="ghost" onclick={() => open = true}>Launch another server →</Button></div><p class="muted">Launch this map with the latest promoted files using a fresh server ID and port. This does not repair the existing server.</p><LaunchRecipe recipe={detail.recipe} historical /></section><DeployServerDialog bind:open sourceDeploymentId={deployment.id} />{/if}
|
||||
<OperationHistory steps={steps.map((label,index): OperationStep => ({label,state:deployment.state === 'cancelled' ? 'pending' : index < currentStep ? 'done' : index === currentStep ? (!active && ['failed','interrupted'].includes(deployment.state) ? 'failed' : 'current') : 'pending'}))} events={detail?.events ?? []} {active} />
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
<script lang="ts">
|
||||
import RuntimeVersion from '$ui/components/RuntimeVersion.svelte';
|
||||
import MatchHistory from '$ui/components/MatchHistory.svelte';
|
||||
import RoundTelemetry from '$ui/components/RoundTelemetry.svelte';
|
||||
import TelemetryDiagnostics from '$ui/components/TelemetryDiagnostics.svelte';
|
||||
@@ -70,6 +71,7 @@
|
||||
status?: { players: number; map: string | null; gametype: string | null; roster: RosterPlayer[] };
|
||||
}
|
||||
interface Server {
|
||||
plutonium?: import('$lib/plutonium.js').PlutoniumVersion;
|
||||
telemetry_after?: string | null;
|
||||
status_source?: 'telemetry' | 'rcon' | null;
|
||||
rcon_status?: import('$lib/server-observation.js').GameObservation;
|
||||
@@ -251,6 +253,7 @@
|
||||
</SectionHeader>
|
||||
|
||||
<CopyValue value={name} label="Server ID" />
|
||||
<RuntimeVersion {name} version={server.plutonium} after={server.telemetry_after} ready={effectiveHealth === 'online'} />
|
||||
<div class="metrics">
|
||||
<div class="metric-cell name-cell">
|
||||
<StatusPip health={effectiveHealth} size="md" />
|
||||
|
||||
@@ -3,7 +3,6 @@
|
||||
import { availableTimeZones, DEFAULT_TIME_ZONE } from '$lib/time-zone.js';
|
||||
import { onMount } from 'svelte';
|
||||
import SectionHeader from '$ui/components/primitives/SectionHeader.svelte';
|
||||
import PlutoniumMaintenance from '$ui/components/PlutoniumMaintenance.svelte';
|
||||
let exampleDate = $state<Date | null>(null);
|
||||
const zones = $derived([...new Set([...availableTimeZones(), $timeZone])].sort());
|
||||
onMount(() => { exampleDate = new Date(); });
|
||||
@@ -23,7 +22,7 @@
|
||||
{#if exampleDate}<p>Example: {$timeFormat.dateTime(exampleDate)}</p>{/if}
|
||||
{#if $timeZonePersistenceError}<p role="status">Browser storage is unavailable. Your timezone applies for this visit but cannot be saved.</p>{/if}
|
||||
</section>
|
||||
<PlutoniumMaintenance />
|
||||
<p id="plutonium">Server updates and restart schedules are now in <a href="/maintenance">Operations → Maintenance</a>.</p>
|
||||
</section>
|
||||
|
||||
<style>
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
import { redirect } from '@sveltejs/kit';
|
||||
export function load({ url }: { url: URL }) { redirect(308, `/maintenance${url.search}`); }
|
||||
@@ -1,168 +0,0 @@
|
||||
<script lang="ts">
|
||||
import { onMount } from 'svelte';
|
||||
import { page } from '$app/stores';
|
||||
import { goto } from '$app/navigation';
|
||||
import { watchdogStateLabels, type WatchdogOverview, type WatchdogState } from '$lib/watchdog.js';
|
||||
import { pollVisible } from '$ui/poll.js';
|
||||
import { timeFormat } from '$ui/stores/time-zone.js';
|
||||
import { serverTitle } from '$ui/server-labels.js';
|
||||
import ServerIdentity from '$ui/components/ServerIdentity.svelte';
|
||||
import WatchdogStatus from '$ui/components/WatchdogStatus.svelte';
|
||||
import Badge from '$ui/components/primitives/Badge.svelte';
|
||||
import Button from '$ui/components/primitives/Button.svelte';
|
||||
import EmptyState from '$ui/components/primitives/EmptyState.svelte';
|
||||
import ListFooter from '$ui/components/primitives/ListFooter.svelte';
|
||||
import SectionHeader from '$ui/components/primitives/SectionHeader.svelte';
|
||||
|
||||
let overview = $state<WatchdogOverview | null>(null);
|
||||
let loadError = $state('');
|
||||
let actionError = $state('');
|
||||
let notice = $state('');
|
||||
let busy = $state<'scan' | 'toggle' | null>(null);
|
||||
let filterState = $state('active');
|
||||
let kind = $state('all');
|
||||
let server = $state('');
|
||||
let limit = $state(25);
|
||||
let request = 0;
|
||||
const states = Object.keys(watchdogStateLabels) as WatchdogState[];
|
||||
const activeStates = ['open', 'recovering', 'attention'];
|
||||
const active = $derived(overview?.incidents.filter(incident => activeStates.includes(incident.state)).length ?? 0);
|
||||
const attention = $derived(overview?.incidents.filter(incident => incident.state === 'attention').length ?? 0);
|
||||
const matching = $derived(overview?.incidents.filter(incident =>
|
||||
(filterState === 'all' || (filterState === 'active' ? activeStates.includes(incident.state) : incident.state === filterState)) &&
|
||||
(kind === 'all' || incident.kind === kind) &&
|
||||
(!server || `${incident.server} ${serverTitle(incident.server)}`.toLowerCase().includes(server.toLowerCase()))
|
||||
) ?? []);
|
||||
|
||||
$effect(() => {
|
||||
const params = $page.url.searchParams;
|
||||
const selectedState = params.get('state') ?? 'active';
|
||||
filterState = ['active', 'all', ...states].includes(selectedState) ? selectedState : 'active';
|
||||
kind = ['health', 'drift'].includes(params.get('kind') ?? '') ? params.get('kind')! : 'all';
|
||||
server = params.get('server') ?? '';
|
||||
limit = Math.max(25, Number(params.get('limit')) || 25);
|
||||
});
|
||||
|
||||
function saveContext(reset = false) {
|
||||
if (reset) limit = 25;
|
||||
const url = new URL($page.url);
|
||||
for (const [key, value] of Object.entries({ state: filterState === 'active' ? '' : filterState, kind: kind === 'all' ? '' : kind, server, limit: limit > 25 ? String(limit) : '' })) {
|
||||
if (value) url.searchParams.set(key, value); else url.searchParams.delete(key);
|
||||
}
|
||||
void goto(url, { replaceState: true, noScroll: true, keepFocus: true });
|
||||
}
|
||||
|
||||
async function load() {
|
||||
if (busy) return;
|
||||
const current = ++request;
|
||||
try {
|
||||
const response = await fetch('/api/watchdog');
|
||||
if (!response.ok) throw new Error('Could not refresh Watchdog.');
|
||||
const body = await response.json() as WatchdogOverview;
|
||||
if (current !== request) return;
|
||||
overview = body; loadError = '';
|
||||
} catch (error) {
|
||||
if (current === request) loadError = (error as Error).message;
|
||||
}
|
||||
}
|
||||
|
||||
async function update(action: 'scan' | 'toggle') {
|
||||
if (busy || !overview) return;
|
||||
busy = action; actionError = ''; notice = '';
|
||||
const current = ++request;
|
||||
const enabled = !overview.enabled;
|
||||
try {
|
||||
const response = await fetch('/api/watchdog', {
|
||||
method: 'POST', headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(action === 'scan' ? { action: 'scan' } : { enabled })
|
||||
});
|
||||
const body = await response.json();
|
||||
if (!response.ok) throw new Error(body.error ?? 'Could not update Watchdog.');
|
||||
if (current !== request) return;
|
||||
overview = body; loadError = '';
|
||||
notice = action === 'scan' ? (body.scanError ? 'Scan finished with errors. Review the scan details below.' : 'Scan complete.') : enabled ? 'Automatic recovery resumed.' : 'Automatic recovery paused. Queued Watchdog recovery is held; already submitted operations continue.';
|
||||
} catch (error) {
|
||||
if (current === request) actionError = (error as Error).message;
|
||||
} finally { if (current === request) busy = null; }
|
||||
}
|
||||
|
||||
onMount(() => {
|
||||
const stop = pollVisible(load, 5000);
|
||||
return () => { stop(); request++; };
|
||||
});
|
||||
</script>
|
||||
|
||||
<svelte:head><title>Watchdog | Electric Cherry</title></svelte:head>
|
||||
|
||||
<div class="page">
|
||||
<SectionHeader title="Watchdog" subtitle="Automatic server recovery and drift repair.">
|
||||
{#snippet actions()}
|
||||
<Button disabled={!overview || !!busy} loading={busy === 'scan'} onclick={() => update('scan')}>Scan now</Button>
|
||||
<Button disabled={!overview || !!busy} loading={busy === 'toggle'} onclick={() => update('toggle')}>{overview?.enabled ? 'Pause automation' : 'Resume automation'}</Button>
|
||||
{/snippet}
|
||||
</SectionHeader>
|
||||
|
||||
{#if loadError}<div class="ui-error" role="alert">{loadError} {overview ? 'Showing the last successful refresh.' : ''}<Button size="sm" disabled={!!busy} onclick={load}>Retry</Button></div>{/if}
|
||||
{#if actionError}<p class="ui-error" role="alert">{actionError}</p>{/if}
|
||||
{#if notice}<p class="notice" role="status">{notice}</p>{/if}
|
||||
|
||||
{#if overview}
|
||||
<section class="automation" aria-label="Automation status">
|
||||
<div class="automation-heading"><h2>Automatic recovery</h2><Badge variant={overview.enabled ? 'online' : 'warning'}>{overview.enabled ? 'Enabled' : 'Paused'}</Badge></div>
|
||||
<p>{overview.enabled ? 'Watchdog restarts unhealthy servers, then rebuilds if recovery fails.' : 'Watchdog continues detecting incidents while automatic recovery is paused. Queued Watchdog recovery is held; already submitted operations continue.'}</p>
|
||||
<p>Drift repair rebuilds the server with the latest promoted setup and records a campaign. <strong>Rebuilds delete the existing instance and all of its files.</strong> Interruption requires a confirmed stopped container or a fresh zero-player count from RCON or telemetry.</p>
|
||||
<div class="meta"><span>{active} active incident{active === 1 ? '' : 's'} · {attention} need attention</span><span>Last scan: {overview.lastScanAt ? $timeFormat.dateTime(overview.lastScanAt) : 'Not yet scanned'}</span></div>
|
||||
{#if overview.scanError}<p class="ui-error" role="alert">Last scan: {overview.scanError}</p>{/if}
|
||||
</section>
|
||||
|
||||
<section aria-label="Incidents">
|
||||
<div class="filters">
|
||||
<label>Status<select bind:value={filterState} onchange={event => { filterState = event.currentTarget.value; saveContext(true); }}><option value="active">Active incidents</option><option value="all">All incidents</option>{#each states as value}<option value={value}>{watchdogStateLabels[value]}</option>{/each}</select></label>
|
||||
<label>Reason<select bind:value={kind} onchange={event => { kind = event.currentTarget.value; saveContext(true); }}><option value="all">All reasons</option><option value="health">Server health</option><option value="drift">File drift</option></select></label>
|
||||
<label class="server-filter">Server<input type="search" bind:value={server} oninput={event => { server = event.currentTarget.value; saveContext(true); }} placeholder="Filter by server name" /></label>
|
||||
</div>
|
||||
<h2>Incidents <span class="muted">{matching.length}</span></h2>
|
||||
{#if !matching.length}
|
||||
<EmptyState title={filterState === 'active' && kind === 'all' && !server ? 'No active incidents' : 'No matching incidents'} description={filterState === 'active' && kind === 'all' && !server ? 'Detected health and drift problems will appear here automatically.' : 'Change the filters to review other incidents.'} />
|
||||
{:else}
|
||||
<div class="incidents">
|
||||
{#each matching.slice(0, limit) as incident (incident.id)}
|
||||
<article class="incident">
|
||||
<div class="row-heading"><a class="incident-link" href={`/watchdog/${encodeURIComponent(incident.id)}`}><ServerIdentity name={incident.server} /><span class="sr-only">Watchdog incident {incident.id}</span></a><WatchdogStatus state={incident.state} /></div>
|
||||
<p>{incident.summary}</p>
|
||||
<div class="meta"><Badge variant="neutral">{incident.kind === 'drift' ? 'File drift' : 'Server health'}</Badge><span>Detected {$timeFormat.dateTime(incident.first_seen_at)}</span><span>Last seen {$timeFormat.dateTime(incident.last_seen_at)}</span></div>
|
||||
<div class="links"><a href={`/watchdog/${encodeURIComponent(incident.id)}`}>View recovery and activity</a><a href={`/servers/${encodeURIComponent(incident.server)}?tab=diagnostics`}>Diagnose server</a></div>
|
||||
</article>
|
||||
{/each}
|
||||
</div>
|
||||
<ListFooter total={matching.length} bind:limit onchange={() => saveContext()} label="incidents" />
|
||||
{/if}
|
||||
</section>
|
||||
{:else if !loadError}<p class="muted" role="status">Loading Watchdog…</p>{/if}
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.page, section, .incidents { display: grid; gap: 20px; }
|
||||
.page { gap: 28px; }
|
||||
.automation, .incident { border: 1px solid var(--border-subtle); border-radius: var(--radius-md); background: var(--surface); padding: 24px; }
|
||||
.automation { gap: 12px; }
|
||||
.automation-heading, .row-heading, .meta, .links, .filters { display: flex; gap: 12px 20px; flex-wrap: wrap; align-items: center; }
|
||||
.row-heading { justify-content: space-between; }
|
||||
.incident { display: grid; gap: 14px; }
|
||||
.incident-link { color: var(--text-primary); text-decoration: none; font-weight: var(--weight-semibold); }
|
||||
.incident-link:hover { color: var(--accent-primary-hover); text-decoration: underline; }
|
||||
h2 { margin: 0; font-size: var(--text-md); }
|
||||
h2 .muted { margin-left: 8px; }
|
||||
p { margin: 0; line-height: 1.6; overflow-wrap: anywhere; }
|
||||
.automation p { max-width: 960px; font-size: var(--text-sm); color: var(--text-secondary); }
|
||||
.automation strong { color: var(--text-primary); font-weight: var(--weight-medium); }
|
||||
.meta, .muted { color: var(--text-secondary); font-size: var(--text-xs); }
|
||||
.links { font-size: var(--text-sm); }
|
||||
a { color: var(--accent-primary-hover); }
|
||||
.filters { align-items: flex-end; }
|
||||
label { display: grid; gap: 8px; font-size: var(--text-xs); color: var(--text-secondary); }
|
||||
.server-filter { flex: 1; min-width: 200px; }
|
||||
input, select { font: inherit; font-size: var(--text-sm); min-height: 40px; color: var(--text-primary); background: var(--surface-sunken); border: 1px solid var(--border-control); border-radius: var(--radius-sm); padding: 9px 12px; }
|
||||
.notice { color: var(--text-secondary); font-size: var(--text-sm); }
|
||||
@media (max-width: 640px) { .automation, .incident { padding: 18px; } label { width: 100%; } }
|
||||
</style>
|
||||
@@ -0,0 +1,2 @@
|
||||
import { redirect } from '@sveltejs/kit';
|
||||
export function load({ params, url }: { params: { id: string }; url: URL }) { redirect(308, `/maintenance/${encodeURIComponent(params.id)}${url.search}`); }
|
||||
@@ -60,6 +60,16 @@ describe('persisted Fleet operations', () => {
|
||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture(); });
|
||||
afterEach(async () => { await fixture.cleanup(); });
|
||||
|
||||
it('shows failed update verification even after a healthy restart and clears attention on a new attempt', async () => {
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
const handle = db();
|
||||
handle.prepare("INSERT INTO maintenance_jobs(server,kind,state,target_revision,requested_at,detail) VALUES ('server','update','failed',5354,?,'Revision not verified')").run(iso());
|
||||
const { fleetOperations } = await import('../../src/lib/server/fleet/overview.js');
|
||||
expect(fleetOperations(now)).toEqual([expect.objectContaining({ kind: 'maintenance', state: 'attention', detail: 'Revision not verified' })]);
|
||||
handle.prepare("INSERT INTO maintenance_jobs(server,kind,state,target_revision,requested_at) VALUES ('server','update','queued',5354,?)").run(iso(1000));
|
||||
expect(fleetOperations(now)).toEqual([]);
|
||||
});
|
||||
|
||||
it('recovers waiting and recovering campaigns without needing a live event', async () => {
|
||||
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
const handle = db();
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
import { beforeEach, afterEach, describe, it, expect, vi } from 'vitest';
|
||||
import { newFixture, type Fixture } from '../helpers/fixture.js';
|
||||
vi.mock('../../src/lib/server/pipeline/worker.js', () => ({ kick: vi.fn().mockResolvedValue(undefined) }));
|
||||
let fixture: Fixture;
|
||||
beforeEach(async () => { vi.resetModules(); fixture = await newFixture(); });
|
||||
afterEach(async () => { await fixture.cleanup(); });
|
||||
const request = (body: unknown) => new Request('http://localhost/api/maintenance', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
|
||||
|
||||
describe('Maintenance API', () => {
|
||||
it('saves independent policies and the global pause without changing the restart schedule', async () => {
|
||||
const { POST, GET } = await import('../../src/routes/api/maintenance/+server.js');
|
||||
expect((await POST({ request: request({ action: 'schedule', enabled: true, time: '06:00', timeZone: 'America/Indiana/Indianapolis' }) })).status).toBe(200);
|
||||
const before = (await (GET()).json()).runtime.schedule;
|
||||
for (const body of [{ action: 'policy', policy: 'health', enabled: false }, { action: 'policy', policy: 'updates', enabled: true }, { action: 'pause', paused: true }]) {
|
||||
expect((await POST({ request: request(body) })).status).toBe(200);
|
||||
}
|
||||
const data = await GET().json();
|
||||
expect(data.policies).toEqual({ paused: true, health: false, drift: true, updates: true });
|
||||
expect(data.runtime.schedule).toEqual(before);
|
||||
expect((await POST({ request: request({ action: 'update' }) })).status).toBe(400);
|
||||
});
|
||||
it('rejects malformed actions without changing policies or queuing work', async () => {
|
||||
const { POST, GET } = await import('../../src/routes/api/maintenance/+server.js');
|
||||
const before = await GET().json();
|
||||
for (const body of [null, [], {}, { action: 'pause', paused: 'yes' }, { action: 'policy', policy: 'unknown', enabled: true },
|
||||
{ action: 'policy', policy: 'updates', enabled: 1 }, { action: 'schedule', enabled: true, time: '25:00', timeZone: 'UTC' },
|
||||
{ action: 'schedule', enabled: true, time: '06:00', timeZone: 'invalid' }, { action: 'update', servers: [] },
|
||||
{ action: 'update', servers: [1] }, { action: 'retry', id: '1' }]) {
|
||||
expect((await POST({ request: request(body) })).status).toBe(400);
|
||||
}
|
||||
expect(await GET().json()).toEqual(before);
|
||||
});
|
||||
it('keeps old Watchdog URLs working through query-preserving redirects', async () => {
|
||||
const index = await import('../../src/routes/watchdog/+page.server.js');
|
||||
const detail = await import('../../src/routes/watchdog/[id]/+page.server.js');
|
||||
expect(() => index.load({ url: new URL('http://localhost/watchdog?kind=drift&server=test') })).toThrow(expect.objectContaining({ status: 308, location: '/maintenance?kind=drift&server=test' }));
|
||||
expect(() => detail.load({ params: { id: 'abc' }, url: new URL('http://localhost/watchdog/abc?test=1') })).toThrow(expect.objectContaining({ status: 308, location: '/maintenance/abc?test=1' }));
|
||||
});
|
||||
});
|
||||
@@ -233,3 +233,117 @@ describe('persistent maintenance', () => {
|
||||
expect(m.plutoniumStatus().schedule.nextAt).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('Maintenance policies and runtime updates', () => {
|
||||
async function outdated() {
|
||||
const rcon = await import('../../src/lib/server/integrations/rcon.js');
|
||||
vi.mocked(rcon.rconPlutoniumVersion).mockResolvedValue({ revision: 5300, error: null });
|
||||
const m = await api(); await m.checkPlutonium(); return m;
|
||||
}
|
||||
it('preserves the legacy recovery preference without enabling updates', async () => {
|
||||
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
db().prepare("INSERT INTO settings(key,value) VALUES ('watchdog_enabled','0')").run();
|
||||
const p = await import('../../src/lib/server/maintenance/policies.js');
|
||||
expect(p.maintenancePolicies()).toEqual({ paused: false, health: false, drift: false, updates: false });
|
||||
p.setMaintenancePolicy('health', true); p.pauseMaintenance(true); closeDb();
|
||||
expect(p.maintenancePolicies()).toEqual({ paused: true, health: true, drift: false, updates: false });
|
||||
expect(p.maintenanceAllowed('health')).toBe(false);
|
||||
});
|
||||
it('only automatically updates verified outdated servers and requires opt-in', async () => {
|
||||
const m = await outdated();
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
db().prepare('UPDATE plutonium_versions SET revision = NULL WHERE server = ?').run(servers[1]);
|
||||
await m.maintenanceTick();
|
||||
expect(m.plutoniumStatus().jobs).toEqual([]);
|
||||
const p = await import('../../src/lib/server/maintenance/policies.js'); p.setMaintenancePolicy('updates', true);
|
||||
await m.maintenanceTick(); await m.maintenanceTick();
|
||||
expect(m.plutoniumStatus().jobs).toEqual([expect.objectContaining({ server: servers[0], kind: 'update', origin: 'automatic', target_revision: 5354 })]);
|
||||
expect(m.plutoniumStatus().servers.find(s => s.server === servers[1])?.update_status).toBe('unknown');
|
||||
});
|
||||
it('holds queued automatic updates when their policy is disabled, but finishes submitted recovery while paused', async () => {
|
||||
const m = await outdated();
|
||||
const p = await import('../../src/lib/server/maintenance/policies.js'); p.setMaintenancePolicy('updates', true);
|
||||
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
|
||||
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
|
||||
m.queueUpdates([servers[0]], true); p.setMaintenancePolicy('updates', false);
|
||||
await processProvisioning(); expect(komodoRestart).not.toHaveBeenCalled();
|
||||
p.setMaintenancePolicy('updates', true); p.pauseMaintenance(true);
|
||||
await processProvisioning(); expect(komodoRestart).not.toHaveBeenCalled();
|
||||
p.pauseMaintenance(false); await processProvisioning(); expect(komodoRestart).toHaveBeenCalledTimes(1);
|
||||
p.pauseMaintenance(true); vi.setSystemTime(Date.now() + 15_000);
|
||||
await processProvisioning(); await m.maintenanceTick();
|
||||
expect(m.plutoniumStatus().jobs[0]).toMatchObject({ state: 'failed', operation_state: 'done' });
|
||||
expect(komodoRestart).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
it('latches failed update verification across restarts and retention, and supports an explicit retry', async () => {
|
||||
const m = await outdated();
|
||||
const p = await import('../../src/lib/server/maintenance/policies.js'); p.setMaintenancePolicy('updates', true);
|
||||
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
db().prepare("UPDATE servers SET desired_state = 'stopped' WHERE name = ?").run(servers[1]);
|
||||
await maintenanceTick(); vi.setSystemTime(Date.now() + 15_000); await maintenanceTick();
|
||||
const failed = m.plutoniumStatus().jobs[0];
|
||||
expect(failed).toMatchObject({ kind: 'update', state: 'failed', operation_state: 'done' });
|
||||
closeDb(); vi.setSystemTime(Date.now() + 31 * 86_400_000); await maintenanceTick();
|
||||
expect(m.plutoniumStatus().jobs).toHaveLength(1);
|
||||
expect(m.retryMaintenance(failed.id)).toBe(1);
|
||||
expect(m.plutoniumStatus().jobs).toHaveLength(2);
|
||||
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
|
||||
vi.mocked(rconPlutoniumVersion).mockResolvedValue({ revision: 5354, error: null });
|
||||
await maintenanceTick(); vi.setSystemTime(Date.now() + 15_000); await maintenanceTick();
|
||||
expect(m.plutoniumStatus().jobs[0]).toMatchObject({ state: 'done', origin: 'manual' });
|
||||
expect(db().prepare("SELECT 1 FROM server_deployments WHERE action = 'rebuild'").get()).toBeUndefined();
|
||||
});
|
||||
it('does not restart intentionally stopped servers, including jobs queued before the intent changed', async () => {
|
||||
const m = await outdated(); m.queueUpdates([servers[0]]);
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
db().prepare("UPDATE servers SET desired_state = 'stopped'").run();
|
||||
await maintenanceTick();
|
||||
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
|
||||
expect(komodoRestart).not.toHaveBeenCalled();
|
||||
expect(m.plutoniumStatus().jobs[0].state).toBe('cancelled');
|
||||
expect(m.queueUpdates()).toBe(0);
|
||||
expect(m.plutoniumStatus().servers.every(s => s.update_status === 'stopped')).toBe(true);
|
||||
});
|
||||
it('separates scheduled restart success from revision verification and coalesces paused runs', async () => {
|
||||
const m = await outdated(); m.saveMaintenanceSchedule(true, '06:00', 'UTC');
|
||||
const p = await import('../../src/lib/server/maintenance/policies.js'); p.pauseMaintenance(true);
|
||||
vi.setSystemTime(new Date('2026-09-24T07:00:00Z')); await m.maintenanceTick();
|
||||
expect(m.plutoniumStatus().jobs).toHaveLength(0);
|
||||
p.pauseMaintenance(false); await maintenanceTick();
|
||||
expect(m.plutoniumStatus().jobs).toHaveLength(2);
|
||||
expect(m.plutoniumStatus().jobs.every(j => j.kind === 'schedule' && j.target_revision === null)).toBe(true);
|
||||
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
|
||||
vi.mocked(rconPlutoniumVersion).mockResolvedValue({ revision: null, error: 'Unknown reply' });
|
||||
vi.setSystemTime(Date.now() + 15_000); await maintenanceTick();
|
||||
expect(m.plutoniumStatus().jobs.find(j => j.server === servers[0])?.state).toBe('done');
|
||||
expect(m.plutoniumStatus().schedule.nextAt).toBe('2026-09-25T06:00:00.000Z');
|
||||
});
|
||||
it('does not treat stale, pre-restart, or unhealthy observations as current runtime evidence', async () => {
|
||||
const m = await outdated();
|
||||
const { db } = await import('../../src/lib/server/db/index.js');
|
||||
db().prepare('INSERT INTO restart_rollout(server,requested_at,healthy_at) VALUES (?,?,?)').run(servers[0], new Date().toISOString(), new Date().toISOString());
|
||||
expect(m.plutoniumStatus().servers[0].update_status).toBe('unknown');
|
||||
vi.setSystemTime(Date.now() + 3_600_000);
|
||||
expect(() => m.queueUpdates()).toThrow('published revision');
|
||||
const { komodoInspectHealthy } = await import('../../src/lib/server/integrations/komodo.js');
|
||||
const { rconPlutoniumVersion } = await import('../../src/lib/server/integrations/rcon.js');
|
||||
vi.mocked(rconPlutoniumVersion).mockClear();
|
||||
vi.mocked(komodoInspectHealthy).mockResolvedValue({ ok: false, state: 'starting', reason: 'Starting' });
|
||||
await m.checkPlutonium();
|
||||
expect(rconPlutoniumVersion).not.toHaveBeenCalled();
|
||||
expect(m.plutoniumStatus().servers.every(s => s.update_status === 'unknown')).toBe(true);
|
||||
});
|
||||
it('adopts existing maintenance jobs without changing their IDs, schedules, or submitted operations', async () => {
|
||||
const m = await outdated(); m.saveMaintenanceSchedule(true, '06:00', 'America/New_York');
|
||||
m.queueMaintenance([servers[0]]); await maintenanceTick();
|
||||
const before = m.plutoniumStatus();
|
||||
const { db, closeDb } = await import('../../src/lib/server/db/index.js');
|
||||
db().exec('ALTER TABLE maintenance_jobs DROP COLUMN kind; ALTER TABLE maintenance_jobs DROP COLUMN origin;'); closeDb();
|
||||
const after = m.plutoniumStatus();
|
||||
expect(after.schedule).toEqual(before.schedule);
|
||||
expect(after.jobs[0]).toMatchObject({ id: before.jobs[0].id, provisioning_id: before.jobs[0].provisioning_id, kind: 'legacy', origin: 'manual' });
|
||||
vi.setSystemTime(Date.now() + 15_000); await maintenanceTick();
|
||||
const { komodoRestart } = await import('../../src/lib/server/integrations/komodo.js');
|
||||
expect(komodoRestart).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -395,3 +395,36 @@ describe('Watchdog drift campaigns', () => {
|
||||
expect(komodo.komodoDestroyDeployment).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('Maintenance recovery policy boundaries', () => {
|
||||
it('separates health recovery from drift repair and holds an already queued drift rebuild', async () => {
|
||||
const policies = await import('../../src/lib/server/maintenance/policies.js');
|
||||
policies.setMaintenancePolicy('health', false);
|
||||
down();
|
||||
for (let n = 0; n < 3; n++) { await watchdog.watchdogTick(true); advance(watchdog.WATCHDOG_SCAN_MS); }
|
||||
expect(operations()).toEqual([]);
|
||||
expect(onlyIncident().kind).toBe('health');
|
||||
vi.mocked(komodo.komodoInspectHealthy).mockResolvedValue({ ok: true, state: 'running' });
|
||||
await writeFile(join(fixture.t5, server, 'runtime/plutonium/storage/t5/raw/scripts/sp', 'stray.gsc'), 'drift');
|
||||
await watchdog.watchdogTick(true);
|
||||
expect(operations()).toEqual([expect.objectContaining({ action: 'rebuild', state: 'waiting' })]);
|
||||
policies.setMaintenancePolicy('drift', false);
|
||||
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
|
||||
await processProvisioning();
|
||||
expect(operations()[0].state).toBe('waiting');
|
||||
expect(komodo.komodoDestroyDeployment).not.toHaveBeenCalled();
|
||||
});
|
||||
it('global pause blocks escalation without disabling the saved health policy', async () => {
|
||||
await sustainedFailure();
|
||||
const { processProvisioning } = await import('../../src/lib/server/provisioning/worker.js');
|
||||
await processProvisioning();
|
||||
const policies = await import('../../src/lib/server/maintenance/policies.js');
|
||||
policies.pauseMaintenance(true);
|
||||
advance(watchdog.WATCHDOG_RECOVERY_MS); await watchdog.watchdogTick(true);
|
||||
expect(operations()).toHaveLength(1);
|
||||
expect(policies.maintenancePolicies().health).toBe(true);
|
||||
policies.pauseMaintenance(false); await watchdog.watchdogTick(true);
|
||||
expect(operations()).toHaveLength(2);
|
||||
expect(operations()[1].action).toBe('rebuild');
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user