# Maintenance Open **Activity → Maintenance** to keep existing servers healthy and current. Maintenance owns observations, policies, schedules, requests, and verification of maintenance outcomes. Provisioning owns lifecycle execution, player checks, conflicts, rolling capacity, container recovery, and persistent operation history. Campaigns retain managed-file deployment evidence; Collections define lineups. ## Views - **Overview** shows active work, attention, upcoming restarts, and Plutonium version coverage. Filter by capability or server. **Check now** refreshes health, drift, the production release, and running revisions. Failed checks are grouped by cause with links to affected servers; raw paths and responses stay in **Technical details**. Identical revision states share one summary row. Expand it for each server’s check and pending restart, or use **Retry version check** when the running revision is unverified. - **Policies & schedules** controls health recovery, configuration drift, automatic Plutonium updates, and the optional daily restart schedule independently. - **Activity** uses the shared Activity log, scoped to maintenance and recovery. Filter retained findings, update results, and scheduled restarts in one timeline. Follow a finding for evidence or an operation for execution progress. Use Overview’s **Active work & attention** to cancel waiting work or retry failed verification after inspection. Fleet and server details link their revision indicators to Maintenance. Failed runtime maintenance appears in Fleet attention, even when the underlying restart succeeded. Provisioning and repair Campaigns link back to the requesting work. ## Policies and pause Health recovery and drift repair inherit the previous Watchdog enabled setting. Automatic Plutonium updates are **disabled by default**. Existing daily schedules retain their enabled state, timezone, and next occurrence. **Pause all maintenance** holds new remediation, scheduled runs, and unsubmitted maintenance operations. Scans and result verification continue; submitted operations finish recovery. Campaigns, betas, and manually requested Provisioning operations continue independently. Resuming keeps each policy's saved setting. Disabling health recovery or drift repair holds queued work for that capability and prevents further automatic attempts, including health escalation. Disabling automatic updates holds queued automatic update requests. Explicitly requested updates remain operator-authorized work. Disabling a daily schedule prevents future runs but does not cancel requests it already created. All capabilities respect intentionally stopped servers. Start them explicitly through Provisioning before expecting maintenance to act on them. While Provisioning owns a server, Maintenance skips new health/drift observations, version probes, automatic updates, and scheduled restart requests for that server. Queued, waiting, running, and recovering operations count, including failed operations that retain recovery capacity. Other servers continue normally. Existing maintenance tracks its own operation to completion; revision verification waits until any later operation releases the server. ## Plutonium updates The published revision comes from the official production updater manifest. Running revisions come from RCON `version` after checking container readiness. Checks run hourly while the application is running. Failed release checks retain the cached release with an error. Unsupported replies, stale checks, unhealthy containers, and evidence from before a lifecycle submission remain unverified. A missing revision is never treated as current or as an automatic update target. **Update on restart** requests ordinary Provisioning restarts for verified outdated servers. **Update outdated servers on restart** applies to the filtered list. Automatic updates, when enabled, apply to verified outdated servers across the fleet, with one attempt per server and published revision. Existing pending maintenance is reused. Occupied servers and servers with conflicting beta or campaign work wait without consuming a restart slot. Plutainer's startup updater must be enabled (`PLUTAINER_AUTO_UPDATE`). Electric Cherry does not download binaries, pull a container image, or pin the runtime to a particular revision. The observed published revision is recorded as the minimum expected result; the actual revision is queried after recovery. A healthy restart and a successful update are separate outcomes. If the server recovers on an older or unknown revision, the restart is successful in Provisioning and the update needs attention in Maintenance. Update verification never triggers a rebuild or an automatic restart loop. Inspect the updater configuration and server logs, then use **Retry after inspection** to create a new attempt. Ambiguous or failed lifecycle requests require inspection of the existing Provisioning operation and its retained recovery slot first. Update attempts persist as suppression evidence across app restarts and history retention. Cancelling an automatic attempt suppresses another automatic attempt for that same revision; an operator can request an update explicitly. ## Scheduled restarts Daily restarts are independent of update availability. At the saved queue time, Maintenance requests a restart for each server whose desired state is running. These requests have no revision requirement. Servers owned by Provisioning are skipped for that occurrence; the schedule advances normally. Requests already queued can wait for players, campaign work, or beta work to finish. Saving a schedule uses the selected timezone and follows daylight saving changes. Skipped times run at the first valid minute after the gap; repeated times run once on their first occurrence. Legacy UTC schedules keep their timing until saved. Changing the browser display timezone does not change the saved schedule. Schedules survive application restarts. Missed days, including while globally paused, become one run on resume. Pending work is deduplicated per server. The worker starts on the first application request, so keep the app running and reachable. Finished scheduled and legacy maintenance activity is retained for 30 days by default (configurable in Admin → Settings); update attempt records remain durable. ## Health recovery and configuration drift Fleet keeps successful game and running-container reports fresh for 90 seconds. Two consecutive failed monitoring checks mark RCON health stale; a first timeout or send failure remains visible in diagnostics without changing the health badge. Configuration and invalid-reply errors, including authentication failures, take effect immediately, as do failed manual probes. Successful RCON checks reset the failure count; skipped checks do not count. Without fresh telemetry, RCON evidence older than 120 seconds is offline. Explicit stopped/unhealthy container states and lifecycle changes still take precedence. The status monitor collects container and game health every 30 seconds. Once per minute, the watchdog evaluates that completed evidence without issuing its own routine health probes. Samples expire after 90 seconds; missing or stale evidence leaves health unknown and cannot justify recovery. Samples retain their original timestamps and instance identity, and each failure sample counts at most once. A monitoring gap or changed instance resets the sustained failure window. After an application restart, routine watchdog checks wait for a new status-monitor pass. **Check now** and recovery verification still request fresh evidence, and Provisioning retains its checks before executing recovery actions. Three observations over at least two minutes are required before an automatic health restart; starting or restarting containers get five minutes. Unknown host-agent responses are observations, not proof of a stopped container. Repeated manual checks do not accelerate this window. A successful routine or manual game-status probe between scans resets the failure streak. Missing RCON configuration, unresolved ports, and invalid replies remain diagnostic findings and cannot authorize health recovery. A health finding gets one restart, then one rebuild if recovery fails after five minutes and fresh health checks still justify recovery. If current container and game checks succeed before an automatic health restart or rebuild is submitted, the obsolete operation is cancelled and the finding is resolved. Unknown health holds recovery for another check. Drift repairs and requested lifecycle operations retain their own eligibility rules. Escalation transfers only that finding's restart slot. A failed rebuild requires operator attention. **Rebuild deletes all instance files** and uses the latest promoted setup, the same server identity and port, and the saved server key. Missing keys block teardown. Submitted rebuild verification continues while paused. Drift includes missing promoted files, changed bytes, and extra files in managed script, config, and plugin directories. Config comparison includes the composed baseline, overlays, selected map, and the RCON password assigned to the current instance at launch. Launches and beta config writes use the same password rendering as drift checks, including Plutainer's startup newline handling. Unmanaged default config is exempt until a baseline is promoted. Missing mounts are scan errors, not empty directories. Routine drift scans run every five minutes per server, reducing file traffic while health observations continue every minute. **Check now**, a replacement registration, or a new lifecycle submission bypasses the drift interval. Failed file scans retry on the next observer pass; post-rebuild verification still reads fresh files. For older instances whose only difference is Plutainer's expected password rewrite, a successful scan reconciles the recorded config hash without changing server files or rebuilding. The entire file must match the expected runtime config; unexpected passwords and other config edits still count as drift. Current launch credentials remain available through history cleanup while the instance is live. If that launch history was already expired, the check reports unavailable evidence instead of guessing a password or requesting a rebuild. Drift repair creates a Campaign capturing observed differences and requests a Provisioning rebuild. The Campaign observes that rebuild without requesting another restart. Its detection evidence and the actual launch recipe remain available even if promotions change while the work waits. Histories survive instance removal. Active beta and campaign work defers new recovery. **Dismiss** suppresses health/drift attempts until the condition clears. **Retry recovery** begins a fresh bounded attempt after inspection. Existing operations must finish or be cancelled first. Fresh verification resolves the finding. ## Compatibility and API Existing job, incident, and operation IDs are retained. Upgrade migration adds request metadata without replaying commands; existing pending requests keep their original verification requirements. `/watchdog` and `/watchdog/:id` redirect to Maintenance, preserving query parameters. Old Watchdog and Plutonium APIs remain compatible. The previous Watchdog enabled switch controls health and drift only. - `GET /api/maintenance`: policies, recovery findings, release/version coverage, schedule, and recent runtime activity with linked operation stages. - `POST /api/maintenance`: actions `scan`, `pause` (`paused` boolean), `policy` (`policy`: `health`, `drift`, or `updates`; `enabled` boolean), `update` (optional `servers` array), `schedule` (`enabled`, `time`, `timeZone`), `cancel` or `retry` (`id`, the numeric runtime maintenance activity ID). - `/api/maintenance/:id`: GET recovery evidence and attempts; POST `retry` or `dismiss` for a health/drift finding.