9.3 KiB
Maintenance
Open Operations → Maintenance to keep existing servers healthy and current. Maintenance owns observations, policies, schedules, requests, and verification of maintenance outcomes. Provisioning owns lifecycle execution, player checks, conflicts, rolling capacity, container recovery, and persistent operation history. Campaigns retain managed-file deployment evidence; Collections define lineups.
Views
- Overview shows active work, attention, upcoming restarts, and Plutonium version coverage. Filter by capability or server. Check now refreshes health, drift, the production release, and running revisions.
- Policies & schedules controls health recovery, configuration drift, automatic Plutonium updates, and the optional daily restart schedule independently.
- Activity combines recovery findings, update results, and scheduled restarts. Follow a finding for evidence and attempts, or its Provisioning operation for execution progress. Cancel unsubmitted runtime restarts here.
Fleet and server details link their revision indicators to Maintenance. Failed runtime maintenance appears in Fleet attention, even when the underlying restart succeeded. Provisioning and repair Campaigns link back to the requesting work.
Policies and pause
Health recovery and drift repair inherit the previous Watchdog enabled setting. Automatic Plutonium updates are disabled by default. Existing daily schedules retain their enabled state, timezone, and next occurrence.
Pause all maintenance holds new remediation, scheduled runs, and unsubmitted maintenance operations. Scans and result verification continue; submitted operations finish recovery. Campaigns, betas, and manually requested Provisioning operations continue independently. Resuming keeps each policy's saved setting.
Disabling health recovery or drift repair holds queued work for that capability and prevents further automatic attempts, including health escalation. Disabling automatic updates holds queued automatic update requests. Explicitly requested updates remain operator-authorized work. Disabling a daily schedule prevents future runs but does not cancel requests it already created.
All capabilities respect intentionally stopped servers. Start them explicitly through Provisioning before expecting maintenance to act on them.
While Provisioning owns a server, Maintenance skips new health/drift observations, version probes, automatic updates, and scheduled restart requests for that server. Queued, waiting, running, and recovering operations count, including failed operations that retain recovery capacity. Other servers continue normally. Existing maintenance tracks its own operation to completion; revision verification waits until any later operation releases the server.
Plutonium updates
The published revision comes from the official production updater manifest.
Running revisions come from RCON version after checking container readiness.
Checks run hourly while the application is running. Failed release checks retain
the cached release with an error. Unsupported replies, stale checks, unhealthy
containers, and evidence from before a lifecycle submission remain unverified.
A missing revision is never treated as current or as an automatic update target.
Update on restart requests ordinary Provisioning restarts for verified outdated servers. Update outdated servers on restart applies to the filtered list. Automatic updates, when enabled, apply to verified outdated servers across the fleet, with one attempt per server and published revision. Existing pending maintenance is reused. Occupied servers and servers with conflicting beta or campaign work wait without consuming a restart slot.
Plutainer's startup updater must be enabled (PLUTAINER_AUTO_UPDATE). Electric
Cherry does not download binaries, pull a container image, or pin the runtime to
a particular revision. The observed published revision is recorded as the minimum
expected result; the actual revision is queried after recovery.
A healthy restart and a successful update are separate outcomes. If the server recovers on an older or unknown revision, the restart is successful in Provisioning and the update needs attention in Maintenance. Update verification never triggers a rebuild or an automatic restart loop. Inspect the updater configuration and server logs, then use Retry after inspection to create a new attempt. Ambiguous or failed lifecycle requests require inspection of the existing Provisioning operation and its retained recovery slot first.
Update attempts persist as suppression evidence across app restarts and history retention. Cancelling an automatic attempt suppresses another automatic attempt for that same revision; an operator can request an update explicitly.
Scheduled restarts
Daily restarts are independent of update availability. At the saved queue time, Maintenance requests a restart for each server whose desired state is running. These requests have no revision requirement. Servers owned by Provisioning are skipped for that occurrence; the schedule advances normally. Requests already queued can wait for players, campaign work, or beta work to finish.
Saving a schedule uses the selected timezone and follows daylight saving changes. Skipped times run at the first valid minute after the gap; repeated times run once on their first occurrence. Legacy UTC schedules keep their timing until saved. Changing the browser display timezone does not change the saved schedule.
Schedules survive application restarts. Missed days, including while globally paused, become one run on resume. Pending work is deduplicated per server. The worker starts on the first application request, so keep the app running and reachable. Finished scheduled and legacy maintenance activity is retained for 30 days; update attempt records remain durable.
Health recovery and configuration drift
The health observer (the watchdog) scans once per minute. Three observations over at least two minutes are required before an automatic health restart; starting or restarting containers get five minutes. Unknown Komodo responses are observations, not proof of a stopped container. Repeated manual checks do not accelerate this window. A successful routine or manual game-status probe between scans resets the failure streak. Missing RCON configuration, unresolved ports, and invalid replies remain diagnostic findings and cannot authorize health recovery.
A health finding gets one restart, then one rebuild if recovery fails after five minutes and fresh health checks still justify recovery. If current container and game checks succeed before an automatic health restart or rebuild is submitted, the obsolete operation is cancelled and the finding is resolved. Unknown health holds recovery for another check. Drift repairs and requested lifecycle operations retain their own eligibility rules.
Escalation transfers only that finding's restart slot. A failed rebuild requires operator attention. Rebuild deletes all instance files and uses the latest promoted setup, the same server identity and port, and the saved server key. Missing keys block teardown. Submitted rebuild verification continues while paused.
Drift includes missing promoted files, changed bytes, and extra files in managed script, config, and plugin directories. Config comparison includes the composed baseline, overlays, and the selected map. Unmanaged default config is exempt until a baseline is promoted. Missing mounts are scan errors, not empty directories. Routine drift scans run every five minutes per server, reducing file traffic while health observations continue every minute. Check now, a replacement registration, or a new lifecycle submission bypasses the drift interval. Failed file scans retry on the next observer pass; post-rebuild verification still reads fresh files.
Drift repair creates a Campaign capturing observed differences and requests a Provisioning rebuild. The Campaign observes that rebuild without requesting another restart. Its detection evidence and the actual launch recipe remain available even if promotions change while the work waits. Histories survive instance removal. Active beta and campaign work defers new recovery.
Dismiss suppresses health/drift attempts until the condition clears. Retry recovery begins a fresh bounded attempt after inspection. Existing operations must finish or be cancelled first. Fresh verification resolves the finding.
Compatibility and API
Existing job, incident, and operation IDs are retained. Upgrade migration adds
request metadata without replaying commands; existing pending requests keep their
original verification requirements. /watchdog and /watchdog/:id redirect to
Maintenance, preserving query parameters. Old Watchdog and Plutonium APIs remain
compatible. The previous Watchdog enabled switch controls health and drift only.
GET /api/maintenance: policies, recovery findings, release/version coverage, schedule, and recent runtime activity with linked operation stages.POST /api/maintenance: actionsscan,pause(pausedboolean),policy(policy:health,drift, orupdates;enabledboolean),update(optionalserversarray),schedule(enabled,time,timeZone),cancelorretry(id, the numeric runtime maintenance activity ID)./api/maintenance/:id: GET recovery evidence and attempts; POSTretryordismissfor a health/drift finding.