179 lines
11 KiB
Markdown
179 lines
11 KiB
Markdown
# Maintenance
|
|
|
|
Open **Operations → Maintenance** to keep existing servers healthy and current.
|
|
Maintenance owns observations, policies, schedules, requests, and verification of
|
|
maintenance outcomes. Provisioning owns lifecycle execution, player checks,
|
|
conflicts, rolling capacity, container recovery, and persistent operation history.
|
|
Campaigns retain managed-file deployment evidence; Collections define lineups.
|
|
|
|
## Views
|
|
|
|
- **Overview** shows active work, attention, upcoming restarts, and Plutonium version
|
|
coverage. Filter by capability or server. **Check now** refreshes health, drift,
|
|
the production release, and running revisions.
|
|
- **Policies & schedules** controls health recovery, configuration drift, automatic
|
|
Plutonium updates, and the optional daily restart schedule independently.
|
|
- **Activity** combines recovery findings, update results, and scheduled restarts.
|
|
Follow a finding for evidence and attempts, or its Provisioning operation for
|
|
execution progress. Cancel unsubmitted runtime restarts here.
|
|
|
|
Fleet and server details link their revision indicators to Maintenance. Failed
|
|
runtime maintenance appears in Fleet attention, even when the underlying restart
|
|
succeeded. Provisioning and repair Campaigns link back to the requesting work.
|
|
|
|
## Policies and pause
|
|
|
|
Health recovery and drift repair inherit the previous Watchdog enabled setting.
|
|
Automatic Plutonium updates are **disabled by default**. Existing daily schedules
|
|
retain their enabled state, timezone, and next occurrence.
|
|
|
|
**Pause all maintenance** holds new remediation, scheduled runs, and unsubmitted
|
|
maintenance operations. Scans and result verification continue; submitted
|
|
operations finish recovery. Campaigns, betas, and manually requested Provisioning
|
|
operations continue independently. Resuming keeps each policy's saved setting.
|
|
|
|
Disabling health recovery or drift repair holds queued work for that capability
|
|
and prevents further automatic attempts, including health escalation. Disabling
|
|
automatic updates holds queued automatic update requests. Explicitly requested
|
|
updates remain operator-authorized work. Disabling a daily schedule prevents
|
|
future runs but does not cancel requests it already created.
|
|
|
|
All capabilities respect intentionally stopped servers. Start them explicitly
|
|
through Provisioning before expecting maintenance to act on them.
|
|
|
|
While Provisioning owns a server, Maintenance skips new health/drift observations,
|
|
version probes, automatic updates, and scheduled restart requests for that server.
|
|
Queued, waiting, running, and recovering operations count, including failed
|
|
operations that retain recovery capacity. Other servers continue normally. Existing
|
|
maintenance tracks its own operation to completion; revision verification waits
|
|
until any later operation releases the server.
|
|
|
|
## Plutonium updates
|
|
|
|
The published revision comes from the official production updater manifest.
|
|
Running revisions come from RCON `version` after checking container readiness.
|
|
Checks run hourly while the application is running. Failed release checks retain
|
|
the cached release with an error. Unsupported replies, stale checks, unhealthy
|
|
containers, and evidence from before a lifecycle submission remain unverified.
|
|
A missing revision is never treated as current or as an automatic update target.
|
|
|
|
**Update on restart** requests ordinary Provisioning restarts for verified outdated
|
|
servers. **Update outdated servers on restart** applies to the filtered list.
|
|
Automatic updates, when enabled, apply to verified outdated servers across the
|
|
fleet, with one attempt per server and published revision. Existing pending
|
|
maintenance is reused. Occupied servers and servers with conflicting beta or
|
|
campaign work wait without consuming a restart slot.
|
|
|
|
Plutainer's startup updater must be enabled (`PLUTAINER_AUTO_UPDATE`). Electric
|
|
Cherry does not download binaries, pull a container image, or pin the runtime to
|
|
a particular revision. The observed published revision is recorded as the minimum
|
|
expected result; the actual revision is queried after recovery.
|
|
|
|
A healthy restart and a successful update are separate outcomes. If the server
|
|
recovers on an older or unknown revision, the restart is successful in Provisioning
|
|
and the update needs attention in Maintenance. Update verification never triggers
|
|
a rebuild or an automatic restart loop. Inspect the updater configuration and
|
|
server logs, then use **Retry after inspection** to create a new attempt. Ambiguous
|
|
or failed lifecycle requests require inspection of the existing Provisioning
|
|
operation and its retained recovery slot first.
|
|
|
|
Update attempts persist as suppression evidence across app restarts and history
|
|
retention. Cancelling an automatic attempt suppresses another automatic attempt for
|
|
that same revision; an operator can request an update explicitly.
|
|
|
|
## Scheduled restarts
|
|
|
|
Daily restarts are independent of update availability. At the saved queue time,
|
|
Maintenance requests a restart for each server whose desired state is running.
|
|
These requests have no revision requirement. Servers owned by Provisioning are
|
|
skipped for that occurrence; the schedule advances normally. Requests already
|
|
queued can wait for players, campaign work, or beta work to finish.
|
|
|
|
Saving a schedule uses the selected timezone and follows daylight saving changes.
|
|
Skipped times run at the first valid minute after the gap; repeated times run once
|
|
on their first occurrence. Legacy UTC schedules keep their timing until saved.
|
|
Changing the browser display timezone does not change the saved schedule.
|
|
|
|
Schedules survive application restarts. Missed days, including while globally
|
|
paused, become one run on resume. Pending work is deduplicated per server. The
|
|
worker starts on the first application request, so keep the app running and
|
|
reachable. Finished scheduled and legacy maintenance activity is retained for
|
|
30 days by default (configurable in Administration → Settings); update attempt records remain durable.
|
|
|
|
## Health recovery and configuration drift
|
|
|
|
Fleet keeps successful game and running-container reports fresh for 90 seconds.
|
|
Two consecutive failed monitoring checks mark RCON health stale; a first timeout
|
|
or send failure remains visible in diagnostics without changing the health badge.
|
|
Configuration and invalid-reply errors, including authentication failures, take
|
|
effect immediately, as do failed manual probes. Successful RCON checks reset the
|
|
failure count; skipped checks do not count. Without fresh telemetry, RCON evidence
|
|
older than 120 seconds is offline. Explicit stopped/unhealthy container states
|
|
and lifecycle changes still take precedence.
|
|
|
|
The status monitor collects container and game health every 30 seconds. Once per
|
|
minute, the watchdog evaluates that completed evidence without issuing its own
|
|
routine health probes. Samples expire after 90 seconds; missing or stale evidence
|
|
leaves health unknown and cannot justify recovery. Samples retain their original
|
|
timestamps and instance identity, and each failure sample counts at most once.
|
|
A monitoring gap or changed instance resets the sustained failure window. After
|
|
an application restart, routine watchdog checks wait for a new status-monitor pass.
|
|
**Check now** and recovery verification still request fresh evidence, and
|
|
Provisioning retains its checks before executing recovery actions.
|
|
|
|
Three observations over at least two minutes are required before an automatic
|
|
health restart; starting or restarting containers get five minutes. Unknown host-agent responses are observations,
|
|
not proof of a stopped container. Repeated manual checks do not accelerate this
|
|
window. A successful routine or manual game-status probe between scans resets
|
|
the failure streak. Missing RCON configuration, unresolved ports, and invalid
|
|
replies remain diagnostic findings and cannot authorize health recovery.
|
|
|
|
A health finding gets one restart, then one rebuild if recovery fails after five
|
|
minutes and fresh health checks still justify recovery. If current container and
|
|
game checks succeed before an automatic health restart or rebuild is submitted,
|
|
the obsolete operation is cancelled and the finding is resolved. Unknown health
|
|
holds recovery for another check. Drift repairs and requested lifecycle operations
|
|
retain their own eligibility rules.
|
|
|
|
Escalation transfers only that finding's restart slot. A failed rebuild
|
|
requires operator attention. **Rebuild deletes all instance files** and uses the
|
|
latest promoted setup, the same server identity and port, and the saved server key.
|
|
Missing keys block teardown. Submitted rebuild verification continues while paused.
|
|
|
|
Drift includes missing promoted files, changed bytes, and extra files in managed
|
|
script, config, and plugin directories. Config comparison includes the composed
|
|
baseline, overlays, and the selected map. Unmanaged default config is exempt until
|
|
a baseline is promoted. Missing mounts are scan errors, not empty directories.
|
|
Routine drift scans run every five minutes per server, reducing file traffic
|
|
while health observations continue every minute. **Check now**, a replacement
|
|
registration, or a new lifecycle submission bypasses the drift interval. Failed
|
|
file scans retry on the next observer pass; post-rebuild verification still reads
|
|
fresh files.
|
|
|
|
Drift repair creates a Campaign capturing observed differences and requests a
|
|
Provisioning rebuild. The Campaign observes that rebuild without requesting
|
|
another restart. Its detection evidence and the actual launch recipe remain
|
|
available even if promotions change while the work waits. Histories survive
|
|
instance removal. Active beta and campaign work defers new recovery.
|
|
|
|
**Dismiss** suppresses health/drift attempts until the condition clears. **Retry
|
|
recovery** begins a fresh bounded attempt after inspection. Existing operations
|
|
must finish or be cancelled first. Fresh verification resolves the finding.
|
|
|
|
## Compatibility and API
|
|
|
|
Existing job, incident, and operation IDs are retained. Upgrade migration adds
|
|
request metadata without replaying commands; existing pending requests keep their
|
|
original verification requirements. `/watchdog` and `/watchdog/:id` redirect to
|
|
Maintenance, preserving query parameters. Old Watchdog and Plutonium APIs remain
|
|
compatible. The previous Watchdog enabled switch controls health and drift only.
|
|
|
|
- `GET /api/maintenance`: policies, recovery findings, release/version coverage,
|
|
schedule, and recent runtime activity with linked operation stages.
|
|
- `POST /api/maintenance`: actions `scan`, `pause` (`paused` boolean), `policy`
|
|
(`policy`: `health`, `drift`, or `updates`; `enabled` boolean), `update` (optional
|
|
`servers` array), `schedule` (`enabled`, `time`, `timeZone`), `cancel` or `retry`
|
|
(`id`, the numeric runtime maintenance activity ID).
|
|
- `/api/maintenance/:id`: GET recovery evidence and attempts; POST `retry` or
|
|
`dismiss` for a health/drift finding.
|