electric-cherry/docs/guides/maintenance.md

# Maintenance

Open **Activity → Maintenance** to keep existing servers healthy and current.
Maintenance owns observations, policies, schedules, requests, and verification of
maintenance outcomes. Provisioning owns lifecycle execution, player checks,
conflicts, rolling capacity, container recovery, and persistent operation history.
Campaigns retain managed-file deployment evidence; Collections define lineups.

## Views

- **Overview** shows active work, attention, upcoming restarts, and Plutonium version
  coverage. Filter by capability or server. **Check now** refreshes health, drift,
  the production release, and running revisions. Failed checks are grouped by
  cause with links to affected servers; raw paths and responses stay in
  **Technical details**. Identical revision states share one summary row. Expand
  it for each server’s check and pending restart, or use **Retry version check**
  when the running revision is unverified.
- **Policies & schedules** controls health recovery, configuration drift, automatic
  Plutonium updates, and the optional daily restart schedule independently.
- **Activity** uses the shared Activity log, scoped to maintenance and recovery.
  Filter retained findings, update results, and scheduled restarts in one timeline.
  Follow a finding for evidence or an operation for execution progress. Use
  Overview’s **Active work & attention** to cancel waiting work or retry failed
  verification after inspection.

Fleet and server details link their revision indicators to Maintenance. Failed
runtime maintenance appears in Fleet attention, even when the underlying restart
succeeded. Provisioning and repair Campaigns link back to the requesting work.

## Policies and pause

Health recovery and drift repair inherit the previous Watchdog enabled setting.
Automatic Plutonium updates are **disabled by default**. Existing daily schedules
retain their enabled state, timezone, and next occurrence.

**Pause all maintenance** holds new remediation, scheduled runs, and unsubmitted
maintenance operations. Scans and result verification continue; submitted
operations finish recovery. Campaigns, betas, and manually requested Provisioning
operations continue independently. Resuming keeps each policy's saved setting.

Disabling health recovery or drift repair holds queued work for that capability
and prevents further automatic attempts, including health escalation. Disabling
automatic updates holds queued automatic update requests. Explicitly requested
updates remain operator-authorized work. Disabling a daily schedule prevents
future runs but does not cancel requests it already created.

All capabilities respect intentionally stopped servers. Start them explicitly
through Provisioning before expecting maintenance to act on them.

While Provisioning owns a server, Maintenance skips new health/drift observations,
version probes, automatic updates, and scheduled restart requests for that server.
Queued, waiting, running, and recovering operations count, including failed
operations that retain recovery capacity. Other servers continue normally. Existing
maintenance tracks its own operation to completion; revision verification waits
until any later operation releases the server.

## Plutonium updates

The published revision comes from the official production updater manifest.
Running revisions come from RCON `version` after checking container readiness.
Checks run hourly while the application is running. Failed release checks retain
the cached release with an error. Unsupported replies, stale checks, unhealthy
containers, and evidence from before a lifecycle submission remain unverified.
A missing revision is never treated as current or as an automatic update target.

**Update on restart** requests ordinary Provisioning restarts for verified outdated
servers. **Update outdated servers on restart** applies to the filtered list.
Automatic updates, when enabled, apply to verified outdated servers across the
fleet, with one attempt per server and published revision. Existing pending
maintenance is reused. Occupied servers and servers with conflicting beta or
campaign work wait without consuming a restart slot.

Plutainer's startup updater must be enabled (`PLUTAINER_AUTO_UPDATE`). Electric
Cherry does not download binaries, pull a container image, or pin the runtime to
a particular revision. The observed published revision is recorded as the minimum
expected result; the actual revision is queried after recovery.

A healthy restart and a successful update are separate outcomes. If the server
recovers on an older or unknown revision, the restart is successful in Provisioning
and the update needs attention in Maintenance. Update verification never triggers
a rebuild or an automatic restart loop. Inspect the updater configuration and
server logs, then use **Retry after inspection** to create a new attempt. Ambiguous
or failed lifecycle requests require inspection of the existing Provisioning
operation and its retained recovery slot first.

Update attempts persist as suppression evidence across app restarts and history
retention. Cancelling an automatic attempt suppresses another automatic attempt for
that same revision; an operator can request an update explicitly.

## Scheduled restarts

Daily restarts are independent of update availability. At the saved queue time,
Maintenance requests a restart for each server whose desired state is running.
These requests have no revision requirement. Servers owned by Provisioning are
skipped for that occurrence; the schedule advances normally. Requests already
queued can wait for players, campaign work, or beta work to finish.

Saving a schedule uses the selected timezone and follows daylight saving changes.
Skipped times run at the first valid minute after the gap; repeated times run once
on their first occurrence. Legacy UTC schedules keep their timing until saved.
Changing the browser display timezone does not change the saved schedule.

Schedules survive application restarts. Missed days, including while globally
paused, become one run on resume. Pending work is deduplicated per server. The
worker starts on the first application request, so keep the app running and
reachable. Finished scheduled and legacy maintenance activity is retained for
30 days by default (configurable in Admin → Settings); update attempt records remain durable.

## Health recovery and configuration drift

Fleet keeps successful game and running-container reports fresh for 90 seconds.
Two consecutive failed monitoring checks mark RCON health stale; a first timeout
or send failure remains visible in diagnostics without changing the health badge.
Configuration and invalid-reply errors, including authentication failures, take
effect immediately, as do failed manual probes. Successful RCON checks reset the
failure count; skipped checks do not count. Without fresh telemetry, RCON evidence
older than 120 seconds is offline. Explicit stopped/unhealthy container states
and lifecycle changes still take precedence.

The status monitor collects container and game health every 30 seconds. Once per
minute, the watchdog evaluates that completed evidence without issuing its own
routine health probes. Samples expire after 90 seconds; missing or stale evidence
leaves health unknown and cannot justify recovery. Samples retain their original
timestamps and instance identity, and each failure sample counts at most once.
A monitoring gap or changed instance resets the sustained failure window. After
an application restart, routine watchdog checks wait for a new status-monitor pass.
**Check now** and recovery verification still request fresh evidence, and
Provisioning retains its checks before executing recovery actions.

Three observations over at least two minutes are required before an automatic
health restart; starting or restarting containers get five minutes. Unknown host-agent responses are observations,
not proof of a stopped container. Repeated manual checks do not accelerate this
window. A successful routine or manual game-status probe between scans resets
the failure streak. Missing RCON configuration, unresolved ports, and invalid
replies remain diagnostic findings and cannot authorize health recovery.

A health finding gets one restart, then one rebuild if recovery fails after five
minutes and fresh health checks still justify recovery. If current container and
game checks succeed before an automatic health restart or rebuild is submitted,
the obsolete operation is cancelled and the finding is resolved. Unknown health
holds recovery for another check. Drift repairs and requested lifecycle operations
retain their own eligibility rules.

Escalation transfers only that finding's restart slot. A failed rebuild
requires operator attention. **Rebuild deletes all instance files** and uses the
latest promoted setup, the same server identity and port, and the saved server key.
Missing keys block teardown. Submitted rebuild verification continues while paused.

Drift includes missing promoted files, changed bytes, and extra files in managed
script, config, and plugin directories. Config comparison includes the composed
baseline, overlays, and the selected map. Unmanaged default config is exempt until
a baseline is promoted. Missing mounts are scan errors, not empty directories.
Routine drift scans run every five minutes per server, reducing file traffic
while health observations continue every minute. **Check now**, a replacement
registration, or a new lifecycle submission bypasses the drift interval. Failed
file scans retry on the next observer pass; post-rebuild verification still reads
fresh files.

Drift repair creates a Campaign capturing observed differences and requests a
Provisioning rebuild. The Campaign observes that rebuild without requesting
another restart. Its detection evidence and the actual launch recipe remain
available even if promotions change while the work waits. Histories survive
instance removal. Active beta and campaign work defers new recovery.

**Dismiss** suppresses health/drift attempts until the condition clears. **Retry
recovery** begins a fresh bounded attempt after inspection. Existing operations
must finish or be cancelled first. Fresh verification resolves the finding.

## Compatibility and API

Existing job, incident, and operation IDs are retained. Upgrade migration adds
request metadata without replaying commands; existing pending requests keep their
original verification requirements. `/watchdog` and `/watchdog/:id` redirect to
Maintenance, preserving query parameters. Old Watchdog and Plutonium APIs remain
compatible. The previous Watchdog enabled switch controls health and drift only.

- `GET /api/maintenance`: policies, recovery findings, release/version coverage,
  schedule, and recent runtime activity with linked operation stages.
- `POST /api/maintenance`: actions `scan`, `pause` (`paused` boolean), `policy`
  (`policy`: `health`, `drift`, or `updates`; `enabled` boolean), `update` (optional
  `servers` array), `schedule` (`enabled`, `time`, `timeZone`), `cancel` or `retry`
  (`id`, the numeric runtime maintenance activity ID).
- `/api/maintenance/:id`: GET recovery evidence and attempts; POST `retry` or
  `dismiss` for a health/drift finding.