electric-cherry/docs/maintenance.md

# Maintenance

Open **Operations → Maintenance** to keep existing servers healthy and current.
Maintenance owns observations, policies, schedules, requests, and verification of
maintenance outcomes. Provisioning owns lifecycle execution, player checks,
conflicts, rolling capacity, container recovery, and persistent operation history.
Campaigns retain managed-file deployment evidence; Collections define lineups.

## Views

- **Overview** shows active work, attention, upcoming restarts, and Plutonium version
  coverage. Filter by capability or server. **Check now** refreshes health, drift,
  the production release, and running revisions.
- **Policies & schedules** controls health recovery, configuration drift, automatic
  Plutonium updates, and the optional daily restart schedule independently.
- **Activity** combines recovery findings, update results, and scheduled restarts.
  Follow a finding for evidence and attempts, or its Provisioning operation for
  execution progress. Cancel unsubmitted runtime restarts here.

Fleet and server details link their revision indicators to Maintenance. Failed
runtime maintenance appears in Fleet attention, even when the underlying restart
succeeded. Provisioning and repair Campaigns link back to the requesting work.

## Policies and pause

Health recovery and drift repair inherit the previous Watchdog enabled setting.
Automatic Plutonium updates are **disabled by default**. Existing daily schedules
retain their enabled state, timezone, and next occurrence.

**Pause all maintenance** holds new remediation, scheduled runs, and unsubmitted
maintenance operations. Scans and result verification continue; submitted
operations finish recovery. Campaigns, betas, and manually requested Provisioning
operations continue independently. Resuming keeps each policy's saved setting.

Disabling health recovery or drift repair holds queued work for that capability
and prevents further automatic attempts, including health escalation. Disabling
automatic updates holds queued automatic update requests. Explicitly requested
updates remain operator-authorized work. Disabling a daily schedule prevents
future runs but does not cancel requests it already created.

All capabilities respect intentionally stopped servers. Start them explicitly
through Provisioning before expecting maintenance to act on them.

## Plutonium updates

The published revision comes from the official production updater manifest.
Running revisions come from RCON `version` after checking container readiness.
Checks run hourly while the application is running. Failed release checks retain
the cached release with an error. Unsupported replies, stale checks, unhealthy
containers, and evidence from before a lifecycle submission remain unverified.
A missing revision is never treated as current or as an automatic update target.

**Update on restart** requests ordinary Provisioning restarts for verified outdated
servers. **Update outdated servers on restart** applies to the filtered list.
Automatic updates, when enabled, apply to verified outdated servers across the
fleet, with one attempt per server and published revision. Existing pending
maintenance is reused. Occupied servers and servers with conflicting beta or
campaign work wait without consuming a restart slot.

Plutainer's startup updater must be enabled (`PLUTAINER_AUTO_UPDATE`). Electric
Cherry does not download binaries, pull a container image, or pin the runtime to
a particular revision. The observed published revision is recorded as the minimum
expected result; the actual revision is queried after recovery.

A healthy restart and a successful update are separate outcomes. If the server
recovers on an older or unknown revision, the restart is successful in Provisioning
and the update needs attention in Maintenance. Update verification never triggers
a rebuild or an automatic restart loop. Inspect the updater configuration and
server logs, then use **Retry after inspection** to create a new attempt. Ambiguous
or failed lifecycle requests require inspection of the existing Provisioning
operation and its retained recovery slot first.

Update attempts persist as suppression evidence across app restarts and history
retention. Cancelling an automatic attempt suppresses another automatic attempt for
that same revision; an operator can request an update explicitly.

## Scheduled restarts

Daily restarts are independent of update availability. At the saved queue time,
Maintenance requests a restart for each server whose desired state is running.
These requests have no revision requirement. Busy servers can restart later.

Saving a schedule uses the selected timezone and follows daylight saving changes.
Skipped times run at the first valid minute after the gap; repeated times run once
on their first occurrence. Legacy UTC schedules keep their timing until saved.
Changing the browser display timezone does not change the saved schedule.

Schedules survive application restarts. Missed days, including while globally
paused, become one run on resume. Pending work is deduplicated per server. The
worker starts on the first application request, so keep the app running and
reachable. Finished scheduled and legacy maintenance activity is retained for
30 days; update attempt records remain durable.

## Health recovery and configuration drift

The health observer (the watchdog) scans once per minute. Three observations over
at least two minutes are required before an automatic health restart; starting or
restarting containers get five minutes. Unknown Komodo responses are observations,
not proof of a stopped container. Repeated manual checks do not accelerate this
window.

A health finding gets one restart, then one rebuild if recovery fails after five
minutes. Escalation transfers only that finding's restart slot. A failed rebuild
requires operator attention. **Rebuild deletes all instance files** and uses the
latest promoted setup, the same server identity and port, and the saved server key.
Missing keys block teardown. Submitted rebuild verification continues while paused.

Drift includes missing promoted files, changed bytes, and extra files in managed
script, config, and plugin directories. Config comparison includes the composed
baseline, overlays, and the selected map. Unmanaged default config is exempt until
a baseline is promoted. Missing mounts are scan errors, not empty directories.

Drift repair creates a Campaign capturing observed differences and requests a
Provisioning rebuild. The Campaign observes that rebuild without requesting
another restart. Its detection evidence and the actual launch recipe remain
available even if promotions change while the work waits. Histories survive
instance removal. Active beta and campaign work defers new recovery.

**Dismiss** suppresses health/drift attempts until the condition clears. **Retry
recovery** begins a fresh bounded attempt after inspection. Existing operations
must finish or be cancelled first. Fresh verification resolves the finding.

## Compatibility and API

Existing job, incident, and operation IDs are retained. Upgrade migration adds
request metadata without replaying commands; existing pending requests keep their
original verification requirements. `/watchdog` and `/watchdog/:id` redirect to
Maintenance, preserving query parameters. Old Watchdog and Plutonium APIs remain
compatible. The previous Watchdog enabled switch controls health and drift only.

- `GET /api/maintenance`: policies, recovery findings, release/version coverage,
  schedule, and recent runtime activity with linked operation stages.
- `POST /api/maintenance`: actions `scan`, `pause` (`paused` boolean), `policy`
  (`policy`: `health`, `drift`, or `updates`; `enabled` boolean), `update` (optional
  `servers` array), `schedule` (`enabled`, `time`, `timeZone`), `cancel` or `retry`
  (`id`, the numeric runtime maintenance activity ID).
- `/api/maintenance/:id`: GET recovery evidence and attempts; POST `retry` or
  `dismiss` for a health/drift finding.