feat: unify recovery, updates, and schedules under Maintenance
This commit is contained in:
@@ -0,0 +1,138 @@
|
||||
# Maintenance
|
||||
|
||||
Open **Operations → Maintenance** to keep existing servers healthy and current.
|
||||
Maintenance owns observations, policies, schedules, requests, and verification of
|
||||
maintenance outcomes. Provisioning owns lifecycle execution, player checks,
|
||||
conflicts, rolling capacity, container recovery, and persistent operation history.
|
||||
Campaigns retain managed-file deployment evidence; Collections define lineups.
|
||||
|
||||
## Views
|
||||
|
||||
- **Overview** shows active work, attention, upcoming restarts, and Plutonium version
|
||||
coverage. Filter by capability or server. **Check now** refreshes health, drift,
|
||||
the production release, and running revisions.
|
||||
- **Policies & schedules** controls health recovery, configuration drift, automatic
|
||||
Plutonium updates, and the optional daily restart schedule independently.
|
||||
- **Activity** combines recovery findings, update results, and scheduled restarts.
|
||||
Follow a finding for evidence and attempts, or its Provisioning operation for
|
||||
execution progress. Cancel unsubmitted runtime restarts here.
|
||||
|
||||
Fleet and server details link their revision indicators to Maintenance. Failed
|
||||
runtime maintenance appears in Fleet attention, even when the underlying restart
|
||||
succeeded. Provisioning and repair Campaigns link back to the requesting work.
|
||||
|
||||
## Policies and pause
|
||||
|
||||
Health recovery and drift repair inherit the previous Watchdog enabled setting.
|
||||
Automatic Plutonium updates are **disabled by default**. Existing daily schedules
|
||||
retain their enabled state, timezone, and next occurrence.
|
||||
|
||||
**Pause all maintenance** holds new remediation, scheduled runs, and unsubmitted
|
||||
maintenance operations. Scans and result verification continue; submitted
|
||||
operations finish recovery. Campaigns, betas, and manually requested Provisioning
|
||||
operations continue independently. Resuming keeps each policy's saved setting.
|
||||
|
||||
Disabling health recovery or drift repair holds queued work for that capability
|
||||
and prevents further automatic attempts, including health escalation. Disabling
|
||||
automatic updates holds queued automatic update requests. Explicitly requested
|
||||
updates remain operator-authorized work. Disabling a daily schedule prevents
|
||||
future runs but does not cancel requests it already created.
|
||||
|
||||
All capabilities respect intentionally stopped servers. Start them explicitly
|
||||
through Provisioning before expecting maintenance to act on them.
|
||||
|
||||
## Plutonium updates
|
||||
|
||||
The published revision comes from the official production updater manifest.
|
||||
Running revisions come from RCON `version` after checking container readiness.
|
||||
Checks run hourly while the application is running. Failed release checks retain
|
||||
the cached release with an error. Unsupported replies, stale checks, unhealthy
|
||||
containers, and evidence from before a lifecycle submission remain unverified.
|
||||
A missing revision is never treated as current or as an automatic update target.
|
||||
|
||||
**Update on restart** requests ordinary Provisioning restarts for verified outdated
|
||||
servers. **Update outdated servers on restart** applies to the filtered list.
|
||||
Automatic updates, when enabled, apply to verified outdated servers across the
|
||||
fleet, with one attempt per server and published revision. Existing pending
|
||||
maintenance is reused. Occupied servers and servers with conflicting beta or
|
||||
campaign work wait without consuming a restart slot.
|
||||
|
||||
Plutainer's startup updater must be enabled (`PLUTAINER_AUTO_UPDATE`). Electric
|
||||
Cherry does not download binaries, pull a container image, or pin the runtime to
|
||||
a particular revision. The observed published revision is recorded as the minimum
|
||||
expected result; the actual revision is queried after recovery.
|
||||
|
||||
A healthy restart and a successful update are separate outcomes. If the server
|
||||
recovers on an older or unknown revision, the restart is successful in Provisioning
|
||||
and the update needs attention in Maintenance. Update verification never triggers
|
||||
a rebuild or an automatic restart loop. Inspect the updater configuration and
|
||||
server logs, then use **Retry after inspection** to create a new attempt. Ambiguous
|
||||
or failed lifecycle requests require inspection of the existing Provisioning
|
||||
operation and its retained recovery slot first.
|
||||
|
||||
Update attempts persist as suppression evidence across app restarts and history
|
||||
retention. Cancelling an automatic attempt suppresses another automatic attempt for
|
||||
that same revision; an operator can request an update explicitly.
|
||||
|
||||
## Scheduled restarts
|
||||
|
||||
Daily restarts are independent of update availability. At the saved queue time,
|
||||
Maintenance requests a restart for each server whose desired state is running.
|
||||
These requests have no revision requirement. Busy servers can restart later.
|
||||
|
||||
Saving a schedule uses the selected timezone and follows daylight saving changes.
|
||||
Skipped times run at the first valid minute after the gap; repeated times run once
|
||||
on their first occurrence. Legacy UTC schedules keep their timing until saved.
|
||||
Changing the browser display timezone does not change the saved schedule.
|
||||
|
||||
Schedules survive application restarts. Missed days, including while globally
|
||||
paused, become one run on resume. Pending work is deduplicated per server. The
|
||||
worker starts on the first application request, so keep the app running and
|
||||
reachable. Finished scheduled and legacy maintenance activity is retained for
|
||||
30 days; update attempt records remain durable.
|
||||
|
||||
## Health recovery and configuration drift
|
||||
|
||||
The health observer (the watchdog) scans once per minute. Three observations over
|
||||
at least two minutes are required before an automatic health restart; starting or
|
||||
restarting containers get five minutes. Unknown Komodo responses are observations,
|
||||
not proof of a stopped container. Repeated manual checks do not accelerate this
|
||||
window.
|
||||
|
||||
A health finding gets one restart, then one rebuild if recovery fails after five
|
||||
minutes. Escalation transfers only that finding's restart slot. A failed rebuild
|
||||
requires operator attention. **Rebuild deletes all instance files** and uses the
|
||||
latest promoted setup, the same server identity and port, and the saved server key.
|
||||
Missing keys block teardown. Submitted rebuild verification continues while paused.
|
||||
|
||||
Drift includes missing promoted files, changed bytes, and extra files in managed
|
||||
script, config, and plugin directories. Config comparison includes the composed
|
||||
baseline, overlays, and the selected map. Unmanaged default config is exempt until
|
||||
a baseline is promoted. Missing mounts are scan errors, not empty directories.
|
||||
|
||||
Drift repair creates a Campaign capturing observed differences and requests a
|
||||
Provisioning rebuild. The Campaign observes that rebuild without requesting
|
||||
another restart. Its detection evidence and the actual launch recipe remain
|
||||
available even if promotions change while the work waits. Histories survive
|
||||
instance removal. Active beta and campaign work defers new recovery.
|
||||
|
||||
**Dismiss** suppresses health/drift attempts until the condition clears. **Retry
|
||||
recovery** begins a fresh bounded attempt after inspection. Existing operations
|
||||
must finish or be cancelled first. Fresh verification resolves the finding.
|
||||
|
||||
## Compatibility and API
|
||||
|
||||
Existing job, incident, and operation IDs are retained. Upgrade migration adds
|
||||
request metadata without replaying commands; existing pending requests keep their
|
||||
original verification requirements. `/watchdog` and `/watchdog/:id` redirect to
|
||||
Maintenance, preserving query parameters. Old Watchdog and Plutonium APIs remain
|
||||
compatible. The previous Watchdog enabled switch controls health and drift only.
|
||||
|
||||
- `GET /api/maintenance`: policies, recovery findings, release/version coverage,
|
||||
schedule, and recent runtime activity with linked operation stages.
|
||||
- `POST /api/maintenance`: actions `scan`, `pause` (`paused` boolean), `policy`
|
||||
(`policy`: `health`, `drift`, or `updates`; `enabled` boolean), `update` (optional
|
||||
`servers` array), `schedule` (`enabled`, `time`, `timeZone`), `cancel` or `retry`
|
||||
(`id`, the numeric runtime maintenance activity ID).
|
||||
- `/api/maintenance/:id`: GET recovery evidence and attempts; POST `retry` or
|
||||
`dismiss` for a health/drift finding.
|
||||
@@ -49,11 +49,11 @@ Provisioning and observe the resulting operation by ID. Their pages link to that
|
||||
operation, and its history links back to the requesting work. Provisioning owns
|
||||
the empty-server check, conflict handling, restart capacity, remote command, and
|
||||
health verification. The callers retain file deployment, beta evidence, and
|
||||
Plutonium revision verification. Jobs remain visible while waiting for players,
|
||||
maintenance outcome verification. Jobs remain visible while waiting for players,
|
||||
other work, or integration configuration.
|
||||
|
||||
[Watchdog](watchdog.md) also submits recovery operations and links each attempt
|
||||
to its incident. A failed Watchdog restart can hand its reserved slot to a rebuild
|
||||
[Maintenance](maintenance.md) also submits recovery operations and links each attempt
|
||||
to its incident. A failed Maintenance restart can hand its reserved slot to a rebuild
|
||||
after five minutes; unrelated recovery slots remain held. Drift campaigns follow
|
||||
their linked rebuild without requesting another restart. Their file evidence and
|
||||
history remain available after the instance is removed.
|
||||
|
||||
+5
-78
@@ -1,80 +1,7 @@
|
||||
# Watchdog
|
||||
# Watchdog has moved to Maintenance
|
||||
|
||||
**Operations → Watchdog** detects unhealthy servers and drift, records each
|
||||
incident, and links its recovery attempts to Provisioning. File drift also opens
|
||||
a Campaign with the observed file differences. Automation is enabled by default;
|
||||
**Pause automation** persists across application restarts. Scanning and verification
|
||||
continue while paused, but new actions and queued Watchdog actions are held.
|
||||
Already submitted actions continue to completion.
|
||||
Open **Operations → Maintenance** for health recovery and drift repair, together
|
||||
with Plutonium updates and scheduled restarts. Existing Watchdog URLs redirect
|
||||
to Maintenance; incidents, operation IDs, and history are preserved.
|
||||
|
||||
The worker scans once per minute while Electric Cherry is running. **Scan now**
|
||||
requests a scan immediately. Repeated button presses do not count as independent
|
||||
health failures. A health incident must have three observations spanning at least
|
||||
two minutes before an automatic restart is queued. Starting or restarting
|
||||
containers get five minutes before recovery is requested. Unknown Komodo responses
|
||||
are recorded without interpreting them as a stopped container.
|
||||
|
||||
## Recovery
|
||||
|
||||
A health incident gets one restart, then one rebuild if the restart has not
|
||||
recovered within five minutes. Both are normal Provisioning jobs with durable
|
||||
history and shared capacity limits. Escalation transfers only that incident's
|
||||
restart recovery slot to the rebuild. An unsuccessful rebuild requires operator
|
||||
attention; Watchdog does not start an endless restart/rebuild loop. Verification
|
||||
of an already submitted rebuild continues, including after the incident is marked
|
||||
as needing attention.
|
||||
|
||||
An intentional **Stop** is retained as the server's desired state. Watchdog leaves
|
||||
it stopped until an explicit Start, Restart, or Rebuild request changes that intent.
|
||||
Active creations, removals, betas, and campaigns defer new Watchdog actions.
|
||||
|
||||
Provisioning checks current container health before using game status. A disruptive
|
||||
action requires a confirmed stopped container or a fresh zero-player reading from
|
||||
RCON or accepted telemetry. A positive player count from either source prevents
|
||||
an interruption. Telemetry evidence must have been received within 45 seconds;
|
||||
future, malformed, stale, and pre-restart reports cannot verify a new process.
|
||||
A game-end event retains its reported roster; it does not imply zero players.
|
||||
Fresh telemetry can also verify game activity and the selected engine map after
|
||||
Komodo reports the container healthy. A healthy container with neither current
|
||||
RCON status nor current telemetry remains queued for player-count verification.
|
||||
|
||||
## Drift
|
||||
|
||||
Drift includes missing promoted files, changed bytes, and extra files in managed
|
||||
script, config, and plugin destinations. Config comparison uses the composed
|
||||
baseline and overlays with the server's selected map. Plutainer's unmanaged default
|
||||
config remains exempt until a baseline is promoted. Active betas are deferred;
|
||||
missing server mounts are reported as scan errors instead of being treated as
|
||||
empty directories.
|
||||
|
||||
Watchdog rebuilds a drifting server in place using the same identity, map, port,
|
||||
and saved server key. **Rebuild deletes all old instance files.** The new instance
|
||||
uses the latest promoted setup when provisioning starts. A missing saved key
|
||||
leaves the incident needing attention without deleting the old server.
|
||||
|
||||
The linked Campaign captures before/after evidence at detection time. Its activity
|
||||
identifies this as the detected plan, and it follows the rebuild's outcome without
|
||||
queuing an additional restart. Promotions can change while the rebuild is queued;
|
||||
the Provisioning launch recipe records the files actually selected for that launch.
|
||||
Campaign evidence and Watchdog history survive subsequent rebuilds and removals.
|
||||
|
||||
## Review and retry
|
||||
|
||||
The list can be filtered by server, incident type, and status. Each detail page
|
||||
shows the triggering evidence, timestamped events, and links to every attempt.
|
||||
**Dismiss** suppresses further automatic attempts until the condition clears.
|
||||
**Retry** explicitly starts a new bounded attempt after inspecting a dismissed or
|
||||
failed incident. Active Provisioning operations must finish or be cancelled first.
|
||||
A verified recovery resolves the incident automatically.
|
||||
|
||||
## API
|
||||
|
||||
- `GET /api/watchdog`: automation setting, last scan, scan errors, and latest incidents.
|
||||
- `POST /api/watchdog` with `{ "action": "scan" }`: run a scan now.
|
||||
- `POST /api/watchdog` with `{ "enabled": false }` (or `true`): pause/resume automation.
|
||||
- `GET /api/watchdog/:id`: incident, event history, and linked actions.
|
||||
- `POST /api/watchdog/:id` with `{ "action": "retry" }` or `{ "action": "dismiss" }`.
|
||||
|
||||
The legacy `/api/drift` GET and dry-run POST still return previews. Mutating POST
|
||||
requests return HTTP 410 and direct clients to Watchdog; file repair is now a
|
||||
provisioning rebuild instead of an immediate rewrite.
|
||||
See [Maintenance](maintenance.md) for policies, recovery behavior, and the API.
|
||||
|
||||
Reference in New Issue
Block a user