feat: unify recovery, updates, and schedules under Maintenance

This commit is contained in:
2026-09-27 16:00:18 -04:00
parent 8ea696417f
commit fc46cb9da4
42 changed files with 1256 additions and 833 deletions
+26 -43
View File
@@ -67,56 +67,39 @@ status between refreshes. Failed refreshes keep the last snapshot visible.
Plugins support named release versions, GitHub DLL imports and update discovery.
See [Plugin releases](docs/plugin-releases.md) for the import and beta workflow.
## Plutonium updates and scheduled restarts
## Maintenance
Open **Settings → Plutonium updates & restarts** to check the latest published
revision, inspect each server's running revision, and queue a server or fleet
restart. Release and running-version checks also run hourly. The published
revision comes from the [official production updater manifest](https://cdn.plutoniummod.com/updater/prod/info.json),
which exposes the game revision directly and avoids the changelog website's browser challenge;
running revisions come from RCON `version`. Unsupported replies and failed
queries appear as unknown rather than being treated as up to date. A failed
manifest request retains the last release with a visible error.
Open **Operations → Maintenance** for health recovery, configuration drift,
Plutonium updates, and scheduled restarts. Overview shows active work, attention,
version coverage, and upcoming restarts. Policies & schedules controls each
capability independently; Activity links findings and results to Provisioning.
Fleet and server pages show runtime revision indicators and maintenance links.
Daily fleet restarts are optional and disabled by default. Choose a queue time
in your **selected timezone** and save the schedule. The timezone is saved with
the schedule so the daily time follows daylight saving changes even when your
browser is closed. Skipped times run at the first valid minute after the gap;
repeated times run once on their first occurrence. Existing UTC schedules retain
their original timing until saved again. The form displays their next occurrence
in your timezone and identifies the saved zone if it differs. Changing a browser's
display preference does not silently change a fleet schedule; save to apply it.
Every due run queues the current fleet, regardless of whether an update exists.
The schedule and jobs survive app restarts. Missed days become one run, and a
server can have only one pending maintenance restart. Disabling the schedule
does not cancel work already queued; use **Cancel** for those jobs.
Maintenance decides when work is needed and verifies the result. Provisioning
executes every lifecycle operation using the shared safety checks and queues.
Health recovery uses one restart followed by one rebuild if needed; drift repair
requests a rebuild with a linked Campaign. **Rebuild deletes all instance files.**
Maintenance waits for a fresh zero-player RCON or telemetry report and for existing
deployment/beta work, then uses the shared rolling restart limit. Unreachable or
occupied servers remain queued. After container and game-status recovery, Electric
Cherry queries the running revision again. If a fresh published revision was
known when queued, recovery on an older or unknown revision is reported as a
failed verification, without an automatic restart loop. Ambiguous restart
failures likewise require inspection before a new request. Activity is retained
for 30 days, with pending jobs and the latest 100 records shown in Settings.
Plutonium updates are applied by Plutainer's startup updater
(`PLUTAINER_AUTO_UPDATE` must be enabled). Check versions and choose **Update on
restart**, or opt into automatic updates for verified outdated servers. Revision
checks run hourly. Unknown or stale revisions remain unverified. A healthy restart
can still fail update verification; Maintenance reports that outcome without
rebuilding or repeatedly restarting the server. Electric Cherry does not download
binaries or pin a runtime revision.
Updates rely on Plutainer's startup updater (`PLUTAINER_AUTO_UPDATE` must be
enabled); Electric Cherry does not download binaries or pull a container image.
See [Plutainer's startup flow](https://github.com/Ayymoss/Plutainer/blob/main/CLAUDE.md)
and [environment migration reference](https://github.com/Ayymoss/Plutainer/blob/main/MIGRATION.md).
The worker starts on the first application request, so keep the app running and
reachable for scheduled maintenance. Validate RCON revision replies and updater
behavior on your live T5/T6 servers before relying on unattended maintenance.
Optional daily restarts are independent of updates. Schedules use the saved
timezone, follow daylight saving changes, and coalesce missed days into one run.
Intentionally stopped servers are excluded. Existing schedules, pending jobs,
operation IDs, and history survive the move from Settings and Watchdog.
**Pause all maintenance** holds new and queued maintenance while checks and
submitted recovery continue. Each capability retains its own policy. Automatic
updates remain disabled until explicitly enabled. See [Maintenance](docs/maintenance.md)
for verification, retry, scheduling, and compatibility details.
## How restarts stay safe
**Operations → Watchdog** automatically detects unhealthy servers and file drift.
Health incidents receive one restart followed by one rebuild if recovery fails;
drift queues a rebuild with the promoted setup and a linked Campaign. Incidents,
evidence, and attempts survive application restarts. Pause automation, inspect
failures, and explicitly retry from Watchdog. Intentional stops and active beta or
campaign work are respected. See [Watchdog](docs/watchdog.md) for timing and recovery behavior.
Betas support configurable gameplay, participation, stability and manual-review
requirements, saved presets, coverage rules and staged waves. See
[Beta test plans](docs/beta-test-plans.md) for usage and the four-event game telemetry contract.