feat: unify recovery, updates, and schedules under Maintenance
This commit is contained in:
@@ -67,56 +67,39 @@ status between refreshes. Failed refreshes keep the last snapshot visible.
|
||||
Plugins support named release versions, GitHub DLL imports and update discovery.
|
||||
See [Plugin releases](docs/plugin-releases.md) for the import and beta workflow.
|
||||
|
||||
## Plutonium updates and scheduled restarts
|
||||
## Maintenance
|
||||
|
||||
Open **Settings → Plutonium updates & restarts** to check the latest published
|
||||
revision, inspect each server's running revision, and queue a server or fleet
|
||||
restart. Release and running-version checks also run hourly. The published
|
||||
revision comes from the [official production updater manifest](https://cdn.plutoniummod.com/updater/prod/info.json),
|
||||
which exposes the game revision directly and avoids the changelog website's browser challenge;
|
||||
running revisions come from RCON `version`. Unsupported replies and failed
|
||||
queries appear as unknown rather than being treated as up to date. A failed
|
||||
manifest request retains the last release with a visible error.
|
||||
Open **Operations → Maintenance** for health recovery, configuration drift,
|
||||
Plutonium updates, and scheduled restarts. Overview shows active work, attention,
|
||||
version coverage, and upcoming restarts. Policies & schedules controls each
|
||||
capability independently; Activity links findings and results to Provisioning.
|
||||
Fleet and server pages show runtime revision indicators and maintenance links.
|
||||
|
||||
Daily fleet restarts are optional and disabled by default. Choose a queue time
|
||||
in your **selected timezone** and save the schedule. The timezone is saved with
|
||||
the schedule so the daily time follows daylight saving changes even when your
|
||||
browser is closed. Skipped times run at the first valid minute after the gap;
|
||||
repeated times run once on their first occurrence. Existing UTC schedules retain
|
||||
their original timing until saved again. The form displays their next occurrence
|
||||
in your timezone and identifies the saved zone if it differs. Changing a browser's
|
||||
display preference does not silently change a fleet schedule; save to apply it.
|
||||
Every due run queues the current fleet, regardless of whether an update exists.
|
||||
The schedule and jobs survive app restarts. Missed days become one run, and a
|
||||
server can have only one pending maintenance restart. Disabling the schedule
|
||||
does not cancel work already queued; use **Cancel** for those jobs.
|
||||
Maintenance decides when work is needed and verifies the result. Provisioning
|
||||
executes every lifecycle operation using the shared safety checks and queues.
|
||||
Health recovery uses one restart followed by one rebuild if needed; drift repair
|
||||
requests a rebuild with a linked Campaign. **Rebuild deletes all instance files.**
|
||||
|
||||
Maintenance waits for a fresh zero-player RCON or telemetry report and for existing
|
||||
deployment/beta work, then uses the shared rolling restart limit. Unreachable or
|
||||
occupied servers remain queued. After container and game-status recovery, Electric
|
||||
Cherry queries the running revision again. If a fresh published revision was
|
||||
known when queued, recovery on an older or unknown revision is reported as a
|
||||
failed verification, without an automatic restart loop. Ambiguous restart
|
||||
failures likewise require inspection before a new request. Activity is retained
|
||||
for 30 days, with pending jobs and the latest 100 records shown in Settings.
|
||||
Plutonium updates are applied by Plutainer's startup updater
|
||||
(`PLUTAINER_AUTO_UPDATE` must be enabled). Check versions and choose **Update on
|
||||
restart**, or opt into automatic updates for verified outdated servers. Revision
|
||||
checks run hourly. Unknown or stale revisions remain unverified. A healthy restart
|
||||
can still fail update verification; Maintenance reports that outcome without
|
||||
rebuilding or repeatedly restarting the server. Electric Cherry does not download
|
||||
binaries or pin a runtime revision.
|
||||
|
||||
Updates rely on Plutainer's startup updater (`PLUTAINER_AUTO_UPDATE` must be
|
||||
enabled); Electric Cherry does not download binaries or pull a container image.
|
||||
See [Plutainer's startup flow](https://github.com/Ayymoss/Plutainer/blob/main/CLAUDE.md)
|
||||
and [environment migration reference](https://github.com/Ayymoss/Plutainer/blob/main/MIGRATION.md).
|
||||
The worker starts on the first application request, so keep the app running and
|
||||
reachable for scheduled maintenance. Validate RCON revision replies and updater
|
||||
behavior on your live T5/T6 servers before relying on unattended maintenance.
|
||||
Optional daily restarts are independent of updates. Schedules use the saved
|
||||
timezone, follow daylight saving changes, and coalesce missed days into one run.
|
||||
Intentionally stopped servers are excluded. Existing schedules, pending jobs,
|
||||
operation IDs, and history survive the move from Settings and Watchdog.
|
||||
|
||||
**Pause all maintenance** holds new and queued maintenance while checks and
|
||||
submitted recovery continue. Each capability retains its own policy. Automatic
|
||||
updates remain disabled until explicitly enabled. See [Maintenance](docs/maintenance.md)
|
||||
for verification, retry, scheduling, and compatibility details.
|
||||
|
||||
## How restarts stay safe
|
||||
|
||||
**Operations → Watchdog** automatically detects unhealthy servers and file drift.
|
||||
Health incidents receive one restart followed by one rebuild if recovery fails;
|
||||
drift queues a rebuild with the promoted setup and a linked Campaign. Incidents,
|
||||
evidence, and attempts survive application restarts. Pause automation, inspect
|
||||
failures, and explicitly retry from Watchdog. Intentional stops and active beta or
|
||||
campaign work are respected. See [Watchdog](docs/watchdog.md) for timing and recovery behavior.
|
||||
|
||||
Betas support configurable gameplay, participation, stability and manual-review
|
||||
requirements, saved presets, coverage rules and staged waves. See
|
||||
[Beta test plans](docs/beta-test-plans.md) for usage and the four-event game telemetry contract.
|
||||
|
||||
Reference in New Issue
Block a user