feat: add Watchdog recovery and drift rebuild workflows

This commit is contained in:
2026-09-27 15:13:38 -04:00
parent a7044f21a1
commit b9f5902945
36 changed files with 2131 additions and 424 deletions
+12 -3
View File
@@ -81,7 +81,7 @@ The schedule and jobs survive app restarts. Missed days become one run, and a
server can have only one pending maintenance restart. Disabling the schedule
does not cancel work already queued; use **Cancel** for those jobs.
Maintenance waits for a fresh zero-player RCON response and for existing
Maintenance waits for a fresh zero-player RCON or telemetry report and for existing
deployment/beta work, then uses the shared rolling restart limit. Unreachable or
occupied servers remain queued. After container and RCON recovery, Electric
Cherry queries the running revision again. If a fresh published revision was
@@ -100,6 +100,13 @@ behavior on your live T5/T6 servers before relying on unattended maintenance.
## How restarts stay safe
**Operations → Watchdog** automatically detects unhealthy servers and file drift.
Health incidents receive one restart followed by one rebuild if recovery fails;
drift queues a rebuild with the promoted setup and a linked Campaign. Incidents,
evidence, and attempts survive application restarts. Pause automation, inspect
failures, and explicitly retry from Watchdog. Intentional stops and active beta or
campaign work are respected. See [Watchdog](docs/watchdog.md) for timing and recovery behavior.
Betas support configurable gameplay, participation, stability and manual-review
requirements, saved presets, coverage rules and staged waves. See
[Beta test plans](docs/beta-test-plans.md) for usage and the four-event game telemetry contract.
@@ -300,8 +307,10 @@ the full player data, updates Fleet, and applies gameplay progress to active
betas automatically. No discovery request or beta-specific fields are needed.
See [Game reporter protocol](docs/beta-test-plans.md#game-reporter-protocol) for
complete payload examples. Telemetry does not replace RCON's empty-server
check before an automated restart.
complete payload examples. Fresh telemetry can verify player count and game
activity alongside RCON during provisioning and Watchdog recovery. Occupied
reports prevent interruptions; stale reports cannot authorize them. Komodo still
verifies container readiness.
For a missing scoreboard, open the server's **Diagnostics → Telemetry API requests**.
The log shows received requests for that exact server name, including accepted