electric-cherry/docs/watchdog.md

Watchdog

Operations → Watchdog detects unhealthy servers and drift, records each incident, and links its recovery attempts to Provisioning. File drift also opens a Campaign with the observed file differences. Automation is enabled by default; Pause automation persists across application restarts. Scanning and verification continue while paused, but new actions and queued Watchdog actions are held. Already submitted actions continue to completion.

The worker scans once per minute while Electric Cherry is running. Scan now requests a scan immediately. Repeated button presses do not count as independent health failures. A health incident must have three observations spanning at least two minutes before an automatic restart is queued. Starting or restarting containers get five minutes before recovery is requested. Unknown Komodo responses are recorded without interpreting them as a stopped container.

Recovery

A health incident gets one restart, then one rebuild if the restart has not recovered within five minutes. Both are normal Provisioning jobs with durable history and shared capacity limits. Escalation transfers only that incident's restart recovery slot to the rebuild. An unsuccessful rebuild requires operator attention; Watchdog does not start an endless restart/rebuild loop. Verification of an already submitted rebuild continues, including after the incident is marked as needing attention.

An intentional Stop is retained as the server's desired state. Watchdog leaves it stopped until an explicit Start, Restart, or Rebuild request changes that intent. Active creations, removals, betas, and campaigns defer new Watchdog actions.

Provisioning checks current container health before using game status. A disruptive action requires a confirmed stopped container or a fresh zero-player reading from RCON or accepted telemetry. A positive player count from either source prevents an interruption. Telemetry evidence must have been received within 45 seconds; future, malformed, stale, and pre-restart reports cannot verify a new process. A game-end event retains its reported roster; it does not imply zero players. Fresh telemetry can also verify game activity and the selected engine map after Komodo reports the container healthy. A healthy container with neither current RCON status nor current telemetry remains queued for player-count verification.

Drift

Drift includes missing promoted files, changed bytes, and extra files in managed script, config, and plugin destinations. Config comparison uses the composed baseline and overlays with the server's selected map. Plutainer's unmanaged default config remains exempt until a baseline is promoted. Active betas are deferred; missing server mounts are reported as scan errors instead of being treated as empty directories.

Watchdog rebuilds a drifting server in place using the same identity, map, port, and saved server key. Rebuild deletes all old instance files. The new instance uses the latest promoted setup when provisioning starts. A missing saved key leaves the incident needing attention without deleting the old server.

The linked Campaign captures before/after evidence at detection time. Its activity identifies this as the detected plan, and it follows the rebuild's outcome without queuing an additional restart. Promotions can change while the rebuild is queued; the Provisioning launch recipe records the files actually selected for that launch. Campaign evidence and Watchdog history survive subsequent rebuilds and removals.

Review and retry

The list can be filtered by server, incident type, and status. Each detail page shows the triggering evidence, timestamped events, and links to every attempt. Dismiss suppresses further automatic attempts until the condition clears. Retry explicitly starts a new bounded attempt after inspecting a dismissed or failed incident. Active Provisioning operations must finish or be cancelled first. A verified recovery resolves the incident automatically.

API

• GET /api/watchdog: automation setting, last scan, scan errors, and latest incidents.
• POST /api/watchdog with { "action": "scan" }: run a scan now.
• POST /api/watchdog with { "enabled": false } (or true): pause/resume automation.
• GET /api/watchdog/:id: incident, event history, and linked actions.
• POST /api/watchdog/:id with { "action": "retry" } or { "action": "dismiss" }.

The legacy /api/drift GET and dry-run POST still return previews. Mutating POST requests return HTTP 410 and direct clients to Watchdog; file repair is now a provisioning rebuild instead of an immediate rewrite.