feat: add Watchdog recovery and drift rebuild workflows

docs/watchdog.md

@@ -0,0 +1,80 @@
+# Watchdog
+
+**Operations → Watchdog** detects unhealthy servers and drift, records each
+incident, and links its recovery attempts to Provisioning. File drift also opens
+a Campaign with the observed file differences. Automation is enabled by default;
+**Pause automation** persists across application restarts. Scanning and verification
+continue while paused, but new actions and queued Watchdog actions are held.
+Already submitted actions continue to completion.
+
+The worker scans once per minute while Electric Cherry is running. **Scan now**
+requests a scan immediately. Repeated button presses do not count as independent
+health failures. A health incident must have three observations spanning at least
+two minutes before an automatic restart is queued. Starting or restarting
+containers get five minutes before recovery is requested. Unknown Komodo responses
+are recorded without interpreting them as a stopped container.
+
+## Recovery
+
+A health incident gets one restart, then one rebuild if the restart has not
+recovered within five minutes. Both are normal Provisioning jobs with durable
+history and shared capacity limits. Escalation transfers only that incident's
+restart recovery slot to the rebuild. An unsuccessful rebuild requires operator
+attention; Watchdog does not start an endless restart/rebuild loop. Verification
+of an already submitted rebuild continues, including after the incident is marked
+as needing attention.
+
+An intentional **Stop** is retained as the server's desired state. Watchdog leaves
+it stopped until an explicit Start, Restart, or Rebuild request changes that intent.
+Active creations, removals, betas, and campaigns defer new Watchdog actions.
+
+Provisioning checks current container health before using game status. A disruptive
+action requires a confirmed stopped container or a fresh zero-player reading from
+RCON or accepted telemetry. A positive player count from either source prevents
+an interruption. Telemetry evidence must have been received within 45 seconds;
+future, malformed, stale, and pre-restart reports cannot verify a new process.
+A game-end event retains its reported roster; it does not imply zero players.
+Fresh telemetry can also verify game activity and the selected engine map after
+Komodo reports the container healthy. A healthy container with neither current
+RCON status nor current telemetry remains queued for player-count verification.
+
+## Drift
+
+Drift includes missing promoted files, changed bytes, and extra files in managed
+script, config, and plugin destinations. Config comparison uses the composed
+baseline and overlays with the server's selected map. Plutainer's unmanaged default
+config remains exempt until a baseline is promoted. Active betas are deferred;
+missing server mounts are reported as scan errors instead of being treated as
+empty directories.
+
+Watchdog rebuilds a drifting server in place using the same identity, map, port,
+and saved server key. **Rebuild deletes all old instance files.** The new instance
+uses the latest promoted setup when provisioning starts. A missing saved key
+leaves the incident needing attention without deleting the old server.
+
+The linked Campaign captures before/after evidence at detection time. Its activity
+identifies this as the detected plan, and it follows the rebuild's outcome without
+queuing an additional restart. Promotions can change while the rebuild is queued;
+the Provisioning launch recipe records the files actually selected for that launch.
+Campaign evidence and Watchdog history survive subsequent rebuilds and removals.
+
+## Review and retry
+
+The list can be filtered by server, incident type, and status. Each detail page
+shows the triggering evidence, timestamped events, and links to every attempt.
+**Dismiss** suppresses further automatic attempts until the condition clears.
+**Retry** explicitly starts a new bounded attempt after inspecting a dismissed or
+failed incident. Active Provisioning operations must finish or be cancelled first.
+A verified recovery resolves the incident automatically.
+
+## API
+
+- `GET /api/watchdog`: automation setting, last scan, scan errors, and latest incidents.
+- `POST /api/watchdog` with `{ "action": "scan" }`: run a scan now.
+- `POST /api/watchdog` with `{ "enabled": false }` (or `true`): pause/resume automation.
+- `GET /api/watchdog/:id`: incident, event history, and linked actions.
+- `POST /api/watchdog/:id` with `{ "action": "retry" }` or `{ "action": "dismiss" }`.
+
+The legacy `/api/drift` GET and dry-run POST still return previews. Mutating POST
+requests return HTTP 410 and direct clients to Watchdog; file repair is now a
+provisioning rebuild instead of an immediate rewrite.