feat: add Watchdog recovery and drift rebuild workflows

docs/provisioning.md

@@ -30,12 +30,16 @@ failures before making another request.
 
 Start and Restart share the rolling restart limit with campaigns, betas, and
 maintenance. A successful API reply is followed by a recovery delay and actual
-container/RCON checks. Failed or ambiguous restart requests retain their recovery
+container and game-status checks. Failed or ambiguous restart requests retain their recovery
 slot and are not automatically replayed. Stop requires an explicit stopped container state; an
 unreachable Komodo API is not evidence that a stop succeeded.
 
-Disruptive operations require a fresh zero-player RCON result, unless Komodo
-positively reports a stopped container. Runtime actions wait for conflicting
+Disruptive operations require a fresh zero-player result from RCON or accepted
+telemetry, unless Komodo positively reports a stopped container. Fresh occupied
+telemetry vetoes a conflicting empty RCON reading. Reports must be received within
+45 seconds; recovery verification also excludes reports from before the submitted
+operation. Telemetry can verify the game's activity and selected map after Komodo
+reports healthy. Runtime actions wait for conflicting
 campaign, beta, maintenance, or provisioning work. Duplicate pending requests for
 the same action return the existing operation; a different action returns a
 conflict. Requests and events survive application restarts.
@@ -48,6 +52,12 @@ health verification. The callers retain file deployment, beta evidence, and
 Plutonium revision verification. Jobs remain visible while waiting for players,
 other work, or integration configuration.
 
+[Watchdog](watchdog.md) also submits recovery operations and links each attempt
+to its incident. A failed Watchdog restart can hand its reserved slot to a rebuild
+after five minutes; unrelated recovery slots remain held. Drift campaigns follow
+their linked rebuild without requesting another restart. Their file evidence and
+history remain available after the instance is removed.
+
 Campaign changes queued before submission can share one restart. Changes arriving
 after submission require a new operation. A failed or interrupted request keeps
 its original ID and never automatically replays. An explicit beta retry starts a

docs/watchdog.md

@@ -0,0 +1,80 @@
+# Watchdog
+
+**Operations → Watchdog** detects unhealthy servers and drift, records each
+incident, and links its recovery attempts to Provisioning. File drift also opens
+a Campaign with the observed file differences. Automation is enabled by default;
+**Pause automation** persists across application restarts. Scanning and verification
+continue while paused, but new actions and queued Watchdog actions are held.
+Already submitted actions continue to completion.
+
+The worker scans once per minute while Electric Cherry is running. **Scan now**
+requests a scan immediately. Repeated button presses do not count as independent
+health failures. A health incident must have three observations spanning at least
+two minutes before an automatic restart is queued. Starting or restarting
+containers get five minutes before recovery is requested. Unknown Komodo responses
+are recorded without interpreting them as a stopped container.
+
+## Recovery
+
+A health incident gets one restart, then one rebuild if the restart has not
+recovered within five minutes. Both are normal Provisioning jobs with durable
+history and shared capacity limits. Escalation transfers only that incident's
+restart recovery slot to the rebuild. An unsuccessful rebuild requires operator
+attention; Watchdog does not start an endless restart/rebuild loop. Verification
+of an already submitted rebuild continues, including after the incident is marked
+as needing attention.
+
+An intentional **Stop** is retained as the server's desired state. Watchdog leaves
+it stopped until an explicit Start, Restart, or Rebuild request changes that intent.
+Active creations, removals, betas, and campaigns defer new Watchdog actions.
+
+Provisioning checks current container health before using game status. A disruptive
+action requires a confirmed stopped container or a fresh zero-player reading from
+RCON or accepted telemetry. A positive player count from either source prevents
+an interruption. Telemetry evidence must have been received within 45 seconds;
+future, malformed, stale, and pre-restart reports cannot verify a new process.
+A game-end event retains its reported roster; it does not imply zero players.
+Fresh telemetry can also verify game activity and the selected engine map after
+Komodo reports the container healthy. A healthy container with neither current
+RCON status nor current telemetry remains queued for player-count verification.
+
+## Drift
+
+Drift includes missing promoted files, changed bytes, and extra files in managed
+script, config, and plugin destinations. Config comparison uses the composed
+baseline and overlays with the server's selected map. Plutainer's unmanaged default
+config remains exempt until a baseline is promoted. Active betas are deferred;
+missing server mounts are reported as scan errors instead of being treated as
+empty directories.
+
+Watchdog rebuilds a drifting server in place using the same identity, map, port,
+and saved server key. **Rebuild deletes all old instance files.** The new instance
+uses the latest promoted setup when provisioning starts. A missing saved key
+leaves the incident needing attention without deleting the old server.
+
+The linked Campaign captures before/after evidence at detection time. Its activity
+identifies this as the detected plan, and it follows the rebuild's outcome without
+queuing an additional restart. Promotions can change while the rebuild is queued;
+the Provisioning launch recipe records the files actually selected for that launch.
+Campaign evidence and Watchdog history survive subsequent rebuilds and removals.
+
+## Review and retry
+
+The list can be filtered by server, incident type, and status. Each detail page
+shows the triggering evidence, timestamped events, and links to every attempt.
+**Dismiss** suppresses further automatic attempts until the condition clears.
+**Retry** explicitly starts a new bounded attempt after inspecting a dismissed or
+failed incident. Active Provisioning operations must finish or be cancelled first.
+A verified recovery resolves the incident automatically.
+
+## API
+
+- `GET /api/watchdog`: automation setting, last scan, scan errors, and latest incidents.
+- `POST /api/watchdog` with `{ "action": "scan" }`: run a scan now.
+- `POST /api/watchdog` with `{ "enabled": false }` (or `true`): pause/resume automation.
+- `GET /api/watchdog/:id`: incident, event history, and linked actions.
+- `POST /api/watchdog/:id` with `{ "action": "retry" }` or `{ "action": "dismiss" }`.
+
+The legacy `/api/drift` GET and dry-run POST still return previews. Mutating POST
+requests return HTTP 410 and direct clients to Watchdog; file repair is now a
+provisioning rebuild instead of an immediate rewrite.