81 lines
4.6 KiB
Markdown
81 lines
4.6 KiB
Markdown
# Watchdog
|
|
|
|
**Operations → Watchdog** detects unhealthy servers and drift, records each
|
|
incident, and links its recovery attempts to Provisioning. File drift also opens
|
|
a Campaign with the observed file differences. Automation is enabled by default;
|
|
**Pause automation** persists across application restarts. Scanning and verification
|
|
continue while paused, but new actions and queued Watchdog actions are held.
|
|
Already submitted actions continue to completion.
|
|
|
|
The worker scans once per minute while Electric Cherry is running. **Scan now**
|
|
requests a scan immediately. Repeated button presses do not count as independent
|
|
health failures. A health incident must have three observations spanning at least
|
|
two minutes before an automatic restart is queued. Starting or restarting
|
|
containers get five minutes before recovery is requested. Unknown Komodo responses
|
|
are recorded without interpreting them as a stopped container.
|
|
|
|
## Recovery
|
|
|
|
A health incident gets one restart, then one rebuild if the restart has not
|
|
recovered within five minutes. Both are normal Provisioning jobs with durable
|
|
history and shared capacity limits. Escalation transfers only that incident's
|
|
restart recovery slot to the rebuild. An unsuccessful rebuild requires operator
|
|
attention; Watchdog does not start an endless restart/rebuild loop. Verification
|
|
of an already submitted rebuild continues, including after the incident is marked
|
|
as needing attention.
|
|
|
|
An intentional **Stop** is retained as the server's desired state. Watchdog leaves
|
|
it stopped until an explicit Start, Restart, or Rebuild request changes that intent.
|
|
Active creations, removals, betas, and campaigns defer new Watchdog actions.
|
|
|
|
Provisioning checks current container health before using game status. A disruptive
|
|
action requires a confirmed stopped container or a fresh zero-player reading from
|
|
RCON or accepted telemetry. A positive player count from either source prevents
|
|
an interruption. Telemetry evidence must have been received within 45 seconds;
|
|
future, malformed, stale, and pre-restart reports cannot verify a new process.
|
|
A game-end event retains its reported roster; it does not imply zero players.
|
|
Fresh telemetry can also verify game activity and the selected engine map after
|
|
Komodo reports the container healthy. A healthy container with neither current
|
|
RCON status nor current telemetry remains queued for player-count verification.
|
|
|
|
## Drift
|
|
|
|
Drift includes missing promoted files, changed bytes, and extra files in managed
|
|
script, config, and plugin destinations. Config comparison uses the composed
|
|
baseline and overlays with the server's selected map. Plutainer's unmanaged default
|
|
config remains exempt until a baseline is promoted. Active betas are deferred;
|
|
missing server mounts are reported as scan errors instead of being treated as
|
|
empty directories.
|
|
|
|
Watchdog rebuilds a drifting server in place using the same identity, map, port,
|
|
and saved server key. **Rebuild deletes all old instance files.** The new instance
|
|
uses the latest promoted setup when provisioning starts. A missing saved key
|
|
leaves the incident needing attention without deleting the old server.
|
|
|
|
The linked Campaign captures before/after evidence at detection time. Its activity
|
|
identifies this as the detected plan, and it follows the rebuild's outcome without
|
|
queuing an additional restart. Promotions can change while the rebuild is queued;
|
|
the Provisioning launch recipe records the files actually selected for that launch.
|
|
Campaign evidence and Watchdog history survive subsequent rebuilds and removals.
|
|
|
|
## Review and retry
|
|
|
|
The list can be filtered by server, incident type, and status. Each detail page
|
|
shows the triggering evidence, timestamped events, and links to every attempt.
|
|
**Dismiss** suppresses further automatic attempts until the condition clears.
|
|
**Retry** explicitly starts a new bounded attempt after inspecting a dismissed or
|
|
failed incident. Active Provisioning operations must finish or be cancelled first.
|
|
A verified recovery resolves the incident automatically.
|
|
|
|
## API
|
|
|
|
- `GET /api/watchdog`: automation setting, last scan, scan errors, and latest incidents.
|
|
- `POST /api/watchdog` with `{ "action": "scan" }`: run a scan now.
|
|
- `POST /api/watchdog` with `{ "enabled": false }` (or `true`): pause/resume automation.
|
|
- `GET /api/watchdog/:id`: incident, event history, and linked actions.
|
|
- `POST /api/watchdog/:id` with `{ "action": "retry" }` or `{ "action": "dismiss" }`.
|
|
|
|
The legacy `/api/drift` GET and dry-run POST still return previews. Mutating POST
|
|
requests return HTTP 410 and direct clients to Watchdog; file repair is now a
|
|
provisioning rebuild instead of an immediate rewrite.
|