josh/electric-cherry
1
0
Fork 0
Code Issues 18 Pull Requests Actions Packages Projects Releases Wiki Activity

fix: honor Komodo health and prefer fresh telemetry across fleet status

Browse Source
This commit is contained in:
josh
2026-09-27 15:34:30 -04:00
parent 46a89ea4b0
commit 8ea696417f
27 changed files with 488 additions and 315 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+23 -13
View File
@@ -8,10 +8,20 @@ Existing database registrations are preserved, and missing mounts do not remove
servers from the fleet. The legacy `seedServerRegistry` helper is an explicit
import utility only.
Status refreshes check Komodo container readiness before polling RCON. Starting
or restarting containers show **Starting**, without old RCON errors or player
data. Stopped, unhealthy, or unverifiable containers are not polled. Containers
without a Docker health check use their running state as the readiness signal.
Status refreshes check Komodo container readiness, including nested Docker health
status. A running container with `health: starting` is still **Starting**. Starting
or restarting containers hide old game data and RCON errors; stopped, unhealthy,
or unverifiable containers do not trigger routine RCON polling. Containers without
a Docker health check use their running state as the readiness signal.
With fresh container health, validated telemetry received in the last 45 seconds
supplies the current player count, roster, map, and game status. RCON is the fallback
when telemetry is unavailable or stale. Fleet, server details, metrics, collection
coverage, setup readiness, and beta health checks share this policy. Reports from
before the latest start, restart, or rebuild cannot verify the new process.
Historical scoreboards remain visible and are labeled separately from live counts;
RCON diagnostics retain their own results. Telemetry does not report ping or player
capacity, so those fields remain unknown in telemetry-only observations.
## Fleet overview
@@ -33,7 +43,7 @@ status between refreshes. Failed refreshes keep the last snapshot visible.
- SQLite via better-sqlite3 (replaces `script_metadata.json` as the source of
truth)
- Server-sent events for live pipeline / server-status streams
- RCON (UDP) direct-to-game-server for player counts
- Game telemetry for current player counts and rosters, with direct UDP RCON as fallback
- Komodo API for restarts and per-server game-port discovery
- Runs as a single Komodo-managed container on the same host as the game
servers, with `/root/zombies` bind-mounted in
@@ -83,7 +93,7 @@ does not cancel work already queued; use **Cancel** for those jobs.
Maintenance waits for a fresh zero-player RCON or telemetry report and for existing
deployment/beta work, then uses the shared rolling restart limit. Unreachable or
occupied servers remain queued. After container and RCON recovery, Electric
occupied servers remain queued. After container and game-status recovery, Electric
Cherry queries the running revision again. If a fresh published revision was
known when queued, recovery on an older or unknown revision is reported as a
failed verification, without an automatic restart loop. Ambiguous restart
@@ -123,7 +133,7 @@ normally run in submission order; on a shared server, beta activation precedes
campaign activation, and maintenance waits for both. Busy
or unreachable servers stay queued without taking a slot, so other empty
servers can proceed. A successful Komodo restart request retains its slot
until the container is healthy and RCON answers. Health checks start no sooner
until the container is healthy and RCON or fresh telemetry verifies game status. Health checks start no sooner
than 15 seconds after the request returns, avoiding an immediate check of the
old process. An unhealthy server keeps its slot rather than letting failures
spread across the fleet. A timed-out restart also keeps its slot because Komodo
@@ -133,7 +143,7 @@ Slots persist in SQLite across app restarts, and overlapping timer/API triggers
share the active run. Beta options to skip health or match gates still work;
skipping a beta gate does not release the fleet's recovery slot early. New-server
creation uses a separate shared startup queue, holding its slot through container,
RCON, and selected-map verification.
game-status, and selected-map verification.
No token, no session, no auth freeze. If Komodo or RCON aren't configured,
the sweep marks pending rows `manual` so nothing hangs forever.
@@ -167,8 +177,8 @@ page tracks both actions. **Fill missing servers** remains the default.
All jobs and capacity reservations are committed together before provisioning;
one collection page tracks each server's independent progress and failures.
New servers start one at a time by default across all collections and individual
launches. A startup slot stays occupied until Komodo and RCON are healthy and
RCON reports the selected map; an unready server holds the queue until it recovers
launches. A startup slot stays occupied until Komodo is healthy and RCON or
fresh telemetry reports the selected map; an unready server holds the queue until it recovers
or is removed. The queue survives application restarts. The collection preview
shows the startup limit, and its progress page shows active startups and each
server's place in the shared queue. Set `DEPLOYMENT_CREATE_CONCURRENCY` to a
@@ -274,7 +284,7 @@ Rotations use `execgts <ruleset>.cfg map <engine-map>`; Plutainer seeds the requ
baseline. Town, Farm, and Bus Depot share `zm_transit`, so EC keeps their selection
in `map_slug` through creation, beta, promotion, and drift repair. Map-scoped
artifacts target the engine map and therefore apply to all its variants.
Creation health checks verify the engine map reported by RCON; this does not
Creation health checks verify the engine map reported by RCON or fresh telemetry; this does not
independently verify the start location within that map.
Both games share the fleet's port allocator (`SERVER_PORT_MIN`/`SERVER_PORT_MAX`)
@@ -365,9 +375,9 @@ RCON parsing (fake UDP socket).
Campaigns (`/campaigns`) tracks script/config updates, drift reconciliation, restart progress, file diffs, and rollbacks. Existing campaign records are preserved in the original database tables. The campaign API lives at `/api/campaigns`; legacy detail API endpoints remain available for compatibility.
Provisioning (`/provisioning`, `/api/provisioning`) owns Create, Start, Restart, Rebuild, Stop, and Destroy: request persistence, eligibility and occupancy checks, execution, recovery, and history. Campaigns, betas, and scheduled maintenance request lifecycle work through Provisioning and link to the resulting operation. They retain their file, test-evidence, and version-verification workflows. Pending automated restarts appear in Provisioning before execution. History survives server removal. New creation requests return HTTP 202 with an operation ID and open the provisioning detail page. Creation is complete only when Komodo reports healthy, RCON responds successfully, and the RCON map exactly matches the selected map. Queued requests reserve their server name and port; queued creation and pending health checks survive application restarts. The detail page records startup progress and explains any health or map mismatch that is still blocking completion.
Provisioning (`/provisioning`, `/api/provisioning`) owns Create, Start, Restart, Rebuild, Stop, and Destroy: request persistence, eligibility and occupancy checks, execution, recovery, and history. Campaigns, betas, and scheduled maintenance request lifecycle work through Provisioning and link to the resulting operation. They retain their file, test-evidence, and version-verification workflows. Pending automated restarts appear in Provisioning before execution. History survives server removal. New creation requests return HTTP 202 with an operation ID and open the provisioning detail page. Creation is complete only when Komodo reports healthy and RCON or fresh telemetry verifies game status with an engine map that exactly matches the selected map. Queued requests reserve their server name and port; queued creation and pending health checks survive application restarts. The detail page records startup progress and explains any health or map mismatch that is still blocking completion.
Each server deployment opens a detail page with its flow and timestamped activity. Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh RCON player count of zero before teardown; occupied or unreachable servers remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the provisioning detail page.
Each server deployment opens a detail page with its flow and timestamped activity. Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh zero-player RCON or telemetry report, or a verified stopped container, before teardown. Occupied reports from either source block teardown; servers without current verification remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the provisioning detail page.
Use **Provisioning → Manage server**, or a server’s **Server actions**, to start,