Fleet membership lives in SQLite. EC registers servers during creation and removes them during destruction; startup does not discover or import server folders. Existing database registrations are preserved, and missing mounts do not remove servers from the fleet. The legacy seedServerRegistry helper is an explicit import utility only.

Status refreshes check Komodo container readiness, including nested Docker health status. A running container with health: starting is still Starting. Starting or restarting containers hide old game data and RCON errors; stopped, unhealthy, or unverifiable containers do not trigger routine RCON polling. Containers without a Docker health check use their running state as the readiness signal.

With fresh container health, validated telemetry received in the last 45 seconds supplies the current player count, roster, map, and game status. RCON is the fallback when telemetry is unavailable or stale. Fleet, server details, metrics, collection coverage, setup readiness, and beta health checks share this policy. Reports from before the latest start, restart, or rebuild cannot verify the new process. Historical scoreboards remain visible and are labeled separately from live counts; RCON diagnostics retain their own results. Telemetry does not report ping or player capacity, so those fields remain unknown in telemetry-only observations.

Active measures time since the lobby was first observed with players; Match measures time since the script's game_start was received. Telemetry records occupancy changes immediately, so a later heartbeat or status poll does not move the active timer forward. A new match keeps the active timer if the lobby stays occupied. An empty report clears the match timer and begins the inactive period; a later join begins a new active period. These are observed times, not exact player connection timestamps.

Fleet overview

Fleet searches server names, map names, and the full player roster. Filter occupied, empty, unhealthy, testing, or pinned lobbies; switch between cards and a compact table; and group by game or map. Pins and display preferences are saved in this browser. Map groups preserve T6 variants and can preselect the exact map when launching another lobby.

Health summaries exclude stale player counts. The match spotlight uses fresh round telemetry from an occupied, ongoing match. Persisted campaign, beta, creation, removal, and restart-recovery state appears alongside affected servers. Fleet membership and operation progress refresh every ten seconds, with live server status between refreshes. Failed refreshes keep the last snapshot visible.

Metrics explores recorded server use by time range, game, and map. It includes occupied/idle/unknown timelines, sortable rankings, previous-period comparisons, map capacity pressure, and an activity heatmap in the selected timezone. Hosting experiments suggest changes from explicit rules, including tentative advice from sparse observations; they never change the fleet automatically. Background refresh keeps the last report visible if it fails. See Metrics for the definitions, evidence limits, and experimental rules.

Stack

SvelteKit (adapter-node) + TypeScript
SQLite via better-sqlite3 (replaces script_metadata.json as the source of truth)
Server-sent events for live pipeline / server-status streams
Game telemetry for current player counts and rosters, with direct UDP RCON as fallback
Komodo API for restarts and per-server game-port discovery
Runs as a single Komodo-managed container on the same host as the game servers, with /root/zombies bind-mounted in

Layout

src/lib/server/db/ — schema, migrations, better-sqlite3 handle
src/lib/server/fleet/ — file walking, push/beta engines
src/lib/server/integrations/ — Komodo, RCON
src/lib/server/pipeline/ — sweep loop, event bus, worker singleton
src/lib/server/provisioning/ — lifecycle requests, queues, execution, recovery, and history
src/routes/api/ — REST + SSE endpoints
src/routes/ — UI pages (dashboard, campaigns, server deployments, betas, per-server)
Dockerfile, docker-compose.yml, entrypoint.sh — container build
komodo/ — Komodo stack definition
scripts/ — legacy JSON importer, fixture seeder
tests/ — vitest unit tests

Plugin releases

Plugins support named release versions, GitHub DLL imports and update discovery. See Plugin releases for the import and beta workflow.

Maintenance

Open Operations → Maintenance for health recovery, configuration drift, Plutonium updates, and scheduled restarts. Overview shows active work, attention, version coverage, and upcoming restarts. Policies & schedules controls each capability independently; Activity links findings and results to Provisioning. Fleet and server pages show runtime revision indicators and maintenance links.

Maintenance decides when work is needed and verifies the result. Provisioning executes every lifecycle operation using the shared safety checks and queues. Health recovery uses one restart followed by one rebuild if needed; drift repair requests a rebuild with a linked Campaign. Rebuild deletes all instance files. Maintenance skips observations and new automatic work on servers with active Provisioning operations or retained recovery capacity. Other servers continue normally, and existing maintenance still tracks its own operation's outcome.

Plutonium updates are applied by Plutainer's startup updater (PLUTAINER_AUTO_UPDATE must be enabled). Check versions and choose Update on restart, or opt into automatic updates for verified outdated servers. Revision checks run hourly, and Create and Rebuild verify and record the running revision before completing. Unknown or stale revisions remain unverified. A healthy restart can still fail update verification; Maintenance reports that outcome without rebuilding or repeatedly restarting the server. Electric Cherry does not download binaries or pin a runtime revision.

Optional daily restarts are independent of updates. Schedules use the saved timezone, follow daylight saving changes, and coalesce missed days into one run. Intentionally stopped servers are excluded. Existing schedules, pending jobs, operation IDs, and history survive the move from Settings and Watchdog. Servers owned by Provisioning are skipped for that scheduled occurrence.

Pause all maintenance holds new and queued maintenance while checks and submitted recovery continue. Each capability retains its own policy. Automatic updates remain disabled until explicitly enabled. See Maintenance for verification, retry, scheduling, and compatibility details.

Fleet settings

Settings → Fleet settings controls campaign history retention (default 7 days), concurrent restarts (default 1), concurrent server startups (default 1), and concurrent server destructions (default 1). Values are positive whole numbers; retention supports up to 36500 days. Settings are saved in SQLite for all users and apply at the next queue or cleanup check without restarting the app. Lowering a limit leaves active operations running and holds queued work until capacity is available. Collection previews and progress pages display the current startup limit.

Retention removes eligible old campaign records and file diffs. Pending, failed, held, and health-recovery-linked campaigns remain protected; Provisioning history is retained. Shortening retention can remove older history on the next sweep.

On the first startup after upgrading, existing DEPLOY_KEEP_DAYS, DEPLOYMENT_RESTART_CONCURRENCY, and DEPLOYMENT_CREATE_CONCURRENCY environment values initialize the saved settings. Keep custom values for that first startup; afterward, the database values take precedence and those variables can be removed. The supplied Compose and Komodo examples use the built-in defaults. Existing saved settings retain their values when upgraded; the destruction limit starts at 1 and can be changed in Settings.

GET /api/settings reads the four values (deployKeepDays, restartConcurrency, createConcurrency, destroyConcurrency); PUT /api/settings saves all four together as JSON. Invalid or incomplete requests return HTTP 400 without changing saved settings.

How restarts stay safe

Betas support configurable gameplay, participation, stability and manual-review requirements, saved presets, coverage rules and staged waves. See Beta test plans for usage and the four-event game telemetry contract.

File changes are staged sequentially. Their activation uses rolling restarts: by default only one server across the fleet can restart or recover at a time. Promotions, rollbacks, drift reconciliation, and beta restarts share this limit. Adjust Concurrent restarts in Settings → Fleet settings to raise it.

The Provisioning worker runs every 15s. Campaigns, betas, and maintenance submit durable restart requests and read their results. Provisioning checks recovering servers first, then admits queued operations when there is capacity. Requests normally run in submission order; on a shared server, beta activation precedes campaign activation, and maintenance waits for both. Busy or unreachable servers stay queued without taking a slot, so other empty servers can proceed. A successful Komodo restart request retains its slot until the container is healthy and RCON or fresh telemetry verifies game status. Health checks start no sooner than 15 seconds after the request returns, avoiding an immediate check of the old process. An unhealthy server keeps its slot rather than letting failures spread across the fleet. A timed-out restart also keeps its slot because Komodo may have accepted the request.

Slots persist in SQLite across app restarts, and overlapping timer/API triggers share the active run. Beta options to skip health or match gates still work; skipping a beta gate does not release the fleet's recovery slot early. New-server creation uses a separate shared startup queue, holding its slot through container, game-status, selected-map, and Plutonium revision verification.

No token, no session, no auth freeze. If Komodo or RCON aren't configured, the sweep marks pending rows manual so nothing hangs forever.

Collections, provisioning, and launch previews

Provisioning also offers House Party Protocol: review the entire fleet and queue destruction in one action. This includes queued creations and reuses pending destruction requests. Each server waits for a fresh zero-player reading; occupied or unreachable servers remain queued. Containers, server files, and registrations are removed, while saved keys and collections are retained. The preview must be reviewed again if fleet membership changes before confirmation. Servers created after confirmation are not included. Progress appears in deployment history. If any target belongs to an active beta, no removals are queued. Promote or roll back that beta before retrying fleet removal or an exact collection launch.

If deletion fails after Komodo removal, the operation detail page offers Resume cleanup to retry local file and database removal without RCON or repeating Komodo deletion. Requesting destruction again also resumes that cleanup. Recovery is blocked if a newer operation exists for the same server name.

Operations → Collections includes a builder for whole server lineups: choose multiple T5/T6 maps and set the desired total lobbies for each. Collections also has Active launches and Launch history tabs, with progress at /collections/launches/:id. Each server operation remains in Provisioning and links back to its collection launch. Existing collection launch bookmarks redirect to the new location. Conform fleet counts existing and queued servers and creates only the shortfall. The preview shows target/existing/missing counts and IDs, ports, and key requirements for new servers only. Already-satisfied targets create nothing; excess servers are kept. Choose Match collection exactly in the preview to also remove excess lobbies and maps outside the collection. Review the keep/create/remove lists and confirm removals. Matching servers stay untouched; unstarted excess creations are cancelled, and running servers wait for a fresh zero-player check before removal. New lobbies can reserve ports from these removals and wait until they are released. The collection page tracks both actions. Fill missing servers remains the default. All jobs and capacity reservations are committed together before provisioning; one collection page tracks each server's independent progress and failures. New servers start one at a time by default across all collections and individual launches. A startup slot stays occupied until Komodo is healthy, RCON or fresh telemetry reports the selected map, and RCON verifies the current Plutonium production revision or newer. Unknown release or runtime revisions hold the queue while verification retries; a confirmed older runtime fails provisioning. The queue survives application restarts. The collection preview shows the startup limit, and its progress page shows active startups and each server's place in the shared queue. Adjust Concurrent server startups in Settings → Fleet settings to increase the limit. Each worker pass starts at most the available number of slots; immediate failures do not drain the remaining queue. Existing single-server collections can be expanded with Edit. Single-server launches and launching again from recorded recipes remain available. Collections save maps and target lobby counts. Every new server uses the latest promoted scripts, plugins, and configuration when provisioning begins, including promotions made after preview or while queued. Existing collections automatically follow this policy. Deployment history records the files selected for that launch; Launch again uses the current promoted setup for the same game and map. The progress page explains waiting conditions, shows elapsed time, and can resume health verification of an interrupted creation when its original server remains. See Deployment collections for the workflow, promotion policy, and API details.

Saved server keys

Use Keys to add, replace, or remove a Plutonium key for an exact server ID such as t5_zombie_theater_2. IDs use a supported map slug and a lobby number from 1 to 20; the server does not need to be deployed first. During deployment, choose the map. The lowest available lobby is assigned automatically, with the exact server ID shown before deployment. Its saved key is reused automatically, and any newly entered key is saved for that ID, including if deployment fails. Destroying a server preserves its saved key. Updating or removing a saved key does not change an existing server; changes apply when that server ID is next created or rebuilt. Electric Cherry does not check whether Plutonium accepts a saved key. Last saved shows when the key was saved, not a validation result.

Keys are stored in the application's SQLite database at DB_PATH, independently of server records. Preserve that database across container replacements. Key values are not returned by the management API, but are stored unencrypted in SQLite, so database files and backups contain credentials. Existing deployment keys are not automatically imported from Komodo; add them once to retain them.

Local dev

Windows or macOS — no /root/zombies on disk, so seed a fixture tree first.

npm install
npm run seed:fixture

# Point env at the fixture (PowerShell)
$env:SHARED_SCRIPTS = "$PWD/test-fixtures/zombies/shared/scripts"
$env:SERVERS_ROOT   = "$PWD/test-fixtures/zombies/t5"
$env:DB_PATH        = "$PWD/data/cherry.db"

npm run dev
# open http://localhost:3000

Without RCON_PASSWORD / KOMODO_* set, the pipeline marks pending rows manual after a sweep — the UI still works, you just don't get real restarts. Enough to develop the push/beta flow.

Map selection and configs

The map selected when deploying a T5 zombies server controls its startup rotation automatically. No map-choice config overlay is required. New servers use the promoted game baseline and matching overlays when available; otherwise Plutainer seeds its usual defaults and Electric Cherry applies the selected map through a generated configs/electric_cherry_map.cfg, loaded after the main config and before map rotation. This file is tracked and restored by drift reconciliation; operators do not need to create or promote it. The launch order uses Plutainer's extra arguments and map-rotation controls. Plutainer's default main config is preserved until a game baseline is promoted.

Beta previews, promotion, and config drift repair all apply the server's stored catalog map after composing the baseline and overlays. An old sv_maprotation overlay cannot override that selection, but its other settings still apply. Existing servers receive this behavior on their next config deployment or drift reconciliation; this change does not immediately rewrite or restart them. Servers with an unknown map retain their existing config behavior. Rollback restores the exact prior config bytes.

T6 Zombies (Black Ops II)

Choose Black Ops II (T6) in Deploy new server. T6 uses t6_zombie_<slug>_<lobby> IDs, its own saved Plutonium keys, PLUTAINER_GAME=t6zm, and configs/dedicated_zm.cfg. Existing T5 deployments and environment settings keep their current paths. API clients can send game: "t6"; omitting it selects T5.

Before the first deployment, populate the Docker volume t6_game_files on the Komodo host with your Black Ops II game files, including the DLC maps you intend to host. Set PLUTAINER_T6_GAMEFILES_VOLUME to use another volume or an absolute host directory. It is mounted read-only at /home/plutainer/gamefiles. Plutainer supplies its runtime and config seeds, not the base game files. Use a Plutonium server key created for T6 Zombies.

Server state defaults to /root/zombies/t6, or the sibling t6 directory beside SERVERS_ROOT; override with T6_SERVERS_ROOT. Electric Cherry and the Komodo host must see this directory at the same absolute path. The supplied Compose mount already covers both games. GSC files go to runtime/plutonium/storage/t6/scripts/zm (no raw/), and game logs use logs/games_zm.log. Optional overrides are T6_REL_DEST_SCRIPT and T6_CONFIG_TARGET_FILE; a custom config filename needs a promoted baseline or a file supplied at that name, since Plutainer seeds dedicated_zm.cfg.

The catalog includes the classic maps, Survival, Grief, and Turned selections. Rotations use execgts <ruleset>.cfg map <engine-map>; Plutainer seeds the required storage/t6/gamesettings files. Keep seeding enabled, including with a promoted baseline. Town, Farm, and Bus Depot share zm_transit, so EC keeps their selection in map_slug through creation, beta, promotion, and drift repair. Map-scoped artifacts target the engine map and therefore apply to all its variants. Creation health checks verify the engine map reported by RCON or fresh telemetry; this does not independently verify the start location within that map.

Both games share the fleet's port allocator (SERVER_PORT_MIN/SERVER_PORT_MAX) to prevent collisions. EC explicitly sets PLUTAINER_PORT, so T6 need not use its upstream default of 4976. Forward the allocated UDP ports. Use a current Plutainer v2 image: EC relies on PLUTAINER_RCON_PASSWORD, PLUTAINER_MAP_ROTATE=false, and extra arguments loaded after the main config. Current Plutainer automatically whitelists the Docker gateway for T5/T6 RCON; an admin client on another host needs its address allowed in the server config.

Implementation references (checked September 2026):

Game telemetry

The server script sends JSON to POST /api/telemetry:

game_start: server, map, and all players (name and GUID).
round_change: server, map, round, and all players with kills, headshots, downs, revives, and score.
heartbeat: the same fields as round change, sent periodically during a round to refresh player stats.
game_end: the same fields as round change, with final stats.

These reports work with or without an active beta. Electric Cherry records the full player data, updates Fleet, and applies gameplay progress to active betas automatically. No discovery request or beta-specific fields are needed.

See Game reporter protocol for complete payload examples. Fresh telemetry can verify player count and game activity alongside RCON during provisioning and Watchdog recovery. Occupied reports prevent interruptions; stale reports cannot authorize them. Komodo still verifies container readiness.

For a missing scoreboard, open the server's Diagnostics → Telemetry API requests. The log shows received requests for that exact server name, including accepted reports, rejected payloads, HTTP status, errors, and expandable submitted JSON. It refreshes every five seconds while visible and survives app restarts. Logging starts with this version; older requests cannot be reconstructed. See Telemetry diagnostics for retention and how to interpret missing or rejected reports.

The Leaderboard records persistent player totals and map records from these same reports, including highest round, cumulative kills, rounds survived, headshots, revives, downs, and games played. See Leaderboard for counting rules, history import, and the read API. Only completed games count; in-progress and interrupted games remain in Match History without contributing to player rankings or profile totals.

In Match History, select View match on a completed match to open its shareable results page. It shows the map, server, final round, timing, and every recorded player's stats, including players absent from the final report. Completed matches also link from server history and player profiles. Results survive server deletion; missing starts and unrecorded stats remain explicitly unknown.

Running tests

npm test

Covers push (add/update/skip), beta lifecycle (open → rollback, open → promote, conflict blocks), sweep (integrations-off manual branch), and RCON parsing (fake UDP socket).

Deploy to the Komodo host

Build the image once locally or on the host:
```
docker compose build
```
Import komodo/stack.toml into Komodo (or point Komodo at the docker-compose.yml at repo root).

Set the env vars in Komodo:

KOMODO_ADDRESS, KOMODO_API_KEY, KOMODO_API_SECRET
RCON_PASSWORD              # single shared password across every T5/T6 server
RCON_HOST                  # optional; defaults to host.docker.internal

First boot triggers the legacy import: if /root/zombies/script_metadata.json exists and /data/cherry.db is empty, entrypoint.sh runs scripts/import-legacy.ts before starting the server. The source JSON gets renamed to .migrated.
Load the app on the LAN or over tailscale (no web-app auth by design).

Campaigns and provisioning

Campaigns (/campaigns) tracks script/config updates, drift reconciliation, restart progress, file diffs, and rollbacks. Existing campaign records are preserved in the original database tables. The campaign API lives at /api/campaigns; legacy detail API endpoints remain available for compatibility.

Beta deployments, promotions, drift repairs, and rollbacks stage their file changes in SQLite. Live server files and inventory remain unchanged while waiting for an empty or stopped server, other operations, and rolling restart capacity. Provisioning captures rollback snapshots and applies the staged changes immediately before requesting the restart. Failed file activation blocks that restart. Queued changes survive application restarts; cancelling an unapplied beta discards its pending changes. Manual beta rollback holds both file restoration and restart until Resume when empty is selected on its rollback campaign. Finish pending file campaigns before opening a beta on their servers; this keeps an older queued change from overwriting the version tested by the new beta.

Deleting a server preserves its name, recorded beta progress and test evidence, campaign results and file diffs, provisioning and maintenance history, and match records. Historical entries show Deleted and omit live server links and actions. Reusing the same server name starts a new instance; old records remain marked deleted, and old work cannot target the replacement. Servers linked to an active beta cannot be destroyed or rebuilt until that beta is promoted or rolled back. A participant removed outside provisioning still pauses its open beta; roll it back and open a new beta to test the replacement. Normal history retention still applies. Records erased by earlier versions cannot be restored by this upgrade. Existing history is marked deleted where the server is already absent; instances that reused a name before upgrading cannot always be distinguished.

Provisioning (/provisioning, /api/provisioning) owns Create, Start, Restart, Rebuild, Stop, and Destroy: request persistence, eligibility and occupancy checks, execution, recovery, and history. Campaigns, betas, and scheduled maintenance request lifecycle work through Provisioning and link to the resulting operation. They retain their file, test-evidence, and version-verification workflows. Pending automated restarts appear in Provisioning before execution. History survives server removal. New creation requests return HTTP 202 with an operation ID and open the provisioning detail page. Create and Rebuild complete only when Komodo reports healthy, RCON or fresh telemetry verifies the selected engine map and game status, and RCON reports a Plutonium runtime revision at least as new as the current production release. Provisioning records the release and runtime evidence so new servers have verified version status immediately. Unknown release or runtime revisions are retried; a confirmed older runtime fails with a clear error and no automatic restart. Queued requests reserve their server name and port; queued creation and pending verification survive application restarts. The detail page records startup progress and explains any health, map, or revision issue blocking completion.

Each server deployment opens a detail page with its outcome, blocker, permitted recovery actions, grouped execution phases, and timestamped verification evidence. New jobs record command certainty and the source of game-health checks; older records retain their known outcomes without invented evidence. A failed restart's ongoing recovery and held capacity appear separately from its recorded failure. Recent activity is visible immediately, with searchable history loaded in pages. Launch files distinguish the setup captured while queued from the files selected when creation began. The page retains its last snapshot if a refresh fails.

Destroy uses an in-app confirmation modal and immediately returns a queued deployment (DELETE /api/servers/:name returns HTTP 202 with deploymentId). The worker waits for a fresh zero-player RCON or telemetry report, or a verified stopped container, before teardown. Occupied reports from either source block teardown; servers without current verification remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the provisioning detail page.

Settings → Concurrent server destructions limits simultaneous individual and fleet removals (default 1). Each destruction holds its slot through teardown and local cleanup. Lowering the limit lets active removals finish before more begin.

Use Provisioning → Manage server, or a server’s Server actions, to start, restart, stop, or rebuild an existing instance. Rebuild destroys and creates in place: the old container, Komodo deployment, server registration, and all server files are deleted. A fresh instance uses the same server ID, map, and port with the latest promoted setup. No old instance files are carried forward. The key stored separately in Keys must be available before teardown. Stop retains the instance so Start can bring it back. Disruptive operations wait for a fresh empty reading or a confirmed stopped container, and completion requires verification. See Provisioning for lifecycle behavior and recovery.

Existing /deployments page links redirect to /provisioning, preserving filters and operation IDs. Legacy /api/deployments endpoints and stored history remain compatible.

README.md Unescape Escape

Electric Cherry