josh/electric-cherry
1
0
Fork 0
Code Issues 18 Pull Requests Actions Packages Projects Releases Wiki Activity

Add native host management and role-based operator authentication

Browse Source 
Introduce an outbound Python host agent, Debian installer, durable remote
operations, immutable game-file libraries, host-aware deployments and ports,
and occupancy-safe campaigns and draining. Native hosts do not require Komodo.

Add administrator, operator and viewer accounts, protected management routes,
secure sessions, password recovery, audit history and separate telemetry tokens.
Document first-admin setup, reporter migration and live-host acceptance checks.

Validation: 1,343 TypeScript tests and 39 agent tests passed, along with type
checking, the production build, compiled SSR/API smoke tests and administrator
recovery. Real Debian/Docker T5/T6 acceptance remains unrun.
This commit is contained in:
josh
2026-10-03 11:42:36 -04:00
parent a2e2311aa9
commit e5c0fda1ed
127 changed files with 8485 additions and 657 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+105 -465
View File
@@ -1,497 +1,137 @@
# Electric Cherry
Web app that manages a fleet of T5 and T6 Plutonium Zombies servers running under Komodo.
Electric Cherry manages T5 and T6 Plutonium Zombies servers through its own host
agents. Docker and Plutainer run the games; EC handles hosts, game libraries,
collections, configuration, campaigns, maintenance and operation history.
Komodo is not required for native hosts.
Fleet membership lives in SQLite. EC registers servers during creation and removes
them during destruction; startup does not discover or import server folders.
Existing database registrations are preserved, and missing mounts do not remove
servers from the fleet. The legacy `seedServerRegistry` helper is an explicit
import utility only.
## Getting started
Status refreshes check Komodo container readiness, including nested Docker health
status. A running container with `health: starting` is still **Starting**. Starting
or restarting containers hide old game data and RCON errors; stopped, unhealthy,
or unverifiable containers do not trigger routine RCON polling. Containers without
a Docker health check use their running state as the readiness signal.
Run the controller with Docker Compose behind HTTPS:
Status polling runs every 30 seconds with at most four server checks in flight.
Each completed check publishes a current fleet snapshot; a slow check does not
hold back updates or the next poll for servers that have finished. RCON status
queries retry timeouts and send errors twice, with 250ms then 500ms backoff.
Overlapping status and occupancy probes share the in-flight query for the same
server process and endpoint. Configuration and invalid-reply errors are reported
immediately; commands that change server state are never retried by this logic.
```sh
cp .env.example .env
# Set EC_PUBLIC_URL, generate EC_SETUP_TOKEN, and retain EC_DATA_DIR.
docker compose up -d --build
```
With fresh container health, validated telemetry received in the last 45 seconds
supplies the current player count, roster, map, and game status. RCON is the fallback
when telemetry is unavailable or stale. Fleet, server details, metrics, collection
coverage, setup readiness, and beta health checks share this policy. Reports from
before the latest start, restart, or rebuild cannot verify the new process.
Historical scoreboards remain visible and are labeled separately from live counts;
RCON diagnostics retain their own results. Telemetry does not report ping or player
capacity, so those fields remain unknown in telemetry-only observations.
Open EC and create the first administrator using `EC_SETUP_TOKEN` (at least 32
characters; generate it with `openssl rand -hex 32`). After setup, remove that
token from the environment. Administrators can add administrators, operators and
viewers from **Settings → Users**. See [Users and access](docs/authentication.md)
for permissions, sessions, password recovery and upgrade instructions.
**Active** measures time since the lobby was first observed with players;
**Match** measures time since the script's `game_start` was received. Telemetry
records occupancy changes immediately, so a later heartbeat or status poll does
not move the active timer forward. A new match keeps the active timer if the
lobby stays occupied. An empty report clears the match timer and begins the
inactive period; a later join begins a new active period. These are observed
times, not exact player connection timestamps.
Operator pages and management APIs require login. Enrolled agents authenticate
independently and connect outbound over HTTPS; the
controller needs no Docker socket, game-server mounts or inbound RCON access.
The default Compose data mount preserves the previous
`/root/zombies/.electric-cherry` database location. For a new installation, set
`EC_DATA_DIR` to your preferred persistent directory. Back up that directory,
including SQLite metadata and published game-file storage.
Server chat shows real time in your selected timezone, elapsed match time using
the same observed start as Fleet, and the round last reported when each message
was sent. Real time is approximate: the latest log write anchors the engine clock.
Earlier clock segments, missing match starts, and unavailable or stale round
reports show `—` instead of borrowing context from another match. Round history
uses retained accepted telemetry (up to 200 requests per server or seven days).
1. **Hosts → Add Host**: configure a Debian 12/13 amd64 machine and run its
generated enrollment command.
2. Prepare a Plutainer runtime from the host page.
3. **Library → Game Files**: import existing T5/T6 files once, then cache a
published version on the destination host.
4. Add the exact Plutonium server keys in **Keys**, or supply missing keys when
launching.
5. Launch a server or collection with an explicit host, game-file version and
runtime image. EC reserves host-specific ports and verifies startup.
## Fleet overview
See [Native host management](docs/native-hosts.md) for installation, credentials,
storage, failure recovery, API contracts and the live-host acceptance checklist.
Fleet searches server names, map names, and the full player roster. Filter occupied,
empty, unhealthy, testing, or pinned lobbies; switch between cards and a compact
table; and group by game or map. Pins and display preferences are saved in this
browser. Map groups preserve T6 variants and can preselect the exact map when
launching another lobby.
## Fleet and operations
Health summaries exclude stale player counts. The match spotlight uses fresh round
telemetry from an occupied, ongoing match. Persisted campaign, beta, creation,
removal, and restart-recovery state appears alongside affected servers. Fleet
membership and operation progress refresh every ten seconds, with live server
status between refreshes. Failed refreshes keep the last snapshot visible.
The fleet registry lives in SQLite, independently of host filesystems. Status
combines agent container health, fresh game telemetry and agent-local RCON.
Unreachable hosts report unknown instance state; they do not trigger duplicate
replacements. Running containers continue through controller or agent outages.
**Metrics** explores recorded server use by time range, game, and map. It includes
occupied/idle/unknown timelines, sortable rankings, previous-period comparisons,
map capacity pressure, and an activity heatmap in the selected timezone. Hosting
experiments suggest changes from explicit rules, including tentative advice from
sparse observations; they never change the fleet automatically. Background refresh
keeps the last report visible if it fails. See [Metrics](docs/metrics.md) for the
definitions, evidence limits, and experimental rules.
Provisioning owns Create, Start, Stop, Restart, Rebuild and Destroy. Commands and
results have durable operation IDs and immutable instance generations. Queues,
capacity reservations and history survive controller restarts. **Rebuild removes
per-instance writable files** and creates a new generation. Shared immutable game
libraries survive rebuilds and destruction.
## Stack
Disruptive actions wait for a fresh empty lobby or verified stopped container.
The agent rechecks occupancy locally immediately before activation and disruption.
Long player sessions can continue indefinitely. **Drain** prevents new launches
and safely retires existing servers as they become eligible.
- SvelteKit (adapter-node) + TypeScript
- SQLite via better-sqlite3 (replaces `script_metadata.json` as the source of
truth)
- Server-sent events for live pipeline / server-status streams
- Game telemetry for current player counts and rosters, with direct UDP RCON as fallback
- Komodo API for restarts and per-server game-port discovery
- Runs as a single Komodo-managed container on the same host as the game
servers, with `/root/zombies` bind-mounted in
Collections save target maps and lobby counts. Fill missing servers adds only the
shortfall; Match collection exactly also queues safe removal of excess servers.
Launch previews select a host, pinned base-game versions and an independent
runtime image. Every new launch uses the latest promoted configuration, scripts
and plugins when execution starts. Saved Plutonium keys are reused by exact
server ID and retained after removal.
## Layout
## Libraries and campaigns
- `src/lib/server/db/` — schema, migrations, better-sqlite3 handle
- `src/lib/server/fleet/` — file walking, push/beta engines
- `src/lib/server/integrations/` — Komodo, RCON
- `src/lib/server/pipeline/` — sweep loop, event bus, worker singleton
- `src/lib/server/provisioning/` — lifecycle requests, queues, execution, recovery, and history
- `src/routes/api/` — REST + SSE endpoints
- `src/routes/` — UI pages (dashboard, campaigns, server deployments, betas, per-server)
- `Dockerfile`, `docker-compose.yml`, `entrypoint.sh` — container build
- `komodo/` — Komodo stack definition
- `scripts/` — legacy JSON importer, fixture seeder
- `tests/` — vitest unit tests
Game-file versions include game, version, DLC metadata, manifests, checksums and
size. Actual game files use dedicated filesystem storage, never SQLite blobs.
Downloads resume, report progress and verify checksums before becoming usable.
Multiple instances on one host share read-only base files.
## Plugin releases
Scripts, plugins, configuration baselines and overlays retain their versioning,
beta, promotion and rollback workflows. Campaigns stage bytes remotely without
stopping games; activation uses the same occupancy and concurrency gates as
lifecycle operations. Agent before images preserve exact rollback bytes and
operation history.
Plugins support named release versions, GitHub DLL imports and update discovery.
See [Plugin releases](docs/plugin-releases.md) for the import and beta workflow.
T5 Zombies uses `t5sp` and `dedicated_sp.cfg`; T6 uses `t6zm` and
`dedicated_zm.cfg`. Map selection generates `electric_cherry_map.cfg`, loaded
after the main config. T6 variants retain their distinct catalog selections.
Game-file libraries and Plutonium runtime revisions are tracked independently.
## Maintenance
Game telemetry uses its own `EC_TELEMETRY_TOKEN`; configure reporters to send it
in an authorization header. Anonymous reports are rejected before they can
affect occupancy or game history. Existing JSON payloads are unchanged.
Open **Operations → Maintenance** for health recovery, configuration drift,
Plutonium updates, and scheduled restarts. Overview shows active work, attention,
version coverage, and upcoming restarts. Policies & schedules controls each
capability independently; Activity links findings and results to Provisioning.
Fleet and server pages show runtime revision indicators and maintenance links.
## Documentation
Maintenance decides when work is needed and verifies the result. Provisioning
executes every lifecycle operation using the shared safety checks and queues.
Health recovery uses one restart followed by one rebuild if needed; drift repair
requests a rebuild with a linked Campaign. **Rebuild deletes all instance files.**
Maintenance skips observations and new automatic work on servers with active
Provisioning operations or retained recovery capacity. Other servers continue
normally, and existing maintenance still tracks its own operation's outcome.
- [Native hosts, storage and operation recovery](docs/native-hosts.md)
- [Users, roles and authentication](docs/authentication.md)
- [Provisioning and safety queues](docs/provisioning.md)
- [Collections and promoted launch recipes](docs/deployment-blueprints.md)
- [Maintenance](docs/maintenance.md)
- [Beta test plans and game telemetry](docs/beta-test-plans.md)
- [Plugin releases](docs/plugin-releases.md)
- [Metrics](docs/metrics.md)
- [Leaderboard and match history](docs/leaderboard.md)
Plutonium updates are applied by Plutainer's startup updater
(`PLUTAINER_AUTO_UPDATE` must be enabled). Check versions and choose **Update on
restart**, or opt into automatic updates for verified outdated servers. Revision
checks run hourly, and Create and Rebuild verify and record the running revision
before completing. Unknown or stale revisions remain unverified. A healthy restart
can still fail update verification; Maintenance reports that outcome without
rebuilding or repeatedly restarting the server. Electric Cherry does not download
binaries or pin a runtime revision.
Older workflow documents may describe the legacy Komodo backend. Native hosts
use the agent boundary described in the native-host document.
Optional daily restarts are independent of updates. Schedules use the saved
timezone, follow daylight saving changes, and coalesce missed days into one run.
Intentionally stopped servers are excluded. Existing schedules, pending jobs,
operation IDs, and history survive the move from Settings and Watchdog.
Servers owned by Provisioning are skipped for that scheduled occurrence.
## Development
**Pause all maintenance** holds new and queued maintenance while checks and
submitted recovery continue. Each capability retains its own policy. Automatic
updates remain disabled until explicitly enabled. See [Maintenance](docs/maintenance.md)
for verification, retry, scheduling, and compatibility details.
The controller is SvelteKit/TypeScript with SQLite via better-sqlite3, the Node
adapter and server-sent events. The host agent is Python 3 standard-library code
installed as a systemd service. Hosts do not install Node.js or build EC.
## Fleet settings
**Settings → Fleet settings** controls campaign history retention (default 7 days),
concurrent restarts (default 1), concurrent server startups (default 1), and
concurrent server destructions (default 1).
Values are positive whole numbers; retention supports up to 36500 days. Settings
are saved in SQLite for all users and apply at the next queue or cleanup check
without restarting the app. Lowering a limit leaves active operations running
and holds queued work until capacity is available. Collection previews and
progress pages display the current startup limit.
Retention removes eligible old campaign records and file diffs. Pending, failed,
held, and health-recovery-linked campaigns remain protected; Provisioning history
is retained. Shortening retention can remove older history on the next sweep.
On the first startup after upgrading, existing `DEPLOY_KEEP_DAYS`,
`DEPLOYMENT_RESTART_CONCURRENCY`, and `DEPLOYMENT_CREATE_CONCURRENCY` environment
values initialize the saved settings. Keep custom values for that first startup;
afterward, the database values take precedence and those variables can be removed.
The supplied Compose and Komodo examples use the built-in defaults.
Existing saved settings retain their values when upgraded; the destruction
limit starts at 1 and can be changed in Settings.
`GET /api/settings` reads the four values (`deployKeepDays`, `restartConcurrency`,
`createConcurrency`, `destroyConcurrency`); `PUT /api/settings` saves all four together as JSON.
Invalid or incomplete requests return HTTP 400 without changing saved settings.
## How restarts stay safe
Betas support configurable gameplay, participation, stability and manual-review
requirements, saved presets, coverage rules and staged waves. See
[Beta test plans](docs/beta-test-plans.md) for usage and the four-event game telemetry contract.
File changes are staged sequentially. Their activation uses rolling restarts:
by default only **one server across the fleet** can restart or recover at a
time. Promotions, rollbacks, drift reconciliation, and beta restarts share this
limit. Adjust **Concurrent restarts** in **Settings → Fleet settings** to raise it.
The Provisioning worker runs every 15s. Campaigns, betas, and maintenance submit
durable restart requests and read their results. Provisioning checks recovering
servers first, then admits queued operations when there is capacity. Requests
normally run in submission order; on a shared server, beta activation precedes
campaign activation, and maintenance waits for both. Busy
or unreachable servers stay queued without taking a slot, so other empty
servers can proceed. A successful Komodo restart request retains its slot
until the container is healthy and RCON or fresh telemetry verifies game status. Health checks start no sooner
than 15 seconds after the request returns, avoiding an immediate check of the
old process. An unhealthy server keeps its slot rather than letting failures
spread across the fleet. A timed-out restart also keeps its slot because Komodo
may have accepted the request.
Slots persist in SQLite across app restarts, and overlapping timer/API triggers
share the active run. Beta options to skip health or match gates still work;
skipping a beta gate does not release the fleet's recovery slot early. New-server
creation uses a separate shared startup queue, holding its slot through container,
game-status, selected-map, and Plutonium revision verification.
No token, no session, no auth freeze. If Komodo or RCON aren't configured,
the sweep marks pending rows `manual` so nothing hangs forever.
## Collections, provisioning, and launch previews
Provisioning also offers **House Party Protocol**: review the entire fleet and queue
destruction in one action. This includes queued creations and reuses pending
destruction requests. Each server waits for a fresh zero-player reading; occupied
or unreachable servers remain queued. Containers, server files, and registrations
are removed, while saved keys and collections are retained. The preview must be
reviewed again if fleet membership changes before confirmation. Servers created
after confirmation are not included. Progress appears in deployment history.
If any target belongs to an active beta, no removals are queued. Promote or roll
back that beta before retrying fleet removal or an exact collection launch.
If deletion fails after Komodo removal, the operation detail page offers
**Resume cleanup** to retry local file and database removal without RCON or
repeating Komodo deletion. Requesting destruction again also resumes that cleanup.
Recovery is blocked if a newer operation exists for the same server name.
**Operations → Collections** includes a builder for whole server lineups: choose
multiple T5/T6 maps and set the desired total lobbies for each. Collections also
has **Active launches** and **Launch history** tabs, with progress at
`/collections/launches/:id`.
Each server operation remains in Provisioning and links back to its collection
launch. Existing collection launch bookmarks redirect to the new location.
**Conform fleet**
counts existing and queued servers and creates only the shortfall. The preview
shows target/existing/missing counts and IDs, ports, and key requirements for new
servers only. Already-satisfied targets create nothing; excess servers are kept.
Choose **Match collection exactly** in the preview to also remove excess lobbies
and maps outside the collection. Review the keep/create/remove lists and confirm
removals. Matching servers stay untouched; unstarted excess creations are cancelled,
and running servers wait for a fresh zero-player check before removal. New lobbies
can reserve ports from these removals and wait until they are released. The collection
page tracks both actions. **Fill missing servers** remains the default.
All jobs and capacity reservations are committed together before provisioning;
one collection page tracks each server's independent progress and failures.
New servers start one at a time by default across all collections and individual
launches. A startup slot stays occupied until Komodo is healthy, RCON or fresh
telemetry reports the selected map, and RCON verifies the current Plutonium
production revision or newer. Unknown release or runtime revisions hold the queue
while verification retries; a confirmed older runtime fails provisioning. The
queue survives application restarts. The collection preview
shows the startup limit, and its progress page shows active startups and each
server's place in the shared queue. Adjust **Concurrent server startups** in
**Settings → Fleet settings** to increase the limit. Each worker pass starts at most the
available number of slots; immediate failures do not drain the remaining queue.
Existing single-server collections can be expanded with **Edit**. Single-server
launches and launching again from recorded recipes remain available.
Collections save maps and target lobby counts. Every new server uses the latest
promoted scripts, plugins, and configuration when provisioning begins, including
promotions made after preview or while queued. Existing collections automatically
follow this policy. Deployment history records the files selected for that launch;
**Launch again** uses the current promoted setup for the same game and map.
The progress page explains waiting conditions, shows elapsed time, and can resume
health verification of an interrupted creation when its original server remains.
See [Deployment collections](docs/deployment-blueprints.md) for the workflow,
promotion policy, and API details.
## Saved server keys
Use **Keys** to add, replace, or remove a Plutonium key for an exact
server ID such as `t5_zombie_theater_2`. IDs use a supported map slug and a
lobby number from 1 to 20; the server does not need to be deployed first.
During deployment, choose the map. The lowest available lobby is assigned
automatically, with the exact server ID shown before deployment. Its saved key is reused automatically,
and any newly entered key is saved for that ID, including if deployment fails.
Destroying a server preserves its saved key. Updating or removing a saved key
does not change an existing server; changes apply when that server ID is next
created or rebuilt. Electric Cherry does not check whether Plutonium accepts a
saved key. **Last saved** shows when the key was saved, not a validation result.
Keys are stored in the application's SQLite database at `DB_PATH`, independently
of server records. Preserve that database across container replacements. Key
values are not returned by the management API, but are stored unencrypted in
SQLite, so database files and backups contain credentials. Existing deployment
keys are not automatically imported from Komodo; add them once to retain them.
## Local dev
Windows or macOS — no `/root/zombies` on disk, so seed a fixture tree first.
```bash
```sh
npm install
npm run seed:fixture
npm run agent:package
# Export EC_SETUP_TOKEN in your shell using the secret saved for /setup.
EC_AUTH_ALLOW_HTTP=true npm run dev
# Point env at the fixture (PowerShell)
$env:SHARED_SCRIPTS = "$PWD/test-fixtures/zombies/shared/scripts"
$env:SERVERS_ROOT = "$PWD/test-fixtures/zombies/t5"
$env:DB_PATH = "$PWD/data/cherry.db"
npm run dev
# open http://localhost:3000
```
Without RCON_PASSWORD / KOMODO_* set, the pipeline marks pending rows
`manual` after a sweep — the UI still works, you just don't get real
restarts. Enough to develop the push/beta flow.
## Map selection and configs
The map selected when deploying a T5 zombies server controls its startup
rotation automatically. No map-choice config overlay is required. New servers
use the promoted game baseline and matching overlays when available; otherwise
Plutainer seeds its usual defaults and Electric Cherry applies the selected map
through a generated `configs/electric_cherry_map.cfg`, loaded after the main
config and before map rotation. This file is tracked and restored by drift
reconciliation; operators do not need to create or promote it. The launch order
uses Plutainer's [extra arguments and map-rotation controls](https://github.com/Ayymoss/Plutainer/blob/main/docs/configuration.md).
Plutainer's
default main config is preserved until a game baseline is promoted.
Beta previews, promotion, and config drift repair all apply the server's stored
catalog map after composing the baseline and overlays. An old `sv_maprotation`
overlay cannot override that selection, but its other settings still apply.
Existing servers receive this behavior on their next config deployment or drift
reconciliation; this change does not immediately rewrite or restart them.
Servers with an unknown map retain their existing config behavior. Rollback
restores the exact prior config bytes.
## T6 Zombies (Black Ops II)
Choose **Black Ops II (T6)** in Deploy new server. T6 uses `t6_zombie_<slug>_<lobby>`
IDs, its own saved Plutonium keys, `PLUTAINER_GAME=t6zm`, and
`configs/dedicated_zm.cfg`. Existing T5 deployments and environment settings
keep their current paths. API clients can send `game: "t6"`; omitting it selects T5.
Before the first deployment, populate the Docker volume `t6_game_files` on the
Komodo host with your Black Ops II game files, including the DLC maps you intend
to host. Set `PLUTAINER_T6_GAMEFILES_VOLUME` to use another volume or an absolute
host directory. It is mounted read-only at `/home/plutainer/gamefiles`.
Plutainer supplies its runtime and config seeds, not the base game files.
Use a Plutonium server key created for **T6 Zombies**.
Server state defaults to `/root/zombies/t6`, or the sibling `t6` directory beside
`SERVERS_ROOT`; override with `T6_SERVERS_ROOT`. Electric Cherry and the Komodo
host must see this directory at the same absolute path. The supplied Compose
mount already covers both games. GSC files go to
`runtime/plutonium/storage/t6/scripts/zm` (no `raw/`), and game logs use
`logs/games_zm.log`. Optional overrides are `T6_REL_DEST_SCRIPT` and
`T6_CONFIG_TARGET_FILE`; a custom config filename needs a promoted baseline or
a file supplied at that name, since Plutainer seeds `dedicated_zm.cfg`.
The catalog includes the classic maps, Survival, Grief, and Turned selections.
Rotations use `execgts <ruleset>.cfg map <engine-map>`; Plutainer seeds the required
`storage/t6/gamesettings` files. Keep seeding enabled, including with a promoted
baseline. Town, Farm, and Bus Depot share `zm_transit`, so EC keeps their selection
in `map_slug` through creation, beta, promotion, and drift repair. Map-scoped
artifacts target the engine map and therefore apply to all its variants.
Creation health checks verify the engine map reported by RCON or fresh telemetry; this does not
independently verify the start location within that map.
Both games share the fleet's port allocator (`SERVER_PORT_MIN`/`SERVER_PORT_MAX`)
to prevent collisions. EC explicitly sets `PLUTAINER_PORT`, so T6 need not use
its upstream default of 4976. Forward the allocated UDP ports. Use a current
Plutainer v2 image: EC relies on `PLUTAINER_RCON_PASSWORD`,
`PLUTAINER_MAP_ROTATE=false`, and extra arguments loaded after the main config.
Current Plutainer automatically whitelists the Docker gateway for T5/T6 RCON;
an admin client on another host needs its address allowed in the server config.
Implementation references (checked September 2026):
- [Plutainer game requirements](https://github.com/Ayymoss/Plutainer/blob/main/docs/games.md)
- [Plutainer launch, seeding, and RCON implementation](https://github.com/Ayymoss/Plutainer/blob/main/scripts/lib/cod.sh)
- [Plutainer configuration](https://github.com/Ayymoss/Plutainer/blob/main/docs/configuration.md)
- [T6 map/ruleset combinations](https://github.com/xerxes-at/T6ServerConfigs/blob/master/localappdata/Plutonium/storage/t6/dedicated_zm.cfg)
- [Plutonium GSC paths](https://plutonium.pw/docs/modding/loading-mods/)
## Game telemetry
The server script sends JSON to `POST /api/telemetry`:
- `game_start`: server, map, and all players (name and GUID).
- `round_change`: server, map, round, and all players with kills, headshots, downs, revives, and score.
- `heartbeat`: the same fields as round change, sent periodically during a round to refresh player stats.
- `game_end`: the same fields as round change, with final stats.
These reports work with or without an active beta. Electric Cherry records
the full player data, updates Fleet, and applies gameplay progress to active
betas automatically. No discovery request or beta-specific fields are needed.
See [Game reporter protocol](docs/beta-test-plans.md#game-reporter-protocol) for
complete payload examples. Fresh telemetry can verify player count and game
activity alongside RCON during provisioning and Watchdog recovery. Occupied
reports prevent interruptions; stale reports cannot authorize them. Komodo still
verifies container readiness.
For a missing scoreboard, open the server's **Diagnostics → Telemetry API requests**.
The log shows received requests for that exact server name, including accepted
reports, rejected payloads, HTTP status, errors, and expandable submitted JSON.
It refreshes every five seconds while visible and survives app restarts. Logging
starts with this version; older requests cannot be reconstructed. See
[Telemetry diagnostics](docs/beta-test-plans.md#telemetry-diagnostics) for retention
and how to interpret missing or rejected reports.
The **Leaderboard** records persistent player totals and map records from these
same reports, including highest round, cumulative kills, rounds survived,
headshots, revives, downs, and games played. See [Leaderboard](docs/leaderboard.md)
for counting rules, history import, and the read API. Only completed games count;
in-progress and interrupted games remain in Match History without contributing
to player rankings or profile totals.
In **Match History**, select **View match** on a completed match to open its
shareable results page. It shows the map, server, final round, timing, and every
recorded player's stats, including players absent from the final report. Completed
matches also link from server history and player profiles. Results survive server
deletion; missing starts and unrecorded stats remain explicitly unknown.
## Running tests
```bash
npm run check
npm test
npm run test:agent
npm run build
```
Covers push (add/update/skip), beta lifecycle (open → rollback, open →
promote, conflict blocks), sweep (integrations-off manual branch), and
RCON parsing (fake UDP socket).
Builds regenerate the downloadable agent and checksum manifest. Local controller
metadata defaults to `./data/cherry.db`; use a separate test database when
exercising enrollment. Production agent transport requires HTTPS.
## Deploy to the Komodo host
`src/lib/server/hosts` owns enrollment, readiness and command transport;
`src/lib/server/backends` provides instance identity and remote file access;
`src/lib/server/provisioning/native.ts` connects existing lifecycle and campaign
queues to agents; `src/lib/server/game-files` owns immutable libraries and
streaming distribution; `agent/` contains the service, installer and agent tests.
1. Build the image once locally or on the host:
```bash
docker compose build
```
2. Import `komodo/stack.toml` into Komodo (or point Komodo at the
`docker-compose.yml` at repo root).
3. Set the env vars in Komodo:
```
KOMODO_ADDRESS, KOMODO_API_KEY, KOMODO_API_SECRET
RCON_PASSWORD # single shared password across every T5/T6 server
RCON_HOST # optional; defaults to host.docker.internal
```
4. First boot triggers the legacy import: if
`/root/zombies/script_metadata.json` exists and `/data/cherry.db` is
empty, `entrypoint.sh` runs `scripts/import-legacy.ts` before starting
the server. The source JSON gets renamed to `.migrated`.
5. Load the app on the LAN or over tailscale (no web-app auth by design).
## Campaigns and provisioning
Campaigns (`/campaigns`) tracks script/config updates, drift reconciliation, restart progress, file diffs, and rollbacks. Existing campaign records are preserved in the original database tables. The campaign API lives at `/api/campaigns`; legacy detail API endpoints remain available for compatibility.
Beta deployments, promotions, drift repairs, and rollbacks stage their file changes
in SQLite. Live server files and inventory remain unchanged while waiting for an
empty or stopped server, other operations, and rolling restart capacity. Provisioning
captures rollback snapshots and applies the staged changes immediately before
requesting the restart. Failed file activation blocks that restart. Queued changes
survive application restarts; cancelling an unapplied beta discards its pending
changes. Manual beta rollback holds both file restoration and restart until
**Resume when empty** is selected on its rollback campaign.
Finish pending file campaigns before opening a beta on their servers; this keeps
an older queued change from overwriting the version tested by the new beta.
Deleting a server preserves its name, recorded beta progress and test evidence,
campaign results and file diffs, provisioning and maintenance history, and match
records. Historical entries show **Deleted** and omit live server links and
actions. Reusing the same server name starts a new instance; old records remain
marked deleted, and old work cannot target the replacement. Servers linked to an
active beta cannot be destroyed or rebuilt until that beta is promoted or rolled
back. A participant removed outside provisioning still pauses its open beta; roll
it back and open a new beta to test the replacement.
Normal history retention still applies. Records erased by earlier versions cannot
be restored by this upgrade. Existing history is marked deleted where the server
is already absent; instances that reused a name before upgrading cannot always be
distinguished.
Provisioning (`/provisioning`, `/api/provisioning`) owns Create, Start, Restart, Rebuild, Stop, and Destroy: request persistence, eligibility and occupancy checks, execution, recovery, and history. Campaigns, betas, and scheduled maintenance request lifecycle work through Provisioning and link to the resulting operation. They retain their file, test-evidence, and version-verification workflows. Pending automated restarts appear in Provisioning before execution. History survives server removal. New creation requests return HTTP 202 with an operation ID and open the provisioning detail page. Create and Rebuild complete only when Komodo reports healthy, RCON or fresh telemetry verifies the selected engine map and game status, and RCON reports a Plutonium runtime revision at least as new as the current production release. Provisioning records the release and runtime evidence so new servers have verified version status immediately. Unknown release or runtime revisions are retried; a confirmed older runtime fails with a clear error and no automatic restart. Queued requests reserve their server name and port; queued creation and pending verification survive application restarts. The detail page records startup progress and explains any health, map, or revision issue blocking completion.
Each server deployment opens a detail page with its outcome, blocker, permitted
recovery actions, grouped execution phases, and timestamped verification evidence.
New jobs record command certainty and the source of game-health checks; older
records retain their known outcomes without invented evidence. A failed restart's
ongoing recovery and held capacity appear separately from its recorded failure.
Recent activity is visible immediately, with searchable history loaded in pages.
Launch files distinguish the setup captured while queued from the files selected
when creation began. The page retains its last snapshot if a refresh fails.
Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh zero-player RCON or telemetry report, or a verified stopped container, before teardown. Occupied reports from either source block teardown; servers without current verification remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the provisioning detail page.
**Settings → Concurrent server destructions** limits simultaneous individual and
fleet removals (default 1). Each destruction holds its slot through teardown and
local cleanup. Lowering the limit lets active removals finish before more begin.
Use **Provisioning → Manage server**, or a server’s **Server actions**, to start,
restart, stop, or rebuild an existing instance. **Rebuild destroys and creates in
place**: the old container, Komodo deployment, server registration, and all server
files are deleted. A fresh instance uses the same server ID, map, and port with
the latest promoted setup. No old instance files are carried forward. The key
stored separately in **Keys** must be available before teardown. Stop retains the
instance so Start can bring it back. Disruptive operations wait for a fresh empty
reading or a confirmed stopped container, and completion requires verification.
See [Provisioning](docs/provisioning.md) for lifecycle behavior and recovery.
Existing `/deployments` page links redirect to `/provisioning`, preserving filters
and operation IDs. Legacy `/api/deployments` endpoints and stored history remain
compatible.
Automated tests use simulated Docker/game hosts. Real Debian installer and
T5/T6 game-server validation remain deployment acceptance work; see the checklist
in the native-host guide.