electric-cherry/README.md

# Electric Cherry

Web app that manages a fleet of T5 and T6 Plutonium Zombies servers running under Komodo.

Fleet membership lives in SQLite. EC registers servers during creation and removes
them during destruction; startup does not discover or import server folders.
Existing database registrations are preserved, and missing mounts do not remove
servers from the fleet. The legacy `seedServerRegistry` helper is an explicit
import utility only.

Status refreshes check Komodo container readiness, including nested Docker health
status. A running container with `health: starting` is still **Starting**. Starting
or restarting containers hide old game data and RCON errors; stopped, unhealthy,
or unverifiable containers do not trigger routine RCON polling. Containers without
a Docker health check use their running state as the readiness signal.

With fresh container health, validated telemetry received in the last 45 seconds
supplies the current player count, roster, map, and game status. RCON is the fallback
when telemetry is unavailable or stale. Fleet, server details, metrics, collection
coverage, setup readiness, and beta health checks share this policy. Reports from
before the latest start, restart, or rebuild cannot verify the new process.
Historical scoreboards remain visible and are labeled separately from live counts;
RCON diagnostics retain their own results. Telemetry does not report ping or player
capacity, so those fields remain unknown in telemetry-only observations.

**Active** measures time since the lobby was first observed with players;
**Match** measures time since the script's `game_start` was received. Telemetry
records occupancy changes immediately, so a later heartbeat or status poll does
not move the active timer forward. A new match keeps the active timer if the
lobby stays occupied. An empty report clears the match timer and begins the
inactive period; a later join begins a new active period. These are observed
times, not exact player connection timestamps.

## Fleet overview

Fleet searches server names, map names, and the full player roster. Filter occupied,
empty, unhealthy, testing, or pinned lobbies; switch between cards and a compact
table; and group by game or map. Pins and display preferences are saved in this
browser. Map groups preserve T6 variants and can preselect the exact map when
launching another lobby.

Health summaries exclude stale player counts. The match spotlight uses fresh round
telemetry from an occupied, ongoing match. Persisted campaign, beta, creation,
removal, and restart-recovery state appears alongside affected servers. Fleet
membership and operation progress refresh every ten seconds, with live server
status between refreshes. Failed refreshes keep the last snapshot visible.

**Metrics** explores recorded server use by time range, game, and map. It includes
occupied/idle/unknown timelines, sortable rankings, previous-period comparisons,
map capacity pressure, and an activity heatmap in the selected timezone. Hosting
experiments suggest changes from explicit rules, including tentative advice from
sparse observations; they never change the fleet automatically. Background refresh
keeps the last report visible if it fails. See [Metrics](docs/metrics.md) for the
definitions, evidence limits, and experimental rules.

## Stack

- SvelteKit (adapter-node) + TypeScript
- SQLite via better-sqlite3 (replaces `script_metadata.json` as the source of
  truth)
- Server-sent events for live pipeline / server-status streams
- Game telemetry for current player counts and rosters, with direct UDP RCON as fallback
- Komodo API for restarts and per-server game-port discovery
- Runs as a single Komodo-managed container on the same host as the game
  servers, with `/root/zombies` bind-mounted in

## Layout

- `src/lib/server/db/` — schema, migrations, better-sqlite3 handle
- `src/lib/server/fleet/` — file walking, push/beta engines
- `src/lib/server/integrations/` — Komodo, RCON
- `src/lib/server/pipeline/` — sweep loop, event bus, worker singleton
- `src/lib/server/provisioning/` — lifecycle requests, queues, execution, recovery, and history
- `src/routes/api/` — REST + SSE endpoints
- `src/routes/` — UI pages (dashboard, campaigns, server deployments, betas, per-server)
- `Dockerfile`, `docker-compose.yml`, `entrypoint.sh` — container build
- `komodo/` — Komodo stack definition
- `scripts/` — legacy JSON importer, fixture seeder
- `tests/` — vitest unit tests

## Plugin releases

Plugins support named release versions, GitHub DLL imports and update discovery.
See [Plugin releases](docs/plugin-releases.md) for the import and beta workflow.

## Maintenance

Open **Operations → Maintenance** for health recovery, configuration drift,
Plutonium updates, and scheduled restarts. Overview shows active work, attention,
version coverage, and upcoming restarts. Policies & schedules controls each
capability independently; Activity links findings and results to Provisioning.
Fleet and server pages show runtime revision indicators and maintenance links.

Maintenance decides when work is needed and verifies the result. Provisioning
executes every lifecycle operation using the shared safety checks and queues.
Health recovery uses one restart followed by one rebuild if needed; drift repair
requests a rebuild with a linked Campaign. **Rebuild deletes all instance files.**
Maintenance skips observations and new automatic work on servers with active
Provisioning operations or retained recovery capacity. Other servers continue
normally, and existing maintenance still tracks its own operation's outcome.

Plutonium updates are applied by Plutainer's startup updater
(`PLUTAINER_AUTO_UPDATE` must be enabled). Check versions and choose **Update on
restart**, or opt into automatic updates for verified outdated servers. Revision
checks run hourly, and Create and Rebuild verify and record the running revision
before completing. Unknown or stale revisions remain unverified. A healthy restart
can still fail update verification; Maintenance reports that outcome without
rebuilding or repeatedly restarting the server. Electric Cherry does not download
binaries or pin a runtime revision.

Optional daily restarts are independent of updates. Schedules use the saved
timezone, follow daylight saving changes, and coalesce missed days into one run.
Intentionally stopped servers are excluded. Existing schedules, pending jobs,
operation IDs, and history survive the move from Settings and Watchdog.
Servers owned by Provisioning are skipped for that scheduled occurrence.

**Pause all maintenance** holds new and queued maintenance while checks and
submitted recovery continue. Each capability retains its own policy. Automatic
updates remain disabled until explicitly enabled. See [Maintenance](docs/maintenance.md)
for verification, retry, scheduling, and compatibility details.

## Fleet settings

**Settings → Fleet settings** controls campaign history retention (default 7 days),
concurrent restarts (default 1), concurrent server startups (default 1), and
concurrent server destructions (default 1).
Values are positive whole numbers; retention supports up to 36500 days. Settings
are saved in SQLite for all users and apply at the next queue or cleanup check
without restarting the app. Lowering a limit leaves active operations running
and holds queued work until capacity is available. Collection previews and
progress pages display the current startup limit.

Retention removes eligible old campaign records and file diffs. Pending, failed,
held, and health-recovery-linked campaigns remain protected; Provisioning history
is retained. Shortening retention can remove older history on the next sweep.

On the first startup after upgrading, existing `DEPLOY_KEEP_DAYS`,
`DEPLOYMENT_RESTART_CONCURRENCY`, and `DEPLOYMENT_CREATE_CONCURRENCY` environment
values initialize the saved settings. Keep custom values for that first startup;
afterward, the database values take precedence and those variables can be removed.
The supplied Compose and Komodo examples use the built-in defaults.
Existing saved settings retain their values when upgraded; the destruction
limit starts at 1 and can be changed in Settings.

`GET /api/settings` reads the four values (`deployKeepDays`, `restartConcurrency`,
`createConcurrency`, `destroyConcurrency`); `PUT /api/settings` saves all four together as JSON.
Invalid or incomplete requests return HTTP 400 without changing saved settings.

## How restarts stay safe

Betas support configurable gameplay, participation, stability and manual-review
requirements, saved presets, coverage rules and staged waves. See
[Beta test plans](docs/beta-test-plans.md) for usage and the four-event game telemetry contract.

File changes are staged sequentially. Their activation uses rolling restarts:
by default only **one server across the fleet** can restart or recover at a
time. Promotions, rollbacks, drift reconciliation, and beta restarts share this
limit. Adjust **Concurrent restarts** in **Settings → Fleet settings** to raise it.

The Provisioning worker runs every 15s. Campaigns, betas, and maintenance submit
durable restart requests and read their results. Provisioning checks recovering
servers first, then admits queued operations when there is capacity. Requests
normally run in submission order; on a shared server, beta activation precedes
campaign activation, and maintenance waits for both. Busy
or unreachable servers stay queued without taking a slot, so other empty
servers can proceed. A successful Komodo restart request retains its slot
until the container is healthy and RCON or fresh telemetry verifies game status. Health checks start no sooner
than 15 seconds after the request returns, avoiding an immediate check of the
old process. An unhealthy server keeps its slot rather than letting failures
spread across the fleet. A timed-out restart also keeps its slot because Komodo
may have accepted the request.

Slots persist in SQLite across app restarts, and overlapping timer/API triggers
share the active run. Beta options to skip health or match gates still work;
skipping a beta gate does not release the fleet's recovery slot early. New-server
creation uses a separate shared startup queue, holding its slot through container,
game-status, selected-map, and Plutonium revision verification.

No token, no session, no auth freeze. If Komodo or RCON aren't configured,
the sweep marks pending rows `manual` so nothing hangs forever.

## Collections, provisioning, and launch previews

Provisioning also offers **House Party Protocol**: review the entire fleet and queue
destruction in one action. This includes queued creations and reuses pending
destruction requests. Each server waits for a fresh zero-player reading; occupied
or unreachable servers remain queued. Containers, server files, and registrations
are removed, while saved keys and collections are retained. The preview must be
reviewed again if fleet membership changes before confirmation. Servers created
after confirmation are not included. Progress appears in deployment history.
If any target belongs to an active beta, no removals are queued. Promote or roll
back that beta before retrying fleet removal or an exact collection launch.

If deletion fails after Komodo removal, the operation detail page offers
**Resume cleanup** to retry local file and database removal without RCON or
repeating Komodo deletion. Requesting destruction again also resumes that cleanup.
Recovery is blocked if a newer operation exists for the same server name.

**Operations → Collections** includes a builder for whole server lineups: choose
multiple T5/T6 maps and set the desired total lobbies for each. Collections also
has **Active launches** and **Launch history** tabs, with progress at
`/collections/launches/:id`.
Each server operation remains in Provisioning and links back to its collection
launch. Existing collection launch bookmarks redirect to the new location.
**Conform fleet**
counts existing and queued servers and creates only the shortfall. The preview
shows target/existing/missing counts and IDs, ports, and key requirements for new
servers only. Already-satisfied targets create nothing; excess servers are kept.
Choose **Match collection exactly** in the preview to also remove excess lobbies
and maps outside the collection. Review the keep/create/remove lists and confirm
removals. Matching servers stay untouched; unstarted excess creations are cancelled,
and running servers wait for a fresh zero-player check before removal. New lobbies
can reserve ports from these removals and wait until they are released. The collection
page tracks both actions. **Fill missing servers** remains the default.
All jobs and capacity reservations are committed together before provisioning;
one collection page tracks each server's independent progress and failures.
New servers start one at a time by default across all collections and individual
launches. A startup slot stays occupied until Komodo is healthy, RCON or fresh
telemetry reports the selected map, and RCON verifies the current Plutonium
production revision or newer. Unknown release or runtime revisions hold the queue
while verification retries; a confirmed older runtime fails provisioning. The
queue survives application restarts. The collection preview
shows the startup limit, and its progress page shows active startups and each
server's place in the shared queue. Adjust **Concurrent server startups** in
**Settings → Fleet settings** to increase the limit. Each worker pass starts at most the
available number of slots; immediate failures do not drain the remaining queue.
Existing single-server collections can be expanded with **Edit**. Single-server
launches and launching again from recorded recipes remain available.
Collections save maps and target lobby counts. Every new server uses the latest
promoted scripts, plugins, and configuration when provisioning begins, including
promotions made after preview or while queued. Existing collections automatically
follow this policy. Deployment history records the files selected for that launch;
**Launch again** uses the current promoted setup for the same game and map.
The progress page explains waiting conditions, shows elapsed time, and can resume
health verification of an interrupted creation when its original server remains.
See [Deployment collections](docs/deployment-blueprints.md) for the workflow,
promotion policy, and API details.

## Saved server keys

Use **Keys** to add, replace, or remove a Plutonium key for an exact
server ID such as `t5_zombie_theater_2`. IDs use a supported map slug and a
lobby number from 1 to 20; the server does not need to be deployed first.
During deployment, choose the map. The lowest available lobby is assigned
automatically, with the exact server ID shown before deployment. Its saved key is reused automatically,
and any newly entered key is saved for that ID, including if deployment fails.
Destroying a server preserves its saved key. Updating or removing a saved key
does not change an existing server; changes apply when that server ID is next
created or rebuilt. Electric Cherry does not check whether Plutonium accepts a
saved key. **Last saved** shows when the key was saved, not a validation result.

Keys are stored in the application's SQLite database at `DB_PATH`, independently
of server records. Preserve that database across container replacements. Key
values are not returned by the management API, but are stored unencrypted in
SQLite, so database files and backups contain credentials. Existing deployment
keys are not automatically imported from Komodo; add them once to retain them.

## Local dev

Windows or macOS — no `/root/zombies` on disk, so seed a fixture tree first.

```bash
npm install
npm run seed:fixture

# Point env at the fixture (PowerShell)
$env:SHARED_SCRIPTS = "$PWD/test-fixtures/zombies/shared/scripts"
$env:SERVERS_ROOT   = "$PWD/test-fixtures/zombies/t5"
$env:DB_PATH        = "$PWD/data/cherry.db"

npm run dev
# open http://localhost:3000
```

Without RCON_PASSWORD / KOMODO_* set, the pipeline marks pending rows
`manual` after a sweep — the UI still works, you just don't get real
restarts. Enough to develop the push/beta flow.

## Map selection and configs

The map selected when deploying a T5 zombies server controls its startup
rotation automatically. No map-choice config overlay is required. New servers
use the promoted game baseline and matching overlays when available; otherwise
Plutainer seeds its usual defaults and Electric Cherry applies the selected map
through a generated `configs/electric_cherry_map.cfg`, loaded after the main
config and before map rotation. This file is tracked and restored by drift
reconciliation; operators do not need to create or promote it. The launch order
uses Plutainer's [extra arguments and map-rotation controls](https://github.com/Ayymoss/Plutainer/blob/main/docs/configuration.md).
Plutainer's
default main config is preserved until a game baseline is promoted.

Beta previews, promotion, and config drift repair all apply the server's stored
catalog map after composing the baseline and overlays. An old `sv_maprotation`
overlay cannot override that selection, but its other settings still apply.
Existing servers receive this behavior on their next config deployment or drift
reconciliation; this change does not immediately rewrite or restart them.
Servers with an unknown map retain their existing config behavior. Rollback
restores the exact prior config bytes.

## T6 Zombies (Black Ops II)

Choose **Black Ops II (T6)** in Deploy new server. T6 uses `t6_zombie_<slug>_<lobby>`
IDs, its own saved Plutonium keys, `PLUTAINER_GAME=t6zm`, and
`configs/dedicated_zm.cfg`. Existing T5 deployments and environment settings
keep their current paths. API clients can send `game: "t6"`; omitting it selects T5.

Before the first deployment, populate the Docker volume `t6_game_files` on the
Komodo host with your Black Ops II game files, including the DLC maps you intend
to host. Set `PLUTAINER_T6_GAMEFILES_VOLUME` to use another volume or an absolute
host directory. It is mounted read-only at `/home/plutainer/gamefiles`.
Plutainer supplies its runtime and config seeds, not the base game files.
Use a Plutonium server key created for **T6 Zombies**.

Server state defaults to `/root/zombies/t6`, or the sibling `t6` directory beside
`SERVERS_ROOT`; override with `T6_SERVERS_ROOT`. Electric Cherry and the Komodo
host must see this directory at the same absolute path. The supplied Compose
mount already covers both games. GSC files go to
`runtime/plutonium/storage/t6/scripts/zm` (no `raw/`), and game logs use
`logs/games_zm.log`. Optional overrides are `T6_REL_DEST_SCRIPT` and
`T6_CONFIG_TARGET_FILE`; a custom config filename needs a promoted baseline or
a file supplied at that name, since Plutainer seeds `dedicated_zm.cfg`.

The catalog includes the classic maps, Survival, Grief, and Turned selections.
Rotations use `execgts <ruleset>.cfg map <engine-map>`; Plutainer seeds the required
`storage/t6/gamesettings` files. Keep seeding enabled, including with a promoted
baseline. Town, Farm, and Bus Depot share `zm_transit`, so EC keeps their selection
in `map_slug` through creation, beta, promotion, and drift repair. Map-scoped
artifacts target the engine map and therefore apply to all its variants.
Creation health checks verify the engine map reported by RCON or fresh telemetry; this does not
independently verify the start location within that map.

Both games share the fleet's port allocator (`SERVER_PORT_MIN`/`SERVER_PORT_MAX`)
to prevent collisions. EC explicitly sets `PLUTAINER_PORT`, so T6 need not use
its upstream default of 4976. Forward the allocated UDP ports. Use a current
Plutainer v2 image: EC relies on `PLUTAINER_RCON_PASSWORD`,
`PLUTAINER_MAP_ROTATE=false`, and extra arguments loaded after the main config.
Current Plutainer automatically whitelists the Docker gateway for T5/T6 RCON;
an admin client on another host needs its address allowed in the server config.

Implementation references (checked September 2026):

- [Plutainer game requirements](https://github.com/Ayymoss/Plutainer/blob/main/docs/games.md)
- [Plutainer launch, seeding, and RCON implementation](https://github.com/Ayymoss/Plutainer/blob/main/scripts/lib/cod.sh)
- [Plutainer configuration](https://github.com/Ayymoss/Plutainer/blob/main/docs/configuration.md)
- [T6 map/ruleset combinations](https://github.com/xerxes-at/T6ServerConfigs/blob/master/localappdata/Plutonium/storage/t6/dedicated_zm.cfg)
- [Plutonium GSC paths](https://plutonium.pw/docs/modding/loading-mods/)

## Game telemetry

The server script sends JSON to `POST /api/telemetry`:

- `game_start`: server, map, and all players (name and GUID).
- `round_change`: server, map, round, and all players with kills, headshots, downs, revives, and score.
- `heartbeat`: the same fields as round change, sent periodically during a round to refresh player stats.
- `game_end`: the same fields as round change, with final stats.

These reports work with or without an active beta. Electric Cherry records
the full player data, updates Fleet, and applies gameplay progress to active
betas automatically. No discovery request or beta-specific fields are needed.

See [Game reporter protocol](docs/beta-test-plans.md#game-reporter-protocol) for
complete payload examples. Fresh telemetry can verify player count and game
activity alongside RCON during provisioning and Watchdog recovery. Occupied
reports prevent interruptions; stale reports cannot authorize them. Komodo still
verifies container readiness.

For a missing scoreboard, open the server's **Diagnostics → Telemetry API requests**.
The log shows received requests for that exact server name, including accepted
reports, rejected payloads, HTTP status, errors, and expandable submitted JSON.
It refreshes every five seconds while visible and survives app restarts. Logging
starts with this version; older requests cannot be reconstructed. See
[Telemetry diagnostics](docs/beta-test-plans.md#telemetry-diagnostics) for retention
and how to interpret missing or rejected reports.

The **Leaderboard** records persistent player totals and map records from these
same reports, including highest round, cumulative kills, rounds survived,
headshots, revives, downs, and games played. See [Leaderboard](docs/leaderboard.md)
for counting rules, history import, and the read API. Only completed games count;
in-progress and interrupted games remain in Match History without contributing
to player rankings or profile totals.

In **Match History**, select **View match** on a completed match to open its
shareable results page. It shows the map, server, final round, timing, and every
recorded player's stats, including players absent from the final report. Completed
matches also link from server history and player profiles. Results survive server
deletion; missing starts and unrecorded stats remain explicitly unknown.

## Running tests

```bash
npm test
```

Covers push (add/update/skip), beta lifecycle (open → rollback, open →
promote, conflict blocks), sweep (integrations-off manual branch), and
RCON parsing (fake UDP socket).

## Deploy to the Komodo host

1. Build the image once locally or on the host:

   ```bash
   docker compose build
   ```

2. Import `komodo/stack.toml` into Komodo (or point Komodo at the
   `docker-compose.yml` at repo root).

3. Set the env vars in Komodo:

   ```
   KOMODO_ADDRESS, KOMODO_API_KEY, KOMODO_API_SECRET
   RCON_PASSWORD              # single shared password across every T5/T6 server
   RCON_HOST                  # optional; defaults to host.docker.internal
   ```

4. First boot triggers the legacy import: if
   `/root/zombies/script_metadata.json` exists and `/data/cherry.db` is
   empty, `entrypoint.sh` runs `scripts/import-legacy.ts` before starting
   the server. The source JSON gets renamed to `.migrated`.

5. Load the app on the LAN or over tailscale (no web-app auth by design).

## Campaigns and provisioning

Campaigns (`/campaigns`) tracks script/config updates, drift reconciliation, restart progress, file diffs, and rollbacks. Existing campaign records are preserved in the original database tables. The campaign API lives at `/api/campaigns`; legacy detail API endpoints remain available for compatibility.

Beta deployments, promotions, drift repairs, and rollbacks stage their file changes
in SQLite. Live server files and inventory remain unchanged while waiting for an
empty or stopped server, other operations, and rolling restart capacity. Provisioning
captures rollback snapshots and applies the staged changes immediately before
requesting the restart. Failed file activation blocks that restart. Queued changes
survive application restarts; cancelling an unapplied beta discards its pending
changes. Manual beta rollback holds both file restoration and restart until
**Resume when empty** is selected on its rollback campaign.
Finish pending file campaigns before opening a beta on their servers; this keeps
an older queued change from overwriting the version tested by the new beta.

Deleting a server preserves its name, recorded beta progress and test evidence,
campaign results and file diffs, provisioning and maintenance history, and match
records. Historical entries show **Deleted** and omit live server links and
actions. Reusing the same server name starts a new instance; old records remain
marked deleted, and old work cannot target the replacement. Servers linked to an
active beta cannot be destroyed or rebuilt until that beta is promoted or rolled
back. A participant removed outside provisioning still pauses its open beta; roll
it back and open a new beta to test the replacement.
Normal history retention still applies. Records erased by earlier versions cannot
be restored by this upgrade. Existing history is marked deleted where the server
is already absent; instances that reused a name before upgrading cannot always be
distinguished.

Provisioning (`/provisioning`, `/api/provisioning`) owns Create, Start, Restart, Rebuild, Stop, and Destroy: request persistence, eligibility and occupancy checks, execution, recovery, and history. Campaigns, betas, and scheduled maintenance request lifecycle work through Provisioning and link to the resulting operation. They retain their file, test-evidence, and version-verification workflows. Pending automated restarts appear in Provisioning before execution. History survives server removal. New creation requests return HTTP 202 with an operation ID and open the provisioning detail page. Create and Rebuild complete only when Komodo reports healthy, RCON or fresh telemetry verifies the selected engine map and game status, and RCON reports a Plutonium runtime revision at least as new as the current production release. Provisioning records the release and runtime evidence so new servers have verified version status immediately. Unknown release or runtime revisions are retried; a confirmed older runtime fails with a clear error and no automatic restart. Queued requests reserve their server name and port; queued creation and pending verification survive application restarts. The detail page records startup progress and explains any health, map, or revision issue blocking completion.

Each server deployment opens a detail page with its outcome, blocker, permitted
recovery actions, grouped execution phases, and timestamped verification evidence.
New jobs record command certainty and the source of game-health checks; older
records retain their known outcomes without invented evidence. A failed restart's
ongoing recovery and held capacity appear separately from its recorded failure.
Recent activity is visible immediately, with searchable history loaded in pages.
Launch files distinguish the setup captured while queued from the files selected
when creation began. The page retains its last snapshot if a refresh fails.

Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh zero-player RCON or telemetry report, or a verified stopped container, before teardown. Occupied reports from either source block teardown; servers without current verification remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the provisioning detail page.

**Settings → Concurrent server destructions** limits simultaneous individual and
fleet removals (default 1). Each destruction holds its slot through teardown and
local cleanup. Lowering the limit lets active removals finish before more begin.


Use **Provisioning → Manage server**, or a server’s **Server actions**, to start,
restart, stop, or rebuild an existing instance. **Rebuild destroys and creates in
place**: the old container, Komodo deployment, server registration, and all server
files are deleted. A fresh instance uses the same server ID, map, and port with
the latest promoted setup. No old instance files are carried forward. The key
stored separately in **Keys** must be available before teardown. Stop retains the
instance so Start can bring it back. Disruptive operations wait for a fresh empty
reading or a confirmed stopped container, and completion requires verification.
See [Provisioning](docs/provisioning.md) for lifecycle behavior and recovery.

Existing `/deployments` page links redirect to `/provisioning`, preserving filters
and operation IDs. Legacy `/api/deployments` endpoints and stored history remain
compatible.