electric-cherry/README.md

# Electric Cherry

Web app that manages a fleet of T5 and T6 Plutonium Zombies servers running under Komodo.

Fleet membership lives in SQLite. EC registers servers during creation and removes
them during destruction; startup does not discover or import server folders.
Existing database registrations are preserved, and missing mounts do not remove
servers from the fleet. The legacy `seedServerRegistry` helper is an explicit
import utility only.

Status refreshes check Komodo container readiness, including nested Docker health
status. A running container with `health: starting` is still **Starting**. Starting
or restarting containers hide old game data and RCON errors; stopped, unhealthy,
or unverifiable containers do not trigger routine RCON polling. Containers without
a Docker health check use their running state as the readiness signal.

With fresh container health, validated telemetry received in the last 45 seconds
supplies the current player count, roster, map, and game status. RCON is the fallback
when telemetry is unavailable or stale. Fleet, server details, metrics, collection
coverage, setup readiness, and beta health checks share this policy. Reports from
before the latest start, restart, or rebuild cannot verify the new process.
Historical scoreboards remain visible and are labeled separately from live counts;
RCON diagnostics retain their own results. Telemetry does not report ping or player
capacity, so those fields remain unknown in telemetry-only observations.

## Fleet overview

Fleet searches server names, map names, and the full player roster. Filter occupied,
empty, unhealthy, testing, or pinned lobbies; switch between cards and a compact
table; and group by game or map. Pins and display preferences are saved in this
browser. Map groups preserve T6 variants and can preselect the exact map when
launching another lobby.

Health summaries exclude stale player counts. The match spotlight uses fresh round
telemetry from an occupied, ongoing match. Persisted campaign, beta, creation,
removal, and restart-recovery state appears alongside affected servers. Fleet
membership and operation progress refresh every ten seconds, with live server
status between refreshes. Failed refreshes keep the last snapshot visible.

## Stack

- SvelteKit (adapter-node) + TypeScript
- SQLite via better-sqlite3 (replaces `script_metadata.json` as the source of
  truth)
- Server-sent events for live pipeline / server-status streams
- Game telemetry for current player counts and rosters, with direct UDP RCON as fallback
- Komodo API for restarts and per-server game-port discovery
- Runs as a single Komodo-managed container on the same host as the game
  servers, with `/root/zombies` bind-mounted in

## Layout

- `src/lib/server/db/` — schema, migrations, better-sqlite3 handle
- `src/lib/server/fleet/` — file walking, push/beta engines
- `src/lib/server/integrations/` — Komodo, RCON
- `src/lib/server/pipeline/` — sweep loop, event bus, worker singleton
- `src/lib/server/provisioning/` — lifecycle requests, queues, execution, recovery, and history
- `src/routes/api/` — REST + SSE endpoints
- `src/routes/` — UI pages (dashboard, campaigns, server deployments, betas, per-server)
- `Dockerfile`, `docker-compose.yml`, `entrypoint.sh` — container build
- `komodo/` — Komodo stack definition
- `scripts/` — legacy JSON importer, fixture seeder
- `tests/` — vitest unit tests

## Plugin releases

Plugins support named release versions, GitHub DLL imports and update discovery.
See [Plugin releases](docs/plugin-releases.md) for the import and beta workflow.

## Maintenance

Open **Operations → Maintenance** for health recovery, configuration drift,
Plutonium updates, and scheduled restarts. Overview shows active work, attention,
version coverage, and upcoming restarts. Policies & schedules controls each
capability independently; Activity links findings and results to Provisioning.
Fleet and server pages show runtime revision indicators and maintenance links.

Maintenance decides when work is needed and verifies the result. Provisioning
executes every lifecycle operation using the shared safety checks and queues.
Health recovery uses one restart followed by one rebuild if needed; drift repair
requests a rebuild with a linked Campaign. **Rebuild deletes all instance files.**

Plutonium updates are applied by Plutainer's startup updater
(`PLUTAINER_AUTO_UPDATE` must be enabled). Check versions and choose **Update on
restart**, or opt into automatic updates for verified outdated servers. Revision
checks run hourly. Unknown or stale revisions remain unverified. A healthy restart
can still fail update verification; Maintenance reports that outcome without
rebuilding or repeatedly restarting the server. Electric Cherry does not download
binaries or pin a runtime revision.

Optional daily restarts are independent of updates. Schedules use the saved
timezone, follow daylight saving changes, and coalesce missed days into one run.
Intentionally stopped servers are excluded. Existing schedules, pending jobs,
operation IDs, and history survive the move from Settings and Watchdog.

**Pause all maintenance** holds new and queued maintenance while checks and
submitted recovery continue. Each capability retains its own policy. Automatic
updates remain disabled until explicitly enabled. See [Maintenance](docs/maintenance.md)
for verification, retry, scheduling, and compatibility details.

## Fleet settings

**Settings → Fleet settings** controls campaign history retention (default 7 days),
concurrent restarts (default 1), and concurrent server startups (default 1).
Values are positive whole numbers; retention supports up to 36500 days. Settings
are saved in SQLite for all users and apply at the next queue or cleanup check
without restarting the app. Lowering a limit leaves active operations running
and holds queued work until capacity is available. Collection previews and
progress pages display the current startup limit.

Retention removes eligible old campaign records and file diffs. Pending, failed,
held, and health-recovery-linked campaigns remain protected; Provisioning history
is retained. Shortening retention can remove older history on the next sweep.

On the first startup after upgrading, existing `DEPLOY_KEEP_DAYS`,
`DEPLOYMENT_RESTART_CONCURRENCY`, and `DEPLOYMENT_CREATE_CONCURRENCY` environment
values initialize the saved settings. Keep custom values for that first startup;
afterward, the database values take precedence and those variables can be removed.
The supplied Compose and Komodo examples use the built-in defaults.

`GET /api/settings` reads the three values (`deployKeepDays`, `restartConcurrency`,
`createConcurrency`); `PUT /api/settings` saves all three together as JSON.
Invalid or incomplete requests return HTTP 400 without changing saved settings.

## How restarts stay safe

Betas support configurable gameplay, participation, stability and manual-review
requirements, saved presets, coverage rules and staged waves. See
[Beta test plans](docs/beta-test-plans.md) for usage and the four-event game telemetry contract.

File changes are staged sequentially. Their activation uses rolling restarts:
by default only **one server across the fleet** can restart or recover at a
time. Promotions, rollbacks, drift reconciliation, and beta restarts share this
limit. Adjust **Concurrent restarts** in **Settings → Fleet settings** to raise it.

The Provisioning worker runs every 15s. Campaigns, betas, and maintenance submit
durable restart requests and read their results. Provisioning checks recovering
servers first, then admits queued operations when there is capacity. Requests
normally run in submission order; on a shared server, beta activation precedes
campaign activation, and maintenance waits for both. Busy
or unreachable servers stay queued without taking a slot, so other empty
servers can proceed. A successful Komodo restart request retains its slot
until the container is healthy and RCON or fresh telemetry verifies game status. Health checks start no sooner
than 15 seconds after the request returns, avoiding an immediate check of the
old process. An unhealthy server keeps its slot rather than letting failures
spread across the fleet. A timed-out restart also keeps its slot because Komodo
may have accepted the request.

Slots persist in SQLite across app restarts, and overlapping timer/API triggers
share the active run. Beta options to skip health or match gates still work;
skipping a beta gate does not release the fleet's recovery slot early. New-server
creation uses a separate shared startup queue, holding its slot through container,
game-status, and selected-map verification.

No token, no session, no auth freeze. If Komodo or RCON aren't configured,
the sweep marks pending rows `manual` so nothing hangs forever.

## Provisioning collections and launch previews

Provisioning also offers **House Party Protocol**: review the entire fleet and queue
destruction in one action. This includes queued creations and reuses pending
destruction requests. Each server waits for a fresh zero-player reading; occupied
or unreachable servers remain queued. Containers, server files, and registrations
are removed, while saved keys and collections are retained. The preview must be
reviewed again if fleet membership changes before confirmation. Servers created
after confirmation are not included. Progress appears in deployment history.

If deletion fails after Komodo removal, the operation detail page offers
**Resume cleanup** to retry local file and database removal without RCON or
repeating Komodo deletion. Requesting destruction again also resumes that cleanup.
Recovery is blocked if a newer operation exists for the same server name.

**Operations → Collections** includes a builder for whole server lineups: choose
multiple T5/T6 maps and set the desired total lobbies for each. **Conform fleet**
counts existing and queued servers and creates only the shortfall. The preview
shows target/existing/missing counts and IDs, ports, and key requirements for new
servers only. Already-satisfied targets create nothing; excess servers are kept.
Choose **Match collection exactly** in the preview to also remove excess lobbies
and maps outside the collection. Review the keep/create/remove lists and confirm
removals. Matching servers stay untouched; unstarted excess creations are cancelled,
and running servers wait for a fresh zero-player check before removal. New lobbies
can reserve ports from these removals and wait until they are released. The collection
page tracks both actions. **Fill missing servers** remains the default.
All jobs and capacity reservations are committed together before provisioning;
one collection page tracks each server's independent progress and failures.
New servers start one at a time by default across all collections and individual
launches. A startup slot stays occupied until Komodo is healthy and RCON or
fresh telemetry reports the selected map; an unready server holds the queue until it recovers
or is removed. The queue survives application restarts. The collection preview
shows the startup limit, and its progress page shows active startups and each
server's place in the shared queue. Adjust **Concurrent server startups** in
**Settings → Fleet settings** to increase the limit. Each worker pass starts at most the
available number of slots; immediate failures do not drain the remaining queue.
Existing single-server collections can be expanded with **Edit**. Single-server
launches and launching again from recorded recipes remain available.
Collections save maps and target lobby counts. Every new server uses the latest
promoted scripts, plugins, and configuration when provisioning begins, including
promotions made after preview or while queued. Existing collections automatically
follow this policy. Deployment history records the files selected for that launch;
**Launch again** uses the current promoted setup for the same game and map.
The progress page explains waiting conditions, shows elapsed time, and can resume
health verification of an interrupted creation when its original server remains.
See [Deployment collections](docs/deployment-blueprints.md) for the workflow,
promotion policy, and API details.

## Saved server keys

Use **Keys** to add, replace, or remove a Plutonium key for an exact
server ID such as `t5_zombie_theater_2`. IDs use a supported map slug and a
lobby number from 1 to 20; the server does not need to be deployed first.
During deployment, choose the map. The lowest available lobby is assigned
automatically, with the exact server ID shown before deployment. Its saved key is reused automatically,
and any newly entered key is saved for that ID, including if deployment fails.
Destroying a server preserves its saved key. Updating or removing a saved key
does not change a running server; changes apply to future deployments.

Keys are stored in the application's SQLite database at `DB_PATH`, independently
of server records. Preserve that database across container replacements. Key
values are not returned by the management API, but are stored unencrypted in
SQLite, so database files and backups contain credentials. Existing deployment
keys are not automatically imported from Komodo; add them once to retain them.

## Local dev

Windows or macOS — no `/root/zombies` on disk, so seed a fixture tree first.

```bash
npm install
npm run seed:fixture

# Point env at the fixture (PowerShell)
$env:SHARED_SCRIPTS = "$PWD/test-fixtures/zombies/shared/scripts"
$env:SERVERS_ROOT   = "$PWD/test-fixtures/zombies/t5"
$env:DB_PATH        = "$PWD/data/cherry.db"

npm run dev
# open http://localhost:3000
```

Without RCON_PASSWORD / KOMODO_* set, the pipeline marks pending rows
`manual` after a sweep — the UI still works, you just don't get real
restarts. Enough to develop the push/beta flow.

## Map selection and configs

The map selected when deploying a T5 zombies server controls its startup
rotation automatically. No map-choice config overlay is required. New servers
use the promoted game baseline and matching overlays when available; otherwise
Plutainer seeds its usual defaults and Electric Cherry applies the selected map
through a generated `configs/electric_cherry_map.cfg`, loaded after the main
config and before map rotation. This file is tracked and restored by drift
reconciliation; operators do not need to create or promote it. The launch order
uses Plutainer's [extra arguments and map-rotation controls](https://github.com/Ayymoss/Plutainer/blob/main/docs/configuration.md).
Plutainer's
default main config is preserved until a game baseline is promoted.

Beta previews, promotion, and config drift repair all apply the server's stored
catalog map after composing the baseline and overlays. An old `sv_maprotation`
overlay cannot override that selection, but its other settings still apply.
Existing servers receive this behavior on their next config deployment or drift
reconciliation; this change does not immediately rewrite or restart them.
Servers with an unknown map retain their existing config behavior. Rollback
restores the exact prior config bytes.

## T6 Zombies (Black Ops II)

Choose **Black Ops II (T6)** in Deploy new server. T6 uses `t6_zombie_<slug>_<lobby>`
IDs, its own saved Plutonium keys, `PLUTAINER_GAME=t6zm`, and
`configs/dedicated_zm.cfg`. Existing T5 deployments and environment settings
keep their current paths. API clients can send `game: "t6"`; omitting it selects T5.

Before the first deployment, populate the Docker volume `t6_game_files` on the
Komodo host with your Black Ops II game files, including the DLC maps you intend
to host. Set `PLUTAINER_T6_GAMEFILES_VOLUME` to use another volume or an absolute
host directory. It is mounted read-only at `/home/plutainer/gamefiles`.
Plutainer supplies its runtime and config seeds, not the base game files.
Use a Plutonium server key created for **T6 Zombies**.

Server state defaults to `/root/zombies/t6`, or the sibling `t6` directory beside
`SERVERS_ROOT`; override with `T6_SERVERS_ROOT`. Electric Cherry and the Komodo
host must see this directory at the same absolute path. The supplied Compose
mount already covers both games. GSC files go to
`runtime/plutonium/storage/t6/scripts/zm` (no `raw/`), and game logs use
`logs/games_zm.log`. Optional overrides are `T6_REL_DEST_SCRIPT` and
`T6_CONFIG_TARGET_FILE`; a custom config filename needs a promoted baseline or
a file supplied at that name, since Plutainer seeds `dedicated_zm.cfg`.

The catalog includes the classic maps, Survival, Grief, and Turned selections.
Rotations use `execgts <ruleset>.cfg map <engine-map>`; Plutainer seeds the required
`storage/t6/gamesettings` files. Keep seeding enabled, including with a promoted
baseline. Town, Farm, and Bus Depot share `zm_transit`, so EC keeps their selection
in `map_slug` through creation, beta, promotion, and drift repair. Map-scoped
artifacts target the engine map and therefore apply to all its variants.
Creation health checks verify the engine map reported by RCON or fresh telemetry; this does not
independently verify the start location within that map.

Both games share the fleet's port allocator (`SERVER_PORT_MIN`/`SERVER_PORT_MAX`)
to prevent collisions. EC explicitly sets `PLUTAINER_PORT`, so T6 need not use
its upstream default of 4976. Forward the allocated UDP ports. Use a current
Plutainer v2 image: EC relies on `PLUTAINER_RCON_PASSWORD`,
`PLUTAINER_MAP_ROTATE=false`, and extra arguments loaded after the main config.
Current Plutainer automatically whitelists the Docker gateway for T5/T6 RCON;
an admin client on another host needs its address allowed in the server config.

Implementation references (checked September 2026):

- [Plutainer game requirements](https://github.com/Ayymoss/Plutainer/blob/main/docs/games.md)
- [Plutainer launch, seeding, and RCON implementation](https://github.com/Ayymoss/Plutainer/blob/main/scripts/lib/cod.sh)
- [Plutainer configuration](https://github.com/Ayymoss/Plutainer/blob/main/docs/configuration.md)
- [T6 map/ruleset combinations](https://github.com/xerxes-at/T6ServerConfigs/blob/master/localappdata/Plutonium/storage/t6/dedicated_zm.cfg)
- [Plutonium GSC paths](https://plutonium.pw/docs/modding/loading-mods/)

## Game telemetry

The server script sends JSON to `POST /api/telemetry`:

- `game_start`: server, map, and all players (name and GUID).
- `round_change`: server, map, round, and all players with kills, headshots, downs, revives, and score.
- `heartbeat`: the same fields as round change, sent periodically during a round to refresh player stats.
- `game_end`: the same fields as round change, with final stats.

These reports work with or without an active beta. Electric Cherry records
the full player data, updates Fleet, and applies gameplay progress to active
betas automatically. No discovery request or beta-specific fields are needed.

See [Game reporter protocol](docs/beta-test-plans.md#game-reporter-protocol) for
complete payload examples. Fresh telemetry can verify player count and game
activity alongside RCON during provisioning and Watchdog recovery. Occupied
reports prevent interruptions; stale reports cannot authorize them. Komodo still
verifies container readiness.

For a missing scoreboard, open the server's **Diagnostics → Telemetry API requests**.
The log shows received requests for that exact server name, including accepted
reports, rejected payloads, HTTP status, errors, and expandable submitted JSON.
It refreshes every five seconds while visible and survives app restarts. Logging
starts with this version; older requests cannot be reconstructed. See
[Telemetry diagnostics](docs/beta-test-plans.md#telemetry-diagnostics) for retention
and how to interpret missing or rejected reports.

The **Leaderboard** records persistent player totals and map records from these
same reports, including highest round, cumulative kills, rounds survived,
headshots, revives, downs, and games played. See [Leaderboard](docs/leaderboard.md)
for counting rules, history import, and the read API.

## Running tests

```bash
npm test
```

Covers push (add/update/skip), beta lifecycle (open → rollback, open →
promote, conflict blocks), sweep (integrations-off manual branch), and
RCON parsing (fake UDP socket).

## Deploy to the Komodo host

1. Build the image once locally or on the host:

   ```bash
   docker compose build
   ```

2. Import `komodo/stack.toml` into Komodo (or point Komodo at the
   `docker-compose.yml` at repo root).

3. Set the env vars in Komodo:

   ```
   KOMODO_ADDRESS, KOMODO_API_KEY, KOMODO_API_SECRET
   RCON_PASSWORD              # single shared password across every T5/T6 server
   RCON_HOST                  # optional; defaults to host.docker.internal
   ```

4. First boot triggers the legacy import: if
   `/root/zombies/script_metadata.json` exists and `/data/cherry.db` is
   empty, `entrypoint.sh` runs `scripts/import-legacy.ts` before starting
   the server. The source JSON gets renamed to `.migrated`.

5. Load the app on the LAN or over tailscale (no web-app auth by design).

## Campaigns and provisioning

Campaigns (`/campaigns`) tracks script/config updates, drift reconciliation, restart progress, file diffs, and rollbacks. Existing campaign records are preserved in the original database tables. The campaign API lives at `/api/campaigns`; legacy detail API endpoints remain available for compatibility.

Provisioning (`/provisioning`, `/api/provisioning`) owns Create, Start, Restart, Rebuild, Stop, and Destroy: request persistence, eligibility and occupancy checks, execution, recovery, and history. Campaigns, betas, and scheduled maintenance request lifecycle work through Provisioning and link to the resulting operation. They retain their file, test-evidence, and version-verification workflows. Pending automated restarts appear in Provisioning before execution. History survives server removal. New creation requests return HTTP 202 with an operation ID and open the provisioning detail page. Creation is complete only when Komodo reports healthy and RCON or fresh telemetry verifies game status with an engine map that exactly matches the selected map. Queued requests reserve their server name and port; queued creation and pending health checks survive application restarts. The detail page records startup progress and explains any health or map mismatch that is still blocking completion.

Each server deployment opens a detail page with its flow and timestamped activity. Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh zero-player RCON or telemetry report, or a verified stopped container, before teardown. Occupied reports from either source block teardown; servers without current verification remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the provisioning detail page.


Use **Provisioning → Manage server**, or a server’s **Server actions**, to start,
restart, stop, or rebuild an existing instance. **Rebuild destroys and creates in
place**: the old container, Komodo deployment, server registration, and all server
files are deleted. A fresh instance uses the same server ID, map, and port with
the latest promoted setup. No old instance files are carried forward. The key
stored separately in **Keys** must be available before teardown. Stop retains the
instance so Start can bring it back. Disruptive operations wait for a fresh empty
reading or a confirmed stopped container, and completion requires verification.
See [Provisioning](docs/provisioning.md) for lifecycle behavior and recovery.

Existing `/deployments` page links redirect to `/provisioning`, preserving filters
and operation IDs. Legacy `/api/deployments` endpoints and stored history remain
compatible.