electric-cherry/README.md

# Electric Cherry

Web app that manages a fleet of T5 and T6 Plutonium Zombies servers running under Komodo.

Fleet membership lives in SQLite. EC registers servers during creation and removes
them during destruction; startup does not discover or import server folders.
Existing database registrations are preserved, and missing mounts do not remove
servers from the fleet. The legacy `seedServerRegistry` helper is an explicit
import utility only.

Status refreshes check Komodo container readiness before polling RCON. Starting
or restarting containers show **Starting**, without old RCON errors or player
data. Stopped, unhealthy, or unverifiable containers are not polled. Containers
without a Docker health check use their running state as the readiness signal.

## Fleet overview

Fleet searches server names, map names, and the full player roster. Filter occupied,
empty, unhealthy, testing, or pinned lobbies; switch between cards and a compact
table; and group by game or map. Pins and display preferences are saved in this
browser. Map groups preserve T6 variants and can preselect the exact map when
launching another lobby.

Health summaries exclude stale player counts. The match spotlight uses fresh round
telemetry from an occupied, ongoing match. Persisted campaign, beta, creation,
removal, and restart-recovery state appears alongside affected servers. Fleet
membership and operation progress refresh every ten seconds, with live server
status between refreshes. Failed refreshes keep the last snapshot visible.

## Stack

- SvelteKit (adapter-node) + TypeScript
- SQLite via better-sqlite3 (replaces `script_metadata.json` as the source of
  truth)
- Server-sent events for live pipeline / server-status streams
- RCON (UDP) direct-to-game-server for player counts
- Komodo API for restarts and per-server game-port discovery
- Runs as a single Komodo-managed container on the same host as the game
  servers, with `/root/zombies` bind-mounted in

## Layout

- `src/lib/server/db/` — schema, migrations, better-sqlite3 handle
- `src/lib/server/fleet/` — file walking, push/beta engines
- `src/lib/server/integrations/` — Komodo, RCON
- `src/lib/server/pipeline/` — sweep loop, event bus, worker singleton
- `src/routes/api/` — REST + SSE endpoints
- `src/routes/` — UI pages (dashboard, campaigns, server deployments, betas, per-server)
- `Dockerfile`, `docker-compose.yml`, `entrypoint.sh` — container build
- `komodo/` — Komodo stack definition
- `scripts/` — legacy JSON importer, fixture seeder
- `tests/` — vitest unit tests

## Plugin releases

Plugins support named release versions, GitHub DLL imports and update discovery.
See [Plugin releases](docs/plugin-releases.md) for the import and beta workflow.

## Plutonium updates and scheduled restarts

Open **Settings → Plutonium updates & restarts** to check the latest published
revision, inspect each server's running revision, and queue a server or fleet
restart. Release and running-version checks also run hourly. The published
revision comes from the [official production updater manifest](https://cdn.plutoniummod.com/updater/prod/info.json),
which exposes the game revision directly and avoids the changelog website's browser challenge;
running revisions come from RCON `version`. Unsupported replies and failed
queries appear as unknown rather than being treated as up to date. A failed
manifest request retains the last release with a visible error.

Daily fleet restarts are optional and disabled by default. Choose a queue time
in your **selected timezone** and save the schedule. The timezone is saved with
the schedule so the daily time follows daylight saving changes even when your
browser is closed. Skipped times run at the first valid minute after the gap;
repeated times run once on their first occurrence. Existing UTC schedules retain
their original timing until saved again. The form displays their next occurrence
in your timezone and identifies the saved zone if it differs. Changing a browser's
display preference does not silently change a fleet schedule; save to apply it.
Every due run queues the current fleet, regardless of whether an update exists.
The schedule and jobs survive app restarts. Missed days become one run, and a
server can have only one pending maintenance restart. Disabling the schedule
does not cancel work already queued; use **Cancel** for those jobs.

Maintenance waits for a fresh zero-player RCON response and for existing
deployment/beta work, then uses the shared rolling restart limit. Unreachable or
occupied servers remain queued. After container and RCON recovery, Electric
Cherry queries the running revision again. If a fresh published revision was
known when queued, recovery on an older or unknown revision is reported as a
failed verification, without an automatic restart loop. Ambiguous restart
failures likewise require inspection before a new request. Activity is retained
for 30 days, with pending jobs and the latest 100 records shown in Settings.

Updates rely on Plutainer's startup updater (`PLUTAINER_AUTO_UPDATE` must be
enabled); Electric Cherry does not download binaries or pull a container image.
See [Plutainer's startup flow](https://github.com/Ayymoss/Plutainer/blob/main/CLAUDE.md)
and [environment migration reference](https://github.com/Ayymoss/Plutainer/blob/main/MIGRATION.md).
The worker starts on the first application request, so keep the app running and
reachable for scheduled maintenance. Validate RCON revision replies and updater
behavior on your live T5/T6 servers before relying on unattended maintenance.

## How restarts stay safe

Betas support configurable gameplay, participation, stability and manual-review
requirements, saved presets, coverage rules and staged waves. See
[Beta test plans](docs/beta-test-plans.md) for usage and the four-event game telemetry contract.

File changes are staged sequentially. Their activation uses rolling restarts:
by default only **one server across the fleet** can restart or recover at a
time. Promotions, rollbacks, drift reconciliation, and beta restarts share this
limit. Set `DEPLOYMENT_RESTART_CONCURRENCY` to a positive integer to raise it.

A background sweep runs every 15s. It checks recovering servers first, then
admits queued deployments in oldest-first order when there is capacity. Busy
or unreachable servers stay queued without taking a slot, so other empty
servers can proceed. A successful Komodo restart request retains its slot
until the container is healthy and RCON answers. Health checks start no sooner
than 15 seconds after the request returns, avoiding an immediate check of the
old process. An unhealthy server keeps its slot rather than letting failures
spread across the fleet. A timed-out restart also keeps its slot because Komodo
may have accepted the request.

Slots persist in SQLite across app restarts, and overlapping timer/API triggers
share the active run. Beta options to skip health or match gates still work;
skipping a beta gate does not release the fleet's recovery slot early. New-server
creation uses a separate shared startup queue, holding its slot through container,
RCON, and selected-map verification.

No token, no session, no auth freeze. If Komodo or RCON aren't configured,
the sweep marks pending rows `manual` so nothing hangs forever.

## Deployment blueprints and launch previews

Deployments also offers **House Party Protocol**: review the entire fleet and queue
destruction in one action. This includes queued creations and reuses pending
destruction requests. Each server waits for a fresh zero-player reading; occupied
or unreachable servers remain queued. Containers, server files, and registrations
are removed, while saved keys and blueprints are retained. The preview must be
reviewed again if fleet membership changes before confirmation. Servers created
after confirmation are not included. Progress appears in deployment history.

If deletion fails after Komodo removal, the operation detail page offers
**Resume cleanup** to retry local file and database removal without RCON or
repeating Komodo deletion. Requesting destruction again also resumes that cleanup.
Recovery is blocked if a newer operation exists for the same server name.

Deployments includes a blueprint builder for whole server collections: choose
multiple T5/T6 maps and set the desired total lobbies for each. **Conform fleet**
counts existing and queued servers and creates only the shortfall. The preview
shows target/existing/missing counts and IDs, ports, and key requirements for new
servers only. Already-satisfied targets create nothing; excess servers are kept.
Choose **Match blueprint exactly** in the preview to also remove excess lobbies
and maps outside the blueprint. Review the keep/create/remove lists and confirm
removals. Matching servers stay untouched; unstarted excess creations are cancelled,
and running servers wait for a fresh zero-player check before removal. New lobbies
can reserve ports from these removals and wait until they are released. The collection
page tracks both actions. **Fill missing servers** remains the default.
All jobs and capacity reservations are committed together before provisioning;
one collection page tracks each server's independent progress and failures.
New servers start one at a time by default across all collections and individual
launches. A startup slot stays occupied until Komodo and RCON are healthy and
RCON reports the selected map; an unready server holds the queue until it recovers
or is removed. The queue survives application restarts. The collection preview
shows the startup limit, and its progress page shows active startups and each
server's place in the shared queue. Set `DEPLOYMENT_CREATE_CONCURRENCY` to a
positive integer to increase the limit. Each worker pass starts at most the
available number of slots; immediate failures do not drain the remaining queue.
Existing single-server blueprints can be expanded with **Edit**. Single-server
launches and launching again from recorded recipes remain available.
Blueprints save maps and target lobby counts. Every new server uses the latest
promoted scripts, plugins, and configuration when provisioning begins, including
promotions made after preview or while queued. Existing blueprints automatically
follow this policy. Deployment history records the files selected for that launch;
**Launch again** uses the current promoted setup for the same game and map.
The progress page explains waiting conditions, shows elapsed time, and can resume
health verification of an interrupted creation when its original server remains.
See [Deployment blueprints](docs/deployment-blueprints.md) for the workflow,
promotion policy, and API details.

## Saved server keys

Use **Keys** to add, replace, or remove a Plutonium key for an exact
server ID such as `t5_zombie_theater_2`. IDs use a supported map slug and a
lobby number from 1 to 20; the server does not need to be deployed first.
During deployment, choose the map. The lowest available lobby is assigned
automatically, with the exact server ID shown before deployment. Its saved key is reused automatically,
and any newly entered key is saved for that ID, including if deployment fails.
Destroying a server preserves its saved key. Updating or removing a saved key
does not change a running server; changes apply to future deployments.

Keys are stored in the application's SQLite database at `DB_PATH`, independently
of server records. Preserve that database across container replacements. Key
values are not returned by the management API, but are stored unencrypted in
SQLite, so database files and backups contain credentials. Existing deployment
keys are not automatically imported from Komodo; add them once to retain them.

## Local dev

Windows or macOS — no `/root/zombies` on disk, so seed a fixture tree first.

```bash
npm install
npm run seed:fixture

# Point env at the fixture (PowerShell)
$env:SHARED_SCRIPTS = "$PWD/test-fixtures/zombies/shared/scripts"
$env:SERVERS_ROOT   = "$PWD/test-fixtures/zombies/t5"
$env:DB_PATH        = "$PWD/data/cherry.db"

npm run dev
# open http://localhost:3000
```

Without RCON_PASSWORD / KOMODO_* set, the pipeline marks pending rows
`manual` after a sweep — the UI still works, you just don't get real
restarts. Enough to develop the push/beta flow.

## Map selection and configs

The map selected when deploying a T5 zombies server controls its startup
rotation automatically. No map-choice config overlay is required. New servers
use the promoted game baseline and matching overlays when available; otherwise
Plutainer seeds its usual defaults and Electric Cherry applies the selected map
through a generated `configs/electric_cherry_map.cfg`, loaded after the main
config and before map rotation. This file is tracked and restored by drift
reconciliation; operators do not need to create or promote it. The launch order
uses Plutainer's [extra arguments and map-rotation controls](https://github.com/Ayymoss/Plutainer/blob/main/docs/configuration.md).
Plutainer's
default main config is preserved until a game baseline is promoted.

Beta previews, promotion, and config drift repair all apply the server's stored
catalog map after composing the baseline and overlays. An old `sv_maprotation`
overlay cannot override that selection, but its other settings still apply.
Existing servers receive this behavior on their next config deployment or drift
reconciliation; this change does not immediately rewrite or restart them.
Servers with an unknown map retain their existing config behavior. Rollback
restores the exact prior config bytes.

## T6 Zombies (Black Ops II)

Choose **Black Ops II (T6)** in Deploy new server. T6 uses `t6_zombie_<slug>_<lobby>`
IDs, its own saved Plutonium keys, `PLUTAINER_GAME=t6zm`, and
`configs/dedicated_zm.cfg`. Existing T5 deployments and environment settings
keep their current paths. API clients can send `game: "t6"`; omitting it selects T5.

Before the first deployment, populate the Docker volume `t6_game_files` on the
Komodo host with your Black Ops II game files, including the DLC maps you intend
to host. Set `PLUTAINER_T6_GAMEFILES_VOLUME` to use another volume or an absolute
host directory. It is mounted read-only at `/home/plutainer/gamefiles`.
Plutainer supplies its runtime and config seeds, not the base game files.
Use a Plutonium server key created for **T6 Zombies**.

Server state defaults to `/root/zombies/t6`, or the sibling `t6` directory beside
`SERVERS_ROOT`; override with `T6_SERVERS_ROOT`. Electric Cherry and the Komodo
host must see this directory at the same absolute path. The supplied Compose
mount already covers both games. GSC files go to
`runtime/plutonium/storage/t6/scripts/zm` (no `raw/`), and game logs use
`logs/games_zm.log`. Optional overrides are `T6_REL_DEST_SCRIPT` and
`T6_CONFIG_TARGET_FILE`; a custom config filename needs a promoted baseline or
a file supplied at that name, since Plutainer seeds `dedicated_zm.cfg`.

The catalog includes the classic maps, Survival, Grief, and Turned selections.
Rotations use `execgts <ruleset>.cfg map <engine-map>`; Plutainer seeds the required
`storage/t6/gamesettings` files. Keep seeding enabled, including with a promoted
baseline. Town, Farm, and Bus Depot share `zm_transit`, so EC keeps their selection
in `map_slug` through creation, beta, promotion, and drift repair. Map-scoped
artifacts target the engine map and therefore apply to all its variants.
Creation health checks verify the engine map reported by RCON; this does not
independently verify the start location within that map.

Both games share the fleet's port allocator (`SERVER_PORT_MIN`/`SERVER_PORT_MAX`)
to prevent collisions. EC explicitly sets `PLUTAINER_PORT`, so T6 need not use
its upstream default of 4976. Forward the allocated UDP ports. Use a current
Plutainer v2 image: EC relies on `PLUTAINER_RCON_PASSWORD`,
`PLUTAINER_MAP_ROTATE=false`, and extra arguments loaded after the main config.
Current Plutainer automatically whitelists the Docker gateway for T5/T6 RCON;
an admin client on another host needs its address allowed in the server config.

Implementation references (checked September 2026):

- [Plutainer game requirements](https://github.com/Ayymoss/Plutainer/blob/main/docs/games.md)
- [Plutainer launch, seeding, and RCON implementation](https://github.com/Ayymoss/Plutainer/blob/main/scripts/lib/cod.sh)
- [Plutainer configuration](https://github.com/Ayymoss/Plutainer/blob/main/docs/configuration.md)
- [T6 map/ruleset combinations](https://github.com/xerxes-at/T6ServerConfigs/blob/master/localappdata/Plutonium/storage/t6/dedicated_zm.cfg)
- [Plutonium GSC paths](https://plutonium.pw/docs/modding/loading-mods/)

## Game telemetry

The server script sends JSON to `POST /api/telemetry`:

- `game_start`: server, map, and all players (name and GUID).
- `round_change`: server, map, round, and all players with kills, headshots, downs, revives, and score.
- `heartbeat`: the same fields as round change, sent periodically during a round to refresh player stats.
- `game_end`: the same fields as round change, with final stats.

These reports work with or without an active beta. Electric Cherry records
the full player data, updates Fleet, and applies gameplay progress to active
betas automatically. No discovery request or beta-specific fields are needed.

See [Game reporter protocol](docs/beta-test-plans.md#game-reporter-protocol) for
complete payload examples. Telemetry does not replace RCON's empty-server
check before an automated restart.

For a missing scoreboard, open the server's **Diagnostics → Telemetry API requests**.
The log shows received requests for that exact server name, including accepted
reports, rejected payloads, HTTP status, errors, and expandable submitted JSON.
It refreshes every five seconds while visible and survives app restarts. Logging
starts with this version; older requests cannot be reconstructed. See
[Telemetry diagnostics](docs/beta-test-plans.md#telemetry-diagnostics) for retention
and how to interpret missing or rejected reports.

The **Leaderboard** records persistent player totals and map records from these
same reports, including highest round, cumulative kills, rounds survived,
headshots, revives, downs, and games played. See [Leaderboard](docs/leaderboard.md)
for counting rules, history import, and the read API.

## Running tests

```bash
npm test
```

Covers push (add/update/skip), beta lifecycle (open → rollback, open →
promote, conflict blocks), sweep (integrations-off manual branch), and
RCON parsing (fake UDP socket).

## Deploy to the Komodo host

1. Build the image once locally or on the host:

   ```bash
   docker compose build
   ```

2. Import `komodo/stack.toml` into Komodo (or point Komodo at the
   `docker-compose.yml` at repo root).

3. Set the env vars in Komodo:

   ```
   KOMODO_ADDRESS, KOMODO_API_KEY, KOMODO_API_SECRET
   RCON_PASSWORD              # single shared password across every T5/T6 server
   RCON_HOST                  # optional; defaults to host.docker.internal
   ```

4. First boot triggers the legacy import: if
   `/root/zombies/script_metadata.json` exists and `/data/cherry.db` is
   empty, `entrypoint.sh` runs `scripts/import-legacy.ts` before starting
   the server. The source JSON gets renamed to `.migrated`.

5. Load the app on the LAN or over tailscale (no web-app auth by design).

## Campaigns and server deployments

Campaigns (`/campaigns`) tracks script/config updates, drift reconciliation, restart progress, file diffs, and rollbacks. Existing campaign records are preserved in the original database tables. The campaign API lives at `/api/campaigns`; legacy detail API endpoints remain available for compatibility.

Deployments (`/deployments`, `/api/deployments`) now tracks server creations and destructions with live stages and persistent history, including failures, warnings, and interrupted operations. History starts with operations performed after this update and survives server removal. New creation requests return HTTP 202 with a deployment ID and open the deployment detail page. Creation is complete only when Komodo reports healthy, RCON responds successfully, and the RCON map exactly matches the selected map. Queued requests reserve their server name and port; queued creation and pending health checks survive application restarts. The detail page records startup progress and explains any health or map mismatch that is still blocking completion.

Each server deployment opens a detail page with its flow and timestamped activity. Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh RCON player count of zero before teardown; occupied or unreachable servers remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the deployment detail page.