electric-cherry/README.md

# electric-cherry

Web app that manages a fleet of T5 Plutonium servers running under Komodo.
Successor to `push_scripts.sh` — same job (push shared GSC scripts, run beta
tests, restart empty servers, track campaigns) with a live UI and real
audit diffs.

## Stack

- SvelteKit (adapter-node) + TypeScript
- SQLite via better-sqlite3 (replaces `script_metadata.json` as the source of
  truth)
- Server-sent events for live pipeline / server-status streams
- RCON (UDP) direct-to-game-server for player counts
- Komodo API for restarts and per-server game-port discovery
- Runs as a single Komodo-managed container on the same host as the game
  servers, with `/root/zombies` bind-mounted in

## Layout

- `src/lib/server/db/` — schema, migrations, better-sqlite3 handle
- `src/lib/server/fleet/` — file walking, push/beta engines
- `src/lib/server/integrations/` — Komodo, RCON
- `src/lib/server/pipeline/` — sweep loop, event bus, worker singleton
- `src/routes/api/` — REST + SSE endpoints
- `src/routes/` — UI pages (dashboard, campaigns, server deployments, betas, per-server)
- `Dockerfile`, `docker-compose.yml`, `entrypoint.sh` — container build
- `komodo/` — Komodo stack definition
- `scripts/` — legacy JSON importer, fixture seeder
- `tests/` — vitest unit tests

## How restarts stay safe

File changes are staged sequentially. Their activation uses rolling restarts:
by default only **one server across the fleet** can restart or recover at a
time. Promotions, rollbacks, drift reconciliation, and beta restarts share this
limit. Set `DEPLOYMENT_RESTART_CONCURRENCY` to a positive integer to raise it.

A background sweep runs every 15s. It checks recovering servers first, then
admits queued deployments in oldest-first order when there is capacity. Busy
or unreachable servers stay queued without taking a slot, so other empty
servers can proceed. A successful Komodo restart request retains its slot
until the container is healthy and RCON answers. Health checks start no sooner
than 15 seconds after the request returns, avoiding an immediate check of the
old process. An unhealthy server keeps its slot rather than letting failures
spread across the fleet. A timed-out restart also keeps its slot because Komodo
may have accepted the request.

Slots persist in SQLite across app restarts, and overlapping timer/API triggers
share the active run. Beta options to skip health or match gates still work;
skipping a beta gate does not release the fleet's recovery slot early. New-server
creation is separate from deployment restarts and still starts its container
directly.

No token, no session, no auth freeze. If Komodo or RCON aren't configured,
the sweep marks pending rows `manual` so nothing hangs forever.

## Saved server keys

Use **Server keys** to add, replace, or remove a Plutonium key for an exact
server ID such as `t5_zombie_theater_2`. IDs use a supported map slug and a
lobby number from 1 to 20; the server does not need to be deployed first.
During deployment, choose the map. The lowest available lobby is assigned
automatically, with the exact server ID shown before deployment. Its saved key is reused automatically,
and any newly entered key is saved for that ID, including if deployment fails.
Destroying a server preserves its saved key. Updating or removing a saved key
does not change a running server; changes apply to future deployments.

Keys are stored in the application's SQLite database at `DB_PATH`, independently
of server records. Preserve that database across container replacements. Key
values are not returned by the management API, but are stored unencrypted in
SQLite, so database files and backups contain credentials. Existing deployment
keys are not automatically imported from Komodo; add them once to retain them.

## Local dev

Windows or macOS — no `/root/zombies` on disk, so seed a fixture tree first.

```bash
npm install
npm run seed:fixture

# Point env at the fixture (PowerShell)
$env:SHARED_SCRIPTS = "$PWD/test-fixtures/zombies/shared/scripts"
$env:SERVERS_ROOT   = "$PWD/test-fixtures/zombies/t5"
$env:DB_PATH        = "$PWD/data/cherry.db"

npm run dev
# open http://localhost:3000
```

Without RCON_PASSWORD / KOMODO_* set, the pipeline marks pending rows
`manual` after a sweep — the UI still works, you just don't get real
restarts. Enough to develop the push/beta flow.

## Map selection and configs

The map selected when deploying a T5 zombies server controls its startup
rotation automatically. No map-choice config overlay is required. New servers
use the promoted game baseline and matching overlays when available; otherwise
Plutainer seeds its usual defaults and Electric Cherry applies the selected map
through a generated `configs/electric_cherry_map.cfg`, loaded after the main
config and before map rotation. This file is tracked and restored by drift
reconciliation; operators do not need to create or promote it. The launch order
uses Plutainer's [extra arguments and map-rotation controls](https://github.com/Ayymoss/Plutainer/blob/main/docs/configuration.md).
Plutainer's
default main config is preserved until a game baseline is promoted.

Beta previews, promotion, and config drift repair all apply the server's stored
catalog map after composing the baseline and overlays. An old `sv_maprotation`
overlay cannot override that selection, but its other settings still apply.
Existing servers receive this behavior on their next config deployment or drift
reconciliation; this change does not immediately rewrite or restart them.
Servers with an unknown map retain their existing config behavior. Rollback
restores the exact prior config bytes.

## Round telemetry experiment

Send `POST /api/telemetry` with `Content-Type: application/json` to the Electric
Cherry host (port 3000 in local development):

```bash
curl http://localhost:3000/api/telemetry \
  -H 'Content-Type: application/json' \
  -d '{"event":"round_change","round":17,"server":"t5_zombie_prototype_1","players":4}'
```

The server must already be registered in Electric Cherry. `round` and `players`
are required nonnegative integers. Success returns HTTP 200 with
`{ "ok": true, "telemetry": { "server": "...", "round": 17, "players": 4, "receivedAt": "..." } }`.
Invalid payloads return 400, unknown/destroyed servers 404, and non-JSON content
types 415. This endpoint uses the app's existing LAN/Tailscale access model;
there is no API key requirement.

The latest report is saved across app restarts and streamed immediately to the
dashboard and server detail page, including after browser reconnection. The UI
labels it as the last reported round with its age: round-change events alone
cannot detect a stopped script or a match ending. Send the initial round on each
new match, then each round change. The latest received request wins (including
lower rounds for new matches); there is no event ordering or history yet.
Telemetry player counts are displayed with the report and never replace RCON
occupancy or affect automated restart gates.

## Running tests

```bash
npm test
```

Covers push (add/update/skip), beta lifecycle (open → rollback, open →
promote, conflict blocks), sweep (integrations-off manual branch), and
RCON parsing (fake UDP socket).

## Deploy to the Komodo host

1. Build the image once locally or on the host:

   ```bash
   docker compose build
   ```

2. Import `komodo/stack.toml` into Komodo (or point Komodo at the
   `docker-compose.yml` at repo root).

3. Set the env vars in Komodo:

   ```
   KOMODO_ADDRESS, KOMODO_API_KEY, KOMODO_API_SECRET
   RCON_PASSWORD              # single shared password across every T5 server
   RCON_HOST                  # optional; defaults to host.docker.internal
   ```

4. First boot triggers the legacy import: if
   `/root/zombies/script_metadata.json` exists and `/data/cherry.db` is
   empty, `entrypoint.sh` runs `scripts/import-legacy.ts` before starting
   the server. The source JSON gets renamed to `.migrated`.

5. Load the app on the LAN or over tailscale (no web-app auth by design).

## What still lives in the shell script

Nothing that's needed at runtime. Move `push_scripts.sh` to `archive/` once
you've run a side-by-side parity check between the shell's dry-run and the
app's `POST /api/push { dryRun: true }`.

## Non-goals

- **Editing GSC files in-browser.** Do that in your editor of choice; the
  web app only cares about what's on disk in `SHARED_SCRIPTS`.
- **Multi-user auth.** LAN/tailscale scoping is the security model.
- **Two-way sync with `script_metadata.json`.** The importer runs once. If
  the shell script is ever used again after that, it'll write to the JSON
  file and the web app won't see it.

## References into the old script

Every port carries a `// Ports push_scripts.sh:NN` comment pointing back to
the original for the parts that had non-obvious behavior. Grep those if a
behavior seems wrong — the shell script is the reference implementation
until the parity check passes.

## Campaigns and server deployments

Campaigns (`/campaigns`) contains script/config rollouts, drift reconciliation, restart progress, file diffs, and rollbacks. Existing rollout records are preserved in the original database tables. The campaign API lives at `/api/campaigns`; legacy rollout detail API endpoints remain available for compatibility.

Deployments (`/deployments`, `/api/deployments`) now tracks server creations and destructions with live stages and persistent history, including failures, warnings, and interrupted operations. History starts with operations performed after this update and survives server removal. Creation completion means the server was provisioned and Komodo accepted the container start request; runtime health remains on the Fleet page.