electric-cherry/README.md

# Electric Cherry

Web app that manages a fleet of T5 Plutonium servers running under Komodo.

## Stack

- SvelteKit (adapter-node) + TypeScript
- SQLite via better-sqlite3 (replaces `script_metadata.json` as the source of
  truth)
- Server-sent events for live pipeline / server-status streams
- RCON (UDP) direct-to-game-server for player counts
- Komodo API for restarts and per-server game-port discovery
- Runs as a single Komodo-managed container on the same host as the game
  servers, with `/root/zombies` bind-mounted in

## Layout

- `src/lib/server/db/` — schema, migrations, better-sqlite3 handle
- `src/lib/server/fleet/` — file walking, push/beta engines
- `src/lib/server/integrations/` — Komodo, RCON
- `src/lib/server/pipeline/` — sweep loop, event bus, worker singleton
- `src/routes/api/` — REST + SSE endpoints
- `src/routes/` — UI pages (dashboard, campaigns, server deployments, betas, per-server)
- `Dockerfile`, `docker-compose.yml`, `entrypoint.sh` — container build
- `komodo/` — Komodo stack definition
- `scripts/` — legacy JSON importer, fixture seeder
- `tests/` — vitest unit tests

## How restarts stay safe

File changes are staged sequentially. Their activation uses rolling restarts:
by default only **one server across the fleet** can restart or recover at a
time. Promotions, rollbacks, drift reconciliation, and beta restarts share this
limit. Set `DEPLOYMENT_RESTART_CONCURRENCY` to a positive integer to raise it.

A background sweep runs every 15s. It checks recovering servers first, then
admits queued deployments in oldest-first order when there is capacity. Busy
or unreachable servers stay queued without taking a slot, so other empty
servers can proceed. A successful Komodo restart request retains its slot
until the container is healthy and RCON answers. Health checks start no sooner
than 15 seconds after the request returns, avoiding an immediate check of the
old process. An unhealthy server keeps its slot rather than letting failures
spread across the fleet. A timed-out restart also keeps its slot because Komodo
may have accepted the request.

Slots persist in SQLite across app restarts, and overlapping timer/API triggers
share the active run. Beta options to skip health or match gates still work;
skipping a beta gate does not release the fleet's recovery slot early. New-server
creation is separate from deployment restarts and still starts its container
directly.

No token, no session, no auth freeze. If Komodo or RCON aren't configured,
the sweep marks pending rows `manual` so nothing hangs forever.

## Saved server keys

Use **Server keys** to add, replace, or remove a Plutonium key for an exact
server ID such as `t5_zombie_theater_2`. IDs use a supported map slug and a
lobby number from 1 to 20; the server does not need to be deployed first.
During deployment, choose the map. The lowest available lobby is assigned
automatically, with the exact server ID shown before deployment. Its saved key is reused automatically,
and any newly entered key is saved for that ID, including if deployment fails.
Destroying a server preserves its saved key. Updating or removing a saved key
does not change a running server; changes apply to future deployments.

Keys are stored in the application's SQLite database at `DB_PATH`, independently
of server records. Preserve that database across container replacements. Key
values are not returned by the management API, but are stored unencrypted in
SQLite, so database files and backups contain credentials. Existing deployment
keys are not automatically imported from Komodo; add them once to retain them.

## Local dev

Windows or macOS — no `/root/zombies` on disk, so seed a fixture tree first.

```bash
npm install
npm run seed:fixture

# Point env at the fixture (PowerShell)
$env:SHARED_SCRIPTS = "$PWD/test-fixtures/zombies/shared/scripts"
$env:SERVERS_ROOT   = "$PWD/test-fixtures/zombies/t5"
$env:DB_PATH        = "$PWD/data/cherry.db"

npm run dev
# open http://localhost:3000
```

Without RCON_PASSWORD / KOMODO_* set, the pipeline marks pending rows
`manual` after a sweep — the UI still works, you just don't get real
restarts. Enough to develop the push/beta flow.

## Map selection and configs

The map selected when deploying a T5 zombies server controls its startup
rotation automatically. No map-choice config overlay is required. New servers
use the promoted game baseline and matching overlays when available; otherwise
Plutainer seeds its usual defaults and Electric Cherry applies the selected map
through a generated `configs/electric_cherry_map.cfg`, loaded after the main
config and before map rotation. This file is tracked and restored by drift
reconciliation; operators do not need to create or promote it. The launch order
uses Plutainer's [extra arguments and map-rotation controls](https://github.com/Ayymoss/Plutainer/blob/main/docs/configuration.md).
Plutainer's
default main config is preserved until a game baseline is promoted.

Beta previews, promotion, and config drift repair all apply the server's stored
catalog map after composing the baseline and overlays. An old `sv_maprotation`
overlay cannot override that selection, but its other settings still apply.
Existing servers receive this behavior on their next config deployment or drift
reconciliation; this change does not immediately rewrite or restart them.
Servers with an unknown map retain their existing config behavior. Rollback
restores the exact prior config bytes.

## Round telemetry experiment

Send `POST /api/telemetry` with `Content-Type: application/json` to the Electric
Cherry host (port 3000 in local development):

```bash
curl http://localhost:3000/api/telemetry \
  -H 'Content-Type: application/json' \
  -d '{"event":"round_change","round":17,"server":"t5_zombie_prototype_1","players":4}'
```

The server must already be registered in Electric Cherry. `round` and `players`
are required nonnegative integers. Success returns HTTP 200 with
`{ "ok": true, "telemetry": { "server": "...", "round": 17, "players": 4, "receivedAt": "..." } }`.
Invalid payloads return 400, unknown/destroyed servers 404, and non-JSON content
types 415. This endpoint uses the app's existing LAN/Tailscale access model;
there is no API key requirement.

The latest report is saved across app restarts and streamed immediately to the
dashboard and server detail page, including after browser reconnection. The UI
labels it as the last reported round with its age: round-change events alone
cannot detect a stopped script or a match ending. Send the initial round on each
new match, then each round change. The latest received request wins (including
lower rounds for new matches); there is no event ordering or history yet.
Telemetry player counts are displayed with the report and never replace RCON
occupancy or affect automated restart gates.

## Running tests

```bash
npm test
```

Covers push (add/update/skip), beta lifecycle (open → rollback, open →
promote, conflict blocks), sweep (integrations-off manual branch), and
RCON parsing (fake UDP socket).

## Deploy to the Komodo host

1. Build the image once locally or on the host:

   ```bash
   docker compose build
   ```

2. Import `komodo/stack.toml` into Komodo (or point Komodo at the
   `docker-compose.yml` at repo root).

3. Set the env vars in Komodo:

   ```
   KOMODO_ADDRESS, KOMODO_API_KEY, KOMODO_API_SECRET
   RCON_PASSWORD              # single shared password across every T5 server
   RCON_HOST                  # optional; defaults to host.docker.internal
   ```

4. First boot triggers the legacy import: if
   `/root/zombies/script_metadata.json` exists and `/data/cherry.db` is
   empty, `entrypoint.sh` runs `scripts/import-legacy.ts` before starting
   the server. The source JSON gets renamed to `.migrated`.

5. Load the app on the LAN or over tailscale (no web-app auth by design).

## Campaigns and server deployments

Campaigns (`/campaigns`) contains script/config rollouts, drift reconciliation, restart progress, file diffs, and rollbacks. Existing rollout records are preserved in the original database tables. The campaign API lives at `/api/campaigns`; legacy rollout detail API endpoints remain available for compatibility.

Deployments (`/deployments`, `/api/deployments`) now tracks server creations and destructions with live stages and persistent history, including failures, warnings, and interrupted operations. History starts with operations performed after this update and survives server removal. New creation requests return HTTP 202 with a deployment ID and open the deployment detail page. Creation is complete only when Komodo reports healthy, RCON responds successfully, and the RCON map exactly matches the selected map. Queued requests reserve their server name and port; queued creation and pending health checks survive application restarts. The detail page records startup progress and explains any health or map mismatch that is still blocking completion.

Each server deployment opens a detail page with its flow and timestamped activity. Destroy uses an in-app confirmation modal and immediately returns a queued deployment (`DELETE /api/servers/:name` returns HTTP 202 with `deploymentId`). The worker waits for a fresh RCON player count of zero before teardown; occupied or unreachable servers remain queued and are checked each worker tick. Waiting requests survive application restarts, while interrupted teardown is marked for inspection. Completed destruction stays on the deployment detail page.