electric-cherry/README.md

# Electric Cherry

Electric Cherry manages T5 and T6 Plutonium Zombies servers through its own host
agents. Docker and Plutainer run the games; EC handles hosts, game files,
collections, configuration, campaigns, maintenance and operation history.

## Getting started

Run the controller with Docker Compose behind HTTPS:

```sh
cp .env.example .env
# Set EC_PUBLIC_URL, generate EC_SETUP_TOKEN, and retain EC_DATA_DIR.
docker compose up -d --build
```

Open EC and create the first administrator using `EC_SETUP_TOKEN` (at least 32
characters; generate it with `openssl rand -hex 32`). After setup, remove that
token from the environment. Administrators can add administrators, operators and
viewers from **Administration → Users**. See [Users and access](docs/authentication.md)
for permissions, sessions, password recovery and upgrade instructions.

Operator pages and management APIs require login. Enrolled agents authenticate
independently and connect outbound over HTTPS; the
controller needs no Docker socket, game-server mounts or inbound RCON access.
The default Compose data mount preserves the previous
`/root/zombies/.electric-cherry` database location. For a new installation, set
`EC_DATA_DIR` to your preferred persistent directory. Back up that directory,
including SQLite metadata and published game-file storage.

1. **Library → Game Files**: upload the complete local folder under T5 or T6,
   including its DLC files. There are no library names or version labels to enter.
   Each upload becomes the latest files for that game after verification.
   Uploads work before any host is enrolled.
2. **Fleet → Hosts → Add Host**: configure a Debian 12/13 amd64 machine and run its
   generated enrollment command. The host automatically prepares its Plutainer
   runtime and the latest T5 and T6 files, when available. The host page
   shows setup progress. Automatic agent updates are enabled by default.
3. Add the exact Plutonium server keys in **Administration → Keys**, or supply missing keys when
   launching.
4. Launch a server or collection. Collection previews show cached capacity
   estimates; Apply saves the queue and opens its progress page. Provisioning
   checks capacity and assigns each server a host and port when it can start.
   The latest game files and default runtime are selected automatically.
   CPU/memory allowances learn
   from actual container usage, using temporary defaults while calibrating.

See [Native host management](docs/native-hosts.md) for installation, credentials,
storage, failure recovery, API contracts and the live-host acceptance checklist.

**Capacity** shows available CPU, memory, ports, estimated additional servers,
and the shared startup queue. Filter by host or game, inspect each host's limiting
resource, and explore up to seven days of resource headroom.

Existing hosts need the current installer once to enable automatic updates.
Generate it under **Agent access** and rerun it on the same host without
`--reenroll`; identity and game data are preserved. Thereafter hosts check their
controller about every five minutes, verify compatible releases and update agent
code between operations. Failed updates roll back; running game containers
continue. The host page shows update status and the last check.

## Fleet and operations

The sidebar groups pages into **Fleet**, **Library**, **Operations**, **Analytics**,
and **Administration** (administrators only). On small screens, open the navigation
menu in the header. The account menu contains **My account**, including your
browser's display timezone; keyboard shortcuts are available through **?**.

**Operations → Overview** combines work that needs attention, running workflows,
and recent results. Related provisioning attempts and host commands stay grouped
under their workflow. Normal waits for empty lobbies remain under **Running**;
successful routine checks remain in **Host operation history**. Full history is
available on each workflow page.

Server details use **Live**, **Matches**, **Configuration**, **Diagnostics**, and
**History** sections. Hosts use **Overview**, **Servers**, **Setup**, and **Settings**.
Contextual Back links preserve the originating filters and views, while breadcrumbs
show each page's place in the navigation. Existing URLs and server tab links work.

The fleet registry lives in SQLite, independently of host filesystems. Status
combines agent container health, fresh game telemetry and agent-local RCON.
Unreachable hosts report unknown instance state; they do not trigger duplicate
replacements. Running containers continue through controller or agent outages.

Provisioning owns Create, Start, Stop, Restart, Rebuild and Destroy. Commands and
results have durable operation IDs and immutable instance generations. Queues,
capacity reservations and history survive controller restarts. **Rebuild removes
per-instance writable files** and creates a new generation. Shared immutable game
libraries survive rebuilds and destruction.

Disruptive actions wait for a fresh empty lobby or verified stopped container.
The agent rechecks occupancy locally immediately before activation and disruption.
Long player sessions can continue indefinitely. **Drain** prevents new launches
and safely retires existing servers as they become eligible.

Collections save target maps and lobby counts. Fill missing servers adds only the
shortfall; Match collection exactly also queues safe removal of excess servers.
Launch previews use the latest verified files for each game and select an
independent runtime image. If newer game files become available before launch,
refresh the preview to confirm the updated files. Every new launch uses the
latest promoted configuration, scripts and plugins when execution starts. Saved
Plutonium keys are reused by exact server ID and retained after removal.

## Libraries and campaigns

Game Files has one upload slot each for T5 and T6. Uploads preserve the folder's
directory layout and included DLC; there is no separate DLC declaration.
EC records immutable upload IDs, file manifests, checksums and sizes automatically.
Uploads use bounded chunks, support pause/resume and verify checksums before
publication. Actual game files
use dedicated filesystem storage, never SQLite blobs. The last upload to finish
verification becomes that game's latest files, even if it started earlier.
Active hosts always prepare the latest T5 and T6 files when available, including
hosts enrolled later. Hosts first download all files with up to four concurrent
transfers, then verify all checksums in a separate pass before the files become
usable. Both steps show progress, elapsed time, measured speed and an estimated
time remaining for the current step. Interrupted transfers resume. Failed preparation
retries automatically with backoff; the host page reports any blocking condition.
Set `PLUTAINER_IMAGE` to change the default runtime prepared on hosts. Existing
servers retain their existing files and runtime without a forced restart.
Previous uploads, verified host copies and server references are preserved.
Existing named libraries appear together under their game, with no renaming or
reupload required. Multiple instances on one host share read-only base files.

Scripts, plugins, configuration baselines and overlays retain their versioning,
beta, promotion and rollback workflows. Campaigns stage bytes remotely without
stopping games; activation uses the same occupancy and concurrency gates as
lifecycle operations. Agent before images preserve exact rollback bytes and
operation history.

T5 Zombies uses `t5sp` and `dedicated_sp.cfg`; T6 uses `t6zm` and
`dedicated_zm.cfg`. Map selection generates `electric_cherry_map.cfg`, loaded
after the main config. T6 variants retain their distinct catalog selections.
Game-file uploads and Plutonium runtime revisions are tracked independently.

Game telemetry uses its own `EC_TELEMETRY_TOKEN`; configure reporters to send it
in an authorization header. Anonymous reports are rejected before they can
affect occupancy or game history. Existing JSON payloads are unchanged.

## Documentation

- [Native hosts, storage and operation recovery](docs/native-hosts.md)
- [Users, roles and authentication](docs/authentication.md)
- [Provisioning and safety queues](docs/provisioning.md)
- [Collections and promoted launch recipes](docs/deployment-blueprints.md)
- [Maintenance](docs/maintenance.md)
- [Beta test plans and game telemetry](docs/beta-test-plans.md)
- [Plugin releases](docs/plugin-releases.md)
- [Metrics](docs/metrics.md)
- [Leaderboard and match history](docs/leaderboard.md)

## Development

The controller is SvelteKit/TypeScript with SQLite via better-sqlite3, the Node
adapter and server-sent events. The host agent is Python 3 standard-library code
installed as a systemd service. Hosts do not install Node.js or build EC.

```sh
npm install
npm run agent:package
# Export EC_SETUP_TOKEN in your shell using the secret saved for /setup.
EC_AUTH_ALLOW_HTTP=true npm run dev

npm run check
npm test
npm run test:agent
npm run build
```

Builds regenerate the downloadable agent, standalone updater and checksum manifest,
and bundle the metrics/leaderboard workers under `build/reports` and native launch
preparation under `build/launch`. `npm run dev` and `npm test` compile those workers
automatically. After editing a report or launch-preparation implementation during
development, restart `npm run dev` to rebuild its worker.
Reports use separate read-only SQLite connections and bounded revision-aware caches;
metrics snapshots can be reused for up to 30 seconds when their inputs are unchanged.

Set `EC_PERF_LOG=true` to log slow request timings, report computation/cache timings,
and event-loop delay. Responses include `Server-Timing`, and browser navigations
record an `ec:navigation` performance measure. `npm run benchmark:leaderboard`
uses an isolated synthetic database; pass `-- --scenario all --baseline-ref <commit>`
to compare full response parity and warm query timings with an earlier revision.

Local controller
metadata defaults to `./data/cherry.db`; use a separate test database when
exercising enrollment. Production agent transport requires HTTPS.

`src/lib/server/hosts` owns enrollment, readiness and command transport;
`src/lib/server/backends` provides instance identity and remote file access;
`src/lib/server/provisioning/native.ts` connects existing lifecycle and campaign
queues to agents; `src/lib/server/game-files` owns immutable game-file uploads and
streaming distribution; `agent/` contains the service, installer and agent tests.

Automated tests use simulated Docker/game hosts. Real Debian installer and
T5/T6 game-server validation remain deployment acceptance work; see the checklist
in the native-host guide.