electric-cherry/README.md

# Electric Cherry

Electric Cherry manages T5 and T6 Plutonium Zombies servers through its own host
agents. Docker and Plutainer run the games; EC handles hosts, game files,
collections, configuration, campaigns, maintenance and operation history.

## Getting started

Run the controller with Docker Compose behind HTTPS:

```sh
cp .env.example .env
# Set EC_PUBLIC_URL, generate EC_SETUP_TOKEN, and retain EC_DATA_DIR.
docker compose up -d --build
```

Open EC and create the first administrator using `EC_SETUP_TOKEN` (at least 32
characters; generate it with `openssl rand -hex 32`). After setup, remove that
token from the environment. Administrators can add administrators, operators and
viewers from **Administration → Users**. See [Users and access](docs/authentication.md)
for permissions, sessions, password recovery and upgrade instructions.

Operator pages and management APIs require login. Enrolled agents authenticate
independently and connect outbound over HTTPS; the
controller needs no Docker socket, game-server mounts or inbound RCON access.
The default Compose data mount preserves the previous
`/root/zombies/.electric-cherry` database location. For a new installation, set
`EC_DATA_DIR` to your preferred persistent directory. Back up that directory,
including SQLite metadata and published game-file storage.

1. **Library → Game Files**: upload the complete local folder under T5 or T6,
   including its DLC files. There are no library names or version labels to enter.
   Each upload becomes the latest files for that game after verification.
   Uploads work before any host is enrolled.
2. **Fleet → Hosts → Add Host**: configure a Debian 12/13 amd64 machine and run its
   generated enrollment command. The host automatically prepares its Plutainer
   runtime and the latest T5 and T6 files, when available. The host page
   shows setup progress. Automatic agent updates are enabled by default.
3. Add the exact Plutonium server keys in **Administration → Keys**, or supply missing keys when
   launching.
4. Launch a server or collection. Collection previews show cached capacity
   estimates; Apply saves the queue and opens its progress page. Provisioning
   checks capacity and assigns each server a host and port when it can start.
   The latest game files and default runtime are selected automatically.
   CPU/memory allowances learn
   from actual container usage, using temporary defaults while calibrating.

See [Native host management](docs/native-hosts.md) for installation, credentials,
storage, failure recovery, API contracts and the live-host acceptance checklist.

**Capacity** shows available CPU, memory, ports, estimated additional servers,
and the shared startup queue. Filter by host or game, inspect each host's limiting
resource, and explore up to seven days of resource headroom.

Existing hosts need the current installer once to enable automatic updates.
Generate it under **Agent access** and rerun it on the same host without
`--reenroll`; identity and game data are preserved. Thereafter hosts check their
controller about every five minutes, verify compatible releases and update agent
code between operations. Failed updates roll back; running game containers
continue. The host page shows update status and the last check.

## Fleet and operations

The sidebar groups pages into **Fleet**, **Library**, **Operations**, **Analytics**,
and **Administration** (administrators only). On small screens, open the navigation
menu in the header. The account menu contains **My account**, including your
browser's display timezone; keyboard shortcuts are available through **?**.

**Operations → Overview** combines work that needs attention, running workflows,
and recent results. Related provisioning attempts and host commands stay grouped
under their workflow. Normal waits for empty lobbies remain under **Running**;
successful routine checks remain in **Host operation history**. Full history is
available on each workflow page.

Server details use **Live**, **Matches**, **Configuration**, **Diagnostics**, and
**History** sections. Hosts use **Overview**, **Servers**, **Setup**, and **Settings**.
Contextual Back links preserve the originating filters and views, while breadcrumbs
show each page's place in the navigation. Existing URLs and server tab links work.

The fleet registry lives in SQLite, independently of host filesystems. Status
combines agent container health, fresh game telemetry and agent-local RCON.
Unreachable hosts report unknown instance state; they do not trigger duplicate
replacements. Running containers continue through controller or agent outages.

Provisioning owns Create, Start, Stop, Restart, Rebuild and Destroy. Commands and
results have durable operation IDs and immutable instance generations. Queues,
capacity reservations and history survive controller restarts. **Rebuild removes
per-instance writable files** and creates a new generation. Shared immutable game
libraries survive rebuilds and destruction.

Disruptive actions wait for a fresh empty lobby or verified stopped container.
The agent rechecks occupancy locally immediately before activation and disruption.
Long player sessions can continue indefinitely. **Drain** prevents new launches
and safely retires existing servers as they become eligible.

Collections save target maps and lobby counts. Fill missing servers adds only the
shortfall; Match collection exactly also queues safe removal of excess servers.
Launch previews use the latest verified files for each game and select an
independent runtime image. If newer game files become available before launch,
refresh the preview to confirm the updated files. Every new launch uses the
latest promoted configuration, scripts and plugins when execution starts. Saved
Plutonium keys are reused by exact server ID and retained after removal.

## Libraries and campaigns

Game Files has one upload slot each for T5 and T6. Uploads preserve the folder's
directory layout and included DLC; there is no separate DLC declaration.
EC records immutable upload IDs, file manifests, checksums and sizes automatically.
Uploads use bounded chunks, support pause/resume and verify checksums before
publication. Actual game files
use dedicated filesystem storage, never SQLite blobs. The last upload to finish
verification becomes that game's latest files, even if it started earlier.
Active hosts always prepare the latest T5 and T6 files when available, including
hosts enrolled later. Hosts first download all files with up to four concurrent
transfers, then verify all checksums in a separate pass before the files become
usable. Both steps show progress, elapsed time, measured speed and an estimated
time remaining for the current step. Interrupted transfers resume. Failed preparation
retries automatically with backoff; the host page reports any blocking condition.
Set `PLUTAINER_IMAGE` to change the default runtime prepared on hosts. Existing
servers retain their existing files and runtime without a forced restart.
Upload history and server references are preserved; unused version files follow the
configured retention policy.
Existing named libraries appear together under their game, with no renaming or
reupload required. Multiple instances on one host share read-only base files.

Scripts, plugins, configuration baselines and overlays retain their versioning,
beta, promotion and rollback workflows. Campaigns stage bytes remotely without
stopping games; activation uses the same occupancy and concurrency gates as
lifecycle operations. Agent before images preserve exact rollback bytes and
operation history.

Artifact bodies are stored once by SHA-256 in SQLite. Versions, staging plans,
rollback snapshots and launch commands reference those shared bytes. The first
startup after this upgrade backfills existing rows transactionally; preserve the
data-directory backup when upgrading. Current agents fetch assigned artifacts
from an authenticated endpoint and verify the hash. Older agents continue to
receive inline content until their automatic update completes.

T5 Zombies uses `t5sp` and `dedicated_sp.cfg`; T6 uses `t6zm` and
`dedicated_zm.cfg`. Map selection generates `electric_cherry_map.cfg`, loaded
after the main config. T6 variants retain their distinct catalog selections.
Game-file uploads and Plutonium runtime revisions are tracked independently.

Game telemetry uses its own `EC_TELEMETRY_TOKEN`; configure reporters to send it
in an authorization header. Anonymous reports are rejected before they can
affect occupancy or game history. Existing JSON payloads are unchanged.

## Documentation

- [Fleet concurrency and retention settings](docs/settings.md)

- [Native hosts, storage and operation recovery](docs/native-hosts.md)
- [Users, roles and authentication](docs/authentication.md)
- [Provisioning and safety queues](docs/provisioning.md)
- [Collections and promoted launch recipes](docs/deployment-blueprints.md)
- [Maintenance](docs/maintenance.md)
- [Beta test plans and game telemetry](docs/beta-test-plans.md)
- [Plugin releases](docs/plugin-releases.md)
- [Metrics](docs/metrics.md)
- [Leaderboard and match history](docs/leaderboard.md)

## Development

The controller is SvelteKit/TypeScript with SQLite via better-sqlite3, the Node
adapter and server-sent events. The host agent is Python 3 standard-library code
installed as a systemd service. Hosts do not install Node.js or build EC.

```sh
npm install
npm run agent:package
# Export EC_SETUP_TOKEN in your shell using the secret saved for /setup.
EC_AUTH_ALLOW_HTTP=true npm run dev

npm run check
npm test
npm run test:agent
npm run build
```

Builds regenerate the downloadable agent, standalone updater and checksum manifest,
and bundle the metrics/leaderboard workers under `build/reports` and native launch
preparation under `build/launch`. `npm run dev` and `npm test` compile those workers
automatically. After editing a report or launch-preparation implementation during
development, restart `npm run dev` to rebuild its worker.
Reports use separate read-only SQLite connections and bounded revision-aware caches;
metrics snapshots can be reused for up to 30 seconds when their inputs are unchanged.

Set `EC_PERF_LOG=true` to log slow request timings, report computation/cache timings,
and event-loop delay. Responses include `Server-Timing`, and browser navigations
record an `ec:navigation` performance measure. `npm run benchmark:leaderboard`
uses an isolated synthetic database; pass `-- --scenario all --baseline-ref <commit>`
to compare full response parity and warm query timings with an earlier revision.

Set `EC_SLOW_SQL_MS=20` to log runtime statements taking at least 20 ms; bound
values and string literals are omitted. It is disabled by default. Before and
after database changes, capture these timings under the same fleet load and run
`npm run diagnostics:database -- /data/cherry.db` inside the controller to report
the largest tables/indexes, database and WAL sizes, and free pages. This command
opens an existing database read-only and does not run app migrations.
The controller uses a 128 MiB page cache and each reader worker uses 32 MiB.
Main-database migrations use a 16 MiB cache with memory mapping disabled; SQLite
temporary storage stays disk-backed during migrations and normal operation.
The container uses `/data/.sqlite-tmp` for SQLite scratch files so a memory-backed
`/tmp` cannot turn a database rebuild into an allocation of the whole database.
Override `SQLITE_TMPDIR` only with another writable, disk-backed directory.
Startup logs identify migration phases, elapsed time and resident memory. A
completed migration is retained across restarts; SQLite rolls back interrupted
transactions and startup retries those migrations. The one-time vacuum still needs disk space for its temporary
copy. Do not remove `cherry.db-wal` to recover from an interrupted upgrade.
For a local data disk, set `EC_SQLITE_MMAP_MB=256` to enable memory-mapped reads.
Query statistics are optimized at startup and hourly; WAL reuse is capped at
64 MiB after checkpoints (active transactions can temporarily grow it larger).

Local controller
metadata defaults to `./data/cherry.db`; use a separate test database when
exercising enrollment. Production agent transport requires HTTPS.

`src/lib/server/hosts` owns enrollment, readiness and command transport;
`src/lib/server/backends` provides instance identity and remote file access;
`src/lib/server/provisioning/native.ts` connects existing lifecycle and campaign
queues to agents; `src/lib/server/game-files` owns immutable game-file uploads and
streaming distribution; `agent/` contains the service, installer and agent tests.

Automated tests use simulated Docker/game hosts. Real Debian installer and
T5/T6 game-server validation remain deployment acceptance work; see the checklist
in the native-host guide.