Clean up README and organize documentation
This commit is contained in:
@@ -1,228 +1,79 @@
|
||||
# Electric Cherry
|
||||
|
||||
Electric Cherry manages T5 and T6 Plutonium Zombies servers through its own host
|
||||
agents. Docker and Plutainer run the games; EC handles hosts, game files,
|
||||
collections, configuration, campaigns, maintenance and operation history.
|
||||
A self-hosted dashboard for managing Plutonium T5 and T6 Zombies servers.
|
||||
Electric Cherry handles your fleet, game files, releases, and maintenance from
|
||||
one web interface. Game servers run on your hosts using Docker, Plutainer, and
|
||||
a small Python agent.
|
||||
|
||||
|
||||
|
||||
*Fleet overview with sample data.*
|
||||
|
||||
## Features
|
||||
|
||||
- Launch individual servers or saved collections of maps and lobbies.
|
||||
- Monitor players, host capacity, match history, and leaderboards.
|
||||
- Manage scripts, plugins, and configuration with beta testing and rollback.
|
||||
- Queue maintenance around active games and scale spare lobbies with opt-in policies.
|
||||
- Control access with administrator, operator, and viewer accounts.
|
||||
|
||||
## Getting started
|
||||
|
||||
Run the controller with Docker Compose behind HTTPS:
|
||||
You'll need Docker Compose for the controller, an HTTPS reverse proxy, and
|
||||
Debian 12 or 13 amd64 game hosts. Bring your own game files and Plutonium server keys.
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
# Set EC_PUBLIC_URL, generate EC_SETUP_TOKEN, and retain EC_DATA_DIR.
|
||||
openssl rand -hex 32
|
||||
```
|
||||
|
||||
Edit `.env`: use the generated secret for `EC_SETUP_TOKEN`, set `EC_PUBLIC_URL`
|
||||
to your HTTPS address, and choose a persistent `EC_DATA_DIR`. Then start the
|
||||
controller and proxy your HTTPS address to port 3000:
|
||||
|
||||
```sh
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
Open EC and create the first administrator using `EC_SETUP_TOKEN` (at least 32
|
||||
characters; generate it with `openssl rand -hex 32`). After setup, remove that
|
||||
token from the environment. Administrators can add administrators, operators and
|
||||
viewers from **Administration → Users**. See [Users and access](docs/authentication.md)
|
||||
for permissions, sessions, password recovery and upgrade instructions.
|
||||
Open the dashboard and create your first administrator with the setup token.
|
||||
Remove the token from `.env` afterward and recreate the controller with
|
||||
`docker compose up -d`.
|
||||
|
||||
Operator pages and management APIs require login. Enrolled agents authenticate
|
||||
independently and connect outbound over HTTPS; the
|
||||
controller needs no Docker socket, game-server mounts or inbound RCON access.
|
||||
The default Compose data mount preserves the previous
|
||||
`/root/zombies/.electric-cherry` database location. For a new installation, set
|
||||
`EC_DATA_DIR` to your preferred persistent directory. Back up that directory,
|
||||
including SQLite metadata and published game-file storage.
|
||||
1. Upload your T5 or T6 folder, including DLC, under **Library → Game Files**.
|
||||
2. Add a machine under **Fleet → Hosts** and run its generated enrollment command.
|
||||
3. Save your Plutonium keys under **Administration → Keys**.
|
||||
4. Wait for host setup to finish, then launch a server or collection.
|
||||
|
||||
1. **Library → Game Files**: upload the complete local folder under T5 or T6,
|
||||
including its DLC files. There are no library names or version labels to enter.
|
||||
Each upload becomes the latest files for that game after verification.
|
||||
Uploads work before any host is enrolled.
|
||||
2. **Fleet → Hosts → Add Host**: configure a Debian 12/13 amd64 machine and run its
|
||||
generated enrollment command. The host automatically prepares its Plutainer
|
||||
runtime and the latest T5 and T6 files, when available. The host page
|
||||
shows setup progress. Automatic agent updates are enabled by default.
|
||||
3. Add the exact Plutonium server keys in **Administration → Keys**, or supply missing keys when
|
||||
launching.
|
||||
4. Launch a server or collection. Collection previews show cached capacity
|
||||
estimates; Apply saves the queue and opens its progress page. Provisioning
|
||||
checks capacity and assigns each server a host and port when it can start.
|
||||
The latest game files and default runtime are selected automatically.
|
||||
CPU/memory allowances learn
|
||||
from actual container usage, using temporary defaults while calibrating.
|
||||
|
||||
See [Native host management](docs/native-hosts.md) for installation, credentials,
|
||||
storage, failure recovery, API contracts and the live-host acceptance checklist.
|
||||
|
||||
**Capacity** shows available CPU, memory, ports, estimated additional servers,
|
||||
and the shared startup queue. Filter by host or game, inspect each host's limiting
|
||||
resource, and explore up to seven days of resource headroom.
|
||||
|
||||
Existing hosts need the current installer once to enable automatic updates.
|
||||
Generate it under **Agent access** and rerun it on the same host without
|
||||
`--reenroll`; identity and game data are preserved. Thereafter hosts check their
|
||||
controller about every five minutes, verify compatible releases and update agent
|
||||
code between operations. Failed updates roll back; running game containers
|
||||
continue. The host page shows update status and the last check.
|
||||
|
||||
## Fleet and operations
|
||||
|
||||
The sidebar groups pages into **Fleet**, **Library**, **Operations**, **Analytics**,
|
||||
and **Administration** (administrators only). On small screens, open the navigation
|
||||
menu in the header. The account menu contains **My account**, including your
|
||||
browser's display timezone; keyboard shortcuts are available through **?**.
|
||||
|
||||
**Operations → Overview** combines work that needs attention, running workflows,
|
||||
and recent results. Related provisioning attempts and host commands stay grouped
|
||||
under their workflow. Normal waits for empty lobbies remain under **Running**;
|
||||
successful routine checks remain in **Host operation history**. Full history is
|
||||
available on each workflow page.
|
||||
|
||||
Server details use **Live**, **Matches**, **Configuration**, **Diagnostics**, and
|
||||
**History** sections. Hosts use **Overview**, **Servers**, **Setup**, and **Settings**.
|
||||
Contextual Back links preserve the originating filters and views, while breadcrumbs
|
||||
show each page's place in the navigation. Existing URLs and server tab links work.
|
||||
|
||||
The fleet registry lives in SQLite, independently of host filesystems. Status
|
||||
combines agent container health, fresh game telemetry and agent-local RCON.
|
||||
Unreachable hosts report unknown instance state; they do not trigger duplicate
|
||||
replacements. Running containers continue through controller or agent outages.
|
||||
|
||||
Provisioning owns Create, Start, Stop, Restart, Rebuild and Destroy. Commands and
|
||||
results have durable operation IDs and immutable instance generations. Queues,
|
||||
capacity reservations and history survive controller restarts. **Rebuild removes
|
||||
per-instance writable files** and creates a new generation. Shared immutable game
|
||||
libraries survive rebuilds and destruction.
|
||||
|
||||
Disruptive actions wait for a fresh empty lobby or verified stopped container.
|
||||
The agent rechecks occupancy locally immediately before activation and disruption.
|
||||
Long player sessions can continue indefinitely. **Drain** prevents new launches
|
||||
and safely retires existing servers as they become eligible.
|
||||
|
||||
Collections save target maps and lobby counts. Fill missing servers adds only the
|
||||
shortfall; Match collection exactly also queues safe removal of excess servers.
|
||||
Launch previews use the latest verified files for each game and select an
|
||||
independent runtime image. If newer game files become available before launch,
|
||||
refresh the preview to confirm the updated files. Every new launch uses the
|
||||
latest promoted configuration, scripts and plugins when execution starts. Saved
|
||||
Plutonium keys are reused by exact server ID and retained after removal.
|
||||
|
||||
## Libraries and campaigns
|
||||
|
||||
Game Files has one upload slot each for T5 and T6. Uploads preserve the folder's
|
||||
directory layout and included DLC; there is no separate DLC declaration.
|
||||
EC records immutable upload IDs, file manifests, checksums and sizes automatically.
|
||||
Uploads use bounded chunks, support pause/resume and verify checksums before
|
||||
publication. Actual game files
|
||||
use dedicated filesystem storage, never SQLite blobs. The last upload to finish
|
||||
verification becomes that game's latest files, even if it started earlier.
|
||||
Active hosts always prepare the latest T5 and T6 files when available, including
|
||||
hosts enrolled later. Hosts first download all files with up to four concurrent
|
||||
transfers, then verify all checksums in a separate pass before the files become
|
||||
usable. Both steps show progress, elapsed time, measured speed and an estimated
|
||||
time remaining for the current step. Interrupted transfers resume. Failed preparation
|
||||
retries automatically with backoff; the host page reports any blocking condition.
|
||||
Set `PLUTAINER_IMAGE` to change the default runtime prepared on hosts. Existing
|
||||
servers retain their existing files and runtime without a forced restart.
|
||||
Upload history and server references are preserved; unused version files follow the
|
||||
configured retention policy.
|
||||
Existing named libraries appear together under their game, with no renaming or
|
||||
reupload required. Multiple instances on one host share read-only base files.
|
||||
|
||||
Scripts, plugins, configuration baselines and overlays retain their versioning,
|
||||
beta, promotion and rollback workflows. Campaigns stage bytes remotely without
|
||||
stopping games; activation uses the same occupancy and concurrency gates as
|
||||
lifecycle operations. Agent before images preserve exact rollback bytes and
|
||||
operation history.
|
||||
|
||||
Artifact bodies are stored once by SHA-256 in SQLite. Versions, staging plans,
|
||||
rollback snapshots and launch commands reference those shared bytes. The first
|
||||
startup after this upgrade backfills existing rows transactionally; preserve the
|
||||
data-directory backup when upgrading. Current agents fetch assigned artifacts
|
||||
from an authenticated endpoint and verify the hash. Older agents continue to
|
||||
receive inline content until their automatic update completes.
|
||||
|
||||
T5 Zombies uses `t5sp` and `dedicated_sp.cfg`; T6 uses `t6zm` and
|
||||
`dedicated_zm.cfg`. Map selection generates `electric_cherry_map.cfg`, loaded
|
||||
after the main config. T6 variants retain their distinct catalog selections.
|
||||
Game-file uploads and Plutonium runtime revisions are tracked independently.
|
||||
|
||||
Game telemetry uses its own `EC_TELEMETRY_TOKEN`; configure reporters to send it
|
||||
in an authorization header. Anonymous reports are rejected before they can
|
||||
affect occupancy or game history. Existing JSON payloads are unchanged.
|
||||
See the [setup guide](docs/guides/getting-started.md) for telemetry, storage,
|
||||
and the first launch. Host agents connect outbound over HTTPS; the controller
|
||||
doesn't need a Docker socket or direct access to your game-server files.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Fleet concurrency and retention settings](docs/settings.md)
|
||||
|
||||
- [Native hosts, storage and operation recovery](docs/native-hosts.md)
|
||||
- [Users, roles and authentication](docs/authentication.md)
|
||||
- [Provisioning and safety queues](docs/provisioning.md)
|
||||
- [Lobby autoscaling implementation plan](docs/lobby-autoscaling-plan.md)
|
||||
- [Lobby autoscaling and pilot procedure](docs/autoscaling.md)
|
||||
- [Collections and promoted launch recipes](docs/deployment-blueprints.md)
|
||||
- [Maintenance](docs/maintenance.md)
|
||||
- [Beta test plans and game telemetry](docs/beta-test-plans.md)
|
||||
- [Plugin releases](docs/plugin-releases.md)
|
||||
- [Metrics](docs/metrics.md)
|
||||
- [Leaderboard and match history](docs/leaderboard.md)
|
||||
- [Documentation index](docs/README.md) — setup, operations, releases, and analytics.
|
||||
- [Development guide](docs/development/README.md) — architecture, checks, and diagnostics.
|
||||
- [Autoscaling plan](docs/plans/lobby-autoscaling-plan.md) — design and remaining pilot work.
|
||||
- [Idle CPU research](docs/research/t5-t6-idle-cpu-research.md) — experiments and tradeoffs.
|
||||
|
||||
## Development
|
||||
|
||||
The controller is SvelteKit/TypeScript with SQLite via better-sqlite3, the Node
|
||||
adapter and server-sent events. The host agent is Python 3 standard-library code
|
||||
installed as a systemd service. Hosts do not install Node.js or build EC.
|
||||
The controller uses SvelteKit, TypeScript, and SQLite. The host agent uses the
|
||||
Python 3 standard library. For local development, install Node.js 22 and Python 3:
|
||||
|
||||
```sh
|
||||
npm install
|
||||
npm ci
|
||||
npm run agent:package
|
||||
# Export EC_SETUP_TOKEN in your shell using the secret saved for /setup.
|
||||
export EC_SETUP_TOKEN=$(openssl rand -hex 32)
|
||||
export BETA_ARCHIVE_DIR=./data/beta-archive
|
||||
EC_AUTH_ALLOW_HTTP=true npm run dev
|
||||
```
|
||||
|
||||
Open `http://localhost:3000` and use the setup token from your shell. The local
|
||||
database defaults to `./data/cherry.db`; keep it separate from production data.
|
||||
|
||||
```sh
|
||||
npm run check
|
||||
npm test
|
||||
npm run test:agent
|
||||
npm run build
|
||||
```
|
||||
|
||||
Builds regenerate the downloadable agent, standalone updater and checksum manifest,
|
||||
and bundle the metrics/leaderboard workers under `build/reports` and native launch
|
||||
preparation under `build/launch`. `npm run dev` and `npm test` compile those workers
|
||||
automatically. After editing a report or launch-preparation implementation during
|
||||
development, restart `npm run dev` to rebuild its worker.
|
||||
Reports use separate read-only SQLite connections and bounded revision-aware caches;
|
||||
metrics snapshots can be reused for up to 30 seconds when their inputs are unchanged.
|
||||
|
||||
Set `EC_PERF_LOG=true` to log slow request timings, report computation/cache timings,
|
||||
and event-loop delay. Responses include `Server-Timing`, and browser navigations
|
||||
record an `ec:navigation` performance measure. `npm run benchmark:leaderboard`
|
||||
uses an isolated synthetic database; pass `-- --scenario all --baseline-ref <commit>`
|
||||
to compare full response parity and warm query timings with an earlier revision.
|
||||
|
||||
Set `EC_SLOW_SQL_MS=20` to log runtime statements taking at least 20 ms; bound
|
||||
values and string literals are omitted. It is disabled by default. Before and
|
||||
after database changes, capture these timings under the same fleet load and run
|
||||
`npm run diagnostics:database -- /data/cherry.db` inside the controller to report
|
||||
the largest tables/indexes, database and WAL sizes, and free pages. This command
|
||||
opens an existing database read-only and does not run app migrations.
|
||||
The controller uses a 128 MiB page cache and each reader worker uses 32 MiB.
|
||||
Main-database migrations use a 16 MiB cache with memory mapping disabled; SQLite
|
||||
temporary storage stays disk-backed during migrations and normal operation.
|
||||
The container uses `/data/.sqlite-tmp` for SQLite scratch files so a memory-backed
|
||||
`/tmp` cannot turn a database rebuild into an allocation of the whole database.
|
||||
Override `SQLITE_TMPDIR` only with another writable, disk-backed directory.
|
||||
Startup logs identify migration phases, elapsed time and resident memory. A
|
||||
completed migration is retained across restarts; SQLite rolls back interrupted
|
||||
transactions and startup retries those migrations. The one-time vacuum still needs disk space for its temporary
|
||||
copy. Do not remove `cherry.db-wal` to recover from an interrupted upgrade.
|
||||
For a local data disk, set `EC_SQLITE_MMAP_MB=256` to enable memory-mapped reads.
|
||||
Query statistics are optimized at startup and hourly; WAL reuse is capped at
|
||||
64 MiB after checkpoints (active transactions can temporarily grow it larger).
|
||||
|
||||
Local controller
|
||||
metadata defaults to `./data/cherry.db`; use a separate test database when
|
||||
exercising enrollment. Production agent transport requires HTTPS.
|
||||
|
||||
`src/lib/server/hosts` owns enrollment, readiness and command transport;
|
||||
`src/lib/server/backends` provides instance identity and remote file access;
|
||||
`src/lib/server/provisioning/native.ts` connects existing lifecycle and campaign
|
||||
queues to agents; `src/lib/server/game-files` owns immutable game-file uploads and
|
||||
streaming distribution; `agent/` contains the service, installer and agent tests.
|
||||
|
||||
Automated tests use simulated Docker/game hosts. Real Debian installer and
|
||||
T5/T6 game-server validation remain deployment acceptance work; see the checklist
|
||||
in the native-host guide.
|
||||
|
||||
Reference in New Issue
Block a user