electric-cherry/docs/native-hosts.md

Native host management

Electric Cherry controls Docker/Plutainer through an outbound host agent. A native instance never uses Komodo or a filesystem mounted into the controller. The controller owns desired state, campaigns, collections, safety queues and history; the agent owns local execution and its durable operation journal.

Install and launch

1. Run the controller with persistent /data storage. Set EC_PUBLIC_URL to its HTTPS origin and proxy TLS to port 3000. Set a random EC_SETUP_TOKEN and create the first administrator at /setup. Management pages/APIs require user login and roles; agent endpoints authenticate their own host credentials. See Users and access for setup and password recovery.
2. Open Hosts → Add Host, choose a name, local game-port range and bind address, and generate an enrollment. Run the displayed command on Debian 12/13 amd64.
3. The installer validates the platform, installs Python 3 and an absent Docker installation, verifies the downloaded agent SHA-256 over HTTPS, enrolls it and enables electric-cherry-agent.service. It needs no Git, Node.js or local build.
4. In Hosts, prepare a Plutainer runtime image. In Game Files, import your existing t5_game_files or t6_game_files Docker volume on an enrolled source host. Name the library and version and record its DLC metadata.
5. Cache the published version on each destination host. Wait for verification. Hosts report connection and readiness separately; deployment also checks the particular library, image, capacity and ports you selected.
6. Launch a server or collection. Select the host, immutable game-file version(s) and runtime image. EC selects lobby IDs and reserves UDP/TCP ports atomically. Saved Plutonium server keys remain associated with their exact server names.

A fresh destination host downloads centrally stored game files without manual copying or per-container setup. EC does not supply the game content itself.

The default host root is /var/lib/electric-cherry; installer --data-dir changes it. Data is separated into libraries/<game>/<version>/content, instances/<UUID>/<generation>/app, and agent/operations.sqlite3. Base libraries are mounted read-only and instance deletion removes only that instance's files. The controller stores large files under GAME_FILES_ROOT (default beside the SQLite database), with manifests/checksums and operation metadata in SQLite.

The agent uses Plutainer's documented game-file and app mounts. Its log reader supports relative log links that resolve inside the instance app directory, while file writes reject symlinks and paths outside that directory.

Identity, transport and credentials

Enrollment tokens are random, hashed at rest, single-use and expire after 15 minutes. Consumption and credential issuance are transactional. Enrollment only works at the enrollment endpoint. Each host receives a distinct bearer credential and the controller only stores its hash. Host and enrollment bearer credentials are rejected outside /api/agent/.

The agent uses HTTPS, operating-system certificate validation, outbound heartbeats and polling; Docker has no exposed remote API. Agent endpoints reject HTTP unless EC_AGENT_ALLOW_HTTP=true is explicitly enabled for development. The agent's insecure development option is limited to loopback. Public release assets contain no credentials. The release checksum is authenticated by the controller's HTTPS origin; there is no separate offline release-signing infrastructure.

Browser sessions do not authorize agents. Similarly, host credentials never grant access to operator pages. Game telemetry uses a separate EC_TELEMETRY_TOKEN, configured in each reporter's authorization header; see the telemetry contract.

Hosts offers credential rotation and revocation. Rotation persists the new secret on the agent before switching controller credentials, so a lost response can be retried using the same proposed secret. Revocation affects one host only and does not stop its running game containers. Re-enrollment requires a fresh token and an explicit installer --reenroll when local credentials already exist.

API outline:

• POST /api/agent/enroll: consume enrollment token and return host credentials.
• POST /api/agent/heartbeat: authenticated inventory, current operation and caches.
• GET /api/agent/operations: deliver/reconcile durable operation IDs.
• POST /api/agent/operations/:id: authenticated acceptance, progress or result.
• /api/agent/imports/... and /api/agent/libraries/...: host-scoped uploads and authenticated game-file downloads.

The operation envelope is independent of polling and can be carried over a persistent transport later. No unrestricted shell operation is available.

Safety and durable operations

Each remote request has an immutable ID and payload, plus an instance UUID and generation where applicable. Reusing a key with a different request is rejected. The controller records remote IDs before dispatch. The agent persists accepted, running, completed, failed and unknown states in SQLite. Lost acknowledgments return the recorded result; restarting the controller never creates a new mutation simply because the previous reply was lost.

The agent reconciles ambiguous results using owned Docker labels, generations and persisted execution checkpoints when those observations prove the outcome. Otherwise the result remains Unknown and holds dependent work. Running alone is insufficient evidence that a particular restart happened. EC neither assumes an unreachable server stopped nor launches a duplicate replacement elsewhere.

Campaign bytes are staged and verified while players remain connected. Only Provisioning can dispatch activation. It checks source eligibility and shared capacity, and the agent repeats a fresh local RCON occupancy check immediately before activation/restart, stop, removal or rebuild. Busy or unverifiable lobbies wait indefinitely, including multi-hour Zombies sessions. Before images returned by the agent populate existing campaign/beta rollback history. A beta cannot be closed while its remote activation receipt is still unresolved.

Rebuild uses a new generation, fresh instance storage and the latest promoted configuration/scripts/plugins. It retains the selected game-file version and runtime image. Old generation commands cannot target the replacement. Start, restart and create complete only after post-operation health evidence; creation and rebuild also verify the selected map and running Plutonium revision. The base-game library and runtime revision are independent records.

Drain blocks new deployments and queues safe retirement of existing lobbies. Active beta ownership and other operations continue to gate retirement. Players may still join until an eligible empty moment; draining does not kick them. Disable holds agent dispatch. Neither mode, controller downtime nor agent communication loss stops unrelated running containers.

Libraries and ports

Imports scan the source volume and generate per-file SHA-256 manifests. Uploads stream to temporary controller storage, are bounded by their manifest sizes and verified before atomic publication. Published versions cannot be updated; changing files requires a new version. Hosts download with HTTP Range resumption, report progress, verify all checksums and atomically promote the complete cache. Corrupt or incomplete content never becomes Ready. Cache inventory is reconciled on heartbeats, including vanished caches. Older immutable versions remain usable when their exact manifest is verified.

Reservations include host ID, local bind address, protocol and port. Wildcard binds conflict with concrete addresses on the same host. UDP and TCP reservations for one launch commit together; the same number can be used independently on another host. Optional public/NAT address and port are recorded separately and do not change local Docker bindings. EC does not configure external NAT/firewalls.

Development and validation

npm run agent:package
npm run check
npm test
npm run test:agent
npm run build

The build packages agent/ec_agent.py and agent/install.sh into the static release, generates its checksum manifest and serves those same assets in the Node-adapter image. The Python agent uses only its standard library.

Automated tests cover enrollment/authentication, rotation/revocation, host readiness, scoped ports, immutable library publication, streaming/ranges, checksums, interrupted downloads, instance generation ownership, local occupancy holds, controller restart/reconnect, ambiguous results, occupied campaign staging, activation before images, collections and draining. Docker/UDP and host observations are simulated in these tests.

A real Debian/Docker/Plutonium acceptance run is still required before production rollout. This workspace has no Docker engine or Debian game host, so automated coverage is not evidence that a real T5/T6 server has been deployed here. Run:

• Installer on Debian 12 and 13 amd64, then rerun without changing identity/data.
• T5 and T6 deployments and a mixed collection from a fresh destination host.
• Start, stop, restart, rebuild, destroy; verify RCON, map and runtime revision.
• Read console and game logs, including Plutainer's relative game-log links.
• File campaign and rollback, with a player joining between controller check and agent activation, and during file activation before restart; verify that the agent holds disruption.
• Drain occupied lobbies and verify shared libraries survive their retirement.
• Corrupt/interrupted library downloads, resumed transfer and verified promotion.
• Controller restart, agent restart and network loss before execution, during execution and after execution but before acknowledgement. Verify the same operation IDs and absence of duplicate containers/restarts.

Legacy backend code remains for existing database registrations. New native launches do not require its configuration, API, network or mounted directories.