electric-cherry/docs/native-hosts.md

Native host management

Electric Cherry controls Docker/Plutainer through an outbound host agent. A native instance never uses Komodo or a filesystem mounted into the controller. The controller owns desired state, campaigns, collections, safety queues and history; the agent owns local execution and its durable operation journal.

Install and launch

1. Run the controller with persistent /data storage. Set EC_PUBLIC_URL to its HTTPS origin and proxy TLS to port 3000. Set a random EC_SETUP_TOKEN and create the first administrator at /setup. Management pages/APIs require user login and roles; agent endpoints authenticate their own host credentials. See Users and access for setup and password recovery.
2. Open Hosts → Add Host, choose a name, local game-port range and bind address, and generate an enrollment. Run the displayed command on Debian 12/13 amd64.
3. The installer validates the platform, installs Python 3 and an absent Docker installation, verifies the downloaded agent and updater SHA-256 over HTTPS, enrolls it and enables electric-cherry-agent.service and its automatic update timer. It needs no Git, Node.js or local build.
4. In Game Files, select T5 or T6 and upload your local game folder. Name the library and version and publish it after the upload finishes. Include the DLC files in that folder; there is no separate DLC declaration. Uploading also works before enrolling any host.
5. Hosts automatically prepare the configured Plutainer runtime and the latest published version of every game library. Automatic host setup reports progress and errors. Wait for verification before launching. Hosts report connection and readiness separately; deployment also checks the particular library, image, capacity and ports you selected.
6. Launch a server or collection. Select the host, immutable game-file version(s) and use the automatically selected default runtime, or an advanced override. EC selects lobby IDs and reserves UDP/TCP ports atomically. Saved Plutonium server keys remain associated with their exact server names.

A fresh destination host downloads centrally stored game files without manual copying or per-container setup. EC does not supply the game content itself.

The controller reconciles automatic setup on agent heartbeats and operation polls. It first pulls PLUTAINER_IMAGE (default ghcr.io/ayymoss/plutainer:latest), then prepares the newest published version of each library, one setup operation at a time. Publishing more files or enrolling another host needs no manual propagation. Setup waits for an active, connected, healthy host with sufficient storage. Failed operations retry automatically with backoff from one minute to fifteen minutes; their errors remain visible in the host's setup panel and operation history. Disconnected hosts resume when they reconnect. Draining and disabled hosts do not receive new automatic setup work.

Verified older versions remain cached and running servers keep their selected files and runtime. Automatic setup ensures that the configured image is present; it does not refresh an already cached mutable image tag. Advanced controls on the host and game-file pages can explicitly pull a custom runtime, prepare an older library version or retry preparation immediately.

The default host root is /var/lib/electric-cherry; installer --data-dir changes it. Use a dedicated, root-owned directory. For a separate disk, use a child of its mount point (for example /mnt/games/electric-cherry), with no nested mounts, so host deletion can remove the entire directory. The installer records ownership and refuses to adopt unrelated files. Data is separated into libraries/<game>/<version>/content, instances/<UUID>/<generation>/app, and agent/operations.sqlite3. Base libraries are mounted read-only and instance deletion removes only that instance's files. The controller stores large files under GAME_FILES_ROOT (default beside the SQLite database), with manifests/checksums and operation metadata in SQLite.

The agent uses Plutainer's documented game-file and app mounts. Its log reader supports relative log links that resolve inside the instance app directory, while file writes reject symlinks and paths outside that directory.

Automatic agent updates

New installations enable automatic agent updates by default. Each host checks its controller's release over HTTPS about every five minutes, beginning about two minutes after boot, with up to 30 seconds of jitter. Updating the controller therefore makes its compatible, newer agent release available to enrolled hosts. There is no controller command to schedule and no update button to press.

Existing hosts need the current installer once. From the host's Agent access, generate an installer command and rerun it on the same machine, with the same controller and data directory. Do not use --reenroll for this upgrade. The installer preserves host identity, credentials, game files and running containers. Agent version 1.2.0 includes automatic update support; the host page shows Setup required until a fresh heartbeat confirms the updater is installed.

The Automatic agent updates panel reports the running version, target version, last check, progress and any error. Offline hosts show their last reported status. Deferred updates report why they are waiting: for example an active agent operation, an unavailable controller or a newer updater bootstrap. Checks also defer while the agent service is stopped; an update does not start a service that was stopped outside the update workflow. The independent updater holds the same execution lock, verifies the release checksum, version and supported protocol, replaces agent code, and restarts the agent service. It never restarts Docker or game containers. It confirms an authenticated heartbeat from the new agent process before accepting the update; failure restores the previous agent code. Its durable journal permits recovery after a host reboot or interrupted update.

The updater only installs newer compatible versions; rolling the controller back does not downgrade agents. A rejected version/checksum pair stays blocked until the controller publishes a different pair that is still newer than the installed agent. Inspect the reported error when a host shows Needs attention. The updater itself is a stable bootstrap component: it changes only when the installer is rerun. If a future release requires an unsupported updater protocol, rerun that release's installer on the host.

When publishing changed agent code, bump its numeric VERSION in agent/ec_agent.py and rebuild the controller. Packaging derives the manifest version and checksums from the sources; an already installed version is never replaced just because its bytes changed.

The installer places the updater at /usr/local/libexec/electric-cherry-update.py and enables electric-cherry-update.timer, which starts the oneshot electric-cherry-update.service. Its state is kept under /etc/electric-cherry, including update-status.json, update/journal.json, agent-health.json and the shared update.lock. Check the timer and worker logs on the host with:

systemctl status electric-cherry-update.timer
journalctl -u electric-cherry-update.service -n 100

Identity, transport and credentials

Enrollment tokens are random, hashed at rest, single-use and expire after 15 minutes. Consumption and credential issuance are transactional. Enrollment only works at the enrollment endpoint. Each host receives a distinct bearer credential and the controller only stores its hash. Host and enrollment bearer credentials are rejected outside /api/agent/.

The agent uses HTTPS, operating-system certificate validation, outbound heartbeats and polling; Docker has no exposed remote API. Agent endpoints reject HTTP unless EC_AGENT_ALLOW_HTTP=true is explicitly enabled for development. The agent's insecure development option is limited to loopback. Public release assets contain no credentials. The release checksum is authenticated by the controller's HTTPS origin; there is no separate offline release-signing infrastructure.

Browser sessions do not authorize agents. Similarly, host credentials never grant access to operator pages. Game telemetry uses a separate EC_TELEMETRY_TOKEN, configured in each reporter's authorization header; see the telemetry contract.

Hosts offers credential rotation and revocation. Rotation persists the new secret on the agent before switching controller credentials, so a lost response can be retried using the same proposed secret. Revocation affects one host only and does not stop its running game containers. Re-enrollment requires a fresh token and an explicit installer --reenroll when local credentials already exist.

API outline:

• GET /api/hosts/:id: host details, operation history and deletion blockers.
• DELETE /api/hosts/:id: administrator requests cleanup of an unused host; returns 202 while remote removal is pending, 200 for an unenrolled host, or 409 when the agent needs updating or unresolved ownership remains.
• POST /api/agent/enroll: consume enrollment token and return host credentials.
• GET /api/agent/release/manifest.json: public agent version, protocol and checksum plus updaterProtocolVersion and updaterSha256 for installer bootstrap.
• GET /api/agent/release/ec-agent.py and ec-agent-update.py: public release source, verified against that HTTPS manifest before installation.
• POST /api/agent/heartbeat: authenticated inventory, current operation and caches.
• GET /api/agent/operations: deliver/reconcile durable operation IDs.
• POST /api/agent/operations/:id: authenticated acceptance, progress or result.
• POST /api/agent/decommission/:id: cleanup worker reports file removal and agent uninstall using its own operation-scoped receipt credential.
• POST /api/game-files: administrator creates a folder upload with game, name, version, optional description and files: [{path, size, lastModified}].
• GET /api/game-files/:id/upload: durable per-file offsets and publication status.
• PUT /api/game-files/:id/upload/files/:path: administrator uploads a raw chunk with X-Upload-Offset, X-Upload-Length and X-Content-SHA256 headers.
• POST /api/game-files/:id/upload/complete: administrator starts asynchronous verification/publication; poll the upload endpoint for completion.
• /api/agent/libraries/...: authenticated game-file downloads. Legacy /api/agent/imports/... endpoints remain compatible with already queued imports.

The operation envelope is independent of polling and can be carried over a persistent transport later. No unrestricted shell operation is available.

Safety and durable operations

Each remote request has an immutable ID and payload, plus an instance UUID and generation where applicable. Reusing a key with a different request is rejected. The controller records remote IDs before dispatch. The agent persists accepted, running, completed, failed and unknown states in SQLite. Lost acknowledgments return the recorded result; restarting the controller never creates a new mutation simply because the previous reply was lost.

The agent reconciles ambiguous results using owned Docker labels, generations and persisted execution checkpoints when those observations prove the outcome. Otherwise the result remains Unknown and holds dependent work. Running alone is insufficient evidence that a particular restart happened. EC neither assumes an unreachable server stopped nor launches a duplicate replacement elsewhere.

Campaign bytes are staged and verified while players remain connected. Only Provisioning can dispatch activation. It checks source eligibility and shared capacity, and the agent repeats a fresh local RCON occupancy check immediately before activation/restart, stop, removal or rebuild. Busy or unverifiable lobbies wait indefinitely, including multi-hour Zombies sessions. Before images returned by the agent populate existing campaign/beta rollback history. A beta cannot be closed while its remote activation receipt is still unresolved.

Rebuild uses a new generation, fresh instance storage and the latest promoted configuration/scripts/plugins. It retains the selected game-file version and runtime image. Old generation commands cannot target the replacement. Start, restart and create complete only after post-operation health evidence; creation and rebuild also verify the selected map and running Plutonium revision. The base-game library and runtime revision are independent records.

Drain blocks new deployments and queues safe retirement of existing lobbies. Active beta ownership and other operations continue to gate retirement. Players may still join until an eligible empty moment; draining does not kick them. Disable holds agent dispatch. Neither mode, controller downtime nor agent communication loss stops unrelated running containers.

Administrators can choose Delete host on a host card or at the top of its detail page. The confirmation lists anything that must be resolved first: registered servers, reserved ports, or operations already sent to the agent. Drain the host or remove its servers, then let outstanding operations reconcile. An unenrolled host is removed immediately. An enrolled host stays visible as Deleting until its agent confirms cleanup. Offline hosts wait for their next connection; their absence is never treated as proof of removal. Existing agents need the current installer: generate a fresh command under Agent access, rerun it on the host, and wait for a heartbeat advertising uninstall support.

Deletion blocks new operations, cancels queued commands that were never delivered, and invalidates unused installer links. A separate cleanup service removes the host's downloaded game libraries, instance files, agent journal and configuration, owned containers, and recorded runtime images unused by other containers. It stops and removes the agent service and executable, automatic update timer, update service, updater executable and update state, then removes its own files. The controller hides the host and revokes its ordinary credentials only after receiving the final uninstall receipt. Adding the machine again requires a new host enrollment.

Game folders uploaded through Electric Cherry belong to the controller. Those original uploads remain available for other hosts and future deployments; only the deleted host's copies are removed. Controller operation history is also kept. Unrelated containers, files and shared dependencies such as Docker and Python are preserved. Historical entries in shared system logs are not erased.

Cleanup retries after connection loss or partial file removal. Failures before handoff offer Retry cleanup after the installation is repaired; failures in the independent worker retry automatically. The worker stays installed until the controller acknowledges data cleanup. After removing its final files it retries the completion receipt from memory. A power loss in that final window can leave the controller awaiting confirmation even though removal succeeded; inspect the machine if that status persists. The controller does not guess.

Libraries and ports

Choose the actual game folder, not its parent, a ZIP archive or the Plutonium launcher data folder. The selected folder's name is removed once; nested paths such as zone/english/... and main/... stay intact. Plutonium's T5 server guide and T6 server guide use a copy of the game directory, separately from the launcher/runtime. The uploader preserves the supplied files rather than attempting to infer installed maps from a manually maintained DLC list.

Folder uploads use chunks of at most 8 MiB, each checked against its SHA-256 before being appended to temporary controller storage. Memory usage does not grow with the size of an individual game file. Pause/retry uses the controller's saved file offsets. After a page reload, choose Resume upload and reselect the original, unchanged folder; relative paths, sizes and modification times must match. Keep the page open while transferring. Once publication starts, the controller hashes every complete file and publishes the directory atomically; closing the browser does not stop that verification. An interrupted controller can retry publication. Published versions cannot be updated; changing files requires a new version. Hosts download up to four files concurrently with HTTP Range resumption, report progress, verify all checksums and atomically promote the complete cache. Checksums are calculated while bytes arrive, avoiding another complete read of newly downloaded files. Resumed files include their saved prefix in verification. Progress reports are rate-limited so controller round trips do not stall every transferred chunk. Corrupt or incomplete content never becomes Ready. Cache inventory is reconciled on heartbeats, including vanished caches. Older immutable versions remain usable when their exact manifest is verified.

The controller and reverse proxy must accept 8 MiB upload requests and folder manifests up to 32 MiB. Compose sets BODY_SIZE_LIMIT=Infinity; application routes still enforce these limits, declared file sizes and checksums. Set a proxy body limit of at least 32 MiB (for example client_max_body_size 32m in nginx). Existing published versions and cached host libraries remain usable after the upload workflow replaces source-host imports.

Reservations include host ID, local bind address, protocol and port. Wildcard binds conflict with concrete addresses on the same host. UDP and TCP reservations for one launch commit together; the same number can be used independently on another host. Optional public/NAT address and port are recorded separately and do not change local Docker bindings. EC does not configure external NAT/firewalls.

Development and validation

npm run agent:package
npm run check
npm test
npm run test:agent
npm run build

The build packages the agent, standalone updater and installer into the static release, generates the agent/updater checksum manifest and serves those same assets in the Node-adapter image. Development serves the freshly packaged static release even when an older production build exists. The Python programs use only the standard library.

Automated tests cover enrollment/authentication, rotation/revocation, host readiness, scoped ports, immutable library publication, streaming/ranges, checksums, interrupted downloads, instance generation ownership, local occupancy holds, controller restart/reconnect, ambiguous results, occupied campaign staging, activation before images, collections and draining. Host deletion tests cover ownership guards, interrupted cleanup, uninstall receipts, credential revocation and retention of controller uploads/history. Docker/systemd/UDP and host observations are simulated in these tests.

A real Debian/Docker/Plutonium acceptance run is still required before production rollout. This workspace has no Docker engine or Debian game host, so automated coverage is not evidence that a real T5/T6 server has been deployed here. Run:

• Installer on Debian 12 and 13 amd64, then rerun without changing identity/data.
• Bootstrap an existing host with the current installer, then publish a newer compatible agent release. Verify a timer-driven update, its authenticated heartbeat and unchanged game containers. Exercise an active-operation hold, bad checksum, failed startup/heartbeat rollback, controller downgrade, network loss and host reboot during update. Verify updater protocol mismatch requests an installer rerun.
• T5 and T6 deployments and a mixed collection from a fresh destination host.
• Start, stop, restart, rebuild, destroy; verify RCON, map and runtime revision.
• Read console and game logs, including Plutainer's relative game-log links.
• File campaign and rollback, with a player joining between controller check and agent activation, and during file activation before restart; verify that the agent holds disruption.
• Drain occupied lobbies and verify shared libraries survive their retirement.
• Delete a drained host; verify its game copies, data directory, agent and cleanup services, update timer, configuration and executables are removed, controller uploads/history stay, and unrelated Docker workloads continue. Repeat with connection loss during cleanup and after the final receipt, and a reboot during data removal.
• Corrupt/interrupted library downloads, resumed transfer and verified promotion.
• Controller restart, agent restart and network loss before execution, during execution and after execution but before acknowledgement. Verify the same operation IDs and absence of duplicate containers/restarts.

Legacy backend code remains for existing database registrations. New native launches do not require its configuration, API, network or mounted directories.