electric-cherry/docs/guides/native-hosts.md

Native host management

Electric Cherry controls Docker/Plutainer through an outbound host agent. The controller does not need mounted game-server files. It owns desired state, campaigns, collections, safety queues and history; the agent owns local execution and its durable operation journal.

Install and launch

1. Run the controller with persistent /data storage. Set EC_PUBLIC_URL to its HTTPS origin and proxy TLS to port 3000. Set a random EC_SETUP_TOKEN and create the first administrator at /setup. Management pages/APIs require user login and roles; agent endpoints authenticate their own host credentials. See Users and access for setup and password recovery.
2. Open Hosts → Add Host, choose a name, local game-port range and bind address, and generate an enrollment. Run the displayed command on Debian 12/13 amd64.
3. The installer validates the platform, installs Python 3 and an absent Docker installation, verifies the downloaded agent and updater SHA-256 over HTTPS, enrolls it and enables electric-cherry-agent.service and its automatic update timer. It needs no Git, Node.js or local build.
4. In Game Files, upload your complete local folder under T5 or T6. Include the DLC files in that folder; there is no separate DLC declaration, library name or version label. The upload becomes the latest files for that game after verification. Uploading also works before enrolling any host.
5. Hosts automatically prepare the configured Plutainer runtime and the latest verified T5 and T6 files whenever available. Automatic host setup reports progress and errors. Wait for verification before launching. Hosts report connection and readiness separately; deployment also checks the particular game files, image, capacity and ports required for the launch.
6. Open Create server, choose a game and map, and review the automatic readiness checks. Launch collections separately from Fleet → Collections. The latest game files and default runtime are selected automatically. Advanced settings in the single-server dialog includes a runtime override for images an admin has already prepared on the destination host, along with optional network overrides. If the latest files change after the checks run, let the dialog recheck the setup before launching; Retry checks also rechecks it. Launching saves the requested lobby IDs. Provisioning selects a host and atomically reserves UDP/TCP ports when capacity is available. An available Plutonium key from the game pool is reserved for each launch. Its assignment survives stops and rebuilds and ends after confirmed removal.

A fresh destination host downloads centrally stored game files without manual copying or per-container setup. EC does not supply the game content itself.

Docker installation

The installer uses Debian's Docker packages: docker.io on Debian 12, plus docker-cli on Debian 13. Rerunning it after an earlier Debian 13 install that omitted the client adds the missing client without restarting the existing engine. An already available Docker installation (including Docker CE) is reused. If the daemon is unavailable, the installer reports the Docker error and stops before enrollment; inspect systemctl status docker and journalctl -u docker.

Automatic host setup

The controller reconciles automatic setup on agent heartbeats. Command polls only deliver work already queued. Setup first pulls PLUTAINER_IMAGE (default ghcr.io/ayymoss/plutainer:latest), then prepares the latest verified upload for T5 and for T6 when available, one setup operation at a time. After those prerequisites, capable agents also prepare the optional Plutonium runtime cache described below. Uploading new files or enrolling another host needs no manual propagation. Setup waits for an active, connected, healthy host with sufficient storage. Failed operations retry automatically with backoff from one minute to fifteen minutes; their errors remain visible in the host's setup panel and Hosts → Operation history. This shared history lists operations across hosts, newest first, with 25 rows per page and host, target, and source details. It defaults to Actions & errors; Routine checks and All operations include successful monitoring reads. Older records may lack source details. Disconnected hosts resume when they reconnect. Draining and disabled hosts do not receive new automatic setup work.

Operation history retention

Settings → Routine operation history retention (days) controls cleanup of successful monitoring and file-read records. The default is one day; zero keeps them indefinitely. Actions, failures, negative checks, and unresolved operations are retained. Agent 1.4.9 also cleans up successful routine journal records after the controller confirms their final receipt. Older unconfirmed journal entries remain until a replay supplies that confirmation. Controller cleanup runs every five minutes and drains eligible records in batches, yielding between batches.

Log-read operation history stores metadata only. The waiting reader receives the log content directly; saved match chat remains with its match. Background cleanup also removes log bodies left in controller history by earlier versions.

Agent 1.8.5 adds incremental game-log reads for chat collection. After the initial tail, the controller requests only appended bytes, handles rotation and truncation, and buffers incomplete lines. Unchanged logs need no content transfer or chat parsing. Background collection skips offline hosts, stopped servers and known empty servers, with one final read after players leave. Opening chat still reads on demand. Older agents use tail reads until they update.

New game files and cached versions

An automatic download that has not been delivered to the agent can be replaced when a newer upload becomes available. A delivered operation finishes safely before the host prepares the latest files. Failed or incomplete uploads never replace the latest verified files.

Verified previous uploads remain cached and running servers keep their existing files and runtime; uploading new files does not restart servers. Automatic setup ensures that the configured image is present; it does not refresh an already cached mutable image tag. Advanced controls on the host and game-file pages can explicitly pull a custom runtime, prepare an older upload or retry preparation immediately.

Plutonium runtime cache

Agent 1.8.0 adds a host cache for the files Plutainer normally downloads from Plutonium on a new container's first start. This is separate from the cached Docker image and uploaded base game libraries. Automatic preparation warms it after the image and libraries are ready and refreshes it for new production revisions. The controller only queues this work for agents that advertise support. Cache preparation is optional: a missing or corrupt cache does not prevent a launch from using Plutainer's normal download path.

Cache preparation has lower priority than deployments and other host changes. If work arrives during a download, the next heartbeat tells the agent to yield between transfer steps. Preparation resumes under the same operation ID after that work finishes, reusing files that have already passed verification.

The agent downloads the official production manifest and its listed vendor files, verifies the SHA-1 hashes published in that manifest, and makes a snapshot available only after the complete set passes verification. It keeps the cache under agent/plutonium-cache within the host data directory. It does not populate the cache from existing lobbies, so lobby configuration, keys, custom scripts, plugins and runtime state are not collected into it. Preparation retains the current snapshot and the resumable download, removes older snapshots, and clears abandoned temporary files before checking available storage or publishing the new snapshot. A failed refresh preserves the previous complete cache.

For new instances and rebuilds using ghcr.io/ayymoss/plutainer image references, the agent seeds app/runtime/plutonium from a valid snapshot, including its matching cdn_info.json, before applying the instance's launch files. Each instance receives its own writable copy; changes made by a lobby or its updater cannot change the host cache or another lobby. Custom runtime images keep their existing startup behavior.

Plutainer's updater remains enabled. When a cached revision is current, it can skip the initial vendor-file download; when production is newer, it can download the update normally. Launch readiness still verifies the running Plutonium revision against production. Refreshing the cache does not restart or update running lobbies. Ordinary restarts retain the instance's existing files and do not need to seed from the cache again.

Deleting a lobby preserves the host cache for subsequent launches. Deleting the host removes it with the rest of the managed data directory. This is intended to reduce repeated first-start downloads; deployment-time improvements still need measurement on a real Debian/Docker/Plutonium host.

Storage layout

The default host root is /var/lib/electric-cherry; installer --data-dir changes it. Use a dedicated, root-owned directory. For a separate disk, use a child of its mount point (for example /mnt/games/electric-cherry), with no nested mounts, so host deletion can remove the entire directory. The installer records ownership and refuses to adopt unrelated files. Data is separated into libraries/<game>/<version>/content, instances/<UUID>/<generation>/app, and agent/operations.sqlite3. Base libraries are mounted read-only and instance deletion removes only that instance's files. The controller stores large files under GAME_FILES_ROOT (default beside the SQLite database), with manifests/checksums and operation metadata in SQLite.

The agent uses Plutainer's documented game-file and app mounts. Its log reader supports relative log links that resolve inside the instance app directory, while file writes reject symlinks and paths outside that directory.

Automatic agent updates

New installations enable automatic agent updates by default. Each host checks its controller's release over HTTPS about every five minutes, beginning about two minutes after boot, with up to 30 seconds of jitter. Updating the controller therefore makes its compatible, newer agent release available to enrolled hosts. There is no controller command to schedule and no update button to press.

Existing hosts need the current installer once. From the host's Agent access, generate an installer command and rerun it on the same machine, with the same controller and data directory. Do not use --reenroll for this upgrade. The installer preserves host identity, credentials, game files and running containers. Agent version 1.2.0 includes automatic update support; the host page shows Setup required until a fresh heartbeat confirms the updater is installed.

The Automatic agent updates panel reports the running version, target version, last check, progress and any error. Offline hosts show their last reported status. Deferred updates report why they are waiting: for example an active agent operation, an unavailable controller or a newer updater bootstrap. Checks also defer while the agent service is stopped; an update does not start a service that was stopped outside the update workflow. The independent updater holds the same execution lock, verifies the release checksum, version and supported protocol, replaces agent code, and restarts the agent service. It never restarts Docker or game containers. It confirms an authenticated heartbeat from the new agent process before accepting the update; failure restores the previous agent code. Its durable journal permits recovery after a host reboot or interrupted update.

The updater only installs newer compatible versions; rolling the controller back does not downgrade agents. A rejected version/checksum pair stays blocked until the controller publishes a different pair that is still newer than the installed agent. Inspect the reported error when a host shows Needs attention. The updater itself is a stable bootstrap component: it changes only when the installer is rerun. If a future release requires an unsupported updater protocol, rerun that release's installer on the host.

Publishing releases and inspecting updates

When publishing changed agent code, bump its numeric VERSION in agent/ec_agent.py and rebuild the controller. Packaging derives the manifest version and checksums from the sources; an already installed version is never replaced just because its bytes changed.

The installer places the updater at /usr/local/libexec/electric-cherry-update.py and enables electric-cherry-update.timer, which starts the oneshot electric-cherry-update.service. Its state is kept under /etc/electric-cherry, including update-status.json, update/journal.json, agent-health.json and the shared update.lock. Check the timer and worker logs on the host with:

systemctl status electric-cherry-update.timer
journalctl -u electric-cherry-update.service -n 100

Placement and resource capacity

Host inventory and eligibility

The Inventory panel shows CPU usage with agent 1.4.2 and later. It measures host-wide busy time across all cores on a 0–100% scale between inventory samples (normally about 15 seconds). Idle and I/O wait are excluded; stolen CPU time counts as unavailable capacity. The first sample after startup, unavailable counters, and older agents show Not reported. Offline hosts retain their last reported value. The controller defaults to retaining CPU and memory samples for seven days (missing measurements remain unknown). Scheduling checks capacity against the latest heartbeat and uses the last five minutes to rank eligible hosts; the Inventory panel shows the latest reading.

New servers are placed automatically. Collection previews use cached availability estimates and submission saves unplaced jobs without running capacity checks. Provisioning checks each job asynchronously: a connected, active host must have verified game files, its runtime, storage, a free UDP/TCP port, and valid CPU and memory measurements. Hosts with stale or missing measurements cannot accept new servers. Existing API clients may still explicitly constrain a launch to a host.

Queued jobs retain their accepted game-file version. Automatic host preparation also downloads the game files needed by unplaced jobs; a newer upload does not remove files still needed by the queue. Only the configured default runtime is pulled automatically. Custom images must have been prepared by an admin from the selected host's page, or on an active host for automatic placement, before a launch is accepted. An image cached by another Docker workload does not qualify. Agent 1.8.6 reports these exact prepared references from runtime-images.json while their image IDs remain cached; the controller also trusts its configured default when cached. Host pages and APIs expose only prepared runtimes. Older agents can still run the configured default, but must update before accepting custom runtimes. Queued launches and migrations never trigger custom image pulls, and container creation cannot implicitly pull a missing image.

Agent 1.8.7 also checks a runtime's current image ID against its recorded preparation before creating or rebuilding a container. If another program retags a prepared image, an admin must prepare it again from the host page before launching it. Containers are created using the validated immutable image ID, so a tag change after validation cannot substitute another image. Existing containers keep running.

Resource samples and learned allowances

Only provisioning reserves a host and port. Agent 1.4.6 fixes container discovery for resource sampling (the earlier collector omitted label= from its Docker filter). It samples each managed running container with a bounded docker stats batch during inventory collection. CPU is normalized to cores (100% Docker CPU is one core); memory is the Linux working usage reported by Docker, excluding reclaimable cache. Missing or failed samples stay unknown instead of becoming zero usage. Samples must match the host's live instance and generation. Measurements arrive through /api/agent/heartbeat, normally every 15 seconds plus collection and delivery time, independently of RCON or inspection operations. Sampling failures appear on the host's Learned server capacity panel. Existing containers can begin contributing running samples as soon as the agent updates; startup samples require observing the first five minutes of a container's run.

The controller defaults to retaining per-instance minute peaks for 24 hours and caches learned CPU/memory allowances. The first five minutes after a container starts count as startup; later samples train the running allowance. Both phases use the 95th percentile of minute peaks, with no per-server headroom. Each calculation considers up to the latest 10,000 minute peaks. Host-local evidence is preferred to account for different hardware; new hosts can borrow the fleet's learned allowance. Startup allowances are never lower than running allowances. The host page shows both allowances and their source. Expand View samples in Learned server capacity to inspect retained minute peaks for the last 24 hours. Filter by startup/running phase or switch from this host to the entire fleet. Rows show time, server and generation, CPU cores, and memory in MiB; CPU and memory peaks may come from different heartbeats. Results are paginated, and Refresh samples loads the latest stored measurements without contacting agents. The allowance count may lag the table until the next calculation or reflect a different source.

Lower allowances require at least three startup minute samples spanning two minutes, or twenty running minute samples spanning five minutes. Until then, 0.5 core and 1 GiB are temporary cold-start allowances, raised immediately if observations exceed them. They are replaced by measured allowances once the evidence threshold is met; a tiny 0.01-core floor prevents zero-CPU admission. Models expire after 24 hours without measurements. Recalculation happens during heartbeat ingestion, at most once per minute per host/fleet phase; new peaks raise the cache immediately. Existing startup estimates refresh when new startup measurements arrive. Launch admission reads only the cached model.

Admission and placement

Scheduling still leaves 5% host CPU and 500 MB RAM (500,000,000 bytes) for the OS and bursts. These allowances are scheduling estimates, not Docker limits. Collection previews estimate how many new servers fit at their running allowance. Jobs may still be queued when that estimate is insufficient. Current host CPU and available memory account for existing containers and other processes. Observed containers are not topped up to historical startup or running allowances, even while awaiting health verification. Established instances are covered by host usage when individual container measurements are unavailable.

Dispatch checks the next launch's startup allowance against live host usage. Unplaced queued launches wait their turn without holding physical resources. Any previously assigned reservations remain protected. Submitted or uncertain launches missing a matching-generation measurement still consume a full startup allowance until telemetry accounts for them. This lets collections start as capacity becomes available, without stacking every queued server's startup allowance. Capacity errors report projected usage, usable capacity and reservations.

When startup capacity is available, provisioning checks each eligible host’s current CPU and available memory against reservations and headroom. Among hosts that fit, it chooses the lowest projected CPU/memory utilization, using the busier of the latest reading and the five-minute average for ranking. A historical resource dip cannot block a host whose current capacity has recovered. Historical ranking runs in a worker thread. The controller rechecks current capacity and reserves the selected host and port atomically, then records them in the launch recipe. Host network defaults are resolved again for the final host; explicit overrides are preserved. If no host qualifies, the request waits and is checked again. Once remote staging or creation has been submitted, placement stays fixed to prevent duplicate instances. Cancelling an unsubmitted launch releases its reservation; uncertain remote work retains capacity until cleanup is confirmed.

Capacity overview

Open Capacity (/capacity) to see the same cached estimates used by collection previews. Available totals include hosts ready for the default runtime and latest files for the selected game. All games counts shared resources once, including hosts ready for either game; it is not a guarantee that every game fits every host. The host breakdown shows readiness, missing measurements, available ports, learned running/startup allowances, and the resource limiting additional servers.

Startup slots and queued jobs are fleet-wide, even with a host or game selected. Server estimates use running allowances; provisioning still checks the larger startup allowance before starting each job. Port availability uses each host's configured bind address. Queued unplaced jobs do not reserve physical resources.

The history graph offers the last hour, 24 hours, or seven days of measured CPU and memory headroom, after the usual OS reserve. It averages each host's reports within each period, then sums hosts for fleet views. Missing reports appear as gaps. Historical measurements cover the selected currently registered hosts and do not reconstruct past readiness, port reservations, or startup slots. Reports refresh automatically while the page is visible, and historical queries run in a separate reader worker. GET /api/capacity supports range=1h|24h|7d, host=<host ID>, and game=all|t5|t6; host/game selections also remain in the page URL.

Identity, transport and credentials

Enrollment tokens are random, hashed at rest, single-use and expire after 15 minutes. Consumption and credential issuance are transactional. Enrollment only works at the enrollment endpoint. Each host receives a distinct bearer credential and the controller only stores its hash. Host and enrollment bearer credentials are rejected outside /api/agent/.

The agent uses HTTPS, operating-system certificate validation, outbound heartbeats and polling; Docker has no exposed remote API. Agent endpoints reject HTTP unless EC_AGENT_ALLOW_HTTP=true is explicitly enabled for development. The agent's insecure development option is limited to loopback. Public release assets contain no credentials. The release checksum is authenticated by the controller's HTTPS origin; there is no separate offline release-signing infrastructure.

Browser sessions do not authorize agents. Similarly, host credentials never grant access to operator pages. Game telemetry uses a separate EC_TELEMETRY_TOKEN, configured in each reporter's authorization header; see the telemetry contract.

Hosts offers credential rotation and revocation. Rotation persists the new secret on the agent before switching controller credentials, so a lost response can be retried using the same proposed secret. Revocation affects one host only and does not stop its running game containers. Re-enrollment requires a fresh token and an explicit installer --reenroll when local credentials already exist.

Host and game-file API

• GET /api/hosts/:id: host details and deletion blockers.
• GET /api/hosts/operations?page=1&filter=important: operation history across hosts, newest first, with 25 summaries per page and filtered total/page counts. Filters are important (default), routine, and all. Summaries include the server, safe target path or status command, and initiating source when recorded. Payloads and results are omitted.
• DELETE /api/hosts/:id: administrator requests cleanup of an unused host; returns 202 while remote removal is pending, 200 for an unenrolled host, or 409 when the agent needs updating or unresolved ownership remains.
• POST /api/agent/enroll: consume enrollment token and return host credentials.
• GET /api/agent/release/manifest.json: public agent version, protocol and checksum plus updaterProtocolVersion and updaterSha256 for installer bootstrap.
• GET /api/agent/release/ec-agent.py and ec-agent-update.py: public release source, verified against that HTTPS manifest before installation.
• POST /api/agent/heartbeat: authenticated inventory, current operation and caches.
• GET /api/agent/operations: deliver/reconcile durable operation IDs.
• POST /api/agent/operations/:id: authenticated acceptance, progress or result.
• POST /api/agent/decommission/:id: cleanup worker reports file removal and agent uninstall using its own operation-scoped receipt credential.
• GET /api/game-files: returns {libraries, hosts}. libraries always contains two game slots, T5 and T6, including empty slots. Each slot has a nullable latestVersionId identifying its latest verified upload, plus upload history.
• POST /api/game-files: administrator creates a folder upload with {game: "t5" | "t6", files: [{path, size, lastModified}]}. Upload IDs and internal revision metadata are generated automatically; no library name, version label or description is required.
• GET /api/game-files/:id/upload: durable per-file offsets and publication status.
• PUT /api/game-files/:id/upload/files/:path: administrator uploads a raw chunk with X-Upload-Offset, X-Upload-Length and X-Content-SHA256 headers.
• POST /api/game-files/:id/upload/complete: administrator starts asynchronous verification/publication; poll the upload endpoint for completion.
• /api/agent/libraries/...: authenticated game-file downloads. Legacy /api/agent/imports/... endpoints remain compatible with already queued imports.

The operation envelope is independent of polling and can be carried over a persistent transport later. No unrestricted shell operation is available.

Safety and durable operations

Each remote request has an immutable ID and payload, plus an instance UUID and generation where applicable. Reusing a key with a different request is rejected. The controller records remote IDs before dispatch. The agent persists accepted, running, completed, failed and unknown states in SQLite. Lost acknowledgments return the recorded result; restarting the controller never creates a new mutation simply because the previous reply was lost.

The agent reconciles ambiguous results using owned Docker labels, generations and persisted execution checkpoints when those observations prove the outcome. Otherwise the result remains Unknown and holds dependent work. Running alone is insufficient evidence that a particular restart happened. EC neither assumes an unreachable server stopped nor launches a duplicate replacement elsewhere.

An interrupted rebuild can settle as Failed when its last checkpoint is disrupting, Docker confirms both the old and replacement containers are absent, and any existing generation marker still matches the original instance. The agent removes only the partial replacement files, preserves the old generation's files and marker, and retains the original diagnostic in its failure report. It does not replay the interrupted rebuild. A separately queued rebuild can then proceed through normal eligibility checks. A later creation checkpoint, a changed generation, an inspection error, or failed partial-file cleanup leaves the result Unknown and keeps dependent work waiting.

Player checks and activation

Campaign bytes are staged and verified while players remain connected. Only Provisioning can dispatch activation. It checks source eligibility and shared capacity, and the agent repeats a fresh local RCON occupancy check immediately before activation/restart, stop, removal or rebuild. Busy or unverifiable lobbies wait indefinitely, including multi-hour Zombies sessions. Before images returned by the agent populate existing campaign/beta rollback history. A beta cannot be closed while its remote activation receipt is still unresolved.

An empty Fleet card does not replace this final host-local check: the card may use telemetry, and the controller's RCON parser is more permissive than the agent's disruption guard. Agent 1.4.1 includes the local RCON failure reason in the operation's hold message (configuration, timeout, unreachable endpoint, authentication, invalid packet, or an unrecognized player count). For a parser hold, inspect the raw status reply in Diagnose connection. Unknown counts remain held; RCON availability alone does not prove the lobby is empty. It also recognizes T6's num score bot ping header, which earlier agents rejected even when the complete player table was empty.

Rebuild uses a new generation, fresh instance storage and the latest promoted configuration/scripts/plugins. It retains the selected game-file version and runtime image. Old generation commands cannot target the replacement. Start, restart and create complete only after post-operation health evidence; creation and rebuild also verify the selected map and running Plutonium revision. The base-game library and runtime revision are independent records.

Rebalance instances

Open Fleet → Capacity → Rebalance to spread existing instances across active hosts. Hosts also provides a shortcut. Operators and administrators can run a rebalance; viewers can inspect its history.

Choose at least two participating hosts, a game selection, and a maximum of 1–20 moves (initially five), then choose Preview rebalance. The preview shows current and projected instance counts, CPU and memory utilization, proposed source/destination pairs, and exclusions. A larger host can carry more instances: the planner balances resource utilization relative to each host's capacity. The selected game limits which instances can move; the estimates include all workloads on the participating hosts.

The planner uses current measurements and conservative five-minute host history, current-generation instance resource samples, destination running/startup allowances, and existing reservations. Each move must reduce the source/destination pair's peak CPU-or-memory utilization by at least five percentage points. A successfully moved instance is excluded for 30 minutes to limit repeated moves. Unplaced queued launches do not reserve physical resources.

Only healthy instances with fresh, confirmed empty occupancy and recoverable deployed files qualify. Occupied, stopped, unknown, stale, and incompatible instances are excluded, as are beta, campaign, maintenance, and unresolved recovery work. Positive current occupancy from either RCON or telemetry vetoes an empty claim. A partially balanced result is normal when eligible instances cannot improve the distribution enough.

Choose Start rebalance within five minutes of the preview. Submission rechecks the proposed pairs; a changed plan requires a new preview. One durable run executes at a time, and it never adds moves beyond the approved list. Before each move it rechecks occupancy, identity, capacity, and competing work. A move that no longer qualifies is skipped. After each healthy completion the run waits for fresh reports from both affected hosts before considering its next move.

Provisioning reserves the destination before removing the source, then recreates and health-verifies the lobby. The source host stays active. This is an empty-lobby rebuild with a temporary outage; its address or port may change. The logical identity, map, saved key, runtime credentials, runtime image, game-file library, and recorded deployed configuration/scripts/plugins are retained. Untracked local mutable files are not copied. If deployed bytes or credentials are no longer available, the instance is excluded instead of silently upgrading to promoted files.

Stop remaining moves withdraws work that has not committed source removal. Once source removal has been queued, that move continues through destination creation and health verification. Stopping does not undo completed moves. Unknown agent results retain their original operation and reservations until reconciliation; they never authorize a duplicate instance. Failures stop the remaining moves and link to Provisioning recovery. Recovering a move there does not resume the rest of its run. Runs and operation links survive controller restarts; completed/stopped runs are retained for 31 days.

Autoscaling retains ownership of lobby counts. A move is represented as a rebuild, so the affected map's scaling decisions wait for it to settle instead of treating the temporary outage as missing capacity.

GET /api/rebalancing returns host choices and the 20 most recent runs, with active runs first. POST /api/rebalancing accepts preview with settings (hostIds, game: all|t5|t6, maxMoves), start with previewId, or stop with the run id. Writes use the existing operator authorization and same-origin checks. Repeating Start with the same preview returns its original run. The controller planner and run state live in src/lib/server/rebalancing/; all host mutations remain in Provisioning. Rebalancing is manual; there is no periodic automatic policy.

Drain and enable hosts

Drain host asks for confirmation, blocks new deployments, and migrates instances to other ready hosts with capacity. Each migration waits for an empty lobby, reserves destination CPU, memory and ports, removes the source instance, then rebuilds the same server identity and map on the destination using its saved key and current promoted configuration/scripts/plugins. The address or port may change. Active beta ownership and other operations still gate migration, and the agent checks occupancy again before removal. Unavailable capacity leaves the source running; unknown removal results retain their reservation until the original operation reconciles. Migrations survive controller restarts.

Once all migrations pass health verification and the source is empty, it becomes Disabled automatically. There is no manual Disable action. Enable starts automatic setup while the host remains outside the capacity pool: the configured runtime is pulled again and both latest published BO1 (T5) and BO2 (T6) libraries are downloaded as needed and fully verified, including cached copies. Both games must be published. Failed setup or stale inventory keeps the host disabled; only successful preparation and fresh inventory return it to Active.

Delete a host

Administrators can choose Delete host on a host card or at the top of its detail page. The confirmation lists anything that must be resolved first: registered servers, reserved ports, or operations already sent to the agent. Drain the host or remove its servers, then let outstanding operations reconcile. An unenrolled host is removed immediately. An enrolled host stays visible as Deleting until its agent confirms cleanup. Offline hosts wait for their next connection; their absence is never treated as proof of removal. Existing agents need the current installer: generate a fresh command under Agent access, rerun it on the host, and wait for a heartbeat advertising uninstall support.

Deletion blocks new operations, cancels queued commands that were never delivered, and invalidates unused installer links. A separate cleanup service removes the host's downloaded game libraries, instance files, agent journal and configuration, owned containers, and recorded runtime images unused by other containers. It stops and removes the agent service and executable, automatic update timer, update service, updater executable and update state, then removes its own files. The controller hides the host and revokes its ordinary credentials only after receiving the final uninstall receipt. Adding the machine again requires a new host enrollment.

Game folders uploaded through Electric Cherry belong to the controller. Those original uploads remain available for other hosts and future deployments; only the deleted host's copies are removed. Controller operation history is also kept. Unrelated containers, files and shared dependencies such as Docker and Python are preserved. Historical entries in shared system logs are not erased.

Interrupted cleanup

Cleanup retries after connection loss or partial file removal. Failures before handoff offer Retry cleanup after the installation is repaired; failures in the independent worker retry automatically. The worker stays installed until the controller acknowledges data cleanup. After removing its final files it retries the completion receipt from memory. A power loss in that final window can leave the controller awaiting confirmation even though removal succeeded; inspect the machine if that status persists. The controller does not guess.

Game files

Upload a game folder

Game Files always shows T5 and T6, each with its own upload button and latest verified files. Successful uploads become latest in publication completion order, not by their start date or an entered label. A newer upload still being transferred or verified leaves the current files available. Previous uploads appear in history, and interrupted uploads can be resumed separately.

Choose the actual game folder, not its parent, a ZIP archive or the Plutonium launcher data folder. The selected folder's name is removed once; nested paths such as zone/english/... and main/... stay intact. Plutonium's T5 server guide and T6 server guide use a copy of the game directory, separately from the launcher/runtime. The uploader preserves the supplied files rather than attempting to infer installed maps from a manually maintained DLC list.

Transfer and verification

Folder uploads use chunks of at most 8 MiB, each checked against its SHA-256 before being appended to temporary controller storage. Memory usage does not grow with the size of an individual game file. Pause/retry uses the controller's saved file offsets. After a page reload, choose Resume upload and reselect the original, unchanged folder; relative paths, sizes and modification times must match. Keep the page open while transferring. Once publication starts, the controller hashes every complete file and publishes the directory atomically; closing the browser does not stop that verification. An interrupted controller can retry publication. Verified uploads are immutable; changing files means uploading another folder for the same game. Hosts first download every file with up to four concurrent transfers and HTTP Range resumption. After all transfers finish, a separate verification pass reads every file from disk and checks its complete SHA-256, including any resumed prefix. Only then is the complete cache promoted atomically. Corrupt staged files are removed so the next attempt downloads them again. Separating the phases removes checksum work from the transfer loop and exposes disk-verification time separately; it does not guarantee a shorter total duration.

The Game Files and host setup pages show Download and Verify as separate steps, each with its own progress. Elapsed time records active work across resumptions and stops when work finishes or fails. Remaining time and speed use the current step's measured progress; a download estimate does not include the verification pass that follows. Estimates appear after enough progress is measured and are hidden for stale or offline hosts. Existing agents without timing reports still show byte progress. Agent 1.2.2 provides phase timing; automatic updates install it between host operations. Phase transitions are reported promptly and periodic heartbeats carry progress; controller round trips do not stall every transferred chunk.

Cache cleanup and retention

Corrupt or incomplete content never becomes Ready. Cache inventory is reconciled on heartbeats, including vanished caches. The latest published version of each game is always retained. With agent 1.4.7 or newer, an older host copy is automatically eligible for removal once no live instance reservation on that host references it. The game-file minimum count, supersession grace period and explicit protections in Admin → Settings also apply. Stopped, provisioning, and rebuilding instances still pin their version. The agent also refuses deletion if any Docker container (including stopped containers) mounts those files. Cleanup includes partial downloads and retries after interruption.

The controller removes an older version's files after every host has reported its copy absent and no instance or unresolved operation references that version. Offline hosts keep their last reported copies pinned until they reconnect. Upload metadata, manifests, and operation history remain visible; removed files cannot be downloaded again without a new upload. Cleanup runs on heartbeats, successful publication, and the controller background worker, without needing an open Game Files page.

Proxy limits and older uploads

The container and Compose retain BODY_SIZE_LIMIT=Infinity for legacy source-host imports, which stream whole game files directly to disk and verify manifest sizes and checksums. Routes that buffer request bodies enforce limits before parsing: ordinary JSON requests (including telemetry) are limited to 1 MiB, authentication and user-management JSON to 8 KiB, game-file upload chunks to 8 MiB, and folder manifests to 32 MiB. Existing agent endpoints retain their individual limits, up to 128 MiB for operation results. Game-file uploads also enforce declared sizes and checksums.

Script, plugin and config artifacts are limited to 32 MiB per file, matching the host agent. Multipart uploads allow 33 MiB total, including fields and framing; beta revisions allow 65 MiB total for multiple replacement files. Both the declared request length and actual streamed bytes are checked before multipart parsing. Oversized ordinary JSON and artifact uploads receive HTTP 413. GitHub plugin imports use the same 32 MiB artifact limit.

Set a proxy body limit of at least 128 MiB (for example client_max_body_size 128m in nginx) for buffered endpoints. If legacy source-host imports are still in use, allow the largest game file on /api/agent/imports/*/files/*; these transfers do not use the browser's 8 MiB chunks. Route limits apply even when connecting directly to the controller.

Existing uploads, host caches and server references remain usable. Previously named libraries are grouped under T5 or T6, and the most recently published upload for each game becomes its latest files. Their immutable IDs and stored content still in use are preserved; unused older files are cleaned up automatically. Legacy source-host imports retain their existing IDs and can complete through their original workflow.

Ports and network bindings

Reservations include host ID, local bind address, protocol and port. Wildcard binds conflict with concrete addresses on the same host. UDP and TCP reservations for one launch commit together; the same number can be used independently on another host. Optional public/NAT address and port are recorded separately and do not change local Docker bindings. EC does not configure external NAT/firewalls.

Provisioning progress and diagnostics

Native launches distinguish initial eligibility evaluation from an actual wait for startup capacity. The operation page records payload preparation, command submission, agent acknowledgement and execution, and separate container, game, map and Plutonium revision checks. Evidence remains available after completion.

The Host agent execution panel reads the durable command state directly, including its queue time and latest report time. Agent receipts are recorded in the provisioning history when received; they do not wait for a fleet scan. An agent command marked completed still requires EC readiness verification. The Controller activity panel shows current observations and their elapsed time while the operation is active. It is a live snapshot, not historical proof of which task delayed a previous deployment.

Health checks

Fleet health checks query RCON even when container inspection is unavailable. A fresh successful RCON reply can establish game health while container state stays unknown; confirmed stopped, unhealthy, and restarting states still take precedence. Agent 1.4.4 retries transient failures of the read-only status query up to three times, with 250 ms and 500 ms delays. Commands that change the game are not retried. Status replies still expire normally, and native timeout diagnostics are retained.

Background observations

Campaign, beta and maintenance observers run independently of provisioning. New requests and completed agent commands wake provisioning; an arrival during a pass schedules another pass. Routine file drift scans run every five minutes; manual checks bypass that interval. Agent 1.4.9 supports one hash manifest per drift scan, avoiding separate list/stat/read operations for unchanged files. Older agents use the existing file checks. Overlapping identical native observations share one operation until it completes. Routine status monitoring and maintenance additionally share successful health results for up to 30 seconds from the request start, preserving the original observation time. Changed server identity or lifecycle intent invalidates reuse. Manual checks and disruption safety checks continue to request fresh evidence. Status scans allow two checks per host, so a slow host cannot stall the fleet. Accepted or running health operations are withheld from the other health lane for 30 seconds; an expired replay lease allows recovery with the original ID. Agent 1.4.10 includes generation-tagged container states in its 15-second heartbeat, using the same Docker inspection as resource sampling (including stopped containers). Routine monitoring uses samples younger than 30 seconds, preserving their sample time and falling back to inspection when evidence is missing, stale, or predates lifecycle changes. RCON status queries share one operation per host, with two concurrent local probes and a 20-second batch deadline. Individual failures remain visible per server. Both additions are capability-gated; older agents retain individual inspection and RCON operations. Agent 1.4.11 caches parsed library manifests during inventory collection until their modification time, size, or file identity changes. Deleted manifests leave the cache, while transfer progress and partial-file sizes are sampled each time. Launch and download integrity verification continue to read from disk. Agent 1.4.5 reserves two observation workers for instance health and read-only RCON, and one for files, logs, and host inventory. Background reads arrive in batches of up to four. File receipt uploads release the instance lock before contacting the controller. Older agents retain the shared read lane until upgraded. Startup and disruption safety gates remain enforced at command admission.

Older completed operations whose progress snapshots were already discarded cannot recover missing check results from this change. Their surviving events and host command records remain available without inventing verification times.

Read operations during downloads

Agent 1.4.0 separates read-only inspections, file/log reads, and the exact RCON commands status, getstatus, and version from the serial mutation queue. Two bounded read lanes can run during runtime or library downloads. Instance mutations exclude reads of that instance; credential rotation and host cleanup wait for active reads. Both lanes retain durable operation IDs, generation checks, and the updater activity lock. Older agents continue using the combined queue.

Agents long-poll GET /api/agent/operations?lane=read|write&wait=20; requests without those parameters retain immediate combined delivery. The controller wakes pending polls when work is queued and rechecks authentication after waiting. Long polls release the updater lock; execution reacquires it and rechecks update state before accepting work.

Browser game-folder uploads transfer up to three files concurrently. Chunks of one file remain serial and checksum-verified. Publication waits for admitted file transfers and prevents subsequent writes; pause/resume still uses acknowledged per-file offsets. Host downloads retain their existing bounded concurrency.

Cancel an unfinished creation

On the native creation operation page, Cancel creation and remove server stops the launch and queues removal of its exact instance and generation. It also works when readiness checks are stuck or the creation failed. This explicit cancellation skips player-count checks on both the controller and agent, even if players have joined. Completed creations use the normal safe removal flow.

The cancelled operation links to a durable cleanup operation with agent receipts and progress. An unsubmitted launch releases its reservations immediately. Once an agent command was queued, ports and the instance identity remain reserved until the agent confirms container and instance-file removal. Shared game libraries, runtime images, saved server keys, and operation history are retained. Repeated cancellation requests return the same cleanup operation. A queued normal removal waiting for occupancy is superseded by this cleanup.

Remote cancellation requires agent 1.3.0 or later (instance.cancel-create capability). An older agent is never sent the unsupported cleanup command; the cleanup page explains that it is waiting for an update. Offline or disabled hosts retain the cleanup request until they reconnect or are enabled. Heartbeats notify an executing create to stop between Docker commands; an already executing Docker command must return or time out before cleanup can run. Cancellation does not require the original creation to acknowledge completion. The agent records a removal tombstone before cleanup so a delayed creation command cannot recreate the instance, and resumes interrupted cleanup under the same operation ID.

While cancellation requires a newer agent, the controller pauses all command delivery to that host, including background observations and command replays. Older agents hold their update lock across a whole command batch; continuing to feed them work could otherwise starve the updater. Already delivered work can finish and report normally, and heartbeats continue. The next automatic update check can install the release once the batch has drained. When a heartbeat advertises cancellation support, delivery resumes with cancellation cleanup first. This controller-side hold also recovers cancellations already waiting on older agents. Hosts without automatic updates need the current installer.

Development and validation

npm run agent:package
npm run check
npm test
npm run test:agent
npm run build

The build packages the agent, standalone updater and installer into the static release, generates the agent/updater checksum manifest and serves those same assets in the Node-adapter image. Development serves the freshly packaged static release even when an older production build exists. The Python programs use only the standard library.

Automated tests cover enrollment/authentication, rotation/revocation, host readiness, scoped ports, immutable library publication, streaming/ranges, checksums, interrupted downloads, instance generation ownership, local occupancy holds, controller restart/reconnect, ambiguous results, occupied campaign staging, activation before images, collections and draining. Host deletion tests cover ownership guards, interrupted cleanup, uninstall receipts, credential revocation and retention of controller uploads/history. Docker/systemd/UDP and host observations are simulated in these tests.

A real Debian/Docker/Plutonium acceptance run is still required before production rollout. Automated simulations do not establish successful deployment on real game hosts. Run:

• Installer on Debian 12 and 13 amd64, then rerun without changing identity/data.
• Bootstrap an existing host with the current installer, then publish a newer compatible agent release. Verify a timer-driven update, its authenticated heartbeat and unchanged game containers. Exercise an active-operation hold, bad checksum, failed startup/heartbeat rollback, controller downgrade, network loss and host reboot during update. Verify updater protocol mismatch requests an installer rerun.
• T5 and T6 deployments and a mixed collection from a fresh destination host.
• Warm the Plutonium runtime cache, then create and rebuild T5 and T6 lobbies; verify the running revision and compare first-start downloads and launch times with an empty cache. Test missing, corrupt and interrupted cache preparation, a newer production revision, and a custom runtime image. Confirm launch-file overrides survive seeding, lobby writes cannot alter the cache or another lobby, and cache refresh leaves running lobbies unchanged. Queue a deployment during preparation and verify that downloading yields and later resumes. Interrupt preparation by stopping the agent and verify that retry removes abandoned temporary files while retaining completed downloads.
• Start, stop, restart, rebuild, destroy; verify RCON, map and runtime revision.
• Read console and game logs, including Plutainer's relative game-log links.
• File campaign and rollback, with a player joining between controller check and agent activation, and during file activation before restart; verify that the agent holds disruption.
• Drain occupied lobbies and verify shared libraries survive their retirement.
• Delete a drained host; verify its game copies, data directory, agent and cleanup services, update timer, configuration and executables are removed, controller uploads/history stay, and unrelated Docker workloads continue. Repeat with connection loss during cleanup and after the final receipt, and a reboot during data removal.
• Corrupt/interrupted library downloads, resumed transfer and verified promotion.
• Controller restart, agent restart and network loss before execution, during execution and after execution but before acknowledgement. Verify the same operation IDs and absence of duplicate containers/restarts.