electric-cherry/docs/native-hosts.md

# Native host management

Electric Cherry controls Docker/Plutainer through an outbound host agent. A native
instance never uses Komodo or a filesystem mounted into the controller. The
controller owns desired state, campaigns, collections, safety queues and history;
the agent owns local execution and its durable operation journal.

Fleet health checks query RCON even when container inspection is unavailable. A
fresh successful RCON reply can establish game health while container state stays
unknown; confirmed stopped, unhealthy, and restarting states still take precedence.
Agent 1.4.4 retries transient failures of the read-only `status` query up to three
times, with 250 ms and 500 ms delays. Commands that change the game are not retried.
Status replies still expire normally, and native timeout diagnostics are retained.

## Install and launch

1. Run the controller with persistent `/data` storage. Set `EC_PUBLIC_URL` to its
   HTTPS origin and proxy TLS to port 3000. Set a random `EC_SETUP_TOKEN` and create
   the first administrator at `/setup`. Management pages/APIs require user login
   and roles; agent endpoints authenticate their own host credentials. See
   [Users and access](authentication.md) for setup and password recovery.
2. Open **Hosts → Add Host**, choose a name, local game-port range and bind address,
   and generate an enrollment. Run the displayed command on Debian 12/13 amd64.
3. The installer validates the platform, installs Python 3 and an absent Docker
   installation, verifies the downloaded agent and updater SHA-256 over HTTPS,
   enrolls it and enables `electric-cherry-agent.service` and its automatic update
   timer. It needs no Git, Node.js or local build.
4. In **Game Files**, upload your complete local folder under T5 or T6. Include
   the DLC files in that folder; there is no separate DLC declaration, library
   name or version label. The upload becomes the latest files for that game after
   verification. Uploading also works before enrolling any host.
5. Hosts automatically prepare the configured Plutainer runtime and the latest
   verified T5 and T6 files whenever available. **Automatic host setup** reports
   progress and errors. Wait for verification before launching. Hosts report
   connection and readiness separately; deployment also checks the particular
   game files, image, capacity and ports required for the launch.
6. Launch a server or collection. The host, latest game files and default
   runtime are selected automatically after checking capacity. An advanced runtime override is
   available. If the latest files change after previewing, refresh the preview
   before launching.
   EC selects lobby IDs and reserves UDP/TCP ports atomically.
   Saved Plutonium server keys remain associated with their exact server names.

A fresh destination host downloads centrally stored game files without manual
copying or per-container setup. EC does not supply the game content itself.

The installer uses Debian's Docker packages: `docker.io` on Debian 12, plus
`docker-cli` on Debian 13. Rerunning it after an earlier Debian 13 install that
omitted the client adds the missing client without restarting the existing engine.
An already available Docker installation (including Docker CE) is reused. If the
daemon is unavailable, the installer reports the Docker error and stops before
enrollment; inspect `systemctl status docker` and `journalctl -u docker`.

The controller reconciles automatic setup on agent heartbeats and operation
polls. It first pulls `PLUTAINER_IMAGE` (default
`ghcr.io/ayymoss/plutainer:latest`), then prepares the latest verified upload for
T5 and for T6 when available, one setup operation at a time. Uploading new files
or enrolling another host needs no manual propagation. Setup waits for an active, connected,
healthy host with sufficient storage. Failed operations retry automatically with
backoff from one minute to fifteen minutes; their errors remain visible in the
host's setup panel and operation history. Disconnected hosts resume when they
reconnect. Draining and disabled hosts do not receive new automatic setup work.

An automatic download that has not been delivered to the agent can be replaced
when a newer upload becomes available. A delivered operation finishes safely
before the host prepares the latest files. Failed or incomplete uploads never
replace the latest verified files.

Verified previous uploads remain cached and running servers keep their existing
files and runtime; uploading new files does not restart servers. Automatic setup
ensures that the configured image is present; it does not refresh an already
cached mutable image tag. Advanced controls on the
host and game-file pages can explicitly pull a custom runtime, prepare an older
upload or retry preparation immediately.

The default host root is `/var/lib/electric-cherry`; installer `--data-dir` changes
it. Use a dedicated, root-owned directory. For a separate disk, use a child of its
mount point (for example `/mnt/games/electric-cherry`), with no nested mounts, so
host deletion can remove the entire directory. The installer records ownership
and refuses to adopt unrelated files. Data is separated into `libraries/<game>/<version>/content`,
`instances/<UUID>/<generation>/app`, and `agent/operations.sqlite3`. Base libraries
are mounted read-only and instance deletion removes only that instance's files.
The controller stores large files under `GAME_FILES_ROOT` (default beside the
SQLite database), with manifests/checksums and operation metadata in SQLite.

The agent uses Plutainer's documented [game-file and app mounts](https://github.com/Ayymoss/Plutainer/blob/main/docs/volumes-and-configs.md).
Its log reader supports relative log links that resolve inside the instance app
directory, while file writes reject symlinks and paths outside that directory.

## Automatic agent updates

New installations enable automatic agent updates by default. Each host checks
its controller's release over HTTPS about every five minutes, beginning about
two minutes after boot, with up to 30 seconds of jitter. Updating the controller
therefore makes its compatible, newer agent release available to enrolled hosts.
There is no controller command to schedule and no update button to press.

**Existing hosts need the current installer once.** From the host's **Agent
access**, generate an installer command and rerun it on the same machine, with
the same controller and data directory. Do not use `--reenroll` for this upgrade.
The installer preserves host identity, credentials, game files and running
containers. Agent version 1.2.0 includes automatic update support; the host page
shows **Setup required** until a fresh heartbeat confirms the updater is installed.

The **Automatic agent updates** panel reports the running version, target
version, last check, progress and any error. Offline hosts show their last
reported status. Deferred updates report why they are waiting: for example an
active agent operation, an unavailable controller or a newer updater bootstrap.
Checks also defer while the agent service is stopped; an update does not start a
service that was stopped outside the update workflow.
The independent updater holds the same execution lock, verifies the release
checksum, version and supported protocol, replaces agent code, and restarts the
agent service. It never restarts Docker or game containers. It confirms an
authenticated heartbeat from the new agent process before accepting the update;
failure restores the previous agent code. Its durable journal permits recovery
after a host reboot or interrupted update.

The updater only installs newer compatible versions; rolling the controller back
does not downgrade agents. A rejected version/checksum pair stays blocked until
the controller publishes a different pair that is still newer than the installed
agent. Inspect the reported error when a host
shows **Needs attention**. The updater itself is a stable bootstrap component:
it changes only when the installer is rerun. If a future release requires an
unsupported updater protocol, rerun that release's installer on the host.

The Inventory panel shows CPU usage with agent 1.4.2 and later. It measures
host-wide busy time across all cores on a 0–100% scale between inventory samples
(normally about 15 seconds). Idle and I/O wait are excluded; stolen CPU time
counts as unavailable capacity. The first sample after startup, unavailable
counters, and older agents show **Not reported**. Offline hosts retain their
last reported value. The controller retains CPU and memory samples for seven days (missing measurements
remain unknown). Scheduling checks capacity against the latest heartbeat and uses
the last five minutes to rank eligible hosts; the Inventory panel shows the latest
reading.

New servers are placed automatically. Preview and submission perform local
capacity checks without contacting agents: a connected, active host must have
verified game files, its runtime, storage, a free UDP/TCP port, and valid CPU and
memory measurements. Hosts with stale or missing measurements cannot accept new
servers. Existing API clients may still explicitly constrain a launch to a host.

Admission reserves a provisional host and port atomically, including collection
launches. Agent **1.4.6** fixes container discovery for resource sampling (the
earlier collector omitted `label=` from its Docker filter). It samples each
managed running container with a bounded
[`docker stats`](https://docs.docker.com/reference/cli/docker/container/stats/)
batch during inventory collection. CPU is normalized to cores (100% Docker CPU
is one core); memory is the Linux working usage reported by Docker, excluding
reclaimable cache. Missing or failed samples stay unknown instead of becoming
zero usage. Samples must match the host's live instance and generation.
Measurements arrive through `/api/agent/heartbeat`, normally every 15 seconds
plus collection and delivery time, independently of RCON or inspection operations.
Sampling failures appear on the host's Learned server capacity panel. Existing
containers can begin contributing running samples as soon as the agent updates;
startup samples require observing the first five minutes of a container's run.

The controller retains per-instance minute peaks for 24 hours and caches learned
CPU/memory allowances. The first five minutes after a container starts count as
startup; later samples train the running allowance. Startup uses the observed
maximum and running uses the 95th percentile of minute peaks, with **no per-server
headroom**. Each calculation considers up to the latest 10,000 minute peaks.
Host-local evidence is preferred to account for different hardware; new hosts
can borrow the fleet's learned allowance. Startup allowances are never lower
than running allowances. The host page shows both allowances and their source.
Expand **View samples** in Learned server capacity to inspect retained minute
peaks for the last 24 hours. Filter by startup/running phase or switch from this
host to the entire fleet. Rows show time, server and generation, CPU cores, and
memory in MiB; CPU and memory peaks may come from different
heartbeats. Results are paginated, and **Refresh samples** loads the latest stored
measurements without contacting agents. The allowance count may lag the table
until the next calculation or reflect a different source.

Lower allowances require at least three startup minute samples spanning two
minutes, or twenty running minute samples spanning five minutes. Until then,
**0.5 core and 1 GiB** are temporary cold-start allowances, raised immediately
if observations exceed them. They are replaced by measured allowances once the
evidence threshold is met; a tiny 0.01-core floor prevents zero-CPU admission.
Models expire after 24 hours without measurements. Recalculation happens during
heartbeat ingestion, at most once per minute per host/fleet phase; new peaks
raise the cache immediately. Launch admission reads only the cached model.

Scheduling still leaves **5% host CPU and 500 MB RAM (500,000,000 bytes)** for the OS and bursts.
These allowances are scheduling estimates, not Docker limits. Collection admission
budgets new and never-submitted queued servers at their running allowance, rather
than assuming the entire collection starts simultaneously. Current host CPU and
available memory account for existing containers and other processes. Observed
containers are not topped up to historical startup or running peaks, even while
awaiting health verification. Established instances are covered by host usage
when individual container measurements are unavailable.

Dispatch checks the next launch's startup allowance against live host usage.
Other never-submitted queued launches wait their turn and retain their port
reservations. Submitted or uncertain launches missing a matching-generation
measurement still consume a full startup allowance until telemetry accounts for
them. This lets collections queue based on running demand and start as capacity
becomes available, without stacking every queued server's startup peak.
Capacity errors report projected usage, usable capacity and reservations.

When startup capacity is available, provisioning checks each eligible host’s
current CPU and available memory against reservations and headroom.
Among hosts that fit, it chooses the lowest projected CPU/memory utilization,
using the busier of the latest reading and the five-minute average for ranking.
A historical resource dip cannot block a host whose current capacity has recovered.
Provisioning moves the provisional reservation atomically and records the final host in the launch recipe. Host network defaults
are resolved again for the final host; explicit overrides are preserved. If no
host qualifies, the request waits and is checked again. Once remote staging or
creation has been submitted, placement stays fixed to prevent duplicate instances.
Cancelling an unsubmitted launch releases its reservation; uncertain remote work
retains capacity until cleanup is confirmed.

When publishing changed agent code, bump its numeric `VERSION` in
`agent/ec_agent.py` and rebuild the controller. Packaging derives the manifest
version and checksums from the sources; an already installed version is never
replaced just because its bytes changed.

The installer places the updater at
`/usr/local/libexec/electric-cherry-update.py` and enables
`electric-cherry-update.timer`, which starts the oneshot
`electric-cherry-update.service`. Its state is kept under `/etc/electric-cherry`,
including `update-status.json`, `update/journal.json`, `agent-health.json` and
the shared `update.lock`. Check the timer and worker logs on the host with:

```sh
systemctl status electric-cherry-update.timer
journalctl -u electric-cherry-update.service -n 100
```

## Identity, transport and credentials

Enrollment tokens are random, hashed at rest, single-use and expire after 15
minutes. Consumption and credential issuance are transactional. Enrollment only
works at the enrollment endpoint. Each host receives a distinct bearer credential
and the controller only stores its hash. Host and enrollment bearer credentials
are rejected outside `/api/agent/`.

The agent uses HTTPS, operating-system certificate validation, outbound heartbeats
and polling; Docker has no exposed remote API. Agent endpoints reject HTTP unless
`EC_AGENT_ALLOW_HTTP=true` is explicitly enabled for development. The agent's
insecure development option is limited to loopback. Public release assets contain
no credentials. The release checksum is authenticated by the controller's HTTPS
origin; there is no separate offline release-signing infrastructure.

Browser sessions do not authorize agents. Similarly, host credentials never grant
access to operator pages. Game telemetry uses a separate `EC_TELEMETRY_TOKEN`,
configured in each reporter's authorization header; see the
[telemetry contract](beta-test-plans.md#game-reporter-protocol).

Hosts offers credential rotation and revocation. Rotation persists the new secret
on the agent before switching controller credentials, so a lost response can be
retried using the same proposed secret. Revocation affects one host only and does
not stop its running game containers. Re-enrollment requires a fresh token and an
explicit installer `--reenroll` when local credentials already exist.

API outline:

- `GET /api/hosts/:id`: host details, operation history and deletion blockers.
- `DELETE /api/hosts/:id`: administrator requests cleanup of an unused host;
  returns `202` while remote removal is pending, `200` for an unenrolled host,
  or `409` when the agent needs updating or unresolved ownership remains.
- `POST /api/agent/enroll`: consume enrollment token and return host credentials.
- `GET /api/agent/release/manifest.json`: public agent version, protocol and
  checksum plus `updaterProtocolVersion` and `updaterSha256` for installer bootstrap.
- `GET /api/agent/release/ec-agent.py` and `ec-agent-update.py`: public release
  source, verified against that HTTPS manifest before installation.
- `POST /api/agent/heartbeat`: authenticated inventory, current operation and caches.
- `GET /api/agent/operations`: deliver/reconcile durable operation IDs.
- `POST /api/agent/operations/:id`: authenticated acceptance, progress or result.
- `POST /api/agent/decommission/:id`: cleanup worker reports file removal and
  agent uninstall using its own operation-scoped receipt credential.
- `GET /api/game-files`: returns `{libraries, hosts}`. `libraries` always contains
  two game slots, T5 and T6, including empty slots. Each slot has a nullable
  `latestVersionId` identifying its latest verified upload, plus upload history.
- `POST /api/game-files`: administrator creates a folder upload with
  `{game: "t5" | "t6", files: [{path, size, lastModified}]}`. Upload IDs and
  internal revision metadata are generated automatically; no library name,
  version label or description is required.
- `GET /api/game-files/:id/upload`: durable per-file offsets and publication status.
- `PUT /api/game-files/:id/upload/files/:path`: administrator uploads a raw chunk
  with `X-Upload-Offset`, `X-Upload-Length` and `X-Content-SHA256` headers.
- `POST /api/game-files/:id/upload/complete`: administrator starts asynchronous
  verification/publication; poll the upload endpoint for completion.
- `/api/agent/libraries/...`: authenticated game-file downloads. Legacy
  `/api/agent/imports/...` endpoints remain compatible with already queued imports.

The operation envelope is independent of polling and can be carried over a
persistent transport later. No unrestricted shell operation is available.

## Safety and durable operations

Each remote request has an immutable ID and payload, plus an instance UUID and
generation where applicable. Reusing a key with a different request is rejected.
The controller records remote IDs before dispatch. The agent persists accepted,
running, completed, failed and unknown states in SQLite. Lost acknowledgments
return the recorded result; restarting the controller never creates a new
mutation simply because the previous reply was lost.

The agent reconciles ambiguous results using owned Docker labels, generations and
persisted execution checkpoints when those observations prove the outcome.
Otherwise the result remains **Unknown** and holds dependent work. Running alone
is insufficient evidence that a particular restart happened. EC neither assumes
an unreachable server stopped nor launches a duplicate replacement elsewhere.

Campaign bytes are staged and verified while players remain connected. Only
Provisioning can dispatch activation. It checks source eligibility and shared
capacity, and the agent repeats a fresh local RCON occupancy check immediately
before activation/restart, stop, removal or rebuild. Busy or unverifiable lobbies
wait indefinitely, including multi-hour Zombies sessions. Before images returned
by the agent populate existing campaign/beta rollback history. A beta cannot be
closed while its remote activation receipt is still unresolved.

An empty Fleet card does not replace this final host-local check: the card may
use telemetry, and the controller's RCON parser is more permissive than the
agent's disruption guard. Agent 1.4.1 includes the local RCON failure reason in
the operation's hold message (configuration, timeout, unreachable endpoint,
authentication, invalid packet, or an unrecognized player count). For a parser
hold, inspect the raw `status` reply in **Diagnose connection**. Unknown counts
remain held; RCON availability alone does not prove the lobby is empty.
It also recognizes T6's `num score bot ping` header, which earlier agents
rejected even when the complete player table was empty.

Rebuild uses a new generation, fresh instance storage and the latest promoted
configuration/scripts/plugins. It retains the selected game-file version and
runtime image. Old generation commands cannot target the replacement. Start,
restart and create complete only after post-operation health evidence; creation
and rebuild also verify the selected map and running Plutonium revision. The
base-game library and runtime revision are independent records.

**Drain** blocks new deployments and queues safe retirement of existing lobbies.
Active beta ownership and other operations continue to gate retirement. Players
may still join until an eligible empty moment; draining does not kick them.
**Disable** holds agent dispatch. Neither mode, controller downtime nor agent
communication loss stops unrelated running containers.

Administrators can choose **Delete host** on a host card or at the top of its
detail page. The confirmation lists anything that must be resolved first:
registered servers, reserved ports, or operations already sent to the agent.
Drain the host or remove its servers, then let outstanding operations reconcile.
An unenrolled host is removed immediately. An enrolled host stays visible as
**Deleting** until its agent confirms cleanup. Offline hosts wait for their next
connection; their absence is never treated as proof of removal. Existing agents
need the current installer: generate a fresh command under **Agent access**, rerun it
on the host, and wait for a heartbeat advertising uninstall support.

Deletion blocks new operations, cancels queued commands that were never delivered,
and invalidates unused installer links. A separate cleanup service removes the
host's downloaded game libraries, instance files, agent journal and configuration,
owned containers, and recorded runtime images unused by other containers. It
stops and removes the agent service and executable, automatic update timer,
update service, updater executable and update state, then removes its own files.
The controller hides the host and revokes its ordinary credentials only after
receiving the final uninstall receipt. Adding the machine again requires a new
host enrollment.

Game folders uploaded through Electric Cherry belong to the controller. Those
original uploads remain available for other hosts and future deployments; only
the deleted host's copies are removed. Controller operation history is also kept.
Unrelated containers, files and shared dependencies such as Docker and Python
are preserved. Historical entries in shared system logs are not erased.

Cleanup retries after connection loss or partial file removal. Failures before
handoff offer **Retry cleanup** after the installation is repaired; failures in
the independent worker retry automatically. The worker stays installed until
the controller acknowledges data cleanup. After removing its final files it
retries the completion receipt from memory. A power loss in that final window
can leave the controller awaiting confirmation even though removal succeeded;
inspect the machine if that status persists. The controller does not guess.

## Game files and ports

Game Files always shows T5 and T6, each with its own upload button and latest
verified files. Successful uploads become latest in publication completion
order, not by their start date or an entered label. A newer upload still being
transferred or verified leaves the current files available. Previous uploads
appear in history, and interrupted uploads can be resumed separately.

Choose the actual game folder, not its parent, a ZIP archive or the Plutonium
launcher data folder. The selected folder's name is removed once; nested paths
such as `zone/english/...` and `main/...` stay intact. Plutonium's
[T5 server guide](https://plutonium.pw/docs/server/t5/setting-up-a-server/) and
[T6 server guide](https://plutonium.pw/docs/server/t6/setting-up-a-server/) use a
copy of the game directory, separately from the launcher/runtime. The uploader
preserves the supplied files rather than attempting to infer installed maps from
a manually maintained DLC list.

Folder uploads use chunks of at most 8 MiB, each checked against its SHA-256 before
being appended to temporary controller storage. Memory usage does not grow with
the size of an individual game file. Pause/retry uses the controller's saved file
offsets. After a page reload, choose **Resume upload** and reselect the original,
unchanged folder; relative paths, sizes and modification times must match. Keep
the page open while transferring. Once publication starts, the controller hashes
every complete file and publishes the directory atomically; closing the browser
does not stop that verification. An interrupted controller can retry publication.
Verified uploads are immutable; changing files means uploading another folder
for the same game. Hosts first download every file with up to four concurrent
transfers and HTTP Range resumption. After all transfers finish, a separate
verification pass reads every file from disk and checks its complete SHA-256,
including any resumed prefix. Only then is the complete cache promoted atomically.
Corrupt staged files are removed so the next attempt downloads them again.
Separating the phases removes checksum work from the transfer loop and exposes
disk-verification time separately; it does not guarantee a shorter total duration.

The Game Files and host setup pages show **Download** and **Verify** as separate
steps, each with its own progress. Elapsed time records active work across
resumptions and stops when work finishes or fails. Remaining time and speed use
the current step's measured progress; a download estimate does not include the
verification pass that follows. Estimates appear after enough progress is
measured and are hidden for stale or offline hosts. Existing agents without
timing reports still show byte progress. Agent 1.2.2 provides phase timing;
automatic updates install it between host operations.
Phase transitions are reported promptly and periodic heartbeats carry progress;
controller round trips do not stall every transferred chunk.
Corrupt or incomplete content never becomes Ready. Cache inventory is reconciled
on heartbeats, including vanished caches. The latest published version of each game
is always retained. With agent 1.4.7 or newer, an older host copy is automatically
removed once no live instance reservation on that host references it. Stopped,
provisioning, and rebuilding instances still pin their version. The agent also
refuses deletion if any Docker container (including stopped containers) mounts
those files. Cleanup includes partial downloads and retries after interruption.

The controller removes an older version's files after every host has reported its
copy absent and no instance or unresolved operation references that version.
Offline hosts keep their last reported copies pinned until they reconnect.
Upload metadata, manifests, and operation history remain visible; removed files
cannot be downloaded again without a new upload. Cleanup runs on heartbeats,
successful publication, and the controller background worker, without needing
an open Game Files page.

The controller and reverse proxy must accept 8 MiB upload requests and folder
manifests up to 32 MiB. Compose sets `BODY_SIZE_LIMIT=Infinity`; application routes
still enforce these limits, declared file sizes and checksums. Set a proxy body
limit of at least 32 MiB (for example `client_max_body_size 32m` in nginx).
Existing uploads, host caches and server references remain usable. Previously
named libraries are grouped under T5 or T6, and the most recently published
upload for each game becomes its latest files. Their immutable IDs and stored
content still in use are preserved; unused older files are cleaned up automatically.
Legacy source-host imports retain their existing IDs and can complete through
their original workflow.

Reservations include host ID, local bind address, protocol and port. Wildcard
binds conflict with concrete addresses on the same host. UDP and TCP reservations
for one launch commit together; the same number can be used independently on
another host. Optional public/NAT address and port are recorded separately and do
not change local Docker bindings. EC does not configure external NAT/firewalls.

## Development and validation

```sh
npm run agent:package
npm run check
npm test
npm run test:agent
npm run build
```

The build packages the agent, standalone updater and installer into the static
release, generates the agent/updater checksum manifest and serves those same
assets in the Node-adapter image. Development serves the freshly packaged static
release even when an older production build exists. The Python programs use only
the standard library.

Automated tests cover enrollment/authentication, rotation/revocation, host
readiness, scoped ports, immutable library publication, streaming/ranges,
checksums, interrupted downloads, instance generation ownership, local occupancy
holds, controller restart/reconnect, ambiguous results, occupied campaign staging,
activation before images, collections and draining. Host deletion tests cover
ownership guards, interrupted cleanup, uninstall receipts, credential revocation
and retention of controller uploads/history. Docker/systemd/UDP and host
observations are simulated in these tests.

A real Debian/Docker/Plutonium acceptance run is still required before production
rollout. This workspace has no Docker engine or Debian game host, so automated
coverage is not evidence that a real T5/T6 server has been deployed here. Run:

- Installer on Debian 12 and 13 amd64, then rerun without changing identity/data.
- Bootstrap an existing host with the current installer, then publish a newer
  compatible agent release. Verify a timer-driven update, its authenticated
  heartbeat and unchanged game containers. Exercise an active-operation hold,
  bad checksum, failed startup/heartbeat rollback, controller downgrade, network
  loss and host reboot during update. Verify updater protocol mismatch requests
  an installer rerun.
- T5 and T6 deployments and a mixed collection from a fresh destination host.
- Start, stop, restart, rebuild, destroy; verify RCON, map and runtime revision.
- Read console and game logs, including Plutainer's relative game-log links.
- File campaign and rollback, with a player joining between controller check and
  agent activation, and during file activation before restart; verify that the
  agent holds disruption.
- Drain occupied lobbies and verify shared libraries survive their retirement.
- Delete a drained host; verify its game copies, data directory, agent and cleanup
  services, update timer, configuration and executables are removed, controller uploads/history stay,
  and unrelated Docker workloads continue. Repeat with connection loss during
  cleanup and after the final receipt, and a reboot during data removal.
- Corrupt/interrupted library downloads, resumed transfer and verified promotion.
- Controller restart, agent restart and network loss before execution, during
  execution and after execution but before acknowledgement. Verify the same
  operation IDs and absence of duplicate containers/restarts.

Legacy backend code remains for existing database registrations. New native
launches do not require its configuration, API, network or mounted directories.

### Provisioning progress and diagnostics

Native launches distinguish initial eligibility evaluation from an actual wait
for startup capacity. The operation page records payload preparation, command
submission, agent acknowledgement and execution, and separate container, game,
map and Plutonium revision checks. Native actions use host-agent steps rather
than legacy Komodo deployment steps. Evidence remains available after completion.

The **Host agent execution** panel reads the durable command state directly,
including its queue time and latest report time. Agent receipts are recorded in
the provisioning history when received; they do not wait for a fleet scan.
An agent command marked completed still requires EC readiness verification.
The **Controller activity** panel shows current observations and their elapsed
time while the operation is active. It is a live snapshot, not historical proof
of which task delayed a previous deployment.

Campaign, beta and maintenance observers run independently of provisioning.
New requests and completed agent commands wake provisioning; an arrival during a
pass schedules another pass. Routine file drift scans run every five minutes;
manual checks bypass that interval. Overlapping identical native observations
share one operation until it completes, without caching results for later checks.
Agent 1.4.5 reserves two observation workers for instance health and read-only RCON,
and one for files, logs, and host inventory. Background reads arrive in batches of
up to four. File receipt uploads release the instance lock before contacting the
controller. Older agents retain the shared read lane until upgraded. Startup and
disruption safety gates remain enforced at command admission.

Older completed operations whose progress snapshots were already discarded
cannot recover missing check results from this change. Their surviving events
and host command records remain available without inventing verification times.

### Cancel an unfinished creation

On the native creation operation page, **Cancel creation and remove server**
stops the launch and queues removal of its exact instance and generation. It
also works when readiness checks are stuck or the creation failed. This explicit
cancellation skips player-count checks on both the controller and agent, even
if players have joined. Completed creations use the normal safe removal flow.

The cancelled operation links to a durable cleanup operation with agent receipts
and progress. An unsubmitted launch releases its reservations immediately. Once
an agent command was queued, ports and the instance identity remain reserved
until the agent confirms container and instance-file removal. Shared game
libraries, runtime images, saved server keys, and operation history are retained.
Repeated cancellation requests return the same cleanup operation. A queued
normal removal waiting for occupancy is superseded by this cleanup.

Remote cancellation requires agent 1.3.0 or later (`instance.cancel-create`
capability). An older agent is never sent the unsupported cleanup command;
the cleanup page explains that it is waiting for an update. Offline or disabled
hosts retain the cleanup request until they reconnect or are enabled. Heartbeats
notify an executing create to stop between Docker commands; an already executing
Docker command must return or time out before cleanup can run. Cancellation
does not require the original creation to acknowledge completion. The agent
records a removal tombstone before cleanup so a delayed creation command cannot
recreate the instance, and resumes interrupted cleanup under the same operation ID.

While cancellation requires a newer agent, the controller pauses all command
delivery to that host, including background observations and command replays.
Older agents hold their update lock across a whole command batch; continuing to
feed them work could otherwise starve the updater. Already delivered work can
finish and report normally, and heartbeats continue. The next automatic update
check can install the release once the batch has drained. When a heartbeat
advertises cancellation support, delivery resumes with cancellation cleanup
first. This controller-side hold also recovers cancellations already waiting
on older agents. Hosts without automatic updates need the current installer.

## Responsive observations (agent 1.4.0)

Agent 1.4.0 separates read-only inspections, file/log reads, and the exact RCON
commands `status`, `getstatus`, and `version` from the serial mutation queue.
Two bounded read lanes can run during runtime or library downloads. Instance
mutations exclude reads of that instance; credential rotation and host cleanup
wait for active reads. Both lanes retain durable operation IDs, generation checks,
and the updater activity lock. Older agents continue using the combined queue.

Agents long-poll `GET /api/agent/operations?lane=read|write&wait=20`; requests
without those parameters retain immediate combined delivery. The controller wakes
pending polls when work is queued and rechecks authentication after waiting.
Long polls release the updater lock; execution reacquires it and rechecks update
state before accepting work.

Browser game-folder uploads transfer up to three files concurrently. Chunks of
one file remain serial and checksum-verified. Publication waits for admitted file
transfers and prevents subsequent writes; pause/resume still uses acknowledged
per-file offsets. Host downloads retain their existing bounded concurrency.