electric-cherry/docs/guides/native-hosts.md

# Native host management

Electric Cherry controls Docker/Plutainer through an outbound host agent. The controller
does not need mounted game-server files. It owns desired state, campaigns, collections, safety queues and history;
the agent owns local execution and its durable operation journal.

## Install and launch

1. Run the controller with persistent `/data` storage. Set `EC_PUBLIC_URL` to its
   HTTPS origin and proxy TLS to port 3000. Set a random `EC_SETUP_TOKEN` and create
   the first administrator at `/setup`. Management pages/APIs require user login
   and roles; agent endpoints authenticate their own host credentials. See
   [Users and access](authentication.md) for setup and password recovery.
2. Open **Hosts → Add Host**, choose a name, local game-port range and bind address,
   and generate an enrollment. Run the displayed command on the Debian 12/13
   amd64 host. It downloads the installer directly from the public repository;
   [release trust](#provision-release-trust) is fetched automatically.
3. The installer validates the platform, installs Python 3, `gpgv` and an absent Docker
   installation, verifies the release signature and agent/updater SHA-256 checksums,
   enrolls it and enables `electric-cherry-agent.service` and its automatic update
   timer. It needs no Git, Node.js or local build.
4. In **Game Files**, upload your complete local folder under T5 or T6. Include
   the DLC files in that folder; there is no separate DLC declaration, library
   name or version label. The upload becomes the latest files for that game after
   verification. Uploading also works before enrolling any host.
5. Hosts automatically prepare the configured Plutainer runtime and the latest
   verified T5 and T6 files whenever available. **Automatic host setup** reports
   progress and errors. Wait for verification before launching. Hosts report
   connection and readiness separately; deployment also checks the particular
   game files, image, capacity and ports required for the launch.
6. Open **Create server**, choose a game and map, and review the automatic
   readiness checks. Launch collections separately from **Fleet → Collections**.
   The latest game files and default runtime are selected automatically.
   **Advanced settings** in the single-server dialog includes a runtime override
   for images an admin has already prepared on the destination host, along with
   optional network overrides. If the latest files change after the checks run,
   let the dialog recheck the setup before launching; **Retry checks** also rechecks it.
   Launching saves the requested lobby IDs. Provisioning selects a host and atomically
   reserves UDP/TCP ports when capacity is available.
   An available Plutonium key from the game pool is reserved for each launch.
   Its assignment survives stops and rebuilds and ends after confirmed removal.

A fresh destination host downloads centrally stored game files without manual
copying or per-container setup. EC does not supply the game content itself.

### Provision release trust

The public Electric Cherry repository is the authority for both the installer
and release-signing public keys. The enrollment command downloads `agent/install.sh`
from that repository over HTTPS, saves it locally, and runs it only after the
download succeeds. No manual key transfer is required.

During installation and **before every update check**, the host downloads the
binary OpenPGP public keyring directly from this fixed URL:

```text
https://gitea.thewrightserver.net/josh/electric-cherry/raw/branch/main/agent/release-signing.gpg
```

The URL is built into the trusted installer and updater. Controller configuration,
release metadata and enrollment credentials cannot change it. The host verifies
HTTPS certificates, rejects redirects and incomplete, empty or oversized key
responses, and requests revalidation with `Cache-Control: no-cache`. It uses the
fresh keyring only for that check and deletes its temporary copy afterward.
There is no fallback to a previous host keyring, cached download or controller key.
If the repository is unavailable, new installations stop and automatic updates
wait for the next check; the installed agent and game containers keep running.
Local recovery of an interrupted update does not require the repository.

This deliberately trusts the repository account, its `main` branch and HTTPS
hosting as authorities for root code on game hosts. They must be outside the
controller's write access: a compromised controller must not also have a token,
mount or server access that can change these repository files. A repository
compromise can replace the key or installer. Before initial enrollment, check that
the command downloads from
`https://gitea.thewrightserver.net/josh/electric-cherry/raw/branch/main/agent/install.sh`,
since a compromised controller UI could display a different command. Host provisioning systems can use that known URL directly.

Before deploying the first signed release, the publisher must export and commit
its **public** key as `agent/release-signing.gpg` on `main`, following the
[signing workflow below](#publishing-signed-releases). No production key is
created by packaging. Until the public key and signed release are published,
installation and updates fail closed. Forks must change the fixed repository URLs
in the installer, updater and enrollment command to their own trusted repository;
there is intentionally no controller setting to override them.

### Docker installation

The installer uses Debian's Docker packages: `docker.io` on Debian 12, plus
`docker-cli` on Debian 13. Rerunning it after an earlier Debian 13 install that
omitted the client adds the missing client without restarting the existing engine.
An already available Docker installation (including Docker CE) is reused. If the
daemon is unavailable, the installer reports the Docker error and stops before
enrollment; inspect `systemctl status docker` and `journalctl -u docker`.

### Automatic host setup

The controller reconciles automatic setup on agent heartbeats. Command polls only
deliver work already queued. Setup first pulls `PLUTAINER_IMAGE` (default
`ghcr.io/ayymoss/plutainer:latest`), then prepares the latest verified upload for
T5 and for T6 when available, one setup operation at a time. After those prerequisites,
capable agents also prepare the optional Plutonium runtime cache described below.
Uploading new files or enrolling another host needs no manual propagation.
Setup waits for an active, connected,
healthy host with sufficient storage. Failed operations retry automatically with
backoff from one minute to fifteen minutes; their errors remain visible in the
host's setup panel and **Hosts → Operation history**. This shared history lists
operations across hosts, newest first, with 25 rows per page and host, target,
and source details. It defaults to **Actions & errors**; **Routine checks** and
**All operations** include successful monitoring reads. Older records may lack
source details.
Disconnected hosts resume when they
reconnect. Draining and disabled hosts do not receive new automatic setup work.

### Operation history retention

**Settings → Routine operation history retention (days)** controls cleanup of
successful monitoring and file-read records. The default is one day; zero keeps
them indefinitely. Actions, failures, negative checks, and unresolved operations
are retained. Agent 1.4.9 also cleans up successful routine journal records after
the controller confirms their final receipt. Older unconfirmed journal entries
remain until a replay supplies that confirmation. Controller cleanup runs every
five minutes and drains eligible records in batches, yielding between batches.

Log-read operation history stores metadata only. The waiting reader receives the
log content directly; saved match chat remains with its match. Background cleanup
also removes log bodies left in controller history by earlier versions.

Agent 1.8.5 adds incremental game-log reads for chat collection. After the initial
tail, the controller requests only appended bytes, handles rotation and truncation,
and buffers incomplete lines. Unchanged logs need no content transfer or chat
parsing. Background collection skips offline hosts, stopped servers and known
empty servers, with one final read after players leave. Opening chat still reads
on demand. Older agents use tail reads until they update.

### New game files and cached versions

An automatic download that has not been delivered to the agent can be replaced
when a newer upload becomes available. A delivered operation finishes safely
before the host prepares the latest files. Failed or incomplete uploads never
replace the latest verified files.

Verified previous uploads remain cached and running servers keep their existing
files and runtime; uploading new files does not restart servers. Automatic setup
ensures that the configured image is present; it does not refresh an already
cached mutable image tag. Advanced controls on the
host and game-file pages can explicitly pull a custom runtime, prepare an older
upload or retry preparation immediately.

### Plutonium runtime cache

Agent 1.8.0 adds a host cache for the files Plutainer normally downloads from
Plutonium on a new container's first start. This is separate from the cached
Docker image and uploaded base game libraries. Automatic preparation warms it
after the image and libraries are ready and refreshes it for new production
revisions. The controller only queues this work for agents that advertise support.
Cache preparation is optional: a missing or corrupt cache does not prevent a
launch from using Plutainer's normal download path.

Cache preparation has lower priority than deployments and other host changes.
If work arrives during a download, the next heartbeat tells the agent to yield
between transfer steps. Preparation resumes under the same operation ID after
that work finishes, reusing files that have already passed verification.

The agent downloads the official production manifest and its listed vendor files,
verifies the SHA-1 hashes published in that manifest, and makes a snapshot
available only after the complete set passes verification. It keeps the cache
under `agent/plutonium-cache` within the host data directory. It does not populate
the cache from existing lobbies, so lobby configuration, keys, custom scripts,
plugins and runtime state are not collected into it. Preparation retains the
current snapshot and the resumable download, removes older snapshots, and clears
abandoned temporary files before checking available storage or publishing the
new snapshot. A failed refresh preserves the previous complete cache.

For new instances and rebuilds using `ghcr.io/ayymoss/plutainer` image references,
the agent seeds `app/runtime/plutonium` from a valid snapshot, including its
matching `cdn_info.json`, before applying the instance's launch files. Each
instance receives its own writable copy; changes made by a lobby or its updater
cannot change the host cache or another lobby. Custom runtime images keep their
existing startup behavior.

Plutainer's updater remains enabled. When a cached revision is current, it can
skip the initial vendor-file download; when production is newer, it can download
the update normally. Launch readiness still verifies the running Plutonium
revision against production. Refreshing the cache does not restart or update
running lobbies. Ordinary restarts retain the instance's existing files and do
not need to seed from the cache again.

Deleting a lobby preserves the host cache for subsequent launches. Deleting the
host removes it with the rest of the managed data directory. This is intended to
reduce repeated first-start downloads; deployment-time improvements still need
measurement on a real Debian/Docker/Plutonium host.

### Storage layout

The default host root is `/var/lib/electric-cherry`; installer `--data-dir` changes
it. Use a dedicated, root-owned directory. For a separate disk, use a child of its
mount point (for example `/mnt/games/electric-cherry`), with no nested mounts, so
host deletion can remove the entire directory. The installer records ownership
and refuses to adopt unrelated files. Data is separated into `libraries/<game>/<version>/content`,
`instances/<UUID>/<generation>/app`, and `agent/operations.sqlite3`. Base libraries
are mounted read-only and instance deletion removes only that instance's files.
The controller stores large files under `GAME_FILES_ROOT` (default beside the
SQLite database), with manifests/checksums and operation metadata in SQLite.

The agent uses Plutainer's documented [game-file and app mounts](https://github.com/Ayymoss/Plutainer/blob/main/docs/volumes-and-configs.md).
Its log reader supports relative log links that resolve inside the instance app
directory, while file writes reject symlinks and paths outside that directory.

## Automatic agent updates

New installations enable automatic agent updates by default. Each host checks
the repository public key and then its controller's release over HTTPS about
every five minutes, beginning about two minutes after boot, with up to 30 seconds
of jitter. Updating the controller
therefore makes its compatible, newer signed agent release available to enrolled hosts.
There is no controller command to schedule and no update button to press.

**Existing hosts must migrate to updater protocol 2 with the trusted installer.**
Protocol 1 bootstraps trust controller-provided checksums and remain exposed to
controller-forged updates until this migration is complete. Publishing a signed
release does not upgrade those bootstraps: they request an installer rerun when
they see protocol 2, but a compromised controller could still serve protocol 1.

From the host's **Agent access**, generate and run the command, which fetches
the repository installer and public key automatically. Use the same controller
and data directory. Preserve a custom root with `--data-dir`;
do not use `--reenroll` for this upgrade. To pause old automatic checks during
migration, run `sudo systemctl stop electric-cherry-update.timer`. This does not
interrupt a running update worker; let any active update or journal recovery
finish before installing. The installer preserves host identity, credentials,
game files and running containers, replaces the bootstrap, and enables its timer
on success. Hosts without an updater show **Setup required** until a heartbeat
confirms installation. Hosts using an earlier protocol 2 bootstrap with a
manually provisioned keyring also need this installer rerun to adopt repository
refresh: the updater does not replace itself automatically. An old local
`/etc/electric-cherry/release-signing.gpg` file is ignored by the new bootstrap.

The **Automatic agent updates** panel reports the running version, target
version, last check, progress and any error. Offline hosts show their last
reported status. Deferred updates report why they are waiting: for example an
active agent operation, an unavailable repository/controller or a newer updater bootstrap.
Checks also defer while the agent service is stopped; an update does not start a
service that was stopped outside the update workflow.
The independent updater holds the same execution lock, authenticates the exact
manifest bytes with the freshly fetched repository keyring before parsing them,
verifies the release checksum, version and supported protocol, replaces agent
code, and restarts the agent service. It never restarts Docker or game containers. It confirms an
authenticated heartbeat from the new agent process before accepting the update;
failure restores the previous agent code. Its durable journal permits recovery
after a host reboot or interrupted update.

The updater only installs newer compatible versions; replaying an older signed
release or rolling the controller back does not downgrade agents. Signing cannot
force a controller to publish newer releases or prevent it withholding updates.
A rejected version/checksum pair stays blocked until
the controller publishes a different pair that is still newer than the installed
agent. Inspect the reported error when a host
shows **Needs attention**. The updater itself is a stable bootstrap component:
it changes only when the installer is rerun. If a future release requires an
unsupported updater protocol, rerun that release's installer on the host.

### Publishing signed releases

When publishing changed agent code, bump its numeric `VERSION` in
`agent/ec_agent.py`. Finalize and review all source changes before packaging and
signing. An already installed version is never replaced just because its bytes
changed.

Use a separate offline publisher machine with GnuPG. Keep the private key and its
GnuPG home outside the checkout, Docker build context, controller, and CI. For
example, create a dedicated signing key and inspect its full fingerprint:

```sh
ec_signing_home=/secure/electric-cherry-signing
install -d -m 0700 "$ec_signing_home"
gpg --homedir "$ec_signing_home" --quick-generate-key 'Electric Cherry Release Signing' ed25519 sign 2y
gpg --homedir "$ec_signing_home" --list-keys --with-fingerprint
```

Protect the key with a passphrase and keep its backup and revocation certificate
outside release artifacts. Set the full fingerprint of the signing key, then
export only its public key into the reviewed checkout:

```sh
ec_signing_fingerprint=REPLACE_WITH_FULL_SIGNING_KEY_FINGERPRINT
gpg --homedir "$ec_signing_home" --export "$ec_signing_fingerprint" > agent/release-signing.gpg
```

Inspect the export with `gpg --show-keys --with-fingerprint agent/release-signing.gpg`
and confirm the expected publisher fingerprint. Commit and publish that public
file to the repository's `main` branch before enrolling hosts or releasing code
signed by this key. Never export secret keys into the repository. A YubiKey-held
signing key works with the same workflow: export its public certificate from your
GnuPG home and use the full signing key/subkey fingerprint with the signer. Keep
the YubiKey on the publisher workstation, outside controller and CI access.

From the reviewed checkout on the publisher machine, package and explicitly sign:

```sh
npm run agent:package
python3 agent/sign_release.py --gnupghome "$ec_signing_home" --fingerprint "$ec_signing_fingerprint"
```

The signer defaults to `static/agent`; `--release-dir` selects another packaged
release directory. It checks that the manifest matches the agent, updater and
installer bytes, then produces the ASCII-armored detached `manifest.json.sig`.
The manifest records their SHA-256 hashes as `sha256`, `updaterSha256`, and
`installerSha256`, along with version, platform and protocol metadata. The
installer hash lets recipients check a publisher-distributed installer against
the authenticated manifest before running it.

Commit or distribute the finalized `static/agent` public artifacts together,
including the signature, before building the controller image from the same
reviewed source revision. Never include private keys or the signing home.
Normal `npm run build`, development startup,
and Docker builds never sign: packaging preserves a signature only when the
manifest remains byte-for-byte identical and removes it after any covered source
change. Repackage, review and sign again after such a change. Unsigned builds can
run the controller, but hosts fail closed and leave their installed code intact.

### Signing-key rotation and removal

Review new fingerprints on the publisher workstation, then replace
`agent/release-signing.gpg` on the repository's `main` branch. Every host uses the
new keyring on its next update check; no host visits or controller changes are
needed. For a planned transition, export both approved public keys into this file,
publish the combined keyring, then publish releases signed by the new key. Remove
the retired key once the controller serves the new signatures. Checks already in
progress use the keyring they fetched at the start of that check.

For a compromised key, remove it from the repository keyring immediately and
publish releases signed by a replacement key. Hosts with the refreshed keyring
reject the old signer even if the controller still serves its release. Publishing
a revocation certificate or allowing a key to expire is insufficient: [GnuPG's
`gpgv` documentation](https://www.gnupg.org/documentation/manuals/gnupg/gpgv.html)
states that it does not check key expiration or revocation. Keep the keyring's
contents restricted to currently trusted keys. Removing all keys blocks new
installations and updates; it does not stop or remove already installed agents.

### Inspecting updates

The installer places the updater at
`/usr/local/libexec/electric-cherry-update.py` and enables
`electric-cherry-update.timer`, which starts the oneshot
`electric-cherry-update.service`. Its state is kept under `/etc/electric-cherry`,
including `update-status.json`, `update/journal.json`, `agent-health.json` and
the shared `update.lock`. Check the timer and worker logs on the host with:

```sh
systemctl status electric-cherry-update.timer
journalctl -u electric-cherry-update.service -n 100
```

## Placement and resource capacity

### Host inventory and eligibility

The Inventory panel shows CPU usage with agent 1.4.2 and later. It measures
host-wide busy time across all cores on a 0–100% scale between inventory samples
(normally about 15 seconds). Idle and I/O wait are excluded; stolen CPU time
counts as unavailable capacity. The first sample after startup, unavailable
counters, and older agents show **Not reported**. Offline hosts retain their
last reported value. The controller defaults to retaining CPU and memory samples for seven days (missing measurements
remain unknown). Scheduling checks capacity against the latest heartbeat and uses
the last five minutes to rank eligible hosts; the Inventory panel shows the latest
reading.

New servers are placed automatically. Collection previews use cached availability
estimates and submission saves unplaced jobs without running capacity checks.
Provisioning checks each job asynchronously: a connected, active host must have
verified game files, its runtime, storage, a free UDP/TCP port, and valid CPU and
memory measurements. Hosts with stale or missing measurements cannot accept new
servers. Existing API clients may still explicitly constrain a launch to a host.

Queued jobs retain their accepted game-file version. Automatic host preparation
also downloads the game files needed by unplaced jobs; a newer upload does not
remove files still needed by the queue. Only the configured default runtime is
pulled automatically. Custom images must have been prepared by an admin from the
selected host's page, or on an active host for automatic placement, before a launch
is accepted. An image cached by another Docker workload does not qualify. Agent
1.8.6 reports these exact prepared references from `runtime-images.json` while
their image IDs remain cached; the controller also trusts its configured default
when cached. Host pages and APIs expose only prepared runtimes. Older agents can
still run the configured default, but must update before accepting custom runtimes.
Queued launches and migrations never trigger custom image pulls, and container
creation cannot implicitly pull a missing image.

Agent 1.8.7 also checks a runtime's current image ID against its recorded preparation
before creating or rebuilding a container. If another program retags a prepared
image, an admin must prepare it again from the host page before launching it.
Containers are created using the validated immutable image ID, so a tag change
after validation cannot substitute another image. Existing containers keep running.

### Resource samples and learned allowances

Only provisioning reserves a host and port. Agent **1.4.6** fixes container
discovery for resource sampling (the
earlier collector omitted `label=` from its Docker filter). It samples each
managed running container with a bounded
[`docker stats`](https://docs.docker.com/reference/cli/docker/container/stats/)
batch during inventory collection. CPU is normalized to cores (100% Docker CPU
is one core); memory is the Linux working usage reported by Docker, excluding
reclaimable cache. Missing or failed samples stay unknown instead of becoming
zero usage. Samples must match the host's live instance and generation.
Measurements arrive through `/api/agent/heartbeat`, normally every 15 seconds
plus collection and delivery time, independently of RCON or inspection operations.
Sampling failures appear on the host's Learned server capacity panel. Existing
containers can begin contributing running samples as soon as the agent updates;
startup samples require observing the first five minutes of a container's run.

The controller defaults to retaining per-instance minute peaks for 24 hours and caches learned
CPU/memory allowances. The first five minutes after a container starts count as
startup; later samples train the running allowance. Both phases use the
95th percentile of minute peaks, with **no per-server
headroom**. Each calculation considers up to the latest 10,000 minute peaks.
Host-local evidence is preferred to account for different hardware; new hosts
can borrow the fleet's learned allowance. Startup allowances are never lower
than running allowances. The host page shows both allowances and their source.
Expand **View samples** in Learned server capacity to inspect retained minute
peaks for the last 24 hours. Filter by startup/running phase or switch from this
host to the entire fleet. Rows show time, server and generation, CPU cores, and
memory in MiB; CPU and memory peaks may come from different
heartbeats. Results are paginated, and **Refresh samples** loads the latest stored
measurements without contacting agents. The allowance count may lag the table
until the next calculation or reflect a different source.

Lower allowances require at least three startup minute samples spanning two
minutes, or twenty running minute samples spanning five minutes. Until then,
**0.5 core and 1 GiB** are temporary cold-start allowances, raised immediately
if observations exceed them. They are replaced by measured allowances once the
evidence threshold is met; a tiny 0.01-core floor prevents zero-CPU admission.
Models expire after 24 hours without measurements. Recalculation happens during
heartbeat ingestion, at most once per minute per host/fleet phase; new peaks
raise the cache immediately. Existing startup estimates refresh when new startup
measurements arrive. Launch admission reads only the cached model.

### Admission and placement

Scheduling still leaves **5% host CPU and 500 MB RAM (500,000,000 bytes)** for the OS and bursts.
These allowances are scheduling estimates, not Docker limits. Collection previews
estimate how many new servers fit at their running allowance. Jobs may still be
queued when that estimate is insufficient. Current host CPU and
available memory account for existing containers and other processes. Observed
containers are not topped up to historical startup or running allowances, even while
awaiting health verification. Established instances are covered by host usage
when individual container measurements are unavailable.

Dispatch checks the next launch's startup allowance against live host usage.
Unplaced queued launches wait their turn without holding physical resources.
Any previously assigned reservations remain protected. Submitted or uncertain launches missing a matching-generation
measurement still consume a full startup allowance until telemetry accounts for
them. This lets collections start as capacity
becomes available, without stacking every queued server's startup allowance.
Capacity errors report projected usage, usable capacity and reservations.

When startup capacity is available, provisioning checks each eligible host’s
current CPU and available memory against reservations and headroom.
Among hosts that fit, it chooses the lowest projected CPU/memory utilization,
using the busier of the latest reading and the five-minute average for ranking.
A historical resource dip cannot block a host whose current capacity has recovered.
Historical ranking runs in a worker thread. The controller rechecks current
capacity and reserves the selected host and port atomically, then records them
in the launch recipe. Host network defaults
are resolved again for the final host; explicit overrides are preserved. If no
host qualifies, the request waits and is checked again. Once remote staging or
creation has been submitted, placement stays fixed to prevent duplicate instances.
Cancelling an unsubmitted launch releases its reservation; uncertain remote work
retains capacity until cleanup is confirmed.

## Capacity overview

Open **Capacity** (`/capacity`) to see the same cached estimates used by collection
previews. Available totals include hosts ready for the default runtime and latest
files for the selected game. **All games** counts shared resources once, including
hosts ready for either game; it is not a guarantee that every game fits every host.
The host breakdown shows readiness, missing measurements, available ports, learned
running/startup allowances, and the resource limiting additional servers.

Startup slots and queued jobs are fleet-wide, even with a host or game selected.
Server estimates use running allowances; provisioning still checks the larger
startup allowance before starting each job. Port availability uses each host's
configured bind address. Queued unplaced jobs do not reserve physical resources.

The history graph offers the last hour, 24 hours, or seven days of measured CPU
and memory headroom, after the usual OS reserve. It averages each host's reports
within each period, then sums hosts for fleet views. Missing reports appear as gaps.
Historical measurements cover the selected currently registered hosts and do not
reconstruct past readiness, port reservations, or startup slots. Reports refresh
automatically while the page is visible, and historical queries run in a separate
reader worker. `GET /api/capacity` supports `range=1h|24h|7d`, `host=<host ID>`, and
`game=all|t5|t6`; host/game selections also remain in the page URL.

## Identity, transport and credentials

Enrollment tokens are random, hashed at rest, single-use and expire after 15
minutes. Consumption and credential issuance are transactional. Enrollment only
works at the enrollment endpoint. Each host receives a distinct bearer credential
and the controller only stores its hash. Host and enrollment bearer credentials
are rejected outside `/api/agent/`.

The agent uses HTTPS, operating-system certificate validation, outbound heartbeats
and polling; Docker has no exposed remote API. Agent endpoints reject HTTP unless
`EC_AGENT_ALLOW_HTTP=true` is explicitly enabled for development. The agent's
insecure development option is limited to loopback. Public release assets contain
no credentials. HTTPS protects transport; an offline publisher signature over the
release manifest authenticates agent and updater code using the public keyring
fetched directly from the trusted repository before each check. Controller
credentials do not grant release-signing authority or control the key source.

Browser sessions do not authorize agents. Similarly, host credentials never grant
access to operator pages. Game telemetry uses a separate `EC_TELEMETRY_TOKEN`,
configured in each reporter's authorization header; see the
[telemetry contract](beta-test-plans.md#game-reporter-protocol).

Hosts offers credential rotation and revocation. Replacing a credential requires
an open administrator-requested rotation operation for that host. The controller
replaces the credential and completes that operation in one transaction. Rotation
persists the new secret on the agent before switching controller credentials, so
a lost response can be retried using the same proposed secret. A retry of the
active secret only acknowledges it; it cannot authorize another replacement.
Revocation affects one host only and does not stop its running game containers.
Re-enrollment requires a fresh token and an explicit installer `--reenroll` when
local credentials already exist.

## Host and game-file API

Open **Fleet → Hosts → Operation history** for retained operations across all
hosts, including deleted hosts. A host's **Operation history** action scopes the
same history to that host. Filters and pagination retain the selected scope.

- `GET /api/hosts/:id`: host details and deletion blockers.
- `GET /api/hosts/operations?page=1&filter=important`: operation history across
  hosts, newest first, with 25 summaries per page and filtered total/page counts.
  Filters are `important` (default), `routine`, and `all`. Summaries include the
  server, safe target path or status command, and initiating source when recorded.
  Payloads and results are omitted.
- `DELETE /api/hosts/:id`: administrator requests cleanup of an unused host;
  returns `202` while remote removal is pending, `200` for an unenrolled host,
  or `409` when the agent needs updating or unresolved ownership remains.
- `POST /api/agent/enroll`: consume enrollment token and return host credentials.
- `GET /api/agent/release/manifest.json`: public agent version, protocol and
  checksum plus `updaterProtocolVersion`, `updaterSha256`, and `installerSha256`.
- `GET /api/agent/release/manifest.json.sig`: detached publisher signature over
  the exact manifest bytes; absent for an unsigned build.
- `GET /api/agent/release/ec-agent.py` and `ec-agent-update.py`: public release
  source, verified against the signed manifest before installation.
- `POST /api/agent/heartbeat`: authenticated inventory, current operation and caches.
- `GET /api/agent/operations`: deliver/reconcile durable operation IDs.
- `POST /api/agent/operations/:id`: authenticated acceptance, progress or result.
- `POST /api/agent/decommission/:id`: cleanup worker reports file removal and
  agent uninstall using its own operation-scoped receipt credential.
- `GET /api/game-files`: returns `{libraries, hosts}`. `libraries` always contains
  two game slots, T5 and T6, including empty slots. Each slot has a nullable
  `latestVersionId` identifying its latest verified upload, plus upload history.
- `POST /api/game-files`: administrator creates a folder upload with
  `{game: "t5" | "t6", files: [{path, size, lastModified}]}`. Upload IDs and
  internal revision metadata are generated automatically; no library name,
  version label or description is required.
- `GET /api/game-files/:id/upload`: durable per-file offsets and publication status.
- `PUT /api/game-files/:id/upload/files/:path`: administrator uploads a raw chunk
  with `X-Upload-Offset`, `X-Upload-Length` and `X-Content-SHA256` headers.
- `POST /api/game-files/:id/upload/complete`: administrator starts asynchronous
  verification/publication; poll the upload endpoint for completion.
- `/api/agent/libraries/...`: authenticated game-file downloads. Legacy
  `/api/agent/imports/...` endpoints remain compatible with already queued imports.

The operation envelope is independent of polling and can be carried over a
persistent transport later. No unrestricted shell operation is available.

## Safety and durable operations

Each remote request has an immutable ID and payload, plus an instance UUID and
generation where applicable. Reusing a key with a different request is rejected.
The controller records remote IDs before dispatch. The agent persists accepted,
running, completed, failed and unknown states in SQLite. Lost acknowledgments
return the recorded result; restarting the controller never creates a new
mutation simply because the previous reply was lost.

The agent reconciles ambiguous results using owned Docker labels, generations and
persisted execution checkpoints when those observations prove the outcome.
Otherwise the result remains **Unknown** and holds dependent work. Running alone
is insufficient evidence that a particular restart happened. EC neither assumes
an unreachable server stopped nor launches a duplicate replacement elsewhere.

An interrupted rebuild can settle as **Failed** when its last checkpoint is
`disrupting`, Docker confirms both the old and replacement containers are absent,
and any existing generation marker still matches the original instance. The agent removes
only the partial replacement files, preserves the old generation's files and
marker, and retains the original diagnostic in its failure report. It does not
replay the interrupted rebuild. A separately queued rebuild can then proceed
through normal eligibility checks. A later creation checkpoint, a changed
generation, an inspection error, or failed partial-file cleanup leaves the result
**Unknown** and keeps dependent work waiting.

### Player checks and activation

Campaign bytes are staged and verified while players remain connected. Only
Provisioning can dispatch activation. It checks source eligibility and shared
capacity, and the agent repeats a fresh local RCON occupancy check immediately
before activation/restart, stop, removal or rebuild. Busy or unverifiable lobbies
wait indefinitely, including multi-hour Zombies sessions. Before images returned
by the agent populate existing campaign/beta rollback history. A beta cannot be
closed while its remote activation receipt is still unresolved.

An empty Fleet card does not replace this final host-local check: the card may
use telemetry, and the controller's RCON parser is more permissive than the
agent's disruption guard. Agent 1.4.1 includes the local RCON failure reason in
the operation's hold message (configuration, timeout, unreachable endpoint,
authentication, invalid packet, or an unrecognized player count). For a parser
hold, inspect the raw `status` reply in **Diagnose connection**. Unknown counts
remain held; RCON availability alone does not prove the lobby is empty.
It also recognizes T6's `num score bot ping` header, which earlier agents
rejected even when the complete player table was empty.

Rebuild uses a new generation, fresh instance storage and the latest promoted
configuration/scripts/plugins. It retains the selected game-file version and
runtime image. Old generation commands cannot target the replacement. Start,
restart and create complete only after post-operation health evidence; creation
and rebuild also verify the selected map and running Plutonium revision. The
base-game library and runtime revision are independent records.

## Rebalance instances

Open **Fleet → Capacity → Rebalance** to spread existing instances across active
hosts. Hosts also provides a shortcut. Operators and administrators can run a
rebalance; viewers can inspect its history.

Choose at least two participating hosts, a game selection, and a maximum of
1–20 moves (initially five), then choose **Preview rebalance**. The preview shows
current and projected instance counts, CPU and memory utilization, proposed
source/destination pairs, and exclusions. A larger host can carry more instances:
the planner balances resource utilization relative to each host's capacity.
The selected game limits which instances can move; the estimates include all
workloads on the participating hosts.

The planner uses current measurements and conservative five-minute host history,
current-generation instance resource samples, destination running/startup
allowances, and existing reservations. Each move must reduce the source/destination
pair's peak CPU-or-memory utilization by at least five percentage points. A
successfully moved instance is excluded for 30 minutes to limit repeated moves.
Unplaced queued launches do not reserve physical resources.

Only healthy instances with fresh, confirmed empty occupancy and recoverable
deployed files qualify. Occupied, stopped, unknown, stale, and incompatible
instances are excluded, as are beta, campaign, maintenance, and unresolved
recovery work. Positive current occupancy from either RCON or telemetry vetoes
an empty claim. A partially balanced result is normal when eligible instances
cannot improve the distribution enough.

Choose **Start rebalance** within five minutes of the preview. Submission
rechecks the proposed pairs; a changed plan requires a new preview. One durable
run executes at a time, and it never adds moves beyond the approved list. Before
each move it rechecks occupancy, identity, capacity, and competing work. A move
that no longer qualifies is skipped. After each healthy completion the run waits
for fresh reports from both affected hosts before considering its next move.

Provisioning reserves the destination before removing the source, then recreates
and health-verifies the lobby. The source host stays active. This is an empty-lobby
rebuild with a temporary outage; its address or port may change. The logical
identity, map, saved key, runtime credentials, runtime image, game-file library,
and recorded deployed configuration/scripts/plugins are retained. Untracked
local mutable files are not copied. If deployed bytes or credentials are no longer
available, the instance is excluded instead of silently upgrading to promoted files.

**Stop remaining moves** withdraws work that has not committed source removal.
Once source removal has been queued, that move continues through destination
creation and health verification. Stopping does not undo completed moves.
Unknown agent results retain their original operation and reservations until
reconciliation; they never authorize a duplicate instance. Failures stop the
remaining moves and link to Provisioning recovery. Recovering a move there does
not resume the rest of its run. Runs and operation links survive controller
restarts; completed/stopped runs are retained for 31 days.

Autoscaling retains ownership of lobby counts. A move is represented as a rebuild,
so the affected map's scaling decisions wait for it to settle instead of treating
the temporary outage as missing capacity.

`GET /api/rebalancing` returns host choices and the 20 most recent runs, with active
runs first. `POST /api/rebalancing` accepts `preview` with `settings` (`hostIds`,
`game: all|t5|t6`, `maxMoves`), `start` with `previewId`, or `stop` with the run `id`.
Writes use the existing operator authorization and same-origin checks. Repeating
Start with the same preview returns its original run. The controller planner and
run state live in `src/lib/server/rebalancing/`; all host mutations remain in
Provisioning. Rebalancing is manual; there is no periodic automatic policy.

## Drain and enable hosts

**Drain host** asks for confirmation, blocks new deployments, and migrates
instances to other ready hosts with capacity. Each migration waits for an empty
lobby, reserves destination CPU, memory and ports, removes the source instance,
then rebuilds the same server identity and map on the destination using its saved
key and current promoted configuration/scripts/plugins. The address or port may
change. Active beta ownership and other operations still gate migration, and the
agent checks occupancy again before removal. Unavailable capacity leaves the
source running; unknown removal results retain their reservation until the
original operation reconciles. Migrations survive controller restarts.

Once all migrations pass health verification and the source is empty, it becomes
**Disabled** automatically. There is no manual Disable action. **Enable** starts
automatic setup while the host remains outside the capacity pool: the configured
runtime is pulled again and both latest published BO1 (T5) and BO2 (T6) libraries
are downloaded as needed and fully verified, including cached copies. Both games
must be published. Failed setup or stale inventory keeps the host disabled; only
successful preparation and fresh inventory return it to Active.

## Delete a host

Administrators can choose **Delete host** on a host card or at the top of its
detail page. The confirmation lists anything that must be resolved first:
registered servers, reserved ports, or operations already sent to the agent.
Drain the host or remove its servers, then let outstanding operations reconcile.
An unenrolled host is removed immediately. An enrolled host stays visible as
**Deleting** until its agent confirms cleanup. Offline hosts wait for their next
connection; their absence is never treated as proof of removal. Existing agents
need the current installer: generate a fresh command under **Agent access**, rerun it
on the host, and wait for a heartbeat advertising uninstall support.

Deletion blocks new operations, cancels queued commands that were never delivered,
and invalidates unused installer links. A separate cleanup service removes the
host's downloaded game libraries, instance files, agent journal and configuration,
owned containers, and recorded runtime images unused by other containers. It
stops and removes the agent service and executable, automatic update timer,
update service, updater executable and update state (including any legacy local
signing keyring), then removes its own files.
The controller hides the host and revokes its ordinary credentials only after
receiving the final uninstall receipt. Adding the machine again requires a new
host enrollment; the installer fetches release trust from the repository again.

Game folders uploaded through Electric Cherry belong to the controller. Those
original uploads remain available for other hosts and future deployments; only
the deleted host's copies are removed. Controller operation history is also kept.
Unrelated containers, files and shared dependencies such as Docker and Python
are preserved. Historical entries in shared system logs are not erased.

### Interrupted cleanup

Cleanup retries after connection loss or partial file removal. Failures before
handoff offer **Retry cleanup** after the installation is repaired; failures in
the independent worker retry automatically. The worker stays installed until
the controller acknowledges data cleanup. After removing its final files it
retries the completion receipt from memory. A power loss in that final window
can leave the controller awaiting confirmation even though removal succeeded;
inspect the machine if that status persists. The controller does not guess.

## Game files

### Upload a game folder

Game Files always shows T5 and T6, each with its own upload button and latest
verified files. Successful uploads become latest in publication completion
order, not by their start date or an entered label. A newer upload still being
transferred or verified leaves the current files available. Previous uploads
appear in history, and interrupted uploads can be resumed separately.

Choose the actual game folder, not its parent, a ZIP archive or the Plutonium
launcher data folder. The selected folder's name is removed once; nested paths
such as `zone/english/...` and `main/...` stay intact. Plutonium's
[T5 server guide](https://plutonium.pw/docs/server/t5/setting-up-a-server/) and
[T6 server guide](https://plutonium.pw/docs/server/t6/setting-up-a-server/) use a
copy of the game directory, separately from the launcher/runtime. The uploader
preserves the supplied files rather than attempting to infer installed maps from
a manually maintained DLC list.

### Transfer and verification

Folder uploads use chunks of at most 8 MiB, each checked against its SHA-256 before
being appended to temporary controller storage. Memory usage does not grow with
the size of an individual game file. Pause/retry uses the controller's saved file
offsets. After a page reload, choose **Resume upload** and reselect the original,
unchanged folder; relative paths, sizes and modification times must match. Keep
the page open while transferring. Once publication starts, the controller hashes
every complete file and publishes the directory atomically; closing the browser
does not stop that verification. An interrupted controller can retry publication.
Verified uploads are immutable; changing files means uploading another folder
for the same game. Hosts first download every file with up to four concurrent
transfers and HTTP Range resumption. After all transfers finish, a separate
verification pass reads every file from disk and checks its complete SHA-256,
including any resumed prefix. Only then is the complete cache promoted atomically.
Corrupt staged files are removed so the next attempt downloads them again.
Separating the phases removes checksum work from the transfer loop and exposes
disk-verification time separately; it does not guarantee a shorter total duration.

The Game Files and host setup pages show **Download** and **Verify** as separate
steps, each with its own progress. Elapsed time records active work across
resumptions and stops when work finishes or fails. Remaining time and speed use
the current step's measured progress; a download estimate does not include the
verification pass that follows. Estimates appear after enough progress is
measured and are hidden for stale or offline hosts. Existing agents without
timing reports still show byte progress. Agent 1.2.2 provides phase timing;
automatic updates install it between host operations.
Phase transitions are reported promptly and periodic heartbeats carry progress;
controller round trips do not stall every transferred chunk.

### Cache cleanup and retention

Corrupt or incomplete content never becomes Ready. Cache inventory is reconciled
on heartbeats, including vanished caches. The latest published version of each game
is always retained. With agent 1.4.7 or newer, an older host copy is automatically
eligible for removal once no live instance reservation on that host references it.
The game-file minimum count, supersession grace period and explicit protections in
Admin → Settings also apply. Stopped,
provisioning, and rebuilding instances still pin their version. The agent also
refuses deletion if any Docker container (including stopped containers) mounts
those files. Cleanup includes partial downloads and retries after interruption.

The controller removes an older version's files after every host has reported its
copy absent and no instance or unresolved operation references that version.
Offline hosts keep their last reported copies pinned until they reconnect.
Upload metadata, manifests, and operation history remain visible; removed files
cannot be downloaded again without a new upload. Cleanup runs on heartbeats,
successful publication, and the controller background worker, without needing
an open Game Files page.

### Proxy limits and older uploads

The container and Compose retain `BODY_SIZE_LIMIT=Infinity` for legacy source-host
imports, which stream whole game files directly to disk and verify manifest sizes
and checksums. Routes that buffer request bodies enforce limits before parsing:
ordinary JSON requests (including telemetry) are limited to 1 MiB, authentication
and user-management JSON to 8 KiB, game-file upload chunks to 8 MiB, and folder
manifests to 32 MiB. Existing agent endpoints retain their individual limits, up
to 128 MiB for operation results. Game-file uploads also enforce declared sizes
and checksums.

Script, plugin and config artifacts are limited to 32 MiB per file, matching the
host agent. Multipart uploads allow 33 MiB total, including fields and framing;
beta revisions allow 65 MiB total for multiple replacement files. Both the declared
request length and actual streamed bytes are checked before multipart parsing.
Oversized ordinary JSON and artifact uploads receive HTTP 413. GitHub plugin imports use
the same 32 MiB artifact limit.

Set a proxy body limit of at least 128 MiB (for example
`client_max_body_size 128m` in nginx) for buffered endpoints. If legacy source-host
imports are still in use, allow the largest game file on
`/api/agent/imports/*/files/*`; these transfers do not use the browser's 8 MiB
chunks. Route limits apply even when connecting directly to the controller.

Existing uploads, host caches and server references remain usable. Previously
named libraries are grouped under T5 or T6, and the most recently published
upload for each game becomes its latest files. Their immutable IDs and stored
content still in use are preserved; unused older files are cleaned up automatically.
Legacy source-host imports retain their existing IDs and can complete through
their original workflow.

## Ports and network bindings

Reservations include host ID, local bind address, protocol and port. Wildcard
binds conflict with concrete addresses on the same host. UDP and TCP reservations
for one launch commit together; the same number can be used independently on
another host. Optional public/NAT address and port are recorded separately and do
not change local Docker bindings. EC does not configure external NAT/firewalls.

## Provisioning progress and diagnostics

Native launches distinguish initial eligibility evaluation from an actual wait
for startup capacity. The operation page records payload preparation, command
submission, agent acknowledgement and execution, and separate container, game,
map and Plutonium revision checks. Evidence remains available after completion.

The **Host agent execution** panel reads the durable command state directly,
including its queue time and latest report time. Agent receipts are recorded in
the provisioning history when received; they do not wait for a fleet scan.
An agent command marked completed still requires EC readiness verification.
The **Controller activity** panel shows current observations and their elapsed
time while the operation is active. It is a live snapshot, not historical proof
of which task delayed a previous deployment.

### Health checks

Fleet health checks query RCON even when container inspection is unavailable. A
fresh successful RCON reply can establish game health while container state stays
unknown; confirmed stopped, unhealthy, and restarting states still take precedence.
Agent 1.4.4 retries transient failures of the read-only `status` query up to three
times, with 250 ms and 500 ms delays. Commands that change the game are not retried.
Status replies still expire normally, and native timeout diagnostics are retained.

### Background observations

Campaign, beta and maintenance observers run independently of provisioning.
New requests and completed agent commands wake provisioning; an arrival during a
pass schedules another pass. Routine file drift scans run every five minutes;
manual checks bypass that interval. Agent 1.4.9 supports one hash manifest per
drift scan, avoiding separate list/stat/read operations for unchanged files.
Older agents use the existing file checks. Overlapping identical native
observations share one operation until it completes. Routine status monitoring
and maintenance additionally share successful health results for up to 30 seconds
from the request start, preserving the original observation time. Changed server
identity or lifecycle intent invalidates reuse. Manual checks and disruption
safety checks continue to request fresh evidence.
Status scans allow two checks per host, so a slow host cannot stall the fleet.
Accepted or running health operations are withheld from the other health lane
for 30 seconds; an expired replay lease allows recovery with the original ID.
Agent 1.4.10 includes generation-tagged container states in its 15-second
heartbeat, using the same Docker inspection as resource sampling (including
stopped containers). Routine monitoring uses samples younger than 30 seconds,
preserving their sample time and falling back to inspection when evidence is
missing, stale, or predates lifecycle changes. RCON status queries share one
operation per host, with two concurrent local probes and a 20-second batch
deadline. Individual failures remain visible per server. Both additions are
capability-gated; older agents retain individual inspection and RCON operations.
Agent 1.4.11 caches parsed library manifests during inventory collection until
their modification time, size, or file identity changes. Deleted manifests leave
the cache, while transfer progress and partial-file sizes are sampled each time.
Launch and download integrity verification continue to read from disk.
Agent 1.4.5 reserves two observation workers for instance health and read-only RCON,
and one for files, logs, and host inventory. Background reads arrive in batches of
up to four. File receipt uploads release the instance lock before contacting the
controller. Older agents retain the shared read lane until upgraded. Startup and
disruption safety gates remain enforced at command admission.

Older completed operations whose progress snapshots were already discarded
cannot recover missing check results from this change. Their surviving events
and host command records remain available without inventing verification times.

### Read operations during downloads

Agent 1.4.0 separates read-only inspections, file/log reads, and the exact RCON
commands `status`, `getstatus`, and `version` from the serial mutation queue.
Two bounded read lanes can run during runtime or library downloads. Instance
mutations exclude reads of that instance; credential rotation and host cleanup
wait for active reads. Both lanes retain durable operation IDs, generation checks,
and the updater activity lock. Older agents continue using the combined queue.

Agents long-poll `GET /api/agent/operations?lane=read|write&wait=20`; requests
without those parameters retain immediate combined delivery. The controller wakes
pending polls when work is queued and rechecks authentication after waiting.
Long polls release the updater lock; execution reacquires it and rechecks update
state before accepting work.

Browser game-folder uploads transfer up to three files concurrently. Chunks of
one file remain serial and checksum-verified. Publication waits for admitted file
transfers and prevents subsequent writes; pause/resume still uses acknowledged
per-file offsets. Host downloads retain their existing bounded concurrency.

## Cancel an unfinished creation

On the native creation operation page, **Cancel creation and remove server**
stops the launch and queues removal of its exact instance and generation. It
also works when readiness checks are stuck or the creation failed. This explicit
cancellation skips player-count checks on both the controller and agent, even
if players have joined. Completed creations use the normal safe removal flow.

The cancelled operation links to a durable cleanup operation with agent receipts
and progress. An unsubmitted launch releases its reservations immediately. Once
an agent command was queued, ports and the instance identity remain reserved
until the agent confirms container and instance-file removal. Shared game
libraries, runtime images, saved server keys, and operation history are retained.
Repeated cancellation requests return the same cleanup operation. A queued
normal removal waiting for occupancy is superseded by this cleanup.

Remote cancellation requires agent 1.3.0 or later (`instance.cancel-create`
capability). An older agent is never sent the unsupported cleanup command;
the cleanup page explains that it is waiting for an update. Offline or disabled
hosts retain the cleanup request until they reconnect or are enabled. Heartbeats
notify an executing create to stop between Docker commands; an already executing
Docker command must return or time out before cleanup can run. Cancellation
does not require the original creation to acknowledge completion. The agent
records a removal tombstone before cleanup so a delayed creation command cannot
recreate the instance, and resumes interrupted cleanup under the same operation ID.

While cancellation requires a newer agent, the controller pauses all command
delivery to that host, including background observations and command replays.
Older agents hold their update lock across a whole command batch; continuing to
feed them work could otherwise starve the updater. Already delivered work can
finish and report normally, and heartbeats continue. The next automatic update
check can install the release once the batch has drained. When a heartbeat
advertises cancellation support, delivery resumes with cancellation cleanup
first. This controller-side hold also recovers cancellations already waiting
on older agents. Hosts without automatic updates need the current installer.

## Development and validation

```sh
npm run agent:package
npm run check
npm test
npm run test:agent
npm run build
```

The build packages the agent, standalone updater and installer into the static
release, generates their checksum manifest and serves those same assets in the
Node-adapter image. Sign production artifacts using the
[offline workflow](#publishing-signed-releases) before building. Development
serves the freshly packaged static release even when an older production build
exists. The Python programs use only the standard library; signature verification
also requires the host's `gpgv` executable. Real signing tests require `gpg` and
`gpgv`, installed by CI; local runs skip those tests when either tool is absent.

Automated tests cover enrollment/authentication, rotation/revocation, host
readiness, scoped ports, immutable library publication, streaming/ranges,
signed manifests, forged checksums, unknown signing keys, repository key refresh
and rotation, repository failures, interrupted downloads,
instance generation ownership, local occupancy
holds, controller restart/reconnect, ambiguous results, occupied campaign staging,
activation before images, collections and draining. Host deletion tests cover
ownership guards, interrupted cleanup, uninstall receipts, credential revocation
and retention of controller uploads/history. Docker/systemd/UDP and host
observations are simulated in these tests.

A real Debian/Docker/Plutonium acceptance run is still required before production
rollout. Automated simulations do not establish successful deployment on real
game hosts. Run:

- Publish the public key in the repository and run the generated enrollment
  command on Debian 12 and 13 amd64 without a local keyring, then rerun without
  changing identity/data.
- Migrate a protocol 1 host with the trusted installer, then publish a newer
  compatible signed agent release. Verify a timer-driven update, its authenticated
  heartbeat and unchanged game containers. Exercise an active-operation hold,
  missing/invalid signatures, an unknown key, a forged manifest matching changed
  code, bad checksum, failed startup/heartbeat rollback, controller downgrade,
  network loss and host reboot during update. Verify updater protocol mismatch
  requests an installer rerun. Replace the repository keyring and confirm the
  next check rejects the removed signer and accepts the new signer; block access
  to the repository and confirm no cached key is used and local rollback still works.
- T5 and T6 deployments and a mixed collection from a fresh destination host.
- Warm the Plutonium runtime cache, then create and rebuild T5 and T6 lobbies;
  verify the running revision and compare first-start downloads and launch times
  with an empty cache. Test missing, corrupt and interrupted cache preparation,
  a newer production revision, and a custom runtime image. Confirm launch-file
  overrides survive seeding, lobby writes cannot alter the cache or another lobby,
  and cache refresh leaves running lobbies unchanged. Queue a deployment during
  preparation and verify that downloading yields and later resumes. Interrupt
  preparation by stopping the agent and verify that retry removes abandoned
  temporary files while retaining completed downloads.
- Start, stop, restart, rebuild, destroy; verify RCON, map and runtime revision.
- Read console and game logs, including Plutainer's relative game-log links.
- File campaign and rollback, with a player joining between controller check and
  agent activation, and during file activation before restart; verify that the
  agent holds disruption.
- Drain occupied lobbies and verify shared libraries survive their retirement.
- Delete a drained host; verify its game copies, data directory, agent and cleanup
  services, update timer, configuration and executables are removed, controller uploads/history stay,
  and unrelated Docker workloads continue. Repeat with connection loss during
  cleanup and after the final receipt, and a reboot during data removal.
- Corrupt/interrupted library downloads, resumed transfer and verified promotion.
- Controller restart, agent restart and network loss before execution, during
  execution and after execution but before acknowledgement. Verify the same
  operation IDs and absence of duplicate containers/restarts.

### Bank journals during server teardown

Agent 1.9.0 introduced `bank.settlement`: removal, rebuild, migration and creation
cancellation finish pending TWS bank decisions before deleting the old runtime.
An unavailable controller or corrupt journal leaves the operation unresolved
and preserves storage. Update all agents before enabling the
[player bank](bank.md). EC generates and injects bank credentials per instance;
no token map or manual server-secret provisioning is required.

Agent **1.9.1+** recognizes both the fixed public game-bank endpoint
`https://ec.thewrightserver.net/api/bank` and its own legacy controller-derived
bank endpoint in journals. It stops generating `tws_bank_url` in the private
game configuration. Agent transport still uses the enrolled controller URL and
host credential; journal URLs never determine request destinations. Upgrade
agents first, then [rebuild servers changing bank endpoints](bank.md#upgrading-from-a-controller-derived-game-endpoint)
to drain pending decisions before the new scripts start. See the
[agent changelog](../../agent/CHANGELOG.md) for release and validation details.