Require signed agent releases (Refs #64)

docs/guides/getting-started.md

@@ -9,6 +9,8 @@ HTTPS. Running game containers continue through controller or agent outages.
 
 - A controller machine with Docker Compose and an HTTPS reverse proxy.
 - Debian 12 or 13 amd64 game hosts with storage for game files and instances.
+- A signed agent release, plus its publisher's public key and reviewed installer
+  obtained through a trusted channel independent of the controller.
 - Your complete T5 or T6 game folders, including the DLC you intend to host.
 - Plutonium server keys for each game, with one unused key per new concurrent lobby.
 
@@ -37,6 +39,13 @@ For an existing installation, retain its original data directory. Compose
 defaults to `/root/zombies/.electric-cherry`; choose a dedicated directory for a
 new controller. See [.env.example](../../.env.example) for the other settings.
 
+Before building for game hosts, obtain the publisher's signed `static/agent`
+artifacts and matching source revision, or follow
+[Publishing signed releases](native-hosts.md#publishing-signed-releases)
+for your own build. Normal builds preserve signatures for identical artifacts;
+source changes require a new offline signature. An unsigned controller can run,
+but hosts reject its agent installation and updates.
+
 ```sh
 docker compose up -d --build
 ```
@@ -60,15 +69,24 @@ password recovery.
 1. Open **Releases → Game files** and upload the complete local folder for T5 or
    T6, including DLC. Uploads can run before any host is enrolled. The latest
    verified upload becomes that game's default files.
-2. Open **Fleet → Hosts → Add Host**, configure the host's game-port range and
-   bind address, and run the generated enrollment command on the host.
-3. Wait for automatic setup to prepare the Plutainer runtime and available game
+2. [Provision release trust](native-hosts.md#provision-release-trust): independently
+   verify the publisher key's full fingerprint, install its binary public key at
+   `/etc/electric-cherry/release-signing.gpg`, and copy the reviewed installer to
+   `./electric-cherry-install.sh` on the host. Both come from the trusted publisher.
+3. Open **Fleet → Hosts → Add Host**, configure the host's game-port range and
+   bind address, and run the generated enrollment command from the directory
+   containing that local installer. Do not pipe controller-downloaded code into
+   a root shell.
+4. Wait for automatic setup to prepare the Plutainer runtime and available game
    files. The host page reports download, verification, and readiness progress.
 
 The installer sets up the Python agent as a systemd service and enables automatic
 agent updates. It installs Docker if needed. Multiple instances share read-only
 base game files; each instance has its own writable data. See
 [Native hosts](native-hosts.md) for storage layout, network access, and recovery.
+Existing hosts using updater protocol 1 need a
+[trusted installer migration](native-hosts.md#automatic-agent-updates) before
+controller-independent release verification takes effect.
 
 ## Launch your first server
 

docs/guides/native-hosts.md

@@ -12,9 +12,11 @@ the agent owns local execution and its durable operation journal.
    and roles; agent endpoints authenticate their own host credentials. See
    [Users and access](authentication.md) for setup and password recovery.
 2. Open **Hosts → Add Host**, choose a name, local game-port range and bind address,
-   and generate an enrollment. Run the displayed command on Debian 12/13 amd64.
-3. The installer validates the platform, installs Python 3 and an absent Docker
-   installation, verifies the downloaded agent and updater SHA-256 over HTTPS,
+   and generate an enrollment. First [provision release trust](#provision-release-trust)
+   and copy the reviewed installer to `./electric-cherry-install.sh` on the Debian
+   12/13 amd64 host. Then run the displayed command from that directory.
+3. The installer validates the platform, installs Python 3, `gpgv` and an absent Docker
+   installation, verifies the release signature and agent/updater SHA-256 checksums,
    enrolls it and enables `electric-cherry-agent.service` and its automatic update
    timer. It needs no Git, Node.js or local build.
 4. In **Game Files**, upload your complete local folder under T5 or T6. Include
@@ -41,6 +43,37 @@ the agent owns local execution and its durable operation journal.
 A fresh destination host downloads centrally stored game files without manual
 copying or per-container setup. EC does not supply the game content itself.
 
+### Provision release trust
+
+Obtain the release publisher's public key and reviewed installer through a
+trusted channel independent of the controller. Verify the full public-key
+fingerprint with the publisher through that independent channel. On a trusted
+administrator workstation with GnuPG, inspect the supplied binary public key:
+
+```sh
+gpg --show-keys --with-fingerprint ./release-signing.gpg
+```
+
+Transfer that verified public key and installer to the host, with the installer
+named `electric-cherry-install.sh`. Provision the key before running enrollment:
+
+```sh
+sudo install -d -o root -g root -m 0700 /etc/electric-cherry
+sudo install -o root -g root -m 0644 ./release-signing.gpg /etc/electric-cherry/release-signing.gpg
+```
+
+The keyring and every parent directory must be owned by root, must not be
+symlinks, and must not be writable by other users. Install only the publisher
+keys you intend to trust. The generated enrollment command runs the local
+installer with `sudo bash ./electric-cherry-install.sh`; never replace this with
+a controller download piped into a root shell. An installer supplied by a
+compromised controller could bypass its own verification. The host never obtains
+or replaces its trust key through controller APIs.
+
+If you build your own controller releases, act as the publisher using the
+[signing workflow below](#publishing-signed-releases). Hosts reject unsigned
+releases, including ordinary development builds that have not been signed.
+
 ### Docker installation
 
 The installer uses Debian's Docker packages: `docker.io` on Debian 12, plus
@@ -169,15 +202,25 @@ directory, while file writes reject symlinks and paths outside that directory.
 New installations enable automatic agent updates by default. Each host checks
 its controller's release over HTTPS about every five minutes, beginning about
 two minutes after boot, with up to 30 seconds of jitter. Updating the controller
-therefore makes its compatible, newer agent release available to enrolled hosts.
+therefore makes its compatible, newer signed agent release available to enrolled hosts.
 There is no controller command to schedule and no update button to press.
 
-**Existing hosts need the current installer once.** From the host's **Agent
-access**, generate an installer command and rerun it on the same machine, with
-the same controller and data directory. Do not use `--reenroll` for this upgrade.
-The installer preserves host identity, credentials, game files and running
-containers. Agent version 1.2.0 includes automatic update support; the host page
-shows **Setup required** until a fresh heartbeat confirms the updater is installed.
+**Existing hosts must migrate to updater protocol 2 with the trusted installer.**
+Protocol 1 bootstraps trust controller-provided checksums and remain exposed to
+controller-forged updates until this migration is complete. Publishing a signed
+release does not upgrade those bootstraps: they request an installer rerun when
+they see protocol 2, but a compromised controller could still serve protocol 1.
+
+First provision release trust as described above. From the host's **Agent
+access**, generate a command and run it using the reviewed local installer, with
+the same controller and data directory. Preserve a custom root with `--data-dir`;
+do not use `--reenroll` for this upgrade. To pause old automatic checks during
+migration, run `sudo systemctl stop electric-cherry-update.timer`. This does not
+interrupt a running update worker; let any active update or journal recovery
+finish before installing. The installer preserves host identity, credentials,
+game files and running containers, replaces the bootstrap, and enables its timer
+on success. Hosts without an updater show **Setup required** until a heartbeat
+confirms installation.
 
 The **Automatic agent updates** panel reports the running version, target
 version, last check, progress and any error. Offline hosts show their last
@@ -185,27 +228,91 @@ reported status. Deferred updates report why they are waiting: for example an
 active agent operation, an unavailable controller or a newer updater bootstrap.
 Checks also defer while the agent service is stopped; an update does not start a
 service that was stopped outside the update workflow.
-The independent updater holds the same execution lock, verifies the release
+The independent updater holds the same execution lock, authenticates the exact
+manifest bytes with the pinned keyring before parsing them, verifies the release
 checksum, version and supported protocol, replaces agent code, and restarts the
 agent service. It never restarts Docker or game containers. It confirms an
 authenticated heartbeat from the new agent process before accepting the update;
 failure restores the previous agent code. Its durable journal permits recovery
 after a host reboot or interrupted update.
 
-The updater only installs newer compatible versions; rolling the controller back
-does not downgrade agents. A rejected version/checksum pair stays blocked until
+The updater only installs newer compatible versions; replaying an older signed
+release or rolling the controller back does not downgrade agents. Signing cannot
+force a controller to publish newer releases or prevent it withholding updates.
+A rejected version/checksum pair stays blocked until
 the controller publishes a different pair that is still newer than the installed
 agent. Inspect the reported error when a host
 shows **Needs attention**. The updater itself is a stable bootstrap component:
 it changes only when the installer is rerun. If a future release requires an
 unsupported updater protocol, rerun that release's installer on the host.
 
-### Publishing releases and inspecting updates
+### Publishing signed releases
 
 When publishing changed agent code, bump its numeric `VERSION` in
-`agent/ec_agent.py` and rebuild the controller. Packaging derives the manifest
-version and checksums from the sources; an already installed version is never
-replaced just because its bytes changed.
+`agent/ec_agent.py`. Finalize and review all source changes before packaging and
+signing. An already installed version is never replaced just because its bytes
+changed.
+
+Use a separate offline publisher machine with GnuPG. Keep the private key and its
+GnuPG home outside the checkout, Docker build context, controller, and CI. For
+example, create a dedicated signing key and inspect its full fingerprint:
+
+```sh
+ec_signing_home=/secure/electric-cherry-signing
+install -d -m 0700 "$ec_signing_home"
+gpg --homedir "$ec_signing_home" --quick-generate-key 'Electric Cherry Release Signing' ed25519 sign 2y
+gpg --homedir "$ec_signing_home" --list-keys --with-fingerprint
+```
+
+Protect the key with a passphrase and keep its backup and revocation certificate
+outside release artifacts. Set the full fingerprint of the signing key, then
+export only its public key for independent host provisioning:
+
+```sh
+ec_signing_fingerprint=REPLACE_WITH_FULL_SIGNING_KEY_FINGERPRINT
+gpg --homedir "$ec_signing_home" --export "$ec_signing_fingerprint" > release-signing.gpg
+```
+
+From the reviewed checkout on the publisher machine, package and explicitly sign:
+
+```sh
+npm run agent:package
+python3 agent/sign_release.py --gnupghome "$ec_signing_home" --fingerprint "$ec_signing_fingerprint"
+```
+
+The signer defaults to `static/agent`; `--release-dir` selects another packaged
+release directory. It checks that the manifest matches the agent, updater and
+installer bytes, then produces the ASCII-armored detached `manifest.json.sig`.
+The manifest records their SHA-256 hashes as `sha256`, `updaterSha256`, and
+`installerSha256`, along with version, platform and protocol metadata. The
+installer hash lets recipients check a publisher-distributed installer against
+the authenticated manifest before running it.
+
+Commit or distribute the finalized `static/agent` public artifacts together,
+including the signature, before building the controller image from the same
+reviewed source revision. Never include private keys or the signing home.
+Normal `npm run build`, development startup,
+and Docker builds never sign: packaging preserves a signature only when the
+manifest remains byte-for-byte identical and removes it after any covered source
+change. Repackage, review and sign again after such a change. Unsigned builds can
+run the controller, but hosts fail closed and leave their installed code intact.
+
+### Signing-key rotation and removal
+
+Verify new fingerprints independently and replace each host's root-owned keyring
+through trusted administration before publishing releases signed by a new key.
+For a planned transition, provision both approved public keys temporarily, then
+remove the retired key once every host can verify the new signer. The controller
+cannot authorize key rotation.
+
+Remove compromised or retired keys from every host's keyring. Publishing a
+revocation certificate or allowing a key to expire is insufficient: [GnuPG's
+`gpgv` documentation](https://www.gnupg.org/documentation/manuals/gnupg/gpgv.html)
+states that it does not check key expiration or revocation. Host deletion removes
+the pinned keyring with `/etc/electric-cherry`; provision trust again before
+reenrolling that machine.
+
+### Inspecting updates
 
 The installer places the updater at
 `/usr/local/libexec/electric-cherry-update.py` and enables
@@ -370,8 +477,10 @@ The agent uses HTTPS, operating-system certificate validation, outbound heartbea
 and polling; Docker has no exposed remote API. Agent endpoints reject HTTP unless
 `EC_AGENT_ALLOW_HTTP=true` is explicitly enabled for development. The agent's
 insecure development option is limited to loopback. Public release assets contain
-no credentials. The release checksum is authenticated by the controller's HTTPS
-origin; there is no separate offline release-signing infrastructure.
+no credentials. HTTPS protects transport; an offline publisher signature over the
+release manifest authenticates agent and updater code using the independently
+provisioned host keyring. Controller credentials do not grant release-signing
+authority.
 
 Browser sessions do not authorize agents. Similarly, host credentials never grant
 access to operator pages. Game telemetry uses a separate `EC_TELEMETRY_TOKEN`,
@@ -397,9 +506,11 @@ explicit installer `--reenroll` when local credentials already exist.
   or `409` when the agent needs updating or unresolved ownership remains.
 - `POST /api/agent/enroll`: consume enrollment token and return host credentials.
 - `GET /api/agent/release/manifest.json`: public agent version, protocol and
-  checksum plus `updaterProtocolVersion` and `updaterSha256` for installer bootstrap.
+  checksum plus `updaterProtocolVersion`, `updaterSha256`, and `installerSha256`.
+- `GET /api/agent/release/manifest.json.sig`: detached publisher signature over
+  the exact manifest bytes; absent for an unsigned build.
 - `GET /api/agent/release/ec-agent.py` and `ec-agent-update.py`: public release
-  source, verified against that HTTPS manifest before installation.
+  source, verified against the signed manifest before installation.
 - `POST /api/agent/heartbeat`: authenticated inventory, current operation and caches.
 - `GET /api/agent/operations`: deliver/reconcile durable operation IDs.
 - `POST /api/agent/operations/:id`: authenticated acceptance, progress or result.
@@ -576,10 +687,11 @@ and invalidates unused installer links. A separate cleanup service removes the
 host's downloaded game libraries, instance files, agent journal and configuration,
 owned containers, and recorded runtime images unused by other containers. It
 stops and removes the agent service and executable, automatic update timer,
-update service, updater executable and update state, then removes its own files.
+update service, updater executable, update state and pinned signing keyring, then
+removes its own files.
 The controller hides the host and revokes its ordinary credentials only after
 receiving the final uninstall receipt. Adding the machine again requires a new
-host enrollment.
+host enrollment and independently reprovisioned release trust.
 
 Game folders uploaded through Electric Cherry belong to the controller. Those
 original uploads remain available for other hosts and future deployments; only
@@ -834,14 +946,18 @@ npm run build
 ```
 
 The build packages the agent, standalone updater and installer into the static
-release, generates the agent/updater checksum manifest and serves those same
-assets in the Node-adapter image. Development serves the freshly packaged static
-release even when an older production build exists. The Python programs use only
-the standard library.
+release, generates their checksum manifest and serves those same assets in the
+Node-adapter image. Sign production artifacts using the
+[offline workflow](#publishing-signed-releases) before building. Development
+serves the freshly packaged static release even when an older production build
+exists. The Python programs use only the standard library; signature verification
+also requires the host's `gpgv` executable. Real signing tests require `gpg` and
+`gpgv`, installed by CI; local runs skip those tests when either tool is absent.
 
 Automated tests cover enrollment/authentication, rotation/revocation, host
 readiness, scoped ports, immutable library publication, streaming/ranges,
-checksums, interrupted downloads, instance generation ownership, local occupancy
+signed manifests, forged checksums, unknown signing keys, interrupted downloads,
+instance generation ownership, local occupancy
 holds, controller restart/reconnect, ambiguous results, occupied campaign staging,
 activation before images, collections and draining. Host deletion tests cover
 ownership guards, interrupted cleanup, uninstall receipts, credential revocation
@@ -852,13 +968,15 @@ A real Debian/Docker/Plutonium acceptance run is still required before productio
 rollout. Automated simulations do not establish successful deployment on real
 game hosts. Run:
 
-- Installer on Debian 12 and 13 amd64, then rerun without changing identity/data.
-- Bootstrap an existing host with the current installer, then publish a newer
-  compatible agent release. Verify a timer-driven update, its authenticated
+- Independently provision release trust and run the reviewed installer on Debian
+  12 and 13 amd64, then rerun without changing identity/data.
+- Migrate a protocol 1 host with the trusted installer, then publish a newer
+  compatible signed agent release. Verify a timer-driven update, its authenticated
   heartbeat and unchanged game containers. Exercise an active-operation hold,
-  bad checksum, failed startup/heartbeat rollback, controller downgrade, network
-  loss and host reboot during update. Verify updater protocol mismatch requests
-  an installer rerun.
+  missing/invalid signatures, an unknown key, a forged manifest matching changed
+  code, bad checksum, failed startup/heartbeat rollback, controller downgrade,
+  network loss and host reboot during update. Verify updater protocol mismatch
+  requests an installer rerun and test trusted keyring replacement for rotation.
 - T5 and T6 deployments and a mixed collection from a fresh destination host.
 - Warm the Plutonium runtime cache, then create and rebuild T5 and T6 lobbies;
   verify the running revision and compare first-start downloads and launch times