Add automatic agent updates with verified rollback

docs/authentication.md

@@ -85,6 +85,13 @@ endpoint or accept passwords as command arguments.
 Host agents retain enrollment/host-token authentication under `/api/agent/`.
 Installer and release files remain public so a new machine can enroll. A browser
 login never substitutes for a host credential, and host tokens cannot manage EC.
+The agent and standalone updater source contain no credentials. Automatic
+updates fetch the controller's public HTTPS release manifest and verify its
+checksums and compatibility. The installed host retains its own credential;
+the updater accepts an upgrade only after a fresh authenticated heartbeat from
+the new agent process. Updating agent code does not change browser sessions or
+user roles. See [automatic agent updates](native-hosts.md#automatic-agent-updates)
+for the one-time installer upgrade required by existing hosts.
 Host deletion uses a separate operation-scoped receipt token so the uninstall
 worker can confirm removal and retry its acknowledgment after host credentials
 are revoked. That token authorizes only the corresponding cleanup receipt.

docs/native-hosts.md

@@ -15,8 +15,9 @@ the agent owns local execution and its durable operation journal.
 2. Open **Hosts → Add Host**, choose a name, local game-port range and bind address,
    and generate an enrollment. Run the displayed command on Debian 12/13 amd64.
 3. The installer validates the platform, installs Python 3 and an absent Docker
-   installation, verifies the downloaded agent SHA-256 over HTTPS, enrolls it and
-   enables `electric-cherry-agent.service`. It needs no Git, Node.js or local build.
+   installation, verifies the downloaded agent and updater SHA-256 over HTTPS,
+   enrolls it and enables `electric-cherry-agent.service` and its automatic update
+   timer. It needs no Git, Node.js or local build.
 4. In **Hosts**, prepare a Plutainer runtime image. In **Game Files**, select T5 or
    T6 and upload your local game folder. Name the library and version and publish
    it after the upload finishes. Include the DLC files in that folder; there is
@@ -48,6 +49,59 @@ The agent uses Plutainer's documented [game-file and app mounts](https://github.
 Its log reader supports relative log links that resolve inside the instance app
 directory, while file writes reject symlinks and paths outside that directory.
 
+## Automatic agent updates
+
+New installations enable automatic agent updates by default. Each host checks
+its controller's release over HTTPS about every five minutes, beginning about
+two minutes after boot, with up to 30 seconds of jitter. Updating the controller
+therefore makes its compatible, newer agent release available to enrolled hosts.
+There is no controller command to schedule and no update button to press.
+
+**Existing hosts need the current installer once.** From the host's **Agent
+access**, generate an installer command and rerun it on the same machine, with
+the same controller and data directory. Do not use `--reenroll` for this upgrade.
+The installer preserves host identity, credentials, game files and running
+containers. Agent version 1.2.0 includes automatic update support; the host page
+shows **Setup required** until a fresh heartbeat confirms the updater is installed.
+
+The **Automatic agent updates** panel reports the running version, target
+version, last check, progress and any error. Offline hosts show their last
+reported status. Deferred updates report why they are waiting: for example an
+active agent operation, an unavailable controller or a newer updater bootstrap.
+Checks also defer while the agent service is stopped; an update does not start a
+service that was stopped outside the update workflow.
+The independent updater holds the same execution lock, verifies the release
+checksum, version and supported protocol, replaces agent code, and restarts the
+agent service. It never restarts Docker or game containers. It confirms an
+authenticated heartbeat from the new agent process before accepting the update;
+failure restores the previous agent code. Its durable journal permits recovery
+after a host reboot or interrupted update.
+
+The updater only installs newer compatible versions; rolling the controller back
+does not downgrade agents. A rejected version/checksum pair stays blocked until
+the controller publishes a different pair that is still newer than the installed
+agent. Inspect the reported error when a host
+shows **Needs attention**. The updater itself is a stable bootstrap component:
+it changes only when the installer is rerun. If a future release requires an
+unsupported updater protocol, rerun that release's installer on the host.
+
+When publishing changed agent code, bump its numeric `VERSION` in
+`agent/ec_agent.py` and rebuild the controller. Packaging derives the manifest
+version and checksums from the sources; an already installed version is never
+replaced just because its bytes changed.
+
+The installer places the updater at
+`/usr/local/libexec/electric-cherry-update.py` and enables
+`electric-cherry-update.timer`, which starts the oneshot
+`electric-cherry-update.service`. Its state is kept under `/etc/electric-cherry`,
+including `update-status.json`, `update/journal.json`, `agent-health.json` and
+the shared `update.lock`. Check the timer and worker logs on the host with:
+
+```sh
+systemctl status electric-cherry-update.timer
+journalctl -u electric-cherry-update.service -n 100
+```
+
 ## Identity, transport and credentials
 
 Enrollment tokens are random, hashed at rest, single-use and expire after 15
@@ -81,6 +135,10 @@ API outline:
   returns `202` while remote removal is pending, `200` for an unenrolled host,
   or `409` when the agent needs updating or unresolved ownership remains.
 - `POST /api/agent/enroll`: consume enrollment token and return host credentials.
+- `GET /api/agent/release/manifest.json`: public agent version, protocol and
+  checksum plus `updaterProtocolVersion` and `updaterSha256` for installer bootstrap.
+- `GET /api/agent/release/ec-agent.py` and `ec-agent-update.py`: public release
+  source, verified against that HTTPS manifest before installation.
 - `POST /api/agent/heartbeat`: authenticated inventory, current operation and caches.
 - `GET /api/agent/operations`: deliver/reconcile durable operation IDs.
 - `POST /api/agent/operations/:id`: authenticated acceptance, progress or result.
@@ -142,14 +200,15 @@ Drain the host or remove its servers, then let outstanding operations reconcile.
 An unenrolled host is removed immediately. An enrolled host stays visible as
 **Deleting** until its agent confirms cleanup. Offline hosts wait for their next
 connection; their absence is never treated as proof of removal. Existing agents
-need the 1.1.0 installer: generate a fresh command under **Agent access**, rerun it
+need the current installer: generate a fresh command under **Agent access**, rerun it
 on the host, and wait for a heartbeat advertising uninstall support.
 
 Deletion blocks new operations, cancels queued commands that were never delivered,
 and invalidates unused installer links. A separate cleanup service removes the
 host's downloaded game libraries, instance files, agent journal and configuration,
 owned containers, and recorded runtime images unused by other containers. It
-stops and removes the agent service and executable, then removes its own files.
+stops and removes the agent service and executable, automatic update timer,
+update service, updater executable and update state, then removes its own files.
 The controller hides the host and revokes its ordinary credentials only after
 receiving the final uninstall receipt. Adding the machine again requires a new
 host enrollment.
@@ -217,9 +276,11 @@ npm run test:agent
 npm run build
 ```
 
-The build packages `agent/ec_agent.py` and `agent/install.sh` into the static
-release, generates its checksum manifest and serves those same assets in the
-Node-adapter image. The Python agent uses only its standard library.
+The build packages the agent, standalone updater and installer into the static
+release, generates the agent/updater checksum manifest and serves those same
+assets in the Node-adapter image. Development serves the freshly packaged static
+release even when an older production build exists. The Python programs use only
+the standard library.
 
 Automated tests cover enrollment/authentication, rotation/revocation, host
 readiness, scoped ports, immutable library publication, streaming/ranges,
@@ -235,6 +296,12 @@ rollout. This workspace has no Docker engine or Debian game host, so automated
 coverage is not evidence that a real T5/T6 server has been deployed here. Run:
 
 - Installer on Debian 12 and 13 amd64, then rerun without changing identity/data.
+- Bootstrap an existing host with the current installer, then publish a newer
+  compatible agent release. Verify a timer-driven update, its authenticated
+  heartbeat and unchanged game containers. Exercise an active-operation hold,
+  bad checksum, failed startup/heartbeat rollback, controller downgrade, network
+  loss and host reboot during update. Verify updater protocol mismatch requests
+  an installer rerun.
 - T5 and T6 deployments and a mixed collection from a fresh destination host.
 - Start, stop, restart, rebuild, destroy; verify RCON, map and runtime revision.
 - Read console and game logs, including Plutainer's relative game-log links.
@@ -243,7 +310,7 @@ coverage is not evidence that a real T5/T6 server has been deployed here. Run:
   agent holds disruption.
 - Drain occupied lobbies and verify shared libraries survive their retirement.
 - Delete a drained host; verify its game copies, data directory, agent and cleanup
-  services/configuration/executables are removed, controller uploads/history stay,
+  services, update timer, configuration and executables are removed, controller uploads/history stay,
   and unrelated Docker workloads continue. Repeat with connection loss during
   cleanup and after the final receipt, and a reboot during data removal.
 - Corrupt/interrupted library downloads, resumed transfer and verified promotion.