Remove host files and uninstall agent when deleting hosts

docs/authentication.md

@@ -85,6 +85,9 @@ endpoint or accept passwords as command arguments.
 Host agents retain enrollment/host-token authentication under `/api/agent/`.
 Installer and release files remain public so a new machine can enroll. A browser
 login never substitutes for a host credential, and host tokens cannot manage EC.
+Host deletion uses a separate operation-scoped receipt token so the uninstall
+worker can confirm removal and retry its acknowledgment after host credentials
+are revoked. That token authorizes only the corresponding cleanup receipt.
 
 Game reporters must send a separate secret of at least 32 characters configured
 as `EC_TELEMETRY_TOKEN`:

docs/native-hosts.md

@@ -35,7 +35,10 @@ A fresh destination host downloads centrally stored game files without manual
 copying or per-container setup. EC does not supply the game content itself.
 
 The default host root is `/var/lib/electric-cherry`; installer `--data-dir` changes
-it. Data is separated into `libraries/<game>/<version>/content`,
+it. Use a dedicated, root-owned directory. For a separate disk, use a child of its
+mount point (for example `/mnt/games/electric-cherry`), with no nested mounts, so
+host deletion can remove the entire directory. The installer records ownership
+and refuses to adopt unrelated files. Data is separated into `libraries/<game>/<version>/content`,
 `instances/<UUID>/<generation>/app`, and `agent/operations.sqlite3`. Base libraries
 are mounted read-only and instance deletion removes only that instance's files.
 The controller stores large files under `GAME_FILES_ROOT` (default beside the
@@ -74,12 +77,15 @@ explicit installer `--reenroll` when local credentials already exist.
 API outline:
 
 - `GET /api/hosts/:id`: host details, operation history and deletion blockers.
-- `DELETE /api/hosts/:id`: administrator removes an unused host and revokes access;
-  returns `409` if servers, reservations or unresolved delivered operations remain.
+- `DELETE /api/hosts/:id`: administrator requests cleanup of an unused host;
+  returns `202` while remote removal is pending, `200` for an unenrolled host,
+  or `409` when the agent needs updating or unresolved ownership remains.
 - `POST /api/agent/enroll`: consume enrollment token and return host credentials.
 - `POST /api/agent/heartbeat`: authenticated inventory, current operation and caches.
 - `GET /api/agent/operations`: deliver/reconcile durable operation IDs.
 - `POST /api/agent/operations/:id`: authenticated acceptance, progress or result.
+- `POST /api/agent/decommission/:id`: cleanup worker reports file removal and
+  agent uninstall using its own operation-scoped receipt credential.
 - `POST /api/game-files`: administrator creates a folder upload with game, name,
   version, optional description and `files: [{path, size, lastModified}]`.
 - `GET /api/game-files/:id/upload`: durable per-file offsets and publication status.
@@ -133,13 +139,34 @@ Administrators can choose **Delete host** on a host card or at the top of its
 detail page. The confirmation lists anything that must be resolved first:
 registered servers, reserved ports, or operations already sent to the agent.
 Drain the host or remove its servers, then let outstanding operations reconcile.
-An unused host can be deleted before enrollment or while offline.
+An unenrolled host is removed immediately. An enrolled host stays visible as
+**Deleting** until its agent confirms cleanup. Offline hosts wait for their next
+connection; their absence is never treated as proof of removal. Existing agents
+need the 1.1.0 installer: generate a fresh command under **Agent access**, rerun it
+on the host, and wait for a heartbeat advertising uninstall support.
 
-Deleting a host removes it from the controller's host list and deployment
-choices, revokes its credentials, and invalidates unused installer links.
-Queued commands that have never been sent to the agent are cancelled. Operation
-history and published libraries are retained; the agent and game files remain on
-the machine. Adding the machine again requires a new host enrollment.
+Deletion blocks new operations, cancels queued commands that were never delivered,
+and invalidates unused installer links. A separate cleanup service removes the
+host's downloaded game libraries, instance files, agent journal and configuration,
+owned containers, and recorded runtime images unused by other containers. It
+stops and removes the agent service and executable, then removes its own files.
+The controller hides the host and revokes its ordinary credentials only after
+receiving the final uninstall receipt. Adding the machine again requires a new
+host enrollment.
+
+Game folders uploaded through Electric Cherry belong to the controller. Those
+original uploads remain available for other hosts and future deployments; only
+the deleted host's copies are removed. Controller operation history is also kept.
+Unrelated containers, files and shared dependencies such as Docker and Python
+are preserved. Historical entries in shared system logs are not erased.
+
+Cleanup retries after connection loss or partial file removal. Failures before
+handoff offer **Retry cleanup** after the installation is repaired; failures in
+the independent worker retry automatically. The worker stays installed until
+the controller acknowledges data cleanup. After removing its final files it
+retries the completion receipt from memory. A power loss in that final window
+can leave the controller awaiting confirmation even though removal succeeded;
+inspect the machine if that status persists. The controller does not guess.
 
 ## Libraries and ports
 
@@ -198,7 +225,9 @@ Automated tests cover enrollment/authentication, rotation/revocation, host
 readiness, scoped ports, immutable library publication, streaming/ranges,
 checksums, interrupted downloads, instance generation ownership, local occupancy
 holds, controller restart/reconnect, ambiguous results, occupied campaign staging,
-activation before images, collections and draining. Docker/UDP and host
+activation before images, collections and draining. Host deletion tests cover
+ownership guards, interrupted cleanup, uninstall receipts, credential revocation
+and retention of controller uploads/history. Docker/systemd/UDP and host
 observations are simulated in these tests.
 
 A real Debian/Docker/Plutonium acceptance run is still required before production
@@ -213,6 +242,10 @@ coverage is not evidence that a real T5/T6 server has been deployed here. Run:
   agent activation, and during file activation before restart; verify that the
   agent holds disruption.
 - Drain occupied lobbies and verify shared libraries survive their retirement.
+- Delete a drained host; verify its game copies, data directory, agent and cleanup
+  services/configuration/executables are removed, controller uploads/history stay,
+  and unrelated Docker workloads continue. Repeat with connection loss during
+  cleanup and after the final receipt, and a reboot during data removal.
 - Corrupt/interrupted library downloads, resumed transfer and verified promotion.
 - Controller restart, agent restart and network loss before execution, during
   execution and after execution but before acknowledgement. Verify the same