Remove Komodo backend in favor of native hosts

2026-10-05 18:07:07 -04:00
parent 701066aaf8
commit 4b0f341d64
116 changed files with 557 additions and 8132 deletions
@@ -104,7 +104,8 @@ the current shortfall too.

 All collections and individual launches share one startup queue, ordered by request
 time. By default, one server starts at a time. Its startup slot remains occupied
-until Komodo is healthy, RCON answers, and the reported map matches the selection.
+until the host agent confirms container health, game status and the selected map
+are verified, and RCON reports a current Plutonium revision.
 Only then can the next queued server start. Set **Concurrent server startups** in **Settings → Fleet settings** to
 a positive integer to allow more simultaneous startups. Each worker pass admits
 at most the available slots, even if an attempt fails immediately.
@@ -137,10 +138,10 @@ another launch takes the lobby, submission stops before saving the key;
 refresh the preview and review the newly assigned ID. Refreshing a preview clears
 the typed key so it cannot accidentally be saved against a different lobby.

-Komodo connection settings, target host, RCON configuration, config filename,
-lobby availability, and port availability are checked locally. These checks do
-not probe the host or verify game files, images, Docker volumes, firewall
-rules, or credentials. Container and RCON verification happen after provisioning.
+Host selection, game-file versions, runtime configuration and lobby availability
+are checked before queueing. Provisioning assigns host capacity and physical
+ports when dispatching. The host agent performs container and RCON verification
+locally after launch. See [native hosts](native-hosts.md) for readiness checks.
 T6 variants retain their exact catalog slug and launch ruleset; RCON health checks
 verify the engine map, not the variant's start location.

@@ -122,7 +122,7 @@ an application restart, routine watchdog checks wait for a new status-monitor pa
 Provisioning retains its checks before executing recovery actions.

 Three observations over at least two minutes are required before an automatic
-health restart; starting or restarting containers get five minutes. Unknown Komodo responses are observations,
+health restart; starting or restarting containers get five minutes. Unknown host-agent responses are observations,
 not proof of a stopped container. Repeated manual checks do not accelerate this
 window. A successful routine or manual game-status probe between scans resets
 the failure streak. Missing RCON configuration, unresolved ports, and invalid
@@ -1,8 +1,7 @@
 # Native host management

-Electric Cherry controls Docker/Plutainer through an outbound host agent. A native
-instance never uses Komodo or a filesystem mounted into the controller. The
-controller owns desired state, campaigns, collections, safety queues and history;
+Electric Cherry controls Docker/Plutainer through an outbound host agent. The controller
+does not need mounted game-server files. It owns desired state, campaigns, collections, safety queues and history;
 the agent owns local execution and its durable operation journal.

 Fleet health checks query RCON even when container inspection is unavailable. A
@@ -554,16 +553,12 @@ coverage is not evidence that a real T5/T6 server has been deployed here. Run:
  execution and after execution but before acknowledgement. Verify the same
  operation IDs and absence of duplicate containers/restarts.

-Legacy backend code remains for existing database registrations. New native
-launches do not require its configuration, API, network or mounted directories.
-
 ### Provisioning progress and diagnostics

 Native launches distinguish initial eligibility evaluation from an actual wait
 for startup capacity. The operation page records payload preparation, command
 submission, agent acknowledgement and execution, and separate container, game,
-map and Plutonium revision checks. Native actions use host-agent steps rather
-than legacy Komodo deployment steps. Evidence remains available after completion.
+map and Plutonium revision checks. Evidence remains available after completion.

 The **Host agent execution** panel reads the durable command state directly,
 including its queue time and latest report time. Agent receipts are recorded in
@@ -111,7 +111,7 @@ Separate durable milestones from transient readiness checks. A passed container/
 | --- | --- |
 | `canCancelRemoval` | Offer **Cancel queued removal**, explaining that it is available only before removal begins. Preserve the existing atomic worker/cancellation race protection. |
 | `canResumeVerification` | Offer **Resume health verification** and explain that it observes the existing instance without creating or restarting it. |
-| `canResumeCleanup` | Offer **Resume cleanup**, including eligible deleted identities. Explain that Komodo deletion already succeeded. |
+| `canResumeCleanup` | Offer **Resume cleanup**, including eligible deleted identities. Explain that instance removal already succeeded. |
 | `canResumeRebuild` | Offer **Resume rebuild cleanup** and explain that local cleanup precedes creation in the reserved slot. |
 | Failed/interrupted without a resume capability | Lead with the recorded failure and relevant inspection destination. An uncertain remote response must not produce a generic retry command. |
 | Cancelled | State that the request is cancelled; distinguish recorded prior work from remaining work that did not run. Do not claim continued execution. |
@@ -154,7 +154,7 @@ A suitable small schema extension is nullable `phase_key`, `phase_started_at`, `

 Implementation rules:

-1. Record structured blockers and observations where the worker already makes decisions. Do not build a second eligibility engine in the API or browser. The detail GET remains read-only and performs no Komodo/RCON probes.
+1. Record structured blockers and observations where the worker already makes decisions. Do not build a second eligibility engine in the API or browser. The detail GET remains read-only and performs no host-agent/RCON probes.
 2. Derive current capacity from existing scheduler helpers/tables, with an explicit snapshot time. Persist important milestones against the operation because server-owned recovery rows can later be replaced or deleted. Only attach a live slot when its `operation_id` matches; slot absence alone is not evidence of successful release.
 3. Update related state, observation, and semantic event atomically around each durable checkpoint, keeping network calls outside database transactions. Dual-write existing stage/event text, especially `REMOVING_SERVER`, which participates in recovery eligibility today.
 4. Record meaningful phase, blocker, command, and recovery transitions. Update observation timestamps on repeated checks without appending an event every poll. Changing player counts must not reset the phase's start time.
@@ -12,9 +12,9 @@ server, choose **Manage server** or open its **Server actions** menu.
 | Create | Create a new instance using the latest promoted files and verify container health, game status, the selected map, and the Plutonium revision. |
 | Start | Start the existing container and verify container health and RCON. |
 | Restart | Wait until empty, restart the existing container, and verify container health and RCON. |
-| Rebuild | Wait until empty, destroy the instance, delete all server files and its registration, then create a fresh instance in the same server slot and port. Verify container health, game status, map, and Plutonium revision. |
-| Stop | Wait until empty, stop the container, and verify that Komodo reports it stopped. Keep the instance for a later Start. |
-| Destroy | Wait until empty, remove the container and Komodo deployment, then delete server files and registration. |
+| Rebuild | Wait until empty, replace the instance through the host agent, preserving its server identity and port while advancing its generation. Verify container health, game status, map, and Plutonium revision. |
+| Stop | Wait until empty, stop the container, and verify that the host agent reports it stopped. Keep the instance for a later Start. |
+| Destroy | Wait until empty, remove the container and its files through the host agent, then retire the instance and registration. |

 Servers linked to a running or ready-to-promote beta cannot be destroyed or
 rebuilt. Promote or roll back the beta first, including paused or failed test
@@ -32,11 +32,10 @@ activity belong to Electric Cherry, and are not instance files.

 Rebuild reserves the original ID and port throughout teardown and creation, even
 when a lower lobby is available. It shares the existing creation queue and startup
-limit. A failed removal never proceeds to creation. Once Komodo deletion is
-confirmed, failed local cleanup offers **Resume rebuild cleanup**; creation cannot
-proceed until the old files are fully removed. An interrupted fresh startup can
-resume health verification, without repeating teardown or creation. Inspect other
-failures before making another request.
+limit. The agent records each operation durably, and the controller reconciles its
+receipt after reconnecting or restarting. An interrupted fresh startup can resume
+health verification without repeating the operation. Inspect failures before
+making another request.

 Create and Rebuild finish only after RCON reports a runtime revision at least as
 new as the current Plutonium production release. The worker checks the production
@@ -49,16 +48,15 @@ updater configuration and server logs before retrying. Telemetry can verify game
 status and map, but cannot verify the Plutonium revision.

 Start and Restart share the rolling restart limit with campaigns, betas, and
-maintenance. A successful API reply is followed by a recovery delay and actual
-container and game-status checks. Failed or ambiguous restart requests retain their recovery
+maintenance. A successful agent receipt is followed by actual container and game-status checks. Failed or ambiguous restart requests retain their recovery
 slot and are not automatically replayed. Stop requires an explicit stopped container state; an
-unreachable Komodo API is not evidence that a stop succeeded.
+unreachable host agent is not evidence that a stop succeeded.

 Disruptive operations require a fresh zero-player result from RCON or accepted
-telemetry, unless Komodo positively reports a stopped container. Fresh occupied
+telemetry, unless the host agent positively reports a stopped container. Fresh occupied
 telemetry vetoes a conflicting empty RCON reading. Reports must be received within
 90 seconds; recovery verification also excludes reports from before the submitted
-operation. Telemetry can verify the game's activity and selected map after Komodo
+operation. Telemetry can verify the game's activity and selected map after the host agent
 reports healthy. Runtime actions wait for conflicting
 campaign, beta, maintenance, or provisioning work. Duplicate pending requests for
 the same action return the existing operation; a different action returns a
@@ -78,8 +76,7 @@ after five minutes; unrelated recovery slots remain held. Drift campaigns follow
 their linked rebuild without requesting another restart. Their file evidence and
 history remain available after the instance is removed.

-Campaign changes queued before submission can share one restart. Changes arriving
-after submission require a new operation. A failed or interrupted request keeps
+Each campaign activation has its own durable operation and file selection. A failed or interrupted request keeps
 its original ID and never automatically replays. An explicit beta retry starts a
 new attempt, preserving prior history; paused or closed beta work cannot start a
 queued restart. Recovery of an already submitted restart continues even if its
@@ -177,7 +174,7 @@ heartbeat or completion-time promises.
  literal-text `query`, `limit` (default 25, maximum 100), an exclusive event-ID
  `cursor`, and an upper-ID `watermark`. Keep the returned watermark when loading
  more and use `nextCursor` until it is null. Begin without a watermark to include
-  newly arrived events. These reads do not probe Komodo or RCON.
+  newly arrived events. These reads do not contact host agents or probe RCON.

 Old `/deployments` pages redirect to their `/provisioning` equivalents, preserving
 query parameters. Collection launch links at `/provisioning/collections/:id` or
@@ -185,7 +182,6 @@ query parameters. Collection launch links at `/provisioning/collections/:id` or
 query parameters. `/api/deployments` remains compatible, including old campaign
 detail endpoints. No history or IDs are renamed or discarded.

-Start and Stop use Komodo's corresponding runtime commands. Rebuild uses full
-destruction and creation, not Komodo's redeploy command. See the
-[Komodo execute API](https://docs.rs/komodo_client/latest/komodo_client/api/execute/index.html)
-for the underlying container operations.
+Start and Stop use durable `instance.start` and `instance.stop` host-agent
+operations. Rebuild uses `instance.rebuild` and replaces the instance files.
+See [native host management](native-hosts.md) for command delivery and recovery.
@@ -1,6 +1,6 @@
 # Reducing idle CPU use in T5 and T6 Zombies servers

-Research date: September 19, 2026. Scope: Plutonium T5 Zombies (`t5sp`) and T6 Zombies (`t6zm`), with particular attention to Electric Cherry's Plutainer/Docker/Komodo deployment.
+Research date: September 19, 2026. Scope: Plutonium T5 Zombies (`t5sp`) and T6 Zombies (`t6zm`), with particular attention to Electric Cherry's Plutainer/Docker host deployment.

 **Recommendation:** first benchmark Wine's NTSYNC support and audit the effective runtime configuration. If every lobby must remain immediately available, trial CPU limits that apply only while a server is confirmed empty. If spare lobbies can be unavailable temporarily, keep a smaller pool running and start additional servers when demand rises. Stopping unused instances offers the most predictable reduction in their CPU consumption.

@@ -60,7 +60,7 @@ The policy should handle several cases explicitly:
 - Give startup, map changes, and recovery the normal CPU budget.
 - Restore that budget on a connection attempt if a reliable signal exists. Otherwise, use a conservative idle limit that still supports joining and a detection interval demonstrated to be fast enough.
 - Restore the active budget when occupancy becomes unknown. A separate watchdog or expiring idle-policy lease should restore it if the controller stops running.
- Coordinate quota ownership with Komodo so container recreation does not silently lose the policy or reinstate an obsolete limit.
+- Coordinate quota ownership with the host agent so container recreation does not silently lose the policy or reinstate an obsolete limit.

 Electric Cherry's current 30-second status cycle can leave a newly joined player under the idle cap until the next successful poll. An aggressive cap should not depend on that cycle alone. Quotas can also allow brief bursts followed by throttling within their scheduling period, so evaluate join time and responsiveness, not just average CPU. The kernel exposes `nr_throttled` and `throttled_usec` in cgroup v2 `cpu.stat` to help identify this behavior. [Linux CPU controller](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu).

@@ -44,7 +44,7 @@ Activity and notifications have been removed. Copy results appear next to their
 - Promotion uses `GET /api/betas/:id/promote` to review impact and `POST` with the returned token as `previewToken` to apply it. A changed plan returns HTTP 409 and needs renewed review.
 - Reconciliation uses `POST /api/drift` with `dryRun: true` and optional `servers`, then executes with the returned `expectedFingerprint`. Omitted servers explicitly mean the entire fleet. A stale review returns HTTP 409; changes detected during application are reported as skipped files.
 - `POST /api/deployments/:id/cancel` cancels remaining provisioning work, including restarts and health checks. It does not undo changes already made.
- System status exposes no credential values. Its optional connection check verifies Komodo network reachability, not credential validity. Remote game-file volumes remain explicitly unverified.
+- System status exposes no credential values. It reports host-agent readiness and verified game-file caches.
 - Drafts contain nonsecret beta setup. Launch keys are not stored in browser drafts.
 - Log search/export covers the loaded tail, not an entire remote file or a full diagnostic archive.

@@ -56,4 +56,4 @@ Activity and notifications have been removed. Copy results appear next to their
 - Local browser walkthroughs use an isolated database and synthetic populated API fixtures, with production integration credentials disabled. Fleet, server diagnostics, logs, Settings, Leaderboard, libraries, beta setup/progress, launches, keys, and reconciliation are exercised at desktop and phone widths.
 - Browser checks include the loading button accessibility tree, clickable native labels, URL/Back restoration, retained previews/drafts, selected-tab visibility, pending import ownership, and horizontal overflow. These are targeted checks, not a claim of complete accessibility conformance.

-Live Komodo provisioning, real game-server restarts and removals were not executed during verification. Their boundary behavior is covered by regression tests and simulated UI flows.
+Live host provisioning, real game-server restarts and removals were not executed during verification. Their boundary behavior is covered by regression tests and simulated UI flows.
@@ -10,7 +10,7 @@ Electric Cherry has a coherent visual foundation and unusually capable operation

 This review combines source inspection with a local browser walkthrough. It covers Fleet, server details and diagnostics, Scripts, Configs, Plugins, beta creation and progress, server deployments, blueprints, campaigns, keys, and Leaderboard. Existing design notes and workflow documentation informed the review.

-Browser exploration used an isolated database with Komodo/RCON credentials disabled. Nine top-level pages were examined at 1440×1000 and 390×844. Populated Fleet, server details, and beta scenarios used synthetic API responses. Beta setup, blueprint validation, and the browser accessibility tree were also exercised. Navigation was checked around its responsive breakpoint. Production integrations, actual restarts, and destructive actions were not executed.
+Browser exploration used an isolated database with external connections disabled. Nine top-level pages were examined at 1440×1000 and 390×844. Populated Fleet, server details, and beta scenarios used synthetic API responses. Beta setup, blueprint validation, and the browser accessibility tree were also exercised. Navigation was checked around its responsive breakpoint. Production integrations, actual restarts, and destructive actions were not executed.

 **Evidence labels:** **Rendered** means reproduced in the local browser, sometimes with an explicitly synthetic populated state. **Source** means verified in UI/API implementation. **Opportunity** means a proposed capability or design direction, rather than a defect. Recommendations are expert judgments; they are not findings from end-user interviews or a complete accessibility conformance audit.

@@ -198,7 +198,7 @@ Sources: [Scripts](../src/routes/scripts/+page.svelte), lines 13–23; [Plugins]

 Settings currently leads only to server keys. Integration information is available deep in server diagnostics; the empty Fleet encourages creation even when external services are not configured. The first-run user needs a short path from prerequisites to a functioning lobby.

-Add a read-only System status page and a contextual first-run checklist: Komodo configured/reachable, RCON configured, game files available where verifiable, keys available, and first server ready. Link directly to relevant setup instructions. Distinguish **not configured**, **unreachable**, and **server offline**. Avoid repeatedly displaying the full checklist after setup succeeds.
+Add a read-only System status page and a contextual first-run checklist: host agents enrolled/connected, local RCON responding, verified game files available, keys available, and first server ready. Link directly to relevant setup instructions. Distinguish **not configured**, **unreachable**, and **server offline**. Avoid repeatedly displaying the full checklist after setup succeeds.

 Acceptance: someone with an empty installation can identify the next missing prerequisite without opening an unrelated server or reading the full README.

@@ -306,7 +306,7 @@ The visual language is coherent, but small controls, nested cards, repeated sect

 Use a dedicated editable-control boundary token, reserve strong surfaces for actionable sections, and reduce repeated title/subtitle layers in builders. Keep body text and important form instructions comfortably readable; use compact metadata selectively. Add descriptive route titles, especially server name and lobby number.

-Use “server” consistently for the managed unit; introduce “lobby” as its player-facing meaning. Prefer **Preview missing servers** over “Conform fleet.” Use “Files” consistently for scripts/configs/plugins where a collective noun is needed. Keep raw IDs, RCON, Komodo, and hashes accessible in technical details. Expose the existing keyboard shortcuts through a small help panel.
+Use “server” consistently for the managed unit; introduce “lobby” as its player-facing meaning. Prefer **Preview missing servers** over “Conform fleet.” Use “Files” consistently for scripts/configs/plugins where a collective noun is needed. Keep raw IDs, RCON, host-agent status, and hashes accessible in technical details. Expose the existing keyboard shortcuts through a small help panel.

 Acceptance: primary tasks are identifiable by placement and wording, and multiple browser tabs are distinguishable. Contrast should be measured before claiming a conformance result.